University of Groningen Information-seeking by general ... · Samenvatting 109 List of publications 113. Preface ... 4 Verhoeven AAH. Wetenschappelijk onderzoek door de huisarts:

Information-Seeking by General Practitioners

© 1999 Anita A.H. VerhoevenAll rights reserved. Published in 1999Printed in the Netherlands by van Denderen BV, GroningenTypeface CG Times 12 pt

Cover: Early Library in: Franciscus Philippus Florinus [pseud.]. Oeconomus prudens et legalis Nürnberg,1722. p. 125 18th c. (modified, 1999). Reproduced with permission from the Herzog August BibliothekWolfenbüttel, Wolfenbüttel, Germany.

Financial support for the printing of this thesis has been kindly provided by the Stichting Het Scholten-Cordes Fonds.

NUGI 741ISBN 90-367-1078-2

RIJKSUNIVERSITEIT GRONINGEN

Information-Seeking by General Practitioners

PROEFSCHRIFT

ter verkrijging van het doctoraat in deMedische Wetenschappen

aan de Rijksuniversiteit Groningenop gezag van de

Rector Magnificus, dr. D.F.J. Bosscher,in het openbaar te verdedigen op

woensdag 10 november 1999om 16.00 uur

door

Anna Ada Henrica Verhoeven

geboren op 22 november 1958te Tilburg

Promotor: Prof. dr. B. Meyboom-de Jong

Aan mijn ouders

Huisartsen hoeven niet alles te weten,als zij maar weten waar de informatie te vinden is

Contents

Preface

Dankwoord (Acknowledgements in Dutch)

About the author

About the co-authors

Introduction

1 Information-seeking by general practitioners: an overview 1

2 Use of information sources by family physicians: a literature survey 13

3 Management of bibliographic information by Dutch researchers ingeneral practice 23

4 Information use and needs: a survey among Dutch general practitioners 43

5 A model for retrieval quality of search queries in threemedical bibliographic sources 63

6 Which literature retrieval method is most effective for generalpractitioners? 75

7 General discussion 89

Alphabetical list of references 99

Summary 105

Samenvatting 109

List of publications 113

Preface

A general practitioner by training, I started to work as an information specialist at theUniversity Library and Medical School Library of the University of Groningen in 1988.Since then I have been busy providing medical information to clinicians and researchers.Of course I had not forgotten the general practitioners I used to work with, and Iwondered how I could contribute to their information needs. My interest in literatureresearch methodology was triggered when I read two articles in the Dutch professionaljournal for librarians on the recognition of literature research as an academic discipline.1,2

These articles generated many new ideas and questions: how do general practitionersgather their literature; what do they need in their specific circumstances, such as patientcare and keeping up to date; and do they use an explicit literature research method?I dealt with some of these ideas and questions in two reports that I wrote during mypostdoctoral training in Library and Information Studies at the University of Amsterdam.3,4

But only after the newly appointed professor in general practice, Betty Meyboom-de Jong,examined these reports by chance, did I get the opportunity to fully implement my ideasduring a 9-years lasting PhD research.During this research it became clear that Boerma and Arnold were not the only onestrying to attract more attention for literature research. In 1990, the editor of the Dutchjournal Huisarts en Wetenschap (General Practitioner and Research) required an explicitmethodology for reviews in this journal of the Dutch College of General Practitioners.5 Asa result of the recognition of the importance of evidence-based medicine,6 the systematicreview has become an important issue in the international literature as well.7-9

While I started my research with the goal of stressing the importance of an explicitliterature methodology, I have ended my research with a much broader goal in which myinitial question was incorporated: to study the information-seeking behaviour of generalpractitioners.

By far the largest problem that we faced in our study was how to fund it. The responsiblemedical as well as library and informatics professionals all acknowledged its importance.The largest governmental research funding organisation (NWO) even awarded our researchproposal a high mark (an eight). However, our study on general practitioners’ expertisefell between two stools: the world of medicine ("no patients involved"), and the world ofinformatics ("no computer programmes developed") and libraries ("no money"). In short,no professional organisation accompanied its approval of our research proposal withfunding. Finally, after having written 13 letters for research grants to professional,research, and governmental organisations, we were very happy to receive some financialcontributions from three organisations that were associated with neither medical nor libraryorganisations. In this way, I could be exempted from my daily library activities for oneday a week for three years to do research. Even so, by far the largest part of the studieswas performed unpaid and out of working hours.Another problem we ran into was that of finding appropriate classrooms with enoughcomputers with Internet connections. Also there was ineffective communication betweenmyself as a researcher and general practitioner, and the computer professionals. The

computer professionals seemed to speak another language, which was illustrated by thefollowing answer to our request to use their computers: "No, because we lack NCSApacket drivers for our Fast Aetherlink III network cards with PCI bus".We conclude that the largest barrier to research in Library and Information Sciences in theNetherlands is an ineffective research structure, and therefore a lack of funds (andresearchers). In contrast to many foreign universities, Dutch universities do not have aFaculty for Library and Information Studies, although the University of Amsterdam has asmall Department for Documentary Information as part of the Faculty of Arts, with one,recently appointed, full-time professor. Existing Medical Informatics Departments atuniversities are focused on technical computer science, not on user studies. For thisreason, regularly paid researchers in Library and Information Studies are unknown in theNetherlands, as is academic teaching of Library and Information Sciences, especially witha medical focus. In a medical world which relies more and more on electronic medicalinformation these are fundamental needs.

REFERENCES

1 Boerma EJ. Naar een methodologie literatuuronderzoek: enige wetenschapstheoretische beschouwingen.[Going to a methodology for literature research: some theoretical, scientific considerations. In Dutch].Open 1989;21:2-7.

2 Arnold CA. Literatuuronderzoek onder de loep: van het vermijden van valkuilen naar een methodologie.[Literature research scrutinized: from avoiding pitfalls to a methodology. In Dutch]. Open 1990;22:411-5.

3 Verhoeven AAH. Literatuuronderzoek door huisartsen: methoden van literatuurverzamelen voorwetenschappelijk onderzoek door huisartsen - onderzoeksvoorstel -. [Searching the literature: methodsof literature gathering for research purposes by general practitioners - a research proposal -. In Dutch].Report made for the Postdoctoral Training in Library and Information Studies. University ofAmsterdam; 1991.

4 Verhoeven AAH. Wetenschappelijk onderzoek door de huisarts: een geannoteerde bibliografie overalgemene aspecten van wetenschapsbeoefening voor huisartsen, en het praktisch verrichten vanwetenschappelijk onderzoek in de huisartspraktijk, samengesteld uit publikaties verschenen na 1970.[Research by the general practitioner: an annotated bibliography on general aspects of performingresearch and searching the literature by general practitioners; compiled of publications published after1970. In Dutch]. Report made for the Postdoctoral Training in Library and Information Studies.University of Amsterdam; 1990.

5 Meyman FJ. Het literatuuroverzicht: waardevol door systematiek en structuur. [The literature review:useful because of its methodology and structure. In Dutch]. Huisarts Wet 1990;33:2-3.

6 Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching thepractice of medicine. JAMA 1992;268:2420-5.

7 Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987;106:485-8.8 Chalmers I, Altman DG, editors. Systematic Reviews. London: BMJ; 1995.9 Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence

[editorial]. Ann Intern Med 1997;126:389-91.

Dankwoord (Acknowledgements in Dutch)

Van te voren heb ik me niet gerealiseerd dat het doen van onderzoek zo inspirerendkon zijn. Een aantal mensen heeft substantieel aan deze ervaring bijgedragen.

Mijn promotor, professor dr. Betty Meyboom-de Jong, was degene die met het ideekwam om op het gebied van informatiewetenschappen onderzoek te gaan doen en zijschetste voor mij de randvoorwaarden. Beste Betty, jij was vooral degene die mijenthousiast kon maken. Door jouw energie en werklust werd ik enorm gestimuleerd. Jewist op een efficiënte wijze heel wat voor mij te organiseren, zoals denascholingscursussen, de pilots voor de verschillende studies en de verwerking vanstudieresultaten door studenten. Je bracht me in contact met de juiste mensen. Daarbijkon ik op je aan als je iets beloofde. Je beoordeelde de manuscripten op eenopbouwende manier, waarbij je steeds de link legde naar de medische praktijk. Ik wilje hartelijk danken voor je vertrouwen in mij bij dit voor jou - qua onderwerp - nietalledaagse onderzoek.

Mijn tweede begeleider vanaf het begin, negen jaar geleden, was drs. Edzard Boerma.Beste Edzard, door een artikel van jou kwam ik op het idee mijn promotieonderzoekover literatuuronderzoek te doen. Je zienswijze vanuit de sociale wetenschappen wasverfrissend en deed mij op een nieuwe wijze naar het onderzoek kijken. Je kennis vande bibliotheekwereld kwam mij zeer goed van pas. Hartelijk dank voor het meedenkenen eindeloos corrigeren van manuscripten. Ik ben erg blij dat je er mee instemde omparanimf te zijn.

Mijn collega vakreferent van de Centrale Medische Bibliotheek van het AMC inAmsterdam, drs. Carla van Noort, heeft een aantal jaren belangeloos haar vrije tijdgestoken in het opzetten, verwerken en beoordelen van onze postenquête. Beste Carla,hartelijk dank voor al het werk dat je nauwgezet voor mij gedaan hebt. Je hebthiervoor vaak van Amsterdam naar Groningen moeten reizen. De postenquête heeft jeheel wat nachtelijke uurtjes gekost. Ik ben je dankbaar voor je betrokkenheid en inzet.Drs. Peter Boendermaker, huisarts, mede-auteur en paranimf. Beste Peter, ons verbindtmeer dan research alleen. Hartelijk dank voor je meedenken en meeschrijven, metname voor het artikel over het model als meetinstrument om kwaliteit vanliteratuurverwijzingen te bepalen. Maar eigenlijk was je steeds een klankbord en datheeft een (onderzoeks)mens nodig.Drs. Henk Bosveld was de methodoloog die de eindeloze rij van gegevens verwerktein SPSS en die er interpreteerbare informatie uit kon laten komen. Beste Henk, je hebtme vaak op je kamer ontvangen en geduldig naar me geluisterd wanneer ik het weernét iets anders wilde. Dank je voor meedenken in mijn project.

Gedurende enkele jaren heeft collega-huisarts dr. Jaap Buis mijn taken in de CentraleMedische Bibliotheek voor een dag in de week waargenomen, zodat ik meer tijd kon

besteden aan mijn onderzoek. Beste Jaap, het waren inspirerende uurtjes die wedeelden over ons beider promotieonderzoek. Bedankt voor je bereidheid je te willenverdiepen in online literatuuronderzoek.

Ik wil de Bibliothecaris van de Rijksuniversiteit Groningen, Dr. Alex Klugkist hartelijkdanken voor de mentale en soms ook materiële ondersteuning die ik de afgelopen jarenheb mogen ontvangen. Beste Alex, ik voel me vereerd dat je plaats wilt nemen in decorona en één van mijn opponenten wilt zijn.Dinie During-Buis, hoofd van de Centrale Medische Bibliotheek, wil ik graagdankzeggen voor haar medewerking bij onder andere het gebruik van de faciliteitenvan de CMB tijdens mijn onderzoek.

Twee studenten hebben in een beginfase van het onderzoek voorwerk gedaan. HeleenDouma, studente aan de opleiding Bibliotheek- en Documentaire Informatie (hethuidige IDM), heeft een eerste opzet gemaakt door een bibliografie samen te stellenover informatiebronnen voor huisartsen. Anne Wassenaar, 6de jaars medisch student,heeft zijn wetenschappelijke stage besteed aan de invoer en verwerking van de eerstegegevens voor de vergelijkende studie. Beiden hartelijk dank.

Ineke Brink, beheerder bij de Disciplinegroep Huisartsgeneeskunde, ben ik dankverschuldigd voor haar hulp bij het organiseren van de huisartscursussen en veleandere hand- en spandiensten.

Zonder de huisartsen die deelnamen aan de interviews, de literatuurcursussen en hetinvullen van de vragenlijsten zou deze dissertatie niet tot stand gekomen zijn. Dankvoor de investering van uw kostbare tijd.

De leden van de beoordelingscommissie, de hoogleraren prof. drs. J.S. MackenzieOwen (hoogleraar Documentaire Informatiewetenschap aan de Universiteit vanAmsterdam), prof. dr. ir. J. Nerbonne (hoogleraar Alfa-informatica aan deRijksuniversiteit Groningen), en prof. dr. E. Schadé (hoogleraar Huisartsgeneeskundeaan de Universiteit van Amsterdam) wil ik hartelijk danken voor de snelle beoordelingvan mijn proefschrift.Bovendien ben ik professor Schadé dank verschuldigd voor zijn hulp bij het verkrijgenvan voldoende huisartscursisten voor de vergelijkende studie.

Mijn collega’s van zowel de Universiteitsbibliotheek, de Centrale MedischeBibliotheek, als de collega-huisartsen van de Disciplinegroep Huisartsgeneeskunde wilik hartelijk danken voor hun medeleven met de voortgang van mijn proefschrift.

Als laatsten wil ik mijn ouders bedanken. Zij stonden aan de wieg van mijnwetenschappelijke carrière. Dankzij wat zij mij meegaven in mijn opvoeding en devrijheid die zij mij boden in mijn studie- en beroepskeuze ben ik aangekomen op ditpunt. Lieve vader en moeder, jullie onvoorwaardelijke liefde en vertrouwen in mijhebben mij heel goed gedaan.

About the author

Born as the middle daughter of three in the city of Tilburg, the Netherlands, on 22November 1958, Anita Verhoeven entered the Medical School of the University ofGroningen in 1977. In 1984, she completed her medical studies with a clerkship in thecity of Zwolle. There she was appointed as Medical Examiner at the Blood Bank.Later, she took up a job as teacher at the School for Nursing, and in the eveningstaught First Aid to volunteers. After hours, she took a one-year class in medicalhistory. This was an advantage when she applied for a job as a research assistant inMedical History. For this two-years’ job, she was employed at the Medical School ofthe Free University in Amsterdam from 1985 to 1986.In 1986, she started her vocational training in general practice at the University ofGroningen, which included a half year practising in rural Nieuweschans with an in-house pharmacy and with home baby delivery, and a half year in urban Zwolle in aprimary health care centre.In 1987 and 1988 she worked as a locum in several general practices in the Provinceof Overijssel. Because there was a shortage of available medical practices, sheaccepted a job as medical librarian and information specialist for the Health Sciencesat the University of Groningen in 1988. In 1991, she finished the postgraduateprogramme for Library and Information Studies at the University of Amsterdam.Anita’s areas of expertise and research include information services for primary care,and evidence-based medicine. Her key responsibilities at the library are to fosterinstructional programs in the medical sciences, including user education, curriculum-based teaching and library instruction. Furthermore, she is responsible for referenceand research support, collection development, and subject indexing.In 1990, she had the opportunity to start her PhD research on the information needs ofgeneral practitioners, which she completed in 1999 with this doctoral thesis.She is a member of the Dutch Association of Librarians (the Biomedical Chapter, andthe Informatics Chapter); the European Association of Health Information Libraries(EAHIL); and the American Medical Library Association (MLA).In 1998, she was awarded the Cunningham Memorial International Fellowship by theAmerican Medical Library Association, which gave her the opportunity to visit 33medical libraries throughout the USA and Canada for 4 months. In 1999, she agreed tobe an adviser for library services at the Medical University in Ulaanbaatar, Mongolia.In her private life, she is a member of the local Dutch Reformed Church Council. Asan elder she takes responsibilities for the weekly services, and pays pastoral visits tochurch members. She writes by hand to friends all over the world, and seizes everyopportunity to visit them. She loves art and literature, especially architecture,expressionist paintings, English novels, and books on history and foreign cultures.

About the co-authors

Peter M. Boendermaker, MD, who was born in 1952, was trained as a generalpractitioner. For 14 years he worked in a solo practice in Coevorden, where he alsoserved as a general practitioner trainer. In 1993, he switched to his current job asEducational Coordinator in the Unit of Vocational Training in the Department ofGeneral Practice at the University of Groningen. In this position, he has published onseveral aspects of medical education. In 1997, he started his PhD research on thecharacteristics of the good general practitioner trainer. After working hours he is anenthusiastic basketball player. But, most important, he likes to spend as much time aspossible with his wife and four children.

Edzard J. Boerma, MA, obtained his degree in history in 1973. Since then, he hasworked as Associated Professor at the Department of Education of the University ofGroningen. He has published widely in the fields of educational documentation andinformation, user education, bibliographic instruction and methodology of literatureresearch, and history and peace education, especially content analysis of textbooks forprimary and secondary education. He is a member of the Socrates-Erasmus network onComparative Education. He is interested in visiting historic buildings, tending hisgarden, and restoring his own historic house.

Henk E.P. Bosveld, MA, who was born in 1953, studied psychology and obtained hisdegree in 1988 at the University of Groningen. While still a student, he also worked asa research assistant. His focus was on scale theory, research methods and statistics.After graduation he was involved in research on education, criminology, and healthcare. In 1995 he was appointed as methodologist and statistician in the Unit ofVocational Training in the Department of General Practice of the University ofGroningen, where he is responsible for teaching trainees and supporting researchers ingeneral practice. For recreation, he likes to play tennis, bridge and chess. For severalyears he has been the passionate chairman of the Northern Chess Federation.

Betty Meyboom-de Jong, MD, PhD, who was born in 1939, has been a generalpractitioner since 1965. After she and her husband, also a general practitioner, hadworked in northern Norway for four years, they established a general practice in arural area in the Dutch Province of Friesland. She has always been interested ineducation and research, and was appointed as a group facilitator in the Unit ofVocational Training in the Department of General Practice at the University ofGroningen in 1974. Her thesis on morbidity and functional status in the elderly wasdefended in 1989. She was appointed Professor in General Practice and Head of theDepartment in 1990. Her professional interests are in developing evidence-basedgeneral practice, studying chronic diseases in the elderly, and rationalpharmacotherapy. She is a member of the advisory board of the Dutch Ministry ofHealth.

In her private life she adores reading, in particular English detective novels, and goingfast without motorised power: skating, skiing, cycling and sailing. She loves travellingaround the world and meeting colleagues from many countries.

Carla P. van Noort, MD, who was born in 1964, received her degree from the FreeUniversity in Amsterdam in 1990. She worked as a medical officer in occupationalhealth care for several years. After she had finished a postgraduate programme forLibrary and Information Studies at the University of Amsterdam in 1994, she wasappointed in 1996 as medical librarian at the Medical Library of the AcademicMedical Center in Amsterdam. In addition to her activities as clinical librarian, she isresponsible for user education, collection development, and subject indexing. Whenwork is done, she loves to take part in athletics and orienteering. She enjoys going tothe movies with friends.

Introduction

Information-seeking by general practitioners has been influenced by the informationexplosion, and developments in information technology that could have affected theirinformation needs. To identify and meet these needs, we performed several studiesbetween 1992 and 1997, which are described in this thesis. The importance of ourstudy was underlined by the publication in 1992 of a new paradigm of medicine by theEvidence-Based Medicine Working Group in which research results published in theliterature served as a scientific base for evidence-based decision-making.

To obtain an accurate and complete picture of the information needs of generalpractitioners, a multi-method approach to data collection and analysis is essential.Therefore, we used a variety of research methods: literature studies (chapters 1 and 2),in-depth interviews (chapter 3), a postal questionnaire (chapter 4), a model for retrievalquality of search queries in medical bibliographic sources (chapter 5), and finally acomparative study of three literature retrieval methods (chapter 6).

We will discuss the following three research questions:1. Information sources: Which information sources do general practitioners use?Among the personal, printed and electronic information sources general practitionersseem to prefer specific sources. In chapter 2 we describe which information sourcesgeneral practitioners use as we found in the international literature. In our interviews(chapter 3) we obtained data about the information sources Dutch general practitionerresearchers use. Through a national postal questionnaire (chapter 4) we obtained dataabout the information sources Dutch practising general practitioners use.

2. Information barriers: What problems do general practitioners encounter whenseeking information, and which factors influence the use of information sources?We discuss the problems general practitioners encounter when seeking medicalinformation for patient care (chapter 4) and research (chapter 3).

3. Comparison of three literature retrieval methods: Are electronic information sourcesmore effective for general practitioners than the printed ones?In a comparative study, we describe the results of three groups of general practitionerswho use three literature retrieval methods (chapter 6). The development of themeasurement instrument is described in chapter 5.

The five studies described in this book (chapters 2 to 6) differ in depth and aim, andin the nationality and work roles of the doctors. The nationality of the studied generalpractitioners varied in the literature review described in chapter 2 (American, English,Belgian and Swedish); the interview study (chapter 3), the questionnaire study (chapter4), and the comparative study (chapters 5 and 6) all involved only Dutch generalpractitioners. We studied practising doctors with work roles in patient care (all

studies), as well as practising doctors with work roles in research in general practice(review, interviews). We were looking for general trends in information-seeking(review, questionnaire), but at some point we were more interested in details oninformation-seeking (interviews, comparative study). We used this detailed informationfor developing the extensive questionnaire for the postal survey. Finally, apart fromevaluating new information retrieval systems, the courses on information retrieval inour comparative study also served as a contribution to the continuing education ofgeneral practitioners.

This thesis is composed of five studies submitted to, and partly published in,international journals. This implies the use of the British English in one chapter, andAmerican English spelling in another. An example is the British term generalpractitioner, which in American English is family physician. In the chapters whichwere not written for publication in a journal, we observed the British English spelling.References are in the Vancouver style, as published on the JAMA Websitehttp://www.ama-assn.org/public/journals/jama, updated October 1998, and visited onFebruary 22, 1999.After we had submitted the journal articles, more literature about these topics has beenpublished. The updated literature has been incorporated in the General Discussion.

Information-seeking by general practitioners will be of interest to general practitioners,planners for continuing medical education for general practitioners, educators atMedical Schools and Vocational Training Units in Departments of General Practice,and medical librarians who are involved in training and teaching library skills andevidence-based medicine.

In summary, chapter 1 discusses information-seeking by general practitioners ingeneral. The specific information sources they use are discussed in chapter 2. Generalpractitioners who are involved in research use the literature heavily; the ways in whichthey handle it are described in chapter 3. Chapter 4 deals very specifically with asample of Dutch general practitioners - their use of information sources and thebarriers they experience. Next, two chapters present details of a comparative study ofthree methods of information retrieval: chapter 5 explains the development of themeasuring instrument used for the work of chapter 6, which describes the design andresults of the study. Finally, we discuss the outcomes and the limits of the presentedstudies, and present recommendations for general practitioners, educators and librariansin chapter 7.

1 Information-seeking by general practitioners: an overview

Research is the process of going up alleys to see if they are blindMarston Bates, 1967

ABSTRACT

General practitioners’ roles and related tasks undertaken in the course of daily practice prompt specificinformation needs, which in turn give rise to an information-seeking process. The environment in whichthis takes place is characterized by an information overload as a result of the information explosion ofthe last decades. In this introduction, we will first discuss the influence of the information explosion ontasks of general practitioners. Then we will discuss some aspects of information needs and information-seeking on the basis of the model of Leckie. Finally, we will describe some characteristics of the groupof Dutch general practitioners who took part in our studies.

INFORMATION EXPLOSION

We experience the world around us as random, and we use information to reduce oursense of disorder so that we can cope with the randomness. However, in our society,and especially in the medical sciences, the rapidly increasing volume of informationinfluences the way physicians seek information. It may be that the volume of medicalliterature has become unmanageable.1,2 We will look at the impact of this problem onretrieving and selecting literature; on the way doctors practise medicine, and onmedical education.

To illustrate the size of the information explosion, we have measured the increase ofinformation in three ways: the number of biomedical databases, the number ofbiomedical journals, and the number of articles on general practice indexed inMedline, the medical database most used by physicians.Figure 1 shows that the number of biomedical databases had increased from 433 in1988 to 1154 in 1997, a growth of 267% in 10 years.3

The increase in the number of biomedical journal titles since 1870 has beenexponential, with a doubling time averaging 19 years (figure 2).4 An estimated twomillion articles are published in the biomedical literature each year.5 For a physician toread everything of possible biomedical relevance, it would be necessary to read about6,000 articles a day.6 Although the volume of medical information is increasing, theratio of researchers and publications has not changed.7 In part, the informationexplosion can be attributed to more researchers who publish.

1

Evidence-based medicine is an approach to clinical practice in which clinicians basetheir decisions and actions on appropriate evidence from health care research literature,as well as on their clinical expertise and the circumstances of the patient. Literatureresearch needs to satisfy the same criteria as research in general, including validity,reproducibility and verifiability.11 This need for scientifically based literature searchingbecame obvious in the development of the systematic review.12 Whereas a narrativereview is based on the author’s authority, systematic reviews involve the application ofscientific strategies, in ways that limit bias.13 For example, it is known that studiesyielding statistically significant differences between groups are about two-and-halftimes more likely to be reported than those in which no such differences are found.14

To minimise publication bias, both published and unpublished studies need to beincluded in a systematic review, and the criteria for inclusion and exclusion must beaccounted for.13 Other features that need to be included in a systematic review are aclearly stated question; the sources used and the search performed to detect relevantstudies; the appraisal of studies; the synthesis of studies; and, usually, evidence-basedinferences. Systematic reviews can help practitioners to keep abreast of the medicalliterature by summarizing large bodies of evidence and helping to explain differencesamong studies on the same question.13 The Cochrane Collaboration collects and writessystematic reviews, and publishes them in an electronic journal. This journal, theCochrane Library, includes regularly updated full-text systematic reviews on theeffects of health care with critical appraisal by experts.15

The information explosion has its impact on the way doctors practise medicine.Owing to the information explosion, physicians find it hard to keep up to date withnew developments.2 Many solutions have been proposed: computerization; effectiveinstruction in use of information sources; greater library accessibility; a change injournal articles’ content and structure; development of guidelines based on soundresearch;16 and abstracts of critically appraised journal articles. Supporters of evidence-based medicine focus on evidence as a basis for patient care decisions.17-20 Thisevidence may be obtained from several sources, such as the patient’s history orexamination, laboratory data, and the medical literature. Evidence-based medicine isbased on five principles,21 namely: formulating the question; searching the literaturefor relevant information; selecting the articles; appraising the evidence for validity andusefulness; and, applying the evidence in everyday practice. Critically appraisedarticles are found in the journals Evidence-based Medicine and ACP Journal Club(together published as the electronic version Best Evidence22), and in the Journal ofFamily Practice, named POEMS (Patient-Oriented Evidence that MatterS).23

Finally, the information explosion has its impact on medical undergraduate andpostgraduate education.The doubling time of the biomedical knowledge base is currently about 19 years,4

meaning that medical knowledge will increase fourfold during a professional life time.Consequently, medical students cannot learn everything they need for theirprofessional career during their undergraduate days. In the next century, society willdemand doctors with different competencies and skills. For example, doctors will beexpected to cope efficiently with the information overload, to take advantage of new

4

Work roles and tasksGeneral practitioners have several roles and related tasks from which their informationneeds originate: service provider, learner, researcher, educator, andadministrator/manager. General practitioners spend the majority of their time in therole of direct service provider. The tasks associated with patient care create theirgreatest needs for information - information mainly on treatment and diagnosis.2 Thegeneral practitioner also has a role of learner or student, within which the specifictasks are professional reading, or attending conferences and meetings to stay abreastwith new developments in health care. In continuing medical education, Welsh generalpractitioners considered that the most valuable educational activities occurred withinthe practice, the most valued being contacts with partners.26 Tasks associated with therole of researcher involve writing publications and speaking at conferences. The roleof researcher needs the use of many and complex information sources, includingelectronic sources.27 Tasks associated with the educator role include planning andcurriculum development. In this role the general practitioner needs information toteach medical students and general practice trainees. Finally, the general practitionermust also be an administrator and manager of his own practice.

Characteristics of information needsThe characteristics of the information needs of general practitioners are mainlydetermined by work roles and tasks.We define information needs as the recognition of the existence of uncertainties28.These uncertainties are not arbitrary, but relevant to practical decisions. The variablesthat influence the information needs of general practitioners include individualdemographics (age, career stage, geographic location), context of the situation thatprompts the information need, frequency of need, predictability (expected orunexpected) and complexity.25 The perceived information needs of general practitionersgive rise to information-seeking behaviour. Connelly distinguished two forms ofinformation-seeking: information-seeking behaviour which fulfilled an immediate need,and knowledge-gathering behaviour which added to the practice knowledge base forfuture use (continuing medical education).29 In addition to immediate needs and futureneeds, some researchers make a distinction between levels of perception. In this view,information demands (or, wants) were articulated, conscious needs,30 and unrecognizedneeds were needs that might not be expressed as grammatical questions, or even notbe verbalised at all.31 In chapter 3 and 4 we discuss general practitioners’ recognizedneeds, including pursued and satisfied needs.General practitioners use five types of information for patient care: patient data,population statistics, medical knowledge, logistic information, and social influences.32

Patient data refers to information about a specific person, and includes items of apatient’s history, observations from physical examination, and results of diagnostictesting. Population statistics refer to aggregated data about populations of patients,such as prevalence. Medical knowledge refers to general information known abouthealth and disease, the body of scientific knowledge of medicine. This is the medicalliterature, generalised by scientists, consisting of the primary, secondary and tertiaryliterature. Primary literature is defined as the original reports of scientific findings;secondary literature as the bibliographic indexes to the primary literature; and tertiary

6

literature as the vast summary of the primary literature, from review articles totextbooks and, more recently, hypertexts and practice guidelines.33 Logistic informationrefers to local knowledge about how to get the job done, often specific to a practicesetting or payment mechanism. An example of this type of information is thefollowing question: "Which forms must be completed to obtain a needed service?".Finally, social influences refers to knowledge about the expectations and beliefs ofothers, especially colleagues.The main type of information being considered in our studies published in this thesisis medical knowledge. It may exist as original research and systematic overviews inthe medical literature. It may consist of the classic descriptions of diseasepathophysiology, diagnosis, and treatment found in standard textbooks. It may take theform of informal experience accumulated by practitioners.32

Sources of informationThe first factor that affects information-seeking is the information sources used by thegeneral practitioner. Those used widely by general practitioners are human basedinformation sources i.e. personal communication, and private books.34-37 Electronicinformation sources were used little. Problems with using electronic sources on theWorld Wide Web in primary care are a lack of time and an efficient use of a newvocabulary.38 It seems important that users need to be involved in the development ofnew electronic information systems.

Awareness of informationThe second factor is the awareness of information. Direct or indirect knowledge ofvarious information sources and perceptions about the process, or about theinformation retrieved, play a crucial role in the overall information-seeking process.Familiarity, timeliness, cost, quality and relevance of the information are alsoconsidered as factors influencing information-seeking.25

OutcomesThe third factor is the outcomes of the information-seeking. Outcomes are the resultsof the information-seeking process. The optimal result is that the information need ismet. If the need is not satisfied, further information-seeking is required, and throughfeedback the definition of the information need may be sharpened or changed. It isalso possible that an outcome from one task associated with a particular role may quiteunexpectedly benefit the professional in another role.The outcome of the information-seeking is the end point of using an informationsource, but not the end of using the information. In evidence-based medicine,physicians with clinical information needs are advised to evaluate high-quality originalresearch critically in order to practise evidence-based medicine.19 The Evidence-basedMedicine Working Group developed "User guides to the medical literature" publishedas a series in the Journal of the American Medical Association of which the firstappeared in November 1993.39 Some researchers have underlined the need for criticallyappraising the literature, but recognised that this is unrealistic for busy clinicians. Thefollowing alternatives have been proposed: using guidelines,16 emphasizing Patient-Oriented Evidence that MatterS (POEMS);23 making a balance between the message

7

and the design of the study;40 reading abstracts of critically appraised journal articles;41

using the READER method for critical reading;42 and developing a medical databasewith answers of self-perceived information needs and reference citations supporting theanswers.43 Within evidence-based medicine, the information seeker has reached his orher goal only after having critically appraised the selected literature, applied theinformation in practice, and evaluated the process of information-seeking.

GENERAL PRACTITIONERS

The results of our studies were influenced by the type of subjects we examined. Ourstudy deals with Dutch general practitioners and their information needs. Theirinformation needs are influenced by the general practitioner’s role in the Dutch healthcare system. We will explain this first, then describe the libraries available for generalpractitioners.Different terms for general practitioner are in use. In the Netherlands the Dutch wordfor a general practitioner is "huisarts", this is literally translated "home physician". Inthe United Kingdom the titles are family doctor, or, general practitioner (GP). In theUnited States the common terms are family physician, and primary care physician,which includes family physicians, general internists, gynecologists, and pediatricians.The specialized physician can be translated as medical specialist (The Netherlands),consultant (UK), and specialist (USA).Dutch general practitioners are federated with the Royal Dutch Medical Association(KNMG) as the National Association of General Practitioners (LHV). The scientificinstitution is the Dutch College of General Practitioners (NHG).The Dutch health care system, including the general practitioner, has severalcharacteristics. General practitioners and medical specialists are two different groups ofprofessionals. They work in primary health care in the community, whereas medicalspecialists work in secondary health care, mostly in hospitals. In contrast to thesituation in the USA, Dutch general practitioners never work in hospitals, andspecialists almost never work in primary care. In the Dutch health care system, thegeneral practitioner acts as a gatekeeper. The general practitioner’s role is crucial foran effective and affordable health care system.44 Recourse to health care is high unlessthere is a financial barrier.Second, the Dutch general practitioner provides continuity of care by caring forpatients of all ages, and deals with all kind of problems, severe or mild, acute orchronic and in any phase. In addition, the general practitioner (or his locum, or,substitute) is available 24-hours a day.Third, the Dutch general practitioner cares for a population that is clearly defined byregistration. As a result, the general practitioner provides long-term care, not seldomlifelong.Finally, many Dutch general practitioners work in solo practices (in 1996: 49%) or induo or group practices (31%). Only 9% of the Dutch general practitioners work inhealth care centers (personal communication from the Netherlands Institute forResearch of Health Care, Utrecht, the Netherlands, March 1996).For their information needs Dutch general practitioners may turn to their privatelibraries. Next, most local hospitals to which general practitioners refer patients, also

8

have medical libraries, most of them managed by professional librarians. Morespecialized are the libraries of University Departments of General Practice, sometimesincorporated within the Medical Centre’s library or University Library. These librariesare not often close to the general practitioners’ offices, and this will be a barrier foruse. Nationally, the Netherlands Institute for Research of Health Care (NIVEL) inUtrecht, provides medical information on demand for general practitioners from theirown book and journal collection and bibliographic databases. The Royal NetherlandsAcademy of Arts and Sciences (KNAW) in Amsterdam owns the largest medicaljournal library in the Netherlands, and provides journal articles to general practitioners.

In conclusion, the model of Leckie of the information-seeking of professionals25 is anappropriate frame to explain the information-seeking by general practitioners. It wasintended to capture the complexity of the information-seeking activities ofprofessionals, including general practitioners.General practitioners are gatekeepers within Dutch health care, and they are expectedto manage a wide range of medical problems. The computerization of 93% of theDutch general practitioners, and the Dutch general practitioners’ characteristics mayinfluence their information needs and information-seeking.

REFERENCES

1 Huth EJ. The information explosion. Bull NY Acad Med 1989;65:647-61.2 Williamson JW, German PS, Weiss R, Skinner EA, Bowes F. Health science information

management and continuing education of physicians: a survey of U.S. primary care practitionersand their opinion leaders. Ann Intern Med 1989;110:151-60.

3 Williams ME. The state of databases today: 1998. In: Braun EE, Kumar L, editors. Gale directoryof databases. Detroit, MI: Gale Research; 1998. p. XXV.

4 Wyatt J. Use and sources of medical knowledge. Lancet 1991;338:1368-73.5 Arndt KA. Information excess in medicine: overview, relevance to dermatology, and strategies for

coping. Arch Dermatol 1992;128:1249-56.6 Lundberg GD. Perspective from the editor of JAMA, the Journal of the American Medical

Association. Bull Med Libr Assoc 1992;80:110-4.7 Solla Price D de. The development and structure of the biomedical literature. In: Warren K.S.

Coping with the biomedical literature. New York: Praeger Publications; 1981. p. 3-16.8 Braun EE, Kumar L, editors. Gale Directory of databases. Volume 1: Online databases. Detroit,

MI: Gale Research; 1998. p. 384.9 Durack DT. The weight of medical knowledge. N Engl J Med 1978;298:773-5.

10 Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teachingthe practice of medicine. JAMA 1992;268:2420-5.

11 Bouter LM. Meta-analyse: controleerbaar en reproduceerbaar literatuuronderzoek als basis voorrationele beslissingen in de gezondheidszorg. [Meta-analysis: verifiable and reproducible literatureresearch as basis for rational decision-making in healtcare. In Dutch]. Amsterdam: AmsterdamUniversity Press; 1994. p. 7-8.

12 Mulrow CD. Rationale for systematic reviews. BMJ 1994;309:597-9.13 Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical

decisions. Ann Intern Med 1997;126:376-80.14 Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet

1991;337:867-72.15 Cochrane Library [database on disk, CD-ROM and Internet]. Cochrane Collaboration; Issue 4,

October 1998. Oxford: Update Software; 1998. Updated quarterly.16 Grol R, Dalhuijsen J, Thomas S, Veld C in ’t, Rutten G, Mokkink H. Attributes of clinical

9

guidelines that influence use of guidelines in general practice: observational study. BMJ1998;317:858-61.

17 Gill P, Dowell AC, Neal RD, Smith N, Heywood P, Wilson AE. Evidence based general practice:a retrospective study of interventions in one training practice. BMJ 1996;312:819-21.

18 Geyman JP. Evidence-based medicine in primary care: an overview. J Am Board Fam Pract1998;11:46-56.

19 Straus SE, Sackett DL. Using research findings in clinical practice. BMJ 1998;317:339-42.20 Gardner M. Information retrieval for patient care. BMJ 1997;314:950-3.21 Sackett DL. Evidence-based medicine: how to practice and teach EBM. New York: Churchill

Livingstone; 1997. p. 2-16.22 Best Evidence [database on CD-ROM]. Philadelphia: American College of Physicians; 1996.23 Slawson DC, Shaughnessy AF. Obtaining useful information from expert based sources. BMJ

1997:314:947-9.24 Chan M, Fox NJ, Clamp SE, Dombal FT de. An information technology course in the medical

curriculum. Med Educ 1996;30:112-20.25 Leckie GJ, Pettigrew KE, Sylvain C. Modeling the information seeking of professionals: a general

model derived from research on engineers, health care professionals, and lawyers. Libr Q1996;66:161-93.

26 Owen PA, Allery LA, Harding KG, Hayes TM. General practitioners’ continuing medicaleducation within and outside their practice. BMJ 1989;299:238-40.

27 Crandall S, Elson R, McLaughlin C. Managing and communicating information in a new era. FamMed 1997;29:270-4.

28 Krikelas J. Information-seeking behavior: patterns and concepts. Drexel Libr Q 1983;19:5-20.29 Connelly DP, Rich EC, Curley SP, Kelly JT. Knowledge resource preferences of family

physicians. J Fam Pract 1990;30:353-9.30 Lor PJ. The information needs of the general practitioner: a theoretical model based on an

analysis of the general practitioner’s cognitive environment [dissertation]. Pretoria: University ofSouth Africa; 1979.

31 Forsythe DE, Buchanan BG, Osheroff JA, Miller RA. Expanding the concept of medicalinformation: an observational study of physicians’ information needs. Comput Biomed Res1992;25:181-200.

32 Gorman PN. Information needs of physicians. J Am Soc Inform Sci 1995;46:729-36.33 Pao ML. Concepts of information retrieval. Englewood: Libraries Unlimited; 1989. p. 14.34 O’Mahony B, Culhane A, Rouse JM, Ryan F, Shannon W. Keeping up to date - a challenge for

teaching practices. Ir Med J 1995;88:170-1.35 Thompson ML. Characteristics of information resources preferred by primary care physicians.

Bull Med Libr Assoc 1997;85:187-92.36 Haug JD. Physicians’ preferences for information sources: a meta-analytic study. Bull Med Libr

Assoc 1997;85:223-32.37 Cullen R. The medical specialist: information gateway or gatekeeper for the family practitioner.

Bull Med Libr Assoc 1997;85:348-55.38 Klercker T af, Zetraeus S. Dilemmas in introducing World Wide Web-based information

technology in primary care: a focus group study. Fam Pract 1998;15:205-10.39 Oxman AD, Sackett DL, Guyatt GH, for the Evidence-Based Medicine Working Group. Users’

guides to the medical literature, I: how to get started. JAMA 1993;270:2093-5.40 Edwards AGK, Russell IT, Stott NCH. Signal versus noise in the evidence base for medicine: an

alternative to hierarchies of evidence? Fam Pract 1998;15:319-22.41 McColl A, Smith H, White P, Field J. General practitioners’ perceptions of the route to evidence

based medicine: a questionnaire survey. BMJ 1998;316:361-5.42 MacAuley D, McCrum E, Brown C. Randomised controlled trial of the READER method of

critical appraisal in general practice. BMJ 1998;361:1134-7.43 Ely JW, Osheroff JA, Ferguson KJ, Chambliss ML, Vinson DC, Moore JL. Lifelong self-directed

learning using a computer database of clinical questions. J Fam Pract 1997;45:382-8.44 Sangster B. Health care policy and general practice research. In: Lamberts H, Knottnerus JA,

10

Hofmans SB, Klaassen A, editors. General practice research in Dutch Academia. Proceedings of aworkshop; 1994 April 15; Amsterdam, the Netherlands. Amsterdam: Medical Committee; 1994. p.35-9.

11

2 Use of information sources by family physicians: a literaturesurvey

The mere existence of information does not ensure access to itThomas Mann, American reference librarian, 1993

ABSTRACT

Introduction. Analysis of the use of information sources by family physicians is important for bothpractical and theoretical reasons. First, discovering the ways in which family physicians handleinformation may point to opportunities for improvement. Second, such efforts may lead to improvementsin the methods of literature research in general.Method. We performed a survey of the literature from 1975 to 1992.Results. Eleven relevant research publications were found. The data showed that family physicians usedcolleagues most often as information sources, followed by journals and books. This corresponds withfindings in other professions. Several factors influenced the use of information sources by familyphysicians, including the physical, functional and intellectual accessibility of the source; the physician’sage; participation by the physician in research or teaching; the social context of the physician; practicecharacteristics; and the stage of the information-gathering. The publications studied suggested ways toimprove information-gathering in the areas of computerization, education, library organization, andjournal articles.Conclusion. Family physicians use colleagues most frequently. An important factor influencing the useof information sources is accessibility.

Published as: AAH Verhoeven, EJ Boerma, B Meyboom-de Jong. Use of information sources by familyphysicians: a literature survey. Bull Med Libr Assoc 1995;83:85-90.

INTRODUCTION

Analysis of the use of information sources by family physicians is important for bothpractical and theoretical reasons. First, family medicine is a young academic disciplinewith a growing scientific output. Due to the interdisciplinary nature of this field,effective use of information sources can be a complex challenge, particularly becausebibliographic education is not yet common. Analysis of family physicians’ use ofinformation sources could point to opportunities for improvement, including moreeffective service by librarians and information specialists.Second, analysis of the use of information sources by family physicians could helpenhance the methods for literature research in general. Objectivity, precision,reliability, and validity are essential criteria of research methodology. These criteriaare, however, not yet accepted as fundamental aspects of literature research.1 It isimportant to develop this methodological rigor for literature research in general and tomake such methods concrete in a specific field such as family medicine. If thepublished results of literature research do not include the description of and

13

justification for the process used, the research cannot be repeated and verified. Hence,the methodological reliability and validity of the research can be called into question.2

Library and information sciences supply information search models; a methodology forliterature research could be based upon these models. However, these models do notreflect fully the working habits of scientists. Analysis of the information-gatheringhabits of family physicians could help fill gaps and shortcomings in these models andthus contribute to development of a sound methodological basis for the literatureresearch process.For these reasons, the authors undertook a literature survey to determine what sourcesand methods for information-gathering are used by family doctors and what factorsinfluence this process.

METHODS

The literature search was carried out using numerous information sources (appendix A)spanning the years 1975 to 1992, and an extensive list of keywords (appendix B).Publications were selected according to the following criteria:

they had to describe original research findings;they had to describe how family physicians gather information in general;they had to describe and compare the frequency of use of information sources;they had to be books, dissertations, or articles from journals indexed in one ofthree large bibliographies (Index Medicus, Excerpta Medica or LibraryLiterature) or on the Dutch list of Additional Scientific Journals for HealthSciences Research of the Royal Netherlands Academy of Arts and Sciences;they had to be in English or Dutch; andthey had to have been published in the period 1975-1992.

RESULTS

Eleven publications were found that addressed information-seeking by familyphysicians (table 1, documents A-K). All but three publications (E, H, J) describeAmerican studies. The motivations for information-seeking in the studies were patientcare decision-making (A, B, C, D, F, J) and continuing medical education (K). Nopublications were found on information-gathering for research. Four studies did notdescribe the motivation for the literature search.

Physician sample and research methodsIn most studies, the physician sample was selected from specialized groups, such asphysicians affiliated with a specific hospital or university (A, D, G, I), members ofprofessional medical associations (K), workers in a medical center, or members of asociety for medical information science (J). In four other studies, the investigatorsselected subjects at random from the population of family physicians in a specific geo-graphical area (C, E, F, H). Dee (B) selected interview candidates through personalnetworks.

14

Table 1. Eleven documents published in 1975 to 1992, focusing on the use of information sources byfamily physicians

A. Connelly DP, Rich EC, Curley SP, Kelly JT. Knowledge resource preferences of familyphysicians. J Fam Pract 1990;30:353-9.

B. Dee CR. Information needs of the rural physician: a descriptive study [dissertation]. FloridaState University; 1990.

C. Ely JW, Burch RJ, Vinson DC. The information needs of family physicians: case-specificclinical questions. J Fam Pract 1992;35:265-9.

D. Gruppen LD, Wolf FM, Voorhees C Van, Stross JK. Information-seeking strategies anddifferences among primary care physicians. Möbius 1987;7:18-26.

E. Heal PE. The information needs of general practitioners: to what extent they are satisfied bythe Postgraduate Medical Centre Library [master’s thesis]. Loughborough University ofTechnology; 1978.

F. Means RP. Information seeking behaviors of Michigan family physicians [dissertation].University of Illinois; 1979.

G. Renford BL, Eagleson BK. Profiling family physicians and their use of information sources.Med Ref Serv Q 1982;1:39-52.

H. Stichele R Vander, Heyvaert J, Royen P van, Smet E de. De informatiekanalen van deVlaamse huisarts. [The information channels of the Flemish family physician. In Dutch].Huisarts Nu 1985;14:188-95.

I. Strasser TC. The information needs of practicing physicians in northeastern New York State.Bull Med Libr Assoc 1978;66:200-9.

J. Timpka T, Ekström M, Bjurulf P. Information needs and information seeking behaviour inprimary health care. Scand J Prim Health Care 1989;7:105-9.

K. Williamson JW, German PS, Weiss R, Skinner EA, Bowes F. Health science informationmanagement and continuing education of physicians: a survey of U.S. primary carepractitioners and their opinion leaders. Ann Intern Med 1989;110:151-60.

The sample sizes ranged from 12 (B) to 425 (H). The responses varied from 45% to100%. Two studies investigated nonrespondents (F, H). The habits of these physiciansdid not significantly differ from those of respondents.Methods used in the eleven studies included written questionnaires (A, D, E, F, G, H,I, J), telephone interviews (F, K), face-to-face interviews (B), observations (B, C),diary sheets (E), the critical incident technique (J), and patient chart reviews (B).

Information sourcesAll studies but one (E) showed that family physicians used information sources in thefollowing order of frequency: colleagues, books and journals, libraries, and printed oronline bibliographies. Means (F) considered textbooks and journals as one (printed)source. When books and journals were combined, they were the most frequently usedsource. However, most studies divided these printed sources into two categories,making colleagues the most important source of information. In Heal’s survey (E),books and journals from personal libraries preceded colleagues.

15

Physician characteristics related to information-seeking behaviorYounger physicians used libraries (E, I, K) and printed sources (F) more frequentlythan did older physicians. Ely and Means (C, F) could not relate age or professionalexperience to the use of information sources. Family physicians who engaged inresearch or educational activities used journals, conferences, libraries (I), and onlinedatabases (I, K) more than did physicians who handled only patient care. Stichele (H)found that oral and informal communication was an important information channel.Physicians in practices in which disciplines other than family medicine also wererepresented used information sources less than did single-specialty practices (A).Whereas physicians in solo practices used journals the most (F), physicians in healthcenters usually consulted colleagues (C). In rural areas, conferences were an importantinformation source, as were sales representatives from the pharmaceutical industry (I).In contrast to urban family physicians, rural physicians performed very few onlinesearches (I). An office library, if available, was an important information source (D,E).

Nature of information needed and accessibilityPhysicians needed various kinds of information: general medical information (J);information on drugs (C, K); information on therapy (B, E, F, J) and information ondiagnosis (J). Means (F) stated that the stage of the information-gathering processinfluenced the choice of the information source. In calling attention to newinformation, printed material was the most important source; in the analysis stage,personal contacts were most important; and in the decision stage, refresher courseswere the most important information source.Cost factors, such as time and energy needed to search for information, were moreimportant than quality of the information source (A, J). The most frequently usedsources were those with good physical, functional, and intellectual accessibility (A, B,D, E, F, K); those that were clinically relevant (A, F); and those that were familiar tophysicians (A, B, E, F, H). Connelly (A) defined physical accessibility as "availability"(how close the resource is to the site of clinical practice), functional accessibility as"searchability" (how easy it is to find the knowledge needed in the source at hand),and intellectual accessibility as "understandability" (how easy it is to read andunderstand the information).Physicians faced a number of problems when seeking information: irrelevantinformation (F, K), inappropriate descriptors (K), incorrect and incomplete indexing ofbooks and journals (B, D, F), inadequate organization of journals in the medicalpractice (B, D), and a large time investment (B, E, F, J, K).

Suggestions for improvementAll authors but one (C) recommended ways to improve information-retrieval for familyphysicians. These recommendations can be divided into four categories:1. Computerization. There were recommendations for improved online retrieval

systems, full-text databases, hypertext systems, and expert systems (A, B, D, F,G, J, K). Two studies warned against massive databases with much irrelevantmaterial (D, K).

2. Education. There is a need for effective instruction in the use of information

16

sources through programs such as the medical documentation and bibliographycourse for family physicians in Belgium3 as well as in medical curricula andcontinuing medical education (A, G, H, K).

3. Libraries. Library facilities should be made more accessible to family physiciansthan they are now (B, E, I).

4. Journal articles. Journal articles should be tailored more than they are now to thefamily physician’s daily work (A), and article structure should be improved (K).

DISCUSSION

Family physicians used colleagues most frequently, followed by books and journals, asinformation sources for patient care and continuing medical education. Familyphysicians involved in research or education relied more on formal informationsources, such as printed and electronic sources, than did their colleagues focusing onlyon patient care. Obviously, there are differences in the qualitative and quantitativeinformation needs of medical researchers and medical practitioners. Research requiresmore use of formal (printed or computer) information than of informal sources, suchas colleagues. Clinical work requires precisely the opposite.4-6

Studies of other professions confirmed the results of the literature survey. Physicists,7

engineers,8 social scientists,9 research and development professionals,10 and scientists11

also rely on individuals and journals more than other information sources. In contrastto family physicians, internists prefer journals to colleagues. Gruppen (D) explainedthis discrepancy as the result of differences in the nature of the two professions, andtheir training, working conditions, and tools.Family physicians rarely used online bibliographic databases (B, E, F, G, H, I).However, for physicians in rural areas, online connections were important.12

In addition to familiarity with information sources, physical and functionalaccessibility are clearly important factors in the process of information-gathering.Family physicians placed more importance on accessibility than on factors related tothe quality such as reliability and completeness of an information source. This findingalso was reported by studies of other professionals, such as physicians and medicalstudents,13 engineers,8 and industrial personnel,14 A possible explanation could be thatquality is not immediately apparent whereas cost factors have a more direct impact,15

Improving the quality of a source, in contrast to improving its accessibility, will notalways result in increased use (A).A number of suggestions were made for improving the organization and supply ofinformation through initiatives involving computers, education, library use, and journalarticles. The suggestions were rather technical, without much analysis of why thecomputer was used so little. Research could help identify possibilities for stimulatinguse of the computer.Although family physicians used other individuals as their primary source ofinformation, none of the studies reviewed offered suggestions with regard tostimulating adequate personal contacts. In this regard, the literature of family medicineis comparable to that of other sciences.The existence of an office library increased the availability and use of information (A,D). Improving the clinical applicability and structure of journal articles was

17

recommended as a way of making information more accessible. However, the presentauthors feel that improving the functional accessibility (indexes of books and journals)probably would have more impact than improving the clinical applicability alone. Thisproposition is strengthened by research showing that, even when they are available,books and journals with inadequate indexes hardly ever are used.16

To increase familiarity with information resources, education is necessary. Inworkshops for continuing medical education courses, it is important to provide adviceon use of not only formal sources, such as journals, but also informal sources, such ascolleagues.

CONCLUSION

When constructing an explicit methodology for information-gathering by familyphysicians, it should be noted that these physicians use colleagues frequently, and thataccessibility is an important factor influencing the use of information sources.In addition to instruction in use of traditional, formal sources such as journals, familydoctors must be taught to exploit personal information sources, which evidently are themost popular. Attending conferences can enlarge personal networks, for example.Librarians and information specialists can help improve communication among familyphysicians. They could create new document files, such as international directories offamily practitioners, which could include up-to-date names, addresses, and researchfields of family physicians. Librarians also could provide analyses of citations ofleading research studies related to family medicine.Library and bibliographic instruction would enlarge physicians’ familiarity withinformation sources, and familiarity would enhance perceptions of an informationsource as being accessible.8 A specific bibliographic database for family medicine,including "grey" and national literature, would improve the physical accessibility ofthe family medicine knowledge base. Functional accessibility could be improved withappropriate descriptors in printed or computerized bibliographies of family medicineand extended indexes in books and journals with, for example, cross-references tosynonyms and related terms. Professional organizations and libraries should advisefamily physicians on organizing their professional libraries and on development ofcomputer and manual filing systems.Clearly structured journal articles and abstracts improve intellectual accessibility.Abbreviations should be minimized, because they decrease the readability of the text.In summary, this literature survey uncovered many ways to improve traditional libraryscience models to provide methodologically sound research tools, for family doctors aswell as others.

AcknowledgmentThis study was supported in part through a grant from the Jan Kornelis de CockFoundation, Groningen, the Netherlands, project number 93-16.Copyright by and reprinted with permission of the Medical Library Association.

18

REFERENCES

1 Vorst HCM. Methodische aspecten van literatuuronderzoek. [Methodological aspects of literatureresearch. In Dutch]. In: Kallenberg A, Kalle P. editors. Automatiseringsaspecten vanonderwijsdocumentatie: verslag van een informatiemarkt. [Aspects of computerization ofeducational literature: report of an information market. In Dutch]. Den Haag: SVO; 1984. p. 59-75.

2 Boerma EJ, Arnold CA. The library, information services and user education and the aims of theuniversity: the contribution of the methodology of (literature) research. Iatul Q 1990;4:56-64.

3 Peeters C, Stichele R Vander. A training course in medical documentation for family physicians.In: Deschamps C, Walckiers M, editors. Medical libraries: cooperation and new technologies.Amsterdam: Elsevier; 1987. p. 235-41.

4 Osiobe SA. Sources of information for biomedical decision-making. Methods Inf Med1985;24:225-9.

5 Woolf SH, Benson DA. The medical information needs of internists and pediatricians at anacademic medical center. Bull Med Libr Assoc 1989;77:372-80.

6 Huth EJ. The information explosion. Bull NY Acad Med 1989;65:647-61.7 Chen C. How do scientists meet their information needs? Spec Libr 1974;65:272-80.8 Gerstberger PG, Allen TJ. Criteria used by research and development engineers in the selection of

an information source. J Appl Psychol 1968;52:272-9.9 Line MB. The information uses and needs of social scientists: an overview of INFROSS. Aslib

Proc 1971;23:412-34.10 Rodenburg P. R&D-professionals als informatiegebruikers. Een inventarisatieonderzoek naar het

benutten van informatiebronnen. [Research & development professionals as information users. Aninventory investigation in the utilization of information sources. In Dutch]. Open 1992;24:2-6.

11 Rowland JFB. The scientist’s view of his information system. J Documentation 1982;38:38-42.12 Harris DL, Peay WJ, Lutz LJ. Using microcomputers in rural preceptorships. Fam Med

1989;21:35-7.13 Northup DE, Moore-West M, Skipper B. Teaf SR. Characteristics of clinical information searching:

investigation using critical incident technique. J Med Educ 1983;58:873-81.14 Rosenberg V. Factors affecting the preferences of industrial personnel for information gathering

methods. Inf Storage Retrieval 1967;3:119-27.15 Curley SP, Connelly DP, Rich EC. Physicians’ use of medical knowledge resources: preliminary

theoretical framework and findings. Med Decis Making 1990;10:231-41.16 Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met? Ann

Intern Med 1985;103:596-9.

19

APPENDIX AInformation sources used for the literature search

Medical literature guides1-5, which contain medical literature sourcesIndexes of journals: the volumes 1990-1992 of Annals of Internal Medicine, the Bulletin of theMedical Library Association, and Huisarts & Wetenschap (Family Physician & Science)Printed bibliographies: FAMLI 6 1990-1991, Medical and Health Care Books and Serials in Print7

1990-1992, ARIST 8 1990, and Library Literature9 1990-1992Dutch University CataloguesOnline bibliographic databases from 1975 to 1992: MEDLINE, EMBASE, CATLINE, DHSS,Health Periodicals Database, Science Citation Index, and LISA; the bibliographic book databases:LC MARC-Books, British Books in Print, Books in Print, and Dissertation Abstracts Online;Institutes and conferences: Netherlands Institute for Primary Health Care in Utrecht, theNetherlands; Information and Documentation Centre for Social Science in Amsterdam, theNetherlands; the Royal College of General Practitioners in London, Great Britain; and a Boerhaavecourse, "Future Trends in Biomedical Documentary Information"Citation analysis: with the citing document at hand, tracing literature references by means ofreferred publications

References

1 Morton LT, Godbolt S, editors. Information sources in the medical sciences. 4th ed. London:Bowker-Saur; 1992.

2 Chen C. Health sciences information sources. Cambridge, MA: MIT Press; 1981.3 Jung C. Wie finde ich medizinische und pharmazeutische Literatur? Zweite Aufl. Berlin: Berlin

Verlag; 1981.4 Strickland-Hodge B, Allan BC. Medical information: a profile. London: Mansell; 1986.5 Welch J. King TA, Hands DE. Searching the medical literature: a guide to printed and online

sources. London: Chapman and Hall; 1985.6 FAMLI: family medicine literature index. London, Ontario: World Organization of National

Colleges, Academics and Academic Associations of General Practitioners/Family Physicians(WONCA); Bethesda, MD: National Library of Medicine; 1980-.

7 Medical and health care books and serials in print: an index to literature in the health sciences.New York: Bowker; 1985-.

8 Annual review of information science and technology. Medford, NJ: Learned Information; 1966-.9 Library literature: an index to library and information science. New York: Wilson; 1921-.

20

APPENDIX BKeywords used for the literature search*

Information descriptors: Family physician descriptors:bibliography family doctorinformation family medicineinformation gathering family physicianinformation need family practiceinformation seeking family practitionerinformation source general physicianinformation service general practiceinformation storage and retrieval general practitionerinformation system health careknowledge physicianlibrary practitionerliterature primary health careliterature researchliterature searchmedical bibliographymedical informationmedical literaturemethodologyresearchresearch designresourceretrieval systemsource

* Plural forms were used also.

21

22

3 Management of bibliographic information by Dutch researchersin general practice

If you think knowledge is expensive, think about ignorance

ABSTRACT

Introduction. As a result of changes in information technology and the rapid growth of publications,methods of searching the literature have changed. Systematic searching of the growing literature hasbecome very important. It is not known whether researchers in general practice search systematically,and whether they have incorporated computerized sources in their research practice.Method. We aimed to assess the methods of searching the literature used by GPs, and therefore weinterviewed eight GP researchers, two information specialists and a psychologist working in primaryhealth care organizations.Results. The GP researchers began their search in their private book collections, or asked colleagues forinformation. Later in the search process, they used computerized information sources. Medline onCD-ROM and the computer-based indexes of two Dutch journals were most widely used. The GPsfound it difficult to locate the appropriate terms to describe their research questions. Knowledge of themethods of searching the literature, and skills required to use the information sources efficiently wereboth lacking. There was a need for instruction and training in the use of information sources. GPsneeded a telephone help desk where their questions could be answered adequately.Conclusion. GP researchers search the literature unsystematically. Although computerized databaseswere being used, knowledge and skills related to the use of these information sources should beimproved.

Published as: Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Management of bibliographicinformation by Dutch researchers in general practice. Fam Pract 1997;14:69-72.

INTRODUCTION

As the number of publications in medicine increases, searching the literaturesystematically becomes more important.1 Electronic bibliographic databases withreferences to the medical literature, such as Medline, make it possible to select articles.Medline is the counterpart of the printed Index Medicus, and it contains literaturecitations from 3700 international biomedical journals. The producer of Medline, theNational Library of Medicine, Bethesda, USA, has offered Medline to online userssince 1970. With a computer, a modem and a telephone connection to a vendor,citations and, in 80% of cases, abstracts, can be searched and selected interactively.The citations can be loaded onto a personal computer and later manipulated with aword processor or database management system such as Reference Manager orCardbox. In the Netherlands, only large organizations such as universities use onlinebibliographic databases, because of the costs and the skills required. To meet theseperceived disadvantages of online searching, and as a result of technical improvements,Medline became available on CD-ROM in the 1980s. It is now offered by most

23

hospital libraries and universities through a local network. More recently, it hasbecome possible to search Medline on the Internet, via vendors such as Dialog andDatastar,2 via an interface such as Grateful Med,3 and on compact disk such asSilverplatter.4

However, GPs do not use computerized databases frequently. They are more likely torefer to colleagues and journals for information, followed by books and meetings, andfinally computerized databases.5 We therefore wanted to determine whether GPresearchers search the literature systematically, whether they use computerizedinformation sources as well as traditional sources, and what problems they encounter.

METHODS

After studying the literature, we composed an interview guideline, interview questionsand a candidate list. The questionnaire included (see appendix):

a short list of personal characteristics;25 open questions on experience, knowledge, and practice of literature research;a questionnaire with multiple choice questions on the use of 30 informationsources; andquestions on problems encountered during literature research.

The questions were asked in a fixed order.We composed a list of candidates for interview on the basis of their experience onliterature research; achieving a balanced spread over the eight Dutch Departments ofGeneral Practice and a distribution of the backgrounds of the candidates. Theseincluded GP researchers paid by the Governmental Organization for Research (NWO);GPs who combined patient care with research; GPs who had already obtained theirPhD; professors; editors of medical journals; and medical librarians. Of the 13candidates invited for the interview 11 accepted.Of the eight physicians interviewed, seven were GP researchers, and one physicianundertook research in general practice. For the analysis, we considered theseresearchers as one group: the GP researchers. Of the three non-GP researchers, twowere medical information specialists of the Dutch College of General Practitioners(NHG), the third was a psychologist of the Netherlands Institute of Primary HealthCare (NIVEL). Four of the interviewees were women, one of whom was a GPresearcher. The average age of the GP researchers was 42 years, and of the non-GPresearchers 40 years. The GP researchers had an average of 13 years’ experience inpatient care, eights years’ research experience, and they had published one to morethan 100 articles.The interviews were held from December 1993 until June 1994 at the place of work ofthe interviewees, except for one. All interviews were conducted by the first author.The interviewer made notes during the interviews. In addition, all interviews wererecorded on tape. On average, an interview took 70 minutes. Because no newinformation emerged, we stopped after 11 interviews.

24

RESULTS

Place of literature researchAll the GP researchers were positive about literature research ("lovely", "finger-lickin’good", "it sharpens you"). On the negative sides of literature research the intervieweesmentioned its time-consuming nature, misleading titles of articles, repetition of authors("that is very irritating"), and the need for perseverance. The GP researchers had littleknowledge of literature research before they began their research projects. However,two GP researchers had read articles about medical literature.6 The non-GP researcherswere of the opinion that GP researchers did not prepare their literature research well("general practitioners are priggish, they think they already can manage it").It took the GP researchers a lot of time to trace the appropriate literature: up to 50%of their research time. They prepared their literature search by selecting keywords,journals and key authors of their subject, and browsed in their personal library. Thetwo most experienced GP researchers worked very unsystematically. The mostimportant times for tracing the literature was before the research question was stated,and during writing the report. However, most of the GP researchers said they searchedthe literature throughout their research project.

Practice of literature researchGP researchers typically used three methods at the start of a new project. To make afirst draft for a new research project, they searched their personal files of journalarticles. In addition, they browsed through medical journals by hand ("trial and error","the most surprising information I find accidentally"). They also perused the citationsof known and relevant publications, especially review articles. When the research hadprogressed and the subject was well defined, the GP researchers used keywords toselect articles from journal indexes. Only when the research came to an end did theGP researchers use computerized literature sources such as Medline on CD-ROM. Theliterature search ended when the same articles were found increasingly.After asking about the role of personal relationships, the GP researchers said thatresearchers, as well as librarians, played key roles in obtaining information on thesubject in all phases of the literature research ("experts are very important when youhave limited time or technical problems"). The three non-GP researchers assisted theresearchers in setting up a search strategy for Medline on CD-ROM. They also showedthem how to use the catalogue, as well as how to find books and journals in thelibrary.

The information sourcesIn addition to the open questions, the printed multiple-choice questions showed thatGP researchers used their personal library, books, journals, people, libraries, andelectronic sources. Most frequently, the GP researchers consulted their own bookcollection or article files, often daily. These referred to Dutch books and journals aswell as books and journals in a foreign language, depending on the subject. They alsobrowsed in books regularly, with no particular purpose, although they did not usebooks a lot, because they were considered as becoming out of date too soon. Apopular method was to scan the literature lists in journal articles, reviews and

25

dissertations in order to detect useful citations.GP researchers consulted their colleagues on a monthly basis. Medical specialists,pharmaceutical representatives, congress participants, and librarians were seldomconsulted. The hospital library or the library of the department of general practicewere visited weekly to monthly. All GP researchers except one used computerizedbibliographic databases. Those who worked at an academic centre used Medline onCD-ROM; some delegated the searching to librarians. In general, those who searchedthemselves were satisfied with the search results. One of them said that, in general,people overestimate the computer. One GP researcher even checked computerizedsystems with people ("I never trust computers; I always test the provided informationthrough people").The two major Dutch journals, Huisarts en Wetenschap (General Practitioner andMedical Science) and Het Nederlands Tijdschrift voor Geneeskunde (Dutch MedicalJournal), both have a computer index. Five of eight GP researchers used one of them.Only one GP researcher used the online catalogue of the university library or hospitallibrary (OPAC, online public access catalogue). Some of the GP researchers did noteven know that OPAC existed. Nobody used Current Contents on Diskette (CCOD),containing tables of contents of the latest issues of a wide range of journals.Two GP researchers delegated literature searching to the library of the Royal DutchAcademy of Sciences (KNAW), to a hospital library, to the library of the RoyalCollege of General Practitioners in London, or to a student on work placement.

Choosing an information sourceSeveral factors influenced the choice of an information source. First of all, the GPresearchers preferred an information source which was easy and quick to use ("I havethe computer index of some journals at hand"). Second, the information source shouldbe close to the user ("now I can use Medline at my desk through the universitynetwork, I use it more frequently"). Finally, negative or positive experiences withinformation sources had a large impact on the use of these sources in the future ("ifyou only once had a negative experience, you will never use it again").The three non-GP researchers were convinced that accessibility was the most importantfactor. Librarians should give information on the ins and outs on searching the medicalliterature, and how GP researchers could search on their own in the future.

Storing the literatureOnly two of the GP researchers actually used a computerized system to manageliterature citations (Cardbox). Three other GP researchers had stopped using it, becauseusing it took too much time ("the discomfort of importing the citations and the timeinvolved did not make odds against the results"). Three GP researchers did not use asystem for the citations at all, but filed documents directly by subject or author. OneGP researcher used the International Classification of Primary Care (ICPC) code to filehis journal articles.

Problems and solutions in literature researchThe GP researchers experienced two major problems: translating the subject intokeywords, and the time it took to search the literature. Finding the appropriate

26

keywords was difficult, and different keywords were sometimes used by differentsources. It seemed as if in general practice the terminology was not unequivocal.Sometimes keywords on their subject did not exist at all.It took too much time to visit a library, and to photocopy ("first the problem with themoney for the photocopier, there should be a money-changer next to the photocopier,next there is a line of people waiting for the photocopier, and after this all I have lostmy concentration totally. I would become a donor to the library, as long as I can makemy photocopies"). The GP researchers suggested several solutions. First of all, theywanted to incorporate structured library instruction and use of information sources inthe vocational training for general practice. In addition, the accessibility of librariesshould be improved, either in reality or in virtual reality. In the library thephotocopying facilities should be increased. Three GP researchers passed on somespecific ideas to improve the information facilities. An ideal information facilityshould consist of a local information centre, where help is offered personally, by postor by telephone. At this help desk, an information specialist with expertise in the fieldof general practice and information sciences as well should meet the needs of thegeneral physicians at different levels. Additionally, the non-GP researchers noticed acomputer illiteracy in GPs. Libraries needed clear information leaflets to informvisitors on the library collections where things can be found, and how they can beused. Library personnel should have a medical education ("only with diligence andgood-will you can’t manage"), and be more client centred ("librarians should noticewhen GPs lack the knowledge to use the library, or when they are too shy to ask forhelp").

DISCUSSION

This study shows that the interviewed Dutch GP researchers search the literatureunsystematically, and that they use Medline as the only computerized informationsource. The main problems they encounter are finding the appropriate keywords andtime to search the literature. Although the GP researchers in this survey used Medlineand the electronic indexes of two important Dutch medical journals, they did not usethese sources systematically or efficiently. This was because it was difficult to selectthe appropriate keywords, and because the information source was not available, noteasily accessible, or too difficult to use. In addition, they were ill-informed about otherelectronic information sources, such as the computer catalogue of libraries. Thisinefficient use of electronic sources could be one of the reasons why literaturesearching took so much time.The interviewees suggested some solutions to the problems encountered. Onesuggestion was structured library and computer instruction in the vocational trainingfor GPs. This could reduce the physical and psychological barriers encountered in theuse of libraries. Another problem was the need for professional aid in searching theliterature. GP researchers preferred an information desk accessible either through apersonal appointment, by post or by telephone. The structure of the Dutch library andinformation world does not meet this demand. Information about general practice isscattered around the country, and different organizations have different opening hours,subscription requirements, quality levels, and subject scopes. A separate organization

27

with local departments, for example as a part of the Netherlands Institute of PrimaryHealth Care (NIVEL) or the Dutch College of General Practitioners (NHG), couldfunction as a professional information help desk. This new organization shouldcombine medical knowledge on general practice with skills to manage medicalinformation.The existing literature mentions this need among GPs.7,8 Free teleinformation deskshave been set up, where GPs could receive tailored information, as well as literaturelists on special subjects, and photocopies of journal articles. Other authors9,10 havedescribed an experiment with an online connection between the librarian and the GP’scomputer. In this way a GP could request for an online literature search at home, andat the same time watch and guide the librarian’s search strategy. At the end, the GPreceives the print-outs from his own printer.The present interview study has some limitations. We selected 13 possibleinterviewees, two of whom declined to take part. We therefore interviewed only afraction of the total number of GP researchers. In choosing the interview method, wecould have been given socially acceptable answers. In addition, the interviewer couldbe biased by previous interviews. To reduce this bias, we used a fixed list of interviewquestions. The advantage of the interview technique is that it provides very specificand detailed information.As far as we know, no one has published studies of the literature search habits of GPresearchers as a group. In other countries postal surveys have been carried out toassess the information needs of practicing GPs. In general, these results are quitesimilar to ours, although in our study the use of electronic sources ranked higher.Connelly11 found that GPs used people, journal articles and books most frequently asan information source. Accessibility seemed to be the most important factor inchoosing and using information sources.In summary, this study shows that the interviewed Dutch GP researchers search theliterature unsystematically, and that they use Medline as the only computerizedinformation source. The main problems they encounter are finding the appropriatekeywords and time to search the literature. Still it is unlikely that electronicinformation sources will oust people or printed sources from the first place in the nearfuture.

AcknowledgementThe Matty Brand Fund made it possible to conduct this study.This article is reproduced by permission of Oxford University Press.

28

REFERENCES

1 Oxman AD, Guyatt GH. The science of reviewing research. Ann NY Acad Sci 1994;703:125-34.2 URL: http://www.krinfo.ch3 URL: http://igm.nlm.nih.gov4 URL: http://www.silverplatter.com5 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Use of information sources by family

physicians: a literature survey. Bull Med Libr Assoc 1995;83:85-90.6 Guyatt GH, Rennie D. Users’ guides to the medical literature [editorial]. JAMA 1993;270:2096-7.7 Creth SD. The health information environment: a view of organizational and professional needs

and priorities. Bull Med Libr Assoc 1993;81:414-20.8 Lovas I. A look at LOANSOME DOCtm service. Bull Med Libr Assoc 1994;82:176-80.9 Witiak JL, Prewitt BG, Deschere AR. Online database searching via telephone conferencing.

Online 1979;3:21-3.10 Graham DL, King C, Whitney PJ. Simultaneous remote search: on-line bibliographic library

services for practicing physicians. JAMA 1981;246:1115-6.11 Connelly DP, Rich EC, Curley SP, Kelly JT. Knowledge resource preferences of family

physicians. J Fam Pract 1990;30:353-9.

29

APPENDIXInterview scheme for the interviewer

Instructions for the interviewer:Follow the given order of the questions.Read out loud literally the questions that are printed; read quietly.Check in advance that the tape recorder operates.Each interview should be on a separate cassette.Write the name of respondent on the cassette-label.Just before the first question, make a note of the starting time.Do introduce yourself. Refer to the announcing letter the respondent has received from theresearcher; and also refer to the appointment made by telephone for this interview.Before starting with the questions, bring the following points to the notice of the respondent:- Bodies responsible for the study: University Library and the Department of General Practice

of the University of Groningen.- Name of the respondent was obtained through ..........................................................- The questions will discuss the way general practitioners gather literature references for

research.- Two questions, which are more suitable for written answering, will be handled over to the

respondent as a questionnaire. The other questions have to be answered orally and most ofthem are open questions.

- Purpose of the interview is twofold.1. To gain an impression of the practice of literature-gathering by general practitioners forresearch. In this way the bottlenecks can be detected.2. To collect the basic material for a written questionnaire that will be sent to a sample ofDutch general practitioners.

ONLY FOR THE PILOTS: "Because your interview will be used as a pilot study, it is possible Iwill approach you later, to ask you additional questions by phone. Would you agree to that?"The results of the interviews will be published in a journal.Tape recording is desired. Tapes will be typed and the typed text will be discussed by threepeople.Except for the study of the interviewer, the results of the interview will not be used.Length of the interview: 1½ to 2 hours.Does the respondent have any questions or difficulties? If not, begin!Note: Starting time: .......

QUESTIONS

1We need some personal data about you. Would you be kind enough to fill in this form?(Hand over the appendix on personal data.)Please, switch on the tape recorder now.

2General: When you performed a literature search in the past, how did you go about this, forexample in your last research project?(Stimulate the respondent to mention as many aspects as possible. If necessary, explain thequestion in your own words. But don’t give examples.)Make a note of the answers by means of keywords.

30

3 to 11Place and preparations of the literature search: Now, I want to ask you some more specificquestions about your literature search. I define a literature search as follows: all decisions and actsneeded to locate and to obtain scientific documents, and to give account of the search process in areport, in order to describe what is known in the scientific literature about a specific subject. Thefollowing questions are about the place of the literature search within the whole of your research,and about the possible preparations you made for it. For example, in your last journal article.

3In comparison with the time you spent on the paper as a whole, how much time did yourliterature search take?(If necessary, give some details as follows.)Was it more or less than half the time you spent on the research as a whole? More or less than aquarter or a tenth of the time?Make a note of the answers by means of keywords.

4When do you usually perform a literature search?(If necessary, enumerate the possibilities given below. Tick where appropriate; other answers arepossible.)Make a note of the answers by means of keywords.

Before the phase of question presentationDuring the process of question presentationDuring the collecting of the research dataDuring the reporting of the researchAfter the reporting of the researchAfter publication of the research

5How did you like the work of collecting literature references?(If needed, clarify the question by asking what kind of feelings he or she has while collectingreferences.)Make a note of the answers by means of keywords.

6Did you construct a strategy in advance to search through the literature in order to collect yourreferences?Tick where appropriate:

yes, continue with question 7no, continue with question 8

7What was your strategy for searching the literature?(Stimulate the respondent to mention as many aspects as possible. If necessary, explain thequestion by giving the following examples: what sources and what keywords could be used?)Make a note of the answers by means of keywords.

8Before you started searching for literature references, did you ask advice about the best way to goabout it?Tick where appropriate:


31

9Whom did you ask?Tick where appropriate and/or fill in; other answers are possible.

Colleague general practitionerMedical specialistOther kind of physicianLibrarianOther person, namely: .......................................................................................

10Before you started searching for literature references, did you read about how to perform aliterature search?Tick where appropriate:


11What kind of printed texts did you consult?Tick where appropriate and/or fill in; other answers are possible. (If necessary, give an example,for instance: a book or a journal article)

Book on family medicineMedical book, not family medicineBook on library or information scienceLiterature guideMedical journal articleArticle, otherOther, namely: ................................................................................................

12 to 17Information sources and accessibility

12Now I would like to ask about information sources that could be used in gathering literaturereferences. The definition of an information source is: a document, person or body, whichcontains itself the whole document to which the literature reference refers (e.g. a journal article),or which refers to the original document (e.g. a catalogue or a person).Please would you let me know about which information sources you use by completing theappendix on information sources, which I am now giving you. Indicate how often you use eachsource that you mention.Do you browse sometimes through journals or books in (personal) libraries? If your answer isyes, tick yes in question 1, otherwise no. If your answer is yes, tick also the right box concerningthe frequency (daily, weekly, monthly or less frequent).(Hand over the appendix on information sources.)

13Could you indicate why you use one information source more frequently than another?(Enumerate the sources the respondent uses daily. If necessary, ask what advantages these sourceshave. Mention, if necessary, the following qualities: distance from the source, speed of search,understandability and relevance for research.)Make a note of the answers by means of keywords.

32

14Have you ever stopped searching for references even though you thought that there were probablymore?Tick where appropriate:


15Could you tell some more about it?(Stimulate the respondent to say as much as possible. If necessary, explain the question byformulating it in your own words. For example, ask for the reason for giving up the search forreferences.)Make a note of the answers by means of keywords.

16On the paper you just filled in, you have indicated whether or not you use people as aninformation source.If the answer was:


17Could you tell me anything more about the circumstances in which you get literature referencesfrom others?(Stimulate the respondent to say as much as possible. If necessary, indicate: whom you got theinformation from, the frequency compared to other sources, the circumstances in which thereferences were obtained - by chance, during a meeting, or during a hospital visit - and if thereferences were useful or not, and why.)Make a note of the answers by means of keywords.

18On the paper you just filled in, you have indicated whether or not you use computerized databasesas an information source.If the answer was:


19Could you tell me anything more about your experiences with computerized databases?(If necessary, explain what these systems imply. Ask if the next items have been used; ask whodrew the respondents’ attention to the following items.)

the OPACthe CD-ROMonline literature searches (would he or she like to search Medline or other online databasespersonally?)Current Contents on Disk, Reference UpdateReference Manager, Cardboxcomputerized indexes (table of contents) of Huisarts & Wetenschap and Het NederlandsTijdschrift voor Geneeskundeinstruction in library useinstruction in use of bibliographies

Make a note of the answers by means of keywords.

33

20 to 23Handling literature referencesThe following questions deal with the method of handling literature references.

20Do you use a system for coping with your references, such as a manual or a computer cardindex?Tick where appropriate:


21Which of the both systems do you use?Tick where appropriate:

Manual card systemComputer card system

22Could you tell some more about your card system?(Stimulate the respondent to say as much as possible. Encourage.)Make a note of the answers by means of keywords.

23How do you file your reference papers?(Stimulate the respondent to say as much as possible. If necessary, give instructions by asking: byauthor, subject or number.)Make a note of the answers by means of keywords.

24Do you think that a general practitioner who performs a literature search for patient care actsdifferently than a general practitioner who performs a literature search for a research project?Make a note of the answers by means of keywords.

25 and 26Problems and suggestions

25What problems do you encounter when searching for literature references in your research?(Stimulate the respondent to say as much as possible. Tick the items that the respondent names.Make a note of the answers, other than stated. Enumerate the next subjects, when the respondentis finished.)

To find the appropriate keywordsTo find the appropriate information sourcesTo obtain the documents (document delivery)Keep up to date with the literature on your subjectOrganizational problems (e.g. finding a locum, or problems concerning the library)Lack of timeTable of contents, indexes of books and journalsUnfamiliarity with information sources, such as computerized databases

34

26Do you have any ideas or suggestions to improve the supplying of research information to generalpractitioners?(Stimulate the respondent to say as much as possible. Tick the items that the respondent names.Make a note of the answers, other than stated. Enumerate the following subjects, when therespondent is finished.)

Information on literature research: medical study, education for general practitioners, postgraduate coursesInformation on literature research in books or librariesA more efficient access to books and journalsLibrary organisationComputersExpertise of library personnelDocument deliveryPublic relations of library and its services

Closing off and thanksThis was the last question. You have answered all questions. Thank you very much for your extensiveanswers. We will use your information, and those of other respondents, to write an article. If you wouldlike to receive a copy of the article, we will send it to you. Do you want to receive the article? Encirclethe answer: YES / NO

Do you have any question or remarks?

The end: stop the tape recorder.Time: .............Topics discussed after the official interview: .........................................................................

35

PERSONAL DATA

Personal detailsName :Position :Employer :Age :

ExperienceYears of experience as a GP :Years of research experience :Number of research publications :(one of the authors, in a peer-reviewed journal)

Present positionWeek task: Number of hours patient care :

Number of hours teaching :Number of hours of research :

Tick the appropriate box

Are you formally affiliated with a university department?yes, namely withno

Are you a teacher?yes; kind of students:no

Do you perform research?yes, and the purpose of it is:no

MembershipsAre you a member of a research or professional association?

Dutch College of General Practitioners (NHG)National Association of General Practitioners (LHV)Other, namely:

36

INFORMATION SOURCES

12Do you use the following information sources to locate literature references, and if so, howfrequently do you use these sources?

1. Browsing in journals or booksyes, are they mostly: Dutch

foreignfifty / fifty

frequency: dailyweeklymonthlyless often than monthly

no

2. Consulting your own bookshelvesyes, frequency: daily

weeklymonthlyless often than monthly

no

3. Consulting the bookshelves of friends or colleaguesyes, frequency: daily


no

4. Other general practitionersyes, frequency: daily


no

5. Medical specialistsyes, frequency: daily


no

37

6. Drug salesmen/womenyes, frequency: daily


no

7. Speakers on medical congresses, workshops, or medical continuing education coursesyes, frequency: daily


no

8. Reference librarian or medical information specialistyes, frequency: daily


no

9. Lists of references in medical journal articlesyes, frequency: daily


no

10. Lists of references in review articlesyes, frequency: daily


no

11. Lists of references in a doctoral thesisyes, frequency: daily


no

12. Lists of references in grey literatureyes, frequency: daily


no

38

13. Lists of references in booksyes, frequency: daily


no

14. Library catalogues (computer catalogue = OPAC; or card index)yes, frequency: daily


no

15. Printed bibliographies, such as the Index Medicus or the NIVEL bibliographies)yes, frequency: daily


no

16. Printed index of Het Nederlands Tijdschrift voor Geneeskundeyes, frequency: daily


no

17. Printed index of the journal Huisarts en Wetenschapyes, frequency: daily


no

18. Electronic bibliographies on CD-ROM, such as Medlineyes, frequency: daily


no

19. Online literature research by a medical librarian, such as Medline through DIMDIyes, frequency: daily


no

39

20. Electronic Current Awareness (journal contents), such as Current Contents, or ReferenceUpdate

yes, frequency: dailyweeklymonthlyless often than monthly

no

21. Printed version of Current Contents (Life Sciences or Clinical Medicine)yes, frequency: daily


no

22. Computerized index of Het Nederlands Tijdschrift voor Geneeskundeyes, frequency: daily


no

23. Computerized index of the journal Huisarts & Wetenschapyes, frequency: daily


no

24. Institutions or organisations, such as patients’ associations, or the Dutch College of GeneralPractitioners (NHG)

yes, frequency: dailyweeklymonthlyless often than monthly

no

25. Have you ever asked somebody or some organisation to do a literature search for you?yes, ... times

I asked:no

26. Hospital Libraryyes, frequency: daily


no

40

27. University Libraryyes, frequency: daily


no

28. Library of a Department of General Practiceyes, frequency: daily


no

29. Library of the Netherlands Institute for Research of Health Care (NIVEL) in Utrechtyes, frequency: daily


no

30. Library of the Dutch College of General Practitioners (NHG) in Utrechtyes, frequency: daily


no

31. Library of the Institute for Research and Support (O & O) for general practitionersyes, frequency: daily


no

32. Other library, namely: ..........yes, frequency: daily


no

HERE ENDS QUESTION 12 ON THE USE OF INFORMATION SOURCES

41

42

4 Information use and needs: a survey among Dutch generalpractitioners

All men by nature desire knowledgeAristotle (384-322 BC)

ABSTRACT

Introduction. With the practice of evidence-medicine in mind, this survey was undertaken with twoobjectives: to study the information use and information needs of Dutch general practitioners for patientcare and continuing medical education.Method. We used a postal questionnaire survey of a random sample of Dutch GPs in 1996. We wereespecially interested in familiarity with evidence-based medicine, an analysis of patient-specificquestions arising in practice, use of information sources, and suggestions for improving access toinformation.Results. Of the 226 GPs (45%) who returned the questionnaire, 45% had heard of evidence-basedmedicine. Of the well-informed GPs, 85% expected a great change in patient management as a result ofevidence-based medicine. The 226 GPs had 6.9 patient-specific questions a week, half relating totherapeutic problems. They most frequently used drug reference and other private books, andconsultation with GP colleagues. Although 93% owned a personal computer at work, they seldom usedit for patient-specific questions. Improvements concerned journal articles, which needed be clearlyorganised and focused on daily practice, and evidence-based reviews, which should be easily accessibleby computer. GPs wanted integration of guidelines and patient health information in the GPs’ HealthInformation System.Conclusion. Electronic information sources have not yet changed the patterns of use of informationsources by Dutch GPs, whose needs are focused on easily accessible electronic information.

Co-authors: CP van Noort, HEP Bosveld, EJ Boerma, B Meyboom-de Jong.

INTRODUCTION

Due to the information explosion physicians have difficulties in finding and obtainingrelevant information.1 Only 30 percent of patient-specific questions arising in practiceare ever pursued.2 An answer to these questions is seldom sought in the medicalliterature,1-6 although 46-54% of primary care physicians’ questions could be answeredusing the medical literature.7,8 Moreover, information from Medline and medicallibraries can have a great impact on medical decision-making and also on the qualityof care,9-13 the length of hospital stay and patient care costs.14

Finding relevant information is essential in practising evidence-based medicine.15 In astudy among general practitioners (GPs) in England, most GPs welcomed evidence-based medicine and agreed that its practice improved patient care.16 GPs seemed tomake more use of printed evidence-based information sources than of electronicdatabases.17

43

For Dutch GPs no quantitative data on information use and needs are available,whereas qualitative data is scarce.18 We therefore studied the information use andinformation needs of Dutch GPs, and collected suggestions for the improvement ofinformation access and use for patient care and continuing medical education.

METHODS

In April 1996 we sent a questionnaire to 500 GPs in the Netherlands, representing6.8% of all registered GPs. They were selected by means of random numbers from anational database of the Netherlands Institute of Primary Health Care (NIVEL).The 14-page-questionnaire consisted of 135 items on GPs’ information use and needs(see appendix), classified in five categories:1. personal and practice characteristics;2. familiarity with evidence-based medicine;3. frequency, topics and pursued answers of patient-specific questions arising in

practice;4. use and characteristics of information sources; and5. suggestions for improvement of information access and use.We sent reminders to non-respondents in June and August 1996.To evaluate the most recent patient-specific question raised in practice, the questionswere coded with the International Classification of Primary Care (ICPC), andcompared with the morbidity registered during more than 200.000 episodes in 14Dutch general practices.19

To measure the use of information sources, the GPs rated their use of 11 presentedsources on a six-point scale ranging from "never" (1) to "daily" (6). We determinedthe influence of the GPs’ age; type of practice (solo, duo, group or health care centre);and the GPs’ involvement in teaching and research on the use of three groups ofinformation sources and the sources individually. The information sources were:printed sources (drug reference books, private books and journals, library books andjournals, journal articles received from others); human resources (consultation withGPs and specialists); and electronic sources (CD-ROM, online databases, and the Inter-net). We analysed the data with the Spearman rank correlation (rs), the Kruskal-Wallistest, and the Mann-Whitney U test.To determine the time spent on information sources for continuing medical education,the GPs answered open questions concerning 13 information sources.To assess the importance of 10 source characteristics for choosing an informationsource, the GPs used a three-point scale ranging from "not important" (1) to "veryimportant" (3). The characteristics were six cost characteristics: travelling time, time tosearch, ease of searching, chance of success, financial costs, and understandability; andfour benefit characteristics: reliability, currency, usefulness, and completeness.6 Inaddition, the GPs rated the quality of individual sources in relation to 10characteristics on a three-point scale: "needs improvement" (1), "acceptable" (2) and"very good" (3).To collect suggestions to improve access to medical information, the GPs rated theiropinion on 20 suggestions on a four-points scale from "not desirable" (1) to "very

44

desirable" (4). Additionally, the GPs could bring forward their own ideas in a free textsection.We used SPSS 6.1 for data analysis, and the Students t-test to compare therespondents and the general Dutch GP population.

RESULTS

Of the 500 questionnaires, 226 (45%) were eligible for analysis. The mean age of therespondents was 45.1 years (SD=7.1, range=31-72) and 18.6% were women. Of therespondents 48% worked in a solo practice, 33% in a duo practice, 11% in a grouppractice, and 8% in a health care centre. No significant differences were found incomparison with the Dutch GP population.Of the 222 respondents (not all respondents answered this question), 45% had heard ofevidence-based medicine. More than half of them (56%) said they knew exactly whatevidence-based medicine meant. Of these well informed GPs, 85% expected changesin patient management if evidence-based medicine were to be introduced.The GPs said that questions for which they need answers arose in practice 6.9 times aweek (SD=6.5, range=0.04-50). When we asked to formulate the most recent question,209 GPs recorded 217 questions. More than half of the questions (50.5%) dealt withtherapeutic problems, and a quarter (24.8%) with diagnostic problems. When wecompared the questions raised with the morbidity we could conclude that 6.9 timesmore questions were raised on blood problems than the morbidity figures wouldsuggest, and ear conditions presented only one tenth of the problems that the morbidityfigures would suggest (table 1).Of the most recent questions, an immediate answer was pursued in 76% of the cases,and in 85% a (partial) answer was found. Only 62% of the GPs who did not search forthe answer intended to do so at a later moment, and 11% would wait expecting to findthe answer by chance.

45

Table 1. Relation between 217 most recent patient-specific questions and patient episodes* coded withthe International Classification of Primary Care (ICPC).

Chapter % questions % patientepisodes

% questions/% patientepisodes

A. General 21.7 9.5 2.3

B. Blood 8.3 1.2 6.9

D. Digestive 8.7 6.6 1.3

F. Eye 3.2 3.1 1.0

H. Ear 0.5 5.3 0.1

K. Circulatory 11.1 4.1 2.7

L. Musculoskeletal 9.2 14.6 0.6

N. Neurological 5.5 2.8 2.0

P. Psychological 2.8 4.5 0.6

R. Respiratory 4.1 17.0 0.2

S. Skin 7.8 14.5 0.5

T. Metabolic, Endocrine, Nutritional 4.7 1.4 3.4

U. Urinary 0.9 3.8 0.2

W. Pregnancy. Childbearing, Family planning 5.0 2.9 1.7

X. Female genital 3.7 6.0 0.6

Y. Male genital 2.8 1.0 2.8

Z. Social 0.0 2.2 0.0

* Figures from 1993-1996; Werf GT van der, et al19 (0.3% was filled out incorrectly)

In 1996, 93% of the GPs owned a pc at work; 64% owned a modem; 33% owned aCD-ROM player; and 31% owned software to search medical literature, though only9% subscribed to an Internet provider. Nineteen percent indicated they wereconsidering subscribing to an Internet provider in the next six months.The most frequently used information sources for patient care were drug referencebooks, other private books, and consultations with GPs and specialists (table 2).Several factors influenced the use of information sources positively. To begin with, wefound a significant influence (P<0.05) of the number of GPs working in a practice(solo, duo, or group practice, or health care centre) on the use of human informationresources, especially GP colleagues, and on the use of private journal articles andlibrary books. We found a significant influence of being involved in research on theuse of the printed information sources, especially drug reference books, other privatebooks, and private journal articles. Similarly, we found a significant influence of beinginvolved in research on the use of human resources as a group, especially GPcolleagues. Lastly, we found a significant influence of being involved in teaching onthe use of human resources.Except for journal articles received from others, we found no significant influence ofage on the use of printed or electronic sources, nor on the use of human resources.Remarkably, we also found no significant influence of the type of practice, nor of GP

46

involvement in teaching or research, on the use of any of the electronic informationsources.

Table 2. Frequency of use of 11 information sources for patient care by 226 Dutch GPs. Medianunderlined. Values are percentages.

six-point scale

(1) (2) (3) (4) (5) (6)

Source never 1-3times a

year

1-3times aquarter

1-3times amonth

1-4times aweek

everyday

Printed source

Drug reference book 2 2 7 8 42 39

Private book 1 4 19 29 38 10

Book from library 79 15 5 1 0 0

Journal article from privatecollection

13 38 23 20 6 0

Journal article from library 83 14 2 1 0 0

Journal article received from others 63 32 5 0 0 0

Human resource

Specialist consultation 0 7 24 45 24 0

GP consultation 2 17 14 29 32 7

Electronic source

CD-ROM 93 4 0 1 1 0

Online databases 96 4 0 0 0 0

Internet 97 2 0 1 0 0

For continuing education, the mean time spent on reading printed material was 9.5hours a month (SD=8.75, range=0-40), and on electronic information 0.15 hours amonth (SD=1.4, range=0-20). Because the electronic information sources were used byonly 25 of the 223 respondents, we calculated the mean time spent on electronicsources by users only. So, after omitting the 197 non-users and one extreme user whospent 20 hours a month on electronic information sources, 25 users were left. Theyspent mean 0.5 hours a month (SD=0.8, range=0.04-4) on electronic informationsources, and 5.9 hours a month (SD=5.9, range=0-47) on continuing education classes,congresses, and meetings for GP trainers.Of the individual sources, not only did private journals, continuing education classesand congresses take most of the GPs’ time, they also were the favourite informationsources used by almost all the GPs (table 3).

47

Table 3. Mean time spent on 13 information sources for continuing education by all 224 respondents*and by users only.

Information sourceTime spent by all

respondents(hours a month)

(SD)

Non-users(%)

Time spent byusers only

(hours a month)(SD)

Private journals 6.6 (5.2) 1 6.6 (5.2)

Continuing education classes and congresses 5.1 (5.7) 3 5.3 (5.7)

Private books 2.4 (3.8) 9 2.6 (3.9)

Talks in GP practices 2.0 (7.1) 19 2.5 (7.8)

Talks in hospital 1.7 (1.9) 18 2.1 (2.0)

Pharmaceutical representatives 1.0 (1.9) 43 1.8 (2.3)

Meetings for GP trainers 0.8 (1.8) 76 3.2 (2.3)

Articles from others 0.5 (1.7) 61 1.4 (2.5)

Journals in the library 0.3 (2.3) 91 2.8 (7.2)

CD-ROMs 0.1 (1.3) 93 1.7 (5.1)

Library books 0.1 (0.3) 90 0.9 (0.4)

Internet 0.03 (0.28) 96 0.7 (1.4)

Online databases 0.01 (0.06) 96 0.2 (0.3)

* Two respondents did not answer all the questions.

The most important characteristics for choosing an information source were itsreliability, usefulness, and ease of searching, which scored from 2.89 (reliability,SD=0.35) to 2.76 (ease of searching, SD=0.47) on a three-point scale (1=notimportant, 3=very important). Financial costs were considered as least important,scoring 2.09 (SD=0.59).

Table 4 shows that the GPs wanted improvements in the Internet on fivecharacteristics: ease of searching, financial costs, time to search, chance of success andusefulness. The pharmaceutical representative needed improvements on reliability,completeness, usefulness, and chance of success; the library on time to search,travelling time and ease of searching.

Sixty four percent of the respondents wanted improvements in obtaining information.The most desirable improvements were:1. Better-organised and more practical oriented journal articles;2. More high quality reviews which were evidence-based;3. Computer-accessible abstracts of most important articles;4. Computer-accessible medical questions and answers;5. More medical guidelines.In the free text section, 70 respondents included additional comments. More than halfof them were on information overload and lack of time. Many other suggestions wereon integration of medical information such as protocols and patient education

48

information into GPs’ computer Health Information Systems (HIS), easily accessibleduring patient episodes.

Table 4. Characteristics of six information sources which needed improvement (with improvementscores lower than 2.0 on a three-point scale*).

Information source Characteristic Score (SD) Number ofrespondents(users) (n)

Internet Ease of searchingFinancial costsTime to searchChance of successUsefulness

1.6 (0.8)1.6 (0.5)1.7 (0.8)1.7 (0.8)1.8 (0.8)

101010109

Online database Financial costs 1.8 (0.7) 12

CD-ROM Chance of success 1.9 (0.7) 20

Library Time to searchTravelling timeEase of searching

1.4 (0.6)1.5 (0.7)1.6 (0.6)

656363

Pharmaceutical representative ReliabilityCompletenessUsefulnessChance of success

1.4 (0.6)1.4 (0.6)1.6 (0.6)1.6 (0.6)

138137139144

Continuing education classes and congresses Travelling time 1.9 (0.6) 194

* 1 = needs improvement; 2 = acceptable; 3 = very good. The information sources with a score of 2.0or higher were consultation with GPs, consultation with specialists, books and journals from privatecollection, journal articles sent by others, talks in a GP practice, and talks in the hospital.

DISCUSSION

This study shows that, in 1996, the information use by Dutch GPs involved mostlybooks and colleagues. GPs used electronic information little, although their reportedinformation needs focused on electronic information.Computer assisted searching of medical databases had been proposed as a solution tothe inaccessibility of the medical literature,20 but as our findings show, in 1996 thishad not been yet realised. Although 93% of all Dutch GPs owned a personal computer,they seldom used it for answering patient-specific questions. This was true also of theGPs who were involved in research. Traditional information sources, like books andcolleagues, were the preferred sources, as reported in earlier studies.20-22 Additionally,the most time-consuming continuing medical education (CME) activities - reading thejournal literature and attending CME classes - were still not associated with thecomputer. Our finding that only 9% of the GPs subscribed to an Internet provider wasconfirmed by a small interview survey among 27 GPs in a region of the Netherlandscarried out in 1997.23 Because we did not find a relation between on the one hand theGP’s age, type of practice, being involved in research or teaching, and on the other

49

hand the use of electronic information sources, we cannot define the group of noviceusers of electronic information sources.

The information needs of our respondents arising from practice averaged almost sevenpatient-specific questions a week. Half of them were on therapy, and a quarter ondiagnosis. In the literature the rates of questions per 10 patients seen by primary carephysicians were 0.7,3 5.7,24 and 18.5.25 This variation was due to different definitionsand research methods (observation, interviews, and reviewing videotapedconsultations). Because we do not know how many patients our respondents had seenin a week, we cannot compare these figures. Our finding that considerably morequestions were raised on therapy than on diagnosis is found in many other stu-dies,3,4,9,26 although Timpka found more questions on diagnosis.25

Of all patient-specific questions, 76% were pursued and 85% of these pursuedquestions were answered; this meant that 65% of all recorded questions, pursued ornot, were answered. With Barrie27 and Ely,3 who found 79% and 88% of the questionsanswered, we could not confirm Covell’s findings that only 30% of the physicians’needs were met.2

In the analysis of suggestions for improving information access, the major topicsidentified were the organisation, focus and quality of journal articles including reviews,and the computer-accessibility of medical knowledge. There are several possibilities toimplement these suggestions.The Cochrane Library, readily available on CD-ROM and the Internet,28 has beennamed as the first source for the best evidence.29 It includes regularly updated full-textsystematic reviews on the effects of health care with critical appraisal by experts. ForGPs there are some limitations. Of the 522 completed systematic reviews in March1999, most involve clinical trials on drug treatments. In general practice however, animportant part of medical decision-making concerns diagnostic problems. Moreover,many problems in general practice can be answered only by qualitative means, or, forethical or logistical reasons, by quantitative observational research.30

A second electronic evidence-based source for clinical information is Best Evidence,31

which includes the journals Evidence-Based Medicine and the ACP Journal Club.Another approach to implementing the suggestions of the GPs is the computerdatabase of clinical questions and answers successfully built by Ely.32 This could beimportant for practising evidence-based medicine, for continuing medical educationand for medical decision-making.Our last recommendation would be stimulating software designers to study GPs’information needs, so they can appropriately integrate guidelines and patient educationinformation into GPs’ Health Information System.Although the above suggested electronic sources could be very useful, availabilityalone will not be sufficient. Reasons why GPs still prefer printed clinical effectivenesssources to electronic databases were lack of awareness and of educational andtechnical support.18 These problems need to be tackled first.

We need to interpret our results with caution, because of some limitations. Firstly, thebias in the GPs’ self-reported choices may have distorted the validity of the study’s

50

findings.2 Secondly, we studied only a part of the GPs needs, namely their recognisedneeds. Finally, our response of 45% was not high, though this is not unusual with busyGPs.33 Nevertheless, the questionnaire should have been shorter, so as not to contributeto the information overload of busy GPs.

In summary, this study shows that, in 1996, the information use by Dutch GPsinvolved mostly books and colleagues, with not much recourse to electronicinformation sources. We would recommend that the electronic accessibility ofevidence-based reviews be improved, the computer database with clinical questions beconstructed, and that guidelines and patient education information should be integratedinto the GPs’ Health Information System.

AcknowledgementsWe thank Lammert Postma for his advice on the questionnaire design.

REFERENCES

1 Williamson JW, German PS, Weiss R, Skinner EA, Bowes F. Health science informationmanagement and continuing education of physicians: a survey of U.S. primary care practitionersand their opinion leaders. Ann Intern Med 1989;110:151-60.

2 Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met?Ann Intern Med 1985;103:596-9.

3 Ely JW, Burch RJ, Vinson DC. The information needs of family physicians: case-specific clinicalquestions. J Fam Pract 1992;35:265-9.

4 Timpka T, Ekström M, Bjurulf P. Information needs and information seeking behaviour inprimary health care. Scand J Prim Health Care 1989;7:105-9.

5 Connelly DP, Rich EC, Curley SP, Kelly JT. Knowledge resource preferences of familyphysicians. J Fam Pract 1990;30:353-9.

6 Curley SP, Connelly DP, Rich EC. Physicians’ use of medical knowledge resources: preliminarytheoretical framework and findings. Med Decis Making 1990;10:231-41.

7 Gorman PN, Ash J, Wykoff L. Can primary care physicians’ questions be answered using themedical journal literature? Bull Med Libr Assoc 1994;82:140-6.

8 Chambliss ML, Conley J. Answering clinical questions. J Fam Pract 1996;43:140-4.9 King DN. The contribution of hospital library information services to clinical care: a study in

eight hospitals. Bull Med Libr Assoc 1987;75:291-301.10 Lindberg DAB, Siegel ER, Rapp BA, Wallingford KT, Wilson SR. Use of MEDLINE by

physicians for clinical problem solving. JAMA 1993;269:3124-9.11 Marshall JG. The impact of the hospital library on clinical decision making: the Rochester study.

Bull Med Libr Assoc 1992;80:169-78.12 Urquhart CJ, Hepworth JB. Comparing and using assessments of the value of information to

clinical decision-making. Bull Med Libr Assoc 1996;84:482-9.13 Wood F, Wright P, Wilson T. The impact of information use on decision making by general

medical practitioners [British Library R&D Report]. Boston Spa: British Library; 1995.14 Klein MS, Ross FV, Adams DL, Gilbert CM. Effect of online literature searching on length of

stay and patient care costs. Acad Med 1994;69:489-95.15 Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching

the practice of medicine. JAMA 1992;268:2420-5.16 McColl A, Smith H, White P, Field J. General practitioners’ perceptions of the route to evidence

based medicine: a questionnaire survey. BMJ 1998;316:361-5.

51

17 Prescott K, Lloyd M, Douglas HR, Haines A, Humphrey C, Rosenthal J, Watt I. Promotingclinically effective practice: general practitioners’ awareness of sources of research evidence. FamPract 1997;14:320-3.

18 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Management of bibliographic information byDutch researchers in general practice. Fam Pract 1997;14:69-72.

19 Werf GT van der, Smith RJA, Stewart RE, Meyboom-de Jong B. Spiegel op de huisarts: overregistratie van ziekte, medicatie en verwijzingen in de geautomatiseerde huisartspraktijk. [View onthe general practitioner: on the registration of diseases, medication, and referrals in thecomputerised general practice office. In Dutch]. Groningen: Morbidity Registration NetworkGroningen; 1998. p. 124-36.

20 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Use of information sources by familyphysicians: a literature survey. Bull Med Libr Assoc 1995;83:85-90.

21 Haug JD. Physicians’ preferences for information sources: a meta-analytic study. Bull Med LibrAssoc 1997;85:223-32.

22 Cullen R. The medical specialist: information gateway or gatekeeper for the family practitioner.Bull Med Libr Assoc 1997;85:348-55.

23 Schimmel M. De huisarts in de informatie-maatschappij: een onderzoek naar de informatiebehoefte en -voorziening van huisartsen in de Achterhoek, Liemers en Zuid-Twente. [The GP inthe information society: a survey on the information needs of and information services for GPs inthe Achterhoek, Liemers and South-Twente. In Dutch]. Report of the Hogeschool IJselland,School for Information Studies]. Aalten: Van Schijndel BibliotheekmanagementInformatiebemiddeling; 1998. p. 17.

24 Gorman PN, Helfand M. Information seeking in primary care: how physicians choose whichclinical questions to pursue and which to leave unanswered. Med Decis Making 1995;15:113-9.

25 Timpka T, Arborelius E. The GP’s dilemmas: a study of knowledge need and use during healthcare consultations. Methods Inf Med 1990;29:23-9.

26 Dee C, Blazek R. Information needs of the rural physician: a descriptive study. Bull Med LibrAssoc 1993;81:259-64.

27 Barrie AR, Ward AM. Questioning behaviour in general practice: a pragmatic study. BMJ1997;315:1512-5.

28 Cochrane Library [database on disk, CD-ROM and Internet]. Cochrane Collaboration; Issue 2,March 1999. Oxford: Update Software; 1999. Updated quarterly.

29 Straus SE, Sackett DL. Using research findings in clinical practice. BMJ 1998;317:339-42.30 Pringle M, Churchill R. Randomised controlled trials in general practice: gold standard or fool’s

gold? BMJ 1995;311:1382-3.31 Best Evidence [database on CD-ROM]. Philadelphia: American College of Physicians; 1996.32 Ely JW, Osheroff JA, Ferguson KJ, Chambliss ML, Vinson DC, Moore JL. Lifelong self-directed

learning using a computer database of clinical questions. J Fam Pract 1997;45:382-8.33 Kaner EFS, Haighton CA, McAvoy BR. "So much post, so busy with practice - so, no time!": a

telephone survey of general practitioners’ reasons for not participating in postal questionnairesurveys. Br J Gen Pract 1998;48:1067-9.

52

APPENDIXQuestionnaire "Searching for medical information by general practitioners".

: tick the appropriate box* : cross out whichever is not applicable.... : fill in number of text

A. PERSONAL CHARACTERISTICS AND GENERAL QUESTIONS

1. What is your age?.............. years

2. Are you:malefemale

3. Is your medical practice:a one person practicea two person practicea group practicea health care centreI do not have a medical practiceother, namely: ................................................................................................

4. How much of your time do you actually spend on:a. Your medical practice: ............ hours a week/month/year*

b. Research: ............ hours a week/month/year*

c. Medical teaching: ............ hours a week/month/year*

5. Have you been employed by or are you a member of one of the following organisations ingeneral practice (you may tick more than one answer):

The Dutch College of General Practitioners (NHG)National Association of General Practitioners (LHV)An organisation or study group for the promotion of professionalisationDepartment of General PracticeNone of the aboveOther: ..........................................................................................................

6. Have you ever published in a medical journal?yes If so, in which journal: .................................................................no

7a. Have you ever heard of evidence-based medicine?yes, continue with question 7bno, continue with question 8

53

7b. Do you think you know exactly what evidence-based medicine implies?yes, continue with question 7cno, continue with question 8

7c. Do you expect that there would be changes in your daily activities if you practised evidence-based medicine?

yesno

8. Which of the following medical journals do you read regularly?Dutch general journals:

Nederlands Tijdschrift voor GeneeskundeMedisch ContactOther journals, such as: .....................................................................................

General journals in other languages:British Medical JournalLancetOther journals, such as: .....................................................................................

Dutch journals for family practice:Huisarts en WetenschapHuisarts NuTijdschrift voor Huisartsgeneeskunde (formerly The Practitioner)Other journals, such as: .....................................................................................

Journals in family practice in other languages:Family PracticeBritish Journal of General PracticeEuropean Journal of Family PracticeOther journals, such as: .....................................................................................

9. Do you own any of the following items either at home or at work? If not, are you consideringpurchase in 1996?

own purchase in 1996

yes no yes nocomputer at homecomputer at workfaxfax-modemmodemCD-ROM playerCD-i playersoftware to search the medical literatureCDs or disk with medical knowledgesubscription to Internet

54

B. SEARCHING FOR INFORMATION FOR PATIENT CARE

During or after office hours undoubtedly medical questions will be raised as a result of patientencounters. Sometimes you will search for the answer, sometimes you will not (for example becauseyou do not have time for it). Sometimes you will find an answer quickly, for example in thePharmacotherapeutic Directory (Farmacotherapeutisch Kompas) or after consulting a colleague;sometimes you will not find the answer, although you have searched for a long time. This part of thequestionnaire includes the medical questions you are confronted with in patient care.

10. Please make a note of the most recent question you remember.

........................................................................................................................

11. Did you search for the answer to this question?yes, continue with question 12ano, continue with question 12b

12a. Did you find the answer to this question?yes, continue with question 13no, continue with question 13partly, continue with question 13

12b. Do you still have the intention to search for the answer?yesnoI will wait until I come across the answer by chance

13. Assume you are confronted with questions from patient care. How many times do you search foranswers on such questions? Include answers which you look up in the PharmacotherapeuticDirectory (Farmacotherapeutisch Kompas), or which you obtain from consulting a colleague........... times a week/month/year*

14. Please tick the appropriate box for the answer to the following question: How many times do youlook for the answer in the following information sources for patient care?

never 1-3× 1-3× 1-3× 1-4× dailya year a quarter a month a week

Consulting a GP colleagueConsulting a specialistPharmacotherapeutic book from your own

private libraryOther book from your private libraryBook from a library not your

private libraryJournal article from an institutional libraryJournal article sent to you by othersCD-ROMOnline databaseInternet

55

C. SEARCHING FOR INFORMATION FOR RESEARCH, TALKS AND TEACHING

This part of the questionnaire relates to when you want information for research, a talk in your localgroup, or for teaching.

15a. How frequently do you purposefully seek information for writing articles and research?.......... times a week/month/year*

15b. How frequently do you purposefully seek information for teaching?.......... times a week/month/year*

15c. How frequently do you purposefully seek information for preparing a talk?.......... times a week/month/year*

16. How frequently do you visit a medical library?.......... times a week/month/year*

17. How frequently do you use the following information sources in order to collect information foran article, research, a talk for your GP colleagues, or for teaching. Tick the appropriate box.

never 1-3× 1-3× 1-3× 1-4× dailya year a quarter a month a week

Consulting a GP colleagueConsulting a specialistPharmacotherapeutic book from your own

private libraryOther book from your private libraryBook from a library not your

private libraryJournal article from an institutional libraryJournal article sent to you by othersCD-ROMOnline databaseInternet

56

D. SEARCHING FOR INFORMATION FOR CONTINUING MEDICAL EDUCATION

In this part of the questionnaire we consider the use of information sources to keep up to date with newdevelopments.Of course, you have been keeping up to date by discussing patients with colleagues, or by reading booksafter you have been confronted with certain patient questions. The following questions do not relate tothese specific patient care questions, but relate to patient care questions in general.

18. How much time do you spend on the following forms of medical continuing education?

hours a month............. Reading books from own private library............. Reading books from other libraries............. Reading own medical journals............. Reading medical journals in the library............. Reading medical journal articles which were sent to you by others............. Attending talks by GP colleagues in local groups............. Attending talks in the hospital............. Receiving a pharmaceutical representative in your own practice

hours a year............. Reading CD-ROMs............. Reading online databases............. Reading Internet

days a year............. Attending national continuing medical education courses and national general

practice meetings............. Attending meetings for general practitioner trainers

E. THE CHOICE BETWEEN INFORMATION SOURCES

19. To what extent do the following items play a role in your choice of an information source?

not important somewhat important very important

1. Travelling timeThe time it takes to reach theinformation source

2. Time to searchThe time it takes to extract the informationfrom the information source

3. Ease of searchingThe ease of getting information froman information source

4. The chance of successDid you find what you was looking forwhen you searched the information source before

57

not important somewhat important very important

5. Financial costsThe money you have to pay to usethe information source

6. UnderstandabilityWas the information presented in aclear way, so that it did nottake much effort to understand it

7. ReliabilityTo what extent the information iscorrect and generally accepted

8. CurrencyHow current is the information

9. UsefulnessTo what extent can the informationbe incorporated in daily practice orbe useful for other activities

10. CompletenessTo what extent the source presentsall available information on thisspecial subject

11. Other, namely:

..........................

20a. Tick the appropriate box on the left.

I have used this source travelling time to ease of the chance financialat least once time search searching of success costs

Consulting a GP colleagueConsulting a specialistBooks and journals from

private libraryInstitutional libraryMedical journal articles

sent to you by a third partyCD-ROMOnline databaseInternetAttending talks by GP

colleagues in local groupsAttending talk in the hospitalNational continuing medical

education courses and nationalgeneral practice meetings

A pharmaceutical representative

58

20b. Please answer this question if you have used an information source at least once. There areseveral factors that are important when choosing an information source. In the above scheme youcan find these factors at the top on the right. Please give your opinion on these factors:1 = needs improvement 2 = acceptable 3 = very good

Explanation of the characteristics is given in question 19.

21a. Tick the appropriate box on the left.

I have used this source reliability usefulnessat least once understandability currency completeness

Consulting a GP colleagueConsulting a specialistBooks and journals from

private libraryInstitutional libraryMedical journal articles

sent to you by a third partyCD-ROMOnline databaseInternetAttending talks by GP

colleagues in local groupsAttending talk in the hospitalNational continuing medical

education courses and nationalgeneral practice meetings

A pharmaceutical representative

21b. Please answer this question if you have used an information source at least once. There areseveral factors that are important when choosing an information source. In the above scheme youcan find these factors at the top on the right. Please give your opinion on these factors in theabove scheme:1 = needs improvement 2 = acceptable 3 = very good

Explanation of the characteristics is given in question 19.

F. THE NEED FOR IMPROVEMENTS AND POSSIBLE IMPROVEMENTS

22a. What do you think of your possibilities for finding information for patient care?very inadequateinadequateadequategoodvery good

59

22b. What do yo think of your possibilities for finding information for writing articles and performingresearch?

very inadequateinadequateadequategoodvery good

22c. What do you think of your possibilities for finding information for teaching?very inadequateinadequateadequategoodvery good

22d. What do you think of your possibilities for finding information for presenting talks?very inadequateinadequateadequategoodvery good

22e. What do you think of your possibilities for finding information for continuing education?very inadequateinadequateadequategoodvery good

23a. Are any improvements needed for finding information for patient care?no, not neededyes, somewhat neededyes, neededyes, needed very much

23b. Are any improvements needed for finding information for writing articles and performingresearch?

no, not neededyes, somewhat neededyes, neededyes, needed very much

23c. Are any improvements needed for finding information for teaching?no, not neededyes, somewhat neededyes, neededyes, needed very much

60

23d. Do your own possibilities for finding information for presenting talks need any improvement?no, not neededyes, somewhat neededyes, neededyes, needed very much

23e. Do your own possibilities for finding information for continuing education need anyimprovement?

no, not neededyes, somewhat neededyes, neededyes, needed very much

24. We want your opinion on the following suggestions to improve access to information for GPs.Some of the suggestions are already reality.

not somewhat desirable verydesirable desirable desirable

1. More courses for searching,selecting, critical appraisal andstoring literature

2. More reviews which are evidence-based

3. Journal articles that are betterorganised and are focused on dailypatient care

4. More medical guidelines needto be developed

5. All books and medical articlesneed to be published unabridgedaccessible by computer

6. Selections of the most importantjournal articles need to beaccessible unabridged by computer

7. Abstracts of the most importantarticles need to be accessibleby computer

8. Practical answers on medicalquestions need to be accessibleby computer

9. More compulsory medical educationfor GPs

61

not somewhat desirable verydesirable desirable desirable

10. A medical inquiry service withcommunications by fax

11. A telephone inquiry service formedical questions needs tobe set up

12. Librarians to perform literaturesearches for patient care,and send the article to your home

13. Librarians to perform literaturesearches for your research, andsend the article to your home

14. Librarians to perform literaturesearches for your teaching, andsend the article to your home

15. Librarians to perform literaturesearches for your talks, andsend the article to your home

16. Journal articles according to yourinterests sent to your home

17. Printed continuing education at home

18. Computer-assisted teaching at home

19. More contacts and meetings withcolleagues

20. Better disclosure of medicalliterature on the Internet

25. Do you have other suggestions to improve information access for GPs for patient care?

26. Do you have other suggestions to improve information access for GPs when they have to give talks,to perform research or to teach?

27. Do you have other suggestions to improve information access for GPs to keep up to date with thenewest developments?

T H E E N D : Thank you so much for answering these questions.

Prof. Betty Meyboom-de Jong, MD, PhDAnita Verhoeven, MD, and Carla van Noort, MD, information specialistsDepartment of General Practice, University of Groningen

62

5 A model for retrieval quality of search queries in three medicalbibliographic sources

The real problem is not whether machines think, but whether men doB.F. Skinner, American sociologist, 1969

ABSTRACT

Introduction. For a continuing medical education course in literature searching an evaluation tool wasneeded. Therefore, we developed a model to assess the retrieval quality of search queries performed byDutch general practitioners using the printed Index Medicus, Medline on CD-ROM, and Medlinethrough Grateful Med.Method. Four search queries related to general practice were formulated. The potential relevant citationsselected by the course instructor and by the 103 participants in the course were presented to threejudges, who evaluated them for relevance and quality. The evaluation of quality was based on journalranking, research design, and publication type. Relevant individual citations received a citation qualityscore from 1 (low) to 4 (high). The overall search quality was expressed in a formula which includedthe individual citation quality score of the selected and missed relevant citations, and the number ofselected non-relevant citations. As outcome measures we used the number and quality of the relevantcitations, and the agreement between the judges.Results. Out of 864 citations, 139 were assessed as relevant. Of these 44 citations received an individualcitation quality score of 1; 76 of 2; 19 of 3, and none of 4. The level of agreement between the judgeswas 68% for the relevant citations, and 88% for the non-relevant citations.Conclusion. We describe a model for retrieval quality of search queries based not only on the relevancebut also on the quality of the citations retrieved. With adaptation, this model could be generalized toother professional users, and to other bibliographic sources.

Accepted for publication in Health Libraries Review.Co-authors: PM Boendermaker, EJ Boerma, B Meyboom-de Jong.

INTRODUCTION

Evidence-based medicine requires new skills of physicians including efficient literaturesearching.1,2 Therefore, we have introduced literature searching in our continuingmedical education courses for general practitioners (GPs). In this study we describe thedevelopment of a model that could be used as a reference point in comparing andevaluating searches in bibliographic sources.The approach most used to assess effectiveness of searchers and their searching hasbeen by assessing the quantity of articles retrieved that are relevant to the searchtopic.3 The quantity of articles retrieved has been measured with recall (or,"sensitivity"), and precision (or, "specificity", or, more correctly, "positive predictivevalue").3-6 However, these measures are based only on the number of selected relevantcitations, and take no account of the quality of the relevant citations. We can assumethat a relevant review article in a journal with a high impact factor will probably have

63

more impact on the user than a case report in a journal with a low impact factor.Therefore, there is a need for a measure based not only on the number of relevant andnon-relevant citations found, but also on measures for the ranking of the journal, theresearch design and the publication type.In developing such a measure the concept of relevance is important. No consensusexists on its meaning.7,8 Harter9 stated that relevance is multidimensional, bringingtogether psychological, situational and topical aspects assessed by the person with theinformation need. Relevance judgments are affected by four categories of factors:10

user’s perceptions concerning elements of the citation, such as quality of thejournal, style of the title of the article, and status of the author;internal context, such as experience with journals and authors, and awareness ofpublished literature;external context, such as goal of the search and stage of research - the situationalrelevance;problem context, or the characteristics of the user’s information problem, e.g.whether the citation cast a new or different light on the problem.

For the purpose of our model, we decided to use a simple measure of topicalrelevance, and three measures of the quality of the retrieved citations. This did notinclude the outcome-oriented relevance, which measures the use that is made of thesearch results.11

The aim of this article is to describe the development of a model to evaluate theretrieval quality of search queries on more than quantitative measures, performed byDutch GPs in medical bibliographic sources.

METHODS

The model was developed as part of a 1-day continuing medical education course inliterature searching for Dutch GPs. The course was offered from 1994 to 1997, and intotal 103 GPs took part. The GPs were randomized blockwise in three groups and weassigned each group to one of the following bibliographic sources: the 1992 printedIndex Medicus, the 1992 Medline on CD-ROM (Silverplatter version 3.1), and Medline(citations with publication year 1992) through Grateful Med (version 6.0) providedonline by the Karolinska Institute in Sweden. We formulated four search queries ontopics with varying complexity related to general practice, namely haemorrhoids,sudden infant death, the telephone, and the gatekeeper (table 1). The topics of thesearch queries were collected from discussions with practising GPs.

64

Table 1. The four search queries for a course in literature searching for Dutch GPs.

I.You have been invited to give a talk on haemorrhoids to a group of colleagues in your town. Tocollect information, you want to do a literature search using the Index Medicus/Medline on CD-ROM/Medline through Grateful Med of the year 1992.

II.You want to keep up with the latest developments in the prevention of sudden infant death.You want to do a literature search using the Index Medicus/Medline on CD-ROM/Medline throughGrateful Med of the year 1992.

III.The editor of the Dutch journal Huisarts & Wetenschap (General Practitioner & Medical Science)has asked you to submit an article on the use of the telephone in the physician’s office. You regardthis request as a challenge and you decide to do a literature search using the Index Medicus/Medlineon CD-ROM/Medline through Grateful Med of the year 1992.

IV.You are supervising a GP trainee in your practice. She wants to discuss the gatekeeper role of theGP. You want to be well prepared for this discussion, so you decide to do a literature search on thistopic using the Index Medicus/Medline on CD-ROM/Medline through Grateful Med of the year1992.

After 2 hours of instruction, the GPs each searched one of the three sources forrelevant citations. First they retrieved a batch of citations and then decided which toselect as relevant for the four search queries.To evaluate the number and quality of the selected citations we developed a goldstandard. Three judges decided which of the citations were relevant, and then theyassessed the quality level of these relevant citations. With this information we couldassess the retrieval quality of search queries in the form of an overall search qualityscore. The relevance judges were all GPs by training: a professor in general practice(judge A), a trainer of GPs (judge B), and an experienced information specialist (judgeC).

Relevance of citationsIt was impossible to consider all available citations from the three sources, thereforethe judges evaluated all citations selected as relevant by the 103 participants on thecourse as well as the citations selected by the course instructor. The definition ofrelevance used by the judges was "whether the journal article represented by thecitation would give relevant information for answering the question that prompted thesearch", based upon the title, the language, and if available, the abstract and theMedical Subject Headings. Relevant languages were English, German, French andDutch. A letter was only relevant if it was published in a general practice journal, orin a general medical journal, such as the Lancet or the New England Journal ofMedicine. Editorials were considered as articles. Disagreements between the judgeswere resolved by discussion. We then calculated the number of citations the individualjudge’s performances had in common with the gold standard of the consensus result in

65

two ways: 1) agreement on the relevant citations (sensitivity), and 2) agreement on thenon-relevant citations (specificity).

Quality assessment of citationsWe chose three formal criteria: journal ranking, research design, and publication type.In the literature, peer review and impact factors have been used for assessing theimportance of medical journals.12 Because the peer review process was not describedexplicitly in most of the journals it was not considered a workable method for ourstudy. We therefore relied on the impact factor of the journal. The impact factor is ameasure of the frequency with which the "average article" in a journal has been citedin a particular year,13 and therefore gives a rough indication of relative influence of thejournal. In addition, we used the impact factor for pragmatic reasons. Researchinstitutions all over the world, including those receiving Dutch governmental researchfunds, and Dutch medical schools, use the impact factor as a measure for the scientificoutput of researchers.We used the most recent available data from the Journal Citation Reports database(science and social science editions).14,15 If the journal appeared in both indexes, theScience Citation Index prevailed. Because the Citation Indexes have a strong US bias,we also used the Dutch list of Additional Scientific Journals for Health SciencesResearch of the Royal Netherlands Academy of Arts and Sciences.16 This list wascompiled by members of the Dutch medical profession, and it covered 106 journalsrelevant for Dutch GPs (see appendix).When a relevant citation was published in a journal that had an impact factor abovethe median for its subject category, or if it was published in a journal covered by theDutch list of Additional Scientific Journals for Health Sciences Research, it receivedone point.The second quality criterion was research design. The citation received one point if thetitle, abstract, or publication type field contained one of the following components ofresearch design: clinical trial, placebo-controlled study, double-blind study, follow-upor prospective study.The final quality criterion was publication type. If the citation was a review it receivedone point, because a review gives an overview of the most recent studies on thatspecial subject. Moreover, it provides many other citations for further study. A citationwas counted as a review article if:

the title contained the word review or overview; orthe citation in Index Medicus mentioned the number of references (in the IndexMedicus this is the standard presentation for all reviews); orthe citation from the CD-ROM or Grateful Med contained review or meta-analysisin the Publication Type (PT) field.

For each selected relevant citation an individual citation quality score was calculated.This score consisted of the sum of the points, ranging from 1 to 4, according to thefollowing criteria:1. one point for topical relevance (only relevant citations were included, so every

citation received 1 point);

66

2. one point for journal ranking (above the median impact factor for its subjectgroup, or listing in the Dutch list of Additional Scientific Journals for HealthSciences Research);

3. one point for research design (clinical trial, placebo-controlled study, double-blindstudy, follow-up or prospective study);

4. one point for review.

Overall search quality scoreUsing the scores of the individual relevant citations, we expressed the overall searchretrieval quality in a new quality measure instrument: the search quality score. Thisconsisted of the sum of the scores of the selected, relevant citations, reduced by thesum of the scores of the missed relevant citations, and the number of the selected non-relevant citations. To avoid negative values, we added 100 points to these scores.In other words, the search quality score = a - b - c + 100, in which a = total score ofthe selected relevant citations, b = total score of the missed relevant citations, and c =number of selected non-relevant citations.

RESULTS

Relevance of citationsAfter we deleted duplicate citations which appeared in more than one source, our listof potential relevant citations consisted of 864 unique citations. From these 864 uniquecitations, 139 were assessed as relevant by the judges, and 725 as non-relevant. Fromthe 864 unique citations, 639 were selected by the course instructor, and an additional225 new citations were selected by the 103 course participants. The judges indicated139 citations as relevant, 109 from the search strategies of the course instructor, and30 additional citations from the search strategies of the course participants (table 2).

Table 2. Number of unique citations retrieved by the course instructor, and the number of additionalnew unique citations retrieved by the 103 course participants.

Number of unique citations retrievedby course instructor

Number of additional unique citationsretrieved by course participants

Total Relevant Non-relevant Total Relevant Non-relevantSearch query

Haemorrhoids 62 22 40 6 0 6

SIDS 66 31 35 92 23 69

Telephone 142 36 106 33 5 28

Gatekeeper 369 20 349 94 2 92

All search queries 639 109 530 225 30 195

The judges checked relevance differences between the course participants and thejudges, but they found no reason to readjust their decisions.

67

First, all 17 citations selected by more than 10 course participants but assessed by thejudges as not relevant were re-evaluated. Of these 17 citations 10 were assessed againas non-relevant for formal reasons (in a language other than English, German, Frenchor Dutch; a letter in a journal other than a general practice or general medical journal).Seven citations were assessed again as non-relevant because they were too specializedfor a GP.Second, all 13 citations which were relevant according to the judges, but were selectedby none of the course participants, were re-evaluated. All of the 13 citations wereagain assessed as relevant by the judges after careful reconsideration.

An analysis of the agreement between the judges and the consensus result (table 3)showed that:

on average, each judge indicated 68% of the relevant citations; andon average, each judge correctly indicated 88% of the non-relevant citations.

In other words, the sensitivity of the judges’ selection process was 68%, and thespecificity was 88%.

Table 3. Percentage (and number of) citations the three relevance judges and the consensus resulthave in common.

Percentage Agreement (n)

Search query Judge A Judge B Judge C Mean perquery

Mean allqueries

Relevant citations(n=139)

Haemorrhoids 96 (21) 27 (6) 64 (14) 62 (14)

SIDS 75 (41) 67 (36) 93 (50) 78 (42) 68

Telephone 83 (34) 44 (18) 61 (25) 63 (26)

Gatekeeper 46 (10) 64 (14) 64 (14) 59 (13)

Non-relevant citations(n=725)

Haemorrhoids 70 (32) 100 (46) 98 (45) 89 (41)

SIDS 81 (44) 94 (51) 88 (48) 88 (48) 88

Telephone 84 (113) 99 (133) 90 (121) 91 (122)

Gatekeeper 98 (432) 98 (432) 96 (423) 97 (429)

Quality assessment of citationsOf the 89 journals in which the relevant articles of our study were published, 58 werecovered by the Science Citation Index and/or by the Social Science Citation Index.Thirty-nine had an impact factor above the median of their subject category, and anadditional four were covered by the Dutch list of Additional Scientific Journals forHealth Sciences Research.

68

Of the 139 relevant citations, 15 had the described components of research design. Ofthe 139 citations, 27 referred to a review article. None of the reviews was a systematicreview.Table 4 shows the number of relevant citations with a score from 1 to 4. None of thepapers received a point for both study design and publication type.

Table 4. Number of relevant citations with an individual citation quality score from 1 to 4.

Individual citation quality score

Search query 1 2 3 4

Haemorrhoids 7 11 4 0

SIDS 18 25 11 0

Telephone 12 26 3 0

Gatekeeper 7 14 1 0

All search queries 44 76 19 0

Overall search quality scoreThe overall search quality score of a search was calculated from the individual citationquality scores and the number of selected non-relevant citations. To illustrate how theformula worked we give an assessment of two specific searches undertaken by courseparticipants A and B.Participant A performed a search in Index Medicus to find references on haemorrhoids.Participant A found six citations under the Medical Subject Heading "hemorrhoids", allof which were relevant according to the judges. The total of the score of the selectedcitations was nine (a). The total score of the missed citations was 10 (b). The numberof the selected non-relevant citations was 0 (c). Hence, the overall search quality scorefor participant A was a-b-c+100: 9-10-0+100=99. Participant B performed a search inMedline on CD-ROM to find references on the gatekeeper role of the GP. ParticipantB retrieved 154 citations with the combination of the Medical Subject Headings"physicians-family", "family-practice", and "referral-and-consultation". Five citationswere selected, but none was relevant. The total score of the selected citations was 0(a). The total score of the missed citations was 11 (b). The number of the selectednon-relevant citations was five (c). Hence, the overall search quality score forparticipant B was 0-11-5+100=84.

DISCUSSION

We have described the development of a model to evaluate retrieval quality of searchqueries expressed in an overall search quality score. This score included theassessment of retrieved citations by quality as well as by a narrowly defined conceptof relevance.New in this study is that we developed a quality measure to assess the overall searchresults in addition to the determination of the number of relevant citations. In this

69

context it is related to the evidence-based quality filters developed and described byHaynes,17 which have been built into PubMed. However, our model differs at somepoints. First, our model does not influence the course of a search but uses a way ofmeasuring search quality once the search has been performed. Second, it can be usednot only for clinical queries that fit into four disease-related study categories (therapy,diagnosis, etiology, prognosis), but also for non-clinical queries such as the use of thetelephone in the physician’s office, and the gatekeeper role of the GP. Third, inaddition to research design, our model also includes journal ranking and publicationtype. Finally, our model not only assesses individual retrieved citations, but also theoverall search results, including the missed citations.

The methodology of the study raises several points of discussion.The relevance assessment was not made by the one who prompted the search query,but by independent judges. Although the search questions were linked to dailypractice, they were not formulated and judged by the real end user. However, Janes18

found that in general, judges’ relevance scores compared reasonably well with those ofthe original users. Saracevic19 pointed to the importance of the judges’ subjectiveexpertise, knowledge, and academic and professional training. He found that the morejudges knew about a query, the higher was the agreement among judges on relevancejudgements and the more stringent the judgements became. The judges in our studyhad a high level of knowledge, expertise and training in the field of general practice.Furthermore, one judge was experienced in online information retrieval.Another factor is that the selection was made on title, abstract and controlledvocabulary which does not always represent the content of the article correctly. Wedeliberately chose this method, because we wanted to use this model in a continuingmedical education course for GPs, who usually select relevant citations only on thebasis of data in the bibliographic source.Finally, many factors determine the selection process of citations. The judges in ourstudy initially considered the topical relevance and language. At a later stage, therelevant citations were assessed by the journal in which they were published, theirresearch design and their publication type. However, more factors can influence thedecision process. In an interview study among clinicians, Sievert20 found that factorsbeyond relevance that most often influenced the decision process to choose a citationnot only pertained to methodological rigor and document types, but also to authors,their institutional affiliations and population studied. Furthermore, users’ criteria forrelevance could vary as they interacted with the information.21

Although the quality of citations is influenced by objective factors such as relevance,journal ranking, research design and publication type, other factors could be consideredas well. Factors from the external and problem context of the information need10 areoften personal, subjective factors and therefore difficult to grasp. These factors includethe user’s purpose in obtaining information, whether the information adds to user’sknowledge, and the applicability of the obtained knowledge. Su investigated theappropriateness of 20 measures for evaluating interactive information retrievalperformance, representing four major evaluation criteria: precision and recall,efficiency, utility and user satisfaction.6 Recall appeared to be more important than

70

precision to users. Users’ satisfaction with precision of the search and withcompleteness of search results were important as well.The ultimate test of end-user searching is whether it actually improves patientoutcomes.4 Hersh pointed to the outcomes-oriented relevance,11 namely the impact ofrelevant information on users and their tasks. Some research has been carried out onthe impact of the library and information services on medical decision-making andpatient care in hospitals,22-26 but the impact of information services on medicaldecision-making and patient care in primary health care is scarce.27,28 Our overallsearch quality score can be used to assess high-quality searches; future research couldassess the impact of these high-quality searches on medical decision-making andpatient care in primary health care.

When we started the study in 1992, Grateful Med was only available in Europe as astandalone with a modem connection to a host computer at the Karolinska Institute inStockholm. However, when in 1997 the US National Library of Medicine madeGrateful Med freely available on the Internet (http://igm.nlm.nih.gov), the KarolinskaInstitute stopped offering Grateful Med online services. Although the data and searchprinciples are similar, it is not known whether the two software systems are similar inuser friendliness and search results.

In summary, this study describes an overall search quality score that can be used forevaluation of retrieval quality of search queries by GPs in medical bibliographicsources. Although this model was developed for Dutch GPs who retrieved citationsfrom three specific sources, it could be adapted and generalized to other professionalusers, and to other bibliographic sources, such as Medline on the Internet (InternetGrateful Med, PubMed, or Medline through another Web interface). As a nationalfactor in our model, the journal list can be replaced by a journal list appropriate toanother country.

AcknowledgementThe authors thank the Janivo Foundation for their financial support.This article is reproduced by permission of Blackwell Science Ltd.

REFERENCES

1 Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teachingthe practice of medicine. JAMA 1992;268:2420-5.

2 Geyman JP. Evidence-based medicine in primary care: an overview. J Am Board Fam Pract1998;11:46-56.

3 Hersh WR, Hickam DH. How well do physicians use electronic information retrieval systems? Aframework for investigation and systematic review. JAMA 1998;280:1347-52.

4 McKibbon KA, Walker-Dilks CJ. The quality and impact of MEDLINE searches performed byend users. Health Libr Rev 1995;12:191-200.

5 Su LT. Evaluation measures for interactive information retrieval. Inform Processing &Management 1992;28:503-16.

6 Su LT. The relevance of recall and precision in user evaluation. J Am Soc Inform Sci1994;45:207-17.

71

7 Schamber L, Eisenberg MB, Nilan MS. A re-examination of relevance: toward a dynamic,situational definition. Inform Processing & Management 1990;26:755-76.

8 Mizzaro S. Relevance: the whole history. J Am Soc Inform Sci 1997;48:810-32.9 Harter SP. Psychological relevance and information science. J Am Soc Inform Sci 1992;43:602-

15.10 Park TK. The nature of relevance in information retrieval: an empirical study. Libr Q

1993;63:318-51.11 Hersh W. Relevance and retrieved evaluation: perspectives from medicine. J Am Soc Inform Sci

1994;45:201-6.12 Weller AC. Editorial policy and the assessment of quality among medical journals. Bull Med Libr

Assoc 1987;75:310-6.13 Garfield E. SCI Journal Citation Reports; a bibliometric analysis of science journals in the ISI

database. Philadelphia: Institute for Scientific Information; 1994. p. 10.14 Garfield E. SCI Journal Citation Reports; a bibliometric analysis of science journals in the ISI

database. Philadelphia: Institute for Scientific Information; 1994. p. 72-93.15 Garfield E. SSCI Journal Citation Reports; a bibliometric analysis of social science journals in the

ISI database. Philadelphia: Institute for Scientific Information; 1995. p. 40-6.16 Wetenschappelijke tijdschriften gezondheidszorgonderzoek. In: Richtlijnen aanlevering gegevens

ten behoeve van het Discipline-advies Geneeskunde 1998. Commissie Geneeskunde van deKoninklijke Nederlandse Akademie van Wetenschappen en het Discipline OverlegorgaanMedische Wetenschappen van de Vereniging van Samenwerkende Nederlandse Universiteiten.[Additional scientific journals for health sciences research. In: Guidelines for provision ofinformation for the Discipline Report on (Bio)Medical and Health Sciences Research 1998.Medical Committee of the Royal Netherlands Academy of Arts and Sciences, and the Council ofMedical Faculties in the Netherlands of the Association of Universities in the Netherlands. InDutch]. Amsterdam, Utrecht; 1996. p. 57.

17 Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal searchstrategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc1994;1:447-58.

18 Janes JW, McKinney R. Relevance judgments of actual users and secondary judges: acomparative study. Libr Q 1992;62:150-68.

19 Saracevic T. Relevance: a review of and a framework for the thinking on the notion ininformation science. J Am Soc Inform Sci 1975;26:321-43.

20 Sievert ME, McKinin EJ, Johnson ED, Reid JC, Mitchell JA. Beyond relevance - characteristicsof key papers for clinicians: an exploratory study in an academic setting. Bull Med Libr Assoc1996;84:351-8.

21 Quintana Y. Intelligent medical information filtering. Intern J Med Informatics 1998;51:197-204.22 King DN. The contribution of hospital library information services to clinical care: a study in

eight hospitals. Bull Med Libr Assoc 1987;75:291-301.23 Marshall JG. The impact of the hospital library on clinical decision making: the Rochester study.

Bull Med Libr Assoc 1992;80:169-78.24 Lindberg DAB, Siegel ER, Rapp BA, Wallingford KT, Wilson SR. Use of MEDLINE by

physicians for clinical problem solving. JAMA 1993;269:3124-9.25 Klein MS, Ross FV, Adams DL, Gilbert CM. Effect of online literature searching on length of

stay and patient care costs. Acad Med 1994;69:489-95.26 Urquhart CJ, Hepworth JB. Comparing and using assessments of the value of information to

clinical decision-making. Bull Med Libr Assoc 1996;84:482-9.27 Wood F, Wright P, Wilson T. The impact of information use on decision making by general

medical practitioners. [British Library R&D Report]. Boston Spa: British Library; 1995.28 Wood F, Palmer J, Ellis D, Simpson S, Bacigalupo R. Information in primary health care. Health

Libr Rev 1995;12:295-308.

72

APPENDIXAdditional 106 scientific journals for health sciences research (Blue List)16

Acta HospitaliaAgeing SocietyAlc TreatmentAlcohol TreatmentAm J Resp DisAnn EpidemiolAnxiety Stress CopAppl Occup Environ HArch Fam MedArs AequiBeleidswetenschBioethicsBrit J Occup MedCambr Q Hlth Car EthCardiov Dis Ep NewslComm Mental Hlth JCommun Dent Oral EpCommun Med Public HContrib Epid BiostatEcon Stat BerichtenEpidemiol BullEur Addiction ResEur J DemographyEur J Gen PractEur J Health LawEur J Health SciEur J Publ HealthFam MedFam Pract Res JFilosofie PraktijkGedragGedragstherapieHealth Care Man RevHealth EconHealth Hum RightsHealth Plan Manage

Health PromotionHuisarts NuHuisarts WetenschInt Digest Healt LegInt J Health SciInt J Hlth Plan ManInt J Ind ErgonomInt J Meth Psyc ResInt J Soc PsychiatInt J Tech Ass H CarInt Q Comm Hlth EdIntegr PsychiatJ Addict DisorderJ Addict ResJ Am B Fam PracticeJ Am Stat SocJ Clin EthicsJ Community HealthJ Consult Clin PsJ Health Admin EducJ Hyg Epid Microb IJ Law Med EthicsJ Manage MedJ PsychophysiolLaw Med Health CarMbl Geest VolksgezMed Care RevMed LawMedical Law RevMens MaatschappijNed JuristenbladNed Tijdschr GeneeskNed Tijdschr PsycholNJCM BullOral Epidemiol

Pers Indiv DifferPharmacol EconPsychiat DevPsychiat ServicesPsycho OncolPsychosoc Rehab JPublic Health PapQual SociolQuality Life ResRev Droit SanteScan J Prim HealthSoc Psychiat EpidemTijdschr Alcohol DrTijdschr Bedr Verz GTijdschr Burg RechtTijdschr ErgonomieTijdschr Fam JeugdreTijdschr Geneesk EthTijdschr Geron GeriatTijdschr Gezondh BeTijdschr Gezondh ReTijdschr JeugdgezondTijdschr KindergenTijdschr OrthopedagTijdschr PsychiatrieTijdschr PsychotherTijdschr Soc GeneeskTijdschr Soc GezondhTijdschr Soc RechtTijdschr Verzek KundVerpleegkundeWestern J Nurs ResWHO BullWorld Health ForumWorld Hlth Stat Quart

73

74

6 Which literature retrieval method is most effective for generalpractitioners?

If it works, it’s out of dateStafford Beer, British scientist, 1972

ABSTRACT

Introduction. Evidence-based medicine requires new skills of physicians including literature searching.We want to determine which literature retrieval method is most effective for general practitioners (GPs),the printed Index Medicus, Medline through Grateful Med, or Medline on CD-ROM.Methods. The design was a randomized comparative study. In a continuing medical education course,three groups of health care professionals (87 GPs and 16 other health care professionals) used one of theliterature retrieval methods to retrieve citations on four search topics related to general practice. For theanalysis in pairs, we used the search results of the 75 participants who completed all four assignments.As outcome measures we used precision, recall, and an overall search quality score; we also had a post-course questionnaire on personal characteristics, experience with computers, handling medical literature,and satisfaction with course instruction and search results.Results. The recall and overall search quality scores in the Index Medicus groups (n=32) were higher(P≤0.001) than those in the CD-ROM groups (n=31). In addition, the search quality scores in theGrateful Med groups (n=12) were higher (P<0.003) than those in the CD-ROM groups. There were nodifferences in precision.Conclusion. In the period 1994 to 1997, the printed Index Medicus was the most effective literatureretrieval method for GPs. For inexperienced GPs there is a need for training in electronic literatureretrieval methods.

Accepted for publication in Family Practice. Co-authors: EJ Boerma, B Meyboom-de Jong.

INTRODUCTION

Due to the rapid expansion of medical knowledge and publications, physicians havedifficulties in locating the medical information they need.1 Experience, basic medicalknowledge, and skills alone are not enough to practise evidence-based medicine.2

Physicians need to develop skills in retrieving and interpreting information,3 e.g. howto identify relevant diagnostic studies.4 The medical literature is underused,5 although46-54% of primary care physicians’ questions could be answered using the medicalliterature.6 7

In order to teach GPs the most effective literature retrieval method, we developed anexperiment to compare three of these methods, namely a printed, an online, and a CD-ROM version of the Index Medicus/Medline. The most effective method should give ahigh proportion of relevant citations of good quality, as well as few non-relevantcitations. Furthermore, the ideal literature retrieval method should be wide available,easy to use, convenient, quick to learn, and not expensive or time-consuming.8

75

In our experimental study, we answered the following research question: Whichliterature retrieval method is most effective for GPs?

METHODS

ParticipantsTo invite GPs for the 1-day continuing medical education course "How to retrieveinformation", we sent an announcement to all 970 registered GPs in the north of theNetherlands with reminders to university affiliated GPs.

RandomizationPrior to sending the invitations, we determined the dates of the courses. Werandomized the interested GPs to the Index Medicus course, the CD-ROM course, orthe Grateful Med course in order of receipt of the registration forms, and depending onthe participants’ day of choice. The participants did not know they would berandomized, because the invitation only mentioned one course including three methodsin information retrieval.

CoursesFrom 1994 to 1997, we offered our 1-day course 15 times. A professional librarian,who was a GP by training, served as course instructor.In each course we focused on one of the following three literature retrieval methods:the 1992 printed Cumulated Index Medicus; Medline through Grateful Med - version6.0 (citations with publication year 1992) provided online by the Karolinska Institutein Sweden; and the 1992 Medline on CD-ROM (Silverplatter Dos version 3.1).The experimental part of the course consisted of a two hours’ introduction in literatureretrieval methods, and an on-site training session. In the introduction, the content andstructure of one of the three methods were discussed, including controlled vocabulary,subheadings, and, if appropriate, free-text searching. Sample searches weredemonstrated.The training session started with a try-out search by the participants in one of the threeretrieval methods. Next, all three groups received the same four assignments forretrieving and selecting bibliographic citations (appendix 1). The four assignmentscovered search topics related to general practice: haemorrhoids, sudden infant death,the use of the telephone, and the gatekeeper role (table 1). For the Grateful Med andthe CD-ROM groups the order of the assignments was fixed and of increasingcomplexity. However, for organisational reasons, the assignments of the IndexMedicus groups were performed in a varied order. The Index Medicus groups couldspend 90 minutes in total on the four assignments. The Grateful Med and the CD-ROM group, however, could spend 80 minutes in total on the four assignments: 20minutes on each assignment. Additionally for these last two groups, the courseinstructor gave a 10 minutes feedback directly after each assignment.After the experimental part of the course with one of the three retrieval methods, thetwo other methods were discussed and practised as well.

Table 1. The four search queries for a course in literature searching for Dutch GPs.

76

I.You have been invited to give a talk on haemorrhoids to a group of colleagues in your town. Tocollect information, you want to do a literature search using the Index Medicus/Medline on CD-ROM/Medline through Grateful Med of the year 1992.

II.You want to keep up with the latest developments in the prevention of sudden infant death.You want to do a literature search using the Index Medicus/Medline on CD-ROM/Medline throughGrateful Med of the year 1992.

III.The editor of the Dutch journal Huisarts & Wetenschap (General Practitioner & Medical Science)has asked you to submit an article on the use of the telephone in the physician’s office. You regardthis request as a challenge and you decide to do a literature search using the Index Medicus/Medlineon CD-ROM/Medline through Grateful Med of the year 1992.

IV.You are supervising a GP trainee in your practice. She wants to discuss the gatekeeper role of theGP. You want to be well prepared for this discussion, so you decide to do a literature search on thistopic using the Index Medicus/Medline on CD-ROM/Medline through Grateful Med of the year1992.

QuestionnaireAfter the course, the participants filled in a questionnaire (appendix 2), which coveredpersonal characteristics, experience with computers, handling medical literature, andsatisfaction with the course instruction and the search results.

Outcome measuresThe effectiveness of the searches was assessed by three measures: precision, recall,and an overall search quality score. Precision is the number of relevant citations as aproportion of the total number of citations retrieved; recall is the number of relevantcitations retrieved from the total number of relevant citations in a subset of thebibliography (table 2). In our study, this subset was formed from all citationsidentified by the 103 course participants and the course instructor. Three judgesassessed the relevance and the quality of these citations. Because GPs are moreinterested in quality than in numbers, we developed an overall search quality score.For this score, we calculated a citation quality score for each relevant citation, basedon the following criteria: coverage of the journal by the Science or Social ScienceCitation Index, or the Dutch list of Additional Scientific Journals for Health SciencesResearch of the Royal Netherlands Academy of Arts and Sciences; the journal’simpact factor; study design; and whether the citation was a review. To avoid negativevalues, we added 100 points to these scores. In a formula: the overall search qualityscore = a - b - c + 100, in which a = total of quality scores of the selected relevantcitations, b = total of quality scores of the missed relevant citations, and c = numberof the selected non-relevant citations.

77

Table 2. Calculation of precision and recall of search results

Assessed by the judges as

Assessed by course participants as Relevant Non-relevant

Relevant a b

Non-relevant c d

precision = a / (a + b)recall = a / (a + c)

Statistical analysisWe used SPSS version 8 for statistical analysis.To check the equal allocation of the participants to the three groups, we compared sex,age, years of experience as general practitioner, and type of practice with the chi-square and Student’s t-tests.To identify the influence of the method on precision, recall and overall search qualityscores, a repeated measures analysis of variance was used for each of the threemethods (Index Medicus, Medline through Grateful Med, and Medline on CD-ROM).The four assignments constituted the repeated factor in this set up, the method was thebetween-subjects factor.To identify differences in precision, recall, and search quality scores in pairs due tothe method, in each of the four assignments, post hoc multiple comparisons forobserved means were used according to Bonferroni.To identify pairwise differences in precision, recall and overall search quality scores ineach group for the four assignments in total, simple contrasts were used.

RESULTS

ParticipantsOf the 970 invited GPs, 87 participants (9%) took part in the study. Additionally, 16other health care professionals participated: 7 nursing home physicians, 6 medicalresearchers in general practice, 2 physiotherapists, and a hospital manager.Of the 103 participants, 73% were male. The mean age of the participants was 43years. The GPs had 14 years of experience; 55% worked in an urban area; and 36%had a solo practice. Eighty-six percent of all participants were affiliated with aUniversity Department of General Practice.

RandomizationThe 103 participants were randomized blockwise in 15 groups of 3-12 people: 37participants were randomized to the Index Medicus course (6 groups), 26 to theGrateful Med course (4 groups), and 40 to the CD-ROM course (5 groups).

Outcome measuresThe participants performed 366 searches: 310 by the 87 GPs, and 56 by the otherhealth care professionals.

78

The allocation of the participants to the three groups was performed equally: the chi-square and t-tests showed no significant difference (P<0.01) between the IndexMedicus, Grateful Med, and CD-ROM groups in sex or age, nor for the GPs in yearsof experience or type of practice.For the analysis of variance and differences in pairs, we used the search results of the75 participants who completed all four assignments.The repeated measures analysis of variance showed a significant influence of themethod on recall and search quality score (P<0.01).After combining the search results of the four search topics, the recall and overallsearch quality scores in the Index Medicus groups were significant higher (P≤0.001)than the recall and overall search quality scores in the CD-ROM groups (table 3). Inaddition, the search quality scores in the Grateful Med groups were higher (P<0.003)than those in the CD-ROM groups. We found no difference in precision. Theprecision, recall, and search quality scores of the individual search topics did not showspecific patterns (figure 1).

Table 3. Precision, recall, and search quality scores for 300 searches on 4 search topics in totalperformed in one of the three sources by 75 course participants. Results are means and (SD).

Literature retrieval method(n = participants who completed all 4 assignments)

Outcome measureIndex Medicusn=32

Grateful Medn=12

CD-ROMn=31

Precision 55 (15) 50 (24) 56 (19)

Recall 26a (12) 22 (10) 16 (6)

Search quality score 83b (5) 71c (6) 65 (6)

(a) p=0.001 vs CD-ROM(b) p<0.001 vs Grateful Med and CD-ROM(c) p=0.003 vs CD-ROM

QuestionnaireThe questionnaire showed that 91% of the 103 participants had possessed a personalcomputer for mean 5.3 years: 88% used it for patient care, but only 14% for retrievingliterature. Although 72% had written an article at least once, 45% never had visited aninstitutional library.Almost all participants (99%) were satisfied with the course instruction, and 73% withthe search results. Searching was interpreted as easy by 62%. The Index Medicus,though, was never used by 70% of the Index Medicus groups; Grateful Med was neverused by 99% of the Grateful Med groups; and CD-ROM was never used by 77% ofthe CD-ROM groups.

79

DISCUSSION

In this study performed from 1994 to 1997, we compared three methods to retrievebibliographic information by GPs with little experience in literature searching. Thisstudy shows that the printed Index Medicus was the most effective literature retrievalmethod for GPs. Specifically, the printed Index Medicus yielded the best results inrecall and overall search quality scores, whereas Medline on CD-ROM yielded thelowest. Apparently, the Index Medicus with only the Medical Subject Headings asentries was less confusing than Medline on CD-ROM which offered also free textsearching.No significant differences were found for precision. Obviously, the method used didnot influence the critical selection of the retrieved citations. In fact, users considerprecision of less importance than recall.9

New in this study was the use of the overall search quality score. Whereas the recalland precision referred to numbers of retrieved citations, the quality score took intoaccount the quality of the relevant citations as assessed by judges as well.

The results of our study could have been influenced by the following aspects of ourstudy design.Since the course participants were interested volunteers, and 86% were affiliated witha University Department of General Practice, our participants might be moreexperienced in handling literature retrieval methods than GPs in general. The results ofour study, therefore, can not be generalized to the general population of GPs.Second, although the Index Medicus group spent an average of 2½ minutes longer oneach assignment (ten minutes for all four assignments), it is unlikely this hadinfluenced the results significantly.Third, in the CD-ROM and the Grateful Med groups the assignment order was fixedand of increasing complexity. In addition, each assignment was discussed for 10minutes afterwards. These factors could have had a positive result on the learningeffect, and therefore on the search results. In spite of the lack of this learning effect, itis remarkable the Index Medicus group scored significantly better.Fourth, although the topics of the assignments were related to daily practice, they wereperformed in a test situation with a restraint on time, and with a restriction of theinformation source (only one year). Actually, the printed Index Medicus is availableonly in institutional libraries, whereas electronic bibliographic information sources maybe accessed from home. Furthermore, perusing several years of the Index Medicuscould be a time-consuming and tedious task.Finally, the test period was from 1994 to 1997. Because the software of literatureretrieval methods we used was current in the early 90s, the study results are dated.Probably, with more consumer friendly software packages, the same study performedin 1999 may show more favourable results for the electronic sources.Although the printed Index Medicus yielded the best search results in our study, wewould not recommend the Index Medicus to GPs. The Index Medicus may be the mosteffective literature retrieval method, but it seems not to be the most efficient. GPseven prefer availability of information sources to quality.10

81

The implications of this study may be found in training programmes for GP traineesand continuing medical education courses. Because many search possibilities confusedthe inexperienced end users, training sessions need to be concise, clear, and simple. Aliterature searching training session as part of an evidence-based medicine program isrecommended.Because our study involved inexperienced end users with dated results, a repeatedstudy with more experienced participants may give smaller or even reversed outcomemeasures of the three literature retrieval methods.

CONCLUSION

In summary, our study, carried out from 1994 to 1997, shows that, out of three, theprinted Index Medicus is the most effective literature retrieval method for GPs.Because inexperienced end users had the highest recall and overall search qualityscores using the simplest method, and the lowest recall and search quality scores usingthe method with the most search possibilities, we suggest the development of user-friendly electronic systems, with the provision of extensive training for end users.

AcknowledgmentsWe thank Henk Bosveld for his help with the statistical analysis.This article is reproduced by permission of Oxford University Press.

REFERENCES

1 Huth EJ. The information explosion. Bull N Y Acad Med 1989;65:647-61.2 Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching

the practice of medicine. JAMA 1992;268:2420-5.3 Greenhalgh T. How to read a paper: the basics of evidence based medicine. London: BMJ; 1997.4 Weijden T van der, IJzermans CJ, Dinant GJ, Duijn NP van, Vet R de, Buntinx F. Identifying

relevant diagnostic studies in Medline. The diagnostic value of the erythrocyte sedimentation rate(ESR) and dipstick as an example. Fam Pract 1997;14:204-8.

5 Huth EJ. The underused medical literature. Ann Intern Med 1989;110:99-100.6 Gorman PN, Ash J, Wykoff L. Can primary care physicians’ questions be answered using the

medical journal literature? Bull Med Libr Assoc 1994;82:140-6.7 Chambliss ML, Conley J. Answering clinical questions. J Fam Pract 1996;43:140-4.8 Smith R. What clinical information do doctors need? BMJ 1996;313:1062-8.9 Su LT. The relevance of recall and precision in user evaluation. J Am Soc Inform Sci

1994;45:207-17.10 Connelly DP, Rich EC, Curley SP, Kelly JT. Knowledge resource preferences of family

physicians. J Fam Pract 1990;30:353-9.

82

APPENDIX 1Three assignment examplesI. Index Medicus assignment

You have been invited to give a talk on haemorrhoids to a group of colleagues in your town. Tocollect information, you want to do a literature search using the Index Medicus of the year 1992.

(Please read the instructions before you start your search)

STARTING TIME:

VOLUME year .......... part of the alphabet .....................

index .......... volume number .....................

SUBJECT subject heading .....................................................

used subheading(s) .....................................................

ARTICLES FOUND (1st word of title + first author, e.g. Risk, Taylor)

1. 6.

2. 7.

3. 8.

4. 9.

5. 10.

83

II. Grateful Med assignment

You want to keep up with the latest developments in the prevention of sudden infant death.You want to do a literature search using Medline via Grateful Med.

LIMIT YOUR SEARCH TO 1992

1. Fill in the screen up to and including JOURNAL ABBREV (use the arrow key).

2. Please copy the entries from the screen:

AUTHOR NAME:

TITLE WORDS:

SUBJECT WORDS:

2ND SUBJECT:

3RD SUBJECT:

4TH SUBJECT:

ENGLISH ONLY:

PUBL TYPE:

JOURNAL ABBREV:

3. A. Go one line down. You want literature references with abstracts.B. Go one line down. Type "S" to answer the question OK TO GO ON TO SEARCH? This will

save your search strategy. Give the file the name: a:sids.If you want to save a second search strategy give it the name a:sids2.

C. You want to retrieve both literature references and Medical Subject Headings (MeSH).Please call one of the course instructors! Enter "Y" and make the connection with Stockholm.When you think you have a sufficient number of articles, you can close the connection bypressing Esc.

4. How many references on the prevention of sudden infant death have you found? .... references

5. Read the list of references.If you consider a reference relevant, you type "F" to save this reference. The first time you type"F", you are asked to enter the name of the file in which the references will be saved. Enter:a:sids. The second time enter a:sids2. Press ENTER twice.

When you have finished your search, press Esc. You have two options:1. change the search strategy and search again: go to EDIT.2. restart (for the next question): go to ACTION.

84

III. CD-ROM assignment

The editor of the Dutch journal Huisarts & Wetenschap (General Practitioner & Medical Science)has asked you to submit an article on the use of the telephone in the physician’s office. You regardthis request as a challenge and you decide to do a literature search using Medline on CD-ROM ofthe year 1992.

1. Remove the sets of the previous assignment (F10, Clear + number of the set you wish toremove).

2. Which term(s) do you use for your search on the use of the telephone in the ...................physician’s office?

...................

3. How many references about the telephone in the physician’s officehave you found? ....... references

4. Study the list of literature references you have found.Mark the references you consider relevant to the article for the journal General Practitioner &Medical Science.

5. Before entering the final command, please call the assistant!Download the references you have marked and your search strategy to your disk, and give the filethe name a:tele.

85

APPENDIX 2Questionnaire example

QUESTIONNAIRE MEDLINE ON CD-ROM

PERSONAL DETAILS please tick the appropriate answer, orwrite your answer on the dotted line

1. Sex: manwoman

2. Age: .... years

3. Number of years experience as general practitioner .... years(if you are not a general practitioner please state your occupationor profession: .............................................................)

4. How would you characterise your practice? town practicerural practice

not applicable

5. What type of practice do you work in? solo practicetwo-person practice

group practicemedical centrenot applicable

6. Do you have formal relationships with an Academic Centre for General Practice?(e.g. as supervisor of general practitioner trainees or students in your practice, yesor as a medical researcher) no

7. Are you a member of: both LHV and NHG(LHV = National Association of General Practitioners; LHVNHG = Dutch College of General Practitioners) NHG

EXPERIENCE WITH COMPUTERS

8. Do you own a personal computer- at home yes no- in your office yes no(if you answer no to both questions, go to question 10)

9. If you own a personal computer, do you actually use it? yes no

If so, for what purposes?- personal use yes no- patient care yes no- to retrieve information from literature databases using a modem yes no

10. How many years of experience do you have in using a computer? .... years

86

MEDICAL LITERATURE please tick the appropriate answer

11. Have you ever written a report or an article for which you needed literature? noyes, once or twice

yes, 3-5 timesyes, more than 5 times

12. Do you read any of the following journals regularly?Nederlands Tijdschrift voor Geneeskunde yes no

(Dutch Medical Journal)Huisarts & Wetenschap yes no

(General Practitioner & Medical Science)general English medical journals, yes no

e.g. the Lancet or the British Medical JournalEnglish journals for general practice, yes no

e.g. Family Practice

13. Have you ever used an academic medical library? neveronce a year at the most

several times a year

14. Have you ever used the computerised catalogue of the university neverlibrary/academic hospital library (the OPAC)? once a year at the most

several times a year

15. Before this course had you ever heard ofIndex Medicus yes

no

Medline yesno

16. How many times have you used the printed Index Medicus? never1-3 times

more than 3 times

17. How many times have you asked someone else to do a Medline search neveron CD-ROM for you? 1-3 times

more than 3 times

18. Prior to this course, have you ever used Medline on CD-ROM? yesno

19. How many times have you asked someone else to do an online search for you? never1-3 times

more than 3 times

87

SEARCH RESULTS EVALUATION please tick the appropriate answer, orwrite your answer on the dotted line

20. Are you satisfied with your search results? yesno

21. Using Medline on CD-ROM was: difficulteasy

22. In carrying out the assignments did you find it difficult to:find the appropriate search terms yes nouse the thesaurus yes noperform the technical procedures, e.g. downloading to the disk

or using "show" yes nonarrow or broaden the search; or using AND, OR or NOT yes no

23. The next time you need literature on a particular subject, are you going to:use Medline on CD-ROM? yes noorder a search in Medline on CD-ROM? yes noorder an online literature search? yes nodo a manual search of journals? yes no

24. Suppose you were actually faced with one of the course assignments,would you be willing to pay DFl 100 to have an expert do a literature search for you?

for the talk to your colleagues yes nofor keeping up with the literature yes nofor writing a article yes nofor the discussion with the general practitioner trainee yes no

25. If you had a CD-ROM player with a subscription on Medline at home, nothingyou would be able to search as extensively and as often as you like. How < DFl 250much would you be willing to invest as a once-only investment into equipment? > DFl 250

26. Would you be willing to pay up to DFl 1500 a year for a subscription to yesMedline on CD-ROM? no

27. If you require more literature references than you have found now,what would you do?

go to a library and ask help yes nogo to a (medical) expert yes nolook through my own books yes nolook through journals yes no

28. In your opinion, did you receive sufficient instruction prior to the assignments? yesno

29. Was the instruction given in the course better than you had expected or were you disappointed?Please comment.

........................................................................................................................

88

7 General discussion

Paved roads create trafficAmerican saying

BRIEF ANSWERS TO THE RESEARCH QUESTIONS

To answer three questions on information-seeking by general practitioners, we usedseveral research methods. These methods were a review of the literature publishedfrom 1975 to 1992; in-depth interviews carried out in 1993 and 1994; a postalquestionnaire sent to general practitioners in 1996; and an experimental comparativestudy performed from 1994 to 1997. The research questions and the answers were:

1. Information sources: Which information sources do general practitioners use?For patient care, general practitioners most frequently used books and human basedinformation resources (consultation with other physicians). For continuing medicaleducation most time was spent in reading journals and attending continuing educationclasses and congresses. Electronic information sources were used neither for medicaldecision-making nor for continuing medical teaching. General practitioners involved inresearch or teaching most frequently used journals and human based informationresources (consultation with physicians and librarians). Some electronic informationsources were used, but not efficiently.

2. Information barriers: What problems do general practitioners encounter whenseeking information, and which factors influence the use of information sources?The problems general practitioners encounter when seeking information for patientcare are: information overload; time restraints; badly organised journal articles; and ashortage of articles focused on daily practice. General practitioners involved inresearch experienced difficulties with locating the appropriate terms describing theirresearch questions, and with the skills to use the information sources efficiently.Factors influencing the use of information sources were the physical, functional andintellectual accessibility of information sources, familiarity with information sources,and the reliability, usefulness, and ease of searching of information sources.

3. Comparison of three literature retrieval methods: Are electronic information sourcesmore effective for general practitioners than the printed ones?The electronic information retrieval methods we examined, Medline on CD-ROM andMedline through Grateful Med, seemed not more effective than the printed one, theIndex Medicus, in terms of recall and quality of the selected references. Actually, theprinted Index Medicus yielded significantly better results in recall and in overallquality of the search than Medline on CD-ROM.

89

INFORMATION SOURCES, BARRIERS AND FACTORS INFLUENCING THEUSE OF INFORMATION SOURCES

Following Leckie’s model of the information-seeking of professionals,1 we answeredthe research questions for the different roles and tasks of general practitioners.For patient care, general practitioners still use traditional information sources such asbooks (especially drug reference books) and human based information sources i.e.personal communication. This has been confirmed by more recent studies.2-4

Apparently, technological and educational change did not radically alter physicians’information-seeking behaviour.2 In our studies, we found that, in general, the mostimportant factors in choosing an information source included reliability, usefulness,and ease of searching (chapter 4), and the physical, functional, and intellectualaccessibility of the source (chapter 2).5 More specifically, a reason for the popularityof books and human based information sources might be that both are widelyavailable, easy accessible, and easy to use. Because most questions arising in patientcare involved therapy, general practitioners used their familiar drug reference booksheavily (chapter 4). Familiarity is an important factor in using information.6

Several reasons may explain why human based resources were so popular. Firstly,people can anticipate on the complexity and the patient-, problem-, and practitioner-specific nature of the questions.7 Secondly, human based resources may meet morethan just specific clinical information needs, i.e. support, guidance, and approval ofwhat the general practitioners are doing, psychological affirmation, sympathy,judgment, and feedback.8 Thirdly, medical specialists best meet criteria important togeneral practitioners, such as availability, ease of searching, understandability,credibility, and applicability.9 Moreover, medical specialists may be able to provideevidence-based information, and to apply this evidence to a specific patient. Fourthly,general practitioners are used to working with oral information: their training and carefor patients heavily rely on this kind of information (for example history taking),rather than on more formal information. Finally, the type of individual who enters thespeciality of general practice may be more interested in human relations and oralcommunication than those who enter other specialities.10

Several studies confirmed that general practitioners use electronic information sourcesvery little,2,6,9,11 and, if they use them, it is predominantly for research activities(chapter 4).5 Overall use of information retrieval systems by physicians in hospitalsand at universities is also small.12 Considering our finding that 65% of generalpractitioners information needs are being met (chapter 4), and that about half of theseneeds can be met by searching Medline,13 then it is clear that information retrievalsystems such as Medline are not playing a major role in meeting physician informationneeds. Since drug reference books were used most frequently for patient care by Dutchgeneral practitioners (chapter 4), we would suggest making them available and easysearchable on the Internet. In the Netherlands, the most used drug reference book isone published by the Dutch Medical Insurance Board (Nederlandse Ziekenfondraad),but this has not been published electronically yet.Low use of electronic information sources in our study could not be explained by alow computer ownership: 93% of the Dutch general practitioners owned a personalcomputer at work in 1996 (chapter 4). The reason must be sought either in the

90

characteristics of the source, or within the physician.On the one hand we found that many characteristics of electronic information sourcesneeded to be improved: ease of searching, financial costs, investment of time, chanceof success and usefulness (chapter 4). On the other hand, general practitioners lackedskills for using electronic information sources efficiently, including locatingappropriate keywords (chapter 3).14 After purchasing personal computers, traininginformation skills should be the next step to information literacy.Apart from a low perceived reliability, usefulness, ease of searching, familiarity, andaccessibility, an important barrier to the use of information sources was the generalpractitioners’ very high workload and consequent shortage of time. We found thatmost time for continuing medical education was spent on reading medical journals (6.6hours a month) and attending continuing medical education classes and congresses (5.1hours a month). Many researchers expected that electronic information sources wouldreduce their time constraints.5 For instance, continuing medical education modules onthe Internet could be accessed at home, thereby saving time. However, this expectationhas not been realised yet. One reason may be that computer literacy demands aninvestment of time first. Another reason may be general practitioners’ preference forrelating with colleagues in continuing medical education.15

COMPARISON OF THREE LITERATURE RETRIEVAL METHODS

Although not many physicians use the printed Index Medicus anymore and rely on theelectronic version Medline, in our comparative study we found that the printed IndexMedicus seemed more effective than Medline on CD-ROM. General practitioners whoused the printed Index Medicus retrieved significantly more relevant citations, and asignificantly higher quality of search results, than the general practitioners who usedMedline on CD-ROM. Thus, although availability and time restraints do limit the useof the printed Index Medicus, it retrieved better search results.To place these findings in context, we have to consider several aspects of our study.First, the general practitioners who participated in the study were novice searchers.From this point of view, we might conclude that it is easier to learn to use a printedthan an electronic information source. Knowledge source providers should study theways the printed Index Medicus is used, and apply the principles to electronic sources.In this way, electronic systems could be made easier to learn and to use. In the future,when general practitioners have more search experience with electronic sources,Medline may become more popular. Secondly, only one year of the Index Medicus orMedline was searched, which could be unrealistic for researchers. Yet, generalpractitioners may find enough citations when they search a few years of the IndexMedicus: when they need a few citations or a single review for the preparation of atalk for colleagues, for example.

91

LIMITATIONS OF THE STUDIES

The studies discussed in this thesis were performed between 1992 and 1997. The greatchanges that have taken place in information communication and technology, haveconsequences for the work, especially for our comparative study of three literatureretrieval methods. Successive interfaces, versions of software programmes and themedia were developing rapidly. At the start of this comparative study, we decided touse the literature retrieval methods currently in use at that time: the printed IndexMedicus, the Medline version on CD-ROM, and the stand-alone Medline versionthrough Grateful Med. However, when after 1992 the Internet developed as popularmedium for electronic communication, Medline became available through severalinterfaces, such as Healthgate, Infotrieve and Biomednet.16 In 1997 the producer ofMedline, the National Library of Medicine, launched free Medline searching on theInternet: PubMed (http://www.nbi.nlm.nih.gov/PubMed), and Internet Grateful Med(http://igm.nlm.nih.gov). If we repeated the study, Pubmed would be one of theselected electronic information sources in addition to Internet Grateful Med.The comparative study took four years because it was difficult to include generalpractitioners, to find classrooms with enough personal computers with access to theInternet, and to find funding.

The types of research methods may have given rise to bias.In the interviews and questionnaire, bias in the general practitioners’ self-reportedchoices may have distorted the validity of the study’s findings. Next, in thecomparative study, we could not totally randomise the participants to the groupsbecause we did not control the general practitioner choice of day of the course.However, the randomisation was blind for the participants, and the allocation of thecourse participants to the three groups was performed equally with regard to sex, age,years in practice, and type of practice. If we repeated the study, we would use arandomisation table within the restraints of course days to assign the participants to thethree groups. In such a repeated study we also would clock the time spent on the fourassignments more precisely. Finally, the last factor is the moderate response of thepostal questionnaire (45%), although this is not unusual among busy generalpractitioners. For a 14 page questionnaire it was possibly a high response. Although nosignificant differences were found between subjects and the Dutch general practitionerpopulation in age, sex and type of practice, it may be that it was the more highlymotivated general practitioners who responded.

EVIDENCE-BASED MEDICINE

Our studies include data important for evidence-based medicine: general practitioners’familiarity with evidence-based medicine, questions raised during patient care, and thesources general practitioners use and want to use.Although familiarity with evidence-based medicine was only moderate in 1996 (45%),the great majority (85%) of those who knew exactly what it was expected changes indaily patient care if evidence-based medicine was to be introduced (chapter 4). In astudy in an English training practice, researchers found that already 81% of the

92

interventions were based on evidence.17 This shows that practising evidence-basedmedicine in general practice is possible.Many questions were raised during the general practitioners’ daily work that were thenpursued in order to obtain answers (7 questions a week of which 5 were pursued).This showed that Dutch general practitioners were well motivated and determined toobtain answers to their questions. However, the topics of the questions posed needsome consideration. When we compared the number of questions posed by generalpractitioners with the morbidity rates of the actual diseases concerned, we canconclude that general practitioners tended to be obsessed with rarer conditions andignore the commoner ones.The information sources general practitioners used were authority based, such aspeople and books, rather than evidence-based sources. Electronic sources such as theCochrane Library, which includes systematic reviews, were not or hardly used. This isnot unusual. A recent Australian study found that only 4% of the general practitionershad ever used it.18 However, the Dutch general practitioners expressed a need forreviews which were evidence-based, more medical guidelines, and computer-accessiblemedical information. Since evidence-based reviews19 and medical guidelines20 havebeen published, and computer-accessible medical information will increasingly beavailable on the Internet, the focus must now be on training and developing skills tosearch efficiently. At the same time, general practitioners’ need for improvement in thequality of the Internet should be taken seriously. The Internet is not subject to peerreview and we should have a debate on this, as we might obtain erroneous informationfrom it; certainly patients do.To conclude, Dutch general practitioners seemed to have a slightly positive attitude toevidence-based medicine. They believe evidence-based medicine could have an impacton patient care, they raise and pursue patient care questions, and they express a needfor evidence-based information. However, they do not use evidence-based informationsources. Human based resources still are the preferred sources, and it is not surewhether they will wish to change human based resources, with the accompanyingvaluable social aspects, to the impersonal computer based information sources. Wecan, however, continue to point them to the new electronic information sources thatcould be important to them. Only after we have directed much energy intofamiliarising general practitioners with evidence-based information sources and intotraining activities, will we know which information sources general practitioners preferto use.

RECOMMENDATIONS

Recommendations to general practitionersGeneral practitioners used easily available information sources such as colleagues andbooks. In our literature study we found that the existence of an office library increasedthe availability and use of information. Thus, in larger practices and in collaborativegroups of general practitioners, we would recommend the establishment of a practicelibrary with easy access to a variety of information sources.11 An appointed practicelibrarian for half a day a week could make a significant impact on the management ofprinted information and on facilitating access to external electronic sources - a

93

university information network, for example. This service should include an onlinefacility to search sources such as Medline on keywords for articles and to look forunbiased and concise information on effectiveness and evidence-based informationfrom the Cochrane Library and Best Evidence.19,21 Subscription to 15 to 35 journals,and a collection of 300-450 books would be an appropriate collection.22

General practitioners experience an information overload and time restraints. Theycould consider delegating their questions to medical librarians, as intermediaries, whocould provide tailored information services.23,24 Professional organisations for generalpractitioners could consider setting up a telephone, e-mail or fax medical informationservice which could provide bibliographic searches performed by physicians trained inliterature evaluation methods and computerised databases.25

Although some tasks could be delegated, general practitioners need to have surgery-based access to medical databases, and therefore to learn the required information-searching skills in post-graduate education courses.

Recommendations to researchers in general practice and library and informationstudiesBecause general practitioners experience the journal literature as rather inaccessible,researchers in general practice should focus on clinical applicability and the easilyaccessible structure of journal articles. Furthermore, general practitioners needelectronic evidence-based summaries of the medical literature relevant to them.Although some products already exist,19,26 high quality, patient-focused, and easy touse products need to be developed.More studies should be performed that use patient outcome as their endpoint. Thesestudies could look for a positive correlation between physicians’ use of the medicalliterature and clinical outcome.4,27-31 Rapid access to the medical literature favourablyinfluences patient outcome,30 and patients for whom Medline searches were performedearlier in their hospitalisations had significantly lower costs, charges, and length ofstay than did those whose searches were conducted later.31

Recommendations for medical educatorsMedical educators have a role in teaching, planning and curriculum development.General practitioners’ information-seeking has two goals: fulfilling an immediate need,and adding to the knowledge base for future use. These two goals need differentstrategies. Whereas an immediate need for patient care questions may be quickly metby searching databases such as Medline and evidence-based electronic informationsources such as the Cochrane Library, continuing medical education goals could bestbe met by screening either a restricted number of relevant journals, or summarized andcritically appraised journal articles such as those in the journals Evidence-BasedMedicine and ACP Journal Club.32

But more than accessibility is needed. For general practitioners, key aspects of apractice information strategy are likely to be the education and training of staff in thevalue and use of information and information technology.11,32 In addition to appropriateequipment, and tailored and easy accessible information sources, general practitionersneed training in information-handling and searching skills, in critical appraisal and inthe sources that are most appropriate for different information needs, including the

94

Internet. Self-paced continuing education modules, on CD-ROM or on the Internethave the potential to be readily available, interactive, and clinically focused, and couldbe a solution to the time restraints general practitioners experience. However, generalpractitioners appreciate relating with colleagues during a continuing education course.Additionally, they prefer to leave their practices and patients when they want to workundisturbed.Teaching information skills should start in the medical school curriculum, and in thevocational training concentrate on specific sources for general practitioners. Astructured examination should be developed for locating, selecting, and appraising themedical literature for students33 as well as for general practitioners trainees,35,36 andgeneral practitioners.36

Recommendations for medical librariansMedical librarians can provide knowledge resources that are suited to generalpractitioners’ needs: patient oriented clinical research, review journals and serials,clinical guidelines, and consensus statements.22 Medical librarians can work withknowledge resource providers to modify existing resources, encouraging thepublication of more patient-oriented research and suggesting bibliographic terms thathelp practitioners identify patient-oriented material. Medical librarians can work withcontinuing medical education developers to utilize information technologies andlearning strategies that support relevant, interactive, and readily available learningopportunities, for example teaching electronic workshops via the Internet.24 Medicallibrarians can improve primary care physicians’ familiarity with the wide array ofavailable resources and advise health care providers on how to assess the likelihoodthat a resource can supply the desired information. Medical librarians can considerdoing the information-seeking for busy general practitioners. They can assist generalpractitioners by increasing their familiarity with and access to computers. To maximisethe usefulness of computer-based information and resources, librarians mustdemonstrate to practitioners that the resources are faster and easier to use and containinformation that is equal to or better than information found in print resources in termsof clinical applicability.6 Functional accessibility could be improved when appropriatedescriptors in printed or computerized bibliographies of general practice are beingdeveloped. Finally, in addition to instruction in use of traditional, formal sources suchas journals, family doctors must be taught to exploit personal information sources, forinstance by using e-mail.

REFERENCES

1 Leckie GJ, Pettigrew KE, Sylvain C. Modeling the information seeking of professionals: a generalmodel derived from research on engineers, health care professionals, and lawyers. Libr Q1996;66:161-93.



4 Wood F, Wright P, Wilson T. The impact of information use on decision making by generalmedical practitioners. [British Library R&D Report]. Boston Spa: British Library; 1995.

5 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Use of information sources by family

95

physicians: a literature survey. Bull Med Libr Assoc 1995;83:85-90.6 Thompson ML. Characteristics of information resources preferred by primary care physicians.

Bull Med Libr Assoc 1997;85:187-92.7 Gorman PN. Information needs of physicians. J Am Soc Inform Sci 1995;46:729-36.8 Forsythe DE, Buchanan BG, Osheroff JA, Miller RA. Expanding the concept of medical

information: an observational study of physicians’ information needs. Comput Biomed Res1992;25:181-200.


10 Gruppen LD, Wolf FM, Voorhees C Van, Stross JK. Information-seeking strategies anddifferences among primary care physicians. Möbius 1987;7:18-26.

11 O’Mahony B, Culhane A, Rouse JM, Ryan F, Shannon W. Keeping up to date - a challenge forteaching practices. Ir Med J 1995;88:170-1.


13 Gorman PN, Ash J, Wykoff L. Can primary care physicians’ questions be answered using themedical journal literature? Bull Med Libr Assoc 1994;82:140-6.



16 Zaat J. Worstelend met de literatuur: zoeken in Medline via Internet - een vergelijking van vijfzoekmachines [Struggling with the literature: searching Medline on the Internet - a comparison offive search engines. In Dutch]. Huisarts Wet 1999;42:11-7.

17 Gill P, Dowell AC, Neal RD, Smith N, Heywood P, Wilson AE. Evidence based general practice:a retrospective study of interventions in one training practice. BMJ 1996;312:819-21.

18 Young JM, Ward JE. General practitioners’ use of evidence databases. Med J Aus 1999;170:56-8.19 Cochrane Library [database on disk, CD-ROM and Internet]. Cochrane Collaboration; Issue 4,

October 1998. Oxford: Update Software; 1998. Updated quarterly.20 Grol R, Dalhuijsen J, Thomas S, Veld C in ’t, Rutten G, Mokkink H. Attributes of clinical


21 Best Evidence [database on CD-ROM]. Philadelphia: American College of Physicians, 1996.22 Bryant SL. Practice libraries: managing printed information and meeting the information needs of

staff in general practice. Health Libr Rev 1997;14:9-21.23 Lundeen GW, Tenopir C, Wermager P. Information needs of rural health care practitioners in

Hawaii. Bull Med Libr Assoc 1994;82:197-205.24 Coggan JM, Crandall LA. Expanding rural primary care training by employing information

technologies: the need for participation by medical reference librarians. Med Ref Serv Q1995:14:9-16.

25 Nobili A, Gebru F, Rossetti A, Schettino F, Zahn RW, Nicolis E, Macario G, Celani L, Acik VO,Farina M, Naldi L. Doctorline: a private toll-free telephone medical information service; fiveyears of activity: old problems and new perspectives. Ann Pharmacother 1998;32:120-5.

26 De Geïnformeerde Huisarts [database on disk]. Houten: Bohn, Stafleu & Van Loghum; 1992.27 Marshall JG. The impact of the hospital library on clinical decision making: the Rochester study.

Bull Med Libr Assoc 1992;80:169-78.28 Urquhart CJ, Hepworth JB. Comparing and using assessments of the value of information to

clinical decision-making. Bull Med Libr Assoc 1996;84:482-9.29 King DN. The contribution of hospital library information services to clinical care: a study in

eight hospitals. Bull Med Libr Assoc 1987;75:291-301.30 Lindberg DAB, Siegel ER, Rapp BA, Wallingford KT, Wilson SR. Use of MEDLINE by

physicians for clinical problem solving. JAMA 1993;269:3124-9.31 Klein MS, Ross FV, Adams DL, Gilbert CM. Effect of online literature searching on length of

stay and patient care costs. Acad Med 1994;69:489-95.

96

32 Dawes MG. On the need for evidence-based general and family practice. Evidence-BasedMedicine 1996;1:68-9.

33 Woods SE, Francis BW. MEDLINE as a component of the objective structured clinicalexamination: the next step in curriculum integration. Bull Med Libr Assoc 1996;84:108-9.

34 Commissie Wetenschappelijk Onderzoek NHG. Kritisch lezen van artikelen; een leidraad om dewaarde van onderzoeksartikelen te bepalen. [Critically reading of articles; a guideline to assess thevalue of research articles. In Dutch]. Commissie Wetenschappelijk Onderzoek NHG. 3e druk.Utrecht: Nederlands Huisartsen Genootschap; 1996.

35 Moore R. MRCGP examination, a guide for candidates and teachers. London: Royal College ofGeneral Practitioners; 1994.

36 Wood F, Palmer J, Ellis D, Simpson S, Bacigalupo R. Information in primary health care. HealthLibr Rev 1995;12:295-308.

97

98

Alphabetical list of references

1 Arndt KA. Information excess in medicine: overview, relevance to dermatology, and strategies forcoping. Arch Dermatol 1992;128:1249-56.

2 Arnold CA. Literatuuronderzoek onder de loep: van het vermijden van valkuilen naar eenmethodologie. [Literature research scrutinized: from avoiding pitfalls to a methodology. In Dutch].Open 1990;22:411-5.


4 Best Evidence [database on CD-ROM]. Philadelphia: American College of Physicians; 1996.5 Boerma EJ. Naar een methodologie literatuuronderzoek: enige wetenschapstheoretische

beschouwingen. [Going to a methodology for literature research: some theoretical, scientificconsiderations. In Dutch]. Open 1989;21:2-7.

6 Boerma EJ, Arnold CA. The library, information services and user education and the aims of theuniversity: the contribution of the methodology of (literature) research. Iatul Q 1990;4:56-64.

7 Bouter LM. Meta-analyse: controleerbaar en reproduceerbaar literatuuronderzoek als basis voorrationele beslissingen in de gezondheidszorg. [Meta-analysis: verifiable and reproducible literatureresearch as basis for rational decision-making in healtcare. In Dutch]. Amsterdam: AmsterdamUniversity Press; 1994. p. 7-8.

8 Braun EE, Kumar L, editors. Gale Directory of databases. Volume 1: Online databases. Detroit,MI: Gale Research; 1998. p. 384.

9 Bryant SL. Practice libraries: managing printed information and meeting the information needs ofstaff in general practice. Health Libr Rev 1997;14:9-21.

10 Chalmers I, Altman DG, editors. Systematic Reviews. London: BMJ; 1995.11 Chambliss ML, Conley J. Answering clinical questions. J Fam Pract 1996;43:140-4.12 Chan M, Fox NJ, Clamp SE, Dombal FT de. An information technology course in the medical

curriculum. Med Educ 1996;30:112-20.13 Chen C. How do scientists meet their information needs? Spec Libr 1974;65:272-80.14 Cochrane Library [database on disk, CD-ROM and Internet]. Cochrane Collaboration; Issue 2,

March 1999. Oxford: Update Software; 1999. Updated quarterly.15 Cochrane Library [database on disk, CD-ROM and Internet]. Cochrane Collaboration; Issue 4,

October 1998. Oxford: Update Software; 1998. Updated quarterly.16 Coggan JM, Crandall LA. Expanding rural primary care training by employing information

technologies: the need for participation by medical reference librarians. Med Ref Serv Q1995:14:9-16.

17 Commissie Wetenschappelijk Onderzoek NHG. Kritisch lezen van artikelen; een leidraad om dewaarde van onderzoeksartikelen te bepalen. [Critically reading of articles; a guideline to assess thevalue of research articles. In Dutch]. Commissie Wetenschappelijk Onderzoek NHG. 3e druk.Utrecht: Nederlands Huisartsen Genootschap; 1996.

18 Connelly DP, Rich EC, Curley SP, Kelly JT. Knowledge resource preferences of familyphysicians. J Fam Pract 1990;30:353-9.

19 Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinicaldecisions. Ann Intern Med 1997;126:376-80.

20 Covell DG, Uman GC, Manning PR. Information needs in office practice: are they being met?Ann Intern Med 1985;103:596-9.

21 Crandall S, Elson R, McLaughlin C. Managing and communicating information in a new era. FamMed 1997;29:270-4.

99

22 Creth SD. The health information environment: a view of organizational and professional needsand priorities. Bull Med Libr Assoc 1993;81:414-20.


24 Curley SP, Connelly DP, Rich EC. Physicians’ use of medical knowledge resources: preliminarytheoretical framework and findings. Med Decis Making 1990;10:231-41.

25 Dawes MG. On the need for evidence-based general and family practice. Evidence-BasedMedicine 1996;1:68-9.

26 Dee C, Blazek R. Information needs of the rural physician: a descriptive study. Bull Med LibrAssoc 1993;81:259-64.

27 Durack DT. The weight of medical knowledge. N Engl J Med 1978;298:773-5.28 Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet

1991;337:867-72.29 Edwards AGK, Russell IT, Stott NCH. Signal versus noise in the evidence base for medicine: an

alternative to hierarchies of evidence? Fam Pract 1998;15:319-22.30 Ely JW, Osheroff JA, Ferguson KJ, Chambliss ML, Vinson DC, Moore JL. Lifelong self-directed

learning using a computer database of clinical questions. J Fam Pract 1997;45:382-8.31 Ely JW, Burch RJ, Vinson DC. The information needs of family physicians: case-specific clinical

questions. J Fam Pract 1992;35:265-9.32 Evidence-Based Medicine Working Group. Evidence-based medicine: a new approach to teaching

the practice of medicine. JAMA 1992;268:2420-5.33 Forsythe DE, Buchanan BG, Osheroff JA, Miller RA. Expanding the concept of medical

information: an observational study of physicians’ information needs. Comput Biomed Res1992;25:181-200.

34 Gardner M. Information retrieval for patient care. BMJ 1997;314:950-3.35 Garfield E. SCI Journal Citation Reports; a bibliometric analysis of science journals in the ISI

database. Philadelphia: Institute for Scientific Information; 1994. p. 10.36 Garfield E. SSCI Journal Citation Reports; a bibliometric analysis of social science journals in the

ISI database. Philadelphia: Institute for Scientific Information; 1995. p. 40-6.37 Garfield E. SCI Journal Citation Reports; a bibliometric analysis of science journals in the ISI

database. Philadelphia: Institute for Scientific Information; 1994. p. 72-93.38 De Geïnformeerde Huisarts [database on disk]. Houten: Bohn, Stafleu & Van Loghum; 1992.39 Gerstberger PG, Allen TJ. Criteria used by research and development engineers in the selection of

an information source. J Appl Psychol 1968;52:272-9.40 Geyman JP. Evidence-based medicine in primary care: an overview. J Am Board Fam Pract

1998;11:46-56.41 Gill P, Dowell AC, Neal RD, Smith N, Heywood P, Wilson AE. Evidence based general practice:

a retrospective study of interventions in one training practice. BMJ 1996;312:819-21.42 Gorman PN. Information needs of physicians. J Am Soc Inform Sci 1995;46:729-36.43 Gorman PN, Ash J, Wykoff L. Can primary care physicians’ questions be answered using the

medical journal literature? Bull Med Libr Assoc 1994;82:140-6.44 Gorman PN, Helfand M. Information seeking in primary care: how physicians choose which

clinical questions to pursue and which to leave unanswered. Med Decis Making 1995;15:113-9.45 Graham DL, King C, Whitney PJ. Simultaneous remote search: on-line bibliographic library

services for practicing physicians. JAMA 1981;246:1115-6.46 Greenhalgh T. How to read a paper: the basics of evidence based medicine. London: BMJ; 1997.47 Grol R, Dalhuijsen J, Thomas S, Veld C in ’t, Rutten G, Mokkink H. Attributes of clinical


48 Gruppen LD, Wolf FM, Voorhees C Van, Stross JK. Information-seeking strategies anddifferences among primary care physicians. Möbius 1987;7:18-26.

49 Guyatt GH, Rennie D. Users’ guides to the medical literature [editorial]. JAMA 1993;270:2096-7.50 Harris DL, Peay WJ, Lutz LJ. Using microcomputers in rural preceptorships. Fam Med

1989;21:35-7.

100

51 Harter SP. Psychological relevance and information science. J Am Soc Inform Sci 1992;43:602-15.


53 Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC. Developing optimal searchstrategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc1994;1:447-58.


55 Hersh W. Relevance and retrieved evaluation: perspectives from medicine. J Am Soc Inform Sci1994;45:201-6.

56 Huth EJ. The information explosion. Bull N Y Acad Med 1989;65:647-61.57 Huth EJ. The underused medical literature. Ann Intern Med 1989;110:99-100.58 Janes JW, McKinney R. Relevance judgments of actual users and secondary judges: a

comparative study. Libr Q 1992;62:150-68.59 Kaner EFS, Haighton CA, McAvoy BR. "So much post, so busy with practice - so, no time!": a

telephone survey of general practitioners’ reasons for not participating in postal questionnairesurveys. Br J Gen Pract 1998;48:1067-9.

60 King DN. The contribution of hospital library information services to clinical care: a study ineight hospitals. Bull Med Libr Assoc 1987;75:291-301.

61 Klein MS, Ross FV, Adams DL, Gilbert CM. Effect of online literature searching on length ofstay and patient care costs. Acad Med 1994;69:489-95.

62 Klercker T af, Zetraeus S. Dilemmas in introducing World Wide Web-based informationtechnology in primary care: a focus group study. Fam Pract 1998;15:205-10.

63 Krikelas J. Information-seeking behavior: patterns and concepts. Drexel Libr Q 1983;19:5-20.64 Leckie GJ, Pettigrew KE, Sylvain C. Modeling the information seeking of professionals: a general

model derived from research on engineers, health care professionals, and lawyers. Libr Q1996;66:161-93.

65 Lindberg DAB, Siegel ER, Rapp BA, Wallingford KT, Wilson SR. Use of MEDLINE byphysicians for clinical problem solving. JAMA 1993;269:3124-9.

66 Line MB. The information uses and needs of social scientists: an overview of INFROSS. AslibProc 1971;23:412-34.

67 Lor PJ. The information needs of the general practitioner: a theoretical model based on ananalysis of the general practitioner’s cognitive environment [dissertation]. Pretoria: University ofSouth Africa; 1979.

68 Lovas I. A look at LOANSOME DOCtm service. Bull Med Libr Assoc 1994;82:176-80.69 Lundberg GD. Perspective from the editor of JAMA, the Journal of the American Medical

Association. Bull Med Libr Assoc 1992;80:110-4.70 Lundeen GW, Tenopir C, Wermager P. Information needs of rural health care practitioners in

Hawaii. Bull Med Libr Assoc 1994;82:197-205.71 MacAuley D, McCrum E, Brown C. Randomised controlled trial of the READER method of

critical appraisal in general practice. BMJ 1998;361:1134-7.72 Marshall JG. The impact of the hospital library on clinical decision making: the Rochester study.

Bull Med Libr Assoc 1992;80:169-78.73 McColl A, Smith H, White P, Field J. General practitioners’ perceptions of the route to evidence

based medicine: a questionnaire survey. BMJ 1998;316:361-5.74 McKibbon KA, Walker-Dilks CJ. The quality and impact of MEDLINE searches performed by

end users. Health Libr Rev 1995;12:191-200.75 Meyman FJ. Het literatuuroverzicht: waardevol door systematiek en structuur. [The literature

review: useful because of its methodology and structure. In Dutch]. Huisarts en Wet 1990;33:2-3.76 Mizzaro S. Relevance: the whole history. J Am Soc Inform Sci 1997;48:810-32.77 Moore R. MRCGP examination, a guide for candidates and teachers. London: Royal College of

General Practitioners; 1994.78 Mulrow CD. The medical review article: state of the science. Ann Intern Med 1987;106:485-8.

101

79 Mulrow CD. Rationale for systematic reviews. BMJ 1994;309:597-9.80 Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of

evidence [editorial]. Ann Intern Med 1997;126:389-91.81 Nobili A, Gebru F, Rossetti A, Schettino F, Zahn RW, Nicolis E, Macario G, Celani L, Acik VO,

Farina M, Naldi L. Doctorline: a private toll-free telephone medical information service; fiveyears of activity: old problems and new perspectives. Ann Pharmacother 1998;32:120-5.

82 Northup DE, Moore-West M, Skipper B. Teaf SR. Characteristics of clinical informationsearching: investigation using critical incident technique. J Med Educ 1983;58:873-81.

83 O’Mahony B, Culhane A, Rouse JM, Ryan F, Shannon W. Keeping up to date - a challenge forteaching practices. Ir Med J 1995;88:170-1.

84 Osiobe SA. Sources of information for biomedical decision-making. Methods Inf Med1985;24:225-9.


86 Oxman AD, Guyatt GH. The science of reviewing research. Ann NY Acad Sci 1994;703:125-34.87 Oxman AD, Sackett DL, Guyatt GH, for the Evidence-Based Medicine Working Group. Users’

guides to the medical literature, I: how to get started. JAMA 1993;270:2093-5.88 Pao ML. Concepts of information retrieval. Englewood: Libraries Unlimited; 1989. p. 14.89 Park TK. The nature of relevance in information retrieval: an empirical study. Libr Q

1993;63:318-51.90 Peeters C, Stichele R Vander. A training course in medical documentation for family physicians.

In: Deschamps C, Walckiers M, editors. Medical libraries: cooperation and new technologies.Amsterdam: Elsevier; 1987. p. 235-41.

91 Prescott K, Lloyd M, Douglas HR, Haines A, Humphrey C, Rosenthal J, Watt I. Promotingclinically effective practice: general practitioners’ awareness of sources of research evidence. FamPract 1997;14:320-3.

92 Pringle M, Churchill R. Randomised controlled trials in general practice: gold standard or fool’sgold? BMJ 1995;311:1382-3.

93 Quintana Y. Intelligent medical information filtering. Intern J Med Informatics 1998;51:197-204.94 Rodenburg P. R&D-professionals als informatiegebruikers. Een inventarisatieonderzoek naar het

benutten van informatiebronnen. [Research & development professionals as information users. Aninventory investigation in the utilization of information sources. In Dutch]. Open 1992;24:2-6.

95 Rosenberg V. Factors affecting the preferences of industrial personnel for information gatheringmethods. Inf Storage Retrieval 1967;3:119-27.

96 Rowland JFB. The scientist’s view of his information system. J Documentation 1982;38:38-42.97 Sackett DL. Evidence-based medicine: how to practice and teach EBM. New York: Churchill

Livingstone; 1997. p. 2-16.98 Sangster B. Health care policy and general practice research. In: Lamberts H, Knottnerus JA,

Hofmans SB, Klaassen A, editors. General practice research in Dutch Academia. Proceedings of aworkshop; 1994 April 15; Amsterdam, the Netherlands. Amsterdam: Medical Committee; 1994. p.35-9.

99 Saracevic T. Relevance: a review of and a framework for the thinking on the notion ininformation science. J Am Soc Inform Sci 1975;26:321-43.

100 Schamber L, Eisenberg MB, Nilan MS. A re-examination of relevance: toward a dynamic,situational definition. Inform Processing & Management 1990;26:755-76.

101 Schimmel M. De huisarts in de informatie-maatschappij: een onderzoek naar de informatiebehoefte en -voorziening van huisartsen in de Achterhoek, Liemers en Zuid-Twente. [The GP inthe information society: a survey on the information needs of and information services for GPs inthe Achterhoek, Liemers and South-Twente. In Dutch]. Report of the Hogeschool IJselland,School for Information Studies]. Aalten: Van Schijndel BibliotheekmanagementInformatiebemiddeling; 1998. p. 17.

102 Sievert ME, McKinin EJ, Johnson ED, Reid JC, Mitchell JA. Beyond relevance - characteristicsof key papers for clinicians: an exploratory study in an academic setting. Bull Med Libr Assoc1996;84:351-8.

102

103 Slawson DC, Shaughnessy AF. Obtaining useful information from expert based sources. BMJ1997:314:947-9.

104 Smith R. What clinical information do doctors need? BMJ 1996;313:1062-8.105 Solla Price D de. The development and structure of the biomedical literature. In: Warren K.S.

Coping with the biomedical literature. New York: Praeger Publications; 1981. p. 3-16.106 Straus SE, Sackett DL. Using research findings in clinical practice. BMJ 1998;317:339-42.107 Su LT. Evaluation measures for interactive information retrieval. Inform Processing &

Management 1992;28:503-16.108 Su LT. The relevance of recall and precision in user evaluation. J Am Soc Inform Sci

1994;45:207-17.109 Thompson ML. Characteristics of information resources preferred by primary care physicians.

Bull Med Libr Assoc 1997;85:187-92.110 Timpka T, Ekström M, Bjurulf P. Information needs and information seeking behaviour in

primary health care. Scand J Prim Health Care 1989;7:105-9.111 Timpka T, Arborelius E. The GP’s dilemmas: a study of knowledge need and use during health

care consultations. Methods Inf Med 1990;29:23-9.112 URL: http://igm.nlm.nih.gov113 URL: http://www.silverplatter.com114 URL: http://www.krinfo.ch115 Urquhart CJ, Hepworth JB. Comparing and using assessments of the value of information to

clinical decision-making. Bull Med Libr Assoc 1996;84:482-9.116 Verhoeven AAH. Wetenschappelijk onderzoek door de huisarts: een geannoteerde bibliografie

over algemene aspecten van wetenschapsbeoefening voor huisartsen, en het praktisch verrichtenvan wetenschappelijk onderzoek in de huisartspraktijk, samengesteld uit publikaties verschenen na1970. [Research by the general practitioner: an annotated bibliography on general aspects ofperforming research and searching the literature by general practitioners; compiled of publicationspublished after 1970. In Dutch]. Report made for the Postdoctoral Training in Library andInformation Studies. University of Amsterdam; 1990.

117 Verhoeven AAH. Literatuuronderzoek door huisartsen: methoden van literatuurverzamelen voorwetenschappelijk onderzoek door huisartsen - onderzoeksvoorstel -. [Searching the literature:methods of literature gathering for research purposes by general practitioners - a research proposal-. In Dutch]. Report made for the Postdoctoral Training in Library and Information Studies.University of Amsterdam; 1991.


119 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Use of information sources by familyphysicians: a literature survey. Bull Med Libr Assoc 1995;83:85-90.

120 Vorst HCM. Methodische aspecten van literatuuronderzoek. [Methodological aspects of literatureresearch. In Dutch]. In: Kallenberg A, Kalle P. editors. Automatiseringsaspecten vanonderwijsdocumentatie: verslag van een informatiemarkt. [Aspects of computerization ofeducational literature: report of an information market. In Dutch]. Den Haag: SVO; 1984. p.59-75.

121 Weijden T van der, IJzermans CJ, Dinant GJ, Duijn NP van, Vet R de, Buntinx F. Identifyingrelevant diagnostic studies in Medline. The diagnostic value of the erythrocyte sedimentation rate(ESR) and dipstick as an example. Fam Pract 1997;14:204-8.

122 Weller AC. Editorial policy and the assessment of quality among medical journals. Bull Med LibrAssoc 1987;75:310-6.

123 Werf GT van der, Smith RJA, Stewart RE, Meyboom-de Jong B. Spiegel op de huisarts: overregistratie van ziekte, medicatie en verwijzingen in de geautomatiseerde huisartspraktijk. [View onthe general practitioner: on the registration of diseases, medication, and referrals in thecomputerised general practice office. In Dutch]. Groningen: Morbidity Registration NetworkGroningen; 1998. p. 124-36.

124 Wetenschappelijke tijdschriften gezondheidszorgonderzoek. In: Richtlijnen aanlevering gegevensten behoeve van het Discipline-advies Geneeskunde 1998. Commissie Geneeskunde van de

103

Koninklijke Nederlandse Akademie van Wetenschappen, en het Discipline OverlegorgaanMedische Wetenschappen van de Vereniging van Samenwerkende Nederlandse Universiteiten.[Additional scientific journals for health sciences research. In: Guidelines for provision ofinformation for the Discipline Report on (Bio)Medical and Health Sciences Research 1998.Medical Committee of the Royal Netherlands Academy of Arts and Sciences, and the Council ofMedical Faculties in the Netherlands of the Association of Universities in the Netherlands. InDutch]. Amsterdam, Utrecht; 1996. p. 57.

125 Williams ME. The state of databases today: 1998. In: Braun EE, Kumar L, editors. Gale directoryof databases. Detroit, MI: Gale Research; 1998. p. XXV.

126 Williamson JW, German PS, Weiss R, Skinner EA, Bowes F. Health science informationmanagement and continuing education of physicians: a survey of U.S. primary care practitionersand their opinion leaders. Ann Intern Med 1989;110:151-60.

127 Witiak JL, Prewitt BG, Deschere AR. Online database searching via telephone conferencing.Online 1979;3:21-3.

128 Wood F, Wright P, Wilson T. The impact of information use on decision making by generalmedical practitioners [British Library R&D Report]. Boston Spa: British Library; 1995.

129 Wood F, Palmer J, Ellis D, Simpson S, Bacigalupo R. Information in primary health care. HealthLibr Rev 1995;12:295-308.

130 Woods SE, Francis BW. MEDLINE as a component of the objective structured clinicalexamination: the next step in curriculum integration. Bull Med Libr Assoc 1996;84:108-9.

131 Woolf SH, Benson DA. The medical information needs of internists and pediatricians at anacademic medical center. Bull Med Libr Assoc 1989;77:372-80.

132 Wyatt J. Use and sources of medical knowledge. Lancet 1991;338:1368-73.133 Young JM, Ward JE. General practitioners’ use of evidence databases. Med J Aus 1999;170:56-8.134 Zaat J. Worstelend met de literatuur: zoeken in Medline via Internet - een vergelijking van vijf

zoekmachines [Struggling with the literature: searching Medline on the Internet - a comparison offive search engines. In Dutch]. Huisarts Wet 1999;42:11-7.

104

Summary

This thesis is about how general practitioners look for medical information.

Chapter 1 provides an overview of the main aspects of information-seeking bygeneral practitioners. The topics discussed are: the influence of the informationexplosion, general practitioners’ information needs, an information-seeking model, andthe subjects who took part in our studies.

Chapter 2 describes the literature relating to the information sources generalpractitioners use. Analysis of the use of information sources by general practitioners isimportant for both practical and theoretical reasons. First, discovering the ways inwhich general practitioners handle information may point to opportunities forimprovement. Second, such efforts may lead to improvements in the methods ofliterature research in general.Eleven relevant research publications published from 1975 to 1992 were found. Thedata showed that general practitioners used colleagues most often as informationsources, followed by journals and books. This corresponds with findings in otherprofessions.Several factors influenced the use of information sources by general practitioners,including:

physical, functional and intellectual accessibility of the source;age and social context of the physician;participation by the physician in research or teaching;practice characteristics; andstage of the information-gathering process.

The publications studied suggested ways to improve information-gathering in the areasof computerization, education, library organization, and journal articles.

Chapter 3 reports on the management of bibliographic information by generalpractitioners involved in research. As a result of changes in information technologyand the rapid growth of publications, methods of searching the literature have changed.Systematic searching of the growing literature has become very important. It is notknown whether researchers in general practice search systematically, and whether theyhave incorporated computerized sources in their research practice.We aimed to assess the methods of searching the literature used by generalpractitioners, and therefore interviewed eight general practitioner researchers, twoinformation specialists and a psychologist working in primary health careorganisations.The results showed that general practitioner researchers began their search in theirprivate book collections, or asked colleagues for information. Later in the search

105

process, they used computerized information sources. Medline on CD-ROM and thecomputer-based indexes of two Dutch journals were most widely used. The generalpractitioners found it difficult to locate the appropriate terms to describe their researchquestions. Knowledge of the methods of searching the literature, and the skills requiredto use the information sources efficiently were both lacking. There was a need forinstruction and training in the use of information sources. General practitioners neededa telephone help desk where their questions could be answered adequately.The conclusion is that general practitioner researchers search the literatureunsystematically. Although computerized databases were being used, knowledge andskills related to the use of these information sources should be improved.

Chapter 4 presents our study of the information use and needs of a sample of Dutchgeneral practitioners. This included the collection of suggestions for the improvementof information access for patient care and continuing medical education.We used a questionnaire, sent to a randomly selected sample of 500 registered Dutchgeneral practitioners, of whom 226 (45%) replied.We were especially interested in:

familiarity with evidence-based medicine;analysis of patient-specific questions arising in practice;use of information sources; andsuggestions for improvement of information access.

The results showed that 45% of the respondents had heard of evidence-basedmedicine, and 85% of the well-informed general practitioners expected a great changein patient management as a result of evidence-based medicine. The respondents had6.9 patient-specific questions a week, half relating to therapeutic problems. They mostfrequently used drug reference and other private books, and consultations with generalpractitioner colleagues. Although 93% owned a personal computer at work, theyseldom used it for patient-specific questions. Improvements concerned journal articles,which needed to be clearly organised and focused on daily practice, and evidence-based reviews, which should be easily accessible by computer. General practitionerswanted integration of guidelines and patient health information in the generalpractitioners’ Health Information System. We conclude that electronic informationsources have not yet changed the patterns of use of information sources by Dutchgeneral practitioner, whose needs were focused on easily accessible electronicinformation.

Chapter 5 sets out the development of a model to evaluate retrieval quality of searchqueries performed by general practitioners using the printed Index Medicus, Medlineon CD-ROM, and Medline through Grateful Med. Four search queries related togeneral practice were formulated for a continuing medical education course inliterature searching. The search queries were on:

haemorrhoids;prevention of sudden infant death;use of the telephone in general practice; andthe general practitioner as gate-keeper.

The potentially relevant citations selected by the course instructor and by the 103

106

participants in the course were presented to three judges, who evaluated them forrelevance and quality. The evaluation of quality was based on journal ranking, researchdesign and type of publication. Relevant individual citations received a citation qualityscore from 1 (low) to 4 (high). The overall search quality was expressed in a formula,which included the individual citation quality score of the selected and missedcitations, and the number of selected non-relevant citations. The outcome measureswere the number and quality of the relevant citations, and the agreement between thejudges.The results showed that, out of 864 citations, 139 were assessed as relevant. Of these,44 citations received an individual citation quality score of 1, 76 of 2, 19 of 3 andnone of 4. The level of agreement between the judges was 68% for the relevantcitations, and 88% for the non-relevant citations.The chapter thus proposes a model for the evaluation of retrieval quality of searchqueries, based not only on the relevance, but also on the quality of the citationsretrieved. With adaptation, this model could be generalized to other professional users,and to other bibliographic sources.

In chapter 6 the results of the comparative study of three methods of literatureretrieval are presented. Evidence-based medicine requires new skills of physicians,including literature searching. We determined which literature retrieval method wasmost effective for general practitioners: the printed Index Medicus, Medline throughGrateful Med, or Medline on CD-ROM.The design was a randomized comparative study. In a continuing medical educationcourse, three groups of health care professionals (87 general practitioners and 16 otherhealth care professionals) used one of the literature retrieval methods to retrievecitations on four search topics related to general practice. For the analysis in pairs, weused the search results of the 75 participants who completed all four assignments. Asoutcome measures we used:

precision and recall;an overall search quality score; anda post-course questionnaire on personal characteristics, experience with computers,

handling medical literature, and satisfaction with course instruction and search results.The results showed that the recall and overall search quality scores in the IndexMedicus groups (n=32) were higher (P≤0.001) than those in the CD-ROM groups(n=31). In addition, the search quality scores in the Grateful Med groups (n=12) werehigher (P<0.003) than those in the CD-ROM groups. There were no differences inprecision.We conclude that in the period from 1994 to 1997, the printed Index Medicus seemedthe most effective literature retrieval method for general practitioners. Forinexperienced general practitioners there is a need for training in electronic literatureretrieval methods.

In our final chapter, chapter 7, we discuss the results of our studies. In addition, wemake recommendations for general practitioners; researchers in general practice andlibrary and information studies; medical educators; and medical librarians.

107

Samenvatting

Deze dissertatie beschrijft het zoekgedrag naar medische informatie van huisartsen.

Hoofdstuk 1 geeft een overzicht van enkele aspecten van het zoekgedrag vanhuisartsen: de invloed van de informatie-explosie, de informatiebehoeften vanhuisartsen en een model voor hun zoekgedrag. Tenslotte beschrijven we enkelekenmerken van de beroepsgroep Nederlandse huisartsen.

In hoofdstuk 2 wordt d.m.v. een literatuuronderzoek geanalyseerd welkeinformatiebronnen huisartsen gebruiken. Om zowel praktische als theoretische redenenis het belangrijk het gebruik van informatiebronnen te analyseren. Ten eerste kan deanalyse blootleggen welke verbeteringen nodig zijn voor een meer efficiënt gebruikvan de juiste informatiebronnen. Ten tweede kan de analyse leiden tot verbetering vande methoden van literatuuronderzoek in het algemeen.Voor de analyse werd een literatuuronderzoek verricht naar het gebruik vaninformatiebronnen door huisartsen in de periode van 1975 tot 1992. Ditliteratuuronderzoek leverde elf relevante onderzoekpublikaties op. Hieruit bleek dat debelangrijkste informatiebron consultatie van collega-artsen was, gevolgd doortijdschriften en boeken. Deze gegevens komen overeen met die van andereberoepsgroepen.Er bleken verschillende factoren te bestaan die het gebruik van informatiebronnenbeïnvloedden:

fysieke, functionele en intellectuele toegankelijkheid van de bron,leeftijd en sociale context van de huisarts,verrichten van onderzoek en geven van onderwijs door huisartsen,kenmerken van de huisartsenpraktijk enstadium van het informatie-zoekproces.

De publikaties vermeldden vier gebieden die voor verbetering vatbaar waren:toepassing van computers, onderwijs, organisatie van de bibliotheek, en de inhoud enorganisatie van tijdschriftartikelen.

Hoofdstuk 3 doet verslag van een onderzoek naar het omgaan met bibliografischeinformatie door huisartsonderzoekers. De aanleiding voor het onderzoek was deveronderstelling dat de informatietechnologie en de snelle toename van publikatiesaanleiding zouden geven tot een veranderend zoekpatroon. Met name het systematischzoeken in deze snel groeiende hoeveelheid informatie is erg belangrijk geworden. Hetwas niet bekend of onderzoekers binnen de huisartsgeneeskunde wel systematischzoeken en of zij elektronische informatiebronnen in hun onderzoekspraktijk gebruiken.Daarom stelden we ons in dit hoofdstuk ten doel de methoden van literatuuronderzoekdoor huisartsonderzoekers te bestuderen. Voor dit doel interviewden we acht

109

huisartsonderzoekers, twee informatiespecialisten en een psycholoog. De laatste driewaren net als de huisartsonderzoekers werkzaam in de eerstelijnsgezondheidszorg.De resultaten laten zien dat huisartsonderzoekers hun literatuuronderzoek begonnenmet het doorzoeken van hun eigen boeken, of dat ze collega’s om informatie vroegen.Van de elektronische informatiebronnen werden Medline op CD-ROM en de indexenvan twee Nederlandse tijdschriften het meest gebruikt. Een probleem hierbij was hetvinden van de juiste zoektermen die de onderzoeksvraag beschreven. Daarbij ontbrakhet de huisartsonderzoekers aan kennis over de methoden van literatuuronderzoek ende vaardigheden die vereist zijn voor het efficiënt gebruik maken van elektronischeinformatiebronnen. Er bleek grote behoefte te zijn aan instructie en training in hetgebruik van informatiebronnen. De huisartsen suggereerden dat een telefonischehelpdesk voor literatuurvragen heel waardevol zou kunnen zijn.De conclusie van dit onderzoek is dat huisartsonderzoekers de literatuur niet op eensystematische wijze doorzoeken. Hoewel elektronische databases worden gebruikt,moeten kennis en vaardigheden over het gebruik van deze informatiebronnen verbeterdworden.

In hoofdstuk 4 onderzochten we het informatiegebruik en de informatiebehoeften vanNederlandse huisartsen. Ons doel was om naast het vaststellen van het gebruik enbehoeften van informatie ook suggesties te verzamelen om de toegang tot informatie teverbeteren voor zowel de patiëntenzorg als voor nascholing. Voor dit doel gebruiktenwe een zelf ontworpen vragenlijst. Deze stuurden we toe aan 500 huisartsen, diesteekproefsgewijs waren geselecteerd uit alle geregistreerde huisartsen in Nederland.We ontvingen 226 (45%) ingevulde vragenlijsten terug. Onze uitkomstmaten wareno.a.:

vertrouwdheid met evidence-based medicine,analyse van patiënt-specifieke vragen uit de praktijk,gebruik van informatiebronnen ensuggesties voor verbetering van de toegang tot informatie.

De resultaten laten zien dat 45% van de reflectanten had gehoord van evidence-basedmedicine en dat 85% van de goed geïnformeerde huisartsen ook een grote veranderingverwachtte in de patiëntenzorg als gevolg van het praktiseren van evidence-basedmedicine. De reflectanten hadden 6.9 patiënt-specifieke vragen per week, waarvan dehelft therapeutische problemen betrof. De meest gebruikte informatiebronnen warennaslagwerken voor geneesmiddelen, andere boeken uit eigen bezit en consultatie vancollega-huisartsen. Ofschoon 93% een personal computer op het werk bezat, werd dezezelden gebruikt voor patiënt-specifieke vragen. Suggesties voor verbetering van detoegang tot informatie waren het beter organiseren en meer praktisch gericht makenvan tijdschriftartikelen. Daarnaast had men behoefte aan evidence-basedoverzichtsartikelen, die toegankelijk moesten zijn via de computer. Als laatste wildende huisartsen integratie van richtlijnen en voorlichtingsmateriaal voor patiënten in hetHuisartsen Informatie Systeem (HIS).We concluderen dat elektronische informatiebronnen nog niet het patroon van hetgebruik van informatiebronnen door huisartsen hebben beïnvloed. Toch zijn deinformatiebehoeften van huisartsen gericht op gemakkelijk toegankelijke elektronischeinformatiebronnen.

110

Hoofdstuk 5 beschrijft de ontwikkeling van een model ter evaluatie van deopbrengstkwaliteit van zoekacties. Het betreft zoekacties die verricht zijn doorhuisartsen in drie medische bibliografische bronnen: de gedrukte Index Medicus,Medline op CD-ROM, en Medline via Grateful Med. De zoekopdrachten vormden eenonderdeel van een nascholingscursus in het literatuurzoeken. Er waren vierverschillende zoekopdrachten geformuleerd die allen te maken hadden met dehuisartsenpraktijk. Zij betroffen:

de ziekte aambeien,de preventie van wiegendood,het gebruik van de telefoon in de huisartsenpraktijk ende functie van de huisarts als poortwachter.

De geselecteerde, potentieel relevante literatuurverwijzingen van de cursusleider en de103 cursisten vormden samen de basisset van literatuurverwijzingen. Dezeliteratuurverwijzingen werden door drie beoordelaars geëvalueerd op hun relevantie enkwaliteit, waarbij de laatste gebaseerd was op tijdschriftrangorde, onderzoeksontwerpen publikatie-type. Relevante individuele literatuurverwijzingen kregen eenkwaliteitsscore (citation quality score) die varieerde van 1 (laag) tot 4 (hoog). Dekwaliteit van de hele zoekactie kwam tot uitdrukking in een formule waar deindividuele kwaliteitsscore van de geselecteerde en gemiste relevante literatuurverwij-zingen en tevens het aantal geselecteerde niet relevante literatuurverwijzingen inverdisconteerd waren (overall search quality score). Onze uitkomstmaten waren dus dekwaliteit en het aantal van de relevante literatuurverwijzingen, en de overeenstemmingtussen de beoordelaars.De resultaten van onze studie lieten zien dat van de 864 literatuurverwijzingen er 139als relevant door de beoordelaars werden aangemerkt. Van deze relevanteliteratuurverwijzingen ontvingen er 44 een individuele kwaliteitsscore van 1, 76 van 2,19 van 3, en geen enkele van 4. De overeenstemming tussen de beoordelaars was 68%voor de relevante literatuurverwijzingen, en 88% voor de niet-relevanteliteratuurverwijzingen.Samengevat beschrijven we in dit hoofdstuk een model voor de evaluatie vanzoekacties dat niet alleen gebaseerd is op de relevantie, maar ook op de kwaliteit vande verkregen literatuurverwijzingen. Na geringe aanpassing kan dit model zowel doorandere professionals als voor andere bibliografische bronnen worden gebruikt.

In hoofdstuk 6 worden de resultaten van de vergelijkende studie gepresenteerd vandrie methoden voor het verkrijgen van literatuurverwijzingen. Evidence-based medicinevereist heden ten dage nieuwe vaardigheden van huisartsen, waaronder hetliteratuurzoeken. Om vast te stellen welke van drie literatuursystemen (de gedrukteIndex Medicus, Medline op CD-ROM of Medline via Grateful Med) het meesteffectief is voor huisartsen ontwierpen we een gerandomiseerde vergelijkende studie.Op een bijscholingscursus gebruikten 87 huisartsen en 16 andere werkers uit degezondheidszorg een van de drie literatuursystemen om literatuurverwijzingen teverzamelen over vier zoekvragen relevant voor de huisartsgeneeskunde.

111

Als uitkomstmaten gebruikten we:precisie en recall,de algehele zoekkwaliteitsscore (overall search quality score) eneen vragenlijst over de persoonsgegevens, ervaring met computers, omgaan met

medische literatuur en de evaluatie van de instructie gegeven in de cursus en dezoekresultaten.Voor de analyse in paren gebruikten we de zoekresultaten van alleen de 75 cursistendie alle vier zoekvragen hadden voltooid.De uitkomsten van ons vergelijkend onderzoek laten zien dat de recall en de algehelezoekkwaliteitsscores in de Index Medicus groepen (n=32) hoger waren (P≤0.001) dandie in de CD-ROM groepen (n=31). Tevens waren de algehele zoekkwaliteitsscores inde Grateful Med groepen (n=12) hoger (P<0.003) dan die in de CD-ROM groepen. Erwaren geen verschillen in precisie.We concluderen dat in de periode van 1994 tot 1997 voor huisartsen de gedrukteIndex Medicus de meest effectieve methode voor literatuurverzamelen was. Wil meneffectief gebruik maken van elektronische informatiebronnen, dan moeten huisartsenzich hierin trainen.

In het laatste hoofdstuk, hoofdstuk 7, worden de resultaten besproken van devoorgaande studies. Tevens doen we aanbevelingen voor huisartsen, onderzoekers inde huisartsgeneeskunde en in de bibliotheek- en informatiewetenschappen, docenten inhet medisch onderwijs en medewerkers van medische bibliotheken.

112

List of publications

Journal articles and letters1 Bakker S, Verhoeven AAH. Thuis in geautomatiseerd literatuuronderzoek. [At home in computerized

literature research. In Dutch]. Tijdschr Med Informatica 1992;21:70-4.2 Volkers ACW, Verhoeven AAH, Otten RHJ, Dijkman JA, Deurenberg-Vos HWJ, Brand-de Heer DL,

Bakker S. Medline op CD-ROM; een evaluatie. [Medline on CD-ROM: an evaluation. In Dutch].Login 1993;17:13-8.

3 Verhoeven AAH. Internationale toegankelijkheid H&W [letter]. [International accessibility of H&W.In Dutch]. Huisarts Wet 1994;37:410.

4 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Use of information sources by family physicians:a literature survey. Bull Med Libr Assoc 1995;83:85-90.

5 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Het gebruik van informatiebronnen doorhuisartsen: een literatuurstudie. [Use of information sources by family physicians: a literature survey.In Dutch]. Huisarts Wet 1996;39:12-5.


7 Verhoeven A. Citing old research may mislead readers [letter]. BMJ 1997;314:905.8 Verhoeven AAH, Boendermaker PM, Boerma EJ, Meyboom-de Jong B. A model for retrieval quality

of search queries in three medical bibliographic sources. Health Libr Rev. In press.

Conference proceedings and abstracts1 Verhoeven AAH. Het gebruik van informatiebronnen door huisartsen: een literatuurstudie. [Use of

information sources by general practitioners: a literature survey. In Dutch]. In: Online InformatieKonferentie Nederland; 1994 Apr 19-20; de Doelen, Rotterdam: lezingen. Den Haag: NBLC; 1994.p. 119-25.

2 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Which literature retrieval method is mosteffective? Teaching the Index Medicus/Medline in post-graduate courses for family physicians. In:The Seventh Ottawa International Conference on Medical Education and Assessment; 1996 June 25-28; Maastricht, The Netherlands: abstract book. Maastricht: Faculty of Medicine of the Universityof Limburg, The Netherlands; 1996. p. 442.

3 Omgaan met wetenschappelijke literatuur [verslag van de lezingen van A. Verhoeven op het NHG-Jubileumcongres 1996]. [Managing scientific literature - report of lectures of A. Verhoeven at theJubilee Congres of the Dutch College of General Practitioners -. In Dutch]. NHG-katern bij HuisartsWet 1997;40:137.

4 Verhoeven AAH, Boerma EJ, Meyboom-de Jong B. Which bibliographic retrieval method is mosteffective? In: Bakker S, editor. Health information management: what strategies? Proceedings of the5th European Conference of Medical and Health Libraries; 1996 Sep 18-21; Coimbra, Portugal.Dordrecht: Kluwer, 1997. p. 118-9.

Posters1 Nederlands Huisarts Genootschap Referatendag; Amsterdam, The Netherlands; 1996 June 14. Title:

"Which bibliographic retrieval method is most effective for general practitioners?"2 Library Association Health Libraries Group; Exeter, U.K.; 1996 Sep 6-8. Title: "Which bibliographic

retrieval method is most effective for general practitioners?"3 Fifth European Conference of Medical and Health Libraries; Coimbra, Portugal; 1996 Sep 18-21.

Title: "Which bibliographic retrieval method is most effective for general practitioners?"

113

Documents

University of Groningen Information-seeking by general ... · Samenvatting 109 List of publications 113. Preface ... 4 Verhoeven AAH. Wetenschappelijk onderzoek door de huisarts: