Cdss Practitioner Performance and Patient Outcomes

REVIEW

Effects of Computerized Clinical DecisionSupport Systems on Practitioner Performanceand Patient OutcomesA Systematic ReviewAmit X. Garg, MDNeill K. J. Adhikari, MDHeather McDonald, MScM. Patricia Rosas-Arellano, MD, PhDP. J. Devereaux, MDJoseph Beyene, PhDJustina Sam, BHScR. Brian Haynes, MD, PhD

COMPUTERIZED CLINICAL DECI-s ion suppor t sys t ems(CDSSs) are information sys-tems designed to improveclinical decision making. Characteris-tics of individual patients are matchedto a computerized knowledge base, andsoftware algorithms generate patient-specific recommendations. Practition-ers, health care staff, or patients canmanually enter patient characteristicsinto the computer system; alterna-tively, electronic medical records canbe queried for retrieval of patient char-acteristics. Computer-generated rec-ommendations are delivered to the cli-nician through the electronic medicalrecord, by pager, or through printoutsplaced in a patients paper chart. Suchsystems have been developed for amyriad of clinical issues, including di-agnosis of chest pain, treatment of in-fertility, and timely administration ofimmunizations. These systems pro-vide several modes of decision sup-port, including alerts of critical val-

See also pp 1197 and 1261.

Author Affiliations:Division of Nephrology (Drs Gargand Rosas-Arellano) and Department of Epidemiol-ogy and Biostatistics (Dr Garg), University of West-ern Ontario, London; Departments of Clinical Epide-miology and Biostatistics (Drs Garg, Adhikari,Devereaux, andHaynes andMsMcDonald) andMedi-cine (Drs Devereaux and Haynes), McMaster Univer-sity, Hamilton, Ontario; Department of Critical CareMedicine, Sunnybrook andWomens College Health

Sciences Centre and Interdepartmental Division of Criti-cal Care (DrAdhikari), PopulationHealth Sciences, Hos-pital for SickChildren (Dr Beyene), and Faculty ofMedi-cine (Ms Sam),University of Toronto, Toronto,Ontario.Corresponding Author: R. Brian Haynes, MD, PhD,Clinical Epidemiology andBiostatistics, Faculty ofHealthSciences,McMasterUniversity, 1200Main StW, Room2C10B, McMaster University, Hamilton, Ontario,Canada L8N 3Z5 ([email protected]).

Context Developers of health care software have attributed improvements in pa-tient care to these applications. As with any health care intervention, such claims re-quire confirmation in clinical trials.

Objectives To review controlled trials assessing the effects of computerized clinicaldecision support systems (CDSSs) and to identify study characteristics predictingbenefit.

Data Sources We updated our earlier reviews by searching theMEDLINE, EMBASE,Cochrane Library, Inspec, and ISI databases and consulting reference lists through Sep-tember 2004. Authors of 64 primary studies confirmed data or provided additionalinformation.

Study Selection We included randomized and nonrandomized controlled trials thatevaluated the effect of a CDSS compared with care provided without a CDSS on prac-titioner performance or patient outcomes.

Data Extraction Teams of 2 reviewers independently abstracted data on methods,setting, CDSS and patient characteristics, and outcomes.

Data Synthesis One hundred studies met our inclusion criteria. The numberand methodologic quality of studies improved over time. The CDSS improvedpractitioner performance in 62 (64%) of the 97 studies assessing this outcome,including 4 (40%) of 10 diagnostic systems, 16 (76%) of 21 reminder systems, 23(62%) of 37 disease management systems, and 19 (66%) of 29 drug-dosing orprescribing systems. Fifty-two trials assessed 1 or more patient outcomes, of which7 trials (13%) reported improvements. Improved practitioner performance wasassociated with CDSSs that automatically prompted users compared with requiringusers to activate the system (success in 73% of trials vs 47%; P=.02) and studies inwhich the authors also developed the CDSS software compared with studies inwhich the authors were not the developers (74% success vs 28%; respectively,P=.001).

Conclusions Many CDSSs improve practitioner performance. To date, the effectson patient outcomes remain understudied and, when studied, inconsistent.JAMA. 2005;293:1223-1238 www.jama.com

2005 American Medical Association. All rights reserved. (Reprinted) JAMA, March 9, 2005Vol 293, No. 10 1223

at Capes Consortia on September 12, 2011jama.ama-assn.orgDownloaded from

ues, reminders of overdue preventivehealth tasks, advice for drug prescrib-ing, critiques of existing health care or-ders, and suggestions for various ac-tive care issues.

As with any health care innovation,CDSSs should be rigorously evaluatedbefore widespread dissemination intoclinical practice. Various stages in thisassessment process have been previ-ously described. Iterative qualitative andquantitative assessment begin early inthe softwaredevelopment cycle.1,2Whenpreliminary testing suggests that aCDSSimproves clinical care or patient out-comes, confirmatory controlled trials arewarranted.Wepreviously reviewed con-trolled trials of computer-aided qualityassurance3 and CDSSs published up to19924 and 1998.5 This field is rapidlyevolving because of technological ad-vances, increasing access to computersystems in clinical practice, and grow-ing concern about the process and qual-ity of medical care. We therefore up-dated previous reviews to provide acumulative summary of controlled trialsevaluating the effectiveness ofCDSSs onpractitioner performance and patientoutcomes.

METHODSResearch Questions

The primary questions of this reviewwere (1) Do CDSSs improve practi-tioner performance or patient out-comes? and (2)WhichCDSS and study-level factors are associatedwith effectiveCDSSs? A priori, we hypothesized thatstudies reporting better outcomeswould assess CDSSs that automati-cally prompted users (vs requiring theuser to actively initiate the system),were built into an electronic medicalrecord or computer order entry sys-tem (vs a stand-alone system), pro-vided reminders (vs information ondis-ease management, drug dosing, ordiagnosis), were tested using less rig-orous study methods, were studied bytheir software developers (vs by evalu-ators not involved in the CDSS de-sign), described pilot testing, and de-scribed user training.

Studies Eligible for ReviewWe included English-language ran-domized andnonrandomized trialswitha contemporaneous control group thatcompared patient care with a CDSS toroutine carewithout a CDSS and evalu-ated clinical performance (ie, a mea-sure of process of care) or a patient out-come.We stipulated that the CDSS hadto provide patient-specific advice thatwas reviewed by a health care practi-tioner before any clinical action. Stud-ies were excluded if the system (1) wasused solely by medical students, (2)only provided summaries of patient in-formation, (3) provided feedback ongroups of patients without individualassessment, (4) only provided com-puter-aided instruction, or (5)was usedfor image analysis. Studies assessingCDSS diagnostic performance againsta defined gold standard were not in-cluded in this review unless clinical useof the diagnostic CDSS was also com-paredwith routine care. Based on thesecriteria, we reevaluated all studies fromour previous reviews for inclusion.

Finding Relevant Studies

Wehavepreviously describedourmeth-ods for finding relevant studies untilMarch 1998.5 For this update, we ex-amined citations from MEDLINE,EMBASE, Evidence-Based Reviews da-tabases (Cochrane Database of System-atic Reviews, ACP Journal Club, Data-base of Abstracts of Reviews of Effects,and Cochrane Central Register of Con-trolled Trials), and Inspec biblio-graphic databases from 1998 throughSeptember 2004. All citations weredownloaded into Reference Manager,version 10.0 (Thomson ISI Research-Soft, Philadelphia, Pa). An experiencedlibrarian developed the search strate-gies using sensitive terms for identify-ing clinical studies of CDSSs.We pilot-tested search strategies and modifiedthem to ensure that they identifiedknown eligible articles. The final strat-egies used the terms computer-assisteddecisionmaking, computer-assisted diag-nosis, computer-assisted therapy, deci-sion support systems, reminder systems,hospital information systems, random-

ized controlled trial, and cohort studies(complete strategies available from theauthors). Pairs of reviewers indepen-dently evaluated the eligibility of all stud-ies identified in our search. Disagree-mentswere resolved by a third revieweror by consensus. Full-text articles wereretrieved if any reviewer considered a ci-tation potentially relevant. Supplemen-tary methods of finding studies in-cluded a review of article reference lists,articles citing included studies as listedin the Science Citation Index, PubMedrelated articles feature, informatics con-ference proceedings, information pro-vided by primary study authors, andother recent reviews.6-11Wheredata froma trial were distributed in more than 1publication, we cited the principal pub-lication.

Data Abstraction

Pairs of reviewers independently ab-stracted the following data from allstudiesmeeting eligibility criteria: studysetting, study methods, CDSS charac-teristics, patient characteristics, andout-comes. Disagreementswere resolved bya third reviewer or by consensus. Weattempted to contact primary authorsof all included studies to confirm dataand provide missing data.All studies were scored for method-

ological quality on a 10-point scale con-sisting of 5 potential sources of bias,whichwe have described elsewhere.5 Inbrief, we considered the method of al-location to study groups (random, 2, vsquasi-random, 1, vs selected concur-rent controls, 0), the unit of the alloca-tion (a cluster such as a practice, 2, vsphysician, 1, vs patient, 0), the pres-ence of baseline differences between thegroups that were potentially linked tostudy outcomes (of particular impor-tance for observational studies; no base-line differences present or appropriatestatistical adjustments made for differ-ences, 2, vs baseline differences presentand no statistical adjustments made, 1,vs baseline characteristics not reported,0), the objectivity of the outcome (ob-jective outcomes or subjective out-comes with blinded assessment, 2, vssubjectiveoutcomeswithnoblindingbut

DECISION SUPPORT, PRACTITIONER PERFORMANCE, AND PATIENT OUTCOMES

1224 JAMA, March 9, 2005Vol 293, No. 10 (Reprinted) 2005 American Medical Association. All rights reserved.


clearly defined assessment criteria, 1, vssubjective outcomes with no blindingand poorly defined, 0), and the com-pleteness of follow-up for the appropri-ate unit of analysis (90%, 2, vs 80 to90%, 1, vs 80% or not described, 0).The unit of allocation was included be-cause of the possibility of group con-tamination in trials in which interven-tions were applied to clinicians eventhough individual patients were allo-cated to the intervention and controlgroups.12Contaminationbiaswould leadto underestimating the effect of a CDSS.The studies substantially differed in

the type and number of outcomes as-sessed. In addition, themajority of stud-ies did not define a single outcome forstatistical testing. We aimed to effi-ciently summarize the benefits ofCDSSsand to identify CDSS and study charac-teristics that predicted success. For agiven study we abstracted all reportedpractitioner performance and patienthealth outcomes. Situations where theCDSS worsened outcomes were rare.Thus, for each study we defined the ef-fects of CDSSs in terms of success, de-fined as an improvement in at least 50%of outcomesmeasured, each at a 2-sidedsignificance level less than .05.

Statistical Analysis

Reviewer agreement on study eligibil-ity was quantified using the Cohen .13

Study andCDSS characteristics predict-ing success were analyzed and inter-preted with the study as the unit ofanalysis. Data were summarized usingdescriptive summarymeasures, includ-ing proportions for categorical vari-ables andmean (standard deviation) forcontinuous variables. Univariable andmultivariable logistic regression mod-els, adjusted for study methodologicalquality, were used to investigate asso-ciations between the outcomes of inter-est and study-specific covariates de-fined in our a priori hypotheses. Allanalyses were carried out using the SASstatistical package, version 8.2 (SAS In-stitute Inc, Cary, NC). We interpretedP.05 as indicating statistical signifi-cance; allP values are 2-sided.When re-porting results from individual studies,

we cited themeasures of association andP values reported in the studies.

RESULTSFinding and Selecting Studies

From 3997 screened citations, we re-trieved 226 full-text articles, and 100trials met our criteria for review. Thechance-corrected agreement between 2independent reviewers for article in-clusion was good (=0.81; 95% con-fidence interval [CI], 0.73-0.88).

Description of Studies

The 100 trials examined more than3826 practitioners or practices (me-dian, 42; range, 2-300 [when re-ported]) caring for more than 92895patients (median, 488; range, 19-12989 [when reported]) from 1973 to2004.14-113 The number of eligible trialsincreased with time: 1 in 1970-1974, 4in 1975-1979, 10 in 1980-1984, 13 in1985-1989, 20 in 1990-1994, 26 in1995-1999, and 26 in 2000Septem-ber 2004.Of these 100 trials,most wereconducted in the United States (69%),followedby theUnitedKingdom(14%),Canada (5%), Australia (4%), Italy(2%), andAustria, France,Germany, Is-rael, Norway, and Switzerland (1%each). Sixty-nine percent of trials de-scribed funding from the public sec-tor and 16% from the private sector. De-velopers of CDSS software were alsostudy authors in 72% of trials. Ninety-seven trials described the effect of CDSSon at least 1 measure of health carepractitioner performance. Fifty-twotrials assessed at least 1 patient out-come. We successfully contacted au-thors of 91 trials, and authors of 64 trialsprovided additional information or con-firmed the accuracy of abstracted data.*

Methodological QualityAssessment

Trial methodological rigor increasedwith time36%of trials before the year2000 were cluster randomized, com-paredwith 67% after this time (P=.01).

Of all trials, 88% were randomized. Ofthe randomized trials, 49% were clus-ter randomized and 40% used a clusteras the unit of analysis or adjusted forclustering in the analysis. Twenty-fourrandomized trials and 1 cohort study re-ported a power calculation for a pre-specifed difference between groups ona specific outcome. Fifteen of these trials(60%) calculated sample size based ona practitioner performance outcome, 9(36%) based on a patient outcome, and1 (4%) based on the cost of prescribedmedications. Only 2 studies examinedpatient outcomes without measuringpractitioner performance.Of the 88 ran-domized trials, 52% described an ap-propriate method of generating ran-dom numbers and 28% reportedallocation concealment. On the 10-pointmethods scale, themean scorewas7 (SD, 1.7) and the range was 2 to 10.

Description of Users and CDSSs

The 100 trials had the following char-acteristics: 92% of trials enrolled physi-cians as primary users, 48% enrolledtraining health care practitioners (in-terns and residents) as users, 34% de-scribed pilot testing with users prior toimplementation, 42%described user in-structional training at the time of imple-mentation, 76% took place in academiccenters, and 33% were inpatient-based.In 47% of studies, the CDSS was part ofan electronic medical record or com-puter order entry system. Most of thesewereearlygenerationsystems lacking thefull functionality of current systems. In15% of studies, the CDSS had a graphi-cal user interface. Feedback from theCDSSoccurred at the timeofpatient carein 88% of studies; in 60% the user wasautomatically prompted to use the sys-tem(vs theuser actively initiating the sys-tem), and in 91% the CDSS suggestednew orders (vs critiquing existing or-ders). Expert physician opinion or clini-calpracticeguidelinesusually formed theknowledge base for the CDSS.The process of data entry into the

CDSSwas clear in 80%of trials, some ofwhich usedmore than 1method. Exist-ing personnel most often entered data(attending or training physician, 38%;

*References 15-18, 20, 21, 24-33, 35-40, 42, 43, 46,47, 49, 51, 52, 56, 60-64, 67, 68, 71, 73-75, 80, 81,83-98, 101, 106, 113-115.




otherhealthcare staff [eg,nurses, clerks],29%), although many trials used staffpaid by research funds (21%) or auto-mated data capture from an electronicmedical record (30%).Themethodofde-livering computer recommendations tothe clinician was clear in 81% of trials.Most CDSSs directly provided the rec-ommendation on a computer screenviewed by the practitioner (41% of alltrials) or generated printed reports thatwere placed inmedical charts by healthcare staff (29%) or by staff paid by re-search funds (16%). Only 13% of trialsevaluated the impact of theCDSS on cli-nicianworkflow,withmore than half oftheseCDSSs requiringmore time and ef-fort from the user comparedwith paper-based methods.

Systems for Diagnosis

Therewere 10 trials evaluating diagnos-tic systems (TABLE 1). All studies mea-sured practitioner performance, and theCDSSwas beneficial in 4 studies (40%).Two of the 4 successful CDSSs were di-agnostic systems for cardiac ischemia intheemergencydepartment, and thesede-creased the rate of unnecessary hospitalor coronary care admissions by 15%(P.05).18,20 The third increased mooddisorder screening in a posttraumaticstress disorder clinic by 25%(P=.008).15

The fourth improved the time to diag-nosis of acute bowel obstruction (1hourwhen computer was used vs 16 hourswhen diagnosis was made with con-trast radiography; P.001).23 Of the 5trials assessing patient outcomes, nonereported an improvement.

Reminder Systems for Prevention

There were 21 trials evaluating re-minder systems for prevention(TABLE 2). All trials measured practi-tioner performance, and the CDSS wasbeneficial in 16 studies (76%). Perfor-mance outcomes were usually rates ofscreening, counseling, vaccination, test-ing, medication use, or the identifica-tion of at-risk behaviors. Successful useof CDSSs was typically demonstratedin ambulatory care, although 1 systemwas successful in hospitalized pa-tients.44 The single trial measuring pa-

tient outcomes failed to demonstrate animprovement in the primary analy-sis.34 Post hoc subgroup analyses, how-ever, demonstrated a significant reduc-tion in winter hospitalization andemergency department visits in pa-tients eligible for pneumococcal or in-fluenza vaccination. One trial exam-ined the effect of adding a cervicalcancer screening reminder to an exist-ing mammography reminder sys-tem.30 This trial suggested no interac-tion between the 2 reminders onscreening efficacy.

Systems for Disease Management

There were 40 studies of CDSSs for ac-tive health conditions. TheseCDSSs im-proved practitioner performance in 23(62%) of 37 studies evaluating thisoutcome. Of the 27 trials measuringpatient outcomes, 5 (18%) demon-strated improvements.For diabetes care, practitioner per-

formance was usually judged by ratesof retinal, foot, urine protein, bloodpressure, and cholesterol examina-tions, with 5 (71%) of 7 trials report-ing improvements (TABLE 3). Simi-larly, in studies of cardiovascularprevention, performancewas judged byblood pressure and cholesterol assess-ment, identificationof smoking, anduseof cardioprotective medications, with5 (38%) of 13 trials reporting improve-ments (TABLE 4). One CDSS providedelectrocardiogram recommendations toimprove thrombolytic prescribing inemergency departments.61OtherCDSSsvaried in purpose, providing recom-mendations for urinary incontinence,human immunodeficiency virus infec-tion management, functional assess-ment, and acute respiratory distresssyndrome, with 6 of 9 reporting im-provements (TABLE 5). Clinical deci-sion support system corollary orderswere used tomonitor the effects of otherprescribed treatments, such as the needfor renal biochemistry measurementsin patients receiving amphotericin B,79

with all 4 trials reporting improve-ments (TABLE 6). Trials testing CDSSperformance to reduce unnecessaryhealth care utilizationmeasured the fre-

quency of redundant testing and un-necessary hospital admissions and hos-pital length of stay, with 3 of 4 trialsreporting improvements (Table 6).FiveCDSSs (18%) examining patient

outcomesdescribed improvements.OneCDSS improved blood pressure control(70% of patients had controlled bloodpressurewithCDSSusevs52%withrou-tine care; P.05).54 A second CDSSreduced urinary incontinence in nurs-inghomeresidentsovera10-weekperiod(23%incontinentwithCDSSvs69%withroutine care; P.01).66 A third CDSSimprovedscoresofbarotrauma(P.001)and organ dysfunction (P= .04) inmechanically ventilated patients withacute respiratory distress syndrome.70

Oneparticipatingcenter in this trial pro-videddatademonstrating lower tidalvol-umes (P.03) and a reduction in expo-sure to high plateau pressures in thegroup receivingCDSS-guidedmechani-cal ventilation (P.001).114 A fourthCDSS reduced patient-reported asthmaexacerbations (8% vs 17%; odds ratio,[OR], 0.43; 95% CI, 0.21-0.85), emer-gencynebulizeruse(1%vs5%;OR,0.13;95% CI, 0.01-0.91), and the need foradditionalconsultations forasthmaman-agement(22%vs34%;OR,0.59;95%CI,0.37-0.95)over6months.73A fifthCDSSreduced hospital length of stay (P=.02)forpatientswithavarietyofgeneralmedi-cal diagnoses.83

In post hoc secondary or subgroupanalyses, some trials described statis-tically significant improvements inthrombolytic prescribing with theCDSS,61 as well as patient outcomes ofdisease-specific emergency depart-ment visits,65 hospital length ofstay,45,54,116,117 body weight,54,116,117 dia-stolic blood pressure,59,115,118 serum lip-ids,51,58 and a reduced estimated risk offuture cardiovascular events.58

Systems for Drug Dosingand Drug Prescribing

There were 29 trials of drug dosing andprescribing (TABLE 7 and TABLE 8).Single-drug dosing improved practi-tioner performance in 15 (62%) of 24studies, and 2 of the 18 systems assess-ing patient outcomes reported an im-




provement (Table 7 and Table 8). An-other 5 systems used computer orderentry for multidrug prescribing(Table 8). Four of these systems im-proved practitioner performance, butnone improved patient outcomes.

The 24 single-drug dosing systemsranged from a simple calculator for par-enteral nutrition to more complex al-gorithms that considered the pharma-cokinetics ofwarfarin, aminoglycosides,or theophylline. Most studies evalu-

ated the serum drug level in medica-tions with a high risk of toxicity. In astudy of heparin dosing for patients re-ceiving thrombolysis formyocardial in-farction, the proportion of individualswith an adverse thrombotic or cardiac

Table 1. Trials of Computer-Assisted Diagnosis*

SourceMethodsScore

No. ofSites Indication Performance Outcomes

PatientOutcomes

Improvementin PractitionerPerformance

Improvementin PatientOutcomes

Diagnostic systems formental health

Lewis et al,141996

5 1 Common mentaldisorders foroutpatients

Rate of patient referral tomental health,psychotropicmedications,psychologicalconsultations

Symptom scoreafter 6 wk

No No

Cannon et al,152000

7 1 Mental healthdiagnosis foroutpatients

Screening for mooddisorder, completedocumentation formajor depressivedisorder

. . . Yes . . .

Schriger et al,162001

6 1 Psychiatric interviewand diagnosis inemergencydepartment

Psychiatric diagnosis andreferrals,documentation ofcomplete psychiatrichistory

. . . No . . .

Rollman et al,200217

9 17 Major depressiondiagnosis foroutpatients

Use of antidepressants,discussion aboutdepression withpatients

Depression scoreafter 6 mo

No No

Diagnostic systems foracute cardiac ischemia

Pozen et al,181984

3 1 Acute cardiacischemia inemergencydepartment

Inappropriate coronarycare unit admission forpatients withoutischemic heart disease

. . . Yes . . .

Wyatt,19 1989 6 1 Chest pain inemergencydepartment

Time to transfer tocoronary care unit,time to see physician,total time inemergencydepartment

. . . No . . .

Selker et al,201998

4 10 Electrocardiograminterpretation forcardiac ischemia inemergencydepartment

Inappropriate hospital orcoronary care unitadmission for patientswithout acute ischemicheart disease

Mortality in first 30 d,in-hospitalcomplications,need forrehospitalization

Yes No

Diagnostic systems forother conditions

Wexler et al,211975

2 1 Admitted pediatricinpatients withoutclear diagnosis

No. of consultationsrequested, time todiagnosis, orders forunnecessarylaboratory tests

. . . No . . .

Wellwoodet al,22 1992

6 1 Acute abdominal painin emergencydepartment

Appropriate diagnosis forappendicitis,unnecessary hospitaladmissions

Unnecessarysurgery withnegative findings

No No

Boguseviciuset al,23 2002

7 1 Acute small bowelobstruction insurgical inpatients

Time to diagnosis, correctdiagnosis

Bowel necrosis,morbidity,mortality,hospital lengthof stay

Yes No

*Ellipses indicate outcome was not assessed. Methods score based on 10-point scale (see the Methods section).Improvement was defined as a statistically significant positive effect on at least 50% of outcomesmeasured. Most studies had inadequate statistical power to detect a clinically importantimprovement in patient outcomes.




Table 2. Trials of Computer-Assisted Reminders for Cancer Screening, Vaccination, and Other Types of Preventive Care

SourceMethodsScore

No. ofSites Indication

Improvementin PractitionerPerformance*

Reminders primarily forcancer screening

Turner et al,241989

8 1 Outpatient screening (stool occult blood, digital rectal examination, Papanicolaou test, breastexamination, mammography)

No

McPhee et al,251989

9 1 Outpatient screening (stool occult blood, digital rectal examination, sigmoidoscopy, pelvicexamination, Papanicolaou test, breast examination, mammography)

Yes

McPhee et al,261991

10 Multiple Outpatient screening (digital rectal examination, stool occult blood, sigmoidoscopy, pelvicexamination, Papanicolaou test, breast examination, mammography) and preventivecounseling (smoking assessment and counseling, dietary assessment and counseling)

Yes

Burack et al,271994

7.5 5 Mammography for outpatients Yes

Burack et al,281996


Burack andGimotty,291997


Burack et al,302003

7.5 3 Papanicolaou test for outpatients; in addition to physician prompt, a patient reminder (personalletter) was generated by the system and was part of the intervention

Yes

Reminders primarily forvaccination

Chamberset al,31 1991

9 1 Influenza vaccination for outpatients Yes

Flanagan et al,321999

7.5 1 Tetanus, hepatitis, pneumococcal, measles, and influenza vaccination for outpatients No

Tang et al,331999

5 1 Influenza vaccination for outpatients Yes

Reminders forpreventive care

McDonaldet al, 341984

8 1 Cancer screening (stool occult blood, mammogram), counseling (weight reduction),immunization (influenza, pneumococcal) in addition to1000 physician behavior rules foroutpatients

Yes

Tierney et al,351986

5 1 Cancer screening (stool occult blood, Papanicolaou test, mammogram), pneumococcalvaccination, tuberculosis skin test, use of antidepressants, metronidazole for trichomonas,cardiovascular medications (-blockers, long-acting nitrates, aspirin), prophylacticantacids, and calcium supplements for outpatients

Yes

Ornstein et al,361991

9 1 Cancer screening (stool occult blood, mammography, Papanicolaou test), cholesterolmeasurement, and tetanus vaccination for outpatients

No

Rosser et al,371991

7.5 1 Cancer screening (Papanicolaou test), blood pressure measurement, assessment of smokingstatus, and vaccination (influenza, tetanus toxoid) in outpatients

Yes

Tape andCampbell,381993

7.5 1 Cancer screening (stool occult blood, Papanicolaou test, mammogram,proctosigmoidoscopy), thyroid function screening, vaccination (tetanus, pneumococcal,influenza) for outpatients

Yes

Turner et al,391994

6 44 Cancer screening (stool occult blood, Papanicolaou test, breast examination, mammogram)and influenza vaccination for outpatients

No

Frame et al,401994

6 5 Cancer screening (stool occult blood, Papanicolaou test, breast examination, mammogram),cardiovascular disease preventive screening (blood pressure, cholesterol, body weight),identification of at-risk behavior (smoking), patient education (self-examination, recognitionof postmenopausal bleeding), and vaccination (tetanus) in outpatients

Yes

Overhageet al,411996

10 1 Cancer screening (Papanicolaou test, mammogram), cardiovascular disease preventivescreening and medications (cholesterol, -blockers, aspirin), diabetes care reminders(retinal examination, urinalysis), vaccination (pneumococcal, rubella, hepatitis B), and anadditional 11 reminders for hospital inpatients

No

Bonevski et al,421999

7 Multiple Cancer screening (Papanicolaou test), cardiovascular disease preventive screening (bloodpressure, cholesterol), and identification of 3 risk behaviors (smoking, excessive alcoholuse, benzodiazepine use) in outpatients

Yes

Demakis et al,432000

10 12 Screening (urinalysis, retinal examination, foot examination), monitoring (glycated hemoglobin),and counseling (dietary advice) to prevent diabetic complications in outpatients; remindersfor other conditions including vaccination, smoking cessation, appropriate -blocker andnonsteroidal anti-inflammatory use; cholesterol screening

Yes

Dexter et al,442001

10 1 Vaccination (pneumococcal, influenza), prophylactic heparin and aspirin use for hospitalinpatients

Yes

*Improvement was defined as a statistically significant positive effect on at least 50% of outcomes measured. Practitioner performance outcomes were the rate of screening, medicationuse, and/or identification of at-risk behaviors. Improvement in patient outcomeswas not assessed except inMcDonald et al,34 in which there was no improvement in bodyweight, bloodpressure, hospitalizations, or emergency department visits.

These systems were designed for more than 1 type of condition, including cancer screening, vaccination, and cardiovascular disease prevention.




eventwas significantly loweredwith theCDSS (0/25 with the CDSS vs 6/26 inusual care; P= .02).97 One warfarin-dosing CDSS reduced hospital lengthof stay from 20 to 13 days (P=.01).87

Two systems reduced hospital lengthof stay in patients receiving theophyl-line (from 8.7 to 6.3 days; P=.03)98 andaminoglycosides (20.3 to 16.0 days;P=.03),104 although themajority of pa-tient outcomesmeasured were not im-proved in these trials.

Study Factors AssociatedWith CDSS Success

Given sparse data for patient outcomes,we only assessed study-level predictorsof improved practitioner performance.

Studies in which users were automati-cally prompted to use the system de-scribed better performance comparedwith studies in which users had to ac-tively initiate the system(success in44/60studies [73%] vs 17/36 studies [47%];P=.02; unadjustedOR, 2.8; 95%CI, 1.2-6.6; OR adjusted for methodologicalquality, 3.0; 95% CI, 1.2-7.1). Simi-larly, studies in which the authors alsocreated the CDSS reported better per-formance comparedwith those inwhichthe trialists were independent of theCDSS development process (success in51/69 studies [74%] vs 5/18 studies[28%];P=.001;unadjustedOR,6.7; 95%CI, 1.7-25.3; OR adjusted for method-ological quality, 6.6; 95%CI, 1.7-26.7).

No other predefined study-level covar-iate was associated with CDSS success.In a post hoc analysis of the 85 studiesthat measured practitioner perfor-mance and enrolled physicians, we didnot find an association (P= .40) be-tween performance and physician expe-rience (trainee vs attending physician).

COMMENTWe identified 100 randomized andnon-randomized trials testing a wide vari-ety of CDSSs, with the number of trialsand their methodological quality in-creasing over time. Of the 97 con-trolled trials assessing practitioner per-formance, themajority (64%) improveddiagnosis, preventive care, disease

Table 3. Trials of Computer-Assisted Diabetes Management*

SourceMethodsScore

No. ofSites Indication Patient Outcomes



Thomas et al,45 1983 4 1 Computer-generated remindersfor outpatients

Change in blood pressure,obesity, glucose,hospitalization, emergencydepartment visits

Yes No

Mazzuca et al,46 1990 8 1 Counseling (exercise and dietaryadvice), glucose controlmonitoring, medication use,education for outpatients

. . . No . . .

Nilasena andLincoln,47 1995

8 2 Screening (foot examination,retinal examination, renaltests), cardiovascular diseaseprevention, neurologicalassessment, and glycemiccontrol in outpatients

. . . No . . .

Lobach andHammond,481997

9 1 Screening (foot examination,complete physical, retinalexamination, cholesterol,urine protein), vaccination(influenza andpneumococcal), as well asglycated hemoglobinmonitoring for outpatients

. . . Yes . . .

Montori et al,49 2002 5 2 Screening (microalbuminuria,retinal examination,cholesterol, foot examination)and counseling (exercise anddietary advice, smokingcessation) to preventcomplications in outpatients;system also identified drugcontraindications

Glycated hemoglobin, totalcholesterol, blood pressure,calculated 10-yFramingham risk score

Yes No

Filippi et al,50 2003 9 Multiple Aspirin use in outpatients . . . Yes . . .

Meigs et al,51 2003 7 1 Screening (retinal examination,foot examination, glycatedhemoglobin, blood pressure,cholesterol), use ofcholesterol-reducing andblood pressure medicationsin outpatients

Change in glycatedhemoglobin, low-densitylipoprotein cholesterol,blood pressure

Yes No

*Ellipses indicate outcome was not assessed.Improvement was defined as a statistically significant positive effect on at least 50% of outcomes measured. Most studies had inadequate statistical power to detect a clinicallyimportant improvement in patient outcomes.

Practitioner performance outcomes were the rate of screening (such as retinal examination or urine protein measurement), medication use, and/or identification of at-risk behaviors.




Table 4. Trials of Computer-Assisted Cardiovascular Disease Management and Prevention*

SourceMethodsScore

No. ofSites Indication Patient Outcomes

Improvement inPractitioner

Performance


Coe et al,52 1977 6 2 Blood pressure management inoutpatients

Diastolic blood pressure, drugadverse effects

. . . No

Barnett et al,541983

4 1 Follow-up for patients with elevatedblood pressure

Diastolic blood pressure100mm Hg or receiving treatment

Yes Yes

Rogers et al,531984

6 1 Management of hypertension, obesity,and renal disease in outpatients

Systolic and diastolic bloodpressure, hospitalization,weight (improved), hospitallength of stay (improved)

Yes No

Brownbridgeet al,55 1986

4 3 Hypertension management inoutpatients: prompts forhypertension care (such as urineprotein measurement, pulseassessment and retinalexamination)

. . . Yes . . .

McAlister et al,561986

7 50 Recommendations forantihypertensive use

Diastolic blood pressure90 mm Hg

No No

Rossi and Every,571997

8 1 Alerts to substitute calcium channelblocker antihypertensives to thoserecommended in practiceguidelines in outpatienthypertensives

. . . Yes . . .

Lowensteyn et al,581998

7 Multiple Calculating coronary risk factor profilefor outpatients

Blood pressure, body mass index,smoking cessation,cholesterol (total, LDL,total/ HDL ratio) (improved)

Predicted 8-y coronary risk factorscore (improved)

Yes No

Hetlevik et al,591999

9 29 Diagnosis, treatment, and follow-uprecommendations forhypertension, diabetes mellitus,and hypercholesterolemia inoutpatients; identification ofsmokers

Glycated hemoglobin, smokingstatus, body mass index,cholesterol, risk score forfuture myocardial infarction,diastolic blood pressure(improved)

No No

Montgomeryet al,60 2000

8 27 Calculation of risk of newcardiovascular event in outpatients

Predicted 5-y cardiovascular riskscore

No No

Selker et al,61 2002 6 28 Thrombolytic prescribing inemergency department, withrecommendations printed onelectrocardiograms

Mortality, stroke, bleeding No No

Ansari et al,62 2003 9 1 -Blocker use in outpatients withcongestive heart failure

Emergency department visit orhospitalization, mortality

No No


8 1 Appropriate medications for patientswith ischemic heart disease andcongestive heart failure; exercisepromotion, weight loss, andsmoking cessation; treatment ofhypertension andhypercholesterolemia

Quality of life (SF-36), emergencydepartment visits for heartdisease, hospitalizations,chronic heart diseasequestionnaire

No No

Weir et al,64 2003 9 16 Antiplatelets and anticoagulantprescribing in patients with anacute ischemic stroke or transientischemic attack; included bothinpatients and outpatients

Predicted relative risk reduction offuture ischemic vascularevents, hemorrhagic vascularevents

No No

Murray et al,652004

9 1 Hypertension management and drugprescriptions for outpatients(2 2 factorial trial; randomizationfor physician to receive CDSS,and randomization for pharmacistto receive CDSS)

Health-related quality of life,emergency department visitsand hospitalizations, systolicand diastolic blood pressure

No No

Abbreviations: CDSS, clinical decision support system; HDL, high-density lipoprotein; LDL, low-density lipoprotein; SF-36, Short Form 36.*Ellipses indicate outcome was not assessed.Improvement was defined as a statistically significant positive effect on at least 50% of outcomesmeasured. Most studies had inadequate statistical power to detect a clinically importantimprovement in patient outcomes.

Practitioner performance outcomes were adherence to recommended guidelines that usually included assessment of cardiac risk factors (blood pressure, cholesterol, smoking) and theuse of cardioprotective medications. The exception was Selker et al,61 in which practitioner performance outcomes were proportion receiving thromobolytics, use of thrombolyticswithin 1 hour of initial electrocardiogram, and achievement of cardiac reperfusion.




management, drug dosing, or drugprescribing. However, the effects ofthese systems on patient health re-main understudiedand inconsistent

when studied. Fifty-two trials as-sessed patient outcomes, often in alimited capacity without adequate sta-tistical power to detect clinically im-

portant differences. Only 7 trials re-ported improved patient outcomeswiththe CDSS, and no study reported ben-efits for major outcomes such as mor-

Table 5. Trials of Computer-Assisted Management for Other Active Health Conditions*

SourceMethodsScore

No. ofSites Indication Performance Outcomes Patient Outcomes



Petrucci et al,661991

7 2 Recommendations fornurse management ofurinary incontinence innursing homes

Nurse knowledge ofincontinence

Rate of urinaryincontinence

Yes Yes

Rubensteinet al,67 1995

8 1 Detection and managementof functional statusimpairments inoutpatients; patientself-reportedinformation wascollected forcomputer-assistedsystem

Physician recognition offunctional statusproblems,recommendedinterventionsundertaken to improvepatient functioning

Functional status (physical,psychological, andsocial) at 6 mo asmeasured byquestionnaire

Yes No

Safran et al,681995

6 1 Screening, treatment, andmanagementrecommendations foroutpatients with humanimmunodeficiency virusinfection

Vaccination,ophthalmologicreferral, CD4 cell countand blood cell count,Pneumocystis jiroveciprophylaxis

Need for physician visits;emergency andhospital admission;mortality

Yes No

Dexter et al,691998

10 1 Reminders to discuss andcomplete advanceddirectives in outpatients

Rates of discussions anddocumentation

. . . Yes . . .

East et al,70 1999 8 10 Mechanical ventilationmanagement(respiratory evaluation,oxygenation, ventilation,weaning, andextubation) in critically illpatients with acuterespiratory distresssyndrome

. . . Survival to hospitaldischarge, intensivecare unit length of stay,barotrauma score(improved), multiorgandysfunction score(improved)

. . . Yes

Kupermanet al,71 1999

6 1 Automated physician alertsvia pager for criticallaboratory results forhospital inpatients

Time to ordering oftreatment for criticallaboratory value, timeto resolution of alertingcondition

Adverse events (death,cardiac arrest, transferto intensive care unit,stroke, renalimpairment) within 48 hof alerting event

Yes No

Christakis et al,722001

8 1 Recommendations forantibiotic use inoutpatient childrenwith otitis media

Unnecessary antibioticprescriptions,prescriptions ofexcessive duration

. . . Yes . . .

McCowanet al,73 2001

6 17 Recommended guidelinesfor treatment ofasthma in outpatients

Review ofself-management plan,inhaler technique, andtreatment adherencewith patient; issuanceof peak flow meter

Symptoms, need for oralsteroid, need forhospital services,patient-initiatedconsultation to manageasthma (improved),exacerbation of asthma(self-report; improved),emergency nebulizeruse (improved)

No Yes

Eccles et al,742002

9 62 Recommendations forangina and asthmamanagement inoutpatients

Adherence to guidelinesincluding medicationprescribing, screening,and assessment ofat-risk behaviors

Self-reported quality of life(generic anddisease-specificmeasures), symptoms

No No

Lesourd et al,752002

7 3 Hormonal ovarianstimulation forinfertile women

No. of missed menstrualcycles

Pregnancy No No

*Ellipses indicate outcome was not assessed.Improvement was defined as a statistically significant positive effect on at least 50% of outcomesmeasured. Most studies had inadequate statistical power to detect a clinically importantimprovement in patient outcomes.




tality. Surrogate patient outcomes suchas blood pressure and glycated hemo-globin were not meaningfully im-proved in moststudies.

Determinants of CDSS SuccessRecent literature has called for a betterunderstanding of factors that predictCDSS success.119 Barriers to implemen-

tation include failure of practitioners touse the CDSS, poor usability or integra-tion into practitionerworkflow, or prac-titioner nonacceptance of computer

Table 6. Trials of Computer Use to Monitor the Effects of Other Prescribed Treatments (Corollary Orders) or to Reduce Unnecessary HealthCare Utilization*

SourceMethodsScore

No. ofSites Indication Practitioner Outcomes Patient Outcomes



Systems to monitor theeffects of corollaryorders

McDonald,761976

5 1 Laboratory tests to monitorpotential medicationadverse effects (suchas measurement ofserum creatinine,potassium orhemoglobin) in adiabetes clinic

Adherence torecommended care

. . . Yes . . .

Young,77 1981 4 1 Recommendedinvestigations for 79medical problems inhospital inpatients

Adherence torecommendedordering

. . . Yes . . .

Fihn et al,781994

8 6 Frequency ofanticoagulantmonitoring inoutpatients

Ability to increaseintervals betweenvisits, proximity totarget internationalnormalized ratiovalue

Hemorrhagic andthromboemboliccomplications

Yes No

Overhage et al,791997

10 1 Recommended tests ortreatments to monitoror ameliorate the effectsof other tests ortreatments forhospital inpatients

Compliance with orders Hospital length of stay,maximum creatininelevel duringhospitalization

Yes No

Systems to reduceunnecessary healthcare utilization


5 1 Prompts to dissuadeordering of routineunnecessary diagnostictests, such aselectrolyte levels, bloodcounts, chestradiographs, andelectrocardiograms inoutpatients

Frequency ofunnecessary testing

. . . Yes . . .


9 1 Alerts for drug allergiesand drug interactions,choices for cost-effective testing aspart of inpatientcomputerized orderentry for medications,tests, and nursingorders

Cost per patientadmission

Hospital length of stay,need for hospitalreadmission

Yes No

Hales et al,821995

6 1 Computer system forhospital admissionscreening

Rate of unnecessaryadmissions

. . . No . . .

Shea et al,831995

7 1 Messages for hospitalinpatients on diagnosis,expected length of stay

. . . Hospital length of stay . . . Yes

Bates et al,841999

8 1 Reminders for redundantclinical laboratory testsin hospital inpatients

Rate of redundant testordering

. . . Yes . . .

*Ellipses indicate outcome was not assessed.Improvement was defined as a statistically significant positive effect on at least 50% of outcomesmeasured. Most studies had inadequate statistical power to detect a clinically importantimprovement in patient outcomes.




Table 7. Trials of Computer-Assisted Anticoagulant Dosing*

SourceMethodsScore


Practitioner PerformanceOutcomes Patient Outcomes



WarfarinAbbrecht

et al,851982

6 1 Warfarin initiation inpostoperativecardiac surgeryinpatients

Proportion of days intherapeutic range, averagenumber of days to achievetherapeutic INR

. . . Yes . . .

Carter et al,861987

6 1 Warfarin initiation forhospital inpatients

Time to achieve therapeuticstable INR

Time to hospital dischargeafter first dose

No No

White et al,871987

8 2 Warfarin initiation forhospital inpatients

Time to achieve therapeuticINR, time to reach stablewarfarin dose, timeabove therapeutic INR

Bleeding complications,hospital length of stay(improved)

Yes Yes

White andMungall,881991

8 1 Warfarin maintenancefor outpatients

Proportion of time withtherapeutic INR, need forfollow-up appointments foranticoagulation adjustment

. . . No . . .

Poller et al,891993


Proportion achieving targetINR, average follow-uptime for appointmentsneeded for anticoagulation

Bleeding complications,mortality

No No

Fitzmauriceet al,901996


Proportion of time withtherapeutic INR, numberof follow-up appointmentsneeded to adjustanticoagulation

Mortality, bleeding,and thromboticcomplications

Yes No

Vadher et al,911997

6 1 Warfarin initiation andmaintenance forinpatients

Time to achieve therapeuticINR, time with therapeuticINR, number ofsupratherapeutic andsubtherapeutic INR levels

Bleeding and thromboticcomplications

No No

Vadher et al,921997

6 1 Warfarin maintenancefor outpatients;system used bynurse practitionercompared withtraining physiciansin routine care

Proportion of time withtherapeutic INR, numberof days between INRtesting, number of testmeasurements

Thrombotic episodes,bleeding complications

Yes No

Ageno andTurpie,931998

7 1 Warfarin maintenancefor outpatientswith mechanicalheart valves

Proportion of time withtherapeutic INR,proportion of INRs withintherapeutic range, numberof required doseadjustments; number ofINR measurements

. . . No . . .

Poller et al,941998

8 5 Warfarin initiation andmaintenance foroutpatients

Time to achieve therapeuticINR, proportion of timewith therapeutic INR

. . . Yes . . .

Fitzmauriceet al,952000

8 12 Warfarin maintenancefor outpatients;nurse-led clinicwith point-of-caretesting

Proportion of time withtherapeutic INR,proportion of patients withtherapeutic INR

Mortality, adverse events(bleeding or thrombosis)

Yes No

Manotti et al,962001


Proportion of time withtherapeutic INR over 1 y,proportion of patientsachieving therapeuticstable INR at 1 mo,number of physicianfollow-up appointments foranticoagulation control

. . . Yes . . .

HeparinMungall

et al,971994

8 2 Heparin dosing usedwith acutemyocardialinfarction treatedwith thrombolytic

Therapeutic anticoagulationafter 24 h

Composite of cardiovascularevents (ie, recurrent chestpain, need for additionalthrombolytics, stroke,cardiac arrest) (improved)

Yes Yes

Abbreviation: INR, international normalized ratio.*Ellipses indicate outcome was not assessed.Improvement was defined as a statistically significant positive effect on at least 50% of outcomesmeasured. Most studies had inadequate statistical power to detect a clinically importantimprovement in patient outcomes.




recommendations.120 Inour review, stud-ies in which users were automaticallyprompted to use the system describedbetter performance compared withstudies in which users were requiredto actively initiate the system. A similarfinding was also reported in a meta-

regression of 11 studies of computer or-der entry.121 Comparedwithmanual ini-tiation, automatic prompting mayimprove integration into practitionerworkflow as well as provide better op-portunities to correct inadvertent defi-ciencies in care. In this review, we also

identified better performance in studiesin which the trial authors also devel-oped the CDSS software. Potential ex-planations of this finding include themotivational effect of a developers en-thusiasm, creationofmoreusable and in-tegrated software, better access to tech-

Table 8. Trials of Computer-Assisted Drug Dosing and Prescribing*

SourceMethodsScore


PractitionerPerformanceOutcomes Patient Outcomes


Performance


Drug-dosing systemsTheophylline/aminophylline

Hurley et al,981986

6 1 Theophylline dosing forinpatients

Proportion of patientswithin therapeuticrange

Theophylline toxicity,peak expiratory flowrate, asthmasymptomquestionnaire,mortality, durationof hospitalization(shorter with CDSS)

Yes No

Gonzalez et al,991989

7 1 Aminophylline dosingin emergencydepartment


Aminophylline toxicity,emergencydepartmentdischarge, peakexpiratory flow rate

No No

Verner et al,1001992

6 1 Theophylline dosing inemergencydepartment


Clinical score ofrespiratory status,peak expiratory flowrate

Yes No

Casner et al,1011993

6 1 Theophylline dosingfor inpatients

Proportion of timewithin therapeuticrange

Theophylline toxicity,number of hospitaldays

No No

AminoglycosidesBegg et al,102

19896 1 Aminoglycoside dosing

for inpatientsProportion of patients

within therapeuticrange

Mortality, decrease increatinine clearance

Yes No

Hickling et al,1031989

5 1 Aminoglycoside dosingin intensive careunit


Estimated creatinineclearance

Yes No

Burton et al,1041991

7 1 Aminoglycoside dosingfor inpatients

Peak concentration ofaminoglycosidewithin therapeuticrange

Mortality due to infection,response to therapy,increase in serumcreatinine level,hospital length of stay(shorter with CDSS)

Yes No

Other medicationsPeck et al,105

19737 1 Digoxin dosing for

outpatients withcongestive heartfailure

Achievement ofactual digoxinconcentrationrelative to targetconcentration

Digoxin toxicity, changein heart failuremedications

No No

Rodman et al,1061984

8 1 Lidocaine dosing forhospital inpatients

Proportion needingadditional lidocainedose, achievementof therapeutic dosewithin 30 min

Lidocaine toxicity Yes No

Ryff-de Lecheet al,1071992

4 1 Insulin dosing foroutpatients

Blood glucose withintherapeutic range,glucose level4.0mmol/L (72 mg/dL)

Hypoglycemic events,glycated hemoglobinlevel

Yes No

Horn et al,1082002

5 1 Parenteral nutritiondosing for hospitalinpatients

Time required tocalculate nutritioncomposition andamount,inappropriateordering

. . . No . . .

(continued)




nical support and training, improvedon-site promotion and tailoring, biasesin assessingoutcomes, and selectivepub-lication of successful trials. Most of theCDSSs in this review were homegrown, and the importance of localchampions to facilitate implementationcannot be underestimated.

Strengths and Weaknessesof This Review

We identified relevant controlled trialsthrough a comprehensive search of theliterature.We extendedour previous re-view from 1998 in a number of impor-tant ways.5 Using better-defined inclu-sion criteria, we reconsidered all priorarticles and identified 37 new articles.To identify CDSS and study character-istics that predicted positive effects, weabstracted relevant data from all ar-ticles in duplicate, confirmed our ab-stractionswith amajority of primary au-

thors, and conducted a multivariableanalysis of study-level covariates.However, limitations of this review

shouldbe appreciated.We includedonlyEnglish-language studies. The CDSSswere grouped into categories based onclinical applications rather thanonotheraspects ofCDSSdesign.122 Although trialmethods are improving with time, thissummary is limited by themethodsusedin the primary studies. We were unableto usemeta-analysis to pool effect sizes,given substantial differences among pri-mary studies in the types of CDSSs andoutcomes evaluated. In addition,we de-fined improvement as apositive effect onat least 50%of outcomesmeasured. Thisapproach, along with the strict inclu-sion criteria of this review,mayhaveun-derestimated the influence of some sys-temand studymethodological factors onCDSS success. The wide confidence in-tervals for the statistically significant de-

terminants of CDSS success imply sub-stantial imprecision in the strength ofthese associations, which may be non-causal. Furthermore, it is possible thatCDSSs for disease management pro-moted the implementation of ineffec-tive therapies, or thatCDSSsof drugdos-ing used incorrect pharmacokineticmodels. Although this appears to be anunlikely explanation for the lackof effecton patient outcomes, we did not evalu-ate the appropriateness of CDSS algo-rithmsor recommendations. Finally,wesummarized controlled trials of CDSSsand did not consider less rigorous butmore common designs, such as before-after studies.

When to Adopt a CDSSfor Practice

The decision to adopt a CDSS for localpatient care is complex and is influ-enced by many considerations. Those

Table 8. Trials of Computer-Assisted Drug Dosing and Prescribing (cont)

SourceMethodsScore


PractitionerPerformanceOutcomes

PatientOutcomes*


Performance

Improvementin Patient

Outcomes*

Drug-prescribing systemsMcDonald,109 1976 7 1 390 Recommended

managementprotocols guidingdrug use, recognitionof adverse drugreactions, andlaboratory tests inoutpatients

Adherence torecommendations

. . . Yes . . .

McDonald et al,1101980

8 1 410 Computerizedmanagement rulesdealing primarily withuse and follow-up ofmedications inoutpatients


. . . Yes . . .

White et al,111 1984 7.5 1 Alerts of potential druginteractions andtoxicity with digoxinin inpatients


. . . Yes . . .

Rotman et al,1121996

7 1 Recommendation forless expensive drugsubstitute whenavailable, alerts fordrug interactions inoutpatients


Adverse druginteractions

No No

Tamblyn et al,1132003

8 Multiple Alerts for prescribingerrors, including drugcontraindications inoutpatients

Inappropriateprescriptions per1000 visits,discontinuationof potentiallyinappropriateprescriptions

. . . Yes . . .

Abbreviation: CDSS, computerized clinical decision support system.*Ellipses indicate outcome was not assessed.Improvement was defined as a statistically significant positive effect on at least 50% of outcomesmeasured. Most studies had inadequate statistical power to detect a clinically importantimprovement in patient outcomes.




responsible for CDSS implementationare typically administrators, informa-tion technology managers, and clini-cians, all of whom are increasinglypushed by technology and guided bygovernment regulations.123 Importantissues include CDSS user acceptance,workflow integration, compatibilitywith legacy applications, system ma-turity, and upgrade availability. Someare concerned about increased practi-tioner dependence on CDSSs, witheroded capacity for independent deci-sion making.31 Finally, cheaper, non-computerized alternatives may beequally or more effective in improvingcare and reducingmedical errors.124-127

One of the primary considerations inadopting a CDSS is its clinical effective-ness: Towhat extent should it be provenbeneficial before mass deployment?Clearly, some testing is required, as aCDSS can have unanticipated effectswhen used in patient care.85 Some high-light the need for multicenter cluster-randomized controlled trials demon-strating improvements in importantpatient outcomes.12 Using such a stan-dard, this review suggests that the ma-jority of available systems are not yetready for mainstream use. Most trialswere unable to enroll enough clusters orpatients for adequate statistical power todetect improvements in patient out-comes. Unfortunately, this situation isunlikely to change soon, given the sub-stantial time and resources needed toconduct such trials, particularly in thearea of preventive health. Furthermore,CDSSs are limited by the cumulativeknowledge used to program their rec-ommendations. It would be unrealisticto require repeatCDSS testing every timeadvances in the knowledge base be-come available. Thus, for initial consid-eration, it may be reasonable to requireproofofCDSSeffectivenessonlyonprac-titioner performance, particularly if suchoutcomes represent current acceptedstandards in care. In our review, manysystems met this requirement. How-ever, this does not preclude the need forsubsequent trials or in-practice assess-ment to confirm systemperformance inimproving patient health. Institutions

need to measure effects on local out-comes and be prepared to iterativelymodify their system in response to prac-tice-based knowledge.2

While some perceive that CDSSs im-prove efficiency and reduce costs, thecurrent supporting evidence is limited.Although some studies have assessedthe costs when outcomes wereimproved,40,45 ,79-81 ,84 ,128 the cost-effectiveness of these systems remainsunknown. Many studies suggestedthe CDSS was inefficient, requiringmore time and effort from the usercompared with paper-based meth-ods.14,15,38,64,81,95,112 Finally, most CDSSsused research funding to facilitate imple-mentation. As highlighted in this re-view, up to 21% of trials used staff paidby research funds for data entry orCDSSrecommendationdelivery.When invest-ing in a commercially available system,funding for support personnel is anadditional cost to be considered.There is currentlywidespread enthu-

siasm for introducing electronic medi-cal records, computerized physicianorder entry systems, andCDSSs intohos-pitals and outpatient settings. In othercommercial, industrial, and scientificspheres of activity, computers have be-come ubiquitous and have improvedsafety, productivity, and timeliness.Given this progress, computerization ofthe health care environment should of-fer tremendous benefits. However, up-take has been slow, and multiple chal-lenges have arisen at every phase ofsoftware development, testing, andimplementation. The progress ofCDSSshas mirrored these trends. Systems areproliferating, their technical perfor-mance and usability are improving, andthe number and quality of evaluations isincreasing. These evaluations haveshown that many CDSSs improve prac-titioner performance. However, furtherresearch is needed to elucidate the ef-fects of such systems on patient health.

Author Contributions:Drs Garg, Adhikari, and Rosas-Arellano had full access to all of the data in the studyand take responsibility for the integrity of the data andthe accuracy of the data analysis.Study concept and design: Garg, Adhikari, Haynes.Acquisition of data: Garg, Adhikari, McDonald,Rosas-Arellano, Sam, Haynes.

Analysis and interpretation of data: Garg, Adhikari,McDonald, Rosas-Arellano, Devereaux, Beyene, Sam,Haynes.Drafting of the manuscript: Garg, Adhikari, Haynes.Critical revision of the manuscript for important in-tellectual content: Garg, Adhikari, McDonald,Rosas-Arellano, Devereaux, Beyene, Sam, Haynes.Statistical analysis: Garg, Adhikari, Beyene.Obtained funding: Garg, Haynes.Administrative, technical, or material support: Garg,Rosas-Arellano, Haynes.Study supervision: Garg, Haynes.

Financial Disclosures: None reported.Funding/Support: Dr Garg was supported by a Ca-nadian Institutes of Health Research (CIHR) ClinicianScientist Training Award. Dr Devereaux was sup-ported by a CIHR Senior Research Fellowship.Role of the Sponsors: No funding source or sponsorhad any role in the design and conduct of the study;collection, management, analysis, or interpretation ofthe data; or preparation, review, or approval of themanuscript.Acknowledgment:Weacknowledge thework of LindaSheridan, who provided administrative help, and TomFlemming, the librarian who helped with the litera-ture searches. We thank Dereck Hunt, MD,MSc, andWilliam Clark, MD, for their help and support.

REFERENCES

1. Wyatt J, Spiegelhalter D. Evaluating medical ex-pert systems: what to test and how? Med Inform(Lond). 1990;15:205-217.2. Sim I, Gorman P, Greenes RA, et al. Clinical deci-sion support systems for the practice of evidence-basedmedicine. J AmMed InformAssoc. 2001;8:527-534.3. Haynes RB, Walker CJ. Computer-aided qualityassurance. Arch Intern Med. 1987;147:1297-1301.4. JohnstonME, Langton KB, Haynes RB,Mathieu A.Effects of computer-based clinical decision support sys-tems on clinician performance and patient outcome.Ann Intern Med. 1994;120:135-142.5. Hunt DL, Haynes RB, Hanna SE, Smith K. Effectsof computer-based clinical decision support systemson physician performance and patient outcomes: a sys-tematic review. JAMA. 1998;280:1339-1346.6. Kaushal R, Shojania KG, Bates DW. Effects of com-puterized physician order entry and clinical decisionsupport systems on medication safety: a systematicreview. Arch Intern Med. 2003;163:1409-1416.7. Payne TH. Computer decision support systems.Chest. 2000;118:47S-52S.8. Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Com-puter-based guideline implementation systems: a sys-tematic review of functionality and effectiveness. J AmMed Inform Assoc. 1999;6:104-114.9. Kaplan B. Evaluating informatics applicationsclinical decision support systems literature review. IntJ Med Inform. 2001;64:15-37.10. Gordon RB, Grimshaw JM, Eccles M, Rowe RE,Wyatt JC. On-screen computer reminders: effects onprofessional practice and health care outcomes.Coch-rane Database Syst Rev. 1998;2:CD001096.11. Mitchell E, Sullivan F. A descriptive feast but anevaluative famine: systematic review of published ar-ticles on primary care computing during 1980-97.BMJ.2001;322:279-282.12. Chuang JH, Hripcsak G, Heitjan DF. Design andanalysis of controlled trials in naturally clustered en-vironments: implications for medical informatics. J AmMed Inform Assoc. 2002;9:230-238.13. Fleiss JL, Cohen J. The equivalence of weightedkappa and the intraclass correlation coefficient asmea-suresof reliability.EducPsycholMeas. 1973;33:613-619.14. Lewis G, Sharp D, Bartholomew J, Pelosi AJ. Com-puterized assessment of commonmental disorders in




primary care: effect on clinical outcome. Fam Pract.1996;13:120-126.15. Cannon DS, Allen SN. A comparison of the ef-fects of computer and manual reminders on compli-ance with a mental health clinical practice guideline.J Am Med Inform Assoc. 2000;7:196-203.16. Schriger DL, Gibbons PS, Langone CA, Lee S,Altshuler LL. Enabling the diagnosis of occult psychi-atric illness in the emergency department: a random-ized, controlled trial of the computerized, self-administered PRIME-MDdiagnostic system.AnnEmergMed. 2001;37:132-140.17. Rollman BL, Hanusa BH, Lowe HJ, et al. A ran-domized trial using computerized decision support toimprove treatment ofmajor depression in primary care.J Gen Intern Med. 2002;17:493-503.18. Pozen MW, DAgostino RB, Selker HP, Syt-kowski PA, HoodWB. A predictive instrument to im-prove coronary-care-unit admission practices in acuteischemic heart disease: a prospective multicenter clini-cal trial. N Engl J Med. 1984;310:1273-1278.19. Wyatt JR. Lessons learnt from the field trial ofACORN, an expert system to advise on chest pain. In:Barber B, Cao D, Quin D, Wagner G, eds. Proceed-ings of the 6th World Conference on Medical Infor-matics; Singapore; 1989:111-115.20. Selker HP, Beshansky JR, Griffith JL, et al. Use ofthe acute cardiac ischemia time-insensitive predictiveinstrument (ACI-TIPI) to assist with triage of patientswith chest pain or other symptoms suggestive of acutecardiac ischemia: a multicenter, controlled clinical trial.Ann Intern Med. 1998;129:845-855.21. Wexler JR, Swender PT, Tunnessen WWJ, OskiFA. Impact of a system of computer-assisted diagnosis.AJDC. 1975;129:203-205.22. Wellwood J, Johannessen S, Spiegelhalter DJ. Howdoes computer-aided diagnosis improve themanage-ment of acute abdominal pain? Ann R Coll Surg Engl.1992;74:40-46.23. Bogusevicius A, Maleckas A, Pundzius J, Skau-dickas D. Prospective randomised trial of computer-aided diagnosis and contrast radiography in acute smallbowel obstruction. Eur J Surg. 2002;168:78-83.24. Turner BJ, Day SC, Borenstein B. A controlled trialto improve delivery of preventive care: physician orpatient reminders? J Gen InternMed. 1989;4:403-409.25. McPhee SJ, Bird JA, Jenkins CN, Fordham D. Pro-moting cancer screening: a randomized, controlled trialof three interventions.Arch InternMed. 1989;149:1866-1872.26. McPhee SJ, Bird JA, Fordham D, Rodnick JE, Os-born EH. Promoting cancer prevention activities by pri-mary care physicians: results of a randomized, con-trolled trial. JAMA. 1991;266:538-544.27. Burack RC, Gimotty PA, George J, et al. Promot-ing screening mammography in inner-city settings: arandomized controlled trial of computerized remind-ers as a component of a program to facilitatemammography. Med Care. 1994;32:609-624.28. BurackRC,Gimotty PA,George J, SimonMS,DewsP, Moncrease A. The effect of patient and physicianreminders on use of screening mammography in ahealth maintenance organization: results of a ran-domized controlled trial.Cancer. 1996;78:1708-1721.29. Burack RC, Gimotty PA. Promoting screeningmammography in inner-city settings: the sustained ef-fectiveness of computerized reminders in a random-ized controlled trial. Med Care. 1997;35:921-931.30. Burack RC, Gimotty PA, SimonM, et al. The effectof adding Pap smear information to a mammogra-phy reminder system in an HMO: results of random-ized controlled trial. Prev Med. 2003;36:547-554.31. Chambers CV, Balaban DJ, Carlson BL, Gras-berger DM. The effect of microcomputer-generatedreminders on influenza vaccination rates in a university-based family practice center. J Am Board Fam Pract.1991;4:19-26.32. Flanagan JR, Doebbeling BN, Dawson J, Beek-

mann S. Randomized study of online vaccine remind-ers in adult primary care. Proc AMIA Symp. 1999:755-759.33. Tang PC, LaRosa MP, Newcomb C, Gorden SM.Measuring the effects of reminders for outpatient in-fluenza immunizations at the point of clinicalopportunity. J AmMed InformAssoc. 1999;6:115-121.34. McDonald CJ, Hui SL, Smith DM, et al. Remind-ers to physicians from an introspective computermedi-cal record: a two-year randomized trial. Ann InternMed. 1984;100:130-138.35. TierneyWM,Hui SL,McDonald CJ. Delayed feed-back of physician performance versus immediate re-minders to perform preventive care: effects on phy-sician compliance. Med Care. 1986;24:659-666.36. Ornstein SM, Garr DR, Jenkins RG, Rust PF, Ar-non A. Computer-generated physician and patient re-minders: tools to improve population adherence to se-lected preventive services. J FamPract. 1991;32:82-90.37. Rosser WW, McDowell I, Newell C. Use of re-minders for preventive procedures in family medicine.CMAJ. 1991;145:807-814.38. Tape TG, Campbell JR. Computerizedmedical re-cords and preventive health care: success depends onmany factors. Am J Med. 1993;94:619-625.39. Turner RC, Peden JGJ, OBrien K. Patient-carried card prompts vs computer-generated promptsto remind private practice physicians to perform healthmaintenance measures. Arch Intern Med. 1994;154:1957-1960.40. Frame PS, Zimmer JG,Werth PL, et al. Computer-based vs manual health maintenance tracking: a con-trolled trial. Arch Fam Med. 1994;3:581-588.41. Overhage JM, TierneyWM,McDonald CJ. Com-puter reminders to implement preventive care guide-lines for hospitalized patients.Arch InternMed. 1996;156:1551-1556.42. Bonevski B, Sanson-Fisher RW, Campbell E, Car-ruthers A, Reid AL, Ireland M. Randomized controlledtrial of a computer strategy to increase general practi-tioner preventive care. Prev Med. 1999;29:478-486.43. Demakis JG, BeauchampC, CullWL, et al. Improv-ing residents compliance with standards of ambula-tory care: results from the VA Cooperative Study onComputerizedReminders. JAMA. 2000;284:1411-1416.44. Dexter PR, Perkins S, Overhage JM, Maharry K,Kohler RB, McDonald CJ. A computerized remindersystem to increase the use of preventive care for hos-pitalized patients. N Engl J Med. 2001;345:965-970.45. Thomas JC,Moore A,Qualls PE. The effect on costof medical care for patients treated with an automatedclinical audit system. J Med Syst. 1983;7:307-313.46. Mazzuca SA, Vinicor F, Einterz RM, Norton JA,Kalasinski LA. Effects of the clinical environment onphysicians response to postgraduate medicaleducation. Am Educ Res J. 1990;27:473-488.47. Nilasena DS, Lincoln MJ. A computer-generatedreminder system improves physician compliance withdiabetes preventive care guidelines. Proc Annu SympComput Appl Med Care. 1995:640-645.48. Lobach DF, Hammond WE. Computerized deci-sion support based on a clinical practice guideline im-proves compliance with care standards. Am J Med.1997;102:89-98.49. Montori VM, Dinneen SF, Gorman CA, et al. Theimpact of planned care and a diabetes electronic man-agement system on community-based diabetes care.Diabetes Care. 2002;25:1952-1957.50. Filippi A, Sabatini A, Badioli L, et al. Effects of anautomated electronic reminder in changing the anti-platelet drug-prescribing behavior among Italian gen-eral practitioners in diabetic patients: an interventiontrial. Diabetes Care. 2003;26:1497-1500.51. Meigs JB, Cagliero E, Dubey A, et al. A con-trolled trial of Web-based diabetes disease manage-ment: the MGH Diabetes Primary Care Improve-ment Project. Diabetes Care. 2003;26:750-757.52. Coe FL, Norton E, Oparil S, et al. Treatment of

hypertension by computer and physician: a prospec-tive controlled study. J Chronic Dis. 1977;30:81-92.53. Rogers JL, Haring OM, Goetz JP. Changes in pa-tient attitudes following the implementation of amedi-cal information system.QRBQual Rev Bull. 1984;10:65-74.54. Barnett GO, Winickoff RN, Morgan MM, Ziel-storff RD. A computer-based monitoring system forfollow-up of elevated blood pressure.MedCare. 1983;21:400-409.55. Brownbridge G, Evans A, Fitter M, Platts M. Aninteractive computerized protocol for the manage-ment of hypertension: effects on the general practi-tioners clinical behaviour. J R Coll Gen Pract. 1986;36:198-202.56. McAlister NH, Covvey HD, Tong C, Lee A,WigleED. Randomised controlled trial of computer assistedmanagement of hypertension in primary care. BrMedJ (Clin Res Ed). 1986;293:670-674.57. Rossi RA, Every NR. A computerized interven-tion to decrease the use of calcium channel blockersin hypertension. J Gen InternMed. 1997;12:672-678.58. Lowensteyn I, Joseph L, Levinton C, Abraham-owicz M, Steinert Y, Grover S. Can computerized riskprofiles help patients improve their coronary risk? theresults of the Coronary Health Assessment Study(CHAS). Prev Med. 1998;27:730-737.59. Hetlevik I, Holmen J, KrugerO. Implementing clini-cal guidelines in the treatment of hypertension in gen-eral practice: evaluation of patient outcome relatedto implementation of a computer-based clinical de-cision support system. Scand J PrimHealth Care. 1999;17:35-40.60. Montgomery AA, Fahey T, Peters TJ, MacIntoshC, Sharp DJ. Evaluation of computer based clinical de-cision support system and risk chart for managementof hypertension in primary care: randomised con-trolled trial. BMJ. 2000;320:686-690.61. Selker HP, Beshansky JR, Griffith JL. Use of theelectrocardiograph-based thrombolytic predictive in-strument to assist thrombolytic and reperfusion therapyfor acute myocardial infarction: a multicenter, ran-domized, controlled, clinical effectiveness trial.Ann In-tern Med. 2002;137:87-95.62. Ansari M, ShlipakMG, Heidenreich PA, et al. Im-proving guideline adherence: a randomized trial evalu-ating strategies to increase beta-blocker use in heartfailure. Circulation. 2003;107:2799-2804.63. TierneyWM,Overhage JM,MurrayMD, et al. Ef-fects of computerized guidelines formanagingheart dis-ease in primary care. J Gen Intern Med. 2003;18:967-976.64. Weir CJ, Lees KR, MacWalter RS, et al. Cluster-randomized, controlled trial of computer-baseddecisionsupport for selecting long-termanti-thrombotic therapyafter acute ischaemic stroke.QJM. 2003;96:143-153.65. Murray MD, Harris LE, Overhage JM, et al. Fail-ure of computerized treatment suggestions to im-prove health outcomes of outpatients with uncom-plicated hypertension: results of a randomizedcontrolled trial. Pharmacotherapy. 2004;24:324-337.66. Petrucci K, Petrucci P, Canfield K, et al. Evaluationof UNIS: urological nursing information systems. ProcAnnu Symp Comput Appl Med Care. 1991:43-47.67. Rubenstein LV, McCoy JM, Cope DW, et al. Im-proving patient quality of life with feedback to phy-sicians about functional status. J Gen InternMed. 1995;10:607-614.68. Safran C, Rind DM, Davis RB, et al. Guidelines formanagement of HIV infection with computer-basedpatients record. Lancet. 1995;346:341-346.69. Dexter PR,Wolinsky FD, Gramelspacher GP, et al.Effectiveness of computer-generated reminders for in-creasing discussions about advance directives andcompletion of advance directive forms: a randomized,controlled trial. Ann Intern Med. 1998;128:102-110.70. East TD, Heermann LK, Bradshaw RL, et al. Effi-cacy of computerized decision support for mechani-




cal ventilation: results of a prospectivemulti-center ran-domized trial. Proc AMIA Symp. 1999:251-255.71. Kuperman GJ, Teich JM, TanasijevicMJ, et al. Im-proving response to critical laboratory results with au-tomation: results of a randomized controlled trial.J Am Med Inform Assoc. 1999;6:512-522.72. Christakis DA, Zimmerman FJ, Wright JA, et al.A randomized controlled trial of point-of-care evi-dence to improve the antibiotic prescribing practicesfor otitis media in children. Pediatrics. 2001;107:E15.73. McCowan C, Neville RG, Ricketts IW,Warner FC,Hoskins G, Thomas GE. Lessons from a randomizedcontrolled trial designed to evaluate computer deci-sion support software to improve the management ofasthma.Med Inform InternetMed. 2001;26:191-201.74. Eccles M,McColl E, Steen N, et al. Effect of com-puterised evidence based guidelines on manage-ment of asthma and angina in adults in primary care:cluster randomised controlled trial.BMJ. 2002;325:941.75. Lesourd F, Avril C, BoujennahA, Parinaud J. A com-puterized decision support system for ovarian stimu-lation by gonadotropins. Fertil Steril. 2002;77:456-460.76. McDonald CJ. Use of a computer to detect andrespond to clinical events: its effect on clinicianbehavior. Ann Intern Med. 1976;84:162-167.77. YoungDW. Improving the consistencywithwhichinvestigations are requested.Med Inform (Lond). 1981;6:13-17.78. FihnSD,McDonellMB,VermesD,et al.Acomput-erizedinterventiontoimprovetimingofoutpatientfollow-up:amulticenter randomizedtrial inpatients treatedwithwarfarin. J Gen Intern Med. 1994;9:131-139.79. Overhage JM, TierneyWM, Zhou XH,McDonaldCJ. A randomized trial of corollary orders to pre-vent errors of omission. J AmMed InformAssoc. 1997;4:364-375.80. Tierney WM, McDonald CJ, Hui SL, Martin DK.Computer predictions of abnormal test results: effectson outpatient testing. JAMA. 1988;259:1194-1198.81. TierneyWM,MillerME,Overhage JM,McDonaldCJ. Physician inpatient order writing on microcom-puter workstations: effects on resource utilization.JAMA. 1993;269:379-383.82. Hales JW, Gardner RM, Jacobson JT. Factors im-pacting the success of computerized preadmissionscreening. Proc Annu Symp Comput Appl Med Care.1995:728-732.83. Shea S, Sideli RV, DuMouchelW, Pulver G, AronsRR, Clayton PD. Computer-generated informationalmessages directed to physicians: effect on length ofhospital stay. J AmMed Inform Assoc. 1995;2:58-64.84. Bates DW, Kuperman GJ, Rittenberg E, et al. Arandomized trial of a computer-based intervention toreduce utilization of redundant laboratory tests. AmJ Med. 1999;106:144-150.85. Abbrecht PH, OLeary TJ, Behrendt DM. Evalu-ation of a computer-assisted method for individual-ized anticoagulation: retrospective and prospectivestudies with a pharmacodynamic model. Clin Phar-macol Ther. 1982;32:129-136.86. Carter BL, Taylor JW, Becker A. Evaluation of threedosage-prediction methods for initial in-hospital stabi-lization ofwarfarin therapy.Clin Pharm. 1987;6:37-45.87. White RH, Hong R, Venook AP, et al. Initiationof warfarin therapy: comparison of physician dosingwith computer-assisted dosing. J Gen Intern Med.1987;2:141-148.88. White RH, Mungall D. Outpatient managementof warfarin therapy: comparison of computer-predicted dosage adjustment to skilled professionalcare. Ther Drug Monit. 1991;13:46-50.89. Poller L,Wright D, RowlandsM. Prospective com-parative study of computer programs used for man-agement of warfarin. J Clin Pathol. 1993;46:299-303.90. Fitzmaurice DA, Hobbs FD, Murray ET, BradleyCP, Holder R. Evaluation of computerized decision sup-port for oral anticoagulationmanagement based in pri-mary care. Br J Gen Pract. 1996;46:533-535.

91. Vadher B, Patterson DL, Leaning M. Evaluationof a decision support system for initiation and controlof oral anticoagulation in a randomised trial.BMJ. 1997;314:1252-1256.92. Vadher BD, Patterson DL, Leaning M. Compari-son of oral anticoagulant control by a nurse-practitioner using a computer decision-support sys-temwith that by clinicians. Clin Lab Haematol. 1997;19:203-207.93. Ageno W, Turpie AG. A randomized compari-son of a computer-based dosing program with amanual system tomonitor oral anticoagulant therapy.Thromb Res. 1998;91:237-240.94. Poller L, Shiach CR, MacCallum PK, et al. Multi-centre randomised study of computerised anticoagu-lant dosage. Lancet. 1998;352:1505-1509.95. Fitzmaurice DA, Hobbs FD,Murray ET, Holder RL,Allan TF, Rose PE. Oral anticoagulation managementin primary care with the use of computerized decisionsupport and near-patient testing: a randomized, con-trolled trial. Arch Intern Med. 2000;160:2343-2348.96. Manotti C, Moia M, Palareti G, Pengo V, Ria L,Dettori AG. Effect of computer-aidedmanagement onthe quality of treatment in anticoagulated patients: aprospective, randomized, multicenter trial of AP-ROAT (Automated PRogram for Oral AnticoagulantTreatment). Haematologica. 2001;86:1060-1070.97. Mungall DR, Anbe D, Forrester PL, et al. A pro-spective randomized comparisonof theaccuracyof com-puter-assisted versusGUSTOnomogramdirected hep-arin therapy. Clin Pharmacol Ther. 1994;55:591-596.98. Hurley SF, Dziukas LJ, McNeil JJ, Brignell MJ. Arandomized controlled clinical trial of pharmacoki-netic theophylline dosing. Am Rev Respir Dis. 1986;134:1219-1224.99. Gonzalez ER, Vanderheyden BA,Ornato JP, Com-stock TG. Computer-assisted optimization of amino-phylline therapy in the emergency department. Am JEmerg Med. 1989;7:395-401.100. Verner D, Seligmann H, Platt S, et al. Computerassisteddesignof a theophylline dosing regimen in acutebronchospasm: serum concentrations and clinicaloutcome. Eur J Clin Pharmacol. 1992;43:29-33.101. Casner PR, Reilly R, Ho H. A randomized con-trolled trial of computerized pharmacokinetic theoph-ylline dosing versus empiric physician dosing.Clin Phar-macol Ther. 1993;53:684-690.102. Begg EJ, Atkinson HC, Jeffery GM, Taylor NW.Individualised aminoglycoside dosage based on phar-macokinetic analysis is superior to dosage based onphysician intuition at achieving target plasma drugconcentrations.Br J Clin Pharmacol. 1989;28:137-141.103. Hickling K, Begg E, Moore ML. A prospectiverandomised trial comparing individualised pharmaco-kinetic dosage prediction for aminoglycosideswith pre-diction based on estimated creatinine clearance in criti-cally ill patients. IntensiveCareMed. 1989;15:233-237.104. Burton ME, Ash CL, Hill DP Jr, Handy T, Shep-herdMD, VaskoMR. A controlled trial of the cost ben-efit of computerized Bayesian aminoglycosideadministration.Clin Pharmacol Ther. 1991;49:685-694.105. PeckCC, Sheiner LB,MartinCM, et al. Computer-assisted digoxin therapy.NEngl JMed. 1973;289:441-446.106. Rodman JH, Jelliffe RW, Kolb E, et al. Clinicalstudieswith computer-assisted initial lidocaine therapy.Arch Intern Med. 1984;144:703-709.107. Ryff-de Leche A, Engler H, Nutzi E, Berger M,BergerW.Clinical application of two computerized dia-betes management systems: comparisonwith the log-book method. Diabetes Res. 1992;19:97-105.108. Horn W, Popow C, Miksch S, et al. Develop-ment and evaluation of VIE-PNN, a knowledge-based system for calculating the parenteral nutritionof newborn infants.Artif IntellMed. 2002;24:217-228.109. McDonald CJ. Protocol-based computer remind-ers, the quality of care and the non-perfectability ofman. N Engl J Med. 1976;295:1351-1355.

110. McDonald CJ, Wilson GA, McCabe GP Jr. Phy-sician response to computer reminders. JAMA. 1980;244:1579-1581.111. White KS, LindsayA, Pryor TA, BrownWF,WalshK. Application of a computerized medical decision-making process to the problem of digoxin intoxication.J Am Coll Cardiol. 1984;4:571-576.112. Rotman BL, Sullivan AN, McDonald TW, et al.A randomized controlled trial of a computer-based phy-sician workstation in an outpatient setting: implemen-tation barriers to outcome evaluation. J Am Med In-form Assoc. 1996;3:340-348.113. TamblynR,HuangA, Perreault R, et al. Themedi-cal office of the 21st century (MOXXI): effectivenessof computerized decision-making support in reduc-ing inappropriate prescribing in primary care. CMAJ.2003;169:549-556.114. McKinley BA,Moore FA, Sailors RM, et al. Com-puterized decision support for mechanical ventilationof trauma induced ARDS: results of a randomized clini-cal trial. J Trauma. 2001;50:415-424.115. Hetlevik I, Holmen J, Kruger O, Kristensen P,Iversen H, Furuseth K. Implementing clinical guide-lines in the treatment of diabetes mellitus in generalpractice: evaluation of effort, process, and patient out-come related to implementation of a computer-based decision support system. Int J Technol AssessHealth Care. 2000;16:210-227.116. Rogers JL, Haring OM, Wortman PM, WatsonRA, Goetz JP. Medical information systems: assess-ing impact in the areas of hypertension, obesity andrenal disease. Med Care. 1982;20:63-74.117. Rogers JL, Haring OM. The impact of a com-puterized medical record summary system on inci-dence and length of hospitalization.Med Care. 1979;17:618-630.118. Hetlevik I, Holmen J, Kruger O, et al. Imple-menting clinical guidelines in the treatment of hyper-tension in general practice. Blood Press. 1998;7:270-276.119. Holbrook A, Xu S, Banting J. What factors de-termine the success of clinical decision support systems?AMIA Annu Symp Proc. 2003:862.120. ReismanY. Computer-based clinical decision aids:a review of methods and assessment of systems.MedInform (Lond). 1996;21:179-197.121. Kawamoto K, Lobach DF. Clinical decisionsupport providedwithin physician order entry systems:a systematic review of features effective for changingclinician behavior. AMIA Annu Symp Proc. 2003:361-365.122. Berlin A, Sorani M, Sim I. Characteristics of out-patient clinical decision support systems: a taxo-nomic description. Medinfo. 2004;2004:578-581.123. Fox J,

Documents

Cdss Practitioner Performance and Patient Outcomes