Open Access Research Mapping physicians admission ... · Subhash Chandra,1,2 Rahul Kashyap,1,2 Cesar A Trillo-Alvarez,2 Mykola Tsapenko,2,3 Murat Yilmaz,2 Andrew C Hanson,2 Brian

Mapping physicians’ admissiondiagnoses to structured concepts towardsfully automatic calculation of acutephysiology and chronic healthevaluation score

Subhash Chandra,1,2 Rahul Kashyap,1,2 Cesar A Trillo-Alvarez,2

Mykola Tsapenko,2,3 Murat Yilmaz,2 Andrew C Hanson,2 Brian W Pickering,2,4

Ognjen Gajic,1,2 Vitaly Herasevich2,4

ABSTRACTObjective: Acute Physiology and Chronic HealthEvaluation (APACHE) is most widely used asa mortality prediction score in US intensive careunits (ICUs), but its calculation is onerous. Theauthors aimed to develop and validate automaticmapping of physicians’ admission diagnoses tostructured concepts for automated APACHE IVcalculation.

Methods: This retrospective study was conducted inmedical ICUs of a tertiary healthcare and academiccentre. Boolean-logic text searches were used to mapadmission diagnoses, and these were compared withconventional APACHE database entry by bedsidenurses and a gold-standard physician chart review.The primary outcome was APACHE IV predictedhospital mortality. The tool was developed in a largercohort of ICU patients.

Results: In a derivation cohort of 192 consecutivecritically ill patients, the diagnosis coefficient coded bythree different methods had a positive correlation,highest between manual and gold standard (r2¼0.95;mean square error (MSE)¼0.040) and least betweenmanual and automatic tool (r2¼0.88; MSE¼0.066).The automatic tool had an area under the curve (95%CI) value of 0.82 (0.74 to 0.90) which was similar tothe physician gold standard, 0.83 (0.75 to 0.91) andstandard manual entry, 0.81 (0.73 to 0.89). TheHosmereLemeshow goodness-of-fit testdemonstrated good calibration of automaticallycalculated APACHE IV score (c2¼6.46; p¼0.6). Theautomatic tool demonstrated excellent discriminationwith an area under the curve value of 0.87 (95% CI0.83 to 0.92) and good calibration (p¼0.58) in thevalidation cohort of 593 patients.

Conclusion: A Boolean-logic text search is an efficientalternative to manual database entry for mapping ofICU admission diagnosis to structured APACHE IVconcepts.

INTRODUCTIONIntensive care medicine consumes a largeproportion of hospital budgets (10e30%)and national healthcare expenditures.1

Owing to increased demands for qualityassessment, qualification of patient treatmentand costebenefit analysis, the need for anaccurate outcome prediction score hasincreased.2 One of the earliest modern risk-adjustment systems, the Acute Physiology and

To cite: Chandra S, KashyapR, Trillo-Alvarez CA, et al.Mapping physicians’admission diagnoses tostructured concepts towardsfully automatic calculation ofacute physiology and chronichealth evaluation score. BMJOpen 2011;1:e000216.doi:10.1136/bmjopen-2011-000216

< Prepublication history forthis paper is available online.To view these files pleasevisit the journal online (http://bmjopen.bmj.com).

Received 22 June 2011Accepted 30 September 2011

This final article is availablefor use under the terms ofthe Creative CommonsAttribution Non-Commercial2.0 Licence; seehttp://bmjopen.bmj.com

1Division of Pulmonary andCritical Care Medicine,College of Medicine, MayoClinic, Rochester, Minnesota,USA2MultidisciplinaryEpidemiology andTranslational Research inIntensive Care, Rochester,Minnesota, USA3Department of Nephrology,Rochester, Minnesota, USA4Department ofAnesthesiology, Rochester,Minnesota, USA

Correspondence toDr Vitaly Herasevich;[email protected]

ARTICLE SUMMARY

Article focus- To develop a fully automated APACHE IV

calculator.- To evaluate the efficiency of automatic tool.- To validate the automated APACHE IV calculator

on a large cohort of ICU patients.

Key messages- Fully automated calculation of the APACHE IV

prognostic score with good discrimination andcalibration is possible.

- A Boolean logic text search is feasible to map themedical ICU admission diagnosis to the corre-sponding APACHE IV disease group.

Strengths and limitations of this study- To our knowledge, this study is the first to

describe a fully automatic calculation of theAPACHE IV score.

- The automated tool presented in this study hasa number of limitations. The tool was developedand validated using medical ICU populations ina single institution. Another limitation of thepresented tool is related to the difficulty in codingthe reason for ICU admission from unstructuredclinical notes.

Chandra S, Kashyap R, Trillo-Alvarez CA, et al. BMJ Open 2011;1:e000216. doi:10.1136/bmjopen-2011-000216 1

Open Access Research

on June 10, 2020 by guest. Protected by copyright.

http://bmjopen.bm

j.com/

BM

J Open: first published as 10.1136/bm

jopen-2011-000216 on 14 Novem

ber 2011. Dow

nloaded from

http://bmjopen.bmj.com/

Chronic Health Evaluation (APACHE) score, was intro-duced in 1981.3 Modified versions (APACHE II,APACHE III and APACHE IV) and various other scoreshave been developed over the past 20 years.4e6

A major limitation in the use of any prediction score isthe amount of time required for its calculation. Forexample, to calculate the APACHE II score,4 medicalpersonnel have to collect 12 physiological parameters,age, patient’s chronic diseases and ICU admission diag-nosis. The number of collected variables increased to 26in APACHE IV along with the number of admissiondisease categories (116 different groups).APACHE IV shows a better discrimination ability in

predicting hospital mortality than other mortality-prediction models such as MPM0 III and SAPS III.However, the data collection for the APACHE IV calcu-lation takes twice as long as SAPS, and three timesas long as the MPM0 III7 calculations. The average timerequired to calculate APACHE IV manually is 37.3 min(95% CI 28.0 to 46.6 min) per patient. Even theonline interfaces offered to calculate the APACHE IVscore require a manual entry of up to 52 data points.8

The development of fully automated calculators,which exploit the strengths of high-fidelity ElectronicMedical Records (EMRs), could support the useof better prediction models without the additionaldata-collection burden usually associated with theiradoption.Shabot et al already demonstrated the usefulness of

automatic extraction of data from a computerisedintensive care unit (ICU) flow sheet for the calculationof an intensity-intervention score.9 Today, programs forautomatic calculation of manually entered values aremore widely available. Some patient-data-managementsystems now offer ‘automatic score calculation.’10

Nevertheless, in all these systems, some or all of the datavalues must be entered manually through a separateinterface. In addition to saving time, completelycomputerised score calculation can reduce interobserverand intraobserver variability and transcriptionerror.11e13 The ideal system would search EMRs auto-matically for all the components of score calculationincluding demographics, hospital monitoring, medica-tion administration, laboratory values, and physician andnursing narrative clinical notes.14

For an automated APACHE IV calculation to succeed,the major challenge to be overcome is that associatedwith ‘mapping’ (matching) chronic conditions and ICUadmission diagnoses to structured APACHE diseasegroups. With this challenge in mind, the specific aims ofthis study were:< to develop a fully automated APACHE IV calculator,

which reliably maps free text physician’s notes tostructured APACHE IV diagnostic disease groups,using Boolean logic text search of the EMRs ofmedical ICU patients;

< to evaluate the efficiency of this tool by comparing itsperformance with conventional APACHE manual

data entry by bedside nurses and a gold-standardposthoc physician review;

< to validate the tool’s performance in a larger cohort.

METHODSThe study was conducted at the Mayo Clinic inRochester, Minnesota, an academic medical centre with1900 beds and 135 000 hospital admissions per year. Thecombined capacity of the ICUs is 204 beds and 14 800admissions per year. Saint Mary’s Hospital has 183 ICUbeds: 24 general medical, 16 medical cardiology, 25cardiac surgery, eight transplant surgery, 20 thoracic orvascular surgery, 24 trauma critical care, 20 neurological,26 neonatal (with the option of dual-occupancy stay fortwins in four of them) and 16 paediatric. RochesterMethodist Hospital has a 21-bed medical-surgical ICU.The Mayo Clinic Institutional Review Board approvedthe study protocol and waived the need for informedconsent for this minimal-risk observational study(approval number 07-005642).

Subject selectionIn this study, we included a retrospective cohort ofpatients admitted to Medical ICU. For the derivationcohort, we evaluated consecutive patients’ EMRs over50 days (OctobereNovember 2006). Randomly selectedpatients from the entire year 2006 (excluding derivationcohort) were included in the validation cohort. Patientswho had an ICU stay of less than 24 h were excluded.

Data source and data collectionThe structural query language (SQL)-based integrativeMultidisciplinary Epidemiology and TranslationalResearch in Intensive Care (METRIC) database(METRIC Data mart) accumulates data within 1 h fromits entry into the EMRs.15 METRIC Data mart was theprimary data source, providing the linked demographic,monitoring, laboratory, intervention and outcome datarequired for the automated APACHE IV score calculator.Age was recorded as a continuous variable and calcu-

lated from the date of birth to the date of admission toICU. For acute physiological variables, the mostabnormal value available in the first 24 h of ICU wasused. Chronic health variables were extracted from theAPACHE database and were collected manually bynurses. To capture the required ICU admission diagnosis(the reason for ICU admission), a free-text search wasapplied to the physician’s ICU admission note. In a pilotstudy, we compared natural-language processing anda Boolean-logic text search to map the ICU admissiondiagnosis.16 The performance of the Boolean logic free-test search was equivalent. Also, the use of natural-language processing required additional hardware andsoftware resources that increased the complexity.Because of this, we chose to use the Boolean-logicfree-test search in this project.The first diagnosis mentioned under the subheading

of ‘Impression’ (Problems/diagnoses) was captured, and

2 Chandra S, Kashyap R, Trillo-Alvarez CA, et al. BMJ Open 2011;1:e000216. doi:10.1136/bmjopen-2011-000216

Mapping physicians’ admission diagnoses to structured concepts


http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


this was mapped to the structured APACHE IV diag-nostic groups. The rules which matched the impressiondiagnosis with the APACHE IV diagnostic group weredeveloped by the authors, who linked them directly tothe disease group. When the ICU admission diagnosiswas unavailable or not coded by the automatic tool, thecorresponding predictive coefficient was replaced by the‘generic’ adjusted diagnosis coefficient of �0.42772.Adjusted diagnosis coefficients were calculated usingmean structured diagnosis coefficients, adjusted fordiagnosis prevalence.

Free-text searchA free-text search is a technique where the designedsearch engine screens all the words in a document ordatabase to match the provided search words. Whenscreening large databases, the major limitation of a free-text search is precision. Several techniques have beendescribed to improve the precision. In the currentproject, we have used field-restricted search and Booleanlogic to perform a more specific search. A field-restrictedsearch enables the search to be limited to a particularsection of the document. We have limited the search tothe first diagnosis mentioned under the ‘Impression’section of the clinical notes. The Boolean logic oroperators (eg, AND, OR, NOT) further refine the searchbased on the logic used. The use of AND operator limitsthe search until both the given search terms are matchedwhere the OR operator includes either of the termsmatched.

APACHE scoreAPACHE is the most widely used mortality predictionmodel in adult ICUs in the USA. In the APACHE score,the physiological variables are derived from the worstvalues in the first 24 h period of the patients’ ICU stay.4 5

The score is also derived from textual conceptsincluding chronic health status, physiological measuresand acute diagnoses.17

Standard manual mapping of admitting diagnosis forAPACHE score calculationAs a standard practice in the host institution, diagnosismapping is performed by trained bedside nurses, anddata entered into the APACHE database in the 24 h timeframe after admission to the ICU. Diagnosis mapping isbased on the nurses’ interpretation of the free-textadmission diagnosis. As a nurse standard practice weused APACHE III coded diagnoses recorded in hospitalEMR. For the purposes of this study, APACHE III struc-tured diagnoses (78) were mapped to APACHE IVstructured diagnoses (116) by a coinvestigator clinicianintesivist (OG).

Development of the gold standard for mapping of admissiondiagnosis for APACHE score calculationICU admission diagnosis was mapped using the ICUadmission note of the attending physician. Attendingphysicians at our institution are present on site 24/7 and

dictate their notes which are transcribed with priorityand 24 h a day. Admission notes are usually availablewithin 2e6 h after their dictation and within 24 h ofpatients’ ICU admissions. Admission diagnoses weredefined as originally described by Zimmermanet al6:‘injuries, surgical procedures, or events that weremost immediately threatening to the patient andrequired the services of the ICU.’ Two physicianresearchers reviewed cases and assigned APACHE diag-nostic codes (k¼0.58). First reviewer utilised all clinicalinformation available in EMR and second only pertinentadmission note. Mismatched cases were analysed bya physician coinvestigators (CAT-A). Agreement betweentwo of the three reviewers was considered the goldstandard. Where there was no agreement between thethree reviewers, a super reviewer (physician researcher)was utilised to adjudicate. In five cases, the gold standardcould not be determined, as the super reviewer refusedto accept the diagnosis of any of the three earlierreviewers, and so the records were excluded from thestudy.

Automatic calculation of APACHE IV predicted mortalityThe automatic tool was an SAS program that retrieves allinformation necessary for APACHE IV calculation datafrom METRIC Data mart using SQL queries. For textprocessing, a Boolean-logic text search of predefinedterms was used. APACHE IV outcome data were savedback to METRIC Data mart. The SAS programran automatically using a schedule and requiredminimal ongoing support. For APACHE IV calculation,the automated APACHE IV calculation system was basedon the equations available at Cerner Corporation webpage (http://www.cerner.com/public/filedownload.asp?LibraryID¼40394).

Statistical analysisCorrelation statistics and BlandeAltman plots were usedto compute the agreement in coding diagnosis coeffi-cient using different mapping methods, manual, goldstandard and automatic. Receiver operating character-istic curves were plotted to calculate the area under thecurve (AUC) and determine the accuracy of theAPACHE IV prediction of hospital mortality. An AUC of>0.70 is considered evidence of a good predictivevalue.18 HosmereLemeshow goodness-fit statistics wereused to test the calibration of the automated calculator.All statistical analyses were performed using JMP andSAS statistical software packages.

RESULTSAfter excluding patients who did not have researchauthorisation (n¼14), those for whom the ICU lengthwas less than 24 h (N¼138) and patients whose goldstandard diagnosis could not be determined (n¼5),a total of 192 patients were enrolled in the derivationcohort. Complete data on physiological parameters,chronic health conditions and admission diagnosis




http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


required for APACHE IV calculation and hospitalmortality were available for all patients in the cohort.Clinical and demographic characteristics of the deriva-tion cohort are shown in table 1.The diagnosis coefficient coded by three different

methods had a positive correlation, the highest correla-tion being between the manual and the gold standard(r2, mean square error (MSE)¼0.95, 0.040), the lowestbetween the manual and the automatic calculation tool(r2, MSE¼0.88, 0.066) and an intermediate correlationbetween the automatic tool and the gold standard (r2,MSE¼0.91 (0.058)). The bias in value of diagnosiscoefficient was least when manual calculation wascompared with the gold standard, 0.013 (95% CI �0.547to 0.574) and maximal when comparing the manual withthe automatic calculation tool; 0.115 (95% CI �0.778 to1.008). On drawing BlandeAltman plot for diagnosiscoefficient coded by three methods, bias betweengold standard and automatic calculation tool, calcula-tion was intermediate: �0.102 (95% CI �0.881 to 0.677)(figure 1).Table 2 shows the mismatch in coding admission

diagnosis by manual, gold standard and automatic tools.A Boolean-logic text search did not code ICU admission

diagnoses for 37 (19.3%) subjects. Among diagnoses thatwere not coded by a Boolean-logic text search, hypo-tension was most prevalent (in 10 subjects) followed byaltered mental status (in four subjects) and alcoholintoxication (in two subjects). ‘Hypotension,’ ‘alteredmental status’ and ‘alcohol intoxication’ are not directlyavailable in the list of APACHE IV diagnoses.For the remaining patients (n¼155), where a first ICU

diagnosis was available in the APACHE diagnoses list, theautomatic tool mapped APACHE IV diagnoses correctlyin 143 (93.8%) patients and miscoded in 12 patients(6.2%). Among diagnoses which were miscoded, respi-ratory distress was the most common (six subjects withrespiratory distress were coded as ‘RESPCA’ which isallotted for ‘Cancer, laryngeal/oral/tracheal/lung’). Acommon minor mismatch was the coding of lower GIbleeding as GIBLEED (unspecified GI bleed, threesubjects) and SGIBLEE (surgery for GI bleed, onesubject).On plotting the receiver operating characteristic curve

of predicted hospital mortality, the automatic calculationtool using a Boolean logic text search showed an AUC(95% CI) value of 0.82 (0.74 to 0.90), which was similarto the physician gold standard, 0.83 (0.75 to 0.91) and

Table 1 Characteristics of the derivation and validation cohorts

Variables Derivation cohort (n[192) Validation cohort (n[593) p Value

Age (years), mean6SD 61619.6 60.8620.9 0.92Gender, male (%) 100 (52) 308 (51.8) 0.97APACHE III score, median (IQR) 56 (40e75) 51 (32e71) <0.05Most common APACHE IV diagnosisgroups; n (%)

1. OD, 26 (13.5)2. RESPOTH, 19 (9.9)3. BACPNEU, 18 (9.4)

1. OD, 129 (21.8)2. BACPNEU, 53 (9.0)3. GIBLEED, 49 (8.3%)

ICU mortality (%) 9.4 4.7 0.01Hospital mortality (%) 16.1 12.3 0.17ICU length if stay, median (IQR) 1.6 (0.8e3.0) 1.1 (0.7e1.9) <0.01Hospital length of stay, median (IQR) 5.6 (2.6e10.1) 3.7 (1.8e6.8) <0.01

APACHE, Acute Physiology and Chronic Health Evaluation; BACPNEU, pneumonia, bacterial or other; GIBLEED, bleeding, GI, upper orunknown location; ICU, intensive care unit; OD, overdose, drug withdrawal; RESPOTH, sleep apnoea, atelectasis, pulmonary haemorrhage/haemoptysis, haemothorax, primary/idiopathic hypertensiondpulmonary, near-drowning accident, pneumothorax, respiratorydmedical, other,restrictive lung disease (ie, sarcoidosis, pulmonary fibrosis), smoke inhalation, weaning from mechanical ventilation (transfer from another unitor hospital only).

Figure 1 BlandeAltman plot of the predictive mortality coefficient showing the correlation between manual and automaticcalculation (A), gold standard and automatic calculation (B), and gold standard and manual calculation (C) in the derivation cohort.




http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


Table 2 Disagreement among automatic tool, manual entry and the gold standard, and the corresponding differences inpredictive coefficients




http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


standard manual entry, 0.81 (0.73 to 0. 89) (figure 2).The HosmereLemeshow goodness-of-fit test demon-strated sufficient calibration of automatically calculatedAPACHE IV scores (c2¼6.4651; 8 degrees of freedom;p¼0.5953).

Validation cohortBased on the analysis of the derivation cohort, additionalconcepts for automatic calculation were added; likewise,‘alcohol intoxication’ was coded as ‘OD,’ ‘code 45/cardiac arrest’ as ‘CARDARR’ and ‘hypokalemia’ as‘ACIDBASE.’ Modified rules were tested on 593 randomsubjects. The automatic tool did not code ICU admissiondiagnoses for 192 (32.2%) patients. On plotting theBlandeAltman plot using the difference and mean valueof the diagnosis coefficient coded manually and by theautomatic tool, the bias between methods in codingdiagnosis coefficient was found to be 0.168 (95% CI�0.799 to 1.135) (figure 3). The discriminatory power ofAPACHE IV score calculated using the automatic toolremained excellent (AUC¼0.87 (0.83 to 0.92)) and wassimilar to manual coding of the admission diagnosis:AUC (95% CI)¼0.88 (0.84 to 0.93) (figure 4). TheHosmereLemeshow goodness-of-fit test showed a goodcalibration for the APACHE IV score calculated by theautomatic tool (c2 6.6381; 8 degrees of freedom;p¼0.5761).

DISCUSSIONIn this retrospective study, we developed and internallyvalidated a model for automatic calculation of APACHEIV using Boolean logic text search for mapping medicalICU admission diagnosis. The automatic model showeda modest agreement in coding medical ICU admissiondiagnosis with routinely performed manual coding

by trained bedside nurses, and the study initiateda physician gold standard. Despite this limitation, theAPACHE IV calculated using the developed automaticmodel demonstrated excellent discrimination inpredicting hospital mortality. The discriminatory abilityof the automatic tool was improved by reviewing themismatches, and this was confirmed in the larger vali-dation cohort. Having an excellent prognostic value inspite of moderate interobserver agreement with theGold Standard is likely due to the modest specificcontribution of ‘diagnosis’ to the overall APACHE IVcalculation which also takes into account multiplephysiological and laboratory values. Therefore, the smalldifferences seen between the coefficients of clinically

Figure 4 Receiver operating characteristic (ROC) curveshowing the predictive performance of the Acute Physiologyand Chronic Health Evaluation (APACHE) IV calculation whenthe diagnosis was mapped using an automatic tool or manualentry (validation cohort).

Figure 3 BlandeAltman plot of predictive mortality coefficienton manual and automatic calculation in the validation cohort.The correlation between manual and automatic model codingof the predictive mortality coefficient was less than it was in thederivation cohort (r2, mean square error¼0.42, 0.423).

Figure 2 Receiver operating curve showing the predictiveperformance of the Acute Physiology and Chronic HealthEvaluation (APACHE) IV calculation when the diagnosis wasmapped by an automatic tool, manual entry and a goldstandard (derivation cohort).




http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


related diagnoses were unlikely to influence the overallaccuracy of the fully automatic APACHE IV calculation.This study also demonstrates the poor interobserveragreement in medical ICU admission diagnosis mappingfor the purpose of APACHE IV calculation.The major factors influencing the use of any mortality

prediction model in ICU include the electronic avail-ability of risk scores, resources and technology.19 Theincreasing availability of EMR in conjunction with thesignificant burden associated with manual collection andcalculation of mortality prediction parameters will likelydrive the development of automated alternatives such asthat presented in this paper.In the past, little effort was made in developing auto-

matic calculations based on mortality-prediction scoressuch as APACHE IV. Some of the key barriers to theirdevelopment included the unavailability of data withinEMRs and difficulties associated with mapping admis-sion diagnoses and chronic health status to structuredAPACHE IV concepts. Automated calculation of theAPACHE II score from EMRs in the ICU has beenattempted previously. Junger and colleagues at Univer-sity Hospital Giessen, Germany,20 used SQL scripts ona dataset of 524 patients. In their retrospective study,physiological parameters and age were extracted directlyfrom the EMR database, and International Classificationof Diseases, Version 9 (ICD-9) was used to map chronicdiseases. The AUC for the automatically calculatedmodified APACHE II score was 0.790 (95% CI 0.712 to0.825) and a goodness-of-fit test showed good calibra-tion.20 They found all the acute physiological parameterseasy to collect, but chronic health conditions, which areentered manually as free text by the medical personnel,were difficult to map. The major limitation of their studywas the unavailability of the comparison group, no-manual-entry comparison group and the absence of thegold standard. Mapping from APACHE IV classificationto Systematized Nomenclature of MedicinedClinicalTerms has also been carried out in previous studies.Eighty-four per cent of diagnostic categories in APACHEIV could be mapped to Systematized Nomenclature ofMedicinedClinical Terms concepts.21

Similar efforts were made to calculate the SAPS IIscore automatically in a retrospective cohort of 524patients from an academic surgical ICU at UniversityHospital Giessen, Germany.2 The study cohort had manymissing laboratory values and clinical parametersrequired for SAPS II score calculation. Despite theselimitations, their automatic tool demonstrated a gooddiscriminatory power and calibration.To our knowledge, this study is the first to describe

a fully automatic calculation of the APACHE IV score.Several limitations need to be acknowledged for appro-priate interpretation of our results. The automated toolpresented in this study was developed and validatedusing medical ICU populations in a single institution.The tool has not been developed for the surgical ICUpopulation, and an algorithm for surgical diagnoses

needs to be included and tested prior to deployment inthis environment. To ensure external validity, themethodology should be replicated in other institutionsequipped with EMRs and using the expertise of localclinical experts. The major limitation of the presentedtool is related to the difficulty in coding the reason forICU admission from unstructured clinical notes. In thisstudy, we used the first diagnosis mentioned under theheading ‘impression’ in the ICU admission note,assuming this to be the primary reason for medical ICUadmission. On other hand, the Boolean-logic text searchwas not run on all the diagnoses in the admission note. Alarger number of diagnoses would reduce the discrep-ancy, as the computer algorithm developed did not havethe ability to determine the primary reason for ICUadmission from a list of diagnoses. The automatic toolcoded the first diagnosis accurately in three-quarters ofpatients. In missed subjects, the diagnosis mentioned inthe ICU admission note was not present in the APACHEdiagnosis groups (unspecified ‘hypotension’). Despitecoding the admission diagnosis with good accuracy, thebias between gold-standard and automatic calculationssuggests that the first listed diagnosis in the ICU admis-sion note is not always the primary reason for ICUadmission in our setting. An effort to distinctly docu-ment the primary ICU admission diagnosis couldpotentially improve the efficacy of such computer-basedautomatic calculations.Alternative solutions such as mapping of the APACHE

IV diagnosis to ICD-9 codes are problematic, as the ICD-9 coding in the ICU is often delayed until after hospitaldischarge. Moreover, in many health systems (includingUS), ICD-9 codes are used for billing, which limits itsclinical accuracy.22

On reviewing mismatching in coding admission diag-nosis, it was observed that relatively similar diagnoses wereallotted different diagnosis groups for APACHE calcula-tion. For example, gastrointestinal (GI) bleeding andlower GI bleeding are coded as GIBLEED and GIBLEUL,respectively. The structured diagnosis coefficients forthese conditions are very similar to each other, �0.55183and �0.57947, respectively. As a result, many of themiscoded diagnoses did not affect predicted mortality toany great extent. In 16 subjects, mismatched codescontributed to a significant difference in structureddiagnosis coefficient (ie, >0.35). This was largely attrib-uted to the fact that many of these patients had than oneadmission diagnosis and that the first diagnosis in the listof diagnosis in ICU admission note was not always theprimary reason for ICU admission.

CONCLUSIONThis study outlines the development and validation ofa fully automated calculation of the APACHE IV score,which utilised a Boolean-logic text search to map themedical ICU admission diagnosis to the correspondingAPACHE IV disease group. The tool developed heredemonstrated consistent and good discrimination and




http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


calibration compared with the established and gold-standard references, when used for medical ICUmortality prediction.

Funding This publication was made possible by Grant Number 1 KL2RR024151 from the National Center for Research Resources, a component ofthe National Institutes of Health, the National Institutes of Health Roadmap forMedical Research and the Mayo Foundation. This study was supported in partby National Heart, Lung and Blood Institute K23 HL78743-01A1.

Competing interests None.

Ethics approval Ethics approval was provided by Mayo Clinic InstitutionalReview Board.

Contributors SC contributed to data management and drafting of themanuscript. RK contributed to data collection, case reviews and management,and revision of the manuscript. CAT-A contributed to case reviews and revisionof the manuscript. MY contributed to data collection, case reviews and revisionof the manuscript. MT contributed to case reviews and revision of themanuscript. ACH contributed to automatic tool development, data analysis andrevision of the manuscript. BWP contributed to the design of the study andrevision of the article. OG contributed to the design of the study and revision ofthe article. VH contributed to the design of the study and revision of the article,and takes responsibility for the integrity of the data and the accuracy of thedata analysis. All authors approved the final version of manuscript.

Provenance and peer review Not commissioned; externally peer reviewed.

Data sharing statement No additional data available.

REFERENCES1. Polderman KH, Metnitz PG. Using risk adjustment systems in the

ICU: avoid scoring an ‘own goal’. Intensive Care Med2005;31:1471e3.

2. Engel JM, Junger A, Bottger S, et al. Outcome prediction in a surgicalICU using automatically calculated SAPS II scores. Anaesth IntensiveCare 2003;31:548e54.

3. Knaus WA, Zimmerman JE, Wagner DP, et al. APACHEdacutephysiology and chronic health evaluation: a physiologically basedclassification system. Crit Care Med 1981;9:591e7.

4. Knaus WA, Draper EA, Wagner DP, et al. APACHE II: a severity ofdisease classification system. Crit Care Med 1985;13:818e29.

5. Knaus WA, Wagner DP, Draper EA, et al. The APACHE III prognosticsystem. Risk prediction of hospital mortality for critically ill hospitalizedadults. Chest 1991;100:1619e36.

6. Zimmerman JE, Kramer AA, McNair DS, et al. Acute Physiology andChronic Health Evaluation (APACHE) IV: hospital mortalityassessment for today’s critically ill patients. Crit Care Med2006;34:1297e310.

7. Kuzniewicz MW, Vasilevskis EE, Lane R, et al. Variation in ICU risk-adjusted mortality: impact of methods of assessment and potentialconfounders. Chest 2008;133:1319e27.

8. APACHE � IV Calculator : Middle East Critical Care Assembly, 2010.http://www.mecriticalcare.net/icu_scores/apacheIV.php.

9. Shabot MM, Leyerle BJ, LoBue M. Automatic extraction of intensity-intervention scores from a computerized surgical intensive care unitflowsheet. Am J Surg 1987;154:72e8.

10. Metnitz PG, Lenz K. Patient data management systems in intensivecaredthe situation in Europe. Intensive Care Med 1995;21:703e15.

11. Chen LM, Martin CM, Morrison TL, et al. Interobserver variability indata collection of the APACHE II score in teaching and communityhospitals. Crit Care Med 1999;27:1999e2004.

12. Polderman KH, Christiaans HM, Wester JP, et al. Intra-observervariability in APACHE II scoring. Intensive Care Med2001;27:1550e2.

13. Polderman KH, Jorna EM, Girbes AR. Inter-observer variability inAPACHE II scoring: effect of strict guidelines and training. IntensiveCare Med 2001;27:1365e9.

14. Herasevich V, Keegan MT, Tines D, et al. ICU Data Mart: InformaticsInfrastructure for Automatics Calculation of Critical Care PrognosticScores. 2008 Summit of Translation Bioinformatics Proceeding,AMIA, 2008:156.

15. Herasevich V, Pickering BW, Dong Y, et al. Informatics infrastructurefor syndrome surveillance, decision support, reporting, and modelingof critical illness. Mayo Clin Proc 2010;85:247e54.

16. Trillo Alvarez CA, Trusko B, Hanson AC, et al. Automatic Calculationof APACHE Admission Diagnoses: Natural language Processing(NLP) vs. ‘Natural Brain Processing’. AMIA 2009, San Fransisco,AMIA, 2009:1054.

17. Lemeshow S, Le Gall JR. Modeling the severity of illness of ICUpatients. A systems update. JAMA 1994;272:1049e55.

18. Wagner DP, Knaus WA, Draper EA. Statistical validation of a severityof illness measure. Am J Public Health 1983;73:878e84.

19. Poon EG, Jha AK, Christino M, et al. Assessing the level ofhealthcare information technology adoption in the United States:a snapshot. BMC Med Inform Decis Mak 2006;6:1.

20. Junger A, Bottger S, Engel J, et al. Automatic calculation ofa modified APACHE II score using a patient data managementsystem (PDMS). Int J Med Inform 2002;65:145e57.

21. Bakhshi-Raiez F, Cornet R, de Keizer NF. Cross-mapping APACHEIV ‘reasons for intensive care admission’ classification to SNOMEDCT. Stud Health Technol Inform 2008;136:779e84.

22. Shenefelt PD. Limits of ICD-9-CM code usefulness in epidemiologicalstudies of contact and other types of dermatitis. Am J Contact Dermat1998;9:176e8.

PAGE fraction trail=7.75




http://bmjopen.bm

j.com/

BM



ber 2011. Dow

nloaded from


Documents

Open Access Research Mapping physicians admission ... · Subhash Chandra,1,2 Rahul Kashyap,1,2 Cesar A Trillo-Alvarez,2 Mykola Tsapenko,2,3 Murat Yilmaz,2 Andrew C Hanson,2 Brian