Emergency Medicine Journal Club - lluem.org

Emergency Medicine Journal Club Wednesday, October 26, 2016

This journal club is sponsored by RCRMC, and will be held in the upstairs room at Mu Restaurant (309 W. State Street, Redlands) on October 26th from 6 to 9 pm. The topic is:

Glasgow Coma Scale: Origin, Application, and Limitations

The GCS has been a fundamental part of the culture of prehospital care and emergency medicine for decades. Where did it come from? Why is it so popular? How is it useful? What are its limitations? What can we learn from the GCS that applies to other scores and scales? The following individuals have been assigned to present articles (maximum of 5-10 minutes each). Please read the articles (particularly those to be presented by residents) and to be prepared to critically discuss them! Bring an electronic or paper copy with you. Page Presenter: Item

2 Steve Green: Sasser. CDC Guidelines for Field Triage of Injured Patients: Recommendations of the National Expert Panel on Field Triage, 2011

3 Tuan Le: Teasdale G, Jennett B. Assessment of coma and impaired consciousness. A practical scale. Lancet 1974 Jul 13;2(7872):81–4.

6 Matt Reddoch: Gill. Interrater reliability of Glasgow Coma Scale scores in the emergency department. Ann Emerg Med. 2004;43:215-223.

15 Radu Dudas: Feldman. Randomized controlled trial of a scoring aid to improve GCS scoring by EMS providers. Ann Emerg Med 2015; 65(3):325–329.

22 Steve Green: Riechers. Physician knowledge of the Glasgow Coma Scale. J Neurotrauma. 2005;22: 1327-1334.

23 Steve Green: Bassi S, Buxton N, Punt JA, et al. Glasgow Coma Scale: a help or a hindrance? Br J Neurosurg. 1999;13:526.

24 Steve Green: Wiese MF. British hospitals and different versions of the Glasgow coma scale: telephone survey. BMJ. 2003 Oct 4;327(7418):782–3.

25 Steve Green: Teoh. Glasgow Coma Scale: variation in mortality among permutations of specific total scores. Intensive Care Med 2000; 26(2):157–61.

26 Steve Green: Healey. Improving the Glasgow Coma Scale Score: Motor Score Alone Is a Better Predictor. The Journal of Trauma 2003 Apr 1;54(4):671–80.

27 Steve Green: Teasdale G, Jennett B, Murray L, et al. Glasgow Coma Scale: to sum or not to sum. Lancet. 1983;2:678.

28 Sarah Peterson: Kupas: Glasgow coma scale motor component (“Patient does not follow commands”) performs similarly to total glasgow coma scale in predicting severe injury in trauma patients. Ann Emerg Med 2016, in press.

35 Steve Green: Wears RL. Lessons From the Glasgow Coma Scale. Ann Emerg Med. 2012 Apr 1;59(4):338.

Page 1

Mu Restaurant, 309 W. State Street, Redlands

Described uses for the GCS • originally described for monitoring depth of coma over time in a neurosurgical unit • categorises severity of traumatic brain injury into mild (13-15), moderate (9-12) and

severe (8 or less) • used in Brain Trauma Foundation guidelines as part of the indications for ICP

monitoring (e.g. GCS 8 or less and abnormal CT head) • used for determining the need for CT head in TBI by tools such as the Canadian CT

Head Rule (GCS<15 at 2 hours post-injury) • Traditional ATLS mantra is “GCS 8, intubate” • used in APACHE II

CDC Guidelines for Field Triage of Injured Patients: Recommendations of the National Expert Panel on Field Triage, 2011 The Panel recommended transport to a facility that provides the highest level of care within the defined trauma system if any of the following are identified: Step One: Physiologic Criteria

• Glasgow Coma Scale ≤13, or • SBP of <90 mmHg, or • respiratory rate of <10 or >29 breaths per minute (<20 in infant aged <1 year), or need

for ventilatory support. Step Two: Anatomic Criteria • all penetrating injuries to head, neck, torso, and extremities proximal to elbow or knee; • chest wall instability or deformity (e.g. flail chest); • two or more proximal long-bone fractures; • rushed, degloved, mangled, or pulseless extremity; • amputation proximal to wrist or ankle; pelvic fractures; • open or depressed skull fractures; or paralysis.

Page 2

Page 3

Page 4

Page 5

N E U R O L O G Y / O R I G I N A L R E S E A R C H

Michelle R. Gill, MDDavid G. Reiley, MDSteven M. Green, MD

From the Department ofEmergency Medicine, LomaLinda University School ofMedicine, Loma Linda, CA.

0196-0644/$30.00

Copyright © 2004 by the AmericanCollege of Emergency Physicians.

doi:10.1016/mem.2004.421

Interrater Reliability of Glasgow Coma Scale

Scores in the Emergency Department

F E B R U A R Y 2 0 0 4 4 3 : 2 A N N A L S O F E M E R G E N C Y M E D I C I N E 2 1 5

Study objective: Emergency physicians often use the Glasgow Coma Scale (GCS)to help guide decisions in patient care, yet the reliability of the GCS has never beentested in a typical broad sample of emergency department (ED) patients. We deter-mined the interrater reliability of the GCS between emergency physicians when adultpatients with altered levels of consciousness are assessed.

Methods: In this prospective observational study at a university Level I trauma cen-ter, we enrolled a convenience sample of ED patients older than 17 years who pre-sented with an altered level of consciousness. Two residency-trained attendingemergency physicians independently assessed and recorded the GCS score and itscomponents (eye, verbal, and motor) in blinded fashion within a 5-minute period. Datawere analyzed for interrater reliability by using standard ordinal calculations. Wealso created scatter plots and Bland-Altman plots for each GCS component and forthe GCS score.

Results: One hundred thirty-one patients were screened and enrolled in the study, with15 excluded because of protocol violations. Of the 116 remaining patients, the agree-ment percentage for exact total GCS was 32% ( -b=0.739; Spearman =0.864; Spearman

2=75%). Agreement percentage for GCS components were eye 74% ( -b=0.715; Spear-man =0.757; Spearman 2=57%), verbal 55% ( -b=0.587; Spearman =0.665; Spearman

2=44%), and motor 72% ( -b=0.742; Spearman =0.808; Spearman 2=65%). OurSpearman’s analyses found that only approximately half (44% to 65%) of the observedvariance could be explained by the relationship between the paired component mea-sures. For GCS components, only 55% to 74% of paired measures were identical, and6% to 17% of them were 2 or more points apart.

Conclusion: We found only moderate degrees of interrater agreement for the GCSand its components.[Ann Emerg Med. 2004;43:215-223.]

Page 6

I N T E R R A T E R R E L I A B I L I T Y O F T H E G L A S G O W C O M A S C A L E Gill, Reiley & Green

2 1 6 A N N A L S O F E M E R G E N C Y M E D I C I N E 4 3 : 2 F E B R U A R Y 2 0 0 4

GCS is universally applied and incorporated intotrauma and life support education programs,41,42 it isimportant for this scale to be valid and reliable.

Goals of This Investigation

We wished to quantify the interrater reliability of theGCS and its components in adult patients presenting tothe ED with altered levels of consciousness.

M A T E R I A L S A N D M E T H O D S

Study Design and Setting

This prospective observational study examined theinterrater reliability of the GCS at a Level I trauma cen-ter with an annual ED census of 53,000 patients.Patients were entered into the study between March2002 and March 2003. This study was approved by thehospital’s institutional review board.

Selection of Participants

Patients were included if they were older than 17years and were judged by an enrolling physician to havean altered level of consciousness because of either med-ical or traumatic illness. Exclusion criteria were theadministration of paralytic agents within 1 hour ofassessment or the failure of both physician-observers tocomplete their evaluations of the patient within a 5-minute period.

Patients were enrolled by convenience sampling,with every attempt to enroll them at the earliest possi-ble time after arrival at the ED. Two residency-trainedattending emergency physicians (as available on a con-venience basis) independently assessed the GCS within5 minutes of each other while blinded to each other’sscores. Physicians assessed and recorded GCS scores inaccordance with their standard practice, did not receivenotice before the study, and were intentionally not pro-vided with any additional training or instruction in thisassessment. They recorded separate GCS componentson data forms listing GCS score categories (eye, verbal,and motor) together with their standard currentlyaccepted definitions (Figure 1).43

Methods of Measurement

We assessed interrater reliability separately for theeye, verbal, motor, and total GCS scores. First, we cal-culated agreement percentage and the unweighted andweighted statistic. The unweighted treats agreementin a binary (perfect agreement versus no agreement)manner and therefore weights all misses equally, where-

I N T R O D U C T I O N

Background

Since its creation in 1974 by Teasdale and Jennett1

for the purpose of assessing the depth and duration ofimpaired consciousness and coma, the Glasgow ComaScale (GCS) has become a universal tool for the quan-tification of altered level of consciousness. The GCS isincreasingly being tested for its ability to predict out-comes from traumatic brain injury,2-24 nontraumaticcoma,25-27 cerebrovascular events,16 cardiac arrest,28

and toxic ingestions.29 Classification schemes and pre-diction trees for head injury have been developedaccording to GCS scores,12,23,30 with radiographicstudies and ultimately patient treatment based on thepatient’s GCS score.12,21,23,31-33

Importance

The Brain Trauma Foundation recently stated thatthe GCS can be “fairly reliably measured by trainedmedical personnel”20; however, we found no evidencethat it has ever been evaluated in a typical broad sampleof emergency department (ED) patients with an alteredlevel of consciousness. A single ED-based study includedonly patients with a GCS score of 13 or greater.32 Otherauthors have studied the reliability of the GCS in theICU34-39 or through the use of videotaped assess-ments,36,40 but these studies are unlikely to be applica-ble to the unique environment of the ED. Because the

Capsule Summary

What is already known on the topicThe Glasgow Coma Scale (GCS) is assumed to be a reliable andreproducible measure of neurologic function when measured bytrained personnel. No previous studies have measured interraterreliability in the broad spectrum of patients seen in an emergencydepartment (ED) setting.

What question this study addressedThis study measured interrater reliability of the GCS in 116patients evaluated by 2 independent blinded observers in the EDsetting.

What this study adds to our knowledgeThe GCS showed only moderate interrater reliability in a broadspectrum of ED patients. The GCS cannot be used as a precisemeasurement of neurologic function or a precise measurementof changes in neurologic function in ED patients.

How this might change clinical practiceA 1- or 2-point change in GCS score can be easily consistentwith interrater variation and may not reflect true changes inneurologic function.

Page 7


Thirty-eight percent of paired total GCS measureswere identical, 67% were within 1 point of each other,

as the weighted adjusts the calculation (being off by 1GCS point is not as bad as being off by 2 points). Wethen calculated Kendall’s -b, created scatter plots, andcalculated Spearman’s . We also report the confidenceintervals (CIs) of Spearman’s and the square of this (ie, the percentage of the observed variance explainedby the relationship between the paired measures). Wethen created histograms and Bland-Altman plots44 ofthe differences between paired scores. Given the possi-bility that intubation might adversely affect the reliabil-ity of verbal scores,45-48 we performed a similar subsetanalysis of this measure in nonintubated patients.

Data Analysis and Processing

All analyses were performed with Stata 8 software(Stata Corporation, College Station, TX), except for CIsaround the Spearman estimates. These were calcu-lated with JACKBOOT, a bootstrapping macro in SASsoftware (version 8.2, SAS Institute, Inc., Cary, NC)that uses a bias-corrected accelerated method.

R E S U L T S

A total of 131 patients were screened and enrolled in thestudy. We then excluded 15 protocol violations (1 under-age patient and 14 with paired ratings >5 minutes apart).The remaining 116 patients (71 men, 45 women) aredescribed in Table 1. Nineteen attending emergencyphysicians participated in the study.

Eighteen (16%) of the 116 patients were intubated attheir assessments; in all circumstances, both ratersassigned a verbal score of 1 for these patients.

Measures of interrater reliability demonstrated mod-erate levels of agreement (Table 2), with approximatelyhalf (44% to 65%) the observed variance explained by therelationship between the paired component measures.Scatter plots are shown in Figure 2, and differencesbetween paired scores are displayed in Figures 3 and 4.


Figure 1.The Glasgow Coma Scale.43

Motor Activity

6. Obeys commands5. Localizes pain4. Withdraws to pain3. Flexion to pain2. Extension to pain1. None

Verbal Activity

5. Oriented4. Confused3. Inappropriate2. Incomprehensible1. None

Eye Opening

4. Spontaneous3. To command2. To pain1. None

Table 1.Patient characteristics.

Variable No. (%) GCS Score Range

Total number of patients 116Median age, y (range) 54 (18–90)SexMale 71 (61)Female 45 (39)Intubated at assessment 18 (16)DiagnosisTrauma 32 (28) 3–15Infection/sepsis 19 (16) 3–15Metabolic disturbance 16 (14) 3–15Drug or alcohol intoxication 11 (9) 3–14Cerebrovascular accident 10 (9) 3–15Nontraumatic intracranial hemorrhage 7 (6) 4–15Seizure disorder 6 (5) 3–14Dementia 4 (3) 7–15Other* 11 (9) 3–15Head CT scan resultsHead CT scan not done 39 (34) 3–15Intracranial hemorrhage 20 (17) 3–15Normal 18 (16) 3–15Atrophy 17 (15) 3–15Infarction 11 (9) 3–15Other† 11 (9) 3–15

CT, Computed tomography.*Included hypotension, shunt malfunction, hypoxia, dehydration, malignancy, andunknown etiology.†Included cerebral contusion, abnormal with no interval changes, and hydrocephalus.

Table 2.Measures of interrater reliability between paired ratings.

Measure Eye Verbal* Motor Total

Agreement, % 74 55 72 32†

Kendall’s -b‡ 0.715 0.587 0.742 0.739Spearman’s ‡ 0.757 0.665 0.808 0.864(95% CIs) (0.612–0.851) (0.519–0.765) (0.700–0.877) (0.804–0.904)Spearman’s 2,§ % 57 44 65 75

, unweighted 0.59 0.37 0.58 0.00ll

, weighted¶ 0.72 0.48 0.63 0.40 *A subset analysis of this measure in the subset of nonintubated patients (n=98)yielded slightly less reliable results: percentage agreement 47%, 0.24 (“fair”),Kendall’s -b 0.452, Spearman’s 0.515, 2 27%.†Thirty-two percent had perfect agreement of all 3 score elements; an additional 6%had identical total GCS scores with differing components.‡P<.0001 for all measures.§The square of Spearman’s represents the percentage of the observed varianceexplained by the relationship between the paired measures.llCalculated with the requirement that all 3 components have agreement. If the overallagreement of the total is considered even when components disagree, the is 0.29 or“fair.”¶Exact matches weighted 1.0, misses of 1 point weighted 0.5, and misses of 2 pointsweighted as 0. The calculation for total GCS is based on the overall agreement of thetotal even when components disagree.

Page 8

assigned verbal scores of 1 in intubated patients,effectively enhancing (if only slightly) the reliabilityof this assessment for the total sample when com-pared with just the nonintubated patients (Table 2).Thus, our data suggest that the GCS verbal score isnot unreliable for intubated patients but instead issimply not applicable or helpful. We suggest that, forsimplicity, this portion of the GCS be omitted in intu-bated patients.

A second potential limitation is that we intentionallychose not to provide our physicians with specific train-ing on techniques of GCS score assignment because wewanted to duplicate the way the GCS score is currentlyassigned in actual practice. It is possible that a struc-


81% were within 2 points of each other, and 90% werewithin 3 points of each other (Table 3). For GCS com-ponents, 55% to 74% of paired measures were identical,83% to 94% were within 1 point of each other, and 92%to 95% were within 2 points of each other.

L I M I T A T I O N S

One limitation of our study is that we chose to includeintubated patients despite questions about the relia-bility of verbal score assessments in this patient sub-set.45-48 We included them with the intent of realisti-cally mimicking the full spectrum of ED clinicalscenarios. We noted that our physicians uniformly


5

3

2

11 2 3 5

Verbal–second measure

Verb

al–f

irst m

easu

re

4

4

B 15

5

01 5 15

Total GCS –second measure

Tota

l GCS

–firs

t mea

sure

10

10

D

Figure 2.Plots of paired ratings (n=116). The smoothing line was generated by LOWESS (locally weighted least squares regression). Theslight offset of data points (ie, jitter) is created solely to display data points that might otherwise overlap. A, Eye component; B,verbal component; C, motor component; D, total.

4

3

2

11 2 3 4

Eye–second measure

Eye–

first

mea

sure

A6

3

2

11 2 3 6

Motor–second measure

Mot

or–f

irst m

easu

re

4

4

5

5

C

Page 9


D I S C U S S I O N

We measured the interrater reliability of the GCS in abroad sample of ED adults with altered level of con-sciousness and were disappointed to find just moderatedegrees of test reliability. Our Spearman’s analysesfound that only approximately half of the observed vari-ance could be explained by the relationship betweenthe paired component measures. For GCS components,only 55% to 74% of paired measures were identical, and6% to 17% of them were 2 or more points apart.Although our paired measures were naturally statisti-cally associated, the magnitude of this association wassubstantially less than we would consider desirable.

Although we have also reported the statistics forthe GCS and its components (Table 2), these values

tured intensive training program might result in im-proved interrater reliability; however, given the sim-plicity and straightforwardness of this test, we believethat this result is unlikely.

Because our paired measures could occur up to 5minutes apart, it is possible that some of our observeddifferences represent actual patient changes in functionover time rather than interrater variation. However, webelieve that appreciable changes within such a shortspan are unlikely.

A final consideration is that we performed our studyin a clinically diverse population, including a mix ofages and diagnoses representative of our ED popula-tion. It is possible that the GCS demonstrates greaterreliability when used in specific patient subsets or con-ditions.


Difference between verbal measures

60

40

20

0–10 –5 0 5 10

Freq

uenc

y

B

Difference between total GCS measures

40

30

20

10

0–10 –5 0 5 10

Freq

uenc

y

D

Figure 3.Histograms of the difference between the paired ratings. A, Eye component; B, verbal component; C, motor component; D, total.

Difference between eye measures

80

60

40

20

0–10 –5 0 5 10

Freq

uenc

y

A

Difference between motor measures

80

60

40

20

0–10 –5 0 5 10

Freq

uenc

y

C

Page 10

as bad as being off by 10). Thus, the ordinal measures ofreliability (ie, Kendall’s -b, Spearman) and weighted reported in Table 2 should be considered more accuratea depiction of GCS test performance than the unweighted

. A disadvantage of the weighted , however, is that theresults will differ, depending on the arbitrary weightingscheme selected.

An alternative manner of putting the current resultsin some sort of context is by comparing them with thereported reliability of other clinical features. The GCScomponents are approximately as reliable as pulmonaryauscultation for rales ( =0.24 to 0.65),49 the examina-tion of tympanic membranes ( =0.30),50 and murmurdetection ( =0.30 to 0.48).51 GCS components appear,in general, less reliable than the assessment of National


must be interpreted with caution. The unweighted technique classifies differences of just a single point ascomplete disagreement, and accordingly its summaryvalues are applicable only for the purposes of judgingidentical rather than partial agreement. When weviewed results in this fashion, we found that the totalGCS exhibited a value of 0.00, which is consistentwith random guessing. However, complete agreementmay be too stringent a requirement for the GCS because1- or 2-point differences in these scales may be consid-ered clinically acceptable. Furthermore, the unweighted

values we discovered contrast with the visual appear-ance of overall correlation seen in our scatter plots(Figure 2), which is likely because an unweighted counts all errors as equally bad (ie, being off by 1 is just


Means of paired verbal ratings

5

3

2

1

01 2 3 5

Diffe

renc

e be

twee

n ve

rbal

ratin

gs

95th percentile

4

4

B

Means of paired total GCS ratings

6

5

4

2

03 4 7 15

Diffe

renc

e be

twee

n to

tal G

CS ra

tings

95th percentile6

1

10 11

3

7

5 6 8 9 12 13 14

D

Figure 4.Bland-Altman plots of absolute differences versus means. The slight offset of data points (ie, jitter) is created solely to displaypoints that might otherwise overlap. A, Eye component; B, verbal component; C, motor component; D, total.

Means of paired eye ratings

4

3

2

1

01 2 3 4

Diffe

renc

e be

twee

n ey

e ra

tings

95th percentile

A

Means of paired motor ratings

6

3

4

2

01 2 3 6

Diffe

renc

e be

twee

n m

otor

ratin

gs

95th percentile

5

1

4 5

C

Page 11


total GCS but did not separately report values for GCScomponents.

Three other investigators have examined GCS scoresin the ICU by using the more traditional . Tesseris etal37 report a of 0.35 between 2 physicians assessingthe GCS in 46 ICU patients. Elliott38 reports a of 0.95for GCS score agreement between 2 nurses for 22 intu-bated ICU patients. Lindsay et al39 report a of 0.69between neurosurgery attending physicians and resi-dents for rating 15 patients with subarachnoid hemor-rhage into the following 3 categories: GCS score of 6 to12, 13 to 14, and 15.

Our study is novel because, to our knowledge, it isthe first to include a broad spectrum of patients with analtered level of consciousness, which is more likely torepresent the typical ED population than are resultsfrom previous ICU studies. Accordingly, we believe thatour results may be more applicable to typical ED prac-tice.

The widespread acceptance and implementation ofthe GCS is likely due to its face validity, because itmerely quantifies those assessments of responsivenessthat clinicians already routinely perform. However, theclinical utility of a test requires more than face validity,specifically, criterion-related validity, construct valid-ity, and content validity.55

Criterion-related validity refers to the correspond-ence between the test in question and a reference crite-rion standard. GCS fails this best-available measure oftrue validity in that there is no criterion standard for thequantification of altered level of consciousness, andtherefore it has been impossible to calculate the accu-racy of this test. Indeed, through default the GCS haseffectively become the essential criterion standard forthis purpose.56-59

Construct validity refers to the consistency of one testwith respect to another test that measures similar char-acteristics.60 In the case of the GCS, one would considerthat it has construct validity in regard to demonstratedstatistical associations with outcomes such as theGlasgow Outcome Score,2,3,9,10,13,17,18,23-28,30 mortal-ity,3,5,6,14,24,26,27,61 and the need for neurosurgicalinterventions.12,16,18,21,32,62 Despite this evidence, it isnot clear that the GCS can accurately predict these spe-cific outcomes in any given patient.3 In this manner,it may be similar to the WBC count, which, despitebeing statistically associated with appendicitis, hassuch a weak association that the test cannot be used toaccurately guide clinical decisionmaking for specificpatients.63

Emergency X-Radiography Utilization Study (NEXUS)cervical spine injury risk criteria (percentage agree-ment 83% to 96%; =0.58 to 0.86)52 and the ascertain-ment of witnessed loss of consciousness from patienthistory ( =0.83).53 If this alternative and more forgiv-ing standard of comparison is used, the GCS score per-haps does not seem as unreliable. However, somedegree of imprecision must be tolerated in these otherclinical features in that there is no better alternative fortheir assessment. The GCS, in contrast, is an arbitraryscale that is widely believed to enhance clinical prac-tice, and we believe that it should be held to a more rig-orous standard.

Three sets of researchers have evaluated interraterreliability of the GCS in nonacute care settings by usinga “disagreement rate” ranging from 0 to 0.5.34-36 Intro-duced in the original GCS study by Teasdale and Jennett,1

and no longer generally accepted as a measure of inter-rater reliability,37,54 disagreement rates are defined by 1of the authors36 as “low” when they fall between 0 and0.299 and high when between 0.3 and 0.5.36 Teasdale etal34 found that 7 neurosurgeons displayed low dis-agreement when assessing 12 ICU patients with theGCS according to this calculation (disagreement rates:eye=0.143, verbal=0.0054, and motor=0.109). Rowleyand Fielding35 noted similar results in an unspecifiednumber of neurosurgical ward patients (eye=0.005 to0.163, verbal=0.000 to 0.035, motor=0.004 to 0.064)when evaluated by trained or untrained observers. For7 videotaped ICU patients, Juarez and Lyons36 foundthat disagreement rates for individual GCS componentsvaried, depending on the patient being evaluated(eye=0.00 to 0.29, verbal=0.00 to 0.06, and motor=0.00to 0.22). Juarez and Lyons also reported in summaryfashion that scores ranged from 0.39 to 0.79 for the


Table 3.Absolute differences between paired ratings.

AbsoluteDifference Eye (%) Verbal (%) Motor (%) Total (%)

0 86 (74) 64 (55) 83 (72) 44 (38)1 23 (20) 32 (28) 19 (16) 34 (29)2 1 (1) 11 (9) 5 (4) 16 (14)3 6 (5) 8 (7) 4 (3) 11 (9)4 0 1 (1) 4 (3) 4 (3)5 0 0 1 (1) 3 (3)6 0 0 0 1 (1)7 0 0 0 3 (3)

Page 12

Received for publication April 24, 2003. Revisions received July 17,2003; July 29, 2003; and August 5, 2003. Accepted for publicationAugust 7, 2003.

Presented at the Society for Academic Emergency Medicine west-ern regional meeting, Scottsdale, AZ, April 2003; and the Society forAcademic Emergency Medicine national meeting, Boston, MA,May 2003.

The authors report this study did not receive any outside funding orsupport.

Reprints not available from the authors.

Address for correspondence: Michelle R. Gill, MD, Department ofEmergency Medicine, Loma Linda University Medical Center, 11234Anderson Street, PO Box 2000, Rm. A-108, Loma Linda, CA 92354;909-558-4344, fax 909-558-0121; E-mail [email protected].

R E F E R E N C E S1. Teasdale G, Jennett B. Assessment of coma and impaired consciousness: a practi-cal scale. Lancet. 1974;2:81-84.2. Teasdale G, Jennett B. Assessment and prognosis of coma after head injury. ActaNeurochir. 1976;34:45-55.3. Jennett B, Teasdale G. Predicting outcome in individual patients after severe headinjury. Lancet. 1976;1:1031-1034.4. Teasdale G, Jennett B. Assessment of coma and severity of brain damage.Anesthesiology. 1978;49:225-226.5. Teasdale G, Murray G, Parker L, et al. Adding up the Glasgow Coma Score. ActaNeurochir. 1979(Suppl);28:13-16.6. Jennett B, Teasdale G, Braakman R, et al. Prognosis of patients with severe headinjury. Neurosurgery. 1979;4:283-289.7. Meredith W, Rutledge R, Hansen AR, et al. Field triage of trauma patients based uponthe ability to follow commands: a study in 29,573 patients. J Trauma. 1995;38:129-135.8. Jagger J, Jane JA, Rimel R. The Glasgow Coma Scale: to sum or not to sum [let-ter]? Lancet. 1983;2:97. 9. Woertgen C, Rothoerl RD, Metz C, et al. Comparison of clinical, radiologic, and serummarker as prognostic factors after severe head injury. J Trauma. 1999;47:1126-1130.10. Waxman K, Sundine MJ, Young RF. Is early prediction of outcome in severe headinjury possible? Arch Surg. 1991;126:1237-1241.11. Zafonte RD, Hammond FM, Mann NR, et al. Relationship between Glasgow ComaScale and functional outcome. Am J Phys Med Rehabil. 1996;75:364-369.12. Servadei F, Nasi MT, Cremonini AM, et al. Importance of a reliable admissionGlasgow Coma Scale score for determining the need for evacuation of posttraumaticsubdural hematomas: a prospective study of 65 patients. J Trauma. 1998;44:868-873.13. Kohi YM, Mendelow AD, Teasdale GM, et al. Extracranial insults and outcome inpatients with acute head injury: relationship to the Glasgow Coma Scale. Injury.1984;16:25-29.14. Alvarez M, Nava J, Rue M, et al. Mortality prediction in head trauma patients: per-formance of Glasgow Coma Score and general severity systems. Crit Care Med.1998;26:142-148.15. Norwood SH, McAuley CE, Berne JD, et al. A prehospital Glasgow Coma Scalescore 14 accurately predicts the need for full trauma team activation and patient hos-pitalization after motor vehicle collisions. J Trauma. 2002;53:503-507. 16. Price T, Miller L, deScossa M. The Glasgow Coma Scale in intensive care: a study.Nurs Crit Care. 2000;5:170-173.17. Zafonte RD, Wood DL, Harrison-Felix CL, et al. Penetrating head injury: a prospec-tive study of outcomes. Neurol Res. 2001;23:219-226.18. Harris OA, Lane B, Lewen A, et al. Infratentorial traumatic brain hemorrhage: mayoutcome be predicted by initial Glasgow Coma Scale? J Trauma. 2000;49:1076-1082.19. Ross SE, Leipold C, Terregino C, et al. Efficacy of the motor component of theGlasgow Coma Scale in trauma triage. J Trauma. 1998;45:42-44.20. The Brain Trauma Foundation, The American Association of NeurologicalSurgeons, The Joint Section on Neurotrauma and Critical Care. Glasgow coma scalescore. J Neurotrauma. 2000;17:563-571.21. Borczuk P. Predictors of intracranial injury in patients with mild head trauma.Ann Emerg Med. 1995;25:731-736. 22. Tellier A, Della Malva LC, Cwinn A, et al. Mild head injury: a misnomer. Brain Injury.1999;13:463-475.


Content validity refers to the ability of the test todirectly assess the true qualities that it intends to mea-sure. In the case of the GCS, this would be the ability toquantify the true functional level of the central nervoussystem. However, the GCS does not measure completeneurologic capacity but rather includes 3 practical sur-rogate markers that cannot be expected to appraise thefull functional domain of the central nervous system. Infact, Teasdale and Jennett4 have stated that “we havenever recommended using the GCS alone, either as ameans of monitoring coma, or to assess the severity ofbrain damage or predict outcome.” Nonetheless, clini-cians widely use the GCS for all of these things.

Despite its popularity and face validity, the GCSlacks the degree of criterion-related validity, constructvalidity, and content validity that would be necessaryto consider it a truly valid clinical tool. Given the near-universal application of the GCS and its prominence intrauma and life support education programs,41,42 wemust regard the GCS as a gross estimate of neurologicresponsiveness rather than as a precise measurement.Our findings of only moderate interrater reliability maybe regarded by some as insufficient to support the cur-rent widespread application of this technique for EDpatient treatment and decisionmaking, as well as theprominence of this tool in trauma and life support edu-cation programs.

In retrospect, although our study found ample dis-agreement in GCS assessments between physicians, ourmethodology was unable to characterize whether suchdifferences were caused by differential quality of physi-cian assessment or just the inherent underlying subjec-tivity of these clinical measures. We could have exploredthese differences by having the 2 physicians review theirscores with each other, reexamine the patient, and thenreport on the etiology of the discrepancy.

In conclusion, we found only moderate degrees ofinterrater agreement for the GCS and its components.

We thank Jason S. Haukoos, MD, MS, for his assistance with statis-tical analysis and Renee Schroetlin for her assistance with dataacquisition.

Author contributions: MRG and SMG conceived of and designed thestudy. MRG oversaw the data collection. SMG provided statisticaladvice on study design and analyzed the data. DGR assisted withdata collection and manuscript development. All authors partici-pated in drafting the manuscript and contributed to its revisions.MRG and SMG take responsibility for the paper as a whole.


Page 13

mailto:[email protected]


56. Gill M, Green SM, Krauss B. Can the bispectral index monitor quantify altered levelof consciousness in emergency department patients? Acad Emerg Med. 2003;10:175-179.57. Johnstone AJ, Lohlun JC, Miller JD, et al. A comparison of the Glasgow ComaScale and the Swedish Reaction Level Scale. Brain Injury. 1993;7:501-506.58. Starmark J, Stålhammar D, Holmgren E, et al. A comparison of the Glasgow ComaScale and the Reaction Level Scale (RLS85). J Neurosurg. 1988;69:699-706.59. Stanczak DE, White JG III, Gouview WD, et al. Assessment of level of con-sciousness following severe neurological insult: a comparison of the psychometricqualities of the Glasgow Coma Scale and the Comprehensive Level of ConsciousnessScale. J Neurosurg. 1984;60:955-960.60. Karras DJ. Statistical methodology II: reliability and validity assessment in studydesign, part A. Acad Emerg Med. 1997;4:144-171.61. Teoh LS, Gowardman JR, Larsen PD, et al. Glasgow Coma Scale: variation in mor-tality among permutations of specific total scores. Intensive Care Med. 2000;26:157-161.62. Servadei F, Teasdale G, Merry G, et al. Defining mild head injury in adults: a pro-posal based on prognostic factors, diagnosis, and management. J Neurotrauma.2001;7:657-664.63. Williams R, Mackway-Jones K. White cell count and diagnosing appendicitis inadults. Emerg Med J. 2002;19:429-430.

23. Choi SC, Muizelaar JP, Barnes TY, et al. Prediction tree for severely head-injuredpatients. J Neurosurg. 1991;75:251-255.24. Nissen JJ, Jones PA, Signorini DF, et al. Glasgow Head Injury Outcome PredictionProgram: an independent assessment. J Neurol Neurosurg Psychiat. 1999;67:796-799.25. Levy DE, Bates D, Caronna JJ, et al. Prognosis in nontraumatic coma. Ann Int Med.1981;94:293-301.26. Hamel MB, Goldman L, Teno J, et al. Identification of comatose patients at highrisk for death or severe disability. JAMA. 1995;273:1842-1848.27. Sacco RL, VanGool R, Mohr JP, et al. Nontraumatic coma: Glasgow coma scoreand coma etiology as predictors of 2-week outcome. Arch Neurol. 1990;47:1181-1184.28. Mullie A, Verstringe P, Buylaert W, et al. Predictive value of Glasgow Coma Scalefor awakening after out-of-hospital cardiac arrest: Cerebral Resuscitation Study Groupof the Belgian Society for Intensive Care. Lancet. 1988;1:137-140.29. Chan B, Gaudry P, Grattan-Smith TM, et al. The use of the Glasgow Coma Scale inpoisoning. J Emerg Med. 1993;11:579-582.30. Choi C, Narayan RK, Anderson RL, et al. Enhanced specificity of prognosis insevere head injury. J Neurosurg. 1988;69:381-385.31. Nagy KK, Joseph KT, Krosner SM, et al. The utility of head computed tomographyafter minimal head injury. J Trauma. 1999;46:268-270.32. Stiell IG, Wells G, Vandemheem K, et al. The Canadian CT Head Rule for patientswith minor head injury. Lancet. 2001;357:1391-1396.33. Haydel MJ, Preston CA, Mills TJ, et al. Indications for computed tomography inpatients with minor head trauma. N Engl J Med. 2000;343:100-105. 34. Teasdale G, Knill-Jones R, Van der Sande J. Observer variability in assessingimpaired consciousness and coma. J Neurol Neurosurg Psychiatry. 1978;41:603-610.35. Rowley G, Fielding K. Reliability and accuracy of the Glasgow Coma Scale withexperienced and inexperienced users. Lancet. 1991;337:535-538.36. Juarez VJ, Lyons M. Interrater reliability of the Glasgow Coma Scale. J NeurosciNurs. 1995;27:283-286.37. Tesseris J, Pantazidis N, Routsi C, et al. A comparative study of the Reaction LevelScale (RLS 85) with Glasgow Coma Scale (GCS) and Edinburgh-2 Coma Scale (Modified)(E2CS(M)). Acta Neurochir (Wien). 1991;110:65-76.38. Elliott M. Interrater reliability of the Glasgow Coma Scale. J Neurosci Nurs.1996;28:213-214.39. Lindsay KW, Teasdale GM, Knill-Jones RP. Observer variability in assessing theclinical features of subarachnoid hemorrhage. J Neurosurg. 1983;58:57-62. 40. Menegazzi J, Davis EA, Sucov AN, et al. Reliability of the Glasgow Coma Scalewhen used by emergency physicians and paramedics. J Trauma. 1993;34:46-48.41. American College of Surgeons Committee on Trauma. Advanced Trauma LifeSupport Manual. 6th ed. Chicago, IL: American College of Surgeons; 1997. 42. Cummins RO, ed. Advanced Cardiac Life Support Provider Manual. Dallas, TX:American Heart Association; 2001.43. Jennett B, Teasdale G, Galbraith S, et al. Severe head injuries in three countries.J Neurol Neurosurg Psychiatry. 1977;40:291-298. 44. Bland JM, Altman DG. Statistical methods for assessing agreement between twomethods of clinical measurement. Lancet. 1986;1:307-310.45. Marion DW, Carlier PM. Problems with initial Glasgow Coma Scale assessmentcaused by prehospital treatment of patients with head injuries: results of a nationalsurvey. J Trauma. 1994;36:89-95.46. Buechler CM, Blostein PA, Koestner A, et al. Variation among trauma centers’ cal-culation of Glasgow Coma Scale score: results of a national survey. J Trauma.1998;45:429-432.47. Rutledge R, Lentz CW, Fakhry S, et al. Appropriate use of the Glasgow Coma Scalein intubated patients: a linear regression prediction of the Glasgow verbal score fromthe Glasgow eye and motor scores. J Trauma. 1996;41:514-522.48. Meredith W, Rutledge R, Fakhry SM, et al. The conundrum of the Glasgow ComaScale in intubated patients: a linear regression prediction of the Glasgow verbal scorefrom the Glasgow eye and motor scores. J Trauma. 1998;44:839-845.49. Wipf J, Lipsky B, Hirschmann J, et al. Diagnosing pneumonia by physical examina-tion: relevant or relic? Arch Intern Med. 1999;159:1082-1087.50. Steinbach W, Sectish T, Benjamin D, et al. Pediatric residents’ clinical diagnosticaccuracy of otitis media. Pediatrics. 2002;109:993-998.51. Etchells E, Bell C, Robb K. Does this patient have an abnormal systolic murmur?JAMA. 1997;277:564-571.52. Mahadevan S, Mower W, Hoffman J, et al. Interrater reliability of cervical spineinjury criteria in patients with blunt trauma. Ann Emerg Med. 1998;31:197-201. 53. Stiell I, Wells G, Vandemheen K, et al. The Canadian CT Head Rule for patients withminor head injury. Lancet. 2001;357:1391-1396.54. Cohen JA. A coefficient of agreement for nominal scales. Educ Psychol Measurem.1960;20:37-46.55. Karras DJ. Statistical methodology II: reliability and validity assessment in studydesign, part B. Acad Emerg Med. 1997;4:144-147.


Page 14

EMERGENCY MEDICAL SERVICES/BRIEF RESEARCH REPORTPage 15

Randomized Controlled Trial of a Scoring Aid to Improve GlasgowComa Scale Scoring by Emergency Medical Services Providers

Amanda Feldman, BS; Kimberly W. Hart, MA; Christopher J. Lindsell, PhD; Jason T. McMullan, MD*

*Corresponding Author. E-mail: [email protected].

Volume 6

Study objective: Emergency medical services (EMS) personnel frequently use the Glasgow Coma Scale (GCS) to assessinjured and critically ill patients. This study assesses the accuracy of EMS providers’ GCS scoring, as well as theimprovement in GCS score assessment with the use of a scoring aid.

Methods: This randomized, controlled study was conducted in the emergency department (ED) of an urban academictrauma center. Emergency medical technicians or paramedics who transported a patient to the ED were randomlyassigned one of 9 written scenarios, either with or without a GCS scoring aid. Scenarios were created by consensus ofexpert attending emergency medicine, EMS, and neurocritical care physicians, with universal consensus agreement onGCS scores. c2 And Student’s t tests were used to compare groups.

Results: Of 180 participants, 178 completed the study. Overall, 73 of 178 participants (41%) gave a GCS score thatmatched the expert consensus score. GCS score was correct in 22 of 88 (25%) cases without the scoring aid. GCS wascorrect in 51 of 90 (57%) cases with the scoring aid. Most (69%) of the total GCS scores fell within 1 point of the expertconsensus GCS score. Differences in accuracy were most pronounced in scenarios with a correct GCS score of 12 orbelow. Subcomponent accuracy was eye 62%, verbal 70%, and motor 51%.

Conclusion: In this study, 60% of EMS participants provided inaccurate GCS score estimates. Use of a GCS scoring aidimproved accuracy of EMS GCS score assessments. [Ann Emerg Med. 2015;65:325-329.]

Please see page 326 for the Editor’s Capsule Summary of this article.

A podcast for this article is available at www.annemergmed.com.

0196-0644/$-see front matterCopyright © 2014 by the American College of Emergency Physicians.http://dx.doi.org/10.1016/j.annemergmed.2014.07.454

INTRODUCTIONBackground and Importance

First introduced in 1974, the Glasgow Coma Scale (GCS) iscommonly used to describe the level of consciousness in and topredict outcomes of a wide variety of patients, including those inthe out-of-hospital setting.1-3 Out-of-hospital providers usebaseline and changes in GCS score assessments to indicate theseverity of injuries or illness, and to aid in patient triage.2,4 Inaddition to its clinical utility, the GCS is commonly used inresearch for ascertainment of participant eligibility and as anoutcomes assessment or adjustment for baseline severity.5

Because the GCS can play a role in the initial and ongoingtreatment of the patient, quick and accurate evaluation is necessary.The ability of out-of-hospital providers to accurately score theGCShas not been well reported, yet anecdotally they are often criticizedfor inaccurate GCS score assessment. There are only limited datacharacterizing the degree of emergency medical services (EMS)GCS inaccuracy.6 Furthermore, interrater reliability of GCSscoring is known to be low, including in the out-of-hospitalsetting.4,7,8 An aid to facilitate quick recall of the GCS in real timecould improve scoring accuracy.9

5, no. 3 : March 2015

Goals of This InvestigationThis study assessed the accuracy of EMS providers’ GCS

scoring of written scenarios and estimated the potential for aGCS scoring aid to improve accuracy. We hypothesized thatproviders who were assisted by a GCS scoring table would assessGCS more accurately than those who were not.

MATERIALS AND METHODSStudy Design and Setting

This randomized controlled study of the utility of a GCSscoring table aidwas conducted in the emergency department (ED)of an urban, academic Level I trauma center. The University ofCincinnati Institutional Review Board approved the study.

Selection of ParticipantsParticipants were emergency medical technicians or

paramedics who had transported a patient to the ED. Weenrolled subjects during times of study personnel availability,which included weekdays, nights, and weekends. Providers werepermitted to participate only once and had to be older than 18

Annals of Emergency Medicine 325


http://annemergmed.com/content/podcast

http://www.annemergmed.com

http://dx.doi.org/10.1016/j.annemergmed.2014.07.454

Scoring Aid to Improve GCS Scoring by EMS Feldman et alPage 16

Editor’s Capsule Summary

What is already known on this topicEmergency medical services (EMS) personnelassessments of Glasgow Coma Scale (GCS) score areoften inaccurate.

What question this study addressedDoes the use of a scoring aid improve the accuracy ofEMS GCS score assessments?

What this study adds to our knowledgeAmong mock written scenarios evaluated by 178EMS personnel, GCS score accuracy was higher with(57%) than without (25%) a scoring aid.

How this is relevant to clinical practiceThis scenario-based study found that GCS scoringaccuracy for EMS personnel was low even whenassisted by a scoring aid, further undermining thevalue of this already poor tool for neurologicassessment.

years. The study was approved by the institutional review board,and all participants provided informed consent.

Methods of MeasurementNine standardized brief patient scenarios (Table E1, available

online at http://www.annemergmed.com) were modified from 3widely used EMS textbooks.10-12 Attending physicians specializingin emergency medicine, EMS, and neurocritical care reviewed thescenarios and provided revisions until there was universalagreement on GCS scores. The scenarios depicted patients withGCS scores corresponding tomild (GCS score 13 to 15), moderate(GCS score 9 to 12), and severe (GCS score 3 to 8) traumatic braininjuries. The test scenarios and expert consensus GCS scores wereverified by an independent team of paramedic instructors.

Scenarios with or without the scoring table were placed intosequentially numbered, sealed envelopes for distribution toparticipants. Each participant was randomly assigned todetermine GCS scores on one of the 9 scenarios, with or withoutthe scoring table. No blinding methods were used afterrandomization. Participants were asked to provide the total GCSscore of the patient in the scenario, as well as the eye, verbal, andmotor subcomponent scores. The participants’ demographicinformation was collected, including experience, level of training,and EMS practice habits.

Outcome MeasuresThe primary outcome was the absolute agreement between

the participants’ assigned GCS scores and the correct GCS scoredetermined by the attending physician review. Secondary

326 Annals of Emergency Medicine

outcomes included the frequency of scores falling within 1 pointof the correct score, accuracy of subcomponent scores, andaccuracy for the different levels of severity.

A sample size of 90 in each group would have 80% power todetect an absolute difference of 15% of the proportion of subjectsable to correctly determine the GCS score with or without theGCS scoring aid when a¼.05 and conservatively assuming awide SD.

Data Collection and ProcessingThe 1:1 randomization sequence was generated with nQuery

Adviser (version 7.0; Statistical Solutions, Boston, MA) anddesigned to ensure equal distribution of the 9 scenarios amongthose receiving the GCS aid and those not receiving it.Participants’ responses were entered into an electronic database(REDCap; Vanderbilt University, Nashville, TN). Out-of-rangeGCS scores were queried and confirmed. Missing data wereminimal and left missing. We compared participant GCS scorewith expert consensus GCS ratings, using the c2 test or Fisher’sexact test, as appropriate, to test for differences in proportionsand calculated 95% confidence intervals (CIs) for the effectsize. We adjusted for multiple comparisons with Sidak’smethod.

All statistical analyses were conducted with SPSS (version22.0; IBM Corporation, Armonk, NY). Graphics were createdwith R (gplots). Differences in means and proportions and 95%CIs were calculated.

RESULTSCharacteristics of Study Subjects

Between April 2013 and June 2013, 261 subjects werescreened; 16 declined participation and 65 did not meet inclusioncriteria. Of 180 subjects enrolled, 2 participants were excludedbecause of incomplete GCS scores, leaving 178 cases in the analysis(Figure E1, available online at http://www.annemergmed.com).

Participant characteristics are described in Table 1.Approximately half (52%) were paramedics. Participants weredrawn from 41 EMS departments or agencies, which werediverse and included rural, suburban, and urban settings; paidand volunteer staffing models; and annual call volumes rangingfrom less than 500 to greater than 55,000. The mean length ofexperience was 12 years (SD 8). Most participants (70%)reported they had been refreshed on GCS material through acourse, recertification, or training within the past year, and 56%stated they consistently use some sort of aid in the field to helpdetermine the GCS score. The 2 study arms were well matchedin experience and certification levels, and no protocol deviationsoccurred.

Overall, 73 of 178 participants (41%) gave a GCS score thatmatched the correct GCS score (Table 2; Figure). Amongparticipants who did not receive the standard GCS scoring tableas an aid, the GCS score was correct in 22 of 88 cases (25%)compared with 51 of 90 (57%) for those who did receive thetable aid (difference in proportions 32%; 95% CI 18% to 46%).

Volume 65, no. 3 : March 2015



Table 1. Participant characteristics.

No Table Aid (n[88) Table Aid (N[90) Total (n[178)

Age, mean (SD), y 37 (10) 36 (9) 36 (9)Race, No. (%)White 72 (81.8) 76 (84.4) 148 (83.1)Black 15 (17.0) 11 (12.2) 26 (14.5)American Indian/Alaskan Native 0 (0) 2 (2.2) 2 (1.1)Other 1 (1.1) 0 (0) 1 (0.6)Asian/Pacific Islander 0 (0) 1 (1.1) 1 (0.6)Male, No. (%) 80 (90.9) 77 (85.6) 157 (88.2)Level of EMS certification, No. (%)EMT-basic 44 (50.0) 39 (43.3) 83 (46.9)EMT-intermediate 2 (2.3) 0 (0) 2 (1.1)Paramedic 42 (47.7) 51 (56.7) 93 (52.0)Years of experience, mean (SD) 12 (8) 11 (7) 12 (8)Refreshed on GCS material within the past year, No. (%) 58 (65.9) 67 (74.4) 125 (70.2)EMS instructor, No. (%) 6 (6.8) 7 (7.8) 13 (7.8)Use aid to determine the GCS in the field, No. (%) 54 (61.4) 45 (50.0) 99 (50.0)

Feldman et al Scoring Aid to Improve GCS Scoring by EMSPage 17

Overall, 123 of 178 scores (69%) fell within 1 point of thecorrect GCS score. There was equal likelihood to overestimate(29.2%) and underestimate (29.8%) the total GCS score. Morescores were correct within 1 point in the group who received thetable aid than the group who did not (82.2% versus 55.7%;difference 26.5%; 95% CI 13.5% to 39.6%). The meandifference between actual and participant-assigned GCS scoresfor the group without the table was 2.6, and the meanparticipant-assigned GCS scores for the group with the table was2.1 (difference of means 0.5; 95% CI –0.3 to 1.3).

The difference in accuracy between the 2 groups was mostpronounced in the moderate (GCS score 9 to 12) and severe

Table 2. Scoring of patient scenarios by EMS providers.*

Total (n[178) No Table Aid (n[88)

No. % No. %

All GCS scenariosTotal 73 (41.0) 22 (25.0)Eye 110 (61.8) 38 (43.2)Verbal 125 (70.2) 48 (54.5)Motor 90 (50.6) 27 (30.7)Mild TBI scenarios (GCS score 13–15)Total 32 (54.2) 13 (44.8)Eye 41 (69.5) 16 (55.2)Verbal 47 (79.7) 21 (72.4)Motor 44 (74.6) 17 (58.6)Moderate TBI scenarios (GCS score 9–12)Total 17 (28.8) 3 (10.3)Eye 37 (62.7) 12 (41.4)Verbal 41 (69.5) 15 (51.7)Motor 21 (35.6) 6 (20.7)Severe TBI scenarios (GCS score 3–8)Total 24 (40.0) 6 (20.0)Eye 32 (53.3) 10 (33.3)Verbal 37 (61.7) 12 (40.0)Motor 25 (41.7) 4 (13.3)

TBI, Traumatic brain injury.*Results are presented as the proportion of absolutely correctly assigned composite andscenarios.


(GCS score 3 to 8) scenarios (Table 2; Figure). Twelveparticipants (7%) gave subcomponent scores that are notpossible on the scale. Eye component accuracy improved from43% without the table aid to 80% with the table aid, the verbalfrom 55% to 86%, and motor from 31% to 70% (Table 2;Figure).

LIMITATIONSScoring a written scenario in a controlled environment and

assigning a GCS score during the immediate evaluation andtreatment phase of an acutely ill or injured patient are inherently

Table Aid (n[90)

% Difference

95% CI

No. % Lower Upper

51 (56.7) 31.9 18.3 45.672 (80.0) 37.3 24.1 50.577 (85.6) 31.6 19.0 44.363 (70.0) 39.7 26.2 53.1

19 (63.3) 14.3 �6.1 34.625 (83.3) 18.5 �6.5 43.526 (86.7) 28.2 5.7 50.627 (90.0) 29.3 6.1 52.5

14 (46.7) 31.4 10.5 52.325 (83.3) 34.9 13.1 56.826 (86.7) 36.3 15.3 57.315 (50.0) 40.0 17.4 62.6

18 (60.0) 40.0 16.9 63.122 (73.3) 42.0 19.6 64.325 (83.3) 43.3 21.3 65.421 (70.0) 56.7 36.2 77.1

component GCS scores and further stratified by mild, moderate, and severe TBI



different. Although we have mirrored previously usedmethodologies,9,13 our approach may overestimate accuracybecause the EMS providers are not subject to the task saturationof clinical care. Conversely, it is possible that the informationgained from examining a live patient is more useful than thatpresented in a written scenario, which could improve theaccuracy of GCS estimates.

We did not record whether subjects sought help in scoring thescenarios either from another EMS provider or from theirpersonal GCS scoring aids, although no such activity waswitnessed. Use of a scoring aid in the group not given one as partof the study would bias toward improved accuracy, and failure touse the aid provided would cause the opposite. In either case, theactual effect sizes would be greater than we observed.

Although we identified a deficiency in GCS scoring by EMSproviders, we are unable to speculate about the reasons such adeficiency exists. Use of a scoring aid does improve accuracy, butdiscovery of other potential causes—and solutions—would beuseful.

DISCUSSIONThese results suggest that GCS score assessment with a scoring

table improves the accuracy of EMS providers’ GCS scoring of

Figure. Dot plot of assigned composite and component GCS scoresa single respondent.


patients, using written scenarios. However, even with use of a GCStable, accuracy of GCS scoring by EMS providers was low.

The GCS has been criticized as somewhat complex, and thefundamental utility and appropriateness of GCS scores inemergency medicine, and out-of-hospital care by proxy, have beenchallenged.14 We agree that GCS is imperfect and we support callsfor a better tool. However, despite these criticisms,14 GCS is stillthe tool that has been universally adopted in clinical care. Until theGCS can be replaced, accurate scoring using the GCS should beemphasized. The inability of EMS providers to accurately assess aninjured patient and communicate the findings is a problemregardless of the tool used.

Our data empirically quantify the inaccuracy, offeringproviders information that should be useful in interpreting anout-of-hospital GCS score. Clinically, a 1-point discrepancy inthe GCS score may be acceptable, and when a scoring table aidwas made available to providers, 82% of scores were within 1point of the correct score. In other situations, even a 1-pointerror may prompt inappropriate field triage to a trauma center,exclusion from a clinical trial, or consideration of a procedure (ie,intubation). Providers were just as likely to overestimate andunderestimate scores, and the magnitude of the difference wasfrequently enough to change the assigned category in the mild/moderate/severe classification scheme (Figure).

for each scenario. Each circle represents the score assigned by



Although our findings of inaccuracy when relying onmemory alone are, unfortunately, not unique,9 we show thathaving a scoring table aid readily available more than doubles(25% to 57%) the number of accurate scores. To ourknowledge, this is the first intervention shown to improve GCSscoring accuracy. In accordance with our observations, EMSproviders should be given GCS scoring cards, with real-time usestrongly encouraged.

Some health care providers have advocated abandonment ofthe full GCS and suggest simplifications or using only the motorcomponent.15-18 In our sample, the motor score was the leastreliable of the subcomponents. Proposed alternatives to the GCSthat simplify assessment of consciousness include the FOURscore and the Emergency Coma Scale.19,20 However, thesescoring methods may also suffer from accuracy limitationsbecause the eye and motor components are similar to those of theGCS. Additionally, many of the articles that compare GCS witha newer tool of mental status assessment rely on retrospectivelyrecorded out-of-hospital GCS values as the criterionstandard.18,19 Our results do not support abandoning the fullGCS in favor of these alternatives.

Our findings provide the key insights about the inaccuracy ofGCS scoring by EMS and support the need for improved toolsfor evaluating out-of-hospital patients with neurologicemergencies. Until a new method of evaluating altered mentalstatus in the setting of trauma is developed, validated, andadopted, use of a GCS scoring aid may help to improve theaccuracy of the EMS GCS score assessments.

Supervising editor: Henry E. Wang, MD, MS

Author affiliations: From the Northeast Ohio Medical University,Akron, OH (Feldman); and the University of Cincinnati Departmentof Emergency Medicine, Cincinnati, OH (Hart, Lindsell, McMullan).Ms. Feldman was formerly affiliated with the University ofCincinnati Department of Emergency Medicine.

Author contributions: ALF, CJL, and JTM conceived and designedthe study. ALF implemented the study, conducted all datacollection, and drafted the article. KWH conducted the statisticalanalysis. All authors contributed to the interpretation of results,provided critical review and revisions for intellectual content for thearticle, and approved the final version. ALF takes responsibility forthe paper as a whole.

Funding and support: By Annals policy, all authors are required todisclose any and all commercial, financial, and other relationshipsin any way related to the subject of this article as per ICMJEconflict of interest guidelines (see www.icmje.org). The authorshave stated that no such relationships exist and provided thefollowing details: This project was supported in part by anInstitutional Clinical and Translational Science Award, NIH/NCRRgrant 5UL1RR026314-03.

Publication dates: Received for publication February 18, 2014.Revisions received June 23, 2014; and July 9, 2014. Accepted forpublication July 29, 2014. Available online September 6, 2014.


Presented as a poster at the National Association of EMSPhysicians annual meeting, January 2014, Tucson, AZ.

REFERENCES1. Teasdale G, Jennett B. Assessment of coma and impaired

consciousness. A practical scale. Lancet. 1974;2:81-84.2. Davis DP, Serrano JA, Vilke GM, et al. The predictive value of

field versus arrival Glasgow Coma Scale score and TRISScalculations in moderate-to-severe traumatic brain injury. J Trauma.2006;60:985-990.

3. Badjatia N, Carney N, Crocco TJ, et al. Guidelines for prehospitalmanagement of traumatic brain injury 2nd edition. Prehosp EmergCare. 2008;12(suppl 1):S1-S52.

4. Kerby JD, MacLennan PA, Burton JN, et al. Agreement betweenprehospital and emergency department glasgow coma scores.J Trauma. 2007;63:1026-1031.

5. Bulger EM, May S, Brasel KJ, et al. Out-of-hospital hypertonicresuscitation following severe traumatic brain injury: a randomizedcontrolled trial. JAMA. 2010;304:1455-1464.

6. Bazarian JJ, Eirich MA, Salhanick SD. The relationship between pre-hospital and emergency department Glasgow Coma Scale scores.Brain Inj. 2003;17:553-560.

7. Baez AA, Giraldez EM, De Pena JM. Precision and reliability of theGlasgow Coma Scale score among a cohort of Latin Americanprehospital emergency care providers. Prehosp Disaster Med.2007;22:230-232.

8. Gill MR, Reiley DG, Green SM. Interrater reliability of Glasgow ComaScale scores in the emergency department. Ann Emerg Med.2004;43:215-223.

9. Lane PL, Baez AA, Brabson T, et al. Effectiveness of a Glasgow ComaScale instructional video for EMS providers. Prehosp Disaster Med.2002;17:142-146.

10. Hsieh A, Boone K. The Paramedic Companion: A Case-based Worktext.Boston, MA: McGraw-Hill; 2008:240-249.

11. Caroline NL. Study Guide for Emergency Care in the Streets. 4th ed.Boston, MA: Little, Brown & Co; 1991:94.

12. Dalton A, Walker RA. Mosby’s Paramedic Refresher and Review: ACase Studies Approach. St. Louis, MO: Mosby; 1999:295-297.

13. Heim C, Schoettker P, Gilliard N, et al. Knowledge of Glasgow ComaScale by air-rescue physicians. Scand J Trauma Resusc Emerg Med.2009;17:39.

14. Green SM. Cheerio, laddie! bidding farewell to the Glasgow ComaScale. Ann Emerg Med. 2011;58:427-430.

15. Gill M, Martens K, Lynch EL, et al. Interrater reliability of 3 simplifiedneurologic scales applied to adults presenting to the emergencydepartment with altered levels of consciousness. Ann Emerg Med.2007;49:403-407; 7.e1.

16. Healey C, Osler TM, Rogers FB, et al. Improving the Glasgow ComaScale score: motor score alone is a better predictor. J Trauma.2003;54:671-678; discussion 8-80.

17. Gill M, Steele R, Windemuth R, et al. A comparison of five simplifiedscales to the out-of-hospital Glasgow Coma Scale for the prediction oftraumatic brain injury outcomes. Acad Emerg Med. 2006;13:968-973.

18. Thompson DO, Hurtado TR, Liao MM, et al. Validation of the SimplifiedMotor Score in the out-of-hospital setting for the prediction ofoutcomes after traumatic brain injury. Ann Emerg Med.2011;58:417-425.

19. Wijdicks EF, Bamlet WR, Maramattom BV, et al. Validation of a newcoma scale: the FOUR score. Ann Neurol. 2005;58:585-593.

20. Takahashi C, Okudera H, Origasa H, et al. A simple and useful comascale for patients with neurologic emergencies: the Emergency ComaScale. Am J Emerg Med. 2011;29:196-202.


http://www.icmje.org/

http://refhub.elsevier.com/S0196-0644(14)00193-0/sref1


























































Figure E1. CONSORT flow diagram of subjects.


329.e1 Annals of Emergency Medicine Volume 65, no. 3 : March 2015

Table E1. Patient scenarios randomly provided to on-duty EMS providers for GCS scoring, with the correct component and compositescores for each.

Eye Verbal Motor Total

Mild TBI1 You respond to the scene of a 50-y-old man who was injured in a bicycle accident. A car pulled in front of

him, forcing him off the road, and he fell in a grassy median. He is sitting up, inspecting his helmet, andis only complaining of road rash to his arms and legs. He is giving the police a description of the carinvolved and limps to the ambulance to have his wounds dressed.

4 5 6 15

2 You respond to an 18-y-old man involved in a single-car MVC, in which he struck a tree. There is moderatedamage to the car, and he is sitting on the curb. When you ask him the date, he has slurred speech andstates, “December 12, 2002” (it is actually February 14, 2013). When you ask him to show you 2fingers, he giggles and flips you off with both hands. You note a strong odor of alcohol on his breath andpossible track marks on his arms.

4 4 6 14

3 You respond to a 28-y-old woman struck in the head by a canoe oar. She was pulled from the river bybystanders, who state she was unconscious. She was wearing a life jacket. She awakens when you askwhether she is okay and asks what happened over and over again. She shows you 2 fingers on eachhand when you ask her to do so.

3 4 6 13

Moderate TBI4 You respond to a 16-y-old woman who was a pedestrian struck by a car and find her lying in the street.

She has a boggy hematoma to the right side of her scalp and abrasions to her arms and legs. Herpupils are equal, round, and reactive to light. She opens her eyes when you call her name but she isconfused. She knows her name and the year, but not the date or month. When you ask her to show you2 fingers, she looks confused and does nothing. When you attempt to insert an IV line in her right arm,she quickly pulls away, swats at you with her left hand, and says, “Stop it!”

3 4 5 12

5 You respond to a 24-y-old male assault victim who was struck in the side of his head with a baseball batduring a bar fight. You find him breathing but unconscious. When you perform a sternal rub, he openshis eyes and tries to pull away, and stops when you stop stimulating him. When you constantly sternallyrub him, he will talk to you and thinks the year is 1963 when asked (it is 2013).

2 4 4 10

6 You respond to a call for a “man down” by the railroad tracks and find a disheveled 45-y-old manfacedown and parallel to the tracks. As you approach, he is moaning but you cannot understand whathe is trying to say. He will not answer questions or follow commands. His breathing is normal, he iscovered in blood, and he has obvious deformities to his right arm and leg. When you apply a sternal rub,he reaches for your hand and briefly opens his eyes; his pupils are equal.

2 2 5 9

Severe TBI7 You respond to a 22-y-old woman who was pushed down the stairs during a fight with her boyfriend. She

fell down 12 wooden steps and landed on the cement basement floor. She is bleeding from the noseand mouth and has an obvious deformity to her left wrist. She will briefly open her eyes to a sternal rub,and her pupils are normal-sized and sluggishly reactive to light. She tries to pull away when you pinchher shoulder and mumbles something you cannot understand, but settles when you stop applyingstimulation.

2 2 4 8

8 You respond to a motorcycle accident in which an unhelmeted rider hit a car that unexpectedly pulled outof a parking lot. He is found lying supine in the road 20 feet from the site of impact. Initially, you noticethat he has irregular, snoring respirations and has obvious trauma to his head, face, and right leg. He isunresponsive and does not open his eyes to a deep sternal rub. You pull his eyelids open and discoverthat his left pupil is 2 mm larger than his right. Other than noisy respirations, he makes no sounds atany time. When you apply a sternal rub, his arms pull into his chest and his legs straighten out.

1 1 3 5

9 You respond to an 18-y-old man who fell out of a tree and landed on his head. He is briskly bleeding fromhis scalp but has no other obvious injuries. His only response to a deep sternal rub is to moan andgroan. On your secondary examination, you find that his pupils are equal, dilated, and sluggishlyreactive to light. There are several empty beer bottles at the base of the tree, and there is a strong odorof alcohol on his breath.

1 2 1 4


Volume 65, no. 3 : March 2015 Annals of Emergency Medicine 329.e2

JOURNAL OF NEUROTRAUMAVolume 22, Number 11, 2005© Mary Ann Liebert, Inc.Pp. 1327–1334

Physician Knowledge of the Glasgow Coma Scale

RONALD G. RIECHERS II,1,3 ANTHONY RAMAGE,2 WILLIAM BROWN,2AUDREY KALEHUA,3 PETER RHEE,4 JAMES M. ECKLUND,5

and GEOFFREY S.F. LING1,3,5

ABSTRACT

Appropriate triage is critical to optimizing outcome from battle related injuries. The Glasgow ComaScale (GCS) is the primary means by which combat casualties, who have suffered head injury, aretriaged. For the GCS to be reliable in this critical role, it must be applied accurately. To determinethe level of knowledge of the GCS among military physicians with exposure and/or training in thescale we administered a prospective, voluntary, and anonymous survey to physicians of all levels oftraining at military medical centers with significant patient referral base. The main outcome mea-sures were correct identification of title and categories of the GCS along with appropriate scoringof each category. Overall performance on the survey was marginal. Many were able to identify what“GCS” stands for, but far fewer were able to identify the titles of the specific categories, let aloneidentify the specific scoring of each category. When evaluated based on medical specialties, those insurgical specialties outperformed those in the medical specialties. When comparing the different lev-els of training, residents and fellows performed better than attending staff or interns. Finally, thosewith Advanced Trauma Life Support (ATLS) certification performed significantly better than thosewithout the training. Physician knowledge of the GCS, as demonstrated in this study, is poor, evenin a population of individuals with specific training in the use of the scale. It is concluded that, tooptimize outcome from combat related head injury, methods for improving accurate quantitationof neurologic state need to be explored.

Key words: brain trauma; Glasgow Coma Scale; human; outcome; triage

1327

1Departments of Neurology and 5Neurosurgery, Walter Reed Army Medical Center, Washington, D.C.2Department of Internal Medicine, Eisenhower Army Medical Center, Augusta, Georgia.3Department of Neurology, Uniformed Services, University of the Health Sciences, Bethesda, Maryland.4Navy Trauma Training Center, Los Angeles County Medical Center, Los Angeles, California.

INTRODUCTION

TRAUMATIC BRAIN INJURY (TBI) accounts for up to 20%of injuries incurred in battle by the U.S. military in

Operation Iraqi Freedom (OIF) and Operation EnduringFreedom (OEF). TBI is also a leading cause of death anddisability among civilians (Kraus and McArthur, 1996).

When treating TBI casualties in the far forward environ-ment, triage is paramount. For triage to be optimal, mili-tary medical first providers require an accurate and reli-able clinical assessment of TBI. This is being done usingthe Glasgow Coma Scale (GCS) (Teasdale and Jennett,1974). An important but unsubstantiated assumption is thatthe GCS is being properly administered.

Page 22

Page 23

Page 24

Introduction

The Glasgow Coma Scale (GCS) was developed byTeasdale and Jennett in 1974 [1] to assess the depthand severity of head trauma. The [GCS] assesses patientresponsiveness through increasing degrees of dysfunc-tion in three features: eye responses (E), motor respons-es (M) and verbal responses (V). Subsequently, scoreswere allocated to performance on the three separatecomponents and the total Glasgow Coma Score(GCSc) is derived from the summation of these [2]. Acorrelation between GCSc and outcome has been dem-onstrated in patients with head injury [2]. In addition

to being used in patients with head injuries, GCS hasbeen studied in a number of other patient subgroups.Correlations between outcome and total GCSc (T)have been reported in patients with infectious disease[3], respiratory infections [4], poisoning [5] and in postcardiac arrest patients [6]. The GCS and GCSc are com-monly used in the intensive care setting to assess abroad spectrum of patients with some form of brain fail-ure, as a component in other scoring systems such as theAcute Physiology and Chronic Health Evaluation(APACHE), as a guide to severity of illness and as auseful and objective assessment of a patient's progressover time.

L.S. G.TeohJ. R. GowardmanP.D. LarsenR. GreenDC Galletly

Glasgow Coma Scale:variation in mortality amongpermutations of specific total scores

Received: 18 AprilFinal revision received: 4 October 1999Accepted: 13 October 1999

L. S. G. Teoh ´ P.D. Larsen ´DC Galletly ())Section of Anaesthesia,Wellington School of Medicine,Wellington, New Zealande-mail: [email protected].: + 64(4)385 5558Fax: + 64(4)3 895318

R. GreenDepartment of Public Health,Wellington School of Medicine,Wellington, New Zealand

J. R.GowardmanIntensive Care Unit, Wellington Hospital,Wellington, New Zealand

Abstract Objective: The objectiveof this study was to determinewhether different score permuta-tions of the Glasgow Coma Scale(GCS) giving the same GCS totalscore were associated with signifi-cantly different mortality.Design: For each GCS total wecompared the mortality associatedwith each of the different GCS per-mutations using a Fisher's exact test.The relationship between compo-nents of the GCS score and mortali-ty was also examined using uni- andmultivariate logistic regression.Setting: Data were collected fromthe intensive care unit at WellingtonHospital, a multidisciplinary, tertia-ry referral unit.Patients: We analysed the GCS andmortality data from all admissionsover a 4 year period (January1994±January 1998). Patients withGCS scores of 3 or 15 were exclud-

ed, since these two total scores donot have multiple permutations,leaving 1390 patients with GCSscores of 4±14 for analysis.Results: The incidence of mortalitywas significantly different for thedifferent permutations for totalGCS scores of 7, 9, 11 and 14.Conclusions. It is possible for pa-tients to have the same total GCSscore, but significantly differentrisks of mortality due to differencesin the GCS profile making up thatscore. This suggests that GCS scoresmay be more useful reported interms of profiles rather than totals.This could also have implications forthe use of other scoring systems suchas Acute Physiology and ChronicHealth Evaluation and SimplifiedAcute Physiology Score.

Key words Glasgow Coma Scale ´Outcome ´ Permutation

Intensive Care Med (2000) 26: 157±161Ó Springer-Verlag 2000 ORIGINAL

Page 25

A GCS score of 4 predicts a mortality rate of 48% if calculated 1+1+2 for eye, verbal, and motor, a mortality of 27% if calculated 1+2+1, but a mortality of only 19% if calculated 2+1+1.

Page 26

Comments from the creators of the GCS

“We have never recommended using the GCS alone, either as a means of monitoring coma, or to assess the severity of brain damage or predict outcome.”

Teasdale G, Jennett B. Assessment of coma and severity of brain damage. Anesthesiology. 1978;49:225-226.

The total GCS “assumes an equal weighting for the three responses. More importantly, the information conveyed by the [total] coma score is less than that contained in the three responses separately … Indeed in Glasgow patients are always described by the three separate responses and never by the total.”

Teasdale G, Jennett B, Murray L, et al. Glasgow Coma Scale: to sum or not to sum. Lancet. 1983;2:678.

Page 27

TRAUMA/ORIGINAL RESEARCHPage 28

Volume -, no.

Glasgow Coma Scale Motor Component(“Patient Does Not Follow Commands”) Performs

Similarly to Total Glasgow Coma Scale inPredicting Severe Injury in Trauma Patients

Douglas F. Kupas, MD*; Eric M. Melnychuk, DO; Amanda J. Young, MS

*Corresponding Author. E-mail: [email protected].

Study objective: Trauma victims are frequently triaged to a trauma center according to the patient’s calculated GlasgowComa Scale (GCS) score despite its known inconsistencies. The substitution of a simpler binary assessment of GCS-motor (GCS-m) score less than 6 (ie, “patient does not follow commands”) would simplify field triage. We compare totalGCS score to this binary assessment for predicting trauma outcomes.

Methods: This retrospective analysis of a statewide trauma registry includes records from 393,877 patients from 1999to 2013. Patients with initial GCS score less than or equal to 13 were compared with those with GCS-m score less than6 for outcomes of Injury Severity Score (ISS) greater than 15, ISS greater than 24, death, ICU admission, need forsurgery, or need for craniotomy. We judged a priori that differences less than 5% lack clinical importance.

Results: The relative differences between GCS and GCS-m scores less than 6 were less than 5% and thus clinicallyunimportant for all outcomes tested, even when statistically significant. For the 6 outcomes, the differences in areasunder receiver operating characteristic curves ranged from 0.014 to 0.048. Total GCS score less than or equal to 13was slightly more sensitive (difference 3.3%; 95% confidence interval 3.2% to 3.4%) and slightly less specific (difference–1.5%; 95% confidence interval –1.6% to –1.5%) than GCS-m score less than 6 for predicting ISS greater than 15, withsimilar overall accuracy (74.1% versus 74.2%).

Conclusion: Replacement of the total GCS score with a simple binary decision point of GCS-m score less than 6, or apatient who “does not follow commands,” predicts serious injury, as well as the total GCS score, and would simplify out-of-hospital trauma triage. [Ann Emerg Med. 2016;-:1-7.]

Please see page XX for the Editor’s Capsule Summary of this article.

0196-0644/$-see front matterCopyright © 2016 by the American College of Emergency Physicians.http://dx.doi.org/10.1016/j.annemergmed.2016.06.017

INTRODUCTIONBackground

Rapid and accurate assessment of a patient’s condition isessential for trauma triage in the field. Emergency medicalservices (EMS) providers of all levels must be able toquickly evaluate and classify patients for appropriatetransport destination while providing medical care.Avoiding undertriage by transporting patients withpotentially serious injuries to an appropriate trauma centerreduces mortality, but overtriage causes a strain onresources and is inconvenient for patients.

The guidelines for field triage of injured patients weredesigned for use by EMS providers to identify patients withpotentially serious injuries and determine the mostappropriate level of care.1,2 The 2011 version consists of 4steps to determine the appropriate destination for patients.

- : - 2016

Step 1 includes physiologic criteria, including assessment ofvital signs and the Glasgow Coma Scale (GCS) score, andrecommends that a patient with a GCS score of less than orequal to 13 be transported to a trauma center, preferably tothe highest level of care within the defined trauma system.

ImportanceRecent research on step-specific field triage has shown

the motor component of the GCS (GCS-m) to be a morespecific and simpler tool for patient assessment.3-8 Acalculated GCS score of less than or equal to 13 may be astatistically more sensitive and less specific indicator ofserious injury than the GCS-m score, which may lead toovertriaging of patients and thus transporting patients tomore distant resources that may not be needed for them.Small differences may not be clinically significant, and field



http://dx.doi.org/10.1016/j.annemergmed.2016.06.017

Motor Component of the Glasgow Coma Scale Kupas, Melnychuk & YoungPage 29

Editor’s Capsule Summary

What is already known on this topicThe Glasgow Coma Scale (GCS) is widely used as acriterion for field triage of injured patients to traumacenters.

What question this study addressedDoes a single GCS element (GCS motor componentscore <6 or “patient does not follow commands”)predict trauma outcomes, as well as the widely usedthreshold of total GCS score less than or equal to 13?

What this study adds to our knowledgeIn this analysis of a 393,877-adult statewide traumaregistry, the differences observed between the newdecision point and GCS score less than or equal to 13were all below the prespecified 5% threshold ofclinical importance for 8 trauma outcomes.

How this is relevant to clinical practiceThe full GCS is unnecessarily complicated for out-of-hospital field triage and can be effectively replaced bythe single decision point “patient does not followcommands.”

use of GCS-m score may be more reliable than the totalcalculated GCS score. The GCS score is only 1 parameterof trauma triage; therefore, these relatively small differencesin sensitivity have an even smaller influence on overalltrauma triage sensitivity. The National Expert Panel onField Triage considered emerging evidence for the use ofGCS-m score during their literature review whendeveloping the 2011 guidelines, but this group ultimatelydid not include use of the GCS-m score in the currentguidelines because of “lack of confirmatory evidence, thelong standing use of total GCS and its familiarity amongcurrent EMS practitioners, the inclusion of the motor scorewithin the total GCS, and complications because of thedifficulty of comparing scoring systems.”1 However, severalstudies have indicated a significant interobserver variabilityin tallying the total GCS score, with discrepancies as highas 3 points.9 Even the assessment of the GCS-m scoresuffers from lack of standardization, with variations basedon type of painful stimuli applied to elicit responses andvariations because of provider education.10,11 Gill et al12

studied the interrater differences among emergencyphysicians in determining the GCS score and found thatthe agreement percentage for exact total GCS score was32%, whereas the agreement percentage for the motor


component was 72%. It is generally accepted that themotor component of the GCS is the most influential onewhen a patient’s severity of injury is assessed.

Goals of This InvestigationWe wished to compare the total GCS score less than or

equal to 13 with the GCS-m score less than 6 (“patientdoes not follow commands”) in predicting trauma-relatedoutcomes.

MATERIALS AND METHODSStudy Design and Setting

We retrospectively analyzed the prospectivelymaintained Pennsylvania Trauma System Foundation’sregistry, which included trauma patients admitted to thestate’s Level I, II, III, and IV trauma centers from 1999 to2013. The Pennsylvania Trauma System Foundationregistry captures all patients with a diagnosis of trauma whoare admitted to a Foundation-accredited Level I, II, III, orIV trauma center and patients presenting to the traumacenter dead on arrival. This includes all trauma transferadmissions and trauma deaths. Solitary hip fractures areexcluded. Patients do not need a minimum Injury SeverityScore (ISS) to be included into the registry. The majority ofaccredited trauma centers in Pennsylvania during this studywere Level I and II. Level IV trauma centers were firstrecognized in Pennsylvania in November 2013, and duringthe study (the last 2 months), there was only 1 Level IVaccredited center. Pennsylvania has an exclusive traumasystem, and the Pennsylvania Trauma System Foundationdata exclude patients who were treated only at facilities thatare not accredited trauma centers, although statewide EMStriage criteria and hospital referral patterns generally directseriously injured trauma patients to accredited traumacenters, either initially or by interfacility transfer.Additional description of the Pennsylvania Trauma SystemFoundation trauma registry can be found at http://www.ptsf.org/index.php/resources.

Quality assurance and improvement measures for thePennsylvania Trauma Outcome Study include internal datavalidation of the data entry system at each trauma center.Each trauma center’s data are abstracted locally andcollected in the Pennsylvania Trauma Outcome Study bytrained trauma registrars. Additional reviews are performedat the central site, including a random sampling program,which generates case reviews. Data are also validated againstobjective coding software, and foundation staff randomlyselect several cases from an institution and review themedical records at sites for consistency, accuracy, andcompleteness.

Volume -, no. - : - 2016

http://www.ptsf.org/index.php/resources

http://www.ptsf.org/index.php/resources

Table 1. Sensitivity, specificity, positive likelihood ratio, and negative likelihood ratio of total GCS versus GCS-m only scores in predictingoutcomes.

Outcome GCS Score £13 (95% CI) GCS-m Score <6 (95% CI) Relative Difference (95% CI)

ISS >15, %Sensitivity 31.3 (31.0 to 31.6) 28.0 (27.7 to 28.3) 3.3 (3.2 to 3.4)Specificity 91.3 (91.2 to 91.4) 92.8 (92.7 to 92.9) –1.5 (–1.6 to –1.5)LRþ 2.54 (2.52 to 2.56) 2.56 (2.54 to 2.59)LR– 0.53 (0.53 to 0.54) 0.51 (0.51 to 0.52)ISS >24, %Sensitivity 47.2 (46.7 to 47.6) 43.7 (43.3 to 44.2) 3.5 (3.3 to 3.6)Specificity 89.3 (89.2 to 89.4) 91.1 (91.0 to 91.2) –1.8 (–1.9 to –1.8)LRþ 4.97 (4.89 to 5.05) 5.11 (5.03 to 5.19)LR– 0.67 (0.66 to 0.67) 0.64 (0.64 to 0.65)Died, %Sensitivity 69.8 (69.2 to 70.4) 67.3 (66.7 to 67.9) 2.5 (2.2 to 2.7)Specificity 88.1 (88.0 to 88.2) 90.1 (90.0 to 90.2) –2.0 (–2.1 to –1.9)LRþ 12.88 (12.52 to 13.25) 13.551 (13.183 to 13.920)LR– 0.76 (0.75 to 0.76) 0.726 (0.722 to 0.730)ICU admission, %Sensitivity 27.3 (27.1 to 27.5) 23.9 (23.7 to 24.1) 3.39 (3.29 to 3.50)Specificity 91.6 (91.4 to 91.8) 92.7 (92.6 to 92.9) –1.15 (–1.24 to –1.06)LRþ 1.52 (1.52 to 1.53) 1.51 (1.50 to 1.51)LR– 0.37 (0.37 to 0.38) 0.37 (0.37 to 0.38)Intubation, %Sensitivity 83.7 (83.3 to 84.2) 81.3 (80.9 to 81.8) 2.4 (2.1 to 2.6)Specificity 90.0 (89.9 to 90.1) 92.0 (91.9 to 92.1) –2.0 (–2.1 to –1.9)LRþ 28.62 (27.72 to 29.53) 28.70 (27.85 to 29.55)LR– 0.62 (0.62 to 0.62) 0.57 (0.57 to 0.58)Trauma care need, %Sensitivity 28.2 (27.9 to 28.4) 25.1 (24.9 to 25.3) 3.0 (2.9 to 3.1)Specificity 93.7 (93.6 to 93.8) 95.0 (94.9 to 95.1) –1.3 (–1.4 to –1.3)LRþ 2.19 (2.17 to 2.20) 2.21 (2.19 to 2.22)LR– 0.38 (0.37 to 0.38) 0.35 (0.34 to 0.35)Surgery, %Sensitivity 33.5 (33.0 to 34.0) 30.5 (30.0 to 31.0) 3.0 (2.8 to 3.2)Specificity 86.5 (86.4 to 86.6) 88.4 (88.3 to 88.5) –1.9 (–2.0 to –1.9)LRþ 2.81 (2.75 to 2.87) 2.89 (2.83 to 2.96)LR– 0.87 (0.87 to 0.88) 0.86 (0.86 to 0.87)Craniotomy, %Sensitivity 51.4 (50.2 to 52.5) 46.5 (45.4 to 47.7) 4.9 (4.3 to 5.4)Specificity 85.9 (85.8 to 86.0) 87.8 (87.7 to 87.9) –2.0 (–2.0 to –1.9)LRþ 6.03 (5.76 to 6.30) 5.88 (5.61 to 6.12)LR– 0.94 (0.94 to 0.94) 0.94 (0.94 to 0.94)

CI, Confidence interval; LRþ, positive likelihood ratio; LR–, negative likelihood ratio.

Kupas, Melnychuk & Young Motor Component of the Glasgow Coma ScalePage 30

This study was approved by the Geisinger HealthSystem institutional review board, with the specificdetermination that this study met exempt criteria for fullinstitutional review board review. Data obtained from thePennsylvania Trauma Systems Foundation State Registrywere approved by Pennsylvania Trauma SystemFoundation.

Selection of ParticipantsThe database contained 393,877 adults aged 18 years and

older. The out-of-hospital total GCS score, out-of-hospitalGCS-m score, and ISS were obtained from each patientrecord. The primary outcome by which we compared totalGCS scores with GCS-m scores was ISS greater than 15.

Volume -, no. - : - 2016

Secondary outcomes were also collected from each patient,which included ISS greater than 24, death, ICU admission,need for craniotomy, any surgery (defined as intrathoracic,abdominal, vascular, or cranial surgery), intubation (at thescene or in the trauma care center), and a composite variable,trauma care need. Trauma care need was defined as ISSgreater than 15, ICU admission 24 hours or greater, need forsurgery, or death before discharge. Values of GCS and GCS-m scores, systolic blood pressure, and respiratory rate werecaptured at first report (at either the scene or trauma center).If these physiologic criteria were available in an out-of-hospital patient care report, then this was used. Otherwise,the first physiologic criteria obtained by the hospitalproviders on arrival to the trauma center were captured. In


Figure. AUC for GCS-m and GCS scores.


this study, the first reported physiologic data were availablefrom out-of-hospital records 48% of the time, and the datafrom initial hospital assessment were used for the remainder.After exclusion of patients with missing GCS score, GCS-mscore, respiratory rates, and systolic blood pressure; withtrauma year out of range; who do not fit into the traumaregistry inclusion and exclusion criteria according to thePennsylvania Trauma Outcome Study documentation; andwho were missing ISS, the analysis data set included370,392.

Methods of MeasurementWe used descriptive analyses for our data (Tables E1 and

E2, available online at http://www.annemergmed.com) andcontrasted the GCS-m score with the total GCS score withthe threshold less than or equal to 13 and with receiver


operating characteristic curves. Recognizing that our largesample would likely identify some small differences asstatistically significant, we defined a priori differences of lessthan 5% (ie, <0.05 for the area under a receiver operatingcharacteristic curve) as clinically unimportant regardless ofstatistical probability.

We performed sensitivity analyses excluding patientswho meet criteria for transport directly to a traumacenter because of other trauma triage criteria within theguidelines (including those with systolic blood pressure lessthan 90 mm Hg and respiratory rate less than 10 or greaterthan 29 breaths/min, as well as for anatomic reasonsincluding flail chest, skull fracture, paralysis, amputation,pelvic fractures, bone fractures, and penetrating injuries),generating a sample of 315,034. In a second sensitivityanalysis, we restricted the sampling to only patients with

Volume -, no. - : - 2016


Figure. Continued.


out-of-hospital reported total GCS score versus GCS-mscore.

Primary Data AnalysisWe used the SAS (version 9.4; SAS Institute, Inc., Cary,

NC) for data analysis.

RESULTSWe found that the differences between total GCS score

less than or equal to 13 and GCS-m score less than 6 were allbelow our prespecified 5% threshold for clinical importance,ranging from 2.5% to 4.9% for sensitivity and –1.2%to –2.0% for specificity (Table 1). All such differences had95% confidence intervals that did not overlap zero. Wefound similar results in our 2 sensitivity analyses.

Differences in areas under receiver operating characteristiccurves ranged from 0.014 to 0.048 (Figure, Tables 2 and E3

Volume -, no. - : - 2016

[available online at http://www.annemergmed.com]), all alsobelow our prespecified 0.05 threshold for clinicalimportance.

LIMITATIONSThis study is from a single state and may not be

representative elsewhere, although Pennsylvania includeslarge urban, suburban, and rural areas. A large proportionof the patients in the registry were victims of blunt trauma.In addition, for analysis the values of total GCS and GCS-m scores, systolic blood pressure and respiratory rate werecaptured at first report (either in the out-of-hospital settingor at the trauma center). A further limitation is thatapproximately half of the first reported GCS scores weredetermined by hospital providers rather than in the field byEMS providers, and the outcomes might have differed ifEMS providers had routinely provided GCS scores.



Table 2. Area under the receiver operating characteristic curve for each outcome.

Outcome

AUC (95% CI)

GCS Score £13 GCS-m Score <6 Relative Difference

ISS >15 0.648 (0.646–0.650) 0.606 (0.605–0.608) 0.042 (0.041–0.043)ISS >24 0.719 (0.716–0.721) 0.680 (0.677–0.682) 0.039 (0.038–0.041)Died 0.831 (0.828–0.834) 0.803 (0.800–0.806) 0.028 (0.026–0.030)ICU admission 0.625 (0.623–0.626) 0.583 (0.581–0.584) 0.042 (0.041–0.043)Intubation 0.904 (0.902–0.907) 0.884 (0.882–0.887) 0.020 (0.019–0.021)Trauma care need 0.641 (0.639–0.642) 0.603 (0.602–0.604) 0.038 (0.037–0.039)Surgery 0.612 (0.608–0.615) 0.597 (0.595–0.600) 0.014 (0.013–0.016)Craniotomy 0.724 (0.718–0.730) 0.676 (0.670–0.682) 0.048 (0.044–0.052)

AUC, Area under the receiver operation characteristic curve.


However, our sensitivity analysis of just this subgroupshowed similar results.

DISCUSSIONTotal GCS score has historically been an important

physiologic component of field triage used to predicttrauma outcomes. This relatively complicated 13-pointscale has shown inaccuracy among health care workers,however, putting its reliability in question.9

A simpler assessment of cerebral function is the binaryclinical determination of whether a patient “followscommands” (GCS-m score¼6) or does not. Althoughprevious studies have shown greater agreement amongemergency physicians assessing the GCS-m score comparedwith total GCS score, it is reasonable to posit that thestraightforward assessment of “following commands”would be as accurate as or more accurate than theassessment of all parts of the GCS-m score. This simplebinary assessment of whether a patient “follows commands”is practical and appealing when one considers ease ofeducation and use by all levels of EMS providers when theymake field triage determinations.

We found that the differences between total GCS scoreless than or equal to 13 and GCS-m score less than 6 werebelow our prespecified 5% threshold for clinical importance,despite statistically significant associations that predictablyresulted from our extremely large sample size. Our data thusconfirm the findings of previous studies that our simplerdecision point is just as predictive of trauma outcomes as thefull GCS.13,14 A simplified field triage score for battlefieldcasualties, which includes the GCS-m, has shown promisingresults for use as a practical instrument in the combat zone.15

Additionally, the use of motor response in children after theysustain a traumatic head injury has shown to predict long-term outcome, as well as the full GCS score, with betterinterobserver agreement.16 A 2012 study identified GCS-mscore as part of a prognostic model for predicting mortality at


30days andunfavorable outcome at 6months after traumaticbrain injury.17 A retrospective study found that if patientshad a GCS-m score less than 6 and a systolic blood pressureless than 90 mm Hg, 95% of them needed a lifesavingintervention.18 A similar study showed that GCS-m andGCS verbal scale scores, alongwith pulse character, predictedneed of out-of-hospital lifesaving intervention.19 Theperformance of GCS-m in previous studies and in this study,as well as ease of using GCS-m in the out-of-hospital setting,make a strong argument for simplifying the nationalGuidelines for Field Triage by changing to the use of GCS-mscore less than 6 or “patient does not follow commands”when making out-of-hospital trauma triage decisions.

In conclusion, during trauma triage a simple binarydecision point of GCS-m score less than 6, or a patientwho “does not follow commands,” predicts serious injurysimilarly to the more complicated calculation of total GCSscore. For all outcomes, the relative differences inspecificity, sensitivity, and area under the receiveroperation characteristic curve between GCS-m score andtotal GCS score were clinically unimportant; therefore, werecommend our simpler binary assessment as areplacement for the total GCS score for field traumatriage.

Supervising editor: Steven M. Green, MD

Author affiliations: From the Department of Emergency Medicine(Kupas, Melnychuk) and Center for Health Research (Young),Geisinger Health System, Danville, PA.

Author contributions: DFK and EMM conceived and designed thestudy and obtained institutional review board approval and waiver.AJY obtained the data and provided statistical analysis. All authorsanalyzed and interpreted the data, drafted the article, andcontributed substantially to its revision. DFK takes responsibilityfor the paper as a whole.

Funding and support: By Annals policy, all authors are required todisclose any and all commercial, financial, and other relationships

Volume -, no. - : - 2016


in any way related to the subject of this article as per ICMJE conflictof interest guidelines (see www.icmje.org). The authors have statedthat no such relationships exist and provided the following details:Dr. Kupas is a member of the board of directors of thePennsylvania Trauma Systems Foundation.

Publication dates: Received for publication December 4, 2015.Revisions received June 1, 2016, and June 3, 2016. Accepted forpublication June 13, 2016.

REFERENCES1. Sasser SM, Hunt RC, Faul M, et al. Guidelines for field triage of injured

patients: 2011 report.Morb Mortal Wkly Rep. 2012;61:1-20. Availableat: http://www.cdc.gov/mmwr/preview/mmwrhtml/rr6101a1.htm?s_cid¼rr6101a1_w. Accessed June 1, 2012.

2. Prehospital Trauma Life Support Committee of the NationalAssociation of Emergency Medical Technicians; Committee on Traumaof the American College of Surgeons. PHTLS: Prehospital Trauma LifeSupport. 6th ed. St. Louis, MO: Mosby; 2007.

3. Caterino JM, Raubenolt A. The prehospital simplified motor score is asaccurate as the prehospital Glasgow Coma Scale: analysis of astatewide trauma registry. Emerg Med J. 2012;29:492-496.

4. Eftekhar B, Zarei MR, Ghodsi M, et al. Comparing logistic modelsbased on modified GCS motor component with other prognostic toolsin prediction of mortality: results of study in 7226 trauma patients.Injury. 2005;36:900-904.

5. Gill M, Steele R, Windemuth R, et al. A comparison of five simplifiedscales to the out-of-hospital Glasgow Coma Scale for the prediction oftraumatic brain injury outcomes. Acad Emerg Med. 2006;13:968-973.

6. Haukoos JS, Gill MR, Rabon RE, et al. Validation of the simplified motorscore for the prediction of brain injury outcomes after trauma. AnnEmerg Med. 2007;50:18-24.

7. Thompson DO, Hurtado TR, Liao MM, et al. Validation of the simplifiedmotor score in the out-of-hospital setting for the prediction ofoutcomes after traumatic brain injury. Ann Emerg Med. 2011;58:417-425.

Volume -, no. - : - 2016

8. Hoffmann M, Lefering R, Rueger JM, et al. Pupil evaluation in additionGlasgow Coma Scale components in prediction of traumatic braininjury and mortality. Br J Surg. 2012;99:122-130.

9. Namiki J, Yamazaki M, Funabiki T, et al. Inaccuracy and misjudgedfactors of Glasgow Coma Scale scores when assessed byinexperienced physicians. Clin Neurol Neurosurg. 2011;113:393-398.

10. Reith FC, Brennan PM, Maas AI, et al. Lack of standardization in theuse of the Glasgow Coma Scale. Results of international surveys.J Neurotrauma. 2016;33:89-94.

11. Bledsoe BE, Casey MJ, Feldman J, et al. Glasgow Coma Scale scoringis often inaccurate. Prehosp Disaster Med. 2015;30:46-53.

12. Gill MR, Reiley DG, Green SM. Interrater reliability of Glasgow ComaScale scores in the emergency department. Ann Emerg Med.2004;43:215-223.

13. Beskind DL, Stolz U, Gross A, et al. A comparison of the prehospitalmotor component of the Glasgow Coma Scale (mGCS) to theprehospital total GCS (tGCS) as a prehospital risk adjustment measurefor trauma patients. Prehosp Emerg Care. 2014;18:68-75.

14. Brown JB, Forsythe RM, Stassen NA, et al. Evidence-basedimprovement of the National Trauma Triage Protocol: the GlasgowComa Scale versus Glasgow Coma Scale motor subscale. J TraumaAcute Care Surg. 2014;77:95-102.

15. Eastridge BJ, Butler F, Wade CE, et al. Field triage score (FTS) inbattlefield casualties: validation of a novel triage technique in acombat environment. Am J Surg. 2010;200:724-727.

16. Fortune PM, Shann F. The motor response to stimulation predictsoutcome as well as the full Glasgow Coma Scale in children withsevere head injury. Pediatr Crit Care Med. 2010;11:339-342.

17. Yuan F, Ding J, Chen H, et al. Predicting outcomes after traumatic braininjury: the development and validation of prognostic models based onadmission characteristics. J Trauma Acute Care Surg. 2012;73:137-145.

18. Holcomb JB, Niles SE, Miller CC, et al. Prehospital physiologic dataand lifesaving interventions in trauma patients. Mil Med. 2005;170:7-13.

19. Holcomb JB, Salinas J, McManus JM, et al. Manual vital signs reliablypredict need for life-saving interventions in trauma patients. J Trauma.2005;59:821-829.


http://www.icmje.org/

http://www.cdc.gov/mmwr/preview/mmwrhtml/rr6101a1.htm?s_cid=rr6101a1_w






























































Spatsuip

scjt

RDUJCIL

d

Fdich

1

2

3

4

5

6

7

CorrespondencePage 35

Background, organization, and clinical survey. JAMA.1977;237:882-886.

2. Hankey GJ, Warlow CP. Transient Ischaemic Attacks of the Brainand Eye. London, England: WB Saunders; 1994.

3. Levy DE. How transient are transient ischemic attacks? Neurology.1988;38:674-677.

4. National Institute of Neurological Disorders and Stroke rt-PA StrokeStudy Group. Tissue plasminogen activator for acute ischemicstroke. N Engl J Med. 1995;333:1581-1587.

5. Crisostomo RA, Garcia MM, Tong DC. Detection of diffusion-weighted MRI abnormalities in patients with transient ischemicattack: correlation with clinical characteristics. Stroke.2003;34:932-937.

Lessons From the Glasgow Coma Scale

To the Editor:Dr. Green’s comments on the persistence of the Glasgow

Coma Scale (GCS)1 were right on the mark but have muchbroader implications than he observed. Nothing persists insocial systems unless it benefits someone, somewhere, somehow,so when a seemingly useless ritual such as the GCS lasts foryears, especially in the face of sound arguments against it, weshould look for who it benefits and how. As Dr. Greenobserved, the persistence of the GCS is due not to its clinicalutility but rather its psychological comfort. It gives theappearance of reducing uncertainty, creating order, andrationalizing clinical practice. That it does none of these isimmaterial to the benefits it provides in reducing cognitive andaffective stress. There are undoubtedly social benefits as well, inthe form of providing seemingly objective justification foractions taken or not taken.

I would argue that the same factors are playing out in manyother areas of health care, eg, the proliferation of decision rules,the desire for guidelines, the quest for standardization and theaversion to variation or heterogeneity, the faith in “evidence-based medicine,” the yearning for quantitative measurement,the fascination with templates and checklists, and themagical thinking about information technology. Althoughthere are sometimes beneficial aspects to these efforts, muchof the impetus behind them stems from their ability toincrease the appearance of rationality and order, to make theintractable seem tractable and the ineffable, effable.

This might not matter if there were not potential drawbackshidden in these ideas. There are 3 issues here. First, they canlead to miscalibration, an unwarranted sense of confidence in
one’s understanding of a situation, an illusion of control.2

econd, there is abundant evidence from multiple fields thateople will accept guidance from external sources or decisionids that is inferior to that which they could have producedhemselves, if unaided.3-6 Finally, these notions can produce aort of socially agreed-on agnosia, a total dependence on rationalnderstanding in which everything must be decomposed before

t can be understood, and that which cannot be decomposed isresumed not important.7

Sorting through the complex motivations lying behind thesechemes to organize and rationalize care so we can identify thehanges that might actually improve clinical work, rather thanust make us feel better about it, should be an important task forhose who wish to make emergency care better.

obert L. Wears, MD, MSepartment of Emergency Medicineniversity of Florida

acksonville, FLlinical Safety Research Unit

mperial Collegeondon, UK

oi:10.1016/j.annemergmed.2011.11.025

unding and support: By Annals policy, all authors are required toisclose any and all commercial, financial, and other relationships

n any way related to the subject of this article as per ICMJEonflict of interest guidelines (see www.icmje.org). The authoras stated that no such relationships exist.

. Green SM. Cheerio, laddie! Bidding farewell to the Glasgow ComaScale. Ann Emerg Med. 2011;58:427-430.

. Langer EJ. The illusion of control. J Pers Soc Psychol. 1975;32:311-328.

. Klein G, Moon B, Hoffman RR. Making sense of sensemaking 1:alternative perspectives. IEEE Intelligent Syst. 2006;21:70-73.

. Layton C, Smith PJ, McCoy CE. Design of a cooperative problem-solving system for en-route flight planning: an empirical evaluation.Hum Factors. 1994;36:94-119.

. Park S, Rothrock L. Systematic analysis of framing bias in missiledefense: implications toward visualization design. Eur J Oper Res.2007;182:1383-1398.

. Smith GF. Representational effects on the solving of anunstructured decision problem. IEEE Trans Syst Man Cybern.1989;19:1083-1090.

. Dreyfuss HL, Dreyfuss SE. Mind Over Machine: The Power ofHuman Intuitive Expertise in the Era of the Computer. New York,
NY: Free Press; 1986.
Volume , . : April

Original text:

givenname

Original text:

surname

http://www.icmje.org

Documents

Emergency Medicine Journal Club - lluem.org