T

CH

a

ARRA

KIFNNN

1

dotCdhat“c

3

0d

Accident Analysis and Prevention 43 (2011) 724–731

Contents lists available at ScienceDirect

Accident Analysis and Prevention

journa l homepage: www.e lsev ier .com/ locate /aap

oo many or too few unintentional firearm deaths in official U.S. mortality data?

atherine Barber ∗, David Hemenwayarvard Injury Control Research Center, Harvard School of Public Health, Boston, MA, USA

r t i c l e i n f o

rticle history:eceived 5 August 2010eceived in revised form 12 October 2010ccepted 18 October 2010

eywords:njury surveillanceirearm accidentational Vital Statistics Systemational Violent Death Reporting Systemosology

a b s t r a c t

We examined the accuracy of data on unintentional firearm fatalities in the United States. We beganwith data from the National Violent Death Reporting System (NVDRS) and examined every firearm deathreported as an accident by any NVDRS data source – the NVDRS abstracter, the State Vital Statistics Reg-istry (i.e., the ICD-10 Underlying Cause of Death code and manner of death from the death certificate), themedical examiner or coroner report, and the police Supplementary Homicide Report. After carefully read-ing the information from all sources, we then classified each case as either unintentional or intentionalusing NVDRS definitions.

Comparing our classification with that of the NVDRS abstracter, we conclude that NVDRS data accu-rately report unintentional firearm deaths (sensitivity 98%; positive predictive value [PPV] 99%). We thencompared our classification with that of the State Vital Statistics Registry based on the ICD-10 UnderlyingCause of Death code and found great inaccuracy (sensitivity 62%; PPV 58%). Thirty-eight percent of truecases were missed and 42% of reported cases were false positives. As has been previously reported inthe literature, over half of unintentional shootings that were inflicted by another person were classifiedas homicides, not accidents. This is a particular problem for the data on young victims. What was unex-pected, however, was the opposite problem: we found many unambiguous suicides and homicides thatwere reported as accidents. We believe this is due to the ICD-10 coding practice of assigning injury deathsto the “accident” category when manner of death is “pending” or left blank.

Finally, at the state-level, we compared our overall results for each state and year with the officialNational Vital Statistics System count (reported on CDC’s WONDER website). We found evidence of evengreater over-reporting of unintentional firearm deaths there.

In answer to the question, “Are there too many or too few unintentional firearm deaths in officialmortality data?” the best answer is, “Both.” Many true accidents are missed, while many suicides andhomicides are mistakenly reported as accidents. By contrast, the NVDRS applies a case definition for

ths w

unintentional firearm dea

. Introduction

Efforts to improve public safety depend on reliable mortalityata to track whether particular causes of death are rising, falling,r static. Reports have indicated a drop in deaths due to unin-entional firearm injury (Frattaroli et al., 2002; National Researchouncil, 2005). However the accuracy of mortality data on acci-ental firearm death has been questioned. A number of studiesave concluded that when one person shoots another unintention-

lly – for example, a hunter mistakenly shoots another hunter –he manner of death is classified as “homicide” more often thanaccident” (Barber et al., 1998, 2002; Schaechter et al., 2003). Offi-ial data therefore appear to under-report firearm accidents that

∗ Corresponding author at: Harvard School of Public Health, 677 Huntington Ave.,rd floor, Boston, MA 02115, USA. Tel.: +1 617 432 1143; fax: +1 617 432 3699.

E-mail address: cbarber@hsph.harvard.edu (C. Barber).

001-4575/$ – see front matter © 2010 Elsevier Ltd. All rights reserved.oi:10.1016/j.aap.2010.10.018

ith consistency and accuracy.© 2010 Elsevier Ltd. All rights reserved.

are other-inflicted (vs. self-inflicted). However, a recent reportfrom Oklahoma indicated that the Vital Statistics System thereover-reported accidental firearm deaths (Comstock et al., 2005).Researchers compared death certificate data with medical exam-iner reports and found that among 20 cases of unintentional firearmdeaths identified by Oklahoma vital statistics data, only 9 weretrue unintentional firearm deaths; the remainder were false posi-tives.

Do official mortality data over- or under-estimate unintentionalfirearm deaths? To answer this question, we used a relativelynew surveillance system, the Centers for Disease Control and Pre-vention’s (CDC) National Violent Death Reporting System, andcompared results from that system with official National Vital

Statistics System mortality data. The purpose of our investi-gation was to evaluate the extent of false positives and falsenegatives for unintentional firearm injuries in both systems andto examine the type of incidents most likely to be misclassi-fied.

C. Barber, D. Hemenway / Accident Analysis and Prevention 43 (2011) 724–731 725

Funeral Director Completes the victim demographics portion

of death certificate

State Vital Statistics Registry Death certificate sent to local

registrar and state vital statistics registry for assembly, data entry

Coroner /Medical Examiner Completes the cause and manner of death portion of

death certificate

ICD-10 Cause of Death

Codes Assigned at State

Vital Statistics Registry or NCHS

(depending on state)

State Vital Statistics Registry

Death certificate data,

including: * ICD-10 cause of death codes

* Manner of death

* Demo-graphics

Coroner/ Medical

Examiner • Death

investigation report

• Cause and manner of death (text) • Autopsy and toxi-cology results

Law Enforcement

• Case investigation information • Crime lab

ballistics data • FBI-UCR data

(NIBRS; Supplementary

Homicide Report)

State NVDRS Abstractor • Links and codes data

• Writes incident narratives • Assigns manner of death

CDC National Center for Health Statistics (NCHS)

Assembles data from states and disseminates

Optional Sources • Child Fatality

Review • ATF Reports

• Other sources

Personal identifiers stripped

Na�onal Vital Sta�s�cs System Death Cer�ficate-based Mortality Data

Na�onal Violent Death Repor�ng System (NVDRS)

CDC NVDRS Assembles data from states and

disseminates

ted de

2

2

bloteiactllD(tdOthd

ofgecoa

system.

Fig. 1. Sources of data on firearm-rela

. Materials

.1. Data sources

The National Violent Death Reporting System (NVDRS) is a state-ased surveillance system that links data from the death certificate,

aw enforcement records, and coroner/medical examiner recordsn deaths due to suicide, homicide, legal intervention, uninten-ional firearm injury, and injuries of unknown intent (Paulozzit al., 2004; Hemenway et al., 2009). (See Fig. 1.) The records arencident-based and include information on the persons (victimsnd suspects), weapons, and circumstances involved. In addition tooded data, the abstracter writes incident narratives summarizinghe findings from the coroner/medical examiner’s records and theaw enforcement records. Data are collected and linked at the stateevel, stripped of personal identifiers, and forwarded to the CDC.ata for this study are from the NVDRS’s Restricted Access Dataset

RAD) (obtained in January 2009) for all states participating duringhe period 2003–2006; these include seven states that providedata all four years: Alaska, Maryland, Massachusetts, New Jersey,regon, South Carolina, Virginia; six states that provided data for

hree years, 2004–2006: Colorado, Georgia, North Carolina, Okla-oma, Rhode Island, and Wisconsin; and three states that providedata for two years, 2005–2006: Kentucky, New Mexico, and Utah.

The National Vital Statistics System is the nation’s official sourcef mortality data. It is assembled by the CDC’s National Centeror Health Statistics (NCHS). Firearm-related deaths are investi-ated by local law enforcement and by the coroner or medical

xaminer in the jurisdiction in which the injury was inflicted. Theoroner/medical examiner fills out the “cause of death” sectionf the death certificate, identifying the manner of death (natural,ccident, suicide, homicide, pending, or undetermined), and sup-

aths in two U.S. surveillance systems.

plies brief phrases to indicate the conditions leading to death (e.g.,“traumatic pneumothorax due to shotgun wound”). In most juris-dictions, the certificate is sent to the State Vital Statistics Registry,which assembles data from local jurisdictions and sends it to theNational Center for Health Statistics. The cause of death statementsare assigned codes using the International Statistical Classificationof Disease and Related Health Problems, Tenth Revision (ICD-10)(World Health Organization, 1992) with the assistance of a familyof software programs provided by the National Center for HealthStatistics. The coding may be done either by the state, by NCHS, ora combination, depending on each state’s arrangement with NCHS.One cause of death is identified as the underlying cause of death; forinjury deaths, the ICD-10 underlying cause is an “external cause ofdeath” code that characterizes both the mechanism and the intentof the injury (e.g., “X73-intentional self-harm by rifle, shotgun andlarger firearm discharge”). Data from the National Vital StatisticsSystem were obtained in aggregate from the WONDER websitehttp://wonder.cdc.gov/mortSQL.html.

Use of the data in this study was approved by the Harvard Schoolof Public Health’s Institutional Review Board.

2.2. Case definition

Deaths from NVDRS and Vital Statistics were based on the dece-dent’s state of residence and on the year of death or, in caseswhen year of actual death is unknown or in the past (e.g., dis-covery of skeletal remains), year of entry into the mortality data

Vital Statistics – unintentional firearm deaths in the VitalStatistics database were defined as deaths to residents of statesparticipating in the NVDRS that were coded as W32 (accidentaldischarge of handgun), W33 (accidental discharge of rifle or shot-

7 nalysis

gT

dawtPhwstpuiicri

pidmddctbfn

3

3

fidt

•

•

•

odornsd

2tem(du

26 C. Barber, D. Hemenway / Accident A

un), or W34 (accidental discharge of other or unspecified firearm).he latter category includes both firearms and non-powder guns.

NVDRS – NVDRS defines unintentional firearm injuries as “Aeath resulting from a penetrating injury or gunshot wound fromweapon that uses a powder charge to fire a projectile when thereas a preponderance of evidence that the shooting was not inten-

ionally directed at the victim” (Centers for Disease Control andrevention, 2003). Examples include a hunter mistaking anotherunter for an animal, a man shooting himself or another personith a gun he believed was unloaded, and a child age 5 or under

hooting a person (even if “intentionally”). Firearm-related deathshat do not meet the case definition are those that occur to anothererson during the commission of a violent crime or when a gun issed to intentionally harm, control or frighten another person, even

f the person who dies was an unintended target. So, for example,f during a robbery a passer-by is shot unintentionally, the death islassified as a homicide, not an accident. Russian roulette (firing aevolver at oneself after spinning a cylinder that the shooter knowss partially loaded) is classified as suicide in NVDRS.

NVDRS abstracters were trained (by the first author and CDCersonnel) in 2003–2005 to apply the NVDRS case definitions

n assigning type (manner) of death. The abstracter’s type ofeath should agree with the coroner or medical examiner’s (C/ME)anner of death except in cases in which the C/ME applied a

ifferent case definition (e.g., calls an unintentional hunting acci-ent a homicide) or when new information from an official sourcelearly establishes a different manner of death. The abstracter israined not to reinterpret evidence or “second guess” the C/ME,ut rather to apply a uniform set of case definitions. When theacts of a case are unclear, the abstracter defaults to the C/ME man-er.

. Methods

.1. Case review

The first author reviewed the incident narratives for each NVDRSrearm-related death that met any of the following criteria toetermine whether it met the NVDRS case definition for an unin-entional firearm incident:

The manner of death was classified as an accident by the NVDRSabstracter, death certificate, or coroner/medical examiner report;The Underlying Cause of Death code was W32, W33, or W34(firearm accident) or Y86 (sequelae of non-transport accidentalinjury);The Supplementary Homicide Report classified the death as“manslaughter by negligence.”

A total of 549 potential firearm accidents involving residentsf an NVDRS state were reviewed out of a total of 24,709 firearmeaths. When the first author and the NVDRS abstracter disagreedn whether the case qualified as unintentional, the second authoreviewed the case to resolve disagreements. The authors’ desig-ation of unintentional firearm death was considered the goldtandard. When the facts of a case were unclear, the study authorsefaulted to the abstracter’s manner.

It was outside the scope of the paper to review the more than4,000 firearm suicides and homicides in the database that neitherhe abstracter nor any data source coded as unintentional. How-

ver, because gun accidents that are inflicted by another personay be coded as homicides, we reviewed a random 5% sample

n = 552) of firearm cases in which all indicators for manner ofeath agreed on homicide. Among these cases, only one was a likelynintentional shooting (a teenager was shot by another teenager

and Prevention 43 (2011) 724–731

who thought the gun was unloaded; the shooter was charged withnegligent homicide in the case).

3.1.1. Review of context in which accident occurredThe NVDRS abstracter indicated whether an unintentional

firearm injury was self-inflicted or other-inflicted and codedthe context in which a firearm accident occurred, coding asmany as applied from among the categories of hunting, play-ing/fooling around, self-defense, target shooting, celebratoryfiring, loading/unloading, cleaning, showing to others, and other.In 20% of cases no context was selected. We reviewed nar-ratives for all cases to identify whether contexts could befurther characterized; unknowns were reduced to 12%. To ana-lyze whether the context in which a firearm accident occursis associated with whether it is correctly classified as an acci-dent, we assigned contexts to a collapsed, mutually exclusivelist using the following hierarchy: hunting, playing, clean-ing/loading/unloading/repairing, other, and unknown (for detailson circumstances see Hemenway 2009 and the NVDRS codingmanual available at http://www.cdc.gov/ncipc/pub-res/nvdrs-coding/VS2/default.htm).

3.2. Analysis

For NVDRS data, we used the unintentional firearm deathsidentified by the authors as the gold standard. We calculated thesensitivity and positive predictive value (PPV) of the abstracter’smanner of death and the coded data supplied by the State VitalStatistics Registry (i.e., the ICD-10 Underlying Cause of Death codeand the manner of death). Sensitivity is defined as the proportionof gold standard cases that are captured by the data sources. PPVis defined as the proportion of cases reported by the data source aspositive that are true positives.

We also compared NVDRS results with National Vital StatisticsSystem data for each state and year. Because NVDRS data do nothave personal identifiers, and we did not obtain individual-levelVital Statistics data from NCHS, we did not attempt to link the twodatasets. We therefore could not calculate sensitivity and PPV forVital Statistics. As an alternate performance metric, we calculatedwhat we call the “overall percentage error.” This is the sum of theabsolute value of the difference between the gold standard andVital Statistics for each state in each year, divided by the overalltotal number of gold standard cases. This metric is not bounded by100%.

We examined the rate at which fatal firearm mishaps occuracross demographic groups based on National Vital Statistics Sys-tem data and NVDRS data. Crude incidence rates per 100,000population were supplied by the WONDER website; incidencerates based on the NVDRS cases were calculated using state/yearpopulation provided by WONDER. Counts and incidence rates bymetropolitan status were based on the decedent’s county of resi-dence using the NCHS Urban-Rural Classification Scheme (Ingramand Franco, 2006). We used the chi square test to examine whetherthe observed distribution of victim demographic characteristics bydata source varied significantly from the expected distribution.

Among the gold standard NVDRS cases, we examined the extentto which the ICD-10 Underlying Cause of Death code (“ICD-10code”) that was assigned to the case by the State Vital Statistics

Registry correctly classified the death as a firearm accident, usingbivariate logistic regression to identify demographic and incidentcharacteristics associated with higher rates of misclassification.

Throughout, we use the phrases “unintentional injury” and“accident” interchangeably.

C. Barber, D. Hemenway / Accident Analysis and Prevention 43 (2011) 724–731 727

Table 1Unintentional firearm deaths among residents of NVDRS states, 2003–2006, by data source.

From NVDRS Dataset (549 cases reviewed)

State Vital Statistics Registry Death Certificate Data

National Vital Statistics System Gold standard NVDRS abstracter ICD-10 Underlying Cause ofDeath code

Manner of death

Deaths coded as firearm accidents 489 381 378 404 265Percent over/under gold standard 28% ref −1% 6% −30%Confirmed accidents (true positives) na ref 374 236 242Missed accidents (false negatives) na ref 7 145 139PPV (true positives) na ref 99% 58% 91%Sensitivity na ref 98% 62% 64%

Decedent’s state of residenceAlaska (2003–2006) 9 24 24 8 10Colorado (2004–2006) 17 21 18 14 14Georgia (2004–2006) 60 50 50 46 37Kentucky (2005–2006) 39 34 34 36 32Maryland (2003–2006) 19 11 11 14 4Massachusetts (2003–2006) 9 5 5 6 4New Jersey (2003–2006) 39 9 9 16 2New Mexico (2005–2006) 9 10 10 4 4North Carolina (2004–2006) 69 50 50 73 43Oklahoma (2004–2006) 35 29 29 32 13Oregon (2003–2006) 22 16 15 15 14Rhode Island (2004–2006) 2 2 2 1 2South Carolina (2003–2006) 92 43 45 62 34Utah (2005–2006) 3 2 2 2 2Virginia (2003–2006) 47 59 60 61 37Wisconsin (2004–2006) 18 16 14 14 13Overall percentage error 50.9% – 4.5% 40.2% 30.4%

Notes. The NVDRS dataset included 549 firearm deaths in which at least one data source indicated that the case was an accident. Definitions: National Vital Statistics – firearmaccidents according to official mortality data from CDC’s WONDER website, National Center for Health Statistics. National Vital Statistics data were obtained in aggregateand could not be linked at the individual level; therefore cases could not be confirmed as accidents nor could sensitivity or PPV be calculated. Gold standard – NVDRS firearmdeaths confirmed as accidents after review by study authors; no missings. Abstracter – firearm accidents according to code assigned by state NVDRS abstracter; no missings.ICD-10 Cause of Death code – firearm accident according to Underlying Cause of Death code from State Vital Statistics Registry’s coded death certificate data; missing = 10.M Registry’s coded death certificate data; missing = 12. Confirmed cases – cases reported byt ives). Missed cases – true cases missed by the data source (false negatives). PPV (positivep percent of “true” cases (gold standard) that were identified by source. Overall percentagee r each state-by-year subtotal divided by 381 (the total gold standard count).

4

wdfiawVtodPdcTs

t2d

dtoSadV

anner of death – accident according to manner of death from State Vital Statisticshe data source that were confirmed as firearm accidents by the authors (true positredictive value) – percent of reported cases that were confirmed cases. Sensitivity –rror – sum of the absolute difference between the source and the gold standard fo

. Results

A total of 549 firearm cases were reviewed. Among these, 381ere identified by the study authors as unintentional firearmeaths (Table 1). The NVDRS abstracter identified 378 unintentionalrearm deaths, 374 of which coincided with those identified by theuthors. Sensitivity (98%) and PPV (99%) of the abstracter codingere very high. Based on the ICD-10 code supplied by the Stateital Statistics Registries, 404 cases received codes in the acciden-

al firearm injury range. As indicated in Fig. 2, among these 404,nly 236 coincided with gold standard cases, and 145 gold stan-ard cases were missed, leading to both low sensitivity (62%) andPV (58%) for the ICD-10 coding. The death certificate manner ofeath field identified 265 firearm accidents, 242 of which wereonfirmed as accidents by the authors; 139 cases were missed.herefore, while PPV was relatively high for manner of death (91%),ensitivity (64%) was low.

During the same period and in the same states, the Vital Statis-ics System reported 489 unintentional firearm deaths, a figure8% higher than that identified using the gold standard NVDRSata.

Table 2 compares the demographic characteristics of gun acci-ent victims identified via Vital Statistics with those identified viahe gold standard NVDRS cases. The NVDRS victims were younger

n average, with a mean age of 29.8, compared with 37.1 for Vitaltatistics. Over half (52%) of the NVDRS decedents were under thege of 25, compared with a third (34%) of the Vital Statistics dece-ents. A greater number of firearm accidents were identified byital Statistics in both metropolitan and non-metropolitan coun-

Fig. 2. Accuracy of State Vital Statistics Registry reporting of deaths due to unin-tentional firearm injury based on ICD-10 Underlying Cause of Death codes.(Source:National Violent Death Reporting System, participating states, 2003–2006).

ties, for both males and females, and for both black and whitenon-Hispanics, but not for other racial/ethnic groups. The larger

numbers reported by Vital Statistics on average did not hold acrossstates; in some states (notably Alaska), Vital Statistics reportedfewer deaths, in others Vital Statistics reported considerably more(notably New Jersey, North Carolina, and South Carolina), and in

728 C. Barber, D. Hemenway / Accident Analysis and Prevention 43 (2011) 724–731

Table 2Demographic characteristics of victims of unintentional firearm injury death by data source, NVDRS states, 2003–1006.

National Vital Statistics System NVDRS Gold standard

# % Rate/100 k # % Rate/100 k

All decedents 489 100.0 0.19 381 100.0 0.15Mean age 37.1 29.7

Age group** 0–14 years 48 9.8 0.09 73 19.2 0.1415–24 years 119 24.3 0.33 126 33.1 0.3525–34 years 88 18.0 0.25 56 14.7 0.1635–44 years 71 14.5 0.18 41 10.8 0.1045–54 years 68 13.9 0.18 38 10.0 0.1055+ years 95 19.4 0.16 45 11.8 0.08Unknown – – – 2 0.5 –

Sex Female 58 11.9 0.04 35 9.2 0.03Male 431 88.1 0.34 346 90.8 0.27

Race/ethnicity Amer. Indian, NH 10 2.0 0.34 18 4.7 0.61Asian, NH 1 0.2 0.01 3 0.8 0.03Black, NH 100 20.4 0.24 70 18.4 0.17White, NH 360 73.6 0.19 271 71.1 0.15Hispanic 18 3.7 0.08 19 5.0 0.08

County of residence Metropolitan 325 66.5 0.15 247 64.8 0.11Non-metropolitan 164 33.5 0.38 131 34.4 0.30Unknown – – – 3 0.8 –

N cludes( isconH g souf

ficbe(

TPi

s

t

otes. Case definition is based on victim’s state of residence and year of death and in2003–2006); Colorado, Georgia, North Carolina, Oklahoma, Rhode Island, and Wispanic. Asterisks refer to the probability that the observed frequency by reportin

requency in a chi square test. *p < .05, **p < .01, ***p < .001.

ve states the numbers were roughly comparable. The overall per-

entage error for Vital Statistics (the sum of the absolute differenceetween gold standard and Vital Statistics cases for each state inach year as a percentage of total gold standard cases) was 51%Table 1).

able 3roportion of unintentional firearm deaths that received an ICD-10 Underlying Cause oncident.

Gold standard firearm accident

#

Decedent characteristicsAll decedentsc 371c

0–9 years 3310–19 years 10620–29 years 8830–39 years 3940–49 years 4250+ years 61Age unknown 2

White, non-Hispanic 267Black, non-Hispanic 67All other 37

Male 337Female 34

Incident characteristicsShot by. . .

Other person 169Self 170Unknown 32

ContextHunting 69Playing/fooling around 134Cleaning/loading/fixing 35Other 78Unspecified 41

a Among the 135 true cases that were NOT coded as firearm accidents, the distributiouicide – 12%, undetermined – 10%, sequelae of accidental injury (mechanism unspecified

b Probability that the proportion of cases that received an ICD-10 code in the firearm ache reference group (“ref”) in bivariate logistic regression. *p < .05. **p < .01. ***p < .001.

c Excludes 10 cases for which the ICD-10 Code supplied by the State Vital Statistics Reg

Alaska, Maryland, Massachusetts, New Jersey, Oregon, South Carolina, and Virginiasin (2004–2006); and Kentucky, New Mexico, and Utah (2005–2006). NH = Non-rce for the listed demographic categories differed significantly from the expected

Table 3 shows those cases that were identified as true

accidents by the study authors and indicates whether thecase was captured as such by the ICD-10 Underlying Causeof Death code provided by the State Vital Statistics Registry.True cases that were most likely to be missed were those

f Death code indicating firearm accident, by characteristics of the decedent and

Firearm accident according to ICD-10 Underlying Cause of Death code?

Yes Noa % Yes pb

236 135 63.620 13 60.648 58 45.3 ***54 34 61.4 *31 8 79.534 8 81.047 14 77.0 ref

2 0 –

186 81 69.7 ref36 31 53.7 *14 23 37.8 ***

218 119 64.7 ref18 16 52.9

70 99 41.4 ***142 28 83.5 ref24 8 75.0

53 16 76.8 ref62 72 46.3 ***23 12 65.767 11 85.931 10 75.6

n of ICD-10 underlying cause of death categories was as follows: homicide – 68%,) – 8%, natural – 1%.

cident range (W32–W34)varied significantly in the listed subgroup compared with

istry was missing in the NVDRS data.

nalysis

aagwdag

dassd

5

eaSaiRtciRddTmoaIduoarc

opwofiWca(g

ItcwcfiTI

Drid

C. Barber, D. Hemenway / Accident A

ges 10–19, 47% of whom were classified as homicide. Victimsges 30 and over were least likely to be misclassified. Otherroups that were significantly more likely to be misclassifiedere racial/ethnic groups other than white non-Hispanics, acci-ents that were inflicted by another person (vs. self-inflicted),nd accidents that occurred in the context of playing with aun.

Among the gold standard cases, 64% were classified as acci-ents by the ICD-10 code, 25% as homicide, 4% as suicide, and 7%s other. The “other” category included “undetermined” (n = 14),equelae of an earlier accident (a coding category that does notpecify the mechanism that caused the injury) (n = 11), and naturaleath (n = 2).

. Discussion

We began with data from the NVDRS states and made a detailedxamination of every firearm death reported in the NVDRS as anccident by any source – the NVDRS abstracter, the State Vitaltatistics Registry (i.e., the ICD-10 Underlying Cause of Death codend manner of death from the death certificate), the medical exam-ner or coroner report, and the police Supplementary Homicideeport. Comparing our findings (the gold standard) with those ofhe NVDRS abstracter, we conclude that NVDRS data accuratelylassify unintentional firearm deaths. We then compared our find-ngs with the ICD-10 codes supplied by the State Vital Statisticsegistries and found, as has been reported in the literature, thateath certificate data often report other-inflicted shooting acci-ents as homicides (Barber et al., 2002; Schaechter et al., 2003).he ICD-10 codes missed 38% of the accidents, most of which wereisclassified as homicide. Quite unexpectedly, however, in spite

f this under-reporting, the ICD-10 codes classified more casess accidents than did the gold standard. A total of 42% of theCD-10-identified cases were false positives. On reviewing the inci-ent narratives and other data sources for these cases, most werenambiguous homicides or suicides, not gray area cases or mattersf definition. We believe this results largely from the practice ofssigning ICD Underlying Cause of Death codes to the “accident”ange when the manner of death for an injury death is initiallylassified as pending or left blank (discussed further below).

Finally, at the state level, we compared our gold standard countf unintentional firearm deaths by state and year with the countrovided by the National Vital Statistics System on the WONDERebsite. We found what appears to be even greater over-reporting

f cases by Vital Statistics. Overall, Vital Statistics reported 489rearm accidents, compared with the 381 gold standard accidents.hile sensitivity and PPV for the Vital Statistics cases could not be

alculated (since the data were not linked at the individual level),t the level of state/year subtotals, there were a total of 194 casesor 51% of the gold standard total) on which Vital Statistics and theold standard disagreed.

The classification of deaths in Vital Statistics is based on theCD-10 Underlying Cause of Death code assigned to the cases. Inhe NVDRS data, both sensitivity (62%) and PPV (58%) of the ICD-10oding was low for firearm accidents; that is, many “true” accidentsere missed, while many suicides and homicides were mistakenly

lassified as accidents. For the death certificate “manner of death”eld, sensitivity was also low, but there were few false positives.he false positive problem, therefore, was largely restricted to theCD-10 coding.

Because of the importance of the ICD-10 Underlying Cause ofeath code in official U.S. mortality data, we summarize below the

easons for the dual problems of false positives and low sensitiv-ty that we observed in the ICD-10 codes for unintentional firearmeath.

and Prevention 43 (2011) 724–731 729

5.1. Reasons for false positives in ICD-10 coding (n = 168/404)

5.1.1. Major reason – cause-of-death coding rulesStarting in 2003, NCHS required all states to use SuperMICAR

software to assist in coding cause of death data from the deathcertificate (Glenn, 2010). Literals from the death certificate “causeof death” section are entered into the software program whichrearranges the words, replaces certain words with synonyms, andselects a numeric code. The MICAR100 program then translatesthe numeric codes to the appropriate ICD-10 codes, and, whenmultiple causes of death are coded, the ACME program selectsone code as the underlying cause in accordance with WHO guide-lines. In accordance with ICD-10 coding policy, the SuperMICARsystem defaults to assigning “external cause of injury” codes tothe “accident” range for injury deaths unless a manner of deathof “homicide,” “suicide,” or “could not be determined” is entered. Ifmanner of death is “pending” or left blank, the ICD-10 code assumesthe manner is “accident” (Mckenzie et al., 2010). States are sup-posed to update the field when the manner of death is established,but this apparently does not always happen. We raised the problemof likely false positives to Donna Glenn, the head of the Mortal-ity Medical Classification Branch at the National Center for HealthStatistics. To investigate, Ms. Glenn read through the records of allsupposed accidental firearm deaths in one year for an NVDRS statein which Vital Statistics-identified accidents outnumbered NVDRS-identified accidents more than 3-to-1 in 2006. Most were “pending”or “missing” for manner of death. It appears likely that the codingalgorithm results in pending/missing cases being incorrectly cate-gorized as unintentional deaths, creating false positives for firearmaccidents.

Variation across states in the extent to which the coding soft-ware results in false positives is likely to be a function of the degreeto which the state resubmits data to NCHS when certificates with“pending” and “missing” manners of death are reclassified to a sub-stantive manner other than “accident.” It is perplexing that VitalStatistics reported larger numbers than were obtained using theICD-10 codes in the NVDRS data, since Vital Statistics are basedon the ICD-10 codes provided by the states. This may in part bebecause the NVDRS system obtains the state death certificate dataat a later point when “pending” or missing manners of death havebeen resolved, or because the NVDRS abstracter continues to checkback on pending/missing cases to learn when new information hasbeen entered.

5.1.2. Minor reason – case definition issues: Russian roulette,non-powder guns, and legal intervention

To a much lesser extent, three definitional and coding issuesplay some role in the false positives. Russian Roulette is classifiedas suicide in the NVDRS but as accident by some (not all) coronersand medical examiners. The National Association of Medical Exam-iners recommends coding Russian Roulette as suicide (Hanzlicket al., 2002). Review of the incident narratives for the ICD-10 falsepositives indicates 6 Russian Roulette cases for which the deathcertificate manner of death was “accident.”

Unintentional shootings that occur as a direct result of a policeaction (e.g., a police officer catches up with a suspect whose gunis drawn and throws him into a fence and the suspect’s gun dis-charges) should receive an ICD-10 code for a legal interventiondeath, not accident. This applied to one case. Finally, non-powderguns, such as BB guns, are classified in the ICD-10 system as “otherand unspecified firearms.” This accounted for only one “false posi-

tive” in the NVDRS database.

5.1.3. Minor reason – jurisdictional issuesThe case definition in this study was based on the decedent’s

state of residence. If, for example a South Carolina hunter were

7 nalysis

kdakwnoaNktd

5(

5h

tMsccetmtaattfanioaD

5

titfiTa

5

c

5

rawlisrdaO

30 C. Barber, D. Hemenway / Accident A

illed while hunting in North Carolina (another NVDRS state), theeath would be picked up in Vital Statistics and in this study asSouth Carolina state. However, if a South Carolina hunter were

illed while hunting in Tennessee (not an NVDRS state), the deathould appear in Vital Statistics as a South Carolina death but mayot be captured in NVDRS (since the state in which the incidentccurred is the reporting state in NVDRS). This is likely to explainvery small percentage of the gap between Vital Statistics andVDRS-identified cases since most firearm accident victims areilled in their own state. Only 1.6% of the people killed in unin-entional firearm accidents in NVDRS states were residents of aifferent state.

.2. Reasons for false negatives (low sensitivity) in ICD-10 codingn = 145/381)

.2.1. Major reason: other-inflicted accidents classified asomicide

Among the 145 firearm accidents missed by the ICD-10 codes inhe NVDRS data, 70% were shootings inflicted by another person.

any coroners and medical examiners consider the unintentionalhooting of one person by another a homicide. In 54% of the NVDRSases in which one person unintentionally shot another, the ICD-10ode was in the homicide range. This comports with findings fromarlier studies and is unlikely to change in the near future. In 2002,he National Association of Medical Examiners released recom-

ended guidelines for classifying manner of death and suggestedhat deaths involving the unintentional shooting of one person bynother be classified as “homicide” unless the incident involvedmalfunctioning firearm (Hanzlick et al., 2002). NVDRS classifies

hese deaths as accidents. ICD-10 guidelines also call for codinghese deaths as accidents. The ICD guidelines indicate that when,or example, a medical examiner classifies a drunk driving crashs a vehicular homicide, the death should be coded as an accident,ot homicide, unless there is clear evidence that the death was

ntended. Judging from the NVDRS data, this coding guideline isften not followed. For each false positive, we checked whetherfirearm accident code appeared in any of the Multiple Cause ofeath codes applied to the case; there were none.

.2.2. Minor reason: late effects deathsDeaths that occur over a year after an accidental injury was sus-

ained are coded in the ICD-10 as sequelae and are divided onlynto transport and non-transport-related sequelae, without fur-her elaboration of the mechanism causing the injury. Late effectsrearm deaths therefore cannot be identified in ICD-10 coding.here were 11 of these among the gold standard firearm accidents,ccounting for 8% of the cases missed by the ICD-10 codes.

.3. Implications

Findings of our comparison of gold standard cases with ICD-10oded cases have four major implications.

.3.1. Accuracy of NVDRS dataThe National Violent Death Reporting System appears to be a

eliable source of data on unintentional firearm deaths. The NVDRSbstracters are applying the case definition for firearm accidentith good accuracy. False positives and false negatives were very

ow; both sensitivity and PPV were over 98%. Unfortunately, NVDRSs not yet a national system; as of 2010, federal funding for the

ystem has enabled only 18 states to participate. Although weeviewed all firearm cases for which any of the sources indicated theeath was unintentional, it is possible that some true unintention-ls are among the cases that all data sources agreed were homicides.ur review of a 5% random sample of these cases found one such

and Prevention 43 (2011) 724–731

case among 552. If this rate, or half or double this rate, held forthe other 95% of firearm homicides, there could be another 10–40firearm accidents in the NVDRS dataset beyond the 381 alreadyidentified.

5.3.2. Inccuracy of official mortality dataFirearm accidents are inaccurately reported in mortality data.

The ICD-10 codes failed to identify 38% of the firearm accidents.Most of these were other-inflicted accidents, which dispropor-tionately occur among younger victims (Hemenway et al., 2010).More concerning, 42% of the cases that the ICD-10 codes identi-fied as accidents were not accidents. While a few of these werematters of interpretation (e.g., Russian Roulette cases), the vastmajority appear to result from a coding policy that misclassifiessuicides and homicides as accidents. The problem appears to beexacerbated at the national level, when comparing National VitalStatistics System data with NVDRS data. The result of the missedtrue accidents and the misclassified intentional deaths is that offi-cial mortality data over-report gun accidents among adults andunder-report them among youths, particularly accidents involv-ing horseplay. Importantly, the over-reporting is not distributedequally across states, with some states showing under-reportingand some gross over-reporting. These data have been used to eval-uate the impact of public policies such as Child Access Protectionlaws (laws requiring safe storage of firearms) (Cummings et al.,1997; Webster and Starnes, 2000; Schaechter et al., 2003; Hepburnand Hemenway, 2004). Researchers wishing to compare states’unintentional firearm death rates are cautioned that the data aredistorted. Further, researchers are cautioned against using unin-tentional firearm injury rates as a proxy for state-level firearmownership rates (Lester, 1988).

5.3.3. Unknown reliability of trend dataAs a result of the coding problems, it is impossible to say with

certainty whether the fall in unintentional firearm death ratescharted by mortality data is accurate; there may have been an evensharper fall if the false positive problem is recent, or perhaps little orno fall if the false positive problem is getting better or the false neg-ative problem getting worse. The policy of defaulting injury deathsto “accident” when manner of death is missing or pending appliesto both hand-coding and software-assisted coding, so the practicepredates SuperMICAR. Over the past few years, at the urging ofNCHS, the proportion of death certificates with incomplete man-ners of death has been declining, which would lead to a decline infalse positive accidents. For example, between 2001 and 2002 thenumber dropped from 10,504 to 6,559 (Minino et al., 2006). How-ever, it is unknown whether there have been any changes over timein the proportion of states that update NCHS when a death cer-tificate with an incomplete manner of death is finally completed.Additionally, it is unknown whether coroners and medical exam-iners have changed behavior with respect to the proportion thatclassify other-inflicted accidents as homicides.

5.3.4. Impact of coding algorithm on other injury deathsFor most mechanisms of injury – transport, drowning, falls, fire

– the practice of defaulting manner of death to accident in caseswhere manner is missing or pending will not distort national dataappreciably since the majority of injury deaths are unintentional,and relatively few injury deaths reported to NCHS have pending ormissing manner (4% in 2002) (Minino et al., 2006). Firearms are oneof the few mechanisms for which the vast majority of deaths (97%)

are intentional, not accidental, and thus the SuperMICAR systemintroduces considerable distortion. Assigning even a tiny propor-tion of firearm suicides and homicides to the accident categoryinflates the accident category considerably. The other mechanismof injury death where intentional cases vastly outnumber uninten-

nalysis

ttTe1udmtw

5

SofiuAsomosDN

Soo1otes

fiwrrwb

6

uauntaicutpl

Pt 1), 741–744.

C. Barber, D. Hemenway / Accident A

ional cases are sharp instruments (94% intentional). It is possiblehat unintentional sharp instruments data may be similarly biased.his general problem is similar to the false-positive problem for rarevents that is well-known in epidemiology (Hennekens and Buring,987; Hemenway, 1997). Poisonings may also be affected; althoughnintentional poisonings are not rare (over 70% are coded as acci-ents), manner of death may be pending or missing for poisoningsore often than for other types of injuries because the results of

oxicologic testing are often not available to the death certifier foreeks.

.4. Limitations

This paper examines the accuracy with which NVDRS and Vitaltatistics capture unintentional firearm deaths. There are a numberf limitations to using NVDRS data. While we found that the classi-cation of firearm deaths within NVDRS is very accurate, we werenable to assess overall sensitivity of firearm deaths in NVDRS.ccording to Deborah Karch, chief scientific officer for NVDRS, notudy of the overall sensitivity of NVDRS in capturing violent deaths,r firearm deaths in particular, has been published to date. A rougheasure, however, indicates that sensitivity is high: in 2005, the

verall number of firearm deaths (all intents) in the 16 NVDRStates was 8051 according to Vital Statistics (on the CDC WON-ER website) and 7828 (97% the Vital Statistics total) according toVDRS.

This study was limited by the fact that we did not link Vitaltatistics and NVDRS data at the individual level. We relied insteadn comparing the gold standard with the ICD-10 Underlying Causef Death code in the NVDRS data. It is possible that some of the ICD-0 codes in the NVDRS data that were misclassified were the resultf data entry errors by the NVDRS abstracter, not actual errors inhe mortality data. Several NVDRS states, however, import the datalectronically from their state’s mortality file, so distortions of thisort will be isolated to states using hand data entry.

Finally, our analysis deals only with the correct classification ofrearm deaths as either intentional or unintentional. For example,e did not examine whether some homicides are reported incor-

ectly as suicides, nor whether some firearm suicides are incorrectlyeported as non-firearm suicides. Our analysis has little to say abouthether the total number of firearm deaths are accurately reported

y either NVDRS or WONDER.

. Conclusions

In answer to the question, “Are there too many or too fewnintentional firearm deaths in official mortality data?” the bestnswer is, “Both.” In over half of the cases in which one personnintentionally shoots another, the case is coded as a homicide,ot accident, for underlying cause of death. However, at the sameime, many firearm homicides and suicides are mistakenly captureds accidents, apparently largely as a result of a coding algorithmn the software that was designed to improve and standardize

ause-of-death coding. Official mortality data are therefore annreliable source of data on unintentional firearm deaths becausehey miss many true positives and report many false positives. Theroblem is particularly acute in certain states. The National Vio-

ent Death Reporting System, on the other hand, applies a case

and Prevention 43 (2011) 724–731 731

definition for unintentional firearm deaths with consistency andaccuracy.

Acknowledgements

Funding was provided by the Joyce Foundation. We thank theCDC’s Center for Injury Prevention and Control for providing theNVDRS data; Donna Glenn, Chief of Mortality Medical ClassificationBranch at the National Center for Health Statistics, for informationabout ICD-10 coding practices; and Grace Galvin for assistance inpreparing the manuscript.

References

Barber, C., Hemenway, D., Hochstadt, J., Azrael, D., 2002. Underestimates of uninten-tional firearm fatalities: comparing supplementary homicide report data withthe National Vital Statistics System. Inj. Prev. 8 (3), 252–256.

Barber, C., Ozonoff, V.V., Schuster, M., Hume, B.C., Mclaughlin, H., Jannelli, L., Saltz-man, L.E., 1998. Massachusetts weapon-related injury surveillance system. Am.J. Prev. Med. 15 (Suppl. 3), 57–66.

Centers for Disease Control and Prevention, 2003. National Violent Death Report-ing System Coding Manual. National Center for Injury Prevention and Control,Centers for Disease Control and Prevention.

Comstock, R.D., Mallonee, S., Jordan, F., 2005. A comparison of two surveillancesystems for deaths related to violent injury. Inj. Prev. 11 (1), 58–63.

Cummings, P., Grossman, D.C., Rivara, F.P., Koepsell, T.D., 1997. State gun safe storagelaws and child mortality due to firearms. JAMA 278 (13), 1084–1086.

Frattaroli, S., Webster, D.W., Teret, S.P., 2002. Unintentional gun injuries, firearmdesign, and prevention: what we know, what we need to know, and what canbe done. J. Urban Health 79 (1), 49–59.

Glenn, D., 2010. In: Barber, C. ed., Donna Glenn, Chief of Mortality Medical Classifi-cation Branch at the National Center for Health Statistics.

Hanzlick, R., Hunsaker, J.C.I., Davis, G.J., 2002. A Guide for Manner of Death Classifi-cation, first ed. National Association of Medical Examiners.

Hemenway, D., 1997. The myth of millions of annual self-defense gun uses: a casestudy of survey overestimates of rare events. Chance 10 (3), 6–10.

Hemenway, D., Barber, C., Gallagher, S.S., Azrael, D.R., 2009. Creating a National Vio-lent Death Reporting System: a successful beginning. Am. J. Prev. Med. 37 (1),68–71.

Hemenway, D., Barber, C., Miller, M., 2010. Unintentional firearm deaths: a com-parison of other-inflicted and self-inflicted shootings. Accid. Anal. Prev. 42 (4),1184–1188.

Hennekens, C.H., Buring, J.E., 1987. Epidemiology in Medicine, first ed. LippincottWilliams & Wilkins, Philadelphia.

Hepburn, L., Hemenway, D., 2004. Firearm availability and homicide: a review of theliterature aggressive and violent behavior 9 (4), 417–440.

Ingram, D.D., Franco, S., 2006. 2006 NCHS Urban-Rural ClassificationScheme for Counties. The Centers for Disease Control and Prevention,http://wonder.cdc.gov/wonder/help/CMF/Urbanization-Methodology.html(accessed July 2010).

Lester, D., 1988. Firearm availability and the incidence of suicide and homicide. ActaPsychiatr. Belg. 88 (5–6), 387–393.

Mckenzie, K., Fingerhut, L., Harrison, J., 2010. Restructuring the externalcauses of injury chapter for icd-11: Recommendations paper. Unpublished.Available online at http://sites.google.com/site/injextcauseicd11/chapter-20-revision. Accessed July 2010.

Minino, A.M., Anderson, R.N., Fingerhut, L.A., Boudreault, M.A., Warner, M., 2006.Deaths: injuries, 2002. Natl. Vital Stat. Rep. 54 (10).

National Research Council, 2005. Firearms and Violence: A Critical Review. TheNational Academies Press, Washington, DC.

Paulozzi, L.J., Mercy, J., Frazier Jr., L., Annest, J.L., 2004. CDC’s National Violent DeathReporting System: background and methodology. Inj. Prev. 10 (1), 47–52.

Schaechter, J., Duran, I., De Marchena, J., Lemard, G., Villar, M.E., 2003. Are “acciden-tal” gun deaths as rare as they seem? A comparison of medical examiner mannerof death coding with an intent-based classification approach. Pediatrics 111 (4

Webster, D.W., Starnes, M., 2000. Reexamining the association between child accessprevention gun laws and unintentional shooting deaths of children. Pediatrics106 (6), 1466–1469.

World Health Organization, 1992. ICD-10: International Statistical Classification ofDiseases and Related Health Problems, tenth ed. WHO, Geneva.