Are There Heterogeneous Effects of Electronic Medical Record Adoption on Patient Health Outcomes?

Seth Freedman, Haizhen Lin and Jeffrey Prince*

Preliminary and incomplete: please do not circulate or cite

November 2013

Abstract

This paper examines the effect of hospital adoption of electronic medical records (EMRs) on health outcomes, particularly patient safety indicators (PSIs). We use a newly integrated dataset that merges information on hospital EMR adoption to a database of inpatient discharges from a nationally representative sample of hospitals. Previous studies of hospital EMR adoption and health outcomes have focused on Medicare and elderly patients and have found small or no impacts. To our knowledge this is the first large-scale study of the effect of EMR adoption on health outcomes that includes non-Medicare patients in addition to Medicare patients. This broader sample allows us to test whether EMRs have differential health effects across age groups. Preliminary results suggest that EMR adoption does decrease the prevalence of preventable adverse events measured by PSIs and has a notably greater impact on the non-Medicare population. These findings extend prior work by showing the modest impacts previously found for seniors may not be representative of the entire population. We also find that the effect of EMRs on patient outcomes may be lagged, consistent with previous findings about the relationship between EMRs and costs.

1. Introduction

The increasing availability and adoption of electronic medical records (EMRs) of various forms

has generated substantial optimism concerning possible consequent improvements in productivity, costs,

and quality within the healthcare sector (e.g., Hillestad et al., 2005). This optimism has proven

substantial enough to even spur U.S. policy to create incentives for adoption (i.e., the Medicare and

* Freedman is at the School of Public and Environmental Affairs at Indiana University; Lin and Prince are at the Department of Business Economics and Public Policy in the Kelley School of Business at Indiana University. We thank Leila Agha, Chris Forman, Shane Greenstein, Avi Goldfarb, and Jeffrey McCullough for useful discussion and comment, and Noah Hammarlund for excellent research assistance. We also acknowledge the Health Information Management Systems Society (HIMSS) for providing access and assistance to their data. We are responsible for all remaining errors.

1

Medicaid EMR/EHR Incentive Programs). In contrast, the extant literature measuring various impacts of

EMR adoption provides little indication of dramatic returns (e.g., Ahga 2012, McCullough et al. 2010,

McCullough et al. 2013, Parente & McCullough 2009). However, the scope of these analyses,

particularly with regard to health outcomes, has largely been limited due to data constraints. In particular,

previous studies have focused on mortality among Medicare patients as the primary health outcome. The

lone large scale study that we are aware of that uses data from non-Medicare patients is Miller and Tucker

(2011) who find that the availability of EMRs within a county decreases infant mortality rates.1

In this paper, we build and utilize a newly integrated dataset to analyze the effect of EMR

adoption on health outcomes for a broader patient population. Our data allow us to examine whether the

effect of EMR adoption differs across age groups (seniors vs. non-seniors). There are various reasons to

think that EMR adoption may have a notably different impact on health outcomes for non-seniors. On one

hand, EMRs may have more impact on higher severity patients and may therefore be less important for

the care of non-seniors. Alternatively, aspects of EMRs that provide the clinician with decision support

such as reminders, treatment protocols, etc. may have more of an impact for more straightforward cases

rather than cases with multiple interacting comorbidities. There also may be reason to believe EMRs may

have a larger effect on a younger population due to differing physician/patient dynamics. For example,

Mold et al. (2004) indicates that duration of a patient’s relationship with his/her physician is increasing

with age (perhaps, e.g., due to lower propensity to change home residence). This suggests that the value

of EMRs, which can facilitate the transfer of information, may be substantially greater for the more

transient, non-senior group.

If such heterogeneity exists, it can have a significant impact on public policy and subsequent

research concerning EMR adoption and usage. It can have a powerful impact on how the U.S.

government should be setting policy as pertains to EMR. It will help to better assess the potential social

value of incentive programs and whether targeted incentives may be warranted. Further, it can help

1 Athey and Stern (2002) find information technology linking 911 caller identification to a location database speeds emergency response and reduces short-term mortality and hospital costs.

2

inform (potential) patients in their choice of care. If benefits for health outcomes from EMRs exist, aware

patients can self-select toward hospitals utilizing these beneficial technologies, improving patient welfare

and providing further incentive for non-adopters to adopt sooner.

A significant group of studies has analyzed the effect of EMR adoption on health outcomes,

focusing on only the senior population. Within this group of studies, Agha (2012) finds little effect of

EMR adoption on patient mortality, medical complication rates, adverse drug events, and readmission

rates. Kazley & Ozcan (2008) find some improvements in process indicators, but overall conclude that

the evidence of a relationship between EMRs and quality is limited. McCullough et al. (2010) find some

evidence of improvement in quality measures, but temper this finding by concluding that achieving

substantive benefits from EMR adoption at a national level may be a lengthy process. McCullough et al.

(2013) find evidence of improvement in mortality rates, but only for the most complex cases. Parente &

McCullough (2009) is the paper most similar to ours. They find some improvement in patient safety due

to EMR adoption, but conclude there is not enough evidence to draw a strong link between EMR and

improvements in patient safety for the Medicare population.

Other studies analyzing the effect of EMR adoption on health outcomes have included the non-

senior population, but have utilized small, focused sets of data, primarily at the hospital level. These

studies appear to have more positive findings. For example, Bates et al. (1998) find a significant

reduction in serious medication errors due to EMR adoption; Bates et al. (1999) find a similar

improvement for a different type of EMR. The only large scale study of the non-senior population that we

are aware of is Miller and Tucker (2011). They find important effects of EMRs on infant mortality rates,

and our study adds to this by examining health effects for a broader subset of the non-elderly population.

In addition, our main outcomes of interest are measures of patient safety, instead of mortality. Beyond

being a more relevant measure for the non-senior population, patient safety can shed light on the impacts

of EMRs on important, but non-deadly, adverse health events.

Prior findings pertaining to costs generally have been negative. For example, Agha (2012) finds

no cost savings following EMR adoption, even five years after the fact. Dranove et al. (2012) find that

3

EMRs, on average, generate a slight increase in costs. However, this average effect is a combination of

cost reductions for hospitals in “favorable” locations (i.e., due to complementarities) and large cost

increases for hospitals in “unfavorable” locations. Sidorov (2006) surveys a range of medical studies on

EMRs, arguing that many of these studies suggest costs increased after adoption. In contrast, Wang et al.

(2003) argue that cost savings are possible, using a hypothetical cost/benefit analysis within an

ambulatory care setting. In addition, Chaudhry et al. (2006) find some efficiency gains in a survey of

institution-level data. However, despite these relatively more sanguine findings concerning costs, the

majority of the literature suggests cost savings have been small or non-existent.

To test for the effect of EMR adoption on PSIs, we employ a fixed effect approach, exploiting the

fact that many hospitals adopted new EMR systems during our time period. We are therefore able to

control for fixed differences between adopting and non-adopting hospitals and identify how adopting an

EMR changes PSI rates within the adopting hospital. In addition to estimating the overall effect of EMR

adoption on PSIs, we estimate the relationship separately for seniors and non-seniors. Our primary data

sets include the 2003 through 2010 Health Information and Management Systems Society (HIMSS)

Analytics Database and the Nationwide Inpatient Sample (NIS) collected by the Agency for Healthcare

Research and Quality’s (AHRQ) Healthcare Cost and Utilization Project (HCUP). To our knowledge, this

is the first study to combine data on EMR adoption with a nationally representative sample of hospital

discharges and examine the effects of hospital EMR adoption on such a broad patient population.

Beyond construction of a novel andintegrated dataset, the other primary innovation of this study

is the direction in which it takes analysis of the effects of EMR adoption. Prior work has consistently

analyzed the effects of EMR adoption for very specific subgroups of the population – in particular, the

senior population or very small groups of hospitals. This study calls attention to the real and

consequential possibility that EMR adoption may have different impacts across various population

subgroups.

4

Our empirical analysis focuses on two EMR applications, Computerized Physician Order Entry

(CPOE) and Physician Documentation, which have experienced large variation in adoption during our

study period. Our results suggest that CPOE decreases the occurrence of preventable adverse events as

measured by PSIs. This stands in contrast to previous results focusing on mortality as an outcome and

suggests EMRs may have important effects on patient outcomes less severe than mortality. We also find

that CPOE leads to larger improvements for non-seniors as compared to seniors and increasing effects

over time. Results for physician documentation are also suggestive of improvements of patient safety that

grow over time.

These findings suggest interesting implications about the effectiveness of electronic medical

records in improving health outcomes. First, our findings suggest that they may play a role in improving

patient wellbeing by decreasing preventable adverse events. While other research has found little impact

on mortality, our results highlight the importance of exploring other health outcomes. Additionally, as

discussed below, patient safety indicators have been linked to longer hospital stays and higher hospital

charges. Therefore our results may have implications for EMRs role in reducing the cost of care by

limiting adverse events with additional downstream costs. Finally, the finding of a larger effect of EMRs

on patient safety among the non-elderly suggests important heterogeneity. In future work we will explore

mechanisms that might lead to larger effects among the non-elderly. In particular, we plan to explore how

the effect of EMRs differs by the multidimensionality of a patient’s comorbidities. This may explain the

age differences we find, and we hope to explore this further.

2. Electronic Medical Records and Health Outcomes

2.1. What are EMR Technologies?

As noted in Dranove et al. (2012), an electronic medical record (EMR) is a “catchall expression

used to characterize a wide range of technologies used by hospitals to keep track of utilization, costs,

outcomes, and billings.” The technologies generally classified as EMRs include: Enterprise EMR,

5

Clinical Data Repository (CDR), Clinical Decision Support System (CDSS), Order Entry, Computerized

Practitioner Order Entry (CPOE), and Physician Documentation. Wang (2012) considers all six of these

technologies in her analysis, and Dranove et al. (2012) consider all but Enterprise EMR.

The functionality of these EMR technologies perhaps is best presented via categorization. Both

Dranove et al. (2012) and Wang (2012) break EMR technologies into two broad groups, which can

essentially be labeled “basic” and “advanced.” The basic group includes Enterprise EMR, Clinical Data

Repository, Clinical Decision Support System, and Order Entry. As Wang (2012) describes, this basic

group contains applications that “can be used to store, organize and retrieve patients’ information.”

Clinical Decision Support can also provide diagnosis and treatment recommendations based on clinical

information. The advanced group includes Computerized Practitioner Order Entry and Physician

Documentation. Wang (2012) notes that these applications present medical history, recommend drugs,

and help health care providers make better decisions. Dranove et al. (2012) note that these advanced

applications “are more difficult to implement and more difficult to operate successfully due to the need

for physician training and involvement.” It is worth noting that Agha (2012) has a slightly different

means of characterizing these technologies. Her first group consists of applications whose primary

functions are record keeping; and the second being Clinical Decision Support (CDS) whose primary

functions are decision support.

We focus our initial analysis on the advanced technologies, CPOE and Physician Documentation.

We make this choice for two reasons. First, these two technologies show the most variation during our

data period (details in Section 3). . Additionally, these two technologies may be expected to have direct

links to patient safety. As described in McCullough et al. (2013), CPOE allows physicians to directly

input orders, potentially reducing miscommunication and errors. Additionally, rules-based protocols,

treatment guidelines, and prescription error checking are often built into CPOE products. 2 These types of

features that provide automatic reminders, check lists, and error checking may be expected to have direct

2 As in McCullough et al. (2013) we do not focus on the Clinical Decision Support application itself do to inconsistent reporting in the HIMSS data.

6

impacts on preventable adverse events that the patient safety indicators we examine are intended to

measure. Physician Documentation allows physicians to input information and receive alerts and

protocols at the point of care. It also generates diagnostic codes from clinical information. These codes

can be used both for billing purposes, but also to enhance communication between practitioners through

standard coding (Dranove et al. 2012). Physician Documentation may be expected to reduce adverse

events when care is administered by multiple practitioners who must communicate efficiently and

coordinate a patient’s care.

2.2. Why Do Hospitals Adopt EMR Technologies?

The primary reasons cited for hospitals to adopt EMR technologies revolve around healthcare

quality and costs. For example, President Obama stated on January 8, 2009 the following: “To improve

the quality of our health care while lowering its cost, we will make the immediate investments necessary

to ensure that within five years, all of America’s medical records are computerized. This will cut waste,

eliminate red tape, and reduce the need to repeat expensive medical tests. But it just won’t save billions

of dollars and thousands of jobs – it will save lives by reducing the deadly but preventable medical errors

that pervade our health care system.”

Adoption of EMRs can reduce costs for hospitals by eliminating redundancy, as noted by the

President. Further, as noted in Hillestad et al. (2005), EMR adoption can lower costs by reducing drug,

radiology, and laboratory usage, reducing clerical staff, reducing nursing time, lowering medical errors,

and shortening inpatient lengths of stay. Adoption of EMRs can improve healthcare quality by reducing

errors and improving disease prevention and chronic disease management (Hillestad et al., 2005). In

addition to these direct benefits to an adopting hospital, as Wang (2012) notes, EMR adoption may

generate externalities, meaning its value to one hospital depends on the adoption decisions of other

hospitals. Specifically, the value of adopting EMR for a given hospital may increase as a function of the

number of other hospitals with EMR, since information transfer becomes easier as more hospitals

7

participate. However, the opposite may be true if EMR adoption attracts more patients, such that

ultimately the profits of adoption go down as the number of adopters increases.

In deciding whether to adopt EMR, a hospital must balance the above (potential) benefits against

the costs of adopting. The Congressional Budget Office (CBO, 2008) estimates the cost of EMR

adoption for a typical urban hospital to range between $3 and $9 million, along with between $700,000

and $1.35 million per year for maintenance. The costs and benefits of adoption certainly change over

time, as well as awareness levels across hospitals and patients. Hence, as we discuss in Section 3, there is

significant variation in hospitals’ timing of adoption of EMR technologies. This variation is important for

us to identify the health effects of these technologies, and our econometric methods are designed to

account for potential factors that may concurrently influence EMR adoption and health outcomes, as we

discuss in our Methods section.

2.3. What are Patient Safety Indicators?

In identifying the effect of EMR adoption on health outcomes, a useful measure is patient safety.

This is both because there exist well-defined and -established patient safety indicators (PSIs), and because

these indicators are more variable than other health outcome measures in our data, such as mortality.

These indicators, developed by AHRQ, are intended to measure preventable in-hospital complications and

adverse events. Therefore, these indicators allow us to examine effects of EMR adoption on meaningful

health outcomes that may be less severe than mortality.

In addition to representing a significant indicator of patient well-being, these indicators are likely

linked to increased healthcare utilization and cost. Zhan and Miller (2003) use the 2000 Nationwide

Inpatient Sample to examine how adverse events measured by patient safety indicators impact health care

utilization and eventual mortality. They use multivariable matching estimator to compare length of stay,

hospital charges, and in-hospital mortality for patients experiencing an adverse event to observably

similar patients within the same hospital not experiencing an adverse event. They find statistically

significant differences in these three outcomes for all of the PSIs that we examine in this paper. For

8

example, they find that patients with postoperative pulmonary embolism or deep vein thrombosis spend

5.36 additional days in the hospital, have higher hospital charges by $21,709, and have a 6.56% higher in

hospital mortality rate. While these differences may be partially driven by unobserved severity, they

motivate that PSIs have important implications for downstream health care utilization and costs, and in

addition to representing decreased patient well-being can lead to potential increases in mortality.

As mentioned above, we know of one prior study, Parente & McCullough (2009), that has

analyzed the effect of EMR adoption on PSIs, but this was limited to a sample of only seniors. The PSIs

they utilized were: infection due to medical care, postoperative hemorrhage or hematoma, and

postoperative pulmonary embolism or deep vein thrombosis. We look at a broader list of PSIs; however,

due to data changes, we do not include infection due to medical care.

We focus our analysis on the following PSIs: death in low-mortality diagnosis related groups,

pressure ulcer rate, postoperative hemorrhage or hematoma, postoperative physiologic and metabolic

derangement rate, postoperative respiratory failure rate, and postoperative pulmonary embolism or deep

vein thrombosis. Appendix Table 1 presents definitions of these outcomes. We also present results

aggregating the postoperative category into a single measure of those experiencing at least one of these

four adverse events. Parente & McCullough (2009) made their choice of PSIs based on the opinion of

clinical experts, and we have followed a similar strategy. We have chosen PSIs that are most likely to be

impacted by the availability of EMRs, in particular those that measure adverse events that can be

prevented by checklists and reminders and result from a failure to provide appropriate medication and

physical activity (potentially in combination). We exclude from our analysis PSIs that measure adverse

events tied more directly to surgical skill, physical accident, or those that occur with extremely low

incidence.

2.4. Possible Heterogeneous Effects of EMR Adoption

The fact that prior studies have found, at best, a modest impact of EMR adoption on health

outcomes when using national samples suggests that enthusiasm about the potential gains from these new

9

technologies may be misguided. However, these prior national studies of health outcomes have also

focused on the senior population, using the Center for Medicare and Medicaid Services (CMS) data. This

rich, individual-level data set has allowed researchers to observe outcomes for large samples of patients

who receive a large amount of hospital care. However, the fact that previous research has been restricted

to the Medicare population poses the question of whether the lack of EMR impact for seniors necessarily

extends to the population as a whole. This question becomes even more compelling in the context of

small-scale studies not limited to seniors which have found significant impacts (Bates et al., 1998 &

1999).

There is reason to believe the impact of EMRs may differ between seniors and non-seniors. It is

possible that any effect of EMRs on health outcomes will be greater for seniors than non-seniors. For

example, McCullough et al. (2013) find that EMR adoption results in larger mortality improvements for

higher severity Medicare patients. Given the fact that seniors on average will have higher severity as

compared to non-seniors, we might expect to find larger impacts among seniors. However, there is also

reason to believe that any effect of EMRs on health outcomes will be greater for non-seniors. While

McCullough et al. (2013) find the largest effects of EMRs among high severity patients, they also

hypothesize that certain aspects of EMRs, particular decision support systems that provide treatment

guidelines, protocols, and reminders, may play a larger role in moderate to low complexity cases. As

McCullough et al. (2013) argue, “Standard treatment guidelines are rarely implemented for complex

combinations of diagnoses.” Within Medicare data, they do not find empirical evidence for the prediction

that decision support applications will have meaningful health effects in less complex cases. However, by

expanding to a more diverse patient population, we may expect larger impacts of EMRs on the health

outcomes of younger and less complex patients through this mechanism. Note another distinction of our

paper is that we use patient safety indictors, which might better capture differences in health outcomes for

less complex cases.

In addition to this mechanism, Mold et al. (2004) show that duration of a patient’s relationship

with his/her physician is increasing with age. Given EMRs are often believed to assist in learning a

10

patient’s medical history and/or following proper medical procedures, these effects may be dampened in

the case where a doctor is already quite familiar with his/her patient. In contrast, the value of EMRs

toward health outcomes may be more pronounced for the large, more transient, non-senior component of

the population. Hospitals may also utilize EMRs differently depending on the patients’ type of coverage.

Given seniors generally often are using Medicare as their primary coverage, and non-seniors certainly are

not, differences in EMR utilization across coverage types will yield differences in their effectiveness

across seniors and non-seniors.

We estimate the implications of EMR adoption on patient safety for both seniors and non-seniors

using nationally representative hospital inpatient data allows. By separating these factors, we can (1)

compare the results for patient safety outcomes to previous results using CMS data (Parente &

McCullough 2009), (2) understand if EMR adoption impacts other health outcomes besides mortality for

a population of previously studied patients, and (3) shed light on whether EMR adoption has differential

impacts on seniors and non-seniors.

We also explore heterogeneity by time of EMR adoption. The adoption of a new EMR system

may not immediately improve health outcomes. Physicians, nurses, and other staff must be trained and

learn how to use new systems. Once they have learned how these systems operate, it may take additional

time to learn how to use them to optimally impact patient’s health. Dranove et al. (2012) find that cost

savings from EMRs in IT-intensive do not occur immediately and instead materialize 3 years after a

system is put in place.

There is also reason to believe the impact of EMRs may differ across EMR types. For example,

Dranove et al. (2012) find a small but notable difference in cost savings between basic and advanced

EMRs in IT-intensive markets. Given the fundamentally different functions of basic and advanced EMRs

toward influencing physician behavior (e.g., diagnosis), this difference may be much larger when

considering health outcomes rather than costs.

11

3. Data

3.1. Data Construction

The data we use for this study come from several sources, and to our knowledge, this is the first

study using such integrated data. Our first source of data is the Healthcare Information and Management

Systems Society (HIMSS) Analytics Database. HIMSS conducts an annual survey of health care

providers, including over 3,000 hospitals nationwide with more than 100 beds. The survey collects a wide

range of information on more than 100 different health information technology applications, including

CPOE and Physician Documentation. For each of these applications we construct variables for whether or

not a hospital has a system installed in a given year.3 The HIMSS data we have span 2003 to 2010.

Our second data source is the Nationwide Inpatient Sample (NIS), collected by the Agency for

Healthcare Research and Quality’s (AHRQ) Healthcare Cost and Utilization Project (HCUP). Using

these data, we are able to build measures of patient safety, and our coverage also spans the years 2003 to

2010. The NIS is a 20-percent, nationally representative, stratified sample of U.S. community hospitals.

Since NIS includes the universe of inpatient discharge records from these sampled hospitals, we are able

to observe both Medicare and non-Medicare insured patients. For each discharge record, the data set

includes information such as diagnosis and procedure codes, admission and discharge status, patient

demographics, expected source of payment, length of stay, and hospital charges. The NIS also reports

basic hospital characteristics including size, location, ownership type, and number of total discharges.

As noted above, our main outcomes of interest are Patient Safety Indicators (PSIs). We calculate

these indicators using a module provided by AHRQ. This module uses information in the discharge

record, such as age, diagnosis related groups, diagnosis codes, and procedure codes to identify the

subpopulation of patients for whom a particular adverse event is relevant and those who are likely to have

experienced the adverse event. For example, for the PSI indicating Postoperative Hemorrhage or

3 Following the guidance from HIMSS, we consider an application as installed if its status in the HIMSS data is live and operational, automated, to be replaced, or replaced.

12

Hematoma, the module first identifies patients who have received operations and might be at potential

risk, and then it determines which of these patients have experienced a hemorrhage or hematoma.

We supplement the HIMSS and NIS data with American Hospital Associate (AHA) data. The

AHA data is used to build a crosswalk between the HIMSS and NIS data. The only external hospital

identifier in the HIMSS data is the hospital’s Medicare provider numbers. The only external hospital

identifier in the NIS data is the hospital’s AHA ID number. AHA data contain both identification

numbers, thus allowing us to merge the HIMSS and NIS data at the hospital-year level.

We observe a total of 8,293 observations on 3,858 unique hospitals in the 2003 to 2010 NIS data.

Of these, 5,132 observations on 2,377 unique hospitals have AHA identification numbers available in the

data.4 Of this set, we are able to merge 4,105 observations from 2,010 unique hospitals with the HIMSS

data. While the NIS is not a panel of hospitals, a large fraction of hospitals appear in the data in multiple

years. For the years 2003 to 2010, 1,223 of the 2,010 unique hospitals that we observe appear at least

twice, with a total of 3,318 observations.5 This allows us to relate changes in patient safety to changes in

EMR adoption within hospitals and over time. Others have used the fact that hospitals appear in the NIS

in multiple years to exploit within-hospital changes in other contexts (e.g. Kolstad and Kowalski, 2012).

3.2. Summary Statistics

Table 1 shows the fraction of hospitals in our analysis sample that have adopted CPOE and

Physician Documentation by year. CPOE adoption grew from 7% to 31% of hospitals from 2003 to 2010.

Physician Documentation grew from 18% to 39% from 2005 to 2010.6 This rapid diffusion provides the

key variation we use to identify the effect of EMR adoption on patient safety.

4 AHA identification numbers are only available for hospitals from a subset of states in the NIS, as some states have not authorized HCUP to release information that would specifically identify hospitals. 5 The sample size for each regression described below varies for each PSI, as not all hospitals have patients in the population eligible to experience each PSI. 6 Note that Physician Documentation was first added to HIMSS in 2005. We are able to uncover the status of adoption in 2003 and 2004 for non-adopters (those that did not adopt between 2005 to 2011) and late-adopters (those that adopted between 2005 and 2011). For those that we observe adoption in 2005, we are not able to tell the year of adoption so we do not fill in these missing values.

13

[Table 1 about here]

Table 2 presents summary statistics of our PSI measures. For each PSI used in our analysis, we

count the number of patients that the AHRQ module indicates as experiencing the adverse event (our

dependent variable) and the number of patients with characteristics that make the PSI a relevant measure

for them (one of our control variables). These numbers can be thought of as the numerator and

denominator of the rate of PSI occurrence among its relevant population. In Table 2, we present the mean

hospital-year level values of these measures from our analyzed samples of seniors, non-seniors, and the

full population. Focusing on all patients, the number of eligible patients varies by PSI, with mean values

between 1,000 and 2,400. The incidence of PSIs also varies. Each hospital has on average 1 patient

suffering a death in a low mortality DRG, 53 patients experiencing a pressure ulcer, and 43 patients

experiencing at least one postoperative PSI. When we separately count patients eligible and experiencing

PSIs by age, the eligible population is generally larger among those below 65, but the number of patients

experiencing PSIs is generally higher. In other words, elderly patients have a higher rate of experiencing

adverse events.7

[Table 2 about here]

4. Empirical Model

Our general empirical strategy for testing the impact of EMR adoption on patient safety outcomes

is to relate within-hospital changes in patient safety over time to within-hospital changes in the

availability of EMRs. Hospitals that do and do not have EMRs at a point in time may be very different

from each other. Therefore we exploit over time variation in EMR adoption. The key identifying

7 This is particularly true for deaths in low mortality DRG, which has a much larger eligible population among the non-elderly. This is not surprising, as elderly patients are more likely to have higher mortality DRGs.

14

assumption is that trends in the prevalence of PSIs are not correlated with unobserved adoption trends. In

other words, our empirical strategy hinges on the idea that when a hospital adopts EMRs there are no

concurrent events, left unaddressed by our controls, that would have an impact on patient safety. If this

assumption is satisfied, we can attribute changes in patient safety to EMR adoption. McCullough et al.

(2013) and Agha (2012) provide extensive evidence that EMR adoption is unlikely to be correlated with

pre-existing trends in patient outcomes or severity. In future work, we will confirm that these findings

apply to the data used in this paper.

The PSI outcomes that we utilize are generally low-incidence events. We therefore aggregate

their occurrence at the hospital-year level. Hospital-year level PSI prevalence is most naturally thought of

as a count variable, and we therefore use the fixed effect Poisson model as our main empirical

specification. Our baseline empirical model therefore assumes that PSIht , which represents the number of

occurrences of a PSI in hospital h during year t, follows the Possion distribution with a mean given by

equation 1:

𝐸(𝑃𝑆𝐼ℎ𝑡|𝛼ℎ,𝛿𝑡 ,𝑋ℎ𝑡,𝐸𝑀𝑅ℎ𝑡) = 𝛼ℎ exp(𝛿𝑡 + 𝑋ℎ𝑡𝛽1 + 𝐸𝑀𝑅ℎ𝑡𝛽2) = exp (𝛾ℎ + 𝛿𝑡 + 𝑋ℎ𝑡𝛽1 + 𝐸𝑀𝑅ℎ𝑡𝛽2)

Where 𝛾ℎ = ln (𝛼ℎ). 𝛼ℎ is the individual hospital effect, EMRht is a dummy variable for the presence of

an EMR system installed in hospital h at year t. We run two versions of this analysis, one using CPOE

and one using Physician Documentation as our measure of EMR adoption. Xht is a set of hospital-year

control variables. These control variables include hospital characteristics (hospital bed size, urban vs.

rural location, and ownership type), the number of patients admitted in the hospital-year for whom the PSI

is relevant (including a linear, squared, and cubic term), and a set of control variables as suggested by

AHRQ’s risk adjustment procedures. These include age, diagnosis related groups, and other

comorbidities. These control variables are tailored to each PSI to reflect relevant comorbidities and

characteristics of the population for which the PSI is calculated. All regressions are weighted by the

number of total discharges in a hospital’s initial year of observation.

15

Following Cameron and Trivedi (2005), we use conditional maximum likelihood to eliminate the

hospital specific effects and estimate the other parameters of equation 1. We also cluster standard errors

for all regression estimates at the hospital level. The coefficient estimate of 𝛽2 can be interpreted as semi-

elasticity: the percentage change in PSI prevalence in response to a change in the EMR dummy from zero

to one.8

After estimating the overall effect of EMR adoption on patient safety, we estimate heterogeneity

of this effect along two dimensions. The first dimension of heterogeneity is patient age. As discussed

above, there may be differential effects of EMR for the elderly and the non-elderly. In order to explore

these differences, we aggregate the data by counting PSI occurrences in hospital, year, and age group

cells, where age groups are defined as above and below 65 years of age. We then estimate a fixed effect

Poisson model where the conditional mean is instead expressed by equation 2:

𝐸(𝑃𝑆𝐼ℎ𝑎𝑡| 𝛼ℎ,𝛿𝑡 ,𝑋ℎ𝑡,𝐸𝑀𝑅ℎ𝑡,𝑛𝑜𝑛𝑒𝑙𝑑𝑎)

= 𝛼ℎ exp(𝛿𝑡 + 𝑛𝑜𝑛𝑒𝑙𝑑𝑎 ∗ 𝛿𝑡 + 𝑋ℎ𝑡𝛽1 + 𝐸𝑀𝑅ℎ𝑡𝛽2 + 𝐸𝑀𝑅ℎ𝑡 ∗ 𝑛𝑜𝑛𝑒𝑙𝑑𝑎𝛽3)

= exp (𝛾ℎ + 𝛿𝑡 + 𝑛𝑜𝑛𝑒𝑙𝑑𝑎 ∗ 𝛿𝑡 + 𝑋ℎ𝑡𝛽1 + 𝐸𝑀𝑅ℎ𝑡𝛽2 + 𝐸𝑀𝑅ℎ𝑡 ∗ 𝑛𝑜𝑛𝑒𝑙𝑑𝑎𝛽3)

Notation is as above with nonelda denoting a dummy that equals one for cells representing patients below

65 years of age. We interact this dummy variable with the EMRht dummy and add age-group specific year

fixed effects. Again, the coefficients have a straightforward semi-elasticity interpretation. Estimates of 𝛽2

represent the percentage change in the occurrence of a PSI in response to EMR adoption for the elderly,

and estimates of 𝛽2 + 𝛽3 represent percentage change in the occurrence of a PSI in response to EMR

adoption for the non-elderly.

8 The marginal effect of EMR on E(PSI), 𝜕𝐸(𝑃𝑆𝐼)𝜕𝐸𝑀𝑅

, is equal to 𝛽2 exp(𝛾ℎ + 𝛿𝑡 + 𝑋ℎ𝑡𝛽1 + 𝐸𝑀𝑅ℎ𝑡𝛽2) = 𝛽2𝐸(𝑃𝑆𝐼).

Therefore, the semi-elasticity 𝜕𝐸(𝑃𝑆𝐼)𝜕𝐸𝑀𝑅

× 1𝐸(𝑃𝑆𝐼)

is equal to 𝛽2.

16

Second, we explore how the impact of EMR adoption on patient safety differs by time since

adoption. For this analysis we replace the EMR dummy in equation 1 with four different dummies

indicating the first, second, third, or fourth or more year of adoption. This specification allows for the fact

that it may take some time to fully and optimally incorporate EMR usage into practice patterns.

Coefficient estimates from this specification reveal how this process evolves from the first year of

adoption through later years. This specification also mitigates a main limitation of the specification in

equation 1. Because the NIS is not a true panel, equation 1 treats hospitals who have adopted EMRs

between observation years the same, regardless of which year they actually adopted the technology. By

utilizing the HIMSS data to calculate the number of years an application has been installed, we can more

precisely differentiate the relationship between actual adoption year and changes in patient safety.9

5. Results

Estimates of the overall effect of CPOE and Physician Documentation on patient safety (equation

1) are presented in Table 3. While we find no statistically significant effect of CPOE adoption on Deaths

in Low Mortality DRGs, coefficient estimates are negative and statistically significant for all other PSIs.

These estimates imply that adopting CPOE reduces Pressure Ulcers by 3.2% and Postoperative PSIs by

between 3.2% and 11.3%. Taking all four Postoperative PSIs together, CPOE reduces the number of

patients experiencing at least one of these events by 6.3%. In contrast with previous studies finding little

to no effect of EMRs on mortality, these results suggest that CPOE can impact the occurrence of adverse

patient safety events. Physician Documentation has a negative and statistically significant effect on three

of the PSIs, namely Pressure Ulcers, Postoperative Hemorrhage or Hematoma, and Physiological &

Metabolic Derangement. Like CPOE, Physician Documentation has no impact on Deaths in Low

Mortality DRGs. In contrast to CPOE, it does not impact two of the postoperative PSIs and has no

9 Note that the sample size for this set of regressions is smaller as for those hospitals who have adopted EMRs prior to 2003, we are not able to track down the number of years since adoption.

17

statistically significant impact on the overall measure of the number of patients experiencing at least one

of these postoperative adverse events.

[Table 3 about here]

Table 4 separates the effect of CPOE and Physician Documentation on patient safety by age

group and presents the results of equation 2. Consistent with the overall effect in Table 3, CPOE has no

impact on Deaths in Low Mortality DRGs for either seniors or non-seniors. The main effect of CPOE,

which represents the effect of adoption on safety for seniors, is negative for all of the other PSIs. This

negative effect is statistically significant at the 5% level for each of these remaining PSIs, except Pressure

Ulcers and Respiratory Failure. The results in the final column imply that CPOE decreases the number of

seniors experiencing at least one adverse postoperative event by 4.6%. Results for non-seniors are a bit

mixed, but are generally more negative than for seniors. In particular, the effect of CPOE adoption is far

greater for non-seniors for Physiologic and Metabolic Derangement, and Pulmonary Embolism or Deep

Vein Thrombosis. Overall non-seniors are 7.1% (4.6 + 2.5) less likely to experience at least one adverse

postoperative event when CPOE is installed, and the non-senior effect is statistically different from the

senior effect at the 5% level. This suggests that CPOE has additional effects on non-seniors safety above

and beyond the effect on seniors.

[Table 4 about here]

Results in Table 4 for Physician Documentation are a bit more mixed. Pressure Ulcers appear to

decrease for seniors, but not for non-seniors. The p-value of the combined effect for non-seniors is 0.15.

Of the Postoperative PSIs, only Physiological & Metabolic Derangement shows improvements for seniors

and additional improvement for non-seniors, and Hemorrhage or Hematoma shows a similar pattern, but

the effect for seniors is not statistically significant. In fact, the overall measure of having at least one

18

postoperative PSI actually increases with Physician Documentation adoption for seniors. On the other

hand, the combined effect for non-seniors is negative (0.23-.040=-.017, p=.048) and statistically

significant at the 5% level. Contrasting the results in Table 4 for CPOE and Physician documentation

suggests that there may be important differences between these two applications in their ability to

improve care for different types of patients.

In order to attempt to disentangle the mechanisms behind these age differences, we also examine

how the effect of EMR adoption varies by case complexity. As discussed above, decision support and

treatment protocols may be more beneficial in cases with fewer interacting comorbidities. The AHRQ

module that we utilize to calculate PSIs also creates indicators for 27 different comorbidity measures. As

a preliminary look at how case complexity impacts EMR effectiveness we simply sum up the number of

comorbidities of each patient. Figure 1 presents histograms of the number of comorbidities for seniors and

non-seniors. As expected, the number of comorbidities is correlated with age. Non-seniors are much more

likely to have 0 comorbidities, and seniors are more likely to have two or more comorbidities. We use a

binary indicator of having fewer than two comorbidities as an indicator of a “simple” case. Having two or

more comorbidities is considered a complex case. Table 5 shows that 63% of non-seniors have a simple

case while only 40% of seniors have a simple case.

[Figure 2 about here]

[Table 5 about here]

In Table 6 we present regression results where we aggregate the data into hospital, year, and case

complexity cells and interact the ERM indicator with an indicator for cells representing simple cases.

While we find some statistically significant coefficients for Death in Low Mortality DRGs and Pressure

Ulcers, neither the complex case effect nor the combined simple case effect are statistically significant at

the 5% level. In contrast to our age results, CPOE decreases the likelihood of experiencing a post-

operative PSI equally for complex and simple cases.

19

6. Conclusions

By combining Healthcare Information and Management Systems Society Analytics Database

with the National Inpatient Sample, we test how adoption of advanced electronic medical records (CPOE

and physician documentation) affect the incidence of patient safety indicators, and whether this effect

differs across the senior and non-senior population. We find a notable reduction in several of these health

measures, and on several occasions, a larger impact on the non-senior population. We also find that the

effect, when there, is generally increasing over time. The findings have important implications

concerning the impact of EMR adoption on health outcomes, particularly more advanced EMRs, which

allow for procedural recommendations, among other features. In future work, we plan to attempt to

establish the source of the (differential) effects we find, particularly by assessing whether the effect

differs by the dimensionality of the patient’s health issues (comorbidities).

20

REFERENCES

Agha, L. 2012. The Effects of Health Information Technology on the Costs and Quality of Medical Care. Working paper, Massachusetts Institute of Technology.

Athey, S and Stern, S. 2002. The Impact of Information Technology on Emergency Health Care Outcomes. Rand Journal of Economics 33: 399-432.

Bates, D., Leape, L., Cullen, D., Laird, N., Petersen, L., Teich, J., Burdick, E., Hickey, M., Kleefield, S., Shea, B., Vliet, M., and Seger, D. 1998. Effect of Computerized Physician Order Entry and a Team Intervention on Prevention of Serious Medication Errors. Journal of the American Medical Association 280: 1311-1316.

Bates, D., Teich, J., Lee, J., Seger, D., Kuperman, G., Ma’Luf, N., Boyle, D., and Leape, L. 1999. The Impact of Computerized Physician Order Entry on Medication Error Prevention. Journal of the American Medical Informatics Association 6: 313-321.

Cameron, A and Trivedi, P. 2005 Microeconometrics: Methods and Applications. 1st Ed., New York: Cambridge University Press.

Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojlca, W., Roth, E., Morton, S. and Shekelle, P. 2006. Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care. Annals of Internal Medicine 144: 742-752.

Congressional Budget Office. 2008. Evidences on the Costs and Benefits of Healthcare Information Technology. http://www.cbo.gov/ftpdocs/91xx/doc9168/maintext.3.1.shtml. Accessed on January 16, 2013.

Dranove, D., Forman, C., Goldfarb, A., and Greenstein, S. 2012. The Trillion Dollar Conundrum: Complementarities and Health Information Technology. NBER Working paper.

Hillestad, R., Bigelow, J., Bower, A., Girosi, F., Meili, R., Scoville, R., and Taylor, R. 2005. Can Electronic Medical Record Systems Transform Health Care? Potential Health Benefits, Savings, and Costs. Health Affairs 24: 1103-1117.

Kazley, A. and Ozcan, Y. 2008. Do Hospitals with Electronic Medical Records (EMRs) Provide Higher Quality Care? An Examination of Three Clinical Conditions. Medical Care Research and Review 65: 496-513.

Kolstad, J. and Kowalski, A. 2012. The Impact of Health Care Reform on Hospital and Preventative Care: Evidence from Massachusetts. Journal of Public Economics 96: 909-929.

McCullough, J., Casey, M., Moscovice, I., and Prasad, S. 2010. The Effect of Health Information Technology on Quality in U.S. Hospitals. Health Affairs 29: 647-654.

McCullough, J., Parente, S., and Town, R. 2013. Health Information Technology and Patient Outcomes: The Role of Organizational and Informational Complementarities. NBER working paper.

21

Miller, A. and Tucker, C. 2011. Can Health Information Technology Save Babies? Journal of Political Economy 119: 289-324.

Mold, J., Fryer, G., and Roberts, M. 2004. When Do Older Patients Change Primary Care Physicians? Journal of the American Board of Family Medicine 17: 453-460.

Parente, S. and McCullough, J. 2009. Health Information Technology and Patient Safety: Evidence from Panel Data. Health Affairs 28: 357-360.

Sidorov, J. 2006. It Ain’t Necessarily So: The Electronic Health Record and the Unlikely Prospect of Reducing Health Care Costs. Health Affairs 25: 1079-1085.

Wang, S., Bardon, C., Kittler, A., and Sussman, A. 2003. A Cost-Benefit Analysis of Electronic Medical Records in Primary Care. The American Journal of Medicine 114: 397-403.

Wang, Y. 2012. Cooperation and Competition: The Multilevel Adoption of Electronic Medical Records in U.S. Hospitals. Working paper, Boston University.

Zhan, C. and Miller, M. 2003. Excess Length of Stay, Charges, and Mortality Attributable to Medical Injuries During Hospitalization. Journal of the American Medical Association 290: 1868-1874.

22

Table 1: EMR adoption by year

Year CPOE Physician

Documentation 2003 0.070 - 2004 0.090 - 2005 0.165 0.182 2006 0.175 0.211 2007 0.198 0.234 2008 0.268 0.288 2009 0.304 0.391 2010 0.313 0.386

Notes: This table lists the fraction of hospitals with CPOE and Physician Documentation installed by year. The sample includes all hospitals for which we have at least two observations in the merged HIMSS and NIS data.

23

Table 2: Summary Statistics of Patient Safety Indicators

Deaths in Low Mort DRG Pressure Ulcer Hemorrhage or

Hematoma

Physiologic & Metabolic

Derangement Repertory Failure

Pulmonary Embolism or Deep Vein Thrombosis

One or More Postoperative PSI

PSI Eligible PSI Eligible PSI Eligible PSI Eligible PSI Eligible PSI Eligible PSI Eligible Panel A: All Patients Mean 0.73 2269.18 52.68 2358.73 6.37 2364.74 1.70 1378.79 11.46 1099.79 27.42 2370.59 43.06 2375.09 SD 1.28 2564.96 79.66 2643.78 9.95 2945.92 3.63 1808.35 16.94 1394.32 49.00 2951.31 69.31 2957.35 N 3288

3312

3213

2591

2586

3213

3213

Panel B: Non-Seniors Mean 0.32 2057.02 11.54 1080.64 3.66 1375.68 0.68 818.97 5.35 701.70 12.95 1379.72 20.81 1382.26

SD 0.89 2380.02 18.72 1433.89 6.34 1769.90 1.81 1100.98 920.67 9.37 25.76 1773.80 37.66 1777.54 N 3239

3268

3145

2530

2524

3145

3145

Panel C: Seniors Mean 0.42 246.13 41.75 1305.92 2.85 1039.95 1.06 593.64 6.41 426.71 15.07 1041.88 23.17 1043.61

SD 0.78 251.74 64.39 1370.91 4.29 1260.29 2.22 756.32 8.55 514.67 25.45 1261.99 34.52 1264.38 N 3242 3275 3142 2524 2511 3142 3143

Notes: This table presents summary statistics of each PSI measure. The unit of observation is a hospital-year. For each PSI, the first column represents the mean number of patients coded as experiencing the adverse event, and the second column is the mean number of patients in the relevant population that the PSI is calculated from.

24

Table 3: Effect of CPOE and Physician Documentation on Patient Safety Postoperative

Deaths in Low Mort DRG

Pressure Ulcer

Hemorrhage or

Hematoma

Physiologic & Metabolic Derangement

Repertory Failure

Pulmonary Embolism or Deep

Vein Thrombosis

One or More Postoperative

PSI PSI 2 PSI 3 PSI 9 PSI 10 PSI 11 PSI 12 Agg 9-12 CPOE 0.001 -0.032** -0.069*** -0.113*** -0.032** -0.079*** -0.063***

(0.058) (0.016) (0.015) (0.036) (0.016) (0.023) (0.016) N 2,131 3,147 2,671 1,469 2,241 2,862 2,932

Phys Doc 0.016 -0.090*** -0.058*** -0.191*** -0.012 0.016 0.005

(0.065) (0.017) (0.018) (0.047) (0.019) (0.011) (0.009) N 1,863 2,809 2,373 1,280 1,978 2,543 2,607

Notes: Coefficient estimates are from separate fixed effect Poisson regressions of PSIs on dummies indicating adoption of CPOE and Physician Documentation at the hospital-year level. All regressions control for hospital fixed effects, year fixed effects, time varying hospital characteristics, the number of patients for whom the PSI is relevant, and PSI specific risk adjustment controls. All standard errors are clustered at the hospital level. *** - p<.01, ** - p < .05, * - p < .10

25

Table 4: Differential Effects by Age Group Postoperative

Deaths in Low Mort DRG

Pressure Ulcer

Hemorrhage or

Hematoma

Physiologic & Metabolic Derangement

Repertory Failure

Pulmonary Embolism or Deep

Vein Thrombosis

One or More Postoperative

PSI PSI 2 PSI 3 PSI 9 PSI 10 PSI 11 PSI 12 Agg 9-12 CPOE -0.024 -0.021 -0.048*** -0.099*** -0.028* -0.053** -0.046***

(0.059) (0.015) (0.016) (0.036) (0.016) (0.022) (0.015) CPOE X -0.025 0.022* 0.002 -0.198*** -0.021 -0.026*** -0.025*** Non-Seniors (0.051) (0.011) (0.014) (0.033) (0.014) (0.009) (0.008) N 4,873 7,536 6,338 3,407 5,345 6,880 7,095

Phys Doc -0.071 -0.087*** -0.026 -0.159*** -0.007 0.033*** 0.023***

(0.066) (0.017) (0.018) (0.044) (0.019) (0.010) (0.008) Phys Doc X 0.106* 0.062*** -0.054*** -0.119*** -0.012 -0.026*** -0.040*** Non-Seniors (0.063) (0.011) (0.015) (0.032) (0.014) (0.009) (0.008) N 4,343 6,885 5,766 3,034 4,839 6,272 6,475

Notes: Coefficient estimates are from separate fixed effect Poisson regressions of PSIs on dummies indicating adoption of CPOE and Physician Documentation at the hospital-year-age level. All regressions control for hospital fixed effects, year fixed effects, non-senior dummy, time varying hospital characteristics, the number of patients for whom the PSI is relevant, and PSI specific risk adjustment controls. All standard errors are clustered at the hospital level. *** - p<.01, ** - p < .05, * - p < .10

26

Appendix Table 1: Descriptions of Patient Safety Indicators PSI Number PSI Name Description 2 Death in Low-Mortality Diagnosis

Related Groups Death when diagnoses included in list of diagnoses with less than .5% mortality rates

3 Pressure Ulcer A pressure ulcer is an area of skin that breaks down when something keeps rubbing or pressing against the skin

9 Postoperative Hemorrhage or Hematoma

Bleeding or bruising after operations

10 Postoperative Physiological and Metabolic Derangement

Metabolic derangement (deficiency in the amount of oxygen reaching body tissues) or other physiological complications that were not present before surgery

11 Postoperative Respiratory Failure Conditions that affect breathing function or the lungs themselves

12 Postoperative Pulmonary Embolism or Deep Vein Thrombosis

Blockage of the main artery of the lung or one of its branches by a substance that has travelled from elsewhere in the body or a blood clot in a deep vein

27