MEASUREMENT and DATA COLLECTION

As we have noted in our prior classes, and will keep repeating until you can’t stand to hear it any more, a focus of Quality Improvement involves eliminating unwarranted variation. Eliminating variation requires

identifying unwarranted variation when it occurs, determining the cause of this variation, changing the way care is delivered (that is changing the process) in order to eliminate the

variation and then determining if the change resulted in an improvement.

Before jumping into data collection, we will spend a few minutes talking about the general principles of Quality Management to provide further context for our discussion.

The basic, relatively simple idea that underlies quality management is collecting information on the issue that concerns us. That seems pretty straightforward doesn’t it? You may recall the famous scene in the movie, Apollo 13, when Gene Krantz warns the staff in mission control:

And yet, oftentimes staff approach improvement with a series of guesses.

So instead of guessing and making changes based on those guesses, quality management involves choosing appropriate measures and measuring them correctly.

This obviously requires choosing the right tools to get an accurate measure that meets the needs of what is being measured. For example, the above shows a measuring cup used to measure flour for a cake recipe. How helpful would a tape measure be for this purpose?

Probably not very. So again, selecting measures involves defining a measure that describes the quality issue to be assessed and determining how to measure it correctly.

Obviously then, measurement is the place where quality management activities begin. As you can see, this approach relies on the collection of data that can be turned into information that can be used to drive improvement. The decisions we make about how well our system is performing and how are improvement efforts are working depend on the quality of data that we use. It is a classic, “Garbage in, garbage out” problem.

This week we will focus on the sources of data outlined in the textbook and also consider some questions that need to be addressed before we begin collecting data as outlined in Robert Lloyd’s video from IHI. You will note that the IHI uses the Model for Improvement as its Quality Improvement Model, however, the points raised in his lecture will work no matter what quality improvement model you use. In this course we refer frequently to the IHI model for improvement because this approach focuses a lot on the thought processes associated with quality improvement. It is essential to understand the rationale for using the methods and tools we are applying.

Let’s begin by understanding a very important point about data in general. Data collection involves an important trade-off.

It costs both time and money to collect data, so it is important to note that in any quality improvement project the aim is to collect the required amount of relevant data for the minimum cost. Improvement teams want to focus on collecting “just enough” data that is required to identify and manage quality issues and avoid collect unnecessary or irrelevant data.

The first question we must think about is what kind of information is helpful when it comes to quality improvement work?

Let’s think back to the definition of Quality Care as defined by the IOM, that is the STEEEP criteria—safe, timely, efficient, effective, equitable, and patient-centered. Our measurements should, then, provide us information on these areas. Therefore, categories of data include:

Clinical quality (including both process and outcome measures) Financial performance Patient, physician, and staff satisfaction Functional status

Before collecting data, a series of questions should be considered:

1.) Why are we doing this project? Are we at the initial stage of getting baseline data? Have we found a problem we are trying to improve something? Are we doing this as part of an internal project or are we trying to satisfy external regulators who require certain information?

2.) Will the measurement we have selected help us identify the improvement that is consistent with our project?

3.) What is the time frame for the measure? Will this be a one-time study? Is this a required metric that will need to be monitored indefinitely at regular intervals?

4.) What category of data will provide sufficient information for the least cost? How complete will the data be? How accurate? Should the data be retrospective or real time?

5.) Are there “unintended consequences” associated with the measures we have chosen? Can we collect “balancing measures” to be sure we are not creating new problems as we begin a quality project?

ACCURACY IN MEASUREMENT

We want to make sure that the data we collect is accurate and useful. Accuracy asks “did we get it right? Are our numbers correct? For example, are the numbers we are reporting correctly describe the wait time patients experienced in a waiting room. An aspect of evaluating if numbers are accurate, is to ask:

Are our measurements reliable? - are we measuring them the same way every time? Are our measures valid? - do they measure the item that is relevant to the issue we are trying

to solve?

For example, we might want to understand if patients thought the providers were empathic. We might develop a patient survey that asks if the care was good. However, every patient who reads the survey might have a different understanding of “good care.” Some might think about wait times, others might think of how the food tasted, others the pleasantness of the nurses. This measure might not produce the same results each time, and, since patients may interpret the meaning differently, it is not really a reliable or valid measure of provider empathy.

It should be also noted that in addition to looking at measures from a structure, process and outcome perspective as we noted in our first week- we can also look at based on where they occur. For example, system-level measures report on the overall performance that is made up of multiple, interdependent processes. But we can also measure what is happening at the level of one process, perhaps within a microsystem. How long does it take for a nurse to respond to a call light, for instance? This is known as an activity-level measure. For the first part of Unit 2, we will think about measurement at the activity-level. Later in Unit 2 and in Unit 3 we will consider how those measures combine to create system level measures and how that data might be of use to people in leadership.

STRATICATION AND SAMPLING

Finally, prior to collecting data, we need to consider 2 aspects related to data collection.

The first aspect is the issue of stratification and the second is the issue of sampling.

Stratification refers to the recognition of confounders- or aspects of a process that might have a hidden impact on an outcome. Stratification requires that someone familiar with a process identify aspects that might influence that process before the data is collected. An example of a confounder might be the impact on the results if extremely ill patients were included in a measure of compliance with preventative screening tests. It might not be appropriate to obtain screening tests on patients with a terminal illness. One might want to understand how including these patients in the data collection might affect the accuracy of the data. Robert Lloyd will discuss other potential confounders in this week’s video lecture.

The second issue to address prior to begin collecting data is the issue of sampling. For some improvement projects, the data set is so large that it would be impossible to monitor the entire set. In this instance, a “sample” must be taken to make the project feasible. The goal of sampling is to choose a sample that gives an accurate representation of the true data set. Sampling bias is frequently encountered in patient satisfaction surveys since it is often the dissatisfied patients who respond to surveys—and hearing the results of their feedback could make organizations believe they are doing worse than they actually are.

A common approach in Quality Improvement work is to use judgement data. This differs from the sampling techniques based on probability that is typically used in research, again because the goal of data collection in quality improvement is to get “just enough” data. Judgment sampling relies on the opinions of someone who is an expert in the process that one is looking to improve. As an expert, this individual is able to identify the type of data to collect. For example, if one was interested in learning the opinion mountain climbers had about the care received in a hospital sports clinic, it would make sense to focus data collection on mountain climbers rather than get a broad sample from the general population. Of course, judgement sampling does have the possibility of bias, but that risk is overshadowed by the benefits of the approach.

Consider this story from the IHI.

The Sunnybrook Health Sciences Centre in Toronto, Ontario, Canada participated in the Institute for Healthcare Improvement's (IHI) Improving Asthma Care Breakthrough Series Collaborative in 1995. The team sought some baseline data regarding the age distribution of patients visiting the Emergency Department (ED) for asthma care. So, they put in a formal request to their Information Systems (IS) department for a report with this information. And then they waited and waited and waited and waited

Impatient that the request was taking too long, one team member, an emergency doctor, decided to take the matter into his own hands, literally. This physician went through the charts for all ED patients in a three-month period, and manually noted the ages of asthma patients. His informal research provided a sampling of 94 patients, with an age distribution as shown in the graphs below.

The team had enough confidence in this data that it decided to proceed with its improvement agenda.

Sure enough, when the IS department finally delivered the report nearly two months later, it confirmed the team's instincts. With a sample more than four times the informal sample taken two months earlier, the results were nearly identical:

This team would have learned nothing more by waiting for more data. They saved valuable time by sampling and depending on their own sound judgment to know when enough data was enough.

UNINTENDED CONSEQUENCES OF QUALITY MEASURES

Finally, it is important to recognize that when we chose a measure—and reward people for meeting that measure (as we see with KPI’s – key performance indicators)- the result may be that we encourage behaviors that we did not anticipate. A classic example of this is hospitals that try to lower their length of stay. If they do this by improving their processes and eliminate waste, the quality improvement effort can be thought to be successful. However, if the staff, feeling pressure to discharge patients do not evaluate and eliminate waste in their process and, instead, just discharge patients before they are ready, it is possible patients might be discharged but readmitted the next day. Unintended consequences can be detected by balancing measures. The “rule of thumb” to describe this situation is, “whenever you solve one problem, always think about what new problem you might have created.”

THE QUALITY MANAGEMENT CYCLE

After measurements have been obtained, the next step is to assess the results. These assessments are used to evaluate performance—that is has an acceptable level of quality been achieved? Again, the answer to this depends on what you have set as your target, but it also assumes you have chosen the right measure, collected it correctly and accurately, and documented the results accurately.

Let’s, for the moment, assume we have done everything correctly. When we assess our performance, we then compare it to our target. Did we meet the target? Yes? Rah! Now we can continue to monitor our performance—follow the trends and make sure our performance remains stable.

But suppose we did not meet our performance goals? ☹ Well the next step would be to figure out how we do so. That is, how do we improve? This repetition of measurement, assessment, monitoring/improving is known as the quality management cycle which is shown below.

SOURCES OF DATA

Let us now look at some common sources of data that are used in Quality Improvement Initiatives.

• Administrative databases

• This data source includes billing systems, claims data, hospital registration systems.• This data source gives information on types and location of service, units of service

(length of stay in a hospital for example), diagnosis and procedure codes, amounts billed, and amounts reimbursed

• Advantages of Administrative Data• Available electronically.• Less expensive than conducting medical record review.• Can provide information for large groups of patients.• Use CPT and ICD coding which provides standard information for comparisons.

• Limitations of Administrative Data• Incomplete clinical information.• Accuracy is questionable as the main purpose of the data is billing. • Timeliness

• Medical record review (retrospective)

• This data source is the record of individual patient care • This data source provides detailed information on the care rendered to a specific patient

including the rational for care, the time of care, etc. • Advantage of Medical Record Review

• Clinical detail is present• Information obtained from medical record has more credibility with providers

• Limitations of Medical Record Review• The cost and time to compile information can be large, less so with electronic

health records than with paper records• Comparisons can be difficult if different forms of documentation are used

• Prospective data collection, data collection forms, and scanners

• “Real time” medical record review• Advantage of Prospective Data Collection

• Similar to advantages to retrospective medical review, but “real time” analysis allows for early detection of negative tends

• Limitations of Prospective Data Collection• Expensive (expense can be lessened by careful design of forms, use of

technology and matching data collection to work- flow

• Patient surveys: satisfaction and functional status

• Information from the patient about their experience including care received, services, impression of outcomes, and other areas where patient is considered the best source of information. Surveys typically done by mail, by telephone, or via the Internet.

• Advantage of Patient Survey

• First- hand information from people who received the care.• Well-designed survey captures the issues of concerns to patients.

• Limitations of Patient Survey• Requires validation of questions. Pre-validated surveys can be expensive• Subject to sampling bias (accurate representative sample of patients not

obtained)• Subject to response bias (inaccurate conclusions because of lack of response)

• Functional status surveys

• Assessments of patient ability to perform tasks, participate in activities of daily living, social interactions, presence of pain, etc.--generally before and after treatment. Generally, use standardized, validated forms. Collected by mail, telephone or internet

• Advantages of Functional Status Surveys• Standardized, often include physiologic measures

• Helpful in determining the cost-effectiveness of certain therapies• Limitations of Functional Status Surveys

• Require validated tools• Poor response rate in elderly patients

• Health plan databases

• Collected from billing data for services rendered to patients• Advantages of Health Plan Data Bases

• Provides information for population management studies• Can be used to track use of preventative services• Can identify patients who would benefit from disease management programs• Helpful in monitoring care of patient across the continuum of care

• Limitations of Health Plan Data Bases• Similar to the limitations for administrative data—this information is primarily

used for billing so there are potential concerns about accuracy and timeliness• Information on outcome of testing is not available

So, the activities for this week include reading the text to learn more about sources of data and then watch Robert Lloyd’s video on things to consider when collecting data. In addition, Dr Lloyd will again discuss the issue of variation. You can never get too much discussion about variation! 😊

For the class activity this week, you will be asked to observe a process that deteriorates over time. See if you can detect why. What type of variation did you identify? What data would you collect to determine if you are correct?