Heather Kaplan, MD, MSCE - Vermont Oxford Kaplan, MD, MSCE April 5, 2016 1 ... Congenital Heart Disease Screening ... • Do not include jargons or acronyms

Measurement 201:Planning and Executing Advanced Measurement

Heather Kaplan, MD, MSCE

April 5, 2016 1



VON NICQ Next2 Jacksonville Meeting

Disclosures

• I have nothing to disclose relevant to the content of this workshop



April 5, 2016 2

Objectives

• Recognize instances where audits and tallies are insufficient for measurement and more advanced strategies are needed

• Identify approaches to using existing electronic data sources to streamline data collection

• Gain a basic understanding of the fundamental requirements for survey development

• Identify any measures for your current project that utilize an advanced data collection strategy (electronic or survey data) and develop a workplan & timeline to acquire that data

The Model for Improvement

• 3‐4 Key Process Measures• Main “Outcome” Measure

(SMART Aim)



April 5, 2016 3

For your NICQ Next2 Project….

•Your key 5‐6 performance measures including:•SMART Aim (“outcome”) measure•Process measures…3‐4 measures of your key PBPs

Simple Data Strategies

• Random Audits and Tallies are useful when:• Data on entire population either easy to obtain or not needed• Measures have complex specifications (inclusion/exclusion criteria, patient‐specific protocols)

• Data available within regular workflow• Measure definition may change over course of project

• QI team has adequate resources to collect data and calculate measures



April 5, 2016 4

Electronic Data Sources

Electronic Data Sources

• Advantages• Can examine every instance• Once the measure is operationalized, will continue to populate with minimal effort• Minimizes human error in calculating measure values

• Limitations• Refinements difficult to operationalize (extensive coding to replicate human reasoning)

• Inclusions and exclusions may be difficult to operationalize• Samples are more costly than examining everyone

• Resource considerations (“who writes the code?”)• You may need to rely on someone in another department who may

• Not know the process (requires extensive clarification)• May have competing priorities• May not be available for revisions



April 5, 2016 5

What kinds of data & resources to use

Applicability of Measure

Measure’s “life cycle” stage

New, experimental, under development, Operational Definition changes with new

learningEstablished, widely used, Operational

Definition seldom changes

Local, specific.In‐depth knowledge of particular process required.

Manual data collection with local, site‐specific resources, familiar with local process.

Electronic, preferably developed & produced with local resources familiar with site‐specific electronic data

Widespread, generic across areas of organization.

EITHER: Manual data collection in one pilot area, with spread to follow,

OR: Electronic generation thru simple Clarity report, to demonstrate proof of concept Electronic, with institutional support

Data from the EHR

•Create note templates to enable manual data collection

•Reports from the EHR (e.g., Clarity Report from Epic)• Get needed fields to calculate measure for entire population

• Get name/MRN for a particular population to facilitate chart review



April 5, 2016 6

Congenital Heart Disease Screening

• Report in excel for the entire newborn population• Data available in report to calculate % infants failing CHD screen and also % where screen completed

• Reports used for:• Quality Improvement

• Communication with Cardiology to ensure infants have follow up Echo

• Communication with birth registrar to ensure data is captured accurately on the birth certificate



April 5, 2016 7

Neonatal Abstinence Syndrome

• Report in excel of any infant who had a Finnegan score entered

• More accurate than using Problem List at our hospital

• Report used to facilitate manual chart review for Ohio Perinatal Quality Collaborative (traditional manual data collection)



April 5, 2016 8

Tips for successfully getting EHR Data

• Plan for a 6‐10 week turn around time• May need to build flowsheet or add to existing flowsheet before creating a report

• Have a specific point person from IS and also from your unit

• Link your need to a strategic priority of the organization• Make it easy for the person in IS by:

• Giving them a template excel file

• Specify your desired variables with row ID (code to help IS find the specific field you want the data from)



April 5, 2016 9



April 5, 2016 10

Congenital Heart Disease Screen #1 Congenital Heart Disease Screen #2MRN Name DOB date/time pre ductal post ductal pass/fail date/time pre ductal post ductal pass/fail

1636000053 1631100072 1631100073 1631100098 1636000054 1631100075 1631100076 163110099

Row ID from flowsheet that tells IS how to find the data in Epic

Provide IS a Template for How to Give You Data

How to Find Row ID’s from a Flowsheet in Epic

1. Click on the word Epic in the upper left corner of the screen2. Go to Help and then to Help Desk Reports3. Click on "Report and Print Group Assistant". It will say that the Assistant

is now enabled.4. Then go into a patient chart and into the flowsheet that you are

interested in.5. Open the side panel that is says "show/hide docked detail report" when

you hover over the side tab arrow. 6. Find the flow row that you are interested in7. In the side panel that you opened, scroll down until you find something

that says "rowID". It is sometimes in the section titled "Debug Print Group“



April 5, 2016 11

Existing Internal Data Sources

• Adverse events, incident reports, senitnel events• Infection rates, isolates• Length of stay• Publicly reported measures (Joint Commission, Medicaid HEDIS)• Service utilization data including diagnostics, pharmacy, specific procedures• Billing data• Customer complaints• Expenditure reports• Patient satisfaction• Use of high‐cost medication items

Hybrid Clinical Data and Census ReportingExample: NEC QI in Cincinnati

• Clinical Data (Cases of NEC—Numerator)

• Unit dietitian collects name of each case of NEC at 3 NICUs

• Cases reviewed at regional task force

• Billing/Census Data (VLBW patient days—Denominator)

• NICU financial analyst counts number of VLBW infants by day if charge for services is recorded based on BW billing codes



April 5, 2016 12

Hybrid Clinical Data and Census ReportingExample: NEC QI in Cincinnati

• Benefits of collecting as cases/VLBW days• Available in real‐time

• Can track temporally, instead of counting at discharge

• Allows for multiple cases of NEC in a single infant

• Adjusting for patient‐days accounts for differential opportunity for outcome based on census

• Minimal burden of data collection

Data from Devices

• Pulse oximeter and monitor data• Alarm Frequency

• Compliance with unit alarm limits

• Percent time in goal O2 Saturation range

• Percent time with O2Saturation <80%

• Frequency of apnea or bradycardia

• Availability and process for accessing data depend on the type of monitor and your organizational structure

• To access data from medical devices, discuss with biomedical engineering and IS at your hospital



April 5, 2016 13

How Can the Charter Help?

Ask senior leaders for support with data:•Can they help identify what is already being collected that can be fed back to you?•Can they assist you in interactions with IS?

Surveys



April 5, 2016 14

When to Use Surveys

• Surveys are typically used in QI to assess attitudes and behaviors of staff or families

• Surveys are not easy to develop, use existing surveys (that have been validated) when possible

• Make sure you understand how the survey was originally used (are single items appropriate, are full scales needed, etc.)

• If you have to use your own survey:• Get help from someone with experience and training in developing surveys

• Test the survey on a small group of your intended audience to identify problems with interpretation and understanding

Examples of Validated Surveys in NICU QI

• NAS Attitudes Survey

• Safety Attitudes Questionnaire

• Maslach Burnout Inventory

• How’s Your Baby Survey

• “The Parent Feedback Questionnaire”

• Institute of Patient & Family Centered Care Self‐Assessment Inventory



April 5, 2016 15

Tips for Developing Your Own Survey• Make sure why you are collecting the data is clear to your audience

• Use “good” survey questions that:• Are specific• Are easy to read and understand• Ask for knowledge or opinion, NOT both• Are not loaded or leading• Do not ask more than one question (double‐barreled)

• Do not include jargons or acronyms

• Provide reasonable ranges of variation in response options• Are unlikely to elicit socially desirable answers

Examples of bad survey questions

• Leading question• Hand washing is critical for infection prevention, to what extent do you feel it is your responsibility to ensure all providers perform hand hygiene before and after patient contact?

• Do you think our new process for medication reconciliation is more efficient?

• Double‐barreled question• How happy were you with the communication of the doctors and nurses at this hospital?

• NICU leadership could improve communication by soliciting nursing input at staff meetings (Strongly Agree to Strongly Disagree)

• Social desirability risk• You have adequate resources to care for your baby after discharge (Strongly Agree to Strongly Disagree)

• How worried were you when you brought your baby home from the NICU?



April 5, 2016 16

Tips for Developing Your Own Survey

• Surveys that do not give valid or reliable data could cause you to make false conclusions about changes leading to improvement

• Questions must remain the same over repeated assessments

• Questions must be well‐written and clear

• Survey must be administered in the same way over repeated assessments

• There should be a consistent interval for repeated assessment based on your knowledge of when changes should be seen

Exercise



April 5, 2016 17

Exercise

• Select one measure for your project and begin the process of operationalizing it using the worksheet

• Focus particularly on:• What data is needed for your measure?

• Where will you obtain the data (data source)?

• Who will you speak to in IT/IS? Finance?

• Developing a timeline for survey development (if applicable)

• Developing a timeline for getting electronic data (if applicable)• Meet with data managers and IS Staff (ASAP)

• Make edits to flowsheets if needed

• Draft mock measures report

• Complete prioritization and request paperwork (at least 6‐10 weeks before data needed)

Summary

• Advanced data collection strategies including use of surveys and electronic data sources can supplement standard manual data collection in QI

• Use of advanced strategies requires significant up front planning and additional resources, but can be time‐saving in the end

• Electronic data is best if you need to examine the entire population or if you need to automate manual data collection of measures that have already been developed and tested

• Think creatively about using existing data sources to supplement manual data collection

• Surveys are useful when you need to understand the attitudes and behaviors of individuals. It is best to use already validated surveys, however if you need to develop your own, engage survey development experts and ensure adequate time to pilot test



April 5, 2016 18

Questions?

Documents

Heather Kaplan, MD, MSCE - Vermont Oxford Kaplan, MD, MSCE April 5, 2016 1 ... Congenital Heart Disease Screening ... • Do not include jargons or acronyms