TRICARE Data Quality Training Course

September 6, 2006

Mr. Duane RutanTRICARE Management Activity (TMA)Executive Information Decision Support (EIDS)United States Department of Defense Military Health System

2

ObjectivesWhy data quality mattersHow our tools affect data qualityHow you can use this information in your data quality program

EIDS MHS centralized data storeReceive, analyze, process, and store 100+ terabytes of dataThousands of users worldwide

Overview

3

EIDS Architecture

4

MDR (MHS Data Repository)

Centralized data capture and validation of MHS data worldwide

More than 5 billion records on-line with 10+ years of data

Provides repository for other systems/applications to receive extracts

Typical users: small cadre of high-level data analysts

Military Health System Data Repository

Military Health System Data Repository

Military Health System Data Repository

5

M2 (MHS Management Analysis & Reporting Tool)

Powerful ad hoc query tool for detailed trend analysis such as patient and provider profiling

Typical users: Data analysts skilled in Business Objects software

6

EIDS Data Quality Requirements

Capture and catalog data filesAssess and monitor data completeness

• “Best Practice” is to perform real time Data Quality AssessmentsDevelop and implement data quality software that:

• Performs automatic data quality checks• Implements “run time” data quality assessments• Meets needs for daily DQ assessment• Provides metrics and manages perspective of the files’ data quality

“A thousand miles can lead so many ways…Just to know who is driving… what a help it would be” The Moody Blues

7

Source Data Quality

• Essential components of an emphasis on data quality1

• Performance of basic data analysis or profiling• Completeness• Value frequency distribution• Simple volumetrics• Outlier analysis of low or high occurrence values• Other reasonability analyses

[1] Larry English, “Mistakes to Avoid if Your Data Warehouse is to Deliver Quality Information”, DMReview, June 2002

8

EIDS Data Quality Metrics

Relevancy: is it appropriate?• Reliability: is it rationally correlated?

• e.g. Groupers/Regrouping• Validity: is it sound?

• e.g. Test Data

Consistency: is it free from contradiction?• Uniqueness: is it free from duplication?• De-duplication logic

Timeliness: is it available when needed?• IAW Batch Schedules and Process Control

Completeness: is it whole?• Extensive metrics at file, record, and field level

Accuracy: is it free from error?

9

DQ Guiding Principles

• Are data completeness evaluation processes “Out of Sync” with usage of the data?

• Process design rationale• Rationale: Encounter data is increasingly being used in real time for decision

making or alerting

• One way to classify healthcare data is according to timeliness requirements for its use.

• Daily: • If data is used daily, data completeness monitoring needs to have a daily

component

• Non-Daily: • Data completeness monitoring requirements are based on arbitrary reporting

cycles

• The ability to monitor for the absence of expected data is a key requirement

10

DQ Guiding Principles

• Does “processing” of data delay use for data completeness monitoring?

• Process design rationale• Rationale: Ingest, aggregation, processing, and loading of data into large

databases or extensive datamarts can delay availability of the data for completeness analysis.

• Significant processing delay before availability in an end user datamart or database is a justification for

• Ingest “run time” monitoring and alerting tools • e.g. Nothing arrived today where something should have arrived

• A “run-time” datamart or ODS fed with selected “data quality indicator” data elements.

11

DQ Best Practice

• “Operations” Best Practices

• For data that is ingested and used daily, best practice is to have processes to identify and correct data receipt problems on a daily basis.

• For data that has a cyclical processing cycle (e.g. weekly, monthly) operations best practice

• Will identify data that is not present PRIOR to processing

• Takes action to recover or repair the data gap or problem

• To caveat the processed data if data gaps or data problems are identified.

12

Determining Completeness

Root Cause Analysis:

Related to “Mechanics” or “Processes”

• Site related coding or other business process related issue• Software Interface Faults• WAN Connectivity• Host SW Faults• Processing Faults

13

Tools

Completeness Tool• What needs to be examined?

• Structure Checking• Identifies failed ingest (e.g. malformed or truncated files)• Both file and record level of detail

• Content Checking• Reasonability• Field Level

• Absence of the expected• A positive affirmation of nothingness• When reporting on the completeness of a set of data, how do you mark a data element as

missing if the record is not in the database? How do you represent “what did not happen” in the database?

“….There is no such thing as nothingness, and zero does not exist. Everything is something. Nothing is nothing. Man lives more by affirmation than by bread…”

Les Miserables, Victor Hugo 1862

14

Determining Data Completeness

Completeness Process Components• Monitoring

• Script based “monitors” (data feed monitoring tools)• Examination of volumetric

• Alerting• Based on process control or arbitrary thresholds• Alerts for “What is late” and “What is missing”• Alerting functions call upon benchmarks of current data against historical data

• Reporting• Requires a “History”

• A Data Quality/Completeness Datum art (or ODS)• Populated automatically in real time• Few selected data completeness measures for identified data types• Provides the history used by alerting functions

“There is in all visible things….a hidden wholeness.”

Thomas Merton

15

Real-time key data quality/completeness DB2 database for:SIDR SADR HL7 PDTS Appointment Ancillary

Database updated daily and scripted to provide “event-driven” alerts via e-mail for critical data quality areas

“Real time” and “snapshot” views of key data completeness measures by DMIS ID

Web and front-end access for standard reportsMultilayer data comparisons from raw to processed data for procedure-

based actionsStatistical process control algorithms and control charts to detect data

anomalies

Data Quality Tools

16

Tools: Data Tracker and the Portal

Essentially a “Mini MDR/M2” where data are processed in real timeData Tracker tools and reports

SIDR and SADR, HL7, Appointment, Ancillary, TED Inst/Non-Inst primary reports provide: • File based accounting (e.g. Gap reports)• Treatment based accounting (e.g. reports based on care date) • Timeliness reporting (e.g. lag from care rendered date to ingest)• Other statistical reports including benchmarking against WWR• Statistical Process Control Alerting for SADR anomalies

Other Data Tracker tools and reports • Monthly benchmarking reports (SIDR and SADR vs. WWR)• Ad Hoc Queries to the Data Tracker

Current Data Tracker reports on the EIDS Web site • Daily SADR by HOST DMIS (The “What Was Received Yesterday” Report)• Daily SADR by Treatment ID – 90 Day (The daily “90 Day Roller” Report) • Monthly SIDR by Treatment DMIS • Weekly HL7 Gaps

17

Location of data quality reports on EIDS Web portal.

EIDS Web Portal Resource

18

Data Quality Assurance

Facilities showing gaps in daily outpatient encounter data receipt. Investigation anddata recovery actions required.

Start with Run Charts

NAVCAMS EASTPAC WAHIAWA DMIS 0284

0

10

20

30

40

50

60

7/1/2005

7/8/2005

7/15/2005

7/22/2005

7/29/2005

8/5/2005

8/12/2005

8/19/2005

8/26/2005

Appointment

SADR

19

Data Completeness Determination

“Completeness” as a Process Control Problem

• Amenable to Statistical\Process Control

• Examine for Special Cause Variation

• Signals when a problem has occurred

• Detects variation • Allows “Process

Characterization” • Reduces inspection need

20

Compare Each Day To Itself

Red Boxes/X’s/etc indicate “Alerts” sent to DQ Team via automated email

Holiday Logic Pending

Chart: Encounters by Day

Project previousdata to today thencompare thisprojection withnewly arriveddata.

21

Identifying Data Completeness Problems

How do you identify and present“possible” problems when:• the “problem” is transient,• it is one data point in a series,• it is from one of a vast number of

daily input data sources?

Red boxes/Xs/etc. indicate automatic e-mail “Alerts” to the EIDS data quality team.

Essentially a projection of previous data forward in time to today then a comparison of this projection with the newly arrived data.

Alerting and Notification Issue

22

Examples

23

Data Tracker Report Series Example: SADR vs. Appointment Tracking (a real-time Benchmark Report ) SADR vs. Appointment Delta Alerting

24

• Daily Ancillary Data Report - 90 Day Roller

25

Web site with pull down listing of Standard Data

Quality and Data Completeness

Reports

26

The HL7 Weekly Tracker – Sorted by Service. Posted on the EIDS Web site and updated weekly from the Data Tracker database.

27

Data Quality Tools

Partial List of Standard Reports from the EIDS Web Portal DataTracker Database

• HL7 tracking: Displays a tabular view of file submission history for each HL7 site. • SADR gaps: Displays a list of sites, by ADS version, that did not report data for at least a fixed number of days• SADR lags: Displays the mean and standard deviation of the reporting lag for each site, by ADS version. • SADR scores: Displays a SADR transmission completeness report. For each site, by ADS version, a completion percentage is

listed. assumed.• SADR tracking: A tabular view of file and record submission history for each site, by ADS version. Each column corresponds to

a file date. • SADR treatment DMIS ID gaps: Displays a list of treatment DMIS IDs that did not report data for at least a fixed number of days. • SADR treatment DMIS ID scores: A SADR transmission completeness report. For each treatment DMIS ID, a completion

percentage is listed. • SADR treatment DMIS ID tracking: Displays a tabular view of record submission history for each treatment DMIS ID. • SADR treatment DMIS ID (by visit type) tracking: Displays a tabular view of record submission history for each treatment DMIS

ID. The displayed counts indicate the number of unique SADR data records, determined by appointment prefix and appointment identifier number.

• SIDR gaps: A list of reporting sites that did not report data for a fixed number of SIDR months, up to and including the ending SIDR month

• SIDR tracking: Displays a tabular view of file and record submission history for each reporting site. • SIDR treatment DMIS ID tracking: Displays a tabular view of SIDR completion history for each treatment DMIS ID.• GCPR gap: Displays a list of sites that did not report data for at least a fixed number of days.

28

Data Quality Tools (cont)

• GCPR sites: Displays a list of GCPR sites by Service, region, and DMIS ID, allowing the user to review the mapping of GCPR sites to DMIS IDs.

• GCPR tracking: Displays a tabular view of file submission history for each GCPR site. Each column corresponds to a date within the range specified.

• HL7 gap: Displays a list of sites that did not report data for at least a fixed number of days, as specified by the user query.• PDTS gap: Displays a line if PDTS data has not been reported for at least a fixed number of days, as specified by the user query.• PDTS tracking: Displays a tabular view of file submission history for PDTS. Each column corresponds to a file date within the range

specified.• Ancillary Tracking: Displays a tabular view of file and record submission history for each reporting DMIS ID. Each column

corresponds to a file date within the selected range.• Ancillary Gap: Displays a list of reporting DMIS IDs, that did not report data for at least a fixed number of days.• Ancillary treatment DMIS ID Tracking: Displays a tabular view of record submission history for each ancillary performing DMIS ID.

Each column corresponds to a service date within the range specified. The displayed counts indicate the number of unique ancillary data records, as determined by the accession number for laboratory, exam number for radiology, and prescription number for pharmacy.

• Ancillary treatment DMIS ID Gap: Displays a list of performing DMIS IDs that did not report data for at least a fixed number of days, as specified by days, up to and including the ending service date, as specified.

• Appointment treatment DMIS ID Tracking: Displays a tabular view of record submission history for each appointment treatment DMIS ID. Each column corresponds to an appointment date within the inclusive range specified by the beginning appointment date, bgndate, and the ending appointment date, end date. The displayed counts indicate the number of unique appointment data records, as determined by the appointment identifier number and the node seed name.

• Appointment treatment DMIS ID Gap: Displays a list of treatment DMIS IDs that did not report data for at least a fixed number of days, as specified by days, up to and including the ending appointment date, as specified.

29

Data Quality Tools

Allow EIDS to:

Catalog data filesMonitor data completenessProvide metrics to assess data quality/completeness of

data receivedDesign, develop and maintain data quality software

30

Partnering with our users to maximize information sharing

Questions?

The Key To Data Quality Success

31

Web Portal https://eids.ha.osd.mil account information product information

MHS Help Deskphone: 800-600-9332e-mail: eids@mhshelpdesk.com

Contact EIDS