Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
June 12, 2018
Setting the Standard:NPCR and SEER Join Forces to
Establish Data Quality BenchmarksSerban Negoita, Clara Lam, Rebecca Ehrenkranz, Amy Solis,
Reda Wilson, Manxia Wu, Vicki Benard
!2
NCI SRP and CDC CSB: Joint Project on Data Quality
Objectives:
▪ Minimize work duplication, optimize the use of resources
▪ Develop benchmarks to be applied consistently independent of the funding agency
▪ Create reference points to assess the effects of 2018 NAACCR standards implementation
▪ Develop infrastructure for real-time monitoring of data quality
!3
Benchmarks - Definition
▪ Benchmark: A slang or jargon term, usually meaning a measurement or point of reference taken at the beginning of a survey or project, used for comparison with subsequent measurements of the same variable; sometimes it means the best or most desirable value of the variable. Alternatively, an acceptable standard in evaluation (e.g., of air quality, of performance).
Dictionary of Epidemiology, 2014 edition
▪ ‘point of reference’ ‘most desirable’ ‘acceptable standard’
!4
Benchmark Development: Grade (Differentiation) and Special Grade Systems
▪ Grade data element suggested by the CDC NPCR based on the results of the NPCR quality audits
▪ Data element frequently used in research
▪ Reference points for grade necessary: significant changes in 2018 for Grade data collection standards ▪ See NAACCR Grade Coding Instructions and Tables Manual
!5
Phase I: Planning/ Protocol Development
▪ Select period of observation, tumor behavior, histology criteria, etc.
▪ Select a two-grade system tumor site: urothelial cancers
▪ Select a three-grade system tumor site: stomach
▪ Select a four-grade system tumor site: colon and rectum
▪ Select cancer type-specific grade variables: ▪ Gleason Score for prostate cancer (SSF 6,8,10)
▪ Bloom - Richardson Score for breast cancer (SSF 7)
!6
Pre-analytic: SEER Program Coding and Staging Manual - Grade History Timeline
SEER Program Coding and Staging Manual: Grade History Timeline: Gleason Score, Prostate
2001 2002 2003200
4 2005200
6200
7 2008200
9201
0201
1201
2201
3 2014 2015 2016
Gleason's Score Grading Gleason's Score GradingSEER Value
Gleason's Score Grading
SEER Value
2--4I, well
differentiated 2--4 I, well differentiated 1 2--6 I, well differentiated 1
5--7II, moderately differentiated 5,6
II, moderately differentiated 2 7 II, moderately
differentiated 2
8--10III, poorly
differentiated 7--10 III, poorly differentiated 3 8--10 III, poorly differentiated 3
!7
Pre-analytic Phase: Collaborative Stage - Grade History Timeline
Collaborative Stage: Grade History Timeline: Gleason Score, Prostate
2001 2002 2003200
4 2005200
6200
7 2008200
9201
0201
1201
2201
3 2014 2015 2016
Collaborative Stage did not exist
SSF 6: Gleason's Score SSF 8: Gleason's Score on Needle Core Biopsy/TURPCode Description Code Description
0 Test not done 998No needle core biopsy/TURP
performed2 -- 10 Gleason's score 2--10 Gleason's score
988 - 999 NA/Unk 988, 999 NA/UnkSSF 10: Gleason's Score on Prostatectomy/Autopsy
Code Description998 No prostatectomy/autopsy performed
2--10 Gleason's score988,999 NA/Unk
!8
Pre-analytic Phase: AJCC - Grade History Timeline
AJCC: Grade History Timeline: Gleason Score, Prostate
2001-2002 2003200
4 2005200
6200
7 2008200
9201
0201
1201
2201
3 2014 2015 2016AJCC 5 AJCC 6 AJCC 7Gleason's Score (Description) Gleason's Score Description Gleason's Score Description2--4 (well differentiated) 2--4
Well differentiated/slight anaplasia
1 -- 6
Well differentiated/slight anaplasia5,6 (moderately differentiated) 5--6
Moderately differentiated/moderate anaplasia
7 (moderately poorly differentiated) 7--10
Poorly differentiated/undifferentiated/marked
anaplasia7
Moderately differentiated/moderate anaplasia
8--10 (poorly differentiated) 8--10
Poorly differentiated/undifferentiated/marked anaplasia
X Grade cannot be assessed X Gleason score cannot be processed
!9
Phase II: Pre-analytic – search for grade collection data quality standards
▪ Verify if benchmarks for grade collection available: ▪ NAACCR members
▪ IARC
▪ European cancer registries
▪ Lit research grade distribution reported by clinical trials and observational studies ▪ Gleason Score ranges across all studies in lit review: n>1,000
▪ GS 2- 6: 29 - 63%
▪ GS 3+4 = 7: 10 - 56%
▪ GS 4+3 = 7: 3 - 25%
▪ GS 8 -10: 1 - 20%
!10
Phase III: Analytic – Outliers Detection for Gleason Score
▪ Selection ▪ Analytic dataset provided by CDC team ▪ Includes SEER and NPCR funded registries, 2001-2016
▪ Gleason score analysis restricted to dx. years 2010-2016
▪ Selection criteria: prostate cancer, positive histologic confirmation
▪ Gleason score post-prostatectomy: surgery code 30-80 (confirming prostatectomy)
▪ Indicators (proportions) of interest: ▪ % cases Gleason Score Unknown (999)
▪ % cases Low Gleason Score 2-6 (prognostic group 1, least aggressive)
▪ % cases High Gleason Score 9-10 (prognostic group 5, most aggressive)
!11
Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry
SSF 10, Gleason Score = 999, Dx. Years 2010-2015
0%
3%
6%
9%
12%
T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A
!12
Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry (n = 36)
SSF 10, Gleason Score = 999, Dx. Years 2010-2015
0%
3%
6%
9%
12%
T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A
Q1: 25th
percentile
Q3: 75th
percentile
!13
Benchmarking Method: IQR Rules for Outliers
▪ Benchmarking method: Fences at Q1-1.5*IQR and Q3+1.5*IQR
% Gleason Score Unknown (SSF 10 = 999)
Q1 (25th percentil
e)Q3 (75th
percentile) IQRLower Fence Upper Fence
1.3% 2.6% 1.4% -0.8% 4.7%
% Low Gleason Score (SSF 10 = 2-6)25
Percentile
75 Percentile IQR
Lower Fence Upper Fence
21.4% 29.2% 7.8% 9.6% 41.0%
% High Gleason Score (SSF 10 = 9-10)25
Percentile
75 Percentile IQR
Lower Fence Upper Fence
5.9% 7.9% 2.0% 2.9% 10.9%
!14
Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry
SSF 10, Gleason Score = 999, Dx. Years 2010-2015
0%
3%
6%
9%
12%
T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A
!15
Proportion Unknown Gleason Score Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year
SSF 10, Gleason Score = 999, Dx. Years 2010-2015
0%
3%
6%
9%
12%
T II L G H GG N M AA B D O CC E DD JJ U J Z S W K HH P EE I F FF R X Y C V Q BB A
!16
Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry
SSF 10, Gleason Score 002-006, Dx. Years 2010 - 2015
0%
11%
23%
34%
45%
K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C
!17
Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year
SSF 10, Gleason Score 002-006, Dx. Years 2010 - 2015
0%
11%
23%
34%
45%
K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C
!18
Proportion High Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry
SSF 10, Gleason Score 009-010, Dx. Years 2010 - 2015
0%
5%
10%
15%
20%
C S EE DD Q J HH X W G AA N I GG Z U M F II D P B L R JJ CC Y T K FF E H BB O A V
!19
Proportion High Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry and Diagnosis Year
SSF 10, Gleason Score 009-010, Dx. Years 2010 - 2015
0%
5%
10%
15%
20%
C S EE DD Q J HH X W G AA N I GG Z U M F II D P B L R JJ CC Y T K FF E H BB O A V
SSF 10, Gleason Score 009-010, Dx. Year 2016
0%
5%
10%
15%
20%
!20
Conclusions
▪ Large number of registries is needed to develop valid benchmarks
▪ Pre-analytic phase is important: ▪ Subject mater expertise necessary to select the right targets (indicators)
▪ Lit. research, research of international standards – limited return
▪ Numerous statistical methods available – additional research necessary to select the optimal method ▪ IQR-based fence method – relatively easy to implement
▪ Immediate applicability – it identified outliers in the distribution of Gleason Score for cases diagnosed in 2016
▪ Collaboration really helped, minimized effort to obtain datasets and history timelines
▪ Feasible, inexpensive method that resulted in reference points for assessing data completeness and frequency distribution Gleason Score
!21
Many Thanks to:
NCI Division of Cancer Control and Population Sciences Leadership
CDC Division of Cancer Control Leadership
Analysts at CDC and IMS:
Jessica King (CDC)
Denise Duran(CDC)
MaryBeth Freeman(CDC)
Jennifer Stevens (IMS) Registrars and all those
contributing to cancer surveillance
www.cancer.gov www.cancer.gov/espanol
!23
Benchmarks in Cancer Surveillance
Cancer system outcomes = cancer data sets Measurable characteristics of the cancer data sets Missing data (% unknown) Precision (% NOS) Validity (trueness) : re-abstraction/ re-coding : observed vs expected distribution
!24
Quality Indicators
▪ Current data quality indicators evaluate: ▪ Processes ▪ Case ascertainment - completeness
▪ Death clearance - DCO rate
▪ De-duplication - duplicate rate
▪ Data editing - edits-failing cases
▪ Quality of output (data sets, data elements) ▪ Sex, age, race, county at dx – all about demographics
▪ Both important, process indicators difficult to interpret and use ▪ No quality indicators for clinically-relevant data elements!
!25
Population-Based High-Quality Clinically-Relevant Data
▪ Reference points (benchmarks) ▪ define quality
▪ Methods to compare against reference points (benchmarking) ▪ Measure and track improvement
▪ Infrastructure/ institutional support (standards setters) ▪ Select the reference point and measurement methods
▪ Integrate tracking/monitoring in registry operations
▪ Use benchmarks to promote population-based cancer data sets
!26
Institutional support needed to develop benchmarks for cancer surveillance
!27
Providing High-Quality Cancer Data
▪ How does the cancer surveillance community: ▪ Defines high-quality?
▪ Measures improvement toward high-quality?
▪ Makes optimal resource allocation decisions to achieve high-quality?
▪ How to inform cancer data users on the high-quality of population-based surveillance data ▪ Differentiate our data products from other data sources
▪ Substantiate the need for additional resources
▪ Accomplish cancer control and research mission
!28
Proportion Low Gleason Score, Among Tumors with Confirmed Prostatectomy Specimen, by Central Registry
SSF 10, Gleason Score 002-006, Dx. Years 2010 - 2015
0%
11%
23%
34%
45%
K V R F FF O W T II Z DD CC E A D GG L S EE Q AA B N JJ U H M J Y HH X P I BB G C
Lowest % based on lit. research
!29
Complexity for Benchmark Selection
▪ Select reference points and measurement methods
▪ Prioritization decisions on data items that need benchmarks ▪ Treatment, stage, prognostic factors, cancer identification?
▪ Requires research, analysis, interpretation of existing data and publications
▪ Are there any existing benchmarks use by registries or oncology organizations, domestically or internationally?
▪ Statistical methods:
▪ Outlier detection
▪ Trend-based forecasting
▪ Measures of dispersion
▪ Observed versus expected categorical tests
▪ Other?
!30
Methods for Benchmark Selection
▪ Use of external datasets to select benchmarks: ▪ Administrative/Claims data for treatments
▪ Patterns of care: treatment, stage, prognostic factors
▪ Re-abstraction/recoding studies: ▪ Manual re-abstraction/re-coding ▪ Large sample size
▪ Multiple registrars
▪ Adjudications of best answers
▪ Natural Language Processing used for re-abstractions, automated re-coding/re-consolidation ▪ NLP algorithms need validation
!31
Project Timeline
▪ NCI-CDC team meetings started December 2017
▪ Multiphase project ▪ Planning/ Protocol development
▪ Pre-Analytic Assessment
▪ Analysis – evaluation of statistical methods ▪ Detection of outliers
▪ Trend-based forecasting
▪ Interpretation and recommendations
▪ Implementation in program activities ▪ Implementation in registry systems dashboards