BSERVATIONAL Some Recent OMOP Research …2013/11/12 · Database in the OmopCommon Data Model for Active Drug Safety Surveillance. Drug Safety: 1 -16. DOI: 10.1007/s40264-012-0009-3

OBSERVATIONAL MEDICAL OUTCOMES PARTNERSHIP


Some Recent OMOP Research Results

William DuMouchel on behalf of the OMOP research team

MidWest Biopharmaceutical Statistics Workshop

Muncie, IN 21 May 2013


2

• Overview of OMOP publications of the last year with focus on three papers

• Explanation and Example of OMOP Proposal: An Empirical Approach to Measuring and Calibrating for Error in Observational Analyses

Agenda


3

OMOP Publications (2012-2013)

• Madigan D, Ryan PB, Schuemie M, Stang PE, Overhage JM, Hartzema AG, et al. Evaluating the Impact of Database Heterogeneity on Observational Study Results. American journal of epidemiology. May 5, 2013.

• Fox BI, Hollingsworth JC, Gray MD, Hollingsworth ML, Gao J, Hansen RA. Developing an expert panel process to refine health outcome definitions in observational data. Journal of Biomedical Informatics. 2013: In press.

• Ryan PB, Suchard MA, Schuemie M & Madigan D (2013): Learning from Epidemiology: Interpreting Observational Database Studies for the Effects of Medical Products, Statistics in Biopharmaceutical Research, DOI:10.1080/19466315.2013.791638

• Madigan D, Ryan PB, Schuemie M. Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies. Therapeutic Advances in Drug Safety. February 25, 2013 2013.

• Hansen RA, Gray M, Fox BI, Hollingsworth J, Gao J, Hollingsworth M, Carpenter DM. Expert panel assessment of acute liver injury identification in observational data. Research in Social and Administrative Pharmacy 2012 (in press).

• Statistical Methods in Medical Research. February 2013; 22 (1). Special Issue: Effectiveness Research. Guest editors: Xiaochun Li, Lingling Li and Patrick Ryan.

• Suchard MA, Simpson SE, Zorych I, Ryan P, Madigan D. Massive Parallelization of Serial Inference Algorithms for a Complex Generalized Linear Model. ACM Trans Model Comput Simul. 2013;23(1):1-17.

• Ryan, P. B. (2012). Using exploratory visualization in the analysis of medical product safety in observational healthcare data. In A. Krause & M. O’Connell (Eds.), A picture is worth a thousand tables: Graphics in life sciences (pp. 391-413). New York, New York: Springer-Verlag.

• Ryan PB, Madigan D, Stang PE, Overhage JM, Racoosin JA and Hartzema AG. (2012), Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership. Statist. Med. doi: 10.1002/sim.5620.

• Reich C, Ryan PB, Stang PE, Rocca M. Evaluation of alternative standardized terminologies for medical conditions within a network of observational healthcare databases. Journal of Biomedical Informatics 2012; 45: 689-696.

• Page D, Santos Costa V, Natarajan S, Barnard A, Peissig P, and Caldwell M. Identifying adverse drug events by relational learning. In AAAI-12, pages 1599-1605, Toronto, 2012.

• Harpaz R, DuMouchel W, Shah NH, Madigan D, Ryan P, Friedman C. Novel Data-Mining Methodologies for Adverse Drug Event Discovery and Analysis. Clinical Pharmacology & Therapeutics (2 May 2012)

http://www.aaai.org/ocs/index.php/AAAI/AAAI12/paper/viewFile/4941/5307


4

Publications of Interest (2012-2013) • Zhou, X., Murugesan, S., Bhullar, H., Liu, Q., Cai, B., Wentworth, C., Bate A. (2013) An Evaluation of the Thin

Database in the Omop Common Data Model for Active Drug Safety Surveillance. Drug Safety: 1-16. DOI: 10.1007/s40264-012-0009-3.

• DeFalco F, Ryan P, Soledad Cepeda M (2012) Applying standardized drug terminologies to observational healthcare databases: a case study on opioid exposure. Health Services and Outcomes Research Methodology: 1-10. DOI 10.1007/s10742-012-0102-1.

• Harpaz R, Vilar S, DuMouchel W, Salmasian H, Haerian K, et al. (2013) Combing signals from spontaneous reports and electronic health records for detection of adverse drug reactions. J Am Med Inform Assoc 2013;20:413–419.

• Kahn MG, Batson D, Schilling LM (2012) Data model considerations for clinical effectiveness researchers. Med Care 50 Suppl: S60-67.

• Kahn MG, Raebel MA, Glanz JM, Riedlinger K, Steiner JF (2012) A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med Care 50 Suppl: S21-29.

• Schuemie MJ, Coloma PM, Straatman H, Herings RM, Trifiro G, et al. (2012) Using Electronic Health Care Records for Drug Safety Signal Detection: A Comparative Evaluation of Statistical Methods. Med Care.

• Platt, R. and Carnahan, R. (2012), The U.S. Food and Drug Administration's Mini-Sentinel Program. Pharmacoepidem. Drug Safe., 21: 1–303. doi: 10.1002/pds.3230

• Robb, M. A., Racoosin, J. A., Sherman, R. E., Gross, T. P., Ball, R., Reichman, M. E., Midthun, K. and Woodcock, J. (2012), The US Food and Drug Administration's Sentinel Initiative: Expanding the horizons of medical product safety. Pharmacoepidem. Drug Safe., 21: 9–11. doi: 10.1002/pds.2311

• Curtis, L. H., Weiner, M. G., Boudreau, D. M., Cooper, W. O., Daniel, G. W., Nair, V. P., Raebel, M. A., Beaulieu, N. U., Rosofsky, R., Woodworth, T. S. and Brown, J. S. (2012), Design considerations, architecture, and use of the Mini-Sentinel distributed data system. Pharmacoepidem. Drug Safe., 21: 23–31. doi: 10.1002/pds.2336


5

Review Paper: Harpaz R, DuMouchel W, Shah NH, Madigan D, Ryan P, Friedman C. Novel Data-Mining Methodologies for Adverse Drug Event Discovery and Analysis. Clinical Pharmacology & Therapeutics (2 May 2012)

Growth of Pharmaco-Vigilance Data Mining Literature


6

Analyses of Spontaneous Reports

• Enhancements: – Bayesian Shrinkage – Covariate Adjustments – Drug Interactions – Concomitant Drugs


7

Claims Records and EHRs

• Designs: Cohort, Case-Control, Self-Controlled, … – Many variations on each basic design


8

Non-Standard Data Sources

• Quantitative Structure Activity Relationship (QSAR) models – Matthews et al. from the FDA CDER – Commercial QSAR software applied to drug candidates from AERS DPAs – Separate QSAR models for several Cardiac, Liver and Urinary ADRs – Reported 78% Specificity and 56% Sensitivity

• PubChem Assays: National Center for Biotechnology Information – 487,000 drug activity screens vs Canadian ADR database – Pouliot, Chiang and Butte (Clin. Pharm. Ther. 90) Use Logistic Regression – Reported 75% Specificity when compared to literature or drug labels

• Biomedical Literature: NLP Analyses Focusing on Drug-AE Pairs – Shetty and Dalal, J. Am. Med. Inf. Ass. 18 report 70% sensitivity, 40% PPV

• User-Generated Content in Health Forums – Leaman et al, Proc 2010 Wkshp Biomed Nat Lang Process – Web crawler, NLP extracts ADR clinical concepts: 78% precision, 70% recall


9

Harpaz R, Vilar S, DuMouchel W, Salmasian H, Haerian K, et al. (2013) Combining signals from spontaneous reports and electronic health records for detection of adverse drug reactions. J Amer Med Inform Assoc

• Objective Data-mining algorithms that can produce accurate signals of potentially novel adverse drug reactions (ADRs) are a central component of pharmacovigilance. We propose a signal-detection strategy that combines the adverse event reporting system (AERS) of the Food and Drug Administration and electronic health records (EHRs) by requiring signaling in both sources. We claim that this approach leads to improved accuracy of signal detection when the goal is to produce a highly selective ranked set of candidate ADRs.

• Materials and methods Our investigation was based on over 4 million AERS reports and information extracted from 1.2 million EHR narratives. Well-established methodologies were used to generate signals from each source. The study focused on ADRs related to three high-profile serious adverse reactions. A reference standard of over 600 established and plausible ADRs was created and used to evaluate the proposed approach against a comparator.

• Results The combined signaling system achieved a statistically significant large improvement over AERS (baseline) in the precision of top ranked signals. The average improvement ranged from 31% to almost threefold for different evaluation categories. Using this system, we identified a new association between the agent, rasburicase, and the adverse event, acute pancreatitis, which was supported by clinical review.

• Conclusions The results provide promising initial evidence that combining AERS with EHRs via the framework of replicated signaling can improve the accuracy of signal detection for certain operating scenarios. The use of additional EHR data is required to further evaluate the capacity and limits of this system and to extend the generalizability of these results.


10

• Comparing the Top K Signals from 2 Methods – AERS alone using MGPS – Minimum of AERS and EHR Signal Scores – Using both gives much better Positive Predictive Values


Empirical Bayes to Combine Signals

Outcome AERS Healthcare Combined Relative Improvement

Acute Renal Failure 0.86 0.81 0.94 56% Upper GI Bleed 0.89 0.73 0.94 49% Acute Liver Injury 0.70 0.76 0.85 37% Acute Myocardial Infarction 0.64 0.70 0.76 20% Average 0.77 0.75 0.87 40%


12

Madigan, Ryan , Schuemie, Stang , Overhage, Hartzema et al. Evaluating the Impact of Database Heterogeneity on Observational Study Results. American journal of epidemiology. May 5, 2013

• Studies of the same issue in different databases can and do generate different results, sometimes with strikingly different clinical implications.

• Relative risk estimates for 53 drug-outcome pairs • 2 study designs (cohort studies and self-controlled case series) • 10 observational databases • Wildly discordant estimates across the 10 databases

– Statistically significant decreased risk to a statistically significant increased risk – 11 of 53 (21%) of drug-outcome pairs that use a cohort design – 19 of 53 (36%) of drug-outcome pairs that use a self-controlled case series design

• Consistent estimates (both direction and significance) across the 10 DBs – 9 of 53 drug-outcome pairs (17%) for cohort studies – 5 of 53 drug-outcome pairs (9%) for self-controlled case series

• Observational studies can be sensitive to the choice of database. • More attention needs to be paid to this issue!


13

• A proposal based on the 2012 OMOP experiments • Initial review established a ‘gold standard’ of 400 drug-event

pairs with established positive or negative standing as to causality

• Restricted to four health outcomes of interest – Acute kidney injury – Acute liver injury – Acute myocardial infarction – GI bleed

• Extensive methodological experimentation – Study bias and variability of estimation across drug-event pairs,

estimation methods, real databases and simulations

An Empirical Approach to Measuring and Calibrating for Error in Observational Analyses

OBSERVATIONAL MEDICAL OUTCOMES PARTNERSHIP OMOP 2012 Experiments

http://omop.fnih.org/


15

Consider a typical observational database study: Exploring clopidogrel and upper gastrointestinal bleeding

p<.001

Error = distance from the point estimate to the true effect How far away from truth is RR=2.07?

Bias = expected value of the error distribution

When applying this type of analysis to this type of data for this type of outcome, how far on average is the estimate from the true value?

Coverage = probability that true effect is contained within confidence interval

When applying this type of analysis to this type of data for this type of outcome, do the 95% confidence intervals (1.66 to 2.58 in this case) actually contain the true relative risk 95% of the time?


16

• Their recommendation: Use 3-4 negative controls, in addition to target outcome, as a means of assessing the plausibility of an observational analysis result

• Our recommendation: Use a large sample of negative (and positive) controls to empirically measure analysis operating characteristics and use them to calibrate your study finding

Learning from what's already known


17

• Develop a standardized implementation of the analysis strategy – Study design: Case-control – Nesting within indication (unstable angina) – Case definition: First episode of upper GI hemorrhage – 10 controls per case, matched on age, gender, and index date – Exposure definition: Length of exposure + 30d – Exclusion criteria: <180d of observation before case

• Systematically apply the analysis across a network of databases, consistently for a large sample of positive and negative controls

– GI Bleeding: 24 positive controls, 67 negative controls – Criteria for negative controls:

• Event not listed anywhere in any section of active FDA structured product label • Drug not listed as ‘causative agent’ in Tisdale et al, 2010: “Drug-Induced Diseases” • Literature review identified no evidence of potential positive association

• Record all effect estimates (RR, CI) from all analysis-database-drug-outcome combinations and summarize analysis*database performance

– If we assume drugs identified as negative controls truly have no effect on outcome, then we can assume true RR = 1 as a basis for measuring error

OMOP approach to methodological research

Standard approach yields similar results as initial study: Opatrny 2008 in CRPD: 2.07 (1.66, 2.58) OMOP 2012 in CCAE: 1.86 (1.79, 1.93)


18

CC

: 2000314, CC

AE

, GI B

leed Case-control estimates for GI bleed negative controls


• If 95% confidence interval was properly calibrated, then 95%*65 = 62 of the estimates should cover RR = 1

• We observed 29 of negative controls did cover RR=1

• Estimated coverage probability =

29 / 65 = 45% • Positive tendency: 74% of

estimates have RR>1

• Error distribution demonstrates positive bias (expected value > 1) and substantial variability


19

• Bias – expected difference between true RR and estimated RR • Mean squared error – sum of variance and squared bias of the estimated RR • Coverage probability - % of drugs where true RR is contained within estimated 95%

confidence interval • Real data: negative controls, assume true RR = 1 • Can’t use positive controls in real data if you don’t know true RR • Simulated data: positive controls, inject true RR = 1, 1.25, 1.5, 2, 4, 10

• Discrimination (AUC) – probability that estimate can distinguish between no effect and positive effect

• AUC can use any rank-order statistic (RR, p-value) • AUC only assumes true RR should be bigger for positive controls than negative controls • Can be/has been studied in both real and simulated data

• Sensitivity/specificity – expected operating characteristics of a procedure at a defined decision threshold

• Decision threshold can be any dichotomous criteria (ex: RR>2, p<0.05, LBRR>1.5) • Sensitivity - % of positive controls that meet decision threshold • Specificity - % of negative controls that do not meet decision threshold • Can set desired sensitivity or specificity to determine decision threshold • Can be/has been studied in both real and simulated data

Measures of accuracy used in OMOP’s evaluations


CM: 21000214 New user cohort, propensity score stratification, with active comparator (drugs known to be negative controls for outcome)

Comparing accuracy of cohort and self-controlled designs Data: MarketScan Medicare Supplemental Beneficiaries (MDCR)

HOI: ‘GI Bleeding’ broad definition

Discrimination Error Coverage

SCCS: 1955010 Multivariate self-controlled case series, including all events, and defining time-at-risk as all-time post-exposure

OS: 403002 Self-controlled cohort design, including all exposures and outcomes, defining time-at-risk and control time as length of exposure + 30d

Observation: Analyses have different error distributions, but all methods have low coverage probability Potential solution: empirical calibration to adjust estimate/standard error for observed bias and residual error

Bias: -0.40

MSE: 0.31

Mean SE:

0.03

Bias: -0.32

MSE: 0.22

Mean SE:

0.05

Bias: -0.21

MSE: 0.31

Mean SE:

0.10


21

OBSERVATIONAL MEDICAL OUTCOMES PARTNERSHIP C

C: 2000314, C

CA

E, G

I Bleed

Using theoretical null: 55% have p < .05

Using empirical null: 6% have p < .05

Case-control estimates for GI bleed negative controls

Intuition for empirical calibration:

You can use empirical null to adjust original estimate by ‘shifting’ for bias and

‘stretching’ for variance in error distribution at each

true effect size

Ex: Clopidogrel-bleeding: Pre-calibration: (1.79-1.93) Post-calibration: (0.79-4.57)


Original coverage probability = 75% Calibrated coverage probability = 96%

6 original estimates that did not contain true RR=1

After calibration, only 1 estimate does not contain true RR = 1

Applying case-control design and calibrating estimates of positive controls in simulated data, RR=1.00











CM: 21000214 New user cohort, propensity score stratification, with active comparator (drugs known to be negative controls for outcome)

Comparing accuracy of cohort and self-controlled designs, after empirical calibration

Data: MDCR; HOI: ‘GI Bleeding’ broad

Discrimination Error Coverage

SCCS: 1955010 Multivariate self-controlled case series, including all events, and defining time-at-risk as all-time post-exposure

OS: 403002 Self-controlled cohort design, including all exposures and outcomes, defining time-at-risk and control time as length of exposure + 30d

Bias: 0.04

MSE: 0.33

Mean SE:

0.67

Bias: 0.00

MSE:

0.11

Mean SE: 0.25

Bias: -0.02

MSE: 0.38

Mean SE:

0.36

Observation: Calibration does not influence discrimination, but tends to improve bias, MSE, and coverage


27

• Systematic exploration of negative and positive controls can be used to augment observational studies to measure analysis operating characteristics • Errors in observational studies were observed to be differential by analysis

design, data source, and outcome • Magnitude and direction of bias varied, but all analyses had error distributions

far from nominal • Traditional interpretation of 95% confidence interval, that the CI covers

the true effect size 95% of the time, may be misleading in the context of observational database studies – Coverage probability was much lower across all methods and all outcomes – Sampling variability is small portion of the true uncertainty in any study

• Empirical calibration is one approach to attempt to account for residual error that should be expected within any observational analysis

Concluding Thoughts


OMOP holds an annual symposium to publicly share insights from the partnership's ongoing research with all stakeholders. Day 1 is a hands on tutorial designed for

data users interested in analysis and engagement in the OMOP community. Day 2 is a research meeting to hear and discuss new OMOP research findings.

Join us for two days and be a part of an exciting and growing community!

DATE & TIME: November 5 – 6, 2013 | 9:00 a.m. - 5:00 p.m. Eastern Time Opening Poster Session – Evening of November 5th

LOCATION: Hyatt Regency Bethesda | One Bethesda Metro Center

Bethesda, MD 20814, USA

POSTER APPLICATIONS and ONLINE REGISTRATION COMING SOON! For questions contact Emily Welebob, [email protected]

Observational Medical Outcomes Partnership

Fourth Annual Symposium

November 5 – 6, 2013 Hyatt Regency Bethesda Bethesda, Maryland

Documents

BSERVATIONAL Some Recent OMOP Research …2013/11/12 · Database in the OmopCommon Data Model for Active Drug Safety Surveillance. Drug Safety: 1 -16. DOI: 10.1007/s40264-012-0009-3