Establishing a High-Quality Real-World · Can place study results in a context of contemporary real...

Establishing a High-Quality Real-World

Data Ecosystem

healthpolicy.duke.eduSubscribe to our monthly newsletter at

dukemargolis@duke.edu

Broadcast to Begin Shortly

Data Ecosystem

Virtual Public Workshop — July 13 & July 14, 2020

Day 1 – Monday, July 13th

Welcome and OverviewMarta Wosińska

Deputy Director, Duke-Margolis Center for Health Policy

Jacqueline Corrigan-CurayDirector, Office of Medical Policy (CDER)

U.S. Food & Drug Administration

What is a High-Quality Data Ecosystem and Why is it Needed?

Lesley CurtisProfessor and Chair, Department of Population Health Sciences

Duke University School of Medicine

Cultural and Institutional Considerations to Align Stakeholders Around a High-

Quality RWD Ecosystem

Session 1: Identifying Data Capture

Challenges and Understanding Stakeholder

Needs1:10 pm – 1:30 pm

Modernizing and designing evaluation frameworks for connected sensor technologies in medicine

Establishing a High-Quality Real-World Data Ecosystem

Duke-Margolis

July 13, 2020

2020 | ELEKTRALABS.COM

Andy CoravosCEO @ Elektra Labs

Research Collaborator @ Harvard-MIT Center for

Regulatory Science

Advisor @ Biohacking Village at DEFCON

If we don’t get to your questions today, feel free to message me on Twitter: @andreacoravos

Formerly

Why collect digital measurements in real-time at home?

Remote sensing offers a more holistic view of a person’s lived experience

Data collected in traditionalclinical trials.

Data collected from decentralizedtrials, at home.

VISIBLE DATA POINTSEpisodic, in the clinic.

Source: “Visible vs. Invisible Data” chart designed by Evidation Health

INVISIBLE DATA POINTSContinuous, in daily life

Chronic conditions and their

interventionsimpact a person’s

daily life, from vitals to activity to

sleep.

The future is now: across the industry, sponsors are remotely collecting patient information to support primary and secondary endpoints

Source: The Digital Medicine Society (DiMe) Crowdsourced Digital Endpoint Library https://www.statnews.com/2019/11/06/digital-endpoints-library-clinical-trials-drug-developmen

Digital primary, secondary, or label claim

Exploratory Only

>50% of examples

N = 39 Digital Endpoints

Development starts early Endpoint positioning

•••••••• 8 Primary Endpoints

••••••••••••••• 15 Secondary

Endpoints

• 1 Label claim

•••••••••••••••• 16 Exploratory

39 Unique Endpoints

Elektra Labs advances remote monitoring by enabling the safe and effective use of connected biosensors at home.

We support and build organizations collecting digital measurements and endpoints in clinical trials.

Source: https://www.fda.gov/drugs/development-approval-process-drugs/public-workshop-patient-focused-drug-development-guidance-4-incorporating-clinical-outcome

Select your measures BEFORE selecting your technology

Use a process derived from best-practice Patient-Focused Drug Development (PFDD)

Step 1: Meaningful Aspect of Health

Step 3: Measure

Step 4: Technology Selection

Step 5: Finalize Endpoint Measurement

Step 2: Concept of Interest

Determine target measures BEFORE technology selection

Example concept

● A change in activity in XX population

Develop hypotheses for target measurement

● Duration, examples:○ Minutes / day walking○ Cumulative daily min per bout of

sedentariness● Frequency, examples:

○ # of daily sedentary bouts○ # of vigorous intensity bouts

● Intensity, examples:○ Average accelerometer counts/day○ Total step counts/week○ METs per minute○ ...

Step 3: Measure

Compare the risks/benefits of certain technologies

Step 3: Measure

A deep dive on these four components is available in the Appendix.

Think of an evaluation process like a drug or

nutrition label, balancing five components

based on needs in a clinical trial.

A moment to define “real world data”

The 21st Century Cures Act, passed in 2016, places additional focus on the use of these types of data to support regulatory decision making, including approval of new indications for approved drugs.

Congress defined RWE as data regarding the usage, or the potential benefits or risks, of a drug derived from sources other than traditional clinical trials.

Real-world data are the data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. RWD can come from a number of sources, for example:

● Electronic health records (EHRs)● Claims and billing activities● Product and disease registries● Patient-generated data including in home-

use settings● Data gathered from other sources that can

inform on health status, such as mobile devices

SOURCE: https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence

Ernesto Ramirez, PhDDesign Lead: Research, Analytics, and Learning, Evidation Health@eramirez

Identifying Data Capture Challenges and Understanding Stakeholder Needs

ESTABLISHING A HIGH-QUALITY REAL-WORLD DATA ECOSYSTEM

JULY 13, 2020

Evidation believes there is a

massive opportunity to change

how we understand and

measure health:

– all via the now shortest

route, the individual managing

health and disease in their

everyday lives

Individually-generated

and Individually-permissioned

data sources

Passive

Continuous

Remote

Objective

Sensitive

The Evidation Way for the Science of Patient Input and Participant Engagement

Patient Journey MapOnset of symptoms, diagnosis (who/when/how), treatment (decision making/access), disease management and monitoring

Voice of Patient Studies for Medical Product DevelopmentMost burdensome symptoms, daily impacts, perspectives on available treatments, benefits/risks, and unmet need

Patient Input into Measure DevelopmentMeaningful aspects of health, concepts of interest, content validity, feasibility

Participant Input on Study DesignsFeedback from target population on inclusion/exclusion criteria, endpoints/outcomes, benefits/risks, ICF, participant burden, ideal participant experience

Participant Input on Value and IncentivesParticipant motivations, participant feedback on proposed incentives (informational, financial, social) and perceived value to be tested and scaled

Patient involvement is always moderated through an informative and clear consent process and attention to return of value to participants.

Study 1 Study 3Study 2 Study 4

Consent Consent Consent Consent

Data Data Data Data

Evidation, along with collaborators at Eli Lilly and Apple, recently completed a study using PGHD in participants with cognitive decline

Sent and received text

messages

Accessed the web before bedtime

Interrupted sleep

Woke up before 8am

Opened weather, phone, message,

and map apps

Took a nap

Walked and talked on the

Chen, Richard, et al. "Developing measures of cognitive impairment in the real world from consumer-grade multimodal sensor streams." Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019. DOI.ORG/10.1145/3292500.3330690

Ensuring data quality with PGHD is a multistep and continuous process

Issue Mitigation Strategy

Missing data! Develop deep understanding of data flows. Real world data = real world problems.

Missing data! Keep it, it could be informative!

This data doesn’t make sense! Develop methods robust to outliers. Examine outliers, they may be important!

Do I need this much data? What is the minimum amount needed to support inferences and outcomes?

How do I keep people engaged?

Use the features of PGHD to inform real-time compliance monitoring.

evidation.com

Ernesto Ramirez, PhD

Research, Analysis, and Learning

Evidation Health, Inc.eramirez@evidation.com

Session 1 — Audience Q&AIdentifying Data Capture Challenges and Understanding Stakeholder Needs

Break2:10 pm – 2:20 pm

Session 2: Emerging Insights and Lessons

Learned from Initiatives to Optimize Data

Capture at the Point of Care2:20 pm – 3:30 pm

Monica BertagnolliRichard E. Wilson Professor of Surgery in the Field of Surgical Oncology

Harvard Medical School and Brigham and Women’s Hospital

Optimizing Data Capture at the Point of Care

Monica M. Bertagnolli, MDDuke-Margolis Center for Health PolicyJuly 13, 2020

Integrating

Clinical trials

Real world

Endpoints

Study-specific case report forms (CRFs)Completed by site clinical research staff

NCTN clinical sitesMultiple EHR vendors

Proprietary Electronic Research

Data Management System

Study-specific CRF Data, Proprietary Format

Alliance Statistics and Data Center

Alliance Data Innovation Lab

DATA LAB

CLINICAL TRIAL PARTICIPANTS

OBSERVATIONAL COHORT

NON-PARTICIPANTS

• treatment planned vs received

• treatment outcomes survivaltreatment responsePROS

• resources utilized

DATA COLLECTED

ALL PATIENTS TREATED AT A CLINICAL SITE WITH A SPECIFIC DIAGNOSIS

• Meet trial eligibility criteria

• Consent to trial participation

• May or may not meet trial eligibility criteria

• Consent to collection of EHR data

• Do not wish to participate in research

Integrating Clinical Trials and Real World Endpoints Data

Advantages of using the EHR to conduct clinical trials

Single data input system maximizes efficiency, reduces cost, and eliminates errors associated with double entry

Can place study results in a context of contemporary real world clinical care by collecting data on both protocol-eligible and non-protocol patients

Provides a mechanism for longitudinal follow up of study patients

Improves the quality and utility of real world data by providing an environment where new EHR data standards can be developed

Example: EHR-based collection of overall survival can continue long after clinical trial completion

Challenges

Heterogeneous data and a lack of interoperability

Lack of meaningful outcome data

Is the treatment working?

How is the patient feeling?

Significant existing clinician burden

• Purpose: To develop and maintain standard computable data formats, known as Minimal Common Oncology Data Elements (mCODE), to achieve data interoperability and enable progress in clinical care quality initiatives, clinical research, and healthcare policy development

CRF – mCODE Mapping

mCODE Element Implementation

Priority

STEAM Team Epic

ImplementationEpic Playbook

Clinical site Epic mCODE

Implementation

usedby

resultsin

usedby

CRF-mCODEGaps

insights into

possibleavenue

to address

STEAM = Scalable Technology for EHR Adoption of mCODE

▪ Builds community and platform to accelerate interoperable data modeling and implementation around mCODE

▪ Run pilots at scale

Source: http://build.fhir.org/ig/HL7/fhir-mCODE-ig/branches/master/index.html#Modeling

How well does mCODE v1.0 cover the data elements required to complete case report forms for Alliance treatment trials?

Characterization VariablesDiagnosis of concernDemographicsCo-morbiditiesFunctional statusConcomitant medications

Outcome VariablesSurvivalResponse to treatmentRequired change in treatmentAdverse eventsTreatment administeredQuality of lifeResource utilization

mCODE v1.0under development

CRFCRF

mCODE data elementsMapped

A071701, A021703, PALLAS,

PATINA, etcElements rolled

up to profiles

mCODE profiles

• mCODE 1.0 released, includes disease status data element– Mechanism for tracking disease natural

history and treatment response

Under development:Linkage of disease status data element to other relevant data:

imagingpathologylaboratory assessment

Data Input

▪ Low burden data collection

▪ Epic implemented an enterprise solution for capturing

ICAREdata in EHR via SmartData Elements

▪ Available via Special Update (released in March)

or May 2020 Epic release

▪ https://userweb.epic.com/Topic/549

ICAREdata Implementation Priority: Streamlining Clinical Workflows

▪ Software for data sharing available

▪ MITRE-developed ICAREdata client available for

sharing ICAREdata in mCODE standard

▪ Epic is implementing an enterprise solution for

extracting ICAREdata from EHR and mapping it to

mCODE standard

FHIR Message

Automated Interface to

Extract and Share

IT Staff

Extract mCODE

Share mCODE

EHR Database

Alliance Statistics and Data Center

NCTN clinical sitesMultiple EHR vendors

FHIR transfer of eCRF data:mCODE core data elements

CT scans

EHR-Enabled Research Protocol

Alliance Foundation Data Innovation LabDATA LAB

• EHR-ready study-specific order sets• eCRF formats for EHR data capture

Integrating

Clinical trials

Real world

Endpoints

ICAREdata®

Thanks to:

Laura EssermanDirector, Breast Care Center

The University of California, San Francisco

47QLHC

Transforming data capture to Integrate clinical

care and research

7.12.2020

Collaborators: Adam Asare, Cal Collins, Ronak Ahir, Clarence So, Mitra Rocca, Frank Weichold,Sue

Dubman, Mark Wheeldon, Heidi Collins

UCSF Breast Care Center and Staff, WISDOM study team, I SPY 2 team, I SPY COVID team, UCSF IT team

Laura Esserman, MD, MBA (UCSF, QLHC)

Alfred A. de Lorimier Chair in General Surgery

Professor of Surgery and Radiology

Director, UCSF Carol Franc Buck Breast Care Center

48QLHC

Why do we need better systems? T H E O N E S O U R C E S O L U T I O N

“There are so many people working so hard and achieving so little”

– Andy Grove

CommentaryJuly 27, 2005Efficiency in the Health Care Industries A View From the OutsideAndrew S. Grove, PhDAuthor Affiliations Article InformationJAMA. 2005;294(4):490-492. doi:10.1001/jama.294.4.490The health science/health care industry and the microchip industry are similar in some important ways: both are populated by extremely dedicated and well-trained individuals, both are based on science, and both are striving to put to use the result of this science. But there is a major difference between them, with a wide disparity in the efficiency with which results are developed and then turned into widely available products and services

49QLHC

Old Processes yield Old Results

ONESOURCE SOLUTION

50QLHC

Before Electronic Health Records

THE ONESOURCE SOLUTION

51QLHC

It’s our clinical systems that are the problem

AFTER Electronic Health Records

52QLHC

Enable improvements in technology with changes to clinical workflows

VISION: Business Process Change

Principles of One Source

Systems orientation

Quality at the source

Enter once, use many

Mind sets, skill sets, tool sets

CALM “One Source puts patients at the center of new data collection

Puts new info into user configurable apps, drives meaningful analytics

MARKET ENVIRONMENT FDA / CERSI supported

• We should be practicing to improve▪ Not just practicing

• That means we have to know what we are doing and what the consequences are▪ Assessing Outcomes Should not be the heroic effort of chart review ▪ We should not be asking for permission to evaluate outcomes▪ We should only seek permission if we choose NOT to evaluate outcomes (and cost)

• Clinical research is just a special case of clinical care▪ Trials summaries should visible to clinicians▪ Trials often create more discipline about process, which should then be applied to all patierts

• Refocus the clinic on gathering the mission critical data to support decision making▪ And create a framework to assess outcomes → Integrate clinical care and research

What is the Purpose of Clinical Medicine?

54UCSF

• Problem▪ Clinical data collected at the point of care is unstructured

▪ Manual data extraction and re-entry into disparate data platforms required to facilitate research and quality improvement

▪ Creates a workflow that separates clinical care and research, thereby increasing costs, decreasing efficiency and compromising quality, and slowing knowledge turns

• Solution▪ Implement technological and process changes that drive high quality, structured data

capture at the point of care

Background

EHRProvider notes

Clinical results

Patient demographics

Administrative data

Trial CRFs

Research Registries

Data Repositories

Current solution examples

Source Data Capture from EHRs to Improve Quality and Efficiency

55AMIA 2019 Annual Symposium | amia.org

CRFs ask for data that is not in or easy to find in the notes

56QLHC

Structured data as “source”Enable improvements in technology with changes to clinical workflows

Point of Care Data Collection

EHR system

57UCSF

OneSource Technology Stack

Providers

• Electronic Health Record

Using a FHIR standard allows an EHR

agnostic approach

58UCSF

• Checklist data entry & write back to EHR

• Clinical Care, Research Registry and Trial Case Report Forms (CRFs)

Read discrete data from and write back to EHRs (via FHIR standards & APIs)

Providers

Form Entry

Single Sign On: Log into EHR and access OpenClinica CRF

Bidirectional flow of information

59UCSF

• Terminology mappings• FDA submission standards

Providers

Form Entry

Data standards mapping

Standards in the background

60UCSF

Providers

Form Entry

• Research data repository

Direct access of discrete data

Researchers

Data seamlessly flows to

trials with consent

61UCSF

Providers

Patients

Form Entry

• Patient surveys (i.e., I-SPY electronic patient reported outcomes)

Researchers

Patient reported information integrated whether for clinical care or trial

62UCSF

• Decision support displays • “Northstar” metrics tracking

Read access

Providers

Patients

Form Entry

• Patient surveys (i.e., I-SPY electronic patient reported outcomes)

Researchers

QI, metrics seamlessly integrated !!

63QLHC

“Enter Once, Use Many”

Domain / Data ElementsNew PATIENT

(AT ANY PHASE) SCREENING DIAGNOSTICTREATMENT PLANNING SURGERY

SYSTEMATIC TREATMENT

RADIATION TREATMENT FOLLOW-UP RESEARCH

Patient-Reported Outcomes

Patient Health History

Imaging

Biopsy Pathology

Clinical Exam and Stage

Clinical Trial Matching

Treatment

Final Pathology

New data Confirm or Update View Only Confirm/Additional Data Added

64QLHC

Process is the keyMap: AS IS and TO BE

7/15/2020Presentation Title and/or Sub Brand Name Here64

65UCSF

Use CasesBreast surgery checklists: What clinicians need for care

I-SPY 2 case report forms

Electronic patient reported outcomes survey platform

Trial adverse events collection

I-SPY COVID TRIAL

66UCSF

• “Enter once, use many”

• Diagnostic Checklist is the firs phase of a patient’s journey

• Allows surgeons and radiologists and nursing staff to share a common tool to track breast lesions

Breast Surgery Checklists

• “Enter once, use many”

• Surgical Checklist is just one phase of a patient’s journey

• Clinician-vetted discrete data elements: 105 including branched logic fields

• Links patient’s procedure, pathology, operative key elements, surgical complications, and action taken

Breast Surgery Checklists

Surgery Final Pathology Pathologic Stage Complications

Use CasesBreast surgery checklists

I-SPY 2 case report forms: What clinicians need for care are in the case report form

I-SPY COVID TRIAL

I-SPY | The right drug. The right patient. The right time. Now.™

The Original Goal was to Capture Data from Care to Trials

• I-SPY 1: used caBIG tools and platforms (caINTEGRATOR)

• ISPY 2: Tolven→ Salesforce→OpenClinica/Formedix/Salesforce/Tableau

I SPY 2: Neoadjuvant therapy for high risk breast cancer

July 1: Complete CRF redesign • 33 forms to 15• 75% of data elements removed• Focus on critical fields for care and

research• Integrated workflow• Primary endpoint entered as

SOURCE

Aug 1: Complete form design and testing

August 31: Launch new system

Electronic patient reported outcomes survey platform: Incorporating patient reported data

I-SPY COVID TRIAL

71UCSF

• Quality-of-life (QoL) prospective sub-study

▪ Explores how patient and tumor subtypes, exposure to investigational

therapies, and residual cancer impact QoL for I-SPY 2 trial participants

▪ Allows assessment of Clinical Benefit Index (efficacy AND toxicity)

▪ Surveys administered from screening, drug treatment, and surgery through

two years postoperatively

• Patient-Reported Outcomes Measurement Information System (PROMIS)

• Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse

events (PRO-CTCAE)

Tracking Patient Reported Outcomes in I-SPY 2

72UCSF

• Current state: Paper-based surveys in the trial

▪ At least 1 survey collected from 1150+ patients

▪ Low completion rate: ~38%

▪ Inefficient process for patients and clinical research coordinators

▪ Manual data entry prone to transcription error and delayed reporting

• Electronic data in the clinic: >75% of patients fill it out

Tracking Patient Reported Outcomes in I-SPY 2

73UCSF

Mapping the I-SPY QoL Survey Workflow (PROMIS 1/2)

Screening surveyDate open: Screening consent signed

Date close: Scheduled date/time of

C1D1 infusion

CRC action required• Before visit

• After patient is registered and

subject ID is generated, enter

patient’s email in OpenClinica

• Note: Subject record will be

automatically created in OC

• During visit

• On C1D1, check if patient has

completed survey. If not,

provide tablet to patient to

complete before starting

infusion

• Enter C1D1 date/time in OC

and be responsible for keeping

it updated

• Note: Randomization date will

be automatically pushed from

• During visit

• On C1D1, check if patient has

completed both baseline and

C1D1 surveys. If not, provide

tablet to patient to complete.

Can complete during infusion

• Enter AC day 1 date in OC and

be responsible for keeping it

updated

• During visit

• On AC day 1, check if patient

has completed survey. If not,

provide tablet to patient to

complete

C1D1 surveyDate open: 3 days before C1D1 OR

Randomization date (whichever comes later)

Date close: C1D1 at 11:59pm

Inter-regimen surveyDate open: 7 days before AC day 1

Date close: AC day 1 at 11:59pm

• Enter surgery date/time in OC

and be responsible for keeping

it updated

• **During visit

• Prior to surgery date, check if

patient has completed survey. If

not, provide tablet to patient to

complete

End of drug regimen 2 / pre-op surveyDate open: 14 days before surgery date

Date close: Scheduled date/time of surgery

74UCSF

Mapping the I-SPY QoL Survey Workflow (PROMIS 2/2)

Follow up 30-dayDate open: 21 days after surgery

Date close: 60 days after surgery at

11:59pm

• No action required IF follow-up surveys are sent based on surgery date

• Note: important for CRCs to have the correct surgery date in OC

Follow up 6-monthDate open: 5 months after surgery

Date close: 8 months after surgery at

11:59pm

Follow up 1-yearDate open: 11 months after surgery

11:59pm

Follow up 2-yearDate open: 23 months after surgery

11:59pm

75UCSF

Translating Workflow to Business Requirements

Requirements for automated survey scheduling system adapted for I-SPY 2 TRIAL

76UCSF

ePRO Survey Pathway

Coordinator triggers survey

notification to patient

Patient completes

eSurvey prior to or

during appointment on

own device or clinic

tabletResearchers have direct & immediate

access to composite survey data

Sets off cascade of automated

scheduled surveys

77UCSF

Trial adverse events collection: Tracking what matters to clinicians, regulators, and pharma

I-SPY COVID TRIAL

78UCSF

• Current I-SPY TRIAL process at UCSF

▪ Source of truth is the oncologist’s note of patient’s treatment course

▪ Coordinators assess note to document AE and Grade

▪ Oncologist reviews / verifies information and assigns Attribution (whether AE is related to study treatment)

• Terminology mapping

▪ Common Terminology Criteria for Adverse Events (CTCAE)

Adverse Events: Current Process

79UCSF

• Streamline AE coding

▪ Simplify the set of allowable CTCAE terminology that can be chosen

▪ Pre-populate official Grade definitions to support Grade designation

• Start with the key questions:

▪ Is there an adverse event to track?

▪ Will the patient continue on therapy?

▪ If no, why not

▪ If yes, dose modification?

Adverse Events Redesign: Form Examples (1/2)

80UCSF

I-SPY COVID TRIAL: Adverse events→Visual; Write back standard

81UCSF

7/15/2020 81

• Innovative ways to display AE data

• Differs from traditional reports

• Mind numbing tables

• Rapidly find Signal above noise

• (std of care arm)

82UCSF

Write Back

Write Back is Crucial

• What should go back?

• Trial summary

information

• Trial arm assignments

• Ability to populate notes,

problem list from data

captured in a structured

Scalable Std for Write Back

• FHIR writeback API

• Standard in the EPIC

• Scalable across sites

• Hopefully across all EHRs

83UCSF

• Reconciliation of data elements in I-SPY and clinic redesign CRFs

• Completing integration of single sign on (SSO) in OpenClinica from Epic

• Development of dashboards to enable decision support with clinical data we already collect

OneSource Clinic Redesign: Looking Ahead

Acknowledgements : It takes a village

• UCSF

• Adam Asare• Heidi Collins (EPIC IT)

• Andrew Robinson (EPIC IT)

• Amrita Basu

• Ronak Ahir

• Sarah Schrupp

• Jasmine Wong

• Hope Rugo

• Michelle Melisko

• Merisa Piper

• Garry Peterdson

• QuantumLeap Healthcare Collaborative• Adam Asare

• Lisa Weiss

• Steven Cosari

• Amy Wilson

• Sue Dubman

• James Palazzolo

• UCSF Breast Care Center

• I SPY TRIAL TEAM

• I SPY COVID TRIAL TEAM

• FDA• Mitra Rocca• Frank Weichold• Amy Abernethy• Jacqueline Corrigan- Curay• Many others

• Salesforce/Tableau• Clarence So• John Kim• Andrew Conn• Vidya Balikrishna• CJ Callendar

• Open Clinica• Cal Collins

• Formedix• Mark Wheeldon

Session 2 — Audience Q&AEmerging Insights and Lessons Learned from Initiatives to Optimize Data Capture at

the Point of Care

Spotlight: RWE Accelerator3:30 pm – 4:00 pm

Amy AbernethyPrincipal Deputy Commissioner of Food and Drugs

U.S. Food & Drug Administration

Real-world Evidence Accelerator –Lessons Learned from COVID-19

Amy P. Abernethy, MD PhD

Principal Deputy Commissioner

Acting Chief Information Officer

U.S. Food and Drug Administration

“reliable information is the best weapon we have against Covid-19.”

Charlie Warzel, NYT Opinon April 3, 2020

NOT FOR DISTRIBUTION

WHY RWD? • Urgent need to rapidly understand the natural history of COVID-19• Many critical clinical evidence needs but limited clinical trial resources

(patients, time, competing tasks)– RWD evaluation of treatment patterns and impact provides understanding– RWD can help prioritize research questions to be answered with clinical trials– RWD can improve study design and support participant enrollment– Pragmatic and platform/adaptive study designs can improve efficiency and

generalizability

• Near real-time performance of diagnostics authorized under EUA• Near real-time vaccine performance of future potential vaccines

Real-World Data for COVID-19

Sits within a larger RWD Community

Evidence

Accelerator

RUF/FOCR managed workstream

Health Data &

Technology Partners

A community of

data and analytic

partners ready to

urgently address

questions about

COVID-19

Meetings and forum for rapid cycle feedback

and learning

RUF/FOCR*

Evidence

Accelerator

Common protocol for repeated analysis of priority

research questions across multiple data partners (the

“parallel analysis”)

Prioritized research questions

Common data elements and translation tables

between common data models

Our Tools

*Reagan-Udall Foundation (RUF) for

the FDA /Friends of Cancer

Research (FOCR)

Individual Accelerator communities focused on

specific topics (e.g., therapeutics, diagnostics)

RUF/FOCR*

Evidence

Accelerator

*Reagan-Udall Foundation (RUF) for the FDA

/Friends of Cancer Research (FOCR)

https://evidenceaccelerator.org/

RUF/FOCR*

Evidence

Accelerator

Research (FOCR)

Prioritized Research Questions

RUF/FOCR*

Evidence

Accelerator

Research (FOCR)

Parallel Analysis Projects

Collaborative

group effort to

align and work

on a common

research

question

Summary Analysis

Data Source #1

Results

Data Source #1Analysis

ParallelAnalysis

Data Source #2

Results

ParallelAnalysis

Data Source #3

Results

ParallelAnalysis

Results

Data Source #4, 5,..

Data Source #4, 5,…Analysis

ParallelAnalysis

RUF/FOCR*

Evidence

Accelerator

Research (FOCR)

Lab Meeting

UC Health DailyDashboards

https://twitter.com/UofCAHealth

UC Health DailyDashboards

https://twitter.com/UofCAHealth

Veterans Affairs study

Rentsch CT, Kidwai-Khan F, Tate JP, et al. Covid-19 by Race and Ethnicity: A National Cohort Study of 6 Million United States Veterans. Preprint. medRxiv. 2020;2020.05.12.20099135. Published 2020 May 18. doi:10.1101/2020.05.12.20099135

RUF/FOCR*

Evidence

Accelerator

Tailored to Specific Topics

Identifying

unique

populations,

connecting the

*Reagan-Udall Foundation (RUF) for the FDA /Friends of Cancer Research (FOCR)

Data or analytic EA partners participate in one or more groups

[VACCINES EA]

o Research questions generated within each accelerator

o Work groups cross boundaries of the different EAs

RUF / FOCR* Evidence Accelerator Work Streams

*Reagan-Udall Foundation (RUF) for the FDA /Friends of Cancer Research (FOCR)

DIAGNOSTICS EA

THERAPEUTICS

Weekly Lab Meeting

Weekly Parallel Analysis

Weekly Lab Meeting

COVID-19 Evidence

Accelerator (EA)

Launch

April 16, 2020

First “Lab” Meeting

May 28, 2020

HCQ +/- azithromycin in

hospitalized patients

Parallel Analysis

Project 1

April 22, 2020

Oncology

Workgroup

June 2020

Diagnostics EA

Launch

Weekly

Lab EA

Weekly

Lab EA

Weekly

Lab EA

Weekly

Lab EA

Weekly

Lab EA

Weekly

Lab EA

Weekly

Therapeutics

Lab EA

June 4th

Real-World Data for COVID-19

NIH, VA, PCORI, etcReal-World Data

Approaches

GovernmentFDA

Sits within a larger RWD Community

Evidence

Accelerator

RUF/FOCR managed workstream

Health Data &

Technology Partners

Responsibility

Research (FOCR)

The EA with

RUF/FOCR provides

a “safe space” for

key players across

the ecosystem to

lead, scrutinize and

“get this right”

• Data selection

• Protocol design

• Transparency

• Data provenance

• Data quality

• Analytical integrity

• Peer review

• Press interactions

RWD Priorities

DRAFT PRINCIPLES for the EARespect for patient privacy01

Act fast, Traceability and provenance – understand data generation, processing, curation, and analytics02

Transparency, ruthless transparency03

Traceability and provenance – understand data generation, processing, curation, and analytics04

Sharing – show process, explore limitations, pitfalls, and celebration successes – bring work and

learnings to the community05

Build trust – show processes. Show curation approaches. Show comparisons. Curation is

expensive and takes time, many “eyes” along the way, yields trust, understanding, and

confidence in the results

Embrace convergence and discordance to facilitate understanding 07

Learning is additive, and continuously integrated to improve knowledge and

understanding 08

Dissemination – responsible evidence generation ( show what good looks like)09

Spotlight — Audience Q&ARWE Evidence Accelerator

Closing RemarksMarta Wosińska

Deputy Director, Duke-Margolis Center for Health Policy

Join us for Day 2 Tomorrow, Tues July 14th

Register Online… …and Follow

DukeMargolis

@DukeMargolis

Duke Margolis

healthpolicy.duke.edu

Subscribe to our monthly newsletter at

DC office: 202-621-2800

Durham office: 919-419-2504

1201 Pennsylvania Avenue, NW, Suite 500

Washington, DC 20004

Data Ecosystem

Broadcast to Begin Shortly

Data Ecosystem

Virtual Public Workshop — July 13 & July 14, 2020

Day 2 – Tuesday, July 14th

Welcome and Opening CommentsMark McClellan

Director, Duke-Margolis Center for Health Policy

Session 3: Translating Early Successes of

Core Data Element Programs to

Therapeutic Areas beyond Oncology1:10 pm – 2:20 pm

Leslie HarroldChief Scientific Officer

Corrona, LLC

The Corrona Independent Registry Model for

Autoimmune Conditions

The Corrona independent registry model is the gold standard for

deep, real-world evidence across autoimmune diseases

120Confidential and proprietary information. Not for distribution.

Rheumatoid Arthritis

>54,000 pts

PsA/SpA

> 4,000 pts

CERTAINRheumatoid

Arthritis bio samples

2,800 pts

Psoriasis

>10,000 pts

Japan RA

≈ 1,900 pts

2001 2010 2013 2015 2016

Portfolio of six registries in five diseases, collecting data from >500 sites in the USA, Canada, and Japan

IBD (Crohn’s & UC)

≈ 2,000 pts

Multiple Sclerosis

≈ 1,000 pts

US US US US/Can

Key Benefits of Corrona’s Approach

1. It is “fit for purpose” and flexible, with prospectively designed custom data collection tools, avoiding the data gaps found in retrospective approaches

2. It is comprehensive, providing substantial insight into clinical decision making by collecting information directly from physicians and PROs from patients

3. It is credible with the scientific community – over 160 full length manuscripts and over 500 abstracts using Corrona data in top tier journals

4. It is unique with systematic collection of RCT-level outcomes in autoimmune patients cared for in a real-world setting

How does Corrona collect high

quality regulatory grade Real World

Data ?

Corrona generates Fit for Use RWD that is reliable and relevant for research & global regulatory agencies

• In Corrona, virtually all (99%) data are prospectively collected using structureddata fields in registry questionnaires completed by physicians and patients, and “fit for purpose” as defined by 21st Century Cures – not extracted from free text EMR notes

• Systematic collection of variables including validated outcomes typically used in RCTs and patient reported outcomes at every visit (approximately every 6 months)

• Comprehensive, granular safety data are collected, including information about serious adverse events (SAEs) that may not be captured or asked about in a routine clinical encounter with a specialist (e.g., UTI, bronchitis, pneumonia treated by PCP)

• Detailed source documents are collected to support reporting of SAEs and AEs of special interest (Targeted AEs), with case validation and processing by Corrona’spharmacovigilance team of drug safety specialists

Corrona collects data using a structured approach to study both real-world drug effectiveness and drug safety

Provider FormCollected from physician at

enrollment and Q6 month follow up

Patient FormCollected from patient at baseline and Q6 month follow up

Targeted Safety EventsCollected from physicians starting

after enrollment during follow up

Data Collection Process

✓ Data entry into a part 11 compliant EDC system

✓ q6 month data collection

✓ Questionnaires are revised based on changes in the relevant field

✓ Extensive QA/QC process to ensure data completeness and accuracy of the analytic database

✓ Source documents requested for SAEs and TAEs for case validation

Over the past 20+ years, Corrona has developed & evolved the

approach to collect high quality data across 500+ sites

Questionnaires

Electronic Data Capture

Analytic database

▪ Questionnaire design

▪ Site selection and

training

▪ Site monitoring

▪ Site feedback

• EDC design and

automatic edit

checks

• Routine site

performance reports

• Routine analytic

data checks

based on logic

Data Capture

Reports

An illustration of the longitudinal nature & data depth

in the registry

Rheumatoid Arthritis

Corrona RA Registry USA – more than 54,000 patients enrolled

✓ 193 private and academic sites

✓ 827 participating rheumatologists

✓ Sites in 42 US states

✓ Over 400,000 registry visits

✓ Total follow-up time of 199,111 person years

✓ Median follow-up is 4.5 years

Summary of key variables collected in the Corrona RA Registry

Demographics

• Age

• Gender

• Race/ethnicity

• Height/weight

• BMI

• Education

• Work status

• Insurance type

• Smoking Status

• Alcohol use

Treatment History

• Conventional synthetic DMARD, Biologic DMARD and Small

Molecules, Biosimilar DMARD, Targeted synthetic DMARD,

NSAID use, Other concomitant medications

Disease Characteristics and Disease

Activity

• RA year of onset and diagnosis

• Comorbidity and medical history

• ACR functional class

• Subcutaneous nodules

• Secondary Sjögren’s syndrome

• Physician global assessment of arthritis

• 28 tender and swollen joint count

Patient Reported Outcomes

• Health Assessment Questionnaire and usual abilities

• Pain VAS (0-100), Patient global assessment VAS (0-100)

• Fatigue VAS (0-100), Stiffness VAS (0-100)

• Days unable to work due to RA

Lab Measures*

• CBC, Liver function, Inflammatory markers, autoantibodies

(CCP and Rheumatoid factor), Lipid panel, 25-OH Vitamin

D, Vectra DA, Joint Radiographs: abnormalities present/not

present in erosions, joint space narrowing, deformities; Chest

Radiographs; Joint MRI: synovitis, erosions, osteitis, bone

marrow edema; Ultrasound (Synovitis, Erosions, Doppler

signal); Bone densitometry (machine type, lumbar spine,

femoral neck, 1/3 radius)

Targeted Adverse Events

• Serious Infections, Malignancy, Cardiovascular, Neurologic,

Hepatic, General Serious, Anaphylaxis, Autoimmune,

Gastrointestinal, Pregnancy Event

Patient

Reported

Patient

Reported

Physician

Reported

Physician

Reported

Corrona has built a strong publication track record in both comparative effectiveness and safety studies, with 500+ abstracts and 160+ manuscripts

Select Comparative Effectiveness Publications

Harrold LR, et al. Comparative effectiveness of abatacept versus tocilizumab in rheumatoid

arthritis patients with prior TNFi exposure in the US Corrona registry. Arthritis Res Ther. 2016 Dec

1;18(1):280; 1-8.

Harrold LR, et al. The comparative effectiveness of abatacept versus anti-tumor necrosis

factor switching for rheumatoid arthritis patients previously treated with an anti-tumor

necrosis factor. Ann Rheum Dis 2015 Feb; 74(2):430-6.

Harrold LR, et al. Comparative effectiveness and safety of rituximab versus subsequent anti-

tumor necrosis factor therapy in patients with rheumatoid arthritis with prior exposure to anti-tumor necrosis factor therapies in the United States Corrona registry. Arthritis Res Ther

2015;17:256.

Select Drug Safety Publications

Harrold LR et al. One-year risk of serious infection in patients treated with certolizumab

pegol as compared with other TNF inhibitors in a real-world setting: data from a national

U.S. rheumatoid arthritis registry. Arthritis Res Ther. 2018 Jan 2;20(1):2.

Pappas DA, et al. Herpes zoster reactivation in patients with rheumatoid arthritis:

Analysis of disease characteristics and disease modifying anti-rheumatic drugs. Arthritis Care Res (Hoboken). 2015 Dec 67(12):1671-8.

Harrold LR et al. Risk of infection associated with subsequent biologic agent use after

rituximab: results from a national rheumatoid arthritis patient registry. Arthritis Care Res

(Hoboken). 2016 Dec;68(12):1888-1893.

The registry data capture model enables evaluation of disease factors associated with outcomes

Model CModel BModel A

HighModerateLowRemission

Disease activity

Solomon DH et al. Arthritis Rheumatol. 2015

Model A is adjusted for age and gender only, Model B is adjusted

for age, gender, and CV risk factors; Model C is adjusted for all

variables in Model B PLUS RA THERAPIES

IRRs of SIEs, malignancies and CVEs comparison of a later TNFi to earlier

TNFis before (a) and after (b) PS matching for baseline characteristics

associated with line of therapy

The impact of line of therapy when

comparing IRRs within a drug class

Harrold LR et al. Arthritis Res Ther. 2018

A. Before PS Matching

B. After PS Matching

Impact of cumulative disease activity

on CVD (MACE) events

Select use cases of Corrona’s data

Corrona RWE data has been leveraged to support regulatory approvals, label expansion and market

access

• Post-authorization Safety Study (PASS) to meet FDA and EMA requirements as part of new drug approval post marketing requirements (PMR)

• Effectiveness and safety data for new formulation approval for European regulators based on US real-world Corrona registry data

• Secondary indication approval by leveraging safety data in primary indication from Corronaregistry as part of regulatory submission

• US regulatory approval to extend label for approval as monotherapy, extending the current approval as combination therapy

• US market access negotiations with payers to expand market access for use as earlier line of

therapy

• European regulator negotiations to approve use of biologic in earlier lines of therapy

Confidential and proprietary information. Not for distribution. 131

Results of 5-Year Post-Authorization Safety Study for Pfizer on Tofacitinib vs Biologic DMARDs as part of FDA Commitment at Approval (PASS analytic protocol nested in the Corrona RA Registry)

132Confidential and proprietary information. Not for

distribution.

Hazard Ratios for MACE, SIEs and HZ

tofacitinib vs biologic DMARDs

Note: HZ includes non-serious and serious events; bbDMARD initiators were the reference population for calculation of HRs

Kremer J et al. EULAR, June 2019, OP0028

0.1 1.0 10.0

PS-trimmed

PS-matched

Lower rate with tofacitinib Lower rate with bDMARDs

1.15 (0.85, 1.55)

0.99 (0.72, 1.36)

2.28 (1.41, 3.68)

2.12 (1.22, 3.66)

0.66 (0.34, 1.29)

0.60 (0.30, 1.18)

0.001*

0.007*

1.01 (0.73, 1.38)

1.01 (0.73, 1.39)

2.27 (1.33, 3.88)

2.26 (1.28, 3.99)

0.58 (0.29, 1.15)

0.58 (0.30, 1.15)

0.003*

0.005*

Unadjusted HR

Adjusted HR

Generating RWE from Corrona RA Registry to support acceptance by European regulators for earlier use of abatacept for RA patients

Corrona demonstrated favorable Benefit/Risk profile for Rituximab vs Anti-TNF switching for RA patients with prior anti-TNF use

Trimmed populations

TNF inhibitor Rituximab

Events PY Event/100 PY Events PY Event/100 PY

Malignancy 13 760.9 1.7 (1.0,2.9) 5 270.5 1.8 (0.8,4.4)

Serious Infection 19 760.9 3.2 (2.0,5.0) 3 270.5 1.5 (0.5,4.5)

CVD Event 12 760.9 1.6 (0.9,2.8) 5 270.5 1.8 (0.8,4.4)

Adjusted Odds

Ratios for Major RA

Outcomes

Harrold et al. Arthritis Research & Therapy (2015) 17:256

Corrona RWE clinical effectiveness and safety data support EMA approval of new formulation

Cohen S, et al. Clinical Effectiveness of Tofacitinib 11mg Once Daily (QD) Versus Tofacitinib 5mg Twice Daily (BID) in the Corrona US RA

Registry [abstract]. Arthritis Rheumatol. 2018; 70 (suppl 10).

Primary and secondary outcomes at 6-month visit for patients who initiated

tofacitinib 11mg QD vs 5mg BID

Thank you

Chhaya ShadraVice President, Product Management - Data and Informatics

Verana Health, Inc.

CONFIDENTIAL AND PROPRIETARY

Chhaya ShadraJuly 2020

Generating Regulatory-Grade Real World Evidence Beyond Oncology

CONFIDENTIAL AND PROPRIETARY140

Oncology Longitudinal careBinary

(mortality rate)

Complete - patient experience

centralized in oncology care

Other specialtie

sEpisodic care

Variable, measured with various tools,

processes

Incomplete -snapshot of patient experience, focused on symptomatic care

Generating real world evidence in specialty areas outside oncology presents unique challenges

Continuity of care

Primary outcome measure

Completeness of health record

Verana Health combines deep clinical expertise with a highly automated and scalable next generation data infrastructure to generate RWE across specialty areas

Expert team of clinicians, biostatisticians and epidemiologists

Close partnerships with leading specialty societies

Specialty agnostic data architecture

EHR data curation through semantic harmonization & NLP

Data linking for enrichment

Deep ClinicalExpertise

Scalable DataInfrastructure

On the Forefront of Clinical Care

Trusted Relationships with Trial Sites and Patients

Leadership in Regulatory Engagement

Partnerships with specialty societies enable medical expertise and high trust relationships with physicians

• Influence clinical guidelines nationally• Deep experience in study design and challenges

• Relationships with large and prominent trial sites• Product feedback loop enabled by physician experts

• High credibility with regulatory agencies (FDA, CMS)• Leaders to advance adoption of real-world data for trials

The Verana team is rooted in deep clinical experience

Data Science and Epidemiology Expertise

Medical Expertise

● Academic Institutions – Stanford University, Duke University, Dana-Farber Cancer Institute, Broad Institute of MIT and Harvard

● Leading Companies – Flatiron, Syapse, ORNL, IQVIA, Aetion, Prometheus

● Dr. Matthew Roe, Chief Medical Officer ○ Duke University ○ Leadership of PCORnet- and FDA-

funded projects leveraging EHR data for prospective randomized studies

○ 15+ years of leadership experience in the National Cardiovascular Data Registry with ACC

● Team of specialty-focused medical directors, medical advisors and clinical consultants that span clinical practice, academic research and bioinformatics

● Through our specialty association partnerships, access and thought partnerships with key opinion leaders

30% of the Verana team has an MPH, MD, and/or PhD.Over 60% of the Verana team has a Master’s degree or higher.

The scale of Verana’s partner registries presents unique challenges

The Axon Registry®

• 5 years of longitudinal EHR data

• 12.4 million patient visits

• 2.4 million unique patients

• 1,126 contributing physicians

• 37 EHRs integrated

The IRIS® Registry

• 343 million patient visits

• 72 million unique patients

As of April 1, 2020 As of May 1, 2020

The AQUA Registry

• 37.8 million patient visits

• 6.3 million unique patients

As of December 31, 2019

Data challenges are amplified by the number of unique EHRs integrated and nature of specialty care delivery

Data variability, inconsistency, & incompletenessProvider behaviors & EHR layouts vary

Timeliness and lag dependent on practice delivery scheduleData continuity depends on how frequently practices push/allow pull of data

Setting represented in registries skewed towards specialty community careMissing health system encounters from PCPs, inpatient, other specialists

EHR data is our starting point to create regulatory-grade data sets with specialty depth

Access data from large patient populations by collaborating with medical societies and

existing registries

Specific details about patient populations becomes crucial as our understanding of disease improves and drugs become

more targeted

Assess safety and efficacy in the real world using outcomes data

(e.g. laterality, IOP, EDSS)

FootprintBreadth

ClinicalDepth

Outcomes

● EHR data collected from specialists

● Diagnosis, procedures, notes, specialist meds

● Structured/unstructured data streams

Linking additional data enables specific targeted analyses and insights

Commercial / Man. Medicare Rx

Images

Lab Data & Genomics

CMS Claims Data

EHR Alone

Linked Data Type Potential Analyses

● Outcomes associated with Tx regimen, adherence, mono vs. combo use, line of therapy● Health-resource utilization● Holistic patient demographics

● Comorbidity burden in elderly● Medicare health-resource utilization

● Imaging confirmation of Dx● Disease staging and segmentation by imaging● Validation of imaging CT endpts

● Patient journey

● Real world comparative effectiveness

● Potential safety issues in new therapies

● Potential strategies for trial enrichment

Insights

● Population segmentation ● PASS / safety associated with treatment● AI analytics for predictors of response

Verana’s data architecture is highly automated and specialty agnostic

Fit for purpose information models: Data ingestion

Shape: Database Model

Patient Medication

Encounter

Shape: ReSTful (JSON)

{ … }

[ … ]

{ … }

[ … ]

Shape: Delimited

Clinical Document

Shape: CCDA

Shape: VH Common Model

● Design Considerations○ C-CDA○ HL7 Feed formats○ FHIR Schema○ Custom Domain Fields○ Unstructured Text Fields○ Binary Image Fields○ Genomic Data Support

Fit for purpose information models: Observational research

Concepts, Ontologies and Vocabularies

patient identifier patient identifier patient identifierpatient identifier

Design Considerations

● PCORnet● OMOP● i2b2● Sentinel CDM

Patient data

● Demographics○ Birth year○ Gender○ Race○ Ethnicity○ Region

● Medical history

● Comorbidities

● Insurance type

Coded encounter data

● Visit dates and practices

● Diagnoses harmonized to ICD-10-CM, ICD-9-CM, SNOMED-CT

● Procedures & visit types harmonized to CPT, HCPCS

● Medications harmonized to RxNorm

Algorithmic enhancements derived from unstructured

● Real-world endpoints○ VA○ IOP○ T25-FW○ EDSS○ Gleason

● Disease severity measures

Additional data sources

● Linkages possible○ Medical claims○ Pharmacy

claims○ Lab test

results ○ Genetic test

results○ Images○ Non-specialist

EMRs○ Prescription

records○ PROs

Provider data

● NPI identifier

● Region

● Practice type○ Academic

Center○ Multi-Specialty

Group○ Urban/Rural

Data curation: Semantic harmonization

DOMAIN TERMINOLOGY OR ONTOLOGY

Conditions ICD-9-CM or ICD-10-CM or SNOMED-CT

Medications RxNorm

Procedures CPT or HCPCS or ICD-9-PCS or ICD-10-PCS

Observations (Relevant Subset) LOINC

Marital Status HL7 Administrative Gender

Race CDC Race Category

Ethnicity HL7 Ethnicity

Verana has developed a custom disease ontology that is relevant for clinical use cases in our specialty areas

• Disease ontology that is relevant for clinical use cases

• Based on how providers categorize and think about diseases

• We do not use an out of the box administrative classification to fit all purposes

• Clinical input is a key component of our Data Module specification and QA process

Verana disease ontology for Age-Related Macular Degeneration (AMD)

An end-to-end data quality framework is applied across the entire data pipeline

Cornerstones of Data Quality

Doesthe Data...

Technical Clinical Scientific

Completeness encompass the entire clinical picture?

Field completeness is assessed among fields where data is expected. (e.g. each diagnosis must have a documented date)

Data completeness in clinical context exists (e.g., intraocular pressure is expected to be documented for patients with a diagnosis of Glaucoma)

Factors (e.g., confounders) have been considered in study design & analyses

Accuracyaccurately reflect patient chart/reality?

Data conforms to expected data types & constraints

EHR effectively captures patient journey & provider patterns

Results are within range of scientific acceptability

Traceability contain provenance back to source? Data elements & transformations are clear and auditable during ingestion and curation

Study specifies a clear, auditable patient cohort

Study design, methods, and analysis are clear and transparent

Consistencymaintain integrity across structures, time, releases?

Data are represented in a consistent data model, under congruent architecture & format

Cohort-specific trends & rates are tracked across time

Data is validated against published studies & external sources

Generalizability represent a minimally-biased sample?Data elements are harmonized to industry standards

Biases have been assessed & accounted for in clinical interpretation

External comparisons are used to identify and adjust for biases

Timeliness reflect recent practice patterns?Data is refreshed at appropriate frequency

Current practice patterns, treatments are incorporated

Data timeframe is relevant to current study

1. Miksad, RA and Abernethy, AP. Harnessing the Power of Real-World Evidence (RWE): A Checklist to Ensure Regulatory-Grade Data Quality. Clin Pharm & Ther 2017;103:202-205.

2. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), Center for Devices and Radiological

Health (CDRH), Use of Electronic Health Record Data in Clinical Investigations Guidance for Industry, July 2018.

3. U.S. Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (CDER), Center for Biologics Evaluation and Research (CBER), Center for Devices and Radiological

Health (CDRH), Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data. May 2013.

Scaled approach to data quality: Overall scores

Appropriateness - Disease Burden . Patient Journey . CT Pre-Screening . Market Tracking

HEOR . Comparative Effectiveness Research . CT Site Selection and Recruitment . Scientific

Publications . Clinical Outcomes Research . Post-Authorization Safety Studies

CMS submissions for VBC/MIPS Quality Reporting . FDA Submission Support to Life-Sciences

Business Grade

ResearchGrade

Regulatory Grade

AGGREGATED DATA QUALITY SCORES

Ophthalmology

EHRs: 56 | Practices: 3128 | Providers: 18,234 | Patients: 60,231,521

Completeness Consistency Traceability Timeliness Accuracy Overall

0.82 0.79 0.65 0.95 0.83 0.81

Data Quality Scores Compared to Previous Data Refresh

(+10%) (+7%) (-5%) (+12%) (+5%) (+10%)

Key Takeaways

• There are meaningful opportunities to expand upon the successes of RWE in oncology for impact in other specialty areas, but also unique challenges

• Specialty society partnerships facilitate access to some of the largest specialty data sets in medicine and enables direct collaboration with physicians for data enrichment

• A scalable specialty-agnostic data platform can be built with generalizable data curation processes

• Data quality can be measured at scale yet must be adjusted to be fit for use case, though applying stringent data quality thresholds may lead to reduced cohort sizes

• Better data standardization and documentation in original systems of record like EHRs will reduce information loss across the data life-cycle

Session 3 — Audience Q&ATranslating Early Successes of Core Data Element Programs to Therapeutic Areas

beyond Oncology

Session 4: Emerging Tools & Technologies

to Support High-Quality Data Capture2:30 pm – 3:40 pm

Emerging Tools & Technologies to Support High-Quality Data Capture

Stephanie Reisinger, VP and GM, Veradigm Life Sciences

Automated Delivery• Drones• Self driving cars

Augmented Reality• Connected eyewear• Operating Room of

the future

Genetic Technologies• Low-cost sequencing• Gene editing

Internet of Everything• Smart appliances• At-home diagnostics• Connected clothing

Cheaper Computing• Private data cloud• Big data analytics

Robotics & Automation• Robotic surgery• Robotic caregivers• Exoskeleton

Blockchain• Patient medical records• Drug supply chain integrity• Clinical trials

3D Printing• Bioprinting – organs,

bones, teeth• Surgical instruments• Devices (e.g. pacemakers

Device Miniaturization• Smart tattoos and bandages• Medical grade wearables• Digital pills

Artificial Intelligence• Drug discovery/

development• Diagnosis• Patient Monitoring

DIGITAL HEALTH DEL IVERY IS CREATING AN

INFORMATION EXPLOSION…

… (real-world) data can be generated and disseminated nearly simultaneously

Macro Healthcare Trends

Research In the Context of Digital Health Delivery

<2% >95%

Of people in the U.S. participate in Clinical

Research

Of people in the U.S. receive Digital Healthcare

Services

• Specialized treatment protocols with narrow patient selection criteria

• Safety and efficacy focus

• Difficult to generalize across populations

• Broad heterogenous patient populations with many comorbidities

• Diverse treatment protocols

• A much better model to understand treatment generalizability

Would be willing to participate in Research

Ideal World Real World

McHugh, Kelly & Swamy, Geeta & Hernandez, Adrian. (2018). Engaging patients throughout the health system: A landscape analysis of cold-call policies and recommendations for future policy change. Journal of Clinical and Translational Science. 2(6). 384-392. 10.1017/cts.2019.1.

Today’s Reality:

Research is disconnected from digital healthcare delivery

Operating model developed before healthcare digitization

Data captured de-novo for each study

Forms-based data capture

Study data is siloed, difficult to integrate and reuse

Disconnected from point of care workflows - main focus on data curation post-hoc

Current research model is unable take advantage of digital and consumer-driven healthcare delivery

The Integrated Research Strategy

Research capabilities embedded into existing point of care workflows & technology

“Research-enabled” Point of Care

ENGAGED PROVIDERS

Research data a natural output of care process

ENGAGED PATIENTS

Alerts

EMR-embedded research

Messaging, Apps

Surveys, PROs

InsightsInsights

InsightsStudy Protocols

CONNECTED RESEARCH SPONSORS

Opportunities for Integrated Research (partial list)

Study Feasibility: data about patient characteristics & availability, site performance and investigator availability centralized & real-time

Digitized Protocols: standardize key protocol elements in quarriable format

Protocol, Physician, Patient Match-making: real-time identification of patients matching study protocols “at scale”

Provider Alerting: at the point of care

Patient Engagement: utilizing patient portals, and EHR patient engagement technologies

360° Insights: analytics customized to needs of different healthcare stakeholders from one “source of truth”

Digital Healthcare Journey: interoperability standards enable disparate digital healthcare data to be linked and connected

Synthetic Control Arms: modeled from real-world data previously collected

Virtual Trials and Remote Patient Monitoring

Telehealth in the time of COVID

~0 minutes per week in Jan. to 75,000-100,000 minutes per day in April-June

Telehealth utilization in PracticeFusion EHR Platform (~30,000 independent and small practice doctors)

Session 4: Emerging Tools & Technologies

to Support High-Quality Data Capture2:30 pm – 3:40 pm

Session 4 — Audience Q&AEmerging Tools & Technologies to Support High-Quality Data Capture

Data Ecosystem

Broadcast to Resume at 3:40 pm ET

Session 5: Next Steps Toward Establishing

a High-Quality RWD Ecosystem3:40 pm – 4:30 pm

Session 5 — Audience Q&ANext Steps Toward Establishing a High-Quality RWD Ecosystem

Closing RemarksMark McClellan

Director, Duke-Margolis Center for Health Policy

Thank You!

Contact Us Follow Us

DukeMargolis

@DukeMargolis

Duke Margolis

healthpolicy.duke.edu

Subscribe to our monthly newsletter at

DC office: 202-621-2800

Durham office: 919-419-2504

1201 Pennsylvania Avenue, NW, Suite 500

Washington, DC 20004

Establishing a High-Quality Real-World · Can place study results in a context of contemporary real...

Documents

Adobe’s Real Time Messaging Protocol

H.323 and Real Time Streaming Protocol

Real World Patients: The Intersection of Real World ... · Real World Patients: The Intersection of Real World Evidence and Episode of Care Analytics, continued 6 SAS REAL WORLD EVIDENCE

Real-world verification: the case of security protocol ... · Real-world veri cation: the case of security protocol standards Marko Horvat MPI-SWS 27/11/2017 1/12. Introduction Actor

P16-253 protocol pdfa - ClinicalTrials.govP16-253 Protocol Abstract Real World Study of the Effectiveness of Paritaprevir/r — Ombitasvir, + Dasabuvir without Ribavirin in Patients

Mobilizador 5G · RTK Real Time Kinematic RTP Real Time Protocol RTCP Real Time Control Protocol SDK Software Development Kit SDN Software Defined Networking SDP Session Discovery

How Real World Data can support Clinical Drug Development€¦ · 10 IMS Health Real-World Evidence Solutions Optimising Protocol Design Conducting Feasibilities in Clinical Trials:

SSL / TLS Case Studyshmat/courses/cs395t_fall04/02ssl.pdf · Introduction to the SSL / TLS protocol • Widely deployed, “real-world” security protocol Protocol analysis case

Real-World Challenges, Performance Measures Real-World Solutions · 2020-03-16 · Real-World Challenges, Real-World Solutions Using big data and information analytics Enabling Effective

QoE Aware Real-time Multimedia Streaming in Software ...€¦ · RTP Real-Time Transport Protocol RTCP Real-time Control Protocol RTSP Real-Time Streaming Protocol SCTP Stream Control

Open Access Protocol Real-world effects of medications for

RTP Real-Time Transport Protocol

Protocol Representation Model in the Real WorldProtocol Representation Model in the Real World Joshua Pines ... • Increased quality, consistency of clinical info • Align objectives,

The Real Time Transport Protocol (RTP)

Protocol Representation Model (PRM) in the Real World

· Problem Solving REAL WORLD . REAL WORLD roblem Solving . REAL WORLD roblem Solving

SVMPharma Real World Evidence – Real World Evidence & Adaptive Pathways

[MS-RTPRAD]: Real-Time Transport Protocol …...The Real-Time Transport Protocol (RTP/RTCP): Redundant Audio Data Extensions (RTPRAD) Protocol is a method for encoding redundant audio

Systematic Mapping Protocol - arXiv · 2016-09-22 · 2016 Systematic Mapping Protocol HAVE SYSTEMATIC REUSE BENEFITS BEEN TRANSFERRED TO REAL-WORLD SETTINGS? FINAL VERSION: 2016/08/02

Protocol Representation Model in the Real World - PhUSE€¦ · Protocol Representation Model in the Real World ... Brief Intro - The Theory Behind PRM! ... IMMS CIL / EST PIER 16