Using Metrics to improve study performance

© 2013 BioClinica, Inc. – Proprietary and Confidential

Global clinical trial solutions. Real-world results.

© 2013 BioClinica, Inc. – Proprietary and Confidential

Brian Bialkowski, PhD

Welcome to today’s BioClinica WebinarUsing Metrics to Improve Study Quality

See this content presented in an on-demand webinar @

bioclinica.com/resources/webinars

© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential 2

Defining Study Quality

A “quality” study is one that yields what the sponsor needs, when they need it.

• Enough Sites• Enough Subjects• Enough Study Drug• Enough Quality Data

What They Need

• Key Study Milestones• Budget• Shared Resources Schedules

When They Need It


Am I on Time?

Defining Study Quality

Do I Have What I Need?

Am I under Budget?

Measure Assess and Take Action

SiteABC357ABC579ABC123ABC456ABC789ABC246ABC468ABC135

???????

???????

???????

© 2013 BioClinica, Inc. – Proprietary and Confidential 4© 2013 BioClinica, Inc. – Proprietary and Confidential 4

• First Site• First Subject In• FPFV• Enroll Complete• LPLV• Data Lock

Milestones

• Site Statuses• Screening• Screen Failures• Enrollment• Dropouts

Enrollment

• Protocol Deviations• SAEs• Monitoring Visits• Out-of-Window

Visits

Site Monitoring

• Status• Expiration Dates

Documents

• Data Collection• SDV• Data Lock• Queries• Cycle Times

Data Management

• Shipments• Shipment Contents• Kit Statuses• Dosing Visits

Supply Chain


Selecting and Using Metrics

• Understand types of metrics and their uses• Know your audience and their key questions• Sequence metrics and action


What Is a Metric?

A metric is, simply put, a measurement.

What

WhyHow


How We Measure It

(Subject)

Detail and Context

Site

(Individual Records)

Aggregation

Country

Program

Study

RegionRates

Variance

KPI

Focus Metric Type

Summary and Judgment

2012 2014Trending and Forecasting

2013


Sample Exercise

• 10 Sites with enrollment data• Each site has an enrollment target.• All sites began enrollment in January 2012.• All were expected to complete enrollment in

March 2013.• Track # of subjects enrolled each month at the

sites.


Simple Aggregation

• Clear message.• Suitable for description.• Clear indicator of scale.• No built-in interpretation of the results.


Normalized Measurements (Rates and Variance

• Normalize measurements across time and scale.• Other normalized metrics: SAEs per visit; Protocol deviations per subject; Queries per 100

data items• Abstracting from raw data adds context for evaluatiion. • Base for trending (“When did things go wrong?”) and forecasting (“Where is my study

going?”)


KPI (Visual Indicator)

• Most easily digestible type of metric• No context on actual enrollment or target numbers; color values set using thresholds for

performance against targets.• Many options for indicators—traffic lights, arrows, gauges, etc.• Drives decision to take action, but not a tool for the action itself.

Green: >95%

Yellow: 80%-95%

Red: <80%


How We Measure It

(Subject)

Detail and Context

Site

(Individual Records)

Aggregation

Country

Program

Study

RegionRates

Variance

KPI

Focus Metric Type

Summary and Judgment


What Could You Measure?

• First Site• First Subject In• FPFV• Enroll Complete• LPLV• Data Lock

Milestones

• Site Statuses• Screening• Screen Failures• Enrollment• Dropouts

Enrollment

• Protocol Deviations• SAEs• Monitoring Visits• Out-of-Window

Visits

Site Monitoring

• Status• Expiration Dates

Documents

• Data Collection• SDV• Data Lock• Queries• Cycle Times

Data Management

• Shipments• Shipment Contents• Kit Statuses• Dosing Visits

Supply Chain


My data management team...

“Which forms are generating the most queries?”

Audience, Questions, and Outcome

My executive…“How many sites are below their enrollment target?”

My supply chain manager…

“Does my depot have enough study drug for the next 6 weeks of visits?”

And then…


There’s Never Just One Question…

• Am I enrolling enough subjects today...• And am I on pace to hit my enrollment targets (# and date) in

12 months…• And did I build enough screen fails and early terminations

into my budget…• And are some sites losing more subjects than others…• And do I have enough capacity to make up enrollment

shortfalls at other sites…• And are treatment groups divided proportionately across all

my sites?


Sequence Your Use of Metrics

• Which questions do you need to ask first?• What information do you need to answer them?• What questions do they lead to?

Question 1(Metric 1)

Question 2 (Metric 2)


Question 5(Metric 5)


Assess Analyze Action


Picking the Right Metrics for Your Study


Early Study

% Enroll vs. Target to Date

% Active Sites vs. Target to Date

CRF Entry Lag

Site Statuses

Screen Fail Rate Document Collection

Screen Fail ReasonsDiscontinued Rate

Discontinued Reasons

Site Milestone Targets vs Actuals

Study

Site


Query Rates Query Counts by Form


Early Study Dashboard

1001 1002 1003 1004 1005 1006-30%-25%-20%-15%-10%

-5%0%5%

10%15%

Enrollment vs Target

1001

1002

1003

1004

1005

1006

0% 5% 10% 15% 20% 25% 30% 35%

Screen Fail, Discontinue Rates

Discontinued Screen Fail

Target to DateActiveOpen

Start-UpEvaluationIdentified

0 1 2 3 4 5 6 7

Site Statuses

1001

1002

1003

1004

1005

1006

0 5 10 15 20 25 30

Queries/100 Data Items

Current Enrollment: 85 Target to Date: 95Current # Sites: 6 Target to Date: 5.8

1001

1002

1003

1004

1005

1006

0 2 4 6 8 10 12 14

CRF Entry Lag (Days)


Mid-Study

% Enroll vs. Target to Date

# Out-of-Window Subjects

Shipment History

Screen Fail Rate

Discontinued Reasons

Shipment Damaged/Lost

Rates

Discontinued Rate

Query Cycle Times

Expired/Destroyed Study Drug

Screen Fail Reasons Study

Site


% CRF SDV/Locked

Late Action Items

Inventory (Kit Statuses)

Query Cycle TimesQuery Rates

CRF Status


Mid-Study Dashboard

1001 1002 1003 1004 1005 1006-30%-25%-20%-15%-10%

-5%0%5%

10%15%

Enrollment vs Target

1001 1002 1003 1004 1005 10060

2

4

6

8

10

Out of Window Subjects

1001

1002

1003

1004

1005

1006

0% 10% 20% 30% 40% 50% 60% 70%

% Collected CRFs SDV and Locked

Locked Verified

1001

1002

1003

1004

1005

1006

0 5 10 15 20 25 30

Queries/100 Data Items

Current Enrollment: 85 Target to Date: 95 Late Action Items: 24

1001 1002 1003 1004 1005 100605

10152025303540

Kit Statuses

Available ExpiredQuarantined


Late Study

Study MilestoneTargets vs Actuals

% Data Locked

Queries Open/Answered >7

days

Open Action Items

CRF Statuses

Study

Site

Assess Analyze/Action

Query Statuses

Upcoming Site Milestones


Late Study Dashboard

1001

1002

1003

1004

1005

1006

0 20 40 60 80 100 120

Open/Answered Queries

Answered > 7 days Open > 7 Days

1500

1200

25000

Query Statuses

Open Answered Accepted

1001

1002

1003

1004

1005

1006

0 5000 10000 150002000025000 30000

CRF Statuses

Locked Verified ILB Entered

Milestone Baseline Revised Actual

Enrollment Complete 04-Feb-2014 08-Mar-2014 12-Mar-2014

LPLV 15-May-2014 21-May-2014

Data Lock 15-Jun-2014 1-Jul-2014

Final Report 1-Jul-2014 15-Jul-2014

Study Close 1-Jul-2014 15-Jul-2014

% Data Locked: 80%

Study Milestones Pending Site Milestones

Site Milestone Baseline Revised

1004 LPLV 24-Apr-2014 18-May-2014

1002 LPLV 15-May-2014 21-May-2014

1003 LPLV 1-May-2014 14-May-2014

1004 Last CRF 23-May-2014 30-May-2014

1005 Last CRF 24-May-2014 4-Jun-2014


Data Management Metrics

% Expected Pages Received

% Forms Locked

Cycle Times (Open, Answered, Accepted)

Data Entry Lag

Query Counts by Form

CRF Page Status

Queries Open/Answered >

7 days

Study

Site


Query Counts by Status

% SDV


Data Management Dashboard

Visit 1 Subject Questionnaire Visit InfoVisit 2 Pain Questionnaire

Visit 1 PAGI-Sym QuestionnaireVisit 3 Pain Questionnaire

Visit 3 McGill Pain QuestionairreVisit 4 Pain Questionnaire

Visit 4 McGill Pain Questionnaire

0 100 200 300 400 500 600 700

Queries by Form

% Expected CRFs received 85%

% SDV 55%

% Locked 25%

2400

1200

8000

Query Statuses

Open Answered Accepted

100110021003100410051006

0 2 4 6 8 10 12 14

CRF Entry Lag (Days)

100110021003100410051006

05000

1000015000

2000025000

30000

CRF Statuses

LockedVerifiedILBEntered

1001

1002

1003

1004

1005

1006

0 20 40 60 80 100 120

Open/Answered Queries

Answered > 7 daysOpen > 7 Days


Planning and Implementation

• Agree on most important questions.Define• Ensure your system(s) can supply the necessary

data.Validate• Build a model/system that will generate your

metrics.Automate• Combine metrics with action, and execute on

that plan.Value• Share metrics with employees; build them into

processes.Share

• Understand types of metrics• Know your key questions• Sequence metrics and actions they’ll

trigger

Other steps…• Work with the data you have• Range of technical options

© 2013 BioClinica, Inc. – Proprietary and Confidential© 2013 BioClinica, Inc. – Proprietary and Confidential

Thank You!

Brian [email protected]

See this content presented in an on-demand webinar @

bioclinica.com/resources/webinars

Health & Medicine

Using Metrics to improve study performance