The World of

FDA Quality Metrics

Yesterday, Today and Tomorrow

2

CDRH: Resource Concerns

FDA can only inspect a portion of firms

5

9566

Inspected

Not inspected

Percent of firms inspectedDomestic; Annual

Percent of firms inspectedForeign; Annual

91 Warning Letters were issued in the 2011 fiscal year –3% of total investigations1 resulted in a Warning Letter

Presented by Steve Silverman at 2012 FDA/Xavier MedCon Conference. 3

CDER: Historical Focus of Staff

FDA Staffing vs. Patient Exposures

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

16000000

18000000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Year Post IND Submission

Pa

tien

tE

xp

osu

res

0

2

4

6

8

10

12

14

FD

AS

taff

ing

Patients Exposed

FDA Staff

PRE-MARKET FOCUS

Presented by Russ Wesdyk at 2014 FDA/Xavier PharmaLink Conference. 4

End Result

Use metrics to risk rank

companies so FDA has data

to support where it is

focusing its resources.

And where FDA shouldfocus its resources

5

6

FDA Safety and Innovation Act“FDASIA”

Main Driver:

Passed into Law

July 9, 2012

7

FDA Authority Differences

Drugs– FDASIA officially allows FDA to use risk based decisions to determine how best

to use limited resources for drug inspections (Sec. 705)

– FDASIA allows FDA to request records in advance of or in lieu of an inspection(Sec. 706)

– FDA is not expressly given authority to use 3rd parties to assist with druginspection coverage

Devices– FDASIA does not officially allow FDA to use risk based decisions to determine

how best to use limited resources for device inspections

– FDASIA does not allow FDA to request records in advance of or in lieu of aninspection

– FDA already had authority to use 3rd parties to assist with device inspectioncoverage

CDER Call for Metrics

CDRH Case for Quality

8

9

• FDA does not have the authority to ask for “metrics” since thereis no regulatory requirement for industry to track metrics.

• FDA can ask for “data” since industry is required to monitor the safety andeffectiveness of its products

• FDA would have to convert all the data received into metrics

CDER: Call for Metrics

• CDER first determined how to address their resource constraints = throughmetrics

• CDER then asked for industry input on what to measure

– Industry associations and some companies provided CDER with metrics ideas

– Xavier expressed concern about the call for metrics approach, and launched aninitiative with 32 industry professionals to identify metrics for industry use.

– CDER spent 2 years in public meetings to gather industry input

– CDER took all the ideas into consideration, then published a draft guidance July 28,2015 with the “data” it is proposing to collect

10

CDRH: Case for Quality

• Steve Silverman (former Director, CDRH Office of Compliance)launched the Case for Quality in 2011

– CDRH wanted to shift the mentality of FDA and industry from compliance toquality, and therefore, defined what

– CDRH then turned to industry to explore how

– This is opposite of the CDER approach

• CDRH led National Case for Quality forums to gather ideas on how tocollaboratively focus on quality (including the 2013 Xavier NationalForum)

• Silverman asked Xavier to partner with FDA to explore ways to“measure” quality risk

11

Focus on shifting the Right First Time mentality as close to the initialdays of development as possible

ORA Perspective

12

Collaborative Effort

(multiple relevantsources of info)

Presented by Ellen Morrison at FDA/Xavier PharmaLink Conference, March 2016.

ORA Perspective

13Presented by Ellen Morrison at FDA/Xavier PharmaLink Conference, March 2016.

14

Metrics Teams: Side-by-Side

Device

• Co-Leader: Kristin McNamara,FDA Senior Advisor

• Number of Team Members: 46– FDA Members: 7

• Number of Companies: 25

• Timeframe:– September 2014 – August 2016

Pharma

• Co-Leader: Sam Venugopal,Partner PwC

• Number of Team Members: 32– FDA Members: 0

• Number of Companies: 23

• Timeframe:– August 2014 – January 2016

15

Goal: Identify MeaningfulMetrics for Industry

To provide a system of metrics across the TotalProduct Lifecycle that:

1. Informs internal company decisions and triggers action

2. Shifts the Right-First-Time mentality closer to the initialdays of development

Xavier Pharma and Device Metrics

16

Total Product Lifecycle View

Pre-Production ProductionTransfer

ProductionContinual Improvement

& Risk Mgmt.

Enterprise-WideContinual Improvement

R&DContinual Improvement

& Risk Mgmt.

Post-Production

Metrics Comparison Summary

TPLC PhasesFDA/Xavier

Device MetricsXavier/PwC

Pharma MetricsCDER

Draft Metrics

Pre-Production

Design Robustness

Design Space

Supply Chain Assurance

TechnologyTransfer

RFT of Analytical Transfer

RFT of Process Validation

Production RFT for Production

RFT for ProductionLot Acceptance Rate

CAPA Effectiveness Rate

Commitment IndexInvalidated OOS Rate

Supplier Risk Index

Post-Production Post-Market Index Market Reliability Index Complaint Rate

Enterprise-wide Management ReviewQbD Effectiveness

(APR on Time Rate)Root Cause of RFT 18

To Inform Decisions and Trigger Action

Step 1. Severity Classifications

Severity

Classification

Severity

Weighted

Value

Severity Definition

Catastrophic50

Potential for Death

Critical30

Potential for Serious Injury

Marginal19

Potential for Non-Serious Injury

Negligible

1

Minor Customer Annoyance, Cosmetic Issue, No

injury to patient

Risk Analysis of Complaints

Step 2. Risk of Each Complaint

1st Quarter Product A Risk Profile Score:

1000 Complaints for period1 Critical Complaint/1000 Complaints X 30 X 100 = 33 Marginal Complaints/1000 Complaints X 19 X 100 = 5.7996 Minor Complaints/1000 Complaints X 1 X 100 = 99.6Total Risk Profile Score = 108.3

2nd Quarter: Product A Risk Profile Score:

1000 Complaints for period2 Critical Complaints/1000 Complaints X 30 X 100= 64 Marginal Complaints/1000 Complaints X 19 X 100 = 7.6995 Minor Complaints/1000 Complaints X 1 X 100 = 99.5Total Risk Profile Score = 113.1

Step 3. Rate of Occurrence

QuarterNumber of

Complaints

Number of

units

released

Complaint

Rate TotalMultiplier

Reported

rate

1st 1,000 10,000 0.1 1,000 100

2nd 1,000 15,000 0.067 1,000 67

3rd 500 18,000 0.028 1,000 28

4th 2,500 10,000 0.25 1,000 250

Step 4. Side-by-Side Comparison

1st Quarter

Product A

2nd Quarter

Product A

3rd Quarter

Product A

4th Quarter

Product A

Complaint Rate 100 67 28 250

Complaint Risk

Profile Score108.3 113.1 134 136

Heat Map Correlation

Y-axis =InternalRisk Score

X-axis =ExternalRisk Score

Each Point =Total Risk to Product Quality

“Internal” includespre-production andproduction metrictotal risk score

“External” includes thetotal post-productionrisk score ofappropriate indicators 25

Enterprise-Wide Assessment

Pre-Production ProductionTransfer

ProductionContinual Improvement

& Risk Mgmt.

Enterprise-WideContinual Improvement

R&DContinual Improvement

& Risk Mgmt.

Post-Production

Spider Chart

28

Reminder: Metrics Comparison

TPLC PhasesFDA/Xavier

Device MetricsXavier/PwC

Pharma MetricsCDER

Draft Metrics

Pre-Production

Design Robustness

Design Space

Supply Chain Assurance

TechnologyTransfer

RFT of Analytical Transfer

RFT of Process Validation

Production RFT for Production

RFT for ProductionLot Acceptance Rate

CAPA Effectiveness Rate

Commitment IndexInvalidated OOS Rate

Supplier Risk Index

Post-Production Post-Market Index Market Reliability Index Complaint Rate

Enterprise-wide Management ReviewQbD Effectiveness

(APR on Time Rate)Root Cause of RFT 29

CDER: Call for Metrics

Date Description

2012 FDASIA Passed

2013 CDER Quality Metrics Initiative launched

2013 – 2015 Call for metrics from industry• ISPE, PDA, GPhA, PhRMA, BIO, etc. provided CDER with metrics

ideas• Xavier/PwC initiative focused on metrics for industry use

2015 (July 28) Draft Guidance: Request for Quality Metrics• FDA’s response to comments have not been shared publically

2016 (June 27) Draft Guidance: Quality Metrics Technical Conformance Guide• Provides guidance on how industry would be expected to

provide data to FDA

2016 (Nov. 25) Revision 1 of the July 2015 Draft Guidance. Now entitled:“Submission of Quality Metrics Data”• Reduces number of metrics, and makes the program voluntary

30

Additional CDER Timeline

The November 2016 Draft Guidance “Submission of QualityMetrics Data” provides the following information:

• 2017:– Voluntary Program

– Anticipate that the electronic submission platform will be available to test in2017

• 2018:– “after evaluation of the results, the FDA intends to initiate notice and

comment rulemaking under existing statutory authority to develop amandatory quality metrics reporting program.”

31

CDER November Draft Guidance

Beyond the Voluntary Program, CDER intends to use the metrics inthe following way:

• Establish a signal detection program as one factor in identifyingestablishments and products that may pose significant risk toconsumers

• Identify situations in which there may be a risk for drug supplydisruption

• Improve the effectiveness of establishment inspections

• Improve FDA’s evaluation of drug manufacturing and controloperations.

32

CDRH: Case for Quality

Date Description

2011 The Case for Quality was launched by Steve Silverman (formerDirector, CDRH Office of Compliance)

2011 - 2013 CDRH led national forums to gain industry input

2012 FDASIA Passed

2014 - 2016 Xavier University partnered with FDA to lead a team of 46 FDAofficials and industry members to identify key metrics, and pilotthose metrics, for industry to assess its own risk to product quality

2015 - 2016 The FDA/Xavier Initiative was adopted by the Medical DeviceInnovation Consortium (MDIC), and is the basis for FDA priorities.

2017 FDA to determine how to incorporate product metrics, culture andquality system metrics into risk decisions. HOWEVER – CDRH is stillfocused on Industry ownership of quality. 33

CDRH 2016-2017 Strategic Priorities

Promote a Culture of Quality andOrganizational Excellence

• By September 30, 2016, develop metrics, successful industry practices,standards, and tools that manufacturers can use to evaluate product andmanufacturing quality beyond compliance with regulatory requirements.

• By December 31, 2016, pilot voluntary use of product and manufacturingquality metrics and evaluation tools.

• By December 31, 2017, propose a voluntary program to recognizeindependent evaluation of product and manufacturing quality.

Goal: Strengthen Product and Manufacturing QualityWithin the Medical Device Ecosystem

34

What Does This Mean?

FDA Regulatory Paradigm Shift

What does a focus on quality and organizational excellence mean for FDA andinnovation?

• Increased manufacturing and product confidence• Faster time to markets, better information to drive regulatory decisions,

improved resource allocation• A focus on what is most important to patients

Remove participants from the agency work plan for routine inspections

Waive pre-approval inspections where appropriate

Engagement and meetings on issue resolution

Modifying submission requirements and faster FDA response

Accelerated approval path

Competitive market around product excellence

36

Impact of Metrics Use

FDA should not use metrics out of context

– Leads to unsubstantiated conclusions

– Drives unintended consequences

37

Therefore

Assess metrics on inspection However

This does not alleviate resource constraints

Focus on Mutual Reliance and MDSAP

So then

Metrics are a Piece of the Puzzle

38Presented by Ellen Morrison at FDA/Xavier PharmaLink Conference, March 2016.

Xavier Asked: What does industry want?

No inspections from FDA?– Not really. Industry wants the touch point with FDA to verify

alignment with expectations.

– More time that passes could result in significant findings if thecompany was not completely aligned

So then what does industry want?1. Inspections of competitors to level the playing field

2. Combined inspections of multiple regulatory authorities -mutual reliance (pharma) and MDSAP (device)• Just a warning that these inspections carry more weight!

3. Faster approval of submissions for demonstrating low risk andhigh control

Reactions and Ideas?

40

Marla PhillipsDirector, Xavier HealthXavier Universityphillipsm4@xavier.edu

www.XavierHealth.org