Upload
vodung
View
447
Download
34
Embed Size (px)
Citation preview
Test Method Validation (TMV) | Mar 2016 | 1
Mai Nguyen-Misra and George Omae
Medtronic
Test Method Validation (TMV)
How to Conduct TMV for Medical Device Industry
Test Method Validation (TMV) | Mar 2016 | 2
Presenter Bio’s
Mai Misra, Ph.D. Sr. Principal Quality Engineer, Medtronic Ph.D., Polymer Science; M.S., Physics (Solid State); M.S., Chemical Engineering; B.S., Applied Mathematics; B.S., Chemical Engineering Dr. Mai Misra, a chemical engineer by training, has diverse education & work experience. Mai received her PhD from University of Akron in Polymers Science in 1995. She also has a BS degree in Chemical Engineering and Applied Mathematics as well as a MS in Chemical Engineering and a MS in Solids State Physics. She has worked for over 20 years in process and product development for various applications serving diverse industries: US EPA (Reformulated Gasoline and Alternative Fuels), H.B. Fuller (Packaging, Window, Footwear etc), Boston Scientific (Cardiovascular Drug-Eluting Stents), and Medtronic (CRDM Pacing Leads). Mai is a co-inventor of 4 patents and has authored many peer-reviewed papers. She has been in the Medical Device industry for 9 years. Mai has been in the current role as a SME on TMV for the validation group (part of Neuromodulation Ops Quality) since 2014.
George Omae, M.S. ASQ CSSB, CQE Sr Engineering Manager, Medtronic B.S. Mechanical Engineering, Nairobi University M.S. Manufacturing Systems Engineering, Univ. of St. Thomas M.S. Applied Statistics, Rochester Institute of Technology MBA, Carlson School of Management, University of Minnesota George Omae, a mechanical engineer by training, has over 19 years experience in the Medical device industry and has worked in roles of increasing responsibility in Quality, Process Development and R&D. He currently manages a test development group. He has had significant experience developing processes and products for 510k and PMA while working at Guidant, Boston Scientific, American Medical Systems and Medtronic. As a Six Sigma Black Belt, he has trained and mentored engineers and other technical professionals in the application of quality engineering tools. He is also an experienced trainer for Test Method Validation, Process Validation and Sampling Plans. George is also an adjunct instructor at the St Cloud State University graduate school (Medical Technology Quality (M.S.))
Test Method Validation (TMV) | Mar 2016 | 3
Course Objective & Approach
Objectives: • Provide an overview of what is required to plan and execute
TMV activities
• Provide guidance on how to document TMV records to be in compliance with prescribed validation requirements
• Share best practices
Approach: • Presentation,
• Case Study,
• Audience Interaction
Test Method Validation (TMV) | Mar 2016 | 4
Course Flow
Section 1: TMV Overview
Presenter: Mai Nguyen-Misra
Section 2
Case study (Variables TMV)
1. TMV Planning
2. Executing & analyzing
3. Documenting TMV
4. Maintaining the validated state
Presenter: George Omae
Lessons Learned Mai Nguyen-Misra
1 2
3
Test Method Validation (TMV) | Mar 2016 | 5
Section 1
Test Method Validation (TMV) Overview
Test Method Validation (TMV) | Mar 2016 | 6
Benefits: • Assures consistence & compliance in the output • Ensures requirements are accurately tested throughout the
Product Lifecycle • Builds basis for evidence of requirements lifecycle traceability
TMV Purpose and Benefits
Purpose:
To demonstrate that a test or inspection method is suitable for its intended purpose and is capable of producing valid results.
Test Method Validation (TMV) | Mar 2016 | 7
Definitions
• Test Method: A definitive procedure that produces a test result. It includes any sample preparation, measurements, observations and calculations.
• Test Method Validation (TMV): The process by which a test method has been demonstrated with objective evidence to be suitable for its intended purpose and capable of producing valid results.
Test Method Validation (TMV) | Mar 2016 | 8
TMV Overview (inputs & output) – Risk-based
Test Method Validation (TMV) | Mar 2016 | 9
TMV Types
Analytical
QMSWI16303
Pharma
PHAP1191
Fluke A Fluke B
Test Method Validation (TMV) | Mar 2016 | 10
Group Discussion – Sharing best practices
• What types of TMV does your organization commonly work with?
• Does your organization create separate work instruction for each type of TMV?
Test Method Validation (TMV) | Mar 2016 | 11
Factors that influence TM Variability
Workpiece (Part)
Measurement System
Variability/Error
Cause Effect
Standard/Reference Instrument
(Gage)
Person (Appraiser)
Environment
Traceability Design
Maintenance
Skill
Procedure
Ergonomics (e.g. lighting)
Vibration
Processing variability
Cleanliness Stability
Influencing factors to consider for control
Stability
Test Method Validation (TMV) | Mar 2016 | 12
TMV Process Flow
The governing procedure defines the process for demonstrating that a test or inspection method is suitable for its intended purpose and is capable of producing valid results.
Test Method Validation (TMV) | Mar 2016 | 13
Process Activities: Lifecycle Approach
4 TM Phases:
I – Good science/engineering
II – Preliminary Test Method
III – Characterized Test Method
IV – Validated or Verified Test Method
• TMs at Phase I,II,III require full documentation of characterization elements
• Phase IV TMs require validation to either Level 1 or Level 2 based on the Risk Identification
No requirement to go through these sequentially or document evidence of prior phase completion
Test Method Validation (TMV) | Mar 2016 | 14
TM Minimum Requirements
Test Method Validation (TMV) | Mar 2016 | 15
Process Inputs (pre-requisites) of TMV
Big 6!
Requirements
Equip qualified
Software validated
Tooling/Fixtures released
Harm/Risk classification
Documented test method
Training documented
Pre-requisites Tool/Source Comment
1
2
4
5
6
Product/print specE.g. Lead body tensile ≥ 7 lbf,
DC resistance ≤ 107Ω, Length: 45cm ± 2cm
E.g. IQ NRP1086-35723 (protocol), NRP1086-36375 (report),
OQ NRP1086-35724 (protocol), NRP1086-36376 (report)
Critical/Major/High risk need TMV,
Provides validation level & sample sizes
Example of content in QMS16001
Train to TM, store in Saba/other system
Per QMS1848E.g. MEDN-3285 (IQ)
E.g. PCPWI8362
E.g. QMS1799, DFMEA/PFMEA/PCRW
QMS16001MEDN-0002
E.g. MEDN-31261 MEDN-0093
Inspection report
3
Test Method Validation (TMV) | Mar 2016 | 16
TMV Level (Risk-based)
Determine TMV level based on risk:
Level 1 TMV – Moderate RL Gathering & documenting evidence to demonstrate TM capability Documentation : Report only Level 2 TMV – High RL Level 1 + additional study data to demonstrate TM capability Documentation: Protocol and Report
Test Method Validation (TMV) | Mar 2016 | 17
Maintaining a Validated State
Active TMs only
• Control Plans are required for Active Test Methods
Test Method Validation (TMV) | Mar 2016 | 18
Controls and Monitoring
• Document: • Type of controls, including frequency & criteria
• Required general controls (e.g., training, equipment calibration & maintenance)
• TM-specific controls where applicable (e.g., periodic re-training, quality control sample checks)
• Type & Location of evidence of implementation
• Document controls and actions to be taken if controls fail in TMV report
• Conduct Periodic Reviews
Test Method Validation (TMV) | Mar 2016 | 19
Maintaining a validated state (cont’d)
To Manage /Control Changes:
• Assess impact of changes – Consider impact of cumulative effects of all changes since validation
• Re-validate or provide justification
• Document decision and/or results
Some examples of changes
• TM changes: TM parameters, Equipment/Tooling, SW
• TM Intended Use Requirement Changes: Specifications, Risk level
Test Method Validation (TMV) | Mar 2016 | 20
Group Discussion – Sharing best practices
• What is your process for Maintaining Validated
State?
Test Method Validation (TMV) | Mar 2016 | 21
Section 2
Variables Test Method Validation
Test Method Validation (TMV) | Mar 2016 | 22
Module 2: TMV - Variables Key Elements Covered:
• Evidence required to demonstrate completion of pre-requisites
• How to design the validation study (sample size, sample part selection, appraiser selection, validation study steps)
• Case Study: Manual DC Resistance Test Method
Section Agenda
Test Method Validation (TMV) | Mar 2016 | 23
Variables TMV Overview
Full verification (100% inspection) – L2b Process control – L2c Acceptance sampling – L2a
Population
Inference
Sample
Test Method Validation (TMV) | Mar 2016 | 24
Module 2: TMV Performance Characteristic Requirements
Key differences
Test Method Validation (TMV) | Mar 2016 | 25
Variables Test Method Case Study: Manual DC Resistance
Test Method Validation (TMV) | Mar 2016 | 26
Case Study - Introduction
Twist-Lock DC Resistance Test Method
A test engineer is getting ready to conduct 100% incoming inspection and design verification testing. One of the requirements (specification) is to test the manual DC resistance of an improved cable used for initial evaluation for patients that have overactive bladder. The resistance per conductor must be between 0 - 10Ω. The risk level is high for this requirement.
Measure this resistance Verify Evaluation System
Test Method Validation (TMV) | Mar 2016 | 27
Case Study - Introduction
TMV Questions:
1. What is the minimum validation level required?
2. What is the %Resolution?
3. What is the %Tolerance?
4. What is required to maintain the validated state of the TM?
Measure this resistance Verify Evaluation System
Test Method Validation (TMV) | Mar 2016 | 28
Case Study - Select Validation Level
Start
Phase IV (100% Inspection) High Risk
Level 2 TMV (Variables nondestructive)
Test Method Validation (TMV) | Mar 2016 | 29
Case Study - Conduct TMV
Start
Test Method Validation (TMV) | Mar 2016 | 30
Case Study - Plan the study
Key planning items Requirement
Pre-requisites: Ensure all are released and controlled
Number of appraisers: 2
Number of parts: 10
Number of trials x3 per appraiser
Part selection: Select from typical/nominal processing conditions or surrogates with rationale
Label & randomize: Label & create randomization scheme using Minitab
Measurement location: Identify location of measurement
Sample size
Other considerations
Test Method Validation (TMV) | Mar 2016 | 31
Case Study - Plan the study
MXXXX-XXX Report Template used
Question: What validation level is selected for 100% inspection?
Test Method Validation (TMV) | Mar 2016 | 32
Case Study - Plan the study
Brief Description of the TM and Requirements
Test Method Validation (TMV) | Mar 2016 | 33
Case Study - Plan the study
Sample size & justification
Equip info
Test Method Validation (TMV) | Mar 2016 | 34
Case Study - Execute protocol
Identify: Validation execution roles
Document: Step-by-step procedure for executing validation/verification. Include role performing each step.
Plan Study Verify Pre-requisites
Execute Protocol
Analyze & Determine
Acceptability
Document results
Test Method Validation (TMV) | Mar 2016 | 35
Case Study - Execute protocol
Randomize Data
Collection
Blind Data Collection
Test Method Validation (TMV) | Mar 2016 | 36
Case Study - Analyze
Resolution
Show your calculations
Plan Study Verify Pre-requisites
Execute Protocol
Analyze & Determine
Acceptability
Document results
Test Method Validation (TMV) | Mar 2016 | 37
Case Study - Analyze
% Tolerance
%Tolerance = 4.27
Test Method Validation (TMV) | Mar 2016 | 38
Do:
• State how the data was analyzed
• Present results with enough detail to show how acceptance criteria were met
• State where evidence is located (attaching in Appendices preferred)
Case Study - Document
Plan Study Verify Pre-requisites
Execute Protocol
Analyze & Determine
Acceptability
Document results
Test Method Validation (TMV) | Mar 2016 | 39
Case Study - Document
Cite references used for the validation
Must have been
mentioned in report
List supporting data attached to report
Must be mentioned in
the report
Test Method Validation (TMV) | Mar 2016 | 40
Case Study - Maintain the Validated State
Workpiece (Part)
Measurement System
Variability/Error
Cause Effect
Standard/Reference Instrument
(Gage)
Person (Appraiser)
Environment
Traceability Design
Maintenance
Skill
Procedure
Ergonomics (e.g. lighting)
Vibration
Elastic deformation
Cleanliness Stability
Influencing factors to consider for control
Calibration & Preventative maintenance schedule
e.g. Annual recertification
Test Method Validation (TMV) | Mar 2016 | 41
Case Study - Document
Use SWIPE model to help document
controls
Describe general controls
Describe TM specific controls
State where change impact assessment is captured when
changes occur
Test Method Validation (TMV) | Mar 2016 | 42
Raw Data Attachment
For Legibility: Use the Snipping Tool or similar to capture and attach evidence in your TMV record
Test Method Validation (TMV) | Mar 2016 | 43
Case Study - Document (raw data)
Equipment used identified
Legible raw data
Clear sample identification
Randomization demonstrated
Test Method Validation (TMV) | Mar 2016 | 44
Lessons Learned
Test Method Validation (TMV) | Mar 2016 | 45
Lessons Learned
Key Challenges:
• Inconsistent record quality
Solutions:
• Provide templates to ensure consistency and completeness
• Training and proficiency assessment
Test Method Validation (TMV) | Mar 2016 | 46
TMV Templates
• Ensure requirements were met
• Provide consistent format to “tell the story”
Help Ensure TMV Records Are:
• Compliant – ensure Author, Approver or someone else can present/defend the work
• Clear/Concise - make it easy for readers to review evidence and understand your thinking; include rationale; legible
• Complete – verify all requirements addressed; attach (training, raw data) or reference (pre-requisite reports) evidence, include data analysis records
• Correct - templates can not replace the need for Authors and Approvers to carefully read and check the work or to think critically
Test Method Validation (TMV) | Mar 2016 | 47
TMV Documentation Tips - Protocols
General Considerations » Good planning and a clear and complete protocol is key to successful validation
» Use tables, flow charts, graphs, photos to create clear & concise records
Samples » Represent intended use (e.g., at nominal, bracket specification limit(s))
» Explain creation of “surrogate” samples and why they are representative of
actual samples
Avoid Ambiguity » Include equations with units
» Explain how data will be collected in compliance with GDP; consider creating
data collection forms
Test Method Validation (TMV) | Mar 2016 | 48
TMV Documentation Tips – Execution
Attributable • Individual: Signature, name/Employee ID
• Test Method: TM Title/Number • Equipment: Equip ID
Legibility • Clear, readable, concise
• Changes must not hide/obscure original record
Contemporaneous • Recorded & signed on date performed
Original • Original record or verified copy.
Accurate • Recorded as observed. No reprocessing. Meaning
maintained through life of data
During Execution:
• Follow the protocol
• If deviation occurs: STOP, document deviation, investigate, document root cause & action plan, obtain approval, then PROCEED
• When collect data:
Test Method Validation (TMV) | Mar 2016 | 49
Group Discussion – Sharing Best Practices Raw Data Integrity …
• Electronic Data and Audit Trails – What do you do?
Test Method Validation (TMV) | Mar 2016 | 50
Key Points for Conducting Successful TMV
Say what you do Written procedures
Do what you say
Follow procedures
Record what you did
Keep records
Check the results
Did ‘it’ pass or fail
Act on the difference
Procedures for deviations/departures
Test Method Validation (TMV) | Mar 2016 | 51
Wrap-up and Final Questions
Test Method Validation (TMV) | Mar 2016 | 52
Group Discussion – Sharing Best Practices Closing Thoughts …
• What are the key challenges (in validation areas) has your organization encountered?
• … and what have you put in place to effectively deal with those challenges?
• TMV Training: How do you ensure your TMV authors/approvers are properly trained on the TMV process?
• Finally - Any other best practices to share?
Test Method Validation (TMV) | Mar 2016 | 53
TMV TOOLS AND RESOURCES
Test Method Validation (TMV) | Mar 2016 | 54
References
• Measurement System Analysis (1995), 4th Ed, Automotive Industry Group, Southfield, MN
o ANOVA Method – for Level 2b & 2c (variables)
o Range Method – for Level 2a (variables)
• ASTM E2782-11e1 – Standard Guide for Measurement System Analysis (MSA)