2018 Focus on Compliance 9/12/2018 Next Generation

2018 Focus on ComplianceNext Generation Sequencing: What Does Compliance Look Like?

9/12/2018

1© 2018 College of American Pathologists. Materials are used with the permission of the faculty.

2018 Focus on Compliance

Karl Voelkerding, MD, FCAP

Birgit Funke, PhD, FACMG

Avni Santani, PhD, FACMG

September 12, 2018

Next Generation Sequencing: What Does Compliance Look Like?

Disclosure

© 2018 College of American Pathologists. All rights reserved.

The following authors/planners/reviewers have financial interests/ relationships to disclose:

Andrew Goodwin, MD, FCAP

Haematologic Technologies

Medical Director Monetary stipend

Lee Hilborne, MD, MPH, DLM(ASCP)

Avni Santani, PhD, ABMGG


Quest Diagnostics

CHOPVeritasAgilent

PierianDx

Employee

ConsultantConsultantLicense agreement

Scientific advisor

Compensation, stock

Consulting feeConsulting feeFees

Consulting fee

2

Disclosure, cont’d.


Birgit Funke, PhD, ABMGG, ACMG Michael Talbert, MD

Gregory Gagnon, MD, FCAP Carlos Machicao, MD, FCAP

S. Robert Freedman, MD, FCAP Jacob Mayer, Jr PhD, HCLD

Kenneth Klein, MD, FCAP Justin Rueckert, DO, MLS(ASCP)cm

Ritu Nayar, MD, MBBS, FCAP Marian Briggs, MS, MT(ASCP)

The following authors/planners/reviewers have no financial interests/relationships to disclose:

3


9/12/2018


AccreditationThe College of American Pathologists (CAP) is accredited by the Accreditation Council for Continuing Medical Education (ACCME) to provide continuing medical education for physicians.

CE (CONTINUING EDUCATION FOR NON-PHYSICIANS)The CAP designates this educational activity for a maximum of 1 credit/hour of continuing education. Each participant should only claim those credits/hours he/she actually spent in the activity.

ASCP STATEMENTThe American Society for Clinical Pathology (ASCP) Board of Certification (BOC) Certification Maintenance Program (CMP) accepts this activity to meet the continuing education requirements.

CALIFORNIA AND FLORIDA STATEMENTThis activity is approved for continuing education credit in the states of California and Florida.

© 2018 College of American Pathologists. All rights reserved. 4

Today’s Presenter


Dr. Voelkerding is a professor of pathology at the University of Utah School of Medicine and past president of the Association for Molecular Pathology. He is the former chair of the College of American Pathologists’ Next-Generation Sequencing Project Team, which developed laboratory accreditation requirements and proficiency testing programs for clinical next-generation sequencing. Dr. Voelkerding is currently chair of the College of American Pathologists’ Genomic Medicine Resource Committee. Dr. Voelkerding received his MD from the University of Cincinnati College of Medicine and is board certified in clinical pathology and molecular genetic pathology. His research focuses on the translation of genomics technologies into clinical diagnostics.


Today’s Presenter


Birgit Funke, PhD, FACMG

• Associate Professor of Pathology (part-time), Harvard Medical School

• Vice President of Clinical Affairs, Veritas Genetics

6


9/12/2018


Today’s Presenter

Avni Santani, PhD, FACMG

• Director - Clinical Laboratories, Strategic Partnerships and Innovation

• Director - Center for Applied Genomics, The Children’s Hospital Of Philadelphia

• Associate Professor of Clinical Pathology & Lab Medicine

• Education - Perelman School of Medicine, University of Pennsylvania


Objectives

• Explain the accreditation requirements for Next Generation Sequencing (NGS)

• Describe practical steps to ensure appropriate implementation of NGS testing

• Demonstrate compliance with the accreditation requirements


• First Published in the 2012 edition of the Molecular Pathology Checklist

• Revised Yearly as NGS Clinical Testing Has Advanced

CAP NGS Laboratory Accreditation Requirements



9/12/2018


• Guiding Principle – Finding the Right Balance

• Foster Innovation and Technology Adoption – Assure Patient Safety



Wet Bench Process• Sample Handling

• Library Preparation• Sequence Generation

Bioinformatics “Dry Bench” Process• Sequence Alignment to Reference

• Variant Identification• Variant Annotation (eg CFTR

p.Arg205Lys)

Data Interpretation and Reporting Medical Practice

FASTQ File

Annotated Variants

Analytical Laboratory Process

CAP Definition of a NGS Test


Majority of Laboratories Perform NGS Testing

In One Primary Physical Laboratory

Wet Bench ProcessSequencing

Bioinformatics Process

Interpretation and Reporting

Single CLIA License

Data IntegritySecurityPrivacy



9/12/2018


Some Laboratories OutsourceIndividual Process Steps



InterpretationReporting

Primary Laboratory

Reference Laboratory

Primary Lab Receives Patient Sample

Outsource




Primary Laboratory



Outsource




• Written to Accommodate Diversity of Testing Configurations

• Single Laboratory to Primary and Reference Laboratory



Diversity

15


9/12/2018






Primary Laboratory



Outsource

“Distributive” Testing Model Concept

CAP Requires Integrated Methods Based Validation




Primary Laboratory



Outsource



“Distributive” Testing Model Concept

CAP Requires Integrated Methods Based Validation


Primary/Referring Laboratory

NGS

Reference Lab

Policy

NGS Confirmatory Testing Policy

NGS Sample Provenance

Tracking

General

Exception Log/Record

Data Storage

Local/Cloud

Data Transfer

Confidentiality

Wet Bench

NGS Wet Bench Process

Documentation


Validation

Quality Management

Plan

Laboratory Records

Monitoring of Upgrades

Bioinformatics

NGS Bioinformatics

Pipeline Documentation

NGS Bioinformatics

Pipeline Validation

Quality Management

Plan

Pipeline Version Traceability

Monitoring of

Upgrades

Interpretation

Sequence Variants


Reporting of Incidental Genetic Findings

Current CAP NGS Laboratory Accreditation Requirements -2018



9/12/2018



NGS

Reference Lab

Policy



Tracking

General


Data Storage

Local/Cloud

Data Transfer

Confidentiality

Wet Bench


Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan


Monitoring of

Upgrades

Interpretation

Sequence Variants



Current CAP NGS Laboratory Accreditation Requirements -2018






p.Arg205Lys)

Data Interpretation and Reporting

FASTQ File

Annotated Variants

CAP Requires an Integrated Methods Based Validation

IntegratedMethods Based

Validation






p.Arg205Lys)

Data Interpretation and Reporting

IntegratedMethods Based

Validation

FASTQ File

Annotated Variants

Validation Specimens Contain a

Representative Spectrum of Variant

Types Test is Designed to Detect

EstablishAssay Performance




9/12/2018


NGS Test Validation – Choice of Specimens

Validation Specimens Contain a Representative Spectrum of Variant Types Test is Designed to Detect

For Gene PanelsInclude Specimens that

Contain Common Mutations

(e.g. Cancer Hot Spots, CFTR p.Phe508del)

Methods Based

Gene or Analyte Based

Hybrid


Add Extra Specimens that Contain Variants That are Technically

Difficult to Detect


• We are sequencing every base instead of assaying known disease variants -- novel variant detection

– Requires new validation approaches

– Methods–based validation

• The amount of data exceeds ability to process it manually

– Increased reliance on computational resources and bioinformatics

• Changing role of laboratory director

– Moving from predominantly technical focus to increasingly interpretive role

• Ability to sequence large numbers of genes

– Increased burden on test content design

NGS has changed many things….

We need to develop new approaches and standards


Adapted from: Rehm et al. 2013

LIFECYCLE OF AN NGS BASED CLINICAL TEST


TARGET

CATPURE

TESTVALIDATION

LIBRARY PREP CAPTURE SEQ ANALYSIS

QUALITY MANAGEMENT

PATIENT TESTING

QC PT

GE

NE

PN

L -

A

GE

NE

PN

L -

B

GE

NE

PN

L -

C

E X

O M

E

G E

N O

M E

P L A T F O R M(cumulative data)

Optimize components and entire test

• General assay conditions• Coverage• Sample pooling• Analysis setting/thresholds

Establish protocol for entire workflow

Validate entire test using set conditions

• Sensitivity• Specificity• Robustness• Reproducibility

Include samples representing

• Different variant types (in/dels, substitutions, CNVs)

• Disease specific aspects (common pathogenic variants)

TEST OPTIMIZATION

Quality Control (QC)

• Every run

Proficiency Testing (PT)

• Periodically

TEST DESIGN

Technical design

Content design

Added dimension Greatly increased complexity

24


9/12/2018


A GROWING LIST OF PROFESSIONAL NGS GUIDELINES

Organization/Entity PMID Year Title

CDC 23138292 2012 Assuring the quality of next-generation sequencing in clinical laboratory practice.

CDC 26154004 2015Good laboratory practice for clinical next-generation sequencing informatics pipelines.

ACMG 23887774 2013 ACMG clinical laboratory standards for next-generation sequencing.

ACMG/AMP 25741868 2015

Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology

AMP/CAP 28341590 2017

Guidelines for Validation of Next-Generation Sequencing-Based Oncology Panels: A Joint Consensus Recommendation of the Association for Molecular Pathology and College of American Pathologists.

AMP/CAP 29154853 2018

Standards and Guidelines for Validating Next-Generation Sequencing Bioinformatics Pipelines: A Joint Recommendation of the Association for Molecular Pathology and the College of American Pathologists.

CAP 25152313 2015College of American Pathologists' laboratory standards for next-generation sequencing clinical tests.

CAP 28322587 2017Development and Validation of Targeted Next-Generation Sequencing Panels for Detection of Germline Variants in Inherited Diseases.

CAP 28362156 2017Development and Validation of Clinical Whole-Exome and Whole-Genome Sequencing for Detection of Germline Variants in Inherited Disease.

CLSI N/A 2014MM09: Nucleic Acid Sequencing Methods in Diagnostic Laboratory Medicine, 2nd Edition

New York State N/A 2015Guidelines for Validation Submissions of Next Generation Sequencing (NGS) assays under the NYS Testing Category of Genetic Testing– Molecular

FDA N/A 2016Use of Standards in FDA Regulatory Oversight of Next Generation Sequencing (NGS)-Based In Vitro diagnostics (IVDs) Used for Diagnosing Germline Diseases

Santani et al. 2018 (manuscript under review, JMD)


THERE IS A NEED FOR MORE….

….. concrete guidance

• Labs want to do the right thing but it’s not easy to translate recommendations into an SOP for how to set up high quality NGS testing

• Need step by step instructions that “get into the weeds”



THE CAP CHECKLIST HAS TRIED TO ACCOMMODATE THIS NEED

From the 2017 CAP checklist:

This is a SINGLE requirement

27


9/12/2018


EVOLUTION OF THE CAP NGS CHEKLIST(MOL.36010 – Analytical Wet Bench Process Documentation)


2013 2015

The complexity of NGS is not compatible with what the checklist was designed for

2017

28


**REVISED** 08/21/2017MOL.36015 NGS Analytical Wet Bench Process Validation Phase II

The laboratory validates the analytical wet bench process and revalidates the entire process and/or confirms that the performance of the components of the process is acceptable when modifications are made.

NOTE: The output of the NGS analytical wet bench process is a collection of sequence data that requires additional bioinformatics processing and analysis to determine whether the sequence is of sufficient quality and quantity for the intended test. To determine this, and to ensure acceptable beginning-to-end test performance, validation of the NGS analytical wet bench process must be integrated with the bioinformatics process validation for the intended test (see MOL.36115).

The analytical wet bench process and the bioinformatics process for a test may occur within a single laboratory, or in a combination of primary and referral laboratories (see MOL.35840). Whether performed in a single laboratory, or in a distributive model involving primary and referral laboratories, the validations of the wet bench and bioinformatics processes for an intended test must be integrated to ensure acceptable beginning-to-end test performance. It is the responsibility of the laboratory director or designee meeting CAP director qualifications to review and approve all validations relevant to the intended test for processes performed within their laboratory, and to review all validations relevant to the intended test for processes performed in referral laboratories, if applicable.

Laboratories validating NGS tests must comply with MOL.31015 and/or MIC.64770, as applicable, and meet the requirement that validations are performed with samples for each type of specimen expected for the assay (e.g. blood, fresh/frozen tissue, paraffin embedded tissue, prenatal specimens, saliva, culture isolates). Validations can be augmented by, but not supplanted with, additional reference standards (e.g. cell lines such as NIST NA12878).• Analytical validations must consist of a baseline methods-based validation that establishes the test's general

performance for the detection of the sequence variant type(s) that the test is designed to identify. For tests that are offered for specific clinical indications (e.g. diagnostic gene panels) the laboratory should investigate whether any analyte, disease or gene specific needs exist that necessitate inclusion of additional specimens or reference samples in the validation. For example, this may include samples containing prevalent pathogenic variants, especially if they are technically difficult to accurately detect (e.g. large indels). For detailed requirements on validation see MOL.36115 (NGS Analytical Bioinformatics Process Validation).

“…I understand what a methods‐based validation approach is ‐ but how exactly do I go about this?”

YET….WE ARE DEFINING MOSTLY WHAT NEEDS TO BE DONE, BUT NOT HOW…

29

A “COMPANION RESOURCE” FOR THE CAP CHECKLIST

Manuscript in revision at the Journal of Molecular Diagnostics



9/12/2018


FILLABLE WORKBOOKS THAT GUIDE THE WAY THROUGH THE NGS TEST LIFECYCLE


TEST DESIGN


Background Detailed Instructions32


This information is critical for appropriate test design, examples:- CNVs rare: supplemental assay less important- Gene not well associated with disease –careful when including

into gene panel !

Goal: ensure both clinical and technical validity

TEST DESIGN

33


9/12/2018



Now you’ve chosen your genes – preparing for technical design

TEST DESIGN

34


TEST VALIDATION

35


Formulas to help translate

Helpful comments and concrete examples

Common sense translation

TEST VALIDATION

36


9/12/2018



TEST VALIDATION

37


TEST VALIDATION

• Build your validation runs• Start with assembling sample list• Annotate with pertinent information

- Which validation “layer” is served by the sample- Specific variants of interest present?- Which metric is served (intra/inter-run variability)

38


• Place samples on a run grid• Example here:

• HiSeq 2500• 10 samples/lane• Run 1: Same libraries run 3X • Run 2+3: New libraries of same samples

TEST VALIDATION

39


9/12/2018



METHODS‐BASED TEST VALIDATION ‐PERFORMANCE METRICS SUMMARY

40

NEXT STEPS?

Short Term• Make worksheets publicly available • Invite comment from users (public access, CAP

website)• Adapt to incorporate other applications (somatic

NGS)

Medium Term: Make it a “living document”• Iterate on first version based on user feedback and

evolving evolution of NGS

Long Term• Convert to web based application


Demonstrate Compliance with the Accreditation Requirements

• Preparing for an inspection of an NGS laboratory

• Integrated inspection (MOL, COM, GEN, TLC)

- Refer to latest checklist edition

• The highlights of NGS section of the MOL Checklist will be covered

• This webinar is not a substitute for the comprehensive CAP Checklists



9/12/2018


PREPARATION

Think like an Inspector!


Review the Checklists

• New or revised questions?

• Requirement AND NOTES are applicable

• Evidence of Compliance suggests ways to ensure compliance

• Review R.O.A.D instructions to inspector; documents and records must be available

• Is the available documentation acceptable to you if you were the inspector?


• High volume tests

• Low volume tests

• High patient impact tests

• Tests from variant PT report

• Concerns from previous deficiencies

• New analyte, new instrument

Most Likely Choices During an Inspection



9/12/2018



Top Molecular Deficiencies

MOL.30785 For each test there is a validation summary addressing analytical and clinical performance parameters.

MOL.34495 Quality control data are reviewed and assessed at least monthly by the laboratory director or designee

MOL.49570 The final report includes an appropriate summary of the methods, the loci or variants tested and the analytic interpretation. When appropriate, the final report includes the clinical interpretation.

MOL.35350 Nucleic acid amplification procedures (e.g. PCR) are designed to minimize carryover (false positive results) usingappropriate physical containment and procedural controls.

MOL.34229 For qualitative tests, positive, negative and sensitivity controls are included for each assay, when appropriate, in every runand as specified in the manufacturer's instructions (as applicable) and laboratory procedure.


MOL.36105 The laboratory has a written procedure that describes the steps included in recording the bioinformatics process used to analyze, interpret, and report NGS test results.

MOL.36115 The laboratory validates the analytical bioinformatics process (also termed pipeline) and revalidates the entire process and/or confirms the performance of the components of the process as acceptable when modifications are made.

MOL.31145 The results of each validation study include a sufficient number of samples with repeated analyses to provide a high degree of assurance of the test's precision or reproducibility

MOL.34352 The results of controls are reviewed for acceptability before reporting of results.

MOL.36010 The laboratory has a written procedure for performing the analytical wet bench process used to generate next generation sequencing data.

Top Molecular Deficiencies cont.



9/12/2018




NGS

Reference Lab

Policy



Tracking

General


Data Storage

Local/Cloud

Data Transfer

Confidentiality

Wet Bench

NGS Wet Bench Process Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan

Pipeline Version

Traceability

Monitoring of

Upgrades

Interpretation

Sequence Variants



Current CAP NGS Laboratory Accreditation Requirements - 2018

Demonstrating Compliance• Reference Laboratory: Records of evaluations of referral

laboratories for NGS referral testing and copies of valid CLIA and/or CAP certificates

• Specimen Provenance: Specimen handling, chain of custody and data transfer starting from initial NGS test order to the final report

• Confirmatory Studies: Determine during validation if and when confirmatory testing of NGS identified variants should be performed

• If certain variants are NOT confirmed, then provide rationale for, and data supporting the rationale

• Records of review of correlation of NGS test results and confirmatory test results over time





NGS

Reference Lab

Policy



Tracking

General


Data Storage

Local/Cloud

Data Transfer

Confidentiality

Wet Bench

NGS Wet Bench Process Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan

Pipeline Version

Traceability

Monitoring of

Upgrades

Interpretation

Sequence Variants




9/12/2018


Demonstrating Compliance cont.

• Review of exception log and corrective action

• Data Security: records of policies and protocols to ensure patient confidentiality, security and data integrity

• Encryption, control of physical and virtual access to data, system backups and redundancy

• Records of audit trails and copies of HIPAA Associate Agreements

• Data Storage: Policy must be in accordance with local, state and federal requirements for storage of data

• Written policy that describes files, type of data to be retained and length of retention





NGS

Reference Lab

Policy



Tracking

General


Data Storage

Local/Cloud

Data Transfer

Confidentiality

Wet Bench


Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan


Monitoring of

Upgrades

Interpretation

Sequence Variants




• Review all centrally located records

• Review documentation of maintenance, reagent labeling, testing performed, QC, and results reporting

• Ensure that policies and procedures are available to testing personnel


54


9/12/2018


Demonstrating Compliance cont.• Monitoring of Proficiency Testing (PT)

– Required for all analytes regardless of CLIA complexity classification

– PT must be specific to the analyte and system matrix– If PT is not available, another mechanism must be

defined to assess accuracy and reliability of the system (alternative assessment)

– PT must be performed by, and rotated among, the same individuals who perform the routine testing

– Corrective action must be documented and reviewed for unacceptable results and for ungraded results



• Quality Control

– QC must be verified prior to reporting patient results– Corrective action must be documented and reviewed– QC must be handled in the same manner as patient

samples and performed by those performing patient testing

© 2018 College of American Pathologists. All rights reserved.56

PRE-ANALYTIC QC FOR NGS -Examples



9/12/2018


ANALYTIC QC FOR NGS - Examples


POST-ANALYTIC QC FOR NGS -Examples





NGS

Reference Lab

Policy



Tracking

General


Data Storage

Local/Cloud

Data Transfer

Confidentiality

Wet Bench


Documentation


Validation

Quality Management

Plan

Laboratory Records


Bioinformatics

NGS Bioinformatics


NGS Bioinformatics

Pipeline Validation

Quality Management

Plan


Monitoring of

Upgrades

Interpretation

Sequence Variants




9/12/2018



• Variant Interpretation: Provide validated algorithm for classifying and interpreting variants

• The ACMG guidelines for classification and interpretation of germline variants in inherited disorders (Richards et al. 2015)

• Standard nomenclature (eg., HGVS and HUGO) – also consider: Infectious disease, somatic variants

• Incidental and Secondary findings: Describe approach for reporting findings unrelated to the clinical purpose of testing

© 2018 College of American Pathologists. All rights reserved.61

College of American Pathologists (CAP) – led effortCreating practical, step‐by‐step guidance for

molecular laboratories

Manuscript in revision at the Journal of Molecular Diagnostics


Remaining 2018 Focus on Compliance Webinars



9/12/2018



64

Documents

2018 Focus on Compliance 9/12/2018 Next Generation