CASE STUDY OF MOLECULAR ASSAY VALIDATION: 2ND TIER

CASE STUDY OF MOLECULAR ASSAY VALIDATION: 2ND TIER QUALITATIVE SCREENING AND SUBSEQUENT MONITORING

Rachel Lee, PhD

March 7, 2017

Identify Purposes of 2nd-tier Test

• Reducing false positives • Clarifying an ambiguous primary

result • Providing additional or just-in-time

information• Minimizing invasive diagnostic

testing• Improving healthcare equality

VLCAD – Why is needed?• Implemented statewide December 2016• Primary analyte C14:1; secondary analytes C14 and C14:1/C2

• Reported VLCAD Abnormal, Borderline, or Normal

• 1,426 presumptive positives and 91 cases of VLCAD

• Diagnostic test – skin biopsy – invasive, costly, and takes a while

• Inconsistent insurance coverage

Genes associated with the condition• Single gene or multiple genes• Autosomal recessive, dominant, X-

linked• Genotype/phenotype correlation

• Phenotypic spectrum• Ethnic or geographical differences

VLCAD – Gene Information• Single gene – ACADVL – 20 exons,

~5.4 kb in size • Autosomal recessive• Strong genotype/phenotype correlation

• 81% of pathogenic truncating variants are associated with severe early onset form

• Milder childhood and adult forms often resulted from one or two pathogenic missense variants

Testing Method Options• Mutation panel or sequencing?

• Common variants• Any existing protocol/kit

• Publication• Peer laboratory• Commercial vendor• FDA approval

• Existing methods or equipment

Testing Method Options – cont.• Cost• Expertise• Turnaround time• Implementation timeline• Capacity/Throughput/Automation• LIMS interface• Testing algorithm• Future expansion for other disorders

VLCAD – Which Method?• ~200 variants reported in ClinVar• Diverse mutational spectrum• No common variants identified in TX cases

VLCAD – Which Method?• Existing protocol

• Publications

• Peer laboratories• Existing methods and equipment

• Beta Globin gene using ABI310

VLCAD – Which Method?• Cost – higher than mutation panel• Expertise – ok with existing protocol• Turnaround time – shorter than skin biopsy• Implementation timeline - flexible• Capacity/Throughput/Automation – ok but

need to improve efficiency• LIMS interface – existing model• Testing algorithm – existing model• Future expansion for other disorders - yes

Expected Proportion of Abnormal Alleles DetectedMutation Detection

Rate

Proportion of CF patients for which a givennumber of abnormal alleles is detected

2 AbnormalAlleles

1 AbnormalAllele

0 AbnormalAllele

98% 96% 4% 0%95% 90% 10% 0%90% 81% 18% 1%85% 72% 26% 2%80% 64% 32% 4%75% 56% 38% 6%70% 49% 42% 9%60% 36% 48% 16%50% 25% 50% 25%40% 16% 48% 36%30% 9% 42% 49%

Moskowitz et al, 2008, Genetics in Medicine 10(12): 851-868

2nd-tier VLCAD DNA Testing

Implementation

Method Development: Optimization &

Scale-up

Method Validation: Testing &

QAO/Management Approvals

LIMS: Modification & Validation

Variant Database: Journal Article

Review & Compile List with Clinical

Significance

Follow-up Algorithm

Development

Stakeholders Input

SOP: Development &

QAO/Management Approvals

Communication: List Serv

Announcement

• Collect in-house specimens needed for method development and validation

• Request specimens from other states or laboratories

• Positive and negative controls with known genotypes

• Number of specimens

Materials Used for Development and Validation

Method Development / Optimization• Testing process – workflow evaluation

• Punch• Extraction protocol• Testing protocol• Data analysis / Result interpretation

• Automation• Staff training• Troubleshooting• PT exchange program

Bidirectional Sanger Sequencing Analysis

• Accuracy• Precision• Sensitivity• Specificity• Reportable range• Reference range• Stability • Carryover study

Validation Plan

Accuracy• Determined by testing 11 specimens carrying 16 different

variants and comparing the sequencing results to the results reported by a metabolic specialist for each specimen. In order for specimen sequencing results to be acceptable, they must be more than 95% in agreement with the metabolic specialist’s result.

• Finding: 100% in agreement with the metabolic specialists’ results, but four additional variants were identified probably due to the primer design and their coverage for sequencing regions

Precision• Determined by assessing the day to day, run to run, and

within run variation, as well as technician variability. • Two positive and two negative specimens were run in

triplicate for five days. • In order for the specimen sequencing results to be

acceptable, repeat testing of the same specimen over time should offer consistent results independent of the technician performing the task.

• The sequence identified variant should be more than 95% in agreement with the metabolic specialist’s result.

Precision• Finding: The sequencing results produced clear,

readable electropherograms and the identified variants are 100% in agreement within each run.

Testing Date Technician

#4 (Exons 12-13)

#11 (Exon 20)

WT 2 (Exons 12-

13)WT 4

(Exon 20)

Day 1 9/20/2016 ST

Heterozygous c.1322G>A

(G441D)


(R615Q)No variant identified

No variant identified

Day 2 9/28/2016 YS


(G441D)




Day 3 9/29/2016 YS


(G441D)




Day 4 9/30/2016 ST


(G441D)




Day 5 10/21/2016 YS


(G441D)




Clinical Sensitivity• Calculated by dividing the number of true positives by the

sum of true positives plus false negatives. The acceptable criteria for diagnostic sensitivity must be ≥95%.

• [TP/ (TP+FN)] x 100%

• Finding: All of the eleven specimens were tested by Sanger sequencing across 20 exons and identified all variants reported by the metabolic specialists as well as four additional variants. Clinical sensitivity is 100% in agreement with the known results.

Clinical Specificity• Calculated by dividing the number of true negatives by the

sum of true negatives plus false positives. The acceptable criteria for diagnostic specificity must be ≥95%.

[TN/ (TN+FP)] x 100%

• Finding: The sequencing results of the negative controls identified two benign variants, c.-63_-49dup15bp and c.1605+6T>C. No other sequencing results identified any variants. Clinical specificity is 100% in agreement with the known results.

Reportable Range• Reportable range was documented by the presence of variants within the amplified region of the ACADVL gene.

• Variants were identified by comparing to the published reference gene sequence NG_007975.1, and the Genbank accession number is AC120057.9.

• Test results reported as • Homozygous variant• Heterozygous variant • No variant identified

Reference Range• The reference range or ‘normal’ value represents specimens that are variant-free at a targeted sequencing region of the ACADVL gene and not necessarily a variant-free patient sample.

• Normal specimens should not contain identified variants or variants causing VLCAD. The test results were reported as “No variant identified”.

Stability Study• Determine if the current TX newborn screening DNA

Analysis specimen acceptance criterion of 4 months after Date of Collection is applicable and how long the mutations are stable at room temperature storage. Two newborn specimens that were received within 1 or 2 days after Date of Collection were selected. Punches were made, extracted, and tested on the same day (Day 1) and month 4.

Carryover Study• Determine potential cross contamination caused by using

the same puncher head to punch samples without cleaning between punches will be evaluated by punching 8 blank filter paper spots (CP) after punching 80 newborn DBS specimens

1 2 3 4 5 6 7 8 9 10 11 12

A patient patient patient patient patient patient patient patient patient patient CP

B patient patient patient patient patient patient patient patient patient patient CP

C patient patient patient patient patient patient patient patient patient patient CP

D patient patient patient patient patient patient patient patient patient patient CP

E patient patient patient patient patient patient patient patient patient patient CP

F patient patient patient patient patient patient patient patient patient patient CP

G patient patient patient patient patient patient patient patient patient patient CP

H patient patient patient patient patient patient patient patient patient patient CP

Carryover Study (cont)• Determine potential cross contamination during extraction

and assay by testing plates with alternating newborn specimens with no template control in a checker board pattern

1 2 3 4 5 6 7 8 9 10 11 12

A patient NTC patient NTC patient NTC patient NTC patient NTC patient

B NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC

C patient NTC patient NTC patient NTC patient NTC patient NTC patient

D NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC

E patient NTC patient NTC patient NTC patient NTC patient NTC patient

F NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC

G patient NTC patient NTC patient NTC patient NTC patient NTC patient

H NTC Patient NTC Patient NTC Patient NTC Patient NTC Patient NTC

• Accuracy• Precision• Sensitivity• Specificity• Reportable range• Reference range• Stability study• Carryover study

Validation Plan– FDA cleared kit

Other Activities• Validation study write-up and approval• Scale up (workflow, coordination with 1st tier)• Result notes (interpretation, recommendation)• Method limitation• Variant database• SOP write-up and approval• Reporting / follow-up algorithm• LIMS modification• Seeking stakeholder inputs• Inform and educate healthcare providers

QC Monitoring• Turnaround time - % of specimens meet TAT• Failed run log

• % of runs failed• Causes – look for pattern or trend in exons, reagent, equipment, staff

• Total number of tests• Update variant database• Diagnosed cases

• Compare our results with clinical results • Frequency of variant alleles in diagnosed cases• Any missed variants• Age of diagnosis• Incidence rate

• PT performance

Problems Encountered• Low PCR amplification

• Poor extraction• Low genomic DNA• Primer design • PCR reaction condition

• Sequence data• High background noise or low signal• Ski slope• Dye blot• Spikes

• Variants in negative control specimens• Incorrect variant call• Classification and interpretation of variants

Take Home Messages….• Each NBS program has different needs - define your

goals and choose wisely

• Need help? Just ask

• Be familiar with CLIA and CAP requirements and CLSI guidelines on validation

• Continuous monitoring is important

• Document…..document…document

Documents

CASE STUDY OF MOLECULAR ASSAY VALIDATION: 2ND TIER