Metrology and Standards Needs for Gene Expression Technologies

Krishna Ghosh

Agilent Technologies

June 10, 2003

June 10, 2003 2

Overview of Presentation

• History of Gene Expression Standards Development• Microarray/Scanner Fluorescent Standards

– Status update

– Next Steps

• Universal RNA Standards– Status update

– Next Steps

• NIST Feed back-Next Steps

June 10, 2003 3

Development of Standards for Gene Expression

• Kickoff meeting (NIST, Oct 2002)

– Workshops identified: Microarray /Scanner & Universal RNA

– Collaborative model development and workgroups

• Follow up meeting (NIST, Dec 2002)

– Technical overview of microarray readers and their performance

– Monthly teleconferences of working group to develop/detail artifact(s)

• Universal RNA Standards Workshop (Stanford, March 2003)

• Microarray Fluorescence Standards Working Group Meeting, (NIST, May 2003)

June 10, 2003 4

NIST Working Group Microarray/Scanner Fluorescence Standards

• Working Group Objectives:– Develop an artifact to measure microarray reader performance using a

standardized method. The artifact(s) will be used primarily by manufacturers to standardize the measurement of specifications.

– Identify the appropriate fluorescent material(s) and manufacturing technology to produce a NIST-certified artifact

– Provide the required procedures and tools required to analyze the artifact

• Accomplishments to Date:– Assembled representatives from microarray reader manufacturers

(Affymetrix, Agilent, Axon, Perkin Elmer, Biorad, Arrayit and others)– Created a working draft of the artifact’s features and characteristics– Reviewed different options for fluorescent materials and manufacturing

technologies (organic and inorganic dyes, metal oxides, nanocrystal composites, polymer coating, fluorescent glasses, Sol-gel)

June 10, 2003 5

Artifacts to Measure Scanner Performance Using a Standardized Method*

• Uniformity Artifact: Measures scanner’s uniformity and Signal / Noise for bright features.

• Detection Limit Artifact: Measures scanner’s limit of detection.

• Artifacts will be manufactured by someone other than NIST, BUT will be qualified by NIST.

• Artifact(s) intended for use primarily by manufacturers to standardize the measurement of specifications.

June 10, 2003 6

Preliminary Scanner Specification Decisions

• Artifacts will be uniformly coated (except for fiducials and background regions).

• There will be at least two artifacts per dye; one in the middle of the dynamic range meant for uniformity measurement, the other for measuring signal/noise for very dim signals approaching the detection limit.

• The dimmer slide should approximate about 0.5 chromophores per square micron of fluorescent material.

• The glass non-flatness should not exceed +-10 microns.• The parallelism will be <1 mrad• The preferred substrate material is glass.• The artifact will be available as 1.00 mm thick.• The artifact will be 1x3 inches and can be fitted to other users.• The dye choice should match Cy3/Cy5 as closely as possible.• Photostable and environmentally stable

June 10, 2003 7

Coating Requirements*

1. Transparent coating on high purity, low background emission slide glass.

2. Mimic Cy-3 and Cy-5 dye properties;

a. Cy-3 Excitation at 532 nm (frequency doubled Nd-YAG laser)

b. Cy-3 Emission at ~562 nm.

c. Cy-5 Excitation at 633 nm (Helium-Neon laser)

d. Cy-5 Emission at ~660 nm

3. Long-Term Stability 

4. Intensity uniformity over coated area (~1% CV)

* From presentation of E.Pope

June 10, 2003 8

Coated Area

Background area

82 mm

1 mm4 mm6 mm

<1 mm

3 mm

94 mm

2.7 mm3 mm

17 mm

2.7 mm

25.4 mm

barcode

65 mm

Cut

Present NIST Artifact Slide Layout*

* From presentation of J.Corson

June 10, 2003 9

Summary of Workshop Presentations14th May 2003

• Metal oxide glasses are less prone to photobleaching than organic dyes and have promising spectral characteristics

• There are a variety of ways to prepare fluorescent standards to meet the Artifact Slide requirements(Matech)

• Fluorescent microspheres offer photostability and compatibility with different attachment chemistries(Molecular Probes)

• Nanocrystals (semi-conductor materials) can mimic emission wavelengths of fluorescent dyes while using common laser and broadband sources (Evident Technologies)

June 10, 2003 10

Microarray/Scanner Fluorescence Standards- Next Steps -

• Select fluorescent material for the artifact– Silica doped with metal oxides

– Q –dots/Nanocrystals

– Alternate organic photostable fluorescent dye

• Assemble manufacturers capable of making specified artifact with uniform coating expertise

• Define recommended use and data analysis procedures

• Refine artifact specifications depending on the limitations

June 10, 2003 11

Universal RNA Standards WorkshopStanford University, March 28-29, 2003

UniversalRNA Standard

ArrayManufacturers

Regulators

ArrayUsers

June 10, 2003 12

Goals of Universal RNA Standards Workshop

• Educational: provide participants a forum to share various methods and techniques that are relevant to defining a standard for Gene Expression and RT-PCR technologies

• Awareness: determine areas of agreement and disagreement on issues of different standards, and where there is a need for additional information

• Guidance: help define how NIST could best help to develop the RNA standard(s) and promote its use

• Requirements: a reliable, reproducible and manufacturable standard to support the use of gene expression results for IVD and NDA filings (proficiency testing, comparison of submitted data with published data, comparison of different gene expression platforms), AND can be used by manufacturers to certify their products

June 10, 2003 13

Gene Expression Workflow- Standardization Requirements

MicroarrayData Quality

Labeling- Unbalanced Channels- Dyes & Reagents- Primer Incorporation- Labeling Efficiency

ArrayFabrication- Pen Shifts- Sub-Array Variations- Binding Chemistry - Plate Contamination- Depletion

Scanning- Scanner Settings- Optical Measurements- Overlap of channels- Image Orientation

Subpopulations

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Fluo

resc

ence

Time (seconds)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

17 22 27 32 37 42 47 52 57 62 67 72

1

10

100

1000

10000

100000

1 10 100 1000 10000 100000

Cy3 S ignal

1

10

100

1000

10000

100000

1 10 100 1000 10000 100000

50 vs 100 ng

100 vs 50 ng

Probe- Sequence Verification- Probe Length & Sequence- Sequence Homology- Oligo/Clone Specific Noise- Cross-Reactivity

SamplePreparation- mRNA Extraction- RNA Concentration- Sample Contamination- Signal Amplification- Sample Purification

Hybridization- Hybridization Conditions- Hybridization Gradients- Amount of Sample Applied- Cross-Reactivity- Hybridization Efficiency

June 10, 2003 14

Generalized Workflow

•Cell line•Tissue type•Donor•Collection method

•RNA stability•Purity•Integrity

•Microarray cDNA Oligo•QPCR•Other

•Dye•Imager / Scanner•Detector

•Feature extraction•Std curve & normalizing

How many standards are needed to address the workflow differences due to Instrumentation, Reagents, Sample, Operator and Analysis?

Acquire SampleRNAExtraction Assay Detection

Data Analysis

June 10, 2003 15

Session 1Standardization of Biological Component of RNA Based

Molecular Assays

• Focus: Review the needs for gene expression measurement standards in support of: Safety and efficacy claims of therapeutic products, and Human clinical in vitro diagnostics.

• Session Summary: – Standards can be used for proficiency testing, benchmarking, cross-

platform comparisons, inter-laboratory comparisons– Significant sources of variation are: Array Platform, Lab and Array-to-

Array etc..– Standards, analysis tools & metrics, and training are needed to help assure

that platform and sample processor are capable of detecting the “biological truth”

– Reference methods and reference materials (traceable, assignable) are key to defining analytical and clinical performance characteristics

– Consideration should be given to “bioinformational standards” with approved gene lists and indicator patterns; used as training sets for decision rules

June 10, 2003 16

Design for Experiment Design for Experiment

1.  Tissue extractionTissue extraction: liver vs. 5-tissue pool from 22 C57 : liver vs. 5-tissue pool from 22 C57 black, adult male mice (NIEHS NTP/OHSU)black, adult male mice (NIEHS NTP/OHSU)

2.2. Standard RNAsStandard RNAs: liver, 5-tissue pool (NIEHS): liver, 5-tissue pool (NIEHS)3.3. ChipsChips

StandardStandard = 18K mouse oligo (Duke) = 18K mouse oligo (Duke)Resident Resident = cDNA, oligo, Affymetrix, Agilent= cDNA, oligo, Affymetrix, Agilent

4. 4. HybridizationsHybridizations (4): liver vs. liver; liver vs. pooled with (4): liver vs. liver; liver vs. pooled with fluor dye flipsfluor dye flips

5. 5. Data qualityData quality: Arabidopsis10-gene set in standard : Arabidopsis10-gene set in standard RNAs and on chips RNAs and on chips (Wang et al, 2002)(Wang et al, 2002)

6.6. OHSU Data WarehouseOHSU Data Warehouse: : Web hosting and data Web hosting and data sharing, MIAME sheet for experimental details, gsharing, MIAME sheet for experimental details, gene ene annotations, resident analysis and stats toolsannotations, resident analysis and stats tools

June 10, 2003 17

Sources of Variation – Preliminary Trends*

Array Platform

Lab

Array-to-Array

Residual

Tissue*Platform

Tissue*Lab

Tissue

Dye

* From presentation of B.Weiss

June 10, 2003 18

FDA Funded collaborative Project for Evaluation of Performance Standards for Toxicogenomic Studies

Using benchmark genes within mixed tissue samples

• Identify tissue-selective, low variance rat genes from control animal data in large databases (populated using a consistent protocol).

• Select tissues with large difference in number of tissue-selective genes from control animal data

• Model a pilot set of tissue mixtures for the standard using database info and test on arrays

• Identify probe sets corresponding to benchmark genes on different platforms

• Evaluate the added value of exogenous spike-in standards (platform-dependent).

*Hsiao et al., Physiol. Genomics 7: 97-104, 2001

June 10, 2003 19

Expected Initial Outcomes

• Identification of probes that can perform similarly across platforms

• Determine normal range of false positive/false negative rates for MTS

• Determine normal range of lab-to-lab variance for MTS

• Determine normal range of cross-platform variance for MTS

• Publication of findings.

June 10, 2003 20

Session 2Metrics for Universal Standard:

Expression Arrays

• Focus: What standard parameters/metrics are needed for existing expression array technologies to ensure both intra-laboratory and inter-laboratory comparison? What are the key functions and features of the standard?

• Session Summary:

– RNA sample quality can impact usefulness of microarray results (

– Not all RNA is created equal. An RNA quality index, taking into account a number of physical metrics, can be formulated

– RNA spikes can be used to assess and limit variability for this reasonably complex procedure

– Transcript pooling offers an organism specific approach to assessing sensitivity, specificity and reproducibility

June 10, 2003 21

Session 3Appropriate Standards to Meet Metrics

• Focus: What are currently implemented intra-laboratory controls and their effectiveness? Are these applicable to inter-laboratory comparisons and cross-platform comparisons?

• Session Summary:

– Depending on its purpose, more than one RNA standard is probably required, and will evolve with time

– “Begin with the end in mind”; what are the design specifications to meet the user requirements?

– Complex synthetic sample offers a sample of intermediate manufacturing difficulty and intermediate customer relevance versus oligo-only and complex natural samples

– Universal reference RNA’s are a commercially available blend of total RNA isolated from multiple cell lines designed to maximize gene expression profiling

June 10, 2003 22

Session 4 Metrics for Universal Standard:

Quantitative RT-PCR

• Focus: What are the key functions and features of standards for RT-PCR? What are currently implemented intra-laboratory controls and their effectiveness?

• Session Summary:– Emerging genetic tests will be evaluated for analytical validity, clinical

validity, clinical utility and ethical, legal and social implications

– Internal quantitative standards (e.g., HSK) can demonstrate the impact of sample collection and processing methods on data quality

– Internal calibrators or standard curves are required for accurate absolute quantification; relative quantification is possible with endogenous controls

– Human universal reference RNA can be used for QRT-PCR

– QRT-PCR has applied normalization to control variation in total RNA mass and amplification efficiency, and ratio-ing to “verified” invariant genes

June 10, 2003 23

Themes from the Universal RNA Standard Workshop

• Multiple sources of data variability– Different laboratories, platforms, sample types, extraction

methods, etc.

– Evolving technologies

– Probe/primer design

• Difficulties sharing data– MIAME is a start

– Annotation problems abound

– Is the same analyte being measured?

• Standard methods and metrics for proficiency testing– Terminology and definitions are needed

– Differences between analytical and clinical applications

June 10, 2003 24

All RNA is Not Created Equal

ProcurementSourceTissueIsolation methodStorage

Tissue procurement is probably the most uncontrolled and variable step in the pathway to microarray data collection

Not sure how we can control this. Must influence collection site process. Having easy and alternative methods available will help to minimize degradation.

June 10, 2003 25

RNA Quality Index

• Sample isolated with minimal degradation• RNA extracted using the same method• RNA is proven stable during study• RNA has a 28S:18S ratio above 1.5• Free of DNA for Real-time validation

The procedures are available but not all tools are easily used or in simple kit formats.

June 10, 2003 26

A Good Standard Should*

• Allow performance validation of any single platform over time.• Facilitate comparison between various platforms used to assay

gene expression.• Be constructed in such a manner as to assure consistency over

time.• Include a well defined protocol describing how it is made and

validated. • Include two or more samples that allow one to make both

absolute and relative measurements of the abundance of individual transcripts.

• Not be limited to hybridization-based approaches, but should be amenable to use with other assays such as QRT-PCR

June 10, 2003 27

Standards to evaluate platform (and laboratory) performance

• Standard should be relatively invariant and regenerable. • Standard should be formed on probes in common between

platforms. • Standard should resemble test samples (e.g., rat tissue

standard for toxicogenomics) and also query a wide range of features on the arrays

• A set of mixed tissue standards that have varying ratios of components would contain real and quantifiable gene expression changes between samples and use endpoints measurable on most platforms

June 10, 2003 28

Microarray Performance Characteristics and Controls*

• Characterization of Array– Design and fabrication, e.g. platform type, surface type, composition

and spatial layout, number of elements (spot), number of replicates, etc.

– Spot elements, e.g. clone, sequence, PCR primer pairs, probe length, gene name, etc.

– Built-in controls, e.g. housekeeping genes, etc.

• Microarray Controls– Internal controls (housekeeping genes, synthetic RNA)

– Pooled RNA from cell lines

– Pooled RNA from test samples

– RNA and oligonucleotides from plants and bacteria

June 10, 2003 29

Session 5Proposed Workshop Recommendations

• Standards are required for several purposes in gene expression RNA analysis regardless of whether quantitative RT-PCR or microarray gene expression technologies are employed.– Periodic laboratory proficiency testing– Platform performance validation and baseline monitoring– Cross-platform performance validations– Inter-laboratory performance validation– Providing a reference point for regulatory agencies evaluating gene

expression profiling data

• A consistent definition of terminology is needed• The consensus of the attendees was to develop two types of

RNA standards, External Synthetic RNA Standard Reference Material & Internal RNA Reference Standard

June 10, 2003 30

Session 5Proposed Workshop Recommendations

• External Synthetic RNA Standard Reference Material– Moderately complex pool of highly characterized synthetic mRNA targets – Used across all RNA gene expression profiling platforms, including both RT-

PCR based and array based methods.– Measures the accuracy, dynamic range, sensitivity and specificity of each

platform under any given set of conditions.

• Internal RNA Reference Standard– Spike-in RNA standards used in conjunction with standard probe sequences– Compatible with all array formats, and available to labs making “in-house”

arrays– Provides an internal measure of the quality of any particular array experiment

(sensitivity, dynamic range, and specificity)– Array manufacturers could also use this material for quality control testing their

products

• Reference Method– Instructions on the correct use of the reference materials are required

June 10, 2003 31

External Synthetic RNA Standard Reference Material

• Modular components– Individually characterized and pooled– Extendable and upgradable

• Number of human sequences = 96• Maximum of 2000 bases beginning with 3’ end• Selected component characteristics

– Conserved, minimally polymorphic sequence– Well characterized tissue specificity, splice variants, gene family

members, etc.– Cloned into expression vector, sequenced, high purity and stability

• Represents 106 fold range of absolute quantitative expression

June 10, 2003 32

Internal RNA Reference Standard

• “Spike-in” standard to array probes serves as internal positive control of array performance

• One set of 12 sequences– Target sequence > 600 nucleotides– Sequence content acceptable to different array platforms– Cloned into expression vector, sequenced, high purity– Covers dynamic range of the platform

• All array platforms have optimized probes with 5-10X redundancy across the array

• Instructions on correct use of spike-in standard to be provided (standard reference method)

June 10, 2003 33

Open Questions and Next steps

•NIST Guidance document published by end of June 2003•Formation of working group members•Will NIST take up this project with limited resources and budget?•Will FDA make a formal request to NIST for specific standards to facilitate acceptance of Tox /Pharmacogenetic data submissions?•Should the platform manufacturers develop their own standards to establish platform performances?