Immune assay harmonization and external quality assurance:

Results of the ongoing Proficiency PanelProgram of the Cancer Vaccine Consortium

Sylvia Janetzki M DSylvia Janetzki, M.D.ZellNet Consulting, Inc.

Introduction – CVC (1)Introduction CVC (1)

I th it id b hi i ti• Immunotherapy community-wide membership organization• Run under Sabin Vaccine Institute umbrella (2002-2007) • Now program of the Cancer Research Institute (CRI)

VisionTo cure and prevent cancer throughTo cure and prevent cancer through the advancement of cancer vaccines and immunotherapeuticsand immunotherapeutics.

Cancer Vaccine Consortium - History

2002 2003 2004 2005 2006 2007

Immuno-Assessment

FDA2nd ElispotCVC 1st Elispot

Satellite Symposium

Colloquium and Annual Meeting

FDA Workshop

2 Elispot Panel

3rd Elispot,

CVCFounded

1 Elispot Panel

Potency Clinical Trials p ,1st ICS, CFSE, and Multimer

yTestingWorkshop

Commercialization

Clinical TrialsWorking Group

Strategy Meeting

Phase 3 Clinical TrialsWorkshop

Proficiency Panel Program ObjectivesProficiency Panel - Program Objectives

• To offer an external assay validation program• To enhance assay harmonizationy

Goal:Goal:

Establish immune assays as standardEstablish immune assays as standard monitoring tools in clinical trials and clinical

ti ttiroutine settings

R i d R f h it tiReminder: Reasons for hesitationmeasuring immune endpoints

1 L k li it d k l d b t1. Lack or very limited knowledge about correlation with clinical outcome

2. Lack of confidence in data obtained from bioassays Validity of data

1. Correlates of protection: there are examples

Dudley: Science 2002; Melanoma (repopulation with TILs – tumor regression)

Morgan: Science 2006; Melanoma (transfer of genetically modified lymphocytes – cancer regression)modified lymphocytes cancer regression)

Todryk: PLos 2008, Correlation of memory T cell response and protection against Malaria

Forrest: Clin Vaccine Immunology 2008; CorrelationForrest: Clin Vaccine Immunology 2008; Correlation vaccination – cellular immune response (Elispot) – protectionagainst Influenzaagainst Influenza

Challenges• Lack of consistent correlation (countless publications)

g

• Limited knowledge about Tcell mediated efficacy (Merck STEP trial)

• Rather qualitative than quantitative monitoring (Appay)

• Polyfunctional effector cells (Darrah, Roederer)y ( , )

• Migration of effector cells to tissue (DeVries/tetramers)

Maximum recovery of relevant immunological data

Reliable immune monitoring techniques

2 C fid i d t bt i d f bi2. Confidence in data obtained from bio-assays

Negative example: Lack of acknowledgment of immune monitoring data from Sipuleucel-T trial (phase 3; Small 2006) d it i ifi t l ti ith i ti t tdespite significant correlation with vaccination status Reason: lack of bioassay validation (proliferation assay)

Do YOU believe immune monitoring data published?Always?Always?

Data need to be reliable, repeatable, plausible= VALID= VALID

Validity of DataValidity of Data: Ensure that the results of a study involvingGLP y g

human subjects areReliable– Reliable

– Repeatable– Auditable– RecognizablegGLP includes training, SOP, Reagents and instruments audits and external qualityinstruments , audits and external quality assurance.

= proficiency panel testing

Program ObjectivesProgram ObjectivesCVC Proficiency Panel

• To offer an external assay validation program• To enhance assay harmonizationTo enhance assay harmonization

Goal:

Establish immune assays as standard monitoring tools in clinical trials and clinicalmonitoring tools in clinical trials and clinical routine settings

Establish immune assays as a platform for biomarker discovery and developmentdiscovery and development

Assay Harmonization

T ll bilit f lt l b

• Identification of crucial protocol variables that

To allow comparability of results across labs

pinfluence assay outcome

• Formulation of acceptable and feasibleFormulation of acceptable and feasible recommendations to the field

• Enhancement of their implementation and• Enhancement of their implementation and confirmation of their usefulnessA li ti f h i d i li i l t i l• Application of harmonized assays in clinical trials

The CVC Proficiency panel program

The CVC Proficiency Panel Program2005: 1st Elispot panel (36 labs, 10 countries)

The CVC Proficiency Panel Program

2006: 2nd Elispot panel (29 labs, 7 countries)2007: 3rd Elispot panel (35 labs)

1st Multimer panel (29 labs)1st ICS panel (28 labs)

10 countries

1st CFSE panel (21 labs)

Central lab: PBMC (pretested), antigen, shippingScientific panel leadersCVC office for central administrationIndependent statistician

Panelists 2005 - 2007Panelists 2005 - 2007

Total: 63 entities (some with multiple labs participating)Total: 63 entities (some with multiple labs participating)

1. Academia/hospitals/non-profit: 332. Biotech/biopharma/services: 193. Big Pharma: 43. Big Pharma: 44. Government / Armed forces: 3(5 Supplies/central lab: 4 )(5. Supplies/central lab: 4 )

Cancer: 46/59 = 78% Infectious diseases: 11/59 = 19% (HIV, TB, other)Other: 2/59 = 3%

Elispot panel: ProtocolElispot panel: Protocol

• 4 pre-tested PBMC from same lot (3 runs, 2 protocols)p ( , p )• Pre-tested CEF and CMV peptide pool from same lot• PBS/DMSO from same lotPBS/DMSO from same lot• Same plate layout (6 replicates; 24 medium alone)• Same number of cells and amount of reagentsSame number of cells and amount of reagents• IFN- Elispot, no pre-stimulation

• Lab-specific protocol, materials and equipment(based on premise that each participating lab has a SOP)(based on premise that each participating lab has a SOP)

• No PHA or PMA/Iono (confluence and counting issues)

Response DefinitionResponse Definition

None: < 10 spots/200,000 PBMCLow: 10 – 50 spots/200,000 PBMCp ,Medium: 51 – 100 spots/200,000 PBMCHigh: >100 spots/200 000 PBMCHigh: >100 spots/200,000 PBMC

Response detection (yes/no):

10 spots/200,000 PBMC 3x background (PBMC alone) 3x background (PBMC alone)

EPP-1EPP 1

F2 all participants (medium strong response – CEF)

4/36 labs outliers = missed to detect > 50% of responses17/36 labs missed to detect low responder against CMV17/36 labs missed to detect low responder against CMV13/36 labs missed to detect low responder against CEF

Challenges to detect crucial protocol variablesChallenges to detect crucial protocol variables

St d d i ith t l h i• Study design with open protocol choices

• Wide range in protocol and reagent choicesg p g

(e.g. 7 ab sources, 10 enzyme sources, 7 readers, cell

i i b 2 d 20 h iresting times between 2 and 20 hours, unique

serum choices for each lab etc.))

• Dispersion of protocol choices small sample sizes

• No adequately powered statistical analysis possible

EPP2Panel participation available for all CVC memberswith sufficient experience

EPP2

with sufficient experience

F2

No outliers4 labs missed to detect the low responder 47% in EPP1 14% in EPP2(+ 5 marginal)

Different protocols – same results

D1

Lab 1 Lab 2 Lab 3Plate PVDF MCE PVDFAntibodies Mabtech Mabtech Own KitDevelopment HRP/AEC AP/BCIP HRP/NovaredDevelopment HRP/AEC AP/BCIP HRP/NovaredSerum Human AB FCS FBSDNase No Yes YesResting No Yes/ON Yes/ONResting No Yes/ON Yes/ONCell counting Hemacyto Hemacyto Automated/GuavaReader Zeiss AID CTL

Similar protocols – different resultsp

D3

Lab 4 Lab 5

Plate MCE MCEAntibodies Mabtech MabtechDevelopment HRP/AEC HRP/AECSerum Human AB AIM-VDNase No NoResting No NoCell counting Hemacytometer HemacytometerCell counting Hemacytometer HemacytometerReader Zeiss Zeiss

Influence of automated cell counter use

1000

100

s pe

r wel

l • No outlier• 2 labs missed

10

Spot

cou

nt • 2 labs missed weak responder

D4-med

D4-CEF

D4-CMV

D3-med

D3-CEF

D3-CMV

D2-med

D2-CEF

D2-CMV

D1-med

D1-CEF

D1-CMV

1Donor & Reagent

DDDDDDDD

Mean cell viabilityGuava users Non-Guava users

Donor 1 80.8 86.7

Donor 2 78.4 85.6p<0 01

Apoptosis!

Donor 3 84.0 89.8

Donor 4 79.2 85.5

p<0.01

Advantage of overnight resting before platingAdvantage of overnight resting before plating

900

700800900

per

wel

l 2 hours rest 20 hours rest

400500600

ot c

ount

p

100200300

Mea

n sp

o

0D1-

CMVD1-CEF

D2-CMV

D2-CEF

D3-CMV

D3-CEF

D4-CMV

D4-CEF

Donor and Reagent

Background reactivity: 0-5 spots

Plate evaluation is operator- and SOP-dependent

Counts Lab X Lab Y Lab Z

Plate evaluation is operator and SOP dependent

Own Central Own Central Own Central

Mean 81.5 3.3 16.7 4.8 9.7 1.7

SD 104.7 1.8 25.9 2.9 5.3 1.6

Median 34.5 3.5 6 5.5 10 1.5

Mi i 3 1 3 1 2 0Minimum 3 1 3 1 2 0

Maximum 270 6 69 9 18 4

Background counts (PBMC alone)

Initial Elispot Harmonization Guidelines

A. Establish lab Elispot SOP for:A1. Counting method for apoptotic cells in order to

determine adequate cell dilution for platingA2. Overnight resting of cells prior to plating

B. Use only pretested serum with optimal signal:noise ratioB. Use only pretested serum with optimal signal:noise ratioC. Establish SOP for plate reading, including:C1 Human auditing during reading processC1. Human auditing during reading processC2. Adequate adjustment for technical artifacts

D. Only let well trained personnel conduct assay

Janetzki et al., CII. 2008 Mar;57(3):303-15

EPP 3 (35): new PBMC, 13 labs new( ) ,(Recommendation to implement harmonization guidelines)

3/22 repeaters missedresponse = 14%p

No outliers

4/13 new labs missed response = 31%

EPP3: Validation PhaseEPP3: Validation Phase

20 labs said that they did comply with all20 labs said that they did comply with all Harmonization guidelines (57%).9 of these labs were “top” performers (9/9 =100%)9 of these labs were top performers (9/9 =100%)

15 labs said they did not comply with all15 labs said they did not comply with all Harmonization guidelines (43%).5 of those labs missed (a) response(s) (=33%).5 of those labs missed (a) response(s) ( 33%).

13 labs introduced changes to their protocol in g presponse to the harmonization guidelines (37%).

- 7 added overnight restingg g- 2 changed medium/serum

Overnight rest of cellsOvernight rest of cells

EPP1 EPP3

T t l # f l b 36 35Total # of labs 36 35

Yes 17 (47%) 29 (83%)Yes 17 (47%) 29 (83%)No 19 (53%) 6 (17%)

>8hrs 10 (59%) 22 (76%)<8 hrs 7 (41%) 7 (24%)

Automated cell counter useAutomated cell counter useApoptosis assessment

EPP1 EPP3

Total lab# 36 35

Yes 7 (19%) 14 (40%)No 29 (81%) 21 (60%)

Yes- but missedResponse(s) 2 (29%) 0 (0%)Response(s) 2 (29%) 0 (0%)

Confirmation of guidelinesConfirmation of guidelines

• 7/13 responses missed due to serum choice(high background reactivity)( g g y)

• 4/13 responses missed due to plate reading strategiesstrategies

Next round (EPP4):Next round (EPP4):• Tackle of serum issue • Integration of guidelines

Serum TaskSerum TaskGoal: to identify at least 1 serum-free medium thatperforms at least equally well as lab’s medium/serumWho: “Top” performersWho: Top performersWhat: Own medium/serum, 3 serum free media( C )(ExVivo, AIM-V, CTL)How: EPP pretested PBMC and antigenSame setup for all 4 platesSame protocolSame protocolSame experiment

CVC Multimer panel 1CVC Multimer panel 1Scientific leaders: Pedor Romero, LICR/Lausanne

& Cedrik Britten, Leiden

• 27 labs from 7 countriesPBMC f 5 t t d d ith i ti it

,

• PBMCs from 5 pre-tested donors with various reactivity against HLA-A2-restricted

Influenza M1 58 66 (GILGFVFTL)- Influenza-M1 58-66 (GILGFVFTL)- Melan A/Mart-1 26-35 (ELAGIGILTV)

St d d lti ( ti ) id d• Standard multimers (antigens) were provided:• Tetramers from Beckman Coulter

P t f P I• Pentamers from ProImmune• Labs used their own staining protocol

MPP1 OutcomeMPP1 - OutcomeThree clear recommendations could be deducedThree clear recommendations could be deduced(Britten, Romero; manuscript in preparation)

Next steps:• Integration of new recommendations

• Discussion about background controlDiscussion about background control

• Systematic examination of protocol variablesSystematic examination of protocol variables identified which might have influence on final resultsresults

ICS Panel 1: Rude awakening

Panelist received:

Scientific leader: Holden Maecker & Maria Jaimes, BD

Panelist received:1) PBCMs 3 donors with different CEF and CMVpp65 reactivity 2) Lyoplate containing antigens for stimulation of cells (triplicates)2) Lyoplate containing antigens for stimulation of cells (triplicates)3) Lyoplate containing staining reagents:a) CD4 FITC / IFNg+IL-2 PE / CD8 PerCP-Cy5.5 / CD3 APC b) Comp Control beads

and instructions for acquisition and analysis using those beads4) EDTA FACS Lysing solution FACS Permeabilizing Solution24) EDTA, FACS Lysing solution, FACS Permeabilizing Solution2

Highly standardized protocol and reagentsNote: protocol and reagents validated at BD and via NIH ICS panel

ICS panel 1ICS panel 1“Gold standard”

Example1Example1

Example2CD4/LP13

Next Steps ICSNext Steps - ICSThe 1st ICS panel has shown: it is not a matter of using “someone's”p gvalidated protocol and reagents - own validation, training per ownSOP, experience and own standards are key!

• Standardized protocol

Own protocol

Fi t d ti• First recommendations:

Compensation and gating strategies p g g gBackground control issue Experience? SOPs?

CFSE panelp

Example: CD8 - all

Scientific leader: Jean Boyer, Dan Schullery, UPenn

8090

LP7 LP19 LP27(negative)p

4050607080

0102030

0.89 0.97 0.040 0.76 0.51 0.003 0.64 0.58 0.011pp(Relative to DMSO)(Relative to DMSO)

17 labs: 3 donors against CMV pp65 and CEFProtocol recommendation, but only few strictrequirements

Challenges for CFSE panelChallenges for CFSE panel

• Huge variability• Huge variability• Huge variability• Very few acceptable responses observed• Huge variability• Very few acceptable responses observed• How much do we know about the assay and

its applicability in immune monitoring of • How much do we know about the assay and

its applicability in immune monitoring of cancer vaccines?

• Diverse sera sources (serum known to be of cancer vaccines?

• Diverse sera sources (serum known to be of (huge influence in CFSE assay – NIAID panel)

• Vague gating strategies

(huge influence in CFSE assay – NIAID panel)

• Vague gating strategiesVague gating strategiesVague gating strategies

Discontinued due to questionable positionin cancer field

SummarySummary

• Proficiency panel program has shown to be of• Proficiency panel program has shown to be of high value for external quality assurance and for assay harmonizationassay harmonization

• First harmonization guidelines for Elispot• First recommendations for Multimer and ICS• First recommendations for Multimer and ICS• CFSE with questionable current position in

cancer vaccine monitoringcancer vaccine monitoring• Further systematic examination of crucial

protocol variables ongoingprotocol variables ongoing• Open for collaboration

AcknowledgementAcknowledgement• The Cancer Vaccine Consortium, its Executive,

Council, Executive office, and umbrella organizationorganization.

• The Steering Committee of the Assay Working GroupGroup.

• All supporters and suppliers (BD, Beckman C lt CTL NIH/ Di i i f AIDSCoulter, CTL, NIH/ Division of AIDS, Proimmune, Seracare).

• Our participants.