Www.hpa-midas.org.uk Evaluations and Validations post Carter Keith Perry, Head of Evaluations Evaluations & Standards Laboratory BSMT, November 2006

www.hpa-midas.org.uk

Evaluations and Validations post Carter

Keith Perry, Head of Evaluations

Evaluations & Standards Laboratory

BSMT, November 2006


• Carter review of NHS Pathology

• Ways we currently assess

• Evaluations post-Carter

• Evaluations & managed introduction of new technology?

Today’s presentation


2002

Need for standardised methods

Need forco-ordinated system for evaluation managed intro of new technology


Need for standardised methods

Need forco-ordinated system for evaluation managed intro of new technology

2006


• Little evidence of a strategic programme of investment for new & replacement technology

• Little automated ‘front end’ & sample tracking

• NHS too focussed on cost not benefit

• Difficult to cost pathology services

• Silos

• Fragmented pathology services

• Relatively small investment can make a difference

• Labs can influence expensive treatments

Carter review of NHS Pathology

What was found ?


• Managed introduction of new technology - Benefits realisation - Competition

• Appropriate investment

• Work equipment harder

• Healthcare closer to home

• More PoC testing – accreditation & links with labs

• Common national framework

• Standardisation of test results

• CEP – PASA role

• Evaluations

Carter review of NHS Pathology

Recommendations


Ways we currently assess

But does not cover:

• Ease of Use

• Monitoring of internal QCs

• External QA

• Use of combinations ie kits/equipment

• Comparative data

• Evaluation and validation

CE Marking


• Focus on device performance

• Comparative

• Independent and unbiased

• Provide informative reports

• Must be accurate and timely

• Relevant to standardisation of methods

• Encourage development

• Should influence managed introduction of new technology


Evaluations

www.hpa-midas.org.uk= combined antigen-antibody = immunometric = Class specific antibody capture = antiglobulin / indirect

Biotest Anti-HIV

TETRA ELISA

Wellcozyme

Anti-HIV

50 10 15 2020

*Ear

liest

an

ti-H

IV d

etec

tio

n

days

OrthoAb-capt.ELISA

Vitros ECianti-HIV

1/2

MurexHIV 1/2VK84/85

IMxHIV1/2III plus

MurexHIV 1.2.0GE94/95

VironostikaHIV Uni-form II Ag/Ab

EnzygnostHIV 1/2

plusBiotest Anti-HIV TETRA

ELISA

Clonesystems Detect-HIV

v1

InnotestHIV-1/-2

Murex ICEHIV 1+2

AxSYM HIV 1/2 gO

Abbott3rd gen

Plus

PasteurGenscreenVersion 2

Access HIV 1/2 NEW

Ear

liest

HIV

det

ecti

on

Enzygnost HIV

Integral

VIDAS HIV DUO

VironostikaHIV Uniform II

plus O

GENSCREEN PLUS HIV Ag-

Ab

AxSYM HIV Ag/Ab Combo

Murex HIV Ag/Ab

combination BiotestHIV 1/2

recombinant

Timing of detection of primary HIV following seroconversion



Molecular Assessments with CVN in collaboration with ESL to reassure users that their in-house assays give comparable results

• In-house real-time PCR assays

• Assays assessed to date (DNA viruses: HSV1 and 2; ADV; CMV)

• Viral targets chosen via CVN committee & members

• Labs submit SOPs

• Panel of 25/target, comprising dilutions of strong or tissue-culture grown material extracted using two methods (9), plus clinical specimens (6) sent out for assessment

• Next RSV, VZV, Norovirus, & Enterovirus


Assessments


• Any new equipment or kit

• Molecular extraction machines

• New media


Quick Look/See


• Hospital based study of IDI-MRSA (direct detection of MRSA in nasal swabs by a rapid molecular method )

• bioMérieux Vitek


Clinical Assessments


Validation is the confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled (ISO 17025:2005)

Validation is evaluation of the whole process & ability to achieve the correct result (not the kit or reagent in isolation)


Validation


VALIDATIONFILE


Evidence of CE Marking

Independent Evaluation Reports

Published papers

Manufacturer's instructions/literature

In-house R & D

WorkbooksLaboratory validation on known samples

IQA

EQA

QC data

Relevant SOPs

Error logging

Customer complaints




‘Evaluation’ issues post-Carter

www.pasa.nhs.uk/evaluation

National framework



• The CEP-PASA Prioritisation Board meets 3 times per year to consider proposals

• Projects that are innovative, related to government priorities or purchasing exercises will gain higher scores

• Next closing date for proposals: 31st January 2007

• Next meeting: 14th March 2007

• website www.pasa.nhs,uk.evaluation/propose_project

Prioritisation


Extent of NHS use

Evaluation

Adoption/purchase

Time

Evaluation project proposals

• Chair • Head of CEP• Business Planning Manager• Head of Commissioning and Delivery• Technical Advisor• PASA / Hubs (2 reps)• NICE (1 rep)• NIIII (1 rep)• HTA (1 rep)• NPSA (1 rep)• MHRA (1 rep)

Prioritisation Board

National Evaluations Programme for Microbiology

Priorities to be aligned to customer requirements& peer-organisations

Anyone cansubmit proposals




• High rate of introduction of new diagnostic tests & automation

• Many subject areas are not covered by evaluations

Scope


Products to reduce HAIs

Virology

Bacteriology

Parasitology

Ser

olog

y

Mol

ecul

ar

Aut

omat

ion

Iden

tific

atio

n

HIV EIAsHCV EIAs

HBsAg EIAsPoCTs

ChlamydiaNAATs

ToxoplasmaEIAs

Extraction,Processors,

Closed systems

HSV Real-time

PCRs

Kiestraplate

automation

SyphilisEIAs

Culture media


Scope



Evidence & processes

• Well designed studies

• Well defined procedures

• Challenging specimen panel

• Expert input

• Technical quality

• Comprehensive quality system

• Safeguards / checks in place

• Project management





Recognised evaluation protocols

• e.g. for evaluations of lab automation

• Developed in consultation with experts / users

• Developed with reference to National Standard Methods

• Process for review and authorisation



National Evaluations Register

• Study title• Status (proposal, preparation, technical, report in draft, completed)

• Expected Output (evaluation report, look/see, literature review, market review)

• Lead centre • Contact name, telephone, email

Awareness



• Evaluating relevance & utility is time consuming

• Products are often in use prior to evaluation report

• Matching evaluations and tender processes

Timing



Mechanism

• The mechanism for where and how evaluations are done is still insufficiently defined

• Central front-door through CEP evaluation centres?

• Evaluation networks co-ordinated through CEP centres?

- with expert working groups - professional groups - reference laboratories- clinical user groups



• How large is the cost?

• How large is the benefit?

• Is introducing the test/device cost effective?

• Resources

• Integral to the work of CEP-PASA

• CEP has Health Economist in post

Cost benefit


• Government priority

• To tackle a specific issue (Chlamydia screening programme)

• Money made available

• Implementation of new technology preceded evaluation (used literature reviews)

• Multi-site evaluation took >2 years

Evaluations & managed introduction of new technology?

Chlamydia NAATs


Data collation,analysis,& report HPA-ESL

Urine25mls+

RoutineScreen

Source Labs (3)

Retest positives

Report 1

Routine EvaluationSpecimen sourcing Transport

Identifydiscordants

HPA-ESL

SDA 2b

TMA 3b

PCR 4b

Aliquot,Label &

Send

Portsmouth

Liverpool

UCLH / Kings

HPA-STBRL

Add to TMA tube

SDA 2a

TMA 3a

PCR 4a

R-timePCR

5

SDA pouch

Aim for 4 day turnaround

Chlamydia NAATs evaluation

Evaluations & managed introduction of new technology?


• Responding to the need for a wide range of microbiological device evaluations

• How to prioritise

• Joining device performance with procurement

• Earlier assessment and usage of innovative technologies

• Improved access to available evaluation results eg National Evaluations Register

• Improved sharing of information

• Impact on lab autonomy to choose equipment/kits

• Funding of evaluations (eg PASA)

• Funding for implementation (capital & revenue)

Challenges


• All collaborators / advisors

• All in Evaluations & Standards Laboratory, including:

- Valerie Bevan, Director ESL

- Standards, Quality Control Reagents & Quality System Units

- The Evaluations Team:

Katrina Barlow

Christine Burgess

Johanna Curtis

Laura Dean

Galit Gonen

Fu Li

Thanks to….

Documents

Www.hpa-midas.org.uk Evaluations and Validations post Carter Keith Perry, Head of Evaluations Evaluations & Standards Laboratory BSMT, November 2006