Luisa Moro, Enrico Ricchizzi - Maria Luisa Moro.pdfAnnual HCAI & AMR meeting, Warsaw 2011 Validationstudy Maria Luisa Moro, Enrico Ricchizzi

Annual HCAI & AMR meeting, Warsaw 2011

ValidationValidation studystudy

Maria Luisa Moro, Enrico Ricchizzi

Validity

The degree to which an assessment

measures what it is supposed to

measure

Why a validation study?

Validation of surveillance data is necessary to:

ensure its scientific credibility,

to identify methodological problems within the

surveillance programme,

to help increase compliance and participation in the

surveillance programme,

to identify data quality issues at local level.

McCoubrey J. JHI 2005

Why a validation study in HALT2?

The accuracy of infection data collection should be

measured in order to estimate the true prevalence of

infections/antimicrobial usage and to be able to

meaningfully compare countries

The gap between residents with signs/symptoms of

infection and those with infections satisfying the

modified McGeer criteria should be better understood

Validation studies of HAIs surveillance

Chart review by externalexperts

200100 reported NI + 100 at random

ICUsFrance


148115 reported NI + 60 at random per ICU

ICUsGermany"


1300All reported NI + 20% at random

ICUsBelgiumMorales I

Chart review by externalexperts, structuredinterview, capture‐recapture

39710 orthopedicoperations with, and 40 non‐SSIs

SSIFinlandHuotari, 2007


8595 SSIs and the 20 most recent non‐SSIs

SSINetherlandsMannien, 2007


60215 reported SSIs and 60 non‐SSIs

SSIScotlandMcCoubrey, 2005


1,136All patients duringstudy period

SSIUSAEmori, 1998

Bedside inspection of allsurgical wounds

4552 consecutive prevalence studies

SSIDenmarkPoulsen, 1996

Gold standardNSamplingType ofinfectionCountryStudy, year

Validation study: how to organize it?• Bedside validation

– retrospective chart review by trained experts Advantages: higher number of patients may be reviewed; no influence on ongoing surveillance

Disadvantages: undocumented information cannot be retrieved; depending on quality of patient charts

– parallel prospective collection by trained expertsAdvantages: presence of clinical caregivers and additional information on the patient or treatment not recorded in the chart

Disadvantages: lower number of patients may be reviewed; may influence parallel data retrieval by local surveyors

• Validation by case studies

• Capture‐recapture (different data soruces)


Carley S, Emerg Med J 2005

Simple nomograms to calculate sample size in diagnostic studies

Validation study: how to organize it?

What should be validated

• quality of data collection (completeness and reliability)

• interpretation of the criteria for assessing an infection

(sensitivity and specificity)


Gold standard

• Truth (100% sensitivity and specificity)

• The best available method in applying surveillance

criteria (consistency, reliability, accuracy)

Pilot validation protocolacute care hospitals PPS

Different methods will be tested:

• Validation of HAI reported during the PPS (during the PPS or after the PPS day‐shortly or longer time after)

• Inter‐rater agreement

• Validation and accurateness of denominatordata

• Other “validation” methods: online case studies

ECDC Pilot validation study – acute care hospitals

• a validation team visited the nursing

homes in May or June 2010 and

assessed blinded, based on information

of nurses, all residents of the

participating nursing homes on having

an infection (gastro‐enteritis, influenza‐

like illness and probable pneumonia).

• Just before the visit, the local

surveyors had to fill in the register form

indicating specific infections of the

residents.

• The outcome measures are the

negative predictive value, positive

predictive value and the inter‐rater

agreement using Cohen’s kappa

coefficient.

• Gastroenteritis (n=5):Sensitivity: 80%PPV: 0.67Specificity: 99.8%NPV: 1.00

• Influenza‐like illness (n=2):Sensitivity: 0%PPV: 0Specificity: 100%NPV: 1.00

• Probable pnumonia (n=5):Sensitivity: 20%PPV: 0.25Specificity: 99.8%NPV: 1.00

N = 1429 residents

Validation study in LTCFs

The variable quality of resident charts does not

allow to conduct retrospective data collection

The average size and the prevalence rate of the

LTCFs requires a large number of facilities to

reach the sample size required to assess precisely

the sensitivity and specificity

Validation study: what

Facility form: structure and

process indicators

Aggregated data:

demographic characteristics,

care load & risk factors

Infections and antimicrobial

use

Validation study: gold standard

The reference or gold standard is the HALT2

protocol and codebook, applied by a

validation team of at least one trained expert

from (and/or acting on behalf of) the

national/ regional PPS coordinating centre

(external to the validated NH), accompanied

by a LTCF staff member.


National/regional validation team

• Trained staff of the PPS coordinating centre:– At least 1 senior expert with experience in HAI surveillance

(especially case definitions) in charge of the “gold standard” data collection

– Less experienced surveillance staff can joint the VT to speed up the data collection process

• Hospital staff: the hospital staff member does not participate in the validation process/data collection as such, but he/she is crucial to:– At the start of the VT visit, introduce the VT to other hospital staff

members where needed, have a short explanatory meeting with the VT, explain how the PPS was performed in the hospital, explain the hospital information system

– Ensure that the VT is provided access to all relevant data sources, at least to the information of the patients which were selected in the validation sample

– Accompany the VT to the different wards and introduce the VT


Validation study: how

Parallel data collection

• an external trained expert will visit a random

sample of LTCFs the PPS day

• the data collection will be blinded from the

local surveyors

Pilot validation study – Long Term Care Facilities

Validation study: how

Timing of data collection

• Simultaneous validation: same day in same

ward, after the primary data collection, good

approximation of the availability of results at

the time of the survey


Validation study: howBlinded collection

• Results of PPS data collectors are not disclosed during or immediately after the validation of a patient, but only at the end of the complete validation data collection in the ward


Advantage:

• Better guarantee for independent data collection

• Random sampling of patients

• Can be done at the same time as the primary data collection

Disadvantage:

• No indication of which data were found by the primary data collectors and that should be verified by the VT – therefore the quality of the validation may be less good than in the semi‐blinded method

• No differential sampling possible (eg all positives) to improve precision of specificity estimation

• Possibly more time consuming



process indicators

Aggregated data:




use



process indicators

The external surveyor will assess presence of item that can be documented by NH staff (e.g.. presence of written protocol, both written that electronic, etc.)

Validation study:Institutional questionnaire

Structure and process indicators

D – INFECTION CONTROL PRACTICE IN THE FACILITY

4. Which of the following tasks are in operation in the facility?

□ Registration of residents colonized/infected with multi‐resistant microorganisms

□ Feedback on surveillance results to the nursing/medical staff of the facility

8. In the facility, is a written protocol available for:

□ the management of MRSA carriers?

□ hand hygiene?

□ the management of urinary catheters?

□ the management of venous catheters/lines?

□ the management of enteral feeding?


Structure and process indicators

E – ANTIBIOTIC POLICY

4. Which of following elements are present/available in the facility?□ written guidelines for appropriate AB use (good practice) in the facility□microbiological samples taken for guidance of best AB choice□ permission from a designated person(s) for prescribing of restricted Abs, not included in local formulary □ therapeutic formulary, comprising a list of antibiotics□ feedback to the GPs on AB consumption in the facility

5. If written therapeutic guidelines are present in the facility, are they on:□ Respiratory tract infections?□ Urinary tract infections?□Wound and soft tissue infections?



process indicators

Aggregated data:




use


Care load & Risk factors

• Representative random sample of eligible population

• Which data will be collected using medical and nursing documentation & staff interviews:

• care load factors

• risk factors

• Data will be collected using the Resident questionnaire

This allows to simultaneously assess both the reliability of aggregated data and the presence of false negative cases



process indicators

Aggregated data:




use

Validation study:Resident questionnaire

Antibiotics and Infections

• All cases identified within the residents random sample (care load & risk factors) with signs & symptoms of infection or atb treatment + All cases detected by the internal surveyor

• Blinded data collection

• Data collection from clinical records and NH staff (not performing survey)


Antibiotics and Infections • Data collection of resident data:resident case mix must match the case mix recorded in the ward list!


• Data on therapeutic use of ATB:if the antibiotics use is reported infections signs and symptoms should be recorded!


• McGeer criteria for definition of infection:use of modified McGeer checklist to report signs and symptoms of infection and confirm HAI.


Antibiotics and Infections

• Infections: the external surveyor will decide for each infection if it satisfies Modified McGeer criteria, or not, or if it is impossible to judge from the available information

• Additional information on the quality of clinical records:

All the information used for the definition of infections were present in the clinical record

Some or all the information were collected from staff

Validation study:

pilot study

• LTCFs

• December 2011 ‐ February 2012

• 18 Italian NH, ranging from 20 to 190 beds

• Total population of 1200 residents

• No experience in PPS and Infection surveillance

One day course on surveillance and HALT protocol

No previous experience on HALT survey

Validation study:pilot study

•External surveyor: Geriatric specialist trainee

•Experience in infections and antibiotics survey

•Trained on the HALT protocol and definitions

•Data collection:

•Representative random sample of LTCF eligible population

•Aid of LTCF contact point

Discussion


• Sustainability at the European level of a parallel

validation study with an external reviewer?

• Feasable sample size (low prevalence + small

facilities + few facilities enrolled in each country)?

• Other possible validation approaches (eg case

studies)?

Documents

Luisa Moro, Enrico Ricchizzi - Maria Luisa Moro.pdfAnnual HCAI & AMR meeting, Warsaw 2011 Validationstudy Maria Luisa Moro, Enrico Ricchizzi