The case for PPS validation

Professor Jacqui Reilly Health Protection Scotland and Glasgow Caledonian University

UK

Introduction to validation

• Validation studies are essential to assess the sensitivity (Se), specificity (Sp), positive predictive value (PPV) and negative predictive value (NPV) of PPS.

• Reproducibility (or reliability) is also an important concept for ensuring data is robust.

• Validation studies are rarely published, there are a variety of approaches described:

Less than half of the prevalence surveys published to date included an evaluation of the data.

Of those that included either a validation or an IRR study, the results were varied substantially

Underscoring the need for a formal evaluation to add confidence to the interpretation of the data.

Ref: Llata E, Gaynes RP, Fridkin S. Measuring the scope and magnitude of hospital-associated infection

in the United States: the value of prevalence surveys. Clin Infect Dis 2009 05/15;48(10):1434-1440.

Why does it matter?

3Cs: Consistency, comparisons and confidence

Low sensitivity (false negatives, or underreporting) of HAIs is a frequently encountered problem in HAI surveillance systems.

Low specificity (false positives, or over reporting) is usually less of a problem

Both may be related to one or more of following factors:

Difficulty in confirming the case definition of an infection if signs and symptoms were not well documented in the patient’s records

If diagnostic tests included in the case definition of a particular HAI type were not done

Non compliance with the definition of the key term ‘healthcare-associated’: even if the case definition of an infection is matched due to cultural or financial/ political incentives and disincentives at a hospital or country level

Prevalence validation

The case for validation in multi-country studies

In order to investigate variation between countries the first question to ask is: is it the data? Validity Reliability

Ref: European Centre for Disease Prevention and Control. Point prevalence survey of healthcare associated infections and antimicrobial use in European acute care hospitals. Stockholm: ECDC; 2013.

Validation results for PPS 2012

Ref: European Centre for Disease Prevention and Control. Point prevalence survey of healthcare associated infections and antimicrobial use in European acute care hospitals. Stockholm: ECDC; 2013.

Interpretation

The large differences observed between HAI prevalence

across Member States are in reality smaller The overall weighted HAI prevalence of 5.7% is likely to be a

slight underestimate. .

International PPS validation

Indonesia: – Significant differences were noted between the teams of data collectors

in terms of:

completeness of data, identifying patients who had undergone surgery in the previous month, and most importantly in the number of HAI detected.

SSI, UTI and septicaemia (p=0.01). (κ) did not exceed 0.60 and 0.59 respectively for any infection type.

– Ascertainment was affected by underreporting in medical records, the retrospective nature of data collection for validation purposes, and suboptimal adherence to the PPS protocol and case definitions.

Ref: Duerink DO, Roeshadi D, Wahjono H, Lestari ES, Hadi U, Wille JC, et al. Surveillance of healthcare-associated infections in Indonesian hospitals. J Hosp Infect 2006 02;62(2):219-229

Challenges with undertaking validation remain…. No internationally agreed standard/ protocol?

– ECDC now have a validation protocol for use in EU for national validation

Practical issues?

– Time/ funding – Identifying an external gold standard /independent of the

primary data collection

National and international validation

National validation: – Required for interpretation of HAI prevalence + burden

estimates – At the same time as the primary PPS – Recommended: re-examine 750 patients in 25 hospitals – Minimum: 250 patients in 5 hospitals – Support contracts with ECDC (10 000 EUR / country),

budget to be spread over (at least) 2 years (2016-2017) International validation:

– Validation of national validation teams (VT) – Accompany national VTs in 1-2 hospitals/country – Who?: Part of HAI-Net support call for tender (published

soon): contractors + ECDC experts

How can ECDC help?

Contracts for validation- financial support, feedback of

national validation results Contracts for International validation

Summary

Validation is a key component of surveillance for comparisons, consistency and confidence Without it we do not know the true prevalence of HAI Without it we cannot investigate reasons for variation

in HAI prevalence between hospitals and/ or countries

Knowing the true burden makes the case for infection prevention and control measures and enables improvement in HAI