Evaluation of a surveillance system Pawel Stefanoff

Evaluation of a surveillance system

Pawel Stefanoff

Evaluation of a surveillance system:

a cyclic process

Gaps between needs and capacities

Implementation

Monitoring and evaluation

Planning for improvements

WHAT do we want to assess?

Technique

Disease specific surveillance

Surveillance system Complexity

The way of defining the problem methods solution.

• To assess the capacity of the system to meet its purpose and objectives 

• In order to – improve its operation

– modify it to address: • priorities

• epidemiology

– optimize the available resources

Why evaluate?

• Description of the needs– Country context

– Disease patterns

– Target audience

• Description of the surveillance system– Objectives

– Process

– Structure: resources, system

• Analysis of attributes

• Analysis of capacities /needs

• Implementation of prioritized modifications

Plan of evaluation

• Description of the needs– Country context

– Disease patterns

– Target audience

• Description of the surveillance system– Objectives

– Process

– Structure: resources, system

• Analysis of attributes

• Analysis of capacities /needs

• Implementation of prioritized modifications

Plan of evaluation

Description of surveillance needs

• Description of the country context– Population

– Health system

– Political structure and administration

• Description of disease patterns:– Disease characteristics (epidemic potential)

– Risk factors (country-specific?)

What’s the target audience of surveillance?

• Public Health professionals?

• Clinicians / Microbiologists / Control of Infection staff?

• Environmental Health professionals?

• Health service managers?

• Health educators / teachers?

• Government / Politicians?

• Public?

• Description of the needs– Country context

– Disease patterns

– Target audience

• Description of the surveillance system– Objectives

– Process

– Structure: resources, system

• Analysis of attributes

• Analysis of capacities /needs

• Implementation of prioritized modifications

Plan of evaluation

Description of the surveillance system

• The description of the surveillance system is a scientific process– Poor methods will generate unreliable

results

• When describing a surveillance system:– Report your methods in the “methods”

section

– Write the description in the “results” section

Methods that may be used to describe a surveillance system

1. Review of documents

2. Review of products/ outputs

3. Qualitative interviews of key informants

4. Quantitative interviews of participants

1. Reviewing documents

• Consider:– Work plans

– Guidelines

– Operation manuals

– Forms

• Generates information on the theoretical surveillance system

2. Reviewing products/ outputs

• Consider:– Records

– Data

– Feedback reports

• Generates information on the actual surveillance system

Elements of the surveillance system

to describe (1/2)

• Diseases under surveillance

• Forms used

• Case definition

• Population under surveillance– Total, sentinel or special

• Reporting mode – Passive, stimulated or active

Elements of the surveillance system

to describe (2/2)

• Data structure– Individual or aggregated

• Data transmission– Flow chart

• Data analysis procedures– Type, frequency

• Feedback – Type, frequency

• Action taken

Define objectives of a surveillance system

• Identify or help formulating the actual or desirable objective(s) of the surveillance system– Identify the goal of the programme served

– Identify what decisions may need to be taken

• Review the attributes of the system

• Review whether the system’s attributes are satisfactory to reach the objective(s)

• Description of the needs– Country context

– Disease patterns

– Target audience

• Description of the surveillance system– Objectives

– Process

– Structure: resources, system

• Analysis of attributes

• Analysis of capacities /needs

• Implementation of prioritized modifications

Plan of evaluation

• Simplicity

• Flexibility

• Acceptability

• Data quality

• Sensitivity

• Positive predictive value

• Representativeness

• Timeliness

• Cost

System attributes

Simplicity

• Structure– information needed

– number and type of sources

– number of information users

• Ease of operation– data transmission

– system maintenance

– data analysis

– information dissemination

Flowchart for surveillance of HIV/AIDS in Norway

Referencelaboratory

Primary HIV reporting form,Blood sample for HIV test laboratory part 1

Lab report and HIV reporting form

HIV reporting form, part 2HIV infection Primary care (Prompting if necessary) National Institute

physician of Public Health

AIDS reporting formAIDS Hospital physician Semiannual check

Oral informationDeath, emigration Semiannual check

Patient

• Ability of the system to accommodate changes with little additional time, persons or allocated funds– New event to follow-up

– New case definition

– New data about an event

– New sources of information

Flexibility

Acceptability

• Knowledge: – Case definition, notification procedures

• Notification conditions– Sufficient stock of notification forms

– Working charge

– Simplicity

• Motivation– Understanding the importance

– Risk perception

– Existence of a regular feedback

Data quality

Completeness and validity of the data recorded

• Completenness of records – blanks in the surveillance forms

• Errors when computing data

• Influenced by– Simplicity of surveillance form

– Clarity of electronic surveillance forms

– Training

– Validation

Sensitivity

• Two aspects:– proportion of cases of a disease or health condition

detected by the surveillance system.

– ability to detect outbreaks, to monitor changes (out of outbreaks, how many were detected)

• Requires (in principle)– to validate the information collected and

– to collect information outside of the system to determine the frequency of the condition in the community.

Factors influencing case detection

• The probability of medical consultation– Severity of symptoms

– Access to health care

– Social and cultural values and habits

• The probability of clinical diagnosis– Pathogen causes a clinical condition also caused by

other pathogens

• The probability of lab diagnosis– Demand for lab-testing

– Existence of laboratories with the specific technique

Quantitative estimates of the sensitivity of a surveillance system

• Health care facility survey– Proportion of cases seen in the health care

facility that are reported / captured by the surveillance system

• Population survey– Proportion of cases occurring in the

population:• Seen in health care facilities• Reported / captured by the surveillance system

• Mathematical model– Cases captured /expected

Sensitivity of measles surveillance in Haora district, West Bengal,

India• Methods used

– Cluster sample with door to door case search– Review of health care facility and surveillance records

• Key results:– Of 240 cases identified in the population, 8 (3.3%)

were seen in public health care facilities and reported

• Conclusion:– Measles surveillance is only 3.3% sensitive

• Recommendation:– Promote vitamin A management in the population to

increase the use of health system in the case of measles

Positive Predictive Value

• PPV depends on– sensitivity– specificity, and the – prevalence of the condition in the population

• Consequence of a low PPV– frequent "false-positive" report– inappropriate follow-up of non-cases– incorrect identification of epidemics (artifacts)– wastage of resources– inappropriate public concern (credibility)

Measles case definitions

• Suspect case– rash and fever

• Probable case– rash, fever, and (cough, coryza or

conjunctivitis)

– epidemiological link to a confirmed case

• Laboratory confirmed– saliva/serum IgM positive

Predictive value of notified measlesEffect of change in incidence

1

10

100

1000

10000

100000

1000000

Pre-vaccine Low coverage High coverage Near elimination

Num

ber o

f cas

es

0 %

20 %

40 %

60 %

80 %

100 %

PV+

Non-measles Genuine measles

• Representative = accurately describing the distribution of a event health in the population by place and time

• Related to – data quality

– bias of data collection

– completeness of reporting

Representativeness

Time

Timeliness

• Delay / speed between steps in surveillance– onset

– diagnosis

– report

– data entry

– analysis

– interpretation

– intervention

• Chronic vs acute disease

Notification completed

Notification received

Intervention

0 2 4 6 8 10 12 14

# days since onset of symptoms

(n = 255)

(n = 264)

(n = 216)

Median delay between notification and intervention by public health

servicesShigellosis, Delaware, 1991

Cost

• Cost is the estimated economic cost of the system

• Critical evaluation:– Could the system do an identical or better

job while using less resources?

SensitivityRepresentativeness

Predictive value positive

TimelinessAcceptability

FlexibilitySimplicity

Cost

Buehler's balance of systems attributes

• Description of the needs– Country context

– Disease patterns

– Target audience

• Description of the surveillance system– Objectives

– Process

– Structure: resources, system

• Analysis of attributes

• Analysis of capacities /needs

• Implementation of prioritized modifications

Plan of evaluation

Gaps between needs and capacities

• Is the system able to– adapt to changing needs

– detect cases and outbreaks

– investigate and intervene

– disseminate and exchange information

identification of weaknesses

• Description of the needs– Country context

– Disease patterns

– Target audience

• Description of the surveillance system– Objectives

– Process

– Structure: resources, system

• Analysis of attributes

• Analysis of capacities /needs

• Implementation of prioritized modifications

Plan of evaluation

Weaknesses and their causes - - - - - - -

Prioritisation - - - - - -

Objectives for improvement - - - -

Prioritisation of weaknesses to be addressed

Example 1

• Surveillance of HIV/AIDS in Spain– notification on AIDS cases

– aggregate reporting by laboratories on new HIV diagnoses

– sentinel clinics report on HIV testing

• Which attributes are relevant?

Example 2

• Surveillance for surgical site infections in a hospital– each patient after surgery is observed

– complications are recorded on the observation card

• Which attributes are relevant?

Example 3

• Surveillance collecting information on viral meningitis in Poland– detection of enteroviral outbreaks

– investigation for poliovirus

– monitoring tick-borne encephalitis foci

– identification of new neurotropic viruses

• Which atributes are relevant?