Statistical Methods for Alerting Algorithms in Biosurveillance

Howard S. BurkomThe Johns Hopkins University Applied Physics Laboratory

National Security Technology Department

Washington Statistical Society SeminarFebruary 3, 2006

National Center for Health StatisticsHyattsville, MD

• ESSENCE: An Electronic Surveillance System for the Early Notification of Community-based Epidemics

• Monitoring health care data from ~800 military treatment facilities since Sept. 2001

• Evaluating data sources– Civilian physician visits– OTC pharmacy sales– Prescription sales– Nurse hotline/EMS data– Absentee rate data

• Developing & implementing alerting algorithms

ESSENCE Biosurveillance Systems

Outline of Talk

• Prospective Syndromic Surveillance: introduction, challenges

• Algorithm Evaluation Approaches• Statistical Quality Control in Health

Surveillance• Data Modeling and Process Control• Regression Modeling Approach• Generalized Exponential Smoothing• Comparison Study• Summary & Research Directions

Required Disciplines: Medical/Epi

Medical/Epidemiological• filtering/classifying clinical

records => syndromes• interpretation/response to

system output• coding/chief complaint

interpretation

Required Disciplines: Informatics

Information Technology• surveillance system

architecture• data ingestion/cleaning • interface between health

monitors and system

Required Disciplines: Analytics

Analytical• Statistical hypothesis tests• Data mining/automated

learning• Adaptation of methodology to

background data behavior

Essential Task Interaction in Volatile Data Background

Medical/Epidemiological• filtering/classifying clinical

records => syndromes• interpretation/response to

system output• coding/chief complaint

interpretation

Information Technology• surveillance system

architecture• data ingestion/cleaning • interface between health

monitors and system

Analytical• Statistical hypothesis tests• Data mining/automated learning• Adaptation of methodology to

background data behavior

The Multivariate Temporal Surveillance Problem

Multivariate Nature of Problem:• Many locations

• Multiple syndromes

• Stratification by age, gender, other covariates

Surveillance Challenges:

• Defining anomalous behavior(s)

– Hypothesis tests--both appropriate and timely

• Avoiding excessive alerting due to multiple testing

– Correlation among data streams

– Varying noise backgrounds

• Communication with/among users at different levels

• Data reduction and visualization

Varying Nature of the Data:• Scale, trend, day-of-week, seasonal

behavior depending on grouping:

Data issues affecting monitoring

– Statistical properties• Scale and random dispersion

– Periodic effects• Day-of-week effects, seasonality

– Delayed (often variably) availability in monitoring system– Trends: long/short term: many causes, incl. changes in:

• Population distribution or demographic composition• Data provider participation• Consumer health care behavior• Coding or billing practices

– Prolonged data drop-outs, sometimes with catch-ups– Outliers unrelated to infectious disease levels

• Often due to problems in data chain• Inclement weather• Media reports (example: the “Clinton effect”)

Most suitable for modeling without data-specific information

Forming the Outcome Variable: Binning by Diagnosis Code

Rash Syndrome Grouping of Diagnosis Codes

Rash ICD-9-CM Code List

ICD9CM ICD9DESCR Consensus 050.0 SMALL POX, VARIOLA MAJOR 1 050.1 SMALL POX, ALASTRIM 1 050.2 SMALL POX, MODIFIED 1 050.9 SMALLPOX NOS 1 051.0 COWPOX 1 051.1 PSEUDOCOWPOX 1 052.7 VARICELLA COMPLICAT NEC 1 052.8 VARICELLA W/UNSPECIFIED C 1 052.9 VARICELLA NOS 1 057.8 EXANTHEMATA VIRAL OTHER S 1 057.9 EXANTHEM VIRAL, UNSPECIFI 1 695.0 ERYTHEMA TOXIC 1 695.1 ERYTHEMA MULTIFORME 1 695.2 ERYTHEMA NODOSUM 1 695.89 ERYTHEMATOUS CONDITIONS O 1 695.9 ERYTHEMATOUS CONDITION N 1 692.9 DERMATITIS UNSPECIFIED CA 2 782.1 RASH/OTHER NONSPEC SKIN E 2 026.0 SPIRILLARY FEVER 3 026.1 STREPTOBACILLARY FEVER 3 026.9 RAT-BITE FEVER UNSPECIFIED 3 051.2 DERMATITIS PUSTULAR, CONT 3 051.9 PARAVACCINIA NOS 3 053.20 HERPES ZOSTER DERMATITIS E 3

053.79 HERPES ZOSTER WITH OTHER SPECIF COMPLIC

3

053.8 H.Z. W/ UNSPEC. COMPLICATION 3 053.9 HERPES ZOSTER NOS W/O COM 3 054.0 ECZEMA HERPETICUM 3 054.79 HERPES SIMPLEX W/OTH.SPEC 3 054.8 HERPES SIMPLEX, W/UNS.COM 3 054.9 HERPES SIMPLEX NOS 3 055.79 MEASLES COMPLICATION NEC 3 055.8 MEASLES COMPLICATION NOS 3 055.9 MEASLES UNCOMPLICATED 3 056.79 RUBELLA COMPLICATION NEC 3 056.8 RUBELLA COMPLICATION NOS 3 056.9 RUBELLA UNCOMPLICATED 3 057.0 ERYTHEMIA INFECT.(5TH DIS 3 074.3 HAND/FOOT AND MOUTH DISEA 3 078.0 MOLLUSCUM CONTAGIOSUM 3 082.0 ROCKY MOUNTAIN SPOTTED FE 3 083.2 RICKETTSIALPOX 3 695.3 ROSACEA 3 695.4 LUPUS ERYTHEMATOSUS 3

www.bt.cdc.gov/surveillance/syndromedef/word/syndromedefinitions.doc

Chief Complaint Query

Simulated Data

Dynamic Detection

Simulated Data

Dynamic Detection

Example with Detection Statistic Plot

Threshold

Injected Cases Presumed

Attributable to Outbreak Event

Comparing Alerting AlgorithmsCriteria:

• Sensitivity– Probability of detecting an outbreak signal– Depends on effect of outbreak in data

• Specificity ( 1 – false alert rate )– Probability(no alert | no outbreak )– May be difficult to prove no outbreak exists

• Timeliness– Once the effects of an outbreak appear in

the data, how soon is an alert expected?

Modeling the Signalas Epicurve of Primary Cases

• Need “data epicurve”: time series of attributable counts above background

• Plausible to assume proportional to epidemic curve of infected

• Sartwell lognormal model gives idealized shape for a given disease type

Observed vs Modeled Incubation Period Distribution: Sverdlovsk 1979 Outbreak

0

2

4

6

8

10

12

0 10 20 30 40 50

Days after Exposure

Nu

mb

er o

f Cas

es

observed

modeled

Sartwell, PE. The distribution of incubation periods of infectious disease. Am J Hyg 1950; 51:310:318

Signal Modeling: Realizations of Smallpox Epicurve

“maximum likelihood” epicurve

Each symptomatic case a random draw

Assessing Algorithm Performance

Sensitivity/Specificity as a function of threshold: Receiver Operating Characteristic

(ROC)

Timeliness/Specificity as a function of threshold:Activity Monitor OperatingCharacteristic

(AMOC)

False Alert Rate (1 – specificity)

Detection Probability(sensitivity)

False Alert Rate (1 – specificity)

Timeliness Score (e.g. Mean or Median Time to Alert)

threshold

threshold

Summary processing: measure dependence of sensitivity or timeliness on false alert rate (ROC or AMOC curves or key sample values at practical rates)

Detection Performance Comparison

Fever_Labbaji, lognormal signal

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0 10 20 30 40 50 60 70 80 90

Background Recurrence (days)

De

tec

tio

n P

rob

ab

ility

EWMA

EARS C2

EARS C3 (CUSUM)

Quality Control Charts and Health Surveillance

Benneyan JC, Statistical Quality Control Methods in Infection Control and Hospital Epidemiology, Infection and Hospital Epidemiology, Vol. 19, (3)194-214

Part I: Introduction and Basic TheoryPart II: Chart use, statistical properties, and research issues

• 1998 Survey article gives 135 references• Many applications: monitoring surgical wound infections, treatment

effectiveness, general nosocomial infection rate, …

Monitoring process for “special causes” of variation• Organize data into fixed-size groups of observations• Look for out-of-control conditions by monitoring mean, standard deviation,

…• General 2-phase procedure:

Phase I: Determine mean , standard deviation of process from historical “in-control” data; control limits often set to 3

Phase II: Apply control limits prospectively to monitor process graphically

Adaptation of Traditional Process Control to Early Outbreak Detection

On adapting statistical quality control to biosurveillance:Woodall , W.H. (2000). “Controversies and Communications in

Statistical Process Control”, Journal of Quality Technology 32, pp. 341-378.

• “Researchers rarely…put their narrow contributions into the context of an overall SPC strategy. There is a role for theory, but theory is not the primary ingredient in most successful applications.”

Woodall , W.H. (2006, in press). “The Use of Control Charts in Health Care Monitoring and Public Health Surveillance”

• “In industrial quality control it has been beneficial to carefully distinguish between the Phase I analysis of historical data and the Phase II monitoring stage”

• “It is recommended that a clearer distinction be made in health-related SPC between Phase I and Phase II…”

Does infectious disease surveillance require an “ongoing Phase I” strategy to maintain robust performance?

Statistical Process Control in Advanced Disease Surveillance

Key application issues:

• Background data characteristics change over time – Hospital/clinic visits, consumer purchases not governed

by physical science, engineering– But monitoring requires robust performance: algorithms

must be adaptive

• Target signal: effect of infectious disease outbreak– Transient signal, not a mean shift– May be sudden or gradual

The Challenge of Data Modeling for Daily Health Surveillance

• Conventional scientific application of regression– Do covariates such as age, gender affect treatment?

Does treatment success of differ among sites if we control for covariates?

– Studies use static data sets with exploratory analysis

• In surveillance, we model to predict data levels in the absence of the signal of interest – Need reliable estimates of expected levels to recognize

abnormal levels– Data sets dynamic—covariate relationships change

The Challenge of Data Modeling for Daily Health Surveillance, cont’d

Modeling to generate expected data levels – Predictive accuracy matters, not just strength of

association or overall goodness-of-fit– For a gradual outbreak, recent data can “train” model

to predict abnormal levels

Alerting decisions based on model residualsResidual = observed value – modeled value

Conventional approach: – assume residuals fit a known distribution (normal,

Poisson,…)– hypothesis test for membership in that distribution

For surveillance, can also apply control-chart methods to residuals

Monitoring Data Series with Systematic Features

• Problem: How to account for short-term trends, cyclic data features in alerting decisions?

• Approaches– Data Modeling

• Regression: GLM, ARIMA, others & combinations

– Signal Processing• LMS filters and wavelets

– Exponential Smoothing: generalizes EWMA

Example: OTC Purchasing BehaviorInfluenced by Many Factors

Example: Tracking Daily Sales of Flu Remedies

Loglinear Regression

Log(Y) = 0 + 1-6d + 7t + 8-9h +10w + 11p + day of week(6 indicators)

harmonic(seasonal)

salespromotion(indicator)

lineartrend

weather(temp.)

deviation(Poisson

dist.)

daily countof anti-flu

sales

Modeling emergency department visit patterns for infectious disease complaints: results and application to disease surveillance

Judith C Brillman , Tom Burr , David Forslund , Edward Joyce , Rick Picard and Edith Umland

BMC Medical Informatics and Decision Making 2005, 5:4, pp 1-14http://www.biomedcentral.com/content/pdf/1472-6947-5-4.pdf

Modeling visit counts on day d:

Let S(d) = log ( visits(day d) + 1 ), the “started log”

S(d) = [Σi ci × Ii(d)] + [c8 + c9 × d] + [c10 × cos(kd) + c11 × sin(kd)],

k = 2π / 365.25 c1-c7 day-of-week effects c9 long-term trend c10-c11 seasonal harmonic terms

Training period:3036 days ~ 8.33 years Test period: 1 year

Recent Surveillance MethodBased on Loglinear Regression

Brillman et. al. Figure 1

EWMA Monitoring

• Exponential Weighted Moving Average

• Average with most weight on recent Xk:

Sk = S k-1 + (1-)Xk,

where 0 < • Test statistic:

Sk compared to expectation from sliding baseline

Basic idea: monitor

(Sk – k) / k

Exponential Weighted Moving Average

0

10

20

30

40

50

60

02/25/94 03/02/94 03/07/94 03/12/94 03/17/94 03/22/94 03/27/94 04/01/94

Daily Count

Smoothed

• Added sensitivity for gradual events• Larger means less smoothing

EWMA Concept & Smoothing Constant

Brown, R.G. and Meyer, R.F. (1961), "The Fundamental Theorem of Exponential Smoothing," Operations Research, 9, 673-685.

• Exponential smoothing represents “an elementary model of how a person learns”:

xk = xk-1 + xk - xk-1)where 0 < • For the smoothed value Sk,

Sk = S k-1 + (1-)Xk ,The variance of Sk is SX

• So a smaller is preferred because it gives a more stable Sk; values between 0.1 and 0.3 often used

• But Chatfield: changes in global behavior will result in a larger optimal

Generalized Exponential Smoothing

Forecast Function:

)) ((ˆ|

cbkmyksnnnnkn

where: mj = level at time j, bj = trend at time j, cj = periodic multiplier at time j s = periodic interval k = number of steps ahead

and mj, bj, cj are updated by exponential smoothing

http://www.statistics.gov.uk/iosmethodology/downloads/Annex_B_The_Holt-Winters_forecasting_method.pdf

Holt-Winters Method: modeling level, trend, and seasonality

Holt-Winters Updating Equations

Updating Equations, multiplicative method:

Level at time t:

Slope at time t:

Periodic multiplierat time t:

10,)1( stt

tt c

m

yc

10,)1()( 11 tttt bmmb

10,)()1( 11

ttst

tt bm

c

ym

And choice of initial values m0, b0, c0,…cs-1 should be calculated from available data

Forecasting Local Linearity:Automatic vs Nonautomatic Methods

Chatfield, C. (1978), "The Holt-Winters Forecasting Procedure," Applied Statistics, 27, 264-279.

Chatfield, C.and Yar, M. (1988), "Holt-Winters Forecasting: Some Practical Issues, " The Statistician, 37, 129-140.

• “Modern thinking favors local linearity rather than global linear regression in time…”

• “Local linearity is also implicit in ARIMA modelling…”– Simple EWMA ~ ARIMA(0,1,1)– EWMA + trend ~ ARIMA(0,2,2)– Multiplicative Holt-Winters has no ARIMA equivalent

• “Practical considerations rule out [Box-Jenkins] if there are insufficient observations or …expertise available”

– “Box-Jenkins… requires the user to identify an appropriate… [ARIMA] model”

For “fair” comparison of H-W to B-J, have both automatic or nonautomatic.Assertion: The simplicity of H-W permits easier classification, requiring less

historic data. Can an automatic B-J give robust forecasting over a range of input series

types?

Regression vs Holt-Winters

0 50 100 150 200 250 300 3500

100

200

300

400

500

600

Days

Cou

nts

Results for Data Set: 1; with DOW and Seasonal VariationHW-RMSE = [57.401] RegressedRMSE = [61.1454]

Raw Data

Holt Winters

Regression

0 50 100 150 200 250 300 350-400

-200

0

200

400Residuals

Days

Cou

nts

HoltWinters

Regression

Ongoing study with Galit Shmueli, U. of MD Sean Murphy, JHU/APL

30 time series, 700 days’ data

5 cities3 data types2 syndromes

Respiratory: seasonal & day-of-week behaviorGastrointestinal:

day-of-week effects

Temporal Aggregation for Adaptive Alerting

baseline interval

Used to get some estimate of normal data behavior• Mean, variance• Regression coefficients• Expected covariate distrib. -- spatial -- age category -- % of claims/syndrome

guardband

Avoids contamination of baseline with outbreak signal

Data stream(s) to monitor in time:

• Counts to be tested for anomaly

• Nominally 1 day• Longer to reduce

noise, test for epicurve shape

• Will shorten as data acquisition improves

test interval

Candidate Methods

1. Global loglinear regression of Brillman et. al.2. Holt-Winters exponential smoothing

fixed sets of smoothing parameters for data:with both day-of-week & seasonal behaviorwith only day-of-week behavior

3. Adaptive RegressionLog(Y) = 0 + 1-6d + 7t + 8hol + 9posthol +

56-day baseline, 2-day guardband1-6 = day-of-week indicator coefficient 7 = centered ramp coefficient8 = coefficient for holiday indicator9 = coefficient for post-holiday indicator

1-day ahead and 7-day-ahead predictions

Respiratory Visit Count Data--- Data--- Holt-Winters--- Regression--- Adaptive Regr.

All series display this autocorrelation; good test for published regression model

GI Visit Count Data--- Data--- Holt-Winters--- Regression--- Adaptive Regr.

Stratified Residual Comparisons

--- Data--- Holt-Winters--- Regression--- Adaptive Regr.

Mean Residual Comparison

• When mean residuals favor regression, difference is small, and this difference results from largest residuals

• If the holiday terms in adaptive regression are removed, H-W means uniformly smaller

Median Residual Comparison

Residual Autocorrelation Comparison

--- Data--- Holt-Winters--- Regression--- Adaptive Regr.

Residual Autocorrelation Comparison 1-Day Ahead Predictions

Residual Autocorrelation Comparison 7-Day Ahead Predictions

Summary

• Data-adaptive methods are required for robust prospective surveillance

• Appropriate algorithm selection requires an automated data classification methodology, often with little data history

• Statistical expertise is required to manage practical issues to maintain required detection performance as datasets evolve:– stationarity (causes rooted in population behavior,

evolving informatics, others)– late reporting– data dropouts

Research Directions

• Classification of time series for automatic forecasting– Easier for Holt-Winters than for Box-Jenkins?– Determining reliable discriminants:

• Autocorrelation coefficients• Simple means/medians• Goodness-of-fit measures

– How little startup data history required?• Most effective alerting algorithm using residuals,

given signal of interest– Apply control chart to residuals?– Need to detect both sudden, gradual signals– Detection performance constraints:

• Minimum detection sensitivity• Maximum background alert rate