Anomaly Detection in the WIPER System using A Markov Modulated Poisson Distribution

Ping Yan Tim Schoenharl

Alec Pawling Greg Madey

Outline

n  Background n  WIPER n  MMPP framework n  Application n  Experimental Results n  Conclusions and Future work

Background

n  Time Series n  A sequence of observations measured

on a continuous time period at time intervals

n  Example: economy (Stock, financial), weather, medical etc…

n  Characteristics n  Data are not independent n  Displays underlying trends

WIPER System

n  Wireless Phone-based Emergency Response System

n  Functions n  Detect possible emergencies n  Improve situational awareness

n  Cell phone call activities reflect human behavior

Data Characteristics

n  Two cities: n  Small city A:

n  Population – 20,000 Towers – 4

n  Large city B: n  Population – 200,000 Towers – 31

n  Time Period n  Jan. 15 – Feb. 12, 2006

Tower Activity

0 5 10 15 20 250

100

200

300

400

500

600

700

800

Hour

Cal

l Act

iviti

es

Small City (4 Towers)

0 5 10 15 20 250

100

200

300

400

500

600

700

Hour

Cal

l Act

iviti

esTower Activity

Large City (31 Towers)

0 5 10 15 20 250

200

400

600

800

1000

1200

1400

1600

1800

Every Hour

Cal

l Act

iviti

es

Sun1Mon1Tue1Wed1Thu1Fri1Sat1Sun2Mon2Tue2Wed2Thu2Fri2Sat2Sun3

15-Day Time Period Data

Small City

0 5 10 15 20 250

5000

10000

15000

Every Hour

Cal

l Act

iviti

es

Sun1Mon1Tue1Wed1Thu1Fri1Sat1Sun2Mon2Tue2Wed2Thu2Fri2Sat2Sun3

15-Day Time Period Data

Large City

Observations

n  Overall call activity of a city are more uniform than a single tower

n  Call activity for each day displays similar trend

n  Call activity for each day of the week shares similar behavior

MMPP Modeling

: Observed Data

: Unobserved Data with normal behavior

: Unobserved Data with abnormal behavior

Both and can be modulated as a Poisson Process.

Modeling Normal Data

n  Poisson distribution

n  Rate Parameter: a function of time

~

Adding Day/hour effects

Associated with Monday, Tuesday … Sunday

: Average rate of the Poisson process over one week

: Time interval, such as minute, half hour, hour etc

Requirements:

: Day effect, indicates the changes over the day of the week

: Time of day effect, indicates the changes over the time period j on a given day of i

Day Effect

0 20 40 60 80 100 120 140 160 1800

500

1000

1500

2000

Time Interval (7 Days x 24 Hours)

Cal

l Act

iviti

es

Call activitiesDay effectOverall Average

Time of Day Effect

0 5 10 15 20 25 30 35 40 45 500

200

400

600

800

1000

Time Interval (Half hour)

Cal

l Act

iviti

es

Call activitiesTime of Day EffectOverall Average Day Effect

Prior Distributions for Parameters

Modeling Anomalous Data n  is also a Poisson process with

rate

n  Markov process A(t) is used to determine the existence of anomalous events at time t

Continued

n  Transition probabilities matrix

MMPP ~ HMM

n  Typical HMM (Hidden Markov

Model)

n  MMPP (Markov Modulated Poisson Process)

Apply MCMC

n  Forward Recursion n  Calculate conditional distribution

of P( A(t) | N(t) )

n  Backward Recursion n  Draw sample of and

n  Draw Transition Matrix from Complete Data

Anomaly Detection

n  Posterior probability of A(t) at each time t is an indicator of anomalies

n  Apply MCMC algorithm: n  50 iterations

Results

Continued

Conclusions

n  Cell phone data reflects human activities on hourly, daily scale

n  MMPP provides a method of

modeling call activity, and detecting anomalous events

Future Work

n  Apply on longer time period, and investigate monthly and seasonal behavior

n  Implement MMPP model as part of real time system on streaming data

n  Incorporate into WIPER system