Five Things I Learned While Building Anomaly Detection Tools

(Or: 5 things that bit me in the …)

Toufic Boubez, Ph.D.

Founder, CTO

Metafor Software

toufic@metaforsoftware.com

2

Preamble

• IANA Data Scientist! I’m just an engineer that needed to get stuff done!

• I learned (!) many more things, but cannnot be mentioned!– Because lawyers – But ask me later

• I usually beat up on parametric, Gaussian, supervised techniques– This talk is not an exception,– But more of a “lessons learned” message

• Note: all data real• Note: no y-axis labels on charts – on purpose!!• Note to self: remember to SLOW DOWN!• Note to self: mention the cats!! Everybody loves cats!!

3

• Co-Founder/CTO Metafor Software• Co-Founder/CTO Layer 7 Technologies

– Acquired by Computer Associates in 2013– I escaped

• CTO Saffron Technology• IBM Chief Architect for SOA• Co-Author, Co-Editor: WS-Trust, WS-

SecureConversation, WS-Federation, WS-Policy• Building large scale software systems for >20

years (I’m older than I look, I know!)

Toufic intro – who I am

4

Why Anomaly Detection?

• Watching screens on the “Wall of Charts” cannot scale!– Leads to alert fatigue

• Need to automate detection of anomalous behaviors

• Anomaly detection is the search for items or events which do not conform to an expected pattern. [Chandola, V.; Banerjee, A.; Kumar, V. (2009). "Anomaly detection: A survey". ACM Computing Surveys 41 (3): 1]

Thing 1:Your data is NOT Gaussian

1

6

Gaussian or Normal distribution

• Bell-shaped distribution

– Has a mean and a standard deviation

7

This is Normally distributed data

8

Quick check: Histogram

9

Normal distributions are really useful

• I can make powerful predictions because of the statistical properties of the data

• I can easily compare different metrics since they have similar statistical properties

• There is a HUGE body of statistical work on parametric techniques for normally distributed data

- Confidential - 10

Normal distributions

• Most naturally occurring processes

• Population height, IQ distributions (present company excepted of course)

• Widget sizes, weights in manufacturing

• …

Not

• Your metrics!

Normally distributed vs Not

11

Why is that important?

• Most analytics tools are based on two assumptions:

1. Parametric techniques: Data is normally distributed with a useful and usable mean and standard deviation

2. Supervised Learning techniques: Data is probabilistically “stationary”

12

Example: Three-Sigma Rule

• Three-sigma rule

– ~68% of the values lie within 1 std deviation of the mean

– ~95% of the values lie within 2 std deviations

– 99.73% of the values lie within 3 std deviations: anything else is considered an outlier

13

Aaahhhh

• The mysterious red lines explained

mean

3s

3s

14

Doesn’t work because THIS

15

Histogram – probability distribution

16

3-sigma rule alerts

17

Holt-Winters predictions

18

Or worse, THIS!

19

Histogram – probability distribution

20

3-sigma rule alerts

Thing 2:Yesterday’s anomaly is today’s normal

2

22

Why is that important?

• Most analytics tools are based on two assumptions:

1. Parametric techniques: Data is normally distributed with a useful and usable mean and standard deviation

2. Supervised Learning techniques: Data is probabilistically “stationary”

23

Remember this data?

24

No matter where you look

25

Its characteristics are stationary

26

Meanwhile, in our real world

• Stationarity is not a realistic assumption in the large complex systems with which we’re dealing

• “Concept Drift” is very common

– http://en.wikipedia.org/wiki/Concept_drift

“ … the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.”

27

Meanwhile, in our real world

• Stationarity is not a realistic assumption in the large complex systems with which we’re dealing

• “Concept Drift” is very common

– http://en.wikipedia.org/wiki/Concept_drift

“ … the statistical properties of the target variable, which the model is trying to predict, change over time in unforeseen ways. This causes problems because the predictions become less accurate as time passes.”

28

Supervised learning

• In ML, Supervised Learning is the general set of techniques for inferring a model from a set of observations:– Observations in a Training Set are labelled with the

desired outcomes (e.g. “normal vs. anomalous”, “normal vs. fraudulent”, “red/green/yellow”, etc)

– As observations are fed into the learning system, it learns to differentiate by inferring a model based on these labels

– Once sufficiently “trained”, the system is used in production on “real” unlabelled data and can label the new data based on the inferred model

29

What happens when something changes in your fundamentals?

30

This is your new normal: all red all the time

31

Mean Shift and Breakout Detection

• https://blog.twitter.com/2014/breakout-detection-in-the-wild

Thing 3:Saying Kolmogorov-Smirnov is a great way to

impress everyone

3

33

Why is that important?

• Seriously!?

• Ok, actually non-parametric techniques that make no assumptions about normality or any other probability distribution are crucial in your effort to understand what’s going on in your systems

34

The Kolmogorov-Smirnov test

• Non-parametric test– Compare two probability

distributions– Makes no assumptions (e.g.

Gaussian) about the distributions of the samples

– Measures maximum distance between cumulative distributions

– Can be used to compare periodic/seasonal metric periods (e.g. day-to-day or week-to-week)

http://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test

35

KS with windowing

36

Data from similar windows

37

Cumulative distribution for those windows

38

Data from dissimilar windows

39

Cumulative distribution for those windows

40

Sliding window of KS scores

41

KS anomaly results

Thing 4:Take Scope and Context into account!

4

43

Some data – is that normal?

44

Wider scope

45

Is this an anomlay?

46

Even wider scope

47

Is every weekend an anomaly?

48

Would this be more accurate?

49

Use domain knowledge!

• Domain knowledge is NOT a bad thing!– There is no algorithm that will work on everything

– Know your data and it general patterns• Periodicity/Seasonality

• Known events (maintenance, backups, etc)

– Apply the appropriate algorithms, taking into account enough scope for any inherent periodicity to appear

– Customize your alerts to take into accounts known events

Thing 5:No data != No information

51

Why is that important?

• Some data channels are inherently non-chatty:

– We don’t have the luxury of always generating non-zero values

– There is a lot of useful information in the fact that nothing is happening on a particular channel

• A lot of time series analytics techniques fail on time series with too few values (e.g. RF, adjusted box plot, etc)

52

Communication channel

53

Box plot results

54

55

Simple lookup table with priors

56

Don’t be an analytics snob

• Sparse data is VERY hard to analyze using typical analytics techniques

• Sparse data conveys VERY important information

• Sometimes the simplest rules, thresholds, lookup tables will work

57

Recap

1. Your data is NOT Gaussian

2. Yesterday’s anomaly is today’s normal

3. Kolmogorov-Smirnov is really cool

4. Scope and Context are important

5. No data != No information

58

Questions?

• Shout out to the Metafor Data Science team!

– Fred Zhang

– Iman Makaremi