Five key learnings from building a large scale anomaly detection system

1Page Numberwww.anodot.com |Confidential

Eliminating Insight Latencies…One Anomaly at a Time

Ira Cohen, Chief Data Scientistira@anodot.com

What is the problem?

Business Insight Latency

Delayed business insights cost companies millions of dollars

Business insights at any granularity

Publishers Ad exchanges DevicesConversion Ad types X XX

User types Countries RegionPurchases Item categories X XX

OS Browsers Software verPage views Pages X XX

Sheet type Locations CustomerPrint errors Printer type X XX

Measuring KPIs with many dimensions

Millions of time series metrics to track

How to Track the Millions and Get the Insights?

Automated Anomaly Detection

Aggregate, Detect and Group

Realtime Anomaly Detection: Design Principals

• Collect metrics from ANY source Universal• Realtime tracking of millions of metricsScale• Unique IP for anomaly detection that eliminates FP Accurate• Multivariate anomalies for swift problem isolation Concise

Automatic Anomaly Detection in Five Steps

Normal behavior learning

Abnormal behavior learning

Behavioral Topology Learning

Detection, scoring and grouping anomalies

Feedback based learning

Normal behavior learning

- Adaptive- Detects and learns seasonal patterns + trends- Auto-configuration- Model selection algorithm

P(Anomaly | Duration, Deviation)

Bayesian model combining statistics of:- Abnormal durations- Abnormal intensity- Abnormal pattern

Abnormal behavior learning

- Normal behavior relationship learning- Abnormal behavior relationship learning- Meta-data based relationship learning

Behavioral Topology Learning

Detection, scoring and grouping anomalies

Feedback based learning

Anodot - Summary

• Founded June 2014• Beta since Jan 2015, GA in Sep• Over 20 million metrics tracked• Over 12,000 real-time insights provided to over 200 users • To try us out, contact: info@anodot.com• Follow us on twitter: @TeamAnodot

THANK YOUinfo@anodot.com@TeamAnodot

Five key learnings from building a large scale anomaly detection system

Technology

Anomaly detection- Credit Card Fraud Detection

Intrusion & Anomaly Detection & Preventiongauss.ececs.uc.edu/Courses/c6056/pdf/intrusion.pdfIntrusion & Anomaly Detection & Prevention Intrusion Detection: Monitor events, analyze

Anomaly Detection and Attribution Using Bayesian … · UNCLASSIFIED DSTO{TR{2975 Anomaly Detection and Attribution Using Bayesian Networks Executive Summary Anomaly detection techniques

Anomaly Detection Techniques

Online Nonparametric Anomaly Detection in High-Dimensional ... · 13/2/2018 · Online Nonparametric Anomaly Detection in High-Dimensional Datasets Online Nonparametric Anomaly Detection

Anomaly Detection using Outlier Detection Schemes

From Anomaly Detection to Rumour Detection using Data ... · Local Anomaly Detection (§4). We propose a non-parametric method for anomaly detection at the level of individual entities,

Anomaly Detection Survey

Anomaly Detection Systems

A Survey on Anomaly detection in Evolving Data · Anomaly detection has received con-1In this paper, we use the terms outlier detection and anomaly detection interchangeably siderable

Probabilistic Anomaly Detection in Dynamic Systemspapers.nips.cc/paper/805-probabilistic-anomaly-detection-in... · Probabilistic Anomaly Detection in Dynamic Systems 827 model (HMM)

Introduction to Anomaly Detection - uwyo.educlan/teach/rampe18_anomaly.pdfWhat is anomaly detection? “Anomaly detection refers to the problem of finding patterns in data that do

Anomaly Detection Tutorial

Anomaly Detection Petty

Unsupervised Anomaly Detection in Large Datacenters › ~mgabel › papers › MSc Thesis - Final.pdf · 2014-06-14 · Unsupervised Anomaly Detection ... Unsupervised Anomaly Detection

Chap10 Anomaly Detection

Science of Anomaly detection - v4 Updated for HTM for IT · The#Science#of#Anomaly#Detection!! ! Page 3! Anomaly!Detection!inStreamingData" Early anomaly detection in streaming data

Adaptive anomaly detection with evolving connectionist systemstechlab.bu.edu/files/resources/articles_tt/Adaptive anomaly... · Adaptive anomaly detection with evolving connectionist

Taxonomy of Anomaly Based Intrusion Detection System… · Figure 1: Taxonomy of Anomaly based Intrusion Detection System. III. STATISTICAL ANOMALY BASED INTRUSION DETECTION SYSTEM

Anomaly Detection in Predictive Maintenance - KNIME · 1 Anomaly Detection in Predictive Maintenance Time Alignment and Visualization for Anomaly Detection Rosaria Silipo Rosaria.Silipo@knime.com