Upload
ido-shilon
View
271
Download
3
Embed Size (px)
Citation preview
Online LearningThe Future of Audience Segmentation is Here
Kevin Lyons + Yakir Buskilla
Models that build profitable marketing audiences at scale...
Finding more of your best customers: High-income business professional
The Modeling Process, simplified
2012 2015
30 - 40 modelslevering billions of events
Creating 100 million + scores
over 1000 models‘leveraging’ trillions of events
Creating 150 billion+ scores / day
The Challenge
In other words, we simply need ….
A system creates as many models as we want, when
we want them, that dynamically adapts in real-time
to changing conditions
○ Automatically creates, validates, ships, and
monitors models, with a capacity that scales
to 10s of thousands of models
The Opportunity
What we really need:
Online models evolve & adapt over time, in
reaction to a changing environment with each
and every event
Given a complete data set, a batch
model is created in entirety all at once
Introducing Online Learning
Batch Online Learning
Creation Evolution
large-scale data storage
large-scaledata schelping
painful data aggregation
lots of manual everything
Harder to build models, but easier to evaluate
limited data storage, mostly for monitoring
event-leveldata streams
light data aggregation
lots of automatic everything
Easier to build, but harder to evaluate (& support)
Batch Models (Offline) vs. Online Learning
Online LearningBatch Models (Offline)
● Outperformed both L2 and Elastic Net
● Leverages small (‘micro’) batches
● Validates and monitors models in real time
● Alerts team when models are not behaving
Some Techno Mumbo Jumbo
Stochastic gradient descent with L1 regularization
eXelate.com @eXelate
Technical Solutions
How do we do it?
eXpresso Serving Cluster
10B events/day
260 nodes across4 data centers
eXtream Modeling Cluster
160B models/day
85 nodes across4 data centers
JGroups
DistributedMessaging
Serving Layer
Online LearningBatch Models (Offline)
Batch
Predefined ratio
Predefined feature selection
One time Validation
Streaming
Downsampling
Automated feature selection
Ongoing data cleaning
Ongoing validation
The Online Learning Challenge
● All necessary data already exists in eXtream
● The cluster’s processing resources can be better utilized
● eXtream addresses most performance / scalability requirements
● Scoring mechanism already exists
eXtream as a Framework for Online Learning
Why it works...
Online Learning Flow
● Labeling Mechanism - customer defined target audience
Events Classification
● Downsampling mechanism● Burst tolerance● Duplicate entries
Dataset Preparation
● Blacklist● Whitelist● Automatic Tuning
Features Selection
● Sliding window of recent events● 60/40 not-converted/converted ratio● Various accuracy metrics (lift, precision, recall, confusion matrix)● Decide if the model is ready for making predictions
Model Validation
● Two phases (Scoring, Re-code)● Scale vs Accuracy tradeoff
Predictions Mechanism
Scalability / Performance
Thousands of
Concurrent Models: High Throughput:
billions of training events per daytraining, validation, scoring
Why do we need it?● Store the models in one common place
● Persistency
● Built-in replication
● Aerospike has built in limitation for object size - 1MB
○ Developed sharding mechanism for storing models on Aerospike
Scalability / Performance
Why do we need it?
Large object issue on Aerospike
The solution is Aerospike fast built-in replication
Cross Data Center Learning
● Low Volume Models
● Traffic Redirection
Monitoring- Why do we need it?
thousands of models
automatically created by users
some models won’t converge
Monitoring- Real Time
Monitoring- Aggregation
Monitoring- DS Bot
eXelate.com @eXelate
Case study
Working in action
● The ideal candidate for digital media expands and even subtly shifts in real time● Real-time modeling tracks and reacts to these changes as they happen, with 2x CPA
improvement over a batch model
The Times, They Are A-Changin’
Market: Downgrading a country’s credit ratings
● Holiday shopping is very different from the rest of the year, particularly Cyber Monday ● AM changes in Eastern US are applied to the Pacific coast before the madness begins
Audiences: Cyber Monday frenzies
● … after the campaign starts, effecting the ideal audience● No need to panic; modeled audience automagically adjust
Product: A product offering is revised
Scores of self-maintaining models that constantly adapt to our ever changing conditions
Happiness Renewed...