Upload
louis-dorard
View
1.578
Download
0
Embed Size (px)
Citation preview
Pragmatic Machine Learning
@louisdorard #MLSpain - 18 Jan 2016
I’M LAZY
“Programming is for lazy people who want to automate things
— AI is for lazier people who want to automate programming”
• Consider manual classification task
• Automate with ML model?
• Build PoC
• Deploy in production
• Maintain
• Monitor performance
• Update with new data6
The Laz y MLer
Phrase problem as ML task
Engineer features
Prepare data (csv)
Learn model
Make predictions
Deploy model & integrate
Evaluate model
Measure impact
• Florian Douetteau at PAPIs Connect in May 2015)
• Top companies invested more than 5M$ in their ML production platform (Facebook, Amazon, LinkedIn, Spotify…)
7
Cost of ML projec ts
• Real-world ML is/was complicated and costly (especially at web scale)
• Do I really need ML?
• How about Human API? (e.g. Amazon Mechanical Turk)
• → Back to Square 1 (but someone else’s problem!)
• → Baseline! (performance, time, cost)8
The Laz y MLer
Performance evaluation
How do you evaluate the performance of an ML system?
Accuracy
Latency
Throughput
11
Per formance measures
• Go beyond accuracy… example: recommendations
• Get clicks!
• → Simulate how many you’d get with your model
• → Need to learn accurately what people like — not what they dislike
• Better decisions with ML
• Revenue increase (A/B test)
• Decisions can have a cost (e.g. give special offer/pricing to customer)… ROI?13
Domain-specif ic evaluation
Decisions from predictions
1. Descriptive
2. Predictive
3. Prescriptive
15
Types of analyt ics
1. Show churn rate against time
2. Predict which customers will churn next
3. Suggest what to do about each customer (e.g. propose to switch plan, send promotional offer, etc.)
17
Churn analysis
• Who: SaaS company selling monthly subscription
• Question asked: “Is this customer going to leave within 1 month?”
• Input: customer
• Output: no-churn or churn
• Data collection: history up until 1 month ago
18
Churn predic t ion
• #TP (we predict customer churns and he does)
• #FP (we predict customer churns but he doesn’t)
• #FN (we predict customer doesn’t churn but he does)
19
Churn predic t ion accurac y
Assume we know who’s going to churn. What do we do?
• Contact them (in which order?)
• Switch to different plan
• Give special offer
• No action?
20
Churn prevention
“3. Suggest what to do about each customer” → prioritised list of actions, based on…
• Customer representation + context (e.g. competition)
• Churn prediction (& action prediction?)
• Uncertainty in predictions
• Revenue brought by customer & cost of action
• Constraints on frequency of solicitations21
Churn prevention
• Taking action for each TP (and FP) has a cost
• For each TP we “gain”: (success rate of action) * (revenue /cust. /month)
• Imagine…
• perfect predictions
• revenue /cust. /month = 10€
• success rate of action = 20%
• cost of action = 2€
• Which ROI?22
Churn prevention ROI
• We predicted customer would churn but they didn’t…
• That’s actually good! Prevention worked!
• Need to store which actions were taken
• Is ML really helping?
• Compare to baseline,e.g. if no usage for more than 15 days then predict churn
• Is fancy model really improving bottom line?23
Churn prevention evaluation
1. Show past demand against calendar
2. Predict demand for [product] at [store] in next 2 days
3. Suggest how much to ship
• Trade-off: cost of storage vs risk of lost sales
• Constraints on order size, truck volume, capacity of people putting stuff into shelves
24
Replenishment
• Context
• Predictions
• Uncertainty in predictions
• Constraints
• Costs / benefits
• Competing objectives (⇒ trade-offs to make)
• Business rules25
Decis ions are based on…
APIs are key
Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions, uncertainties, constraints, objectives, costs
• Apply given decision27
S eparation of concerns
Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions, uncertainties, constraints, objectives, costs
• Apply given decision28
Operations Research component
Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions, uncertainties, constraints, objectives, costs
• Apply given decision29
M achine Learning components
Software components for automated decisions:
• Create training dataset from historical data (merge sources, aggregate…)
• Provide predictive model from given training set (i.e. learn)
• Provide prediction against model for given context
• Provide optimal decision from given contextual data, predictions, uncertainties, constraints, objectives, costs
• Apply given decision30
Predic t ive APIs
The two methods of predictive APIs:
• model = create_model(‘training.csv’)
• predicted_output = create_prediction(model, new_input)
31
Predic t ive APIs
Amazon ML
BigML
Google Prediction
PredicSis
… or your own company!
32
Providers of REST http Predic t ive APIs
?
Experiment on “ScienceCluster”• Distributed jobs • Collaborative workspace • Serialize chosen model
Deploy model as API on “ScienceOps”• Load balancing • Auto scaling • Monitoring (API calls, accuracy)
• “Open source prediction server” in Scala
• Based on Spark, MLlib, Spray
• DASE framework: Data preparation, Algorithm, Serving, Evaluation
• Amazon CloudFormation template → cluster
• Manual up/down scaling40
→ PAPI+
→ PAPI+
Interesting research problems
45
Concurrenc y for high-throughput ML APIs
Brian Gawalt (Senior Data Scientist at Upwork) Talk at PAPIs ’15
upwork.com use case:
• predict freelancer availability
• huge web platform (millions of users)→ need very high throughput and low latency
• things change quickly → need freshest data & predictions
46
Concurrenc y for high-throughput ML APIs
• event: invitation sent to freelancer
• steps to prediction:
• gather raw data from all sources
• featurize event
• make prediction
Concurrenc y for high-throughput ML APIs
• An actor…
• gets & sends messages
• makes computations
• Actors we need:
• “Historians”: one per data source
• “Featurizer”
• “Scorer”48
Concurrenc y with Ac tor framework
49
Concurrenc y for high-throughput ML APIs
before
50
Concurrenc y for high-throughput ML APIs
after
• Python defacto standard: scikit-learn
• “Sparkit-learn aims to provide scikit-learn functionality and API on PySpark. The main goal of the library is to create an API that stays close to sklearn’s.”
• REST standard: PSI (Protocols & Structures for Inference)
• Pretty similar to BigML API!
• Implementation for scikit available
• Easier benchmarking! Ensembles!51
API standards?
• “AzureML: Anatomy of a machine learning service”
• “Deploying high throughput predictive models with the actor framework”
• “Protocols and Structures for Inference: A RESTful API for Machine Learning”
• Coming soon… JMLR W&CP Volume 50
• Get updates: @papisdotio or papis.io/updates52
PAPIs ’15 Proceedings
53
Simple MLaaS comparison
Amazon Google PredicSis BigML
Accuracy 0,862 0,743 0,858 0,790
Training 135s 76s 17s 5s
Test time 188s 369s 5s 1s
louisdorard.com/blog/machine-learning-apis-comparison
• With SKLL (SciKit Learn Laboratory)
• Wrap each service in a scikit estimator
• Specify evaluations to perform in a config file (datasets, metrics, eval procedure)
• Need to also measure time…
• See papiseval on Github54
Automated B enchmark?
• Return of the Lazy MLer!
• Model selection
• Find optimal values for n (hyper-)parameters → optimisation problem (function in n dimensions)
• Search space of parameters, efficiently → explore vs exploit
• Bayesian optimization?55
AutoML
56
Bayesian Optimization in 1 dimension
From CODE517E
57
Bayesian Optimization in 1 dimension
From CODE517E
• Building ensembles
• Decide to continue training existing model, or to train new one
• Explore vs exploit again!
• Reward is accuracy. Let’s estimate reward for all options.
• Choose option with highest expected reward + uncertainty? (i.e. upper confidence bound)
• Limited computational budget… 58
AutoML
• Zoubin Gharahmani & James Lloyd @ Uni Cambridge
• Gaussian Processes: find (mixture of ) kernel(s) that maximises data likelihood
• Also Bayesian!
59
Automatic Stat ist ic ian
• Spearmint: “Bayesian optimization” for tuning parameters → Whetlab → Twitter
• Auto-sklearn: “automated machine learning toolkit and drop-in replacement for a scikit-learn estimator”
• See automl.org and challenge
60
Open S ource AutoML l ibrar ies
61
S cik it
from sklearn import svmmodel = svm.SVC(gamma=0.001, C=100.)
from sklearn import datasetsdigits = datasets.load_digits() model.fit(digits.data[:-1], digits.target[:-1])
model.predict(digits.data[-1])
62
S cik it
from sklearn import svmmodel = svm.SVC(gamma=0.001, C=100.)
from sklearn import datasetsdigits = datasets.load_digits() model.fit(digits.data[:-1], digits.target[:-1])
model.predict(digits.data[-1])
63
AutoML S cik it
import autosklearnmodel = autosklearn.AutoSklearnClassifier()
from sklearn import datasetsdigits = datasets.load_digits() model.fit(digits.data[:-1], digits.target[:-1])
model.predict(digits.data[-1])
• Before learning:
• Automatic feature extraction from text?
• After learning:
• Monitor new predictions and automatically retrain models when necessary?
• See panel discussion at PAPIs ‘1564
M ore automation ideas…
• Same as Azure ML?
• Scaling up? down?
65
Open S ource Auto S cal ing?
Tech talks:
• Intro to Spark
• Using ML to build an autonomous drone
• Demystifying Deep Learning (speaker needed!)
• Distributed Deep Learning with Spark on AWS
67
PAPIs Connec t (14-15 M arch, Valencia)
Topics:
• Managing technology
• FinTech
• Enterprise, Retail, Operations
• AI for Society (Nuria Oliver, Scientific Director at Telefonica R&D)
• Future of AI (Ramon Lopez de Mantaras, Director AI Research at Spanish Research Council)
68
PAPIs Connec t (14-15 M arch, Valencia)
• Dev? Bring your manager!
• Manager? Bring your devs!
• Discount code: MLSVLC20
• papis.io/connect
69
PAPIs Connec t (14-15 M arch, Valencia)