VSSML16 LR2. Summary Day 2

Class summary

BigML, Inc 2

Day 2 – Morning sessions

BigML, Inc 3

Basic transformations

ExpectationsPoul Petersen

Reality

$

MLready data needs work!!!Any data is always MLready

What does MLready mean?● Machine Learning algorithms consume instances of the question that you want to

model. Each row must describe one of the instances and each column a property of the instance

● Fields can be:– already present in your data– derived from your data– generated using other fields

BigML, Inc 4


● Select the right model for the problem you want to solve: Classification, regression, cluster analysis, anomaly detection, association discovery

● Perform cleansing, denormalizing, aggregating, pivoting, and other data wrangling tasks to generate a collection of instances relevant to the problem at hand. Finally use a very common format as output format: CSV

● Choose the right format to store each type of feature into a field● Feature engineering: Using domain knowledge and Machine

Learning expertise, generate explicit features that help to better represent the instances (Flatline)

ML-ready steps

BigML, Inc 5


Cleansing: Homogenize missing values and different types in the same feature, fix input errors, correct semantic issues, etc.

Denormalizing: Data is usually normalized in relational databases, MLReady datasets need the information denormalized in a single file/dataset.

Aggregation: When data is stored as individual transactions, as in log files, an aggregation to get the entity might be needed

Pivoting: Different values of a feature are pivoted to new columns in the result dataset

Regular time windows: Create new features using values over different periods of time

Preprocessing data

BigML, Inc 6


For numeric features: – Discretization: percentiles, within percentiles, groups– Replacement– Normalization– Exponentiation– Shocks (speed of change compared to stdev)

For text features:– Mispellings– Length– Number of subordinate sentences– Language– Levenshtein distance

Stacking

Compute a field using nonlinear combinations of other fields

Feature engineering

BigML, Inc 7


● Define a clear idea of the goal.● Understand what ML tasks will achieve the goal.● Understand the data structure to perform those ML tasks.● Find out what kind of data you have and make it MLReady

– where is it, how is it stored?– what are the features?– can you access it programmatically?

● Feature Engineering: transform the data you have into the

data you actually need.● Evaluate: Try it on a small scale● Accept that you might have to start over….● But when it works, automate it!!!

Holistic approach

BigML, Inc 8


Command line tools:

join, jq, awk, sed, sort, uniq

Automation:

Shell, Python, etc.

Talend

BigML: flatline, bindings, bigmler, API, whizzml

Relational Db:

MySQL

NonRelational Db:

MongoDB

Tools that help

BigML, Inc 9

Feature Engineering

Data + ML Algorithm, is that enough?

The ML Algorithm only knows about the features in the dataset. Features can be useless to the algorithm if:

● They are not correlated to the objective to be predicted● Their values change their meaning when combined with other

features

For ML Algorithms to work there must be some kind of statistical relation between some of the features and the objective. Sometimes, you must transform the available features to find such relations

Feature engineering: the process of transforming raw data into machine learning readydata

Charles Parker

BigML, Inc 10

Feature Engineering

When do you need Feature Engineering?● When the relationship between the feature and the

objective is mathematically unsatisfying● When the relationship of a function of two or more

features with the objective is far more relevant than the one of the original features

● When there is missing data● When the data is timeseries, especially when the

previous time period’s objective is known● When the data can’t be used for machine learning in the

obvious way (e.g., timestamps, text data)

BigML, Inc 11

Feature Engineering

Mathematical transformations● Statistical aggregations (group by, all and allbut)● Better categories– too many detailed categories should be avoided– ordered categories can be translated to numeric values. The model will be able to

extract more information by partinioning the ordered number range● Binning or discretization: consider whether your number is more informative in

ranges (quartiles, deciles, percentiles) even for the objective field● Linearization: nonimportant for decision trees but can be for logistic regression

(watch out for exponential distributions)

Missing data● Missing value induction (replace missings with common values: mean, median,

mode, even with a Machine Learning model)● Missing values presence can be informative, so this can be added as a new feature

BigML, Inc 12

Feature Engineering

Timeseries transformations● Better objective (percent change instead of absolute

values)● Deltas from previous reference time points● Deltas from moving average (time windows)● Recent Volatility...

Problem: Exponential explosion of possible transformations

Caveats:● The regularity in time of the points has to match your training data● You have to keep track of past points to compute your windows● Really easy to get information leakage by including your objective in a

window computation (and can be very hard to detect)!

BigML, Inc 13

Feature Engineering

Datetime features● Cannot be used “as is” in a model. It's a collection of features. BigML is able to

decompose them automatically when they are provided in the most usual formats. With Flatline, you can decompose them all.

● Datetime predicates that the computer does not know (some of them, domain dependent): Working hours? Daylight? Is rush hour?...

Text features● Bag of words: a new feature is associated to each word in the document● Tokenization: how do we select tokens? Do we want ngrams? What about

numbers?● Stemming: grouping forms of the same word in a unique term● Length● Text predicates: Dollar amounts? Dates? Salutations? Please and Thank you?

BigML, Inc 14

Feature Engineering

Machine Learning for Feature engineeringLatent Dirichlet Allocation

• Learn word distributions for topics

• Infer topic scores for each document

• Use the topic scores as features to a model (dimensional reduction)

Distance to cluster Centroids

Stacked Generalization: Classifiers provide new features

BigML, Inc 15

Day 2 – Evening sessions

BigML, Inc 16

REST API, bindings and basic workflows

jao (José Antonio Ortega)

Academics Real world

How do Machine Learning Workflows look like?

We need highlevel tools to face the real world workflows by growing in:

● Automation● Abstraction

BigML, Inc 17


The foundations● REST API first applications: Standards in software development.

First level of abstraction

Client side tools● Web UI: Sitting on top of the REST API. Humanfriendly access and

visualizations for all the Machine Learning resources. Workflows must be defined and executed step by step. Second level of abstraction.

● Bindings: Sitting on top of the REST API. Finegrained accessors for the REST API calls. Workflows must be defined and executed step by step. Second level of abstraction.

● BigMLer: Relying on the bindings. Highlevel syntax. Entire workflows can be created in only one command line. Third level of abstraction.

BigML, Inc 18


.

BigMLer automation● Basic 1click workflows in one command line● Rich parameterized workflows: feature selection, crossvalidation, etc.

● Models are downloaded to your laptop, tablet, cell phone, etc. once and can be used offline to create predictions

Still..

Great for local predictions

BigML, Inc 19


.

Problems of clientside solutions● Complexity Lots of details outside the problem

domain● Reuse No interlanguage compatibility● Scalability Clientside workflows hard to optimize● Extensibility BigMLer hides complexity at the cost of

flexibility● Not enough abstraction

BigML, Inc 20


.Solution: bringing automation and abstraction to the serverside

● DSL for ML workflow automation● Framework for scalable, remote execution of ML workflows

Sophisticated serverside optimizationOutofthebox scalability

Clientserver brittleness removedInfrastructure for creating and sharing ML scripts and libraries

WhizzML

BigML, Inc 21


.

WhizzML's new REST API resources:

Scripts: Executable code that describes an actual workflow, taking a list of typed inputs and producing a list of outputs.

Executions: Given a script and a complete set of inputs, the workflow can be executed and its outputs generated.

Libraries: A collection of WhizzML definitions that can be imported by other libraries or scripts.

BigML, Inc 22


ScriptsCreating scripts

● Usable by any binding (from any language)● Builtin parallelization● BigML resources management as primitives of the language● Complete programming language for workflow definition

Using scripts

Web UI

Bindings

BigMLer

WhizzML

BigML, Inc 23

Advanced WhizzML workflows

Charles Parker

WhizzML offers:● Primitives for all ML resources: (datasets, models, clusters, etc.)

● A complete programming language to compose at will these ML resources.

● Parallelization and Scalability builtin.

This empowers the user to benefit from:● Automated feature engineering: Bestfirst feature selection.

● Automated configuration choice: Randomized parameter optimization, SMACdown.

● Complex algorithms as 1click: Stacked generalization, Boosting.

All of them can be shared, reproduced and reused as one more BigML resource in a languageagnostic way.

BigML, Inc 24


f5 fn

... ...

......

... ...

f5 f7 f5 fn

... ...

......

... ...

f5 f1

Selectedfields

()

(f5)

The best scoreis obtained forthe model with (f5)

The best scoreis obtained forthe model with (f5 f7)

Following iterations don't improve the score for the modelwith (f5 f7), so the process stops

Step 1

Step 2

f1Bestfirst feature selection

BigML, Inc 25


A new dataset is generatedwith the predictions for the

hold out data

A new metamodel is createdfrom this dataset

50%

Hold out

Stacked generalization

BigML, Inc 26


Configurationrandom

generator

... ...

Bestscore

Process stops when you reach the expected performanceor the usergiven iterations limit

+

Randomized parameter optimization

BigML, Inc 27


Configurationrandom

generator

... ...

+ New configurations are filteredaccording to the predictionsof the model of performances

Only promisingconfigurations are analyzed

SMACdown

BigML, Inc 28


… …

The final model is an ensemble of models

T0

F0

T1

F1

T2

F2

F8

T8

Boosting

BigML, Inc 29

Advanced WhizzML workflowsScript it once, for everybody anywhere

Publish scripts in the gallery

Add scripts toyour menus

Data & Analytics

VSSML16 LR2. Summary Day 2