Barga Data Science lecture 2

Deriving Knowledge from Data at Scale


An Introduction to Statistical Learning with Applications in R

The Elements of Statistical Learning

A Programmer’s Guide to Data Mining

Probabilistic Programming & Bayesian Methods for Hackers

Think Bayes, Bayesian Statistics Made Simple

http://www-bcf.usc.edu/~gareth/ISL/

http://statweb.stanford.edu/~tibs/ElemStatLearn/

http://guidetodatamining.com/

http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/

http://greenteapress.com/thinkbayes/


Data Mining and Analysis, Fundamental Concepts and Algorithms

An Introduction to Data Science

Machine Learning

Machine Learning – The Complete Guide

http://www.dataminingbook.info/pmwiki.php/Main/BookDownload

https://docs.google.com/file/d/0B6iefdnF22XQeVZDSkxjZ0Z5VUE/edit?pli=1

http://www.intechopen.com/books/machine_learning

https://en.wikipedia.org/wiki/Book:Machine_Learning_-_The_Complete_Guide


Bayesian Reasoning and Machine Learning

A Course in Machine Learning

Information Theory, Inference and Learning Algorithms

Modeling with Data

Mining of Massive Datasets

Information Theory, Inference and Learning Algorithms

http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage

http://ciml.info/

http://www.inference.phy.cam.ac.uk/mackay/itprnn/book.html

http://modelingwithdata.org/about_the_book.html

http://infolab.stanford.edu/~ullman/mmds.html

http://www.inference.phy.cam.ac.uk/itprnn/book.html



• Introduction to machine learning

• Overview of the machine learning process

• Introduction to select algorithms

• Introduction to concepts we will examine later in course:

bias/variance, generalization, underfitting, overfitting, ensemble

methods, etc.

• Elements of a time series

• Functions to manipulate time series

• Lay a foundation for Prediction and Forecasting…


http://www.cs.waikato.ac.nz/ml/weka/

install by next class, get the developer version…

http://www.cs.waikato.ac.nz/ml/weka/


• Class Discussions 20 minutes

• Machine Learning Primer 60 minutes

• Break 15 minutes

• Time Series & Forecasting, 1/2 60 minutes

• Closing discussion 15 minutes


so we need teams


Discussion on Assigned Reading…

Week One


The Data Science WorkflowLecture 1 Review

Stay in the immediate zone during exploratory modeling, to extent possible:

• 5 to 10 minutes per experiment, results in 100’s per day;

• Small, statistically sound/relevant samples;

• Linear modelling during feature exploration;

• Don’t write ML algorithms, use packages, do learn to write data manipulation code;

Start with a sample of data, as soon as you can get it…

• You will learn a considerable amount from that first data sample

• Quality of the data, what fields are being collected, possible missing values and/or

missing fields, and the questions will begin to flow (dialogue with customer will be at a

much more meaningful level).

If your customer can’t quantify it, you can’t change/improve it…


The Data Science WorkflowLoops within loops…


The Data Science WorkflowDefine the Goal

The first task in a data science project is to define a measurable & quantifiable

goal. At this stage, learn all that you can about the context of your project.

• Why do the project sponsors want the project in the first place? What do they lack, and

what do they need?

• What are they doing to solve the problem now, and why isn't that good enough?

• What resources will you need: what kind of data, how much staff, will you have domain

experts to collaborate with, what are the computational resources?

• How do the project sponsors plan to deploy your results? What are the constraints that

have to be met for successful deployment?

Not We want to get better at finding bad loans.

But We want to reduce our rate of loan charge-offs by at least 10%, using a model

that predicts which loan applicants are likely to default.


A concrete goal begets concrete stopping conditions and concrete

acceptance criteria. The less specific the goal, the likelier that the

project will go unbounded, because no result will be "good

enough." If you don't know what you want to achieve, you don't

know when to stop trying – or even what to try. When the project

eventually terminates – because either time or resources run out –

no one will be happy with the outcome…


The Data Science WorkflowData Collection and Management

This step encompasses identifying the data you need, exploring it, and

conditioning it to be suitable for analysis. This stage is often the most time-

consuming step in the process. It's also one of the most important.

• What data is available to me?

• Will it help me solve the problem?

• Is it enough?

• Is it of good enough quality?

Rule First piece of data is very informative, data set utility is roughly logarithmic in size.

Prefer Direct measurements, but if they are not available identify proxy variables.



The Data Science WorkflowModeling

We get to statistics and machine learning during the modeling stage. Here is where

you try to extract useful insights from the data. Many modeling procedures make

specific assumptions about data distribution and relationships, there will be back-and-

forth between the modeling and data cleaning stages as you try to find the best way to

represent and model the data. The most common data science modeling tasks are:

• Classification: deciding if something belongs to one category or another.

• Scoring: predicting or estimating a numeric value such as a price or probability.

• Ranking: learning to order items by preferences.

• Clustering: grouping items into most similar groups.

• Finding Relations: finding correlations or potential causes of effects seen in the data.

• Characterization: very general plotting and report generation from data.


The Data Science WorkflowModel Evaluation and Critique

Once you have a model, you need to determine if it meets your goals.

• Is it accurate enough for your needs?

• Does it generalize well?

• Does it perform better than "the obvious guess"? Better than whatever

estimate you currently use?

• Do the results of the model (coefficients, clusters, rules) make sense in the

context of the problem domain?

If you've answered "no" to any of the above questions, it's time to loop back to

the modeling step — or decide that the data doesn't support the goal you are

trying to achieve.


The Data Science WorkflowModel Deployment and Maintenance

Finally, the model is put into operation.

In many organizations this means the data scientist no longer has primary

responsibility for the day-to-day operation of the model. However, you still

should ensure that the model will run smoothly and will not make disastrous

unsupervised decisions. You also want to make sure that the model can be

updated as its environment changes. And in many situations, the model will

initially be deployed in a small pilot program.

The test might bring out issues that you didn't anticipate, and you will have to

adjust the model appropriately.


The Data Science WorkflowPresentation and Documentation

Once you have a model that meets your success criteria, you will present your results

to your project sponsor and other stakeholders. You must also document the model

for those in the organization who are responsible for using, running, and maintaining

the model once it has been deployed. Model interpretability may be an issue…


The Data Science WorkflowLoops within loops…



• Machine Learning Primer 60 minutes


• Time Series & Forecasting 60 minutes

• Closing discussion 15 minutes


1 1 5 4 3

7 5 3 5 3

5 5 9 0 6

3 5 2 0 0

training data (expensive) synthetic training data (cheaper)

solve hard problems

value from Big Data

human intelligence

Machine learning enables nearly every

value proposition of web search.

Hundreds of thousands of machines…

Hundreds of metrics and signals per machine…

Which signals correlate with the real cause of a problem?

How can we extract effective repair actions?

solve hard problems

value from Big Data

human intelligence

business analytics

Predicting future performance from historical data

Recommenda-

tion engines

Advertising

analysis

Weather

forecasting for

business

planning

Social network

analysis

IT infrastructure

and web app

optimization

Legal

discovery and

document

archiving

Pricing analysis

Fraud

detection

Churn

analysis

Equipment

monitoring

Location-based

tracking and

services

Personalized

Insurance

Predictive analytics address the likelihood of something happening in the future, even if it is just an instant later…

object encoded with features

(think DB attributes/ OO member fields of primitive types)

𝑑 is the feature dimensionality.

classifier

prediction

(response/dependent variable).

Can be qualitative/quantitative

(classification/regression).

𝑶𝒃𝒋𝒆𝒄𝒕 → 𝑶𝒖𝒕𝒄𝒐𝒎𝒆Entity → Category

Entity → PopularityComplex decision making:

𝑿 → 𝒀

𝑿 = (𝒙𝟎, … , 𝒙𝒅) → 𝒀

input/independent variable

We may know the relation for certain values of 𝑋 and 𝑌:

In fact, we may know the relation for many 𝒙s and 𝑦s:

𝒙, 𝑦

𝒙 𝟏 , 𝑦 𝟏 , … , 𝒙 𝑵 , 𝑦 𝑵 The 𝑖-th 𝑥 is: 𝒙(𝑖) = 𝑥0(𝑖), … , 𝑥𝑑

(𝑖)

TRAINING

Input

𝒙 = (𝑥𝟎, … , 𝑥𝒅)

Online

System

object encoded with features

classifier

prediction

(response/dependent

variable)

Final

Output

𝑦

ModelOffline

Training

Sub-system

Training Data

𝒙 𝟏 , 𝑦 𝟏 , … , 𝒙 𝑵 , 𝑦 𝑵

where 𝒙(𝒊) = 𝑥𝟎(𝒊), … , 𝑥𝒅

(𝒊)

𝑓 𝑋 = 𝑌𝑋 → 𝑌 Task is very complex . Hard to construct good 𝑓.

We construct an approximation to 𝑓: 𝑔(𝑋) ≈ 𝑌Hypothesis space: 𝑔(𝑋) ∈ 𝐻.


The decision is driven by both the nature of your

data and the question you are trying to answer


Out of Class Reading

Week Two

gender age smoker eye color

male 19 yes green

female 44 yes gray

male 49 yes blue

male 12 no brown

female 37 no brown

female 60 no brown

male 44 no blue

female 27 yes brown

female 51 yes green

female 81 yes gray

male 22 yes brown

male 29 no blue

lung cancer

no

yes

yes

no

no

yes

no

no

yes

no

no

no

male 77 yes gray

male 19 yes green

female 44 no gray

?

?

?


male 19 yes green

female 44 yes gray

male 49 yes blue

male 12 no brown

female 37 no brown

female 60 no brown

male 44 no blue

female 27 yes brown

female 51 yes green

female 81 yes gray

male 22 yes brown

male 29 no blue

lung cancer

no

yes

yes

no

no

yes

no

no

yes

no

no

no

male 77 yes gray

male 19 yes green

female 44 no gray

?

?

?

Train ML Model


male 19 yes green

female 44 yes gray

male 49 yes blue

male 12 no brown

female 37 no brown

female 60 no brown

male 44 no blue

female 27 yes brown

female 51 yes green

female 81 yes gray

male 22 yes brown

male 29 no blue

lung cancer

no

yes

yes

no

no

yes

no

no

yes

no

no

no

male 77 yes gray

male 19 yes green

female 44 no gray

yes

no

no

Train ML Model

Always

focuses on abias-variance decomposition We have a lecture dedicated to evaluationmetrics for models.

key

easily the most important factor

key

CITY 1 LAT. CITY 1 LNG. CITY 2 LAT. CITY 2 LNG. DRIVABLE?

123.24 46.71 121.33 47.34 Yes

123.24 56.91 121.33 55.23 Yes

123.24 46.71 121.33 55.34 No

123.24 46.71 130.99 47.34 No

key

Feature engineering

More data wins

Once you’ve defined your input fields, there’s only so much analyticgymnastics you can do. Computer algorithms trying to learn models haveonly a relatively few tricks they can do efficiently, and many of them arenot so very different. Performance differences between algorithms aretypically not large. Thus, if you want better classifiers:

1. Engineer better features

2. Get your hands on more high-quality data

The power of ensembles

models vote for the final prediction

Occam’s Razor

smaller faster to fitmore interpretable

representable

could will

observational data can only show us that twovariables are related, but it cannot tell us the “why”

Freakonomics

tended to have higher standardized test scores


Break, 5 minutes…


boosting)



Ensemble Classification


Let’s look at the Netflix Prize Competition…


Began October 2006



from http://www.research.att.com/~volinsky/netflix/

However, improvement slowed…


Today, the top team has posteda 8.5% improvement.

Ensemble methods are the best performers…


“Thanks to Paul Harrison's collaboration, a simple mix of our solutions improved our result from 6.31 to 6.75”

Rookies


“My approach is to combine the results of many methods (also two-way interactions between them) using linear regression on the test set. The best method in my ensemble is regularized SVD with biases, post processed with kernel ridge regression”

Arek Paterek

http://rainbow.mimuw.edu.pl/~ap/ap_kdd.pdf


“When the predictions of multipleRBM models and multiple SVD models are linearly combined, we achieve an error rate that is well over 6% better than the score of Netflix’s own system.”

U of Toronto

http://www.cs.toronto.edu/~rsalakhu/papers/rbmcf.pdf


Gravity

home.mit.bme.hu/~gtakacs/download/gravity.pdf


“Our common team blends the result of team Gravity and team Dinosaur Planet.”

Might have guessed from the name…

When Gravity and Dinosaurs Unite


And, yes, the top team which is from AT&T…

“Our final solution (RMSE=0.8712) consists of blending 107 individual results. “

BellKor / KorBell



• 83.7% majority vote accuracy

• 99.9% majority vote accuracy

Intuitions


Boosting

Bagging


Leo Breiman


• Resampl few data

• Resample the minority data skew


N=10





Break, 5 minutes…


Time Series & Forecasting, 1/2


median filter, a windowing technique that moves through the data point-by-point, and replaces it with the median value calculated for the current window…


Visual inspection reveals smoothing of the outliers without dampening the naturally occurring peaks and troughs (no signal loss). Prior to smoothing, we could see no correlation in our data, but afterwards, Spearman’s Rho was ~0.5 for almost all parameters.


?

?

?


3.7

12.5

9.0


Forecasting

Quantitative

Causal Model

Trend

Time series

Stationary

Trend

Trend + Seasonality

Qualitative

Expert Judgment

Delphi Method

Grassroots


Year1

Year2

Year3

Year4

Dem

and

fo

r p

rod

uct

or

serv

ice


Year1

Year2

Year3

Year4

Dem

and

fo

r p

rod

uct

or

serv

ice

Trend component

Actual

demand line

Seasonal peaks

Random

variation

Deriving Knowledge from Data at Scale87



• Naïve

• Moving Average

• Exponential Smoothing

• Regression



91


smoothing time

n

A+...+A +A +A =F 1n-t2-t1-tt

1t

Ft+1 = Forecast for the upcoming period, t+1

n = Number of periods to be averaged

A t = Actual occurrence in period t


3

n

A+...+A +A +A =F 1n-t2-t1-tt

1t

MonthSales

(000)

1 4

2 6

3 54 ?5 ?6 ?


MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 ?5 ?

(4+6+5)/3=5

6 ?

n

A+...+A +A +A =F 1n-t2-t1-tt

1t

You’re manager in Amazon’s electronics department. You want to forecast ipod

sales for months 4-6 using a 3-period moving average.


MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 ?

5

6 ?

?


MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 ?

5

6 ?

(6+5+3)/3=4.667


MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 7

5

6 ?

4.667?


MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 7

5

6 ?

4.667(5+3+7)/3=5



100


1


Month Weighted

Moving

Average1 4 NA

2 6 NA

3 5 NA4 31/6 = 5.167

56 ?

??

1n-tn2-t31-t2t11t Aw+...+Aw+A w+A w=F

Sales(000)


Month Sales(000)

Weighted

Moving

Average1 4 NA

2 6 NA

3 5 NA4 3 31/6 = 5.167

5 76

25/6 = 4.16732/6 = 5.333

1n-tn2-t31-t2t11t Aw+...+Aw+A w+A w=F


4


5


106

simple

any

• Previous

time

previous actual and forecasted value


• gives more weight to recent time periods

Ft+1 = Ft + a(At - Ft)

et

Ft+1 = Forecast value for time t+1

At = Actual value at time t

a = Smoothing constant

Need initial

forecast Ft

to start.


Week Demand

1 820

2 775

3 680

4 655

5 750

6 802

7 798

8 689

9 775

10

Given the weekly demand

data what are the exponential

smoothing forecasts for

periods 2-10 using a=0.10?

Assume F1=D1

Ft+1 = Ft + a(At - Ft)i Ai


Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28


a =

F2 = F1+ a(A1–F1) =820+.1(820–820)

=820

i Ai Fi


Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28


a =

F3 = F2+ a(A2–F2) =820+.1(775–820)

=815.5

i Ai Fi


Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28


This process

continues

through week

10

a =

i Ai Fi


Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28


What if the

a constant

equals 0.6

a = a =

i Ai Fi


α• depends on the emphasis you want to place on the

most recent data

α


Ft+1 = a At + a(1- a) At - 1 + a(1- a)2At - 2 + ...

Weights

Prior Period

a

2 periods ago

a(1 - a)

3 periods ago

a(1 - a)2

a=

a= 0.10

a= 0.90

10% 9% 8.1%

90% 9% 0.9%

Ft+1 = Ft + a (At - Ft)

or

w1 w2 w3


115


Trend

Seasonal


7


0

1

2

3

4

5

6

7

1 2 3 4 5 6 7 8 9 10


b0 b1

0

2

4

6

8

10

12

10 11 12 13 14 15 16 17 18 19 20

Best line!

Intercept


deseasonalize

• Reseasonalize


Deseasonalize

Forecast

Reseasonalize

Actual data Deseasonalized data

Example (SI + Regression)


2


123

Year

Sales

($ mil.)

2002 7

2003 10

2004 9

2005 11

2006 13

Linear Trend – Using the Least Squares Method: An Example

Year t

Sales

($ mil.)

2002 1 7

2003 2 10

2004 3 9

2005 4 11

2006 5 13


124


5


126

(Excel function:

=log(x) or log(x,10)


7


SI

SI


year

raw index

quarter

four


The table below shows the quarterly sales for Toys International for the years 2001 through 2006. The sales are reported in millions of dollars. Determine a quarterly seasonal index using the ratio-to-moving-average method.

Seasonal Index – An Example


1


2


3


deseasonalize




• Forecasting, continued (2/2) 45 minutes

• Weka Tutorial 30 minutes


• Decision Trees and Random Forests 60 minutes

• Hands on, decision tree in Weka 15 minutes


That’s all for tonight….

Data & Analytics

Barga Data Science lecture 2