Barga Data Science lecture 2

Preview:

Citation preview

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

An Introduction to Statistical Learning with Applications in R

The Elements of Statistical Learning

A Programmer’s Guide to Data Mining

Probabilistic Programming & Bayesian Methods for Hackers

Think Bayes, Bayesian Statistics Made Simple

Deriving Knowledge from Data at Scale

Data Mining and Analysis, Fundamental Concepts and Algorithms

An Introduction to Data Science

Machine Learning

Machine Learning – The Complete Guide

Deriving Knowledge from Data at Scale

Bayesian Reasoning and Machine Learning

A Course in Machine Learning

Information Theory, Inference and Learning Algorithms

Modeling with Data

Mining of Massive Datasets

Information Theory, Inference and Learning Algorithms

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

• Introduction to machine learning

• Overview of the machine learning process

• Introduction to select algorithms

• Introduction to concepts we will examine later in course:

bias/variance, generalization, underfitting, overfitting, ensemble

methods, etc.

• Elements of a time series

• Functions to manipulate time series

• Lay a foundation for Prediction and Forecasting…

Deriving Knowledge from Data at Scale

http://www.cs.waikato.ac.nz/ml/weka/

install by next class, get the developer version…

Deriving Knowledge from Data at Scale

• Class Discussions 20 minutes

• Machine Learning Primer 60 minutes

• Break 15 minutes

• Time Series & Forecasting, 1/2 60 minutes

• Closing discussion 15 minutes

Deriving Knowledge from Data at Scale

so we need teams

Deriving Knowledge from Data at Scale

Discussion on Assigned Reading…

Week One

Deriving Knowledge from Data at Scale

The Data Science WorkflowLecture 1 Review

Stay in the immediate zone during exploratory modeling, to extent possible:

• 5 to 10 minutes per experiment, results in 100’s per day;

• Small, statistically sound/relevant samples;

• Linear modelling during feature exploration;

• Don’t write ML algorithms, use packages, do learn to write data manipulation code;

Start with a sample of data, as soon as you can get it…

• You will learn a considerable amount from that first data sample

• Quality of the data, what fields are being collected, possible missing values and/or

missing fields, and the questions will begin to flow (dialogue with customer will be at a

much more meaningful level).

If your customer can’t quantify it, you can’t change/improve it…

Deriving Knowledge from Data at Scale

The Data Science WorkflowLoops within loops…

Deriving Knowledge from Data at Scale

The Data Science WorkflowDefine the Goal

The first task in a data science project is to define a measurable & quantifiable

goal. At this stage, learn all that you can about the context of your project.

• Why do the project sponsors want the project in the first place? What do they lack, and

what do they need?

• What are they doing to solve the problem now, and why isn't that good enough?

• What resources will you need: what kind of data, how much staff, will you have domain

experts to collaborate with, what are the computational resources?

• How do the project sponsors plan to deploy your results? What are the constraints that

have to be met for successful deployment?

Not We want to get better at finding bad loans.

But We want to reduce our rate of loan charge-offs by at least 10%, using a model

that predicts which loan applicants are likely to default.

Deriving Knowledge from Data at Scale

A concrete goal begets concrete stopping conditions and concrete

acceptance criteria. The less specific the goal, the likelier that the

project will go unbounded, because no result will be "good

enough." If you don't know what you want to achieve, you don't

know when to stop trying – or even what to try. When the project

eventually terminates – because either time or resources run out –

no one will be happy with the outcome…

Deriving Knowledge from Data at Scale

The Data Science WorkflowData Collection and Management

This step encompasses identifying the data you need, exploring it, and

conditioning it to be suitable for analysis. This stage is often the most time-

consuming step in the process. It's also one of the most important.

• What data is available to me?

• Will it help me solve the problem?

• Is it enough?

• Is it of good enough quality?

Rule First piece of data is very informative, data set utility is roughly logarithmic in size.

Prefer Direct measurements, but if they are not available identify proxy variables.

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

The Data Science WorkflowModeling

We get to statistics and machine learning during the modeling stage. Here is where

you try to extract useful insights from the data. Many modeling procedures make

specific assumptions about data distribution and relationships, there will be back-and-

forth between the modeling and data cleaning stages as you try to find the best way to

represent and model the data. The most common data science modeling tasks are:

• Classification: deciding if something belongs to one category or another.

• Scoring: predicting or estimating a numeric value such as a price or probability.

• Ranking: learning to order items by preferences.

• Clustering: grouping items into most similar groups.

• Finding Relations: finding correlations or potential causes of effects seen in the data.

• Characterization: very general plotting and report generation from data.

Deriving Knowledge from Data at Scale

The Data Science WorkflowModel Evaluation and Critique

Once you have a model, you need to determine if it meets your goals.

• Is it accurate enough for your needs?

• Does it generalize well?

• Does it perform better than "the obvious guess"? Better than whatever

estimate you currently use?

• Do the results of the model (coefficients, clusters, rules) make sense in the

context of the problem domain?

If you've answered "no" to any of the above questions, it's time to loop back to

the modeling step — or decide that the data doesn't support the goal you are

trying to achieve.

Deriving Knowledge from Data at Scale

The Data Science WorkflowModel Deployment and Maintenance

Finally, the model is put into operation.

In many organizations this means the data scientist no longer has primary

responsibility for the day-to-day operation of the model. However, you still

should ensure that the model will run smoothly and will not make disastrous

unsupervised decisions. You also want to make sure that the model can be

updated as its environment changes. And in many situations, the model will

initially be deployed in a small pilot program.

The test might bring out issues that you didn't anticipate, and you will have to

adjust the model appropriately.

Deriving Knowledge from Data at Scale

The Data Science WorkflowPresentation and Documentation

Once you have a model that meets your success criteria, you will present your results

to your project sponsor and other stakeholders. You must also document the model

for those in the organization who are responsible for using, running, and maintaining

the model once it has been deployed. Model interpretability may be an issue…

Deriving Knowledge from Data at Scale

The Data Science WorkflowLoops within loops…

Deriving Knowledge from Data at Scale

• Class Discussions 30 minutes

• Machine Learning Primer 60 minutes

• Break 15 minutes

• Time Series & Forecasting 60 minutes

• Closing discussion 15 minutes

Deriving Knowledge from Data at Scale

1 1 5 4 3

7 5 3 5 3

5 5 9 0 6

3 5 2 0 0

training data (expensive) synthetic training data (cheaper)

solve hard problems

value from Big Data

human intelligence

Machine learning enables nearly every

value proposition of web search.

Hundreds of thousands of machines…

Hundreds of metrics and signals per machine…

Which signals correlate with the real cause of a problem?

How can we extract effective repair actions?

solve hard problems

value from Big Data

human intelligence

business analytics

Predicting future performance from historical data

Recommenda-

tion engines

Advertising

analysis

Weather

forecasting for

business

planning

Social network

analysis

IT infrastructure

and web app

optimization

Legal

discovery and

document

archiving

Pricing analysis

Fraud

detection

Churn

analysis

Equipment

monitoring

Location-based

tracking and

services

Personalized

Insurance

Predictive analytics address the likelihood of something happening in the future, even if it is just an instant later…

object encoded with features

(think DB attributes/ OO member fields of primitive types)

𝑑 is the feature dimensionality.

classifier

prediction

(response/dependent variable).

Can be qualitative/quantitative

(classification/regression).

𝑶𝒃𝒋𝒆𝒄𝒕 → 𝑶𝒖𝒕𝒄𝒐𝒎𝒆Entity → Category

Entity → PopularityComplex decision making:

𝑿 → 𝒀

𝑿 = (𝒙𝟎, … , 𝒙𝒅) → 𝒀

input/independent variable

We may know the relation for certain values of 𝑋 and 𝑌:

In fact, we may know the relation for many 𝒙s and 𝑦s:

𝒙, 𝑦

𝒙 𝟏 , 𝑦 𝟏 , … , 𝒙 𝑵 , 𝑦 𝑵 The 𝑖-th 𝑥 is: 𝒙(𝑖) = 𝑥0(𝑖), … , 𝑥𝑑

(𝑖)

TRAINING

Input

𝒙 = (𝑥𝟎, … , 𝑥𝒅)

Online

System

object encoded with features

classifier

prediction

(response/dependent

variable)

Final

Output

𝑦

ModelOffline

Training

Sub-system

Training Data

𝒙 𝟏 , 𝑦 𝟏 , … , 𝒙 𝑵 , 𝑦 𝑵

where 𝒙(𝒊) = 𝑥𝟎(𝒊), … , 𝑥𝒅

(𝒊)

𝑓 𝑋 = 𝑌𝑋 → 𝑌 Task is very complex . Hard to construct good 𝑓.

We construct an approximation to 𝑓: 𝑔(𝑋) ≈ 𝑌Hypothesis space: 𝑔(𝑋) ∈ 𝐻.

Deriving Knowledge from Data at Scale

The decision is driven by both the nature of your

data and the question you are trying to answer

Deriving Knowledge from Data at Scale

Out of Class Reading

Week Two

gender age smoker eye color

male 19 yes green

female 44 yes gray

male 49 yes blue

male 12 no brown

female 37 no brown

female 60 no brown

male 44 no blue

female 27 yes brown

female 51 yes green

female 81 yes gray

male 22 yes brown

male 29 no blue

lung cancer

no

yes

yes

no

no

yes

no

no

yes

no

no

no

male 77 yes gray

male 19 yes green

female 44 no gray

?

?

?

gender age smoker eye color

male 19 yes green

female 44 yes gray

male 49 yes blue

male 12 no brown

female 37 no brown

female 60 no brown

male 44 no blue

female 27 yes brown

female 51 yes green

female 81 yes gray

male 22 yes brown

male 29 no blue

lung cancer

no

yes

yes

no

no

yes

no

no

yes

no

no

no

male 77 yes gray

male 19 yes green

female 44 no gray

?

?

?

Train ML Model

gender age smoker eye color

male 19 yes green

female 44 yes gray

male 49 yes blue

male 12 no brown

female 37 no brown

female 60 no brown

male 44 no blue

female 27 yes brown

female 51 yes green

female 81 yes gray

male 22 yes brown

male 29 no blue

lung cancer

no

yes

yes

no

no

yes

no

no

yes

no

no

no

male 77 yes gray

male 19 yes green

female 44 no gray

yes

no

no

Train ML Model

Always

focuses on abias-variance decomposition We have a lecture dedicated to evaluationmetrics for models.

key

easily the most important factor

key

CITY 1 LAT. CITY 1 LNG. CITY 2 LAT. CITY 2 LNG. DRIVABLE?

123.24 46.71 121.33 47.34 Yes

123.24 56.91 121.33 55.23 Yes

123.24 46.71 121.33 55.34 No

123.24 46.71 130.99 47.34 No

key

Feature engineering

More data wins

Once you’ve defined your input fields, there’s only so much analyticgymnastics you can do. Computer algorithms trying to learn models haveonly a relatively few tricks they can do efficiently, and many of them arenot so very different. Performance differences between algorithms aretypically not large. Thus, if you want better classifiers:

1. Engineer better features

2. Get your hands on more high-quality data

The power of ensembles

models vote for the final prediction

Occam’s Razor

smaller faster to fitmore interpretable

representable

could will

observational data can only show us that twovariables are related, but it cannot tell us the “why”

Freakonomics

tended to have higher standardized test scores

Deriving Knowledge from Data at Scale

Break, 5 minutes…

Deriving Knowledge from Data at Scale

boosting)

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

Ensemble Classification

Deriving Knowledge from Data at Scale

Let’s look at the Netflix Prize Competition…

Deriving Knowledge from Data at Scale

Began October 2006

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

from http://www.research.att.com/~volinsky/netflix/

However, improvement slowed…

Deriving Knowledge from Data at Scale

Today, the top team has posteda 8.5% improvement.

Ensemble methods are the best performers…

Deriving Knowledge from Data at Scale

“Thanks to Paul Harrison's collaboration, a simple mix of our solutions improved our result from 6.31 to 6.75”

Rookies

Deriving Knowledge from Data at Scale

“My approach is to combine the results of many methods (also two-way interactions between them) using linear regression on the test set. The best method in my ensemble is regularized SVD with biases, post processed with kernel ridge regression”

Arek Paterek

http://rainbow.mimuw.edu.pl/~ap/ap_kdd.pdf

Deriving Knowledge from Data at Scale

“When the predictions of multipleRBM models and multiple SVD models are linearly combined, we achieve an error rate that is well over 6% better than the score of Netflix’s own system.”

U of Toronto

http://www.cs.toronto.edu/~rsalakhu/papers/rbmcf.pdf

Deriving Knowledge from Data at Scale

Gravity

home.mit.bme.hu/~gtakacs/download/gravity.pdf

Deriving Knowledge from Data at Scale

“Our common team blends the result of team Gravity and team Dinosaur Planet.”

Might have guessed from the name…

When Gravity and Dinosaurs Unite

Deriving Knowledge from Data at Scale

And, yes, the top team which is from AT&T…

“Our final solution (RMSE=0.8712) consists of blending 107 individual results. “

BellKor / KorBell

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

• 83.7% majority vote accuracy

• 99.9% majority vote accuracy

Intuitions

Deriving Knowledge from Data at Scale

Boosting

Bagging

Deriving Knowledge from Data at Scale

Leo Breiman

Deriving Knowledge from Data at Scale

• Resampl few data

• Resample the minority data skew

Deriving Knowledge from Data at Scale

N=10

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

Break, 5 minutes…

Deriving Knowledge from Data at Scale

Time Series & Forecasting, 1/2

Deriving Knowledge from Data at Scale

median filter, a windowing technique that moves through the data point-by-point, and replaces it with the median value calculated for the current window…

Deriving Knowledge from Data at Scale

Visual inspection reveals smoothing of the outliers without dampening the naturally occurring peaks and troughs (no signal loss). Prior to smoothing, we could see no correlation in our data, but afterwards, Spearman’s Rho was ~0.5 for almost all parameters.

Deriving Knowledge from Data at Scale

?

?

?

Deriving Knowledge from Data at Scale

3.7

12.5

9.0

Deriving Knowledge from Data at Scale

Forecasting

Quantitative

Causal Model

Trend

Time series

Stationary

Trend

Trend + Seasonality

Qualitative

Expert Judgment

Delphi Method

Grassroots

Deriving Knowledge from Data at Scale

Year1

Year2

Year3

Year4

Dem

and

fo

r p

rod

uct

or

serv

ice

Deriving Knowledge from Data at Scale

Year1

Year2

Year3

Year4

Dem

and

fo

r p

rod

uct

or

serv

ice

Trend component

Actual

demand line

Seasonal peaks

Random

variation

Deriving Knowledge from Data at Scale87

Deriving Knowledge from Data at Scale88

Deriving Knowledge from Data at Scale

• Naïve

• Moving Average

• Exponential Smoothing

• Regression

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

91

Deriving Knowledge from Data at Scale92

smoothing time

n

A+...+A +A +A =F 1n-t2-t1-tt

1t

Ft+1 = Forecast for the upcoming period, t+1

n = Number of periods to be averaged

A t = Actual occurrence in period t

Deriving Knowledge from Data at Scale

3

n

A+...+A +A +A =F 1n-t2-t1-tt

1t

MonthSales

(000)

1 4

2 6

3 54 ?5 ?6 ?

Deriving Knowledge from Data at Scale

MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 ?5 ?

(4+6+5)/3=5

6 ?

n

A+...+A +A +A =F 1n-t2-t1-tt

1t

You’re manager in Amazon’s electronics department. You want to forecast ipod

sales for months 4-6 using a 3-period moving average.

Deriving Knowledge from Data at Scale

MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 ?

5

6 ?

?

Deriving Knowledge from Data at Scale

MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 ?

5

6 ?

(6+5+3)/3=4.667

Deriving Knowledge from Data at Scale

MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 7

5

6 ?

4.667?

Deriving Knowledge from Data at Scale

MonthSales

(000)Moving Average

(n=3)1 4 NA

2 6 NA

3 5 NA4 35 7

5

6 ?

4.667(5+3+7)/3=5

Deriving Knowledge from Data at Scale99

Deriving Knowledge from Data at Scale

100

Deriving Knowledge from Data at Scale10

1

Deriving Knowledge from Data at Scale

Month Weighted

Moving

Average1 4 NA

2 6 NA

3 5 NA4 31/6 = 5.167

56 ?

??

1n-tn2-t31-t2t11t Aw+...+Aw+A w+A w=F

Sales(000)

Deriving Knowledge from Data at Scale

Month Sales(000)

Weighted

Moving

Average1 4 NA

2 6 NA

3 5 NA4 3 31/6 = 5.167

5 76

25/6 = 4.16732/6 = 5.333

1n-tn2-t31-t2t11t Aw+...+Aw+A w+A w=F

Deriving Knowledge from Data at Scale10

4

Deriving Knowledge from Data at Scale10

5

Deriving Knowledge from Data at Scale

106

simple

any

• Previous

time

previous actual and forecasted value

Deriving Knowledge from Data at Scale

• gives more weight to recent time periods

Ft+1 = Ft + a(At - Ft)

et

Ft+1 = Forecast value for time t+1

At = Actual value at time t

a = Smoothing constant

Need initial

forecast Ft

to start.

Deriving Knowledge from Data at Scale

Week Demand

1 820

2 775

3 680

4 655

5 750

6 802

7 798

8 689

9 775

10

Given the weekly demand

data what are the exponential

smoothing forecasts for

periods 2-10 using a=0.10?

Assume F1=D1

Ft+1 = Ft + a(At - Ft)i Ai

Deriving Knowledge from Data at Scale

Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28

Ft+1 = Ft + a(At - Ft)

a =

F2 = F1+ a(A1–F1) =820+.1(820–820)

=820

i Ai Fi

Deriving Knowledge from Data at Scale

Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28

Ft+1 = Ft + a(At - Ft)

a =

F3 = F2+ a(A2–F2) =820+.1(775–820)

=815.5

i Ai Fi

Deriving Knowledge from Data at Scale

Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28

Ft+1 = Ft + a(At - Ft)

This process

continues

through week

10

a =

i Ai Fi

Deriving Knowledge from Data at Scale

Week Demand 0.1 0.6

1 820 820.00 820.00

2 775 820.00 820.00

3 680 815.50 793.00

4 655 801.95 725.20

5 750 787.26 683.08

6 802 783.53 723.23

7 798 785.38 770.49

8 689 786.64 787.00

9 775 776.88 728.20

10 776.69 756.28

Ft+1 = Ft + a(At - Ft)

What if the

a constant

equals 0.6

a = a =

i Ai Fi

Deriving Knowledge from Data at Scale

α• depends on the emphasis you want to place on the

most recent data

α

Deriving Knowledge from Data at Scale

Ft+1 = a At + a(1- a) At - 1 + a(1- a)2At - 2 + ...

Weights

Prior Period

a

2 periods ago

a(1 - a)

3 periods ago

a(1 - a)2

a=

a= 0.10

a= 0.90

10% 9% 8.1%

90% 9% 0.9%

Ft+1 = Ft + a (At - Ft)

or

w1 w2 w3

Deriving Knowledge from Data at Scale

115

Deriving Knowledge from Data at Scale

Trend

Seasonal

Deriving Knowledge from Data at Scale11

7

Deriving Knowledge from Data at Scale

0

1

2

3

4

5

6

7

1 2 3 4 5 6 7 8 9 10

Deriving Knowledge from Data at Scale

b0 b1

0

2

4

6

8

10

12

10 11 12 13 14 15 16 17 18 19 20

Best line!

Intercept

Deriving Knowledge from Data at Scale

deseasonalize

• Reseasonalize

Deriving Knowledge from Data at Scale

Deseasonalize

Forecast

Reseasonalize

Actual data Deseasonalized data

Example (SI + Regression)

Deriving Knowledge from Data at Scale12

2

Deriving Knowledge from Data at Scale

123

Year

Sales

($ mil.)

2002 7

2003 10

2004 9

2005 11

2006 13

Linear Trend – Using the Least Squares Method: An Example

Year t

Sales

($ mil.)

2002 1 7

2003 2 10

2004 3 9

2005 4 11

2006 5 13

Deriving Knowledge from Data at Scale

124

Deriving Knowledge from Data at Scale12

5

Deriving Knowledge from Data at Scale

126

(Excel function:

=log(x) or log(x,10)

Deriving Knowledge from Data at Scale12

7

Deriving Knowledge from Data at Scale

SI

SI

Deriving Knowledge from Data at Scale

year

raw index

quarter

four

Deriving Knowledge from Data at Scale130

The table below shows the quarterly sales for Toys International for the years 2001 through 2006. The sales are reported in millions of dollars. Determine a quarterly seasonal index using the ratio-to-moving-average method.

Seasonal Index – An Example

Deriving Knowledge from Data at Scale13

1

Deriving Knowledge from Data at Scale13

2

Deriving Knowledge from Data at Scale13

3

Deriving Knowledge from Data at Scale

deseasonalize

Deriving Knowledge from Data at Scale

Deriving Knowledge from Data at Scale

• Class Discussions 15 minutes

• Forecasting, continued (2/2) 45 minutes

• Weka Tutorial 30 minutes

• Break 10 minutes

• Decision Trees and Random Forests 60 minutes

• Hands on, decision tree in Weka 15 minutes

Deriving Knowledge from Data at Scale

That’s all for tonight….