Barga Data Science lecture 9

Deriving Knowledge from Data at Scale





What is model overfitting?

How would you identify it?

How would you correct it?


What is model underfitting?

How would you identify it?

How would you correct it?


List methods for dealing with missing values?

But what must you do first?


Describe stratification and give an example of when you would use it…


Why is feature selection important?


Strategy to Incrementally Build on Foundation

• Select a Data Type

•Data Manipulation for Selected Data Type

• Feature Selection, Feature Creation, Transformation,…

•Machine Learning Algorithm

• Techniques and Association Tool(s)



You can make much stronger inferences about a woman named Brittany. That name was very popular from the mid-1980s through the mid-1990s, but it wasn’t all that common before and hasn’t been since. If you know a Brittany, she is probably of college age or just a bit older. Half of living American Brittanys are between the ages of 19 and 25







Optional Reading…


About the Experiment


Key Points


Key Points



Read to Hack Your Machine Learning Development


Read to Hack Your Machine Learning Development


Read to Hack Your Machine Learning DevelopmentCascade Classifier is one of the most popular face detection algorithms and the default choice in OpenCV libraries as well. Highly accurate and very fast…

• 15 times faster than previous work at the time, fast enough for realtime;

• Intuition: Accurate and complex models are computational expensive. The input are subjected to a series of increasing accurate and expensive models, and the most expensive model will be used on only the most promising input. So it asks a series of questions like: 1) Does model one thinks it is a face? If no, stop. If Yes, ask model two. 2) Does model two thinks it is a face? If no, stop. If yes, ask model three. And so on, until the last model also say yes.

• Feature-based then pixel-based. Haar-like features were created for the images. Note working with pixels is generally very computational expensive.

• Haar-like features are basically high-level features over pixels. For example, the feature observation that the region of the eyes is darker than the region of the cheeks. This can be used as a feature input to the model.


Better, Best?Simple (single) Classifier, Bagging, or Boosting?

Bagging: Multiple classifiers trained on different under-sampled subsets and allow classifiers to vote on final decision, in contrast to using just one classifier.

Boosting: Series of classifiers to train on the dataset, but gradually putting more emphasis on training examples that the previous classifiers have failed on, in the hope of that the next classifier will focus on these harder examples. So in the end, you will have a series of classifiers who are in general balanced but slightly more focused on the hard training examples.

In practice, boosting beats bagging, either bagging and boosting will beat a plain classifier. See ‘Bagging, Boosting and C4.5’ by JR Quinlan. Related work and experience is that Random Forest models are as good as boosting.



Retrain on whole dataset after validating model?Suppose you have a dataset split into 80% for training and 20% for validation, do you follow ‘Plan A’ or ‘Plan B’?

Plan A

Plan B


Retrain on whole dataset after validating model?Plan A


Retrain on whole dataset after validating model?Plan A

hyper-parameters


Hyperparameter TuningJust to be sure we’re all on the same page…







Random search

• If the close-to-optimal region of hyperparameters occupies at least 5% of the grid surface, then random search with 60 trials will find that region with high probability.

With its utter simplicity and surprisingly reasonable performance, random search is my to-go method for hyperparameter tuning. It’s trivially parallelizable, just like grid search, but it takes much fewer tries and performance almost as well most of the time


Feature Creation


Process of Feature Engineering


Why?

most important



• Google Refine

• Weka

• Brainstorming







https://www.youtube.com/watch?v=yNccGtn3Wb0

https://www.youtube.com/watch?v=cO8NVCs_Ba0

https://www.youtube.com/watch?v=5tsyz3ibYzk

http://www.youtube.com/watch?v=B70J_H_zAWM







Not all errors are equalWe need better metrics…


True

Label

Predicted Label

Confusion

matrix


True Positive Rate True Negative Rate False Positive Rate False Negative Rate





average per-class accuracyaverage of the accuracy for each class




Frequently, one might look at only the top k items from the ranker, k = 5, 10,

20, 100, etc. Then the metrics would be called “precision@k” and “recall@k.”


Frequently, one might look at only the top k items from the ranker, k = 5, 10,

20, 100, etc. Then the metrics would be called “precision@k” and “recall@k.”

One might average precision and recall scores for each query and look at

“average precision@k” and “average recall@k.” (Analogous to the relationship

between accuracy and average per-class accuracy for classification.)


Once you can compute precision and recall, you are often able to produce

precision/recall curves. Suppose that you are attempting to identify spam. You

run a learning algorithm to make predictions on a test set. But instead of just

taking a “yes/no” answer, you allow your algorithm to produce its confidence.

For instance, using a perceptron, you might use the distance from the

hyperplane as a confidence measure. You can then sort all of your test emails

according to this ranking. You may put the most spam-like emails at the top

and the least spam-like emails at the bottom


Once you can compute precision and recall, you are often able to produce precision/recall curves. Suppose

that you are attempting to identify spam. You run a learning algorithm to make predictions on a test set. But

instead of just taking a “yes/no” answer, you allow your algorithm to produce its confidence. For instance, using

a perceptron, you might use the distance from the hyperplane as a confidence measure. You can then sort all

of your test emails according to this ranking. You may put the most spam-like emails at the top and the least

spam-like emails at the bottom

Once you have this sorted list, you can choose how aggressively you want your

spam filter to be by setting a threshold anywhere on this list. One would hope

that if you set the threshold very high, you are likely to have high precision (but

low recall). If you set the threshold very low, you’ll have high recall (but low

precision). By considering every possible place you could put this threshold,

you can trace out a curve of precision/recall values. This allows us to ask the

question: for some fixed precision, what sort of recall can I get…



• Method 1

• Method 2


Method 1

Method 2


Method 1

Method 2

very small differences quite a large difference


Sometimes we want a single number that informs us of the quality of the

solution. A popular way to combe precision and recall into a single number is

by taking their harmonic mean. This is known as the balanced f-measure:

The reason to use a harmonic mean rather than an arithmetic mean is that it

favors systems that achieve roughly equal precision and recall. In the extreme

case where P = R, then F = P = R. But in the imbalanced case, for instance P =

0.1 and R = 0.9, the overall f-measure is a modest 0.18.


NDCG

treat all retrieved items equally


NDCG


Regression Metrics


Regression Metrics


Regression Metrics


This was not an exhaustive list or coverage, but I hope the next time you encounter a new evaluation metric you can deconstruct it, identify what is it measuring and why – to align with the business objective or function of the model?


10 minutes break…





X

Y

N1

N2

o1

o2

O3


Related problems

* N. Talleb, The Black Swan: The Impact of the Highly Probable?, 2007




Tid SrcIPStart

timeDest IP

Dest

Port

Number

of bytesAttack

1 206.135.38.95 11:07:20 160.94.179.223 139 192 No

2 206.163.37.95 11:13:56 160.94.179.219 139 195 No

3 206.163.37.95 11:14:29 160.94.179.217 139 180 No

4 206.163.37.95 11:14:30 160.94.179.255 139 199 No

5 206.163.37.95 11:14:32 160.94.179.254 139 19 Yes

6 206.163.37.95 11:14:35 160.94.179.253 139 177 No

7 206.163.37.95 11:14:36 160.94.179.252 139 172 No

8 206.163.37.95 11:14:38 160.94.179.251 139 285 Yes

9 206.163.37.95 11:14:41 160.94.179.250 139 195 No

10 206.163.37.95 11:14:44 160.94.179.249 139 163 Yes10


Tid SrcIP Duration Dest IPNumber

of bytesInternal

1 206.163.37.81 0.10 160.94.179.208 150 No

2 206.163.37.99 0.27 160.94.179.235 208 No

3 160.94.123.45 1.23 160.94.179.221 195 Yes

4 206.163.37.37 112.03 160.94.179.253 199 No

5 206.163.37.41 0.32 160.94.179.244 181 No




X

Y

N1

N2

o1

o2

O3


Normal

Anomaly

Deriving Knowledge from Data at ScaleAnomalous Subsequence



Evaluation of Anomaly Detection – F-value


Evaluation of Outlier Detection – ROC & AUC

Standard measures for evaluating anomaly detection problems:

Predicted

class

Confusion

matrix

NC C

NC TN FP Actual

class C FN TP

anomaly class – C

normal class – NC



Using Support Vector Machines

SVM Classifiers

• Main idea

• Normal data records belong to high density data regions

• Anomalies belong to low density data regions

• Use unsupervised approach to learn high density and low density data

regions

• Use SVM to classify data density level

• Main idea

• Data records are labelled (normal network behaviour vs. intrusive)

• Use standard SVM for classification


Using Replicator Neural Networks

Target

variablesInput


Using Support Vector Machines

origin push the hyper plane

away from origin as

much as possible


Nearest Neighbour Based Techniques


Nearest Neighbour Based Techniques


Nearest Neighbor Based Techniques


Distance based Outlier Detection

Nearest Neighbour (NN) approach

• For each data point d compute the distance to the k-th nearest neighbour dk

• Sort all data points according to the distance dk

• Outliers are points that have the largest distance dk and therefore are located

in the more sparse neighbourhoods

• Usually data points that have top n% distance dk are identified as outliers

• n – user parameter


Model-based approaches to anomaly detection construct a profile of normal

instances, then identify instances that do not conform to this profile as anomalies.

Isolation Forrest is a fundamentally different model-based method that explicitly

isolates anomalies instead of profiles normal points. iForests create an algorithm

which has a linear time complexity with a low constant and a low memory

requirement.

Empirical evaluations shows that iForest performs favorably to OCSVM a near-

linear time complexity distance-based method, NN and random forests in terms

of AUC and processing time, and especially in large data sets. iForest also works

well in high dimensional problems which have a large number of irrelevant

attributes, and in situations where training set does not contain any anomalies.


Randomly select a dimension (feature) and then

randomly select a cut point within that feature

space, recurse, build a tree until leaf node has

only n (for example 1).

Create an ensemble of such trees, merge the

results.

Outliers do not have neighbors (mass) and will

be close to the root (top) of the tree…


“Isolation Forest”

• When building a tree, anomalies are likely to be

isolated closer to the root of the tree; whereas

normal points appear deeper in the tree structure

• No need to profile normal data points

• No distance or density measures

• Gap

• Lacks explanatory power



Neural Networks

http://www.youtube.com/watch?v=GrugzF0-V3I

http://www.youtube.com/watch?v=GrugzF0-V3I


X1 X2 X3 Y

1 0 0 0

1 0 1 1

1 1 0 1

1 1 1 1

0 0 1 0

0 1 0 0

0 1 1 1

0 0 0 0

X1

X2

X3

Y

Black box

Output

Input

Output Y is 1 if at least two of the three inputs are equal to 1.


X1 X2 X3 Y

1 0 0 0

1 0 1 1

1 1 0 1

1 1 1 1

0 0 1 0

0 1 0 0

0 1 1 1

0 0 0 0

X1

X2

X3

Y

Black box

0.3

0.3

0.3 t=0.4

Output

node

Input

nodes

otherwise0

trueis if1)( where

)04.03.03.03.0( 321

zzI

XXXIY


X1

X2

X3

Y

Black box

w1

t

Output

node

Input

nodes

w2

w3

)( tXwIYi

ii

Perceptron Model

)( tXwsignYi

ii

or


−

++

+

+

+

++

+ +

+

+ −−

−

−

−

−

−

−

− −

−

−

−

+

+

+

+

+

++

+ +

+

+−

−

−

−

−

−

−

−

−

−

−

−

Linearly separable data: Non-linearly separable data:

+

+

+

−

−

−


Activation

function

g(Si )

Si

Oi

I1

I2

I3

wi1

wi2

wi3

Oi

Neuron iInput Output

threshold, t

Input

Layer

Hidden

Layer

Output

Layer

x1

x2

x3

x4

x5

y

Training ANN means learning the weights of

the neurons




• Inputs

input layer

hidden layer

output layer

feed-forward

nonlinear regression


input layer # of hidden

layers each hidden layer output layer

• Normalize

input

• Output

unacceptable

different network topology different set of

initial weights


𝛻𝐸 𝒘 =𝜕𝐸

𝜕𝑤0,𝜕𝐸

𝜕𝑤1, … ,

𝜕𝐸

𝜕𝑤𝑑


114

2

),( i

iii XwfYE


Backpropagation

Iteratively process a set of training tuples & compare the network's

prediction with the actual known target value

minimize the mean squared error

backwards

backpropagation


Stochastic Gradient Descent



• Cross-validation

0E

0E



WEKA provides user control of training

parameters:

• # of hidden layers

• Learning rate

• # of iterations or epochs (“training time”)

• Increment of weight adjustments in back

propogation (“learning rate”)

• Controls on varying changes to increments

(“momentum”) and weight decay



Multilayered Neural Networks require a lot of experimentation to get right…

a lot…

I’m serious…


Hands On


Are they any good?


rare



That’s all for tonight….

Data & Analytics

Barga Data Science lecture 9