Quiz 1 review

Evaluating Classifiers

Reading: T. Fawcett paper, link on class website,

Sections 1-4

Optional reading: Davis and Goadrich paper, link on class

website

“Confusion matrix” for a given class c

Actual Predicted (or “classified”) True

False(in class c)

(not in class c)

True (in class c) TruePositiveFalseNegative

False (not in class c) FalsePositive TrueNegative

Evaluating classification algorithms

Evaluating classification algorithms

• Accuracy: Fraction of correct answers out of all problems

• Precision: Fraction of true positives out of all predicted positives:

• Recall: Fraction of true positives out of all actual positives:

€

P =TP

TP + FP

€

R =TP

TP + FN

Trading off precision against recall

.

.

.

w1

w2

w64

o

w0

+1x1

x2

x64

€

class(x) = y(x) = sgn(w0 +w1x1 +w2x2 + ...+wnxn )

where sgn(z) =−1 if y < 0 0 if y = 0

+1 if y > 0

⎧ ⎨ ⎪

⎩ ⎪

How can we improve precision (at the expenseof recall) with a fixed classifier?

€

P =TP

TP +FP

€

R =TP

TP + FN

True False(“8”) (“0”)

True (“8”) 40 10

False (“0”) 30 120

True False(“8”) (“0”)

True (“8”) ? ?

False (“0”) ? ?

Old, with threshold of 0

New, with threshold of:

-∞

Example 1: Assume 200 sample digits, of which 50 have class “8”

Precision? Recall?

Actual

Actual

Predicted

Predicted

€

P =TP

TP + FP

€

R =TP

TP + FN

True False(“8”) (“0”)

True (“8”) 40 10

False (“0”) 30 120

True False(“8”) (“0”)

True (“8”) ? ?

False (“0”) ? ?

Old, with threshold of 0

New, with threshold of:

+∞

Precision? Recall?

Actual

Actual

Predicted

Predicted

Example 2: Assume 200 sample digits, of which 50 have class “8”

€

P =TP

TP + FP

€

R =TP

TP + FN

€

P =TP

TP + FP

€

R =TP

TP + FN

Results of classifier

Threshold Accuracy Precision Recall

.9

.8

.7

.6

.5

.4

.3

.2

.1

-∞

Creating a Precision/Recall Curve

9

(“sensitivity”)(1 “specificity”)

10

€

True Positive Rate (= Recall) =TP

TP + FN

€

False Positive Rate =FP

TN + FP

Results of classifier

Threshold Accuracy TPR FPR

.9

.8

.7

.6

.5

.4

.3

.2

.1

-∞

Creating a ROC Curve

12

Precision/Recall versus ROC curves

http://blog.crowdflower.com/2008/06/aggregate-turker-judgments-threshold-calibration/

13

14

15