Logistic RegressionWeek 3 – Soft ComputingBy Yosi Kristian.

Classificationo Email: Spam / Not Spam?o Online Transactions: Fraudulent (Yes / No)?o Tumor: Malignant / Benign ?

0: “Negative Class” (e.g., benign tumor)

1: “Positive Class” (e.g., malignant tumor)

Using Linear Regression ( fail )

Tumor Size

Threshold classifier output at 0.5:

If , predict “y = 1”

If , predict “y = 0”

Tumor Size

Malignant ?

(Yes) 1

(No) 0

0,5

Other Reason

Classification: y = 0 or 1

Linear regression can be > 1 or < 0

Logistic Regression:

Although it has the term regression, Logistic Regression is actually a Classification Algorithm, the name was giver only for historical reasons.

HYPOTHESISREPRESENTATION

Logistic Regression Model

Sigmoid function / Logistic function

Want

1

0.5

0

Interpretation of Hypothesis Output

= estimated probability that y = 1 on input x

Tell patient that 70% chance of tumor being malignant

Example: If

“probability that y = 1, given x, parameterized by ”

DECISION BOUNDARY

Boundaries Logistic regression

Suppose predict “ “ if

predict “ “ if

z

1

Decision Boundary

x1

x2

1 2 3

1

2

3

Predict “ “ if

Non-linear decision boundaries

x1

x2

Predict “ “ if

x1

x2

1-1

-1

1

COST FUNCTION

Linear Regression Cost function (fail)

Linear regression:

“non-convex” “convex”

Logistic regression cost function

If y = 1

10

Logistic regression cost function

If y = 0

10

SIMPLIFIED COST FUNCTION AND GRADIENT DESCENT

Logistic regression cost function

Logistic regression cost function

Output

To fit parameters :

To make a prediction given new :

Gradient Descent

Want :

Repeat

(simultaneously update all )

Gradient Descent

Want :

(simultaneously update all )

Repeat

Algorithm looks identical to linear regression!