Linear Programming

As Used for Discriminant Analysis

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-2

Objectives• Maximize minimum distance from critical

value

• Minimize sum of deviations from critical value– Simple– Direct– Free of statistical assumptions– Flexible

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-3

Requirements to use LP

• LP modeling skills

• Commercial software

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-4

Linear Discriminant Analysis

• Separate data into groups such that– Minimize distance within group– Maximize distance to other groups

• Can have:– Binary (2 groups)– Multiple categories (more than 2 groups)

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-5

Minimize Sum of Deviations (MSD)

Minimize 1 + … + r

Subject to:

A11 x1 + … + A1r xr b + 1 for A1 in B,

…………

An1 x1 + … + Anr xr b - r for An in G,

, … , r 0,

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-6

Maximize Minimum Distance(MMD)

Maximize 1 + … + r

Subject to:

A11 x1 + … + A1r xr b - 1 for A1 in B,

…………

An1 x1 + … + Anr xr b + r for An in G,

1, …, r 0

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-7

ExampleMinimize 1 + 2

Subject to:

6 x1 + 8 x2 b + 1 for A1 in B,

15 x1 + 31 x2 b - 2 for A2 in G,

1 , 2 0

Use b = 9

Optimal solution: x1* = 0, x2* = 0.290323

A1 = 2.35 < 9 so BAD; A2 = 9.00001 > 9 so GOOD

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-8

Perfect SeparationAX* = 9

2.3458409.000013

GoodBad

i

i

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-9

Overlapping Data

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-10

Three-Class Linear Discriminant Analysis

a1

bL1

bU1

bL2

bU2

bL3

bU3

C1

C2

C3X

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-11

MCLP Classification

• Two or more criteria

• Create deviational variables for eachFunctiona + da

- - da+ = Targeta

Objective: Min weighted sum of deviations

IDEAL POINT: all desired deviations = 0

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-12

Fuzzy LP Classification• Not all data precise

• Fuzzy concept:– Membership function 0 ≤ MF ≤ 1– Can have MF for any number of states– 50 degrees

• Cold MF might be 0.7• Warm MF might be 0.4• Hot MF might be 0

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-13

Fuzzy MOLP• Discriminate to various classes available

X-axis is alpha; Y-axis is beta

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-14

Real Application: Credit Card• Outcomes

– Bankruptcy– Good

• Scoring techniques

1. Behavior Score

2. Credit Bureau Scores

3. Proprietary Bankruptcy Score

4. Set Enumeration Decision Tree

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-15

Real Application – Credit Card• LP an alternative to these scoring methods• Classify cardholders in terms of payment• Common variables:

– Balance– Purchase– Payment– Cash advance– State of residence– Job security

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-16

Real Application – Credit Card• FDR model

– 38 original variables over 7 months– 65 derived variables generated

• Separation criteria:– Information value – mean difference/STD– Concordance– Kolmogorov-Smirnov (best)

McGraw-Hill/Irwin ©2007 The McGraw-Hill Companies, Inc. All rights reserved

9-17

Real Application – Credit Cards• Sampled 6,000 records• 2-class output• 65 attributes• 50 LP solutions computed

– Varied fuzzy parameters, setoff limits– Used 1000, 3000, 6000 records– Compared with decision tree, neural network model– MCLP best at not calling actual bad cases good

• But this was on a small test set

– Fuzzy LP best on large test set