Upload
calvin-bruce
View
214
Download
2
Embed Size (px)
Citation preview
Data Mining – Best PracticesPart #2
Richard Derrig, PhD,
Opal Consulting LLC
CAS Spring Meeting
June 16-18, 2008
Data Mining
Data Mining, also known as Knowledge-Discovery in Databases (KDD), is the process of automatically searching large volumes of data for patterns. In order to achieve this, data mining uses computational techniques from statistics, machine learning and pattern recognition.
www.wikipedia.org
AGENDA
Predictive v Explanatory ModelsDiscussion of Methods
Example: Explanatory Models for Decision to Investigate Claims
The “Importance” of Explanatory and Predictive Variables
An Eight Step Program for Buildinga Successful Model
Predictive v Explanatory Models
Both are of the form: Target or Dependent Variable is a Function of Feature or Independent Variables that are related to the Target Variable
Explanatory Models assume all Variables are Contemporaneous and Known
Predictive Models assume all Variables are Contemporaneous and Estimable
Desirable Properties of a Data Mining Method:
Any nonlinear relationship between target and features can be approximated
A method that works when the form of the nonlinearity is unknown
The effect of interactions can be easily determined and incorporated into the model
The method generalizes well on out-of sample data
Major Kinds of Data Mining Methods Supervised learning
Most common situation Target variable
Frequency Loss ratio Fraud/no fraud
Some methods Regression Decision Trees Some neural networks
Unsupervised learning No Target variable Group like records
together-Clustering A group of claims with
similar characteristics might be more likely to be of similar risk of loss
Ex: Territory assignment,
Some methods PRIDIT K-means clustering Kohonen neural networks
The Supervised Methods and Software Evaluated
1) TREENET 7) Iminer Ensemble
2) Iminer Tree 8) MARS
3) SPLUS Tree 9) Random Forest
4) CART 10) Exhaustive Chaid
5) S-PLUS Neural 11) Naïve Bayes (Baseline)
6) Iminer Neural 12) Logistic reg ( (Baseline)
Decision Trees
In decision theory (for example risk management), a decision tree is a graph of decisions and their possible consequences, (including resource costs and risks) used to create a plan to reach a goal. Decision trees are constructed in order to help with making decisions. A decision tree is a special form of tree structure.
www.wikipedia.org
CART – Example of 1st split on Provider 2 Bill, With Paid as Dependent
For the entire database, total squared deviation of paid losses around the predicted value (i.e., the mean) is 4.95x1013. The SSE declines to 4.66x1013 after the data are partitioned using $5,021 as the cutpoint.
Any other partition of the provider bill produces a larger SSE than 4.66x1013. For instance, if a cutpoint of $10,000 is selected, the SSE is 4.76*1013.
1st Split
All Data
Mean = 11,224
Bill < 5,021
Mean = 10,770
Bill>= 5,021
Mean = 59,250
Different Kinds of Decision Trees
Single Trees (CART, CHAID)Ensemble Trees, a more recent development
(TREENET, RANDOM FOREST) A composite or weighted average of many trees
(perhaps 100 or more) There are many methods to fit the trees and prevent
overfitting Boosting: Iminer Ensemble and Treenet Bagging: Random Forest
Neural Networks
=
Three Layer Neural Network
Input Layer Hidden Layer Output Layer(Input Data) (Process Data) (Predicted Value)
Self-Organizing Feature Maps T. Kohonen 1982-1990 (Cybernetics) Reference vectors of features map to OUTPUT
format in topologically faithful way. Example: Map onto 40x40 2-dimensional square.
Iterative Process Adjusts All Reference Vectors in a “Neighborhood” of the Nearest One. Neighborhood Size Shrinks over Iterations
NEURAL NETWORKSNEURAL NETWORKS
FEATURE MAPSUSPICION LEVELS
1 4 7 10 13 16S1
S4
S7
S10
S13
S16
4-5
3-4
2-3
1-2
0-1
FEATURE MAPSIMILIARITY OF A CLAIM
1 5 9
13
17
S1
S4
S7
S10
S13
S16
4-5
3-4
2-3
1-2
0-1
DATA MODELING EXAMPLE: CLUSTERING
Data on 16,000 Medicaid providers analyzed by unsupervised neural net
Neural network clustered Medicaid providers based on 100+ features
Investigators validated a small set of known fraudulent providers
Visualization tool displays clustering, showing known fraud and abuse
Subset of 100 providers with similar patterns investigated: Hit rate > 70% Cube size proportional to annual Medicaid revenues
© 1999 Intelligent Technologies Corporation
Multiple Adaptive Regression Splines (MARS)
MARS fits a piecewise linear regression BF1 = max(0, X – 1,401.00) BF2 = max(0, 1,401.00 - X ) BF3 = max(0, X - 70.00) Y = 0.336 + .145626E-03 * BF1 - .199072E-03 * BF2
- .145947E-03 * BF3; BF1 is basis function BF1, BF2, BF3 are basis functions
MARS uses statistical optimization to find best basis function(s)
Basis function similar to dummy variable in regression. Like a combination of a dummy indicator and a linear independent variable
Baseline Methods:Naive Bayes Classifier
Logistic Regression
Naive Bayes assumes feature (predictor) variables) independence conditional on each category
Logistic Regression assumes target is linear in the logs of the feature (predictor) variables
REAL CLAIM FRAUDDETECTION PROBLEM
Classify all claimsIdentify valid classes
Pay the claim No hassle Visa Example
Identify (possible) fraud Investigation needed
Identify “gray” classes Minimize with “learning” algorithms
The Fraud Surrogates used as Target Decision Variables
Independent Medical Exam (IME) requested
Special Investigation Unit (SIU) referralIME successfulSIU successfulDATA: Detailed Auto Injury Closed Claim
Database for Massachusetts Accident Years (1995-1997)
DMDatabases
Scoring Functions
Graded Output
Non-Suspicious ClaimsRoutine Claims
Suspicious ClaimsComplicated Claims
ROC Curve Area Under the ROC Curve
Want good performance both on sensitivity and specificity
Sensitivity and specificity depend on cut points chosen for binary target (yes/no) Choose a series of different cut points, and
compute sensitivity and specificity for each of them
Graph resultsPlot sensitivity vs 1-specifityCompute an overall measure of “lift”, or area
under the curve
True/False Positives and True/False Negatives: The “Confusion” Matrix
Choose a “cut point” in the model score.Claims > cut point, classify “yes”.
Sample Confusion Matrix: Sensitivity and Specificity
Prediction No Yes Row TotalNo 800 200 1,000 Yes 200 400 600 Column Total 1,000 600
True Class
Correct Total Percent CorrectSensitivity 800 1,000 80%Specificity 400 600 67%
TREENET ROC Curve – IMEAUROC = 0.701
Logistic ROC Curve – IMEAUROC = 0.643
Ranking of Methods/Software – IME Requested
Method/Software AUROC Lower Bound Upper BoundRandom Forest 0.7030 0.6954 0.7107Treenet 0.7010 0.6935 0.7085MARS 0.6974 0.6897 0.7051SPLUS Neural 0.6961 0.6885 0.7038S-PLUS Tree 0.6881 0.6802 0.6961Logistic 0.6771 0.6695 0.6848Naïve Bayes 0.6763 0.6685 0.6841SPSS Exhaustive CHAID 0.6730 0.6660 0.6820CART Tree 0.6694 0.6613 0.6775Iminer Neural 0.6681 0.6604 0.6759Iminer Ensemble 0.6491 0.6408 0.6573Iminer Tree 0.6286 0.6199 0.6372
Variable Importance (IME) Based on Average of Methods
Important Variable Summarizations for IME Tree Models, Other Models and Total
Total Score
Tree Score
Other Score
Variable Variable type
Total Score Rank Rank Rank
Health Insurance F 16529 1 2 1 Provider 2 Bill F 12514 2 1 3 Injury Type F 10311 3 3 2 Territory F 5180 4 4 7 Provider 2 Type F 4911 5 6 4 Provider 1 Bill F 4711 6 5 5 Attorneys Per Zip DV 2731 7 7 14 Report Lag DV 2650 8 10 8 Treatment Lag DV 2638 9 13 6 Claimant per City DV 2383 10 12 9 Provider 1 Type F 1794 11 9 13 Providers per City DV 1708 12 11 11 Attorney F 1642 13 8 16 Distance MP1 Zip to Clt Zip DV 1134 14 18 10 AGE F 1048 15 17 12 Avg. Household Price/Zip DM 907 16 16 15 Emergency Treatment F 660 17 14 18 Income Household/Zip DM 329 18 15 20 Providers/Zip DV 288 19 20 17 Household/Zip DM 242 20 19 19 Policy Type F 4 21 21 21
Claim Fraud Detection Plan
STEP 1:SAMPLE: Systematic benchmark of a random sample of claims.
STEP 2:FEATURES: Isolate red flags and other sorting characteristics
STEP 3:FEATURE SELECTION: Separate features into objective and subjective, early, middle and late arriving, acquisition cost levels, and other practical considerations.
STEP 4:CLUSTER: Apply unsupervised algorithms (Kohonen, PRIDIT, Fuzzy) to cluster claims, examine for needed homogeneity.
Claim Fraud Detection Plan STEP 5:ASSESSMENT: Externally classify claims according
to objectives for sorting. STEP 6:MODEL: Supervised models relating selected
features to objectives (logistic regression, Naïve Bayes, Neural Networks, CART, MARS)
STEP7:STATIC TESTING: Model output versus expert assessment, model output versus cluster homogeneity (PRIDIT scores) on one or more samples.
STEP 8:DYNAMIC TESTING: Real time operation of acceptable model, record outcomes, repeat steps 1-7 as needed to fine tune model and parameters. Use PRIDIT to show gain or loss of feature power and changing data patterns, tune investigative proportions to optimize detection and deterrence of fraud and abuse.