Data Preprocessing - University of Iowauser.engineering.uiowa.edu/~coneng/lectures/Lecture_DM...2 The University of Iowa Intelligent Systems Laboratory Data Mining Software •Enterprise-level:

1

The University of Iowa Intelligent Systems Laboratory

Data Mining(Knowledge discovery in database)


What is Data Mining?

• Data Mining: "The non trivial extraction of implicit, previously unknown,

and potentially useful information from data"

– William J Frawley, Gregory Piatetsky-Shapiro and Christopher J Matheus

• Data mining finds valuable information hidden in large volumes of data.

• Data Mining process involves:

– Databases

– Statistics

– Machine Learning

– High Performance Computing

– Visualization

– Mathematics


Data mining: Basic steps

Feature

Produ

ctP

rodu

ctP

rodu

ctP

rodu

ctP

rodu

ctP

rodu

ctP

rodu

ctP

rodu

ctP

rodu

ctP

rodu

ct

FeatureFeatureFeatureFeatureFeatureFeatureFeatureFeatureFeature

Selection

Preprocessing

Mining

Data

Selected

Data

Preprocessed

Data

Knowledge

Classification,

clustering, association

analysis etc.

Handling missing

values, outlier

removal, labeling

output, feature

selection

Data sampling


Mining tasks

• Classification: YES, NO

• Regression: Predict the actual output value

• Clustering: „Grouping identical items‟

• Association rules: Identifying association among input

attributes.

• Anomaly detection: Detect deviation from normal behavior

{Milk} --> {Coke}, {Diaper, Milk} --> {Beer}

Fraud detection, Network intrusions

Recommender systems e.g. Netflix

Weather forecasting

Patient diagnosis

2


Data Mining Software

• Enterprise-level: (US $10,000 and more)– Fair Isaac, IBM, Insightful, KXEN, Oracle, SAS, and SPSS

• Department-level: (from $1,000 to $9,999)– Angoss, CART/MARS/TreeNet/Random Forests, Equbits,

GhostMiner, Gornik, Mineset, MATLAB, Megaputer, Microsoft

SQL Server, Statsoft Statistica, ThinkAnalytics

• Personal-level: (from $1 to $999):– Excel, See5, MATLAB

• Free:– C4.5, R, Weka, Xelopes


Polling results


Data Mining: WEKA


Outline

•Data preparation

•Preprocessing and “arff” files

•Filters, classifiers, and visualization

•Attribute selection

•Training and testing

•Quality measurements

•Interpretation of results

3


Data Mining Procedures

• Prepare the data into desired formats

• Preprocess the data if necessary

• Select different algorithms based on

application or domain expertise

• Evaluate the results and repeat experiments

again if necessary


Data Sets


Arff file

@relation weather@attribute No real@attribute outlook {sunny,overcast,rainy}@attribute temperature real@attribute humidity real@attribute windy {TRUE,FALSE}@attribute play {yes,no}


Arff file

4


Arff file


Arff file


WEKA


WEKA Explorer

5


WEKA


WEKA


Parameter Distribution


Filters

6


Filters


Filters


Filters


Filters

7


Filters


Classifier


Classifier


Classifiers

8


Classifiers


Classifiers


Decision Tree


Decision Tree

9


Decision Tree


Decision Tree


Decision Tree


Decision Tree

10


Classifier: PART


Classifier: PART


Classifier: PART


Classifier: PART

11


Neural Networks


Neural Networks


Neural Networks


Association Rules

12


Association Rules


Association Rules


Visualization


Visualization

13


Clustering


Clustering


Clustering


Test Data Analysis

14


Test Data Analysis


Test Data Analysis


Test Data Analysis


Test Data Analysis

15


Test Data Analysis


Test Data Analysis


Test Data Analysis


Test Data Analysis

16


UCI repository

•http://archive.ics.uci.edu/ml/index.html
http://archive.ics.uci.edu/ml/index.html

Documents

Data Preprocessing - University of Iowauser.engineering.uiowa.edu/~coneng/lectures/Lecture_DM...2 The University of Iowa Intelligent Systems Laboratory Data Mining Software •Enterprise-level: