Copy

right

R. W

eber

Machine Learning, Data Mining

ISYS370

Dr. R. Weber

Copy

right

R. W

eber

The game

• How did you reason to find the rule?

• According to Michalski (1983) A theory and methodology of inductive learning. In Machine Learning, chapter 4, “inductive learning is a heuristic search through a space of symbolic descriptions (i.e., generalizations) generated by the application of rules to training instances.”

Copy

right

R. W

eber

Learning

• Rote Learning– Learn multiplication tables

• Supervised Learning– Examples are used to help a program identify a concept– Examples are typically represented with attribute-value

pairs– Notion of supervision originates from guidance from

examples• Unsupervised Learning

– Human efforts at scientific discovery, theory formation

Copy

right

R. W

eber

Inductive Learning

• Learning by generalization• Performance of classification tasks

– Also categorization• Rules indicate categories• Goal:

– Characterize a concept

Copy

right

R. W

eber

•Learner uses:–positive examples (instances ARE examples of

a concept) and –negative examples (instances ARE NOT

examples of a concept)

Concept Learning is a Form of Inductive Learning

Copy

right

R. W

eber

• Needs empirical validation• Dense or sparse data determine quality

of different methods

Concept Learning

Copy

right

R. W

eber

• The learned concept should be able to correctly classify new instances of the concept– When it succeeds in a real instance of the

concept it finds true positives – When it fails in a real instance of the concept

it finds false negatives

Validation of Concept Learning i

Copy

right

R. W

eber

• The learned concept should be able to correctly classify new instances of the concept– When it succeeds in a counterexample it

finds true negatives– When it fails in a counterexample it finds

false positives

Validation of Concept Learning ii

Copy

right

R. W

eber

Rule Learning

• Learning widely used in data mining• Version Space Learning is a search

method to learn rules• Decision Trees

Copy

right

R. W

eber

Version Space ii

• Creates tree that includes all possible combinations• Does not learn for rules with disjunctions• Incremental method, trains additional data without

the need to retrain all data

A=1

B=0B=1 B=.5

C=.5 C=.3C=1 C=.5

C=.3C=1

C=.5

C=.3C=1

Outcome ?Outcome ?Outcome ?Outcome ?Outcome ?Outcome ?Outcome ?

Copy

right

R. W

eber

Decision trees

• Knowledge representation formalism• Represent mutually exclusive rules

(disjunction)• A way of breaking up a data set into classes

or categories• Classification rules that determine, for each

instance with attribute values, whether it belongs to one or another class

• Not incremental

Decision treesconsist of:-leaf nodes (classes)

- decision nodes (tests on attribute values)

-from decision nodes branches grow for each possible outcome of the test

From Cawsey, 1997

Copy

right

R. W

eber

Decision tree induction

• Goal is to correctly classify all example data• Several algorithms to induce decision trees:

ID3 (Quinlan 1979) , CLS, ACLS, ASSISTANT, IND, C4.5

• Constructs decision tree from past data• Attempts to find the simplest tree (not

guaranteed because it is based on heuristics)

Copy

right

R. W

eber

•From:– a set of target classes–Training data containing objects of more than one class

•ID3 uses test to refine the training data set into subsets that contain objects of only one class each•Choosing the right test is the key

ID3 algorithm

Copy

right

R. W

eber

• Information gain or ‘minimum entropy’• Maximizing information gain corresponds to minimizing entropy•Predictive features (good indicators of the outcome)

How does ID3 chooses tests

Copy

right

R. W

eber

• Information gain or ‘minimum entropy’• Maximizing information gain corresponds to minimizing entropy•Predictive features (good indicators of the outcome)

Choosing tests

Copy

right

R. W

eber

Clustering

• Data analysis method applied to data• Data should naturally possess groupings• Goal: group data into clusters• Resulting clusters are collections where objects within a

cluster are similar to each other• Objects outside the cluster are dissimilar to objects

inside• Objects from one cluster are dissimilar to objects in other

clusters • Distance measures are used to compute similarity

• http://movielens.umn.edu/main

Copy

right

R. W

eber

Explanation-based learning

• Incorporates domain knowledge into the learning process

• Feature values are assigned a relevance factor if their values are consistent with domain knowledge

• Features that are assigned relevance factors are considered in the learning process

Copy

right

R. W

eber

Familiar Learning Task

• Learn relative importance of features• Goal: learn individual weights• Commonly used in case-based reasoning• Methods include a similarity measure to get

feedback about verify their relative importance: feedback methods

• Search methods: gradient descent• ID3

Copy

right

R. W

eber

Esteem CBR shell has option to use Gradient

Descent to learn weights:

Copy

right

R. W

eber

• Link analysis

• Deviation detection

Data mining tasks ii

Rules: • Association generation• Relationships between entities

• How things change over time, trends

Copy

right

R. W

eber

KDD applications• Fraud detection

– Telecom (calling cards, cell phones)– Credit cards– Health insurance

Loan approval Investment analysis Marketing and sales data analysis

Identify potential customers Effectiveness of sales campaign Store layout

Copy

right

R. W

eber

Text mining

The problem starts with a query and the solution is a set of information (e.g., patterns, connections, profiles, trends) contained in several different texts that are potentially relevant to the initial query.

Copy

right

R. W

eber

Text mining applications

• IBM Text Navigator– Cluster documents by content;– Each document is annotated by the 2 most

frequently used words in the cluster;

• Concept Extraction (Los Alamos)– Text analysis of medical records;– Uses a clustering approach based on trigram

representation;– Documents in vectors, cosine for comparison;

rule-based ES

case-based reasoning

inductive ML, NN

algorithms

deductive reasoning

analogical reasoning

inductive reasoning

search

Problem-solving method

Reasoning type