Upload
lopanath
View
231
Download
0
Embed Size (px)
Citation preview
8/2/2019 Chapter 9 Negnevitsky
1/72
Knowledge Engineering and Data Mining
Jiajuan LinJohn LarimerLopa NathNick RemishSean Ruck
Tyler Purdom
8/2/2019 Chapter 9 Negnevitsky
2/72
What Is Knowledge Engineering?
John Larimer
8/2/2019 Chapter 9 Negnevitsky
3/72
Knowledge Engineering
It is the process of building intelligent systems.
1. Problem Assessment
2. Data & knowledge acquisition
3. Prototype
4. Complete System
5. Evaluation & Revise6. Integration & Maintain System
8/2/2019 Chapter 9 Negnevitsky
4/72
Knowledge Engineering
1. Problem Assessment
Determine the problem's characteristics Identify the main participants in the project
Specify the project's objectives
Determine the resources need for building the systemTypes of problems: Diagnosis, selection, prediction,classification, clustering, optimization, control
E.g. Diagnosis -> domain knowledge, explanation facilities
8/2/2019 Chapter 9 Negnevitsky
5/72
Knowledge Engineering
2. Data and Knowledge Acquisition
Collect and analyse data and knowledge Make key concepts of the system design more explicit
Intelligent System
8/2/2019 Chapter 9 Negnevitsky
6/72
Knowledge Engineering
3. Development of a Prototype System
Choose a tool for building an intelligent system Transform data and represent knowledge Design and implement a prototype
Test the prototype
8/2/2019 Chapter 9 Negnevitsky
7/72
Knowledge Engineering
4. Development of a complete system
Prepare a detailed design for a full-scale system Collect additional data and knowledge Develop the user interface
Implement the complete System
8/2/2019 Chapter 9 Negnevitsky
8/72
Knowledge Engineering
5. Evaluation and revision of the system
Evaluate the system against the performance criteria Revise the system as necessary
8/2/2019 Chapter 9 Negnevitsky
9/72
Knowledge Engineering
6. Integration and Maintenance of the System
Make arrangements for technology transfer Establish an effective maintenance program
8/2/2019 Chapter 9 Negnevitsky
10/72
Case Study 1: Diagnostic Expert System
I want to develop an Intelligent system that can help me to fix malfunctions of my Mac computer.
Will an expert system work for this problem?
8/2/2019 Chapter 9 Negnevitsky
11/72
Case Study 1: Diagnostic Expert System
Phone Call Rule (Firebaugh, 1988) - "Any problem that can be solved by your in-house expert in a 10-30 minute phone call can be developed in an expert system."
Troubleshooting manuals Series of visual inspections Rule structure with domain knowledge
8/2/2019 Chapter 9 Negnevitsky
12/72
Case Study 1: Diagnostic Expert System
Taken from N. Ch9 Pg 309 & 310
8/2/2019 Chapter 9 Negnevitsky
13/72
8/2/2019 Chapter 9 Negnevitsky
14/72
Case Study 2: Classification E.S.
Problem: Identify different classes of sailboat (typical classification problem)o Handled well by both expert systems and NNs
Collect informationo In this case, the sail plan can help identify the class of boat.
Issues with the expert system approacho What if the information is incomplete or inexact? (Rough weather obscuring sails)
Manage incrementally acquired evidence with certainty factors.
8/2/2019 Chapter 9 Negnevitsky
15/72
9.3 Will a fuzzy expert system work for my problem?
A Fuzzy Solution?o Useful when you cannot define a set of exact rules.o Great for inherent imprecise properties and modeling human decision making.
Sometimes parameters are imprecise (a doctor dealing with a patient)o Mainly used in engineering, but has applications in any sector that relies on human
experience or is too complex or uncertain (ex: finance)
8/2/2019 Chapter 9 Negnevitsky
16/72
Case Study 3: Decision-Support Fuzzy System
8/2/2019 Chapter 9 Negnevitsky
17/72
Decision-Support Fuzzy System
Problem: assessing mortgage applicationso Use a Decision-Support Fuzzy System
Steps:o Represent the concept in fuzzy termso Implement the concept in a prototypeo Test and optimize
8/2/2019 Chapter 9 Negnevitsky
18/72
Decision-Support Fuzzy System
Represent the concept in fuzzy terms:
Triangular and trapezoidal fuzzy membership functions are used to represent knowledge.
8/2/2019 Chapter 9 Negnevitsky
19/72
Decision-Support Fuzzy System
Obtain the fuzzy rules:Based off of Von Altrock's
fuzzy model and appliedto mortgages.
8/2/2019 Chapter 9 Negnevitsky
20/72
Decision-Support Fuzzy System
Evaluate and analyse performance:
Despite having 100+ rules, decision-support fuzzy systems can be developed, tested andimplemented relatively quickly.
8/2/2019 Chapter 9 Negnevitsky
21/72
9.4 Will a neural network work for my problem?
Sean Ruck
8/2/2019 Chapter 9 Negnevitsky
22/72
Neural Network Overview
Very powerful, general purpose tools Successfully applied to prediction, classification, and clustering problems Quite popular due to the versatility of neural networks
8/2/2019 Chapter 9 Negnevitsky
23/72
Case Study 4: Character Recognition Neural Networks
Suppose you want to copy a document onto your computer without retyping the whole thing.o How?
Optical Character Recognitiono The ability of a computer to translate character images into a text file using software
o Capture the character images by scanning the document Converts the scanned document into a bit map
8/2/2019 Chapter 9 Negnevitsky
24/72
Choosing The Neural Network Architecture
Architecture and size of neural newtork dependent upon complexity of the problemo Handwritten character recognition is far more complex than computer printed
A 3-layer network will suffice for printed digit recognition
8/2/2019 Chapter 9 Negnevitsky
25/72
Determining an optimal number of hidden neurons
More neurons leads to a more accurate network, but takes longer to train Too many neurons may actually prevent the network from generalising or working for anything
other than training exampleso Overfitting
How to prevent overfittingo Choose the smallest number of neurons that give good results and generalisation
8/2/2019 Chapter 9 Negnevitsky
26/72
cont'd.
We should test out the training of the network with various numbers of hidden neuronso Performance rated by sum of squared errors
The training runs that have a good enough sum of squared errors result have anumber of hidden neurons to consider using
8/2/2019 Chapter 9 Negnevitsky
27/72
Test Examples?
The test set should be entirely independent of the training exampleso Only use the training runs that passed the previous test
Test examples should also contain "noise"o Distortion of the input
The training runs that give us a reasonable error in recognition even with noise have a goodenough number of hidden neurons to useo Use the lowest number for practical purposes
8/2/2019 Chapter 9 Negnevitsky
28/72
Improving Performance
A neural network is only as good as the examples used to train it Improve the network by training it with noisy examples
8/2/2019 Chapter 9 Negnevitsky
29/72
Case Study 5: Prediction Neural Networks
Neural networks are useful in prediction situations such as predicting the market value of a house Using a neural network creates a black box around how the results were reached
o The result is more important than the how anyway
For prediction training examples are critically importanto We need a wide array of examples to cover all possible inputs
8/2/2019 Chapter 9 Negnevitsky
30/72
Determining The Size Of A Training Set
Can be estimated with "Widrow's rule of Thumb": N = nw/eo Where N is the number of training examples, nw is the number of synaptic weights in the
network, and e is the network error permitted
8/2/2019 Chapter 9 Negnevitsky
31/72
Dealing With The Data
Neural networks work best with inputs in the 0 to 1 range, but in cases such as with determiningthe value of a house, our inputs are not all in that rangeo Number of bedrooms, square footage, etc.
So we need to "massage" the data to this rangeo massaged value = (actual value - minimum value) / (maximum value - minimum value)o Good for up to a dozen possible values
8/2/2019 Chapter 9 Negnevitsky
32/72
cont'd
We can also utilize 1 of N codingo Each possible value is taken as its own input each with a value of 0 or 1
8/2/2019 Chapter 9 Negnevitsky
33/72
Dealing With The Results
To validate the results we test the network with never before seen examples, as before Our network is working with values between 0 and 1. We need to convert back to actual values
o We can reverse the "massaging" we did before
To test the importance of certin inputs we can test the network's sensitivity to them: "SensitivityAnalysis"o Set each input one at a time to its minimum and maximum values and measure the results
8/2/2019 Chapter 9 Negnevitsky
34/72
Case Study 6: Classification Neural Networks With Competitive Learning
Using a neural network we can discover significant features of input patterns and separate thedata into different classes
Using competitive learning a single layer neural network can perform clusteringo Combining similar data into groups or clusterso Uses 1 input neuron for each input and 1 competitive neuron for each cluster
8/2/2019 Chapter 9 Negnevitsky
35/72
When Is The Learning Process Completed?
In a competitive neural network, there is no obvious way to know if the network is done learningo We do not know the desired output, so we cannot use the sum of squared errors
Use Euclidean Distance criterion insteado When there has been no noticeable change in the weights of the competitive neurons, the
network can be considered to have converged
8/2/2019 Chapter 9 Negnevitsky
36/72
How Can We Associate Neurons to Specific Classes or Clusters?
Competitive neural networks enable us to identify clusters in input data, but does nothing to labelthe clusterso We can connect a competitive neuron with a cluster/class by analyzing its weights
We can identify exactly which cluster is which by feeding the network test data corresponding toone particular cluster
o The output neuron that most often is utilized is labeled as that class
8/2/2019 Chapter 9 Negnevitsky
37/72
9.5 Will genetic algorithms work for my problem?
Lopa Nath
8/2/2019 Chapter 9 Negnevitsky
38/72
Genetic Algorithm Review
Most applicable to optimization problemso Process of finding a better solution to a problem
More than one solution not of equal quality
Generates a population of competing candidate solutions Causes candidates to evolve through process of natural selection
o Poor solutions die out while better solutions survive and reproduce
Process repetition breeds an optimal solution
8/2/2019 Chapter 9 Negnevitsky
39/72
Case Study 7: The Traveling Salesman Problem
I want to develop an intelligent system that can produce anoptimal itinerary. I am going to travel by car and I want to visitall major cities in Western and Central Europe and then returnhome. Will a genetic algorithm work for this problem?o Known as the traveling salesman problem (TSP)
o Given a finite number of cities, and the cost of travel (or the distance) betweeneach pair of cities, we need to find the cheapest way (or shortest route) forvisiting each city exactly once and returning to the starting point.
o TSP naturally represented in numerous transportation and logisticsapplications.
Arranging routes, scheduling drilling of holes in a circuit board (timeefficient - shortest distance)
Although we can not be completely sure if the selected routeis the best one, after several runs we can be sure that theroute obtained is a good one.
8/2/2019 Chapter 9 Negnevitsky
40/72
How does a genetic algorithm solve the TSP?
Representation
Chromosome where order of integersrepresents order in which cities will bevisited.
8/2/2019 Chapter 9 Negnevitsky
41/72
Genetic Operators in the TSP
Genetic operators used to create new routes Crossover Operatoro Classical form cannot be directly applied because a simple exchange
of parts between parents would contain duplicates and omissions.
Clearly classical crossover with single crossover point does not work.
8/2/2019 Chapter 9 Negnevitsky
42/72
How the Crossover Operator Works
8/2/2019 Chapter 9 Negnevitsky
43/72
Genetic Operators in the TSP Continued
Mutation Operatoro Reciprocal Exchange
Simply swaps two randomly selected cities in the chromosome
o Inversion Selects two random points along the chromosome string and reverses order of cities
between these points
8/2/2019 Chapter 9 Negnevitsky
44/72
Fitness Function in the TSP
Evaluate total length of the routeo Fitness of each individual chromosome is determined as the reciprocal of the route length
Shorter the route, fitter the chromosome
8/2/2019 Chapter 9 Negnevitsky
45/72
9.6 Will a hybrid intelligent system work for my problem?
8/2/2019 Chapter 9 Negnevitsky
46/72
Hybrid Intelligent Systems
Solving complex real-world problems require an application of complex intelligent systems thatcombine the advantages of expert systems, fuzzy logic, neural networks, and evolutionarycomputation.
Such systems can integrate human-like expertise in a specific domain with abilities to learn andadapt to a rapidly changing environment.
8/2/2019 Chapter 9 Negnevitsky
47/72
Case Study 8: Neuro-fuzzy decision-support systems
I want to develop an intelligent system for diagnosing myocardialperfusion from cardiac images. I have a set of cardiac images aswell as the clinical notes and physician's interpretation. Will ahybrid system work for this problem?o Analysis of two SPECT images must be done
One stress image taken 10-15 minutes after injection with radioactive tracer
One rest image taken 2-5 hours after the injectiono Brighter patches on image correspond to well-perfused areas while darker patches
may indicate the presence of an ischemia.
o Visual inspection is highly subjective--intelligent system can help a cardiologistdiagnose.
o One binary feature assigns an overall diagnosis--normal or abnormal
The neuro-fuzzy system in this example has a heterogeneous
structure - the neural network and fuzzy system will work asindependent components but cooperate in solving the problem.
8/2/2019 Chapter 9 Negnevitsky
48/72
Back-Propagation Neural Network to Classifythe SPECT Images into Normal and Abnormal
Each image is divided into 22 regions, so we need 44 input neurons. Since SPECT images are to be classified as either normal or abnormal, we should use two output
neurons.
Good generalization in this study can be obtained with 5 to 7 neurons in the hidden layer.
8/2/2019 Chapter 9 Negnevitsky
49/72
Testing the Neural Network
Testing the network, we find the network's performance is rather pooro 25% normal are misclassified as abnormalo Over 35% abnormal are misclassified as normalo Indicates that the training set may lack some important examples
Can improve this still
8/2/2019 Chapter 9 Negnevitsky
50/72
Neural Network Output
Two outputso First - possibility that the SPECT image belongs to class
normalo Second - possibility that the SPECT image belongs to class
abnormal Examples:
o NORMAL OUTPUT HIGH AND ABNORMAL OUTPUT LOWFirst (normal) output is 0.92 and second (abnormal) is 0.16 -image classified as normal - risk for heart attack is low
o NORMAL OUTPUT LOW AND ABNORMAL OUTPUT HIGHFirst (normal) output is 0.17 and second (abnormal) is 0.51 -image classified as abnormal 0 risk for heart attack is high
o BOTH OUTPUTS ARE CLOSE First (normal) output is 0.51and second (abnormal) is 0.49 - we cannot confidently classifythe image.
8/2/2019 Chapter 9 Negnevitsky
51/72
Adding Fuzzy Logic for Decision-Making in Medical Diagnosis
Fuzzy logic provides us with a means of modeling how thecardiologist asses the risk of a heart attack. Need to determine input and output variables, define fuzzy sets,
and construct fuzzy rules.o Two inputs (NN output 1 and NN output 2) and one output (the risk of
a heart attack).
Inputs [0, 1] and output vary between 0 and 100 percent.o Fuzzy sets shown in Negnevitsky page 342 and 343 - Figure 9.33,
Figure 9.34, and Figure 9.35o Fuzzy rules in Negnevitsky page 343 - Figure 9.36
Examples:1. If (NN-output1 is Low) and (NN_output2 is Low) then (Risk is Moderate)
2. If (NN-output1 is Low) and (NN_output2 is Medium) then (Risk is High)3. If (NN-output1 is Low) and (NN_output2 is High) then (Risk is Very_High)4. If (NN-output1 is Medium) and (NN_output2 is Low) then (Risk is Low)
8/2/2019 Chapter 9 Negnevitsky
52/72
More Certainty
Risk between 30 and 50 percent cannot be classified as either normal or abnormal - uncertain. Apply the following heuristics known by experienced cardiologists to all corresponding regions (22in each image)
1. If perfusion inside region iat stress is higher than perfusion inside the sameregion at rest, then then risk of a heart attack should be decreased.
2. If perfusion inside region iis not higher than perfusion inside the same region atrest, then the risk of a heart attack should be increased.
8/2/2019 Chapter 9 Negnevitsky
53/72
Three Heuristics Implemented In the Diagnostic System
Step 1 Present the neuro-fuzzy system with the cardiac case.
Step 2 If the system's output is less than 30, classify the presented case as normal and then stop. If the outputis greater than 50, classify the case as abnormal and stop. Otherwise go to step 3.
Step 3 For region 1, subtract perfusion at rest fro perfusion at stress. If the result is positive, decrease the
current risk by multiplying its value by 0.99. Otherwise, increase the risk by multiplying its value by1.01. Repeat this procedure for all 22 regions then go to Step 4.
Step 4 If the new risk value is less than 30, classify the case as normal; if the risk is greater than 50, classifythe case as abnormal; otherwise, classify the case as uncertain.
When we now apply the test set to the neuro-fuzzy system, we find that theaccuracy of diagnosis has dramatically improved - the overall diagnostic error doesnot exceed 5 percent, while only 3 percent of abnormal cases are misclassified asnormal.
Although we have not improved the system's performance on normal cases (over 30percent of normal cases are misclassified as abnormal), and up to 20 percent of thetotal number of cases are classified as uncertain, the neuro-fuzzy system canactually achieve even better results in classifying SPECT images than a cardiologistcan.
8/2/2019 Chapter 9 Negnevitsky
54/72
Homogeneous Structure of Neuro-Fuzzy Systems
A typical example of a neuro-fuzzy system with a homogeneous structure is an Adaptive Neuro-Fuzzy Inference System (ANFIS).o It cannot be divided into two independent distinct parts.o An ANFIS is a multilayer neural network that performs fuzzy inferencing.
Case Study 9: Time series predictiono Page 346 of Negnevitsky
8/2/2019 Chapter 9 Negnevitsky
55/72
Data Mining and Knowledge Discovery
Tyler Purdom
8/2/2019 Chapter 9 Negnevitsky
56/72
Data Mining
Definition:o The extraction of knowledge from datao The exploration and analysis of large quantities of data to to discover patterns.
Ultimate goal is to discover knowledge Amount of data doubles every year Important to have fast algorithms to process data
8/2/2019 Chapter 9 Negnevitsky
57/72
Data Warehouses
Definition:o Large databases that store historical data.o Contain millions and in some cases billions of data records.
The data stored is time dependent and integrated Used to help support decision making Query tools are used to discover relationships in the data.
8/2/2019 Chapter 9 Negnevitsky
58/72
Query Tools vs. Data Mining
Query tools are assumption-basedo User must ask the right questions to get resulto User must make assumptionso Can select a specific variable that affects the outcome
Data Mining tools determine the most significant factorso No assumptions are necessaryo Discovers patterns automatically
The representation of data in data warehouses helps facilitate the data mining process
8/2/2019 Chapter 9 Negnevitsky
59/72
Data Mining Practice
Data Mining is a new and evolving field Very popular in the banking, finance, marketing, and telecommunications industries Data Mining uses:
o Determine trends in marketso Detect fraudso Target people most likely to buy a product/use a service
8/2/2019 Chapter 9 Negnevitsky
60/72
Data Mining Tools
People used to use query tools and statistics to solve data mining problemso These techniques are not very efficient for large amounts of datao Can only correlate a few variables at a time
Now, tools are based off of intelligent technologies:o Neural networks, neuro-fuzzy systems, and decision trees
Decision trees are currently the most popular tool used for data mining
8/2/2019 Chapter 9 Negnevitsky
61/72
Decision Tree
A map of the reasoning process These trees do not allow for the use of noisy or incomplete data Uses tree structure to describe the data set Very effective in solving classification problems Popular because they help you visualize the problem Nodes are separated by predictors
o In the book example, homeownership was used to split the tree
8/2/2019 Chapter 9 Negnevitsky
62/72
Decision Tree Example
8/2/2019 Chapter 9 Negnevitsky
63/72
Gini Coefficient
A measure of how well the predictor separates the classes contained in the parent node Introduced by Corrado Gini, an Italian economist He used it to measure the inequality in Italy's income distribution
8/2/2019 Chapter 9 Negnevitsky
64/72
Calculating the Gini Coefficient
Top curve represents the real economy Bottom line represents equal distribution ofwealth
Coefficient:o (shaded area)/ area below bottom
line
8/2/2019 Chapter 9 Negnevitsky
65/72
Gini Split Example
8/2/2019 Chapter 9 Negnevitsky
66/72
Summary
Jiajuan Lin
8/2/2019 Chapter 9 Negnevitsky
67/72
8/2/2019 Chapter 9 Negnevitsky
68/72
8/2/2019 Chapter 9 Negnevitsky
69/72
Summary - Data & Knowledge acquisition
Questions about the datao Range? Continues? Discrete? Precise? Noise Tolerance? Numerical? Symbolic? Data Mining
o analyze data, finding pattern & rules, extracting knowledge from large quantities of datao decision tree
easy to follow
visualization of solution makes clear sets of rules
8/2/2019 Chapter 9 Negnevitsky
70/72
Summary - Prototype
shows understanding ofo the problemo problem-solving strategy
o tool selected
Testo Throw it away if needed
o Forcing wrong tool leads to more time waste in the later development process
o Prototype is there for discovering any inappropriate/wrong decisions made
8/2/2019 Chapter 9 Negnevitsky
71/72
Summary -
Complete System,Evaluation ,Revision, Integration & maintenance
Complete System Developmento plan, schedule, budget Evaluation
o no clear right/wrongo user satisfaction = measurement
Revisiono Modify as limitation & weaknesses discovered
Maintenanceo Knowledge evolves over timeo keep modifying and updating to maintain efficiency and accuracy
8/2/2019 Chapter 9 Negnevitsky
72/72