View
1
Download
0
Category
Preview:
Citation preview
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
Artificial Neural NetworksAn Introductory Look
Sayed Jahed Hussini & Hisham Saleh
Western Michigan UniversityDepartment of Computer ScienceAdvanced Theory of Computation
Dr. Elise de Doncker
February 4, 2016
Hussini & Saleh Artificial Neural Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source: http://gizmodo.com/these-are-the-incredible-day-dreams-of-artificial-
neura-1712226908
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Outline
1 IntroductionProblemSolution
2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
3 Applications
4 Case StudyBankruptcy Prediction
5 Benefits/Limitations
6 Questions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Machines will follow a path that mirrors the evolutionof humans.
“Ray Kurzweil”
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
In 2012, Google received over 2 million search queries perminute
In 2014 it received over 4 million search queries per minute
Every Second:
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source: http://pennystocks.la/internet-in-real-time/
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
In 2012, Google received over 2 million search queries perminute
In 2014 it received over 4 million search queries per minute
Every Second:
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source: http://pennystocks.la/internet-in-real-time/
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
In 2012, Google received over 2 million search queries perminute
In 2014 it received over 4 million search queries per minute
Every Second:
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source: http://pennystocks.la/internet-in-real-time/
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
World is full of data
In today’s interconnected e-world, information can be storedand transmitted instantly
Challange?
To generate useful knowledge from collecteddata
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
World is full of data
In today’s interconnected e-world, information can be storedand transmitted instantly
Challange?
To generate useful knowledge from collecteddata
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
World is full of data
In today’s interconnected e-world, information can be storedand transmitted instantly
Challange?
To generate useful knowledge from collecteddata
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
World is full of data
In today’s interconnected e-world, information can be storedand transmitted instantly
Challange?
To generate useful knowledge from collecteddata
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
Question How do we extract knowledge from noisy mass ofdata?
Traditional computers are too dumb tounderstand patterns or do analysis
Solution Empirical computer models that learn
Interpretation requires data acquisition, cleaning(preparing the data for analysis),Key is to extract information about data fromrelationships buried within the data itself.
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Problem
Question How do we extract knowledge from noisy mass ofdata?
Traditional computers are too dumb tounderstand patterns or do analysis
Solution Empirical computer models that learn
Interpretation requires data acquisition, cleaning(preparing the data for analysis),Key is to extract information about data fromrelationships buried within the data itself.
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Solution
Human Brain is the most powerful computer every invented to doanalysis
However it cannot handle huge amounts of data
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Artificial Neural Networks(ANN)
Rather than programming computers to the most specificdetail of their tasks teach them how to do a job e.g: children
We must built empirical models that can find patterns rapidlyand accurately(to some extent) burried in data
Artificial Intelligence System - AI can do this
ANN is a case of AI
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Artificial Neural Networks(ANN)
Rather than programming computers to the most specificdetail of their tasks teach them how to do a job e.g: children
We must built empirical models that can find patterns rapidlyand accurately(to some extent) burried in data
Artificial Intelligence System - AI can do this
ANN is a case of AI
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Artificial Neural Networks(ANN)
Rather than programming computers to the most specificdetail of their tasks teach them how to do a job e.g: children
We must built empirical models that can find patterns rapidlyand accurately(to some extent) burried in data
Artificial Intelligence System - AI can do this
ANN is a case of AI
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
ProblemSolution
Artificial Neural Networks(ANN)
Rather than programming computers to the most specificdetail of their tasks teach them how to do a job e.g: children
We must built empirical models that can find patterns rapidlyand accurately(to some extent) burried in data
Artificial Intelligence System - AI can do this
ANN is a case of AI
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Outline
1 IntroductionProblemSolution
2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
3 Applications
4 Case StudyBankruptcy Prediction
5 Benefits/Limitations
6 Questions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Artificial Neural Networks(ANN)
ANN is an information processing paradigm that is inspired bythe way biological nervous systems, such as the brain, processinformation
The key element in this paradigm is the novel structure ofinformation processing
ANNs, like people, learn by example
Currently, an ANN is configured for a specific application e.g:pattern recognition, data calssification
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Artificial Neural Networks(ANN)
ANN is an information processing paradigm that is inspired bythe way biological nervous systems, such as the brain, processinformation
The key element in this paradigm is the novel structure ofinformation processing
ANNs, like people, learn by example
Currently, an ANN is configured for a specific application e.g:pattern recognition, data calssification
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Artificial Neural Networks(ANN)
ANN is an information processing paradigm that is inspired bythe way biological nervous systems, such as the brain, processinformation
The key element in this paradigm is the novel structure ofinformation processing
ANNs, like people, learn by example
Currently, an ANN is configured for a specific application e.g:pattern recognition, data calssification
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
How human brain learns?
Human brain is a dense network of approximately 1011
neurons, each connected to, on average, 104 others
Neuron activity is excited or inhibited through connections toother neurons
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
How human brain learns?
Human brain is a dense network of approximately 1011
neurons, each connected to, on average, 104 others
Neuron activity is excited or inhibited through connections toother neurons
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
How human brain learns?
Human brain is a dense network of approximately 1011
neurons, each connected to, on average, 104 others
Neuron activity is excited or inhibited through connections toother neurons
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
From Natural to Artificial Neurons
To build artificial neuron:
Deduce essential features of neurons and their connections
Program a system to simulate the features
Due to imprecise knowledge, our models are necessarily grossidealisations of real networks of neurones
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
From Natural to Artificial Neurons
To build artificial neuron:
Deduce essential features of neurons and their connections
Program a system to simulate the features
Due to imprecise knowledge, our models are necessarily grossidealisations of real networks of neurones
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
From Natural to Artificial Neurons
To build artificial neuron:
Deduce essential features of neurons and their connections
Program a system to simulate the features
Due to imprecise knowledge, our models are necessarily grossidealisations of real networks of neurones
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
A simple neuron
Artificial neuron is a device with many inputs and one output
Two modes:
TrainingUsing
Firing Rule determines when a neuron should fire.
Are very important in neural networks and accounts for theirhigh flexibility
Calcualtions of when neuron should fire are based on inputpatterns
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
A simple neuron
Artificial neuron is a device with many inputs and one output
Two modes:
TrainingUsing
Firing Rule determines when a neuron should fire.
Are very important in neural networks and accounts for theirhigh flexibility
Calcualtions of when neuron should fire are based on inputpatterns
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
A simple neuron
Artificial neuron is a device with many inputs and one output
Two modes:
TrainingUsing
Firing Rule determines when a neuron should fire.
Are very important in neural networks and accounts for theirhigh flexibility
Calcualtions of when neuron should fire are based on inputpatterns
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
A simple neuron
Artificial neuron is a device with many inputs and one output
Two modes:
TrainingUsing
Firing Rule determines when a neuron should fire.
Are very important in neural networks and accounts for theirhigh flexibility
Calcualtions of when neuron should fire are based on inputpatterns
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
A simple neuron
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Perceptrons
A type of atrificial neuron developed in 1905s
Takes several binary inputs and produces a single ouput
To compute the output each input is given a weight, thatexpresses it’s importance
The output is determined:
output =
{0 if
∑j wjxj ≤ threshhold
1 if∑
j wjxj ≥ threshhold
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Preceptrons
Example: Decide whether to go to a festival or not:
How is the weather?(x1)How far is the festival grounds?(x2)Does your boyfriend/girlfriend want to accompany you?(x3)
A complex perceptron:
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Preceptrons
Example: Decide whether to go to a festival or not:
How is the weather?(x1)How far is the festival grounds?(x2)Does your boyfriend/girlfriend want to accompany you?(x3)
A complex perceptron:
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Sigmoid
The problem is that this isn’t what happens when our networkcontains perceptrons
In fact, a small change in the weights or bias of any singleperceptron in the network can sometimes cause the output ofthat perceptron to completely flip, say from 00 to 1
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Sigmoid
The problem is that this isn’t what happens when our networkcontains perceptrons
In fact, a small change in the weights or bias of any singleperceptron in the network can sometimes cause the output ofthat perceptron to completely flip, say from 00 to 1
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Sigmoid
The aforementioned problem is solved by another type ofartificial neuron called Sigmoid neuron
Similar to perceptrons, but modified so that small changes intheir weights and bias cause only a small change in theiroutput
In Sigmoid neurons inputs instead of just being 0 and 1, cantake any value between 0 and 1
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Sigmoid
The aforementioned problem is solved by another type ofartificial neuron called Sigmoid neuron
Similar to perceptrons, but modified so that small changes intheir weights and bias cause only a small change in theiroutput
In Sigmoid neurons inputs instead of just being 0 and 1, cantake any value between 0 and 1
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Sigmoid
The aforementioned problem is solved by another type ofartificial neuron called Sigmoid neuron
Similar to perceptrons, but modified so that small changes intheir weights and bias cause only a small change in theiroutput
In Sigmoid neurons inputs instead of just being 0 and 1, cantake any value between 0 and 1
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Sigmoid
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
Architecture
Feed-Forward Networks Backpropagation Networks
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:http://neuralnetworksanddeeplearning.com/chap1.html
andhttp://nikhilbuduma.com/2015/01/11/a-deep-dive-into-recurrent-neural-networks/
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Artificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
ANN Application Development
Nelson and Illingworth in ”A Practical Guide To Neural Networks”outline following steps on designing a neural network:
1 Variable selection
2 Data collection
3 Training, testing and validation set
4 Network Architecture
Number of hidden layers and neuronsNumber of ouput neuronsTransfer function
5 Neural Network Training
6 implementation
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Outline
1 IntroductionProblemSolution
2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
3 Applications
4 Case StudyBankruptcy Prediction
5 Benefits/Limitations
6 Questions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Applications
Security(e.g: Baggagechecking in airports)
Stock market prediction
Loan approval
Credit rating
Medical diagnosis
Process/Quality control
Pattern recognition
Recognizing genes
Ecosystem evaluation
Kndowledge discovery
Time serie analysis
Sales forecasting
Targetted marketing
HR management
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Outline
1 IntroductionProblemSolution
2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
3 Applications
4 Case StudyBankruptcy Prediction
5 Benefits/Limitations
6 Questions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Problem Statement
There have been a lot of work on developing neural networksto predict bankruptcy using financial ratios and discriminantanalysis
The ANN paradigm selected in the design phase for thisproblem was a three-layer feedforward ANN usingbackpropagation
The data for training the network consisted of a small set ofnumbers for well-known financial ratios, and data wereavailable on the bankruptcy outcomes corresponding to knowndata sets
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Problem Statement
There have been a lot of work on developing neural networksto predict bankruptcy using financial ratios and discriminantanalysis
The ANN paradigm selected in the design phase for thisproblem was a three-layer feedforward ANN usingbackpropagation
The data for training the network consisted of a small set ofnumbers for well-known financial ratios, and data wereavailable on the bankruptcy outcomes corresponding to knowndata sets
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Problem Statement
There have been a lot of work on developing neural networksto predict bankruptcy using financial ratios and discriminantanalysis
The ANN paradigm selected in the design phase for thisproblem was a three-layer feedforward ANN usingbackpropagation
The data for training the network consisted of a small set ofnumbers for well-known financial ratios, and data wereavailable on the bankruptcy outcomes corresponding to knowndata sets
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Application Design
There are five input nodes, corresponding to five financial ratios:
X1: Working capital/total assets
X2: Retained earnings/total assets
X3: Earnings before interest and taxes/total assets
X4: Market value of equity/total debt
X5: Sales/total assets
A single ouput, based on the given input, will indicate a possiblebrankruptcy(0) or nonbankruptcy(1) for a given financial firmThe system must have data and financial ratios of the firms thatdid and did not go bankrupt in the past
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Application Design
There are five input nodes, corresponding to five financial ratios:
X1: Working capital/total assets
X2: Retained earnings/total assets
X3: Earnings before interest and taxes/total assets
X4: Market value of equity/total debt
X5: Sales/total assets
A single ouput, based on the given input, will indicate a possiblebrankruptcy(0) or nonbankruptcy(1) for a given financial firm
The system must have data and financial ratios of the firms thatdid and did not go bankrupt in the past
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Application Design
There are five input nodes, corresponding to five financial ratios:
X1: Working capital/total assets
X2: Retained earnings/total assets
X3: Earnings before interest and taxes/total assets
X4: Market value of equity/total debt
X5: Sales/total assets
A single ouput, based on the given input, will indicate a possiblebrankruptcy(0) or nonbankruptcy(1) for a given financial firmThe system must have data and financial ratios of the firms thatdid and did not go bankrupt in the past
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
ANN Architecture
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source:www.cse.hcmut.edu.vn/ dtanh/download/ANN.ppt
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Training and Testing
Training:
The data set, consisting of 129 firms, was partitioned into atraining set and a test set. The training set of 74 firmsconsisted of 38 that went bankrupt and 36 that did not
Testing:
The test set consisted of 27 bankrupt and 28 non-bankruptfirms. The neural network was able to correctly predict 81.5%of the bankrupt cases and 82.1% of the nonbankrupt casesOverrall, the ANN did much better predicting 22 out of the27 actual cases (the discriminant analysis predicted only 16cases correctly)
Source: R.L. Wilson and R. Sharda, “Bankruptcy Prediction UsingNeural Networks,” Decision Support Systems, Vol. 11, No. 5, June1994, pp. 545-557.
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Bankruptcy Prediction
Training and Testing
Training:
The data set, consisting of 129 firms, was partitioned into atraining set and a test set. The training set of 74 firmsconsisted of 38 that went bankrupt and 36 that did not
Testing:
The test set consisted of 27 bankrupt and 28 non-bankruptfirms. The neural network was able to correctly predict 81.5%of the bankrupt cases and 82.1% of the nonbankrupt casesOverrall, the ANN did much better predicting 22 out of the27 actual cases (the discriminant analysis predicted only 16cases correctly)
Source: R.L. Wilson and R. Sharda, “Bankruptcy Prediction UsingNeural Networks,” Decision Support Systems, Vol. 11, No. 5, June1994, pp. 545-557.
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Outline
1 IntroductionProblemSolution
2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
3 Applications
4 Case StudyBankruptcy Prediction
5 Benefits/Limitations
6 Questions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Benefits
Useful in pattern recognition, classification, abstraction andinterpretation of incomplete and noisy inputs e.g:handwritting
Providing some human characteristics to problem solving thatare difficult using standard system/software
Ability to solve new kinds of problems. ANNs are particularlyeffective at solving problems whose solutions are difficult, ifnot impossible, to define
ANNs tend to be more robust, and have the ability to copewith imcomplete or fuzzy data.
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Limitations
ANNs do not produce an explicit model even though newcases can be fed into it and new results obtained
ANNs lack explanation capabilities. Justifications for results isdifficults to obtain because the connection weights usually donot have obvious interpretaions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Google’s DeepDream
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source: http://gizmodo.com/these-are-the-incredible-day-dreams-of-artificial-
neura-1712226908
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Google’s DeepDream
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Source: http://gizmodo.com/these-are-the-incredible-day-dreams-of-artificial-
neura-1712226908
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Outline
1 IntroductionProblemSolution
2 ConceptsArtificial Neural NetworksHuman and Artificial NeuronsArtificial NeuronsANN Artchitecture
3 Applications
4 Case StudyBankruptcy Prediction
5 Benefits/Limitations
6 Questions
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
Questions
Questions?
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
References
Uhrig, R.E., ”Introduction to artificial neural networks,” inIndustrial Electronics, Control, and Instrumentation, 1995.,Proceedings of the 1995 IEEE IECON 21st InternationalConference on , vol.1, no., pp.33-37 vol.1, 6-10 Nov 1995Introduction to artificial neural networks,” in ElectronicTechnology Directions to the Year 2000, 1995. Proceedings. ,vol., no., pp.36-62, 23-25 May 1995Yuhong Li; Weihua Ma, ”Applications of Artificial NeuralNetworks in Financial Economics: A Survey,” inComputational Intelligence and Design (ISCID), 2010International Symposium on , vol.1, no., pp.211-214, 29-31Oct. 2010Huang, S.H.; Hong-Chao Zhang, ”Artificial neural networks inmanufacturing: concepts, applications, and perspectives,” inComponents, Packaging, and Manufacturing Technology, PartA, IEEE Transactions on , vol.17, no.2, pp.212-228, Jun 1994
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
IntroductionConcepts
ApplicationsCase Study
Benefits/LimitationsQuestions
References
https://www.doc.ic.ac.uk/ nd/surprise96/journal/vol4/cs11/report.htmlhttp ://neuralnetworksanddeeplearning .com/chap1.html
http://nikhilbuduma.com/2015/01/11/a-deep-dive-into-recurrent-neural-networks/
http://natureofcode.com/book/chapter-10-neural-networks/
http://www.theatlantic.com/technology/archive/2015/09/robots-hallucinate-dream/403498/
https://www.technologyreview.com/s/513696/deep-learning/
uhaweb.hartford.edu/ilumokanw/Chap1student.ppt
www.cse.hcmut.edu.vn/ dtanh/download/ANN.ppt
Sayed Jahed Hussini and Hisham M. Saleh Artificial Nerual Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
Artificial Neural NetworksAn Introductory Look
Sayed Jahed Hussini & Hisham Saleh
Western Michigan UniversityDepartment of Computer ScienceAdvanced Theory of Computation
Dr. Elise de Doncker
February 4, 2016
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
What Are They?
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
What Are They?
What Are Learning Algorithmsand How do they work
As Jahed discussed in his presentation, neural networksbecome accurate as they are trained more.Training a neural network and building its neuronconnections requires a set of algorithms that fall within therealm of machine learning. These algorithms are generalartificial intelligence algorithms that can applied to helptrain the neuron network.By the end of this presentation, it is my hope that you willhave an additional two algorithms to add to your list ofthings that confuse you.
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
What Are They?
The Algorithms
Hill Climbing:The first algorithm we will cover is the hill climbingalgorithm, a technique that allows you to conduct a "local"search. This is one of the simplest technique available inthe artificial
Simulated Annealing:The algorithm is used to allow us to approximate theoptimal solution to a problem with too many possiblesolutions to reasonably consider all of them in the search.Simulated Annealing is an algorithm thats based on thesimilar annealing process in metallurgy. We will cover how itworks and its advantages in a few slides.
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
The Algorithm
The basic idea in Hill Climbing is that the solution or goalstate is at the top of the highest hill and you must reach it.Algorithm:
First generate an initial solution.Loop till the crest is reached.
Check neighboring pointIf it is better, choose it as solutionOtherwise, you’ve reached the crest
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
Example
Figure: Image from http://www35.homepage.villanova.edu/abdo.achkar/csc8530/proj.htm
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
Local Maxima/Minima
Figure: Image from http://webspace.ulbsibiu.ro/adrian.florea/html/Planificari/EvolutionaryComputing/Course_3/ppt/Hill_Climbing.ppt
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
Plateau
Figure: Image from http://webspace.ulbsibiu.ro/adrian.florea/html/Planificari/EvolutionaryComputing/Course_3/ppt/Hill_Climbing.ppt
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
The algorithmDisadvantages
Getting out?
How do we get out when we’re stuck?Backtracking to some earlier node and choosing a differentpath.Making a big jumpOthers?
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Definition
Simulated Annealing works by trying to achieve a goalstate without reaching it too fast. What?In search algorithms, we want to focus on solutions thatmight be optimal without ignoring better solutions that wemight end up finding later. We want to make sure we don’tget stuck in the local optimal solutionAnnealing in metallurgy is a process that applies heat to asubstance in order to alter its physical properties in orderto increase its malleability and decrease its hardness.What does that have to do with anything?
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
The Algorithm
1 First thing is to pick a solution. What is its2 while temperature is greater than minimum temperature
"energy value"?1 Store a copy of the current solution2 Change the copy slightly.3 Compare the new energy value with the old energy value.
and keep the better one.4 Reduce the temperature.
3 Repeat all the above (Why?)(Where do we stop?)
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Why does this work
In the beginning, we create a variable to represent thetemperature. We set the temperature to be high.When the temperature is high, we allow the algorithm willbe allowed to more often accept solutions that are not asgood as the one we have right now. (Why?)Reduction of the temperature allows the algorithm toreduce its acceptance of worse solutions, thus allowing itto focus on an area of the search space.Simulated Annealing allows the algorithm to takeadvantage of the fact that a solution is easy to find.
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Advantages
Because simulated annealing can accept worse solutionsat times, it does not get stuck at local optima as much asthe hill climbing technique.In general, simulated annealing is much better atapproximating the global optimal solution. It is muchsimpler than the more complicated genetic techniques yetvery powerful
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Outline
1 Introduction to Learning AlgorithmsWhat Are They?
2 Hill ClimbingThe algorithmDisadvantages
3 Simulated AnnealingDefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Acceptance Function
How do we determine how to accept the solution and stop?Well there’s an equation for that (sort of). We mustdetermine whether the new solution is better than ourcurrent one, or if it’s worse, by how much.T̂hat leads us to the question of how do we quantify worse?The math is pretty simple: exp( (current Energy -neighbor’s Energy) / temperature) Basically, the smaller thechange in energy and the higher the temperature, the morelikely the new solution will be accepted.
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Setting the parameters
1 Initial temperature: The temperature should be chosen ashigh as possible so that initially, the current solution will beaccepted.
2 Temperature change: Setting the cooling rate too high willforce the algorithm into a smaller region without first takinga glance at the entire search space.
3 Minimum Temperature: Setting the temperature too low willforce the algorithm to search in a small region for too long,rather than allowing it to escape from the local area.
4 Acceptance delta: At what point do you simply accept thesolution you have already as a close enoughapproximation of the optimal solution?
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
DefinitionThe algorithmAdvantagesAcceptance Function and Parameter Setting
Effect of Initial Solution
Question: How much does your choice of the initial solutionaffect your final result?
What about in comparison to hill climbing?
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
Summary
Neural Networks should be in every computer scientist’stoolbox.Hill climbing and Simulated Annealing are just twomethods used to teach a neural network.
Hussini & Saleh Artificial Neural Networks
Introduction to Learning AlgorithmsHill Climbing
Simulated AnnealingSummary
Questions
1 How do we extract knowledge from noisy mass of data?2 Explain the difference between Sigmoids and Perceptrons.
Describe when do we use one over the other?3 Describe some of the benefits of Artificial Neural Networks.4 Describe some methods of getting out when the hill
climbing algorithm gets stuck.5 Describe the simulated annealing algorithm and explain its
advantages over hill climbing.6 Explain some of the parameters that must be set for
simulated annealing to work.
Hussini & Saleh Artificial Neural Networks
Recommended