View
213
Download
0
Tags:
Embed Size (px)
Citation preview
Stamatios Cheirdaris, Dmitry Nikelshpur, Charles Tappert, Alexander Cipully, Roberto Rodriguez, Rohit Yalamanchi, Abou Damon, Stephanie Pierce-Jones, and Robert Zucker
Human Visual System Neural Network
The Visual System
• Hubel and Wiesel• 1981 Nobel Prize for work in early 1960s
• Cat’s visual cortex
• cats anesthetized, eyes open with controlling muscles paralyzed to fix the stare in a specific direction
• thin microelectrodes measure activity in individual cells
• cells specifically sensitive to line of light at specific orientation
• Key discovery – line and edge detectors in the visual cortex of mammals
The Study
• Compare Two Neural Networks• One without vertical and horizontal line detectors
• One with vertical and horizontal line detectors
• Objective• Show that the neural network with line detectors
is superior to the one without on the six vertical-horizontal line-segment letters E, F, H, I, L, T
• Also, experiment with the full alphabet• Without line detectors
Uppercase 5x7 Bit-map AlphabetHorizontal-vertical line-segment letters are E, F, H, I, L, T
Neural Network Without Line Detectors
Neural Network SpecificationWithout Line Detectors
Layers
1. Input layer: 20x20 retina of binary units
2. Hidden layer: 50 units (other numbers explored)
3. Output layer: 6 units for letters E, F, H, I, L, T
Weights
• 20,000 (400x50) between input and hidden layer
• 300 (50x6) between hidden and output layer
• Total of 20,300 variable weights, no fixed weights
Neural Network With Line Detectors
Neural Network SpecificationWith Vertical and Horizontal Line Detectors
Layers
1. Input layer: 20x20 retina of binary units
2. 576 simple vertical and horizontal line detectors
3. 48 complex vertical and horizontal line detectors
4. Hidden layer: 50 units (other numbers explored)
5. Output layer: 6 units for letters E, F, H, I, L, T
Weights
• 6336 (576x11) fixed weights from input to simple detectors
• 576 fixed weights from simple and complex detectors
• 2400 (48x50) variable weights from complex detectors to hidden layer
• 300 (50x6) variable weights from hidden to output layer
• Total of 6912 fixed weights and 2700 variable weights
DETECTORS OVERLAP COVERING EACH POSSIBLE RETINAL POSITION FOR A TOTAL OF 288 (18x16) VERTICAL LINE DETECTORS
EACH DETECTOR HAS 5 EXCITATORY AND 6 INHIBITORY INPUTS (11 FIXED WEIGHTS),WITH A THRESHOLD OF 3
Vertical Line Detectors
Horizontal Line Detectors are Similar
Retina Image – Letter “E” in Upper Left Area
Region of possible upper-left corners is shown in green.
Retina Image – Letter “E” in Upper Right Area
Region of possible upper-left corners is shown in green.
Retina Image – Letter “E” in Lower Right Area
Region of possible upper-left corners is shown in green.
Example of Vertical Line Detectoron Line Segment of “E” – Detector Activated
Example of Shifted Vertical Line Detectoron Letter “E” – Detector Not Activated
Example of Shifted Vertical Line Detectoron Letter “E” – Detector Not Activated
24 Vertical Complex Line Detector RegionsAny Simple Line Detector in a RegionActivates the Complex Line Detector
24 Horizontal Complex Line Detector RegionsAny Simple Line Detector in a RegionActivates the Complex Line Detector
Complex Horizontal and Vertical Line Detector Matrix
The Corresponding 48 Complex Horizontal and Vertical Line Detectors
Experiments
Experiment 1o 6 Line-Segment Letters without Line Detectorso 26 Letters without Line Detectors
Experiment 2o 6 Line-Segment Letters with Line Detectors
Experimental Parameter Combinations• Epochs:
50
100
200
400
800
1600
32000 (occasionally)
• Hidden Layer Units:
10
18*
50
100
200 *
300*
500*
*Selected cases
• Noise:
0%
2%
5%
10%
15%
20%
Experiment 1
Simulation View – Peltarion’s Synapse Product
• Weight Layer: Forward Rule: No rule Back Rule: Levenberg-
Marquardt Propagator: Weight Layer
• Function Layer:
• Function: Tanh Sigmoid Forward Rule: No rule Back Rule: Levenberg-
Marquardt Propagator: Function Layer
Simulation Settings Experiment 2 – Line Detectors
6 Line-Segment Letters: E, F, H, I, L, T
6 Letters: no line detectors
50 Hidden layer units
50 Epochs
0% noise
Exp 1 – 6 Letters, No Line Detectors – 35.42% Accuracy
6 Letters: no line detectors
50 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 6 Letters, No Line Detectors – 36.25% Accuracy
6 Letters: with line detectors
50 Hidden layer units
50 Epochs
0% noise
Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy
6 Letters: with line detectors
50 Hidden layer units
50 Epochs
0% noise
Exp 2 – 6 Letters, With Line Detectors – 67.5% Accuracy
6 Letters: no line detectors
10 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 6 Letters, No Line Detectors – 27.69% Accuracy
6 Letters: with line detectors
10 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 6 Letters, With Line Detectors – 82.5% Accuracy
6 Letters: with line detectors
10 Hidden layer units
1600 Epochs
0% noise
Exp 2 – 6 Letters, With Line Detectors – 82.5% Accuracy
26 Letters: no line detectors
50 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy
26 Letters: no line detectors
50 Hidden layer units
1600 Epochs
0% noise
Exp 1 – 26 Letters, No Line Detectors – 27.69% Accuracy
Percent Noise
Epochs 0% 2% 5% 10% 15% 20%
50 35.42 20.83 24.17 20 20.42 20.83
100 36.25 21.67 25 18.75 20.83 20.42
200 35.42 21.67 23.75 18.75 20.83 20.83
400 36.25 22.50 23.75 20.42 20.42 20.83
800 36.67 22.50 23.75 20.42 20.00 20.83
1600 36.25 18.33 23.75 20.42 20.42 20.83
Exp 1 – 6 Letters, No Line DetectorsEpochs versus Percent Added Noise
Exp 1 – 26 Letters, No Line DetectorsEpochs versus Percent Added Noise
Percent Noise
Epochs 0% 2% 5% 10% 15% 20%
50 23.08 8.17 7.21 7.50 6.25 6.83
100 23.56 9.04 7.69 7.31 6.15 4.62
200 25.00 9.81 7.60 5.58 6.06 4.52
400 27.12 9.81 6.92 5.02 5.87 4.62
800 28.37 8.96 6.92 5.48 5.58 5.19
1600 27.69 10.10 7.31 5.67 7.21 6.15
Exp 2 – 6 Letters, With Line DetectorsEpochs versus Percent Added Noise
Epochs 0% 2% 5% 10% 15% 20%
50 67.50 58.75 46.25 56.67 16.67 42.50
100 72.92 59.17 46.25 56.67 17.08 28.00
200 77.08 59.58 50.00 62.50 48.75 43.75
400 75.83 58.33 50.00 62.92 34.58 28.33
800 78.33 60.42 52.08 59.17 32.92 57.08
1600 83.33 59.17 56.67 60.00 35.00 32.08
3200 85.00 66.67 56.67 57.92 52.50 62.92
Comparison of Line / No-Line Detector Networks6 letters, 50 hidden layer units, 1600 epochs, no noise
Experiment PerformanceFixed
WeightsVariable Weights
Total Weights
No Line Detectors 36.3% 0 20,300 20,300
Line Detectors 83.3% 6,912 2,700 9,612
• Character recognition performance and efficiency of the neural network using Hubel-Wiesel-like line detectors in the early layers is superior to that of a network using adjustable weights directly from the retina
• Recognition performance more than doubled
• Line detector network was much more efficient
• order of magnitude fewer variable weights and half as many total weights
• training time decrease of several orders of magnitude
Main Conclusion
• Increasing the number of hidden layer units does not translate to better accuracy, it actually reduces it.
• Increasing the number of epochs increased the accuracy but not always
• For Experiment 2 (6 letters with line detectors) we can achieve perfect training accuracy and very good validation accuracy
• Training time varied from a few minutes to many hours with Experiment 1 – 26 Letters taking the longest out of all, i.e. for 500 hidden layer units it required up to 9 hours.
• When noise is added to the retina image the accuracy of the system drops significantly, even for Experiment 2 with the line detectors
Additional Conclusions