Upload
martha
View
69
Download
2
Tags:
Embed Size (px)
DESCRIPTION
Neural Network Architectures. Aydın Ulaş. 02 December 2004. ulasmehm @boun.edu.tr. Outline Of Presentation. Introduction Neural Networks Neural Network Architectures Conclusions. Introduction. Some numbers… The human brain contains about 10 billion nerve cells (neurons) - PowerPoint PPT Presentation
Citation preview
Neural Network ArchitecturesAydn Ula02 December [email protected]
Aydn Ula
Outline Of PresentationIntroductionNeural NetworksNeural Network ArchitecturesConclusions
Aydn Ula
IntroductionSome numbersThe human brain contains about 10 billion nerve cells (neurons)Each neuron is connected to the others through 10000 synapses
Brain as a computational unit It can learn, reorganize from experienceIt adapts to the environment It is robust and fault tolerantFast computations with too much individual computational units
Aydn Ula
IntroductionTaking the nature as a model. Consider the neuron as a PEA neuron hasInput (dendrites)Output (the axon)The information circulates from the dendrites to the axon via the cell bodyAxon connects to dendrites via synapsesStrength of synapses changeSynapses may be excitatory or inhibitory
Aydn Ula
Perceptron (Artificial Neuron)Definition : Non linear, parameterized function with restricted output range
Aydn Ula
Activation FunctionsLinearSigmoidHyperbolic tangent
Aydn Ula
Neural NetworksA mathematical model to solve engineering problemsGroup of highly connected neurons to realize compositions of non linear functionsTasksClassificationClusteringRegression According to input flowFeed forward Neural NetworksRecurrent Neural Networks
Aydn Ula
Feed Forward Neural NetworksThe information is propagated from the inputs to the outputsTime has no role (Acyclic, no feedbacks from outputs to inputs)
Aydn Ula
Recurrent NetworksArbitrary topologiesCan model systems with internal states (dynamic ones)Delays can be modeledMore difficult to trainProblematic performanceStable Outputs may be more difficult to evaluateUnexpected behavior (oscillation, chaos, )
x1x2
Aydn Ula
LearningThe procedure that consists in estimating the parameters of neurons (setting up the weights) so that the whole network can perform a specific task.2 types of learningSupervised learningUnsupervised learningThe Learning process (supervised)Present the network a number of inputs and their corresponding outputs (Training)See how closely the actual outputs match the desired onesModify the parameters to better approximate the desired outputsSeveral passes over the data
Aydn Ula
Supervised LearningThe real outputs of the model for the given inputs is known in advance. The networks task is to approximate those outputs.A Supervisor provides examples and teach the neural network how to fulfill a certain task
Aydn Ula
Unsupervised learningGroup typical input data according to some function.Data clusteringNo need of a supervisor Network itself finds the correlations between the dataExamples:Kohonen feature maps (SOM)
Aydn Ula
Properties of Neural NetworksSupervised networks are universal approximators (Non recurrent networks)Can act asLinear Approximator (Linear Perceptron)Nonlinear Approximator (Multi Layer Perceptron)
Aydn Ula
Other PropertiesAdaptivityAdapt weights to the environment easilyAbility to generalizeMay provide against lack of data Fault toleranceNot too much degradation of performances if damaged The information is distributed within the entire net.
Aydn Ula
An Example Regression
Aydn Ula
Example ClassificationHandwritten digit recognition16x16 bitmap representationConverted to 1x256 bit vector7500 points on training set3500 points on test set0000000001100000000000011010000000000001000000000000001000000000000001000000000000001000000000000000100000000000000010000000000000001000000000000001000111110000000101100001100000011000000010000001100000001000000100000000100000001000000100000000011111110000
Aydn Ula
TrainingTry to minimize an error or cost functionBackpropogation algorithmGradient DescentLearn the weights of the networkUpdate the weights according to the error function
Aydn Ula
ApplicationsHandwritten Digit RecognitionFace recognitionTime series predictionProcess identificationProcess controlOptical character recognitionEtc
Aydn Ula
Neural NetworksNeural networks are statistical toolsAdjust non linear functions to accomplish a taskNeed of multiple and representative examples but fewer than in other methodsNeural networks can model static (FF) and dynamic (RNN) tasksNNs are good classifiers BUTGood representations of data have to be formulatedTraining vectors must be statistically representative of the entire input spaceThe use of NN needs a good comprehension of the problem
Aydn Ula
Implementation of Neural NetworksGeneric architectures (PCs etc)Specific Neuro-HardwareDedicated circuits
Aydn Ula
Generic architecturesConventional microprocessorsIntel Pentium, Power PC, etc AdvantagesHigh performances (clock frequency, etc)CheapSoftware environment available (NN tools, etc)DrawbacksToo generic, not optimized for very fast neural computations
Aydn Ula
Classification of HardwareNN HardwareNeurochipsSpecial PurposeGeneral Purpose (Ni1000, L - Neuro)NeuroComputersSpecial Purpose (CNAPS, Synapse)General Purpose
Aydn Ula
Specific Neuro-hardware circuitsCommercial chips CNAPS, Synapse, etc.AdvantagesCloser to the neural applicationsHigh performances in terms of speedDrawbacksNot optimized to specific applicationsAvailabilityDevelopment tools
Aydn Ula
CNAPSSIMDOne instruction sequencing and control unitProcessor nodes (PN)Single dimensional array (only right or left nodes)
Aydn Ula
CNAPS 1064
Aydn Ula
CNAPS
Aydn Ula
Dedicated circuitsA system where the functionality is buried in the hardware. For specific applications only not changeableAdvantagesOptimized for a specific applicationHigher performances than the other systemsDrawbacksHigh development costs in terms of time and money
Aydn Ula
What type of hardware to be used in dedicated circuits ?Custom circuitsASIC (Application-Specific Integrated Circuit)Necessity to have good knowledge of the hardware designFixed architecture, hardly changeableOften expensive Programmable logicValuable to implement real time systemsFlexibilityLow development costsLower performances compared to ASIC (Frequency, etc.)
Aydn Ula
Programmable logicField Programmable Gate Arrays (FPGAs)Matrix of logic cells Programmable interconnectionAdditional features (internal memories + embedded resources like multipliers, etc.)ReconfigurabilityWe can change the configurations as many times as desired
Aydn Ula
Real Time SystemsExecution of applications with time constraints.Hard real-time systemsDigital fly-by-wire control system of an aircraft: No lateness is accepted. The lives of people depend on the correct working of the control system of the aircraft.Soft real-time systemsVending machine: Accept lower performance for lateness, it is not catastrophic when deadlines are not met. It will take longer to handle one client with the vending machine.
Aydn Ula
Real Time Systemsms scale real time systemConnectionist retina for image processingArtificial Retina: combining an image sensor with a parallel architectures scale real time systemLevel 1 trigger in a HEP experiment
Aydn Ula
Connectionist RetinaIntegration of a neural network in an artificial retinaScreenMatrix of Active Pixel sensorsCAN 8 bits ADC converter 256 levels of greyProcessing ArchitectureParallel system where neural networks are implemented
ProcessingArchitecture
Aydn Ula
Maharadja Processing ArchitectureMicro-controllerGeneric architecture executing sequential cost with low power consumptionMemory 256 Kbytes shared between processor, PEs, inputStore the network parametersUNE (Unit Neural SIMD Completely pipelined 16 bit internal data bus)Processors to compute the neurons outputsCommand bus manages all different operators in UNEInput/Output moduleData acquisition and storage of intermediate results
Micro-controllerSequencer Command busInput/OutputunitInstruction BusUNE-0UNE-1UNE-2UNE-3MMMM
Aydn Ula
Level 1 trigger in a HEP experimentHigh Energy Physics (Particle Physics)Neural networks have provided interesting results as triggers in HEP.Level 2 : H1 experiment 10 20 s Level 1 : Dirac experiment 2 s Particle RecognitionHigh timing constraints (in terms of latency and data throughput)
Aydn Ula
Neural Network architecture....64128Execution time : ~500 ns
Weights coded in 16 bitsStates coded in 8 bits with data arriving every BC=25ns4Electrons, tau, hadrons, jets
Aydn Ula
Very Fast ArchitectureTanHPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPETanHTanHTanHACCACCACCACC256 PEsMatrix of n*m matrix elementsControl unitI/O moduleTanH are stored in LUTs1 matrix row computes a neuronThe results is back-propagated to calculate the output layer
I/O moduleControl unit
Aydn Ula
PE architectureXAccumulatorMultiplierWeights memInput data816Addr gen +Data incmd busControl ModuleData out
Aydn Ula
Neuro-hardware todayGeneric Real time applicationsMicroprocessors technology (PCs, computers, i.e. software) is sufficient to implement most of neural applications in real-time (ms or sometimes s scale) This solution is cheapVery easy to manageConstrained Real time applicationsIt still remains specific applications where powerful computations are needed e.g. particle physicsIt still remains applications where other constraints have to be taken into consideration (Consumption, proximity of sensors, mixed integration, etc.)
Aydn Ula
ClusteringIdea : Combine performances of different processors to perform massive parallel computations
High speedconnection
Aydn Ula
ClusteringAdvantagesTake advantage of the implicit parallelism of neural networksUtilization of systems already available (university, Labs, offices, etc.)High performances : Faster training of a neural net Very cheap compare to dedicated hardware
Aydn Ula
ClusteringDrawbacksCommunications load : Need of very fast links between computers Software environment for parallel processingNot possible for embedded applications
Aydn Ula
Hardware ImplementationsMost real-time applications do not need dedicated hardware implementationConventional architectures are generally appropriateClustering of generic architectures to combine performancesSome specific applications require other solutionsStrong Timing constraintsTechnology permits to utilize FPGAs FlexibilityMassive parallelism possibleOther constraints (consumption, etc.)Custom or programmable circuits
Aydn Ula
Questions?
Aydn Ula