Upload
omeras78
View
222
Download
0
Embed Size (px)
Citation preview
8/3/2019 Neural Networ;
1/38
For other uses, seeNeural network (disambiguation).
Simplified view of a feedforward artificial neural network
The term neural network was traditionally used to refer to a network or circuit ofbiological
neurons.[1]
The modern usage of the term often refers toartificial neural networks, which arecomposed ofartificial neuronsor nodes. Thus the term has two distinct usages:
1. Biological neural networksare made up of real biological neurons that are connected orfunctionally related in theperipheral nervous systemor thecentral nervous system. In thefield ofneuroscience, they are often identified as groups of neurons that perform a
specific physiological function in laboratory analysis.2. Artificial neural networksare composed of interconnecting artificial neurons
(programming constructs that mimic the properties of biological neurons). Artificial
neural networks may either be used to gain an understanding of biological neural
networks, or for solving artificial intelligence problems without necessarily creating amodel of a real biological system. The real, biological nervous system is highly complex:
artificial neural network algorithms attempt to abstract this complexity and focus on what
may hypothetically matter most from an information processing point of view. Goodperformance (e.g. as measured by good predictive ability, low generalization error), or
performance mimicking animal or human error patterns, can then be used as one source
of evidence towards supporting the hypothesis that the abstraction really captured
something important from the point of view of information processing in the brain.Another incentive for these abstractions is to reduce the amount of computation required
to simulate artificial neural networks, so as to allow one to experiment with larger
networks and train them on larger data sets.
This article focuses on the relationship between the two concepts; for detailed coverage of the
two different concepts refer to the separate articles:biological neural networkandartificialneural
http://en.wikipedia.org/wiki/Neural_network_(disambiguation)http://en.wikipedia.org/wiki/Neural_network_(disambiguation)http://en.wikipedia.org/wiki/Neural_network_(disambiguation)http://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/File:Neural_network_example.svghttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Peripheral_nervous_systemhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Neural_network#cite_note-0http://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neural_network_(disambiguation)8/3/2019 Neural Networ;
2/38
This article is about cells in the nervous system. For other uses, seeNeuron (disambiguation).
"Brain cell" redirects here. For other uses, seeGlial cell.
Neuron: Nerve Cell
Drawing bySantiago Ramn y Cajalof neurons in the
pigeon cerebellum. (A) DenotesPurkinje cells, an example
of a multipolar neuron. (B) Denotesgranule cells, which
are also multipolar.
NeuroLexID sao1417703748
vd e
A neuron ( /njrn/NEWR-on; also known as a neurone or nerve cell) is anelectricallyexcitablecellthat processes and transmits information by electrical and chemicalsignaling.
Chemical signaling occurs viasynapses, specialized connections with other cells. Neurons
connect to each other to formnetworks. Neurons are the core components of thenervous system,which includes thebrain,spinal cord, and peripheralganglia. A number of specialized types ofneurons exist:sensory neuronsrespond to touch, sound, light and numerous other stimuli
affecting cells of thesensory organsthat then send signals to the spinal cord and brain.Motor
neuronsreceive signals from the brain and spinal cord, causemuscle contractions, and affect
glands. Interneurons connect neurons to other neurons within the same region of the brain orspinal cord.
http://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Neuron_(disambiguation)http://en.wikipedia.org/wiki/Neuron_(disambiguation)http://en.wikipedia.org/wiki/Neuron_(disambiguation)http://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/NeuroLexhttp://en.wikipedia.org/wiki/NeuroLexhttp://www.neurolex.org/wiki/sao1417703748http://en.wikipedia.org/wiki/Template:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Glandhttp://en.wikipedia.org/wiki/Glandhttp://en.wikipedia.org/wiki/File:PurkinjeCell.jpghttp://en.wikipedia.org/wiki/File:PurkinjeCell.jpghttp://en.wikipedia.org/wiki/Glandhttp://en.wikipedia.org/wiki/Muscle_contractionhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Motor_neuronhttp://en.wikipedia.org/wiki/Sensehttp://en.wikipedia.org/wiki/Sensory_neuronhttp://en.wikipedia.org/wiki/Gangliahttp://en.wikipedia.org/wiki/Spinal_cordhttp://en.wikipedia.org/wiki/Brainhttp://en.wikipedia.org/wiki/Nervous_systemhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Synapsehttp://en.wikipedia.org/wiki/Signal_(biology)http://en.wikipedia.org/wiki/Cell_(biology)http://en.wikipedia.org/wiki/Electricityhttp://en.wikipedia.org/wiki/Wikipedia:Pronunciation_respelling_keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_English#Keyhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/wiki/Wikipedia:IPA_for_Englishhttp://en.wikipedia.org/w/index.php?title=Template:Infobox_neuron/sandbox&action=edithttp://en.wikipedia.org/wiki/Template_talk:Infobox_neuron/sandboxhttp://en.wikipedia.org/wiki/Template:Infobox_neuron/sandboxhttp://www.neurolex.org/wiki/sao1417703748http://en.wikipedia.org/wiki/NeuroLexhttp://en.wikipedia.org/wiki/Granule_cellshttp://en.wikipedia.org/wiki/Purkinje_cellhttp://en.wikipedia.org/wiki/Santiago_Ram%C3%B3n_y_Cajalhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Neuron_(disambiguation)8/3/2019 Neural Networ;
3/38
A typical neuron possesses a cell body (often called thesoma),dendrites, and anaxon. Dendrites
are filaments that arise from the cell body, often extending for hundreds of micrometres andbranching multiple times, giving rise to a complex "dendritic tree". An axon is a special cellular
filament that arises from the cell body at a site called theaxon hillockand travels for a distance,
as far as 1 m in humans or even more in other species. The cell body of a neuron frequently gives
rise to multiple dendrites, but never to more than one axon, although the axon may branchhundreds of times before it terminates. At the majority of synapses, signals are sent from the
axon of one neuron to a dendrite of another. There are, however, many exceptions to these rules:
neurons that lack dendrites, neurons that have no axon, synapses that connect an axon to anotheraxon or a dendrite to another dendrite, etc.
All neurons are electrically excitable, maintainingvoltagegradients across theirmembranesby
means of metabolically drivenion pumps, which combine withion channelsembedded in the
membrane to generate intracellular-versus-extracellular concentration differences ofionssuch as
sodium,potassium,chloride, andcalcium. Changes in the cross-membrane voltage can alter thefunction ofvoltage-dependent ion channels. If the voltage changes by a large enough amount, an
all-or-none electrochemical pulse called anaction potentialis generated, which travels rapidlyalong the cell's axon, and activates synaptic connections with other cells when it arrives.
Neurons of the adult brain do not generally undergocell division, and usually cannot be replaced
after being lost, although there are a fewknown exceptions. In most cases they are generated byspecial types ofstem cells, althoughastrocytes(a type ofglial cell) have been observed to turn
into neurons as they are sometimespluripotent
Artificial neural network
From Wikipedia, the free encyclopedia
Jump to:navigation,search
This article needs additionalcitationsforverification.Please helpimprove this articleby addingreliable references. Unsourced material may bechallenged
andremoved.(March 2009)
An artificial neural network (ANN), usually calledneural network(NN), is amathematical
modelorcomputational modelthat is inspired by the structure and/or functional aspects of
biological neural networks. A neural network consists of an interconnected group ofartificialneurons, and it processes information using aconnectionistapproach tocomputation. In most
cases an ANN is anadaptive systemthat changes its structure based on external or internal
information that flows through the network during the learning phase. Modern neural networksarenon-linearstatisticaldata modelingtools. They are usually used to model complexrelationships between inputs and outputs or tofind patternsin data.
http://en.wikipedia.org/wiki/Soma_(biology)http://en.wikipedia.org/wiki/Soma_(biology)http://en.wikipedia.org/wiki/Soma_(biology)http://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Sodiumhttp://en.wikipedia.org/wiki/Sodiumhttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Non-linearhttp://en.wikipedia.org/wiki/Non-linearhttp://en.wikipedia.org/wiki/Statisticalhttp://en.wikipedia.org/wiki/Statisticalhttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/File:Question_book-new.svghttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Data_modelinghttp://en.wikipedia.org/wiki/Statisticalhttp://en.wikipedia.org/wiki/Non-linearhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Computationhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Computational_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Mathematical_modelhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edithttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Citing_sources#Inline_citationshttp://en.wikipedia.org/wiki/Artificial_neural_network#p-searchhttp://en.wikipedia.org/wiki/Artificial_neural_network#mw-headhttp://en.wikipedia.org/wiki/Pluripotenthttp://en.wikipedia.org/wiki/Glial_cellhttp://en.wikipedia.org/wiki/Astrocyteshttp://en.wikipedia.org/wiki/Stem_cellhttp://en.wikipedia.org/wiki/Neurogenesis#Adult_neural_stem_cellshttp://en.wikipedia.org/wiki/Mitosishttp://en.wikipedia.org/wiki/Action_potentialhttp://en.wikipedia.org/wiki/Voltage-dependent_ion_channelhttp://en.wikipedia.org/wiki/Calciumhttp://en.wikipedia.org/wiki/Chloridehttp://en.wikipedia.org/wiki/Potassiumhttp://en.wikipedia.org/wiki/Sodiumhttp://en.wikipedia.org/wiki/Ionhttp://en.wikipedia.org/wiki/Ion_channelhttp://en.wikipedia.org/wiki/Ion_transporterhttp://en.wikipedia.org/wiki/Cell_membranehttp://en.wikipedia.org/wiki/Voltagehttp://en.wikipedia.org/wiki/Axon_hillockhttp://en.wikipedia.org/wiki/Axonhttp://en.wikipedia.org/wiki/Dendritehttp://en.wikipedia.org/wiki/Soma_(biology)8/3/2019 Neural Networ;
4/38
An artificial neural network is an interconnected group of nodes, akin to the vast network of
neuronsin thehuman brain.
Contents
[hide]
1 Background 2 Models
o 2.1 Network functiono 2.2 Learning
2.2.1 Choosing a cost functiono 2.3 Learning paradigms
2.3.1 Supervised learning 2.3.2 Unsupervised learning 2.3.3 Reinforcement learning
o 2.4 Learning algorithms 3 Employing artificial neural networks 4 Applications
o 4.1 Real-life applicationso 4.2 Neural networks and neuroscience
4.2.1 Types of models 4.2.2 Current research
5 Neural network software 6 Types of artificial neural networks 7 Theoretical properties
o 7.1 Computational powero 7.2 Capacityo 7.3 Convergence
http://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Artificial_neural_network#Backgroundhttp://en.wikipedia.org/wiki/Artificial_neural_network#Backgroundhttp://en.wikipedia.org/wiki/Artificial_neural_network#Modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Network_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Network_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Choosing_a_cost_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Choosing_a_cost_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_paradigmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_paradigmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Supervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Supervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Unsupervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Unsupervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Reinforcement_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Reinforcement_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_algorithmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_algorithmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Employing_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Employing_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Real-life_applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Real-life_applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_networks_and_neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_networks_and_neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Current_researchhttp://en.wikipedia.org/wiki/Artificial_neural_network#Current_researchhttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Theoretical_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Theoretical_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Computational_powerhttp://en.wikipedia.org/wiki/Artificial_neural_network#Computational_powerhttp://en.wikipedia.org/wiki/Artificial_neural_network#Capacityhttp://en.wikipedia.org/wiki/Artificial_neural_network#Capacityhttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/File:Artificial_neural_network.svghttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Capacityhttp://en.wikipedia.org/wiki/Artificial_neural_network#Computational_powerhttp://en.wikipedia.org/wiki/Artificial_neural_network#Theoretical_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neural_network#Current_researchhttp://en.wikipedia.org/wiki/Artificial_neural_network#Types_of_modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Neural_networks_and_neurosciencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Real-life_applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Applicationshttp://en.wikipedia.org/wiki/Artificial_neural_network#Employing_artificial_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_algorithmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Reinforcement_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Unsupervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Supervised_learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Learning_paradigmshttp://en.wikipedia.org/wiki/Artificial_neural_network#Choosing_a_cost_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Learninghttp://en.wikipedia.org/wiki/Artificial_neural_network#Network_functionhttp://en.wikipedia.org/wiki/Artificial_neural_network#Modelshttp://en.wikipedia.org/wiki/Artificial_neural_network#Backgroundhttp://en.wikipedia.org/wiki/Artificial_neural_networkhttp://en.wikipedia.org/wiki/Human_brainhttp://en.wikipedia.org/wiki/Neuron8/3/2019 Neural Networ;
5/38
o 7.4 Generalization and statisticso 7.5 Dynamic properties
8 Criticism 9 Gallery 10 See also 11 References 12 Bibliography 13 Further reading 14 External links
[edit] Background
The original inspiration for the termArtificial Neural Networkcame from examination ofcentralnervous systemsand theirneurons,axons,dendrites, andsynapses, which constitute the
processing elements ofbiological neural networksinvestigated byneuroscience. In an artificial
neural network, simple artificialnodes, variously called "neurons", "neurodes", "processingelements" (PEs) or "units", are connected together to form a network of nodes mimicking the
biological neural networkshence the term "artificial neural network".
Because neuroscience is still full of unanswered questions, and since there are many levels ofabstraction and therefore many ways to take inspiration from the brain, there is no single formal
definition of what an artificial neural network is. Generally, it involves a network of simple
processing elements that exhibit complex global behavior determined by connections betweenprocessing elements and element parameters. While an artificial neural network does not have to
be adaptive per se, its practical use comes with algorithms designed to alter the strength
(weights) of the connections in the network to produce a desired signal flow.
These networks are also similar to thebiological neural networksin the sense that functions are
performed collectively and in parallel by the units, rather than there being a clear delineation ofsubtasks to which various units are assigned (see alsoconnectionism). Currently, the term
Artificial Neural Network (ANN) tends to refer mostly to neural network models employed in
statistics,cognitive psychologyandartificial intelligence.Neural networkmodels designed withemulation of thecentral nervous system(CNS) in mind are a subject oftheoretical neuroscience
andcomputational neuroscience.
In modernsoftware implementationsof artificial neural networks, the approach inspired by
biology has been largely abandoned for a more practical approach based on statistics and signal
processing. In some of these systems, neural networks or parts of neural networks (such as
artificial neurons) are used as components in larger systems that combine both adaptive and non-adaptive elements. While the more general approach of suchadaptive systemsis more suitable
for real-world problem solving, it has far less to do with the traditional artificial intelligence
connectionist models. What they do have in common, however, is the principle of non-linear,distributed, parallel and local processing and adaptation.
http://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Convergencehttp://en.wikipedia.org/wiki/Artificial_neural_network#Generalization_and_statisticshttp://en.wikipedia.org/wiki/Artificial_neural_network#Generalization_and_statisticshttp://en.wikipedia.org/wiki/Artificial_neural_network#Dynamic_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Dynamic_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Criticismhttp://en.wikipedia.org/wiki/Artificial_neural_network#Criticismhttp://en.wikipedia.org/wiki/Artificial_neural_network#Galleryhttp://en.wikipedia.org/wiki/Artificial_neural_network#Galleryhttp://en.wikipedia.org/wiki/Artificial_neural_network#See_alsohttp://en.wikipedia.org/wiki/Artificial_neural_network#See_alsohttp://en.wikipedia.org/wiki/Artificial_neural_network#Referenceshttp://en.wikipedia.org/wiki/Artificial_neural_network#Referenceshttp://en.wikipedia.org/wiki/Artificial_neural_network#Bibliographyhttp://en.wikipedia.org/wiki/Artificial_neural_network#Bibliographyhttp://en.wikipedia.org/wiki/Artificial_neural_network#Further_readinghttp://en.wikipedia.org/wiki/Artificial_neural_network#Further_readinghttp://en.wikipedia.org/wiki/Artificial_neural_network#External_linkshttp://en.wikipedia.org/wiki/Artificial_neural_network#External_linkshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=1http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=1http://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Statisticshttp://en.wikipedia.org/wiki/Statisticshttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Adaptive_systemshttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Theoretical_neurosciencehttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Neural_networkhttp://en.wikipedia.org/wiki/Artificial_intelligencehttp://en.wikipedia.org/wiki/Cognitive_psychologyhttp://en.wikipedia.org/wiki/Statisticshttp://en.wikipedia.org/wiki/Connectionismhttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Artificial_neuronhttp://en.wikipedia.org/wiki/Node_(neural_networks)http://en.wikipedia.org/wiki/Neurosciencehttp://en.wikipedia.org/wiki/Biological_neural_networkshttp://en.wikipedia.org/wiki/Synapseshttp://en.wikipedia.org/wiki/Dendriteshttp://en.wikipedia.org/wiki/Axonshttp://en.wikipedia.org/wiki/Neuronhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/wiki/Central_nervous_systemhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=1http://en.wikipedia.org/wiki/Artificial_neural_network#External_linkshttp://en.wikipedia.org/wiki/Artificial_neural_network#Further_readinghttp://en.wikipedia.org/wiki/Artificial_neural_network#Bibliographyhttp://en.wikipedia.org/wiki/Artificial_neural_network#Referenceshttp://en.wikipedia.org/wiki/Artificial_neural_network#See_alsohttp://en.wikipedia.org/wiki/Artificial_neural_network#Galleryhttp://en.wikipedia.org/wiki/Artificial_neural_network#Criticismhttp://en.wikipedia.org/wiki/Artificial_neural_network#Dynamic_propertieshttp://en.wikipedia.org/wiki/Artificial_neural_network#Generalization_and_statistics8/3/2019 Neural Networ;
6/38
[edit] Models
Neural network models in artificial intelligence are usually referred to as artificial neuralnetworks (ANNs); these are essentially simple mathematical models defining a function
or a distribution over or both and , but sometimes models are also intimately associated
with a particular learning algorithm or learning rule. A common use of the phrase ANN modelreally means the definition of a class of such functions (where members of the class are obtained
by varying parameters, connection weights, or specifics of the architecture such as the number of
neurons or their connectivity).
[edit] Network function
See also:Graphical models
The word networkin the term 'artificial neural network' refers to the interconnections betweenthe neurons in the different layers of each system. An example system has three layers. The first
layer has input neurons, which send data via synapses to the second layer of neurons, and thenvia more synapses to the third layer of output neurons. More complex systems will have more
layers of neurons with some having increased layers of input neurons and output neurons. Thesynapses store parameters called "weights" that manipulate the data in the calculations.
An ANN is typically defined by three types of parameters:
1. The interconnection pattern between different layers of neurons2. The learning process for updating the weights of the interconnections3. The activation function that converts a neuron's weighted input to its output activation.
Mathematically, a neuron's network function is defined as a composition of other functions, which can further be defined as a composition of other functions. This can be conveniently
represented as a network structure, with arrows depicting the dependencies between variables. A
widely used type of composition is the nonlinear weighted sum, where ,
where (commonly referred to as theactivation function[1]
) is some predefined function, such asthehyperbolic tangent. It will be convenient for the following to refer to a collection of functions
as simply a vector .
ANN dependency graph
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=2http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=2http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=3http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=3http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=3http://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/Hyperbolic_tangenthttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Activation_functionhttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=3http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=28/3/2019 Neural Networ;
7/38
This figure depicts such a decomposition of , with dependencies between variables indicated by
arrows. These can be interpreted in two ways.
The first view is the functional view: the input is transformed into a 3-dimensional vector ,
which is then transformed into a 2-dimensional vector , which is finally transformed into .
This view is most commonly encountered in the context ofoptimization.
The second view is the probabilistic view: therandom variable depends upon the
random variable , which depends upon , which depends upon the random
variable . This view is most commonly encountered in the context ofgraphical models.
The two views are largely equivalent. In either case, for this particular network architecture, the
components of individual layers are independent of each other (e.g., the components of are
independent of each other given their input ). This naturally enables a degree of parallelism inthe implementation.
Recurrent ANN dependency graph
Networks such as the previous one are commonly calledfeedforward, because their graph is a
directed acyclic graph. Networks withcyclesare commonly calledrecurrent. Such networks arecommonly depicted in the manner shown at the top of the figure, where is shown as being
dependent upon itself. However, an implied temporal dependence is not shown.
[edit] Learning
What has attracted the most interest in neural networks is the possibility oflearning. Given aspecific taskto solve, and a class of functions, , learning means using a set ofobservations to
find which solves the task in some optimal sense.
This entails defining acost function such that, for the optimal solution ,
(i.e., no solution has a cost less than the cost of the optimal solution).
Thecost function is an important concept in learning, as it is a measure of how far away a
particular solution is from an optimal solution to the problem to be solved. Learning algorithms
search through the solution space to find a function that has the smallest possible cost.
For applications where the solution is dependent on some data, the cost must necessarily be a
function of the observations, otherwise we would not be modelling anything related to the data. It
http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Directed_acyclic_graphhttp://en.wikipedia.org/wiki/Directed_acyclic_graphhttp://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=4http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=4http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=4http://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/File:Recurrent_ann_dependency_graph.pnghttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/wiki/Cost_functionhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=4http://en.wikipedia.org/wiki/Recurrent_neural_networkhttp://en.wikipedia.org/wiki/Path_(graph_theory)http://en.wikipedia.org/wiki/Directed_acyclic_graphhttp://en.wikipedia.org/wiki/Feedforward_neural_networkhttp://en.wikipedia.org/wiki/Graphical_modelshttp://en.wikipedia.org/wiki/Random_variablehttp://en.wikipedia.org/wiki/Optimization_(mathematics)8/3/2019 Neural Networ;
8/38
is frequently defined as astatisticto which only approximations can be made. As a simple
example, consider the problem of finding the model , which minimizes , for
data pairs drawn from some distribution . In practical situations we would only have
samples from and thus, for the above example, we would only minimize
. Thus, the cost is minimized over a sample of the data rather than the
entire data set.
When some form ofonline machine learningmust be used, where the cost is partiallyminimized as each new example is seen. While online machine learning is often used when is
fixed, it is most useful in the case where the distribution changes slowly over time. In neural
network methods, some form of online machine learning is frequently used for finite datasets.
See also:Optimization (mathematics),Estimation theory, andMachine learning
[edit] Choosing a cost function
While it is possible to define some arbitrary,ad hoccost function, frequently a particular costwill be used, either because it has desirable properties (such asconvexity) or because it arises
naturally from a particular formulation of the problem (e.g., in a probabilistic formulation the
posterior probability of the model can be used as an inverse cost). Ultimately, the cost functionwill depend on the desired task. An overview of the three main categories of learning tasks is
provided below.
[edit] Learning paradigms
There are three major learning paradigms, each corresponding to a particular abstract learning
task. These aresupervised learning,unsupervised learningandreinforcement learning.
[edit] Supervised learning
Insupervised learning, we are given a set of example pairs and the aim is to find a
function in the allowed class of functions that matches the examples. In other words, wewish to inferthe mapping implied by the data; the cost function is related to the mismatch
between our mapping and the data and it implicitly contains prior knowledge about the problem
domain.
A commonly used cost is themean-squared error, which tries to minimize the average squared
error between the network's output, f(x), and the target value y over all the example pairs. When
one tries to minimize this cost usinggradient descentfor the class of neural networks calledmultilayer perceptrons, one obtains the common and well-knownbackpropagation algorithmfor
training neural networks.
Tasks that fall within the paradigm of supervised learning arepattern recognition(also known as
classification) andregression(also known as function approximation). The supervised learning
paradigm is also applicable to sequential data (e.g., for speech and gesture recognition). This can
http://en.wikipedia.org/wiki/Statistichttp://en.wikipedia.org/wiki/Statistichttp://en.wikipedia.org/wiki/Statistichttp://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=5http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=5http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=5http://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=6http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=6http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=6http://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=7http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=7http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=7http://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Multilayer_perceptronhttp://en.wikipedia.org/wiki/Multilayer_perceptronhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Backpropagationhttp://en.wikipedia.org/wiki/Multilayer_perceptronhttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Mean-squared_errorhttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=7http://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Supervised_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=6http://en.wikipedia.org/wiki/Convex_functionhttp://en.wikipedia.org/wiki/Ad_hochttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=5http://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Estimation_theoryhttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Online_machine_learninghttp://en.wikipedia.org/wiki/Statistic8/3/2019 Neural Networ;
9/38
be thought of as learning with a "teacher," in the form of a function that provides continuous
feedback on the quality of solutions obtained thus far.
[edit] Unsupervised learning
Inunsupervised learning, some data is given and the cost function to be minimized, that can beany function of the data and the network's output, .
The cost function is dependent on the task (what we are trying to model) and our a priori
assumptions (the implicit properties of our model, its parameters and the observed variables).
As a trivial example, consider the model , where is a constant and the cost
. Minimizing this cost will give us a value of that is equal to the mean of the
data. The cost function can be much more complicated. Its form depends on the application: forexample, in compression it could be related to themutual informationbetween x and y, whereas
in statistical modeling, it could be related to theposterior probabilityof the model given the data.
(Note that in both of those examples those quantities would be maximized rather thanminimized).
Tasks that fall within the paradigm of unsupervised learning are in general estimationproblems;the applications includeclustering, the estimation ofstatistical distributions,compressionand
filtering.
[edit] Reinforcement learning
Inreinforcement learning, data are usually not given, but generated by an agent's interactionswith the environment. At each point in time , the agent performs an action and the
environment generates an observation and an instantaneous cost , according to some(usually unknown) dynamics. The aim is to discover apolicy for selecting actions that minimizes
some measure of a long-term cost; i.e., the expected cumulative cost. The environment'sdynamics and the long-term cost for each policy are usually unknown, but can be estimated.
More formally, the environment is modeled as aMarkov decision process(MDP) with states
and actions with the following probability distributions: the instantaneous
cost distribution , the observation distribution and the transition ,
while a policy is defined as conditional distribution over actions given the observations. Taken
together, the two define aMarkov chain(MC). The aim is to discover the policy that minimizesthe cost; i.e., the MC for which the cost is minimal.
ANNs are frequently used in reinforcement learning as part of the overall algorithm.
Tasks that fall within the paradigm of reinforcement learning are control problems,gamesand
othersequential decision makingtasks.
See also:dynamic programmingandstochastic control
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=8http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=8http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=8http://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Bayesian_spam_filteringhttp://en.wikipedia.org/wiki/Bayesian_spam_filteringhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=9http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=9http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=9http://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Stochastic_controlhttp://en.wikipedia.org/wiki/Dynamic_programminghttp://en.wikipedia.org/w/index.php?title=Sequential_decision_making&action=edit&redlink=1http://en.wikipedia.org/wiki/Gamehttp://en.wikipedia.org/wiki/Markov_chainhttp://en.wikipedia.org/wiki/Markov_decision_processhttp://en.wikipedia.org/wiki/Reinforcement_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=9http://en.wikipedia.org/wiki/Bayesian_spam_filteringhttp://en.wikipedia.org/wiki/Data_compressionhttp://en.wikipedia.org/wiki/Statistical_distributionshttp://en.wikipedia.org/wiki/Data_clusteringhttp://en.wikipedia.org/wiki/Estimationhttp://en.wikipedia.org/wiki/Posterior_probabilityhttp://en.wikipedia.org/wiki/Mutual_informationhttp://en.wikipedia.org/wiki/Unsupervised_learninghttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=88/3/2019 Neural Networ;
10/38
[edit] Learning algorithms
Training a neural network model essentially means selecting one model from the set of allowed
models (or, in aBayesianframework, determining a distribution over the set of allowed models)
that minimizes the cost criterion. There are numerous algorithms available for training neural
network models; most of them can be viewed as a straightforward application ofoptimizationtheory andstatistical estimation. Recent developments in this field useparticle swarm
optimizationand otherswarm intelligencetechniques.
Most of the algorithms used in training artificial neural networks employ some form ofgradient
descent. This is done by simply taking the derivative of the cost function with respect to the
network parameters and then changing those parameters in agradient-relateddirection.
Evolutionary methods,simulated annealing,expectation-maximizationandnon-parametric
methodsare some commonly used methods for training neural networks.
See also:machine learning
Temporalperceptual learningrelies on finding temporal relationships in sensory signal streams.
In an environment, statistically salient temporal correlations can be found by monitoring thearrival times of sensory signals. This is done by theperceptual network.
[edit] Employing artificial neural networks
Perhaps the greatest advantage of ANNs is their ability to be used as an arbitrary function
approximation mechanism that 'learns' from observed data. However, using them is not so
straightforward and a relatively good understanding of the underlying theory is essential.
Choice of model: This will depend on the data representation and the application. Overlycomplex models tend to lead to problems with learning.
Learning algorithm: There are numerous trade-offs between learning algorithms. Almostany algorithm will work well with the correcthyperparametersfor training on a
particular fixed data set. However selecting and tuning an algorithm for training onunseen data requires a significant amount of experimentation.
Robustness: If the model, cost function and learning algorithm are selected appropriatelythe resulting ANN can be extremely robust.
With the correct implementation, ANNs can be used naturally inonline learningand large data
set applications. Their simple implementation and the existence of mostly local dependenciesexhibited in the structure allows for fast, parallel implementations in hardware.
[edit] Applications
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=10http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=10http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=10http://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Evolutionary_methodshttp://en.wikipedia.org/wiki/Evolutionary_methodshttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=11http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=11http://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=12http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=12http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=12http://en.wikipedia.org/wiki/Online_algorithmhttp://en.wikipedia.org/wiki/Hyperparameterhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=11http://en.wikipedia.org/w/index.php?title=Perceptual_network&action=edit&redlink=1http://en.wikipedia.org/wiki/Perceptual_learninghttp://en.wikipedia.org/wiki/Machine_learninghttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Non-parametric_methodshttp://en.wikipedia.org/wiki/Expectation-maximizationhttp://en.wikipedia.org/wiki/Simulated_annealinghttp://en.wikipedia.org/wiki/Evolutionary_methodshttp://en.wikipedia.org/wiki/Gradient-relatedhttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Gradient_descenthttp://en.wikipedia.org/wiki/Swarm_intelligencehttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Particle_swarm_optimizationhttp://en.wikipedia.org/wiki/Statistical_estimationhttp://en.wikipedia.org/wiki/Optimization_(mathematics)http://en.wikipedia.org/wiki/Bayesian_probabilityhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=108/3/2019 Neural Networ;
11/38
The utility of artificial neural network models lies in the fact that they can be used to infer a
function from observations. This is particularly useful in applications where the complexity ofthe data or task makes the design of such a function by hand impractical.
[edit] Real-life applications
The tasks artificial neural networks are applied to tend to fall within the following broad
categories:
Function approximation, orregression analysis, includingtime series prediction,fitnessapproximationand modeling.
Classification, includingpatternand sequence recognition,novelty detectionandsequential decision making.
Data processing, including filtering, clustering, blind source separation and compression. Robotics, including directing manipulators,Computer numerical control.
Application areas include system identification and control (vehicle control, process control),quantum chemistry,
[2]game-playing and decision making (backgammon, chess, racing), pattern
recognition (radar systems, face identification, object recognition and more), sequence
recognition (gesture, speech, handwritten text recognition), medical diagnosis, financial
applications (automated trading systems),data mining(or knowledge discovery in databases,"KDD"), visualization ande-mail spamfiltering.
[edit] Neural networks and neuroscience
Theoretical andcomputational neuroscienceis the field concerned with the theoretical analysis
and computational modeling of biological neural systems. Since neural systems are intimately
related to cognitive processes and behavior, the field is closely related to cognitive andbehavioral modeling.
The aim of the field is to create models of biological neural systems in order to understand howbiological systems work. To gain this understanding, neuroscientists strive to make a link
between observed biological processes (data), biologically plausible mechanisms for neural
processing and learning (biological neural networkmodels) and theory (statistical learningtheory andinformation theory).
[edit] Types of models
Many models are used in the field defined at different levels of abstraction and modelingdifferent aspects of neural systems. They range from models of the short-term behavior ofindividual neurons, models of how the dynamics of neural circuitry arise from interactions
between individual neurons and finally to models of how behavior can arise from abstract neural
modules that represent complete subsystems. These include models of the long-term, and short-
term plasticity, of neural systems and their relations to learning and memory from the individualneuron to the system level.
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=13http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=13http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=13http://en.wikipedia.org/wiki/Function_approximationhttp://en.wikipedia.org/wiki/Function_approximationhttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Statistical_classificationhttp://en.wikipedia.org/wiki/Statistical_classificationhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Data_processinghttp://en.wikipedia.org/wiki/Data_processinghttp://en.wikipedia.org/wiki/Roboticshttp://en.wikipedia.org/wiki/Roboticshttp://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=14http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=14http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=14http://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=15http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=15http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=15http://en.wikipedia.org/wiki/Biological_neuron_modelshttp://en.wikipedia.org/wiki/Biological_neuron_modelshttp://en.wikipedia.org/wiki/Biological_neuron_modelshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=15http://en.wikipedia.org/wiki/Information_theoryhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Computational_neurosciencehttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=14http://en.wikipedia.org/wiki/E-mail_spamhttp://en.wikipedia.org/wiki/Data_mininghttp://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-Balabin_2009-1http://en.wikipedia.org/wiki/Computer_numerical_controlhttp://en.wikipedia.org/wiki/Roboticshttp://en.wikipedia.org/wiki/Data_processinghttp://en.wikipedia.org/wiki/Novelty_detectionhttp://en.wikipedia.org/wiki/Pattern_recognitionhttp://en.wikipedia.org/wiki/Statistical_classificationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Fitness_approximationhttp://en.wikipedia.org/wiki/Time_series_predictionhttp://en.wikipedia.org/wiki/Regression_analysishttp://en.wikipedia.org/wiki/Function_approximationhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=138/3/2019 Neural Networ;
12/38
[edit] Current research
This section does notciteanyreferences or sources.Please help improve this section by adding citations toreliable sources. Unsourced material may be
challengedandremoved.(June 2010)
While initial research had been concerned mostly with the electrical characteristics of neurons, aparticularly important part of the investigation in recent years has been the exploration of the role
ofneuromodulatorssuch asdopamine,acetylcholine, andserotoninon behavior and learning.
Biophysicalmodels, such asBCM theory, have been important in understanding mechanisms for
synaptic plasticity, and have had applications in both computer science and neuroscience.
Research is ongoing in understanding the computational algorithms used in the brain, with somerecent biological evidence forradial basis networksandneural backpropagationas mechanisms
for processing data.
Computational devices have been created in CMOS for both biophysical simulation andneuromorphic computing. More recent efforts show promise for creatingnanodevicesfor very
large scaleprincipal componentsanalyses andconvolution. If successful, these effort could usherin a new era ofneural computingthat is a step beyond digital computing, because it depends on
learningrather thanprogrammingand because it is fundamentallyanalograther thandigitaleven
though the first instantiations may in fact be with CMOS digital devices.
[edit] Neural network software
Main article:Neural network software
Neural network software is used tosimulate,research,developand apply artificial neural
networks,biological neural networksand in some cases a wider array ofadaptive systems.
[edit] Types of artificial neural networks
Main article:Types of artificial neural networks
Artificial neural network types vary from those with only one or two layers of single directionlogic, to complicated multiinput many directional feedback loop and layers. On the whole, these
systems use algorithms in their programming to determine control and organization of their
functions. Some may be as simple, one neuron layer with an input and an output, and others canmimic complex systems such asdANN, which can mimic chromosomal DNA through sizes atcellular level, into artificial organisms and simulate reproduction, mutation and population
sizes.[3]
Most systems use "weights" to change the parameters of the throughput and the varying
connections to the neurons. Artificial neural networks can be autonomous and learn by inputfrom outside "teachers" or even self-teaching from written in rules.
http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=16http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=16http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=16http://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Biophysicshttp://en.wikipedia.org/wiki/Biophysicshttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/Synaptic_plasticityhttp://en.wikipedia.org/wiki/Synaptic_plasticityhttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Learninghttp://en.wikipedia.org/wiki/Learninghttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=17http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=17http://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=18http://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=18http://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/Artificial_neural_network#cite_note-2http://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects#Software_librarieshttp://en.wikipedia.org/wiki/Types_of_artificial_neural_networkshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=18http://en.wikipedia.org/wiki/Adaptive_systemhttp://en.wikipedia.org/wiki/Biological_neural_networkhttp://en.wikipedia.org/wiki/Technology_developmenthttp://en.wikipedia.org/wiki/Researchhttp://en.wikipedia.org/wiki/Simulationhttp://en.wikipedia.org/wiki/Neural_network_softwarehttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=17http://en.wikipedia.org/wiki/Digitalhttp://en.wikipedia.org/wiki/Analog_signalhttp://en.wikipedia.org/wiki/Programminghttp://en.wikipedia.org/wiki/Learninghttp://en.wikipedia.org/wiki/Neural_computinghttp://en.wikipedia.org/wiki/Convolutionhttp://en.wikipedia.org/wiki/Principal_componenthttp://en.wikipedia.org/w/index.php?title=Nanodevice&action=edit&redlink=1http://en.wikipedia.org/wiki/Neural_backpropagationhttp://en.wikipedia.org/wiki/Radial_basis_networkshttp://en.wikipedia.org/wiki/Synaptic_plasticityhttp://en.wikipedia.org/wiki/BCM_theoryhttp://en.wikipedia.org/wiki/Biophysicshttp://en.wikipedia.org/wiki/Serotoninhttp://en.wikipedia.org/wiki/Acetylcholinehttp://en.wikipedia.org/wiki/Dopaminehttp://en.wikipedia.org/wiki/Neuromodulatorshttp://en.wikipedia.org/wiki/Wikipedia:Verifiability#Burden_of_evidencehttp://en.wikipedia.org/wiki/Template:Citation_neededhttp://en.wikipedia.org/wiki/Wikipedia:Identifying_reliable_sourceshttp://en.wikipedia.org/wiki/Wikipedia:Verifiabilityhttp://en.wikipedia.org/wiki/Wikipedia:Citing_sourceshttp://en.wikipedia.org/w/index.php?title=Artificial_neural_network&action=edit§ion=168/3/2019 Neural Networ;
13/38
[edit] Theoretical properties
[edit] Computational power
The multi-layerperceptron(MLP) is a universal function approximator, as proven by the
Cybenko theorem. However, the proof is not constructive regarding the number of neuronsrequired or the settings of the weights.
Work byHava SiegelmannandEduardo D. Sontaghas provided a proof that a specific recurrentarchitecture with rational valued weights (as opposed to full precisionreal number-valued
weights) has the full power of aUniversal Turing Machine[4]
using a finite number of neurons
and standard linear connections. They have further shown that the use of irrational values for
weights results in a machine withsuper-Turingpower.
[edit] Capacity
Artificial neural network models have a property called 'capacity', which roughly corresponds to
their ability to model any given function. It is related to the amount of information that can be
stored in the network and to the notion of complexity.
[edit] Convergence
Nothing can be said in general about convergence since it depends on a number of factors.Firstly, there may exist many local minima. This depends on the cost function and the model.
Secondly, the optimization method used might not be guaranteed to converge when far away
from a local minimum. Thirdly, for a very large amount of data or parameters, some methodsbecome impractical. In general, it has been found that theoretical guarantees regarding
convergence are an unreliable guide to practical application.
[edit] Generalization and statistics
In applications where the goal is to create a system that generalizes well in unseen examples, the
problem of over-training has emerged. This arises in convoluted