Introduction to Neural Introduction to Neural NetworksNetworks

Neural Nets slides mostly from: Andy Philippides,University of Sussex

Uses of NNs Neural Networks Are ForApplications Science

Character recognition Neuroscience

Optimization Physics, mathematics statistics

Financial prediction Computer science

Automatic driving Psychology

.............................. ...........................

What are biological NNs?• UNITs: nerve cells called neurons, many different

types and are extremely complex• around 1011 neurons in the brain (depending on

counting technique) each with 103 connections• INTERACTIONs: signal is conveyed by action

potentials, interactions could be chemical (release or receive neurotransmitters) or electrical at the synapse

• STRUCTUREs: feedforward and feedback and self-activation recurrent

“The nerve fibre is clearly a signalling mechanism of limited scope. It can only transmit a succession of brief explosive waves, and the message can only be varied by changes in the frequency and in the total number of these waves. … But this limitation is really a small matter, for in the body the nervous units do not act in isolation as they do in our experiments. A sensory stimulus will usually affect a number of receptor organs, and its result will depend on the composite message in many nerve fibres.” Lord Adrian, Nobel Acceptance Speech, 1932.

We now know it’s not quite that simple

• Single neurons are highly complex electrochemical devices

• Synaptically connected networks are only part of the story

• Many forms of interneuron communication now known – acting over many different spatial and temporal scales

The complexity of a neuronal system can be partly seen from a picture in a book on computational neuroscienceedited by Jianfeng

How do we go from real neurons to artificial ones?

Hillock

input

output

Single neuron activity

• Membrane potential is the voltage difference between a neuron and its surroundings (0 mV)

CellCell

CellCell

0 Mv

Membrane potential

Single neuron activity •If you measure the membrane potential of a neuron and print it out on the screen, it looks like:

spike

Single neuron activity

•A spike is generated when the membrane potential is greater than its threshold

Abstraction•So we can forget all sub-threshold activity and concentrate on spikes (action potentials), which are the signals sent to other neurons

Spikes

• Only spikes are important since other neurons receive them (signals)

• Neurons communicate with spikes

• Information is coded by spikes

• So if we can manage to measure the spiking time, we decipher how the brain works ….

Again its not quite that simple

• spiking time in the cortex is random

With identical input for the identical neuron

spike patterns are similar, but not identical

Recording from a real neuron: membrane potential

Single spiking time is meaninglessTo extract useful information, we have to average

to obtain the firing rate r

for a group of neurons in a local circuit where neuron codes the same information over a time window

Local circuit

=

Time window = 1 sec

r =

Hz

So we can have a network of these local groups

w1: synaptic strength

wn

r1

rn

R f w rj j ( )

Hence we have firing rate of a group of neurons

ri is the firing rate of input local circuit

The neurons at output local circuits receives signals in the form

The output firing rate of the output local circuit is then given by R

where f is the activation function, generally a Sigmoidal function of some sort

€

wiri

i=1

N

∑

€

R = f ( wiri

i=1

N

∑ )

wi weight, (synaptic strength) measuring the strength of the interaction between neurons.

Artificial Neural networks

Local circuits (average to get firing rates)

Single neuron (send out spikes)

Artificial Neural Networks (ANNs)

A network with interactions, an attempt to mimic the brain• UNITs: artificial neuron (linear or nonlinear input-

output unit), small numbers, typically less than a few hundred

• INTERACTIONs: encoded by weights, how strong a neuron affects others

• STRUCTUREs: can be feedforward, feedback or recurrent

It is still far too naïve as a brain model and an information processing

xn

x1

x2

Input

(visual input)

Output

(Motor output)

Four-layer networks

Hidden layers

The general artificial neuron model has five components, shown in the following list. (The subscript i indicates the i-th input or weight.)

• A set of inputs, xi.

• A set of weights, wi.

• A bias, u.

• An activation function, f.

• Neuron output, y

m

jijiji bxwfy

1

)(

Thus the key to understanding ANNs is to understand/generate the local input-output relationship