Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons

Computational Intelligence in Complex Decision Systems G. Oltean

Neural Network Concepts

and Paradigms


Biological NeuronNeurons: nerve cells; consist of dendrites, body and an axon;

signals flow through synapses.

Axon(output)

dendrites => body => axon

Axons connects to dendrites via synapses Synapses

vary in strength may be excitatory or inhibitory (+ or -)

an electrically excitable cell that processes and transmits

information through electrical and chemical signals.


Human Brain

on average the human brain has

• on average 86 billions neurons

• ~1000 main modules /~500 neural network each /~100 000 neurons

Properties of the brain:

It can learn, reorganize itself from experience

It adapts to the environment

It is robust and fault tolerant


Artificial Neural Networks

Some differences between biological neurons and

artificial neurons (processing elements - PE):

* Signs of weights (+ or -)

* Signals are ac in biological neurons, dc in PEs

* Many types of neurons in a system; usually only

a few at most in neural networks

* Basic cycle time for PC (<1 ns) faster than

brain (10-100ms) {as far as we know!}

An analysis paradigm very roughly modeled after themassively parallel structure of the brain.

Simulates a highly interconnected, parallel computationalstructure with numerous relatively simple individualprocessing elements (PE ).


What ANNs are and why they are useful

In the simplest terms, ANNs map input vectors to output vectors

ANNs consist of processing elements (PEs) or

neurons and weighted connections

• Operations performed by ANNs can include:* Fitting (approximation) and modeling* Pattern recognition (Classification)* Prediction* Pattern matching and completion* Noise removal* Optimization* Simulation* Control


What NNs are and why they are useful

• Connection weights store the information

• Weight values usually determined by a learning procedure

• Each PE acts independently of all others

• Each PE relies only on local information

• Connection pattern provides redundancy and facilitates fault tolerance

• ANNs are able to “learn” arbitrary non-linear mappings


7 / 20

Housing Price

Prediction

https://www.coursera.org/learn/neural-networks-deep-learning/lecture/eAE2G/what-is-a-neural-network

https://www.coursera.org/learn/neural-networks-deep-learning/lecture/eAE2G/what-is-a-neural-network


8 / 20

https://www.coursera.org/learn/neural-networks-deep-learning/lecture/2c38r/supervised-learning-with-neural-networks



9 / 20




10 / 20


People are just really good at interpreting unstructured data.

It has been much harder for computers to make sense of unstructured data compared to structured data.

Thanks to deep learning, thanks to neural networks, computers are now much better at interpreting unstructured data



11 / 20


Feedforward Neural Network

Input layer Hidden layer Output layer


Principal elements required to specify ANNs

Topology - layers and interconnections

Learning - how network is configured to store information

Recall - How stored information is retrieved


Neuron (PE) Model

- summing component

f - activation function

𝑠 = 𝑤1𝑥1 + 𝑤2𝑥2 + …+ 𝑤𝑁𝑥𝑁 + 𝑏

If N = 1, the weighted sum is in fact the simple linear regression

𝑠 = 𝑤1𝑥1 + 𝑏

If N > 1, the weighted sum is in fact the multiple linear regression

b is the intercept


Neuron (PE)

Model

TNxxxx ,...,, 21

Nwwww ,...,, 21

Input vector

Weight vector

Bias

N

i

ii bxwfysfy1

)(

bwxsbxwsN

i

ii 1

- summing component

f - activation function

b

multiple linear regression


Problem

5.2];5.025.0[

;4

2;2

bw

xN

)(sf

s = ?; y = ?

Implement in a programing language the above neuron


Neuron Model as Logistic Regression

TNxxxx ,...,, 21Input vector

Binary classification

1 – positive (in the class)

0 – negative (not in the class)

features vector (of the objects)

Output 𝑦 ∈ 0, 1

We want our model to predict the chance for the object

described by a set of features x to be in the class

𝑦 ∈ [0, 1] 0 ≤ 𝑦 ≤ 1


Neuron Model as Logistic Regression

TNxxxx ,...,, 21

Binary classification

𝑦 ∈ [0, 1] 0 ≤ 𝑦 ≤ 1

Activation function f

Sigmoidal (logistic function)

𝑦 =1

1+e−𝑠


19 / 20

Phyton 3.5 (Spyder) implementation using NumPy module


20 / 20

Phyton 3.5 (Spyder) results

x=24

w = [0.25, -0.5]b = 2.5

s= 1.0y = 0.7615941559557649

x=24-0.3

w = [0.25, -0.5, 1]b = 0.5

s= -1.3y = -0.8617231593133063

x=24-0.8

w = [0.25, -0.5, 1]b = -0.99

s=-3.29y = -0.9972281483227239


Activation(Transfer) Functions

hardlims

0,1

0,1)(

s

ssy

)(sf

)(sf


Activation(Transfer) Functions

)(sf

)(sf


Simulink Transfer Functions


ANN with 1 layer 2 inputs, 3 outputs

2

1

x

xx

3231

2221

1211

ww

ww

ww

w

3

2

1

b

b

b

b

3

2

1

s

s

s

s

3

2

1

o

o

o

o

bwxs )(sfo

3

2

1

2

1

3231

2221

1211

3

2

1

b

b

b

x

x

ww

ww

ww

s

s

s


Exercise

3

5.0

5.0

5.075.0

11.0

15.0

bw

0

10

4

2

4

1321 xxx

sesf

1

1)(

Compute the output vector o, for each input vector xi

Implement in a programing language the above example


26 / 20

Phyton 3.5 (Spyder) implementation


27 / 20

The one-layer user-defined function is written in a separate

file (ANNfunc.py)


28 / 20

The input is:[[-1. ][-2. ][ 0.33]]

The weights matrix is:[[-0.5 -0.15 -0.1 ][ 0.23 0.11 -0.5 ][-0.5 1. -0.1 ][ 0.045 0.15 -0.05 ]]

The bias is:[[ 0.2 ][-0.5 ][-0.03][ 0.36]]

The activation function is: tanh

The output is:[[ 0.747383 ][-0.80582272][-0.91590514][-0.0015 ]]

Phyton 3.5 (Spyder) results


It is common for different layers to have different numbers of neurons.

Multiple-layer networks are quite powerful.

Feedforward Neural Network

An ANN of two layers, where the first layer is sigmoid and the second layer is linear, can be trained to approximate any function (with a finite number of discontinuities) arbitrarily well - universal aproximator


Two layers: one hidden layer and one output layer

N inputs; H neurons in the hidden layer;

K outputs (neurons in the output layer)

hidden

layer

output

layer

output input

1x

hW

2x

Nx

1

hb1

hs1 hy1

1

1

hf

1

1

1

1o

oW

hs2

h

Hs

hf

hf

of

of

of

hb2

Hb

hy2

h

Hy

os1

os2

o

Ks

2o

Ko

ob1

ob2

o

Kb

∑

∑

∑

∑

∑

∑


hidden layer:

h

HN

h

H

h

H

h

N

hh

h

N

hh

h

www

www

www

w

21

22221

11211

a

H

a

a

a

b

b

b

b2

1

hidden

layer

output

layer

output input

1x

hW

2x

Nx

1

hb1

hs1 hy1

1

1

hf

1

1

1

1o

oW

hs2

h

Hs

hf

hf

of

of

of

hb2

Hb

hy2

h

Hy

os1

os2

o

Ks

2o

Ko

ob1

ob2

o

Kb

∑

∑

∑

∑

∑

∑


output layer:

o

KH

o

K

o

K

o

H

oo

o

H

oo

o

www

www

www

w

21

22221

11211

o

K

o

o

o

b

b

b

b2

1

hidden

layer

output

layer

output input

1x

hW

2x

Nx

1

hb1

hs1 hy1

1

1

hf

1

1

1

1o

oW

hs2

h

Hs

hf

hf

of

of

of

hb2

Hb

hy2

h

Hy

os1

os2

o

Ks

2o

Ko

ob1

ob2

o

Kb

∑

∑

∑

∑

∑

∑


hhhh bxwfy Th

H

hhh yyyy 21

ohoo bywfO

ohhhoo bbxwfwfO

Transfer function:

hidden

layer

output

layer

output input

1x

hW

2x

Nx

1

hb1

hs1 hy1

1

1

hf

1

1

1

1o

oW

hs2

h

Hs

hf

hf

of

of

of

hb2

Hb

hy2

h

Hy

os1

os2

o

Ks

2o

Ko

ob1

ob2

o

Kb

∑

∑

∑

∑

∑

∑


Exporting an ANN trained in Matlabto simulate it in another environment (e.g. Python)

In Matlab, training a ANN generally involves a normalization step that is

applied to both the input vectors and the target vectors in the data set.

The network output always falls into a normalized range, [-1, 1].

The network output can then be reverse transformed back into the units

of the original target data when the network is put to use in the field

(simulated in Matlab).

For correct export of the ANN, remove all preprocessing and

postprocessing functions that MATLAB executes on inputs and outputs,

(or implement those function in the new environment)

For a two-layer ANN, in the script (function) used to create and train the

ANN, write:

net.inputs{1}.processFcns = {}; % for inputs

net.outputs{2}.processFcns = {}; % for targets

Default setting is: net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};

net.outputs{2}.processFcns = {'removeconstantrows','mapminmax’};


Exporting an ANN trained in Matlabto simulate it in another environment (e.g. Python) – cont.

The parameters of the trained ANN (e.g. for a two-layer ANN)

wh=cell2mat(net.IW) %weights of the hidden layer

wo=cell2mat(net.LW) %weights of the output layer

bh=cell2mat(net.b(1)) % bias of the hidden layer

bo=cell2mat(net.b(2)) % bias of the output layer

afunh=net.layers{1}.transferFcn % activation function in the

hidden layer

afunh=net.layers{2}.transferFcn % activation function in the

output layer

In Matlab:

Observation: “net” is the name of the ANN

Documents

Neural Network Concepts and Paradigms · Computational Intelligence in Complex Decision Systems G. Oltean Human Brain on average the human brain has • on average 86 billions neurons