Neural NetworksNeural Networks

Si Wu

Dept. of Informaticssiwu@sussex.ac.uk

PEV III 5c7

Spring 2008

Today’s Topics: Course Organization The origin of neural networks

Teaching Methods

• Lectures -- 2 hours per week,

Thursday, 9-11am, PEV 1 1A06

•Reading week: Week 5• Seminar– 1-2 hour per week (Mr. Thomas Baker)

•3rd: Thursday, 11-12pm, Chichester 1 CI003

•Msc: Monday, 14-16pm, Chichester 1 CI204/5

• Office hour: Thursday 14-16pm, PEV III 5c7

•Lecture notes will be available online:

http://www.informatics.sussex.ac.uk/users/siwu

•The course covers the fundamental theory of artificial neural networks (ANN), and will present basic models and learning algorithms of ANN.

•Seminars will be supervised Mr. Thomas Baker, and will be used to answer concerns related to the course and courseworks.

•Coursework will give you chance to practice ANN.

Course SummaryCourse Summary

Topics to be covered

1. Basics of the neural networks method2. Single layer perceptrons3. Multilayer networks4. Radial basis function networks5. Principle component analysis6. Bayesian inference 7. Support Vector Machines

Assessment

• Only coursework

• Coursework consists of two parts: –The first one, for both 3rd and Msc, counts 40%, due in Thursday 4pm, week 5 –The second one 60%

•For 3rd year, due in Thursday 4pm, week 10•For Msc, due in Monday 12:00 noon, week 1 of Summer Semester

Recommended Textbooks1. Haykin S (1999). Neural networks. Prentice Hall International. Excellent

but quite heavily mathematical 2. Bishop C (1995). Neural networks for pattern recognition. Oxford:

Clarendon Press (good but a bit statistical, not enough dynamical theory)3. Pattern Classification, John Wiley, 2001

R.O. Duda and P.E. Hart and D.G. Stork4. Hertz J., Krogh A., and Palmer R.G. Introduction to the theory of neural

computation (nice, but somewhat out of date)5. Pattern Recognition and Neural Networks by Brian D. Ripley. Cambridge

University Press. Jan 1996. ISBN 0 521 46086 7. 6. Neural Networks. An Introduction, Springer-Verlag Berlin, 1991 B. Mueller

and J. Reinhardt

Find the one that best suits your background.

What is Neural Networks?

• Inspired from real neural systems• Having a network structure, consisting of

nodes (artificial neurons) and weights (neuronal connection)

• A general methodology for function approximation

How neural systems look like?

The structure of neural systems

• Neuron: the fundamental singaling/computational units • Synapses: the connections between neurons• Layer: neurons are organized into layers• Extremely complex: around 1011 neurons in the brain,

each with 103 connections

Why do we learn from neural systems?

• The brain is still superior than modern computers in many aspects

• A different style of computation: parallel distributed processing

• Adaptive and can learn new knowledge• An universal computational architecture:

the same structure carries out many different functions

The function of a single neuron

Hillockinput

output

CellCell

CellCell

Membrane potential

The state of neuron

Membrane potential: the voltage difference between the cell bodyand the surrounding.

spike

Neuron as a computational unit

•Neuron fires when its input is larger than a threshold•Two states: on & off

The Idea of Artificial Neural Networks

• A single neuron’s function is simple, the specialty of brain functions is on the network structure

• ANN is to mimic the network structure of neural systems• The constitution of ANN

– Nodes: artificial neurons, performing a fixed linear or non-linear mapping– Network Weights: interactions between nodes– Layers: nodes are organized into layers– Connection style: feed-forward, feed-back or recurrent

• ANN is more and more engineering-driven nowadays, its biological root is gradually losing. The key of ANN is on the design and training of suitable network structures

An example of one-layer feed-forward neural network

x

w

y

b

m

iii bxwfy

1

)(

xn

x1

x2

Input Output

An Example of Three-Layer Feed-forward Networks

Hidden layers