Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly

Introduction

CS/CMPE 537 – Neural Networks

CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS 2

Biological Inspiration

The brain is a highly complex, nonlinear, and parallel computer Simple processing units called neurons Cycle times in milliseconds Massive number of neurons (estimated at 1011 in humans) Massive interconnection (estimated at 6012 connections)

The brain can perform certain computations (e.g. pattern recognition, perception, etc) many times faster than the fastest digital computers available today


Comparison of the Brain and Digital Computer

Functionality comparison

Brain Digital computer

Fault tolerant Intolerant to errors

Adaptive Preprogrammed

Learn-ability Preprogrammed

Trained Designed

Nonlinear Linear, primarily


Comparison of the Brain and Digital Computer

Structural comparison

Brain Digital computer

Simple processing unit Complex processing unit

Large number of units Lesser number of units

Massive interconnection Little or no interconnection

Dynamic Static

Slow switching Faster switching


What is an Artificial Neural Network? (1)

Conceptual definitions An artificial neural network (neural network) is a

machine that is designed to model the way in which the brain performs a particular task or function of interest; the network is either implemented in hardware or simulated in software on a digital computer.

A neural network mimics the brain or nervous system. In what sense? In structure (simple processing units, massive

interconnection, etc) In functionality (learning, adaptability, fault tolerance, etc)


What is an Artificial Neural Network? (2)

A pragmatic definition A neural network is a massively parallel distributed

computing system that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects: Knowledge is acquired by the network through a learning

process (called training) Interneuron connection strengths known as synaptic weights

are used to store the knowledge Other names for neural networks

Neurocomputers Connectionist networks Parallel distributed processors


Significance of Artificial Neural Networks

Why study neural networks? To develop artificial systems that posses human-like

characteristics (thought and action) Neural computation

To understand the working of the brain Computational neuroscience

These two fields are not mutually exclusive. Knowledge gained in each helps developments in the other.

Our primary concern will be neural computation.


Characteristics of Neural Networks

Nonlinearity Input-output mapping Adaptivity Evidential response Contextual information Fault tolerance VLSI implementability Uniformity of analysis and design Neurobiological analogy


Structural Levels of Organization in the Brain

Conceptual organization of the nervous system Receptors Neural network Effectors


Structural Levels in the Brain


Biological Neuron


Biological Neuron

Cell body – simple processing unit Axon – output link Dendrite – input link Synapse – connection between neuron


Neuron Model (1)

Bias bk

vk


Neuron Model (2)

Properties A set of signals x1 to xp

A set of connection weights w associated with each of the p connecting links (synapses)

A processing unit (or neuron) k that performs the weighted summation of the inputs (also known as adder)

An activation function φ(.) that limits the amplitude of the output, usually to the range {0, 1} or {-1, 1} (also known as squashing or transfer function)

A bias bk that has the effect of lowering or increasing the net input to the activation function


Neuron Model (3)

Mathematically…(using a threshold)

uk = Σj=1p wkjxj (linear combiner)

yk = φ(uk + bk)

Mathematically…(using a bias unit)

vk = Σj=0p wkjxj (linear combiner)

yk = φ(vk)

where vk = uk + bk

and x0 = +1 and wk0 = bk


Activation Functions (1)

Threshold function



Piecewise linear function



Sigmoid function


Signal-Flow Graph Representation

A cleaner (simpler) representation that captures the various elements of a neural network and the flow of information (signals) Directed graph – network of

arrows and nodes

Rules for signal transmission


Signal-Flow Graph Representation of Neuron


Neural Net Definition – Signal-Flow Rep.

A neural net is a directed graph consisting of nodes with interconnecting synaptic and activation links, and which is characterized by four properties: Each neuron is represented by a set of linear synaptic links,

an externally applied threshold, and a nonlinear activation link

The synaptic links of a neuron weight their respective input signals

The weighted sum of the input signals defines the total internal activity level of the neuron in question

The activity link squashes the internal activity level of the neuron to produce an output


Architectural Graph Representation

A signal-flow graph that omits details of flow inside the neuron A partially complete directed graph


Classification of Neural Networks (1)

Number of layers Single layer network Multilayer networks

Direction of information (signal) flow Feedforward Recurrent

Connectivity Fully connected Partially connected


Classification of Neural Networks (2)

Activation function Threshold networks Linear networks Nonlinear networks Radial-basis function networks

Learning methodology Supervised Unsupervised Reinforcement

Training algorithm Static Dynamic


Example


Example


Example


Example


Example


Design Decisions

Architecture No. of layers, units, links; connectivity Activation functions

Learning Supervised, unsupervised, competitive

Training algorithm Iterative, static, dynamic, epoch-based, sample-based


Knowledge Representation (1)

Neural networks encode and make available knowledge for information processing (knowledge representation)

What is knowledge? Information or models used to interpret, predict and

appropriately respond to the outside world (environment) How can knowledge be represented?

Explicit – what is made explicit Physically encoded – how can is physically encoded

The quality of knowledge representation generally translates into the quality of the solution –better representation -> better solution


Knowledge Representation (2)

Knowledge is encoded in the free parameters of the network. Assuming the architecture is fixed, the free parameters are…. Weights and bias values

Knowledge representation in neural nets is complex Few theories exist that relate a given weight, for example, to

a particular piece of information

Hmm, so neural nets are worthless? Nope!


Learning

Learning: to acquire and maintain knowledge of interest

Knowledge of interest: knowledge of environment that will enable the machine (neural net) to achieve its goals Prior information Current information

Knowledge can be built into neural networks from input-output examples by a automatic process of learning (commonly known as training) Training algorithms


Some General Knowledge Rep. Rules

Rule 1: Similar inputs from similar classes should usually produce similar representations inside the network, and should therefore be classified as belonging to the same category

Rule 2: Inputs to be characterized as separate classes should be given widely different representations in the network

Rule 3: If a particular feature is important, then there should be a large number of neurons involved in the representation of that item in the network

Rule 4: Prior knowledge and invariances should be built into the design of the network, thus simplifying learning


Building-in Prior Knowledge

Advantages Specialized structure Benefits of having a specialized structure – biologically

plausible, less complication, fewer free parameters, faster training, fewer examples needed, better generalization, etc

How to build-in prior knowledge? No hard and fast rules In general, use domain knowledge to reduce complexity of

neural network based on what we know about their performance characteristics


Building-in Invariance

Invariance? Fault tolerance Immunity to transformations

Invariance by structure Invariance by training Invariant feature space


Building-in Current Knowledge

Learning and training algorithms Subject of next 2 lectures (chapter 2)


An Example

A neural net for signature verification Prior knowledge

Architecture

Current knowledge Input-output examples

Learning Modification of the free parameters (weights, biases, etc)

Generalization Using the trained network for prediction


Application Areas of Neural Networks

Model estimation Interpolation Extrapolation

Pattern classification Lots of examples

Signal processing Noise reduction, enhancement, echo removal

Optimization Lots of examples

Documents

Introduction CS/CMPE 537 – Neural Networks. CS/CMPE 537 - Neural Networks (Sp 2004/2005) - Asim Karim @ LUMS2 Biological Inspiration The brain is a highly