KULIAH II JST: BASIC CONCEPTS Amer Sharif, S.Si, M.Kom

KULIAH II JST:BASIC CONCEPTS

Amer Sharif, S.Si, M.Kom

INTRODUCTION REVIEW

Neural Network definition: A massively parallel distributed

processor of simple processing units (neuron)

Store experiential knowledge and make it available for use

Knowledge is acquired from the environment through learning process

Knowledge is stored as internerneuron connection strengths (synaptic weights)

INTRODUCTION REVIEW

Benefits: Nonlinearity Input Output Mapping Adaptivity

Evidential Response Contextual Information

Fault Tolerance/Graceful Degrading VLSI Implementability Uniform Analysis & Design

NEURON MODELLING

Basic elements of neuron: A set of synapses or connecting links

Each synapse is characterized by its weight Signal xj at synapse j connected to neuron k

is multiplied by synaptic weight wkj Bias is bk

An adder for summing the input signals An activation function for limiting the

output amplitude of the neuron

NEURON MODELLING

Block diagram of a nonlinier neuron

Inputsignals

Biasbk

x1 wk1

wk2

wkm

x2

xm

Summingjunction

vk

Activationfunction

Output

yk

Synapticweights

.

.

.

.

.

.

m

jjkjk xwu

1

buy kkk

NEURON MODELLING

Note x1, x2,…, xm are input signals wk1, wk2,…, wkm are synaptic weights of

neuron k uk is the linier combiner output bk is bias is the activation function yk is the output signal of the neuron

NEURON MODELLING

If and bias is substituted for a synapse where

x0 = + 1 with weight wk0 = bk

then and

buv kkk

m

jjkjk xwv

0

vy kk

NEURON MODELLING

Modified block diagram of a nonlinier neuron

Inputsignals

wk0=bk (bias)

x1 wk1

wk2

wkm

x2

xm

Summingjunction

vk

Activationfunction

Output

yk

Synapticweights

.

.

.

.

.

.Fixed input x0= +1

wk0

m

jjkjk xwv

0

vy kk

ACTIVATION FUNCTIONS

Activation Function types: Threshold Function

and

also known as the McCulloch-Pitts model

00

01

k

k

k vif

vify

bxwv k

m

jjkjk

1

-2 -1 0 1 2

v

1.2

1

0.8

0.6

0.4

0.2

0

v


Piecewise-Linear Function

0

0.2

0.4

0.6

0.8

1

1.2

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2

2

1,0

2

1

2

1,

2

12

1,1

v

vv

v

v


Sigmoid Function S-shaped Sample logistic

function:

a is the slope parameter: the larger a the steeper the function

Differentiable everywhere

)exp(1

1)(

avv

0

0.2

0.4

0.6

0.8

1

1.2

-10 -8 -6 -4 -2 0 2 4 6 8 10

v

f(v)

0

0.2

0.4

0.6

0.8

1

1.2

increasing

a

NEURAL NETWORKS AS DIRECTED GRAPHS

Neural networks maybe represented as directed graphs: Synaptic links

(linier I/O) Activation links

(nonlinier I/O) Synaptic convergence

Synaptic divergence

wkjxj

yk= wkj xj

xj

xy jk

yk=yi + yj

yi

yj

xj

xj

xj

NEURAL NETWORKS AS DIRECTED GRAPHS Architectural graph: partially complete

directed graph

Outputyk

x0 =+1

xm

x2

x1

.

.

.

FEEDBACK

Output of a system influences some of the input applied to the system

One or more closed paths of signal transmission around the system

Feedback plays an important role in recurrent networks

FEEDBACK

Sample single-loop feedback system

Output signal yk(n) is an infinite weighted summation of present and past samples of input signal xj(n)

0

1 )()(l

jl

klnn xwy )( lnx j

x’j (n)xj(n) yk(n)

w

z-1

w is fixed weight

z-1 is unit-delay operator

is sample of input signal delayed by l time units

FEEDBACK

Dynamic system behavior is determined by weight w

yk(n)

wxj(0)

0 1 2 3 4 nw < 1 System is exponentially convergent/stable System posses infinite memory: Output depends on

input samples extending into the infinite past Memory is fading: influence of past samples is

reduced exponentially with time n

w < 1

FEEDBACK

w = 1 System is linearly

divergent w > 1

System is exponentially divergent

yk(n)

wxj(0)

0 1 2 3 4 n yk(n)

wxj(0)

0 1 2 3 4 n

w = 1

w > 1

NETWORK ARCHITECTURES

Single-Layered Feedforward Networks

input layer ofsource nodes

output layer ofneurons

Neurons are organized in layers

“Single-layer” refers to output neurons

Source nodes supply to output neurons but not vice versa

Network is feedforward or acyclic


One or more hidden layers

Hidden neurons enable extractions of higher-order statistic

Network acquires global perspective due to extra set of synaptic connections and neural interactions

Multilayer Feedforward Networks

Input layer ofsource nodes

Layer of hiddenneurons

Layer of outputneurons

7-4-2 fully connected network:

• 7 source nodes• 4 hidden neurons• 2 output neurons


Recurrent Networks

At least one feedback loop

Feedback loops affect learning capability and performance of the network

z-1

z-1

z-1

z-1

Unit-delayoperators

Inputs

Outputs

KNOWLEDGE REPRESENTATION

Definition of Knowledge: Knowledge refers to stored information or

models used by a person or a machine to interpret, predict, and appropriately respond to the outside world

Issues: What information is actually made explicit How information is physically encoded for

subsequent use Knowledge representation is goal-directed Good solution depends on good

representation of knowledge

KNOWLEDGE REPRESENTATION Challenges faced by Neural Networks:

Learn the model of the world/environment Maintain the model to be consistent with the

real world to achieve the goals desired Neural Networks may learn from a set of

observations data in form of input-output pairs (training data/training sample)

Input is input signal and output is the corresponding desired response

KNOWLEDGE REPRESENTATION Handwritten digit recognition problem

Input signal: one of 10 images of digits Goal: to identify image presented to the

network as input Design steps:

Select the appropriate architecture Train the network with subset of examples (learning

phase) Test the network with presentation of data/digit image

not seen before, then compare response of network with actual identity of the digit image presented (generalization phase)

KNOWLEDGE REPRESENTATION Difference with classical pattern-classifier:

Classical pattern-classifier design steps: Formulate mathematical model of the problem Validate model with real data Build based on model

Neural Network design is: Based on real life data Data may “speak for itself”

Neural network not only provides model of the environment but also process the information

ARTIFICIAL INTELLIGENCE AND NEURAL NETWORKS

AI systems must be able to: Store knowledge Use stored knowledge to solve problem Acquire new knowledge through experience

AI components: Representation

Knowledge is presented in a language of symbolic structures

Symbolic representation makes it relatively easy for human users


Reasoning Able to express and solve broad range of problems Able to make explicit and implicit information

known to it Have a control mechanism to determine which

operation for a particular problem, when a solution is obtained, or when further work on the problem must be terminated

Rules, Data, and Control: Rules operate on Data Control operate on Rules

The Travelling Salesman Problem: Data: possible tours and cost Rules: ways to go from city to city Control: which Rules to apply and when


Learning

Inductive learning: determine rules from raw data and experience

Deductive learning: use rules to determine specific facts

Environment Learning

element

Knowlegdge Base

Performance

element


Parameter Artificial

Intelligence

Neural Networks

Level of Explanation Symbolic representation with sequential processing

Parallel distributed processing (PDP)

Processing Style Sequential Parallel

Representational Structure

Quasi-linguistic structure

Poor

Summary Formal manipulation of algorithm and data representation in top down fashion

Parallel distributed processing with natural ability to learn in bottom up fashion

Documents

KULIAH II JST: BASIC CONCEPTS Amer Sharif, S.Si, M.Kom