Download ppt - Machine Learning in Engineering Problems Jzau-Shenlg Lin ( 林灶生 ) Dept. of Computer Science and Information Engineering, Nat’l Chin-Yi Institute of Technology

Machine Learning in Engineering Problems

Jzau-Shenlg Lin ( 林灶生 )

Dept. of Computer Science and Information Engineering,

Nat’l Chin-Yi Institute of Technology

Outline

Introduction Artificial Neural Networks (ANN) Fuzzy-, Possibilistic-, and Rough- Systems Cerebellar Model Arithmetic Computer (CMAC) Genetic Algorithm (GA) Artificial Immune System (AIS) Ant Colony System (ACS) Support Vector Machine (SVM) Conclusions

Introduction(1/2) Machine learning is a research strategy, in which comput

ers can modeling or implement the humans’ learning behaviors.

It also reconstructs the intelligent architecture in its intelligIt also reconstructs the intelligent architecture in its intelligent base to reinforce the performance for itself.ent base to reinforce the performance for itself.

H.A. Simon indicated that learning is an adaptive activity for a system to causes the system doing the same or similar task more effectively.

R.S. Michalski thought that learning is the representation to configure or revise the experimental tasks.

The experts who design expert systems presented that learning is extracting intelligence.

Introduction(2/2)

Learning is a very important feature for the intelligent behavior.

The applications for the machine learning include: Robots Computer game Signal processing – Compressing, Recognition, watermarking, … Network topology – The shortest path, Channel assignment, … Several optimization problems in engineering

1.Artificial Neural Networks (ANN) (1/6)

An ANN is a Massively parallel distributed Processor that has a natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects:

(a) Knowledge is acquired by the network through a learning process.

(b) Interneuron connection strengths known as synaptic weights are used to

store the knowledge. Neural Networks are referred to in the literature as neurocomputers, c

onnectionist networks, parallel distributed processors, etc.

Neurobiological model

Axon( 神經軸 )— 輸出路徑Soma( 神經核 )— 細胞本體

Synapse( 神經連接線 )-- 依電位變化傳遞

Dendrites( 神經樹 )-- 輸入路徑

1. Artificial Neural Networks (ANN) (2/6)

Models of a Neuron

1kw

2kw

knw

::

)(k

kyOutput

kThreshold

Summingjunction

Activationfunction

nx

2x

1x

Synaptic Weights

Bias input

n

jjjkk xw

1,

)( kkk uy


Type of Activation Function

0

1

0

1)(

v

v

if

ifv

2/1

2/12/1

2/1

0

,

,1

)(

v

v

v

vv )exp(1

1)(

avv

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-2 -1.5 -1 -0.5 0 0.5 1 1.5 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

-10 -5 0 5 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

)(v )(v)(v

(a) Threshold Function (b) Piecewise-linear Function (c) Sigmoid Function

Slope=a


Feedforward Architecture

Input layerof source nodes

Output layerof neurons

Input layerof source nodes

Layer ofhidden neurons

Output layerof neurons

1. Artificial Neural Networks (ANN) (5/6) Recurrent and 2-D Lattice Networks

Input layer of source nodes

Z-1 Z-1 Z-1 Z-1

1. Artificial Neural Networks (ANN) (6/6) Classifications

Neural Networks

Optimal Nets

● Hopfield-Tank net● Annealing net● Bolzmann machine

Fixed Nets

● Hamming net● Hopfield net● Bi-direction associative memory

Unsupervised Nets

● Self- organization map ● ART● Neocognitron● Competitive learning● Principle component analysis (PCA)● Independent component analysis (ICA)

Supervised Nets

● Perceptron● Back-Propagation delay

● Probabilistic net● Multilayer Perceptron● ADALINE● LVQ● Counter propagation net

2.1 Fuzzy-Systems

Fuzzy C-Means (FCM)

Hard-C-Means (HCM) Fuzzy-C-Means (FCM)

n

x

c

iixHCM wzJ

1 1

2

2

1

k

zw

ky y

i 1

n

x

c

iix

mixFCM wzuJ

1 1

2,2

1

1

1)1/(12

)1/(12

,)(

)(

c

mx

mix

ixz

z

n

x

c

i

mix

n

x

c

ix

mix

i

z

w

1 1,

1 1,

2.1 Fuzzy-Systems

Penalized FCM and Compensated FCM

Penalized FCM (PFCM) Compensated FCM (CFCM)

n

x

c

i

mix

n

x

c

ix

mix

i

z

w

1 1,

1 1,

i

mn

x

c

iix

n

x

c

iix

mixPFCM uvwzuJ ln

2

1

2

1

1 1,

1 1

2,

n

x

c

i

mix

n

x

mix

i

1 1,

1,

1

1)1/(12

)1/(12

,)ln(

)ln(

c

mx

miix

ixz

z

n

x

c

i

mix

n

x

c

ix

mix

i

z

w

1 1,

1 1,

i

mn

x

c

iix

n

x

c

iix

mixCFCM uvwzuJ tanh

2

1

2

1

1 1,

1 1

2,

n

x

c

i

mix

n

x

mix

i

1 1,

1,

1

1)1/(12

)1/(12

,)tanh(

)tanh(

c

mx

miix

ixz

z

2.1 Fuzzy-Systems

The curves of ln(i) and tanh (-i) within 0 i 1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-2.5

-2

-1.5

-1

-0.5

0

2.1 Fuzzy-Systems

Fuzzification in the training example --Butterfly

x1

x2

x3

x4

x5

x6

x7

x8x9

x10

x11

x12

x13

x14

x15

1 1.2 1.4 1.6 1.8 20

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

fuzzyfica tion pa ramete r *:FCM +:P FCM o:CFHNN

me

mb

ers

hip

gra

de

s o

f z8

2.2 Possibilistic-System

Possibilistic C-Means (PCM) – Proposed by Krishnapuram and Keller

c

iix

n

xi

n

x

c

iixixPCM twztJ

1,

11 1

2, )1(

2

1

i : Scale parameter at the i-th cluster.

ixt , : Possibilistic typicality value of training sample belonging to the i-th cluster.

xz

Fuzzy Possibilistic C-Means (FPCM) -- Proposed by Pal, Pal, and Bezdek

n

x

c

iixix

mixFPCM wztJ

1 1

2,,2

1

1

1)1/(12

)1/(12

,)(

)(

c

mx

mix

ixz

z

1

1 )1/(12

)1/(12

,)(

)(

n

y miy

mix

ixz

zt

2.3 Fuzzy-, Possibilistic-Systems(1/7)

FPCM-- Membership and Typicality

z1

z2

zn

Training sample

1,1 2,1 c,1

1,2 2,2 c,2

1,n 2,n cn,

Cluster

1 2 ….. c

c

iix

1, 1

Membership function

1,1t 2,1t ct ,1

1,2t 2,2t ct ,2

1,nt 2,nt cnt ,

Cluster

1 2 ….. c

n

xixt

1, 1

Typicality function


Data Set FCM m=3

FPCM m=3, =3

x p1 p2 1,x 2,x 1,x 2,x 1,xt 2,xt 1 -3.34 0.00 0.9524 0.0476 0.9538 0.0462 0.0227 0.0012 2 -3.34 1.67 0.9599 0.0401 0.9572 0.0438 0.0368 0.015 3 -3.34 0.00 0.9972 0.0028 0.9976 0.0024 0.8664 0.0016 4 -1.67 -1.67 0.9218 0.0782 0.9249 0.0751 0.0178 0.0014 5 -1.67 0.00 9.9075 0.0925 0.9060 0.0940 0.0287 0.0031 6 0.00 0.00 0.5000 0.5000 0.5001 0.4999 0.0067 0.0067 7 1.67 0.00 0.0925 0.9075 0.0927 0.9073 0.0028 0.0301 8 3.34 1.67 0.0401 0.9599 0.0415 0.9585 0.0015 0.0385 9 3.34 0.00 0.0028 0.9972 0.0017 0.9983 0.0016 0.8654 10 3.34 -1.67 0.0782 0.9218 0.0745 0.9255 0.0014 0.0193 11 5.00 0.00 0.0476 0.9524 0.0456 0.9546 0.0010 0.0210 12 0.00 10.00 0.5000 0.5000 0.4997 0.5003 0.0005 0.0005

Class center (-3.1947, 0.3138) (3.1946, 0.3134)

(-3.2045, 0.2702) (3.2048, 0.2657)

Simulated data set

x1

x2

x3

x4

x5

x6

x7

x8

x9

x10

x11

x12

Typicality order

1,xt 2,xt

3 9 2 8 5 7 1 11 4 10 6 6 7 5 9 3 8 2

10 4 11 1 12 12


Penalized FPCM

n

x

c

ixixi

mix

n

x

c

iixix

mixPFPCM tuvwztuJ

1 1,,

1 1

2,, lnln

2

1

2

1

1

1)1/(12

)1/(12

,)ln(

)ln(

c

mx

miix

ixz

z

1

1 )1/(12

)1/(12

,)ln(

)ln(

n

y myiy

mxix

ixz

zt

n

x

c

i

mix

n

x

mix

i

1 1,

1,

n

x

c

iix

c

iix

x

t

t

1 1,

1,

=JFPCM - scale factors based on clusters and training samples



Compensated FPCM

n

x

c

ixixi

mix

n

x

c

iixix

mixCFPCM tuwztuJ

1 1,,

1 1

2,, tantan

2

1

2

1

1

1)1/(12

)1/(12

,)tan(

)tan(

c

mx

miix

ixz

z

1

1 )1/(12

)1/(12

,)tan(

)tan(

n

y myiy

mxix

ixz

zt

n

x

c

i

mix

n

x

mix

i

1 1,

1,

n

x

c

iix

c

iix

x

t

t

1 1,

1,

=JFPCM + scale factors based on clusters and training samples


Original Image

DCTDCTAC coefficients

DC coefficient

PFPCM/CFPCM

PFPCM/CFPCM CodebookCodebook

DC + Index

Transmission

Encoder

CodebookCodebook

DC + Index

DC coefficient

AC coefficientsIDCTIDCT

Reconstructed Image Decoder


OriginalImage LBG

DCT +LBG(VQ)

DCT +CFPCM(VQ)

2.4 Rough- System (1/5)

Rough set Let R be a binary equivalence relation defined on a universal set Z is a subset of the Cartesian product, . An equivalence relation is a binary relation, R, that satisfies

R is reflexive : R is symmetric : R is transitive :

can be defined as the union of all equivalence classes in Z/R that are contained in A such that

can be also defined as the union of all equivalence classes in Z/R that overlap with A like the following equation

R Z Z

R Z Z

1 1 1( , )z Z z z R 1 2 1 2 2 1( , ( , ) ) ( , )z z Z z z R z z R

1 2 3 1 2 2 3 1 3( , , ( , ) ( , ) ) ( , )z z z Z z z R z z R z z R

)(AR

},][|]{[)( ZzAzzAR RR

)(AR

},][|]{[)( ZzAzzAR RR


)(AR

Rough set A rough set can be represented by and with the given set A as

And the rough boundary of A by the equivalence classes Z/R is distinct as

Interconnection models in the architecture of rough neurons

R Z Z

)(AR

)(),()( ARARAR

)()()( ARARARB

s

s

soutput

soutput

r

r

s

s

soutput

soutput

r

r

s

s

soutput

soutput

r

r

(a) Fully connected (b) Exciting model (c) Inhibiting model


Rough Neurons (Proposed by Lingras in 1998) Definition for the Exciting model in the rough neurons

R Z Z

s

s

soutput

soutput

r

r

m

rrrss outputwinput

1,

m

rrrss outputwinput

1,

))(),(max( sss inputtinputtoutput

min( ( ), ( ))s s soutput t input t input

xext

1

1)(


Rough Fuzzy Hopfield Neural Network (RFHNN)

R Z Z

Netx,i

:

Netx,i

:

)( y

n

1yx,i;y,i zW

)( y

n

1y

x,i;y,i zW

ixI , xz

ixI , xz

ix,

ix,

ix

2

y

n

yx,i;y,ixx,i IzWzNet ,

1

ix

2

y

n

y

x,i;y,ixx,i IzWzNet ,

1

2 2

1 1 1 1 1 1

1 1( ) ( ) ( ) ( )

2 2

n c n n c nmmx yx,i;y,ix yx,i;y,i x,ix,i

x i y x i y

E z W z z W z

n

x

m

ixixm

x,i

c

ix,i II

1,,

1

)(2

1

n

h

m

ih

m

iyiyixW

1 ,

,,;,

)(

)(

n

h

m

ih

m

iyiyixW

1 ,

,,;,

)(

)(

),min( ,,,ixixix

),max( ,,, ixixix 1

1

1/1

,

,,

c

j

m

jx

ixix

Net

Net

1

1

1/1

,

,,

c

j

m

jx

ixix

Net

Net

Processing to the Multi-Spectral Image using RFHNN

R Z Z


MorphologyProcessing

Result

Fuzzy Competitive Learning Network (FCLN) Penalized Fuzzy Competitive Learning Network (PFCLN) Compensated Fuzzy Competitive Learning Network (CFCLN) Rough Fuzzy Competitive Learning Network (RFCLN) Fuzzy Hopfield Neural Network (FHNN) Penalized Fuzzy Hopfield Neural Network (PFHNN) Compensated Fuzzy Hopfield Neural Network (CFHNN) Fuzzy-Possibilistic Hopfield Neural Network (FPHNN) Rough Fuzzy Hopfield Neural Network (RFHNN)

R Z Z

2.5 Fuzzy-, Possibilistic-, Rough- Systems + Artificial Neural networks

3. Cerebellar Model Arithmetic Computer (CMAC) (1/4)

CMAC, named Cerebella Model Articulation Controller, was proposed by J.S. Albus in1975.

CMAC is a model of associate memory network. In the training phase, the CMAC updates the weights in

memory by using a transformation from input samples. It can easily obtain the outputs by looking up the weights

in memory in accordance with the input vectors in the recognition phase.

Due to a simple manner with memory architecture, the CMAC can be easily implemented into hardware circuit.

3. Cerebellar Model Arithmetic Computer (CMAC) (2/4)

**)()( ,...,2,1, Ai

A

yydWW ai

oldiainewi

Traditional CMAC Architecture

a1

a2

a4

a5

a6

a7

aN

w1

w2

w4

w5

w6

w7

wN

Input Vector

y

a w

AXS YA

P

d

i

yiE1

2)1(

3. Cerebellar Model Arithmetic Computer (CMAC) (3/4)Modified CMAC Architecture with Clustering Memory

m94 m2m1

Input Pattern Memory Output Weights

Class 1

Class 2

Class 16

a0 ~ a15

a0 ~ a15

a0 ~ a15

Sum 94

Sum 2

Sum 1

m94

m94

m2

m2

m1

m1

1

2

3

4

5

6

7

8

coding

coding

.

.

.

.

.

....

.

.

.

.

.

.

.

.

.

61626364

.

.

.

.

.

.

.

.

.

coding

1

2

N

Input signalswith quantizingbinary code

3. Cerebellar Model Arithmetic Computer (CMAC) (4/4)Applied CMAC to Character Recognition

8 error pixels in characters



4. Genetic Algorithm (GA)(1/4)

Evolutionary computing was introduced in the 1960s by I. Rechenberg in his work "Evolution strategies" (Evolutions strategie in original). His idea was then developed by other researchers. Genetic Algorithms (GAs) were invented by John Holland and developed by him and his students and colleagues at University of Michigan, 1970’s . Directed search algorithms based on the mechanics of biological evolution and Survival with a fitness function. Functions of GA:

• Chromosome– string of DNA– consists of genes– a solution of the problem

• Fitness– measure the chromosome– survival or not

• Reproduction– crossover

▪ two chromosomes ▪ combine the genes from parents▪ form new chromosomes

– mutation▪ occurs on single chromosome▪ elements of DNA are a bit changed


Simple Genetic Algorithm(){

Randomly initialize population;evaluate population;while(termination criterion not reached){

select solutions for next population with a fitness function (reproduction);

perform crossover and mutation;evaluate population (to produce new offspring);

}}


Population

Evaluate

Fitness Function

Evolution Circumstance

Reproduction

crossover

mutationNew Offspring

Evolutionary Procedure

Roulette-WheelSelection

Combing GA with other systems GA + Fuzzy Algorithm

GA + Possibilistic Algorithm

GA + Rough algorithm

GA + Artificial Neural Network


The AIS transfers the characteristics of natural immune system with mathematic model into computing system in algorithm manner to solve the engineering problems.

The AIS is based on Jerne’s idiotypic network theory (Jerne, 1973), which suggests that the immune system maintains a network of interconnected B-cells.

5. Artificial Immune System (AIS)(1/9)

Natural immune system in human body

自然殺手細胞

細胞及分泌物

淋巴細胞吞噬細胞補體

T 細胞及淋巴球B 細胞及抗體


The structure of multi-protection and -defense system in immune system


Models in AIS 1. Antibody Network

2. Evolutionary Algorithm Immune Genetic Algorithm (IGA) Immune Evolutionary Programming (IEP) Immune Evolutionary Strategy (IES)

3. Colonel Selection Principle (CSP)


Antibody Network (1)

抗體群Ab population

抗原激發

選擇抗體Ab Selection

相似成熟度Affinity Maturation Selection

重選抗體Ab Re-selection

繁殖 Clone 衰亡 Death

非激發細胞激發細胞


Antibody Network (2)

Inhibit immune reaction

Start Immune System

Input external Ags

B-Cell’s surface

Character of Ags displayed by B-cell

Activate lymphoid cells

Th cells secrete to start immune reaction

Activate Ts to secrete IL¯

Generate Abs

All Ags are removed?

Generate Abs Continuously?

Stop immune reaction

N

Y

YN

IL


Immune Genetic Algorithm with Constraint (IGAC)

抗原與目的函數、制約條件的對應

生成起始抗體群

抗體與抗原親和度 (R1) 計算

親和度 (R1)=1?

抗原排除 ( 結束 )

記憶親和度 (R1)高的抗體

生成新抗體

抗體與記憶親和度 (R2)的計算

排除適量親和度 (R2)高的抗體

促進和抑制

Y

N


Immune Genetic Algorithm with Vaccination (IGAV)

啟動

最佳族群?

選擇、交配、及突變

族群更新及適應因子計算

停止疫苗注射

免疫選擇

Y

N


Antibody Network (2) + Fuzzy Algorithm to Image segmentation


Ant system algorithm, based on behavior of real ants, is a natural approach to establish from their nest to food source.

An ant moves randomly and detects a previously laid pheromone on a path in order to find the shortest way between their nest and the food source.

Ant system algorithm is an important methodology to apply on non-linear optimal problems recently.

It is a parallel architecture to force ants move simultaneously, independently, and without supervisor.

6. Ant Colony System (ACS)(1/4)


Each ant chooses the next position to visit in accordance with the visibility of the position and the ph

eromone intensity. The k-th ant starting from position i decides to visit

position j with the probability defined as follows:

where is the visibility of position j from position i, and are two heuristically defined paramet

ers.

( )( )

( )

ij ijkij

i i

tp t

t

1ij

ijd


We define the pheromone intensity on path (i, j) at time t to be and to assign a random value to it when t = 0.

Along the path from i to j, a trail substance is laid on path (i, j) and defined as:

where Q is a constant and is the tour length of the

k-th ant.

0

kkij

Qif k th ant uses path in its tour

L

otherwise

( )ij t

kL


When the ant has completed a position and a cycle of n iterations is consisted, the laid trail substance is used to update the amount of substance previously laid as the following equation:

and

where is a coefficient of persistence of the tail and is the quantity of trail substance laid on path (i, j) by the k-th ant during a cycle( between time t and t + n).

( ) ( )ij ij ijt n t

1

mk

ij ijk

kij

6.1 Annealing algorithm + ACS (AACS)

In the scheme of ant system algorithm, the total cost function for the network topology from node i to k and cooling schedule can be defined as

where

The probability that the k-th ant starting from node i to visit node j undergo random thermal perturbations at a given temperature T conforms to a Boltzmann distribution

( ) [ ( )] [ ]ik iki k

E t t

1/ ( , )

0ik

ikd if path i k exists

otherwise

( ) /( ) ijE t Tk

ijp t e

( ) /

( ) /( )

ij

i

E t Tkij

E t T

ep t

e

1( ) tanh( ) ( 1)

1tT t w T t

and

6.2 The Application of AACS to Shortest Problem(1/4)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20Source

Destination

230 130

90

4530

65

50

6227

25

136 58

43

15

40

32

17

29

35

60 250

30120

32

220

61

20

88

144

161

71

54 26

61

147

60194

1108977

14

150 40

220 72

22

72

24

16

Length = 142

the conventional ant system algorithm

(a) CM model (b) DM model

0 50 100 150 200 250 30080

82

84

86

88

90

92

94

96

98

100

the number of ants

succ

essf

ul p

erce

ntag

e

o : = 0.2 + : = 0.4* : = 0.6 : = 0.8 : = 1.0

0 50 100 150 200 250 30010

20

30

40

50

60

70

80

90

100

the number of ant

the n

um

ber

of

success

o : = 0.2 + : = 0.4* : = 0.6 : = 0.8 : = 1.0


Conventional ant algorithm with the Roulette wheel selection



0 50 100 150 200 250 30030

40

50

60

70

80

90

100

the number of ants

succ

essf

ul p

erce

ntag

e

0 50 100 150 200 250 30020

30

40

50

60

70

80

90

100

the number of ants

succ

essf

ul p

erce

ntag

e

o:=0.2 +:=0.4*:=0.6 :=0.8 :=1.0The value of

o:=0.2 +:=0.4*:=0.6 :=0.8 :=1.0The value of

Experimental results The annealed ant system algorithm with the Roulette wheel selection


0 50 100 150 200 250 30060

65

70

75

80

85

90

95

100

the number of ants

succ

essf

ul p

erce

ntag

e

0 50 100 150 200 250 3000

10

20

30

40

50

60

70

80

90

100

the number of ants

succ

essf

ul p

erce

ntag

e

o:w=0.98, =0.8x:w=0.98, =0.99+:w=0.998, =0.8 :w=0.998, =0.99

o:w=0.98, =0.8x:w=0.98, =0.99+:w=0.998, =0.8 :w=0.998, =0.99


7. Support Vector Machine (SVM)(1/6)

SVM was first introduced by Boser et al. in 1992. SVM is now regarded as an important example of “

kernel methods”, arguably the hottest area in machine learning.

Class 1

Class 2

Class 1

Class 2

Class 1

Class 2

7. SVM -- Large-margin Decision Boundary(2/6) The decision boundary should be as far away from

the data of both classes as possible We should maximize the margin, m

Class 1

Class 2

m

7. SVM -- Nonlinear Decision Boundary(3/6)

(.)

Input space

( )( )

( )

( )

( )

( )

( )

( )

( )( )

( )

Feature space

( )( )

( )( )

( )

( ) ( )

Nonlinear Transform

7. SVM – Application to the Face Detection(4/6)

Support Vectors :

SVM Training

1 2 3 4 5 i 1i 2i 3i

…

Training

From Sami Romdhani et al.


From Sami Romdhani et al.

1 2 3 4 5 i 1i 2i 3i

…

D D D D D D D D D

Output

2. ClassificationIs this path a face ?

> T Face<= T Non-Face


Fro

m S

ami R

omdh

ani

et a

l.

8. Conclusions Machine-learning research has been making great progress

in many directions such as (1)The improvement of classification accuracy by learning ensembles of classifiers (2) Methods for scaling up supervised learning algorithms (3) reinforcement learning (4) the learning of complex

Machine-learning techniques are being applied to several problems including knowledge discovery in databases, language processing, robot control, and combinatorial optimization, as well as to more traditional problems such as speech recognition, face recognition, handwriting recognition, medical data analysis, and game playing.

Thanks !