1 A TSK-Type Neuro-fuzzy Network Approach to System Modeling Problems Chen-Sen Ouyang Wan-Jui Lee Shie-Jue Lee Presented by : Pujan Ziaie

11

A TSK-TypeA TSK-Type Neuro-fuzzy Neuro-fuzzy Network Approach to System Network Approach to System

ModelingModeling Problems ProblemsChen-Sen Ouyang Wan-Jui Lee Shie-Jue Lee

Presented by: Pujan Ziaie

Hirota labHirota lab 22/45/45

Authors (1)Authors (1)

• Chen-Sen OuyangChen-Sen Ouyang– born in Kin-Men, Taiwan born in Kin-Men, Taiwan – Received Received Ph.D.Ph.D. degree from the degree from the National National

Sun Yat-Sen UniversitySun Yat-Sen University, Kaohsiung, , Kaohsiung, Taiwan, in Taiwan, in 20042004

– Research interests:Research interests:• Soft computing, data mining, pattern recognition, Soft computing, data mining, pattern recognition,

video processingvideo processing

– member of the Taiwanese Association of member of the Taiwanese Association of Artificial Intelligence Artificial Intelligence


Authors (2)Authors (2)

• Wan-Jui LeeWan-Jui Lee – born in Tainan, Taiwan born in Tainan, Taiwan – Received Received B.S.B.S. degree from the degree from the National National

Sun Yat-Sen UniversitySun Yat-Sen University, Kaohsiung, , Kaohsiung, Taiwan, in Taiwan, in 20002000

– Research interests:Research interests:•data mining, fuzzy set theory, neural data mining, fuzzy set theory, neural

networks, and support vector learningnetworks, and support vector learning


Authors (3) -Authors (3) -ProfessorProfessor

• Shie-Jue LeeShie-Jue Lee– born in Kin-Men, Taiwan born in Kin-Men, Taiwan – Received Received Ph.D.Ph.D. degree from the degree from the University ofUniversity of

North CarolinaNorth Carolina , , Chapel HillChapel Hill, in , in 19901990– Research interests:Research interests:

•machine intelligence, data mining, soft computing, machine intelligence, data mining, soft computing, multimedia communications, and chip design multimedia communications, and chip design

– Received the Received the Excellent Teaching AwardExcellent Teaching Award of of National Sun Yat-Sen UniversityNational Sun Yat-Sen University

– Chairman of the Electrical Engineering Chairman of the Electrical Engineering Department from 2000Department from 2000


Universities (Taiwan)Universities (Taiwan)

• I-Shou (C-S. Ouyang)- Kaohsiung

•National Sun Yat-Sen University (W.-J. Lee and S.-J. Lee)


Paper infoPaper info

• Manuscript Manuscript received June 18, 2004received June 18, 2004 • revised November 18, 2004revised November 18, 2004• supported by the National Science supported by the National Science

CouncilCouncil• recommended by Editorrecommended by Editor H.-X. Li H.-X. Li• IEEE TRANSACTIONS ON SYSTEMS,

MAN, AND CYBERNETICS—PART B: CYBERNETICS, VOL. 35, NO. 4, AUGUST 2005


OutlineOutline

• Rudiments of Rudiments of Fuzzy control Fuzzy control && Neural Neural NetworksNetworks

• Paper IntroductionPaper Introduction

• Rule extraction: Merged-based fuzzy Rule extraction: Merged-based fuzzy clustering:clustering:

• Rule Refinement: Neural NetworksRule Refinement: Neural Networks

• Experimental resultsExperimental results

• ConclusionConclusion

Hirota lab 8/45

What isFuzzy logic Proposed by professor Lotfizadeh-1964 Mathematical idea – (worst way to explain)

• Crisp logic: 0 or 1 >> Fuzzy logic [0..1]

• Use a membership function instead of 0,1 Applicative explanation

• A way of describing the word by linguistic, inexact, fuzzy variables

• Explain the behavior of a system through linguistic variables

µAc(t) = 1 - µA(t)

Hirota lab 9/45

Membership functions

Defining fuzzy variables by using membership functions

Common functions:

Example: “youngness”

age

Youngness grade

Natori-san3510

197%100%

Hirota lab 10/45

Fuzzy control concept

Using if-then rules with linguistic variables instead of differential equations

Example: Riding Unicycle

Classic way of stabilization:•large number of non-linearity •unknown and variables such as friction and total mass

Hirota lab 11/45

Fuzzy control example

How much tip according to service and food quality?

inputs output

max

Defuzzification

Centroid Average (CA) Maximum Center Average (MCA) Mean of Maximum (MOM) Smallest of Maximum (SOM) Largest of Maximum (LOM)

Hirota lab 12/45

TSK-type fuzzy rule Proposed by: Takagi, Sugeno, and Kang fuzzy inputs but crisp outputs (constant or a function) If X is X1 and Y is Y1 then z = f1(x,y) Example: If pressure is low and temperature is medium then valve opening is

5*p + 3*t If X is X1 and Y is Y1 then z = f1(x,y)

If X is X2 and Y is Y2 then z = f2(x,y)...If X is Xn and Y is Yn then z = fn(x,y)

Defuzzification: (wi) being the degree of matching (product of µ(xi)) (w1*f1(x,y) + w2*f2(x,y) + … + wn*fn(x,y)) / (w1 + w2 + …

+ wn) = ∑wn*fn(x,y)/ ∑wn

Hirota lab 13/45

Neural Networks

Perform learning to approximate a desired function

Useful when lots of examples of the behaviour we can't formulate an algorithmic solution

w1

w2

w3

wn

I1

I2

I3

In

Activationvector

processorf(I,W, T)

Hirota lab 14/45

Linear Neural Networks(1)

Model the existing in-out data by the simple function: y = w2 x + w1

Hirota lab 15/45

Linear Neural Networks(2)

Consider the simple N-N Adjust the weights to map the

function by minimizing the loss

W1

W2

I1

I2

Σ g (x) O

Inputs

weight

Output

Adder and Activation function

1

x

Hirota lab 16/45

Gradient descent Algorithm Computing the gradient

W1

W2

I1

I2

Σ g (x) O

Inputs

weight

Output

Adder and Activation function

Learning rate

Hirota lab 17/45

Multi-layer N-N Add a hidden layer

Input layer

Hidden layer

Out layer layer

Neurones

Output = w0+w1*f(x1) +.. +wn*f(xn)

Hirota lab 18/45

Multi-layer N-N learning Forward activation

passing (or feeding) the signals through the network

Calculating output error Mean squared error (MSE)

Error backpropagation Chain rule


IntroductionIntroduction

• ModelingModeling– System is unknownSystem is unknown– Measured input-output data availableMeasured input-output data available

• Neuro-fuzzy systemsNeuro-fuzzy systems– Fuzzy rulesFuzzy rules: describe the behavior : describe the behavior

((structurestructure))– Neural NetworksNeural Networks: adjust control parameters : adjust control parameters

and improvement of the system and improvement of the system ((refinementrefinement))


IntroductionIntroduction

• Modeling processModeling process

Self-Constructing Rule Generation

Neural NetworkFuzzy rules

Parameter identificationStructure Identification

Merged-based fuzzy clustering

Hybrid learning algorithm

In-Out data ResultFinalFuzzyRules


Controller StructureController Structure

• Fuzzy rules from DataFuzzy rules from Data– If If x is Mx1x is Mx1 Then Then y is y is

Mz1Mz1– If If x is Mx2x is Mx2 Then Then y is y is

Mz2Mz2

– 2 inputs2 inputs– If If x is Mx2x is Mx2 and and y is y is

My1My1 Then Then y is Mz2y is Mz2z

X

Mz1

Mx1

Mx2

Mz2

y

My1


Our approachOur approach

• Gaussian function for membership Gaussian function for membership functionsfunctions

• J final clusters of dataJ final clusters of data• TSK fuzzy rule:TSK fuzzy rule:

– If If x1 is µ1j(x1) AND x2 is µ2j(x2) AND … AND xn is µnj(xn) THEN

y = b0j +b1jx1+..+bnjxnb0j > m0jb1j..bnj > temporarily 0 (can not be

deduced)

Mean vector(m)

Deviation vector

[38],[42]


ClusteringClustering

• To define input To define input membership membership functions regionsfunctions regions– Data partitioningData partitioning

•Sensitive to input orderSensitive to input order

•Might be redundantMight be redundant

– Clusters mergeClusters merge•Merging similar Merging similar

clustersclusters

•n inputs> x1…xn, one output

•Cj > fuzzy cluster


Data partitioningData partitioning

• N: number of training N: number of training patternspatterns

• tv: pattern (1≤v≤N)tv: pattern (1≤v≤N)

• Sj: Size of cluster CjSj: Size of cluster Cj

• comb > operator to comb > operator to combine Cj and tvcombine Cj and tv


CombinationCombination

• Combining: Changing Gaussian Combining: Changing Gaussian functions of clusterfunctions of cluster

User-defined constant

Initial deviations


Partitioning processPartitioning process

• tv> New training instancetv> New training instance• Calculate Ij(p v) > compare with Calculate Ij(p v) > compare with ρρ

(threshold)(threshold)

• Calculate O'j(y)=comb_y(Oj(y),qv) and Calculate O'j(y)=comb_y(Oj(y),qv) and compare ơ'oj with compare ơ'oj with ηη (threshold)(threshold)

•If both failed > new If both failed > new cluster kcluster k•If not> add to the If not> add to the most appropriatemost appropriate Cluster Using the combination methodCluster Using the combination method

•Largest input-similarity

•With minimum output-variance


Cluster mergeCluster merge

• Partitions > input order dependantPartitions > input order dependant

• Merging > creating efficient clustersMerging > creating efficient clusters


Merging conditionsMerging conditions

• Calculating input and output Calculating input and output similarity measure similarity measure rrII

ijij , , rrOO

ijij for every for every

two clusters (Ctwo clusters (Cii,C,Cjj))

• IfIf r rIIij ij ≥≥ ρρ , , rrOO

ij ij ≥≥ ε ε (threshold)(threshold) Then put Then put

them into the same candidate class Xthem into the same candidate class X


Merging processMerging process• KK Clusters combination: Clusters combination:

• Clusters are combined if ơ'oj ≤ Clusters are combined if ơ'oj ≤ ηη


Merging process Merging process (full (full procedure)procedure)

Input-output similarity

measurement

Mergingclusters

clusters

Increasing ρρ , , εε

To (1+θ)* (ρρ , , εε )

Every candidate class has only one cluster

classes No

Yes

Finished

Merged clusters


Rule RefinementRule Refinement

Final clusters >> Final rules (Final clusters >> Final rules (TSKTSK type)type)• CC11,C,C22,…C,…Cjj >> R={R >> R={R11,R,R22,..R,..Rjj}}

IfIf xx11 ISIS µ1j1j(x11) AND IfAND If x x22 ISIS µ2j2j(x22) ANDAND … ANDAND IfIf x xnn ISIS µnjnj(xnn)

THENTHEN yjj ISIS fjj(x)=b0j0j+b1j1jx11+…+bnjnjxnn

• µijij(x11) == exp[-( (xii-mijij)/ơijij)2]

• b0j0j == m0j0j, b1j1j..bnjnj=0 (will be learned later)


TSK-type fuzzy rule - RemindingTSK-type fuzzy rule - Reminding

fuzzy inputs but crisp outputs (constant or a fuzzy inputs but crisp outputs (constant or a function)function)

If X is X1 and Y is Y1 then z = f1(x,y)If X is X1 and Y is Y1 then z = f1(x,y)If X is X2 and Y is Y2 then z = f2(x,y)If X is X2 and Y is Y2 then z = f2(x,y)......If X is Xn and Y is Yn then z = fn(x,y) If X is Xn and Y is Yn then z = fn(x,y)

DefuzzificationDefuzzification:: (wi) being the degree of matching ((wi) being the degree of matching (product of product of

µ(xi)µ(xi))) Y= (w1*f1(x,y) + w2*f2(x,y) + … + Y= (w1*f1(x,y) + w2*f2(x,y) + … +

wn*fn(x,y)) / (w1 + w2 + … + wn)wn*fn(x,y)) / (w1 + w2 + … + wn)


Rules’ Calculations(1)Rules’ Calculations(1)

αjj(x)=µ1j1j(x11)*µ2j2j(x22)*…*µnjnj(xnn) =Ijj(x)

fjj(x)=b0j0j+b1j1jx11+…+bnjnjxnn

Final output y:

•µijij(x11) == exp[-( (xii-mijij)/ ij)2]

•b0j0j == m0j0j, b1j1j..bnjnj=0

Rule j strength


Rules’ Calculations(2)Rules’ Calculations(2)

µnjnj(xnn) > αjj(x)=∏µij ij > djj(x) > sjj(x) > y


Network structureNetwork structure

Layer 1: Layer 1: µnjnj(xnn)==exp[-( (xii-mijij)/ơijij)2]

x1

.

.

xi

.

.

.

xn

µ1111

µi1i1

µnJnJ

µn1n1

µ1j1j

µijij

µnjnj

µ1J1J

µiJiJ

(mij,ơijij))

Layer 2: Layer 2: αjj(x)=∏µijij

∏∏

∏∏

∏∏

Layer 3: Layer 3: djj(x) = αjj(x)/∑αjj(x)

NN

NN

NN

Layer 4: Layer 4: sjj(x) = djj(x)*fjj(x) x

s11

sjj

sJJ

Layer 5: Layer 5: y = ∑ sjj(x)

∑∑ yαjj(x) djj(x)


Network refinementNetwork refinement

Hybrid Learning AlgorithmHybrid Learning AlgorithmCombination of:Combination of:• Recursive SVD-based least-Recursive SVD-based least-

square estimator square estimator >> refine b refine bijij

• Gradient decent method Gradient decent method >> refine (mij,ơij)refine (mij,ơij)

x1

.

.

xi

.

.

.

xn

µ1111

µi1i1

µnJnJ

µn1n1

µ1j1j

µijij

µnjnj

µ1J1J

µiJiJ

(mij,ơijij))

∏∏

∏∏

∏∏

NN

NN

NN

x

s11

sjj

sJJ


SVD-based LSE

Gradient decent

algorithm


SSingular ingular VValue alue DDecompositionecomposition(1)(1)

NN training patterns: t training patterns: tVV==

||D|| = (trace(DTD))1/

2

trace(D)=∑n1

dii


SSingular ingular VValue alue DDecompositionecomposition(2)(2)

Suppose aSuppose aijij as constant as constant >> Finding Finding XX**: : the optimal solution to minimizethe optimal solution to minimize EE

SVDSVD ensures: A=Uensures: A=U∑VT

• UU: N: NххN orthonormal matrixN orthonormal matrix• VV: J: JххJ orthonormal matrixJ orthonormal matrix• ∑∑: N: NххJ diagonal matrix with J diagonal matrix with eiei: :

eigenvalues of Aeigenvalues of ATTAA

DTD=I<>A-

1=AT


SVD calculationsSVD calculations

Calculating Calculating XX**: : (X the only variable)(X the only variable)

E(E(XX)=||Q-U)=||Q-U∑VTXX||||

E(E(XX)=||U)=||UTTQ-Q-∑VTXX|| || >> VTXX==YY∑= UUTTQ=Q=

E(X)=E(X)= - - YY = =

U: orthonormal

∑’0

Q’- ∑’YQ’’

Q’Q’’

∑’0

Q’Q’’

minimized Q’-∑’Y*=0X*=VY*


Gradient Decent MethodGradient Decent Method

To refine (To refine (mmijij,,ơơijij))


Summary of processSummary of process

Self-Constructing Rule Generation Neural NetworkFuzzy rules

Merged-based fuzzy clustering

Hybrid learning algorithm

ResultFinalFuzzyRules

x1

.

.

xi

.

.

.

xn

µ1111

µi1i1

µnJnJ

µn1n1

µ1j1j

µijij

µnjnj

µ1J1J

µiJiJ

(mij,ơijij))

∏∏

∏∏

∏∏

NN

NN

NN

x

s11

sjj

sJJ



Experimental resultsExperimental results

Modeling the functionModeling the function

7 clusters > 7 Rules4 clusters > 4 Rules

Refinement through Neural Network

x1

.

.

xi

.

.

.

xn

µ1111

µi1i1

µnJnJ

µn1n1

µ1j1j

µijij

µnjnj

µ1J1J

µiJiJ

(mij,ơijij))

∏∏

∏∏

∏∏

NN

NN

NN

x

s11

sjj

sJJ


Final Fuzzy Rules:


Comparison of methodsComparison of methods

Modeling function:Modeling function:

Yen’s System Juang’s System This paper’s System


ConclusionConclusion

Using differential equationsUsing differential equations is is useless useless for many control problemsfor many control problems

Advantages of this approachAdvantages of this approach• flexibilityflexibility• SimplicitySimplicity

DisadvantagesDisadvantages• No explanation on methods usedNo explanation on methods used• Data-set neededData-set needed


Thank you for listeningThank you for listening

Any questions?Any questions?


Thank you for listeningThank you for listening

Any questions?Any questions?


x1

.

.

xi

.

.

.

xn

µ1111

µi1i1

µnJnJ

µn1n1

µ1j1j

µijij

µnjnj

µ1J1J

µiJiJ

(mij,ơijij))

∏∏

∏∏

∏∏

NN

NN

NN

x

s11

sjj

sJJ

∑∑αjj(x) djj(x)

Documents

1 A TSK-Type Neuro-fuzzy Network Approach to System Modeling Problems Chen-Sen Ouyang Wan-Jui Lee Shie-Jue Lee Presented by : Pujan Ziaie