Adaptive fuzzy clustering Kharkiv National University of Radio Electronics Control Systems Research...

Adaptive fuzzy Adaptive fuzzy clusteringclustering

Kharkiv National University of Radio Electronics

Control Systems Research Laboratory

Crisp Approach to Clustering

AB

C

1x

2x

Fuzzy Approach to Clustering

1x

2x

AB

C

),( BAw

),( CBw),( CAw

),,( CBAw

Fuzzy Approach to Clustering

Input Data SetNkXxxxxX kN ,...2,1,},,,...,,{ 21

Goal Function of Clustering

N

k

m

jjkjkjjk cxdwcwE

1 1

2,, ),(),(

Batch Approaches

m

jjk Nkw

1, ,...,1,1

N

kjk mjNw

1, ,...,1,0

mNWwW jk dim},{ ,

Batch Fuzzy Clustering Algorithms: Probabilistic ApproachFuzzy clustering as a constrained optimization problem

nN RxxxX }...,{ ,2,1 m Divide a data set into clusers.

N

k

m

jjkkjjk cxdwcwE

1 1

2, ),,(,),(

,,...,1,11

, Nkwm

jjk

.,...,1,01

, mjNwN

kjk

(1)

(2)

(3)

Lagrange function:

N

k

m

j

N

k

m

jjkkjkjkkjjk wcxdwcwL

1 1 1 1,

2,, .1),(),,( (4)

Batch Fuzzy Clustering Algorithms: Probabilistic ApproachDegrees of membership:

,

)),((

))((

1

1

1

2

1

1

2

,

m

llk

jkjk

cxd

cxdw (5)

prototypes:

N

kjk

N

kkjk

j

w

xw

c

1,

1,

(6)

A Bkx 2,kw1,kw

px

2,pw1,pw

1x

2x

5.02,1,2,1, ppkk wwww

Batch Fuzzy Clustering Algorithms: Possibilistic Approach

2, , ,

1 1 1 1

( , ) ( , ) (1 ) .N m m N

k j j k j k j j k jk j j k

E w c w d x c w

0j

.

),(

1,

1

2,

N

kjk

N

kjkjk

j

w

cxdw

Unconstrained optimizationModified objective function:

Scalar parameters determine the distance at which the degrees of membership are equal to 0.5:

(8)

(7)

Batch Fuzzy Clustering Algorithms: Possibilistic Approach

,),(

1

1

1

12

,

j

jkjk

cxdw

.

1,

1,

N

kjk

N

kkjk

j

w

xw

c

Possibilistic algorithmDegrees of membership:

prototypes:

(10)

(9)

Adaptive Fuzzy Probabilistic Algorithms

m

j

m

jjkkjkjkkjjkk wcxdwcwL

1 1,

2,, ).1(),(),,(

,

)),((

)),((

1

1

1

,2

1

1

,2

,

m

llkk

jkkjk

cxd

cxdw

).( ,1,,,1 jkkjkkjkjk cxwcc

One-step Largange function:

Degrees of membership:

prototypes:

(13)

(11)

(12)

Unsurprised fuzzy competitive learning (UFCL) algorithm

Adaptive Fuzzy Probabilistic Algorithms

, 1 ,

1,,

( ), ,

, ,

k j k k k j

k jk j

c x c j lc

c j l

l

1

2

1,

2,

,

m

llkk

jkkjk

cx

cxw

).( ,12,,,1 jkkjkkjkjk cxwcc

UFCL is an extension of the Kohonen competitive learning rule:

where

When , UFCL transforms into the Gradient-based fuzzy c-means (GBFCM) algorithm

prototypes:

(16)

(14)

(15)

is a number of the “winning” prototype.

Degrees of membership:

Adaptive Fuzzy Possibilistic Algorithms

m

j

m

jjkjjkjkjjkk wcxdwcwE

1 1,

2,, .)1(),(),(

,),(

1

1

1

12

,

j

jkjk

cxdw

).( ,1,,,1 jkkjkkjkjk cxwcc

One-step objective function:

Adaptive possibilistic algorithmDegrees of membership:

prototypes:

(19)

(18)

(17)

Adaptive Fuzzy Possibilistic Algorithms

,2

,

,

jkkj

jjk

cxw

).( ,12,,,1 jkkjkkjkjk cxwcc

When β=2Degrees of membership:

prototypes:

(20)

(21)

Adaptive Fuzzy Possibilistic Algorithms

-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

wkj

mu=0.1mu=0.2mu=0.3

Possibilistic membership function for 3.01.0,2

Matrix Fuzzy C-means Algorithm

1 2

2,{ ( )} , 1,2,... ,n ni iX x k R k N

1

1( ), ( ) ( ( ) ( )),

NT

k

x x k x k Tr x k x kN

1 1

( , ) ( ) ( ( ) )( ( ) )N m

Tj j j j j

k j

E u c u k Tr x k c x k c

1

1,m

jj

u

1

0 ( ) , 1,..., .N

jk

u k N j m

Input Data Set(22)

.Goal Function of Clustering

.Batch Approaches(25)

. (26)

(23)

(24)

Matrix Fuzzy C-means Algorithm

Karush-Kuhn-Tucker equation system:

1 2

1

1

( ( ), , ( ))( ) ( ( ), ) ( ) 0,

( )

( ( ), , ( ))( ) 1 0,

( )

( ( ), , ( ))2 ( )( ( ) ).

( )

j jj j

j

mj j

jjj

Nj j

j jkj

L u k c ku k D x k c k

u k

L u k c ku k

k

L u k c ku k x k c

c k

O

(27)

Matrix Fuzzy C-means Algorithm

Fuzzy clustering algorithm:

12 1

12 1

1

112 1

1

1

1

( ( ( ), ))( ) ,

( ( ( ), ))

( ) ( ( ), ) ,

( ) ( ).

( )

jj m

ll

m

ll

N

jk

j N

jk

D x k cu k

D x k c

k D x k c

u k x kc

u k

(28)

Matrix Fuzzy C-means Algorithm

J. Bezdek’s FCM-algorithm:

1

1

1

2

1

2

1

( ( ( ) )( ( ) ) )( ) ,

( ( ( ) )( ( ) ) )

( ) ( ).

( )

Tj j

j mT

l ll

N

jk

j N

jk

Tr x k c x k cu k

Tr x k c x k c

u k x kc

u k

. (29)

Experiments

MaxSelector

1x

2x

nx

mw

2w

1w

Class number

Classification system based on fuzzy clustering (the number of the rule nodes is equal to the number of classes)

Experiments

Data Fuzzy c-means

Batch possibilistic

Park-Dagher Recursive possibilistic

Iris 7,4% 7,6% 6,9% 7,8%Wine 3,8% 4,4% 3,9% 4,3%

Data Fuzzy c-means

Batch possibilistic

Park-Dagher Recursive possibilistic

Iris, class 3 33,3% 14,0% 33,3% 15,3%Iris, class 1 38,0% 6,0% 38,0% 6,7%Wine, class 3 29,0% 19,6% 29,0% 19,6%Wine, class 1 35,9% 23,6% 35,9% 23,4%Thyroid, class 3

18,1% 20,9% 18,6% 11,6%

Thyroid, class 1

15,4% 17,2% 15,5% 5.6%

Table 1: Classification with known number of classes (error rate on testing data)

Table 2: Classification with one unknown class (error rate on testing data)

Robust Probabilistic Fuzzy Clustering Algorithms

N

kj

m

jjkjjk

R ckxDkwcwE1 1

,, ),()(,

,,,1,1)(1

,

m

jjk Nkkw

.,,1,)(01

,

N

kjk mjNkw

,1 ,)(,

1

1

pckxcxD

pn

i

pjiijk

, ,k jx jic i )1( njk cx ,

Subject to constraints

(31)

(32)

where are th components of -vectors respectively.

(33)

(30)

Robust Probabilistic Fuzzy Clustering Algorithms

,s

cx

s),sSe(c),cp(x

i

ii

iiiii

2sech

2

1

icis

i

iiiiii

cxcxf

coshln),(

i

iiii

xxxf

tanh)()(

n

i

n

i i

jiiijuiij

R ckxckxfckxD

1 1

)(coshln),(),(

(34)

where and are the parameters that define the center and width of the distribution respectively.

, (35)

(36)

(37)

Robust Probabilistic Fuzzy Clustering Algorithms

.)(

coshln),(),(1 1 11 1

N

k

m

j

n

i i

jiiij

N

k

m

jj

Rjjj

E ckxwckxDwckwE

N

k

m

jj

i

jiiN

k

m

j

n

iijjj kwk

ckxkwkckwL

1 11 1 1

1)()()(

coshln)()(,),(

.0))(,),((

,0)(

))(,),((

,0)(

))(,),((

kckwL

k

kckwL

kw

kckwL

jjc

jj

j

jj

j

Objective function for robust clustering

Lagrange function

The system of Kuhn-Tucker equations

(40)

(39)

(38)

Robust Probabilistic Fuzzy Clustering Algorithms

N

kj

Rcjjjc

m

ll

R

ml l

R

jR

j

ckxDwkckwL

ckxDk

ckxD

ckxDkw

jj1

1

1

1

1

11

1

1

1

0),())(,),((

,),()(

,

),(

),(()(

m

jjj

Rm

jjjjk kwkckxDkwkckwL

11

)1)()((),()())(,),((

Local Lagrange function

(42)

(41)

Robust Probabilistic Fuzzy Clustering Algorithms

,)()(

tanh)()()(

)(,),()()()1(

,

),(

),(()(

11

1

1

1

i

jiijji

ji

jjkjiji

ml l

R

jR

prj

kckxkwkkc

c

kckwLkkckc

ckxD

ckxDkw

jikc , i jk

Arrow-Hurwitz-Uzawa procedure

where is the -th component of the -th prototype vector calculated at the -th step.

(43)

Robust Possibilistic Fuzzy Clustering Algorithms

N

kj

m

jj

N

kj

m

jjjjj kwckxDkwckwE

111 1

)(1),()(),),((

N

kj

Rcjjjjc

Nk j

Nk i

Rj

j

j

jR

posj

ckxDwckwE

kw

ckxDkw

ckxDkw

jj1

1

1

1

1

1

.0),(),),((

,)(

),()(

,),(

1)(

The criterion

The system of Kuhn-Tucker equation

(45)

(44)

Robust Possibilistic Fuzzy Clustering Algorithms

n

i

m

jji

i

jiii

m

jj

m

jjj

m

jj

Rjjjj

Rk

kwckx

kw

kwckxDkwckwE

1 11

11

)(1)(

coshln)(

)(1),()(,),(

,)()(

tanh)()()(,),(

)()()1(

,),(

1)(

1

1

1

i

jijji

ji

jjjkjiji

j

jR

posj

kckxkwkkc

c

ckwEkkckc

ckxDkw

Local Lagrange function

Adaptive possibilistic fuzzy clustering algorithm

(47)

(46)

Robust Possibilistic Fuzzy Clustering Algorithms

12 ))(1()( cxxp ii Experiments (48)

Complete data set (left) and its central part (right)

Robust Possibilistic Fuzzy Clustering Algorithms

Cluster prototypes layout obtained by different algorithms

Robust Possibilistic Fuzzy Clustering Algorithms

Table 3: Classification results 

AlgorithmClassification error rate

Training Checking

Bezdek’s fuzzy c-means 17.1 % (1229 samples) 16.6 % (299 samples)

Probabilistic robust clustering (41) 15.6 % (1127 samples) 15.6 % (281 samples)

Possibilistic robust clustering (47) 15.2 % (1099 samples) 14.6 % (263 samples)