Fuzzy Classification Vtu

8/3/2019 Fuzzy Classification Vtu

http://slidepdf.com/reader/full/fuzzy-classification-vtu 1/119

FUZZY CLASSIFICATION

Classification by Equivalence Relations- Crisp Relations, FuzzyRelations, Cluster Analysis, Cluster validity, c-Means Clustering- Hard

c-Means (HCM), Fuzzy c-Means (FCM), Classification Metric,

Hardening the Fuzzy c-Partition, Similarity Relations from Clustering



Hard partition

(i) The partition Covers all data points

(ii)The partition are mutual exclusive



Soft partition

Constrained soft partition



Clustering example



Compact separated clusters

Any two points in a cluster are closer than the

distance between two points in different cluster.

Compact well separated Not Compact

well separated

Compact

Not well separated



Criterion for searching cluster center To to find a minimal J

V is the vector of cluster

centers P is a partition of data set §

! jk C x

k

j

j xC

v||

1



Searching cluster center



Hard C-means algorithm(HCM) (1) calculating the cost J of current partition

(2) modifying the current cluster centers using

gradient descent method to minimize J.

§! jk C x

k

j

j xC v ||

1



Criterion for fuzzy C-means (FCM)

Higher degree of membership will have higher influence

Weighting sum by membership degree



Criterion for fuzzy C-means(FCM)

QCi( x): xci

d i x ci || x ± vi|| = d i Hharmonic mean

X X

v1 X1 X X

X

X v2 XX X

C1 C2

d1d2

1

21 ),()(

1d

d d H x

C

!Q



Harmonic Mean

xy

y x

y x H 11

1),(

!

1

),(e

x

y x H

1),(

e y

y x H



X X

v1

x X X

X

X v2 X

X X

C1 C2

d1

d2

2

21

1

21 ),(),()()(

21d

d d H

d

d d H x x C C ! QQ

]11

)[,(21

21

d d d d H ! 1

),(

1),(

21

21!!

d d H d d H

Constrained soft partition



0),(

)(2

1

2

21

2}}!

d

d

d

d d H x

C Q

1),(

)(

1

21

1}!

d

d d H x

C Q

X X

v1

x X X

X

X v2 X

X X

d1

d2

membership



fuzzy C-means algorithm



Example

Initial v1=(5,5), v2=(10,10)



Example-1



Example-2

Initial v1=(5,5), v2=(10,10)



Fuzzy Clustering - Theory

CLUSTER ANALYSIS

± way to search for structure in a dataset X

± a component of patter recognition

± clusters form a partition

Examples:

± partition all credit card users into two groups, thosethat

are legally using their credit cards and those who are

illegally using stolen credit cards ± partition UCD students into two classes,

those who will go skiing over winter vacation andthose

who will go to the beach



Clustering is a mathematical tool that

attempts to discover structures or

certain patterns in a data set, where

the objects inside each cluster showa certain degree of similarity.

Clustering



Hard clustering assign each feature

vector to one and only one of theclusters with a degree of membership

equal to one and well defined

boundaries between clusters.



F uzzy clustering allows each feature

vector to belong to more than one

cluster with different membership

degrees (between 0 and 1) and

vague or fuzzy boundaries between

clusters.



Difficulties with Fuzzy Clustering

The optimal number of clusters K to becreated has to be determined (the

number of clusters cannot always be

defined a priori and a good cluster validity criterion has to be found).

The character and location of cluster

prototypes (centers) is not necessarilyknown a priori, and initial guesses

have to be made.



Difficulties with Fuzzy Clustering

The data characterized by large

variabilities in cluster shape, cluster

density, and the number of points(feature vectors) in different clusters

have to be handled.



Objectives and Challenges

Create an algorithm for fuzzy clustering that partitions the data set into an optimal number

of clusters.

This algorithm should account for variability

in cluster shapes, cluster densities, and the

number of data points in each of the subsets.

Cluster prototypes would be generated

through a process of unsupervised learning.



The Fuzzy k-Means Algorithm

N ± the number of feature vectors K ± the number of clusters (partitions)

q ± weighting exponent (fuzzifier; q > 1)

uik ± the ith membership function

on the k th vector ( uik : X p [0,1] )

k uik = 1; 0 < iuik < nV i ± the cluster prototype (the mean of all

feature vectors in cluster i or thecenter of cluster i)

J q(U,V) ± the objective function



Partition a set of feature vectors X into K clusters (subgroups) represented as

fuzzy sets F 1, F 2, «, F K

by minimizing the objective function J q(U,V)

J q(U,V) = ik (uik )qd 2( X j ± V i ); K e N

Larger membership values indicate higher

confidence in the assignment of the pattern to

the cluster .




Description of Fuzzy Partitioning

1) Choose primary cluster prototypes V ifor the values of the memberships

2) Compute the degree of membership of

all feature vectors in all clusters:

uij = [1/d 2(X j ± V i )]1/(q-1) /

k [1/ d 2(X j ± V i )]1/(q-1) (1)

under the constraint: iuik = 1



Description of Fuzzy Partitioning

3) Compute new cluster prototypes V i

V i = j[(uij)q X j ] / j(uij)

q (2)

4) Iterate back and force between (1) and (2)

until the memberships or cluster centers

for successive iteration differ by more than

some prescribed value I (a termination

criterion)




Computation of the degree of membership uij depends

on the definition of the distance measure, d 2( X j ± V i ):

d 2( X j ± V i ) = ( X j ± V i )T 7 -1( X j ± V i )

7 = I => The distance is Euclidian, the shape of the

clusters assumed to be hyperspherical

7 is arbitrary => The shape of the clusters assumed

to be of arbitrary shape




For the hyperellipsoidal clusters, an ³exponential´

distance measure, d 2e ( X j ± V i ), based on ML

estimation was defined:

d 2e (X j ± V i ) = [det( F i )] 1/2 /P i exp[(X j ± V i )

T F i -1(X j ± V i )/2]

F i ± the fuzzy covariance matrix of the ith cluster

P i ± the a priori probability of selecting ith cluster

h( i/ X j ) = (1 / d 2e ( X j ± V i ))/ k (1 / d 2e ( X j ± V k ))

h( i/ X j ) ± the posterior probability (the probability of

selecting ith cluster given jth vector)




It¶s easy to see that for q = 2, h( i/ X j

) = uijThus, substituting uij with h( i/ X j ) results in the fuzzy

modification of the ML estimation (FMLE).

Addition calculations for the FMLE:



The Major Advantage of FMLE

O btaining good partition results starting from³good´ classification prototypes.

The first layer of the algorithm, unsupervised

tracking of initial centroids, is based on the fuzzyK-means algorithm.

The next phase, the optimal fuzzy partition, is

being carried out with the FMLE algorithm.



Unsupervised Tracking of Cluster

Prototypes Different choices of classification prototypes

may lead to different partitions.

Given a partition into k cluster prototypes, place

the next (k +1)th cluster center in a region where

data points have low degree of membership in the

existing k clusters.



Unsupervised Tracking of Cluster

Prototypes

1) Compute average and standard deviation of the

whole data set.

2) Choose the first initial cluster prototype at the

average location of all feature vectors.3) Choose an additional classification prototype

equally distant from all data points.

4) Calculate a new partition of the data set

according to steps 1) and 2) of the fuzzyk-means algorithm.

1) If k , the number of clusters, is less than a given

maximum, go to step 3, otherwise stop.



Common Fuzzy Cluster Validity

Each data point has K memberships; so, it is

desirable to summarize the information by a

single number, which indicates how well the

data point ( X k ) is classified by clustering.

i(uik )2 partition coeff icient

i(uik ) loguik cl assi f ication entrop y

maxi uik proportional coeff icient

The cluster validity is just the average of any

of those functions over the entire data set.



Proposed Performance Measures

³Good´ cl ust er s ar e act uall y not ver y fuzzy.

The criteria for the definition of ³optimal

partition´ of the data into subgroups were

based on the following requirements:

1. Clear separation between the resulting

clusters2. Minimal volume of the clusters

3. Maximal number of data points concentrated

in the vicinity of the cluster centroid




Fuzzy hypervolume, F H

V , is defined by:

Where F i is given by:




Average partition density, D PA

, is calculated from:

Where S i, the ³ sum of the central members´, is given by:




The partition density, P D

, is calculated from:



Sample Runs

In order to test the performance of the

algorithm, N artificial m-dimensional

feature vectors from a multivariate normal

distribution having different parameters anddensities were generated.

Situations of large variability of cluster

shapes, densities, and number of data points

in each cluster were simulated.



FCM Clustering with Varying

Density

The higher density cluster attracts all other cluster prototypes

so that the prototype of the right cluster is slightly drawn away

from the original cluster center and the prototype of the left

cluster migrates completely into the dense cluster.





Fig. 3. Partition of 12 clusters generated from five-

dimensional multivariate Gaussian distribution with

unequally variable features, variable densities and

variable number of data points ineach cluster (only threeof the features are displayed).

(a) Data points before partitioning

(b) Partition of 12 subgroups using the UFP-O NC algorithm.

All data points gave been classified correctly.

(a) (b)





Conclusions

The new algorithm, UFP-O NC(unsupervised fuzzy partition-optimal number

of classes), that combines the most favorable

features of both the fuzzy K-means algorithm

and the FMLE, together with unsupervised

tracking of classification prototypes, were

created.

The algorithm performs extremely well insituations of large variability of cluster shapes,

densities, and number of data points in each

cluster .




REMARKS: (1) The dataset, in the case of studentswould include such things as age, school, incomeof parents, number of years as student, maritalstatus

(2) Classical cluster analysis would partitionthe set of student (with respect to theircharacteristics; that is, the items in the dataset) into

disjoint sets Pi so that we would have:

.for }{and

1

ji j

P i

P c

i P

i

{*!

!

+7




Lets suppose that our dataset has:

Age = {17,18,,35}

School = {Arts, Drama, , Civil Engineering, NaturalSciences, Mathematics, Computer Science}

Income = {$0 $500,000}

Note: It is (or should be) intuitively clear that for this

problem the partitions are intersecting since for manystudents there is an equal preference between goingto the beach and going to ski for vacation and thepreferences are not zero/one for most students.




The idea of cluster analysis is to obtain centers (i=1,,cwhere c=2 for the example of skiing and going to thebeach) v

1,,v

cthat are exemplars and radii that will

define the partition. Now, the centers serve asexemplars and an advertising company could sent skiing brochures to the group that is defined by thefirst center and another brochure for beach trips for

students. The idea of fuzzy clustering (fuzzy c-means clustering where c is an a-priori chosennumber of clusters) is to allow overlapping clusterswith partial membership of individuals in clusters.




degreesometoelement(s)some

containmust partitiona, partitionsemptynoarethere- 1)(0

ddistributecompletely bemust pointdataeachthatsays- 1)(

such that

,..., partitionfuzzy

,...,dataset:Given

1

1

1

1

§

!§

!

!

!

!

±À

±¿¾

±°

±¯®

±À

±¿¾

±°

±¯®

j

N

j i

A

j

c

i i A

c

N

x

x

A A P

x x X

Q

Q

Fuzzy Clustering Example (from



A1 = {0.6/x1, 1/x2, 0.1/x3}

A2 = {0.4/x1, 0/x2, 0.9/x3}

Fuzzy Clustering Example (fromKlir&Yuan)

33.1)()()(0

37.1)()()(0

:2Property

0.1)()(

0.1)()(

0.1)()( :1

Property

32

22

12

31

21

11

3

2

3

1

22

21

12

11

!

!

!

!

!

x x x

x x x

x x

x x

x x

A A A

A A A

A A

A A

A A

QQQ

QQQ

QQ

QQ

QQ



Fuzzy Clustering

In general:

¹¹¹¹¹¹¹

¹

º

¸

©©©©©©©

©

ª

¨

¹¹¹¹¹¹¹¹

¹¹

º

¸

©©©©©©©©

©©

ª

¨

!

!

000,60$

20

#17,student particular

example,our For vector.ldimensiona- pa

17

1

smathematic x

x

x

x p

p j

j

j/

T



Fuzzy Clustering

Suppose all components to the vectors in the dataset are numeric, then:

m>1 governs the effect of the membership grade.

(4.1)

)]([

)]([

1

1

m

j

N

j i A

j

m

j

N

j i A

i

x

x x

v

§

§!

!

!

Q

Q



Fuzzy Clustering

Given a way to compute the center vi we need a way tomeasure how good these centers are (one by one).This is done by a performance measure orobjective function as follows:

(4.2) )]([)(2

1 1

i jm

j

N

j

c

i i

Av x x P m J § §!

! !

Q



Fuzzy Clustering: Fuzzy c-means algorithm

Step 1: Set k=0, select an initial partition P(0)

Step 2: Calculate centers vi(k) according to equation

(4.1)

Step 3: Update the partition to P(k+1) according to:

Fuzzy Clustering: Fuzzy c-means algorithm (step 3



Fuzzy Clustering: Fuzzy c means algorithm (step 3continued)

.for 0)(

and1)(

thatsovaluesly)(arbitrariassignthen},,...,1{somefor 0If

.)(

then0,if ),dataset(

)1(

)1(

2)(

1

1

1

2)(

2)(

)1(

2)(

I i x

x

c I iv x

v x

v x x

v x X x

jk

i A

I i j

k

i A

k

i j

c

l

m

k l j

k

i j

jk

i

A

k

i j j

!

!§

!

§

!

"

¼¼¼¼¼¼

¼¼¼¼¼¼¼

½

»

¬¬¬¬¬¬

¬¬¬¬¬¬¬

-

«

¹¹¹

¹¹¹¹¹¹

º

¸

©©©

©©©©©©

ª

¨

Q

Q

Q



Step 4: Compare P(k) to P(k+1) . If || P(k) - P(k+1) || < Ithen stop. Otherwise set k:=k+1 and go to step 2.

Remark: the computation of the updated membershipfunction is the condition for the minimization of theobjective function given by equation (4.2).

The example that follows uses c=2, I!theEuclidean norm and A1 = {0.854/x1 ,, 0.854/x15 }

and

A2 = {0.146/x1 ,, 0.146/x15 }.

For k=6, A1 and A2 are given in the following slidewhere v1

(6)=(0.88,2)T and v2(6)=(5.14,2)T

Fuzzy Clustering: Fuzzy c-means algorithm





Cluster (www.m-w.com)

A number of similar individuals that

occur together as a: two or more

consecutive consonants or vowels in

a segment of speech b: a group of

houses (...) c: an aggregation of

stars or galaxies that appear close

together in the sky and aregravitationally associated.

Cluster analysis (



Cluster analysis (www.m-

w.com)

A statistical classification technique

for discovering whether the

individuals of a population fall into

different groups by making

quantitative comparisons of multiple

characteristics.



Vehicle Example

Vehicle Top speed

km/h

Colour Air

resistance

Weight

Kg

V1 220 red 0.30 1300

V2 230 black 0.32 1400V3 260 red 0.29 1500

V4 140 gray 0.35 800

V5 155 blue 0.33 950

V6 130 white 0.40 600

V7 100 black 0.50 3000

V8 105 red 0.60 2500

V9 110 gray 0.55 3500



Vehicle Clusters

100 150 200 250 300500

1000

1500

2000

2500

3000

3500

Top speed [km/h]

W e i g h t [ k g ] Sports cars

Medium market cars

Lorries



Terminology

100 150 200 250 300500

1000

1500

2000

2500

3000

3500

Top speed [km/h]

W e i g h t [ k g ] Sports cars

Medium market cars

Lorries

Object or data point

featur e

feature space

cluster

featur e

label



Example: Classify cracked tiles



475Hz 557Hz Ok?

-----+-----+---

0.958 0.003 Yes

1.043 0.001 Yes

1.907 0.003 Yes

0.780 0.002 Yes

0.579 0.001 Yes

0.003 0.105 No

0.001 1.748 No

0.014 1.839 No

0.007 1.021 No

0.004 0.214 No

Table 1: frequency

intensities for ten

tiles.

Tiles are made from clay moulded into the right shape, brushed, glazed,and baked. Unfortunately, the baking may produce invisible cracks.

Operators can detect the cracks by hitting the tiles with a hammer, and in

an automated system the response is recorded with a microphone, filtered,

Fourier transformed, and normalised. A small set of data is given in TABLE

1 (adapted from MIT, 1997).



Algorithm: hard c-means (HCM)(also known as k means)



Plot of tiles by frequencies (logarithms). The whole tiles (o) seem

well separated from the cracked tiles (*). The objective is to find

the two clusters.

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

log( intens i ty) 475 Hz

l o g ( i n t e n s i t y ) 5 5 7

H

z

Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres



1. Place two cluster centres (x) at random.

2. Assign each data point (* and o) to the nearest cluster centre (x)

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g ( i n t e n s i t y ) 5 5 7

H

z




-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g ( i n t e n s i t y ) 5 5 7

H

z


1. Compute the new centre of each class

2. Move the crosses (x)



Iteration 2

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g ( i n t e n s i t y ) 5 5 7

H

z




Iteration 3

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g ( i n t e n s i t y ) 5 5 7

H

z




Iteration 4 (then stop, because no visible change)

Each data point belongs to the cluster defined by the nearest centre

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g ( i n t e n s i t y ) 5 5 7

H

z


M =



The membership matrix M:

1. The last five data points (rows) belong to the first cluster (column)

2. The first five data points (rows) belong to the second cluster (column)

M =

0.0000 1.0000

0.0000 1.0000

0.0000 1.0000

0.0000 1.0000

0.0000 1.0000

1.0000 0.0000

1.0000 0.0000

1.0000 0.0000

1.0000 0.0000

1.0000 0.0000



Membership matrix M

±°±̄®

e!otherwi sei f m jk ik

ik

01

22

cucu

data point k cluster centrei

distance

cluster centre j



c-partition

K c

iall f or U C Ø

jiall f or Ø C C

U C

i

ji

c

ii

ee

{!

!!

2

17

All clusters C

together fills thewhole universe U

Clusters do notoverlap

A cluster C isnever empty and itis smaller than thewhole universe U

There must be at least 2clusters in a c-partition

and at most as many asthe number of data

points K



Objective function

§ §§! !

¹¹ º

¸©©ª

¨!!

c

i C k

ik

c

i

i

ik

J J 1

2

,1 u

cu

Minimise the total sumof all distances



Algorithm: fuzzy c-means (FCM)



Each data point belongs to two clusters to different degrees

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g (

i n t e n s i t y ) 5 5 7

H

z




1. Place two cluster centres

2. Assign a fuzzy membership to each data point depending on

distance

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g (

i n t e n s i t y ) 5 5 7

H

z




1. Compute the new centre of each class

2. Move the crosses (x)

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g (

i n t e n s i t y ) 5 5 7

H

z






Iteration 5

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

log( intensi ty) 475 Hz

l o g ( i n t e n s i t y ) 5 5 7

H

z

Ti les data: o = who le t iles, * = cracked t iles, x = ce ntres



Iteration 10

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g (

i n t e n s i t y ) 5 5 7

H

z




Iteration 13 (then stop, because no visible change)

Each data point belongs to the two clusters to a degree

-8 -6 -4 -2 0 2-8

-7

-6

-5

-4

-3

-2

-1

0

1

2


l o g (

i n t e n s i t y ) 5 5 7

H

z




Fuzzy membership matrix




M

§!

¹

¹

º

¸

©

©

ª

¨!

c

j

q

jk

ik

ik

d

d

m

1

1/2

1

ik ik d cu !

Distance from point k

to current cluster centre i

Distance from point k

to other cluster centres j

Point k ¶s membershipof cluster i

Fuzziness

exponent





M

ik m

1/21/22

1/21

1/2

1/21/2

2

1/2

1

1

1/2

111

1

1

1

!

!

¹¹ º

¸©©ª

¨¹¹

º

¸©©ª

¨¹¹

º

¸©©ª

¨!

¹¹

º

¸

©©

ª

¨!

§

qck

qk

qk

q

ik

q

ck

ik

q

k

ik

q

k

ik

c

j

q

jk

ik

d d d

d

d

d

d

d

d

d

d

d

.

.

Gravitation tocluster i relative

to total

gravitation



Electrical Analogy

R1 R2i1 i2U

I

I

i

iU I

U

R

R

R R R

R

R R

R R R

R

R I U

i

i

i

c

i

i

c

!!

!

!

!

11

111

1

1

111

1

21

21

.

. Same form asmik



Fuzzy Membership

1 2 3 4 5

0

0.5

1

Cluster centres

M e m b e r s h i p

o f t e s t p o i n t

o is with q = 1.1, * is with q = 2

Datapoint



Fuzzy c-partition

K c

iall f or U C Ø

jiall f or Ø C C

U C

i

ji

c

ii

ee

{!

!!

2

1

7

All clusters C together fillthe whole universe U.

Remar k: The sum of

member ships for a d ata

point i s 1, and t he t otal for

all point s i s K

Not v al i d : C l ust er s

do overla p

A cluster C isnever empty and itis smaller than thewhole universe U

There must be at least 2clusters in a c-partition

and at most as many asthe number of data

points K

Example: Classify cancer



p y

cells

Normal smear Severely dysplastic smear

Using a small brush, cotton stick, or wooden

stick, a specimen is taken from the uterincervix and smeared onto a thin, rectangular glass plate, a slide. The purpose of the smear

screening is to diagnose pre-malignant cellchanges before they progress to cancer . Thesmear is stained using the Papanicolaumethod, hence the name P a p smear . Different characteristics have differentcolours, easy to distinguish in a microscope.

A cyto-technician performs the screening in amicroscope. It is time consuming and proneto error, as each slide may contain up to

300.000 cells.

Dysplastic cells have undergone precancerouschanges. They generally have longer and darker nuclei, and they have a tendency to cling together inlarge clusters. Mildly dysplastic cels have enlarged

and bright nuclei. Moderately dysplastic cells havelarger and darker nuclei. Severely dysplastic cellshave large, dark, and often oddly shaped nuclei. Thecytoplasm is dark, and it is relatively small.





Classes are nonseparable



Hard Classifier (HCM)

Ok

light

moderate

severeOk

A cell is either one

or the other classdefined by a colour .



Fuzzy Classifier (FCM)

Ok

light

moderate

severeOk

A cell can belong to

several classes to aDegree, i.e., one columnmay have several colours.



Function approximation

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-1.5

-1

-0.5

0

0.5

1

1.5

Input

O u t p u

t 1

Curve fitting in a multi-dimensional space is also calledfunc t i on a ppr ox i mat i on. Lear ning is equivalent to finding afunction that best fits the training data.

Approximation by fuzzy



pp y y

sets

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-2

-1

0

1

2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.2

0.4

0.6

0.8

1



Procedure to find a model

1. Acquire data

2. Select structure

3. Find clusters, generate model

4. Validate model



Conclusions

Compared to neural networks, fuzzy

models can be interpreted by human

beings

Applications: system identification,

adaptive systems



Links

J. Jantzen: Neur ofuzzy Modell ing . Technical University of

Denmark: Oersted-DTU, Tech report no 98-H-874 (nfmod), 1998.

URL http://fuzzy.iau.dtu.dk/download/nfmod.pdf

PapSmear tutorial. URL http://fuzzy.iau.dtu.dk/smear/

U. Kaymak: Data Dr i ven Fuzzy Modell ing . PowerPoint, URL

http://fuzzy.iau.dtu.dk/tutor/ddfm.htm

Exercise: fuzzy clustering



Exercise: fuzzy clustering

(Matlab)

Download and follow the instructions in this text

file: http://fuzzy.iau.dtu.dk/tutor/fcm/exerF5.txt

The exercise requires Matlab (no special

toolboxes are required)





Fuzzy Classification

1 2 3 4 5 6 7 8 9 10

1 1 0 0 1 0 0 1 0 0 1

2 0 1 0 0 1 0 0 1 0 0

3 0 0 1 0 0 1 0 0 1 0

4 1 0 0 1 0 0 1 0 0 1

5 0 1 0 0 1 0 0 1 0 0

6 0 0 1 0 0 1 0 0 1 0

7 1 0 0 1 0 0 1 0 0 1

8 0 1 0 0 1 0 0 1 0 0

9 0 0 1 0 0 1 0 0 1 0

10 1 0 0 1 0 0 1 0 0 1

R =

The relation is reflexive, symmetric and transitive. Hence, the

matrix is an equivalence relation.



Fuzzy Classification

We can group the elements of the universe into classes as:

[1] = [4] = [7] = [10] = {1,4,7,10} with remainder = 1

[2] = [5] = [8] = {2,5,8} with remainder = 2

[3] = [6] = [9] = {3,6,9} with remainder = 0

With these classes, we can prove the three properties

discussed earlier . Hence, the quotient set is:

X | R ={(1,4,7,10),(2,5,8

),(3,6,9)}

Not all relations are equivalent, but a tolerance relation can

become an equivalent one by max-min compositions.



Fuzzy Relations

1 0.8 0 0.1 0.2

0.8 1 0.4 0 0.9

0 0.4 1 0 0

0.1 0 0 1 0.5

0.2 0.9 0 0.5 1

Rt = p R =

By taking P-cuts of fuzzy equivalent relation R at values of

P = 1 , 0.9, 0.8, 0.5, 0.4; we get the following:

1 0.8 0.4 0.5 0.8

0.8 1 0.4 0.5 0.9

0.4 0.4 1 0.4 0.4

0.5 0.5 0.4 1 0.5

0.8 0.9 0.4 0.5 1

1 1 0 1 1

1 1 0 1 10 0 1 0 0

1 1 0 1 1

1 1 0 1 1

1 0

11

1

0 1

R1 R0.9 R0.8 R0.5 R0.4

1 0 0 0 0

0 1 0 0 10 0 1 0 0

0 0 0 1 0

0 1 0 0 1

1 1 0 0 1

1 1 0 0 10 0 1 0 0

0 0 0 1 0

1 1 0 0 1

1 1 1 1 1

1 1 1 1 11 1 1 1 1

1 1 1 1 1

1 1 1 1 1



Fuzzy Relations

The classification can be described as follows:



Fuzzy Relations



u y e at o s

Convert to an equivalent relation by composition.

Fuzzy Relations



y

P-cut P = 0.6, we have



Fuzzy Relations

Four distinct classes are identified:

{1,6,8,13,16}, {2,5,7,11,14}, {3}, {4,9,10,12,15}

From this clustering it seems that only photograph number

3 cannot be identified with any of the families. Perhaps a

lower value of P might assign photograph 3 to one of the

other three classes.

The other three clusters are all correct.



Cluster Analysis

How many clusters?

C-means clustering

Sample set: X = {x1,x2,«,xn}

n points, each xi = {xi1,xi2,«,xim} is an m-dimensional vector .

V2V1 Minimize the distance in

each cluster

Maximize the distance

between clusters



Cluster Analysis

Hard C-means (HCM)

Classify data in crisp sense.

Each data will be one and only one cluster .

nC

X A

ji A A

X A

i

ji

C

i

i

ee

{!

!!

2

1

J

J

7



Cluster Analysis

The objective function for the hard c-means algorithm is

known as a within-class sum of squared errors approach

using a Euclidian norm to characterize distance. It is given

by:

Where,

U: partition matrix V: vector of cluster centers

Dik: Euclidian distance in m-dimensional feature spacebetween the kth data sample and ith cluster center vi, givenby:

§§! !

!n

k

C

i

ik ik d vU J 1 1

2, G

2/1

2

¼½

»¬-

«!!! § ijkjik ik ik v xv xv xd d



Fuzzy Pattern Recognition

Features

Feature Extraction

Partition of feature space




Multi-feature pattern recognition: more features

Multi-dimensional pattern recognition

1. Nearest neighbor classifier .

2. Nearest center classifier .

3. Weighted approaching degree.




Nearest neighbor approach:

Sample Xi has m features

xi = {xi1,xi2,«,xim}

X = {X1,X2,«,Xn}We can use C-fuzzy partitions, then get c-hard partitions

If w

e have new

singleton data X, then

x and xi in the same class

ji A A A X ji

c

i

i {!

!

71

_ ak nk

i x xd x xd ,min,1 ee

!




Nearest Center Classifier:

First got c-clusters, the center for each cluster vi and V =

{V1,V2,«,Vc}

x is in cluster i

_ ak ck i v xd v x D ,min, 1 ee!



Syntactic recognition

Examples include image recognition, fingerprintrecognition, chromosome analysis, character recognition,

scene analysis, etc.

Problem: how to deal with noise?

Solution: a few noteworthy of them are [Fu, 1982]:

The use of approximation

The use of transformational grammarsThe use of similarity and error-correcting parsingThe use of stochastic grammarsThe use of fuzzy grammars





Syntactic recognition

A string x L iff

M: # of derivations

lk: the length of the kth derivation chain

r: ith production used in the kth derivation chain

_ ak

il k mk

L

L

r x

x

k

QQ

Q

eeee!

"

11

minmax

0

Documents

Fuzzy Classification Vtu