View
232
Download
0
Category
Preview:
Citation preview
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 1/119
FUZZY CLASSIFICATION
Classification by Equivalence Relations- Crisp Relations, FuzzyRelations, Cluster Analysis, Cluster validity, c-Means Clustering- Hard
c-Means (HCM), Fuzzy c-Means (FCM), Classification Metric,
Hardening the Fuzzy c-Partition, Similarity Relations from Clustering
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 2/119
Hard partition
(i) The partition Covers all data points
(ii)The partition are mutual exclusive
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 3/119
Soft partition
Constrained soft partition
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 4/119
Clustering example
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 5/119
Compact separated clusters
Any two points in a cluster are closer than the
distance between two points in different cluster.
Compact well separated Not Compact
well separated
Compact
Not well separated
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 6/119
Criterion for searching cluster center To to find a minimal J
V is the vector of cluster
centers P is a partition of data set §
! jk C x
k
j
j xC
v||
1
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 7/119
Searching cluster center
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 8/119
Hard C-means algorithm(HCM) (1) calculating the cost J of current partition
(2) modifying the current cluster centers using
gradient descent method to minimize J.
§! jk C x
k
j
j xC v ||
1
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 9/119
Criterion for fuzzy C-means (FCM)
Higher degree of membership will have higher influence
Weighting sum by membership degree
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 10/119
Criterion for fuzzy C-means(FCM)
QCi( x): xci
d i x ci || x ± vi|| = d i Hharmonic mean
X X
v1 X1 X X
X
X v2 XX X
C1 C2
d1d2
1
21 ),()(
1d
d d H x
C
!Q
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 11/119
Harmonic Mean
xy
y x
y x H 11
1),(
!
1
),(e
x
y x H
1),(
e y
y x H
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 12/119
X X
v1
x X X
X
X v2 X
X X
C1 C2
d1
d2
2
21
1
21 ),(),()()(
21d
d d H
d
d d H x x C C ! QQ
]11
)[,(21
21
d d d d H ! 1
),(
1),(
21
21!!
d d H d d H
Constrained soft partition
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 13/119
0),(
)(2
1
2
21
2}}!
d
d
d
d d H x
C Q
1),(
)(
1
21
1}!
d
d d H x
C Q
X X
v1
x X X
X
X v2 X
X X
d1
d2
membership
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 14/119
fuzzy C-means algorithm
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 15/119
Example
Initial v1=(5,5), v2=(10,10)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 16/119
Example-1
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 17/119
Example-2
Initial v1=(5,5), v2=(10,10)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 18/119
Fuzzy Clustering - Theory
CLUSTER ANALYSIS
± way to search for structure in a dataset X
± a component of patter recognition
± clusters form a partition
Examples:
± partition all credit card users into two groups, thosethat
are legally using their credit cards and those who are
illegally using stolen credit cards ± partition UCD students into two classes,
those who will go skiing over winter vacation andthose
who will go to the beach
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 19/119
Clustering is a mathematical tool that
attempts to discover structures or
certain patterns in a data set, where
the objects inside each cluster showa certain degree of similarity.
Clustering
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 20/119
Hard clustering assign each feature
vector to one and only one of theclusters with a degree of membership
equal to one and well defined
boundaries between clusters.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 21/119
F uzzy clustering allows each feature
vector to belong to more than one
cluster with different membership
degrees (between 0 and 1) and
vague or fuzzy boundaries between
clusters.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 22/119
Difficulties with Fuzzy Clustering
The optimal number of clusters K to becreated has to be determined (the
number of clusters cannot always be
defined a priori and a good cluster validity criterion has to be found).
The character and location of cluster
prototypes (centers) is not necessarilyknown a priori, and initial guesses
have to be made.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 23/119
Difficulties with Fuzzy Clustering
The data characterized by large
variabilities in cluster shape, cluster
density, and the number of points(feature vectors) in different clusters
have to be handled.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 24/119
Objectives and Challenges
Create an algorithm for fuzzy clustering that partitions the data set into an optimal number
of clusters.
This algorithm should account for variability
in cluster shapes, cluster densities, and the
number of data points in each of the subsets.
Cluster prototypes would be generated
through a process of unsupervised learning.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 25/119
The Fuzzy k-Means Algorithm
N ± the number of feature vectors K ± the number of clusters (partitions)
q ± weighting exponent (fuzzifier; q > 1)
uik ± the ith membership function
on the k th vector ( uik : X p [0,1] )
k uik = 1; 0 < iuik < nV i ± the cluster prototype (the mean of all
feature vectors in cluster i or thecenter of cluster i)
J q(U,V) ± the objective function
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 26/119
Partition a set of feature vectors X into K clusters (subgroups) represented as
fuzzy sets F 1, F 2, «, F K
by minimizing the objective function J q(U,V)
J q(U,V) = ik (uik )qd 2( X j ± V i ); K e N
Larger membership values indicate higher
confidence in the assignment of the pattern to
the cluster .
The Fuzzy k-Means Algorithm
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 27/119
Description of Fuzzy Partitioning
1) Choose primary cluster prototypes V ifor the values of the memberships
2) Compute the degree of membership of
all feature vectors in all clusters:
uij = [1/d 2(X j ± V i )]1/(q-1) /
k [1/ d 2(X j ± V i )]1/(q-1) (1)
under the constraint: iuik = 1
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 28/119
Description of Fuzzy Partitioning
3) Compute new cluster prototypes V i
V i = j[(uij)q X j ] / j(uij)
q (2)
4) Iterate back and force between (1) and (2)
until the memberships or cluster centers
for successive iteration differ by more than
some prescribed value I (a termination
criterion)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 29/119
The Fuzzy k-Means Algorithm
Computation of the degree of membership uij depends
on the definition of the distance measure, d 2( X j ± V i ):
d 2( X j ± V i ) = ( X j ± V i )T 7 -1( X j ± V i )
7 = I => The distance is Euclidian, the shape of the
clusters assumed to be hyperspherical
7 is arbitrary => The shape of the clusters assumed
to be of arbitrary shape
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 30/119
The Fuzzy k-Means Algorithm
For the hyperellipsoidal clusters, an ³exponential´
distance measure, d 2e ( X j ± V i ), based on ML
estimation was defined:
d 2e (X j ± V i ) = [det( F i )] 1/2 /P i exp[(X j ± V i )
T F i -1(X j ± V i )/2]
F i ± the fuzzy covariance matrix of the ith cluster
P i ± the a priori probability of selecting ith cluster
h( i/ X j ) = (1 / d 2e ( X j ± V i ))/ k (1 / d 2e ( X j ± V k ))
h( i/ X j ) ± the posterior probability (the probability of
selecting ith cluster given jth vector)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 31/119
The Fuzzy k-Means Algorithm
It¶s easy to see that for q = 2, h( i/ X j
) = uijThus, substituting uij with h( i/ X j ) results in the fuzzy
modification of the ML estimation (FMLE).
Addition calculations for the FMLE:
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 32/119
The Major Advantage of FMLE
O btaining good partition results starting from³good´ classification prototypes.
The first layer of the algorithm, unsupervised
tracking of initial centroids, is based on the fuzzyK-means algorithm.
The next phase, the optimal fuzzy partition, is
being carried out with the FMLE algorithm.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 33/119
Unsupervised Tracking of Cluster
Prototypes Different choices of classification prototypes
may lead to different partitions.
Given a partition into k cluster prototypes, place
the next (k +1)th cluster center in a region where
data points have low degree of membership in the
existing k clusters.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 34/119
Unsupervised Tracking of Cluster
Prototypes
1) Compute average and standard deviation of the
whole data set.
2) Choose the first initial cluster prototype at the
average location of all feature vectors.3) Choose an additional classification prototype
equally distant from all data points.
4) Calculate a new partition of the data set
according to steps 1) and 2) of the fuzzyk-means algorithm.
1) If k , the number of clusters, is less than a given
maximum, go to step 3, otherwise stop.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 35/119
Common Fuzzy Cluster Validity
Each data point has K memberships; so, it is
desirable to summarize the information by a
single number, which indicates how well the
data point ( X k ) is classified by clustering.
i(uik )2 partition coeff icient
i(uik ) loguik cl assi f ication entrop y
maxi uik proportional coeff icient
The cluster validity is just the average of any
of those functions over the entire data set.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 36/119
Proposed Performance Measures
³Good´ cl ust er s ar e act uall y not ver y fuzzy.
The criteria for the definition of ³optimal
partition´ of the data into subgroups were
based on the following requirements:
1. Clear separation between the resulting
clusters2. Minimal volume of the clusters
3. Maximal number of data points concentrated
in the vicinity of the cluster centroid
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 37/119
Proposed Performance Measures
Fuzzy hypervolume, F H
V , is defined by:
Where F i is given by:
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 38/119
Proposed Performance Measures
Average partition density, D PA
, is calculated from:
Where S i, the ³ sum of the central members´, is given by:
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 39/119
Proposed Performance Measures
The partition density, P D
, is calculated from:
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 40/119
Sample Runs
In order to test the performance of the
algorithm, N artificial m-dimensional
feature vectors from a multivariate normal
distribution having different parameters anddensities were generated.
Situations of large variability of cluster
shapes, densities, and number of data points
in each cluster were simulated.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 41/119
FCM Clustering with Varying
Density
The higher density cluster attracts all other cluster prototypes
so that the prototype of the right cluster is slightly drawn away
from the original cluster center and the prototype of the left
cluster migrates completely into the dense cluster.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 43/119
Fig. 3. Partition of 12 clusters generated from five-
dimensional multivariate Gaussian distribution with
unequally variable features, variable densities and
variable number of data points ineach cluster (only threeof the features are displayed).
(a) Data points before partitioning
(b) Partition of 12 subgroups using the UFP-O NC algorithm.
All data points gave been classified correctly.
(a) (b)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 45/119
Conclusions
The new algorithm, UFP-O NC(unsupervised fuzzy partition-optimal number
of classes), that combines the most favorable
features of both the fuzzy K-means algorithm
and the FMLE, together with unsupervised
tracking of classification prototypes, were
created.
The algorithm performs extremely well insituations of large variability of cluster shapes,
densities, and number of data points in each
cluster .
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 46/119
Fuzzy Clustering - Theory
REMARKS: (1) The dataset, in the case of studentswould include such things as age, school, incomeof parents, number of years as student, maritalstatus
(2) Classical cluster analysis would partitionthe set of student (with respect to theircharacteristics; that is, the items in the dataset) into
disjoint sets Pi so that we would have:
.for }{and
1
ji j
P i
P c
i P
i
{*!
!
+7
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 47/119
Fuzzy Clustering - Theory
Lets suppose that our dataset has:
Age = {17,18,,35}
School = {Arts, Drama, , Civil Engineering, NaturalSciences, Mathematics, Computer Science}
Income = {$0 $500,000}
Note: It is (or should be) intuitively clear that for this
problem the partitions are intersecting since for manystudents there is an equal preference between goingto the beach and going to ski for vacation and thepreferences are not zero/one for most students.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 48/119
Fuzzy Clustering - Theory
The idea of cluster analysis is to obtain centers (i=1,,cwhere c=2 for the example of skiing and going to thebeach) v
1,,v
cthat are exemplars and radii that will
define the partition. Now, the centers serve asexemplars and an advertising company could sent skiing brochures to the group that is defined by thefirst center and another brochure for beach trips for
students. The idea of fuzzy clustering (fuzzy c-means clustering where c is an a-priori chosennumber of clusters) is to allow overlapping clusterswith partial membership of individuals in clusters.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 49/119
Fuzzy Clustering - Theory
degreesometoelement(s)some
containmust partitiona, partitionsemptynoarethere- 1)(0
ddistributecompletely bemust pointdataeachthatsays- 1)(
such that
,..., partitionfuzzy
,...,dataset:Given
1
1
1
1
§
!§
!
!
!
!
±À
±¿¾
±°
±¯®
±À
±¿¾
±°
±¯®
j
N
j i
A
j
c
i i A
c
N
x
x
A A P
x x X
Q
Q
Fuzzy Clustering Example (from
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 50/119
A1 = {0.6/x1, 1/x2, 0.1/x3}
A2 = {0.4/x1, 0/x2, 0.9/x3}
Fuzzy Clustering Example (fromKlir&Yuan)
33.1)()()(0
37.1)()()(0
:2Property
0.1)()(
0.1)()(
0.1)()( :1
Property
32
22
12
31
21
11
3
2
3
1
22
21
12
11
!
!
!
!
!
x x x
x x x
x x
x x
x x
A A A
A A A
A A
A A
A A
QQQ
QQQ
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 51/119
Fuzzy Clustering
In general:
¹¹¹¹¹¹¹
¹
º
¸
©©©©©©©
©
ª
¨
¹¹¹¹¹¹¹¹
¹¹
º
¸
©©©©©©©©
©©
ª
¨
!
!
000,60$
20
#17,student particular
example,our For vector.ldimensiona- pa
17
1
smathematic x
x
x
x p
p j
j
j/
T
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 52/119
Fuzzy Clustering
Suppose all components to the vectors in the dataset are numeric, then:
m>1 governs the effect of the membership grade.
(4.1)
)]([
)]([
1
1
m
j
N
j i A
j
m
j
N
j i A
i
x
x x
v
§
§!
!
!
Q
Q
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 53/119
Fuzzy Clustering
Given a way to compute the center vi we need a way tomeasure how good these centers are (one by one).This is done by a performance measure orobjective function as follows:
(4.2) )]([)(2
1 1
i jm
j
N
j
c
i i
Av x x P m J § §!
! !
Q
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 54/119
Fuzzy Clustering: Fuzzy c-means algorithm
Step 1: Set k=0, select an initial partition P(0)
Step 2: Calculate centers vi(k) according to equation
(4.1)
Step 3: Update the partition to P(k+1) according to:
Fuzzy Clustering: Fuzzy c-means algorithm (step 3
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 55/119
Fuzzy Clustering: Fuzzy c means algorithm (step 3continued)
.for 0)(
and1)(
thatsovaluesly)(arbitrariassignthen},,...,1{somefor 0If
.)(
then0,if ),dataset(
)1(
)1(
2)(
1
1
1
2)(
2)(
)1(
2)(
I i x
x
c I iv x
v x
v x x
v x X x
jk
i A
I i j
k
i A
k
i j
c
l
m
k l j
k
i j
jk
i
A
k
i j j
!
!§
!
§
!
"
¼¼¼¼¼¼
¼¼¼¼¼¼¼
½
»
¬¬¬¬¬¬
¬¬¬¬¬¬¬
-
«
¹¹¹
¹¹¹¹¹¹
º
¸
©©©
©©©©©©
ª
¨
Q
Q
Q
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 56/119
Step 4: Compare P(k) to P(k+1) . If || P(k) - P(k+1) || < Ithen stop. Otherwise set k:=k+1 and go to step 2.
Remark: the computation of the updated membershipfunction is the condition for the minimization of theobjective function given by equation (4.2).
The example that follows uses c=2, I!theEuclidean norm and A1 = {0.854/x1 ,, 0.854/x15 }
and
A2 = {0.146/x1 ,, 0.146/x15 }.
For k=6, A1 and A2 are given in the following slidewhere v1
(6)=(0.88,2)T and v2(6)=(5.14,2)T
Fuzzy Clustering: Fuzzy c-means algorithm
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 58/119
Cluster (www.m-w.com)
A number of similar individuals that
occur together as a: two or more
consecutive consonants or vowels in
a segment of speech b: a group of
houses (...) c: an aggregation of
stars or galaxies that appear close
together in the sky and aregravitationally associated.
Cluster analysis (
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 59/119
Cluster analysis (www.m-
w.com)
A statistical classification technique
for discovering whether the
individuals of a population fall into
different groups by making
quantitative comparisons of multiple
characteristics.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 60/119
Vehicle Example
Vehicle Top speed
km/h
Colour Air
resistance
Weight
Kg
V1 220 red 0.30 1300
V2 230 black 0.32 1400V3 260 red 0.29 1500
V4 140 gray 0.35 800
V5 155 blue 0.33 950
V6 130 white 0.40 600
V7 100 black 0.50 3000
V8 105 red 0.60 2500
V9 110 gray 0.55 3500
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 61/119
Vehicle Clusters
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [km/h]
W e i g h t [ k g ] Sports cars
Medium market cars
Lorries
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 62/119
Terminology
100 150 200 250 300500
1000
1500
2000
2500
3000
3500
Top speed [km/h]
W e i g h t [ k g ] Sports cars
Medium market cars
Lorries
Object or data point
featur e
feature space
cluster
featur e
label
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 63/119
Example: Classify cracked tiles
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 64/119
475Hz 557Hz Ok?
-----+-----+---
0.958 0.003 Yes
1.043 0.001 Yes
1.907 0.003 Yes
0.780 0.002 Yes
0.579 0.001 Yes
0.003 0.105 No
0.001 1.748 No
0.014 1.839 No
0.007 1.021 No
0.004 0.214 No
Table 1: frequency
intensities for ten
tiles.
Tiles are made from clay moulded into the right shape, brushed, glazed,and baked. Unfortunately, the baking may produce invisible cracks.
Operators can detect the cracks by hitting the tiles with a hammer, and in
an automated system the response is recorded with a microphone, filtered,
Fourier transformed, and normalised. A small set of data is given in TABLE
1 (adapted from MIT, 1997).
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 65/119
Algorithm: hard c-means (HCM)(also known as k means)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 66/119
Plot of tiles by frequencies (logarithms). The whole tiles (o) seem
well separated from the cracked tiles (*). The objective is to find
the two clusters.
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 67/119
1. Place two cluster centres (x) at random.
2. Assign each data point (* and o) to the nearest cluster centre (x)
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 68/119
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
1. Compute the new centre of each class
2. Move the crosses (x)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 69/119
Iteration 2
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 70/119
Iteration 3
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 71/119
Iteration 4 (then stop, because no visible change)
Each data point belongs to the cluster defined by the nearest centre
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
M =
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 72/119
The membership matrix M:
1. The last five data points (rows) belong to the first cluster (column)
2. The first five data points (rows) belong to the second cluster (column)
M =
0.0000 1.0000
0.0000 1.0000
0.0000 1.0000
0.0000 1.0000
0.0000 1.0000
1.0000 0.0000
1.0000 0.0000
1.0000 0.0000
1.0000 0.0000
1.0000 0.0000
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 73/119
Membership matrix M
±°±̄®
e!otherwi sei f m jk ik
ik
01
22
cucu
data point k cluster centrei
distance
cluster centre j
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 74/119
c-partition
K c
iall f or U C Ø
jiall f or Ø C C
U C
i
ji
c
ii
ee
{!
!!
2
17
All clusters C
together fills thewhole universe U
Clusters do notoverlap
A cluster C isnever empty and itis smaller than thewhole universe U
There must be at least 2clusters in a c-partition
and at most as many asthe number of data
points K
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 75/119
Objective function
§ §§! !
¹¹ º
¸©©ª
¨!!
c
i C k
ik
c
i
i
ik
J J 1
2
,1 u
cu
Minimise the total sumof all distances
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 76/119
Algorithm: fuzzy c-means (FCM)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 77/119
Each data point belongs to two clusters to different degrees
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g (
i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 78/119
1. Place two cluster centres
2. Assign a fuzzy membership to each data point depending on
distance
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g (
i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 79/119
1. Compute the new centre of each class
2. Move the crosses (x)
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g (
i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 81/119
Iteration 5
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intensi ty) 475 Hz
l o g ( i n t e n s i t y ) 5 5 7
H
z
Ti les data: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 82/119
Iteration 10
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g (
i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 83/119
Iteration 13 (then stop, because no visible change)
Each data point belongs to the two clusters to a degree
-8 -6 -4 -2 0 2-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
log( intens i ty) 475 Hz
l o g (
i n t e n s i t y ) 5 5 7
H
z
Ti les d ata: o = who le t iles, * = cracked t iles, x = ce ntres
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 84/119
Fuzzy membership matrix
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 85/119
Fuzzy membership matrix
M
§!
¹
¹
º
¸
©
©
ª
¨!
c
j
q
jk
ik
ik
d
d
m
1
1/2
1
ik ik d cu !
Distance from point k
to current cluster centre i
Distance from point k
to other cluster centres j
Point k ¶s membershipof cluster i
Fuzziness
exponent
Fuzzy membership matrix
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 86/119
Fuzzy membership matrix
M
ik m
1/21/22
1/21
1/2
1/21/2
2
1/2
1
1
1/2
111
1
1
1
!
!
¹¹ º
¸©©ª
¨¹¹
º
¸©©ª
¨¹¹
º
¸©©ª
¨!
¹¹
º
¸
©©
ª
¨!
§
qck
qk
qk
q
ik
q
ck
ik
q
k
ik
q
k
ik
c
j
q
jk
ik
d d d
d
d
d
d
d
d
d
d
d
.
.
Gravitation tocluster i relative
to total
gravitation
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 87/119
Electrical Analogy
R1 R2i1 i2U
I
I
i
iU I
U
R
R
R R R
R
R R
R R R
R
R I U
i
i
i
c
i
i
c
!!
!
!
!
11
111
1
1
111
1
21
21
.
. Same form asmik
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 88/119
Fuzzy Membership
1 2 3 4 5
0
0.5
1
Cluster centres
M e m b e r s h i p
o f t e s t p o i n t
o is with q = 1.1, * is with q = 2
Datapoint
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 89/119
Fuzzy c-partition
K c
iall f or U C Ø
jiall f or Ø C C
U C
i
ji
c
ii
ee
{!
!!
2
1
7
All clusters C together fillthe whole universe U.
Remar k: The sum of
member ships for a d ata
point i s 1, and t he t otal for
all point s i s K
Not v al i d : C l ust er s
do overla p
A cluster C isnever empty and itis smaller than thewhole universe U
There must be at least 2clusters in a c-partition
and at most as many asthe number of data
points K
Example: Classify cancer
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 90/119
p y
cells
Normal smear Severely dysplastic smear
Using a small brush, cotton stick, or wooden
stick, a specimen is taken from the uterincervix and smeared onto a thin, rectangular glass plate, a slide. The purpose of the smear
screening is to diagnose pre-malignant cellchanges before they progress to cancer . Thesmear is stained using the Papanicolaumethod, hence the name P a p smear . Different characteristics have differentcolours, easy to distinguish in a microscope.
A cyto-technician performs the screening in amicroscope. It is time consuming and proneto error, as each slide may contain up to
300.000 cells.
Dysplastic cells have undergone precancerouschanges. They generally have longer and darker nuclei, and they have a tendency to cling together inlarge clusters. Mildly dysplastic cels have enlarged
and bright nuclei. Moderately dysplastic cells havelarger and darker nuclei. Severely dysplastic cellshave large, dark, and often oddly shaped nuclei. Thecytoplasm is dark, and it is relatively small.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 92/119
Classes are nonseparable
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 93/119
Hard Classifier (HCM)
Ok
light
moderate
severeOk
A cell is either one
or the other classdefined by a colour .
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 94/119
Fuzzy Classifier (FCM)
Ok
light
moderate
severeOk
A cell can belong to
several classes to aDegree, i.e., one columnmay have several colours.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 95/119
Function approximation
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-1.5
-1
-0.5
0
0.5
1
1.5
Input
O u t p u
t 1
Curve fitting in a multi-dimensional space is also calledfunc t i on a ppr ox i mat i on. Lear ning is equivalent to finding afunction that best fits the training data.
Approximation by fuzzy
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 96/119
pp y y
sets
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1-2
-1
0
1
2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
0.2
0.4
0.6
0.8
1
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 97/119
Procedure to find a model
1. Acquire data
2. Select structure
3. Find clusters, generate model
4. Validate model
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 98/119
Conclusions
Compared to neural networks, fuzzy
models can be interpreted by human
beings
Applications: system identification,
adaptive systems
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 99/119
Links
J. Jantzen: Neur ofuzzy Modell ing . Technical University of
Denmark: Oersted-DTU, Tech report no 98-H-874 (nfmod), 1998.
URL http://fuzzy.iau.dtu.dk/download/nfmod.pdf
PapSmear tutorial. URL http://fuzzy.iau.dtu.dk/smear/
U. Kaymak: Data Dr i ven Fuzzy Modell ing . PowerPoint, URL
http://fuzzy.iau.dtu.dk/tutor/ddfm.htm
Exercise: fuzzy clustering
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 100/119
Exercise: fuzzy clustering
(Matlab)
Download and follow the instructions in this text
file: http://fuzzy.iau.dtu.dk/tutor/fcm/exerF5.txt
The exercise requires Matlab (no special
toolboxes are required)
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 102/119
Fuzzy Classification
1 2 3 4 5 6 7 8 9 10
1 1 0 0 1 0 0 1 0 0 1
2 0 1 0 0 1 0 0 1 0 0
3 0 0 1 0 0 1 0 0 1 0
4 1 0 0 1 0 0 1 0 0 1
5 0 1 0 0 1 0 0 1 0 0
6 0 0 1 0 0 1 0 0 1 0
7 1 0 0 1 0 0 1 0 0 1
8 0 1 0 0 1 0 0 1 0 0
9 0 0 1 0 0 1 0 0 1 0
10 1 0 0 1 0 0 1 0 0 1
R =
The relation is reflexive, symmetric and transitive. Hence, the
matrix is an equivalence relation.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 103/119
Fuzzy Classification
We can group the elements of the universe into classes as:
[1] = [4] = [7] = [10] = {1,4,7,10} with remainder = 1
[2] = [5] = [8] = {2,5,8} with remainder = 2
[3] = [6] = [9] = {3,6,9} with remainder = 0
With these classes, we can prove the three properties
discussed earlier . Hence, the quotient set is:
X | R ={(1,4,7,10),(2,5,8
),(3,6,9)}
Not all relations are equivalent, but a tolerance relation can
become an equivalent one by max-min compositions.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 104/119
Fuzzy Relations
1 0.8 0 0.1 0.2
0.8 1 0.4 0 0.9
0 0.4 1 0 0
0.1 0 0 1 0.5
0.2 0.9 0 0.5 1
Rt = p R =
By taking P-cuts of fuzzy equivalent relation R at values of
P = 1 , 0.9, 0.8, 0.5, 0.4; we get the following:
1 0.8 0.4 0.5 0.8
0.8 1 0.4 0.5 0.9
0.4 0.4 1 0.4 0.4
0.5 0.5 0.4 1 0.5
0.8 0.9 0.4 0.5 1
1 1 0 1 1
1 1 0 1 10 0 1 0 0
1 1 0 1 1
1 1 0 1 1
1 0
11
1
0 1
R1 R0.9 R0.8 R0.5 R0.4
1 0 0 0 0
0 1 0 0 10 0 1 0 0
0 0 0 1 0
0 1 0 0 1
1 1 0 0 1
1 1 0 0 10 0 1 0 0
0 0 0 1 0
1 1 0 0 1
1 1 1 1 1
1 1 1 1 11 1 1 1 1
1 1 1 1 1
1 1 1 1 1
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 105/119
Fuzzy Relations
The classification can be described as follows:
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 106/119
Fuzzy Relations
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 107/119
u y e at o s
Convert to an equivalent relation by composition.
Fuzzy Relations
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 108/119
y
P-cut P = 0.6, we have
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 109/119
Fuzzy Relations
Four distinct classes are identified:
{1,6,8,13,16}, {2,5,7,11,14}, {3}, {4,9,10,12,15}
From this clustering it seems that only photograph number
3 cannot be identified with any of the families. Perhaps a
lower value of P might assign photograph 3 to one of the
other three classes.
The other three clusters are all correct.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 110/119
Cluster Analysis
How many clusters?
C-means clustering
Sample set: X = {x1,x2,«,xn}
n points, each xi = {xi1,xi2,«,xim} is an m-dimensional vector .
V2V1 Minimize the distance in
each cluster
Maximize the distance
between clusters
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 111/119
Cluster Analysis
Hard C-means (HCM)
Classify data in crisp sense.
Each data will be one and only one cluster .
nC
X A
ji A A
X A
i
ji
C
i
i
ee
{!
!!
2
1
J
J
7
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 112/119
Cluster Analysis
The objective function for the hard c-means algorithm is
known as a within-class sum of squared errors approach
using a Euclidian norm to characterize distance. It is given
by:
Where,
U: partition matrix V: vector of cluster centers
Dik: Euclidian distance in m-dimensional feature spacebetween the kth data sample and ith cluster center vi, givenby:
§§! !
!n
k
C
i
ik ik d vU J 1 1
2, G
2/1
2
¼½
»¬-
«!!! § ijkjik ik ik v xv xv xd d
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 113/119
Fuzzy Pattern Recognition
Features
Feature Extraction
Partition of feature space
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 114/119
Fuzzy Pattern Recognition
Multi-feature pattern recognition: more features
Multi-dimensional pattern recognition
1. Nearest neighbor classifier .
2. Nearest center classifier .
3. Weighted approaching degree.
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 115/119
Fuzzy Pattern Recognition
Nearest neighbor approach:
Sample Xi has m features
xi = {xi1,xi2,«,xim}
X = {X1,X2,«,Xn}We can use C-fuzzy partitions, then get c-hard partitions
If w
e have new
singleton data X, then
x and xi in the same class
ji A A A X ji
c
i
i {!
!
71
_ ak nk
i x xd x xd ,min,1 ee
!
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 116/119
Fuzzy Pattern Recognition
Nearest Center Classifier:
First got c-clusters, the center for each cluster vi and V =
{V1,V2,«,Vc}
x is in cluster i
_ ak ck i v xd v x D ,min, 1 ee!
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 117/119
Syntactic recognition
Examples include image recognition, fingerprintrecognition, chromosome analysis, character recognition,
scene analysis, etc.
Problem: how to deal with noise?
Solution: a few noteworthy of them are [Fu, 1982]:
The use of approximation
The use of transformational grammarsThe use of similarity and error-correcting parsingThe use of stochastic grammarsThe use of fuzzy grammars
8/3/2019 Fuzzy Classification Vtu
http://slidepdf.com/reader/full/fuzzy-classification-vtu 119/119
Syntactic recognition
A string x L iff
M: # of derivations
lk: the length of the kth derivation chain
r: ith production used in the kth derivation chain
_ ak
il k mk
L
L
r x
x
k
Q
eeee!
"
11
minmax
0
Recommended