ANN for pattern recognition applications

Computational Intelligence in Complex Decision Systems G. Oltean

1

ANN for

pattern recognition applications


Pattern recognition includes two main

activities:

• Feature extraction: The features are essential for the particular

recognition / classification process. Typically, the decision taken

at this stage is relatively subjective and depends on practical

considerations such as the accessibility of measurements, their

costs, etc. The criteria for the features selection and their ordering

are based either on their importance in characterizing the forms

or on the contributions they bring to the recognition performance

(the correct / erroneous recognition percentages).

• pattern recognition (pattern classification), meaning decision

making about the membership to one class or another class for

each pattern.


Pattern classification:

Data clustering, uses unsupervised learning, that

is, determining the number of groups, group centers andthe membership of each object (pattern) to each group;

The development and utilization of a patternrecognition systems based on supervised learning.

The once-built and trained recognition system is able torecognize to what class a new pattern belongs;

new pattern - a pattern that did not participate in theprocess of developing (training, testing) the recognitionsystem.


• Recunoastere de catre om prin

perceptie (similaritate, relatii).

• Se folosesc direct obiectele de

clasificat

• Recunoastere de catre calculator -

utilizare formalism matematic: extragerea

trasaturilor in format numeric. Similaritati

matematice pe date, relatii pe date.

•Nu se utilizeaza direct obiectele (formele) ci

descrierea lor prin trasaturi - obiecte de date


The typical structure of a pattern recognition system

Reading SegmentationFeature

extraction

Pattern

classification

Post-

processing

Pattern Decision

Reading - Retrieval and transformation of information from the real world into the space of

patterns, in a format compatible with the way of representing data in the computer (sensors /

translators: video camera, microphone, etc.)

Segmentation (if required) - Isolation of objects of interest from the background or from other

objects (isolation of objects from images or isolation of words)

Feature extraction - Measuring the properties of patterns to build a feature vector. Features are

chosen so that they have similar values for patterns in the same class and different values for

patterns in different classes - distinctive features. PCA - (Principal Component Analysis) tries to

reduce the feature vector size, resulting in an easy-to-process, less-redundant vector.

Pattern classification - Uses the feature vector to assign the pattern to a class: pattern

recognition. Because perfect classification is (almost) impossible, a more general task is to

determine a pattern probability for each class.

Post-processing - Used to recommend certain actions to be taken, each action having a

certain cost (example: decision on what to do next, after the pattern has been recognized.


Start

Collect data

Select features

Select ANN architecture

Train ANN

Evaluate ANN

Accepted

performance?

ANN for pattern

recognition

Workflow for Neural

Network Design(intended for pattern recognition)


Block diagram of a ANN-based pattern recognition system

Each output stands for one class:

1 – the input pattern is recognized in that class

0 – the input pattern is not recognized in that class


Pattern Recognition for Iris flowers

Classify iris flowers based on four features (attributes)

The data set consists of 150 samples.

"irisInputs" is an 4x150 matrix, whose rows are:

1. Sepal length in cm2. Sepal width in cm3. Petal length in cm4. Petal width in cm

"irisTargets" is an 3x150 matrix, where each ith column indicates which category the ith iris belongs to with a 1 in one element (and zeros in the other two elements).

1. Iris Setosa2. Iris Versicolour3. Iris Virginica

One category of iris is linearly separable, the other two are not.


Iris setosa Iris versicolor Iris virginica

Iris dataset is by far the earliest and the most commonly

used dataset in the literature of pattern recognition.

The dataset contains 150 instances of iris flowers

collected in Hawaii.


5.5000 2.5000 4.0000 1.3000 0 1.0000 05.5000 2.6000 4.4000 1.2000 0 1.0000 06.1000 3.0000 4.6000 1.4000 0 1.0000 05.8000 2.6000 4.0000 1.2000 0 1.0000 05.0000 2.3000 3.3000 1.0000 0 1.0000 05.6000 2.7000 4.2000 1.3000 0 1.0000 05.7000 3.0000 4.2000 1.2000 0 1.0000 05.7000 2.9000 4.2000 1.3000 0 1.0000 06.2000 2.9000 4.3000 1.3000 0 1.0000 05.1000 2.5000 3.0000 1.1000 0 1.0000 05.7000 2.8000 4.1000 1.3000 0 1.0000 06.3000 3.3000 6.0000 2.5000 0 0 1.00005.8000 2.7000 5.1000 1.9000 0 0 1.00007.1000 3.0000 5.9000 2.1000 0 0 1.00006.3000 2.9000 5.6000 1.8000 0 0 1.00006.5000 3.0000 5.8000 2.2000 0 0 1.00007.6000 3.0000 6.6000 2.1000 0 0 1.00004.9000 2.5000 4.5000 1.7000 0 0 1.00007.3000 2.9000 6.3000 1.8000 0 0 1.00006.7000 2.5000 5.8000 1.8000 0 0 1.00007.2000 3.6000 6.1000 2.5000 0 0 1.0000

Inputs [cm] Targets

Dataset illustration setosa versicolor virginica


Features of iris flowers

40

45

50

55

60

65

70

75

80

Lungime

sepala

20

25

30

35

40

45

Latime

sepala

10

20

30

40

50

60

70

Lungime

petala

0

5

10

15

20

25

Latime

petala

Setosa

Versicolor

Virginica

Sepal length

Sepal width

Petal length

Petal width


2D representation of iris dataset – only 2 features

45 50 55 60 65 70 75 80

20

25

30

35

40

45

lungime sepala

latim

e s

epala

Setosa

Versicolor

Virginica

Sepal length

Sepal w

idth


>> nprtool >> (nnstrt - nprtool)

Matlab R2016a



18 / 19






number of cases correctly classified

misclassified cases

total percent of correctly classified cases

total percent of misclassified cases

Training: 104 samplesValidation: 23 samplesTest: 23 samples

percent of correctly classified cases

percent of misclassified cases

Confusion matrix

(Error matrix)


Classification results (partial)

3 misclassified out of 150

The ANN produces numerical values on each output:

ideally close to zero - input pattern is not in this class

ideally close to one - input pattern is in this class

The final decision is made considering the maximum between all the outputs


Classification results 3 misclassified out of 150


26 / 19

51 0.0000 0.9959 0.004152 0.0001 0.9909 0.009053 0.0001 0.9813 0.018754 0.0000 0.9983 0.001655 0.0000 0.9863 0.013756 0.0001 0.9949 0.005057 0.0002 0.9574 0.042458 0.0008 0.9990 0.000259 0.0000 0.9978 0.002160 0.0002 0.9965 0.003361 0.0002 0.9996 0.000262 0.0001 0.9936 0.006363 0.0000 0.9998 0.000264 0.0001 0.9870 0.012965 0.0001 0.9993 0.000666 0.0000 0.9979 0.002167 0.0002 0.9686 0.0312

68 0.0000 0.9996 0.000369 0.0000 0.9011 0.098970 0.0000 0.9996 0.000371 0.0002 0.3908 0.609072 0.0000 0.9993 0.000773 0.0001 0.8275 0.172474 0.0000 0.9972 0.002775 0.0000 0.9989 0.001176 0.0000 0.9977 0.002377 0.0000 0.9926 0.007478 0.0001 0.6913 0.308679 0.0001 0.9821 0.017880 0.0000 0.9998 0.000281 0.0000 0.9996 0.000382 0.0000 0.9997 0.000283 0.0000 0.9995 0.000584 0.0001 0.2744 0.7255

85 0.0003 0.9559 0.043886 0.0003 0.9673 0.032387 0.0001 0.9879 0.012088 0.0000 0.9979 0.002189 0.0001 0.9981 0.001890 0.0000 0.9986 0.001491 0.0001 0.9977 0.002392 0.0001 0.9912 0.008893 0.0000 0.9995 0.000594 0.0004 0.9994 0.000295 0.0001 0.9978 0.002196 0.0001 0.9987 0.001297 0.0001 0.9980 0.001998 0.0000 0.9986 0.001499 0.0008 0.9990 0.0002

100 0.0000 0.9986 0.0014

Predicted output for data from index 51 t0 100 (versicolor)


Problem

Implement in Python the ANN pattern trained before to recognize the iris flowers.The parameter of the ANN (weight and bias of hidden layer and for output layer) will be automatically load from text files generated previously from matlab.

Compare the results of your ANN implemented in Python with the results provided by the ANN in matlab, using the same input vectors.


Retraining …. worse results; why?


Classification results 53 misclassified out of 150


1 2 3

1

2

3

13

32.5%

1

2.5%

1

2.5%

86.7%

13.3%

0

0.0%

13

32.5%

2

5.0%

86.7%

13.3%

0

0.0%

0

0.0%

10

25.0%

100%

0.0%

100%

0.0%

92.9%

7.1%

76.9%

23.1%

90.0%

10.0%

Target Class

Ou

tpu

t C

las

s

Confusion Matrix

Exercise The results obtained when testing a

3-class pattern recognition system

are synthesized in the confusion

matrix.

a) How many patterns do we have

in each of the three classes

according to the reference

distribution?

b) Determine how many patterns

have been recognized in each of

the three classes. How many of

these were correctly recognized?

c) Interpret the values in all cells

on 2nd row in the confusion matrix.

d) Interpret the values of all the

cells in the 1st column.

e) Interpret the values in the lower

right cell


Statistical measures of the performance

of a binary classification

The performance of pattern recognition system can be

very well characterized by the recognition rates

considered for each class considering a binary classification

problem.

Thus, for each class

The instance (example, sample) belonging to the class is

considered positive

the instance (example, sample) not belonging to the

class is considered negative.


True positive TP (correctly classified as positive; hit) – an

instance belonging to the class is correctly classified as

belonging to the class;

True negative TN (correctly classified as negative, correct

rejection) – an instance that does not belong to the class is

correctly classified as not belonging to the class;

False pozitiv FP (incorrectly classified as positive, false

alarm) – an instance that does not belong to the class is

classified (incorrectly) as belonging to the class;

False negative FN (incorrectly classified as negative, miss) –

an instance belonging to the class is classified (incorrectly) as

not belonging to the class;

Classification results


True positive rate (sensitivity; recall; hit rate):

TPR = TP / P = TP / (TP + FN)

True negative rate (specificity; selectivity):

TNR = TN / N = TN / (TN + FP)

False positive rate (fall-out):

FPR = FP / N =FP / (TN + FP)

False negative rate (miss rate):

FNR = FN / P = FN / (TP + FN)

TPR + FNR = 1

TNR + FPR = 1

Recognition rates


36 / 19

Precision (positive predicted values):

PPV = TP / (TP + FP)

success probability of making a correct positive class classification

Accuracy:

ACC = (TP + TN) / (P + N) = (TP + TN) / (TP + FN + TN + FP)

success probability of detecting the positive and negative class

Recall (TPR):

Recall = TP / P = TP / (TP + FN)

how sensitive the model is towards identifying the positive class

The 3 pillars of binary classification


Rate

ClassInstances in the

target (reference) class

TPR[%]

TNR [%]

FPR[%]

FNR[%]

123

For a pattern recognition

problem the confusion matrix

is presented in the next figure.

Fill in the next table.

Exercise

Documents

ANN for pattern recognition applications