Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow

Multi-Valued and Universal Binary Neurons:

New Solutions in Neural Networks

Igor AizenbergTexas A&M University-Texarkana,

Department of Computer and Information Sciences

[email protected]

http://www.eagle.tamutedu/faculty/igorhttp://www.freewebs.com/igora

mailto:[email protected]

http://www.freewebs.com/igora

NEURAL NETWORKS

Traditional Neurons

Neuro-Fuzzy Networks

Complex-Valued Neurons

Generalizations of sigmoidalNeurons

Multi-Valued andUniversal Binary Neurons

Multi-Valued and Universal Binary Neurons (MVN and UBN)

MVN’s and UBN’s complex-valued activation functions are phase-dependent: their outputs are completely determined by the weighted sum’s argument and they are always lying on the unit circle.

Initial ideas

Generalization of the principles of Boolean threshold logic for the multiple-valued caseExtension of a set of the Boolean functions that allow learning using a single neuron (overcoming of the Minsky-Papert limitation)Consideration of new corresponding types of neurons

Multi-Valued Neuron

(MVN):learns mappings described by multiple-valued functions (functions of multiple-valued logic including continuous-

valued logic)

Universal Binary Neuron (UBN):learns mappings described by Boolean functions that are not necessarily threshold (linearly separable)

Multi-Valued and Universal Binary Neurons

Contents

Multiple-Valued Logic (k-valued Logic) over the Field of the Complex NumbersMulti-Valued Neuron (MVN)Discrete and continuous-valued MVNLearning Algorithm for MVNThreshold Boolean Functions and P-realizable Boolean FunctionsUniversal Binary Neuron (UBN)Relation between MVN and UBNLearning algorithm for UBN Solving the XOR and parity problems on a single UBNMVN-based multilayer feedforward neural network and its derivative free backpropagation learning algorithmApplicationsFurther work

Multi-Valued Neuron (MVN)

A Multi-Valued Neuron is a neural element with n inputs and one output lying on the unit circle, and with the complex-valued weights.The theoretical background behind the MVN is the Multiple-Valued (k-valued) Threshold Logic over the field of complex numbers

Multiple-Valued Threshold Logic over

the field of Complex numbers and a Multi-Valued Neuron

Multiple-Valued Logic: classical view

The values of multiple-valued(k-valued) logic are traditionally encoded by the integers {0,1, …, k-1}

Multiple-Valued Logic over the field of complex numbers: basic idea

Let M be an arbitrary additive group and its cardinality is not lower than k

Let be a structural alphabet

{ }0 1 1, ,... ,, k kk a a aA A M−= ∈


Let be the nth Cartesian power of

Basic Definition: Any function of nvariables is called a function of k-valued logic over group M

nkA

kA

( )1 | :,..., nnk kfx Af x A→


If then we obtain traditional k-valued logic.

In terms of previous considerations is a group with

respect to mod k

addition.

{ }0,1,..., 1kA k= −

{ }0,1,..., 1kA k= −


Let us take the field of complex numbers as a group MLet , where i is an imaginary unity, be a primitive kth root of unity Let be set

C

exp( 2 )i / kε π=

{ }0 2 1, , ,..., kk ε ε ε ε −Ε = kA

Multiple-Valued (k-valued) logic over the field of complex numbers

(Proposed and formulated by Naum

Aizenberg in 1971-1973)

Important conclusion: Any function of nvariables is a function of k-valued logic over the field of complex numbers according to the basic definition

( )1 | :,..., nnk kff x x Ε →Ε

Multiple-Valued (k-valued) logic over the field of complex numbers

(Proposed and formulated by Naum

Aizenberg in 1971-1973)

/exp( 2 )i kε π=

{0, 1, ..., 1}j k∈ −

exp( 2 / )jj i j kπε→ =

primitive kth

root of unity

regular values of k-valued logici

0

1

k-1

k-2

ε

ε2

εk-1

1one-to-one correspondence

{ }0 2 1, , ,..., kjε ε ε ε ε −∈The

kth

roots of unity

are values

of

k-valued logic over the field of complex numbers

Important advantage

In multiple-valued logic over the field of complex numbers all values of this logic are algebraically (arithmetically) equitable: their absolute values are equal to 1

Discrete-Valued (k-valued) Activation Function

( )= exp 2 / , if

( )/ ( ) ( ) /

ji πj k2 j k arg z 2π j +1 k

P z επ

=

≤ <i

0

1

k-2Z

j-1

J j+1

k-1

Function P

maps the complex plane into the set of the kth

roots of unity

jε

1jε +

Multiple-Valued (k-valued) Threshold FunctionsThe k-valued function

is called a k-valued threshold function, if such a complex-valued weighting vector exists that for all from the domain of the function f:

),...,( 1 nxxf

),...,,( 10 nwww),...,( 1 nxxX =

)...(),...,( 1101 nnn xwxwwPxxf +++=

Multi-Valued Neuron (MVN)

(introduced by I.Aizenberg

& N.Aizenberg

in 1992)

)...(),...,( 1101 nnn xwxwwPxxf +++=f

is a function of k-valued logic

(k-valued threshold function) 1x

nx

),...,( 1 nxxf. . .

P(z)

nn xwxwwz +++= ...110

MVN: main properties

The key properties of MVN: Complex-valued weightsThe activation function is a function of the argument of the weighted sumComplex-valued inputs and output that are lying on the unit circle (kth roots of unity)Higher functionality than the one for the traditional neurons (e.g., sigmoidal)Simplicity of learning

MVN Learning

q sδ ε ε= −

qεi

qs

sε

qε

•Learning is reduced to movement along the unit circle

•No derivative is needed, learning is based on the error-correction rule

- error, which completely determines the weights adjustment

sε-

Desired output

-

Actual output

Learning Algorithm for the Discrete MVN with

the Error-Correction Learning Rule

X

qε

W –

weighting vector; X

-

input vectoris a complex conjugated to X

αr

–

learning rate (should be always equal to 1)r -

current iteration;

r+1 –

the next iteration

is a desired

output (sector)

is an actual

output (sector)

1 ( )( 1)

q srr+ r + ε - ε X

n +W W α

= i

qs

sε

qε

sε

Continuous-Valued Activation Function

(introduced by Igor Aizenberg in 2004)

Continuous-valued case (k ∞):

Arg exp( (arg ))

(

)/ | |

i zP z ez

i zz

= = ==

Function P

maps the complex plane into the unit circle

Z

||/)( zzzP =

i

1

Learning Algorithm for the Continuous MVN with

the Error Correction Learning Rule

XW –

weighting vector; X

-

input vector

is a complex conjugated to Xαr

–

a learning rate (should be always equal to 1)r -

current iteration;

r+1 –

the next iterationZ

– the weighted sum

1 ( 1) | |qr r

rr rW z+ ε - XW

n + zα

+

⎛ ⎞= ⎜ ⎟

⎝ ⎠i

|| zzqε

|| zzq −ε

qε is a desired

output

is an actual

output| |zz | |

q zz

εδ = − -

neuron’s error

A role of the factor 1/(n+1) in the Learning Rule

|| zzq −= εδ -

neuron’s error

nnn xn

wwxn

wwn

ww)1(

~ ; ... ;)1(

~ ;)1(

~11100 +

δ+=

+δ

+=+δ

+=

. ...)1(

...)1()1(

))1(

(...))1(

())1(

(

~...~~~

110

110

11110

110

δδ

δδδ

δδδ

+=++++=

=+

++++

+++

+=

=+

++++

+++

+=

=+++=

zxwxwwn

xwn

xwn

w

xxn

wxxn

wn

w

xwxwwz

nn

nn

nn

nn

The weights after the correction:

The weighted sum after the correction:

Self-Adaptation of the Learning Rate

-is the absolute value of the weighted sum on theprevious (rth) iteration.

rz

is a self-adaptive part of the learning rate1

rz

i

|z|

<1|z|

>1

1

1 ( 1) | |r+qr r

rr r

zW W + ε - Xn+ z zα ⎛ ⎞

= ⎜ ⎟⎝ ⎠

1/|zr

|

is a self-adaptive part of the learning rate

Modified Learning Rules with the Self-Adaptive Learning Rate

1 ( )( 1) | |

q srr r

r

W W + ε - ε Xn+ z

α+ =

1/|zr

|

is a self-adaptive part of the learning rate

1 ( 1) | |qr

r rr

zW W + ε - Xn+ z zα

+

⎛ ⎞= ⎜ ⎟

⎝ ⎠

Discrete MVN

Continuous MVN

Convergence of the learning algorithm

It is proven that the MVN learning algorithm converges after not more than k! iterations for the k -valued activation functionFor the continuous MVN the learning algorithm converges with the precision λ after not more than (π/λ)! iterations because in this case it is reduced to learning in π/λ –valued logic.

MVN as a model of a biological neuron

No impulses Inhibition Zero frequency

Intermediate State Medium frequency

Excitation High frequency

The State of a biological neuron is determined by the frequency of the generated impulses

The amplitude of impulses is always a constant


The state of a biological neuron is determined by the frequency of the generated impulses

The amplitude of impulses is always a constant

The state of the multi-valued neuron is determined by the argument of the weighted sum

The amplitude of the state of the multi-valued neuron is always a constant


Maximal inhibition

Maximal excitation

2π0π

Intermediate State

Intermediate State


Maximal inhibition

Maximal excitation

2π0π

MVN:Learns fasterAdapts betterLearns even highly nonlinear functionsOpens new very promising opportunities for the network designIs much closer to the biological neuronAllows to use the Fourier Phase Spectrum as a source of the features for solving different recognition/classification problemsAllows to use hybrid (discrete/continuous) inputs/output

P-realizable Boolean Functions over

the field of Complex numbers and a Universal Binary Neuron

Boolean Alphabets

{0, 1} –

a classical alphabet

{1,-1} –

an alternative alphabet used here

11 ;1021}1,1{ };1,0{

−→→⇒−=−∈∈

yxxy

Threshold Boolean Functions (a classical definition)

The Boolean function is called a threshold

function, if such a real-valued weighting vector exists that for all from the domain of the function f:

),...,( 1 nxxf

),...,,( 10 nwww),...,( 1 nxxX =

⎩⎨⎧

−=<+++=≥+++

1),...,( if 0, ...1),...,( if ,0...

1110

1110

nnn

nnn

xxfxwxwwxxfxwxww

A threshold function is a linearly separable function

1

1

-1

-1

f(1,1)=

1

f(1,-1)= -1

f(-1,1)= -1

f(-1,-1)=

-1

f (x1

, x2

)

is the

OR function

Threshold Boolean Functions

The threshold function could be implemented on the single neuronThe number of threshold functions is very small in comparison to the number of all functions (104 of 256 for n=3, about 2000 of 65536 for n=4, etc.)Non-threshold functions can not be implemented on a single neuron (Minsky-Papert, 1969)

XOR – a classical non-threshold function

1

1

-1

-1

f(1,1)=1

f(1,-1)= -1

f(-1,1)= -1

f(-1,-1)=1

Is it possible to overcome

the Minsky’s-Papert’s

limitation for the classical perceptron?

Yes !!!

P-realizable Boolean functions (introduced by I. Aizenberg in 1984-1985)

))1(2()arg()2(if,)1()(

/mj+π zj/m =zP j

B

<≤−

π

i

0

1

m-1

m-2Z

1

-1

1-1

-1

1

1=)(zPB

Binary activation function:

P-realizable Boolean functions

The Boolean function is called a P-realizable function, if such a complex-valued weighting vector exists that for all from the domain of the function f:

),...,( 1 nxxf

),...,,( 10 nwww

),...,( 1 nxxX =

)...(),...,( 1101 nnBn xwxwwPxxf +++=

Universal Binary Neuron (UBN) (introduced in 1985)

)...(),...,( 1101 nnBn xwxwwPxxf +++=

1x

nx

),...,( 1 nxxf...

PB(z)

nn xwxwwz +++= ...110

UBN: main properties

The key properties of UBN: Complex-valued weightsThe activation function is a function of the argument of the weighted sumAbility to learn arbitrary (not necessarily threshold) Boolean functions on a single neuronSimplicity of learning

XOR problem

x1 x222110 xwxww

z++=

=)(zPB f(x1, x2)

1 1 1+i 1 1

1 -1 1-i -1 -1

-1 1 -1+i -1 -1

-1 -1 -1-i 1 1

1−=BP

1−=BP

n=2, m=4

–

four sectorsW=(0, 1, i) –

the weighting vector

1

i

-i-1

1=BP

1=BP

Parity problemn=3, m=6 : 6 sectors

W=(0, ε, 1, 1) –

the weighting vector

1

1-1

1

-1 -1

ε

1-1

1X 2X 3X3322

110

xwxwxwwZ

++++=

)(ZPB ),,( 321 xxxf

1 1 1 2+ε 1 1 1 1 -1 ε -1 -1 1 -1 1 ε -1 -1 1 -1 -1 2−ε 1 1 -1 1 1 2+ε− -1 -1 -1 1 -1 ε− 1 1 -1 -1 1 ε− 1 1 -1 -1 -1 2−ε− -1 -1

Dependence of the number of sectors on the number of inputs

A proven estimation for m, to ensure P-realizibility of all functionsEstimation for self-dual and even functions (experimental fact, mathematically open problem)P-realizability of all functions may be reduced to P-realizability of self-dual and even functions

n

mn 222 ≤≤

nmn 32 ≤≤

Relation between P-realizable Boolean functions and k-valued threshold functions

If the Boolean function f is P-realizable with a separation of the complex plane into m sectors then at least one m-valued threshold function fm (partially defined on the Boolean domain) exists, whose weighting vector is the same that for the Boolean function f.The learning of P-realizable function may be reduced to the learning of partially defined m-valued threshold function

Learning Algorithm for UBN

q

= s-1 (mod

m),

if

Z

is

closer

to

(s-1)st

sector

q = s+1 (mod

m), if

Z

is

closer

to

(s+1)st

sector

1( )

1

qr

r+ r

sε - ε+ X ,(n+

W)

W α=

s+1 i s s-1

Semi-supervised/semi-

self-adaptive learning

Main Applications

MVN- based Multilayer Feedforward Neural Network (MLMVN) and a Backpropagation Learning

Algorithm

Suggested and developed by Igor Aizenberg et al in 2004-2006

MVN-

based Multilayer Feedforward Neural Network

Hidden layers Output layer

1

2

N

MLMVN: Key PropertiesDerivative-free learning algorithm based on the error-correction ruleSelf-adaptation of the learning rate for all the neuronsMuch faster learning than the one for other neural networksA single step is always enough to adjust the weights for the given set of inputs independently on the number of hidden and output neuronsBetter recognition/prediction/classification rate in comparison with other neural networks, neuro-fuzzy networks and kernel based techniques including SVM

MLMVN: Key Properties

MLMVN can operate with both continuousand discrete inputs/outputs, as well as with the hybrid inputs/outputs: continuous inputs discrete outputsdiscrete inputs continuous outputshybrid inputs hybrid outputs

A Backpropagation Derivative- Free Learning Algorithm

kmT -

a desired output of the kth

neuron from the mth

(output) layer

kmY -

an actual output of the kth

neuron from the mth

(output) layer

kmkmkm YT −=δ* -

the network error for the kth

neuron from output layer

*1km

mkm s

δ=δ -

the error

for the kth

neuron from output layer

11 += −mm Ns -the number of all neurons on the previous layer (m-1, to which the error is backpropagated) incremented by 1


∑∑++

=

−++

=

+++

δ=δ=δ11

1

111

1

1121

)(1)(11 jj N

i

ijkij

j

N

i

ijkijij

kjkj w

sw

ws

The error

for the kth

neuron from the hidden (jth) layer, j=1, …, m-1

1 ;,...,2 ,1 11 ==+= − smjNs jj

-the number of all neurons on the previous layer (previous to j, to which the error is backpropagated) incremented by 1

The error backpropagation:


Correction rule for the neurons from the mth

(output) layer (kth

neuron of mth

layer):

kmkmkmkm

imkmkmkm

ikmi

nCww

niYnCww

δ+

+=

=δ+

+= −

)1(~

,...,1 ,~)1(

~

00

1


Correction rule for the neurons from the 2nd

through m-1st

layer (kth

neuron of the jth

layer (j=2, …, m-1):

kjkj

kjkjkj

ikjkj

kjkji

kji

znC

ww

niYzn

Cww

δ+

+=

=δ+

+=

||)1(~

,...,1 ,~||)1(

~

00


11

110

10

11

111

||)1(~

,...,1 ,||)1(

~

kk

kkk

ikk

kki

ki

znCww

nixzn

Cww

δ+

+=

=δ+

+=

Correction rule for the neurons from the 1st

hidden

layer:

Criteria for the convergence of the learning process

Learning should continue until either minimum MSE/RMSE

criterion will be satisfied or zero-error will be reached

MSE criterion

λ≤=δ ∑∑∑==

N

ss

N

s kskm E

NW

N 11

2* 1)()(1

λ is a maximum possible MSE for the training data

N is

the number of patterns in the training set

is the network square error for the sth

pattern

* 2( )s kmsk

E δ=∑

RMSE criterion

λ≤=δ ∑∑∑==

N

ss

N

s kskm E

NW

N 11

2* 1)()(1

λ is a maximum possible RMSE for the training data

N is

the number of patterns in the training set

is the network square error for the sth

pattern

* 2( )s kmsk

E δ=∑

MLMVN Learning: Example

1 1

2 2

3 3

(exp(4.23 ),exp(2.10 )) ,

(exp(5.34 ),exp(1.24 )) ,

(exp(2.10 ),exp(0 )) .

X i i T

X i i T

X i i T

= →

= →

= →

Suppose, we need to classify three vectors belonging to three different classes:

( ) ( ) ( )1 2 3exp 0.76 , exp 2.56 , exp 5.35 .T i T i T i= = =

1 2 3, ,T T T

[arg( ) 0.05,arg( ) 0.05], 1, 2,3,j jT T j− + =

Classes are determined in such a way that the argument of the desired output of the network must belong to the interval


Thus, we have to satisfy the following conditions:

( ) ( ) arg arg 0.05iArg zjT - e ≤

iArg ze

, where

is the actual output.

and for the mean square error

20.05 0.0025E ≤ =

.


Let us train the 2 1 MLMVN

(two hidden neurons and the single neuron in the output layer

x1

x2

( )21, xxf

11

12

21

MLMVN Learning: ExampleThe training process converges after 7 training epochs.

Update of the outputs:


E p o c h

1 2 3 4 5 6 7

M S E

2.4213 0.1208 0.2872 0.1486 0.0049 0.0026 0.0009

20.05 0.0025E ≤ =

MLMVN: Simulation Results

Simulation Results: Benchmarks

All simulation results for the benchmark

problems are obtained using the network with n

inputs

n S 1 containing a single hidden layer with S

neurons

and a single neuron in the output layer:

Hidden layer

Output layer

The Two Spirals Problem

Two Spirals Problem: complete training of 194 points

Structure of thenetwork A network type The results

2 40 1

MLMVN Trained completely, no errors

A traditional MLP, sigmoid activation function

still remain 4% errors

2 30 1

MLMVN Trained completely, no errors

A traditional MLP, sigmoid activation function

still remain 14% errors

Two Spirals Problem: cross-validation (training of 98 points

and prediction of the rest 96 points)

The prediction rate is stable for all the networks from 2 26 1 till 2 40 1:

68-72%

The prediction rate of 74-75%

appears 1-2 times per 100 experiments with the network 2 40 1

The best known result obtained using the Fuzzy Kernel Perceptron (Nov. 2002) is 74.5%,

But it is obtained using more complicated and larger network

The “Sonar” problem

208 samples:60 continuous-valued inputs1 binary output

The “sonar”

data is trained completely using the simplest possible MLMVN

60 2 1

(from 817 till 3700 epochs in 50 experiments)

The “Sonar”

problem: prediction

104 samples in the training set104 samples in the testing set

The prediction rate is stable for 100 experiments with the simplest possible MLMVN

60 2 1:

88-93%

The best result for the Fuzzy Kernel Perceptron (Nov., 2002) is 94%.The best result for SVM is 89.5%.However, both FKP and SVM used are larger and their trainingis more complicated task.

Mackey-Glass time series prediction

Mackey-Glass differential delay equation:

),()(1.0)(1

)(2.0)(10 tntx

txtx

dttdx

+−−+−

=ττ

The task of prediction is to predict )6( +tx

from )18( ),12( ),6( ),( −−− txtxtxtx

.


Training Data: Testing Data:


RMSE Training: RMSE Testing:

Mackey-Glass time series predictionTesting Results:

Blue curve

–

the actual series; Red curve

–

the predicted series


# of neurons on the hidden

layer 50 50 40

ε- a maximum possible RMSE 0.0035 0.0056 0.0056

Actual RMSE for the training set (min - max) 0.0032 - 0.0035 0.0053 – 0.0056 0.0053 – 0.0056

Min 0.0056 0.0083 0.0086 Max 0.0083 0.0101 0.0125

Median 0.0063 0.0089 0.0097 Average 0.0066 0.0089 0.0098

RMSE for the testing

set SD 0.0009 0.0005 0.0011 Min 95381 24754 34406 Max 272660 116690 137860

Median 145137 56295 62056

Number of training epochs Average 162180 58903 70051

The results of 30 independent runs:


Comparison of MVN to other models:

MLMVNmin

MLMVNaverage

GEFREXM. Rosso, Generic Fuzzy

Learning, 2000 min

EPNet You-Liu,

Evolution. system, 1997

ANFISJ.S. R. Jang,

Neuro-Fuzzy, 1993

CNNE M.M.Islam

et all., Neural

Networks Ensembles,

2003

SuPFuNISS.Paul et

all, Fuzzy-Neuro, 2002

Classical Backprop.

NN 1994

0.0056 0.0066 0.0061 0.02 0.0074 0.009 0.014 0.02

MLMVN outperforms all other networks in:

•The number of either hidden neurons or supporting vectors

•Speed of learning

•Prediction quality

Real World Problems:

Solving Using MLMVN

Classification of gene expression microarray data

“Lung”

and “Novartis”

data sets

4 classes in both data sets“Lung”: 197 samples with 419 genes“Novartis”: 103 samples with 697 genes

K-folding cross validation with K=5

“Lung”: 117-118 samples of 197 in each training set and 39-40 samples of 197 in each testing set“Novartis”: 80-84 samples of 103 in each training set and 19-23 samples of 103 in each testing set

MLMVN

Structure of the network: n 6 4 (n inputs, 6 hidden neurons and 4 output neurons)The learning process converges very quickly (500-1000 iterations that take up to 1 minute on a PC with Pentium IV 3.8 GHz CPU are required for both data sets)

Results: Classification Rates

MLMVN: 98.55% (“Novartis)MLMVN: 95.945% (“Lung”)kNN (k=1) 97.69% (“Novartis”)kNN (k=2) 92.55% (“Lung”)

Generation of the Genetic Code using MLMVN

Genetic code

There are exactly four different nucleic acids: Adenosine (A), Thymidine (T), Cytidine (C) and Guanosine (G)Thus, there exist 43=64 different combinations of them “by three”. Hence, the genetic code is the mappingbetween the four-letter alphabet of the nucleic acids (DNA) and the 20-letter alphabet of the amino acids (proteins)

Genetic code as a multiple- valued function

Let be the amino acidLet be the nucleic acidThen a discrete function of three variables

, 1, ..., 20jG j =

{ }, , , , 1, 2, 3ix A G C T i∈ =

1 2 3( , )jG f x , x x=

is a function of a 20-valued logic, which is partially defined on the set of four-valuedvariables

Genetic code as a multiple- valued function

A multiple-valued logic over the field of complex numbers is a very appropriate model to represent the genetic code functionThis model allows to represent mathematically those biological properties that are the most essential (e.g., complementary nucleic acids A T; G Cthat are stuck to each other in a DNA double helix only in this order

DNA double helix

The complementary nucleic acids

A T; G C

are always stuck to each other in pairs

Representation of the nucleic acids and amino acids

ND

T

F

SA

AY

G

WM

I

H

Q

T K

ERGC

LV

PC

The complementary nucleic acids

A T; G Care located such that

A=1; G=i

T

= -A= -1

and

C= -

G

= -i

All amino acids

are distributed along the unit circle in the logical way, to insure their closeness to those nucleic acids that form each certain amino acid

Generation of the genetic code

The genetic code can be generated using MLMVN 3 4 1 (3 inputs, 4 hidden neurons and a single output neuron – 5 neurons)There best known result for the classical backpropagation neural network is generation of the code using the network 3 12 2 20 (32 neurons)

Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN

Problem statement: capturing

Mathematically a variety of capturing principles can be described by the Fredholm integral of the first kind

where x,t ℝ2, v(t) is a point-spread function (PSF) of a system, y(t) is a function of a real object and z(x) is an observed signal.

2

2( ) ( , ) ( ) , ,z x v x t y t dt x t= ∈∫

Mic

rosc

opy

Tomog

raphy

∈

Ph

oto

Image deblurring: problem statement

Mathematically blur is caused by the convolution of an image with the distorting kernel.Thus, removal of the blur is reduced to the deconvolution.Deconvolution is an ill-posed problem, which results in the instability of a solution. The best way to solve it is to use some regularization technique.To use any kind of regularization technique, it is absolutely necessary to know the distorting kernel corresponding to a particular blur: so it is necessary to identify the blur.

Image deblurring: problem statement

The observed image given in the following form:

where “*”

denotes a convolution operation, y

is an image, υ is point-spread function

of a system (PSF) , which is exactly

a distorting kernel, and ε is a noise.

In the continuous frequency domain

the model takes the form:

where is a representation of the signal z

in the Fourier domain

and denotes a Fourier transform.

( ) ( )( ) ( )z x v y x xε= ∗ + ,

( ) ( ) ( ) ( )Z V Yω ω ω ε ω= ⋅ + ,

2( ) { ( )}Z F z x Rω ω= , ∈ ,{}F ⋅

Blur Identification

We use MLMVN to recognize Gaussian, motion and rectangular (boxcar) blurs.We aim to identify simultaneously both blur (PSF), and its parameters using a single neural network.

Considered PSFPSF in time domain

PSF in frequency domain

Gaussian Motion Rectangular

Considered PSF

The Gaussian PSF:

is a parameter (variance)

The uniform linear motion:

h

is a parameter (the length of motion)

The uniform rectangular:

h

is a parameter (the size of smoothing area)

2 21 2

2 2

1( ) exp2

t tv t

πτ τ⎛ ⎞+

= −⎜ ⎟⎝ ⎠

2 21 2 1 2

1 , / 2, cos sin ,( )otherwise,0,

t t h t tv t h φ φ⎧⎪ + < == ⎨⎪⎩

1 22

1 , , , ( ) 2 2

otherwise,0,

h ht tv t h

⎧ < <⎪= ⎨⎪⎩

2τ

Degradation in the frequency domain:

True Image Gaussian Rectangular HorizontalMotion

VerticalMotion

Images and log of their Power Spectra log Z

Training Vectors

We state the problem as a recognition of the shape of V , which is a Fourier spectrum of PSF υ and its parameters from the Power-Spectral Density, whose distortions are typical for each type of blur.

Training Vectors

( )( ) ( )( ) ( )

1 2, min

max min

log logexp 2 ( 1) ,

log logk k

j

Z Zx i K

Z Z

ωπ

⎛ ⎞−⎜ ⎟= ⋅ −⎜ ⎟−⎜ ⎟⎝ ⎠

),...,( 1 nxxX =The training vectors are formed as follows:

for

(| ( ) |)Log Z ω

(0,0)ω

1 2 2

1 2

2 1

1,..., / 2 1, for , 1,..., / 2 1,/ 2,..., 2, for 1, 1,..., / 2 1,

1,...,3 / 2 3, for 1, 1,..., / 2 1,

j L k k k Lj L L k k Lj L L k k L

= − = = −⎧⎪ = − = = −⎨⎪ = − − = = −⎩

LxL is a size of an image

the length of the pattern vector is n= 3L/2-3

Examples of training vectors

True Image Gaussian Rectangular

HorizontalMotion

VerticalMotion

Neural Network

Hidden layers Output layer

1

2

N

Blur 1

Blur 2

Blur N

Training (pattern) vectors

Output Layer Neuron

Reservation of domains on the unit circle for the output neuron

Simulation

{ }1, 1.33, 1.66, 2, 2.33, 2.66, 3 ;τ ∈

Experiment 1 (2700 training pattern vectors corresponding to 72 images): six types of blur with the following parameters:MLMVN structure: 5 35 6

1) The Gaussian blur is considered with 2) The linear uniform horizontal motion blur of the lengths 3, 5, 7, 9; 3) The linear uniform vertical motion blur of the length 3, 5, 7, 9; 4) The linear uniform diagonal motion from South-West to North-

East blur of the lengths 3, 5, 7, 9; 5) The linear uniform diagonal motion from South-East to North-

West blur of the lengths 3, 5, 7, 9; 6) rectangular has sizes 3x3, 5x5, 7x7, 9x9.

ResultsClassification Results

Blur

MLMVN, 381 inputs,5 35 6,

2336 weights in total

SVMEnsemble from

27 binary decision SVMs,

25.717.500 support vectors in total

No blur 96.0% 100.0%Gaussian 99.0% 99.4%Rectangular 99.0% 96.4

Motion horizontal 98.5% 96.4Motion vertical 98.3% 96.4

Motion North-East Diagonal 97.9% 96.5Motion North-West Diagonal 97.2% 96.5

Restored images

Blurred noisy image:rectangular 9x9

Restored

Blurred noisy image:Gaussian, σ=2

Restored

Main Publications1. N.N. Aizenberg, Yu. L.Ivaskiv and D.A. Pospelov "About one Generalization of the

Threshold Function", Doklady Akademii Nauk SSSR (The Reports of the Academy of Sciences of USSR), vol. 196, No. 6, 1971, pp. 1287–1290. (in Russian).

2. N.N. Aizenberg, Yu.L. Ivaskiv, D.A. Pospelov and G.F. Hudiakov "Multi-Valued Threshold Functions. 1. The Boolean Complex-Threshold Functions and their Generalization", Kibernetika (Cybernetics), No. 4, 1971, pp. 44-51 (in Russian).

3. N.N. Aizenberg, Z. Toshich. A Generalization of the Threshold Functions, // Proceedings of the Electronic Department University of Belgrade, No.399, 1972, pp.97-99.

4. N.N. Aizenberg, Yu.L. Ivaskiv, D.A. Pospelov and G.F. Hudiakov "Multi-Valued Threshold Functions. 2. Synthesis of the Multi-Valued Threshold Element" Kibernetika(Cybernetics), No. 1, 1973, pp. 86-94 (in Russian).

5. N.N. Aizenberg and Yu.L. Ivaskiv Multiple-Valued Threshold Logic, Naukova Dumka Publisher House, Kiev, 1977 (in Russian).

6. I.N. Aizenberg "Model of the Element with a Complete Functionality", Izvestia AN SSSR. Technicheskaya ("Journal of Computer and System Sciences International "), 1985, No. 2., pp. 188-191 (in Russian).

7. I.N. Aizenberg "The Universal Logical Element over the Field of the Complex Numbers",Kibernetika (Cybernetics and Systems Analysis), No.3, 1991, pp.116-121 (in Russian).

Main Publications8. N.N. Aizenberg and I.N. Aizenberg "Model of the Neural Network Basic Elemets (Cells)

with Universal Functionality and various of Harware Implementation", Proc. of the 2-ndInternational Conference "Microelectronics for Neural Networks", Kyrill & MethodyVerlag, Munich, 1991, pp. 77-83.

9. N.N.Aizenberg and I.N.Aizenberg "CNN Based on Multi-Valued Neuron as a Model ofAssociative Memory for Gray-Scale Images", Proceedings of the Second IEEEInternational Workshop on Cellular Neural Networks and their Applications, TechnicalUniversity Munich, Germany October 14-16, 1992, pp.36-41.

10. N.N.Aizenberg, I.N. Aizenberg and G.A.Krivosheev “Method of Information Processing”Patent of Russia No 2103737, G06 K 9/00, G 06 F 17/10, priority date 23.12.1994.

11. N.N. Aizenberg, I.N. Aizenberg and G.A. Krivosheev "Multi-Valued Neurons: Learning,Networks, Application to Image Recognition and Extrapolation of Temporal Series",Lecture Notes in Computer Science, vol. 930 (J.Mira, F.Sandoval - Eds.), Springer-Verlag, 1995,pp.389-395.

12. I.N. Aizenberg. and N.N. Aizenberg “Universal binary and multi-valued neuronsparadigm: Conception, Learning, Applications”, Lecture Notes in Computer Sciense, vol.1240 (J.Mira,R.Moreno-Diaz, J.Cabestany - Eds.), Springer-Verlag, 1997, pp. 463-472.

13. I.N. Aizenberg “Processing of noisy and small-detailed gray-scale images using cellularneural networks”, Journal of Electronic Imaging, vol. 6, No. 3, July, 1997, pp. 272-285.

Main Publications

14. I. Aizenberg., N. Aizenberg, J. Hiltner, C. Moraga, and E. Meyer zu Bexten., “Cellular Neural Networks andComputational Intelligence in Medical Image Processing”, Journal of Image and Vision Computing, Vol. 19, March,2001, pp. 177-183.

15. I. Aizenberg., E. Myasnikova, M. Samsonova., and J. Reinitz , “Temporal Classification of DrosophilaSegmentation Gene Expression Patterns by the Multi-Valued Neural Recognition Method”, MathematicalBiosciences , Vol.176 (1), 2002, pp. 145-159.

16. I. Aizenberg and C. Butakoff, “Image Processing Using Cellular Neural Networks Based on Multi-Valued andUniversal Binary Neurons”, Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology,Vol. 32, 2002, pp. 169-188.

A book

Igor Aizenberg, Naum Aizenberg and Joos

Vandewalle

“Multi-Valued and Universal Binary Neurons:

Theory, Learning and Applications”,

Kluwer, 2000

The latest publications1.

Aizenberg I., Paliy D., Zurada J., and Astola J., "Blur Identification by Multilayer Neural Network based on Multi-Valued Neurons", IEEE Transactions on Neural Networks, accepted, to appear: April, 2008.

2.

Aizenberg I., "Solving the XOR and Parity n

Problems Using a Single Universal Binary Neuron", Soft Computing, vol. 12, No 3, February, 2008, pp. 215-222.

3.

Aizenberg I. and Moraga C. "Multilayer Feedforward Neural Network based on Multi-Valued Neurons and a Backpropagation Learning Algorithm", Soft Computing, Vol. 11, No 2, February, 2007, pp. 169-183.

4.

Aizenberg I. and Moraga C., "The Genetic Code as a Function of Multiple- Valued Logic Over the Field of Complex Numbers and its Learning using

Multilayer Neural Network Based on Multi-Valued Neurons", Journal of Multiple-Valued Logic and Soft Computing, No 4-6, November 2007, pp. 605-

618.

5.

Aizenberg I. Advances in Neural Networks. Class notes

for the graduate course. University of Dortmund. 2005.

Some Prospective ApplicationsSolving different recognition and classification problems, especially those, where the formalization of the decision rule is a complicated problemClassification and prediction in biologyTime series predictionModeling of complex systems including hybrid systems that depend on many parametersNonlinear filtering (MLMVN can be used as a nonlinear filter)Etc. …

CONCLUSION

The best results are expected in the near

future…

If it be now, 'tis not to come; if it be not to come, it will be now;

if it be not now, yet it will come: the readiness is all…

Shakespeare

Documents

Multi-Valued and Universal Binary Neurons: Theory and Learningbiehl/Coll/aizenberg.pdf · the unit circle. Initial ideas ... Extension of a set of the Boolean functions that allow