Multi-Valued and Universal Binary Neurons:
New Solutions in Neural Networks
Igor AizenbergTexas A&M University-Texarkana,
Department of Computer and Information Sciences
http://www.eagle.tamutedu/faculty/igorhttp://www.freewebs.com/igora
NEURAL NETWORKS
Traditional Neurons
Neuro-Fuzzy Networks
Complex-Valued Neurons
Generalizations of sigmoidalNeurons
Multi-Valued andUniversal Binary Neurons
Multi-Valued and Universal Binary Neurons (MVN and UBN)
MVN’s and UBN’s complex-valued activation functions are phase-dependent: their outputs are completely determined by the weighted sum’s argument and they are always lying on the unit circle.
Initial ideas
Generalization of the principles of Boolean threshold logic for the multiple-valued caseExtension of a set of the Boolean functions that allow learning using a single neuron (overcoming of the Minsky-Papert limitation)Consideration of new corresponding types of neurons
Multi-Valued Neuron
(MVN):learns mappings described by multiple-valued functions (functions of multiple-valued logic including continuous-
valued logic)
Universal Binary Neuron (UBN):learns mappings described by Boolean functions that are not necessarily threshold (linearly separable)
Multi-Valued and Universal Binary Neurons
Contents
Multiple-Valued Logic (k-valued Logic) over the Field of the Complex NumbersMulti-Valued Neuron (MVN)Discrete and continuous-valued MVNLearning Algorithm for MVNThreshold Boolean Functions and P-realizable Boolean FunctionsUniversal Binary Neuron (UBN)Relation between MVN and UBNLearning algorithm for UBN Solving the XOR and parity problems on a single UBNMVN-based multilayer feedforward neural network and its derivative free backpropagation learning algorithmApplicationsFurther work
Multi-Valued Neuron (MVN)
A Multi-Valued Neuron is a neural element with n inputs and one output lying on the unit circle, and with the complex-valued weights.The theoretical background behind the MVN is the Multiple-Valued (k-valued) Threshold Logic over the field of complex numbers
Multiple-Valued Threshold Logic over
the field of Complex numbers and a Multi-Valued Neuron
Multiple-Valued Logic: classical view
The values of multiple-valued(k-valued) logic are traditionally encoded by the integers {0,1, …, k-1}
Multiple-Valued Logic over the field of complex numbers: basic idea
Let M be an arbitrary additive group and its cardinality is not lower than k
Let be a structural alphabet
{ }0 1 1, ,... ,, k kk a a aA A M−= ∈
Multiple-Valued Logic over the field of complex numbers: basic idea
Let be the nth Cartesian power of
Basic Definition: Any function of nvariables is called a function of k-valued logic over group M
nkA
kA
( )1 | :,..., nnk kfx Af x A→
Multiple-Valued Logic over the field of complex numbers: basic idea
If then we obtain traditional k-valued logic.
In terms of previous considerations is a group with
respect to mod k
addition.
{ }0,1,..., 1kA k= −
{ }0,1,..., 1kA k= −
Multiple-Valued Logic over the field of complex numbers: basic idea
Let us take the field of complex numbers as a group MLet , where i is an imaginary unity, be a primitive kth root of unity Let be set
C
exp( 2 )i / kε π=
{ }0 2 1, , ,..., kk ε ε ε ε −Ε = kA
Multiple-Valued (k-valued) logic over the field of complex numbers
(Proposed and formulated by Naum
Aizenberg in 1971-1973)
Important conclusion: Any function of nvariables is a function of k-valued logic over the field of complex numbers according to the basic definition
( )1 | :,..., nnk kff x x Ε →Ε
Multiple-Valued (k-valued) logic over the field of complex numbers
(Proposed and formulated by Naum
Aizenberg in 1971-1973)
/exp( 2 )i kε π=
{0, 1, ..., 1}j k∈ −
exp( 2 / )jj i j kπε→ =
primitive kth
root of unity
regular values of k-valued logici
0
1
k-1
k-2
ε
ε2
εk-1
1one-to-one correspondence
{ }0 2 1, , ,..., kjε ε ε ε ε −∈The
kth
roots of unity
are values
of
k-valued logic over the field of complex numbers
Important advantage
In multiple-valued logic over the field of complex numbers all values of this logic are algebraically (arithmetically) equitable: their absolute values are equal to 1
Discrete-Valued (k-valued) Activation Function
( )= exp 2 / , if
( )/ ( ) ( ) /
ji πj k2 j k arg z 2π j +1 k
P z επ
=
≤ <i
0
1
k-2Z
j-1
J j+1
k-1
Function P
maps the complex plane into the set of the kth
roots of unity
jε
1jε +
Multiple-Valued (k-valued) Threshold FunctionsThe k-valued function
is called a k-valued threshold function, if such a complex-valued weighting vector exists that for all from the domain of the function f:
),...,( 1 nxxf
),...,,( 10 nwww),...,( 1 nxxX =
)...(),...,( 1101 nnn xwxwwPxxf +++=
Multi-Valued Neuron (MVN)
(introduced by I.Aizenberg
& N.Aizenberg
in 1992)
)...(),...,( 1101 nnn xwxwwPxxf +++=f
is a function of k-valued logic
(k-valued threshold function) 1x
nx
),...,( 1 nxxf. . .
P(z)
nn xwxwwz +++= ...110
MVN: main properties
The key properties of MVN: Complex-valued weightsThe activation function is a function of the argument of the weighted sumComplex-valued inputs and output that are lying on the unit circle (kth roots of unity)Higher functionality than the one for the traditional neurons (e.g., sigmoidal)Simplicity of learning
MVN Learning
q sδ ε ε= −
qεi
qs
sε
qε
•Learning is reduced to movement along the unit circle
•No derivative is needed, learning is based on the error-correction rule
- error, which completely determines the weights adjustment
sε-
Desired output
-
Actual output
Learning Algorithm for the Discrete MVN with
the Error-Correction Learning Rule
X
qε
W –
weighting vector; X
-
input vectoris a complex conjugated to X
αr
–
learning rate (should be always equal to 1)r -
current iteration;
r+1 –
the next iteration
is a desired
output (sector)
is an actual
output (sector)
1 ( )( 1)
q srr+ r + ε - ε X
n +W W α
= i
qs
sε
qε
sε
Continuous-Valued Activation Function
(introduced by Igor Aizenberg in 2004)
Continuous-valued case (k ∞):
Arg exp( (arg ))
(
)/ | |
i zP z ez
i zz
= = ==
Function P
maps the complex plane into the unit circle
Z
||/)( zzzP =
i
1
Learning Algorithm for the Continuous MVN with
the Error Correction Learning Rule
XW –
weighting vector; X
-
input vector
is a complex conjugated to Xαr
–
a learning rate (should be always equal to 1)r -
current iteration;
r+1 –
the next iterationZ
– the weighted sum
1 ( 1) | |qr r
rr rW z+ ε - XW
n + zα
+
⎛ ⎞= ⎜ ⎟
⎝ ⎠i
|| zzqε
|| zzq −ε
qε is a desired
output
is an actual
output| |zz | |
q zz
εδ = − -
neuron’s error
A role of the factor 1/(n+1) in the Learning Rule
|| zzq −= εδ -
neuron’s error
nnn xn
wwxn
wwn
ww)1(
~ ; ... ;)1(
~ ;)1(
~11100 +
δ+=
+δ
+=+δ
+=
. ...)1(
...)1()1(
))1(
(...))1(
())1(
(
~...~~~
110
110
11110
110
δδ
δδδ
δδδ
+=++++=
=+
++++
+++
+=
=+
++++
+++
+=
=+++=
zxwxwwn
xwn
xwn
w
xxn
wxxn
wn
w
xwxwwz
nn
nn
nn
nn
The weights after the correction:
The weighted sum after the correction:
Self-Adaptation of the Learning Rate
-is the absolute value of the weighted sum on theprevious (rth) iteration.
rz
is a self-adaptive part of the learning rate1
rz
i
|z|
<1|z|
>1
1
1 ( 1) | |r+qr r
rr r
zW W + ε - Xn+ z zα ⎛ ⎞
= ⎜ ⎟⎝ ⎠
1/|zr
|
is a self-adaptive part of the learning rate
Modified Learning Rules with the Self-Adaptive Learning Rate
1 ( )( 1) | |
q srr r
r
W W + ε - ε Xn+ z
α+ =
1/|zr
|
is a self-adaptive part of the learning rate
1 ( 1) | |qr
r rr
zW W + ε - Xn+ z zα
+
⎛ ⎞= ⎜ ⎟
⎝ ⎠
Discrete MVN
Continuous MVN
Convergence of the learning algorithm
It is proven that the MVN learning algorithm converges after not more than k! iterations for the k -valued activation functionFor the continuous MVN the learning algorithm converges with the precision λ after not more than (π/λ)! iterations because in this case it is reduced to learning in π/λ –valued logic.
MVN as a model of a biological neuron
No impulses Inhibition Zero frequency
Intermediate State Medium frequency
Excitation High frequency
The State of a biological neuron is determined by the frequency of the generated impulses
The amplitude of impulses is always a constant
MVN as a model of a biological neuron
The state of a biological neuron is determined by the frequency of the generated impulses
The amplitude of impulses is always a constant
The state of the multi-valued neuron is determined by the argument of the weighted sum
The amplitude of the state of the multi-valued neuron is always a constant
MVN as a model of a biological neuron
Maximal inhibition
Maximal excitation
2π0π
Intermediate State
Intermediate State
MVN as a model of a biological neuron
Maximal inhibition
Maximal excitation
2π0π
MVN:Learns fasterAdapts betterLearns even highly nonlinear functionsOpens new very promising opportunities for the network designIs much closer to the biological neuronAllows to use the Fourier Phase Spectrum as a source of the features for solving different recognition/classification problemsAllows to use hybrid (discrete/continuous) inputs/output
P-realizable Boolean Functions over
the field of Complex numbers and a Universal Binary Neuron
Boolean Alphabets
{0, 1} –
a classical alphabet
{1,-1} –
an alternative alphabet used here
11 ;1021}1,1{ };1,0{
−→→⇒−=−∈∈
yxxy
Threshold Boolean Functions (a classical definition)
The Boolean function is called a threshold
function, if such a real-valued weighting vector exists that for all from the domain of the function f:
),...,( 1 nxxf
),...,,( 10 nwww),...,( 1 nxxX =
⎩⎨⎧
−=<+++=≥+++
1),...,( if 0, ...1),...,( if ,0...
1110
1110
nnn
nnn
xxfxwxwwxxfxwxww
A threshold function is a linearly separable function
1
1
-1
-1
f(1,1)=
1
f(1,-1)= -1
f(-1,1)= -1
f(-1,-1)=
-1
f (x1
, x2
)
is the
OR function
Threshold Boolean Functions
The threshold function could be implemented on the single neuronThe number of threshold functions is very small in comparison to the number of all functions (104 of 256 for n=3, about 2000 of 65536 for n=4, etc.)Non-threshold functions can not be implemented on a single neuron (Minsky-Papert, 1969)
XOR – a classical non-threshold function
1
1
-1
-1
f(1,1)=1
f(1,-1)= -1
f(-1,1)= -1
f(-1,-1)=1
Is it possible to overcome
the Minsky’s-Papert’s
limitation for the classical perceptron?
Yes !!!
P-realizable Boolean functions (introduced by I. Aizenberg in 1984-1985)
))1(2()arg()2(if,)1()(
/mj+π zj/m =zP j
B
<≤−
π
i
0
1
m-1
m-2Z
1
-1
1-1
-1
1
1=)(zPB
Binary activation function:
P-realizable Boolean functions
The Boolean function is called a P-realizable function, if such a complex-valued weighting vector exists that for all from the domain of the function f:
),...,( 1 nxxf
),...,,( 10 nwww
),...,( 1 nxxX =
)...(),...,( 1101 nnBn xwxwwPxxf +++=
Universal Binary Neuron (UBN) (introduced in 1985)
)...(),...,( 1101 nnBn xwxwwPxxf +++=
1x
nx
),...,( 1 nxxf...
PB(z)
nn xwxwwz +++= ...110
UBN: main properties
The key properties of UBN: Complex-valued weightsThe activation function is a function of the argument of the weighted sumAbility to learn arbitrary (not necessarily threshold) Boolean functions on a single neuronSimplicity of learning
XOR problem
x1 x222110 xwxww
z++=
=)(zPB f(x1, x2)
1 1 1+i 1 1
1 -1 1-i -1 -1
-1 1 -1+i -1 -1
-1 -1 -1-i 1 1
1−=BP
1−=BP
n=2, m=4
–
four sectorsW=(0, 1, i) –
the weighting vector
1
i
-i-1
1=BP
1=BP
Parity problemn=3, m=6 : 6 sectors
W=(0, ε, 1, 1) –
the weighting vector
1
1-1
1
-1 -1
ε
1-1
1X 2X 3X3322
110
xwxwxwwZ
++++=
)(ZPB ),,( 321 xxxf
1 1 1 2+ε 1 1 1 1 -1 ε -1 -1 1 -1 1 ε -1 -1 1 -1 -1 2−ε 1 1 -1 1 1 2+ε− -1 -1 -1 1 -1 ε− 1 1 -1 -1 1 ε− 1 1 -1 -1 -1 2−ε− -1 -1
Dependence of the number of sectors on the number of inputs
A proven estimation for m, to ensure P-realizibility of all functionsEstimation for self-dual and even functions (experimental fact, mathematically open problem)P-realizability of all functions may be reduced to P-realizability of self-dual and even functions
n
mn 222 ≤≤
nmn 32 ≤≤
Relation between P-realizable Boolean functions and k-valued threshold functions
If the Boolean function f is P-realizable with a separation of the complex plane into m sectors then at least one m-valued threshold function fm (partially defined on the Boolean domain) exists, whose weighting vector is the same that for the Boolean function f.The learning of P-realizable function may be reduced to the learning of partially defined m-valued threshold function
Learning Algorithm for UBN
q
= s-1 (mod
m),
if
Z
is
closer
to
(s-1)st
sector
q = s+1 (mod
m), if
Z
is
closer
to
(s+1)st
sector
1( )
1
qr
r+ r
sε - ε+ X ,(n+
W)
W α=
s+1 i s s-1
Semi-supervised/semi-
self-adaptive learning
Main Applications
MVN- based Multilayer Feedforward Neural Network (MLMVN) and a Backpropagation Learning
Algorithm
Suggested and developed by Igor Aizenberg et al in 2004-2006
MVN-
based Multilayer Feedforward Neural Network
Hidden layers Output layer
1
2
N
MLMVN: Key PropertiesDerivative-free learning algorithm based on the error-correction ruleSelf-adaptation of the learning rate for all the neuronsMuch faster learning than the one for other neural networksA single step is always enough to adjust the weights for the given set of inputs independently on the number of hidden and output neuronsBetter recognition/prediction/classification rate in comparison with other neural networks, neuro-fuzzy networks and kernel based techniques including SVM
MLMVN: Key Properties
MLMVN can operate with both continuousand discrete inputs/outputs, as well as with the hybrid inputs/outputs: continuous inputs discrete outputsdiscrete inputs continuous outputshybrid inputs hybrid outputs
A Backpropagation Derivative- Free Learning Algorithm
kmT -
a desired output of the kth
neuron from the mth
(output) layer
kmY -
an actual output of the kth
neuron from the mth
(output) layer
kmkmkm YT −=δ* -
the network error for the kth
neuron from output layer
*1km
mkm s
δ=δ -
the error
for the kth
neuron from output layer
11 += −mm Ns -the number of all neurons on the previous layer (m-1, to which the error is backpropagated) incremented by 1
A Backpropagation Derivative- Free Learning Algorithm
∑∑++
=
−++
=
+++
δ=δ=δ11
1
111
1
1121
)(1)(11 jj N
i
ijkij
j
N
i
ijkijij
kjkj w
sw
ws
The error
for the kth
neuron from the hidden (jth) layer, j=1, …, m-1
1 ;,...,2 ,1 11 ==+= − smjNs jj
-the number of all neurons on the previous layer (previous to j, to which the error is backpropagated) incremented by 1
The error backpropagation:
A Backpropagation Derivative- Free Learning Algorithm
Correction rule for the neurons from the mth
(output) layer (kth
neuron of mth
layer):
kmkmkmkm
imkmkmkm
ikmi
nCww
niYnCww
δ+
+=
=δ+
+= −
)1(~
,...,1 ,~)1(
~
00
1
A Backpropagation Derivative- Free Learning Algorithm
Correction rule for the neurons from the 2nd
through m-1st
layer (kth
neuron of the jth
layer (j=2, …, m-1):
kjkj
kjkjkj
ikjkj
kjkji
kji
znC
ww
niYzn
Cww
δ+
+=
=δ+
+=
||)1(~
,...,1 ,~||)1(
~
00
A Backpropagation Derivative- Free Learning Algorithm
11
110
10
11
111
||)1(~
,...,1 ,||)1(
~
kk
kkk
ikk
kki
ki
znCww
nixzn
Cww
δ+
+=
=δ+
+=
Correction rule for the neurons from the 1st
hidden
layer:
Criteria for the convergence of the learning process
Learning should continue until either minimum MSE/RMSE
criterion will be satisfied or zero-error will be reached
MSE criterion
λ≤=δ ∑∑∑==
N
ss
N
s kskm E
NW
N 11
2* 1)()(1
λ is a maximum possible MSE for the training data
N is
the number of patterns in the training set
is the network square error for the sth
pattern
* 2( )s kmsk
E δ=∑
RMSE criterion
λ≤=δ ∑∑∑==
N
ss
N
s kskm E
NW
N 11
2* 1)()(1
λ is a maximum possible RMSE for the training data
N is
the number of patterns in the training set
is the network square error for the sth
pattern
* 2( )s kmsk
E δ=∑
MLMVN Learning: Example
1 1
2 2
3 3
(exp(4.23 ),exp(2.10 )) ,
(exp(5.34 ),exp(1.24 )) ,
(exp(2.10 ),exp(0 )) .
X i i T
X i i T
X i i T
= →
= →
= →
Suppose, we need to classify three vectors belonging to three different classes:
( ) ( ) ( )1 2 3exp 0.76 , exp 2.56 , exp 5.35 .T i T i T i= = =
1 2 3, ,T T T
[arg( ) 0.05,arg( ) 0.05], 1, 2,3,j jT T j− + =
Classes are determined in such a way that the argument of the desired output of the network must belong to the interval
MLMVN Learning: Example
Thus, we have to satisfy the following conditions:
( ) ( ) arg arg 0.05iArg zjT - e ≤
iArg ze
, where
is the actual output.
and for the mean square error
20.05 0.0025E ≤ =
.
MLMVN Learning: Example
Let us train the 2 1 MLMVN
(two hidden neurons and the single neuron in the output layer
x1
x2
( )21, xxf
11
12
21
MLMVN Learning: ExampleThe training process converges after 7 training epochs.
Update of the outputs:
MLMVN Learning: Example
E p o c h
1 2 3 4 5 6 7
M S E
2.4213 0.1208 0.2872 0.1486 0.0049 0.0026 0.0009
20.05 0.0025E ≤ =
MLMVN: Simulation Results
Simulation Results: Benchmarks
All simulation results for the benchmark
problems are obtained using the network with n
inputs
n S 1 containing a single hidden layer with S
neurons
and a single neuron in the output layer:
Hidden layer
Output layer
The Two Spirals Problem
Two Spirals Problem: complete training of 194 points
Structure of thenetwork A network type The results
2 40 1
MLMVN Trained completely, no errors
A traditional MLP, sigmoid activation function
still remain 4% errors
2 30 1
MLMVN Trained completely, no errors
A traditional MLP, sigmoid activation function
still remain 14% errors
Two Spirals Problem: cross-validation (training of 98 points
and prediction of the rest 96 points)
The prediction rate is stable for all the networks from 2 26 1 till 2 40 1:
68-72%
The prediction rate of 74-75%
appears 1-2 times per 100 experiments with the network 2 40 1
The best known result obtained using the Fuzzy Kernel Perceptron (Nov. 2002) is 74.5%,
But it is obtained using more complicated and larger network
The “Sonar” problem
208 samples:60 continuous-valued inputs1 binary output
The “sonar”
data is trained completely using the simplest possible MLMVN
60 2 1
(from 817 till 3700 epochs in 50 experiments)
The “Sonar”
problem: prediction
104 samples in the training set104 samples in the testing set
The prediction rate is stable for 100 experiments with the simplest possible MLMVN
60 2 1:
88-93%
The best result for the Fuzzy Kernel Perceptron (Nov., 2002) is 94%.The best result for SVM is 89.5%.However, both FKP and SVM used are larger and their trainingis more complicated task.
Mackey-Glass time series prediction
Mackey-Glass differential delay equation:
),()(1.0)(1
)(2.0)(10 tntx
txtx
dttdx
+−−+−
=ττ
The task of prediction is to predict )6( +tx
from )18( ),12( ),6( ),( −−− txtxtxtx
.
Mackey-Glass time series prediction
Training Data: Testing Data:
Mackey-Glass time series prediction
RMSE Training: RMSE Testing:
Mackey-Glass time series predictionTesting Results:
Blue curve
–
the actual series; Red curve
–
the predicted series
Mackey-Glass time series prediction
# of neurons on the hidden
layer 50 50 40
ε- a maximum possible RMSE 0.0035 0.0056 0.0056
Actual RMSE for the training set (min - max) 0.0032 - 0.0035 0.0053 – 0.0056 0.0053 – 0.0056
Min 0.0056 0.0083 0.0086 Max 0.0083 0.0101 0.0125
Median 0.0063 0.0089 0.0097 Average 0.0066 0.0089 0.0098
RMSE for the testing
set SD 0.0009 0.0005 0.0011 Min 95381 24754 34406 Max 272660 116690 137860
Median 145137 56295 62056
Number of training epochs Average 162180 58903 70051
The results of 30 independent runs:
Mackey-Glass time series prediction
Comparison of MVN to other models:
MLMVNmin
MLMVNaverage
GEFREXM. Rosso, Generic Fuzzy
Learning, 2000 min
EPNet You-Liu,
Evolution. system, 1997
ANFISJ.S. R. Jang,
Neuro-Fuzzy, 1993
CNNE M.M.Islam
et all., Neural
Networks Ensembles,
2003
SuPFuNISS.Paul et
all, Fuzzy-Neuro, 2002
Classical Backprop.
NN 1994
0.0056 0.0066 0.0061 0.02 0.0074 0.009 0.014 0.02
MLMVN outperforms all other networks in:
•The number of either hidden neurons or supporting vectors
•Speed of learning
•Prediction quality
Real World Problems:
Solving Using MLMVN
Classification of gene expression microarray data
“Lung”
and “Novartis”
data sets
4 classes in both data sets“Lung”: 197 samples with 419 genes“Novartis”: 103 samples with 697 genes
K-folding cross validation with K=5
“Lung”: 117-118 samples of 197 in each training set and 39-40 samples of 197 in each testing set“Novartis”: 80-84 samples of 103 in each training set and 19-23 samples of 103 in each testing set
MLMVN
Structure of the network: n 6 4 (n inputs, 6 hidden neurons and 4 output neurons)The learning process converges very quickly (500-1000 iterations that take up to 1 minute on a PC with Pentium IV 3.8 GHz CPU are required for both data sets)
Results: Classification Rates
MLMVN: 98.55% (“Novartis)MLMVN: 95.945% (“Lung”)kNN (k=1) 97.69% (“Novartis”)kNN (k=2) 92.55% (“Lung”)
Generation of the Genetic Code using MLMVN
Genetic code
There are exactly four different nucleic acids: Adenosine (A), Thymidine (T), Cytidine (C) and Guanosine (G)Thus, there exist 43=64 different combinations of them “by three”. Hence, the genetic code is the mappingbetween the four-letter alphabet of the nucleic acids (DNA) and the 20-letter alphabet of the amino acids (proteins)
Genetic code as a multiple- valued function
Let be the amino acidLet be the nucleic acidThen a discrete function of three variables
, 1, ..., 20jG j =
{ }, , , , 1, 2, 3ix A G C T i∈ =
1 2 3( , )jG f x , x x=
is a function of a 20-valued logic, which is partially defined on the set of four-valuedvariables
Genetic code as a multiple- valued function
A multiple-valued logic over the field of complex numbers is a very appropriate model to represent the genetic code functionThis model allows to represent mathematically those biological properties that are the most essential (e.g., complementary nucleic acids A T; G Cthat are stuck to each other in a DNA double helix only in this order
DNA double helix
The complementary nucleic acids
A T; G C
are always stuck to each other in pairs
Representation of the nucleic acids and amino acids
ND
T
F
SA
AY
G
WM
I
H
Q
T K
ERGC
LV
PC
The complementary nucleic acids
A T; G Care located such that
A=1; G=i
T
= -A= -1
and
C= -
G
= -i
All amino acids
are distributed along the unit circle in the logical way, to insure their closeness to those nucleic acids that form each certain amino acid
Generation of the genetic code
The genetic code can be generated using MLMVN 3 4 1 (3 inputs, 4 hidden neurons and a single output neuron – 5 neurons)There best known result for the classical backpropagation neural network is generation of the code using the network 3 12 2 20 (32 neurons)
Blurred Image Restoration (Deblurring) and Blur Identification by MLMVN
Problem statement: capturing
Mathematically a variety of capturing principles can be described by the Fredholm integral of the first kind
where x,t ℝ2, v(t) is a point-spread function (PSF) of a system, y(t) is a function of a real object and z(x) is an observed signal.
2
2( ) ( , ) ( ) , ,z x v x t y t dt x t= ∈∫
Mic
rosc
opy
Tomog
raphy
∈
Ph
oto
Image deblurring: problem statement
Mathematically blur is caused by the convolution of an image with the distorting kernel.Thus, removal of the blur is reduced to the deconvolution.Deconvolution is an ill-posed problem, which results in the instability of a solution. The best way to solve it is to use some regularization technique.To use any kind of regularization technique, it is absolutely necessary to know the distorting kernel corresponding to a particular blur: so it is necessary to identify the blur.
Image deblurring: problem statement
The observed image given in the following form:
where “*”
denotes a convolution operation, y
is an image, υ is point-spread function
of a system (PSF) , which is exactly
a distorting kernel, and ε is a noise.
In the continuous frequency domain
the model takes the form:
where is a representation of the signal z
in the Fourier domain
and denotes a Fourier transform.
( ) ( )( ) ( )z x v y x xε= ∗ + ,
( ) ( ) ( ) ( )Z V Yω ω ω ε ω= ⋅ + ,
2( ) { ( )}Z F z x Rω ω= , ∈ ,{}F ⋅
Blur Identification
We use MLMVN to recognize Gaussian, motion and rectangular (boxcar) blurs.We aim to identify simultaneously both blur (PSF), and its parameters using a single neural network.
Considered PSFPSF in time domain
PSF in frequency domain
Gaussian Motion Rectangular
Considered PSF
The Gaussian PSF:
is a parameter (variance)
The uniform linear motion:
h
is a parameter (the length of motion)
The uniform rectangular:
h
is a parameter (the size of smoothing area)
2 21 2
2 2
1( ) exp2
t tv t
πτ τ⎛ ⎞+
= −⎜ ⎟⎝ ⎠
2 21 2 1 2
1 , / 2, cos sin ,( )otherwise,0,
t t h t tv t h φ φ⎧⎪ + < == ⎨⎪⎩
1 22
1 , , , ( ) 2 2
otherwise,0,
h ht tv t h
⎧ < <⎪= ⎨⎪⎩
2τ
Degradation in the frequency domain:
True Image Gaussian Rectangular HorizontalMotion
VerticalMotion
Images and log of their Power Spectra log Z
Training Vectors
We state the problem as a recognition of the shape of V , which is a Fourier spectrum of PSF υ and its parameters from the Power-Spectral Density, whose distortions are typical for each type of blur.
Training Vectors
( )( ) ( )( ) ( )
1 2, min
max min
log logexp 2 ( 1) ,
log logk k
j
Z Zx i K
Z Z
ωπ
⎛ ⎞−⎜ ⎟= ⋅ −⎜ ⎟−⎜ ⎟⎝ ⎠
),...,( 1 nxxX =The training vectors are formed as follows:
for
(| ( ) |)Log Z ω
(0,0)ω
1 2 2
1 2
2 1
1,..., / 2 1, for , 1,..., / 2 1,/ 2,..., 2, for 1, 1,..., / 2 1,
1,...,3 / 2 3, for 1, 1,..., / 2 1,
j L k k k Lj L L k k Lj L L k k L
= − = = −⎧⎪ = − = = −⎨⎪ = − − = = −⎩
LxL is a size of an image
the length of the pattern vector is n= 3L/2-3
Examples of training vectors
True Image Gaussian Rectangular
HorizontalMotion
VerticalMotion
Neural Network
Hidden layers Output layer
1
2
N
Blur 1
Blur 2
Blur N
Training (pattern) vectors
Output Layer Neuron
Reservation of domains on the unit circle for the output neuron
Simulation
{ }1, 1.33, 1.66, 2, 2.33, 2.66, 3 ;τ ∈
Experiment 1 (2700 training pattern vectors corresponding to 72 images): six types of blur with the following parameters:MLMVN structure: 5 35 6
1) The Gaussian blur is considered with 2) The linear uniform horizontal motion blur of the lengths 3, 5, 7, 9; 3) The linear uniform vertical motion blur of the length 3, 5, 7, 9; 4) The linear uniform diagonal motion from South-West to North-
East blur of the lengths 3, 5, 7, 9; 5) The linear uniform diagonal motion from South-East to North-
West blur of the lengths 3, 5, 7, 9; 6) rectangular has sizes 3x3, 5x5, 7x7, 9x9.
ResultsClassification Results
Blur
MLMVN, 381 inputs,5 35 6,
2336 weights in total
SVMEnsemble from
27 binary decision SVMs,
25.717.500 support vectors in total
No blur 96.0% 100.0%Gaussian 99.0% 99.4%Rectangular 99.0% 96.4
Motion horizontal 98.5% 96.4Motion vertical 98.3% 96.4
Motion North-East Diagonal 97.9% 96.5Motion North-West Diagonal 97.2% 96.5
Restored images
Blurred noisy image:rectangular 9x9
Restored
Blurred noisy image:Gaussian, σ=2
Restored
Main Publications1. N.N. Aizenberg, Yu. L.Ivaskiv and D.A. Pospelov "About one Generalization of the
Threshold Function", Doklady Akademii Nauk SSSR (The Reports of the Academy of Sciences of USSR), vol. 196, No. 6, 1971, pp. 1287–1290. (in Russian).
2. N.N. Aizenberg, Yu.L. Ivaskiv, D.A. Pospelov and G.F. Hudiakov "Multi-Valued Threshold Functions. 1. The Boolean Complex-Threshold Functions and their Generalization", Kibernetika (Cybernetics), No. 4, 1971, pp. 44-51 (in Russian).
3. N.N. Aizenberg, Z. Toshich. A Generalization of the Threshold Functions, // Proceedings of the Electronic Department University of Belgrade, No.399, 1972, pp.97-99.
4. N.N. Aizenberg, Yu.L. Ivaskiv, D.A. Pospelov and G.F. Hudiakov "Multi-Valued Threshold Functions. 2. Synthesis of the Multi-Valued Threshold Element" Kibernetika(Cybernetics), No. 1, 1973, pp. 86-94 (in Russian).
5. N.N. Aizenberg and Yu.L. Ivaskiv Multiple-Valued Threshold Logic, Naukova Dumka Publisher House, Kiev, 1977 (in Russian).
6. I.N. Aizenberg "Model of the Element with a Complete Functionality", Izvestia AN SSSR. Technicheskaya ("Journal of Computer and System Sciences International "), 1985, No. 2., pp. 188-191 (in Russian).
7. I.N. Aizenberg "The Universal Logical Element over the Field of the Complex Numbers",Kibernetika (Cybernetics and Systems Analysis), No.3, 1991, pp.116-121 (in Russian).
Main Publications8. N.N. Aizenberg and I.N. Aizenberg "Model of the Neural Network Basic Elemets (Cells)
with Universal Functionality and various of Harware Implementation", Proc. of the 2-ndInternational Conference "Microelectronics for Neural Networks", Kyrill & MethodyVerlag, Munich, 1991, pp. 77-83.
9. N.N.Aizenberg and I.N.Aizenberg "CNN Based on Multi-Valued Neuron as a Model ofAssociative Memory for Gray-Scale Images", Proceedings of the Second IEEEInternational Workshop on Cellular Neural Networks and their Applications, TechnicalUniversity Munich, Germany October 14-16, 1992, pp.36-41.
10. N.N.Aizenberg, I.N. Aizenberg and G.A.Krivosheev “Method of Information Processing”Patent of Russia No 2103737, G06 K 9/00, G 06 F 17/10, priority date 23.12.1994.
11. N.N. Aizenberg, I.N. Aizenberg and G.A. Krivosheev "Multi-Valued Neurons: Learning,Networks, Application to Image Recognition and Extrapolation of Temporal Series",Lecture Notes in Computer Science, vol. 930 (J.Mira, F.Sandoval - Eds.), Springer-Verlag, 1995,pp.389-395.
12. I.N. Aizenberg. and N.N. Aizenberg “Universal binary and multi-valued neuronsparadigm: Conception, Learning, Applications”, Lecture Notes in Computer Sciense, vol.1240 (J.Mira,R.Moreno-Diaz, J.Cabestany - Eds.), Springer-Verlag, 1997, pp. 463-472.
13. I.N. Aizenberg “Processing of noisy and small-detailed gray-scale images using cellularneural networks”, Journal of Electronic Imaging, vol. 6, No. 3, July, 1997, pp. 272-285.
Main Publications
14. I. Aizenberg., N. Aizenberg, J. Hiltner, C. Moraga, and E. Meyer zu Bexten., “Cellular Neural Networks andComputational Intelligence in Medical Image Processing”, Journal of Image and Vision Computing, Vol. 19, March,2001, pp. 177-183.
15. I. Aizenberg., E. Myasnikova, M. Samsonova., and J. Reinitz , “Temporal Classification of DrosophilaSegmentation Gene Expression Patterns by the Multi-Valued Neural Recognition Method”, MathematicalBiosciences , Vol.176 (1), 2002, pp. 145-159.
16. I. Aizenberg and C. Butakoff, “Image Processing Using Cellular Neural Networks Based on Multi-Valued andUniversal Binary Neurons”, Journal of VLSI Signal Processing Systems for Signal, Image and Video Technology,Vol. 32, 2002, pp. 169-188.
A book
Igor Aizenberg, Naum Aizenberg and Joos
Vandewalle
“Multi-Valued and Universal Binary Neurons:
Theory, Learning and Applications”,
Kluwer, 2000
The latest publications1.
Aizenberg I., Paliy D., Zurada J., and Astola J., "Blur Identification by Multilayer Neural Network based on Multi-Valued Neurons", IEEE Transactions on Neural Networks, accepted, to appear: April, 2008.
2.
Aizenberg I., "Solving the XOR and Parity n
Problems Using a Single Universal Binary Neuron", Soft Computing, vol. 12, No 3, February, 2008, pp. 215-222.
3.
Aizenberg I. and Moraga C. "Multilayer Feedforward Neural Network based on Multi-Valued Neurons and a Backpropagation Learning Algorithm", Soft Computing, Vol. 11, No 2, February, 2007, pp. 169-183.
4.
Aizenberg I. and Moraga C., "The Genetic Code as a Function of Multiple- Valued Logic Over the Field of Complex Numbers and its Learning using
Multilayer Neural Network Based on Multi-Valued Neurons", Journal of Multiple-Valued Logic and Soft Computing, No 4-6, November 2007, pp. 605-
618.
5.
Aizenberg I. Advances in Neural Networks. Class notes
for the graduate course. University of Dortmund. 2005.
Some Prospective ApplicationsSolving different recognition and classification problems, especially those, where the formalization of the decision rule is a complicated problemClassification and prediction in biologyTime series predictionModeling of complex systems including hybrid systems that depend on many parametersNonlinear filtering (MLMVN can be used as a nonlinear filter)Etc. …
CONCLUSION
The best results are expected in the near
future…
If it be now, 'tis not to come; if it be not to come, it will be now;
if it be not now, yet it will come: the readiness is all…
Shakespeare