17
I IEEE TRANSACTIONS ON SYSTEMS, MAN, CYBERNTTIC b. VOL. 20, NO. 3, MAY/JUNF 1990 637 Higher-Ordered and Intraconnected Bidirectional Associative Memories Abstracf -Autocorrelation associative memories (autocorrelators) are feedback neural network architectures that store bipolar patterns. Hete- rocorrelation associative memories (heterocorrelators) are feedback neu- ral network architectures that store bipolar pattern pairs. The relation- ship between autocorrelators and heterocorrelators is examined. A review of the encoding algorithms, recall operations, stability proofs, and capacity arguments of first- and higher-order autocorrelators, com- monly referred to as Hopfield associative memories (HAM’S), and first- order heterocorrelators, commonly referred to as bidirectional associa- tive memories (BAM’s) is presented. Higher-ordered BAM’s and first- and higher-ordered intraconnected BAM’s are then introduced. The encoding, recall, and stability procedures of each are discussed. Finally, simulation results that compare the storage capacity and storage effi- ciency of three heterocorrelators, the heteroassociative HAM, the BAM, and the intraconnected BAM, are presented. 1. INTRODUCTION SSOCIATIVE memories encode and recall arbitrary A patterns or pattern pairs. If the associative memory encodes single patterns, then it is an autoassociative memory, and if it encodes pattern pairs, it is a heteroasso- ciative memory. One form of associative memory is the correlation associative memory, a memory that encodes patterns by correlating their values. Hence an autoasso- ciative correlation associative memory is called an auto- correlator, and a heteroassociative correlation associative memory called a heterocorrelator. In the first portion of this paper, first- and higher-order autocorrelators and first-order heterocorrelators are reviewed. In the remain- der of the paper, higher-order heterocorrelators are intro- duced, the heterocorrelator is extended by adding in- tralayer connections, the similarities between this new intraconnected heterocorrelator and autocorrelators are discussed, and comprehensive comparisons of the storage capacity and efficiency of three different heterocorrela- tion topologies (heteroassociative autocorrelator, simple heterocorrelator, and intraconnected heterocorrelator) are provided. 11. AUTOCORRELATION ASSOCIATIVE MEMORIES-HAMS Autocorrelation associative memories (autocorrelators) encode the patterns Z,, Z,,. f e, Z,, where z k E (- 1, + llq, by superimposing pattern outer products on a Manuscript received May 8, 1989; revised November 6, 1989. This work supported by General Dynamics Internal Research and Develop- ment under projects 88015131 and 89015131. The author is with the Electronics Division, General Dynamics, P.O. Box 85310, MZ 7202K, San Diego, CA 92138. IEEE Log Number 8933541. common matrix. Autocorrelators were introduced as a theoretical notion by Donald Hebb in 1943 [26] and first rigorously analyzed by Amari [ 11-[2]. Other researchers that studied the dynamics of autocorrelators include Little [18], Little and Shaw [19], and later, Hopfield [12]. Although Hopfield’s work is considered to be seminal, his primary technical contribution was an alternative stability procedure (using an Ising spin-glass analogy that is now commonly referred to as Lyapunov energy functions) for a constrained version of the autocorrelator that did not allow any recurrent connections (t,, = 0, Vi = 1,2; . ., q) and required asynchronous updates. Amari presented an identical model and several variants in 1972 and strictly analyzed the stability of these models using a method he pioneered, entitled statistical neurodynamics. Surpris- ingly, a description of functions very similar to the Lyapunov functions described by Hopfield can also be found in Amari’s treatment of autocorrelators using stochastic approximation [2]. Without a doubt, it was Hopfield’s promulgation of the autocorrelator’s capability to a large audience that created much of the early excite- ment in this area. Because of this immediate relationship, the autocorrelator is now most easily recognized by the title of Hopfield associative memory (HAM), a title that will be used interchangeably with autocorrelator through- out this paper. First-order autocorrelators form matrix entries by mul- tiplying a pattern’s element with every other pattern’s element (i.e., a one-to-one correlation). Each element of a pattern (vector) represents the activation value of a processing element (PE) of the single-layer autocorrela- tor’s topology (see Fig. 1) and the connections between these PE’s contain the correlations he., this is where all the information is stored). As shown in Fig. 2, the first- order connections strengths, tfl, represent the weight from the ith to the jth PE and are determined by the correla- tion of z, and z,. Restated, t,, = z,zl. These correlations are also commonly referred to as Hebbian learning. Second-order autocorrelators form matrix entries by multiplying two elements of a pattern with each of its other elements (i.e., a two-to-one correlation). Referring to Fig. 3, the second-order connection strengths, s,!~, are formed by correlating the ith and jth PE values with the kth PE value. Restated, s,,~ = z,zIzk. Higher-order corre- lations continue in the same fashion, forming third- order connections, rrlkl = z,zIzkzl and so on, until all possible combinations are exhausted by the scalar-valued (q - 11th-order correlation. Higher-order autocorrelators 0018-9472/90/0.500-0637$01.00 01990 IEEE

Higher-ordered and intraconnected bidirectional associative memories

  • Upload
    pk

  • View
    215

  • Download
    1

Embed Size (px)

Citation preview

Page 1: Higher-ordered and intraconnected bidirectional associative memories

I

IEEE TRANSACTIONS O N SYSTEMS, MAN, CYBERNTTIC b. VOL. 20, NO. 3, MAY/JUNF 1990 637

Higher-Ordered and Intraconnected Bidirectional Associative Memories

Abstracf -Autocorrelation associative memories (autocorrelators) are feedback neural network architectures that store bipolar patterns. Hete- rocorrelation associative memories (heterocorrelators) are feedback neu- ral network architectures that store bipolar pattern pairs. The relation- ship between autocorrelators and heterocorrelators is examined. A review of the encoding algorithms, recall operations, stability proofs, and capacity arguments of first- and higher-order autocorrelators, com- monly referred to as Hopfield associative memories (HAM’S), and first- order heterocorrelators, commonly referred to as bidirectional associa- tive memories (BAM’s) is presented. Higher-ordered BAM’s and first- and higher-ordered intraconnected BAM’s are then introduced. The encoding, recall, and stability procedures of each are discussed. Finally, simulation results that compare the storage capacity and storage effi- ciency of three heterocorrelators, the heteroassociative HAM, the BAM, and the intraconnected BAM, are presented.

1. INTRODUCTION

SSOCIATIVE memories encode and recall arbitrary A patterns or pattern pairs. If the associative memory encodes single patterns, then it is an autoassociative memory, and if it encodes pattern pairs, it is a heteroasso- ciative memory. One form of associative memory is the correlation associative memory, a memory that encodes patterns by correlating their values. Hence an autoasso- ciative correlation associative memory is called an auto- correlator, and a heteroassociative correlation associative memory called a heterocorrelator. In the first portion of this paper, first- and higher-order autocorrelators and first-order heterocorrelators are reviewed. In the remain- der of the paper, higher-order heterocorrelators are intro- duced, the heterocorrelator is extended by adding in- tralayer connections, the similarities between this new intraconnected heterocorrelator and autocorrelators are discussed, and comprehensive comparisons of the storage capacity and efficiency of three different heterocorrela- tion topologies (heteroassociative autocorrelator, simple heterocorrelator, and intraconnected heterocorrelator) are provided.

11. AUTOCORRELATION ASSOCIATIVE MEMORIES-HAMS

Autocorrelation associative memories (autocorrelators) encode the patterns Z , , Z,,. f e , Z,, where z k E

( - 1, + llq, by superimposing pattern outer products on a

Manuscript received May 8, 1989; revised November 6, 1989. This work supported by General Dynamics Internal Research and Develop- ment under projects 88015131 and 89015131.

The author is with the Electronics Division, General Dynamics, P.O. Box 85310, MZ 7202K, San Diego, CA 92138.

IEEE Log Number 8933541.

common matrix. Autocorrelators were introduced as a theoretical notion by Donald Hebb in 1943 [26] and first rigorously analyzed by Amari [ 11-[2]. Other researchers that studied the dynamics of autocorrelators include Little [18], Little and Shaw [19], and later, Hopfield [12]. Although Hopfield’s work is considered to be seminal, his primary technical contribution was an alternative stability procedure (using an Ising spin-glass analogy that is now commonly referred to as Lyapunov energy functions) for a constrained version of the autocorrelator that did not allow any recurrent connections ( t , , = 0, Vi = 1,2; . ., q ) and required asynchronous updates. Amari presented an identical model and several variants in 1972 and strictly analyzed the stability of these models using a method he pioneered, entitled statistical neurodynamics. Surpris- ingly, a description of functions very similar to the Lyapunov functions described by Hopfield can also be found in Amari’s treatment of autocorrelators using stochastic approximation [2]. Without a doubt, it was Hopfield’s promulgation of the autocorrelator’s capability to a large audience that created much of the early excite- ment in this area. Because of this immediate relationship, the autocorrelator is now most easily recognized by the title of Hopfield associative memory (HAM), a title that will be used interchangeably with autocorrelator through- out this paper.

First-order autocorrelators form matrix entries by mul- tiplying a pattern’s element with every other pattern’s element (i.e., a one-to-one correlation). Each element of a pattern (vector) represents the activation value of a processing element (PE) of the single-layer autocorrela- tor’s topology (see Fig. 1) and the connections between these PE’s contain the correlations h e . , this is where all the information is stored). As shown in Fig. 2, the first- order connections strengths, tfl, represent the weight from the ith to the j th PE and are determined by the correla- tion of z , and z,. Restated, t,, = z,zl. These correlations are also commonly referred to as Hebbian learning.

Second-order autocorrelators form matrix entries by multiplying two elements of a pattern with each of its other elements (i.e., a two-to-one correlation). Referring to Fig. 3, the second-order connection strengths, s , ! ~ , are formed by correlating the ith and j th PE values with the kth PE value. Restated, s , , ~ = z ,zIzk . Higher-order corre- lations continue in the same fashion, forming third- order connections, rrlkl = z , z I zkz l and so on, until all possible combinations are exhausted by the scalar-valued ( q - 11th-order correlation. Higher-order autocorrelators

0018-9472/90/0.500-0637$01.00 01990 IEEE

Page 2: Higher-ordered and intraconnected bidirectional associative memories

I

FZ

Fig. 1 . Autocorrelator topology viewed as one-layer neural network. Both first- and second-order connections are shown. First-order con- nections store correlation of activation values between one processing element (PE) and another (e.g., t [ k , t k / , tqkr tkq , t , , . tkk, and tqJ. Second-order connections store correlation of two PE activation val- ues and a third (e.g., s ~ ~ ~ ) .

Fig. 2. Connection topology of first-order connection is shown. Con- nection t,] is formed by correlating PE z, with the PE z I .

n %-o ’ijk

Fig. 3. Connection topology of second-order connection is shown. Connection s , , ~ is formed by correlating two PE’s, z , and z I , with PE zk .

have been described and analyzed by Chen et al. [61, Psaltis et al. [23], [24], Peretto and Niez [22], Giles and Maxwell [8], Giles et al. [9], and Baldi and Venkatesh [5].

A . First- Order Autocotrelators I ) Encoding: The first-order correlation associative

memory, i.e., the first-order HAM, stores the m bipolar row vectors Z , , Z , , . . . , Z , , where z h E { - 1, + llq, by summing together the m outer products with the equation

where T = [ t i j l is the q X q connection matrix. In point- wise notation the equation is

where Z , = ( z : , z ; , . * e , 2:).

2) Recall: The first-order autocorrelation recall equa- tion is a vector-matrix multiplication followed by a point- wise nonlinear threshold operation. The vector-matrix multiplication represents a “synchronous” weighted sum- ming operation for each processing element (PE) in this one-layer associative memory. The recall equation for

TIONS ON SYSTEMS, MAN, A N D (.YRF.RNETIC‘S, VOL. 20, NO. 3, M A Y / J U N E 1990

each F, PE is

z;eW=f CZ,t’,’Z/o’d , Vj=1 ,2 ; . . , q , (3) 1 where the two-parameter bipolar threshold function is

1 i f a > O r -1 i f a < O f ( a , P ) = P i f a = O . (4)

This recall operation is synchronously (all PE’s at once) repeated until all PE’s cease to change. When all PE’s cease to change, the associative memory is said to be stable. This point of stability is where the recalled pattern is found.

3) Stabiliry and Capacity: An important point that con- cerns all feedback autocorrelators is proving the recall process stabilizes (i.e., reaches an equilibrium point) for all possible inputs. There are two methods that are em- ployed to prove recall stability, the first is the less com- mon statistical approach and the second is the more common Lyapunov energy (or Ising spin-glass analogy) approach. The first stability method is also used to deter- mine the information capacity of the HAM and will be only briefly presented here. The second approach will be presented in greater detail and employed in the remain- der of the paper.

The statistical method of proving HAM recall stability uses the strong law of large numbers to show that the probability of perfect recall of all stored patterns is 1, provided the memory capacity does not exceed a logarith- mically decreasing value. Mathematically restated,

lim Pq( m) = 1 ( 5 ) 9 -=

provided the capacity (i.e., number of patterns stored in T ) does not exceed

(6) 4 m =

2logq +loglogq where log is the natural logarithm. The preceding stabil- ity/capacity work was performed by Amari and Maginu [3]. Similar work has also been done by McEliece et al. [21], and Baldi and Venkatesh [5]. The importance of this information-capacity equation is twofold. First, Hopfield’s [12] initial empirical capacity result that stated the num- ber of patterns stored is linearly increasing with q (specifically, m = 0.15q) is now rigorously defined and found to be logarithmically decreasing with q. Second, as the autocorrelator’s dimensionality increases, the capacity ratio (the ratio of the number of patterns stored with respect to the dimensionality), call it r (q ) , decreases to 0 as q becomes infinitely large. Restated mathematically,

1 lim r ( q ) = lim = o (7)

q + m q + m 210gq+l0glogq where r ( q ) = m / q .

The second method of proving HAM recall stability is the use of Lyapunov energy functions. Lyapunov energy functions, used extensively in control theory, are potential energy functions that describe the state of a dynamical

I 1

Page 3: Higher-ordered and intraconnected bidirectional associative memories

I

SIMPSON: HIGHER-ORDERED A N D INTRAC'ONNECTED BIDIRECTIONAL ASSOCIATIVE MEMORIES 639

system. If the time of change of the potential energy function (i.e., the kinetic energy of the system) is decreas- ing or zero, then the system is stable. Hopfield [12] is the first to have used the Lyapunov energy formalism to describe the recall stability of an autocorrelator. Later work by McEliece et al. [21] and Gindi et al. [lo] removed Hopfield's asynchronous updating (only a random subset of the PE's are processed during any given time step) and nonrecurrent connection constraints, making the HAM a more versatile autocorrelator. What follows is the "asynchronous" stability proof for the first-order HAM. For completeness, the "synchronous" proof is presented for the second-order HAM presented in section 11-B-3 so that the reader can examine the differences. It is impor- tant to note that asynchronous stability is not a limitation. Rather, it can allow a much wider range of implementa- tions to be considered, because timing constraints can be removed (for example V U 1 implementations). Con- versely, synchronous stability is very appealing when con- sidering massively parallel machine implementations.

The traditional first-order (asynchronous update) HAM potential energy function is

4 4

E = - Z T Z ~ = - Z , Z I t l l (8) , = I , = I

where Z = [ z , ] is the initial state vector, Z T = [ z,] is the vector transpose of Z , and T = [t,,] is the autocorrelator's connection matrix. Calculating the change in energy with respect to the change in unit time, AE = E""" - Eold, yields

AE = E"e" - Eold 4 4 4 4

= - c c Z , Z ] t , ] - c c Z i Z J t l ] - ( A z , ) 2 t , , (9) i # k ~ = l , = I ~ # k , = I ] = I

where A z l is the change in the ith element of 2. Noting that tkk is always positive (trivially true by construction) and that the memory matrix T is symmetric (i.e., ti, = t , , ) , (9) can be simplified, yielding

4 9 A E = - 2 A z , Z,t,, - ( A Z , ) ' t , k . (10)

r # k ] = I r = l

The individual effects A t , has on AE are now analyzed with respect to the term

4

A21 c ZJtJI. (11) ] = 1

Enumerating all possible effects A z , has on A E results in the following three cases.

Case 1: A z , = 0. Trivially, this means AE = 0. Case 2: Az, > 0. This means that ZY" is greater than

z;ld. The only situation that gives rise to this set of values is when 2:"" = 1 and zpId = - 1. Therefore A z , = 2:"" - zpld = 1 -(- 1) = 2. The only way that z:~" could have changed from a negative to a positive value is if the input sum, C,z,r,,, is greater than 0. Hence both the A z , and

the input sum agree in sign, and the result is a AE < 0. Case 3: A z , < 0. Analogous to case 2, A z , = z,"~" - z,"ld

= - 1 - 1 = - 2. The only way to change from 1 to - 1 is if the input sum is negative. Hence the two terms again agree in sign and AE < 0.

From case 2 and 3 it is observed that as long as there is any nonzero change in z, , the energy will decrease. More- over, when all the changes in z , are zero, the energy is zero and the system is stable.

B. Higher-Order Autocomelators

First-order correlations linearly separate a problem space with a hyperplane. The linear separation quality of first-order correlations creates a restriction in that many problems are highly nonlinear in nature. Higher-order correlations circumvent this restriction and allow nonlin- ear relationships to be captured and stored [8], 191. An- other quality of higher-order correlations is their ability to increase the capacity with respect to the dimensionality of an autocorrelator at the expense of more connections. In light of the low capacity provided by first-order connec- tions, this is a very appealing attribute.

I ) Encoding: The second- and third-order correlation encoding equations are now presented. Extending beyond the third order, up to the ( q - 1)st-order, is direct. The second-order autocorrelation encoding equation (in pointwise notation) is

m

S i l k = z,hz:zt, V i , ; & k = 1 , 2 , ' . ' , q (12) h = l

where S = [ s i l k ] is the q X q X q matrix of second-order connections. Notice that a matrix that contains three axes and contains q 3 - q 2 more connections is created. Con- tinuing, the third-order autocorrelation encoding equa- tion is

m

r,,,, = z,hz:zLz;, V i , ; , k&l = 1,2; . . , q (13) h = l

where R = [r,,,,] is the q x q X q X q matrix of third-order connections.

As an interesting side note, it has been shown by Psaltis et al. [24] that any unique set of bipolar vectors expanded to its highest order and placed into a single vector creates an orthogonal set of vectors. As an example, given a three-dimensional vector Z = (zl, z 2 , z3) , the single full- vector higher-order expansion would yield Z' =

( z , , z 2 , z3 , z , z 2 , zIz3, z 2 z 3 , z ,z2z3) . This full expansion provides the capability of a Boolean look-up table.

2) Recall: Higher-order HAM recall equations are simi- lar to the first-order recall equations (3) in a direct sense, and are additive in a general sense. The "direct" use of second-order correlations results in the recall equation

zy"=f( ~ z , z J s l l k , z ~ l d , V k = 1 , 2 ; . . , q (14)

where the two-parameter nonlinear threshold function is 1 4 4

1 = 1 ] = I

Page 4: Higher-ordered and intraconnected bidirectional associative memories

640 IFIT TRANSACTIONS ON SYSTEMS, MAN, ANI) (.YUFRNFTI<‘S, VOI.. 20. NO. 3 , MAY/JIJNF 1990

described by (4). The “general” second-order recall equa- tion combines the first- and second-order activation val- ues as

1 4 4

Zknew = f ( z j t ] k + Z i Z ~ S ~ l k 3 zk“’d 9

1 = 1 1 = 1

Vk=1 ,2 ; . . , q . (15)

Extending the recall equations to include higher-order correlations, in either the direct or general sense, is immediate. It should be noted that all recall equations are assumed to have an inherent zero threshold, and hence thresholds are not included in our recall or stability equations.

3) Stability: A general higher-order energy equation has been introduced by Chen et al. [6] that is used to describe the stability of higher-order autocorrelators. The potential energy of the gth-order HAM is

m ...

Eg=- ( Z . Z h ) g c l h = l

where Z is the initial pattern and Z, is the hth pattern stored in the gth-order memory and the ‘‘.” is the dot product operator. Using this equation, Chen has shown that all odd gth-order autocorrelators are stable using synchronous updating, and all higher-order autocorrela- tors are stable with asynchronous updating.

Examining the general second-order recall equation (19, stability is shown using Chen’s formalism, resulting in the potential energy equation

E = E , + E , m m

= - (Z.Z,)’- (Z.Z,)’ h = l h = l

4 4 4 4 4

= - Z j Z k t ] k - Z i Z I Z k S , j k (17) j = 1 k = l i = 1 J = I k = l

where E , and E , are the first- and second-order correla- tion contributions to the potential energy, respectively. Calculating the kinetic energy under the synchronous updating condition yields

AE = E”“” - E o l d

4 4 4

= - A z k z ] t l k - A z k Z i Z j S i l k k = l 1 = 1 k = l j = k

1 4 4 4

= - Azk [ z l t jk + Z i z l S i l k . ( I8 ) k = l J = 1 I - 1 J = 1

Employing the case-by-case analysis performed for the first-order HAM stability proof, we find AE < 0 when at least one Azk is nonzero and the system is stable when all

4) Capacity: There are several researchers that have described the storage capacity of higher-ordered HAM’S. The equation that we will use is described by Baldi and

A Z k = 0.

FY

WT

J. Fig. 4. First-order heterocorrelator (BAM) topology viewed as two-

layer neural network. First-order connections store correlation of activation values between PE in one layer and PE in other layer (e.g., w , ~ , wtpr and w,,). Flow of activation in this topology is interfield-syn- chronous, meaning all PE’s in one layer feed all their activations to PE’s in other layer simultaneously. This interlayer feedback always results in immediate stable reverberation.

Venkatesh [51 as

m=- 2q! log q

where q is the dimensionality of the stored patterns, g is the order of the HAM, and m is the number of stored patterns. Again, as was the case with the first-order capacity, the capacity ratio logarithmically decreases as the dimensionality increases.

111. HETEROCORRELATION ASSOCIATIVE MEMORIES -BAM’s

Autocorrelators are excellent for applications that re- quire pattern completion and nearest-neighbor searches for single patterns, but often what is needed is a pattern- matching ability. The extension of autocorrelators to het- erocorrelators, i.e., the encoding of pattern pairs and not just single patterns, is the focus of the next two sections. The key point that is made with heterocorrelators is that feedback between the layers stabilizes and at the point of stability is a stored pattern pair. This operation was introduced by Dunning, et al. [7] for use in an optical resonating associative memory. Later, Kosko [ 171 refined first-order heterocorrelators and gave them the title of bidirectional associative memories (BAM’s). In the follow- ing, Kosko’s first-order heterocorrelator is described and then Kosko’s first-order heterocorrelator is extended to include higher-order correlations. Kosko sketched out a similar higher-ordered scheme for continuous and adap- tive BAM’s [15]-[16], which is formalized here for dis- crete-time bipolar heterocorrelators.

A. First- Order Heterocon-elators

Heterocorrelators store the m pattern pairs (XI, Y,), (X2,Y,);~~,(X,,Ym), where X h ~ { - l , + l ) “ and YhE { - 1, + I}”. The first-order BAM topology (see Fig. 4) encodes the interlayer (between layer) connections from the F, layer to the F , layer in the matrix W. By requiring that the connection strength from the ith F, to the j th F , PE, wi,, be symmetric (meaning wil = w,;), the trans- pose of W automatically contains the connections from

1 - -1

Page 5: Higher-ordered and intraconnected bidirectional associative memories

I

SIMPSON: H I G H E R - O R D E R E D ANI) INTRACONNEC'TED BIDIRECTIONAL ASSOCIATIVE MEMORIES 64 1

F , to F,, an attribute that significantly reduces storage requirements.

1) Encoding: The first-order BAM encoding equation in vector notation is

where the n x p matrix W = [wij] contains the associa- tions for the m pattern pairs. In pointwise notation, the equation is

m

wij= x"y: V i = 1 , 2 ; . . , n , h = l

and

Vj = 1,2; . ' , p

where xh = ( x : , x;; . ., x,") and Yh = (yf , y;; . ., y,"). The preceding construction procedure is identical to

that employed by Anderson's [4] linear associative mem- ory (LAM) and Kohonen's [13] correlation matrix memory (CMM). Although the encoding equations are the same, the LAM and CMM encode real-valued pattern pairs and they employ a purely feedforward recall procedure. Con- versely, the BAM encodes bipolar pattern pairs and relies upon the stabilizing feedback between the F, and F , layers during recall.

As a historical note, Wee [28] introduced another type of feedforward associative memory based upon the gener- alized inverse of the input pattern matrix. Later, Kohonen and Ruohonen [ 141 independently rediscovered the use of the generalized inverse operation and called the resultant memory matrix the optimal linear associative memory (OLAM). Each of these introductions shows that the generalized inverse (pseudo-inverse) of the matrix formed by the collection of input patterns forms the optimal (in a least squares sense) mapping between the input/output data pairs. Additionally, Stone [27] has shown that the Widrow-Hoff LMS algorithm [291 iteratively computes the OLAM.

2) Recall: First-order BAM recall employs interlayer feedback (feedback between layers) in a fashion similar to the intralayer feedback (feedback within layers) of the previously presented autocorrelators (3-4). The recall process originally described by Dunning er al. [71 is as follows: an initial pattern is presented to F,,F, activa- tions are passed through W and create F , activations; the F , activations are thresholded and passed back through W', which creates a possibly new set of F, activations; the F, activations are thresholded and passed back through W, and so on until the recall process stabilizes (i.e., both the F, and F , thresholded activations cease to change). Fig. 5 comparatively summarizes the feedback recall procedures of the feedback autocorrelator and het- erocorrelator.

Autoassociative Feedhack Recall

;ivcn: Initial vector z E 1 - 1 .+I ) q &I: Pass Z through T and thresh- Id to produce a possibly new set of PE alues Z"). Pass Z"' through T and ireshold to produce a possibly new se f PE values Z"'. Continue in this drhion until a11 PE values remain the ame. Symbolically this is represented iy the steps:

Z + F(ZT) + Z"'

zi21 e F(Z( lkT , z l l l 7 3 c z"' + F(Z'?'T) ~ z"'

c z'n + F ( Z k + z'n < ZCn e F(Z'r!TT) e Z'"

4eteroassociative Feedhack Recall

;iven: niti in^ vector x E I - I . + I j N &I: Pass X through W and thresh- Id to produce a new set of PE alues Y"'. Pass Yi') through WT and ireshold to produce a possibly new set f PE values X"'. Continue in t h i h ashion unt i l all PE values remilin the ame Symbolically this is represented y the steps:

X + F(XW) --f Yo '

cX'l' e F(Y"'WT) t Y"'

X") + F(X"'W) + YI2'

C X'" t FiY'2'WT) e YI2'

3 2

-3 X'" + F,X& + Y'" < X'" e F(Y'"WT) t Ycn

Fig. 5. Feedback recall operations for autocorrelator and heterocorre- lator are comparatively summarized. Function F ( X ) represents a threshold operation on each of the components of vector X.

The first-order BAM F, layer recall equation, F ( Y W ' ) , is

xyew=f C y - w ,x;ld , ~ i = 1 , 2 , . . . , n (22)

where xi is the activation value of the ith F, PE. The F , layer recall equation, F ( X W ) , is

yYew=f( ~ ~ x l w , j 7 y , ' " d , V j = 1 , 2 ; . . , p (23)

where yj is the activation value of the j th F , PE and the threshold function, f , is described by (4).

3) Stability: First-order BAM stability is proven using the pctential energy function

( , r l I' )

i 1 = 1

E = - ( E , + E,) = -(XWY'+YW'X')

= - C C xiyjwij - C C Yjxiwji n P P n

(24) 1-1 j = 1 j - 1 i - 1

where E, and E , are the potential energy contributions of the F, and F , layers, respectively. By symmetry, (24) can be re-expressed as

n P

E = - 2 x,y1wil. (25) [ = I j - 1

The kinetic energy is first examined with respect to the change in the F , PE activations and is mathematically expressed as

AE Y - - E Y ~ - E ; ~ = - XW(AY)' (26) where AY = (Ayl, Ay,; . .,Ay,,). In pointwise notation, (26) is expressed as

P n

AEy= - 2 Ayj xiwi,. (27) j = 1 i = 1

1

Page 6: Higher-ordered and intraconnected bidirectional associative memories

~

642

I

I K E E TRANSACTIONS ON SYSTEM^, M A N . A N I ) C.YHFKNFTIC'S, VOL. 20, NO. 3, M A Y / J U N F 1990

Examining the effects of a single PE change, Ay], on A E , leads to the following three cases.

Case 1: Ay, = 0. Trivially, this means A E = 0. Case 2: Ay, > 0. This means that the y,l"" is greater

than the y,OId. The only time this can occur is when y,!'"" = 1 and y,"Id = - 1, therefore Ay, = y,"'" - ,,"Id = 1 - (- 1) = 2. The only way that y/new could have changed from a negative to a positive value is if the input sum, C I x l w l I , is greater than 0. Hence both the AyJ and the input sum agree in sign, and the result is a A E , < 0.

Case 3: Ay, < 0. Analogous to case 2, AyJ = Y,"~"- yPld = - 1 - 1 = - 2. The only way to change from 1 to - 1 is if the input sum is negative. Hence the two terms again agree in sign and A E , < 0.

From cases 2 and 3 it is observed that any nonzero change in y, causes the energy to decrease, and when all the changes in y, are zero, the kinetic energy is zero and the F , portion of the system is stable. Similar analysis of A E , with respect to Axl yields the same result. Hence the BAM is stable. Also, because the energy changes are nontrivial (they are integer valued), the stabilization is immediate.

4) Capacity: Haines and Hecht-Neilsen [Ill, using the results of McEliece, et al. [211, estimate the capacity of the BAM at

r 210g r

m=-

for perfect recall of all stored pattern pairs, where r =

min ( n , p ) . Unfortunately, this theoretical result lacks em- pirical support.

B. Higher-Order Heterocon-elators

Higher-order heterocorrelators are now introduced. The extension to higher-order correlations will increase the storage capacity of the BAM at the expense of more connections. The majority of the explanation will focus on the second-order BAM. Like the higher-order autocorre- lators, extensions to higher-order BAMs is immediate.

1) Encoding: The second-order BAM encodes pattern pairs in two separate matrices. The first matrix, U , is an n X n x p lattice that holds the correlations of F, PE pairs with an F , PE. The second matrix, V , is a p X p X n lattice that holds the correlations of F , PE pairs with an F, PE. Fig. 6 illustrates the two connection topologies formed by the matrices U and V. The bottom-up ( F , to F,) second-order correlation matrix, U = [ U i j k ] , is formed by the equation

and V k = 1,2;

and the top-down ( F , to F,) second-order

' > P (29)

correlation

Fig. 6 . Second-order heterocorrelator (BAM) topologies viewed as two-layer neural networks. (a) Shows second-order connection U,,, that store correlations of F, PE pairs ( x , and x , ) with F , PE ( y ) (b) Shows second-order connection v l p r that store correlations of PE pairs ( y , and yp) with FT PE ( x , ) . Like first-order BAM, flow of activation in this topology is interfield-synchronous and always results in immediate stable reverberation.

matrix, V = [ vk l l ] , is formed by the equation in

V k / , = Y C Y f X : , V k & l = 1,2, ' * ' , p , h = l

and V j = 1,2; . . , n . (30) Continuing, the third-order correlations are also captured in two matrices. The first matrix, D, correlates F, PE triples with an F , PE, and the second matrix, E , corre- lates F , PE triples with an F, PE. Note that the correla- tions of F, PE pairs with F , PE pairs are omitted because it does not allow for pure field to P E interations. How this affects higher-order recall is discussed in a later section. The two encoding equations for D = [ d l l k l ] , and E = [e,,,,I are

d l lk l =

m

x,hx:x;yf, V i , j&k = 1,2; . . , n , h = l

and V l = 1,2;. . , p (31) and

m

I

and V i = 1,2; . . , n (32) respectively.

2) Recall: Higher-order BAM recall works in the same fashion as the first-order BAM. Feedback between the layers ensues upon initial pattern presentation and even- tually stabilizes at an equilibrium point. The "direct" second-order BAM F, recall equation is

I P P \

Page 7: Higher-ordered and intraconnected bidirectional associative memories

SIMPSON: HIGHER-ORDERED A N D INTRAC'ONNEC'TED HIDIRECTIONAL ASSOCIATIVE MEMORIES

Pascal Tree

1 I 1

1 2 1 1 3 3 1

1 4 6 4 1 1 5 10 10 5 I

e e e

643

Binomial Expansion

n + P n2 + 2np + P'

n3 + 3nZp + 3np2 + p3 n4 + 4n3p + 6n2p2 + 4np3 + p4

n'+5n4p+10n3p'+1~'p3+5np4+p' e e 0

and the corresponding F , recall equation is

1 n n

y/yW = f( X ; X ; U ; ; k , y y ,v k = 1,2; . . , p (34) ; = I j = 1

where f is our familiar two parameter threshold function (4). Including the first-order correlations provides the "general" set of recall equations

1

1

P P P

' Y W = f ( YkYI 'k l j + Y k W k j , X P l d 7

k = l I = 1 k = l

v j = 1,2; . . , n (35 ) and

n n

y; '"=.f( x j x ; u j j k + 2 X j W j k , y k O l d 7

; = I j = 1 j = 1

V k = 1,2;. ' , p (36) for the F, and F , PE's, respectively. Third-order corre- lations would replace the second-order terms of (33) and (34) in a direct sense, and they would be added to (35) and (36) in a general sense, and so on, for successively higher-order correlations.

A convenient representation of how the higher-order connection topologies look is found in the Pascal tree shown in Fig.7. The Pascal tree is a representation of the coefficients of the binomial expansion of successively higher orders. Table 1 shows how the Pascal tree is used to compare the connectivity of the BAM with that of the HAM for increasingly higher dimensions. This expansion illustrates two important points concerning higher-order BAM correlations: 1) there are significantly fewer connec- tions than equivalently dimensioned higher-ordered HAM, and 2) the higher the order, the more sparse the BAM interactions are because of the elimination of all but the many-to-one correlations. The trade-off between the more efficient use of connections and the sparser representation is empirically characterized in the last sect ion.

3) Stability: Higher-order BAM stability is easily gener- alized in the spirit of Chen et al. 161. Simpson [25] has shown that the first-order BAM potential energy equation is expressible as a sum of dot products as seen by the equation

E = - M T - J" TYT m

= - 2 (X.Xh)(Y.Yh). (37) h = l

Extending (37) for the gth-order BAM yields the poten- tial energy equation

E g = - ( E g Y + E g x )

m m

= - (X.Xh)g(Y*Yh)- (X.Xh)(Y*Yh)g (38)

where the first term, E,,, is the gth-order potential energy contribution of the F , PE's, and the second term, E g x , is the gth-order contribution of the F, PE's. Using

h = 1 h = l

(381, the potential energy of the second-order BAM can be expressed as

E , = -(E,, + E,,) m m

= - (X'X/,)'(Y'&)- (x'~h)(Y*yh)2. (39) h = l h = l

Analyzing the kinetic energy of the F, contribution to E, yields

m

AE,, = - (AX.X,)(Y.Y,)' h = l

n P P

= - " 1 Y j y k ' j k i . (40) i = l j = l k = l

Performing the familiar three-case analysis shows that AE, , is decreasing for nonzero A x ; , and zero otherwise. Similarly, the kinetic energy contribution of the F , PE's is described by the equation

m

AE,, = - (X.Xh)'( AY.Yh) h = l

P n n

= - ' Y k X i X j U i j k (41) k = l i = l j = 1

which is also decreasing for nonzero A y k , and zero other- wise. Combining the two terms yields

A E , = AE,, + AE,, < 0. (42) Hence the second-order BAM is stable. Moreover, the changes in energy are nontrivial integer values, therefore stabilization is immediate.

IV. AUTOCORRELATION AND

HETEROCORRELATION -1BAM's

One unfortunate side-effect of the BAM encoding pro- cedure is that complement patterns are encoded in a BAM by default. As an example, assume the pattern pair (Xl,Yl), X , = (1 - 1 1) and Y, = (1 - 1 - 1 - 1) is en- coded in a 3 x 4 first-order BAM matrix. By default the pattern pairs ( X ; , Yt), X ; = ( - 1 1 -1) and Y,C = ( - 1 1 1 1) are also encoded. If a new pattern pair (X2,Y2), X, = (- 1 1 - 1) and Y, = (1 1 1 1) need to be

1 -

Page 8: Higher-ordered and intraconnected bidirectional associative memories

I

644 IFFE TRANSACTIONS ON SYSTF.MS, M A N , ANI) C.YRERNETI('S, VOI.. 20, NO. 3. M A Y / J U N E 1990

TABLE I" DIFFERENCE BETWEEN NUMBER OF CONNECTIONS REQUIRED FOR DIMENSIONAL HAM

AND EQUIVALENT n X p BAM, WHERE q = n + p

HAM BAM Connections Order Connections

0 n + P n + P

2 n 7 + 3 n 2 p + 3 n p 2 + p 3 n 2 p + np2 3 n4 + 4 n 3 p + 6 n 2 p 2 + 4 n p 7 + p 4 n3p + np' 4 n4p + np4

1 n2 + 2np + p 2 nP

n5 + 5 n 4 p + 10n3p2 + 10n2p' +5np4 + p s

"This table illustrates how the BAM uses the minimum number of connections to implement a heterocorrelator. To use the HAM as a heterocorrelator would require many more connections, especially for higher-order correlations.

encoded, it would be impossible with the current encod- X Y ing methodology because Xf = X,. It was this problem that prompted the investigation of extending the current encoding scheme. To eliminate this problem, each pattern

pattern. To illustrate how this can be done, an autocorre- lator will be constructed from a heterocorrelator by con- catenating the F, and F, fields into a single F, layer and creating a autocorrelation matrix from the heterocor- relation matrix. The two-layer to one-layer concatenation is performed by making the identifications

X 1

pair encoded in the BAM must be treated as a single - [;;;;I

where q = n + p. The heterocorrelation-to-autocorrela- tion matrix translation is performed as shown in Fig. 8, resulting in the matrix T with n-by-n and p-by-p zero matrices along the diagonal. With this representation it is easier to see how to remove the complement encoding problem-add intraconnections. The heterocorrelation associative memory formed from patterns created by (43) is called the heterocorrelation HAM (HHAM). This con- struct's performance will be compared with that of the BAM and the forthcoming IBAM.

A. First-Order IBAM's

The addition of intralayer connections, connections be- tween the F, and F , PE's, results in improved storage capacity and removal of the complement restriction. The resultant network (shown in Fig. 9) is the intraconnected bidirectional associative memory (IBAM), an extension of the discrete BAM that represents the natural synergism between autocorrelators (intraconnections) and hetero- correlators (interconnections). The intraconnections have the effect of contrast enhancing a layer of PE activation values between the interlayer processing stages, an action that leads to removal of the complement encoding prob- lem as well as greater storage capacity. The following sections describe the encoding, recall, and stability of first-order IBAM's.

Z - zl. . . zi . . . zn . . .Zn+j. . . zq

Fig. 8. Translation of heterocorrelation matrix W into autocorrelation matrix T is shown. Concatenation of vectors X and Y results in q-dimensional vector Z. Heterocorrelation matrix W and its transpose, W', form diagonal blocks, and suitably dimensioned zero matrices are placed along diagonal. Notice that block diagonals represent missing intralayer connections from F, to F, PE's and F , to F , PE's.

1) Encoding: The first-order IBAM stores m pattern pairs ( X , , Y , 1, ( X 2 , Y2), , . , , ( X m , Ym), where

X , E ( -1, +I}", Y,E( -1, +l}P,

X , = ( x : , x i ; . . , x : ) , and Y, = ( y: , y ; ; . . , yp"). The interlayer connections are encoded in the n X p heterocorrelation matrix C = [c , , ] with the equation

m

e = X,'Y,. (44) h = 1

In pointwise notation the equation is m

c , , = x:y," V i = 1 , 2 ; . . , n , & V j = 1 , 2 ; * . , p . h = l

(45)

I

Page 9: Higher-ordered and intraconnected bidirectional associative memories

SIMPSON: HIGHER-ORDERED A N D INTRACONNECTED RlDlRECTlONAl ASSOCIATIVE MEMORIES 645

B Autoassociative and Heteroassociative Feedback Recall

b l l bii b""

4 * A

Fig. 9. First-order intraconnected bidirectional associative memory (IBAM) topology viewed as two-layer neural network. First-order interconnections are stored in matrix C; F, intraconnections are stored in matrix A ; and F , intraconnections are stored in matrix B. Flow of activation in this topology is intrafield-synchronous followed by interfield-synchronous, meaning all PE's in one layer feed all their activations to themselves simultaneously, and then feed all their activations to opposite layer simultaneously. This intralayer feedback followed by interlayer feedback always results in immediate stable reverberation.

The F, intraconnections are encoded in the autocorre- lation matrix A with the equation

m

A = x:xh (46) h = l

where A = [aik] is the n X n autocorrelation connection matrix. In pointwise notation the equation is

m

a i k = c xihx;, V i & k = 1 , 2 ; . . , n . (47)

Similarly, the F , intraconnections are encoded in the h = l

autocorrelation matrix B with the equation m

B = Y,'Yh (48) h = l

where B = [b jk] is the p X p autocorrelation connection matrix. In pointwise notation the equation is

m

bjk = y:y,", V j&k = 1,2; * * , p . (49) h = l

Referring to the heterocorrelation to autocorrelation translation shown in Fig. 8, the zero matrices along the diagonal can be replaced with the intraconnection matri- ces A and B. The resultant autocorrelation translation matrix is shown in Fig. 11.

2) Recall: IBAM recall relies upon both interlayer feedback and intralayer feedback. Like the BAM, an initial input pattern is presented to one of the layers, say

m: I n i t i a l v e c t o r ~ t ( - I . + I I ~ M I : Pass X through C and threshold to produce a new set of PE values Y"'. I'ass Y"' throufh B and threshold to produce a possibly new set of PE v111ue\ Y"'. Pn\s Y"' tliroufh CT and threshold to produce a possibly new sct of 1'1: values X"' Pa\\ X"' through A and threshold to produce a possi- hly iicu set of 1'1: value\ X"' Pa\\ X"' through C and threshold to pro- duce ii new \e! of PE viilues Y"', and so on Continue in this fashion until a11 PI; viilues remain the same. Symbolically [ h i \ is represented by the steps:

3 X + FIXC) i Y"' + FIY"'B1 + Y"'

F(X"1A) x " ' + F(Y'2'CT) y"' c x , 2 , + F,x,+) y ' T 1 + FIY"'B) --t Y ( ~ ) 2 c P e F I X ' I ' A I t x'" t F(Y%T') t Y'4'

XIn 4 FlX'"C1 + Y'g' + F(Y'"B1 + Y 1 0 3

x l r l r , x t . ! F A ) + X ~ F 1 + f:(ylrlc~, + yin

Fig. 10. Feedback recall operations for auto- and heteroassociative recall operation are summarized. Function F ( X ) represents threshold operation on each of components of vector X .

FX FY X

Y J Fig. 11. Convenient pictorial representation of information flow

through IBAM is shown. Initial input, X , is first processed through interconnection matrix C; resulting values are passed through matrix B , then C', then A ; and then cycle repeats. When PE activation values are stable, output pattern is read from F , after C interconnec- tion matrix completes its processing. These processing steps are reminiscent of current optical implementations of feedback associa- tive memories.

F,, and eventually the feedback stabilizes, resulting in the recall of the stored F , pattern. The difference between BAM and IBAM recall is that there are intralayer feed- back operations placed between the interlayer feedback steps. The recall is as follows: an initial pattern is pre- sented to F,; F, activations are passed through the interconnection matrix C and create F , activations; the F , activations are thresholded and passed through the F , intraconnection matrix B to create a possibly new set of

--- 1

Page 10: Higher-ordered and intraconnected bidirectional associative memories

I

646 IEEF. TRANSACTIONS ON SYSTEMS. M A N , A N D CYBERNETICS, VOL. 20, NO. 3 , M A Y / J U N E 1990

F , activations; the F , activations are thresholded and passed back through the interconnection matrix C', which creates a possibly new set of F, activations; the F, activations are thresholded and passed through the F, intraconnection matrix A , and they create a possibly new set of F, activations; the F, activations are thresholded and passed back through the interconnection matrix C; and so on, until the recall process reaches equilibrium (i.e., both the F, and F, thresholded activations cease to change for their respective memories). This recall process is summarized in Fig. 10. The F, PE interconnection recall equation, F ( X C ) , is

, V j = 1 , 2 ; - - , p . ( S O )

where 0 is a suitably dimensioned zero vector represent- ing the fact that during this stage of the processing these activation values are zero. Using the case-by-case analysis presented earlier for the first-order HAM (Section 11-A-3) and the first-order BAM (Section III-A-3), it is found that for all nonzero F, PE changes, A E , < 0; and when all F, PE's cease to change, A E , = 0. Continuing in the same fashion, during the F , to F, processing of PE values, (55) is simplified to yield

A E, = - ( AX)O - ( A X ) CY - o - O( AX) ' n P

= - C ( ' x i ) C Y j c j i i = l ;= 1

< 0. (57) The F , PE intraconnection recall equation, F(YB) , is During all other processing stages, A E , with respect to

AX is zero. Computing the kinetic energy created by the P

y r w = f ( x y k b k j , y ; l d \ , V j = 1 , 2 ; . . , p . (51) changesinthe F, PE'syields \ k = l I

The F, interconnection recall equation, F(YCT), is I P \

A E , = -xAx~-xc(AY)~-(AY)BY~-(AY)c~x~. ( 5 8 )

x?" = f y jc j i , .:Id , V i = 1,2, * . . , IZ .

The F, interconnection recall equation, F ( X A ) , is

( 5 2 ) During the F , to F , processing of PE values, a simplifi- cation of (58) and case-by-case analysis of the effects of A E , with respect to AY results in the relationship

I . j = 1 1 A E , = -o-o(AY) ' - ( A Y ) BY' - ( A Y ) O x y w y f ( X k a k r , X : ' d , V i = 1 , 2 , " ' , n . (53)

P P

= - ( A Y , > C Yk'jk i = 1 k = l

k = l 1 For each of these equations, (50)-(53), f is the familiar

(59) two-parameter bipolar threshold equation, (4). A conve- nient representation of the information flow of the IBAM is shown in Fig. 11. The arrow shows how the PE activa- tion values are passed in a cycle through the various matrices. In the next section, the stability of this recall procedure will be discussed.

3) Stability: To prove IBAM stability, the following potential energy function is employed:

< 0.

Similarly, during the F, to F , processing of PE values, (58) yields the relationship

AE,=-O-XC(AY)'-(AY)O-(AY)O P n

= - C ( A Y j ) C x i c i j ;= 1 i = l

E = - XAXT - XCY ' - YBY ' - YCTXT n n n P

< 0. = - X j X k a ; k - x j y j c j j During all other processing stages, A E , with respect to

AY is zero. All the possible effects of the change in energy with respect to the changes in PE activations have

- Y j y k b j , - C y j x i c j i . (54) been described using the information flow described here, and it shows the A E , + A E , < 0. Therefore the IBAM is

i = l k = l i * l j -1

P P P n

j = 1 k = l ;= 1 ;= 1

From (541, the kinetic energy with respect to the change in F, PE's and with respect to the change in the F , PE's is computed. Computing the kinetic energy with respect to the change in the F, PE's yields

A E, = - ( AX) AXT - ( A X ) C Y T

- Y B Y T - Y C T ( A X ) ' . ( 5 5 ) During the F, to F, processing of PE values, (55) is simplified to yield

stable.

B. Higher-Order IBAMS

IBAM's can be extended to higher-order correlations. Because the extensions are directly extendable from the first-order IBAM and the higher-order BAM and HAM, those efforts will not be duplicated here. In addition, the following general potential energy equation for the gth- order IBAM is presented:

m

E,= - ((X.X,)" '+(Y.Y,)""(X.X,,)"(Y.Y,) A E , = - ( A X ) A X T - ( A X ) O - 0 - O ( AX)' n n h = l

+ ( X - X h ) ( Y . Y h ) ' ) . (61) = - C X k a i k i = l k = l

Page 11: Higher-ordered and intraconnected bidirectional associative memories

I

SIMPSON: H I G H E R - O R D E R E D A N D INTRA('ONNE('TED HIDIRE<'TIONAL ASSO('IATIVE M E M O R I E S 647

TABLE 11 CAPACITY RATIOS PRODUCED FROM OUR SIMLJLATIONS FOR FIRST-ORDER, DIRECT SECOND-ORDER,

AND GENERAL SECOND-ORDER BAM's, IBAM's, AND HHAM's

BAM IBAM HHAM

0.25 1.72 1.63 first-order

1.48 1.38 0.88 general second-order

0.22 0.26 0.34 first-order 10 0.41 0.28 0.45 direct second-order

0.45 0.30 0.50 general second-order

0.14 0.15 0.18 first-order 20 0.38 0.32 0.40 direct second-order

0.37 0.33 0.41 general second-order

0.10 0.11 0.13 first-order 50 0.3 1 0.30 0.39 direct second-order

0.32 0.30 0.38 general second-order

0.05 0.07 0.08 first-order 100 0.13 0.13 0.15 direct second-order

0.14 0.12 0.14 general second-order

4 1.20 0.76 0.73 direct second-order

As the statistics $how, the theoretical capacity ratio is empirically confirmed to be a logarithmically decreasing value with increasing dimensionality. Later graphs of the data make this more evident.

TABLE I11 INFORMATION RATIOS PRODUCED FROM OUR SIMULATIONS FOR FIRST-ORDER, DIRECT SECOND-ORDER,

AND GENERAL SECOND-ORDER BAM's, IBAM's, AND HHAM's

BAM IBAM HHAM

0.25 4 0.30

0.30

8.7X 10W2 10 1 . 6 ~ lo-*

1.6X 10W2

2.7X 20 3 . 8 ~ lo-'

3.6X lo-'

8.1 X lo-' 50 5 . o ~ 10-4

5 . o ~ 10-4

2.1 x 10-3 100 5.3 x 10-5

5 . 4 ~ 10-5

0.57 9 . 4 ~ 0.12

3.4X 5.5 X lo-' 5.3x lo-'

3 . 0 ~ lo-' 1 . 6 ~ lo-' 1.5X lo-'

8.9X 2 . 4 ~ 10-4 2 . 4 ~ 1 0 - 4

1.8 x 10-4 2 . 7 ~ 10-5 2 . 4 ~ 10-4

0.41 4 . 6 ~ lo-' 4 . 4 ~

3.4x 10-2 4.5 x 10-3 4.5x lo-'

1.ox 10-3 9 . 7 ~ 1 0 - ~

1 . 6 ~ 10-4 I S X ~ O - ~

8.1 x 10-4 1 . 5 ~ 10-5 1 . 4 ~ 10-5

9.1 x lo-'

2.6X lo-'

first-order direct second-order

general second-order

first-order direct second-order

general second-order

first-order direct second-order

general second-order

first-order direct second-order

general second-order

first-order direct second-order

general second-order

Higher information ratios represent a more efficient use of connections.

It is obvious that this equation represents a composite of the general higher-order HAM and BAM potential en- ergy equation. Because of its composite nature, this equa- tion is subject to the processing restrictions of its higher- ordered cousins, specifically the asynchronous update restriction for all odd valued g.

V. COMPARATIVE SIMULATIONS

An extensive set of simulation suites was collected to compare the storage capacity and storage efficiency of three different heterocorrelators; a heteroassociative ver- sion of the HAM, the BAM, and the IBAM. The simula- tion suites were conducted for first-order, direct second- order, and general second-order heterocorrelators with a total pattern dimensionality of 4, 10, 20, 50 and 100. Each suite consisted of between 150 and 1000 separate program executions. Each program execution randomly selected a

pattern pair, stored it in the heterocorrelator, and verified that it, and all the previously stored patterns, were com- pletely recallable without errors. Restated, each program returned the number of randomly chosen patterns that it could perfectly store. Each program was written in C and executed on a Sun-3 Workstation. The entire suite of simulations required approximately three months of con- tinuous and simultaneous processing on five Sun-3 Work- stations.

The statistics gathered were capacity ratios and infor- mation ratios. The capacity ratio is the number of pat- terns stored with respect to the pattern dimensionality:

number of patterns stored dimensionality of the patterns capacity ratio = . (62)

Table I1 lists the capacity ratios gathered from our simu- lations. The theoretical results discussed earlier ((6), (19), and (28)) stated that the capacity ratio is a logarithmically

Page 12: Higher-ordered and intraconnected bidirectional associative memories

I

648 IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 20, NO. 3, MAY/IUNE 1990

First Order BAM Direct Second Or ie l F3AM General Second Ordrr RAM

0 10

1.00 -

0.90 -

0.80 -

0.70 -

0.60 -

0.50 -

0.40 -

0.30 -

First Order IBAM Direct Second Order IBPM General Second Order IBAM -

o'oo I 30 40 50 60 70 80 90 100 I I I ~ I I I I

10 20 30 40 50 60 70 80 90 100

(a)

1.00 -

0.90 -

0.80 -

0.70 -

0.60 -

0.50 -

0.40

0.30 .

0.20

0.10

0.00

(b)

First-Order HHAM "A-*-- Direct Second-Order "AM - General Second-Order HHAM

-- 10 20 30 40 50 60 70 80 90 100

(C )

Fig. 12. Comparison of capacity ratios for each heterocorrelator is shown. Combined dimensionality of both pattern pairs (ranging from 0 to 100) is found along horizontal axis and capacity ratio (ranging from 0 to 1) is found along vertical axis. These figures illustrate difference in storage capacity between first- and second-order correlations. (a) BAM. (b) IBAM. (c) HHAM.

1 - 1

Page 13: Higher-ordered and intraconnected bidirectional associative memories

I

SIMPSON: HIGHER-ORDERED AND INTRACONNECTED BIDIRECTIONAL ASSOCIATIVE MEMORIES

1 .oo

0.90

0.80

0.70

0.60

0.50

0.40

0.X

0.21

0.11

0.0

- First-Order HHAM First-Order IBAM First Order BAM

1 00 -

0 90 -

0 80 -

0 70

110 20 30 40 50 60 70 80 90 100 I I I I I I I I I

(a)

1 00

0 90

0 80

0 70

0 60

0 50

C 40

0 3c

0 21

0 11

0 0

0 60

0 50

649

- Direct Second Order "AM Direct Second-Order IBAM Direct Second-Order BAM

0.00 O i 0 a (b) 100

i - General Second~Order HHAM General Second-Order IBAM General Second~Order BAM

\

Fig. 13. Capacity ratios for each of correlation orders is shown. Combined dimensionality of both pattern pairs (ranging from 0 to 100) is found along horizontal axis, and capacity ratio (ranging from 0 to 1) is found along vertical axis. These figures illustrate difference in storage capacity between each heterocorrelator for each order. (a) Comparison of first-order capacity ratios. (b) Comparison of direct second-order capacity ratios. (c) Comparison of general second-order capacity ratios.

Page 14: Higher-ordered and intraconnected bidirectional associative memories

650

10-1

1 o - ~

IO-^

1 o - ~

0.00

I

IEEE T R A N S A ~ I O N S ON SYSTEMS, MAN, AND CYBERNETICS, VOL. 20, NO. 3, MAY/JUNE 1990

First-Order BAM -- Direct Second-Order B4M - General Second-Oroer BAhl

First-Otder IBAM " " "x." ~ Direct Second-Order IBAM - General Second Order IBAM

First-Order BAM -- Direct Second Order B4M - General Second-Oroer BAhl

I I I I I I I 1 1 1 10 20 30 40 50 60 70 80 90 100

(a)

First-Order HHAM - Direct Second-Order HHAM - General Second-Order HHAM

1

0.00 1 I I I I I I I I I 10 20 30 40 50 60 70 80 90 100

(C) Comparison of information ratios for each heterocorrelator is shown. Combined dimensionality of both pattern

pairs (ranging from 0 to 100) is found along horizontal axis, and information ratio (ranging from 0 to lo-') is found along vertical axis. These figures illustrate difference in storage efficiency between first- and second-order correlations. (a) BAM. (b) IBAM. (c) "AM.

Fig. 14.

Page 15: Higher-ordered and intraconnected bidirectional associative memories

I

-

SIMPSON: HIGHER-ORDERED AND INTRACONNECTED BIDIRECTIONAL ASSOCIATIVE MEMORIES

1 o-2

lo9 -

-

lo.!

10.~ -

-

-

-

0 00

65 1

10-1 -

First Order HHAM First Order IBAM First Order 3AM

-

\ --. --a*

\ I

IO4 - 1

I I I I I I I I I I 10 20 30 40 50 60 70 80 90 100

10.2 -I - General Second-Order HHAM

General Second-Order IBAM General Second-Order BAM

- Direct Second Order HHAM Direct Second-Order IBAM Direct Second-Order BAM

10 20 30 40 50 60 70 80 90 100

(C) Fig. 15. Information ratios for each of correlation orders is shown. Combined dimensionality of both pattern pairs (ranging

from 0 to 100) is found along horizontal axis, and information ratio (ranging from 0 to lo-') is found along vertical axis. These figures illustrate difference in storage efficiency between each heterocorrelator for each order. (a) Comparison of first-order information ratios. (b) Comparison of direct second-order information ratios. (c) Comparison of general second-order information ratios.

Page 16: Higher-ordered and intraconnected bidirectional associative memories

652

decreasing value that tends to zero as the dimensionality becomes infinitely large. Our simulations empirically con- cur with these theoretical values. The information ratio is the number of patterns stored with respect to the number of connections:

number of patterns stored number of connections information ratio = . (63)

Table 111 lists the information ratios gathered from our simulations.

The heteroassociative HAM (HHAM) is one of the three heterocorrelators used in the simulations. A hetero- correlator is created from an autocorrelator in a fashion similar to the heterocorrelation-to-autocorrelation proce- dure described by (43) and Fig. 8. Each pattern pair is concatenated into a single pattern. The pattern initially presented to the HAM has the first n dimensions filled with the input pattern and the last p dimensigns filled with zeroes. By filling the last p dimensions of this ( n + p)-dimensional pattern with zeroes, a set of don’t- care states is initially introduced as the unknown pattern. The p-dimensional pattern of zeroes will initially have no effect on the recall of a complete pattern, allowing the first n dimensions to solely act as the recall cue for the entire pattern pair concatenation. In every other way, the HHAM processes information exactly as the HAM described in section 11.

A. Capacity Ratios

One of the advantages of using higher-order correla- tions is improved storage capacity. Figs. 12(a), (b), and (c) display the capacity ratios of the BAM, IBAM, and HHAM, respectively, for first-order, direct second-order, and general second-order correlations. Each heterocorre- lator exhibited a marked increase in capacity for second- order correlations, typically twice the first-order capacity. Also, each heterocorrelator showed no appreciable differ- ence in storage capacity between the direct and general second-order correlations.

Figs. 13(a)-(c) compare the first-order, direct second- order, and general second-order capacities, respectively, for each of the heterocorrelators. For the first-order correlations, Fig. 13(a), it is clear that the HHAM has the best overall capacity performance, followed closely by the IBAM, and then by the BAM. For both the direct and general second-order correlations, Figs. 13(b) and (c), the HHAM again proved to have the best capacity perfor- mance, but surprisingly, the BAM slightly outperforms the IBAM. Apparently, for second-order correlations, the intralayer connections inhibit the IBAM’s recall perfor- mance.

B. Information Ratios

Although higher-order correlations allow greater stor- age capacity, it is at the expense of greater connectivity. The information ratio is a metric that describes the stor- age efficiency. Figs. 14(a)-(c) compare the information

ratios of the BAM, IBAM, and “AM, respectively, for first-order, direct second-order. and general second-order correlations. Each heterocorrelator exhibits significantly better storage efficiency for the first-order correlations, and a marked decrease in the information ratio for the second-order correlations. Also, like the capacity ratios, the difference between the direct and general second- order correlation information ratios is indistinguishable.

Figs. lS(a), 15(b), and 13c) compare the first-order, direct second-order and general second-order information capacities, respectively, for each of the heterocorrelators. For the first-order correlations, Fig. 15(a), the HHAM and the BAM share the best storage efficiency, and the information ratio of the IBAM is discernibly less. For both the direct and general second-order information ratios, Figs. 13b) and 15(c), the efficiency measurements were closely grouped but distinguishably different, with the IBAM performing best, followed by the HHAM and then the BAM.

VI. CONCLUSION

Overall, the performance of the HHAM exceeded that of either the BAM or the IBAM, especially when consid- ering both the information and capacity ratios. The only plausible reason for this is that the synchronous updating of one layer provides superior storage and recall ability to that of either of the two layer heterocorrelators. Although statistics for both higher dimensionality and higher- ordered correlations would have been more desirable, the processing time for these extensive simulations simply becomes too prohibitive. Despite this possible lack of more comprehensive simulations, the empirical results gathered show undeniable trends that agree with the theoretical results presented elsewhere, and it is expected that these trends would continue for higher dimensionali- ties and higher-ordered correlations.

ACKNOWLEDGMENT

I would like to thank my wife Christalyn for her unend- ing patience and support. I would also like to thank J. Harold McBeth and Ho Chung Lui for their helpful comments and advice during the preparation of this pa-

REFERENCES

S. Amari, “Learning patterns and pattern sequences by self- organizing nets of threshold elements,” IEEE Trans. Cornput., vol.

S. Amari, “Neural theory of association and concept formation.” Biol. Cybenz., vol. 26, pp. 175-185. 1977. S . Amari and K. Maginu, “Statistical neurodynamics of associative memory,” Neural Networks, vol. 1, pp. 63-74, 1988. J. Anderson, “A simple neural network generating interactive memory,” Math. Biosci., vol. 14, pp. 107-220, 1972. P. Baldi and S. Venkatesh, “Number of stable points for spin glasses and neural networks of higher orders,” Phys. Rev. Lett., vol. 58, p. 913, 1987.

C-21, pp. 1197-1206, 1972.

Page 17: Higher-ordered and intraconnected bidirectional associative memories

I

SIMPSON: HIGHER-ORDERED AND INTRACONNFCTED BIDIRECTIONAL ASSOCIATIVE MEMORIES 653

[211

H. Chen, Y. Lee, G. Sun, H. Lee, T. Maxwell, and C. Giles, “Higher order correlation model for associative memory,” in AIP Conference Proceedings 15I: Neural Networks for Computing, J. Denker, Ed. New York American Institute of Physics, 1986, pp. 86-99. G. Dunning, E. Marom, Y. Owechko, and B. Soffer, “Optical holographic associative memory using a phase conjugate res- onator,” Proceedings of the SPIE, vol. 625, pp. 205-213, 1986. C. Giles and T. Maxwell, “Learning and generalization in higher order networks,” Appl. Optics, vol. 26, pp. 4972-4978, 1987. C. Giles, R. Griffin, and T. Maxwell, “Encoding geometric invari- ances in higher-order neural networks,” in Neural Information Processing Systems, D. Anderson, Ed. New York American Insti- tute of Physics, 1988, pp. 301-309. G. Gindi, A. Gmitro, and K. Parthasarathy, “Hopfield model associative memory with nonzero-diagonal terms in memory ma- trix,” Appl. Optics, vol. 27, pp. 129-134, 1988. K. Haines and R. Hecht-Nielsen, “A BAM with increased infor- mation storage capacity,” in Proc. of the 1988 IEEE Int. Conf on Neural Networks, vol. I, 1988, pp. 181-190. J. Hopfield, “Neural networks and physical systems with emergent collective computational abilities,” Proceedings of the National Academy of Sciences, vol. 79, 1982, pp. 2554-2558. T. Kohonen, “Correlation matrix memories,” IEEE Trans. Corn-

T. Kohonen and M. Ruohonen, “Representation of associated data by matrix operators,” IEEE Trans. Comput., vol. C-22, 1973,

B. Kosko, “Competitive adaptive bidirectional associative memo- ries,” in Proc. of the IEEE First Int. Conf. on Neural Networks, vol.

~, “Sampling adaptive bidirectional associative memories,” in Proc. of the 21st Asilomar Conf: on Signals, Systems and Computers, Nov. 1987. -, “Bidirectional associative memories,” IEEE Trans. Syst. Man Cybern., vol. SMC-18, pp. 49-60, 1988. W. Little, “The existence of persistent states in the brain,” Math. Biosci., vol. 19, pp. 101-120, 1974. W. Little and G. Shaw, “Analytical study of the memory storage capacity of a neural network,” Math. Biosci., vol. 39, 1978, pp.

H. Loos, “Reflexive associative memories,” in Neural Information Processing Systems, D. Anderson, Ed. New York: American Insti- tute of Physics, 1988, pp. 495-504. R. McEliece, E. Posner, E. Rodemich, and S. Venkatesh, “The capacity of the Hopfield model,” IEEE Trans. Inform. Theoly, vol.

P. Peretto and J. Niez, “Long-term memory storage capacity of multiconnected neural networks,” Biol. Cybern., vol. 54, pp. 53-63, 1986.

put., vol. C-21, 1972, pp. 353-359.

pp. 701-702.

11, 1987, pp. 759-766.

281 -289.

IT-33, 1987, pp. 461-482.

[23] D. Psaltis, C. Park, and J. Hong, “Quadratic optical associative memories,” J . Opti. Soc. Amer. - A , vol. 3, p. 32, 1986.

(241 -, “Higher order associative memories and their optical implementations,” Neur. Networks, vol. 1, pp. 149-164, 1988.

[25] P. Simpson, “Bidirectional associative memory systems,” Heuristis, vol. 1 , no. 2, pp. 52-60, 1988.

[26] -, Artificial Neural Systems: Foundations, Paradigms, Applica- tions and Implementations.

[27] G. Stone, “An analysis of the delta rule and the learning of statistical associations,” in Parallel Distributed Processing, Vol. I , D. Rumelhart, J. McClelland, and the PDP Group, Eds. Cam- bridge: MIT Press/Bradford Books, 1986, pp. 444-459. W. Wee, “Generalized inverse approach to adaptive multiclass pattern classification,” IEEE Trans. Comput., vol. C-17, pp. 1157-1164, 1968. B. Widrow and M. Hoff, “Adaptive switching circuits,” in Proc. of the IRE WESCON Concention Record, Part IV, p. 96, 1960.

Elmsford: Pergamon Press, 1990.

(281

1291

Patrick K. Simpson (M’87) received the B.A. degree in computer science from the University of California at San Diego.

He is the Principal Investigator of the neural networks internal research and development project at General Dynamics Electronics Divi- sion, specializing in the application of neural networks, fuzzy logic, and artificial intelligence to difficult defense-related problems. He teaches neural network courses at the University of Cal- ifornia at the San Diego Extension. He has

published several papers on topics that include associative memory systems, spatiotemporal pattern classifiers, higher-ordered heterocorre- lation associative memories, Hebbian learning law variants, and the application of neural networks to battlefield surveillance, EW/ECM, and diagnostics.

Mr. Simpson is a member of the IEEE, the International Neural Network Society, and the International Association of Knowledge Engi- neers. He was the Tutorial Chair at the 1988 IEEE International Conference on Neural Networks, he is Treasurer of Council to the IEEE Neural Network, and he is the Local Arrangements Chair of the 1990 International Joint Conference on Neural Networks. He has re- cently completed a book, entitled Artificial Neural Systems: Foundations, Paradigms Application and Implementations, that provides a broad overview of neural network technology and succinctly describes 28 neural network paradigms with application potential.