Proceedings of International Joint Conference on Neural Networks, Montreal, Canada, July 31 - August 4, 2005

Dynamical Consistent Recurrent Neural NetworksHans-Georg Zimmermann, Ralph Grothmann, Anton M. Schafer and Christoph Tietz

Siemens AG, Corporate Technology, Information & Communication Division, Dept. of Neural ComputationHans-Georg.Zimmermann@siemens.com

Abstract- Recurrent neural networks are typically consid-ered as relatively simple architectures, which come along withcomplicated learning algorithms. Most researchers focus on theimprovement of these algorithms. Our approach is different:Rather than focusing on learning and optimization algorithms,we concentrate on the design of the network architecture.

As we will show, many difficulties in the modeling of dynam-ical systems can be solved with a pre-design of the networkarchitecture. We will focus on large networks with the taskof modeling complete high dimensional systems (e.g. financialmarkets) instead of small sets of time series. Standard neuralnetworks tend to overfit like any other statistical learning system.We will introduce a new recurrent neural network architecture inwhich overfitting and the associated loss of generalization abilitiesis not a major problem. We will enhance these networks bydynamical consistency.

I. INTRODUCTION

Recurrent neural networks allow the identification of dy-namical systems in form of high dimensional, nonlinear statespace models. They offer an explicit modeling of time andmemory and allow in principle to model any type of dynamicalsystems [Hay94; MJ99; KKO1]. The basic concept is as oldas the theory of artificial neural networks, so e.g. unfolding intime of neural networks and related modifications of the back-propagation algorithm can be found in [Wer74] and [RHW86].Different types of learning algorithms are summarized inthe paper of Pearlmutter [Pearl95]. Nevertheless, over thelast 15 years most feedforward neural networks have beenpredominantly applied to time series problems. The appealingway of modeling time and memory in recurrent networksis opposed to the apparent easier numerical tractability of apattern recognition approach as represented by feedforwardneural networks. Still some researchers enhance the theoryof recurrent neural networks. Recent developments are sum-marized in the books of Haykin [Hay94], Kolen and Kremer[KKO1], Soofi and Cao [SC02] and Medsker and Jain [MJ99].Our approach differs from the outlined research directions

in a significant but at first sight non-obvious way. Insteadof focusing on algorithms, we put network architectures inthe foreground. We show, that the design of a networkarchitecture automatically implies the usage of an adjointsolution algorithm for the parameter identification problem.This correspondence between architecture and equations isvalid for simple as well as complex network architectures. Theunderlying assumption is, that the associated parameter opti-mization problem is solved by error backpropagation throughtime, i.e. a shared weights extension of the standard errorbackpropagation algorithm.

In our previous papers we discussed the modeling of dynam-ical systems based on time-delay recurrent neural networks[NZ98; ZNO1]. We solved the system identification task byunfolding in time, i. e. we transferred the temporal probleminto a spatial architecture, which can be handled by the errorbackpropagation through time [RHW86; Hay94, p. 354-357and p. 751-756]. Proceeding this way, we can enforce thelearning of the autonomous dynamics in an open system byovershooting [ZNO1, p.326-327]. Consequently our recurrentneural networks not only learn from data but also integrateprior knowledge and first principles into the modeling in formof architectural concepts.

However, the question arises if the outlined neural networksare a sufficient framework for the modeling of complexnonlinear and high dimensional dynamical systems, whichcan only be understood by analyzing the interrelationship ofdifferent sub-dynamics. Our experiments indicate, that simplyscaling up the basic time-delay recurrent neural networksby increasing the dimension of the internal state, results inoverfitting due to the high number of free parameters.

In this paper we present architectures which are feasible forlarge recurrent neural networks. These architectures are basedon a redesign of the basic time-delay recurrent neural networks[ZNO1; Hay94, p. 322-323 and p. 739-747]. Remarkably, mostof the resulting networks cannot even be designed with a lowdimensional internal state (see sec. II). We further move froman often arbitrary distinction between input and output to amodeling of observables (see sec. III). In addition, we focuson a consistency problem of traditional statistical modeling:Typically one assumes, that the environment of the systemremains unchanged when the dynamics is iterated into thefuture direction. We show, that this is a questionable statisticalassumption and solve the problem with a dynamical consistentrecurrent neural network (see sec. IV).

II. NORMALIZATION OF RECURRENT NETWORKSFor discrete time grids a basic time-delay recurrent neural

network can be described with a state transition equation stand output equation Yt:

st+i = tanh(Ast + c + But) state transition

Yt = Cst output equation(1)

The state transition equation st is a nonlinear combination ofthe previous state st-i and external influences ut using weightmatrices A and B and a bias c, which handles offsets in theinput variables ut. The network output Yt is computed fromthe present state st employing matrix C. The network output is

0-7803-9048-2/05/$20.00 @2005 IEEE 1 537

therefore a nonlinear composition applying the transformationsA, B and C [ZNO1; Hay94, p. 322-323 and p. 739-747]:As a preparation for the development of large networks

we first separate the state equation of the basic time-delayrecurrent network (Eq. 1) into a past and a future part. In thisframework st is always regarded as the present time state.All states s7 with < t belong to the past part and thosewith > t to the future part. The parameter is herebyalways bounded by the length of the unfolding in time m andthe length of the overshooting n (see [ZNO1; ZNG02]). Wehave C {t-m,...,t+n} forall tC {m,...,T-n}with T as the total number of available data patterns. Thepresent time ( = t) is included in the past part, as these statetransitions share the same characteristics. We get the followingrepresentation of the optimization problem:

< t: sr+1 = tanh(A sr + c+ BuT)

> t: sr+ i = tanh(As±+ c)

As a remedy, we propose the neural network of Eq. 3 whichincorporates besides the bias c only one connector type, thematrix A. The resulting architecture is depicted in Fig. 2.

< t: s, = tanh(Asr- 1+c+01

E0 ur)

Id]> t: sr = tanh(Asr-±+c)

(3)Yr = [IdOO]Sr

T-n t+n

E E (YT_ YTd) mint=m r=t-m A

YT = CSr (2)

T-n t+n

E E __-_ _4_mA,B,C,ct=m Tr=t-mABCc

Using finite unfolding in time, these equations can be easilytransformed into a neural network architecture (see Fig. 1).

Fig. 1. Unfolded recurrent neural network.

In this model, past and future iterations are consistent underthe assumption of a constant future environment. The difficultywith this kind of recurrent neural network is the training withbackpropagation through time, because a sequence of differentconnectors has to be balanced. The gradient computation isnot regular, i. e. we do not have the same learning behaviorfor the weight matrices in the different time steps. Even thetraining itself is unstable due to the concatenated matrices A,B and C. As the training changes weights in all of thesematrices, different effects or tendencies - even opposing ones- can influence them and may superpose. This implies, thatthere results no clear learning direction or change of weightsfrom a certain backpropagated error. In our experiments wefound, that these problems become even more important forthe training of large recurrent neural networks.Now the question arises, how to re-design the basic recur-

rent architecture (Eq. 2), such that the learning behavior andthe stability improves especially for large networks.

Fig. 2. Normalized recurrent neural network.

We call this model a Normalized Recurrent Neural Network.It avoids the stability and learning problems resulting from theconcatenation of the three matrices A, B and C. The modelingis now solely focused on the transition matrix A. The matricesbetween input and hidden as well as hidden and output layerare fixed, i. e. they are not learned. This implies that all freeparameters - as they are combined in one matrix - are nowtreated the same way by error backpropagation.

It is important to note, that the normalization or concen-tration on only one single matrix is paid with an oversized(high dimensional) internal state. At first view it seems, thatin this network architecture (Fig. 2) the external input UT isdirectly connected to the corresponding output Yr. This is notthe case, because we enlarge the dimension of the internalstate si, such that the input uT has no direct influence onthe output Yr. Assuming that we have a number of p networkoutputs, q computational hidden neurons and r external inputs,the dimension of the internal state would be dim(s) = p+q+r.

With the matrix [Id 0 0] we connect only the first p neuronsof the internal state s7, to the output layer Yr. As this connectoris not trained, it can be seen as a fixed identity matrix ofappropriate size. Consequently, the neural network is forcedto generate the p outputs of the neural network at the first pcomponents of the state vector s7.

Let us now focus on the last r state neurons, which are usedfor the processing of the external inputs u.. The connector[O 0 Id]T between the externals U,T and the internal state sr-is an appropriately sized fixed identity matrix. More precisely,

1538

the connector is designed such that the input UT is connectedto the last state neurons. Recalling that the network outputs arelocated at the first p internal states, this composition avoidsa direct connection between input and output. It delays theimpact of the externals uT on the outputs y, by at least onetime step.To additionally support the internal processing and to in-

crease the network's computational power, we add a numberof q hidden neurons between the first p and the last r stateneurons. This composition ensures, that the input and outputprocessing of the network is separated.

Besides the bias vector c the state transition matrix A holdsthe only tunable parameters of the system. Matrix A does notonly code the autonomous and the externally driven part ofthe dynamics, but also the processing of the external inputsuar and the computation of the network outputs Yr.Most remarkably, the normalized recurrent neural network

of Eq. 3 can only be designed as a large neural network. Ifthe internal state of the network is too small, the inputs andoutputs can not be separated, as the external inputs would atleast partially cover the internal states at which the outputs areread out. Thus, the identification of the network outputs at thefirst p internal states would become impossible.Our experiments indicate, that recurrent neural networks

in which the only tunable parameters are located in a singlestate transition matrix (e.g. Eq. 3) show a more stable trainingbehavior, even if the dimension of the internal state is verylarge. Having trained the large network to convergence, manyweights of the state transition matrix will be dispensablewithout affecting the functioning of the network. Unneededweights can be singled out by using a weight decay penaltyand standard pruning techniques (see e. g. [Hay94; NZ98]).

III. MODELING THE DYNAMICS OF OBSERVABLES

In the normalized recurrent neural network (Eq. 3) weconsider inputs and outputs independently. This distinctionbetween externals uT and the network output yr is arbitraryand mainly depends on the application or the view of themodel builder instead of the true underlying dynamical system.Therefore, for the following model we take a different pointof view. We merge inputs and targets into one group ofvariables, which we call observables. So we now look atthe model as a high dimensional dynamical system whereinput and output represent the observable variables of theenvironment. The hidden units stand for the unobservable partof the environment, which nevertheless can be reconstructedfrom the observations. By doing so, we arrive at an integratedview of the dynamical system.We implement this approach by replacing the externals uT

with the (observable) targets yd in the normalized recurrentnetwork. Consequently, the output Yr and the external inputyv have now identical dimensions.

< t: s = tanh(As-I+c+01

L Id J> t: s, = tanh(As,-+c)

(4)YTr [IdOO]sT.

T-n t+n

S: E (YT-y) Amint=m r=t-m A,c

The corresponding model architecture (Fig. 3) changes onlyslightly in comparison to Fig. 2.

Fig. 3. Normalized recurrent neural network modeling the dynamics ofobservables yd.

Note, that due to the one step time delay between inputand output, yr and Yr are not directly connected. Furthermoreit is important to understand, that we now take a totallydifferent view on the dynamical system. In contrast to Eq.3, the network (Eq. 4) not only generates forecasts for thedynamics of interest but also for all external observablesyd. Consequently, the first r state neurons are used for theidentification of the network outputs. They are followed by qcomputational hidden neurons and r state neurons which readin the external inputs.

IV. DYNAMICAL CONSISTENT NEURAL NETWORKSAn open dynamical system is partially driven by an au-

tonomous development and partially by external influences.If the dynamics is iterated into the future, the development ofthe system environment is unknown. Now, one of the standardstatistical paradigms is to assume, that the external influencesare not significantly changing in the future. This means, thatthe expected value of a shift in an external input yd with

> t is zero per definition. For that reason we have so farneglected the external inputs yd in the normalized recurrentneural network at all future time steps, > t, of the unfolding(see Eq. 4).

Especially when we consider fast changing external vari-ables with a high impact on the dynamics of interest, the latterassumption is very questionable. In relation to Eq. 4 it evenposes a contradiction, as the observables are assumed to beconstant on the input, but fluctuate on the output side. Even

1539

in case of a slowly changing environment, long-term forecastsbecome doubtful. The longer the forecast horizon is, the morethe statistical assumption of a constant environment is violated.A statistical model is therefore not consistent from a dynamicalpoint of view. For a dynamical consistent approach, one hasto integrate assumptions about the future development of theenvironment into the modeling of the dynamics.

For that reason we propose a network that uses its own pre-dictions as replacements for the unknown future observables.The resulting dynamical consistent recurrent neural network,DCRNN, is specified in Eq. 5:

< t: Sr = 0 IdO tar

Id 0 0> t: Sr = O Id O tari

LId O OiYr = [IdO O] sr

T-n t+n

(YT-y)2 mint=m -r=t-m A,c

nh(Asr_ + c) +[01

L[Idj

ah(As1.1 + c)

(5)The DCRNN is understood best by analyzing the structure

of the state vector s7-. In the past ( . t) and future ( > t)the structure of the internal state s7- is

YrhT

ST = J r < t: YdL T > t: YT |

I~ ~ ~expectationshidden states

epr< t: observationsI Ir > t: expectations

(6)

In the first r components of the state vector we have theexpectations Y, i.e. the predictions of the model. The qcomponents in the middle of the vector represent the hiddenunits h., They are responsible for the development of thedynamics. In the last r components of the vector we find inthe past ( < t) the observables yT, which the model receivesas external input. In the future ( > t) the model replaces theunknown future observables by its own expectations Y,. Thisreplacement is modeled with two consistency matrices:

Id 0 0 Id 0 0C 0 Id O and C>= O Id 0 (7)

0 O 0 Id 0 0

Let us explain one recursion of the state equation (Eq.5) step by step: In the past ( < t) we start with a statevector s-1, which has the structure of Eq. 6. This vectoris first multiplied with the transition matrix A. After addingthe bias c the vector is sent through the nonlinearity tanh.The consistency matrix then keeps the first r+ q components(expectations and hidden states) of the state vector but deletes(multiplication with zero) the last r ones. These are finallyreplaced by the observables yd, such that s,r has again the

partitioning of Eq. 6. Note that in contrast to the normalizedrecurrent neural network (Eq. 4) the observables are now addedto the state vector after the nonlinearity. This is important forthe consistency structure of the model.The recursion of the future state transition ( > t) differs

from the one in the past in terms of the structure of theconsistency matrix and the missing external input. The latteris now replaced with an additional identity-block in the futureconsistency matrix C > which maps the first r components ofthe state vector, the expectations Yr, to its last r components.In doing so we get the desired partitioning of s7. (Eq. 6) andthe model becomes dynamical consistent.

Fig. 4 illustrates the corresponding architecture. Note, thatthe nonlinearity and the final calculation of the state vectorare separated and hence modeled in two different layers. Thisfollows from the dynamical consistent state equation (Eq. 5),in which the observables are added separate from the nonlinearcomponent.

Fig. 4. Dynamical Consistent Recurrent Neural Network (DCRNN). Atall future time steps of the unfolding the network uses its own forecasts assubstitutes for the unknown development of the environment.

Regarding the transition matrix A, we want to point outthat in a statistical consistent recurrent network (Eq. 4) thematrix has to model the state transformation over time andthe merging of the input information. However, the networkis only triggered by the external drivers up to the present timestep t In a dynamical consistent network we have forecastsof the external influences, which can be used as future inputs.Thus, the transition matrix A is always dedicated to the sametask: modeling the dynamics.

V. CONCLUSIONS

In this article we focused on high-dimensional and dynam-ical consistent recurrent neural networks for the modeling ofopen dynamical systems. These networks allow an integratedview on real-world problems and consequently show bettergeneralization abilities. We concentrated the modeling of thedynamics on one single transition matrix and also enhancedfrom a simple statistical to a dynamical consistent handling ofmissing input information in the future part. The networks arenow able to map integrated dynamical systems (e. g. coherentfinancial markets) instead of only a small set of time series.Remarkably, these recurrent neural networks do not onlyprovide superior forecasts, but also a deeper understandingof the underlying dynamical system.

1540

Current applications for dynamical consistent neural net-works are financial and commodity market price forecasts.

Further research is done on a combination of dynami-cal consistency and error correction neural networks, ECNN[ZNG02]. ECNN use an error correction mechanism for aquantification of the model's misfit and as indicator of short-term effects or external shocks. We expect that dynamicalconsistent recurrent neural networks with error correction willfurther improve the ability of the identification and forecastingof complex and high-dimensional dynamical systems.

ACKNOWLEDGMENTAll computations are performed with the Simulation Environment

for Neural Networks (SENN), a product of Siemens AG.

REFERENCES[Hay94] Haykin S.: Neural Networks. A Comprehensive Foun-

dation, Macmillan College Publishing, New York, 1994.second edition 1998.

[HSW92] Hornik K., Stinchcombe M. and White H.: Multi-layer Feedforward Networks are Universal Approximators,in: White H. et al. [Eds.]: Artificial Neural Networks: Ap-proximation and Learning Theory, Blackwell Publishers,Cambridge, MA, 1992.

[KKO1] Kolen J. F. and Kremer St. [Eds.]: A Field Guide toDynamical Recurrent Networks, IEEE Press, 2001.

[MCOl] Mandic D. P. and Chambers J. A.: Recurrent NeuralNetworks for Prediction: Learning Algorithms, Architec-tures and Stability, Wiley Series in Adaptive Learning Sys-tems for Signal Processing, Communications & Control,S. Haykin [Ed.], J. Wiley & Sons, Chichester 2001.

[MJ99] Medsker L. R. and Jain L. C.: Recurrent Neural Net-works: Design and Application, CRC Press internationalseries on comp. intelligence, No. I, 1999.

[NZ98] Neuneier R. and Zimmermann H. G.: How to TrainNeural Networks, in: Neural Networks: Tricks of theTrade, Springer, Berlin, 1998, pp. 373-423.

[Pearl95] Pearlmatter B.: Gradient Calculations for DynamicRecurrent Neural Networks: A survey, in: IEEE Transac-tions on Neural Networks, Vol. 6, Nr. 5, 1995.

[RHW86] Rumelhart D. E., Hinton G. E. and Williams R. J.:Learning Internal Representations by Error Propagation,in: D.E. Rumelhart, J. L. McClelland, et al., Parallel Dis-tributed Processing: Explorations in The Microstructure ofCognition, Vol. 1: Foundations, Cambridge: M.I.T. Press,pp. 318-362, 1986.

[SC02] Soofi, A. and Cao, L.: Modeling and Forecasting Fi-nancial Data, Techniques ofNonlinear Dynamics, KluwerAcademic Publishers, 2002.

[Wer74] Werbos P. J.: Beyond Regression: New Tools forPrediction and Analysis in the Behavioral Sciences,PhD. Thesis, Harvard University, 1974.

[ZGT02] Zimmermann H. G., Grothmann R. and Tietz Ch.:Yield Curve Forecasting by Error Correction Neural Net-works and Partial Learning, in: M. Verleysen [Ed.], Pro-ceedings of the European Symposium on Artificial NeuralNetworks (ESANN) 2002, Bruges, Belgium, pp. 407-12.

[ZGST05] Zimmermann H. G., Grothmann R., Schafer A.M.and Tietz Ch.: Identification and Forecasting of LargeDynamical Systems by Dynamical Consistent Neural Net-works, in: Haykin S., Principe J., Sejnowski T. andMcWhirter J.[Eds.]: New Directions in Statistical SignalProcessing: From Systems to Brain, MIT Press, forthcom-ing 2005

[ZNO1] Zimmermann H. G. and Neuneier R.: Neural NetworkArchitectures for the Modeling of Dynamical Systems, in:A Field Guide to Dynamical Recurrent Networks, Eds.Kolen, J.F.; Kremer, St.; IEEE Press, 2001, pp. 311-350.

[ZNG02] Zimmermann H. G., Neuneier R. and GrothmannR.: Modeling of Dynamical Systems by Error CorrectionNeural Networks, in: Modeling and Forecasting FinancialData, Techniques of Nonlinear Dynamics, Eds. Soofi, A.and Cao, L., Kluwer Academic Publishers, 2002.

1541