Designing spontaneous behavioral switching via chaotic itinerancy

Katsuma Inoue,∗ Kohei Nakajima,† and Yasuo Kuniyoshi‡Graduate School of Information Science and Technology,

The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan(Dated: February 20, 2020)

Chaotic itinerancy is a frequently observed phenomenon in high-dimensional and nonlinear dynamical sys-tems, and it is characterized by the random transitions among multiple quasi-attractors. Several studies haverevealed that chaotic itinerancy has been observed in brain activity, and it is considered to play a critical role inthe spontaneous, stable behavior generation of animals. Thus, chaotic itinerancy is a topic of great interest, par-ticularly for neurorobotics researchers who wish to understand and implement autonomous behavioral controlsfor agents. However, it is generally difficult to gain control over high-dimensional nonlinear dynamical systems.Hence, the implementation of chaotic itinerancy has mainly been accomplished heuristically. In this study, wepropose a novel way of implementing chaotic itinerancy reproducibly and at will in a generic high-dimensionalchaotic system. In particular, we demonstrate that our method enables us to easily design both the trajectories ofquasi-attractors and the transition rules among them simply by adjusting the limited number of system param-eters and by utilizing the intrinsic high-dimensional chaos. Finally, we quantitatively discuss the validity andscope of application through the results of several numerical experiments.

I. INTRODUCTION

Designing a cognitive architecture that acts spontaneouslyin the real-world environment is one of the ultimate goals inthe field of cognitive robotics [1]. A cognitive agent is ex-pected to have autonomy; i.e., the agent should behave in-dependently of the designer’s control while maintaining itsidentity. Furthermore, adaptability is another requirement forcognitive functionality. Thus, the agent must select the ap-propriate behavior continuously and robustly in response tothe changing environment in real-time. To summarize, theagent’s cognitive behavior should be implemented through thebody-environment interaction while still enabling the agent tomaintain its autonomy and adaptability.

In the conventional context of robotics and artificial intel-ligence, designers often take top-down approaches to providean agent with a hierarchical structure corresponding to the be-havioral category. This representation-based approach has acritical limitation in the design of a cognitive agent: the staticand one-to-one relationship between the behavior and struc-ture makes it difficult to adapt flexibly to the dynamicallychanging environment and developing body. For example, ithas been considered that the motion control systems of liv-ing things, including humans, realize their high-order motionplans by combining the reproducible motor patterns calledmotion primitives [2]. Inspired by this viewpoint, we tend torealize an agent’s behavior control with a predetermined statichierarchical structure. However, such a hierarchical structuredoes not exist in living organisms from the beginning; rather,these structures are cultivated through the body’s developmentand dynamic interactions with the environment. Thus, it is im-portant to introduce a dynamical perspective to understand ahierarchical structure of behavior control generated in animalsthat possesses adaptability and flexible plasticity.

∗ k-inoue@isi.imi.i.u-tokyo.ac.jp† k˙nakaijma@mech.t.u-tokyo.ac.jp‡ kuniyosh@isi.imi.i.u-tokyo.ac.jp

In robotics, approaches based on dynamical systems the-ory have been applied to analyze and control agents beingmodeled as sets of variables and parameters on a phase space[3, 4]. This dynamical systems approach can deal with boththe functional hierarchy and the elementary motion in a uni-fied form by expressing the whole physical constraints of theagent as the temporal development of state variables, namelydynamics. For example, Jaeger [4] sketched a pioneering ideaof an algorithm where the behavior of an agent is expressedas dynamics, and both the behavioral regularity (referred to astransient attractors) and the higher-order relationships amongthem are extracted in a bottom-up manner. Therefore, thedynamical systems approach has the potential to model anagent’s hierarchical behavior as the dynamics of interactionwithout a top-down structure given by the external designer.

Following this dynamical systems perspective, chaotic itin-erancy (CI) [5–7] is a powerful option for modeling sponta-neous behavior with functional hierarchy. CI is a frequentlyobserved, nonlinear phenomenon in high-dimensional dynam-ical systems, and it is characterized by random transitionsamong locally contracting domains, namely quasi-attractors.In general, a chaotic system has an initial sensitivity, anda slight difference in the phase space is exponentially ex-panded in a certain direction with temporal development.Conversely, multiple transiently predictable dynamics can berepeatedly observed in a chaotic system, yielding CI despitethe global chaoticity and initial sensitivity. Interestingly, thistype of hierarchical dynamics frequently emerges from high-dimensional chaos even without hierarchical mechanisms, im-plying that explicit structure is not necessarily needed for im-plementing hierarchical behaviors. Furthermore, the chaotic-ity plays an important role in forming the autonomy of anagent as it is virtually impossible for the designer to predictand control an agent’s behavior completely due to the agent’sinitial sensitivities, which essentially ensures the agent’s in-dependence from the designer. Thus, CI would work as aneffective tool for implementing the intellectual behavior ofa cognitive agent by embedding the behavior in the form ofa quasi-attractor and maintaining the autonomy of the agent

arX

iv:2

002.

0833

2v1

[cs

.RO

] 1

3 Fe

b 20

20

2

with the chaoticity.Historically, CI was first found in a model of optical turbu-

lence [5]. Since its discovery, similar phenomena have beennumerically obtained in various setups [6, 8, 9]. Several physi-ological studies have reported that CI-like dynamics have evenoccurred in brain activity, suggesting that CI plays an essen-tial role in forming cognitive functions [10, 11]. For exam-ple, Freeman et al. [12] revealed that an irregular transitionamong learned states was observed in the electroencephalo-gram pattern of a rabbit olfactory when a novel input wasgiven, indicating that the cognitive conditions correspondingto “I don’t know” are internally realized as CI-like dynamics.Furthermore, a recent observation of rat auditory cortex cellactivity revealed the existence of a random shift among dif-ferent stereotypical activities corresponding to individual ex-ternal stimuli during anesthesia [13]. Based on these reports,Kurikawa et al. [14] suggested the novel idea of “memory-as-bifurcation” to understand the mechanism of the transitoryphenomenon. They reproduced it in an associative memorymodel in which several input-output functions were embeddedby Hebbian learning. An intermittent switching among meta-stable patterns was also observed in a recurrent neural net-work (RNN) by installing multiple feedback loops trained tooutput a specific transient dynamic corresponding to externaltransient inputs [15]. CI-like dynamics arise not only in ner-vous systems but also in interactions between agent’s bodiesand their surrounding environments [16–19]. Kuniyoshi et al.[17] developed a human fetal development model, for exam-ple, by coupling chaotic central pattern generators and a mus-culoskeletal system, and they reported that several commonbehaviors, such as crawling and rolling over, emerge from thephysical constraint. Therefore, CI is a nonlinear phenomenonof high-dimensional dynamical systems and is thought to playa significant role in generating structural behavior.

Inspired by the contribution of CI to the cognitive func-tions and spontaneous motion generation of agents, CI hasbeen utilized for motion control in the field of neuroroboticsand cognitive robotics by designing the CI trajectory. Forexample, Namikawa and Tani [20–22] designed stochasticmotion-switching among predetermined motor primitives ina humanoid robot by using a hierarchical, deterministic RNNcontroller. In this study, it was confirmed that lower-orderRNNs with larger time constants stably produced the trajec-tories of motion primitives, whereas higher-order RNNs withlarger time constants realized a pseudo-stochastic transitionby exploiting self-organized chaoticity. Steingrube et al. [23]designed a robot that skillfully broke a deadlock state in whichthe motion had completely stopped by employing chaos in theRNN controller. Hence, it can be interpreted that CI-like dy-namics were embedded in the coupling of the body and thesurrounding environment.

While CI is a fascinating phenomenon in high-dimensionaldynamical systems, roboticists also find it a useful tool fordesigning an agent’s behavior structure while maintaining theagent’s autonomy. However, it has generally been difficult toembed desired quasi-attractors at will due to their nonlinear-ity and high dimensionality. For example, in Namikawa andTani’s method [20–22], the internal connections of an RNN

was trained with backpropagation through time (BPTT) [24];however, embedding a long-term input-output function in anRNN by the gradient descent method is generally unstable andrequires a large number of the learning epochs, as pointed outin [25]. Furthermore, their method required both a hierarchi-cal structure and the same number of separated modules as themotor primitives, restricting the scalability and the range ofits application. Methods using the associated memory model[9, 26] are also unsuitable for designing output with compli-cated spatiotemporal patterns because the embedded quasi-attractors are limited to fixed-point attractors.

In this study, we propose a novel algorithm, freely design-ing both the trajectories of quasi-attractors and transition rulesamong them in a generic setup of high-dimensional chaoticdynamical systems. Our method employs batch learning com-posed of the following three-step procedure (Fig. 1):

Step 1: Prepare a high-dimensional chaotic system and modifythe interactions (internal parameters) so that the systemreproducibly generates intrinsic chaotic trajectories (in-nate trajectories) corresponding to the type of the dis-crete inputs (named symbol). At the same time, alter thelinear regression model (named readout) to output thedesignated trajectories (output dynamics) by exploitingthe high dimensionality and nonlinearity of the internaldynamics. Our method can be applied to a wide rangeof chaotic dynamical systems since neither modules norhierarchical structures are required. Also, this embed-ding process is accomplished by modifying fewer pa-rameters, utilizing the method of reservoir computing(RC) [27, 28]. Therefore, our scheme is more stable andless computationally expensive than the conventionalmethods using backpropagation to train the network pa-rameters.

Step 2: Add a feedback classifier to the trained chaotic systemsfor generating specific symbol dynamics autonomously.In the training of the feedback discriminator, the net-work’s internal parameters are fixed, as with the read-out in step 1. Thus, by making the most of the infor-mation processing of the innate trajectory, the feedbackdiscriminator achieves multiple symbol transition ruleswith minimum additional computational capacity (i.e.,nonlinearity and memory).

Step 3: Regulate the feedback unit added in step 2 to designdesignated stochastic symbol transition rules. The de-terministic system is expected to imitate the stochas-tic process by employing intrinsic chaoticity. The sys-tem repeatedly generates the quasi-attractors embeddedin step 1 in synchronization with the pseudo-stochasticsymbol transition, meaning that the design of the de-sired CI dynamics is completed.

In this study, we demonstrate that the trajectories of quasi-attractors and their transition rules can be designed using thethree steps described above. In step 1, we show that the de-sired output dynamics can be designed with high operabil-ity by utilizing the embedded internal dynamics reproducibly

3

generated after the innate training. Next, in step 2, we demon-strate that various types of periodic symbol sequences switch-ing at a certain interval can be implemented simply by adjust-ing the parameters of a feedback loop attached to the system.Finally, in step 3, we prepare several stochastic symbol tran-sition rules governed by a finite state machine and show thatthe system can simulate these stochastic dynamics by makinguse of the system’s chaoticity. We also discuss the proposedmethod’s validity and adaptability through several numericalexperiments.

II. PROPOSED METHOD

A. System architecture

In our method, we aimed to embed M types of quasi-attractors and the transition rules among them in an RNN. Weprepared M discrete symbols s ∈ S (S := {s1, s2, · · · sM}).Each symbol corresponds to each individual quasi-attractor.We used echo state network (ESN) [29], one type of RNN,as a high-dimensional chaotic system. As shown in Fig. 1A,we prepared a generic RNN composed of a non-chaoticinput ESN (N in nodes) working as an input transient generatoras well as a chaotic ESN (Nch nodes) yielding chaotic dynam-ics. The dynamics of input ESNxin(t) ∈ RN in

and chaotic ESNxch(t) ∈ RN in

are given as the following differential equations:

τdxin

dt(t) = −xin(t) + tanh

(ginJinxin(t) + uin(s(t))

)(1)

τdxch

dt(t) = −xch(t) + tanh

(gchJchxch(t) + Jicxin(t)

)(2)

where τ ∈ R is a time constant, tanh element-wise hyperbolictangent, gin, gch ∈ R are scaling parameters, uin(s) ∈ RN in

isdiscrete input projected onto input ESN when symbol s isgiven, Jin ∈ RN in×N in

, Jch ∈ RNch×Nchare connection matri-

ces, and Jic ∈ RNch×N inis a feed-forward connection matrix

between input ESN and chaotic ESN. Each element of Jin issampled from a normal distribution N

(0, 1

N in

). Jch is a ran-

dom sparse matrix with density p = 0.1 whose elements arealso sampled from a normal distribution N

(0, 1

pNch

). We used

τ = 10.0, gin = 0.9, gch = 1.5 to make input ESN non-chaoticand chaotic ESN chaotic [30] Also, to prevent chaotic ESNfrom becoming non-chaotic due to the bifurcation caused bythe strong bias term, we tuned Jic before hand to project tran-sient dynamics converging to 0 onto the chaotic ESN when thesame symbol input continues to be given (see the Appendixfor detailed information about the transient dynamics). In anycase, the whole RNN dynamics x(t) ∈ RN in+Nch

concatenatingEqu. (1) and (2) can be represented by the following singleequation (� represents an element-wise product):

τdxdt

(t) = −x(t) + tanh (g � (Jx(t)) + u(s(t))) (3)

where x, g, J,u are defined by the following equations:

x(t) := [xin(t);xch(t)] (4)

g := [gin, · · · gin︸ ︷︷ ︸N in

gch, · · · gch︸ ︷︷ ︸Nch

]T (5)

J :=[Jin 0Jic Jch

](6)

u(s) := [uin(s);0] (7)

The output dynamics are calculated by the linear transforma-tion of the internal dynamics x(t), that is, the linear readoutwout ∈ R

N in+Nchis trained to approximate the following target

dynamics fout(t):

wToutx(t) ≈ fout(t) (8)

The symbol dynamics s(t) itself, which is externally given instep 1, is finally generated autonomously with a closed-loopsystem (Fig. 2B(2)). In the feedback loop, the following soft-max classifier fsoftmax : RN in+Nch

→ S is attached:

fsoftmax (x(t)) := arg maxs∈S

wTs x(t) (9)

where ws ∈ RN in+Nch

represents the connection matrix whoseelements are trained to approximate designated symbol dy-namics s(t) (i.e., s(t) ≈ fsoftmax(x(t))).

To summarize, we designed the desired quasi-attractors,output dynamics, and symbol dynamics by tuning the param-eters of the RNN connections J, the readout wout, and thesoftmax classifier ws, respectively.

B. First-order-reduced and controlled-error (FORCE)learning and innate training

We used two RC techniques called first-order-reduced andcontrolled-error (FORCE) learning [31] and innate training[32]. Both FORCE learning and innate training are methodsthat harness the chaoticity of the system. Below, we brieflydescribe the algorithms of both FORCE learning and innatetraining.

FORCE learning is a method that embeds designated dy-namics in a system by harnessing the chaoticity of dynamicalsystems. Suppose the following ESN dynamics with a singlefeedback loop:

τdxdt

(t) = −x(t) + tanh (gJx(t) + uz(t)) (10)

z(t) = wTx(t) (11)

Typically, the scaling parameter g is set to be greater than 1 tomake the whole system chaotic [30]. In FORCE learning, toembed the target dynamics f (t) in the system, w is trained tooptimize the following cost function CFORCE:

CFORCE := 〈‖z(t) − f (t)‖2〉 (12)

4

chaotic ESN:

input ESN:

discrete input:(s: symbol)

x in

x ch

(N in nodes, g input=0.9)

(N ch nodes, g ch=1.5)

u(s)A

B

reservoir (RNN)

readout

(linear)

symbol

sequence

A

C

B

output dynamics

A, B, C, A, C, A

B, C, B, A, B, …

, A

B C

A, B, C,

A, B, … /

symbol dynamics(periodic / stochastic)

A

C

B

output dynamics

(1) open-loop (2) closed-loop

A

B

C

readout

(linear)

reservoir (RNN)

wout wout

fsoftmax(x)

u(s)

J J

feedback

classifier

input A

Step 1 - open loop generator

input B

input C

Step 3 - chaotic itinerancy

AT=TA

BT=TB

CT=TC

stochastictransition

Step 2 - periodic dynamics

AT=TA

BT=TB

CT=TC

periodictransition

C

FIG. 1. Experimental setups. (A) Schematic diagram of a high-dimensional chaotic system prepared in our experiments. The system can bedivided into two parts: input ESN and chaotic ESN. Input ESN acts as an interface between the discrete input and the chaotic ESN, generatingtransient dynamics projecting onto the chaotic ESN when the symbol input switches. To prevent the chaotic ESN from becoming non-chaoticdue to the bifurcation, the connection between the input ESN and the chaotic ESN is trained to output transient dynamics converging to 0(see the Appendix for the detailed information about the transient dynamics). (B) Two experimental schemes. In the open-loop scheme, thesymbol input is externally given. On the other hand, in the closed-loop one, the symbol input is autonomously generated by the additionalfeedback loop. In our method, we change the elements represented by red arrows to embed desired CI dynamics. (C) Outline diagrams ofour batch learning methods composed of a three-step procedure. In step 1, the parameters of the network and readout are trained to output thequasi-attractors and the output dynamics corresponding to the symbols. In step 2 and step 3, the symbol sequence is autonomously yielded.We prepare periodic symbol transition patterns as the target in step 2 and stochastic symbol transition rules in step 3.

5

Here, the bracket denotes the averaged value over several sam-ples and trials. Especially in the FORCE learning, w is op-timized online with a least-square error algorithm. It wasreported from numerical experiments using ESN that bettertraining performance was obtained when the initial RNN wasin a chaotic regime [31].

Innate training is also a scheme for harnessing chaotic dy-namics and is accomplished by modifying the internal con-nection J using FORCE learning. The novel aspect of innatetraining is that the inner connection of ESN is trained in asemi-supervised manner, that is, the connection matrix J ofthe ESN is modified to minimize the following cost functionCinnate to reproduce the chaotic dynamics yielded by the initialchaotic RNN (xtarget(t), innate trajectory):

Cinnate := 〈‖x(t) − xtarget(t)‖2〉 (13)

Intriguingly, the innate trajectory is reproducibly generatedfor a certain period with the input while maintaining thechaoticity after the training. In other words, innate trainingis a method that allows a chaotic system to reproducibly yieldthe innate trajectory with complicated spatiotemporal patternsby applying the FORCE learning method to the modificationof the internal connection.

In this study, we propose a novel method of designing CIby employing both FORCE learning and innate training tech-niques.

C. Recipe for designing chaotic itinerancy

Our proposed method is a batch-learning scheme consistingof the following three-step process (Fig. 1C).

Step 1. Designing quasi-attractor

In step 1, the connection matrix Jch of the chaotic ESN isadjusted by innate training to design the trajectories of quasi-attractors. First, the target trajectories xs

target(t) are recordedfor M symbols under an initial connection matrix Jinit andsome initial states xs

target(0), where xstarget(t) denotes chaotic

dynamics when the symbol is switched to s at t = 0 ms (forsimplification, the switching time is fixed to t = 0 ms. in step1. Note that the symbol can be switched at any time). In step1, Jch is trained to optimize the following cost function C1-in:

C1-in :=∑s∈S

∫ Linnate

0‖xs(t) − xs

target(t)‖2dt (14)

Here, xs(t) represents the dynamics when the symbol isswitched to s at t = 0 ms, and Linnate the time period of thetarget trajectory. We only modify half the elements of Jch.The connection matrix Jch is trained for 200 epochs for eachs. We finally use Jch recording the minimum C1-in (see theAppendix for the detailed algorithm used in step 1). After theinnate training in step 1, the system is expected to reproducethe recorded innate trajectories xs

target for Linnate.

Similarly, wout is trained to produce designated output dy-namics f s(t) corresponding to symbol s. The following costfunction C1-out is optimized:

C1-out :=∑s∈S

∫ Lout

0‖ f s(t) −wT

outxs(t)‖2dt (15)

Here, note that Linnate does not always match Lout, that is, Loutcan be greater than, Linnate. The training is accomplished byan offline algorithm Ridge regression based on the recordedinternal dynamics xs(t).

Step 2. Embedding autonomous symbol transition

In step 2, we tune a feedback loop fsoftmax to achieve theautonomous symbol transition. We especially prepare targetperiodic transition rules switching every T [ms]. Suppose atarget periodic symbol dynamics sper(t). First, the networkdynamics x(t) of the open-loop setup (Fig. 1B(1)) is recordedwith a symbol dynamics sper(t) for Trec := 500, 000 ms. Basedon the recorded dataset, fsoftmax is tuned to output sper(t) fromx(t). The parameters ws of fsoftmax is trained to optimize thefollowing cost function C2:

C2 := −∑s∈S

∫ Trec

01

{sper(t) = s

}log

ewTs x(t)∑

k∈S ewTk x(t)

dt (16)

As the optimization algorithm, we use the limited-memoryBroydenFletcherGoldfarbShanno (BFGS) algorithm[33].

Step 3. Embedding stochastic symbol transition

In step 3, we implement a stochastic transition rule gov-erned by a finite state machine by modifying a feedback loopfsoftmax. As discussed in the Introduction, the chaoticity of thesystem is expected to be employed to emulate the stochasticprocess in the deterministic setup. The process of the learn-ing is same as that in step 2, that is, the pair of (x(t), ssto(t))recorded in the open-loop setup for 500,000 ms is used to trainthe fsoftmax to emulate ssto(t). Here, we use the following costfunction C3 in the training:

C3 := −∑s∈S

∫ Trec

01 {ssto(t) = s} log

ewTs x(t)∑

k∈S ewTk x(t)

dt (17)

As with the optimization of the cost function C2, C3 is opti-mized with the Limited-memory BFGS algorithm.

III. RESULTS

In this section, we show the demonstration and analytic re-sults of the numerical experiments for each step.

6

B

A

symbol input (random)

network dynamics (10 trials)

output dynamics (2-dim)

tria

ls

A: Lissajous curve B: At sign (@) C: Lorenz attractor

input ESN

chaotic ESN

−1,000 −500 0 500 1,000 1,500

before innate training

time steps [ms]

xin

xch

−1,000 −500 0 500 1,000 1,500

input ESN

chaotic ESN

after innate training

time steps [ms]

xin

xch

0 2,500 5,000 7,500 10,000 12,500 15,000 17,500A B C

time steps [ms]

xin

xch

training condition: (M, Linnate) = (1, 1,000)

training condition: (M, Linnate) = (3, 1,000)

xin

xch

xin

xch

FIG. 2. Demonstration of step 1. (A) The dynamics of the reservoir before and after the innate training. In the figure, we show the RNNdynamics trained under the condition (M, Linnate) = (1, 1, 000). The time-series data of a selected node in the input ESN is shown in the topcolumn. Conversely, the four selected dynamics of the chaotic ESN are displayed in the bottom four columns. In each column, both the innatetrajectory (black dotted) and ten individual trajectories with different initial conditions (red) are exhibited. (B) Demonstration of open-loopdynamics. The network dynamics of the RNN trained under the condition (M, Lin, Lout) = (3, 1, 000, 1, 500) is used in this demonstration. Boththe network dynamics and output dynamics of the trained readout are depicted. The readout is trained to output the Lissajous curve for symbolA, at the sign for symbol B, and the xz coordinates of the Lorenz attractor for symbol C. Note that the intervals of the symbol input wererandomly decided.

7

A

C

E

B

F

0 2,000 4,000 6,000 8,000 10,000

−10

−5

0

5

10

time steps [ms]

local Lyapnov e

xponent 1,000

2,000

3,000

4,000

5,000

target trajectory

length (Linnate [ms])

innate training performance innate training performance

timer task function timer task capacity

local Lyapnov exponent maximum Lyapnov exponent

0 2,000 4,000 6,000 8,000 10,000target pulse wave peak time (tpeak [ms])

0.0

0.2

0.4

0.6

0.8

1.0

R²

valu

e

target trajectory

length (Linnate [ms])

1,000

2,000

3,000

4,000

5,000

0 1,000 2,000 3,000 4,000 5,000 6,000 7,0000.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

capacity

# of node (Nchaos)

1,000

500250

target trajectory length (Linnate [ms])

D

norm

aliz

ed m

ean s

quare

err

or

0 1,000 2,000 3,000 4,000 5,000 6,000 7,000target trajectory length (Linnate [ms])

0.00

0.05

0.10

0.15

0.20

0.25 # of node (Nchaos)

1,000

500250

0 2,000 4,000 6,000 8,000 10,000target trajectory length (Linnate [ms])

norm

aliz

ed m

ean s

quare

err

or

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

# of target

trajectory (M)

1

2

3

4

5

0 2,000 4,000 6,000 8,000 10,000

0

2

4

6

8

10

maxim

um

Lyapnov e

xponent (×

10

-3)

# of node (Nchaos)

1,000

500250

target trajectory length (Linnate [ms])

FIG. 3. (A) Performance of innate training over M symbols. The normalized mean square errors (NMSEs) are calculated from the ten trials.(B) Effect of network size Nch on the performance of innate training. (C) Evaluation of the temporal information capacity with timer task. Theaveraged values for ten trials are plotted. (D) Effect of the system size on timer task capacities. Timer task capacity is defined as the integralvalue of the timer task function. (E) Evaluation of the local Lyapunov exponent (LLE). The LLE is measured with the time development ofthe perturbation of the chaotic ESN (see the Appendix for detailed information about the calculation method of the LLE). (F) Evaluation ofthe system’s maximum Lyapunov exponent (MLE).

8

Step 1. Designing quasi-attractor

As discussed in the previous section, the internal connec-tion of the chaotic ESN Jch is trained to output the correspond-ing innate trajectories reproducibly to the symbol switching.Fig. 2A demonstrates the change of the network dynamics ofa 1,500-node RNN (N in = 500, Nch = 1, 000) whose connec-tion matrix is modified with innate training under the condi-tion (M, Linnate) = (1, 1, 000). The trajectory quickly spreadsbefore t = Linnate in the pre-trained system, whereas the tar-get trajectory xtarget

s (dashed line) is reproducibly yielded for1,000 ms (covered by the yellow rectangle) in the post-trainedsystem. Moreover, intriguingly, the dispersion of the trajecto-ries continues to be suppressed even after t = Linnate.

Next, Fig. 2B displays both the network dynamics and theoutput dynamics. The 1,500-node RNN (N in = 500, Nch =

1, 000) trained under the condition (M, Linnate) = (3, 1, 000)was used. At first, the symbol input was absent, and thensymbols were switched with random intervals from the mid-dle. Also, the 2-dim readout was trained to output the Lis-sajous curve for symbol A, at the sign for symbol B, andthe xz coordinates of the Lorenz attractor for symbol C forLout = 1, 500 ms. It was observed that the desired spa-tiotemporal patterns were stably and reproducibly generatedfor a certain period in every trajectory with different initialstates after the symbol transition (see supplementary video 1).Note that the same linear model wout was used in the demon-stration, implying that the trajectory of each quasi-attractorhas rich enough information to output the designated time-series patterns independently even with the single linear re-gressor. Our scheme for designing transient dynamics wouldbe highly useful in the field of robotics because the processin step 1 is easily achieved by adjusting the partial elementsof a high-dimensional chaotic system. For example, the sys-tem working in a real-world environment should immediatelyand adaptively switch its motion according to the change ofenvironmental input like a system developed by Ijspeert et al.[34], which can be easily accomplished by our computation-ally cheap method. In this way, our method would work ef-fectively in the context of robotics, where fast responsivenessand adaptability are required.

We also examined both the scalability and the validity of in-nate training in detail through several numerical experiments(Fig. 3). First, we examined the relationship between the num-ber of input symbols M and the accuracy of innate training. Toevaluate the performance of innate training, we used the nor-malized mean square error (NMSE) between the output andthe innate trajectory xs

target represented by the following for-mula:

NMS E :=1M

∑s∈S

⟨∫ Linnate

0 ‖xs(t) − xstarget(t)‖

2dt∫ Linnate

0 ‖xstarget(t)‖2dt

⟩(18)

We calculated the NMSE for ten trials. Fig. 3A shows theinnate training performances with the different training con-ditions, suggesting that NMSEs are more likely to increasewith a longer target trajectory and a larger number of sym-bols. This result implies that innate training has its limitation

in the design of the quasi-attractors. We also examined theeffect of network size on the capability to embed the quasi-attractors. We investigated the relationship between the num-ber of nodes in the chaotic ESN Nc and the accuracy of innatetraining under the condition M = 1 ( Fig. 2B), suggesting thatthe NMSEs were less likely to increase with a larger network.To summarize, our analysis indicates that longer trajectoriescan be embedded in a larger network by innate training.

Next, we evaluated the effect of innate training on the ca-pacity of the system’s information processing. We prepareda timer task and measured how long the inputted informationwas stored in the RNN. In the timer task, the pulse-like wavewith a peak tpeak [ms] after the symbol transition was pre-pared as the target, and the performance was defined as theaccuracy of the pulse-like wave reconstruction by a trainedreadout. Here, we defined the R2 value between the outputand the pulse-like wave as the timer task function R2(tpeak).At the same time, we also calculated the integral value of thetimer task function

∫ ∞0 R2(t)dt and define it as the timer task

capacity (see the Appendix for detailed information about thesetup of the timer task). Fig. 4C shows the timer task func-tion with different innate training conditions, indicating thatRNNs trained with the longer-length target trajectory Linnateperform better. It was also observed that the timer task ca-pacity saturated around Linnate = 5, 000 ms in the 1,000-nodeRNN, and the border of the saturation decreased in a smallersystem (Fig. 3D). These results imply that the temporal infor-mation capacity of the system is improved by innate trainingwith the longer target length Linnate but saturates at a certainvalue, which is determined by the system size.

Furthermore, we assessed the effect of innate training onthe system’s chaoticity by measuring the Lyapunov exponentsof the system. Since the transition among quasi-attractors isdriven by the system’s chaoticity, it is necessary to keep thesystem chaotic. In this experiment, we measured the localLyapunov exponent (LLE) to evaluate the degree of trajec-tory variation after the symbol switching. We also measuredthe maximum Lyapunov exponent (MLE) without any inputs(u(t) = 0) to estimate the global chaoticity of the system (seethe Appendix for the detailed calculation algorithm of both theLLE and MLE). Fig. 4E displays the LLE values of the sys-tems with the different target trajectory length Linnate, suggest-ing that the trajectories unevenly expand after the symbol tran-sition. In particular, it was observed from the LLE analysisthat contracting regions existed (regions with negative LLEscorresponding to the lengths of the quasi-attractors) causedby the transient dynamics projected by the input ESN, and thedegree of the expansion became gradual in the trained periodt ∈ [0, Linnate). These results imply that innate training yieldsa locally contractive phase space structure, that is, a quasi-attractor. Moreover, positive MLE values were constantly ob-tained from the MLE analysis depicted in Fig. 3F, supportingthe conjecture that the system chaoticity was maintained espe-cially well with the larger RNNs even after the innate training.(Note that a sharp increase in MLE was observed with shorterLinnate, which is caused by the increase in the spectral radiusof the connection matrix J of the system. See the Appendixfor detailed information of the analysis.)

9

A B

C

output dynamics

network dynamics

(M, Linnate, Lout) = (10, 1,000, 500)

“A-B-C-D-E-F-G-H-I-J” switching every 500 ms

output dynamics

symbol dynamicssymbol dynamics

D

tria

ls

0 5,000 10,000 15,000time steps [ms]

A B C A B C A B C A

x-axis

periodic pattern 1 (A-B-C-B) periodic pattern 2 (A-B-A-B-C)

perturbated trajectory

original trajectory

−5,000 0 5,000 10,000 15,000

perturbation (t = 0)

time steps [ms]

x-a

xis

y-a

xis

perturbed trajectory

original trajectory

enlarged view

time steps

0 5,000 10,000time steps [ms]

15,000

A B A B C A B A B C

tria

ls

A B C B A B C B A B

0 5,000 10,000time steps [ms]

15,000

tria

ls

tria

ls

0 4,000 8,000 12,000time steps [ms]

A B C D E F G H I J A B C D E F G H I J A B C D E FG H I J

(M, Linnate, Lout) = (3, 1,000, 1,500)

“A-B-C” switching every 2,000 ms

y-a

xis

FIG. 4. Demonstrations of closed-loop dynamics in step 2. (A) Three-symbol periodic transition. We prepared an RNN trained under thecondition (M, Linnate) = (3, 1, 000) and a readout trained under the condition Lout = 1, 500 ms to output three Lissajous curves correspondingto the symbol input. The feedback loop fsoftmax realizes the periodic symbol transition A-B-C switching at 2,000-ms intervals. (B) Ten-symbolperiodic transition. We prepared an RNN trained under the condition (M, Linnate) = (10, 500) and a readout trained under the conditionLout = 500 ms to output ten different Lissajous curves corresponding to the symbol input. The feedback loop fsoftmax achieves the periodicsymbol transition A-B-C-D-E-F-G-H-I-J switching at 500-ms intervals. (C) Demonstration of the tasks requiring higher-order memory to besolved. The same RNN was used in the demonstration of (A). The left panel displays the periodic symbol transition pattern A-B-C-B switchingat 2,000-ms intervals. The right one demonstrates the periodic symbol transition pattern A-B-A-B-C switching at 2,000-ms intervals. Thesetasks were accomplished in the same way in demonstrations (A) and (B), that is, only the parameters in fsoftmax were tuned. (D) Two outputdynamics: original trajectory and perturbed trajectory. A small perturbation was given to the original trajectory at t = 0 ms.

10

Step 2. Periodic symbol transition

In step 2, the system autonomously generates a symbol se-quence externally given in step 1. The additional feedbackloop realizes the autonomous periodic switching of the sym-bols. We demonstrate that various types of periodic sym-bol sequences switching at a fixed interval can be easily de-signed simply by modifying the parameter of the feedbackloop fsoftmax. Fig. 4A demonstrates the embedding of the pe-riodic symbol sequence A-B-C (2,000-ms interval and 6,000-ms period) with a trained RNN ((M, Linnate) = (3, 1, 000)).Fig. 4A also exhibits the embedding of the periodic symbolsequence A-B-C-D-E-F-G-H-I-J (500-ms interval and 5,000-ms period), with the same RNN used as the demonstrationin Fig. 4A. In both demonstrations, the system succeeded notonly in generating the desired symbol transition rules but alsoin stably outputting the designated output dynamics with highaccuracy.

We also show that the system can solve tasks requiringhigher-order memory in the same scheme. We prepared thetwo periodic symbol sequences A-B-C-B and A-B-C-B-A.These two symbol sequences are more difficult to embed be-cause the system must change the output according to the pre-vious output. In the symbol transition A-B-C-B, for example,the system must output the next symbol depending on the pre-vious symbol when switching from B, though the total numberof symbols is the same as in the task A-B-C. We used the sameRNN and setup used in the Fig. 4A and only changed the pa-rameters in ffeedback to realize the symbol transitions. Fig. 4Cdisplays the network dynamics and symbol transition of thetwo tasks, showing that the system successfully achieves boththe periodic sequence A-B-C-B with an 8,000-ms period andA-B-A-B-C with a 10,000-ms period. These results suggestthat the trained RNN had the higher-order memory capacity,that is, the generated trajectories have sufficient separability todistinguish the contextual situation depending on the previoussymbol sequence (see supplementary video 2). In robotics,periodic motion control has often been implemented by anadditional oscillator (e.g., a central pattern generator) to yieldlimit cycles [23, 34–36]. Our method in step 2 would be usefulin designing limit cycles with longer periods and more com-plicated patterns.

We also analyzed the effect of perturbation to investigatethe stability of the embedded symbol transition. Fig. 4Dshows the output dynamics of both the original and perturbedtrajectories, clarifying that the trajectory returned to the orig-inal one after the addition of the perturbation. We also cal-culated the MLE values of the system and obtained the value−1.89 × 10−4, which was very close to zero. These analysesindicate that the trained feedback loop finnate made the sys-tem non-chaotic, that is, the generated internal dynamics wasa limit-cycle.

Step 3. Stochastic symbol transition (chaotic itinerancy)

In step 1, we constructed the trajectories of the quasi-attractors and the corresponding output dynamics. In step

2, we showed that periodic transitions among quasi-attractorscan be freely designed by simply tuning the feedback loopffeedback. In step 3, we realize a stochastic transition, that is,CI. As discussed above, the system is expected to employ itschaoticity to emulate a stochastic transition in deterministicdynamical systems.

First, we demonstrate that stochastic transition can be freelydesigned by adjusting ffeedback (see Fig. 5A and supplementaryvideo 3). In this demonstration, we used the same RNN asin Fig. 4A. We prepared a symbol transition rule uniformlyswitching among symbols A, B, and C at 3,000-ms inter-vals. Fig. 5B shows the symbol dynamics, network dynamics,and output dynamics, suggesting that the symbol transitionsstarted to spread at around t = 10, 000 ms and finally settledown to completely different transition patterns. Neverthe-less, the system continued to generate Lissajous curves stably.These demonstrations imply that the system constantly repro-duced quasi-attractors embedded by innate training, while thequasi-stochastic transition was achieved by the global chaotic-ity.

To analyze the flexibility of our method, we measured thestochastic transition matrix and the average symbol intervals(Fig. 5B). We prepared two stochastic symbol transition rulesas the targets: the transition rule governed by the uniform fi-nite state machine (pattern 1) and the transition rule governedby the finite state machine with a limited transition (pattern2). Note that we used the same trained RNN as in the demon-stration in step 2 and embedded the transition rules simplyby adjusting fsoftmax. Fig. 5B shows the results of the ob-tained trajectories, implying that the system successfully em-bedded patterns similar to the target rules, although there weresome errors and variations in the transition probability andthe switching time. The positive MLEs were obtained in bothcases (+2.01 × 10−3 in pattern 1, and +1.71 × 10−3 in pattern2), suggesting that the system was weakly chaotic as a whole.

Finally, we analyzed both the structures of the obtainedchaotic attractors and the symbol dynamics in detail. Fig. 6Ashows the effect of small perturbations on the symbol dynam-ics, implying that the patterns of symbol dynamics varied af-ter a certain period. To analyze the structural change of theterminal symbol state, we measured the symbol dynamics ac-companied by the temporal development of the set of initialstates on a plane constructed by the two selected dimensions(Fig. 6B), clarifying that a complex terminal symbol structureemerges after a certain period (Fig. 6B). Especially in the em-bedding of the pattern 1 rule, the entropy of the terminal sym-bol pattern converges to a value close to the maximum entropylog2 39 ≈ 14.26 (note that the entropy was measured based onthe probability distribution constructed by the frequency of3 × 3 grid patterns). These results indicate that the symboltransition dramatically changed even with a small perturba-tion and was unpredictable after a certain period, that is, theprediction of symbol dynamics required the complete obser-vation of the initial state value and calculation of the temporaldevelopment with infinite precision.

11

AT=3,000

BT=3,000

CT=3,000

0.5

0.5

0.5

0.5

0.5

0.5

AT=3,500

BT=2,000

CT=2,000

1.0

0.5 0.5

0.5

1.0

symbol dynamics (10 trials)

output dynamics (2-dim)

B

A

network dynamics (10 trials)

stochastic pattern 1

stochastic pattern 2

tria

ls0 5,000 10,000 15,000 20,000

time steps [ms]

A B C0

1,000

2,000

3,000

4,000

pe

rio

d

A B C

A

B

C

0.00 0.63 0.37

1.00 0.00 0.00

1.00 0.00 0.00

0.0

0.2

0.4

0.6

0.8

1.0

A B C

A

B

C

0.00 0.45 0.55

0.60 0.00 0.40

0.51 0.49 0.00

0.0

0.2

0.4

0.6

0.8

1.0

A B C0

1,000

2,000

3,000

pe

rio

d

4,000

t ∈ [0, 2,000) t ∈ [8,500, 10,500) t ∈ [22,000, 24,000)

xin

xch

training condition: (M, Linnate, Lout) = (3, 1,000, 1,500)

xin

xch

xin

xch

FIG. 5. Demonstration of step 3. (A) Network dynamics with ffeedback trained to imitate a stochastic transition rule. We used an RNNtrained under the condition (M, Linnate) = (3, 1, 000), and readout trained to output Lissajous curves under the condition Lout = 1, 500 ms.The feedback classifier fsoftmax was trained to uniformly switch the symbol among the three symbols A, B, and C at 3,000-ms intervals. Tendifferent trajectories with small perturbations are overwritten in the figure. (B) Evaluation of the embedding performance of a stochasticsymbol transition. Two different stochastic symbol transition rules (patterns 1 and 2) were prepared as the target. The same RNN was used asin the demonstration of (A). The middle figures show the obtained probability density matrix, and the right ones show the average switchingduration (the error bar represents standard deviation).

12

A B

stochastic pattern 2

tria

ls (

N=

50

)

stochastic pattern 1

0 20,000 40,000

time steps [ms]

(i) t = 0 (ii) t = 6,500 (iii) t = 10,000 (iv) t = 50,000

0 10,000 20,000 30,000 40,000 50,0000

10

(iii) (iv)(ii)(i)

entropy of neighboring pattern

tria

ls (

N=

50

)

time steps [ms]

pattern 1

pattern 2

pa

tte

rn 2

pa

tte

rn 1

x2-a

xis

x1-axis

FIG. 6. Analysis of symbol dynamics and the final state. (A) Effect of a small perturbation on the terminal symbol dynamics. We evaluatedthe two closed-loop set-ups prepared in Fig. 5B. The figures display the symbol dynamics generated by 50 trajectories with 50 different initialvalues. (B) Analysis of symbol dynamics generated by the temporal development of the initial states on a small plane and its entropy of thesymbol pattern. Two dimensions (x1, x2) on the phase space were selected from the chaotic ESN to construct the plane. We observed thesymbol dynamics generated by the temporal development of the states on the plane. To evaluate the randomness of the obtained pattern, wecalculated the entropy of the obtained symbolic pattern based on the probability distribution constructed from the 3× 3 grid patterns. Note thatthe horizontal dotted line shows the maximum entropy (log2 39 ≈ 14.26).

IV. DISCUSSION

In this study, we proposed a novel method of designing CIbased on RC techniques. We also showed that the varioustypes of output dynamics and symbol transition rules could bedesigned with high operability simply by adjusting the par-tial parameters of a generic chaotic system with our three-step recipe. In this section, we discuss the scalability of ourmethod and the mechanism of how CIs are successfully em-bedded by reviewing several numerical analyses that verifythe validity of innate training.

First, the results of the innate training performances dis-played Fig. 3A and B indicate that the number of RNN nodesconstrains the total length of the quasi-attractors that can beembedded in the system by the innate training. However, theLLE analyses in Fig. 3E show that the system has the ex-panded region of the negative LLE even when the NMSE be-tween the innate trajectory and the embedded trajectory be-comes large (e.g., Linnate = 5, 000 ms). These results im-ply that even when the innate trajectories are not success-fully embedded in the system, the system stably yields high-dimensional trajectories with complicated spatiotemporal pat-terns for each symbol transition over Linnate, caused by theweakening of the system chaoticity. In fact, the same RNN

trained under the condition (M, Linnate) = (3, 1, 000) wasrepeatedly used in our series of demonstrations, the desiredoutput dynamics (e.g., the Lissajous curves) being constantlygenerated for Lout = 1, 500 ms periods after the symbol shift(Figs. 2, 4 and 5). We also demonstrated that the system canautonomously generate a symbol transition rule with an in-terval greater than Linnate (Figs. 4 and 5), suggesting that thesystem exploited high-dimensional reproducible trajectorieslonger than Linnate.

Moreover, it is assumed that the length of the quasi-attractors constrains the target stochastic transition rules thatcan be embedded. Indeed, the system failed to imitate thestochastic transition, and the transition became periodic whenthe target transition had a shorter switching interval, whereasthe training of ffeedback became unstable when it had a longerswitching interval. These results suggest that the followingtwo mechanisms should be required in the design of CI in ourmethod: (i) the differences among the trajectories are suffi-ciently enlarged through the temporal development to realizethe stochastic symbol transition, and (ii) a similar spatiotem-poral pattern should be reproducibly yielded until the switch-ing moment to discriminate the switching timing precisely.These two mechanisms are contradictory, of course, and thedesired CI can likely be embedded when both conditions aremoderately satisfied.

13

Next, we discuss the effectiveness and significance of ourmethod from the viewpoint of robotics. It should be notedthat both the trajectory of the quasi-attractor and its transitionrule are freely designed in a high-dimensional chaotic sys-tem by adjusting the reduced number of parameters and uti-lizing the intrinsic high-dimensional chaos. This flexibility isunavailable in the conventional heuristic methods of CI, andour method offers a novel methodology for designing CI ona wider range of chaotic dynamical systems, including real-world robots. Our cheap training method would also be use-ful in designing more sophisticated agents, such as animals.For example, recent physiological studies on the motor cortex[37, 38] support the conjecture that a large variety of behav-iors can be instantaneously generated by the partial plasticityof the nervous system, indicating that adjustments of entireneural circuits are not necessary. In this point, our compu-tationally inexpensive method of reusing and fine-tuning thetrained model would be especially helpful in the context ofbio-inspired robotics, where fast responsiveness and the real-time processing are required.

Another advantage of our method is that it does not requirethe explicit structure of dynamical systems. For example, inthe method proposed by Namikawa and Tani [20–22], thecontroller needs a fixed hierarchical structure and modular-ity. Therefore, the trained controller was specialized in imple-menting a specific behavior, making it difficult to divert it forany other purpose. In contrast, we proposed a method of de-signing CI with a generic setup consisting of a single chaoticESN, auxiliary symbols, and an interface between them (in-put ESN) with high scalability. Thus, our method allows usto design the various trajectories and their transition rules ina single high-dimensional chaotic system and thus greatly ex-pands the scope of application of high-dimensional chaoticdynamical systems. Especially, neuromorphic devices basedon physical reservoir computing framework would be an ex-cellent candidate for implementing our scheme. Sprintronicsdevices, for example, are recently shown to exhibit chaoticdynamics [39, 40] and are actively exploited as physical reser-voirs [41–43]. We expect that this framework would provideone of the promising application scenarios for real-world im-

plementations of our scheme.Our method is also scalable to autonomous symbol gener-

ation required in more advanced functionality. For example,in our method, M kinds of auxiliary symbols are given in ad-vance. However, in a highly autonomous system, such as hu-mans, symbols are dynamically generated and destroyed dueto developmental processes. As demonstrated by Kuniyoshiet al. [17], such self-organizing symbol dynamics can be real-ized by providing an additional automatic labeling mechanismin the system. In other words, it is possible to generate sym-bols spontaneously by embedding an unsupervised learningalgorithm in the system; this is a subject for future work.

Finally, the dynamic phenomena obtained by our methodare significant from the viewpoint of high-dimensional dy-namical systems. As shown in Fig. 6, we demonstrated thatsmall differences in the initial network state were expandedby the chaoticity of the system, which eventually led to dras-tic change in both the global symbol transition pattern s(t) andthe local dynamicsx(t). Such tight interaction between micro-layer and macro-layer is a phenomenon unique to determin-istic dynamical systems; that is, it cannot occur in principlein a system where the higher-order mechanism is completelyseparated from the lower-order one (e.g., independent randomvariables). Also, the global characteristics of dynamical sys-tems are often analyzed by the mean-field theory. However,the analysis by the mean-field approximation cannot capturethe contribution of microscopic dynamics to the macroscopicchange. Thus, our CI design method has a meaningful rolein shedding light on the interaction between micro and macrodynamics in deterministic chaotic dynamical systems.

ACKNOWLEDGMENTS

This work was based on results obtained from a projectcommissioned by the New Energy and Industrial TechnologyDevelopment Organization (NEDO). K.N was supported byJSPS KAKENHI Grant Numbers JP18H05472 and by MEXTQuantum Leap Flagship Program (MEXT Q-LEAP) GrantNumber JPMXS0118067394.

[1] R. Pfeifer and C. Scheier, Understanding intelligence (MITpress, 2001).

[2] T. Flash and B. Hochner, Current opinion in neurobiology 15,660 (2005).

[3] R. D. Beer, Artificial intelligence 72, 173 (1995).[4] H. Jaeger, Identification of behaviors in an agent’s phase space

(Citeseer, 1995).[5] K. Ikeda, K. Otsuka, and K. Matsumoto, Progress of Theoreti-

cal Physics Supplement 99, 295 (1989).[6] K. Kaneko, Physica D: Nonlinear Phenomena 41, 137 (1990).[7] I. Tsuda, World Futures: Journal of General Evolution 32, 167

(1991).[8] I. Tsuda, E. Koerner, and H. Shimizu, Progress of Theoretical

Physics 78, 51 (1987).[9] M. Adachi and K. Aihara, Neural Networks 10, 83 (1997).

[10] I. Tsuda, Behavioral and Brain Sciences 24, 793 (2001).[11] I. Tsuda, Current opinion in neurobiology 31, 67 (2015).[12] W. J. Freeman, Biological cybernetics 56, 139 (1987).[13] A. Luczak, P. Bartho, and K. D. Harris, Neuron 62, 413 (2009).[14] T. Kurikawa and K. Kaneko, PLoS computational biology 9,

e1002943 (2013).[15] H. Suetani, in International Conference on Artificial Neural

Networks (Springer, 2019) pp. 76–81.[16] Y. Kuniyoshi and S. Suzuki, in 2004 IEEE/RSJ International

Conference on Intelligent Robots and Systems (IROS)(IEEECat. No. 04CH37566), Vol. 2 (IEEE, 2004) pp. 2042–2049.

[17] Y. Kuniyoshi and S. Sangawa, Biological cybernetics 95, 589(2006).

[18] T. Ikegami, Journal of Consciousness Studies 14, 111 (2007).[19] J. Park, H. Mori, Y. Okuyama, and M. Asada, PloS one 12,

14

e0182518 (2017).[20] J. Namikawa and J. Tani, Neural Networks 21, 1466 (2008).[21] J. Namikawa and J. Tani, Neural Networks 23, 625 (2010).[22] J. Namikawa, R. Nishimoto, and J. Tani, PLoS computational

biology 7, e1002221 (2011).[23] S. Steingrube, M. Timme, F. Worgotter, and P. Manoonpong,

Nature physics 6, 224 (2010).[24] P. J. Werbos, Proceedings of the IEEE 78, 1550 (1990).[25] Y. Bengio, P. Simard, P. Frasconi, et al., IEEE transactions on

neural networks 5, 157 (1994).[26] M. Oku and K. Aihara, Nonlinear Theory and Its Applications,

IEICE 2, 508 (2011).[27] W. Maass, T. Natschlager, and H. Markram, Neural computa-

tion 14, 2531 (2002).[28] H. Jaeger and H. Haas, science 304, 78 (2004).[29] H. Jaeger, Bonn, Germany: German National Research Cen-

ter for Information Technology GMD Technical Report 148, 13(2001).

[30] H. Sompolinsky, A. Crisanti, and H.-J. Sommers, Physical re-view letters 61, 259 (1988).

[31] D. Sussillo and L. F. Abbott, Neuron 63, 544 (2009).[32] R. Laje and D. V. Buonomano, Nature neuroscience 16, 925

(2013).[33] R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, SIAM Journal on

Scientific Computing 16, 1190 (1995).[34] A. J. Ijspeert, A. Crespi, D. Ryczko, and J.-M. Cabelguen, sci-

ence 315, 1416 (2007).[35] C. Liu, D. Wang, and Q. Chen, IEEE Transactions on Systems,

Man, and Cybernetics: Systems 43, 1206 (2013).[36] D. Owaki, T. Kano, K. Nagasawa, A. Tero, and A. Ishiguro,

Journal of The Royal Society Interface 10, 20120669 (2013).[37] J. P. Stroud, M. A. Porter, G. Hennequin, and T. P. Vogels,

Nature neuroscience 21, 1774 (2018).[38] M. G. Perich, J. A. Gallego, and L. E. Miller, Neuron 100, 964

(2018).[39] T. Taniguchi, N. Akashi, H. Notsu, M. Kimura, H. Tsukahara,

and K. Nakajima, Physical Review B 100, 174425 (2019).[40] T. Yamaguchi, N. Akashi, K. Nakajima, S. Tsunegi, H. Kubota,

and T. Taniguchi, Physical Review B 100, 224422 (2019).[41] J. Torrejon, M. Riou, F. A. Araujo, S. Tsunegi, G. Khalsa,

D. Querlioz, P. Bortolotti, V. Cros, K. Yakushiji, A. Fukushima,et al., Nature 547, 428 (2017).

[42] T. Furuta, K. Fujii, K. Nakajima, S. Tsunegi, H. Kubota,Y. Suzuki, and S. Miwa, Physical Review Applied 10, 034063(2018).

[43] S. Tsunegi, T. Taniguchi, K. Nakajima, S. Miwa, K. Yakushiji,A. Fukushima, S. Yuasa, and H. Kubota, Applied Physics Let-ters 114, 164101 (2019).

[44] I. Shimada and T. Nagashima, Progress of theoretical physics61, 1605 (1979).

Algorithm 1 Innate training1: for i ∈ A do2: Pi ← I3: for s ∈ S do . generating innate trajectory4: initializing xs

target(0) . washing out5: recording xs

target(t) (t ∈ [0, Linnate))

6: for epoch = 1 to 200 do . innate training7: for s ∈ S do8: initializing xs(0) . washing out9: t ← 0

10: count← 011: while t < Linnate do12: for s ∈ S do13: if count = 0 (mod 2) then14: continue . avoiding redundant sampling15: e← xs(t) − xs

target(t)16: for i ∈ A do17: for j ∈ B(i) do18: Ji j ← Ji j − ei

∑k∈B(i) Pi

jkxsk(t)

19: for k ∈ B(i) do20: Qi

jk ← Pijk −

∑m∈B(i)

∑n∈B(i) Pi

jmxsm(t)xs

n(t)Pink

1+∑

m∈B(i)∑

n∈B(i) xsm(t)Pi

mn(t)xsn(t)

21: Pi ← Qi

22: t ← t + ∆t23: count← count + 1

Appendix A: Algorithm of the innate training

The connection matrix J in step 1 is trained by algorithmshown in algorithm 1, where A is a set of the selected nodesin the chaotic ESN (|A| = Nc/2), and B(i) is a set of nodesprojected to node i among the nodes of the chaotic ESN. Inthis study, ∆t = 1 ms was constantly used.

Appendix B: Projected dynamics onto chaotic ESN

To prevent chaotic ESN from becoming non-chaotic due tothe bifurcation caused by the strong bias term, we tuned thefeed-forward connection matrix Jic between input ESN andchaotic ESN before hand to project the following transient dy-namics converging to 0 onto the chaotic ESN when the samesymbol input continues to be given:

Jicxin(t − tswitch) ≈[(t − tswitch) · exp

(−

12

( t − tswitch

50

)2)]vs

(B1)

where the elements of vs ∈ RNch

(corresponding to symbols) are also sampled from a normal distribution N(0, 1). Thistransition dynamics can be replaced with any other dynamicsconverging to 0.

Appendix C: Distribution of eigenvalues after the innatetraining

Fig. S1A shows the effect of innate training on the distribu-tion of eigenvalues under the condition M = 1. The spectral

15

−2 0−2 0−2 0

−2

−1

0

1

2

5002,0008,000

target trajectory

length (Linnate [ms])

0 500 1,000 1,5000

1

2

3S

pe

ctr

al ra

diu

s

Linnate = 500 Linnate = 2,000 Linnate = 8,000

index of eigen value (sorted by the spectral radius)

target time range [0, tpeak+50]

time steps [ms]t = 0 t = tpeak

u(s)

d(t)

0

symbol dynamics

desired output

s(t) spost (≠ spre)

discrete input

spre

u(spost)

u(spre)

BA

FIG. S1. (A) Distribution of eigenvalue and spectral radius of J after the innate training. The results with three different length of the innatetrajectories (Linnate = 500 ms, 2, 000 ms, 8, 000 ms) are displayed. (B) Schematic graph of the timer task. The readout is trained to output apulse wave with a delay tpeak [ms] after symbol shift.

radius greatly exceeds 1 when Lin is small. Conversely, theeigenvalues fit within the unit circle as Lin becomes longer.This result corresponds to the result in Fig. 3F.

Appendix D: Timer task setup

The timer task evaluates the temporal information-processing capability of the system. We evaluate how suc-cessfully the system can reconstruct a pulse-like wave with apeak with a delay tpeak [ms] using only the linear transforma-tion of the internal state x (Fig. S1B); that is, the readoutwoutis trained to approximate a pulse-like wave represented by thefollowing equation:

wToutx(t) ≈ exp

((t − tpeak)2

2 × 102

)(D1)

Note that wout is tuned by Ridge regression with the Ridgeparameter α = 1.0. The task performance is calculated byaveraging ten R2 values between output and target, the R2

value between x and y defined by Cor(x,y)2

σ2xσ

2y

. We express the

task performance as R2(tpeak) (timer task function). Also, thetimer task capacity is defined as the integral of R2(tpeak) (i.e.,∫ ∞

0 R2(t)dt).

Algorithm 2 Maximum Lyapunov exponent1: initializing x(0) ∈ RN . reference trajectory2: sampling ε ∈ RN (ε ∼ N(0, 1))3: ε← lpert

ε‖ε‖

. normalizing sampled perturbation4: initializing y(0) := x(0) + ε . perturbed trajectory5: t ← 06: Lyap = {φ} . set of sampled MLE7: while t < Thorizon do8: Lyap← Lyap ∪

{1

∆T log(‖y(t+∆T )−x(t+∆T )‖

‖y(t)−x(t)‖

)}9: t ← t + ∆T

10: y(t)← x(t) + lperty(t)−x(t)‖y(t)−x(t)‖ . recomputing y(t)

11: calculating MLE by averaging values in Lyap

Appendix E: Maximum Lyapunov exponent

MLE is calculated based on [44]. The algorithm is rep-resented in algorithm 2. In this experiment, we used ∆T =

1, 000 ms, Thorizon = 1, 000, 000 ms, lpert = 10−6 and measuredthe average of ten trials.

Appendix F: Local Lyapunov exponents

LLE with symbol input s is defined by the following for-mula:

LLE(t) :=⟨log

‖xspert(t) − x

s(t)‖

‖xspert(0) − xs(0)‖

⟩ (F1)

where xs represents the internal dynamics with the symbol in-put s starting at time t = 0 ms, xs

pert represents the perturbedtrajectories, a small perturbation being added to the originaltrajectory xs at t = 0 ms (xs

pert(0) = xs(0) + ε, ‖ε‖ = 10−6).The bracket represents an average over several trials. We mea-sured it with ten trials.