Learning the Perceptual Conditions ofSatisfaction of Elementary Behaviors

Matthew Luciw∗, Konstantin Lahkman†, Sohrob Kazerounian∗,Mathis Richter‡, and Yulia Sandamirskaya‡

∗The Swiss AI Lab IDSIA, USI & SUPSI, Galleria 2, Manno CH-6928, Switzerland†NBIC-Centre, National Research Center, Kurchatov Institute, Moscow, Russia‡Ruhr-Universitat Bochum, Institut fur Neuroinformatik, Bochum, Germany

Abstract—A core requirement for autonomousrobotic agents is that they be able to initiate actions toachieve a particular goal and to recognize the resultingconditions once that goal has been achieved. Moreover,if the agent is to operate autonomously in complexand changing environments, the mappings between in-tended actions and their resulting conditions must belearned, rather than pre-programmed. In the presentwork, we introduce a method in which such mappingscan be learned within the framework of Dynamic FieldTheory. We not only show how the learning processcan be implemented using dynamic neural fields, butshow how the adaptive architecture can operate onreal-world inputs while controlling the outgoing motorcommands. The proposed method extends a recentlyproposed neural-dynamic framework for behavioralorganization in cognitive robotics.

I. INTRODUCTION

Complex behaviors performed by cognitiverobotic agents may be segregated in a number ofelementary behaviors (EBs), performed simultane-ously and/or sequentially. Each EB links the agent’sperceptual system to its motor system in a behavior-specific manner [3]. Complex actions require thecoordination between a number of simpler EBs,such that each EB is activated in the appropriateorder, persists as long as necessary in order toachieve its behavioral subgoal, and is ultimatelydeactivated when this goal is achieved. We haverecently introduced a neural-dynamic framework, inwhich such an organization of the robot’s behaviors,composed of structurally simpler EBs, was realized[8]. In this framework, each EB consists of twocoupled dynamical structures, which determine anelementary behavior’s intention and its condition

of satisfaction (CoS). The intention comprises be-havioral parameters, which eventually bring abouteither an overt or an internal action, whereas thecondition of satisfaction is a perceptual indicator,by which the agent recognizes the completion ofthe EB. We have previously demonstrated howsequences of goal-directed actions may be gener-ated in this framework by linking a neural-dynamicarchitecture for behavioral organization to sensorsand motors of a Nao robot [9, 8]. We have alsodemonstrated how sequences of EBs may be learnedfrom delayed rewards by combining the neural-dynamic architecture with reinforcement learningand making use of eligibility traces [7]. However,the structure of an EB, i.e. the coupling structurebetween the intention and the CoS of a behavior inthese prior models was hand-designed as part of thearchitecture. In the present work, we demonstratehow this coupling may emerge autonomously inthe framework of Dynamic Field Theory, using anassociative learning rule, along with a set of built-in internal drives, or needs, and a scalar rewardinginput when the need is satisfied.

II. METHODOLOGICAL BACKGROUND

A. Dynamic Field Theory

Dynamic Field Theory (DFT; [12]) is a mathe-matical framework, which has been used to modelneural-dynamic processes that underly behavior.DFT architectures are well suited for robotic controlsystems due to their ability to form and stabilizerobust categorical outputs from noisy, dynamical,and continuous real-world input. DFT has been

applied in robotics at different levels, from low-levelnavigation dynamics with target acquisition basedon vision [2] to object representation, dynamicscene memory, and spatial language [11], as wellas sequence generation [10].

The basic computational unit in DFT is a dy-namic neural field (DNF). DNFs represent activa-tion distributions of neural populations, as opposedto classical neural network architectures whosecomputational units are at the level of individualneurons. A DNF’s activation is defined over con-tinuous dimensions (e.g., color or space), whichcharacterize sensorimotor systems and task spaceof the agent. This activation develops in continuoustime based on a dynamical equation analyzed byAmari [1]. As a result of non-linearities in theDNF’s dynamics and lateral interactions withinneural fields, stable localized peaks of activationemerge from distributed, noisy, and transient input.These activation peaks represent perceptual objectsor motor goals in the DFT framework. Multiplecoupled DNFs spanning different perceptual andmotor modalities can be composed into complexDFT architectures.

B. Elementary behaviors (EBs)

A generic structure of EBs (Fig. 1) has beenproposed recently in the framework of DynamicField Theory [9]. Each EB consists of an intentionand a condition of satisfaction (CoS) DNFs. Anactive intention DNF either modifies the percep-tual system of the agent or impacts on the motordynamics of the agent directly. The CoS DNFdetects and stabilizes a perceptual signal that the EBhas successfully achieved its intention. To enablethis, two inputs converge on the CoS DNF: onefrom the intention DNF and the second one fromthe perceptual system. If the two inputs match inthe dimension of the CoS DNF, an activity peakemerges in this field, inhibiting the intention DNFof the EB.

The intention and CoS DNFs are associatedwith intention and CoS dynamic nodes respectively,which facilitate the sequential organization of EBs.While the DNFs are relevant for intra-behaviordynamics, such as in selecting the appropriate per-

Fig. 1: Schematic representation of a generic ele-mentary behavior.

ceptual inputs for the behavior, the dynamic nodesplay a role on the level of inter-behavior dynamics(i.e., switching between behaviors). In our previouswork, we have shown how EBs may be chainedaccording to rules of behavioral organization [8, 9],serial order [10, 5, 4], or the value-function of agoal-directed representation [7].

C. Condition of satisfaction

The condition of satisfaction DNF generates asignal, which denotes that the intention of its EBis successfully achieved. For instance, the CoSDNF for the behavior ‘go to the red object’ coulddetect when a large red object is present in thevisual field, or when the robot is below a distancethreshold to the target object. In our neural-dynamicframework, the CoS is specified by the choice of thedimension(s) of the CoS field and by the synapticconnection weights from the intention field to theCoS field. While the dimensions of the field reflectwhich sensory dimensions the robot is sensitive to,the weights shape the pre-activation in the CoS fieldand make specific regions of the field sensitive toperceptual input.

In our previous work, the perceptual input, whichactivated the CoS of all behaviors, was ‘hardcoded’into the architecture. We designed both the dimen-sions of the CoS field and the synaptic weightsconverging on the field to produce a CoS signal (i.e.,

a peak in the CoS field) only in environmental situ-ations that we, the designers, felt appropriate. Witharchitectures built in this way we have successfullyshown autonomous behavior of robotic agents (see,e.g., [8]). Here, we address the question of howsuch architectures could come about by autonomouslearning.

III. LEARNING THE CONDITION OFSATISFACTION

Here, we present a straightforward but effectivemechanism for learning the CoS, using a variantof associative learning, gated by reward. To enableCoS learning, the basic structure of an EB is aug-mented by two components. First, the connectionweights between the intention and the CoS DNFare made plastic, with a learning rule that combinesassociative and reinforcement learning. In particu-lar, when a rewarding signal is received, learningbetween the intention and CoS fields is enabled,i.e. the connection weights between these DNFsare modified according to a simple Hebbian-likelearning rule (section III-A). The rewarding signal,in its turn, comes from the second new element– a number of internal drives, which motivate theagent’s behavior. These drives can most closely becompared to the prototypical drives suggested byWoodworth, e.g. hunger and thirst [13]. Drivessuch as these serve as internal forces that initiatebehaviors and agents are rewarded when the drivesare satisfied [6].

Fig. 2 shows an exemplar mapping between one-dimensional intention and CoS DNFs. In the caseof one-dimensional DNFs, the coupling betweenthem is a 2D mapping. Here, only two locationsof the intention DNF are associated with respectivetwo locations in the CoS DNF. More complex andhigh-dimensional mappings may be realized in DFTin an analogous way [11]. Such associations aredynamically encoded as memory traces, or localizedpreshapes, in the dynamics of the mapping.

A. Reward-gated associative learningThe learning process in our architecture leads to for-

mation of the memory traces in the dynamical mappingbetween the intention and the CoS DNFs. Let u(x, t) bethe intention DNF, which evolves in time according to Eq.

Intention DNF

CoS D

NF

Dynamic mapping

Preshape

Fig. 2: A simple mapping between one-dimensionalintention and CoS dynamic neural fields. Here,the two pre-shape bumps indicate the two learnedmappings from intention to CoS.

1 [1] and v(y, t) be the CoS DNF (Eq. 2). x is the motorparameter, which spans the dimension, over which theintention DNF is defined, y is the respective perceptualparameter of the CoS DNF:

τ u(x, t) = − u(x, t) + hu +

+

∫f(u(x′, t))ω(x′ − x)dx′ −

− c

∫f(v(y, t))dy + It(x, t), (1)

τ v(y, t) = − v(y, t) + hv +

+

∫f(v(y′, t))ω(y′ − y)dy′ −

+

∫W (x, y, t)f(u(x, t))dx+

+ Isens(y, t). (2)

Here, hu, hv are resting levels of the DNF dynamics,f(·) is the sigmoidal non-linearity shaping the outputof the DNFs, ω(·) is the lateral interaction kernels, cis the constant regulating strength of the homogeneousinhibition from the CoS DNF to the intention DNF, Itis the task (motivational) input, Isens is the sensory in-put, W (x, y, t) is the two-dimensional weights function,which maps output of the intention DNF onto CoS DNF(see Fig. 2).

Fig. 3: CoS learning architecture. See main text for details.

The mapping W (x, y, t) is updated according to asimple reward-driven learning rule, Eq. 3:

τlW (x, y, t) = λR(t)(−W (x, y, t) +

+ f(u(x, t)

)× f(v(y, t)

))·

· f(u(x, t)

)× f(v(y, t)

), (3)

where f(u(x, t)

)× f

(v(y, t)

)= f

(fy(u(x, t)) +

fx(v(y, t)))

is a sigmoided sum of the output of theintention DNF, extended along y (the dimension of theCoS DNF) and the output of the CoS DNF, extendedalong x (the dimension of the intention DNF). Theweights W (x, y, t) are updated when a reward signalR(t) is perceived and they are updated at locations, wherethe overlap between the two projections (fx and fy) ispositive. When updated, the weights converge towards theoutputs of the DNFs.

Fig. 3 illustrates a sketch of the learning architecture.In the following we present an implementation of this au-tonomous neural-dynamic architecture in a simple roboticscenario in a physical environment.

IV. IMPLEMENTATION AND RESULT

Here, we present an implementation of the learningmechanism in a simple scenario in a physical environ-ment. It is important to note that the mechanism wedescribed earlier is very general and works with differenttypes of EBs and perceptual inputs other than the ones

Fig. 4: CoS learner’s environment.

we describe here. The robotic system we used in ourexperiments consisted of an ePuck, equipped with a colorcamera (shown in Fig. 4). The robot was put in anenvironment, which contained several object of differentcolors. The robot needed to search its environment forobjects of certain colors in order to satisfy its internaldrives. The drives, provided to the robot, were looselycalled ‘hunger’ and ‘thirst’. The drives became activeat different times: With the hunger drive active, rewardoccurred when a red object was in the image; when thirstwas active, reward was achieved with a yellow object.

The robot could move around the arena, guided bysimple search dynamics. The camera images provideinput to a two-dimensional perceptual field [10], withone dimension as color hue (separated into 15 bins) andthe other as the image columns. Along each column of

the camera image, the hue of the pixels was summedto provide input to a certain location in the perceptualfield. Activity peaks were formed in the perceptual field,detecting color objects along the horizontal dimension ofthe image. Positive activation in the perceptual field wasprojected onto the hue dimension and provided input tothe CoS field. However, the CoS field could not achievea peak without either a reward signal, which uniformlyboosts the CoS field, or a targeted boost (preshape) fromthe intention field. The goal of the learning process wasto learn the connection weights from the intention field.

The function of teacher-provided reward signal was toprovide a boost to the CoS field activation. This boostallowed a peak to emerge in the output. Due to this peak,the CoS field and intention field had nonzero output atthe same time. Under these conditions, the associativelearning rule adapted the weights between the particularintention that is on (corresponding to which basic driveis active) and the CoS field.

However, other colors besides the one causing thereward were perceived and might be incorrectly asso-ciated with the drive’s satisfaction. Thus, the robot hasto get the reward in different perceptual contexts, and,since the true conditions of satisfaction have a lowvariance, and the incorrect perceptual conditions havehigh variance, the incorrect percepts will be averagedout by the associative Hebbian learning dynamics. Fig. 5shows a snapshot of the system in action. After about 5minutes in this simple environment, where we moved theobjects around so many contexts could be experienced,the correct mappings were learned.

Once the weight matrix is learned, the reward becomesunnecessary to achieve satisfaction. The weights providea sufficient boost to activate the CoS. This boost is se-lective for the perceptual conditions under which rewardwas achieved.

V. CONCLUSION AND OUTLOOK

Learning CoS amounts to learning a coupling structurebetween the intention of the EB and the respectiveCoS. In this coupling structure, an anticipation is rep-resented of the forthcoming state, which terminates theaction. To learn this coupling, the agent must be ableto perceive an elementary reward signal, which mightbe driven by genetically encoded internal drives, such ashunger, thirst, curiosity, or by the emotional system of theagent. The learning mechanism presented in this paperis closely related to classical conditioning, where theinternally generated reward corresponds to the reactionto the unconditioned stimulus, and the learned CoS tothe conditioned reaction. This fundamental mechanism isat the core of the cognitive processes, and has been shownto be important for shaping the behavioral repertoire ofan intentional agent.

Fig. 5: Snapshot of the robot’s dynamic fields. Thepeak in the Intention Field reflects the currentlyactive drive. In the Perceptual Field, the coloredobjects (yellow, red, blue) provide inputs at differentlocations. For this drive, the color yellow in thecenter causes a reward, which gates adaptation inthe weights from intention to the CoS field. Whenthe robot experiences the reward in many differentcontexts, the incorrect cues in the CoS weights arediminished over time. The end result is that whenthis drive is active and the robot sees the coloryellow in the center of the image, the CoS fieldpeaks (correctly), and the behavior leading to thedrive satisfaction will complete.

ACKNOWLEDGMENTS

The authors gratefully acknowledge the financial sup-port of the European Union Seventh Framework Pro-gramme FP7-ICT-2009-6 under Grant Agreement no.270247 – NeuralDynamics; and of the DFG SPP Au-tonomous Learning, within Priority Program 1567.

REFERENCES

[1] S Amari. Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics,27:77–87, 1977.

[2] E Bicho, P Mallet, and G Schoner. Target rep-resentation on an autonomous vehicle with low-level sensors. The International Journal of RoboticsResearch, 19:424–447, 2000.

[3] R. A. Brooks. Do elephants play chess? Roboticsand Autonomous Systems, 6(1-2):3–15, 1990.

[4] B Duran and Y Sandamirskaya. Neural dynamicsof hierarchically organized sequences: a roboticimplementation. In Proceedings of 2012 IEEE-

RAS International Conference on Humanoid Robots(Humanoids), 2012.

[5] Boris Duran, Yulia Sandamirskaya, and GregorSchoner. A dynamic field architecture for thegeneration of hierarchically organized sequences.In Alessandro E.P. Villa, Wlodzislaw Duch, PeterErdi, Francesco Masulli, and Gunther Palm, editors,Artificial Neural Networks and Machine Learning– ICANN 2012, volume 7552 of Lecture Notes inComputer Science, pages 25–32. Springer BerlinHeidelberg, 2012. ISBN 978-3-642-33268-5. doi:10.1007/978-3-642-33269-2 4. URL http://dx.doi.org/10.1007/978-3-642-33269-2 4.

[6] C.L. Hull. Principles of behavior: an introductionto behavior theory. Century psychology series. D.Appleton-Century Company, incorporated, 1943.

[7] S Kazerounian, M Luciw, M Richter, and Y San-damirskaya. Autonomous reinforcement of behav-ioral sequences in neural dynamics. In Proceed-ings of the Joint IEEE International Conference onDevelopment and Learning & Epigenetic Robotics(ICDL-EPIROB), 2012.

[8] Mathis Richter, Yulia Sandamirskaya, and GregorSchoner. A robotic architecture for action selectionand behavioral organization inspired by human cog-nition. In IEEE/RSJ International Conference onIntelligent Robots and Systems, pages 2457–2464,2012.

[9] Y. Sandamirskaya, M. Richter, and G. Schoner. Aneural-dynamic architecture for behavioral organi-zation of an embodied agent. In IEEE InternationalConference on Development and Learning and onEpigenetic Robotics (ICDL EPIROB 2011), 2011.

[10] Yulia Sandamirskaya and Gregor Schoner. Anembodied account of serial order: How insta-bilities drive sequence generation. Neural Net-works, 23(10):1164–1179, December 2010. ISSN0893-6080. doi: DOI:10.1016/j.neunet.2010.07.012.URL http://authors.elsevier.com/offprints/NN2760/7f89af22f761d0dea464a43fdb300383.

[11] Yulia Sandamirskaya, Stephan K.U. Zibner, Se-bastian Schneegans, and Gregor Schoner. Us-ing dynamic field theory to extend the embod-iment stance toward higher cognition. NewIdeas in Psychology, (0):–, 2013. ISSN 0732-118X. doi: 10.1016/j.newideapsych.2013.01.002.URL http://www.sciencedirect.com/science/article/pii/S0732118X13000111.

[12] G Schoner. Dynamical systems approaches to cog-nition. In Ron Sun, editor, Cambridge Handbookof Computational Cognitive Modeling, pages 101–126, Cambridge, UK, 2008. Cambridge UniversityPress.

[13] R.S. Woodworth. Dynamic psychology, by RobertSessions Woodworth. Columbia University Press,

1918.