2010 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL ......2010 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT 1 Learning Generalizable Control Programs Stephen Hart and Roderic Grupen

2010 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT 1

Learning Generalizable Control ProgramsStephen Hart and Roderic Grupen

Abstract—In this paper we present a framework for guidingautonomous learning in robot systems. The paradigm we in-troduce allows a robot to acquire new skills according to anintrinsic motivation function that finds behavioral affordances.Affordances—in the sense of Gibson [6]—describe the latentpossibilities for action in the environment and provide a directmeans of organizing functional knowledge in embodied systems.We begin by showing how a robot can assemble closed-loopaction primitives from its sensory and motor resources andthen show how these primitives can be sequenced into multi-objective policies. We then show how these policies can beassembled hierarchically to support incremental and cumulativelearning. The main contribution of this paper demonstrates howthe proposed intrinsic motivator for affordance discovery cancause a robot to both acquire such hierarchical policies usingreinforcement learning and then to generalize these policies tonew contexts. As the framework is described, its effectivenessand applicability is demonstrated through a longitudinal learningexperiment on a bimanual robot.

Index Terms—Cognitive Architectures, Incremental Learning,Generalization, Schema, Intrinsic Motivation, ReinforcementLearning

I. INTRODUCTION

Humans demonstrate a remarkable ability in finding dex-terous solutions to new problems quickly and efficiently. Partof this ability lies in the capacity to re-use skills previouslylearned in one context to solve related problems in differentcontexts. Rather than trying to solve each new problem byapproaching it with a blank slate, a large amount of knowledgeis brought to bear every time an action is taken. As a result,human development tends to be incremental and gradual;rarely are large leaps made. In contrast, the robotics andmachine learning communities often approach each new taskwith exactly this sort of blank slate. Much attention is given tofinding clever representations and algorithms that create justthese types of large leaps, acquiring great amounts of task-specific knowledge for each new problem in one go, even ifthat knowledge has little applicability in other situations.

One key limitation that has prevented robot programmersfrom re-using knowledge in different contexts has been thatthere are no commonly agreed on representations for groundedbehavior that facilitate efficient integration between com-ponents. Although attempts have been made at code-levelmodularity (e.g., applying good software engineering practicesand object-oriented programming), little attention has beengiven to creating a formal substrate by which adaptive behaviorcan be assembled in a principled way to provide strongperformance guarantees. We argue that such an approach is

S. Hart is with the Italian Institute of Technology. 30 Via Morego, 16163Genova, Italy. e-mail: [email protected]

R. Grupen is with the Laboratory for Perceptual Robotics,University of Massachusetts Amherst, Amherst, MA 01003. e-mail:[email protected]

!"#$%&'

()*+,)-'.,)/,&%0'

!$*0),1%)+),'2$0)3,"$0'4'5&67$'()*+,)--$,0'

&"")%%)8&6)*'

&001%1-&6)*'

9*71,)*%

$*+'

&:),8&*"$'810")7$,;'%)67&+),'

Fig. 1. The System Architecture.

necessary in any artificial system that must explore its envi-ronment autonomously to ensure safe and reliable operation.Furthermore, we hypothesize that this type of approach isinherently scalable, in that acquiring new skills and knowledgewill be less likely to result in unwieldy situations whereintegration is difficult or impossible.

In this paper we advocate a novel approach to behav-ioral programming in robots. This approach uses a generalrepresentation for behavior that is robot-centric, rather thantask-centric. Behavioral competency is evaluated through anintrinsic motivation function that encourages the robot to orga-nize its sensory and motor systems into co-articulated controlprograms that couple to the dynamics of the environment. Thedynamic status of these coupled dynamical systems are usedto define environmental affordances. Hierarchical programsare assembled from the bottom-up as the robot finds newways to combine these programs and uncover new affordancesand generalized from the top-down as the robot learns abouthow these programs apply in novel contexts. This approach isconsistent with Piaget’s notions of accommodation, in whichan organism finds new ways to “mold” itself to its environmentas new situations present themselves, and assimilation, inwhich resulting knowledge structures are fit to new situationswith robust contingency plans [24]. Due to this relationship toPiaget, we call the generalizable knowledge structures in theproposed framework “schema.”

Figure 1 shows an overview of the proposed architecture.Through a process of accommodation, a robot assemblescontrol programs by finding combinations of its sensory andmotor resources and “native” control fields (not specific toany one task) that couple the dynamics of the robot with thedynamics of the environment. Through a process of assimila-tion, these programs are transformed into schema as the robotdiscovers resource re-parameterizations that provide solutionsin different (but related) environmental contexts. Both ofthese processes are governed by the robot’s intrinsic drive fordiscovering affordances. Schema may be re-used hierarchically


as atomic actions in higher-level control programs, bringingwith them all their contingency plans and supporting open-ended behavior acquisition.

The remainder of this paper is structured as follows. Sec-tion II introduces the combinatorial basis for control—thecontrol basis—and demonstrates how a robot’s sensory andmotor resources can be combined with native control fieldsto form co-articulated actions. Section III demonstrates howhierarchical control programs are acquired in an adaptivelearning framework and introduces a novel intrinsic rewardfunction for uncovering behavioral affordances. In Section IV,we show how these programs are transformed into schemaand applied to novel contexts. Finally, Section V concludeswith a discussion of the benefits of the proposed approach. InSections III and IV, a longitudinal learning experiment per-formed on the bimanual robot “Dexter” is presented. In eachstage of learning, Dexter either accommodates a new programor assimilates an existing skill in a new context. While thecontext provided to the robot at each stage is structured bythe programmer, the strategies the robot discovers to deal withthat context are learned through the robot’s intrinsic motivator.

II. A COMBINATORIAL BASIS FOR CONTROL

The control basis was originally introduced by Huberand Grupen as a means for robot systems to explore thecombinatorics of sensory and motor control circuits in anautonomous learning framework [13]. These combinatoricsprovide a definition for action that, as discussed in the nextsections, is useful for organizing knowledge into structuresthat facilitate generalization and transfer. The control basisformulates robot learning in a discrete event dynamic sys-tems (DEDS) framework [23] to control the complexity ofstate/action spaces and to ensure that safety and performancespecifications are satisfied during on-line exploration.

In the control basis, state and actions spaces are constructedfrom feedback controllers by combining elements of a setof artificial potential functions, φ ∈ Ωφ, with subsets of therobot’s sensory and motor resources, σ ⊆ Ωσ and τ ⊆ Ωτ ,respectively. Reinforcement learning techniques are applied inthese spaces to adaptively build value functions, Φ ∈ ΩΦ,that define discrete action policies. These policies are thehierarchical generalization of the primitive control laws thatcomprise the control basis. The result is a naturally recursivedescription of knowledge in terms of functions that underliebehavior.

All control expressions constructed from the control basisprovide force or position reference commands to lower-levelclosed-loop motor units that actuate the robot (Figure 2). Thecontrol basis exploits the fact that a small alphabet of primitivecontrol elements Ωφ × Ωσ × Ωτ can yield a large varietyof hierarchical control circuits. The process of assemblingfeedback control loops from potential fields is discussed inthe remainder of this section. How to assemble higher-levelprograms on top of these feedback controllers is discussed inthe next section.

feedback signals

value functions

force, position

references (") (!)

embedded motor circuits*

Φ!

Δ!

Φ!

Δ!

!

Δ!

potential functions

!"Δ!

value functions hierarchy of sensorimotor

programs greedy behavior leads to robust

solutions

.*.*

.*

/*/*

/*

Fig. 2. The hierarchical control basis architecture. Control actions de-scend potential functions by submitting reference inputs to lower-levels andultimately, to embedded motor circuits. H and G represent feedback andfeedforward transfer functions, respectively.

A. Artificial Potential Functions

We now examine a set of artificial potential functions Ωφthat have formal constraints that are of general use in adaptivecontrol systems. All of the control actions employed in thispaper can be characterized by these potential functions, ascan many other control actions for mobile robots or robotmanipulators. The control basis provides a general and unifiedapproach for assembling feedback control laws when thesepotential functions are combined with sensory and motorresources.

Potential function methods for control require constraintson the shape of the function in order to guarantee asymptot-ically stable behavior and to avoid common problems citedin the literature (e.g., local minima). Rimon and Koditschekenumerated the conditions for a class of navigation functionsthat can formally be used as control functions [14], [27]:• Analytic - Potential φ is analytic if it is infinitely

differentiable (i.e., it can be written as a Taylor series),and thus has a gradient that points toward a minimum.

• Polar - The gradient of polar functions create streamlinesthat terminate at a unique minimum.

• Admissible - The gradient ∇φ of a navigation functionmust be bounded so that it can serve as a realistic controlinput without gain scheduling or scaling.

• Morse - Navigation functions are Morse if they containno degenerate critical points (e.g., saddle points).

We next examine a number of artificial potential functionsthat can form a basis for a wide variety of robot behaviorand describe the conditions under which they can be used toconstruct closed-loop controllers.

1) Quadratic Potential Functions: Hooke’s law is an ex-ample of a quadratic potential field that describes the strainenergy stored in a spring. It can be employed in the controlbasis to induce virtual spring-like properties on feedback errorsobserved in many domains. In this context, Hooke’s law isdefined as:

φs(σref , σact) =12

(σref − σact)T (σref − σact) (1)

where the difference between the actual and the referencefeedback signals, σact, σref ⊆ Ωσ , captures errors betweentwo features of the same type. This function is convex and,


if the input error is bounded, then it is also a navigationfunction and can be used for control. We use this kind ofprimitive “intention” repeatedly in the control basis frameworkto, for instance, build tracking controllers in force and positiondomains.

2) Harmonic Functions for Collision-Free Motion: Artifi-cial potential field approaches have also been developed forrobot path planning that make efficient use of sensor feedback.Harmonic functions describe many physical processes that relyon minimum energy configurations such as soap films, laminarfluid flow, the temperature dissipation in thermally conductivemedia, and the voltage distribution in resistive networks.They have been applied in robot systems to find path planswithout local minima and that minimize the probability ofcollisions [4]. Goals and obstacles define the boundaries of thenavigatable free space in this approach. A harmonic potentialfield φh has no local extrema points in the interior of thisspace. Numerical relaxation methods (e.g., Successive Over-Relaxation (SOR), Jacobi iteration, or Gauss-Seidel iteration)can be used to compute harmonic functions quickly in tasksfor low-dimensional systems.

Harmonic functions, unfortunately, do not meet all four ofthe criteria for navigation functions (they are not admissibleand are not analytic at goals). As a result, they produce pathsthat minimize the likelihood of collisions with obstacles, butare not asymptotically stable. This limitation can be overcomeby allowing the system to follow the direction of the potential’sgradient, but shaping its magnitude before sending it to theplant.

3) Kinodynamic Conditioning Functions: Conditioning ac-tions are useful in multi-objective control tasks and can pro-vide a natural way for an embodied system to get the most outof its sensory and motor resources [8]. Several conditioningfields have been devised for use in the control basis—eachof which captures some independent prerogative of a system.Three such fields are discussed next.

a) Range Limits: It is often useful to bias a manipulatoraway from joint range limits in order to provide a greaterlikelihood that global objectives are met. One possible choicefor an n-dimensional system is to create an n-dimensionalcosine field around the center of each joint’s range of motion.For a set of joint angles θ ∈ Cn, we define

φr(θ) = n−n∑i

cos

(θi − θi

θi,max − θi,minπ

). (2)

In this equation, θi,max and θi,min represent the upper andlower limits of joint i’s range of motion, and θi represents thejoint center. This field is a navigation function that provides aconvex potential centered in the center of the robot’s range ofmotion over all degrees of freedom.

b) Manipulability: Yoshikawa’s measure of manipulability(MoM) field moves a kinematic mechanism into configurationsthat allow for a tradeoff in the ability to perceive (input)errors and to impart (output) movements [36]. Informally, itconditions the manipulator Jacobian in a manner that preservesflexibility and least commitment for unknown future circum-stances. Such a methodology is useful when programmingdexterous robots that must behave well in uncertain real-world

environments. In addition, it provides a natural kinematic“sweet-spot” in the system, and pushes the manipulator awayfrom undesirable singular locations. Whether the manipulabil-ity field for a particular robot is a navigation function or notdepends on that robot’s specific configuration. It, however, isoften a useful objective to maximize in practice to providesmooth and natural movements for redundant manipulators.The manipulability field is defined as:

φm(θ) = −√det(J(θ)J(θ)T ) (3)

where θ ∈ Cn is an n-dimensional subset of joints that forma kinematic chain and J(θ) is the manipulator Jacobian.

c) Localizability: Kinematic conditioning can be appliedto any linear transformation and has been generalized tocontrol viewpoint quality in a stereo vision system in orderto maximize localization precision [34], [8]. Let us define ameasure of localizability (MoL) field to achieve this objective.This field is defined in terms of the oculomotor JacobianJ(γl,γr), where γl and γr represent the headings towarda feature viewed by both the left and right cameras of thestereo system. The oculomotor Jacobian transforms visualdisplacements into Cartesian displacements. The localizabilityfield is defined as:

φl(γl,γr) =1√

det(J(γl,γr)J(γl,γr)T ). (4)

This potential field describes how the Jacobian of the stereotriangulation equations amplifies imprecision. Optimizing forlocalizability allows for high precision in stereo-triangulationtasks where the objective is to recover the Cartesian locationof a feature from its visual appearance.

The collection of potential functions presented in thissection, Ωφ = φs, φh, φr, φm, φl, provides a number ofways for a robot to re-code its sensory signals into artificialgradients that precipitate behavior. We argue that this set ofpotentials can support a rich set of behavioral prerogatives inan embodied system. In the remainder of this paper, we supportthis argument with demonstrations in which a bimanual robotemploying only this set of “native” potentials learns how toperform a number of hierarchical manipulation tasks.

B. Sensory and Motor Signals

A robot’s sensory and motor signals provide the specificmeans for an embodied system to observe and interact withits environment. In the control basis, these signals form thesensor and effector sets, Ωσ and Ωτ , that, along with a setof potential functions Ωφ, define the primitive actions of thecontrol basis framework.

Although any given set of sensory resources, Ωσ , is par-ticular to a specific robot, many standard types of signalsare used in the robotics literature to govern behavior. Sensorsignals may come directly from “raw” physical devices (e.g.,encoders, cameras, microphones, force/torque strain gauges,etc.), they may arise through mathematical transformations ap-plied to the information returned by multiple of these devices(e.g., via forward kinematics functions, functions for signal


TABLE ITYPES OF SENSORY INFORMATION.

Type SpaceConfiguration variables Cn

Cartesian coordinates SE(3)Cartesian positions R3

Cartesian orientations SO(3)Wrench coordinates SE∗(3)Force vectors R3

Torques SO∗(3)Headings SO(2)

localization, etc.), or they may be sampled from statisticalmodels that the robot builds as it learns about its world. Table Ishows some common types of sensory information availableto a complex sensorimotor system such as a humanoid robot.

A robot’s set of motor signals, Ωτ , is defined by the degreesof freedom that accept command inputs. Typically, this setconsists of configuration variables or motor torques, but mayconsist of “virtual” DOFs defined by sensory transformations(e.g., the position of a robot’s hand calculated from theforward kinematics). It is often useful to group subsets ofa robot’s effector variables together into “synergies” that arecontrolled concurrently. For example, the motor variables thatcontrol a robot’s arm form a synergy that can allow forreaching movements. Forming motor synergies alleviates someof the challenges of the “degrees of freedom” problem, and isconsistent with Bernstein’s theories of motor control [2].

C. Composite Control Laws

A closed-loop controller in the control basis, c(φ, σ, τ),where φ ∈ Ωφ, σ ⊆ Ωσ , and τ ⊆ Ωτ , describes a circuitthat iteratively computes reference inputs to low-level motorunits. The sensitivity of the potential to changes in the valueof motor variables is captured in the Jacobian J = ∂φ(σ)/∂τ ,where J# is its Moore-Penrose pseudoinverse [21]. Controlsignals are computed by the expression:

∆τ = −J#φ(σ)κ, (5)

where κ is a positive gain. In all of the experiments inthis document κ = 1 for simplicity. Under this control law,the system follows the negative gradient of the potentialtoward stable attractor states where ∇τφ(σ) = 0. By thisdefinition, all controllers in the control basis framework definelinear dynamical systems that suppress disturbances from theenvironment.

Multi-objective control actions are constructed by com-bining control primitives in a prioritized fashion. Considertwo control actions, a higher priority controller c1 and alower priority controller c2. To ensure that the lower priorityobjective does not destructively interfere with the progress ofthe higher priority objective, control objectives are combinedusing nullspace projection [21]. For a two-fold control rela-tionship, the prioritized composite control law is

∆τ = −J#1 φ1(σ1)κ1 − N1

(J#

2 φ2(σ2)κ2

), (6)

where N1 represents the nullspace of the higher prioritycontroller, N1 = (I − J#

1 J1). Equation 6 can be extendedto combinations of n-fold concurrency relationships betweencontrol basis actions. We use the “subject-to” operator “/” to

represent the prioritized combination between any two suchcontrol actions [13]. The control expression c2 /c1—read, “c2subject-to c1”—provides a useful shorthand notation for suchrelationships.

D. Typing

Control actions assembled from combining elements of Ωφ,Ωσ , and Ωτ , require strict adherence to typing constraintsbetween input and output signals to behave as intended. Inparticular, specific potential fields may only be evaluated withrespect to certain types of feedback signals while it may onlybe possible to compute gradients with respect to certain outputvariables. Typing is important for guaranteeing correct controlexpressions as well as for enabling behavioral abstraction.For example, kinematic conditioning metrics, such as thosepresented in Section II-A3, can be applied to any collectionof configuration variables in Cn, regardless of the specificmechanism they represent. Similarly, “reaching” tasks thatmove a robot’s end-effector to a Cartesian position definedby triangulation can be implemented for any combination ofvisual features visible by two (or more) cameras.

Sensory signals of one type can sometimes betransformed—or typecast—into signals of a differenttype; for example, through the forward kinematics or stereotriangulation equations (or their inverses). Typecasting createsdexterous alternatives for controlling robots, and allowscombinations of control tasks to be constructed in differentstate spaces and combined in some joint space.

Enforcing typing constraints and allowing automatic type-casting between signals makes it possible to implement aformal programming specification for the control basis that canfacilitate code re-use and control construction, and can gener-ate search spaces for machine learning algorithms. A ControlBasis Applications Programming Interface (or CBAPI) [11]has been implemented using both Microsoft Robotics Devel-oper’s Studio [19] and YARP [18].

E. Example: Kinematically Conditioned Reaching

We now provide an example in which co-articulated controlbasis expressions are implemented on the robot Dexter (Fig-ure 3) to improve the quality of a reaching task. Dexter has atwo degree of freedom pan/tilt head equipped with two Sonycolor cameras and two 7-DOF Whole-Arm Manipulators (Bar-rett Technologies, Cambridge MA). Each WAM is equippedwith a 3-finger Barrett Hand with a 6-axis F/T load-cell oneach fingertip. Each hand has four degrees of freedom (onefor each finger, and one for the spread angle between two ofthese fingers).

The control expressions discussed in this example areassembled from Dexter’s task-independent resource sets, butused in the context of performing the task of reaching outand making contact with an object. This example emphasizesthe utility of “uncommitted” conditioning actions in task-directed behavior. Another example, demonstrating the abilityof the control basis in increasing the performance at a visualclassification task is provided in [8].


Consider three controllers assembled from Dexter’s nativecontrol basis:

PosturalBias is a kinematic conditioning control action thatbiases the posture of a robot mechanism toward the middle ofits range of motion via φr. In this example, it is employed forthe robot’s right arm and is defined as

POSTURALBIAS(RIGHTARM) , c(φr,θr,arm,θr,arm). (7)

where θr,arm ∈ C7 represents Dexter’s right armConfiguration-space motor synergy.

Manipulability is a kinematic conditioning control actionthat optimizes the manipulability metric according to φm.In this example, we optimize this metric with respect to therobot’s right arm, defined as

MANIPULABILITY(RIGHTARM) , c(φm,θr,arm,θr,arm).(8)

Reach is a spatial tracking control action that uses the virtualspring potential function φs to reduce the Cartesian errorbetween the end-effector and a reference position. In thisexample, the REACH action is implemented to move Dexter’sright arm synergy, θr,arm, toward the position of a highly-saturated object xsat (computed by triangulating the headingstoward highly saturated visual cues (γlsat,γ

rsat) viewable on

both of Dexter’s left and right camera images), and is definedas:

REACH(SAT,RIGHTARM) , c(φs, (xsat, xr,arm),θr,arm).(9)

Let us define an error vector ε=(xsat − xr,arm). UsingEquation 5, effector variable displacements for REACH arecomputed as follows:

∆θr,arm = −(∂φs(xsat, xr,arm)

∂θr,arm

)#

φs(xsat, xr,arm)

= −(∂φs(xsat, xr,arm)

∂xr,arm∂xr,arm∂θr,arm

)#

φs(xsat, xr,arm)

=(εT Jr,arm

)#(12εT ε

)=

12(J#r,armε

), (10)

where Jr,arm is the manipulator Jacobian of the right arm.

In the following demonstration, a highly-saturated objectwas placed in twenty-five uniformly distributed locations onthe table in front of Dexter in a 0.4m × 0.8m square. Threecomposite control laws were constructed and executed, eachwith the reaching controller as the superior objective. Forone control law, this was the only controller employed. Theother two laws employed each of the kinematic conditioningcontrollers as an inferior objective. These three laws are

(a) (b)

Fig. 3. Frames (a) and (b) show the robot before and after reaching to ahighly-saturated object with its right arm.

defined as:

c0 , REACH(SAT,RIGHTARM) (11)c1 , POSTURALBIAS(RIGHTARM)

/ REACH(SAT,RIGHTARM) (12)c2 , MANIPULABILITY(RIGHTARM)

/ REACH(SAT,RIGHTARM) (13)

The change in potentials for all three controllers under allthree laws were recorded over the twenty-five reach actions(even if they were not optimized by the control law beingexecuted). Figure 3 shows an example reach in which the robotmakes contact with the object. The robot begins in the centerof its range of motion.

Figure 4(a) shows the average potential φs for the reachingaction over all runs. We see how the quadratic error functioncauses the system to decrease steadily to the goal. Figure 4(b)shows the average value of the postural bias potential, φr, overthe course of the reaching actions performed under controllaws c0 and c1. Because the robot begins each action atthe minima of this field, the potential only increases as thereaching action is performed. However, in the case where therange of motion controller is optimized in the nullspace, theincrease in this metric grows less, on average, than the casewhere this objective is not optimized, thus staying furtheraway from joint limits. Figure 4(c) shows the manipulabilitypotential, φm, over the course of the reaching actions per-formed under control laws c0 and c2. In the case where themanipulability controller is optimized in the nullspace, theincrease in this metric grows less, on average, than the casewhere this objective is not optimized, thus keeping the robotfurther away from undesirable singular configurations.

III. LEARNING CONTROL PROGRAMS

This discrete nature of controller composition makes thecontrol basis framework amenable to stochastic search. Inthis section, we describe how machine learning algorithmssuch as reinforcement learning [32] can explore the structuredcontrol basis in order to create domain-general behavioralprograms. We present (1) a novel definition of state thatcaptures convergence events in the run-time dynamics ofcontrol actions, and (2) an intrinsic motivator that rewards thediscovery of environment affordances.


0 50 100 150 200 250 3000

0.05

0.1

0.15

0.2

0.25

Cartesian Movement Potential

control iterations

po

ten

tia

l

(a)

0 50 100 150 200 250 3000

0.002

0.004

0.006

0.008

0.01

0.012

Postural Bias Potential

control iterations

po

ten

tia

l

2nd priority

not optimized

(b)

0 50 100 150 200 250 300

0.075

0.08

0.085

0.09

0.095

0.1

0.105

Manipulability Potential

control iterations

po

ten

tia

l

2nd priority

not optimized

(c)

Fig. 4. Frame (a) shows the potential of the Cartesian movement controller averaged over the 25 trials. Frame (b) shows the potential for the postural biascontroller for the same 25 trials when that controller is run as a subordinate control action to the Cartesian reach controller compared to the same metricwhen that controller is not run. Frame (c) shows the same comparison for the manipulability controller run as the secondary objective to the reach action.

The proposed definitions of state, action, and reward allowa robot to learn hierarchical control basis programs through aprocess of accommodation as described in Section I. Hierar-chical representations provide efficient encoding of complexbehavioral programs in a manner that hides complexity andbounds the size of the state and action spaces. They alsoencourage behavioral re-use by providing temporally extendedpolicies as single, invokable actions. In the proposed frame-work, hierarchical programs are built from the bottom up.When a robot learns a new program, it creates an addi-tional means for exploring its environment and enhancingits behavioral capabilities. As a result, behavior is learnedincrementally and in stages, forcing a robot to only addresslearning problems that are just beyond the limits of its currentdevelopment.

A. A State Representation for Dynamic Processes

Complex systems that must learn from on-line experienceare best served by efficient state representations that capturereal-valued sensory and motor signals in compact forms.Useful structure arises when this data is represented as aseries of interacting dynamical systems either generated bythe robot (e.g., control processes), or observed by the robot’ssensors (e.g., environmental forcing functions). The discreteevent dynamic systems (DEDS) approach leverages this factby capturing state in terms of discrete events observed in thecontinuous data [23].

In control processes, the dynamics (φ, φ) created when acontroller interacts with the world provides a natural discreteabstraction of the underlying continuous state space. Huberprovided a binary state representation in which the predicatep(φ, φ) associated with a controller c(φ, σ, τ) is 0 during thetransient response of the controller and transitions to 1 whenthe controller converges to an attractor [13]. The change inpotential, φ, is the observed, time-based derivative and is notto be confused with the gradient of the field at that point, ∇τφ.

In this paper we use a similar discrete state representation asHuber based on quiescence events capturing controller conver-gence [10]. Quiescence events occur when either a controllerreaches an attractor state in its potential or when a lack

X

-

1

0

Fig. 5. This figure shows an iconic representation of the transitions in theproposed state representation.

of progress along the gradient of that potential is observed.Mathematically, we define a predicate p(φ, φ) associated withcontroller c(φ, σ, τ), such that:

p(φ, φ) =

X : φ(σ) controller is not activated− : φ(σ) undef. feedback reference0 : |φ| > ε, transient response1 : |φ| ≤ ε, quiescence,

(14)

where ε is a small positive constant. When the controller isnot activated, the state evaluates to “X.” If the controller isactivated, the state predicate will evaluate to “0” if there is atarget stimuli present in the feedback signal (and φ(σ) can becomputed), or “−” if there is not. The controller runs in thetransient state “0” until it loses the target stimuli or quiescesin state “1.” Figure 5 shows an iconic representation of thepossible state transitions of a controller as it interacts with theenvironment. Given a collection of n distinct primitive controlactions a discrete state space S can be formed where s ∈ Sis defined as s = (p1 . . . pn).

B. An Intrinsic Reward for Affordance Discovery

The state representation of Equation 14 registers conver-gence events that occur in the dynamics of a robot’s controlcircuits. When a convergence event occurs for a controllerthat tracks an environmental forcing function measured byelements of the subset Ωσ(env) ⊆ Ωσ , it creates a closed-loop coupling between the robot and the world that has aspecial designation. We call this coupling an environmentalaffordance. Affordances are thus measured not only in termsof perceptual stimuli, but also in terms of the robot’s ability


to engage these stimuli with its motor resources in a stablecontrol configuration. By modeling affordances defined in thisway, a robot can gain a sense of what it can control and thusincrease its ability to perform tasks in its world. We argue thata robot designed to learn adaptive behavior over the long-termmust be equipped with an imperative to seek out affordancesand the conditions in which they can be asserted.

We now define an intrinsic motivation function for affor-dance discovery, adapted from [10]. In the control basis, thediscovery of an affordance is measured by the convergenceevent

bti =((pt−1i 6= 1) ∧ (pti = 1)

), (15)

where pti is the state of a controller c(φi, σi, τi) at time t. Theintrinsic motivation function provides positive reward for allcontrollers that converge at t according to the function

rti =

1 : if(bti ∧ (σi ⊆ Ωσ(env))

)0 : otherwise (16)

rt =∑i

(htir

ti

), (17)

where hti is a habituation metric that modulates the level ofreward based on past experience. A complete description ofthis metric is beyond the scope of this paper, but the interestedreader is directed towards [12], [7] for more details. For thesake of the current discussion, we will not consider the effectof habituation on learning and only address the simplifiedsituation in which hti = 1 for all t and i.

The restriction that the controller’s feedback signal σi mustbe an element of Ωσ(env) prevents reward from occurring forrandom movements, kinematic conditioning actions, actionsthat track transformed signals, or actions that respond tointernal models the robot may have. As a result, the affordancediscovery reward function partitions control expressions de-rived from the control basis into two disjoint subsets: those thattrack forcing functions originating in external environmentalstimuli and those that do not.

What kinds of controllers produce rewarding events by theaffordance discovery motivator? For Dexter, controllers thatrespond to feedback signals in the set of visual headings andthe set of force/torque measurements originating in externalenvironmental stimuli are rewarding. Consider a visual track-ing controller that moves Dexter’s stereo pair of cameras tofoveate on a brightly colored object. A quiescence event forthis tracking controller will represent the discovery of a new“trackable” affordance with respect to that object. If that objectalso affords a controlled touch response using a controller thattracks a force-domain signal when the robot reaches out to it(and it does not roll away), that object also has a “touchable”affordance. It is important to note that not all controllersreferenced to feedback signals in Ωσ(env) will always providecontrol affordances. An affordance represents a tight couplingbetween a perceptual stimulus and the robot’s body. If a visualfeature, for example, is moving too fast for the robot to trackgiven the limitations of its motor systems, the controller willnot produce a convergence event and thus not provide reward.

Algorithm 1 ACCOMMODATE(m, A, T )1: Let A′ be the set of non-composite actions in A2: Let S be the state space formed from the predicates of

the actions in A′3: n← |A′|, γ = 0.8, α = 0.1, ε = 0.24: for i = 1 to m do5: k ← 0, t← 06: reward← false7: while (reward = false) ∧ (k < T ) do8: st ← (pt1 . . . p

tn)

9: a← π(st) (via ε-greedy selection)10: repeat11: execute a for one iteration12: t← t+ 113: st ← (pt1 . . . p

tn)

14: until st 6= st−1

15: k ← k + 116: evaluate rt according to Eq. 1717: update Φ(s, a) using Q-Learning (Eq 18)18: if rt > 0 then19: reward← true20: update Pr(τ |σ, reward) for all rewarding control

actions c(φ, σ, τ),21: end if22: end while23: end for

C. Skill Accommodation

The procedure for estimating the state/action value functionΦ for a set of control basis actions A and the rewardfunction provided in Equation 17 is shown in Algorithm 1.This procedure is called ACCOMMODATE() because it shapesΦ to provide strategies for uncovering affordances in theenvironment. It executes m reinforcement learning episodesof no more than T state transitions over the state/action spacedefined by A. Specifically, the algorithm uses Q-Learning [35]to estimate the state-action value function, Φ(s, a). The Q-learning update rule is defined as:

Φ(s, a)← Φ(s, a) + α(r + γ maxa′Φ(s, a′)− Φ(s, a)

). (18)

ACCOMMODATE() also estimates probability distributions ofthe form Pr(τ |σ, reward) when rewarding conditions aremet (Line 19). Distributions of this form provide primitivememory structures that encode configurations where the robothas achieved reward in the past, and can be used as virtualsensors to inform future searches.

D. Example: SearchTrack

We now present a simple program Dexter learned usingthe affordance discovery reward function to find visual af-fordances. This program, called SEARCHTRACK, was learnedusing Algorithm 1. It moves Dexter’s pan/tilt head to locationswhere the robot has previously observed highly-saturated pixelregions on its cameras’ image planes, and then tracks theseregions to determine if they afford quiescence measured byEquation 17. The acquisition of SEARCHTRACK represents the


first stage of Dexter’s development it the longitudinal learningexperiment presented in this paper.

This learning problem orients the robot to uncover asingle rewarding control event and is taught to Dexter in asimple, constrained environment designed to make that eventconspicuous. The training context focused exclusively on themost highly-saturated pixels in the robot’s field of view. Weprovide a comparison of learning results from two scenarios:one in which the robot explored co-articulated control actionsand one in which it did not. This comparison demonstratesthe quantitative advantage of co-articulated policies oversequential policies that execute only a single action at a time.Both scenarios employed the following two primitive controlactions:

Track is a control action that pursues a saturation cue on therobot’s left camera by changing the reference pan/tilt headposture, θhead ∈ C2, according to the virtual spring potentialfunction φs. The goal is to keep the coordinate of a highly-saturated visual cue, γlsat ∈ R2, at the left camera’s imagecenter, γl0 = [0 0]T . This controller is defined as:

TRACK(SAT) , c(φs, (γl0,γlsat),θhead). (19)

Defining an error vector ε=(γl0 − γlsat), and plugging theseresources into Equation 5:

∆θhead =−(∂φs(γl0,γ

lsat)

∂θhead

)#

φs(γl0,γlsat)

=−(∂φs(γl0,γ

lsat)

∂γlsat

∂γlsat∂θhead

)#(12εT ε

)≈−

(−εT I2×2

)#(12εT ε

)=

12ε, (20)

where the Jacobian capturing how pan/tilt displacements effectthe view of headings perceived in the robot’s left camera isapproximately the identity matrix I2×2 due to the fact thatthe pan and tilt axes intersect at the camera’s optical center.Given this physical relationship, the perceived heading errorsin the image plane correspond proportionately to variationsin the robot’s pan and tilt configuration. The quiescence ofTRACK is rewarding according to the affordance discoveryintrinsic motivator because γlsat ∈ Ωσ(dir).

Search constructs and reduces a feedback error from twosignals—the current value of the head’s pan/tilt angles θhead,and a head reference posture θhead,ref—according to thegradient of the potential function φs. This controller is definedas:

SEARCH(SAT) , c(φs, (θhead,ref ,θhead),θhead). (21)

θhead,ref is sampled from a probabilistic model of priors forthe search target in terms of pan and tilt head angles,

θhead,ref ∼ Pr(θhead|γlsat = γl0). (22)

θhead,ref ∈ C2 is thus sampled from a distribution of headconfigurations where the environment is likely to have affordedTRACK(SAT) quiescence in the past. It is easy to see thatPr(θhead|γlsat = γl0) is closely related to Pr(τ |σ, reward)

for the TRACK controller that is updated at Line 19 ofthe ACCOMMODATE() procedure. These distributions are thusused interchangeably in this program.

SEARCH orients the head to postures where the targetsaturation is likely to be found on the left camera imageplane. It is not rewarding by the affordance discovery intrinsicmotivator because the reference for the action is not derivedfrom the environment, but rather from probabilistic modelsof past environments.

In conjunction, the SEARCH(SAT) and TRACK(SAT) con-trollers support the construction of the state space Sst whereeach state s ∈ Sst is evaluated such that s = (psearch ptrack),and two possible action sets (dropping the parameter values fornotational simplicity) A1

st = SEARCH, TRACK and A2st =

SEARCH, TRACK, SEARCH / TRACK, TRACK / SEARCH,depending on whether co-articulation is allowed. The fol-lowing experiment compares policies for asserting the trackaffordance using each of these candidate action sets. Dexterlearns policies for SEARCHTRACK according to the procedureshown in Algorithm 1. Ten trials of 50 learning episodeswere performed for each experiment. At the beginning of eachtrial, Φ(s, a) was initialized to zero. For evaluation purposes,the average reward per state transition was recorded for eachepisode and averaged over the trials. Each episode endedwhen a rewarding event occurred (i.e., TRACK quiesced).The distribution Pr(θhead|γlsat = γl0) was estimated as anon-parametric distribution with a small Gaussian smoothingkernel and was initialized at the beginning of each trial to auniform distribution.

In half of the episodes, the experimenter presented a highly-saturated object at a position in front of the robot (in thecamera’s initial field of view), as seen in the image fromDexter’s camera in Figure 6(a). In the other half of the trainingepisodes, no object was presented to the robot. However, othersaturation cues were available in the robot’s environment if therobot “looked around” (e.g., toward the window to its left, asseen in Figure 6(b)).

Figure 6(c) shows one of the learned non-parametric distri-butions at the end of a trial for SEARCHTRACK estimatingPr(θhead|γlsat = γl0). The large peak in the center ofthe robot’s pan range corresponds to locations where theexperimenter held objects in front of the robot (Figure 6(a)),the smaller peak corresponds to the configuration where thesaturated window region could be seen (Figure 6(b)). Thisdistribution reflects the robot’s knowledge at the end of thetrial concerning were TRACK affordances occur. This modelcan thus be used as a prior by the SEARCH controller to informfuture executions to orient the robot to efficiently achievereward by the affordance discovery motivator.

For the case in which the action set did not include co-articulated actions, the robot learned a policy that resulted inthe transitions shown in Figure 7(a). From the start state (XX),the policy chooses action TRACK. If a saturation stimulusis absent, the state transitions to (X−), thereafter entering aloop that iteratively searches using SEARCH and then testsfor stimuli using TRACK. When a saturation cue is detected,the robot enters state (X0) and then continues to execute


X 1!X 0!

1 X!

0 X!

X -!

X X! Track

Search Search

Track

Track

(a)

0 1!0 0!

1 -!

0 -!

X X! Search Track

Search Track

Search Track

Search Track

(b)

Fig. 7. SEARCHTRACK transition diagrams for the policies acquired on Dexter in the first stage of learning. Transitions are shown if they occurred with aprobability greater than 20%. The diagrams are characterized by states s ∈ Sst where s = (psearch ptrack). The policy employs TRACK first, and SEARCHis chosen only when no stimuli is immediately present. The state diagram in (a) shows the policy when only single actions are allowed in the action set. Thestate diagram in (b) shows the policy when composite actions are allowed.

(a) (b)

!"#$%"&'()*+),+-.*/0"1$+,)%+0%.2345+6.$'%.()*+7+89"#)54+:;++

!"#$%&'($

(c)

Fig. 6. Frame (a) shows an image from Dexter’s left camera when a saturatedobject is presented in front of the robot. Frame (b) shows an image fromDexter’s camera while viewing window to its left. Frame (c) shows the non-parametric distribution after 50 training episodes summarizing the pan/tiltconfigurations where Dexter expects to observe a saturation cue.

TRACK until it quiesces in state (X1) and receives reward.The average reward graph for these experiments is shown inred in Figure 8(a).

For the case in which the action set included co-articulatedactions, the robot learned the policy that resulted in thetransitions shown in Figure 7(b). This policy allows the robotto “interrupt” the search process if a saturated stimulus appearsas the robot moves to the reference location. The policysuggests using action SEARCH / TRACK in all states, allowingthe higher priority action, TRACK, to dictate the robot’sbehavior if the stimuli is present, while performing successivesearches in the track controller’s nullspace, if it is not. Theresulting co-articulated policy requires fewer state transitions

0 5 10 15 20 25 30 35 40 45 500.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45SearchTrack Reward (avg. 10 trials)

episodes

aver

age

rew

ard

per s

tate

tran

sitio

n

w/ coarticulationno coarticulation

(a)

0 1 2 3 4 5 6 7 8 9 10 110.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

SearchTrack Reward w/o Exploration (avg. 50 trials)

episodes

av

era

ge

re

wa

rd p

er

sta

te t

ran

sit

ion

w/ coarticulation

no coarticulation

(b)

Fig. 8. Reward plots for the SEARCHTRACK policies learned on Dexter. Plot(a) shows the learning curves for each experiment averaged over 10 trials of50 episodes each (with 20% exploration). Plot (b) shows the reward for thelearned policies averaged over 50 trials of 10 additional episodes in whichthere is no exploration and the stimuli is guaranteed to be found from the firstsearch. The error bars shows a clear statistical advantage of the co-articulatedpolicy (blue) over the sequential action policy (red).

on average than the sequential search-then-track policy seen inFigure 7(a). This is apparent by the fact that the blue line (theaverage reward for the co-articulation policy) in Figure 8(a)seems to be slightly higher, on average, than the red line (theaverage reward for the sequential policy).

Due to the high level of exploration in the learning episodes(20%) and the stochasticity in the search process, the improve-ment of the co-articulated policy over the single-action policyis not statistically significant, despite an apparent advantageseen in Figure 8(a). To show that the co-articulated policy is infact better, a simulation was performed in which both learnedpolicies were run under similar environmental conditions,except that exploration was turned off, and in the 50% ofthe cases where the robot had to employ SEARCH to findthe saturation stimuli, it was guaranteed to find it on the firsttry. Under these conditions, fifty trials of ten episodes wereperformed. The resulting average reward graphs are shown inFigure 8(b), along with the data’s variance. From this graphit is clear that the co-articulated policy achieves more rewardper state transition than the sequential policy.

E. Hierarchical Composition of Control Programs

Value functions are potential functions for discretestate/action spaces. As a result, performing greedy ascent


on a value function may lead an agent toward absorbingstates where Φ=0. The basis for hierarchy in the controlbasis framework depends on the abstraction of sensorimotorprograms—with all the internal state they require—in termsof the single, four-predicate state logic of Figure 5. Althougha program can have a significant amount of internal structure,the hierarchical learning agent views this program as a single,temporally extended control action whose state is representedthe same way as that of all other control actions. Control basisprograms are similar to reinforcement learning options [33]with their own state/action spaces.

We now present the results of additional learning stages inwhich Dexter used the affordance discovery reward functionto acquire hierarchical control basis programs. In each stage,Dexter learned a policy to uncover a new affordance usingthe ACCOMMODATE() procedure. Each of these programsemploys at least one other control basis program hierarchically.When a program is used as a hierarchical single action in anew learning program, we will treat it as having habituated,and will fix its policy in place. Furthermore, these programswill provide no additional reward from the intrinsic motivator.

F. Example: ReachTouch

The first hierarchical program Dexter learned asserts aTOUCH affordance by combining SEARCHTRACK with armcontrol tasks. We call this program REACHTOUCH, and itallows Dexter to engage highly-saturated objects that are notphysically presented to it.

A learning stage for Dexter was constructed to enableDexter to learn a policy for REACHTOUCH. The robotwas provided with three actions: SEARCHTRACK,REACH(SAT,RIGHTARM), and TOUCH(RIGHTHAND).The latter is defined as follows:

Touch uses the virtual spring potential φs to apply a smallmagnitude force in a desired reference direction. In thisexample, Dexter uses TOUCH constructed to control a smallreference force on each of the three fingers of its right handin order to grab objects. This controller is defined as:

TOUCH(RIGHTHAND) , c(φs, (fr,ref , fr,hand),θr,hand), (23)

where θr,hand ∈ C4 are the configuration variables of therobot’s right hand, fr,hand = [fr,1 fr,2 fr,3]T , capturing thesensed forces at each of the robot’s three right-hand finger-tips,fr,ref = [fref,1 fref,2 fref,3]T and fref,i is a reference vectorof 0.2N pointing in the inward (palm) direction of finger i.

TOUCH allows Dexter to hold simple objects like balls orboxes when they are placed close to the robot’s palm. Becauseof the geometric structure of Dexter’s hand, simultaneously“TOUCH-ing” the same object with all three fingers oftenforms a primitive grasp. This is the control basis analogof the palmer grasp reflex in humans, and demonstrateshow an embodied system’s morphology can be exploitedto accomplish behavior. We will call this primitive grasp a“grab.” It should be noted that more robust grasp controlstrategies exist that achieve wrench closure on objects [25],but we will be satisfied with TOUCH in the following

examples due to its simplicity.

These three actions construct an action space Art =SEARCHTRACK, REACH, TOUCH, REACH / TOUCH,TOUCH / REACH1 and a state vector s ∈ Srt wheres = (pst preach ptouch) and pst is the state predicate valueof the entire SEARCHTRACK program. Quiescence of theTOUCH(RIGHTHAND) controller is rewarding by the affor-dance discovery motivator because it signifies an affordancein the environment.

One trial of 25 learning episodes was conducted in thislearning stage to allow Dexter to learn the REACHTOUCHprogram using the ACCOMMODATE() procedure. During ap-proximately half of the learning episodes, a human presentedhighly-saturated objects to Dexter. In most of the otherepisodes the human delivered a highly-saturated object in frontof the robot in a location initially out of view of the robot’stwo cameras. During a small number of episodes (∼10%) noobject was presented to the robot. In the cases where the objectwas presented to the robot, it was able to grab it using TOUCH.

By the end of the learning trial, Dexter had learned apolicy for REACHTOUCH to uncover TOUCH affordances. Thetransition diagram showing the most likely state transitionsthat occur under the greedy policy for this program is shownin Figure 9. The robot starts out in state (XXX) and choosesthe action REACH / TOUCH. If the object is initially inthe field of view, the state transitions to (X0−), meaningthat there is a reach goal, but no reference to TOUCH. Ifthe object is not initially in the robot’s field of view, therobot transitions to state (X−−), from which it employsSEARCHTRACK to find a reach goal. When a goal is foundand tracked, the robot enters state (1XX) and REACH /TOUCH is tried once again, resulting in a transition to state(X0−). From this state, the robot continues executing REACH/ TOUCH until it either reaches state (X11), in which rewardis received by the intrinsic motivator, or the reach actionconverges without bringing the robot’s hand into contact withan object, resulting in a transition to state (X1−). This lattercase occurred in the small number of episodes in which therobot reached toward visual stimuli that did not pertain to anobject within its reachable workspace (e.g., the window inFigure 6(b)).

The simple policy for REACHTOUCH provides a robust wayfor Dexter to turn visual cues into spatial and tactile cues itcan engage in different ways. In [12], three additional controlprograms were presented that allowed Dexter to uncoveradditional visual and tactile affordances using the proposedframework. All three programs were learned by the robot instages of 25 learning episodes using the ACCOMMODATION()algorithm; all use the REACHTOUCH program hierarchically.The first program, BIMANUALTOUCH, allowed the robot toemploy a combination of kinematic conditioning actions tobring a grasped object in contact with its left hand to accom-plish a new TOUCH objective. Figure 10(a) shows the robot

1Co-articulated actions between primitives and programs were not allowed.How to co-articulate discrete state/action policies is an open research question,but see [28] for one possible approach.


X X X

1 X X!X - - !

X 1 -!

X 1 0 ! X 1 1 !X 0 0 !X 0 - !

X

-

1

0

SearchTrack!

Reach Touch

Reach Touch

Reach Touch

Fig. 9. This diagram shows the transition diagram for the policylearned for REACHTOUCH after 25 episodes. The state vector is s =(pst preach ptouch) where pst is the state value of the SEARCHTRACKprogram. The hierarchical use of SEARCHTRACK is indicated by the abstracttransition icon introduced in Figure 5.

at the conclusion of this program. The second program, calledVISUALINSPECT, allows the robot to optimize the measureof localizability with respect to grabbed objects, bringingthem to a location where small visual features that were notinitially visible can be tracked. Figure 10(b) shows the robotafter the conclusion of VISUALINSPECT. The third program,PICKANDPLACE, allowed the robot to pick an object andbring it into contact with another object (e.g., the greentable) using a Harmonic path-plan to prevent collisions withobstacles, and regulating the reaction forces that occur uponcontact. Figures 10(c) and 10(d) show the robot performingPICKANDPLACE, putting the yellow ball in the center of thegreen table.

(a) (b)

(c) (d)

Fig. 10. Frame (a) shows Dexter after the completion of BIMANUALTOUCH.Frame (b) shows Dexter after the completion of VISUALINSPECT. Frames (c)and (d) show Dexter during and after PICKANDPLACE.

IV. PROGRAM GENERALIZATION

We have seen how a robot can acquire new skills in aprocess of accommodation by restricting the environmental

context and the control expressions that it can explore. Suchscaffolding is an appropriate way for a human teacher tobootstrap intrinsically motivated behavior. It is necessary,however, to also consider how the robot can assimilate newcontexts into its knowledge structures to provide dexteroussolutions for achieving reward in more general situations.

In this section we address how a control basis program canbe transformed into a unit of behavior called a schema to pro-vide dexterous contingency plans in a variety of environmentalcontexts. This generalization is accomplished by factoringexisting control programs into declarative and proceduralcomponents [9]. The declarative structure of a program—capturing abstract information concerning which combinationof objectives are required to meet a behavioral goal—canbe transferred to different contexts. The procedural structureexamines the environmental conditions under which reward isreceived and dictates how resources should be allocated to the(abstract) declarative objectives at run-time.

Specifically, our approach allows a robot to find the sta-tistically reliable parameterizations of control basis actionsthat maintain the original typing constraints and transitiondynamics of policies that achieve reward. We next discuss howto factor controllers and control programs into abstract entitiesthat can be re-parameterized, and then discuss how proceduralpolicies can be found that increase the likelihood of achievingintrinsic reward in contexts different from those in which thepolicies were initially acquired.

A. Controller Abstraction

Let T be a set of sensory types, such as those seen inTable I. Control expressions in the control basis provide typingrequirements on the input sensors and output effectors that areused to compute control inputs. As a result, a potential functionφ ∈ Ωφ, when combined with a sensory signal σ ⊆ Ωσ witha characteristic input type (CIT) tin ∈ T , and an effectorresource τ ⊆ Ωτ with characteristic output type (COT) tout ∈T , represents a family of functionally equivalent controllerswe will call an abstract action, a(φ, tin, tout). For example,an abstract action using a harmonic potential φh represents aclass of control actions that provide collision-free motion plansin R3 to a manipulator configuration output in Cn. However,goals and obstacles in φh can be observations derived from alaser scanner, a stereo vision system, a tactile probe, or anyother equivalent sources of position information.

1) Example: SearchTrack Abstraction: The SEARCH-TRACK program that Dexter learned in Section III-D createdmodels of where training features (highly-saturated pixel re-gions) occurred and tracked them on the center of its leftcamera image plane. The strategy acquired, however, couldequally well be applied to different visual features. In a newlearning stage, Dexter executed its SEARCHTRACK policy foran additional fifty training episodes, this time building a modelof where regions of pixel motion occur in its pan/tilt configura-tion space and tracking such motions. This was accomplishedby “swapping out” the feedback signals to SEARCH andTRACK pertaining to high-saturation cues, γlsat ∈ Ωσ , withsignals pertaining to motion cues, γlmotion ∈ Ωσ . For 50% of


(a)

!"#$%"&'()*+),+-.*/0"1$+,)%+0%.2345+<)()*+7+89"#)54+:;++

!"#$%&'($

(b)

!"#$%&'"(&)*+$

,+'#+%,'-$

.&+$

)/'$

0$$$$$$$$$$$$$$$$$$$$$$$$$$$1$$$$$$$$$$$$$$$$$$$$$$$$$$$$2$$$$$$$$$$$$$$$$$$$$$$$$$$$$3$$$$$$$$$$$$$$$$$$$$$$$$$$$4$

5$$$$$$$$$$$$$$$$$$$$$$$$$$$6$$$$$$$$$$$$$$$$$$$$$$$$$$$$7$$$$$$$$$$$$$$$$$$$$$$$$$$$8$$$$$$$$$$$$$$$$$$$$$$$$$$09$

0$$$$$$$$$$$$$$$$$$$$$$$$$$$1$$$$$$$$$$$$$$$$$$$$$$$$$$$$2$$$$$$$$$$$$$$$$$$$$$$$$$$$$3$$$$$$$$$$$$$$$$$$$$$$$$$$$4$

5$$$$$$$$$$$$$$$$$$$$$$$$$$$6$$$$$$$$$$$$$$$$$$$$$$$$$$$$7$$$$$$$$$$$$$$$$$$$$$$$$$$$8$$$$$$$$$$$$$$$$$$$$$$$$$$09$

0$$$$$$$$$$$$$$$$$$$$$$$$$$$1$$$$$$$$$$$$$$$$$$$$$$$$$$$$2$$$$$$$$$$$$$$$$$$$$$$$$$$$$3$$$$$$$$$$$$$$$$$$$$$$$$$$$4$

5$$$$$$$$$$$$$$$$$$$$$$$$$$$6$$$$$$$$$$$$$$$$$$$$$$$$$$$$7$$$$$$$$$$$$$$$$$$$$$$$$$$$8$$$$$$$$$$$$$$$$$$$$$$$$$$09$

(c)

Fig. 11. Panel (a) shows Dexter’s left camera view while tracking motionduring a typical programming trial, and (b) shows the non-parametric dis-tribution of pan/tilt configurations learned for motion cues after 25 trainingepisodes. The single peak corresponds to the place where the experimenterpresented motion cues to the robot during the acquisition of SEARCHTRACK.The thirty panels in (c) show the ten hue, saturation, and intensity histograms,respectively.

these episodes, an object was shaken in front of the robot, asseen in Figure 11(a). The other half of the time, no object waspresented to the robot. Figure 11(b) shows the non-parametricdistribution Dexter learned during these episodes encodingwhich pan/tilt configurations track motion cues.

To show the wide applicability of SEARCHTRACK re-parameterization, Dexter was directed to explore headings to-wards regions of interest in thirty hue, saturation, and intensitychannels in Ωσ (10 channels each, discretizing the full space).The result of this exploration was that Dexter was able togain a comprehensive “understanding” of the affordances inits primitive visual environment. For this training situation,Dexter cycled through each of the thirty HSI channels, pa-rameterizing SEARCHTRACK accordingly, and gathering dataregarding where the environment affords tracking each visualsignal. Dexter attempted to acquire fifty positive samples of

each channel, but gave up if no valid heading was found afterten searches. Figure 11(c) show histograms of pan/tilt locationswhere regions of each of the thirty channels were trackable.Most channels produced some response from the environment,although a few did not (e.g., saturation channels 7 and 10, huechannels 8 and 9, etc.).

In conjunction, these visual signals can be used by Dexteras a primitive background model for what it expects to seefrom its cameras. Given that Dexter is a stationary robot, sucha model provides a prior on the entire visual environment therobot can expect to observe. Furthermore, deviations from thismodel can direct the robot’s attention towards new possibleaffordances. For example, any object placed in front of therobot—which will itself be comprised of a combination of theabove thirty channels—will result in some deviation from therobot’s prior model. It is easy to see how this deviation couldbe used to “trigger” further exploration—for example, therobot could try reaching out and touching the object, pickingit up, etc.

B. Schema

We now describe a process by which a robot can generalizeits acquired control programs to new situations in subsequentstages of learning that preserve the transitional structure (or“intentions”) of the original policies. As a robot generalizes itsprograms to support increasingly dexterous procedural contin-gency plans, these programs are transformed into knowledgestructures we call schema that can produce reward in manycontexts.

The proposed methodology for generalizing control basisprograms into schema is illustrated in Figure 12. A policyπ is first learned over a state and action space A and Susing specific sensory and motor allocations that work in thetraining context. The policy is then factored into declarative(abstract) and procedural components. The abstract actions arethen allocated with type-constrained resources based on theenvironmental context f ∈ F in order to preserve the originaltransition structure of π. Enforcing strict typing constraintsreduces the combinatorial space of possible resource combi-nations, making the search space more efficient for machinelearning algorithms to explore. We now discuss how thisprocess can be represented computationally.

Let the (ordered) declarative and procedural parts of aprioritized control law ci = c(φ0, σ0, τ0) / · · · / c(φn, σn, τn)be defined, respectively, as follows:

declarative(ci) = (a0, · · · , an) (24)procedural(ci) = (ω0, · · · , ωn) (25)

where each am is a single-objective abstract action consistingof a potential function with characteristic input and outputtypes, such that am = a(φm, type(σm), type(τm)), ωm isthe set of sensor and effector resources, ωm = 〈σm, τm〉, thatmeet the original CIT and COT typing constraints of cm, andm = 0 . . . n.

At run-time, each abstract action must be allocated withsensorimotor resources. Let procedural policy ψ for a schema


s0

s1

s3

s2

sensorimotor program

c(!0,"0,#0)

c(!1,"1,#1)

c(!2,"2,#2)

s0

s1

s3

s2

declarative program

a0 !"0,#0"

a1 !"1,#1 "

a2 !"2,#2 "

procedural resource allocation

factorization generalization

s0

s1

s3

s2

declarative program

a0

a1

a2

procedural resource allocation

!"4,#4 " !"3,#3 "

!"0,#0 "

f0

!"5,#5 "

!"2,#2 " f2

f1 !"1,#1 "

Fig. 12. Sensorimotor programs in the control basis can be factored intoprocedural and declarative components and generalized to new environmentalcontexts, by means of the policy ψ(ai, fj), where ai ∈ A and fj ∈ F .

be a mapping from abstract action a ∈ A, and context f ∈ F ,to a set of sensorimotor resources for each controller,

ψ : (a, f) 7→ (ω0, · · · , ωn). (26)

One useful procedural policy for a single-objective abstractaction a(φi, tin, tout) is defined as:

ψ(a, f) = argmaxωiPr(scur, sdes|ci, a, f) (27)

where ωi = 〈σi, τi〉, ci is a controller parameterized by φiand resource model ωi (that obeys the CIT and COT typingconstraints of original control action, such that type(σi) = tinand type(τi) = tout), scur is the current state, and sdes is thedesired next state along the route to a rewarding state underpolicy π. Policy ψ can be extended to multi-objective controllaws by finding the set of procedural parameterizations thatpreserve the desired state transitions.

Examining the procedural context of a particular controlbasis program also supports inferences regarding when rewardis unlikely to occur for any resource allocation. This likelihoodis captured by the probability of achieving reward for a givenpolicy and context, Pr(reward|π, f). Consider a case inwhich Dexter explores REACHTOUCH, but no object is presentin the reachable workspace: in this case, the probability shouldbe sufficiently low. In such a situation, the schema shouldreport that it is not in a region of its state-space where itsgoals can be met, and evaluate to the undefined “−” condition.It can then inform any higher-level program that is using ithierarchically that it is unlikely to be able to achieve its goals.

The procedure for how a control basis program can be gen-eralized into a schema with procedural contingency plans—called ASSIMILATE()—is provided by Algorithm 2. It issimilar to ACCOMMODATE() except that, instead of using Q-Learning to learn an action-value function Φ, it estimates apolicy ψ and various probability distributions that captureprocedural information. ASSIMILATE() takes as input theaction set A, a policy π that maps states in the state spaceS formed by the actions in A to those actions, a set ofresources Ω that can be used to re-parameterize the actionssuggested by π, the number of learning episodes m to beperformed, and a “timeout” parameter T that limits the numberof state transitions for each episode (set to 100 in the following

Algorithm 2 ASSIMILATE(A, π, Ω, m, T )1: Let A′ be the set of non-composite actions in A2: Let S be the state space formed from the predicates of

the actions in A′3: n← |A′|, ε = 0.24: for i = 1 to m do5: k ← 0, t← 06: reward← false7: while (reward = false) ∧ (k < T ) do8: observe features f ∈ F9: st ← (pt1 . . . p

tn)

10: a← declarative(π(st))11: ω ← ψ(a, f) (via ε-greedy selection), where ω ∈ Ω12: allocate a with ω to form control action c13: repeat14: execute c for one iteration15: t← t+ 116: st ← (pt1 . . . p

tn)

17: until st 6= st−1

18: k ← k + 119: update Pr(st−1, st|c, a, f)20: evaluate rt according to Equation 1721: if rt > 0 then22: reward← true23: update Pr(τ |σ, reward) for all rewarding control

actions c(φ, σ, τ),24: end if25: end while26: update Pr(reward|π, f)27: end for

experiments). Line 18 updates the probability distribution usedto evaluate ψ as seen in Equation 27. Line 25 updates theprobability of achieving reward for the given policy and theobserved run-time context f ∈ F .

1) Example: ReachTouch Procedural Knowledge: In thelast section, we demonstrated a program called REACHTOUCHthat Dexter acquired using ACCOMMODATE() in a constrainedsetting. This program provides a policy for the robot touncover TOUCH affordances in its environment using its righthand. The REACHTOUCH strategy, however, can be appliedequally well to uncover left-handed or bimanual TOUCHaffordances. We now present procedural contingency planslearned for REACHTOUCH using ASSIMILATE() demonstrat-ing common sense knowledge concerning handedness andconcerning when objects are out of reach.

During learning, objects of either 50cm or 10cm diameterwere presented to the robot in a variety of positions and witha variety of velocities. In half of the training episodes, a ballof the larger diameter was placed in front of the robot. Inthe rest of the episodes, a smaller ball was presented, placedin a stationary position to the left or right sides of the robotor on one side moving with a velocity of 0.05m/s in thedirection of the opposite hand. The larger objects requirea bimanual strategy to successfully track reference contactsignals. The smaller and moving objects require policies thatconsider handedness and anticipatory reaches. The object was


Y-pos > 0.02 m

vol > 0.008 m3

Y!

N!

N!

Y!

2-hand

left-hand right-hand

ReachTouch Procedural Policy

Y-vel < -0.2 m/s

N! Y!left-hand Y-vel > 0 m/s

Y! N!right-hand

(a)

ReachTouch Reward Condition

X-pos > 1.17 m

Y! N!false

Y-pos < -0.51 m

N! Y!false

Y-pos < 0.42 m

Y! N!true false

(b)

Fig. 13. The decision tree in (a) shows the resulting procedural policy forchoosing which arm to allocate for reaching based on object volume, position,and velocity. The tree in (b) shows conditions under which REACHTOUCH islikely to achieve reward. It captures when objects are out of the robot’s workspace in terms of their position.

occasionally presented outside of the robot’s initial field ofview, requiring the use of the SEARCHTRACK program.

Figure 13(a) shows a decision tree learned using the C4.5algorithm2 after one of the training trials on the robot toestimate ψ. Other trials produced similar results. This treeindicates that if the ball is large (i.e., it has appreciablevolume), then a 2-handed reach should be used. Moreover,small moving objects indicate that the robot should reachwith the arm that anticipates the movement, otherwise, theobject would move out of the workspace of the hand chosen.Stationary objects indicate the use of the hand on the sameside as the object. This policy reflects clear common senseknowledge about handedness, scale, and velocity concerningone- and two-hand REACHTOUCH options.

In an additional 50 learning episodes, objects of varioussizes were placed on the table in front of the robot, half of thetime outside the robot’s reach. The C4.5 algorithm was usedto learn a decision tree based on the position of the object con-cerning what contexts lead to reward using the REACHTOUCHpolicy and which do not. This tree (Figure 13(b)) reflects theprobability distribution Pr(reward|π, f) updated in Line 25of the ASSIMILATE() procedure. We see that for objects placedin x locations greater than 1.17m in front of the robot or toofar over to the robot’s side in the y-direction, the program isnot likely to achieve reward.

2) Example: PickAndPlace Procedural Knowledge: Inthe initial learning stage in which Dexter accommodatedPICKANDPLACE, both the object to be transported as wellas the goal location had to be placed sufficiently close tothe robot’s right hand in order for reward to be achieved. Incases where either were too far away, the program would notsucceed. In a new learning stage, we expanded the contextsin which Dexter applied PICKANDPLACE by placing a ballwith highly-saturated hues in unconstrained initial locationson the table and by exploring more places to transport it.During training, Dexter learned to assimilate these situations

2The C4.5 algorithm, [26], was chosen due to its intuitive and human-readable output. It should be noted, however, that there may be other efficientmeans of estimating the desired probabilities and policies, but a full discussionis ongoing research and beyond the scope of this paper.

PickAndPlace !"#$%&'"()*!#)+$,*

Yobj < -0.01 m

Yobj < 0.06 m

Ygoal < 0.45 m

Ygoal < -0.46 m

ReachTouch*

ReachTouch*

ReachTouch*BimanualTouch*

BimanualTouch*

n*

n*n*

n*

Y*

Y*Y*

Y*

Fig. 14. This learned PICKANDPLACE procedural policy.

into its PICKANDPLACE schema. “Place” goals were visuallydesignated by a blue-colored hue spot that was placed invarious locations on the table. In this expanded context, thereare situations in which a solely right-handed policy (as wasinitially learned) does not yield reward.

The C4.5 decision tree learner to learn a procedural policyfor initially grabbing the object according to the ASSIMI-LATE() procedure over the joint signal space that arises fromthe union of the “pick” and the “place” locations (i.e., theobject and goal locations, respectively). The resulting policyis shown in Figure 14. This decision tree distinguishes caseswhen the object and the goal are sufficiently far away fromeach other in the robot’s y-direction (lateral to the robot).In this case, it should invoke BIMANUALTOUCH in place ofREACHTOUCH. Because both these schema reward the sametype of affordance (TOUCH-ability), they both provide validre-parameterizations of the original declarative policy. In thecurrent training context, the BIMANUALTOUCH accomplishesa form of “hand-transfer” by picking up the object andbringing it into contact with the robot’s other hand to achievea grab. The resulting behavior in these situations is illustratedin the sequence of images shown in Figures 15(a)-15(d). Theresult is that the robot can accomplish a larger variety ofPICKANDPLACE tasks by leveraging a large amount of priorbehavioral knowledge.

V. DISCUSSION

In this paper we introduced a paradigm for programmingadaptive robot control strategies that can be applied in a varietyof contexts. This paradigm advocates three important princi-ples that are related to recent work in the community. The firstprinciple is that behavior should be learned incrementally (ordevelopmentally) and acquired in learning stages that providenew contexts for either accommodating or assimilating newschema. Computational methods for developmental learningin robotic systems were proposed by a number of researchersincluding [30] and [1], and have recently enjoyed a greatdeal of attention in the literature (cf. [17]). The frameworkpresented in this paper contributes to the literature a devel-opmental strategy for acquiring behavior that is inherentlygrounded in a robot’s ability to assemble closed-loop strategiesfor interactive behavior.

The second principle the proposed paradigm advocates isthat control strategies should be learned through an intrinsicmotivator that rewards behavior in ways measurable to the


(a) (b) (c) (d)

Fig. 15. A procedural adaptation for PICKANDPLACE. In (a) the (blue-colored) goal is placed far to the robot’s left, while the object is on its right. (b) and(c) show the robot using BIMANUALTOUCH to pick up the object and pass it between its hands so that it can be brought to this far-away goal location (d).

robot, not just to the human programmer. This approach is indirect contrast to much of the recent work in robot learning thatfocuses on programming by demonstration [3] and imitationlearning [31]. In this work, a robot learns a skill based ona small number of demonstrated examples provided by theteacher. The robot uses the exemplars—often in conjunctionwith a task-specific reward function tailored to the domain—to bootstrap online exploration as it learns how to reliablyaccomplish the goal. Similarly, work in inverse reinforcementlearning allows an agent to extract a reward function fromexample trajectories that can, in turn, be used to learn robustpolicies [22]. The approach we take in this document doesnot rely on demonstrations by a teacher. Instead, the intrinsicreward function for affordance discovery we have presentedis designed to provide a robot with a means to acquire newskills autonomously and with minimal guidance.

The third principle we advocate is that a robot’s knowledgeshould be grounded in the affordances it discovers in itsenvironment. There has been much recent work in affor-dance learning for robots. Montesano et al. and the Multi-Sensory Autonomous Cognitive Systems group have providedformalisms that capture the statistical relationships betweentaking actions on objects and observing the effects [20],[29]. A number of researchers have examined affordance-like structures called “Object-Action Complexes” as the basisfor use in higher-level planning tasks [5], [15], [16]. Theseapproaches are similar in that they formulate robot actionsas pre-programmed activities that require little or no sensoryfeedback to measure “success.” In contrast, this paper con-tributes a novel approach in which the behavioral affordancesare explicitly grounded in the robot’s dynamic sensorimotorinteractions with its environment.

Taken together, these principles support an integrated ap-proach to robot learning that makes the first steps towardrobot systems that learn over the long term with minimalguidance from human programmers—something that has beenelusive to artificial intelligence researchers to this point. Weargue that any robot operating for long periods of time inunstructured and real-world environments must be endowedwith an ability to continually learn new skills on its own, aswell as to adapt existing skills for novel situations previouslyunseen. We believe that the framework presented in this paperprovides formal methodologies for achieving these goals.

ACKNOWLEDGEMENTS

This work was supported by NSF award SGER IIS-0847895. The authors would like to thank Shiraj Sen and ArjanGijsberts for their useful feedback.

REFERENCES

[1] M. Asada, K. MacDorman, H. Ishiguro, and Y. Kuniyoshi, “Cognitivedevelopmental robotics as a new paradigm for the design of humanoidrobots,” Robotics and Autonomous Systems, vol. 37, no. 2, pp. 185–193,2001.

[2] N. Bernstein, “On dexterity and its development,” in Dexterity and itsDevelopment, M. Latash and M. Turvey, Eds. Mahwah, N.J.: LaurenceErlbaum Associates Inc., 1996.

[3] A. Billard, S. Calinon, R. Dillmann, and S. Schaal, “Robot programmingby demonstration,” in Handbook of Robotics, B. Siciliano and O. Khatib,Eds. Secaucus, NJ, USA: Springer, 2008, pp. 1371–1394.

[4] C. Connolly, J. Burns, and R. Weiss, “Path planning using laplace’sequation,” in Proceedings of the IEEE Conference on Robotics andAutomation (ICRA), 1990.

[5] C. Geib, K. Mourao, R. Petrick, N. Pugeault, M. Steedman, N. Kruger,and F. Worgotter, “Object action complexes as an interface for planningand robot control,” in Workshop ‘Toward Cognitive Humanoid Robots’at IEEE-RAS International Conference on Humanoid Robots, Genoa,Italy, 2006.

[6] J. J. Gibson, “The theory of affordances,” in Perceiving, acting andknowing: toward an ecological psychology. Hillsdale, NJ: LawrenceErlbaum Associates Publishers, 1977, pp. 67–82.

[7] S. Hart, “An intrinsic reward for affordance exploration,” in Proceedingsof the 8th IEEE International Conference on Development and Learning(ICDL), Shanghai, China, 2009.

[8] S. Hart and R. Grupen, “Natural task decomposition with intrinsicpotential fields,” in Proceedings of the 2007 International Conferenceon Intelligent Robots and Systems (IROS), San Diego, California, 2007.

[9] S. Hart, S. Sen, and R. Grupen, “Generalization and transfer inrobot control,” in 8th International Conference on Epigenetic Robotics(Epirob08), University of Sussex, Brighton, UK, 2008.

[10] ——, “Intrinsically motivated hierarchical manipulation,” in Proceed-ings of the 2008 IEEE Conference on Robots and Automation (ICRA),Pasadena, California, 2008.

[11] S. Hart, S. Sen, S. Ou, and R. Grupen, “The control basis api - a layeredsoftware architecture for autonomous robot learning,” in 2009 Workshopon Software Development and Integration in Robotics (SDIR) at theIEEE Conference on Robots and Automation (ICRA), Kobe, Japan, 2009.

[12] S. W. Hart, “The development of hierarchical knowledge in robot sys-tems,” Ph.D. dissertation, Department of Computer Science, Universityof Massachusetts Amherst, 2009.

[13] M. Huber, “A hybrid architecture for adaptive robot control,” Ph.D. dis-sertation, Department of Computer Science, University of MassachusettsAmherst, 2000.

[14] D. Koditschek and E. Rimon, “Robot navigation functions on manifoldswith boundary,” Advances in Applied Mathematics, vol. 11, no. 4, pp.412–442, 1990.


[15] D. Kraft, N. Pugeault, E. Baseski, M. Popovic, D. Kragic, S. Kalkan,F. Worgotter, and N. Kruger, “Birth of the object: Detection of ob-jectness and extraction of shape through object action complexes,” inProceedings of the 2008 International Conference on Cognitive Systems,Karlsruhe, Germany, 2008.

[16] N. Kruger, J. Piater, F. Worgotter, C. Geib, R. Petrick, M. Steedman,A. Ude, T. Asfour, D. Kraft, D. Omrcen, B. Hommel, A. Agostino,D. Kragic, J. Eklundh, V. Kruger, and R. Dillmann, “A formal definitionof object action complexes and examples at different levels of the processhierarchy,” http://www.paco-plus.org, 2009.

[17] M. Lungarella, G. Metta, R. Pfeifer, and G. Sandini, “Developmentalrobotics: A survey,” Connection Science, vol. 15, no. 4, 2003.

[18] G. Metta, P. Fitzpatrick, and L. Natale, “Yarp: yet another robotplatform,” Int. Journal on Advanced Robotics Systems, Special Issueon Software Development and Integration in Robotics, March 2006.

[19] Microsoft Co., “Microsoft robotics developers studio,”http://msdn.microsoft.com/en-us/robotics/, 2008.

[20] L. Montesano, M. Lopes, A. Bernardino, and J. Santos-Victor, “Learningobject affordances: From sensory-motor coordination to imitation,” IEEETransactions on Robotics, vol. 24, no. 1, 2008.

[21] Y. Nakamura, Advanced Robotics: Redundancy and Optimization.Addison-Wesley, 1991.

[22] A. Y. Ng and S. Russell, “Algorithms for inverse reinforcement learn-ing,” in Proceedings of the 17th International Conference on MachineLearning, 2000.

[23] J. S. Ostroff and W. M. Wonham, “A temporal logic approach to realtime control,” 24th IEEE Conference on Decision and Control, vol. 24,pp. 656–657, Dec. 1985.

[24] J. Piaget, The Origins of Intelligence in Childhood. InternationalUniversities Press, 1952.

[25] R. Platt, A. H. Fagg, and R. Grupen, “Nullspace composition of controllaws for grasping,” in International Conference on Intelligent Robotsand Systems (IROS). Laussane, Switzerland: IEEE/RSJ, 2002.

[26] J. Quinlan, C4.5: Programs for Machine Learning. San Mateo, CA.:Morgan Kaufmann Publishers, 1993.

[27] E. Rimon and D. Koditschek, “Exact robot navigation using artificialpotential functions,” IEEE Transactions on Robotics and Automation,vol. 8, no. 5, pp. 501–518, 1992.

[28] K. Rohanimanesh and S. Mahadevan, “Coarticulation: An approachfor generating concurrent plans in markov decision processes,” inProceedings of the 22nd International Conference on Machine Learning(ICML-2005), Bonn, Germany, 2005.

[29] E. Sahin, M. Cakmak, M. Dogar, E. Ugur, and G. Ucoluk, “To afford ofnot to afford: A formalization of affordances toward affordance-basedrobot control,” Adaptive Behavior, vol. 4, no. 15, pp. 447–472, 2007.

[30] G. Sandini, G. Metta, and J. Konczak, “Human sensori-motor develop-ment and artificial systems,” in Proceedings of AIR & IHAS, 1997.

[31] S. Schaal, “Is imitation learning the route to humanoid robots?” Trendsin Cognitive Sciences, 1999.

[32] R. Sutton and A. Barto, Reinforcement Learning. Cambridge, Mas-sachusetts: MIT Press, 1998.

[33] R. Sutton, D. Precup, and S. Singh, “Between MDPs and semi-MDPs:A framework for temporal abstraction in reinforcement learning,”Artificial Intelligence, vol. 112, no. 1-2, pp. 181–211, 1999. [Online].Available: citeseer.ist.psu.edu/sutton99between.html

[34] J. Uppala, D. Karuppiah, M. Brewer, C. Ravela, and R. Grupen, “Onviewpoint control,” in International IEEE Conference on Robotics andAutomation, 2002.

[35] D. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no.3,4, pp. 279–292, 1992.

[36] T. Yoshikawa, “Manipulability of robotic mechanisms,” The Interna-tional Journal of Robotics Research, vol. 4, pp. 3–9, 1985.

Stephen Hart is a Post-Doctoral researcher at theItalian Institute of Technology (IIT) investigating de-velopmental learning strategies on the iCub robot. InNovember 2010, he will be joining General Motorsas a research scientist to work with the NASA/GMRobonaut 2. Dr. Hart received his M.S. and Ph.D. atthe Laboratory for Perceptual Robotics at Universityof Massachusetts Amherst in 2009. He received hisB.S. in Computer Systems Engineering in 2002 alsoat UMass Amherst. His work focuses on examininghow robots can be intrinsically motivated to learn

robust control strategies for manipulation.

Roderic Grupen is a professor of Computer Scienceat the University of Massachusetts Amherst wherehe directs the Laboratory for Perceptual Robotics.He conducts research on robotics and autonomousmethods for cumulative learning during extendedinteractions with the environment. His work focuseson applications for personal robotics, mobile ma-nipulation, and healthcare. Grupen received a B.A.in Physics from Franklin and Marshall College, aB.S. in Mechanical Engineering from WashingtonUniversity, a M.S. in Mechanical Engineering from

Penn State University, and a Ph.D. in Computer Science from the Universityof Utah in 1988. He is an editor for AIEDAM (AI in Engineering Designand Manufacture), and Co-Editor-in-Chief Robotics and Autonomous SystemsJournal.

Documents

2010 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL ......2010 IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT 1 Learning Generalizable Control Programs Stephen Hart and Roderic Grupen