Lecture XIII: MLSC - Dr. Sethu Vijayakumar 1

Lecture XIIIDynamical Systems as Movement Policies

Contents:• Differential Equation• Force Fields, Velocity Fields• Dynamical systems for Trajectory Plans

• Generating plans dynamically • Fitting (or modifying) plans• Imitation based learning

Thanks to my collaborator Auke Ijspeert (EPFL) for many of the contents on the slides for this lecture.

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 2

Movement policies as Dynamical Systems

• Represent complex movements in globally stableattractor landscapes of nonlinear autonomous differential equations

• Choose kinematic representation for easy re-use in different workspace location

• Ensure easy temporal and spatial scaling (topological equivalence)

• Use local learning to modify the attractors according to demonstration of teacher and self-learning

Discreet & Rhythmic Movement Primitives

( , , )des desf goalτ =x x x&

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 3

Discreet & Rhythmic Movement superposition

Open loop with oscillators Closed loop control in horizontal plane

Open (vertical) + Closed (horizontal) loop control

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 4

What is a Differential Equation?

• Differential equation: an equation that describes how state variables evolve over time, for instance:

)( ycy −= α&

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 5

Some Definitions

• Ordinary differential equation: differential equation thatinvolves only ordinary derivatives (as opposed to partial derivatives)

• Autonomous equation: differential equation that does not(explicitely) depend on time

• Linear differential equation: differential equation in which thestate variables only appear in linear combinations

• Nonlinear differential equation: differential equation in whichsome state variables appear in nonlinear combinations (e.g. products, cosine,…)

• Fixed point: point at which all derivatives are zero (can be an attractor, a repeller, or a saddle point, cf later)

• Limit cycle: periodic isolated closed trajectory (can only occur in nonlinear systems)

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 6

Interesting Regimes of Differential Equations

From Strogatz 1994

Limit cycles ChaosAttractors

Attractor

saddles

Unstable node

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 7

First Order Linear Systems

• First order linear system:

• How to solve this equation, for a given y(t=0), c, and α ?

• Two methods: analytical solution or numerical integration

• Analytical solution:

• Numerical integration: Euler method, Runge-Kutta,…

)( ycy −= α&

ctcyty +−−= )exp()()( 0 α

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 8

First Order Linear Systems

)( ycy −= α&

ctcyty +−−= )exp()()( 0 α

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 9

Second Order Linear Systems

yxyxcy

=−−=

&

& ))((βα

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 10

Second Order Linear Systems

yxyxcy

=−−=

&

& ))((βα

88

==

βα

18

==

βα

28

==

βα

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 11

Second Order Non-linear System

xyxyxy

coscos2coscos2

−−=−−=

&

&

Attractor

saddles

Repeller

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 12

Learning a movement by demonstration

Movement recognition

Movement execution

(Inv. Dyn.)

Trajectory formation

system

Joint angles

Task of the trajectory formation system: • To encode demonstrated trajectories with high accuracy, • To be able to modulate the learned trajectory when:

• Perceptual variables are varied (e.g. timing, amplitude)• Perturbations occur

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 13

Encoding a trajectory

Traditionally, the problem of replaying a trajectory has been decomposedinto two different issues:• One of encoding the trajectory, and• One of modifying the trajectory, for instance, in case the movement is

perturbed, or when it requires to be modulated.

Our approach: combine both abilities in a nonlinear dynamical system

Aim: to encode the trajectory in a nonlinear dynamical system with welldefined attractor landscape

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 14

Nonlinear Dynamical Systems Approach

y

dy/dt

Single point attractor

Discrete movements

y

dy/dt

Limit cycle attractor

Rhythmic movements

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 15

Two types of movement recordings

« Kinesthetic » demonstrationSensuit

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 16

Pointing Demonstration

position velocity

Demo forOne DOF

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 17

Shaping Attractor Landscapes

Goal: g

( )( )( )( )?

z zz g y zy f z

α β= − −

= +

&

&

( )( )z zz g y zy z

α β= − −=

&

&

Can one create more complex dynamics by non-linearly modifying the dynamic system:

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 18

Discreet Control Policy

Goal: g

Amplitude and phase system

Output System

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 19

Shaping attractor landscapes

( )( )z zz g y zy z

α β= − −=

&

&

y

dy/d

tg = goal Can one create more

complex dynamics by non-linearly modifying the above dynamic system:

( )( )( )( )?

z zz g y zy f z

α β= − −

= +

&

&

g = goal

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 20

Shaping Attractor Landscapes

A globally stable learnable nonlinear point attractor:

( )( )( )( )

( )( )

( )

( )

1

1

2 0

0

,where

,

1exp and 2

z z

y

v v

xk

i ii

k

ii

i i i

z g y zy f x v z

v g x vx v

w b vf x v

w

x xw d x c xg x

α βα

α βα

=

=

= − −

= +

= − −=

=

−⎛ ⎞= − − =⎜ ⎟ −⎝ ⎠

∑

∑

&

&

&

&

Trajectory PlanDynamics

Canonical Dynamics

Local LinearModel Approx.

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 21

Learning the Attractor

Given a demonstrated trajectory y(t)demo and a goal g

Extract movement durationAdjust time constants of canonical dynamics to movement durationUse LWL to learn supervised problem

( )target ,demo

y

yy z f x vα

= − =&

&

[ Stefan Schaal, Sethu Vijayakumar et al, Proc. of Intl. Symp. Rob. Res.(ISRR) (2001) ] Also extended to rhythmic primitives :

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 22

Trajectory following & Generalization

Backhand Demonstration Backhand Reproduction

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 23

Modulation of Goal: Anchor

Lecture XIII: MLSC - Dr. Sethu Vijayakumar 24

Drumming: Modulating Frequency

Drumming: Kinesthetic Demo

Imitation and Modulation