Motor Control. Beyond babbling Three problems with motor babbling: –Random exploration is slow –Error-based learning algorithms are faster but error signals

Motor Control

Beyond babbling

• Three problems with motor babbling:– Random exploration is slow– Error-based learning algorithms are faster but

error signals are available in sensory coordinates only

– Real arms have two many degrees of freedom

Degrees of freedom

1

2

1

2

Averageposition

General Learning Principles

W

X

Y Yd

2

2

1

1,

212

d d

Nd

i ii

d

E f Y Y Y Y

Y Y

E E YW

W Y WY

Y YW

Error

This works if we have equations for this term…

General Learning Principles

W

X*

Y XE

WW

Error

?

?

Forward Modeling

Cortex: InverseModel

Cortex: InverseModel

Desired Position of the hand

Reaching Motor command

ArmArm

Hand Displacement

Forward Modeling

InverseModel

InverseModel

Desired Sensory Change

Motor command

PlantPlant

Sensory Change

InverseModel

InverseModel

Desired Sensory Change: Sd

Motor command (M)

ArmArm

Sensory Change: : S -Sd- S)

212

d

ijij

E S S

Ew

w

Learning starts with desired change, not a spontaneous movement.

Forward Modeling

Forward Modeling

InverseModel

InverseModel

Desired Sensory Change: Sd

Motor command (M)

ArmArm

Sensory Change: : S -ij

ij

ij

Ew

w

E MM w

Major problem:How do we compute

?EM

Sd- S)

Forward Modeling

InverseModel

InverseModel

ForwardModel

ForwardModel

Desired Sensory Change: : Sd

Predicted Sensory Change: : Sp

Motor command

PlantPlant

Sensory Change: S-Sd- Sp)

Forward Modeling

We can’t compute

but we can use instead,

ijij

E Mw

M w

212

P

ijij

P d P

E Mw

M w

E S S

Forward Modeling

This works as long as:

This means that the forward model only needs to be approximately correct

0PE E

M M

Forward Modeling

InverseModel

InverseModel

ForwardModel

ForwardModel

Desired Sensory Change: : S*

Predicted Sensory Change: Sp

Motor command

PlantPlant

Sensory Change: S - S- Sp)

Training the forward model with motor babbling using prediction error

Forward Modeling

InverseModel

InverseModel

ForwardModel

ForwardModel



Motor command

- Sd- Sp)

Off line training of inverse model with predicted performance error

Forward Modeling

InverseModel

InverseModel

ForwardModel

ForwardModel



Motor command

PlantPlant

Sensory Change: S - Sd- S)

On line training of inverse models using performance error

Forward Modeling

• Forward models can be used to predict the consequences of motor commands

• The prediction can be used to drive a linear inverse model (e.g. Jacobian) since they are not subject to long delays

• The prediction can be subtracted from current sensory input to compute the prediction error.

• Errors can be used to improve prediction.

Modular Approach

• Sensorimotor transformations are context dependent, e.g., the force required to move an object depends on the mass of the objects.

• A central controller for all contexts might not be feasible and would take too many resources

• Alternative: use a family of controllers and mix them smoothly across contexts.

Responsibility Estim

ator

Modular Approach

1t

*t tu u

X

X

XX

+

2t

3t

11tu

1t

*tx

1 11

ˆn

i it t t

i

u u

InverseModel 1

fbtu

or

Modular Approach

• The weights of the inverse model are adjusted according to:

*i

i i itt t t ti

t

duu u

d

Responsibility Estim

ator

Modular Approach

1ˆtx

tx 1t

ForwardModel 1

X

X1

+

-

XX

+

ut

2t

3ttx

2/erre

11ˆtx

1t 2

2

ˆ

ˆ

1

it t

jt t

x x

it n

x x

j

e

e

• To compute the responsibility, use the forward models

Favors the model which makes the best prediction at the previous time step

Modular Approach

• The weights of the forward model are adjusted according to:

ˆˆ

ii i itt t t ti

t

dxw x x

dw

Modular Approach

• We can also add a responsibility predictor.

where yt are the sensory cues used for the predictions.

ˆ ,

ˆˆ

i it t t

ii i itt t ti

t

y

d

d

Modular Approach

• The overall responsibility is computed according to:

plays the role of a likelihood function, while is the prior.

2

2

ˆ

ˆ

1

ˆ

ˆ

it t

jt t

x xii tt n

x xjt

j

e

e

ˆit

2ˆi

t tx xe

Controlling a trajectory

• The inverse problem (inverse dynamics)

• Ex: Controlling the trajectory of a spaceship

0 0

Forward Model

( )

Inverse Model

t t

x t a d d

F t ma t mx t

x t

F t



• Ex: a multi-joint arm ((t): joint torques)

1

Forward Model

, ,

Inverse Model

, ,D D D

x t x t x t H t

t H x t x t x t

, ,D D Dx t x t x t

t


• The inverse problem (inverse dynamics) – Feedback models– Feedforward models– Equilibrium point models


• The inverse problem: feedback models• Generate force according to the difference

between desired position and actual position (easy to compute in linear systems).

• Unstable because of sensory delays• Works better if the “actual position” is internally

estimated by a forward model• Adapts on the fly to change of context• Popular model of the oculomotor system


• The inverse problem: feedforward models• Estimating torque and its differentials from

a desired trajectory is a nonlinear mapping• Use a basis function network to implement

the mapping• Subject to the curse of dimensionality• Unable to adapt to change of context

(unless you add context units…)



• Ex: a multi-joint arm ((t): joint torques)

1

Forward Model

, ,

Inverse Model

, ,D D D

x t x t x t H t

t H x t x t x t

, ,D D Dx t x t x t

t


• Equilibrium models

• Basic idea: the motor system only specifies the muscle length that maintain the arm at the desired location (in other words, it only specifies the joint coordinates).


• Equilibrium models• Monkeys can reach accurately without

proprioceptive or visual feedback• If the arm is moved to the end point right at

the onset of a movement, it goes back to the starting point and resume its trajectory. This implies that trajectory are specified by moving the equilibrium point


• Equilibrium models

• How do you control trajectories? The motor system may specify the trajectory of the equilibrium point (virtual trajectory). Unless muscles are very stiff, the actual trajectory will end up being very different. This makes it difficult to control trajectories very precisely.


• There is an infinite number of trajectories between two points. How do we choose one? Do you have to specify one?

– Minimum Jerk (Flash & Hogan)– Maximum accuracy (Harris & Wolpert)– Optimal control (Todorov & Jordan)


• Minimum Jerk

• Goal: find (t) minimizing C. The solution can be found using calculus of variations.

0

23

3

ft

t

dC dt

dt


• Minimum Jerk

• It’s unclear how Jerk is computed in the brain.

• No principled explanation for why the brain would minimize such a quantity.


• Maximum accuracy• Hypothesis: trajectory are optimized to

maximize accuracy• Assumption: motor commands are corrupted by

noise with standard deviation proportional to mean (this is different from Poisson noise!!)

• Question: for a fixed duration and amplitude of a movement, what’s the optimal control signal?


• Maximum accuracy

• Large control signals lead to fast but inaccurate movements

• Small control signals lead to accurate but slow movements.

• Goal: select the control signal that leads to maximum accuracy for a given duration.



• wt: white noise with mean zero and variance kut2

1

11

1 01

t t t t

tt t i

t i ii

x x u w

x x u w

A B

A A B

11

01

11 1 2

0

tt t i

t ii

tTt i t i

t ii

E x x u

Cov x k u

A A B

A B A B



• Goal: minimize cov[xt] under the constraint that E[xt] is equal the desired location for several time steps after the end of the movement. Quadratic problem.

11

01

11 1 2

0

tt t i

t ii

tTt i t i

t ii

E x x u

Cov x k u

A A B

A B A B



• Eye Movements



• Arm Movements

0 100 2000

50

100

150

200

250

Time (ms)

Vel

oci

ty (

cm s

–1 )

0 100 2000

50

100

150

200

250

Time (ms)

0 100 2000

50

100

150

200

250

Time (ms)

a b c O bserved Predicted Param etricsensit iv ity


• Optimal motor control (Todorov-Jordan)• Experts show a lot of variance in their

movements but high accuracy on end points

• Indeed, there are directions in motor space that induce no variance in end points, because of the large number of degrees of freedom


• Optimal motor control (Todorov-Jordan)

• Choose trajectory with maximum accuracy and minimum effort


final

2final final *1 2 0

, 1,2

arg min

i i i i ix ax u u i

E x x X

u

x1

x2

1St solution: bring X to a value X* such that x1*=(X0*)1, x2*=(X0*)2

(X0*)2

(X0*)1

Line where the constraint is verified

*1 2 0x x X


final

2final final *1 2 0

, 1,2

arg min

i i i i ix ax u u i

E x x X

u

x1

x2

(X0*)2

(X0*)1

Additional noise

1St solution: bring X to a value X* such that x1*=(X0*)1, x2*=(X0*)2


final

2final final * 2 21 2 0 1 2

, 1,2

arg min

i i i i ix ax u u i

E x x X r u u

u

x1

x2

2nd solution: To minimize effort, go to the closest point such that x1*+x2*=X0*


final

2final final * 2 21 2 0 1 2

, 1,2

arg min

i i i i ix ax u u i

E x x X r u u

u

x1

x2

2nd solution: To minimize effort, go to the closest point such that x1*+x2*=X0*


final

2final final * 2 21 2 0 1 2

, 1,2

arg min

i i i i ix ax u u i

E x x X r u u

u

x1 x1

x2x2


• Optimal motor control (Todorov-Jordan)

X 1+X 2

-X*

Task e

rror

X1 -X

2

More task errorLess variability in solutions

Open question

• How does the nervous system compute the solutions to those optimization problems?

• Is it done off line (i.e., does the CNS specify a trajectory?) or on line? Attractor nets?

Documents

Motor Control. Beyond babbling Three problems with motor babbling: –Random exploration is slow –Error-based learning algorithms are faster but error signals