Visual feedback in the control of reaching movements

David Knill

and

Jeff Saunders

Two types of motor control

• Ballistic

• Feedback control

Motor planning

Physicalplant

Target state

Initial system state

Motorcommands New System

states

Ballistic control

Feedback control

Motor planning

Physicalplant

Target state

Initial System state

Motorcommands New System

states

Sensorysystem

Baseball examples

• Ballistic control– Hitting– Throwing

• Feedback control– Running to catch a ball

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

Target

0 100 200 300 400 500 600 700 8000

100

200

300

400

500

600

700

Time (msecs.)

X position (mm)

EOG signal

Finger position

Eye-hand coordination

0 100 200 300 400 500 600 700 8000

100

200

300

400

500

600

700

Time (msecs.)

X position (mm)

EOG signal

Finger position

Movement start

0 100 200 300 400 500 600 700 8000

100

200

300

400

500

600

700

800

900

1000

Time (msecs.)

Speed (mm/sec)

Speed profile for pointing movement

Motor planning

Physicalplant

Target state

Initial system state

Motorcommands New System

states

Ballistic control

Questions

• Does the visuo-motor system use visual information about target on-line to update motor program?

Feedback control

Motor planning

Physicalplant

Target state

Initial System state

Motorcommands New System

states

Sensorysystem

Questions

• Does the visuo-motor system use visual information about target on-line to update motor program?

• Does the visuo-motor system use continuous feedback from the hand during a movement to control the movement

Questions

• Does the visuo-motor system use visual information about target on-line to update motor program?

• Does the visuo-motor system use continuous feedback from the hand during a movement to control the movement– What visual information is used?

Question

• Does the visuo-motor system use visual information about target on-line to update motor program?

Question

• Does the visuo-motor system use visual information about target on-line to update motor program?

• Yes - for detectable target motion (e.g. catching a moving object)

Question

• Does the visuo-motor system use visual information about target on-line to update motor program?

• Yes - for detectable target motion (e.g. catching a moving object)

• ?? - for imperceptible changes in target position

Experiment

• Perturb position of target during a saccade (imperceptible change)

• Does motor system correct for change in target position?

infrared markerson finger

tabletop aligned to virtual targets

monitor

mirror

0 50 100 150 200 250 300 350 400-20

-15

-10

-5

0

5

10

15

X (mm)

Y (mm)

Perturbed trials

Unperturbedtrials

0 50 100 150 200 250 300 350

-100

-50

0

50

100

X (mm)

Y (mm)

Results

• Automatically correct for imperceptible target perturbations.

• Correct for perturbations – Perpendicular to movement– In direction of movement

• Reaction time = 150 ms

• Smooth corrections

Question

• Does the visuo-motor system use continuous feedback from the hand during a movement to control the movement?

Hypotheses

• Classic model – Ballistic control during “fast” phase of motion– Feedback control during end, “slow” phase of

motion

• Continuous model– Feedback control throughout movement

Arguments against continuous feedback

• Visuo-motor delay (~100 ms) is too large for effective control during fast phase.

• Removing vision of hand early in motion does not affect end-point error.

• Corrections to target perturbations are just as strong with or w/o vision of hand.

Experiment

• Imperceptibly perturb the position of the hand during a movement and measure motor response.

• Add perturbations early and late in pointing movement.

• Measure reaction time to perturbations.

infrared markerson finger

tabletop aligned to virtual targets

monitor

mirror

(c)

(b)

target

unseenhand

virtual fingertip

(a)

Reaction time predictions

End-phasefeedback

Continuousfeedback

Lateperturbation

Earlyperturbation

X(cm)

0 5 10 15 20 25 30

2

0

-2

X(cm)

0 5 10 15 20 25 30

2

0

-2

X(cm)

0 5 10 15 20 25 30

2

0

-2X(cm)

0 5 10 15 20 25 30

2

0

-2

X(cm)

0 5 10 15 20 25 30

2

0

-2X(cm)

0 5 10 15 20 25 30

2

0

-2

Fast reaches (~450ms)

Fast reaches (~450ms) Slow reaches (~600ms)

Slow reaches (~600ms)

Early perturbation

Mid-reach perturbation

Early perturbation

Mid-reach perturbation

Early perturbation

Mid-reach perturbation

Early perturbation

Mid-reach perturbation

X(cm)

0 5 10 15 20 25 30

2

0

-2

X(cm)

0 5 10 15 20 25 30

2

0

-2

-5 0 5 10 15 20 25 30-8

-6

-4

-2

0

2

4

6

X position (cm)

Y position (cm)

Sample finger paths

Autoregressive model

• Baseline (unperturbed) trajectories

• Perturbed trials

y(t) =w8y(t−8)+w7y(t−7)+L +w1y(t−1)

y(t) =w8y(t−8)+w7y(t−7)+L +w1y(t−1)+wP (t)ΔY

0

0 100 200 300 400 500time after perturbation (ms)

0

0 100 200 300 400 500time after perturbation (ms)

0 100 200 300 400 500time after perturbation (ms)

0

0 100 200 300 400 500time after perturbation (ms)

0

0 100 200 300 400 500time after perturbation (ms)

0

0 100 200 300 400 500time after perturbation (ms)

Fast reaches (~450ms)

Fast reaches (~450ms) Slow reaches (~600ms)

Slow reaches (~600ms)

Early perturbation

Mid-reach perturbation

Early perturbation

Mid-reach perturbation

Early perturbation

Mid-reach perturbation

Early perturbation

Mid-reach perturbation

0

0 100 200 300 400 500time after perturbation (ms)

0

0 100 200 300 400 500time after perturbation (ms)

0

Early Mid-reach0

20

40

60

80

100

120

140

160

180

200

Fast reaches

Early Mid-reach

Slow reaches

N = 6 N = 6

0 100 200 300 400 500 600 700 800 900 10000

5

10

15

20

25

30

Time (ms)

X position (cm)

Subject 1: Trajectories for early perturbed trials

Positive perturbations

Negative perturbations

0 100 200 300 400 500 600 700 800 900 100020

21

22

23

24

25

26

27

28

29

30

Time (ms)

X position (cm)

Subject 1: Trajectories for early perturbed trials

Positive perturbations

Negative perturbations

0 100 200 300 400 500 600 700 800 900 10000

5

10

15

20

25

30

Time (ms)

X position (cm)

Subject 1: Trajectories for late perturbed trials

Positive perturbations

Negative perturbations

0 100 200 300 400 500 600 700 800 900 100020

21

22

23

24

25

26

27

28

29

30

Time (ms)

X position (cm)

Subject 1: Trajectories for late perturbed trials

Positive perturbations

Negative perturbations

0 100 200 300 400 500 600 700 800 900 10000

5

10

15

20

25

30

Time (ms)

X position (cm)

Subject 2: Trajectories for early perturbed trials

Positive perturbations

Negative perturbations

0 100 200 300 400 500 600 700 800 900 10000

5

10

15

20

25

30

Time (ms)

X position (cm)

Subject 2: Trajectories for late perturbed trials

Positive perturbations

Negative perturbations

0 50 100 150 200 250 300 350 400 450 500-3

-2

-1

0

1

2

3x 10-3

Time (ms)

Perturbation weight

Perturbation weight function for in-line perturbations

0 50 100 150 200 250 300 350 400 450 500-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1x 10-3

Time (ms)

Perturbation weight

Perturbation weight function for perpendicular perturbations

Conclusions

• Visuomotor system uses directional error signal for feedback control?

• Position / speed error in direction of movement is not effective feedback signal?

• Why?– Position along path blurred by motion– Insensitivity to acceleration along direction of

motion

Question

• What visual information about hand does visuomotor system use– Position error?– Motion error?– Position and motion?

(c)

(b)

target

unseenhand

virtual fingertip

(a)

(c)

(b)

target

(a)

0 50 100 150 200 250 300 350 400 450-12

-10

-8

-6

-4

-2

0

2

4x 10-4

Time (msecs.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1x 10-3

Time (msecs.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5x 10-3

Time (msecs.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-2.5

-2

-1.5

-1

-0.5

0

0.5x 10-3

Time (msecs.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-4

-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1x 10-3

Time (msecs.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-3.5

-3

-2.5

-2

-1.5

-1

-0.5

0

0.5

1x 10-3

Time (msecs.)

Perturbation weight

Conclusions

• Visuomotor system uses continuous visual feedback to control reaching movements.

• Feedback signals include positional error.

• Feedback signals include motion error.

• System is approximately linear.

0 50 100 150 200 250 300 350 400 450-6

-4

-2

0

2

4

6

8x 10-3

Time (msec.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-0.06

-0.05

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

Time (msec.)

Cumulative perturbation weight

0 50 100 150 200 250 300 350 400 450-16

-14

-12

-10

-8

-6

-4

-2

0

2

4x 10-3

Time (msec.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-0.35

-0.3

-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

Time (msec.)

Cumulative perturbation weight

0 50 100 150 200 250 300 350 400 450-12

-10

-8

-6

-4

-2

0

2

4

6x 10-3

Time (msec.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-0.2

-0.15

-0.1

-0.05

0

0.05

Time (msec.)

Cumulative perturbation weight

0 50 100 150 200 250 300 350 400 450-12

-10

-8

-6

-4

-2

0

2

4

6x 10-3

Time (msec.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-0.25

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

Time (msec.)

Cumulative perturbation weight

0 50 100 150 200 250 300 350 400 450-10

-8

-6

-4

-2

0

2

4

6

8x 10-3

Time (msec.)

Perturbation weight

0 50 100 150 200 250 300 350 400 450-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

Time (msec.)

Perturbation weight