Slide 1

How Gaze Patterns are LearnedNeuroeconomicsFixation on Collider

2Learning to Adjust GazeChanges in fixation behavior fairly fast, happen over 4-5 encounters (Fixations on Rogue get longer, on Safe shorter)

3

Shorter Latencies for Rogue FixationsRogues are fixated earlier after they appear in the field of view. This change is also rapid.

4

target selectionsignals to musclesinhibits SC saccade decisionsaccade commandplanning movementsNeural Circuitry for SaccadesSubstantia nigra pc(Dopamine)5Where do targets come from?Neural Substrate for LearningNeurons in substantia nigra pc in basal ganglia release dopamine.These neurons signal expected reward. Neurons at all levels of saccadic eye movement circuitry are sensitive to reward.This provides the neural substrate for learning gaze patterns in natural behavior, and for modeling these processes using Reinforcement Learning. 6These findings are important because if we want to understand how fixation patterns and timing are so tightly linked to task, need reward mechanisms like this to control learning

Dopaminergic neurons in basal ganglia signal expected reward. (Schultz, 2000)Response to unexpected rewardIncreased firing for earlier or later rewardExpected reward is absent.SNpc7

8

target selectionsignals to musclesinhibits SC saccade decisionsaccade commandplanning movementsNeural Circuitry for SaccadesSubstantia nigra pcSubstantia nigra pc modulates caudate9Where do targets come from?Neurons at all levels of saccadic eye movement circuitry are sensitive to reward.

LIP: lateral intra-parietal cortex. Neurons involved in initiating a saccade to a particular location have a bigger response if reward is bigger or more likely

SEF: supplementary eye fieldsFEF: frontal eye fieldsCaudate nucleus in basal ganglia10

Monkey makes a saccade to a stimulus - some directions are rewarded.

Cells in caudate signal both saccade direction and expected reward.Hikosaka et al, 200011 This provides the neural substrate for learning gaze patterns in natural behavior, and for modeling these processes using Reinforcement Learning. (eg Sprague, Ballard, Robinson, 2007) 12Virtual environments allow direct comparison of human behavior and model predictions in the same natural context.

Use Reinforcement Learning models with an embodied agentacting in the virtual environment.

Modelling Natural Behavior in Virtual Environments. 13

Assume behavior composed of a set of sub-tasksSprague, Ballard, Robinson, 2007; Rothkopf ,2008 Modelling behaviors using virtual agents14Model agent after learning

Pickup litterFollow walkwayAvoid obstacles15Choose the task that reduces uncertainty of reward the mostControlling the Sequence of fixationsobscanside

16

17

Humans walk in the avatars environment18

Reward weights estimated from human behavior using InverseReinforcement Learning - Rothkopf 2008.Human path Avatar path 19

Time fixatingIntersection.

Follow the car.orFollow the car and obeytraffic rules.CarRoadsideRoadIntersectionShinoda et al. (2001)Detection of signs at intersection results from frequent looks.2021What do we know? Previous work on dsn of attn in natural environments:IntersectionP = 1.0Mid-blockP = 0.3Greater probability of detection in probable locationsSuggests Ss learn where to attend/look.

How well do human subjects detect unexpected events? Shinoda et al. (2001)Detection of briefly presented Stop signs. 21Brain-Computer Interface for Control of a Prosthetic Arm

NeuraldecoderNeural signalsSpeech synthesizerSpeech soundsWernickes areaBrocas area

Different regions of motor cortex control different parts of the body

25

Record from cells in motor cortex while monkey controls robot arm with a joystickFind the preferred direction for each cell in the regions thats active.

Find the population vector = vector sum of preferred directions.

Send this signal to the arm.

Add in a correction while the monkey is learning.

Gradually reduce the correction.