Eye Movements andWorking Memory

Marc Pomplun

Department of Computer ScienceUniversity of Massachusetts at Boston

E-mail: marc@cs.umb.eduHomepage:

http://www.cs.umb.edu/~marc/

Eye Movements andWorking Memory

Overview:

• Image Processing: Convolution Filters• Iconic Memory Representations for

Visual Search• Working Memory Use in a Natural Task• The Working Memory - Eye Movement

Tradeoff

Convolution Filters

Grayscale Image:

109244

82013

796105

0103112

92361

1/91/91/9

1/91/91/9

1/91/91/9

Averaging Filter:

Image Processing

Original Image:

109244

82013

796105

0103112

92361

Filtered Image:

00000

00

00

00

00000

1/91/91/9

1/91/91/9

1/91/91/9

value = 11/9 + 61/9 + 31/9 + 21/9 + 111/9 + 31/9 + 51/9 + 101/9 + 61/9 = 47/9 = 5.222

55

Image Processing

Original Image:

109244

82013

796105

0103112

92361

Filtered Image:

00000

00

00

00

00000

1/91/91/9

1/91/91/9

1/91/91/9

value = 61/9 + 31/9 + 21/9 + 111/9 + 31/9 + 101/9 + 101/9 + 61/9 + 91/9 = 60/9 = 6.667

55 77

Image Processing

Original Image:

109244

82013

796105

0103112

92361

Filtered Image:

00000

0 0

0 0

0 0

00000

55 77 55

55

55

55 66

6644

Now you can see the averaging (smoothing) effect of the 33 filter that we applied.

Gaussian Filters

2

22

222

1),(),(

yx

eyxGyxW

•1•4•7•4•1

•4•16•26•16•4

•7•26•41•26•7

•4•16•26•16•4

•1•4•7•4•1

• Discrete version: 1/273

• implement decreasing influence by more distant pixels

Gaussian Filters

original 33 99 1515

Effect of Gaussian smoothing:

Sobel Filters

• Sobel filters are an example for edge detection filters.

• Two small convolution filters are used successively:

Sx

10-1

20-2

10-1

-1-2-1

000

121

Sy

Sobel FiltersSobel filters yield two interesting pieces of information:

• The magnitude of the gradient (local change in brightness):

22|| yx ssG

• The angle of the gradient (tells us about the orientation of an edge):

x

y

s

sarctan

Sobel Filters

Original image (left) and result of calculating the magnitude of the brightness gradient with a Sobel

filter (right)

Rao, Zelinsky, Hayhoe & Ballard (2002):

Eye Movements in Iconic Visual Search

Question: How do people represent items in their memory for efficient visual search?Idea: Iconic (appearance-based) multiscale representationSuch representations were modeled using spatiochromatic convolution filters of different scales and orientations.

Rao, Zelinsky, Hayhoe & Ballard

Convolution filters used for the model.

Rao, Zelinsky, Hayhoe & Ballard

According to the model, iconic visual search proceeds as follows:• The first saccade is aimed at the point in the visual scene whose low-frequency features have the best match with the low-frequency features of the memorized object.• For the programming of the following saccades, higher and higher frequencies are included, until the target is found.

Rao, Zelinsky, Hayhoe & Ballard

Coarse-to-fine scanning mechanism

Rao, Zelinsky, Hayhoe & Ballard

Conclusion: Good correspondence between modeled and empirical scanpaths

Ballard, Hayhoe & Pelz (1995):

Memory Representations in Natural Tasks

Task: Copy a pattern of colored blocks

Ballard, Hayhoe & Pelz

Possible strategies for completing the block copying task. Participants performed the following operations:(M)odel inspection, (P)ickup, and (D)ropoff.

Ballard, Hayhoe & Pelz

Typical hand and gaze trajectories for a single copying step

Ballard, Hayhoe & Pelz

Empirical frequency of individual strategies in the block copying task

Ballard, Hayhoe & Pelz

Conclusions:

• In the block copying task, working memory is only sparsely used.

• Instead, subjects prefer to make additional eye movements.

• Because eye movements are “inexpensive”, subjects use the visual scene as an “external memory” rather than building an internal representation of it.

Eye Movement - Working Memory

Tradeoff (Inamdar & Pomplun, 2003)Based on the previous study by Ballard et al, it

seems that using working memory is clearly more “expensive” than performing eye movements.

So maybe a “cost model” is an adequate way of describing and predicting behavior in visual tasks.

Inamdar & Pomplun

The basic idea is that the visual system (including the cognitive mechanisms that are required for performing the task) optimizes visual behavior, i.e. minimizes its effort (cost).

Is there such a tradeoff between the use of working memory and eye movements?

If so, what exactly is minimized? Can this be quantified?

Inamdar & Pomplun

• Let subjects perform a visual task that requires eye movements and extensive use of visual working memory.

• Vary the “cost” of eye movements.

• Hypothesis: If the assumed tradeoff between eye movements and working memory exists, costlier eye movements should lead to increased use of working memory.

Stimuli in Experiment 1

• Subjects were presented with two columns of simple geometrical objects in three different colors and three different shapes.

• The columns were identical except for one object that differed in either its color or its shape (in target-present trials).

• Subjects had to indicate whether such a target was present or not.

• The objects in the non-attended hemifield were always masked.

• The cost of eye movements was varied by changing the distance between the two columns.

Stimuli in Experiment 1

Stimuli in Experiment 1

Stimuli in Experiment 1

Stimuli in Experiment 1

Eye Movements in Experiment 1

Eye Movements in Experiment 1

Results of Experiment 1

Results of Experiment 1

Results of Experiment 1

Experiment 2

• What happens if the capacity limit of visual working memory is reached?

• By just varying the distance between columns, the cost of eye movements cannot be dramatically increased.

• Idea: “Artificially” increase the cost of eye movements in the present paradigm by delaying the unmasking of objects after any gaze switch between hemifields.

Stimuli in Experiment 2

• We used the same stimuli as in Experiment 1, but only those for the “medium-distance” condition.

• Three visibility delays were used: 0ms, 500ms, and 1000ms.

• During the delays, objects in both hemifields were masked.

Results of Experiment 2

Results of Experiment 2

Results of Experiment 2

Conclusions

• There clearly is a cost-minimizing behavior with regard to eye movements and working memory.

• However, the current data does not allow to build a quantitative model of this phenomenon.

• It seems that people slightly overestimate their working memory capacity when they are forced to heavily increase their working memory load.