A computational model of the reviewing of object-files

1

A computational model of the reviewing of object-files

Michael Liddle

Alistair Knott

Anthony Robins

2

Introduction

• Selective visual attention.

• Object-files and the object-specific advantage (OSA).

• Computational modeling of cortical vision.

• A neural network model of object-file reviewing (first of its kind?)

3

Selective visual attention

4

Managing limited resources

• Retinal image contains an enormous amount of information.

• Processing complexity subject to combinatorial explosion.

• Solution: only bother processing information about one object at a time.

5

Explaining the solution

• What actually happens in the brain when we “attend” to an object?

• Experiments indicate that attention is the means by which feature “conjunction” and “binding” occurs (Treisman & Gelade, 1980).

• What is the medium of this binding? Object-files!

6

Object-files and the object-specific advantage

7

Object-files

• Kahneman & Treisman (1984)

• Provide stable repositories for visual information about four or five objects.

• Maintain identity and continuity of objects during a perceptual episode.

• Analogy: police files for investigations.

8

Object-files

• When attending to an object for the first time: “open” an object-file.

• When reattending to an object: “review” the information in its object-file.

• Reviewing involves reconciling old information with new.

9

The object-specific advantage

• Evidence for a object-specific type of priming (Kahneman, Treisman, & Gibbs, 1992), linked to object-file reviewing.

• Facilitation for perceptually coherent objects, greater than general priming.

• Suggestion is that previous perception of an object allows stored information to speed recognition.

10

Example: Preview

V

Q

11

Example: Linking

12

Example: SO condition

V

13

Example: DO condition

Q

14

Example: NM condition

S

15

Recognition times

450

460

470

480

490

500

510

520

RT (ms)

SO

DONM

16

Computational models of cortical vision

17

Providing a foundation

• Object-files must exist at a relatively high level of visual perception.

• Important to consider both current thought about neurology of visual attention, as well as existing computational models.

18

Models of object detection and recognition

• Models of detection:– Retinotopic maps of salient regions

(saliency maps).– Guide attentional processes.

• Models of recognition:– Hierarchical structures (increasing

selectivity/receptive field size)– Output encoding of feature conjunctions.

19

A neural network model of object-file reviewing

20

Neural network modeling

• Connect collection of simple “neuron-like” components via weighted “synapse-like” components.

• Basic neuron sums its inputs, and applies an “activation-function” to determine output.

• Output is interpreted as a firing rate.

21

Modeling the object-specific advantage

• Need a recognition procedure that can be subject to facilitation (i.e. involves a time course).

• Need to store “bottom-up stimulus” information in an object-specific way.

• Need to provide “top-down expectation” based on stored information for currently attended object.

22

Modeling the OSA

• Correct expectation should lead to facilitation.

• Incorrect expectation should not destroy general priming.

23

Modeling facilitation

• Use type based classification: when type is known, recognition is complete.

• Enforce single winning type by lateral competition.

• Winner is called the “stimulus type”.

• Enhance time factor by using “cascaded activation” neurons.

24


V

Q

S

J

Hierarchical feature encoderType layer

V

25


V

Q

S

J


V

26


V

Q

S

J


V

27


V

Q

S

J


V

Recognised

28

Storing stimulus: object specificity

• FINSTs (Fingers of INSTantiation) identify “proto-objects” in the scene (Pylyshyn, 1989)

• Track their proto-objects as they move and change size/shape.

• Set of four or five FINSTs constantly assigned/reassigned from saliency map

• Provide candidates for attention.

29


• Associate a neuron with each FINST.• Selecting a FINST for attention

activates its neuron.• Associate stimulus type with current

FINST.• Thus a level of indirection is introduced

between retinal location and mental representation.

30


V

Q

S

J V

Q

Association“stuff”

31


V

Q

S

J V

Q

Association“stuff”

32

Association stuff?

V

Q

S

J

33

Association stuff

FINST

Feedforward

Feedback

*Excitatory connections shown only

Type

V

Q

S

J

“Object-file”

34

Storing stimulus: feedback “stuff”

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

V

Q

35

Storing stimulus: feedback “stuff”

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

V

Q

36

Providing expectations: feedforward “stuff”

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

V

Q

37

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

V

Q

Providing expectations: feedforward “stuff”

38

Storing stimulus:opening an object-file

39

Storing stimulus

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

V

Q

40

Storing stimulus

FINST

Feedforward

Feedback


“Object-file”

V

Q

Type

V

Q

S

J

41

Storing stimulus

FINST

Feedforward

Feedback


“Object-file”

V

Q

Type

V

Q

S

J

42

Storing stimulus

FINST

Feedforward

Feedback


“Object-file”

V

Q

Type

V

Q

S

J

43

Storing stimulus

FINST

Feedforward

Feedback


“Object-file”

V

Q

Type

V

Q

S

JRecognised

44

Storing stimulus

FINST

Feedforward

Feedback


“Object-file”

V

Q

Type

V

Q

S

JRecognised

45

Storing stimulus

FINST

Feedforward

Feedback


“Object-file”

V

Q

Type

V

Q

S

JRecognised

Stored

46

Providing correct expectation:

the SO condition

47

Providing correct expectation (SO)

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

V

48


FINST

Feedforward

Feedback


“Object-file”

V

Type

V

Q

S

J

49


FINST

Feedforward

Feedback


“Object-file”

V

Type

V

Q

S

J

50


FINST

Feedforward

Feedback


“Object-file”

V

Type

V

Q

S

JRecognised

51

Providing incorrect expectation:

the DO and NM conditions

52

Providing incorrect expectation (DO/NM)

FINST

Feedforward

Feedback


Type

V

Q

S

J

“Object-file”

Q

53


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

J

54


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

J

55


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

J

56

Correcting incorrect expectation (DO/NM)

FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

J

57


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

J

58


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

J

59


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

JRecognised

60


FINST

Feedforward

Feedback


“Object-file”

Q

Type

V

Q

S

JRecognised

Corrected

61

Preserving general priming:the DO condition

62

Preserving general priming (DO)

V

Q

S

J

63

Recognition times: model

RT

SO

DO

NM

64

Recognition times: empirical

RT

SO

DONM

65

Results

• Got the right order of time courses:– SO < DO < NM

• Got close to the right proportions for time courses: – SO << DO < NM

• Thus:– OSA implemented.– Standard priming retained.

66

Side effects (predictions?)

• Recognition is slowed in DO and NM conditions (interference).

• Corrections are persistent (persistence).• Corrections lost if not given enough time

(masking).• Unassigned FINSTs take on attended

type when attention to one object is prolonged.

67

More to be done…

• Move away from types towards feature-encoding (requiring recognition to be done at a higher level).

• Implement a biologically plausible FINST module on top of a saliency map.

• Combine with hierarchical feature encoding models.

68

The end

Got any questions?

69

References

• Itti, L., & Koch, C. (2001). Computational modeling of visual attention. Nature Reviews Neuroscience, 2(3), 94-203.

• Pylyshyn, Z. (1989). The role of location indexes in spatial attention: a sketch of the FINST spatial-index model. Cognition, 32, 65-97.

70

References

• Kahneman, D., & Treisman, A. (1984). Changing view of attention and automaticity. In R. Parasuraman, & D. Davies (Eds.), Varieties of attention (pp. 29-61). New York: Academic Press.

• Kahneman, D., Treisman, A., & Gibbs. B. (1992). The reviewing of object files: object-specific integration of information. Cognitive Psychology, 24, 174-219.

71

References

• Riesenhuber, M., & Poggio, T. (1999). Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11), 1019-1025.

• Treisman, A., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12, 97-136.

72

The REAL end!

73

This slide intentionally left blank

74

Variations

• Experiments involving more preview objects:– Object-specific advantage decreased for more than four

objects.– Corroborates limited number claim.

• Experiments involving moving frames in linking display:– Results consistent with “static” conditions.– Suggests Object-files associated with objects not locations.

75

Storing stimulus: association “stuff”

• For each FINST neuron:– Feedback layer: raised threshold neurons forcefully transmit

stimulus type to object-file layer. – Object-file layer: self-stabilising (competitive) neurons

“remember” stimulus type for the currently attended FINST.

• Winner in object-file layer called “stored type”.• Type only stored in attended FINST’s object-file.

76

Providing expectations: more association “stuff”

• For each FINST neuron:– Feedforward layer: raised threshold

neurons transmit stored type back to type layer.

• Expectation only provided from attended FINST’s object-file.

77


• What: recognition is slowed in DO and NM conditions.

• Why: interference in the type layer due to incorrect expectation.

• Test: compare recognition times for non-previewed objects.

• Conclusion: seems a plausible “surprise” effect.

78


• What: corrections are persistent.• Why: the expectation provided for an

object is the currently stored type.• Test: an OSA should exist for the most

recently observed type only.• Conclusion: seems a plausible

prediction if object-file theory is accurate.

79


• What: correction is not made if attention to a changed object is too short.

• Why: correction is not instantaneous, corrected feedback is required.

• Test: OSA should only be “updated” is change was seen for long enough.

• Conclusion: seems a plausible object level analogue to masking effects in iconic memory.

80


• What: prolonged attention to an object causes unassigned FINSTs to store current type.

• Why: unassigned FINSTs are very plastic because they have no stored type.

• Test: not really testable.• Conclusion: probably a problem with the

current implementation.

Documents

A computational model of the reviewing of object-files