7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 1 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Linking vision and language:From infant cognition to eye-tracking in the visual worldFranklin Chang and Andrew Jessop —- University of Liverpool
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 2 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Vision and Language
How are vision and language linked?·
Meaning mediated account-
2/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 3 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Evidence for a link between vision and language
Caused motion (transitive, 他動詞) -> Spontaneous motion (intransitives, 自動詞)
Children produce FROM passives: I was caught from you before (Clark & Carpenter, 1989)
·
CM I put the doll here -> SP the doll goes here (Bloom, 1993)
Two perspectives on the same visual scene: Child or Doll
-
-
·
Normal BY passives 受け身: I was caught by you
Children never hear caught from in input
Spatial similarity between Agents and Sources
-
-
-
The girl is running from the man (man = SOURCE 起点)
the girl is chased by the man (man = AGENT 動作主)
-
-
3/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 5 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Linking Visual-spatial cognition and Language
Standard Approach: Vision and language are separate modules/systems
Alternative Approach: Language is built on pre-existing spatial mechanisms
·
Concepts mediate between vision and language-
·
Use infant/adult visual cognition to motivate a rich set of spatial mechanisms andrepresentations
Show how these can be linked to thematic roles in language
Present a connectionist model that uses these representations to explain eye-tracking inthe visual world
-
-
-
5/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 6 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Thematic roles (主題役割)
Agent 動作主 = doer of action (child in I put the doll here)
Patient 被動者 = affected by action (doll)
Theme 主題 = moved by action (doll in the doll goes here)
How do children figure out if the doll is patient or theme?
Existing theories use abstract features (e.g., volition 意志, sentience 感覚性; Dowty, 1991)
·
·
·
·
Reality: Child moves the doll (child is agent)
Imagination: Doll is moving by itself (doll is theme)
-
-
·
These features are not defined clearly
Vision-based relational features are needed
-
-
6/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 7 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
A Vision-based Theory of Thematic Roles
Thematic roles in language make use of mechanisms that developed for object tracking
Vision has evolved powerful mechanisms for tracking objects
·
These two views are consistent with two tigers
Object tracking might show that there is only one tiger
-
-
·
Spatial pointers support object tracking (Pylyshyn & Storm, 1988)- 7/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 8 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Spatial Indexes/Object files (Pylyshyn & Storm, 1988)
Participants see initial scene with identical crosses
Some crosses are highlighted as targets
Crosses move which participants keeps attention in center of screen
Cross is highlighted at test and participant responds whether it is one of the targets
·
·
·
·
8/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 9 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Spatial Pointers and Visual heuristics
Wolf is the agent of chasing (sheep are patients)
Approach visual heuristic
Visual heuristics can be used to identify thematic roles
·
Wolf moves towards sheep (approach)
Wolf and sheep are identical circles (object tracking)
-
-
·
Angle of motion (Gao et al., 2009)
Goal-object motion (Woodward, 1998)
-
-
·
Innate (infants for approach, Luo, 2011)-
9/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 10 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Separation, Distance, and Primacy Heuristic
Separation heuristic
Distance heuristic
Primacy heuristic
·
Reverse of Approach Heuristic-
·
Represents the distance between pairs of objects in the scene
Lew, Bremner, and Lefkovitch (2000) found that 8.5 month old infants encode a hiddentoy’s position between two landmarks after displacement to a new location
-
-
·
Represents the order of motion of objects in scenes
Bullock & Gelman (1979) showed two events where balls went down a ramp, but oneball went down before a jack-in-the-box appeared and another ball went downafterwards. Children labeled the first ball as the cause.
-
-
10/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 11 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
A Connectionist Model of Language Acquisition
Simple recurrent network (Elman 1990) ·
11/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 12 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Dual-path model (Chang 2002)
Simple recurrent network learns sequencing constraints
Message: Fast changing weights between roles and concepts
·
·
12/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 13 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Spatial Dual-path model
Roles are replaced with spatial pointers/indexes (e.g., P1)
Visual heuristics are input to SRN and help in identifying thematic roles
·
·
13/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 14 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Explaining Eye-tracking in Production
Griffin & Bock (2000) eye-tracking during sentence production·
Look at subject before they produce head noun-
14/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 15 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Eyetracking during noun production
Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"produced
·
15/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 16 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Output Message Supports Predictive Eye movements
Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"produced
·
16/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 17 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Output Message Supports Predictive Eye movements
Visual heuristics activate output pointer P2, which shifts attention to agent before "mouse"produced
·
17/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 18 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Explaining Eye-tracking in Comprehension
Altmann (2004) Blank Screen Paradigm·
The boy will eat the cake
Screen is blank when eye-tracking is performed
-
-
18/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 19 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Internal object tracking is needed to explain blank screen studies
Input and output messages are set when picture is visible ·
19/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 20 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Input Message Supports Reactive Eye movements
Attention is shifted to corresponding input pointer, even when screen is blank ·
20/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 21 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Input Message Supports Reactive Eye movements
Attention is shifted to corresponding input pointer, even when screen is blank ·
21/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 22 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Input Message Supports Reactive Eye movements
Attention is shifted to corresponding input pointer, even when screen is blank ·
22/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 23 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Errors and structural changes
Caused motion (transitive, 他動詞) -> Spontaneous motion (intransitives, 自動詞)
Children produce FROM passives: I was caught from you before (Clark & Carpenter, 1989)
·
CM I put the doll here -> SP the doll goes here (Bloom, 1993)
Changing thematic roles: AGENT=CHILD PATIENT=DOLL -> THEME=DOLL
Spatial similarity in visual heuristics creates these changes in the model
-
-
-
·
Normal BY passives: I was caught by you
Changing AGENT=CHILD -> SOURCE=CHILD
Spatial similarity between agents and sources create these errors in the model
-
-
-
23/24
7/31/15, 17:49Linking vision and language: From infant cognition to eye-tracking in the visual world
Page 24 of 24file:///Users/chang/Dropbox/mywork/RPresentation/kit/kit1web.html#1
Spatial Dual-path Model
Object-based pointers and visual heuristics support meaning
Language uses these spatial mechanisms and representations
Powerful generalization abilities have been used to argue for innate language knowledge
·
Infant physical/social cognition (e.g., old-goals have approach information)
Thematic relations (e.g., caused motion = approach + separation)
-
-
·
Eyes move when we understand language (e.g., eye tracking in the visual world)
Spatial biases and errors in development (e.g. FROM passives)
-
-
·
Critics have argued that domain-general statistical learning can explain languageabilities
Language evolved on top of preexisting visual/spatial mechanisms
-
-
24/24