LA

NG

UA

GE

AN

D C

OG

NIT

IVE

PR

OC

ESS

ES,

198

9, 4

, (3/

4) S

I 287

- 335

Sent

ence

Com

preh

ensi

on: A

Par

alle

l Dis

trib

uted

Proc

essi

ng A

ppro

ach

J. L

. McC

lella

nd, M

ark

St. J

ohn

, and

Rom

an T

arab

anD

epar

tmen

t of

Psyc

holo

gy, C

arne

gie

Mel

lon

Uni

vers

ity, P

ittsb

urgh

,

In th

is p

aper

, we

revi

ew b

asic

asp

ects

of

conv

entio

nal a

ppro

ache

s to

sen

tenc

eco

mpr

ehen

sion

and

poi

nt o

ut s

ome

of th

e di

fficu

lties

face

d by

mod

els

that

take

thes

e ap

proa

ches

, We

then

des

crib

e an

ahe

rnat

ive

appr

oach

, bas

ed o

nth

e pr

inci

ples

of

para

llel d

istr

ibut

ed p

roce

ssin

g, a

nd s

how

how

it o

ffer

sdifferent answers to basic questions about the nature of the language proces-

sing

mec

hani

sm, W

e de

scri

be a

n ill

ustr

ativ

e si

mul

atio

n m

odel

that

cap

ture

sth

e ke

y ch

arac

teri

stic

s of

the

appr

oach

, and

illu

stra

te h

ow it

can

cop

e w

ithth

e di

ffic

ultie

s fa

ced

by c

onve

ntio

nal m

odel

s. W

e de

scri

be a

ltern

ativ

e w

ays

ofco

ncep

tual

isin

g ba

sic

aspe

cts

of la

ngua

ge p

roce

ssin

g w

ithin

the

fram

ewor

k of

this

app

roac

h, c

onsi

der

how

it c

an a

ddre

ss s

ever

al a

rgum

ents

that

mig

ht b

ebr

ough

t to

bear

aga

inst

it, a

nd s

ugge

st a

venu

es fo

r fu

ture

dev

elop

men

t.

INT

RO

DU

CT

ION

Wha

t is

cons

truc

ted

men

tally

whe

n w

e co

mpr

ehen

d a

sent

ence

? H

ow d

oes

this

con

stru

ctiv

e pr

oces

s oc

cur?

Wha

t rol

e do

wor

ds p

lay

in th

e co

nstr

uc-

tion

proc

ess?

How

is th

e ab

ility

to construct such a representation ac-

quire

d? T

hese

are

som

e of

the

cent

ral q

uest

ions

that

face

any

atte

mpt

tobuild a model of language processing.

In th

is p

aper

, we

pres

ent a

vie

w th

at d

iffer

s fr

om m

ost e

xist

ing

notio

nsabout the general form of the answers to th

ese

ques

tions

, We

brie

flyou

tline

wha

t we

take

to b

e a

gene

ric

vers

ion

of e

xist

ing

notio

ns. T

hen,

we

poin

t out

som

e di

fficu

lties

with

thes

e no

tions

. Afte

r th

is, w

e present a

sket

ch o

f an

alte

rnat

ive

that

see

ms

to u

s to

hav

e pr

omis

e to

add

ress

thes

epr

oble

ms.

We

illus

trat

e th

e al

tern

ativ

e by

des

crib

ing

a pr

elim

inar

y m

odel

,

Req

uest

s fo

r re

prin

ts s

houl

d be

add

ress

ed to

J. L

. McC

lella

nd, D

epar

tmen

t of Psychol-

ogy,

Car

negi

e M

ello

n U

nive

rsity

, Pitt

sbur

gh. P

A t5

213,

U.

The

wor

k re

port

ed h

ere

was

sup

port

ed b

y N

SF G

rant

s B

NS

86-(

)972

9 an

d B

NS

111

1-

1204

8. O

NR

Con

trac

ts N

OO

OI4

-86-

OI4

6 an

d N

OO

OI4

- 86-

0349

, and

an

NIM

H C

aree

rDevelopment Award MHOO385 to J.

L. M

cCle

lland

.

1990

Law

renc

e E

rlbau

m A

ssoc

iate

s Li

mite

d an

d V

SP

Pub

licat

ions

5128

8M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PO

P51

289

and

cons

ider

how

it g

ives

dif

fere

nt a

nsw

ers

to s

ome

of th

e qu

estio

ns r

aise

dab

ove,

and

not

e ho

w it

add

ress

es th

e pr

oble

ms

face

d by

mod

els

that

giv

eco

nven

tiona

l ans

wer

s to

thes

e qu

estio

ns, W

e ex

amin

e so

me

of th

e ar

gu-

men

ts, b

oth

theo

retic

al a

nd e

mpi

rica

l, th

at h

ave

been

take

n as

cou

ntin

gag

ains

t thi

s so

rt o

f al

tern

ativ

e, a

nd s

how

that

they

can

in f

act b

e co

unte

red,

Fina

lly, w

e de

scri

be f

utur

e di

rect

ions

for

the

furt

her

deve

lopm

ent o

f th

isap

proa

ch,

wor

ds, T

his

view

app

ears

to u

nder

lie F

odor

and

Pyl

yshy

ns

(198

8) p

rinc

iple

of

com

posi

tiona

lity:

A

ccor

ding

to th

is p

rinci

ple

, "

a w

ord

mak

es a

ppro

xi-

mat

ely

the

sam

e se

man

tic c

ontr

ibut

ion

to th

e m

eani

ng o

f ev

ery

sent

ence

inw

hich

it o

ccur

s,"

Let

us

use

thei

r ex

ampl

e:

1. J

ohn

love

s th

e gi

rl.2,

The

gir

l lov

es J

ohn,

CO

NV

EN

TIO

NA

L A

PPR

OA

CH

ES

TO

SE

NT

EN

CE

CO

MPR

EH

EN

SIO

N

Fodo

r an

d Py

lysh

yn u

se th

ese

sent

ence

s to

illu

stra

te w

hat t

hey

mea

n by

com

posi

tiona

lity,

The

y as

k us

to c

onsi

der

the

mea

ning

of

the

wor

d " l

oves

that appears in both of these sentences, T

hey

stat

e th

at th

e re

latio

nshi

pth

at J

ohn

is s

aid

to b

ear

to th

e gi

rl in

the

firs

t sen

tenc

e is

the

sam

ere

latio

nshi

p th

at th

e gi

rl is

sai

d to

bea

r to

Joh

n in

the

seco

nd s

ente

nce,

Thi

sco

mm

on r

elat

ions

hip

can

be ta

ken

to b

e th

e m

eani

ng o

f th

e w

ord

" lov

esan

d it

occurs in the representation of the meaning of both of th

ese

sent

ence

s.

The

com

preh

ensi

on o

f se

nten

ces

has

been

stu

died

ext

ensi

vely

, and

ther

ear

e m

any

disp

arat

e vi

ews

abou

t the

nat

ure

of th

is p

roce

ss. W

e do

not

mea

nto assert that all previous researchers have adhered to the views we

desc

ribe

in th

is s

ectio

n, H

owev

er, q

uite

a b

it of

wor

k ha

s be

en d

one

whi

chw

e be

lieve

eith

er ta

citly

or

expl

icitl

y ad

opts

the

view

s w

e de

scri

be h

ere,

We

tend

to c

ite th

e pa

per

by F

odor

and

Pyl

yshy

n (1

988)

, bec

ause

it a

rtic

ulat

esth

ese

view

s cl

earl

y, W

here

rel

evan

t, w

e w

ill s

ite w

orks

that

app

ly th

ese

idea

s an

d ge

nera

l tex

ts w

here

they

are

use

d or

ass

umed

,H

ow d

oes

the

Pro

cess

of C

onst

ruct

ing

aR

epre

sent

atio

n of

the

Pro

posi

tions

Und

erly

ing

aSentence Occur?

Wha

t is

Con

stru

cted

whe

n W

e C

ompr

ehen

d a

Sent

ence

?O

ften

, thi

s pr

oces

s is

take

n to

be

one

of b

uild

ing

a st

ruct

ural

des

crip

tion

using a system of

stru

ctur

e-se

nsiti

ve

rule

s. F

ollo

win

g F

odor

and

Pyl

yshy

nwe take

stru

ctur

e-se

nsiti

ve

to m

ean

that

the operations that apply to

repr

esen

tatio

ns a

re s

ensi

tive

to th

eir

form

and

not

thei

r co

nten

t (Fo

dor

&Pylyshyn, 1988), This means, for example, that they care only if

som

eco

nstit

uent

is a

n ite

m o

f th

e ri

ght v

ery

gene

ral t

ype

(N, Y

, NP

, YP

, etc

,

It is

typi

cal t

o as

sum

e th

at w

hat i

s co

nstr

ucte

d is

an

inte

rcon

nect

ed s

et o

fpr

opos

ition

s (e

, g, Clark & Clark, 1977),

or

prop

ositi

onal

rep

rese

ntat

ion,

The

exa

ct n

atur

e of

thes

e pr

opos

ition

s va

ries

fro

m im

plem

enta

tion

toim

plem

enta

tion

, but

in g

ener

al th

ey a

re ta

ken

to b

e sy

mbo

lic e

xpre

ssio

nsw

hich

hav

e a

com

bina

toria

l syntax and semantics (Fodor & Pylyshyn

1988

). A

ccor

ding

to F

odor

and

Pyl

yshy

n, c

ombi

nato

rial

rep

rese

ntat

ions

are

thos

e w

hich

exh

ibit

the

follo

win

g pr

oper

ties:

Impl

icit

in m

any

theo

ries

of

com

preh

ensi

on is

the

notio

n th

at w

ords

hav

em

eani

ngs,

and

that

thes

e m

eani

ngs

are

the

cons

titue

nts

of th

e m

eani

ngs

ofth

e pr

opos

ition

s th

at a

re c

onst

ruct

ed f

rom

sen

tenc

es th

at c

onta

in th

ese

How

is th

e A

bilit

y to

Con

stru

ct a

Rep

rese

ntat

ion

Acq

uire

d?

To

the

exte

nt th

at w

e as

sum

e th

at th

e pr

oces

s of

constructing representa-

tions of sentences proceeds by the use of structure-sensitive rules to

stru

ctur

e th

e co

nstit

uent

exp

ress

ions

cor

resp

ondi

ng to

wor

ds, i

t see

ms

natu

ral t

o as

sum

e th

at a

cqui

sitio

n am

ount

s to

a p

roce

ss o

f det

erm

inin

gw

hat t

he r

ules

are

and

wha

t the

con

stitu

ent e

xpre

ssio

ns a

re th

at w

ords

are

used

to d

esig

nate

, Res

earc

hers

inte

rest

ed in

acq

uisi

tion

of c

ompr

ehen

sion

skill

do

not o

f cou

rse

assu

me

that

the

rule

s th

at a

re a

ctua

lly u

sed

inco

mpr

ehen

sion

are

the

sam

e ru

les

that

cha

ract

eris

e th

e ab

stra

ct li

ngui

stic

com

pete

nce

of th

e sp

eake

r-he

arer

, but

they

are

rul

es n

one

the

less

,

. They may be atomic or molecular expressions"

. If th

ey a

re m

olec

ular

, the

y ha

ve c

onst

ituen

ts w

hich

may

be

eith

er a

tom

icor molecular.

. The semantic content of a molecular expression is

a fu

nctio

n of

the

semantic content of each of the parts of the expression and of the

orga

nisa

tion

of th

e co

nstit

uent

s,

Wha

t Rol

e do

Wor

ds P

lay

in th

e C

ompr

ehen

sion

Proc

ess?

5129

0M

cCLE

LLA

ND

. ST

. JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

291

PR

OB

LEM

S F

OR

TH

E C

LAS

SIC

AL

VIE

W O

FS

EN

TE

NC

E C

OM

PR

EH

EN

SIO

N

It is

wid

ely

accepted that the

ultim

ate

interpretation that a sentence

rece

ives

is a

ffec

ted

by c

onte

nt. M

any

rese

arch

ers

acce

pt th

is, b

ut r

esis

t the

idea

that

the

initi

al p

roce

ssin

g of

atta

chm

ent a

mbi

guiti

es is

innu

ence

d by

non-

synt

actic

con

tent

. Thu

s, f

or e

xam

ple,

Fra

zier

(19

86)

has

prop

osed

that

initi

al p

arsi

ng d

ecis

ions

are

bas

ed o

n a

pure

ly s

ynta

ctic

mec

hani

sm th

atpr

opos

es it

s pr

efer

red

alte

rnat

ive

for

cons

ider

atio

n by

sem

antic

pro

cess

es,

Lat

er in

the

pape

r , w

e re

view

em

piric

al e

vide

nce

rele

vant

to th

is c

laim

,F

or th

e m

omen

t , w

e po

int o

ut a

mor

e co

ncep

tual

pro

blem

with

it. T

hedi

ffic

ulty

is th

at th

e de

cisi

on a

s to

whi

ch in

terp

reta

tion

of a

n am

bigu

ous

sent

ence

will

win

out

in th

e en

d do

es n

ot s

eem

in g

ener

al to

be

base

d on

asi

mpl

e ye

s-no

dec

isio

n ab

out t

he a

ccep

tabi

lity

of th

e su

ppos

edly

syn

tact

i-ca

lly p

refe

rred

inte

rpre

tatio

n (C

rain

& S

teed

man

. 198

5), T

hus

in (

5), i

t is

not r

eally

pla

usib

le to

arg

ue th

at th

e in

terp

reta

tion

in w

hich

the

bird

isus

ing

the

bino

cula

rs a

s in

stru

men

t is

stri

ctly

blo

cked

, For

exa

mpl

e, w

eha

ve le

ss d

iffi

culty

acc

eptin

g su

ch a

n in

terp

reta

tion

in "

The

bir

d sa

w it

sprey with binoculars

, eve

n if

we

find

it s

omew

hat o

dd f

or a

bir

d to

be

usin

g an

inst

rum

ent.

Rat

her,

it a

ppea

rs th

at th

e al

tern

ativ

e in

terp

reta

tion

is s

impl

y m

ore

plau

sibl

e in

the

case

of

(5),

It t

hus

appe

ars

that

mor

e th

anon

e al

tern

ativ

e in

terp

reta

tion

mus

t be

eval

uate

d fo

r pl

ausi

bilit

y. th

ereb

yro

bbin

g th

e pa

rser

of a

ny s

peci

al r

ole

in p

rovi

ding

a s

ingl

e al

tern

ativ

e fo

rco

nsid

erat

ion,

. It i

s al

so im

port

ant t

o no

te th

at it

is n

ot s

impl

y th

e ca

se th

at d

ecis

ions

can

eith

er b

e m

ade

by s

ynta

ctic

rul

e or

nee

d to

be

left

for

sem

antic

det

erm

ina-

tion,

As

Mar

cus

(198

0) p

oint

s ou

t, la

ngua

ge c

ompr

ehen

ders

hav

e pr

efer

-en

ces

for

synt

actic

inte

rpre

tatio

n w

hich

mus

t be

seen

as

mat

ters

of

degr

ee,

and

ther

efor

e th

ey s

omet

imes

win

and

som

etim

es lo

ose

whe

n pl

aced

inco

nflic

t with

oth

er c

onsi

dera

tions

, Ver

y cl

ear

exam

ples

of

this

ari

se in

sent

ence

s lik

e (6

) an

d (7

):

6, W

e at

e so

me

food

with

som

e fr

iend

s th

at w

e lik

e,7,

We

foun

d a

pain

ting

in th

e at

tic th

at w

as c

over

ed w

ith c

obw

ebs,

A s

truc

ture

-sen

sitiv

e ru

le w

ould

allo

w u

s to

par

se (

fi)

corr

ectly

. bas

ed o

nth

e id

ea th

at r

elat

ive

clau

ses

shou

ld b

e ta

ken

to a

ttach

to th

e im

med

iate

lypr

eced

ing

noun

phra

se r

athe

r th

an a

n ea

rlie

r on

e, e

spec

ially

whe

n. a

s in

this

cas

e, a

ttach

men

t to

the

earli

er n

oun

phra

se w

ould

vio

late

the

so-c

alle

dno

-cro

ssov

er"

cons

trai

nt. H

owev

er, i

t is

exac

tly th

is c

onst

rain

t tha

t is

viol

ated

in (

7), w

here

it is

the

pain

ting,

rat

her

than

the

attic

, whi

ch n

ativ

esp

eake

rs ta

ke to

hav

e be

en c

over

ed w

ith c

obw

ebs,

Vio

latin

g th

is c

onst

rain

tm

ay m

ake

the

sent

ence

see

m a

bit

awkw

ard

, but

it d

oes

not p

reve

nt th

eco

bweb

s fr

om a

ttach

ing

to th

e pa

intin

g,

Sum

mar

yIn

bri

ef. t

he c

ompr

ehen

sion

of

sent

ence

s is

gen

eral

ly ta

ken

to b

e th

epr

oces

s w

here

by a

list

ener

use

s a

set o

f st

ruct

ure-

sens

itive

rul

es to

con

-st

ruct

a p

ropo

sitio

nal r

epre

sent

atio

n th

at c

onst

itute

s th

e "m

eani

ng"

of th

esc

ntcn

ce, T

he c

onst

ituen

ts o

f thi

s re

pres

enta

tion

incl

ude

the

mea

ning

s of

the

wor

ds in

thc

sent

ence

, Fol

low

ing

Fod

or and Pylyshyn

s te

rmin

olog

y.we call this view the

clas

sica

l vi

ew, T

hese

aut

hors

inte

nd it

to b

e ta

ken

asap

plyi

ng m

ore

broa

dly

than

to ju

st th

e in

terp

reta

tion

of s

ente

nces

, but

they

mak

e cl

ear

that

lang

uage

is a

"pa

radi

gm o

f sy

stem

atic

cog

nitio

n, W

e w

illno

t hav

e an

ythi

ng to

say

abo

ut it

s br

oade

r ap

plic

abili

ty; i

nste

ad, w

e w

illfo

cus

on th

e re

ason

s w

hy w

e fe

el th

at it

may

be

wor

th seeking an

alte

rnat

ive

fram

ewor

k fo

r ad

dres

sing

the

prob

lem

of

lang

uage

com

preh

en-

sum

,

Con

cept

ual G

uida

nce

and

Rul

e C

onfli

cts

A c

entr

al p

robl

em f

or th

e co

nven

tiona

l vie

w is

the

fact

that

sen

tenc

ein

terp

reta

tions

can

not i

n ge

nera

l be

reco

vere

d co

rrec

tly f

rom

str

uctu

re-

sens

itive

rul

es a

lone

, Eve

n th

ose

who

try

to g

o th

e fa

rthe

st u

sing

str

uctu

re-

sensitive rules (Frazier, 19

116;

Mar

cus,

198

0) a

re a

ccur

atel

y aw

are

of th

ispr

oble

m, T

he p

robl

em is

not

just

a c

urio

sity

; it c

omes

up

alm

ost e

very

tim

ea

prep

ositi

onal

phr

ase

is e

ncou

nter

ed, C

onsi

der,

for

exa

mpl

e:

3, The spy saw the policeman with bi

nocu

lars

,, T

he s

py s

aw th

e po

licem

an w

ith a

rev

olve

r,

In (

3). m

ost r

eade

rs in

terp

ret t

he b

inoc

ular

s as

the

inst

rum

ent u

sed

by th

esp

y in

see

ing

the

polic

eman

, In

(4),

mos

t rea

ders

inte

rpre

t the

rev

olve

r as

apo

sses

sion

of

the

polic

eman

, Thi

s si

mpl

e ex

ampl

e ill

ustr

ates

cle

arly

that

itis

nec

essa

ry a

t a m

inim

um to

con

side

r w

heth

er th

e ob

ject

of

the

prep

ositi

o-na

l phr

ase

is a

pla

usib

le c

andi

date

for

use

as

an in

stru

men

t of

the

verb

, In

gene

ral.

as th

e ne

xt e

xam

ple

mak

es c

lear

, it i

s al

so n

eces

sary

to c

onsi

der

whe

ther

in f

act t

he s

ubje

ct o

f th

e se

nten

ce m

ight

be

the

kind

of

acto

r th

atca

n us

e th

e in

stru

men

t:

5, The bird saw the birdwatcher with bi

nocu

lars

.

Inde

ed. O

den

(197

8) h

as s

how

n th

at e

very

con

stitu

ent o

f se

nten

ces

like

(3)-

(5)

can

potentially influence the interpretation of the role of th

eprepositional phrase,

5129

2M

cCLE

LLA

ND

. ST

. JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

293

Con

text

ual S

hadi

ng a

s w

ell a

s S

elec

tion

of W

ord

Mea

ning

The problem of word-meaning in

dete

rmin

acy

also

pos

es a

pro

blem

for

conv

entio

nal a

ppro

ache

s, I

t is,

of course, typical to as

sum

e th

at a

nin

divi

dual

wor

d ca

n ha

ve m

ore

than

one

mea

ning

, The

pro

blem

of

sent

-en

ce in

terp

reta

tion.

then

, is

seen

as

one

of s

elec

ting

the

righ

t mea

ning

fro

ma set of possible m

eani

ngs

that

are

sto

red

in a

"m

enta

l lex

icon

, One

prob

lem

with

this

is th

e po

tent

ial c

ombi

nato

rial

exp

losi

on th

at c

an r

esul

tas

dis

cuss

ed b

elow

, Her

e w

e fo

cus

on a

dif

fere

nt p

robl

em: T

he p

robl

em is

that

it s

eem

s ra

ther

lim

iting

to s

uppo

se th

at th

e ra

nge

of m

eani

ngs

that

aw

ord

can

have

is r

estr

icte

d in

adv

ance

to th

e se

t of

know

n us

ages

of

the

word, Let us consider some examples:

H. T

he h

oste

ss th

rew

the

ball

for

char

ity,

9. T

he s

lugg

er h

it th

e ba

ll ov

er th

e fe

nce,

10, T

he b

aby

rolle

d th

e ba

ll to

her

dad

dy.

The

dis

tinct

ions

bet

wee

n th

e m

eani

ngs

of b

all a

s it

appe

ars

in (

8) a

nd (

9)se

em w

ell e

noug

h ca

ptur

ed b

y th

e id

ea th

at th

e sp

ecifi

catio

n of

a m

eani

ngfo

r th

is w

ord

invo

lves

a s

elec

tion

of o

ne o

f tw

o al

tern

ativ

es, o

ne th

at m

eans

som

ethi

ng li

ke "

fanc

y da

nce

" an

d on

e th

at m

eans

som

ethi

ng li

ke "

sphe

ri-

cal object

. But in (1

0), i

t see

ms

that

the

spec

ific

atio

n of

the

ball

isso

mew

hat d

iffe

rent

fro

m th

e sp

ecifi

catio

n th

at w

e ge

t fro

m (

9). I

t is

poss

ible

to a

sser

t tha

t her

e ag

ain

we

are

sele

ctin

g be

twee

n tw

o al

tern

ativ

em

eani

ngs-

one.

let u

s sa

y, in

whi

ch th

e sp

heri

cal o

bjec

t is

smal

lish,

har

dan

d w

hite

, and

the

othe

r in

whi

ch it

is la

rger

. squ

ishi

er, a

nd p

roba

bly

mul

ticol

oure

d, B

ut ta

ken

to it

s ex

trem

e, th

is v

iew

see

ms

to le

ad to

a v

ast

expl

osio

n of

lexi

cal e

ntri

es, o

ne f

or e

ach

of th

e po

ssib

le b

alls

that

we

can

envi

sage

bei

ng im

plic

itly

desc

ribed

in a

sen

tenc

e. Is

ther

e to

be

a se

para

tele

xica

l ent

ry f

or e

very

sha

de o

f m

eani

ng th

at c

an b

e co

mpr

ehen

ded,

for

ever

y w

ord

in th

e la

ngua

ge?

obsc

ures

the

broa

d co

mm

onal

ity th

at d

oes

exis

t am

ong,

for

exa

mpl

e. th

eco

nstit

uent

s w

hich

we

wou

ld te

nd to

cal

l age

nts

if w

e di

d no

t loo

k to

o

clos

ely,

A Similar Problem with Roles

A s

imila

r pr

oble

m a

rise

s w

hen

we

atte

mpt

to s

peci

fy th

e se

t of t

hem

atic

role

s th

at a

re a

vaila

ble

to b

e fil

led

by w

ord

mea

ning

s in

the

stru

ctur

alde

scri

ptio

n th

at r

epre

sent

s a

sent

ence

, In

earl

y w

ork

on r

oles

(Fi

llmor

e19

68),

atte

mpt

s w

ere

mad

e to

enu

mer

ate

the

set o

f rol

es th

at c

onst

ituen

tsco

uld

fill.

How

ever

, thi

s ef

fort

qui

ckly

ran

into

the

prob

lem

that

ther

e ar

e a

larg

e nu

mbe

r of

slig

ht d

istin

ctio

ns a

mon

g ro

les,

all

of w

hich

hav

e in

terp

re-

tive

sign

ific

ance

. The

pro

blem

is s

o ba

d th

at m

any

wor

kers

hav

e ta

ken

the

tack

of

assu

min

g th

at f

or e

ach

verb

ther

e is

an

idio

sync

ratic

set

of

role

s,T

his

is, o

f cou

rse,

not

terr

ibly

sat

isfa

ctor

y ei

ther

, bec

ause

this

sim

ply

Implied Constituents

The

not

ion

that

the

repr

esen

tatio

n of

a s

ente

nce

cons

ists

of

an as

sem

blag

e

of r

epre

sent

atio

ns o

f co

nstit

uent

s of

a s

ente

nce

fails

to p

rovi

de a

ny d

irec

tw

ay o

f un

ders

tand

ing

why

it is

that

man

y se

nten

ces

conv

ey im

plie

dco

nstit

uent

s w

hich

nat

ive

spea

kers

do

not n

eed

to h

ear

men

tione

d, T

hus

in11 and 12

11. T

he b

oy s

prea

d th

e je

lly o

n th

e br

ead,

12. T

he m

an s

tirre

d hi

s co

ffee,

we

can

infe

r a

knif

e an

d a

spoo

n, re

spec

tivel

y, T

hat s

uch

infe

rred

con

sti-

tuen

ts a

re e

xpec

ted

to b

e pa

rts

of th

e re

pres

enta

tions

we

form

in li

sten

ing

to s

ente

nces

, is

indi

cate

d by

the

fact

that

we

can

refe

r to

them

as

thou

ghth

ey h

ave

been

men

tione

d, T

hus

we

can

say,

for

exam

ple,

13. T

he b

oy s

prea

d th

e je

lly o

n th

e br

ead,

The

kni

fe w

as c

over

ed w

ith p

oiso

n.

and

we

can

expe

ct th

e re

ader

to k

now

that

som

eone

is in

dan

ger

of b

eing

pois

oned

if th

ey e

at th

e sa

ndw

ich,

Now

, typ

ical

ly, i

t wou

ld b

e co

nven

tiona

l to

assu

me

eith

er th

at im

plie

d

cons

titue

nts

are

part

s bu

ilt in

to th

e re

pres

enta

tions

of

the

lexi

cal i

tem

s

(e,g

, the knife is built into the representation of the verb

spre

ad)

or that

they are inferred by post-pr

oces

ses,

How

ever

, it i

s by

no

mea

ns a

n ea

syta

sk to

dec

ide

whe

n so

met

hing

sho

uld

be b

uilt

in; n

or is

it e

asy

to d

ecid

ewhen something should be inferred, We do not

alw

ays

stir

cof

fee

with

a

spoon, and we do not even

nece

ssar

ily

spre

ad je

lly w

ith a

kni

fe; t

here

fore

,

drawing an inference in an al

l-or-

noth

ing

way

can

lead

to over-

com

mitm

ent.

We

mig

ht d

raw

infe

renc

es a

nd as

sign

them

str

engt

hs, b

ut

ther

e is

no

end

to th

e in

fere

nces

that

we

mig

ht d

raw

, Sho

uld

we

draw

all

ofth

em?

Whe

re s

houl

d th

e lin

e be

drawn? These problems have plagued

infe

renc

e-ba

sed

com

preh

ensi

on p

rogr

ams

for

year

s (S

chan

k, 19

81).

Com

bina

tori

al E

xplo

sion

or

Prem

atur

eC

omm

itmen

t?

The

mul

tiplic

ity o

f al

tern

ativ

e m

eani

ngs

of w

ords

and

of

poss

ible

rol

es, a

ndthe wide range of po

ssib

le in

fere

nces

whi

ch m

ight

follow from each

poss

ible

com

bina

tion

of r

oles

and

mea

ning

s, b

ecom

es a

n ex

trem

ely

seri

ous

5129

4M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

295

proh

lem

whe

n w

e co

nsid

er th

e im

plic

atio

ns f

or p

roce

ssin

g, F

amou

s ex

am-

ples

like

14, T

ime

flie

s lik

e an

arr

ow,

the

gene

ral c

hara

cter

istic

s of

nat

ural

lang

uage

s, In

deed

. the

re a

rc r

egul

ar-

ities

in th

e w

ay w

e st

ruct

ure

sent

ence

s w

hich

giv

e cl

ues

to th

e id

eas

we

wis

hth

ese

sent

ence

s to

con

vey;

and

ther

e ar

e re

gula

ritie

s in

the

way

s in

whi

ch

we

use

wor

ds, T

hese

two

fact

s se

em c

onsi

sten

t with

the

idea

that

wor

dsha

ve m

eani

ngs

that

are

par

ts o

f th

e m

eani

ngs

of th

e se

nten

ces

that

they

,oc

cur

in a

nd th

at th

e m

eani

ngs

of th

e w

hole

s ar

e co

nstr

ucte

d fr

om th

ese

part

s by

str

uctu

re-s

ensi

tive

rule

s, F

odor

and

Pyl

yshy

n (1

988)

are

. of

cour

se. c

orre

ct w

hen

they

poi

nt to

the

prod

uctiv

ity a

nd systematicity of

lang

uage

. and

it is

no

mea

n ac

com

plis

hmen

t of

the

clas

sica

l vie

w th

at it

capt

ures

thes

e es

sent

ial c

hara

cter

istic

s of

nat

ural

lang

uage

,B

ut it

is o

ur v

iew

that

the

clas

sica

l app

roac

h is

des

tined

to r

emai

nst

rapp

ed w

ith m

any,

if n

ot a

ll, o

f th

e pr

oble

ms

liste

d ab

ove,

Of

cour

se,

othe

rs h

ave

take

n a

diffe

rent

vie

w, a

nd m

any

prop

osal

s ha

ve b

een

mad

efo

r au

gmen

ting

or tu

ning

cla

ssic

ally

bas

ed m

odel

s of

sen

tenc

e pr

oces

sing

,T

hus,

for

exa

mpl

e, w

e fi

nd th

at m

any

curr

ent r

esea

rche

rs w

orki

ng w

ithin

the

clas

sica

l tra

ditio

n al

low

lexi

cal i

nfor

mat

ion

asso

ciat

ed w

ith th

e he

ads

ofconstituents to be referred to in parsing, Si

mila

rly.

tech

niqu

es s

uch

asbe

am s

earc

h ca

n be

use

d to

fin

d a

reas

onab

le c

ompr

omis

e be

twee

n th

eco

mbi

nato

rial

exp

losi

on th

at r

esul

ts f

rom

com

putin

g al

l pos

sibl

e pa

rses

and

the

prem

atur

e co

mm

itmen

t tha

t aris

es fr

om c

ompu

ting

only

one

; ess

en-

tially

, one

sim

ply

com

pute

s th

e be

st f

ew a

nd h

opes

that

wha

t tur

ns o

ut to

be th

e co

rrec

t int

erpr

etat

ion

happ

ens

to b

e am

ong

them

,It

is, o

f cou

rse,

pos

sibl

e th

at a

n ac

cum

ulat

ion

of in

crem

enta

l fix

es o

f thi

ski

nd w

ill u

ltim

atel

y pr

ovid

e an

ade

quat

e fr

amew

ork

for

mod

ellin

g th

ese

nten

ce c

ompr

ehen

sion

pro

cess

, But

our

bet

is th

at it

will

not

. Usi

ng

lexi

cal i

nfor

mat

ion

asso

ciat

ed w

ith h

eads

of

cons

titue

nts

does

not

sol

ve

the

who

le p

robl

em o

f con

tent

sen

sitiv

ity o

f par

sing

for

rea

sons

we

have

alre

ady

trie

d to

mak

e cl

ear,

and

bea

m s

earc

h is

just

a w

ay o

f elim

inat

ing

som

e, b

ut n

ot a

ll, c

ases

of

prem

atur

e co

mm

itmen

t with

out p

ayin

g to

o hi

gha

cost

in te

rms

of m

aint

aini

ng m

ultip

le p

arse

s,O

ur p

oint

is s

impl

y th

is: M

odel

s fo

rmul

ated

in th

e cl

assi

cal f

ram

ewor

kfa

ce m

any

seri

ous

prob

lem

s-pr

oble

ms

whi

ch a

utho

rs li

ke F

odor

and

Pyly

shyn

do

not a

ckno

wle

dge

whe

n to

utin

g th

e vi

rtue

s of

the

clas

sica

l

appr

oach

, Whi

le w

e ac

know

ledg

e th

e achievements of the

clas

sica

l

appr

oach

, we

sim

ply

belie

ve th

at it

mak

es s

ense

to e

xplo

re th

e po

ssib

ility

of a

n al

tern

ativ

e w

hich

dea

ls d

irec

tly w

ith th

e di

ffic

ultie

s th

at it

fac

es, o

nth

e vi

ew th

at s

uch

an a

ppro

ach

may

turn

out

ulti

mat

ely

to lead to a

supe

rior

over

all a

ccou

nt. T

he r

est o

f thi

s pa

per

is a

n at

tem

pt to

giv

e th

ere

ader

a s

ense

of

wha

t thi

s al

tern

ativ

e m

ay b

e lik

e,

rem

ind

us o

f th

e po

tent

ial c

ombi

nato

rial

exp

losi

on a

ssoc

iate

d w

ith th

em

ultip

licity

of

poss

ihle

wor

d-m

eani

ng a

nd s

truc

tura

l pos

sibi

litie

s th

at a

rise

in p

roce

ssin

g vi

rtua

lly e

very

sen

tenc

e, M

odel

s bu

ilt in

the

clas

sica

l tra

di-

tion

arc

forc

ed to

take

one

of

two

appr

oach

es to

this

pro

blem

: Eith

er th

eyca

n cr

eate

a p

oten

tially

exp

onen

tial n

umbe

r of

pos

sibl

e in

terp

reta

tions

. or

they

can

mak

e an

ear

ly c

omm

itmen

t to

purs

ue o

nly

a lim

ited

rang

e of

alte

rnat

ives

, In

the extreme form, a si

ngle

trac

k is

cho

sen,

sub

ject

tohack tracking if that track turns out to fa

il.

The

Diff

icul

ty o

f Acq

uisi

tion

As

a fin

al n

ote.

we

rem

ind

the

read

er o

f th

e problem of ac

quis

ition

,Se

vera

l ser

ious

pro

hlem

s fa

ce a

nyon

e w

ho a

ttem

pts

to b

uild

a m

odel

of

acqu

isiti

on o

f th

e ru

les

and

wor

d m

eani

ngs

posi

ted

by th

e cl

assi

cal v

iew

:

. The ru

les

arc

ofte

n ov

erri

dden

, as

we

saw

abo

ve,

. The co

rrec

t cho

ice

of r

ules

is d

rast

ical

ly u

nder

dete

rmin

ed b

y th

e ev

i-dence available to the child,

. The feedback ch

ildre

n re

ceiv

e on

the

corr

ectn

ess

of th

eir

cons

truc

tions

is n

otor

ious

ly im

pove

rish

ed,

. A given se

nten

ce m

ay h

ave

mor

e th

an o

ne p

erfe

ctly

acc

epta

ble

inte

rpre

-ta

tion,

Thi

s m

akes

it h

ard

to k

now

whe

n to

rej

ect a

rul

e as

wro

ng o

rsi

mpl

y no

t alw

ays

righ

t.. Correct performance re

quir

es n

ot o

nly

the

know

ledg

e of

the

cons

trai

nts

but h

ow m

uch

wei

ght e

ach

one

shou

ld b

e gi

ven,

. The ch

ild f

aces

a v

ery

seri

ous

boot

-str

appi

ng p

robl

em in

lear

ning

to m

apse

nten

ces

on to

thei

r m

eani

ngs,

Thi

s pr

oble

m is

rev

iew

ed b

y G

leitm

anan

d W

anne

r (1

982)

,

The

se a

nd o

ther

pro

blem

s ha

ve le

d m

any

psyc

holin

guis

ts c

omm

itted

toth

e vi

ew th

at a

cqui

sitio

n in

volv

es le

arni

ng r

ules

to th

e vi

ew th

at a

cqui

sitio

nis

impo

ssib

le, I

nste

ad, i

t has

oft

en b

een

prop

osed

that

the

rule

s of

all

lang

uage

s ar

e in

nate

and

that

acq

uisi

tion

sim

ply

amou

nts

to s

ettin

g pa

ra-

met

ers

whe

re th

ere

are

degr

ees

of f

reed

om, I

t has

eve

n be

en p

ropo

sed

(e, g

, Cho

msk

y. 1

988)

that

it is

not

impl

ausi

ble

to im

agin

e th

at a

ll co

ncep

tsar

e in

nate

,

Sum

mar

y

We

do n

ot w

ish

to m

ake

light

of

clas

sica

l mod

els,

Suc

h m

odel

s do

hav

eco

nsid

erab

le a

ppea

l, an

d th

ey s

eem

to u

s to

cap

ture

app

roxi

mat

ely

som

e of

5129

6M

cCLE

LLA

ND

, ST

, JO

HN

, TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

297

A P

DP

ALT

ER

NA

TIV

Eth

e re

pres

enta

tions

we

form

. in

orde

r to

be

able

to a

nsw

er s

impl

e qu

estio

nsabout the events these sentences describe,

Giv

en th

is c

once

ptio

n of

com

preh

ensi

on, w

e w

ill n

eed

a m

odel

whi

chca

n ac

tual

ly a

ppre

hend

sen

tenc

es a

nd th

en r

espo

nd c

orre

ctly

to a

set

of

prob

es, B

ecau

se w

e do

not

stip

ulat

e ex

actly

wha

t for

m th

e re

pres

enta

tions

mus

t tak

e, w

e m

ust r

ely

on th

e ad

equa

cy o

f the

per

form

ance

of t

he m

odel

to d

eter

min

e if

in f

aet i

ts r

epre

sent

atio

ns a

re a

dequ

ate,

For the purposes of what fo

llow

s, w

e w

ill d

istin

guis

h be

twee

n th

eprocess of comprehension itself-

the

form

atio

n of

a r

epre

sent

atio

n fr

om a

sent

ence

-and

the

use

of th

is r

epre

sent

atio

n to

res

pond

app

ropr

iate

ly to

prob

es, O

ur m

ain

inte

rest

is in

the

form

er, b

ut f

or th

e re

ason

s ju

st g

iven

.th

e la

tter

mus

t be

cons

ider

ed a

s w

ell,

or w

e ha

ve n

o m

easu

reof

suc

cess

ful

perf

orm

ance

,

Den

ied

Pres

uppo

sitio

nsT

he I

'DI'

alte

rnat

ive

whi

ch w

e w

ill p

ropo

se d

enie

s th

e po

int o

f dep

artu

re.

impl

icit

in c

lass

ical

app

roac

hes.

is n

eces

sary

to r

equi

re in

form

atio

n to

be

disp

laye

d in

str

uctu

red

form

in th

e re

pres

enta

tion

itsel

f (va

n G

elde

r, in

pres

s), R

athe

r. w

e as

k on

ly th

at th

e re

pres

enta

tions

pro

vide

a s

uffic

ient

nasi

s fo

r pe

rfor

min

g th

e ta

sk o

r ta

sks

that

are

req

uire

d of

them

, Thu

srepresentations of sentences are not required to ex

hibi

t a s

peci

fiei

ally

propositional format

so lo

ng a

s th

ey c

an b

e IIs

ed to

per

form

the

tl/.~

ks w

e

requ

ire,

Si

mila

rly.

rep

rese

ntat

ions

of

know

ledg

e ab

out h

ow to

for

m r

epre

-se

ntat

ions

are

not

req

uire

d to

take

the

form

of

rule

s ~

long

as

this

know

ledg

e al

/ow

s u,

~ to

act

inla

wfl

ll w

ays

as th

e en

l'iro

nmef

ll de

man

dsan

dre

pres

enta

tions

of w

ord-

spec

ific

know

ledg

e ar

e no

t req

uire

d to

hav

e an

yvi

sibl

e in

tern

al s

truc

ture

rep

rese

ntin

g th

e m

eani

ng o

f th

e w

ord,

Ind

eed

the

know

ledg

e of

rul

es a

nd o

f w

ord-

spec

ific

info

rmat

ion

may

wel

l be

elll"

oded

in a

den

sely

com

pile

d fo

rm. a

s lo

ng a

s th

is in

form

atio

n ca

n be

used

eff

ectiv

ely

to m

eet t

he im

pose

d de

man

ds,

Nat

ure

of th

e T

ask

Con

stra

int S

atis

fact

ion

Proc

essi

ngW

e th

ink

of th

e . p

roce

ss o

f com

preh

ensi

on a

s a

cons

trai

nt s

atis

fact

ion

proc

ess

(Rum

etha

rt. S

mol

ensk

y, M

cCle

lland

, & H

into

n, 1986b), In the

com

preh

ensi

on o

f is

olat

ed s

ente

nces

, there are two sorts of constraints:

thos

e im

pose

d by

the

sequ

ence

of w

ords

, and

thos

e im

pose

d by

kno

wle

dge

abou

t how

suc

h se

quen

ces

are

to b

e in

terp

rete

d, B

oth

type

s of

con

stra

ints

are taken to be

grad

ed,

The

y ar

e as

sum

ed to

act

as

forc

es s

hapi

ng th

efo

rmat

ion

of a

rep

rese

ntat

ion.

and

to h

ave magnitudes which determine

thei

r de

gree

of

infl

uenc

e, F

or o

ur p

urpo

ses,

the

sequ

ence

of

wor

ds in

the

sent

ence

can

be

inst

antia

ted

as a

seq

uenc

e of

pat

tern

s of

act

ivat

ion

over

ase

t of p

roce

ssin

g un

its, A

s ea

ch n

ew w

ord

com

es in

, we

assu

me

that

it is

used

to u

pdat

e th

e se

nten

ce r

epre

sent

atio

n, w

hich

is a

lso

take

n to

he

apa

ttern

of

activ

atio

n ov

er a

set

of

proc

essi

ng u

nits

, In

fact

, if

we

cons

ider

the

proc

ess

at e

ach time-step, it is useful to view it as a constraint

satis

fact

ion

proc

ess

in w

hich

ther

e ar

e tw

o in

puts

: the

sen

tenc

e re

pres

enta

-tio

n fr

om th

e pr

evio

us ti

me-

step

and

the

new

inpu

t. T

hese

two

inpu

ts a

rcus

ed to

pro

duce

an

upda

ted

sent

ence

rep

rese

ntat

ion

for

the

next

tim

e-st

ep,

The

kno

wle

dge

of h

ow th

is u

pdat

ing

is to

be

perf

orm

ed is

sto

red

in th

eco

nnec

tions

that

allo

w th

ese

inpu

ts to

upd

ate

the

sent

ence

rep

rese

ntat

ion,

It m

ay b

e w

orth

not

ing

that

gra

ded

cons

trai

nts

can

vary

in m

agni

tude

from

thos

e th

at a

rc s

o w

eak

that

they

are

ver

y ea

sily

ove

rrid

den

to th

ose

that

are

so

stro

ng th

at th

ey a

re n

early

impo

ssib

le to

ove

rrid

e ev

en h

y a

cons

pira

cy o

f ot

her

quite

str

ong

cons

trai

nts,

Thu

s th

e ex

iste

nce

of c

ases

inw

hich

con

stra

ints

are

not

ove

rrid

den

does

not

arg

ue a

gain

st th

e id

ea th

atth

ey a

re g

rade

d; it

just

indi

cate

s th

at s

omet

imes

they

can

be

very

str

ong.

Aft

er e

ach

upda

te o

f th

e se

nten

ce re

pres

enta

tion,

it c

an b

e us

ed to

resp

ond

to o

ne o

r m

ore

prob

es. R

espo

ndin

g to

thes

e pr

obes

is a

lso

view

edas

a c

onst

rain

t sat

isfa

ctio

n pr

oces

s, w

here

the

goal

is to

pro

duce

ext

erna

lly

Our

fir

st s

tep.

then

. mus

t he

to d

evel

op s

ome

conc

eptio

n of

the

natu

re o

fth

e im

pose

d de

man

ds, A

t a g

ener

al le

vel.

we

thin

k it

is r

easo

nabl

e to

thin

kof

the

sent

ence

com

preh

ensi

on ta

sk in

the

follo

win

g te

rms,

A s

eque

nce

ofw

ords

is p

rese

nted

. and

the

com

preh

ende

r m

ust f

orm

a r

epre

sent

atio

nw

hich

allo

ws

him

to r

espo

nd c

orre

ctly

whe

n pr

obed

in v

ario

us w

ays,

Inge

nera

l, th

e pr

ohes

can

take

a w

ide

rang

e of

diff

ercn

t for

ms.

req

uirin

gac

tions

, ver

hal r

espo

nses

. etc

. Am

ong

the

thin

gs w

e w

ould

exp

ect i

s th

atw

e w

ould

he

ahle

to a

nsw

er v

ario

us q

uest

ions

usi

ng th

is r

epre

sent

atio

n,For example. on apprehending "T

he m

an s

tirre

d th

e co

ffee

, we

wou

ldex

pect

a d

evic

e th

at h

as u

nder

stoo

d th

is s

ente

nce

to h

e ab

le to

giv

e co

rrec

tan

swer

s to

man

y 4u

estio

ns: W

ho d

id th

e st

irri

ng?

Wha

t did

he

stir

?, W

hat

did

he s

tir w

ith?

, and

so

on.

Or~

ours

e. th

ere

are

othe

r as

pect

s to

lang

uage

pro

cess

ing;

for

exa

mpl

e,in

pro

cess

ing

lang

uage

we

have

exp

ecta

tions

for

wha

t the

nex

t wor

d w

illhe

. and

we

can

thin

k of

par

t of

the

task

of

lang

uage

pro

cess

ing

as th

ean

ticip

atio

n of

the

next

wor

d, R

ecen

tly. E

lman

(19

89)

has

appl

ied

anap

proa

ch s

imila

r to

the

one

we

take

her

e to

this

seq

uent

ial a

ntic

ipat

ion

task, In this task

, lis

tene

rs le

arn

to c

onst

ruct

rep

rese

ntat

ions

that

ren

ect

the

pure

ly s

e4ue

ntia

l str

uctu

re o

f la

ngua

ge, H

ere

we

focu

s on

lear

ning

toco

nstr

uct r

epre

sent

atio

ns th

at r

efle

ct th

e co

nstr

aint

s se

nten

ces

impo

se o

n

5129

8M

cCLE

LLA

ND

, ST

, JO

HN

, TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

299

know

ledg

e ac

quir

ed th

at g

over

ns th

e co

nstr

uctio

n of

the

sent

ence

rep

re-

sent

atio

n fr

om th

e se

quen

ce o

f w

ords

, and

the

prod

uctio

n of

app

ropr

iate

outp

uts

to s

ente

nce/

prob

e co

mbi

natio

ns?

The

answer to all of th

ese

ques

tions

is th

e sa

me:

con

nect

ion

stre

ngth

adj

ustm

ent t

hrou

gh e

rror

-correcting learning,

We

assu

me

that

the

outp

ut p

atte

rn a

ctua

lly g

ener

ated

by

the

netw

ork

inre

spon

se to

eac

h pr

obe

is c

ompa

red

to th

e co

rrec

t out

put t

hat i

s pr

ovid

edas

par

t of t

he e

nviro

nmen

t. T

he m

ism

atch

bet

wee

n th

e ne

twor

k's

actu

alou

tput

and

the

corr

ect a

nsw

er is

then

use

d as

the

basi

s fo

r co

nnec

tion

strength adjustment, following the back-pr

opag

atio

n le

arni

ng p

roce

dure

,T

his

conn

ectio

n ad

just

men

t pro

cess

occ

urs

for connections in both the

comprehension network and for connections in the readout network,

grad

ually

lead

ing

the

netw

ork

to le

arn

both

how

to r

epre

sent

the

info

rma-

tion

in e

ach

sent

ence

and

how

to u

se it

to r

espo

nd to

eac

h pr

obe,

Not

e th

at th

is c

onne

ctio

n st

reng

th a

djus

tmen

t pro

cess

can

not a

ctua

llyre

sult

in p

erfe

ct p

erfo

rman

ce, b

ecau

se m

any

of th

e sentences that the

netw

ork

sees

are

in f

act a

mbi

guou

s, F

urth

erm

ore,

ear

ly o

n du

ring

pro

ces-

sing of a particular sentence

, bef

ore

the

who

le s

ente

nce

has

been

pre

-se

nted

, the

net

wor

k ca

n on

ly m

ake

its b

est g

uess as to the answers to

certain questions, w

ithou

t any

pos

sibi

lity

that

it c

an a

lway

s be

rig

ht. I

ttu

rns

out t

hat i

t can

be

show

n th

at th

e le

arni

ng p

roce

dure

is adjusting the

strengths of the connections am

ong

the

units

in th

e ne

twor

k in

the

dire

ctio

n of

min

imis

ing

the

disc

repa

ncy

betw

een

the

activ

atio

n of

eac

h un

itan

d th

e pr

obab

ility

that

it s

houl

d be

act

ive,

giv

en th

e in

put t

hat h

as b

een

pres

ente

d up

to th

is p

oint

. And

, ind

eed,

the

activ

atio

ns c

ome

grad

ually

tore

flec

t the

se p

roba

bilit

ies

reas

onab

ly w

ell.

In o

ur s

imul

atio

ns th

ey te

nd to

grad

ually

app

roac

h an

equ

ilibr

ium

, in

whi

ch th

ey ji

tter

abou

t the

true

prob

abili

ties

base

d on

the

vici

ssitu

des

of th

e m

ost r

ecen

t set

of t

rain

ing

exam

ples

,

FIG, 1 A

ske

tch

of th

e pr

esen

t con

cept

ion

of th

e se

nten

ce c

ompr

ehen

sion

mec

hani

sm. T

heov

als

repr

esen

t gro

ups

of u

nits

, and

the

arro

ws

repr

esen

t mod

ifiab

le c

onne

ctio

ns.

spec

ified

out

puts

in r

espo

nse

to e

xter

nally

pro

vide

d pr

obes

, The

re a

re n

owthree sources of constraint; the sentence representation

, the

pro

be, a

ndkn

owle

dge

abou

t wha

t out

puts

sho

uld

be p

rodu

ced

for

part

icul

ar s

ente

nce/

prob

e co

mbi

natio

ns, B

oth

the

sent

ence

rep

rese

ntat

ion

and

the

prob

e ca

nbe instantiated as patterns of activation over processing units, as can the

desi

red

outp

uts;

and

the

know

ledg

e of

how

to p

rodu

ce th

ese

outp

uts

from

the

corr

espo

ndin

g in

puts

can

be

enco

ded

in th

e co

nnec

tions

am

ong

the

proc

essi

ng u

nits

,So

far

we

have

out

lined

a g

ener

al f

ram

ewor

k fo

r se

nten

ce c

ompr

ehen

-si

on a

nd f

or u

sing

the

resu

lts o

f co

mpr

ehen

sion

to r

espo

nd to

pro

bes,

Ask

etch

of t

he n

etw

ork

that

inst

antia

tes

this

fram

ewor

k is

sho

wn

in F

ig,), In

the

figu

re, t

he o

vals

cor

resp

ond

to p

ools

of

units

and

the

arro

ws

corr

es-

pond

to c

onne

ctio

ns, T

here

is a

poo

l of

units

for

rep

rese

ntin

g th

e su

cces

-si

ve w

ords

: a p

ool o

f un

its f

or r

epre

sent

ing

the

evol

ving

sen

tenc

e re

pres

en-

tation, or

Sentence Gestalt;

a po

ol f

or r

epre

sent

ing

prob

es; a

nd a

poo

l for

repr

esen

ting

resp

onse

s to

the

prob

es, T

he a

rrow

s re

pres

ent c

onne

ctio

ns,

from

eac

h un

it in

the

pool

at t

he s

endi

ng e

nd o

f th

e ar

row

to e

ach

unit

inth

e po

ol a

t the

rec

eivi

ng e

nd, T

he u

nits

in th

e un

labe

lled

pool

s, w

hich

will

simply be called "hi

dden

uni

ts, serve to allow

com

bina

tions

of

asp

ects

of

the patterns on the input side of these pools to constrain the patterns of

activ

atio

n on

the

outp

ut s

ide,

A M

OD

EL

IL

LU

STR

AT

ING

TH

E A

PPR

OA

CH

Lear

ning

by

Con

nect

ion

Adj

ustm

ent

Thr

ee c

ruci

al q

uest

ions

rem

ain.

Firs

t , w

hat d

eter

min

es th

e fo

rm o

f th

ese

nten

ce r

epre

sent

atio

n its

elf?

Sec

ondl

y, h

ow is

the

form

of

this

rep

rese

n-ta

tion

com

mun

icat

ed to

the

inne

r pa

rt o

f th

e ne

twor

k? T

hird

ly, h

ow is

the

The

mod

el w

e de

scri

be h

ere

exem

plif

ies

the

appr

oach

des

crib

ed a

bove

, It

is in

man

y w

ays

high

ly s

impl

ifie

d, I

t will

not

con

vinc

e th

e re

ader

that

we

have

alr

eady

suc

ceed

ed in

pro

vidi

ng a

com

plet

e al

tern

ativ

e to

con

vent

iona

lap

proa

ches

. Rat

her ,

it p

rovi

des

a co

ncre

tisat

ion

of th

e ge

nera

l app

roac

has

wel

l as

an il

lust

ratio

n of

som

e of

the

reas

ons

for

its a

ppea

l, w

hich

we

hope

will

sug

gest

that

the

furt

her

expl

orat

ion

of th

is n

ew fr

amew

ork

isw

orth

whi

le, T

he m

odel

is c

alle

d th

e Se

nten

ce G

esta

lt or

SG

mod

el. I

t is

described briefly here (a fuller de

scri

ptio

n is

ava

ilabl

e in

St.

John

& .

McC

lella

nd, i

n pr

ess)

,

5130

0M

cCLE

LLA

ND

, ST

. JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

301

The

Env

iron

men

t

The

mod

el c

onsi

sts

of a

net

wor

k pl

aced

in a

n en

viro

nmen

t con

sist

ing

ofse

nten

ce/e

vent

des

crip

tion

pair

s. T

he s

ente

nces

arc

of

but o

ne c

laus

e, a

ndth

ey c

onsi

st o

f a

sequ

ence

of

stri

pped

-dow

n co

nstit

uent

s, E

ach

cons

titue

ntco

nsis

ts o

f a s

ingl

e co

nten

tive

(nou

n, verb, or adverb) together with a

sing

le p

repo

sitio

n or

the

verb

al a

uxili

ary

elem

ent "

was

. For

exa

mpl

e, t

heEnglish sentence "T

he s

choo

l gir

l was

kis

sed

by th

e bo

y"

is r

educ

ed to

thre

e co

nstit

uent

s-sc

hool

girl

"

, "

was

kis

sed"

, "

by b

oy, T

he m

ost c

om-

plex

sen

tenc

es in

volv

ed d

ativ

e pa

ssiv

es li

ke "

The

teac

her

was

giv

en a

ros

eby the bus driver

. with

add

ition

al lo

cativ

e, m

anne

r, a

nd/o

r in

stru

men

tal

prep

ositi

onal

phr

ases

pos

sibl

e, d

epen

ding

on

the

verb

.T

he e

vent

des

crip

tions

are

sim

ple

too;

they

con

sist

onl

y of

a li

st o

f ro

lc-

fille

r pa

irs, F

or "

The

sch

oolg

irl w

as k

isse

d by

the

boy

, the

list

is: (

agen

t:bo

y; a

ctio

n: k

iss;

pat

ient

: sch

oolg

irl)

, The

rol

es a

re a

gent

, act

ion

, obj

ect

reci

pien

t. lo

catio

n. m

anne

r, in

stru

men

t, an

d w

hat m

ight

bes

t be

calle

dac

com

pani

st" (as in "the bus driver ate the ice-cream with the school-

girl

" Whi

le th

e se

nten

ces

and

the

even

ts th

ey d

escr

ibe

arc

both

qui

te s

impl

e,th

e re

latio

nshi

ps w

hich

hol

d be

twee

n th

em a

re n

ot. F

or o

ne th

ing,

wor

dsus

ed in

a s

ente

nce

may

be

ambi

guou

s or

vag

ue, a

s in

(15

) an

d (1

6):

15. T

he p

itche

r hi

t the

bal

l with

the

bat.

16. The adult ate something,

thereby forcing the model to rely on the syntactic cues in the sentence.

(Not

e th

at th

e di

ffic

ulty

her

e is

fur

ther

incr

ease

d by

the

fact

that

the

mod

eldo

es n

ot d

istin

guis

h " g

ave

" fr

om "

give

n. W

e si

mpl

y us

e a

sing

le f

orm

for

each

ver

b th

roug

hout

, bec

ause

for

mos

t ver

bs th

e pa

st a

nd p

ast p

artic

iple

are

the

sam

e in

Eng

lish.

The

oth

er s

ourc

e of

rol

e as

sign

men

t diff

icul

ty a

rises

from

the

ambi

guity

of s

urfa

ce r

ole

cues

. Pre

posi

tions

and

word-order information provide

som

e cu

es, b

ut th

ese

cues

are

oft

en q

uite

am

bigu

ous

as to

the

role

s th

atth

ey s

igni

fy. T

hus

in (

19)

and

(20)

,

19. T

he b

us d

rive

r at

e th

e st

eak

with

the

teac

her.

20. T

he b

us d

river

ate

the

stea

k w

ith th

e kn

ife.

the

sem

antic

s of

the

role

- fill

er m

ust b

e co

nsid

ered

in d

eter

min

ing

whe

ther

the object of the with-ph

rase

is a

n in

stru

men

t or

an a

ccom

pani

st,

The

act

ual s

et o

f se

nten

ce-e

vent

pai

rs th

at th

e m

odel

see

s is

gen

erat

edas

fol

low

s. F

irst

, an

actio

n is

sel

ecte

d at

ran

dom

fro

m a

set

of

poss

ible

actio

ns. T

hen

, an

agen

t is

sele

cted

fro

m a

set

of

poss

ible

age

nts

who

mig

htpe

rfor

m th

e ac

tion.

Fol

low

ing

this

, an

obje

ct, a

n in

stru

men

t. or

indi

rect

obje

ct if

app

licab

le, a

nd o

ther

rol

es a

re fi

lled,

An

illus

trat

ion

is g

iven

for

the action "ea

t" in

Fig

, 2. N

ote

that

the

.sel

ectio

n pr

oces

s is

inhe

rent

lyro

babi

listic

, and

that

ther

e ar

e co

mpl

ex d

epen

denc

ies.

Giv

en, f

or e

xam

-

Structure of Events

In b

oth

case

s, th

e m

odel

is a

sked

to d

o its

bes

t to

reco

ver

the

corr

ect e

vent

desc

ript

ion,

In

the

latte

r ca

se, t

he e

vent

des

crip

tion

invo

lves

a s

peci

ficad

ult a

nd a

spe

cifi

c so

met

hing

eat

en. w

hich

may

not

be

uniq

uely

pre

dict

-ab

le (

in th

e sm

all w

orld

of

the

mod

el, t

he a

dult

mig

ht b

e a

teac

her

or a

bus

driv

er; t

he s

omet

hing

mig

ht b

e so

up o

r a

stea

k). T

he m

odel

mus

t do

itsbe

st b

ased

on

the

info

rmat

ion

give

n.C

onst

ituen

ts m

ay a

lso

be le

ft o

ut o

f se

nten

ces ,

as

in (

17):

17, T

he b

us d

river

stir

red

the

coffe

e.

Her

e th

e ne

twor

k is

exp

ecte

d to

und

erst

and

that

the

even

t bei

ng d

escr

ibed

invo

lved

an

inst

rum

ent,

whi

ch in

the

case

of

stir

ring

is a

lway

s a

spoo

n,R

ole

assi

gnm

ent i

s m

ade

diff

icul

t in

two

way

s. F

irst

, bot

h ac

tive

and

pass

ive

cons

truc

tions

are

use

d. T

houg

h th

ere

are

sem

antic

con

stra

ints

that

ofte

n m

ake

a co

rre c

t int

erpr

etat

ion

of p

assi

ves

poss

ible

, thi

s is

not

alw

ays

the case

, as

in s

ente

nces

like

(18

):

18. T

he te

ache

r w

as g

iven

the

rose

by

the

bus

driv

er.

In th

is a

nd o

ther

kin

ds o

f ca

ses,

the

corp

us w

as s

truc

ture

d so

that

the

two

hum

an p

artic

ipan

ts w

ere

equa

lly li

kely

to s

erve

as

agen

t or

as r

ecip

ient

,

Obj

ect

Tea

cher

Bus

Dri

ver

Scho

olG

irl P

itche

r

I~~I

0 I S

oup

Ste

ak 1

1.0 IceCream

11.0

1.0

I

Inst

rum

ent S

poon

Kni

fe

Act

ion

Age

nt

Man

ner

Daintiness Gusto

FIG

, 2 S

truc

ture

of the event generator for the action

eat

used

in tr

aini

ng th

e S

O m

odel

.

5130

2M

cCLE

LLA

ND

, ST

, JO

HN

, TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

303

pie.

that

the

actio

n is

eat

and

the

agen

t is

bus

driv

er. t

he o

bjec

t is

prob

ably

stea

k (P

=

0,7

) but may be soup

(P

= 0

, 1);

the

inst

rum

ent d

epen

ds o

n th

eob

ject

eat

en. t

he m

anne

r on

the

agen

t of e

atin

g,T

his

proc

edur

e pr

oduc

es a

n en

sem

ble

of e

vent

des

crip

tions

whi

ch a

restrongly constrained, These constraints can be absolute or

hard

. so

that

for

exam

ple.

ani

mal

bat

s do

not

sho

w u

p at

all

as th

e in

stru

men

ts o

f hi

tting

;or they may be

.wft

. so

that

. for

exa

mpl

e. s

teak

is th

e pr

efer

red

but n

ot th

eun

ique

obj

ect o

f ea

ting

for

the

bus

driv

er. N

ote

that

the constraints are

fair

ly c

ompl

ex, i

n th

at th

ey d

epen

d on

par

ticul

ar c

onju

nctio

ns o

f ve

rbs

and

role

fille

rs, S

teak

is th

e pr

efer

red

food

onl

y of

the

bus

driv

er. t

he k

nife

isth

e in

stru

men

t of

eatin

g w

hen

the

food

is s

teak

but

not

sou

p. e

tc,

The

ass

ignm

ent o

f w

ords

to e

vent

s is

als

o pr

obab

ilist

ic, T

hus.

the

bus

driver in the eating example might be described with the words

bus

driv

eror with the word

adul

t; the steak might be described as

stea

k or as

food

;the instrument as

knif

e or as

uten

sil,

Thu

s, th

e ac

tual

spe

cific

par

ticip

ants

in th

e ev

ents

can

onl

y be

infe

rred

by

usin

g in

form

atio

n fr

om c

onte

xt.

Som

etim

es, t

he s

ente

nce

cont

ains

suf

fici

ent i

nfor

mat

ion

to r

emov

e al

lun

cert

aint

y w

ith r

espe

ct to

a p

artic

ular

par

ticip

ant (

as in

"th

e bu

s dr

iver

ate

the

food

with

the

knife

; the

food

can

onl

y be

ste

ak),

but

oth

er ti

mes

not (

as in

"th

e bu

s dr

iver

ate

the

food

"), E

ven

here

som

e an

swer

s m

ay b

em

ore

likel

y th

an o

ther

s. th

ough

in s

ome

case

s th

ere

may

be

at le

ast t

wo

equa

lly li

kely

alte

rnat

ives

(in

"th

e ad

ult a

te th

e fo

od" ,

sou

p an

d st

eak

are

equa

lly li

kely

),So

met

imes

, who

le c

onst

ituen

ts a

re s

impl

y le

ft o

ut o

f se

nten

ces

desc

rib-

ing

even

ts in

whi

ch th

eir

refe

rent

s ap

pear

. Thu

s. th

e kn

ife c

an b

e le

ft ou

t of

the

sent

ence

on

the

bus

driv

er e

atin

g st

eak,

The

mod

el a

dher

es to

the

conventions that su

hjec

t and

ver

b ar

e always mentioned (however

vagu

ely)

, but

oth

er c

onst

ituen

ts m

ay g

o un

men

tione

d, d

epen

ding

on

the

specific actions,

The

Tas

k an

d th

e In

terf

ace

to th

e E

nvir

onm

ent

The

mod

el's

task

is to

pro

cess

the

sequ

ence

of

cons

titue

nts

that

rep

rese

nts

a pa

rtic

ular

sen

tenc

e an

d. a

s ea

ch c

onst

ituen

t com

es in

, to update a

repr

esen

tatio

n w

hich

is in

tend

ed to

allo

w it

to r

espo

nd to

pro

bes

quer

ying

its c

ompr

ehen

sion

of t

he e

vent

des

crib

ed b

y th

e se

nten

ce. T

o as

sess

the

mod

el's

per

form

ance

, we

can

actu

ally

pro

be it

afte

r ea

ch c

onst

ituen

t has

been

pro

cess

ed,

Eac

h in

put c

onst

ituen

t con

sist

s of

a c

onte

nt w

ord

and

poss

ibly

a p

repo

si-

tion

or "

was

, Eac

h su

ch w

ord

is r

epre

sent

ed b

y a

sing

le u

nit,

Thu

s th

ere

isa

unit

for

" bat

" (r

egar

dles

s of

mea

ning

), a

uni

t for

"ga

ve, a unit for

adul

t", a

uni

t for

"w

as

, "

with

", "

, etc

, Alto

geth

er, t

here

wer

e un

itsfo

r 58

wor

ds,

A s

imila

r lo

calis

t rep

rese

ntat

ion

sche

me

was

als

o us

ed f

or p

robe

s an

dre

spon

ses,

Res

pond

ing

to a

pro

be c

an b

e th

ough

t of

as c

ompl

etio

n: fi

lling

in a

mem

ber

of a

rol

e-fi

ller

pair

, whe

n pr

obed

with

eith

er th

e ro

le o

r th

efi

ller.

Not

e th

at th

e fi

llers

are

now

con

cept

s ra

ther

than

wor

ds, a

nd th

atfi

llers

in p

artic

ular

eve

nts

are

alw

ays

spec

ific

conc

epts

, rat

her

than

sup

er-

ordi

nate

cat

egor

ies,

The

re w

ere

a to

tal o

f 45

conc

ept u

nits

, cov

erin

gac

tions

, man

ners

, and

noun-concepts, including persons, places, and

thin

gs,

Giv

en th

is s

chem

e, w

e of

ten

find

sen

tenc

e-ev

ent p

airs

whe

re a

wor

d is

used

(sa

y " b

at" )

that

cor

resp

onds

to tw

o di

stin

ct c

once

pts

(bas

ebal

l bat

lfl

ying

bat

), A

s in

rea

l lan

guag

es, i

t is

not t

he w

ord

itsel

f whi

ch te

lls

whi

ch o

f th

e co

ncep

ts is

inte

nded

; the

cor

rect

ans

wer

mus

t be

deri

ved

bym

akin

g us

e of

cue

s pr

ovid

ed b

y th

e co

ntex

t in

whi

ch th

e w

ord

occu

rs.

In s

ome

sim

ulat

ions

usi

ng th

e m

odel

, St.

John

and

McC

lella

nd in

clud

eda

few

uni

ts r

epre

sent

ing

supe

rord

inat

e co

ncep

ts in

add

ition

to th

e un

its f

orsp

ecifi

c co

ncep

ts, I

n th

is c

ase,

a c

once

pt is

rep

rese

nted

not

by

a si

ngle

uni

tbu

t by

a se

t of

units

rep

rese

ntin

g th

e sp

ecif

ic c

once

pt a

nd it

s su

pero

rdin

ate

feat

ures

. Thu

s, f

or e

xam

ple ,

ther

e ar

e un

its fo

r pe

rson

, for

mal

e an

dfe

mal

e, f

or a

dult

and

child

. The

bus

dri

ver

is a

n ad

ult m

ale

and

the

teac

her

is a

n ad

ult f

emal

e, e

tc,

In c

onsi

deri

ng th

e ta

sk o

f th

e ne

twor

k, i

t is

wor

th n

otin

g th

at th

ere

is n

otal

way

s a

sing

le r

ight

ans

wer

. Ind

eed

, ear

ly o

n in

a s

ente

nce.

just

aft

er th

epr

esen

tatio

n of

the

first

con

stitu

ent ,

ther

e is

a g

reat

dea

l of

inde

term

inac

y;th

e in

itial

nou

nphr

ase

need

not

eve

n de

scri

be th

e ag

ent o

f th

e se

nten

ce,

Nev

erth

eles

s , it

is p

ossi

ble

to v

iew

eac

h co

nstit

uent

, as

it is

pre

sent

ed, a

simposing constraints on the possible event-de

scri

ptio

ns th

at m

ight

be

correct, In this context, w

e ca

n ch

arac

teri

se th

e ta

sk o

f th

e ne

twor

k as

bein

g on

e of

indi

catin

g, in

response to each probe, what the range of

poss

ibili

ties

mig

ht b

e, a

nd o

f gi

ving

an

indi

catio

n, b

y th

e ac

tivat

ions

that

itas

sign

s to

the

com

plet

ions

of

the

vari

ous

prob

es. o

f its

est

imat

e of

the

prob

abili

ty a

ssoc

iate

d w

ith e

ach,

Net

wor

k A

rchi

tect

ure

and

Proc

essi

ng

The

arc

hite

ctur

e of

the

netw

ork

, as

show

n in

Fig

. 1, c

an b

e tr

eate

d as

consisting of two basic parts. O

ne p

art i

s th

e actual comprehension

mec

hani

sm it

self

, the

par

t tha

t rea

ds in

the

cons

titue

nts

sequ

entia

lly a

ndup

date

s th

e se

nten

ce r

epre

sent

atio

n; th

e ot

her

part

is th

e ou

tput

mec

han-

ism

, tha

t per

form

s th

e pr

obe

com

plet

ion

task

. The

sen

tenc

e ge

stal

t units

are

in b

oth

part

s , a

nd f

orm

the

inte

rfac

e be

twee

n th

e tw

o.Pr

oces

sing

occ

urs

as f

ollo

ws.

At t

he b

egin

ning

of

a se

nten

ce, t

he p

atte

rnof

act

ivat

ion

on th

e se

nten

ce g

esta

lt un

its is

set

to a

ll O

', a

nd th

e un

it or

units

rep

rese

ntin

g th

e fi

rst i

nput

con

stitu

ent i

n th

e in

put p

ool a

re tu

rned

51 3

04 M

cCLE

LLA

ND

. ST

. JO

HN

. TA

RA

BA

N

on. A

ctiv

atio

n fe

eds

from

the

SG

uni

ts (

via

the

feed

back

loop

) an

d th

ein

put u

nits

to th

e hi

dden

uni

ts in

the

com

preh

ensi

on p

art o

f th

e sy

stem

,an

d fr

om th

ese

it fe

eds

on a

gain

to th

e S

G u

nits

, whe

re th

e in

itial

SG

repr

esen

tatio

n of

all

O's

is r

epla

ced

by a

new

pat

tern

of

activ

atio

n re

flec

ting

the

influ

ence

of t

he fi

rst c

onst

ituen

t of t

he s

ente

nce.

Thi

s re

pres

enta

tion

isno

w p

art o

f th

e in

put a

t the

nex

t tim

e-st

ep, w

hen

the

next

con

stitu

ent i

sin

put i

n pl

ace

of th

e fir

st. T

his

proc

ess

cont

inue

s to

the

end

of th

e se

nten

ce,

Eac

h of

the

units

insi

de th

e ne

twor

k is

a s

impl

e lo

gist

ic p

roce

ssin

g un

it;th

at is

, the

act

ivat

ion

that

a u

nit t

akes

on

is e

qual

to th

e lo

gist

ic f

unct

ion

ofits

net

inpu

t, w

here

the

net i

nput

is s

impl

y th

e su

m o

ver

all c

onne

ctio

nsco

min

g to

the

unit

of th

e in

put o

n ea

ch c

onne

ctio

n. T

he in

put o

n ea

chco

nnec

tion

is ju

st th

e pr

oduc

t of

the

activ

atio

n of

the

send

ing

unit

at th

eend of the connection x the weight on the connection. Activations range

from

0 to

I; w

eigh

ts a

re fl

oatin

g-po

int n

umbe

rs in

itial

ised

in a

ran

gebe

twee

n +

/- 0

., and adjusted according to the le

arni

ng p

roce

dure

described below.

Proc

essi

ng in

the

outp

ut n

etw

ork

is a

lso

quite

sim

ple

, and

can

occ

ur a

tan

y po

int d

urin

g or

aft

er th

e pr

esen

tatio

n of

a s

ente

nce.

The

two

inpu

ts to

the

outp

ut n

etw

ork

are

the

patte

rn o

n th

e SG

uni

ts a

nd th

e pa

ttern

on

the

prob

e un

its. T

his

patte

rn c

onsi

sts

of a

sin

gle

unit

on, r

epre

sent

ing

eith

er a

quer

ied

role

or

the

quer

ied

fille

r. A

ctiv

atio

n fe

eds

forw

ard

from

the

SGun

its a

nd th

e pr

obe

inpu

t uni

ts to

a s

et o

f hi

dden

uni

ts a

nd th

en f

rom

thes

eto the probe output un

its, w

here

the

patte

rn is

take

n to

rep

rese

nt th

ene

twor

k's

resp

onse

to th

e pr

obe,

SEN

TE

NC

E C

OM

PRE

HE

NSI

ON

AN

D P

DP

5130

5

Lea

rnin

g

Lea

rnin

g in

the

netw

ork

occu

rs v

ia th

e ba

ck- p

ropa

gatio

n le

arni

ng p

roce

-du

re. W

hen

a pr

obe

is p

rese

nted

, the response to the probe can be

com

pare

d to

the

resp

onse

that

wou

ld b

e co

rrec

t for

the

curr

ent s

ente

nce-

even

t pai

r, a

nd a

mea

sure

of

erro

r ca

lled

cros

s-en

trop

y ca

n be

com

pute

d.B

ack-

propagation is used to adjust the connection strengths so as to

min

imis

e th

is m

easu

re (

see

SI. J

ohn

& M

cCle

lland

, in

pres

s, f

or d

etai

ls).

It is

impo

rtan

t to

note

that

the

min

ima

in th

is m

easu

re occur at those

poin

ts w

here

the

activ

atio

ns o

f un

its in

par

ticul

ar s

ituat

ions

rep

rese

nt th

epr

obab

ilitie

s th

at th

e un

its s

houl

d be

on

in th

ese

situ

atio

ns. W

e th

ink

of th

eac

tivat

ions

of

the

outp

ut u

nits

as

repr

esen

ting

the

prob

abili

ty th

at th

e un

itsh

ould

be

on, T

he tr

aini

ng p

roce

dure

can

be

seen

as

tryi

ng to

fin

d an

ense

mbl

e of

con

nect

ion

wei

ght v

alue

s th

at a

l\ow

the

netw

ork

to g

et th

ese

probabilities correct.

In tr

aini

ng th

e ne

twor

k, w

e fol\owed the procedure of presenting a

com

plet

e se

t of

prob

es a

fter

the

pres

enta

tion

of e

ach

cons

titue

nt o

f ea

ch o

fa

larg

e nu

mbe

r of

trai

ning

sen

tenc

es. T

he c

ompl

ete

set o

f pr

obes

con

sist

ed

of a

rol

e pr

obe

and

a fil

\er

prob

e fo

r ea

ch r

ole-

fille

r pa

ir in

the

even

tde

scrip

tion

for

the

sent

ence

-eve

nt p

air

curr

ently

bei

ng pr

oces

sed.

Thi

s tr

aini

ng p

roce

dure

was

inte

nded

to a

ppro

xim

ate

the

situ

atio

n in

whi

ch a

lang

uage

lear

ner

has

just

witn

esse

d an

eve

nt, s

o th

at h

e al

read

y ha

sa

desc

ript

ion

of it

; and

hea

rs a

sen

tenc

e sp

oken

abo

ut th

at e

vent

. We

imagine that as the learner processes the sentence, he is co

ntin

ually

(im

plic

itly)

ask

ing

him

self

, "

how

wel

l doe

s th

e m

achi

nery

that

I ha

ve fo

rla

ngua

ge c

ompr

ehen

sion

allo

w m

e to

des

crib

e co

rrec

tly th

e ev

ent I

hav

eju

st w

itnes

sed"

. The

que

stio

n is

pos

ed in

the

form

of

the

set o

f pr

obes

, and

the

answ

er is

the

set o

f re

spon

ses

to th

e pr

obes

, The

mis

mat

ch b

etw

een

the

resp

onse

s to

the

prob

es a

nd th

e co

rrec

t res

pons

es d

icta

ted

by th

e de

scri

p-tion then serves as the basis for learning.

Thi

s pr

oced

ure

has

two

inte

rest

ing

char

acte

ristic

s. F

irst ,

it d

oes

not

prov

ide

the

lear

ner

with

any

spe

cifi

c al

ignm

ent b

etw

een

the

cons

titue

nts

ofth

e se

nten

ce a

nd th

e co

rres

pond

ing

cons

titue

nts

of th

e ev

ent d

escr

iptio

n.T

hus

it fo

rces

the

netw

ork

to d

isco

ver

the

solu

tion

to th

e bo

otst

rapp

ing

prob

lem

men

tione

d ea

rlie

r fo

r its

elf.

Sec

ondl

y, th

e pr

oced

ure

requ

ires

the

netw

ork

to d

o its

bes

t at e

ach

time-

step

to p

redi

ct a

ll of

the

cons

titue

nts

ofth

e ev

ent f

rom

wha

t is

has

seen

so

far.

If learning reaches the global

min

imum

in th

e er

ror

mea

sure

des

crib

ed a

bove

, the

n th

e ac

tivat

ions

will

alw

ays

refl

ect t

he b

est a

chie

vabl

e es

timat

es o

f th

e pr

obab

ilitie

s th

at th

eunits should be on at each point in the ~rocessing of every sentence.

Seve

ral d

iffe

rent

run

s of

the

mod

el h

ave

been

und

erta

ken.

The

one

fro

mw

hich

we

repo

rt r

esul

ts h

ere

invo

lved

630

000

trai

ning

tria

ls, e

ach

invo

lv-

ing

the

pres

enta

tion

of a

n in

depe

nden

tly g

ener

ated

sen

tenc

e-ev

ent p

air.

Som

e se

nten

ce-e

vent

pai

rs o

ccur

ofte

n, w

here

as o

ther

s oc

cur

extr

emel

yra

rely

. The

low

- fre

quen

cy it

ems

also

invo

lve

rela

tivel

y at

ypic

al r

ole-

fille

rsan

d it

is th

e pr

oces

s of

lear

ning

fro

m r

are

erro

rs to

ove

rcom

e th

e te

nden

cyto

giv

e th

e m

ost t

ypic

al a

nsw

er th

at m

akes

lear

ning

take

suc

h a

very

long

time.

A f

ulle

r di

scus

sion

of

the

time-

cour

se o

f ac

quis

ition

is p

rovi

ded

by S

t.Jo

hn a

nd M

cCle

lland

(in

pre

ss).

Res

ults

Aft

er tr

aini

ng, t

he m

odel

was

fir

st te

sted

on

a se

t of

55 r

ando

mly

gen

erat

edse

nten

ces

that

are

una

mbi

guou

s gi

ven

the

hard

con

stra

ints

bui

lt in

to th

eco

rpus

, Tha

t is

, alth

ough

eac

h of

thes

e se

nten

ces

actu

ally

con

tain

ed a

t lea

ston

e am

bigu

ous

wor

d or

uns

peci

fied

fill

er, t

he h

ard

cons

trai

nts

built

into

the

eo~pus were enough to al

\ow

"'it t

o re

spon

d co

rrec

tly to

all

prob

es. F

orex

ampl

e

, "

The

teac

her

ate

the

soup

with

the

uten

sil"

is u

nam

bigu

ous

beca

use

the

only

ute

nsil

that

cou

ld b

e us

ed f

or e

atin

g so

up is

a s

poon

. Aft

erth

e pr

esen

tatio

n of

eac

h se

nten

ce, w

e te

sted

the

full

set o

f pro

bes

for

the

role

- fill

er p

airs

in th

e ev

ent d

escribed by the sentence. The network

5130

6M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

307

activ

ated

all

of th

e co

rrec

t out

put u

nits

mor

e st

rong

ly th

an a

ny o

utpu

t uni

tsit

shou

ld n

ot h

ave

activ

ated

on

mor

e th

an 9

9% o

f th

e pr

obes

,T

he n

etw

ork

was

als

o te

sted

spe

cific

ally

on

seve

ral s

ets

of s

ente

nces

desi

gned

to a

sses

s its

abi

lity

to h

andl

e di

ffer

ent a

spec

ts o

f th

e co

mpr

ehen

-si

on ta

sk, T

he ta

sks

are

brok

en d

own

into

two

broa

d ca

tego

ries

, hav

ing

todo

with

rol

e as

sign

men

t on

the

one

hand

and

spe

cifi

catio

n of

the

iden

tity

ofro

le fi

llers

on

the

othe

r, W

ith r

egar

d to

rol

e as

sign

men

t, St

. Joh

n an

dM

cCle

lland

pro

bed

with

fill

ers

from

the

even

ts d

escr

ibed

by

test

sen

tenc

esan

d ex

amin

ed th

e ro

les

assi

gned

to th

ese

fille

rs, T

he u

se o

f bo

th s

ynta

ctic

and

sem

antic

con

stra

ints

was

exa

min

ed, T

hus,

for

a s

ente

nce

like

" The

scho

olgi

rl s

tirre

d th

e ko

ol-a

id w

ith th

e sp

oon

, sem

antic

con

stra

ints

mus

the used to determine th

at th

e sp

oon

is a

n in

stru

men

t, an

d no

t, fo

rex

ampl

e. a

n ac

com

pani

st o

f th

e sc

hool

girl

(d,

"T

he s

choo

lgir

l stir

red

the

kool

-aid

with

the

teac

her

), I

n ot

her

sent

ence

s, s

ynta

ctic

con

stra

ints

wer

eex

amin

ed, T

hus

, for

the

sent

ence

"T

he b

us d

rive

r w

as g

iven

the

rose

by

the

teac

her

, the

ord

er o

f th

e co

nstit

uent

s, to

geth

er w

ith th

e pr

esen

ce o

f th

epa

ssiv

e m

arke

r an

d th

e pr

epos

ition

", a

re n

eces

sary

to d

eter

min

e th

ecorrect role assignments of "

bus

driv

er" and "

teac

her

, bec

ause

eith

erco

uld

play

the

role

of a

gent

or

reci

pien

t. In

test

s in

volv

ing

five

sen

tenc

esof each of four types (a

ctiv

e. p

assi

ve, c

ross

ed w

ith a

nee

d to

rel

y se

man

tic o

r sy

ntac

tic c

onst

rain

ts),

all

of th

e fi

llers

wer

e assigned to the

corr

ect r

oles

, The

top

part

of Fig, 3 illustrates a passive syntactic role

assi

gnm

ent c

ase,

Exa

mpl

es il

lust

ratin

g th

e ot

her

kind

s of

cas

es m

ay

foun

d in

St.

John

and

McC

lella

nd (

in p

ress

),Fo

r th

e sp

ecif

icat

ion

fille

rs, t

hree

dis

tinct

var

iant

s w

ere

cons

ider

ed: T

hefir

st is

the

stra

ight

forw

ard

reso

lutio

n of

wor

d am

bigu

ity, i

n w

hich

the

netw

ork

is a

sked

sim

ply

to c

hoos

e be

twee

n tw

o al

tern

ativ

e an

d qu

itedi

stin

ct in

terp

reta

tions

of

the

fille

rs o

f on

e or

mor

e ro

les,

For

exa

mpl

e, in

The

pitc

her

hit t

he b

at w

ith th

e ba

t, t

he s

ubje

ct, o

bjec

t, an

d pr

epos

itio-

nal p

hras

e ob

ject

are

all

ambi

guou

s w

ords

in th

e co

rpus

, but

eac

h is

suffi

cien

tly c

onst

rain

ed b

y th

e co

ntex

t , g

iven

the

trai

ning

exp

erie

nce

of th

em

odel

. to

yiel

d a

uniq

ue in

terp

reta

tion,

The

mid

dle

port

ion

of F

ig, 3

illus

trat

es w

hat h

appe

ns in

this

cas

e, W

hen

we

prob

e fo

r th

e ag

ent ,

the

mod

el a

ctiv

ates

the

conc

ept u

nit c

orre

spon

ding

to th

e ba

seba

ll-pl

ayin

gpi

tche

r; w

hen

we

prob

e fo

r th

e pa

tient

, it a

ctiv

ates

the

conc

ept u

nit

corr

c~on

ding

to th

e fly

ing

bat;

and

whe

n w

e pr

obe

for

the

inst

rum

ent ,

itac

tivat

es th

e ba

seba

ll ba

t, T

his

occu

rs b

ecau

se, i

n tr

aini

ng, p

eopl

e bu

t not

pitc

hers

occ

ur a

s ag

ents

of

hitti

ng, f

lyin

g ba

ts b

ut n

ot b

aseb

all b

ats

occu

r as

patie

nts

of h

ittin

g, a

nd b

aseb

all b

ats

but n

ot fl

ying

bat

s oc

cur

as in

stru

-m

ents

of

hitti

ng,

The

sec

ond

vari

ant i

s co

ncep

t ins

tant

iatio

n. T

he s

ente

nce

" The

teac

her

kiss

ed s

omeo

ne" illustrates a particularly interesting case, b

ecau

se th

eso

meo

ne c

anno

t be

reso

lved

uni

quel

y gi

ven

the

cont

ext b

ut c

an b

e re

solv

ed

The

bul

mer

- rea

tile

I'G8e

by

the

tach

er.

00000001 01000000 00100000 10000000

! ,

" ' I J

l.i j.

l.I

" , I ii,

" j, 8

,

" ' I il,

8 ,

" ,

ii-"

~ I.

j' ,"

'~ I. .s.l. ,: ~ I. j.l. I.' 1; I. .s

' ,:

82: ,

8 ,

S ,

budm

er_p

earw

e(II

O8D

)to

cher

The

' pitc

her

lilt t

ile b

IIt w

ItI8

tile

b.L

101100001 OOiO DIDiO olooi

C':"

'.JIJ

I b

t:--

!l

:::--

!!'1

iI""

'

:::- '"

""""

"=

..~'

. ~~

~-

i!i-e B

:ll .

' 1 :s

-gll ~

-g",

!lj~

~! ~

~ J

\II t:

,;,

:e, ,;,

'"

..e !:

... .

';;(

i:j -

5 "'

lI "'

lI

.ert

.8p

Dt

patle

atID

Itnu

nent

The

tach

er k

illed

--.

I~H

liOO

o~o

li~;l

fit

'-'...

,.dea

t

FIG, 3 Activations of relevant output units in response to the indicated probes af

ter

pres

enta

tion

of th

e se

nten

ces

show

n,

part

ially

, In

the

expe

rien

ce o

f th

e ne

twor

k, t

he te

ache

r is

a fe

mal

e, a

nd th

eev

ent g

ener

ator

is c

onst

rain

ed s

o th

at k

issi

ng is

alw

ays

a he

tero

sexu

alac

tivity

; but

the

teac

her

is ju

st a

s lik

ely

to k

iss

the

pitc

her

(a c

hild

) or

the

bus

driv

er (

an a

dult)

, Thu

s w

e w

ould

exp

ect t

he m

odel

to b

e ab

le to

iden

tify

the

som

eone

as

a m

ale

but n

ot to

det

erm

ine

his

age

or w

heth

ersp

ecifi

cally

it w

as th

e pi

tche

r or

the

bus

driv

er, I

n th

e bo

ttom

pan

el o

f Fig

,. t

he o

utpu

t pro

duce

d in

res

pons

e to

a p

robe

for

the patient in "

the

teac

her

kiss

ed s

omeo

ne"

is s

how

n, w

here

the

cont

ext p

artia

lly s

peci

fies

the

5130

8M

cCLE

LLA

ND

. ST

, JO

HN

, TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

309

The adult ate the lieU with daintiness.

Sea

t- G

al"l

A.:I

I\II

IdD

DI

RoI

e/F

Dle

r A

.:II\I

IIlIo

Dl

uaJI

I:J

I:J

asea

lI:

JI8

JI8

JK

:Jpe

non

adul

tI:

Jch

ildm

ale

K:J

fem

ale

I:J

I:J

IKJ

buld

rive

rK

:Jle

ache

rK

:JI:

JIK

Jac

tion

I:J

I:J

I:J

I:J

ale

I:J

shot

I:J

I:J

I:J

I:J

drov

c~tr

ans.

) I:J

drov

e m

otiv

,)1:

Jpa

tient

peno

nK

:JI8

Jad

ult

I:J

IKJ

child

c:J

bwdr

iver

I:J

c:J

c:J

ICho

ollli

rlII

:JI:

JIK

JK

:J!b

ing

K:J

food

I:J

c:J

I:J

IKJ

stea

kc:

JI:

JIK

JI:

Jc:

:Jc:

JIO

IIp

I:J

I:J

c:J

c:J

c:J

c:J

crac

lcen

c:J

I:J

c:J

I:J

amrl

lc:

JpI

GK

:JIK

J

I:J

K:J

I:J

pleU

ll'e

:za

dain

linca

IKJ

FIG

.A

ctiv

atio

n of

a s

ubse

t of

the

sent

ence

ges

talt

units

(on

the

left

) an

d of

rel

evan

t out

put

units

in r

espo

nse

to th

e in

dica

ted

prob

es (

on th

e ri

ght)

aft

er p

rese

ntat

ion

of e

ach

cons

titue

nlof

the

sent

ence

"T

he a

dult

ate

the

stea

k w

ith d

ainl

ines

s, T

he c

olum

ns la

belle

d #

I. #2

, elC

..

refer to the state after eaeh of the successive co

nstit

uent

s (#

1 =

adu

lt. e

tc.

fille

r. H

ere

we

can

see

that

the

sex,

but

not

the

age,

is c

lear

ly s

peci

fied

,(T

here

app

ears

to b

e a

slig

ht p

refe

renc

e fo

r th

e pi

tche

r ov

er th

e bu

s dr

iver

.T

hese

pre

fere

nces

oft

en r

efle

ct th

e ef

fect

s of

spe

cifi

c tr

aini

ng tr

ials

that

occu

rred

just

pri

or to

test

ing.

The

thir

d ty

pe in

volv

es w

hat m

ight

be

calle

d "i

nfer

ence

of i

mpl

icit

argu

men

ts, b

ecau

se in

this

cas

e th

e se

nten

ces

cont

aine

d no

ove

rt in

dica

-tio

n ev

en th

at th

ere

was

a fi

ller

of a

par

ticul

ar r

ole.

For

exa

mpl

e, i

n " T

heteacher ate the soup

, the

re is

no

instrument mentioned; but during

trai

ning

, the

eat

ing

of s

oup

alw

ays

occu

rred

with

the

use

of a

spo

on, a

nd s

oth

e sp

oon

is in

ferr

able

in th

is c

onte

xt. I

n th

is c

ase

(not

sho

wn)

the

mod

elle

arns

to f

ill in

spo

on w

hen

prob

ed f

or th

e in

stru

men

t.T

he m

odel

was

test

ed w

ith f

ive

diff

eren

t exa

mpl

e se

nten

ces

for

each

of

thes

e th

ree

type

s of

fill

er specification cases, In all cases it pe

rfor

med

corr

ectly

(se

e 5t

. Joh

n an

d M

cCle

lland

, in press, f

or f

urth

er d

etai

ls a

ndex

ampl

es).

HO

W D

OE

S T

HE

MO

DE

L W

OR

K?

In this section

, we

begi

n by

fol

low

ing

the

time-

cour

se o

f pr

oces

sing

one

exam

ple

sent

ence

, to

give

the

read

er a

feel

ing

for

the

step

-by-

step

pro

ces-

sing

act

ivity

that

occ

urs

in th

e m

odel

. We

then

ret

urn

to th

e qu

estio

nsra

ised

at t

he b

egin

ning

of t

his

pape

r, to

see

how

the

mod

el g

ives

ver

ydi

ffer

ent a

nsw

ers

to e

ach

of th

ese

ques

tions

,T

he s

ente

nce

we

shal

l stu

dy is

"T

he a

dult

ate

the

stea

k w

ith d

aint

ines

sT

he s

ente

nce

is in

tere

stin

g, in

that

ther

e ar

e th

ree

diff

eren

t sources of

information as to the identity of the subject. One of these is the word

adul

titself. The second is the fact that the adult is eating

stea

k,

And

the

thir

d is

the adverb (with

dain

tines

s);

in th

e m

odel

' s e

xper

ienc

e it

is o

nly

the

teac

her

(a fe

mal

e) w

ho e

ver

eats

with

dai

ntin

ess.

As

we

shal

l see

, the

exa

mpl

eill

ustr

ates

the

mod

el's

abi

lity

to m

ake

use

of a

var

iety

of c

ues

of v

aryi

ngst

reng

th, s

prea

d th

roug

hout

the

sent

ence

, to

iden

tify

a pa

rtic

ular

con

sti-

tuen

t.After the pr

esen

tatio

n of

eac

h co

nstit

uent

(ad

ult ,

ate, steak, with

dain

tines

s), w

e ca

n examine the response of the network to probes

asse

ssin

g th

e fi

llers

of

the

agen

t , a

ctio

n, in

stru

men

t, an

d pa

tient

rol

es (

see

. Fig, 4). Later, we will return to co

nsid

er th

e pa

ttern

of

actic

atio

n ov

er th

eSG

uni

ts, w

hich

pro

vide

s th

e re

pres

enta

tion

of th

e w

hole

sen

tenc

e,W

e co

nsid

er f

irst

the

resp

onse

to "

agen

t", b

ecau

se it

is h

ere

that

we

see

the

effe

cts

of s

ever

al c

onst

ituen

ts o

pera

ting

mos

t cle

arly

, Aft

er th

e pr

esen

-tation of "ad

ult"

, the

mod

el ta

kes

the

agen

t to

be a

n ad

ult p

erso

n; th

ere

isso

me

activ

atio

n of

bot

h m

ale

and

fem

ale,

and

of

both

bus

dri

ver

and

teac

her,

the

only

two

adul

ts in

the

set,

The

re is

a s

light

bia

s fa

vour

ing

mal

e.C

hild

is in

clud

ed to

illu

stra

te th

at it

is n

ot a

ctiv

e at

any

poi

nt. T

here

is li

ttle

chan

ge a

fter

the presentation of the verb

, bec

ause

this

doe

s no

t rea

lly

prov

ide

any

cons

trai

nts

on th

e id

entit

y of

the

adul

t (th

e te

ache

r an

d th

e bu

sdriver appear equally often in sentences involving ea

ting)

, The

pre

sent

a-

tion

of "

stea

k", h

owev

er, p

rodu

ces

a sh

ift i

n th

e di

rect

ion

of m

ale

and

bus

driv

er. T

his

shif

t is

reve

rsed

(tho

ugh

not c

ompl

etel

y) w

hen

the

final

constitutent, "w

ith d

aint

ines

s, i

s pr

esen

ted.

For

the

othe

r ro

les,

the

read

er w

ill n

ote

that

the

mod

el p

erfo

rms

in a

gene

rally

sen

sibl

e w

ay, T

he o

ne s

light

pro

blem

app

ears

in th

e ca

se o

f th

epa

tient

. We

see

the

activ

atio

n of

"st

eak"

, whi

ch w

as q

uite

str

ong

just

afte

r

the presentation of the steak co

nstit

uent

, wea

ken

cons

ider

ably

whe

n

dain

tines

s" is

pre

sent

ed, W

e w

ill r

etur

n to

a c

onsi

dera

tion

of th

is s

peci

fic

aspe

ct o

f th

e m

odel

's p

erfo

rman

ce la

ter.

Giv

en th

e ov

eral

l suc

cess

of

the

mod

el, l

et u

s no

w a

sk w

hat k

inds

of

answ

ers

do w

e ge

t to

the

ques

tions

raised at the begining of this paper

whe

n w

e us

e a

mod

el o

f th

is s

ort?

5131

0M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

311

Wha

t is

Con

stru

cted

whe

n W

e C

ompr

ehen

d a

Sent

ence

?V

erb

Sim

ilarit

y

:!:

In th

is c

ase,

the

answ

er is

not

"a structural description

, Wha

t is

con-

stru

cted

is a

pat

tern

of

activ

atio

n w

hich

per

mits

the

perf

orm

ance

of a

spec

ific

task

or

task

s, In

this

cas

e, th

e ta

sk is

to p

rovi

de a

bas

is fo

rco

mpl

etin

g ro

le- f

iller

pai

rs; b

ut o

ne c

an im

agin

e a

wid

e va

riet

y of

oth

erus

es a

s w

ell.

Wha

teve

r th

e ta

sks

wer

e th

at w

e w

ere

calle

d up

on to

use

the

resu

lts o

f co

mpr

ehen

sion

to p

erfo

rm, a

mod

el w

ith th

e ge

nera

l str

uctu

re o

fth

e on

e us

ed h

ere

coul

d be

use

d to

lear

n to

per

form

that

task

,G

iven

this

. it b

ecom

es a

mat

ter

of e

mpi

rici

al r

esea

rch

to a

scer

tain

just

how

a n

etw

ork

will

cho

ose

to u

se it

s un

its in

lear

ning

to p

erfo

rm th

e ta

sks

that

it is

giv

en to

per

form

, We

know

fro

m o

ther

con

nect

ioni

st r

esea

rch

that

the

answ

ers

to th

ese

ques

tions

are

dep

ende

nt b

oth

on th

e sp

ecifi

c ta

sks

the

netw

ork

is a

sked

to p

erfo

rm, a

nd o

n th

e de

tails

of n

etw

ork

arch

itect

ure

(Hinton. 19R6: McClelland, in press), In this instance

, jus

t per

usin

g th

epa

ttern

of

activ

atio

n in

the

sent

ence

ges

talt

at e

ach

succ

essi

ve p

rese

ntat

ion

of a

new

inpu

t con

stitu

ent,

we

can

see

two

thin

gs, F

irst

, man

y of

the

units

take on graded ac

tivat

ions

, and

sev

eral

of t

hese

see

m o

nly

part

ially

corr

elat

ed w

ith p

artic

ular

rol

e-fi

ller

activ

atio

ns, T

his suggests that the

activ

atio

ns o

f pa

rtic

ular

out

put u

nits

in r

espo

nse

to p

artic

ular

pro

bes

are

gene

rally

det

erm

ined

by

the

join

t inf

luen

ce o

f a n

umbe

r of

hid

den

units

:th

us th

ey p

rovi

de a

dis

trib

uted

. coa

rse-

code

d re

pres

enta

tion

of th

e ro

le-

fille

r in

form

atio

n co

nvey

ed b

y th

e se

nten

ce (

d, Hinton, McClelland

. &

Rum

elha

rt, 1

9R6)

,

w 0

~ ~

is.

.;;

Wha

t Rol

e do

Wor

ds P

lay

in th

e C

ompr

ehen

sion

Proc

ess?

In the present model. as each word is presented

. it c

hang

es th

e pa

ttern

of

activ

atio

n in

the

sent

ence

ges

talt,

In

this

cas

e, w

e se

e ea

ch w

ord

as e

xert

ing

cons

trai

nts

on th

e re

pres

enta

tion,

It w

ill b

e no

ted

that

thes

e co

nstr

aint

s ca

nin

gen

eral

infl

uenc

e th

e re

spon

ses

to a

ll of

the

prob

es w

e m

ight

pre

sent

afte

r pr

esen

tatio

n of

a w

ord,

Thu

s th

e pr

esen

tatio

n of

"at

e " affects not

only

res

pons

es to

pro

bes

for

the

actio

n bu

t als

o pr

obes

for

the

patie

nt; a

ndth

e pr

esen

tatio

n of

ste

ak a

nd d

aint

ines

s ea

ch in

flue

nce

resp

onse

s to

pro

bes

for the agent, the patient, a

nd th

e m

anne

r. T

hus

a w

ord

is a

clu

e th

atco

nstr

ains

the

inte

rpre

tatio

n of

the

even

t as

a w

hole

,T

he in

flue

nce

that

a p

artic

ular

wor

d w

ill h

ave

on th

e co

mpr

ehen

sion

proc

ess

depe

nds

on w

hat h

as a

lrea

dy b

een

pres

ente

d, B

ut, t

here

is a

syst

emat

ic c

ontr

ibut

ion

that

eac

h w

ord

mak

es. T

his

syst

emat

ic c

ontr

ibu-

tion

is r

epre

sent

ed b

y th

e se

t of

conn

ectio

n st

reng

ths

from

the

inpu

t uni

tth

at r

epre

sent

s a

part

icul

ar w

ord

to th

e se

t of

hidd

en u

nits

insi

de th

eco

mpr

ehen

sion

par

t of t

he n

etw

ork,

fti

FIG, 5 Cluster an

alys

is o

f th

e w

eigh

t vec

tors

em

anat

ing

from

eac

h w

ord

inpu

t uni

t II'

Ihe

hidd

en u

nits

in th

e co

mpr

ehen

sion

par

t of

the

SG m

odel

. for

the

units

, rep

rese

ntin

g th

e II

unam

bigu

ous

verb

s sh

own,

The

ver

tical

pos

ition

of

the

hori

zont

al b

ar jo

inin

g tw

o br

anch

esin

dica

tes

the

sim

ilari

ty o

f th

e le

aves

or

bran

ches

join

ed,

To

exam

ine

thes

e co

ntri

butio

ns, S

t, Jo

hn a

nd M

cCle

lland

ext

ract

ed th

eve

ctor

of

conn

ectio

n w

eigh

ts e

man

atin

g fr

om e

ach

wor

d in

put u

nit t

o th

isfi

rst l

ayer

of

hidd

en u

nits

, The

se f

eatu

re v

ecto

rs w

ere

then

ent

ered

into

ahi

erar

chic

al c

lust

er a

naly

sis

(sep

arat

e an

alys

es w

ere

perf

orm

ed f

or th

eno

uns

and

verb

s), T

he a

naly

sis

for

the

verb

s (F

ig, 5

) di

spla

ys c

lear

ly th

atth

e m

odel

has

cap

ture

d th

e si

mila

rity

stru

ctur

e am

ong

the

"fra

mes

repr

esen

ted

by th

ese

verb

s as

use

d in

our

trai

ning

cor

pus,

The

ver

b "g

ivc

is th

e on

ly d

ativ

e ve

rb in

the

corp

us, a

nd is

clu

ster

ed s

epar

atel

y fr

om a

ll th

eot

hers

, The

ver

bs "

ate

, "

dran

k", a

nd "

cons

umed

" al

l tak

e an

imat

e th

ings

as s

ubje

cts

and

inan

imat

e th

ings

(fo

od)

as th

eir

obje

cts;

the

verb

s "s

tirrc

d"and "sp

read

" ea

ch ta

ke a

hum

an s

ubje

ct, f

ood

as a

n ob

ject

, and

a s

poon

or

5131

2M

cCLE

LLA

ND

. ST

. JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

313

Nou

n S

imila

rity

appe

ars

to b

e pi

ckin

g up

wha

t mig

ht b

e ca

lled

the

fram

es th

at th

e no

uns

ente

r in

to, r

athe

r th

an th

eir

indi

vidu

al m

eani

ngs

per se,

Of

cour

se, t

he

deta

ils o

f thi

s de

pend

on

the

part

icul

ar tr

aini

ng c

orpu

s; in

ord

inar

y lif

e,m

uch

happ

ens

in p

arks

bes

ides

the

eatin

g of

ice-

crea

m, I

n ge

nera

l, it

seem

slik

ely

that

nou

n-fr

ames

are

muc

h w

eake

r th

an v

erb-

frames; but to the

exte

nt th

at s

uch

fram

es d

o ex

ist,

they

can

be

capt

ured

by

mod

els

such

as

this

.

c ~

" .2

How

doe

s th

e P

roce

ss o

f Con

stru

ctin

g a

Rep

rese

ntat

ion

of a

Sen

tenc

e O

ccur

?

In th

e co

nnec

tioni

st m

odel

, the

re is

no

sepa

ratio

n of

the

stru

ctur

e-se

nsiti

ve

rule

s an

d th

e le

xica

l con

tent

of

wor

ds, T

he p

roce

ss is

inhe

rent

ly s

usce

ptib

leto

gui

danc

e by

con

tent

as

wel

l as

stru

ctur

al in

form

atio

n,In

som

e se

nse,

the

mod

el r

epre

sent

s th

e st

rong

est p

ossi

ble

alte

rnat

ive

toa

mod

ular

app

roac

h. N

ot o

nly

are

all d

iffe

rent

sou

rces

of

cons

trai

nt ta

ken

into account simultaneously. the kn

owle

dge

unde

rlyin

g ea

ch s

ourc

e of

cons

trai

nt is

inex

tric

ably

inte

rwov

en in

the

conn

ectio

ns,

The

re a

re li

mits

on

the

kind

s of

pro

cess

ing

ambi

guiti

es w

hich

our

mod

el

actu

ally

dea

ls w

ith, a

nd o

n th

e ki

nds

of in

form

atio

n th

at it

bri

ngs

to b

ear

inde

alin

g w

ith th

em, T

he fi

rst o

f the

se li

mits

is d

ue p

rimar

ily to

the

sim

plic

ityof

the

task

we

have

ask

ed th

e ne

twor

k to

per

form

; we

can

stud

y ro

leam

bigu

ities

. bec

ause

we

can

prob

e fo

r th

e fi

llers

of

part

icul

ar r

oles

, but

we

cann

ot s

tudy

atta

chm

ent a

mbi

guiti

es, b

ecau

se th

e ro

le-f

iller

com

plet

ion

task

is to

o lim

ited.

If

the language of the probe completion task were

enri

ched

, how

ever

, it s

houl

d be

pos

sibl

e fo

r th

e ne

twor

k to

lear

n to

dea

lw

ith m

ore

inte

rest

ing

stru

ctur

al a

mbi

guiti

es a

s w

ell,

To

mak

e th

is c

oncr

ete,

supp

ose

we

had

trai

ned

the

netw

ork

to c

ompl

ete

head

-rol

e-fi

ller

trip

les,

rather than simply role-

fille

r pa

irs,

For

exa

mpl

e, c

onsi

der

agai

n se

nten

ces

like

(3)

and

(4):

i!;- Uj C

IO

i!;-

...

FIG

.C

lust

er a

naly

sis

of th

e w

eigh

t vec

tors

em

anat

ing

from

eac

h w

ord

inpu

t uni

t to

the

hidd

en u

nits

in th

e co

mpr

ehen

sion

par

t of

the

SG m

odel

, for

the

units

representing the

unam

bigu

ous

noun

s sh

own.

The

ver

tical

pos

ition

of

the

hori

zont

al b

ar jo

inin

g tw

o br

anch

esin

dica

tes

the

sim

ilari

ty o

f th

e le

aves

or

bran

ches

join

ed,

3. T

he s

py s

aw th

e po

licem

an w

ith th

e re

volv

er.

4, T

he s

py s

aw th

e po

licem

an w

ith th

e bi

nocu

lars

,

Supp

ose

that

the

prob

es s

peci

fied

a h

ead

noun

or

verb

, and

a r

ole,

and

thc

task

was

to f

ill in

the

fille

r, I

n th

e ca

se o

f Y

, we

wou

ld w

ant t

he n

etw

ork

tofi

ll in

the

conc

ept c

orre

spon

ding

to r

evol

ver

whe

n pr

obed

for

the

mod

ifie

rof

po

licem

an,

but not when probed for the instrument of

saw

. In

the

case

of

(4), we would want the network to fill in

bino

cula

rs

whe

n pr

obed

for

the

instrument of saw, but not when probed for the modifier of

polic

eman

,

prin

cipl

e, w

e se

e no

rea

son

why

som

e ve

rsio

n of

this

app

roac

h sh

ould

not

be a

ble

to e

xten

d to

mul

ticla

use

sent

ence

s, th

ough

cle

arly

this

is a

mat

ter

for

furt

her

rese

arch

, A

com

plet

e m

odel

wou

ld, o

f co

urse

, als

o pr

ovid

e so

me

repr

esen

tatio

n of

a knife as the instrument; and "

hit

, "

kick

ed", a

nd "

kiss

ed"

are

all

pass

ivis

ible

in th

e co

rpus

(un

like

the

food

-rel

ated

ver

bs),

and

all

invo

lve

apa

tient

that

may

be

anim

ate.

The

ana

lysi

s fo

r th

e no

uns

(Fig

, 6)

is le

ss c

lear

; it a

ppea

rs th

at th

ere

are

two

orga

nisa

tiona

l prin

cipl

es th

at a

re b

oth

at w

ork,

Sometimes, nouns

clus

ter

by m

eani

ng. T

hus

all t

he h

uman

nou

ns c

lust

er s

epar

atel

y fr

om th

ere

st o

f the

nou

ns, H

owev

er, a

t a fi

ner

grai

n, t

he n

ouns

som

etim

es a

ppea

rto cluster by co-occurrence in the sa

me

even

ts. T

hus

ice-

crea

m c

lust

ers

with

par

k be

caus

e in

our

cor

pus

ice-

crea

m is

eat

en in

the

park

, and

that

isthe only thing that ever happens in the park, O

nce

agai

n, th

e m

odel

5131

4M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

315

cont

ext p

rior

to th

e be

ginn

ing

of th

e cu

rren

t sen

tenc

e, A

s th

e w

ork

ofC

rain

and

Ste

edm

an (

1985

) an

d A

ltman

n an

d St

eedm

an (

1988

) m

akes

clea

r , c

onst

rain

ts a

risin

g fr

om th

e te

nden

cy f

or m

ultis

ente

nce

text

s to

mai

ntai

n re

fere

ntia

l coh

eren

ce c

an in

flue

nce

atta

chm

ent a

mbi

guity

res

olu-

tion,

We

wou

ld e

xpec

t tha

t a m

odel

that

som

ehow

kep

t a r

ecor

d of

pri

orco

ntex

t cou

ld le

arn

to e

xplo

it it

for

the

reso

lutio

n of

atta

chm

ent a

mbi

-gu

ities

.O

f co

urse

. wha

t we

have

just

des

crib

ed a

re h

oped

-for

ext

ensi

ons ,

not

real

res

ults

at t

his

poin

t. So

me

rele

vant

res

earc

h, i

ndic

atin

g an

abi

lity

tous

e co

ntex

t to

reso

lve

ambi

guiti

es o

f ref

eren

ce in

mul

tisen

tenc

e te

xts,

has

been

car

ried

out

(se

e, e

, g, A

llen,

198

7; 1

988;

Miik

kula

inen

& D

yer,

198

9),

but t

o ou

r kn

owle

dge

noth

ing

rele

vant

has

yet

bee

n do

ne w

ith r

egar

d to

the

resolution of structural ambiguity,

proc

ess,

Tha

t pro

cess

ass

igns

str

engt

hs to

the

feat

ures

so

that

the

corr

ect

inte

rpre

tatio

ns a

re a

chie

ved

acro

ss th

e en

tire

corp

us,

CA

N T

HE

PO

P A

PPR

OA

CH

SO

LV

E T

HE

PR

OB

LE

MS

WIT

H C

ON

VE

NT

ION

AL

MO

DE

LS?

Con

text

ual S

hadi

ng a

s w

ell a

s S

elec

tion

of W

ord

Mea

ning

Thi

s ch

arac

teri

stic

of

POP

mod

els

is n

ot il

lust

rate

d so

cle

arly

by

the

pres

ent

mod

el b

ecau

se o

f its

use

of

loca

l rep

rese

ntat

ions

for

con

cept

s, W

e ca

n se

eth

is k

ind

of th

ing

to a

lim

ited

degr

ee in

suc

h ex

ampl

es a

s " T

he a

dult

ate

the

steak with daintiness

. Tho

ugh

" tea

cher" and "

fem

ale" are ultimately

mor

e ac

tive

than

"hu

s dr

iver

" an

d "m

ale

, the

fac

t tha

t it i

s a

stea

k th

at is

eate

n de

fini

tely

sha

des

the

activ

atio

ns in

the

netw

ork

with

mal

enes

s; th

em

odel

see

ms

only

too

natu

ral i

n its

abi

lity

to c

aptu

re s

tere

otyp

es li

ke th

eon

e im

mor

talis

ed in

the

phra

se

, "

real

men

don

t eat

qui

che

, and

to u

sein

nuen

do in

sha

ding

its

repr

esen

tatio

ns.

The

use

of

loca

l rep

rese

ntat

ions

for

con

cept

s m

akes

it p

ossi

ble

to s

eeco

ntex

tual

sha

ding

onl

y in

the

rela

tive

degr

ee o

f ac

tivat

ion

of th

e fe

wsu

pero

rdin

ate

feat

ure

units

that

wer

e in

clud

ed in

the

mod

el. H

owev

er, t

his

use of local representations is not inherent in the connectionist ap

proa

chan

d w

e ad

opte

d th

is u

sage

her

e on

ly f

or e

ase

of te

stin

g an

d to

avo

idbu

ildin

g un

due

amou

nts

of k

now

ledg

e in

to th

e co

ncep

t rep

rese

ntat

ions

.H

owev

er, a

n ea

rlie

r m

odel

that

did

use

dis

trib

uted

rep

rese

ntat

ions

doe

sill

ustr

ate

shad

ing

effe

cts

on a

gra

nder

sca

le (

McC

lella

nd &

Kaw

amot

o.1986), In that model

, con

cept

s w

ere

repr

esen

ted

by fu

lly d

istr

ibut

edpa

ttern

s, T

he m

odel

was

trai

ned

to in

terp

ret a

var

iety

of s

entences involv-

ing

brea

king

one

obj

ect w

ith a

noth

er, a

nd a

ll bu

t one

of

the

obje

cts

that

coul

d oc

cur

as th

e in

stru

men

t sha

red

a fe

atur

e in

dica

ting

that

the

obje

ctw

as h

ard,

The

one

exc

eptio

n, t

he b

all,

was

enc

oded

as

soft

, and

the

mod

elco

rrec

tly tr

eate

d it

as s

uch

whe

n it

occu

rred

in m

ost c

onte

xts.

How

ever

,w

hen

it w

as u

sed

to b

reak

oth

er o

bjec

ts, t

he m

odel

sha

ded

the

repr

esen

ta-

tion

, giv

ing-

it th

e fe

atur

e ha

rd in

stea

d of

sof

t; th

is h

appe

ned

just

bec

ause

thin

gs th

at b

reak

oth

er th

ings

wer

e ty

pica

lly h

ard,

and

the

mod

el b

ecam

ese

nsiti

ve to

this

fac

t, It

is w

orth

not

ing

that

the

resu

lting

pat

tern

was

not

one

of th

e ex

istin

g pa

ttern

s on

whi

ch th

e m

odel

had

bee

n tr

aine

d bu

t an

exte

nsio

n by

the

mod

el o

f th

e en

sem

ble

of p

ossi

ble

conc

epts

,

How

doe

s A

cqui

sitio

n W

ork?

Acq

uisi

tion

wor

ks b

y a

proc

ess

of g

radu

al c

onne

ctio

n st

reng

th a

djus

tmen

t.T

his

is q

uite

diff

eren

t fro

m th

e fo

rmul

atio

n of

a s

yste

m o

f ex

plic

it ru

les,

Cer

tain

pro

blem

s ar

e av

oide

d rig

ht fr

om th

e st

art,

such

as

the

ques

tion

ofw

hen

to f

orm

a r

ule,

and

whe

n to

sim

ply

list e

xcep

tions

. How

ever

, it w

ould

cert

ainl

y no

t be

accu

rate

to s

ugge

st th

at th

e m

odel

we

have

pre

sent

ed h

ere

is a

tabula rasa,

acqu

irin

g kn

owle

dge

of la

ngua

ge w

ithou

t any

pri

orst

ruct

ure,

Ind

eed

, the

inpu

t is

pars

ed f

or th

e m

odel

into

con

stitu

ents

and

wor

ds; a

nd th

e ro

le- f

iller

rep

rese

ntat

ion

of th

e ev

ent d

escr

iptio

ns a

nd th

ese

t of

conc

epts

use

d in

the

outp

ut n

etw

ork

are

pred

eter

min

ed a

s w

ell.

Fina

lly, t

he s

truc

ture

of

the

netw

ork

is p

reor

dain

ed, a

nd ta

ilore

d to

the

task

. The

se f

eatu

res

of th

e m

odel

wer

e no

t ado

pted

out

of

any

belie

f th

atth

eir

adop

tion

was

nec

essa

ry b

ut s

impl

y ou

t of

a de

sire

to e

stab

lish

a si

mpl

eill

ustr

ativ

e m

odel

. Jus

t how

muc

h pr

ior

stru

ctur

e ha

s to

be

built

in, a

nd in

wha

t way

it is

bui

lt in

, rem

ain

basi

c an

d ce

ntra

l iss

ues

for

conn

ectio

nist

mod

els

in th

is a

nd a

num

ber

of o

ther

dom

ains

.

Ear

lier

we

enum

erat

ed a

set

of

prob

lem

s w

ith c

onve

ntio

nal m

odel

s. H

ere

we

cons

ider

how

they

are

or

coul

d be

sol

ved

in m

odel

s of

the

kind

we

have

cons

ider

ed h

ere,

Con

cept

ual G

uida

nce

and

Rul

e C

onfli

cts

The

pro

blem

of

conc

eptu

al g

uida

nce

is n

atur

ally

sol

ved

by th

e in

tegr

ated

handling of both sy

ntac

tic a

nd c

onte

nt- based constraints on processing.

The

pro

blem

of

rule

con

flic

ts is

dea

lt w

ith b

y th

e co

nnec

tion

adju

stm

ent

The

Sim

ilar

Prob

lem

with

Rol

es

The

sha

ding

of c

once

pt r

epre

s.;n

tatio

ns th

at is

cap

ture

d in

McC

lella

nd a

ndK

awam

oto

s m

odel

has

bee

n ap

plie

d to

rol

es b

y T

oure

tzky

and

Gev

a(1

987)

. The

idea

is s

impl

y th

at th

e se

t of

poss

ible

rol

es is

not

som

e fi

xed

set

of

alte

rnat

ives

but

an

exte

nsib

le s

et w

ith a

ric

h si

mila

rity

str

uctu

re, s

uch

as is

nat

ural

ly c

aptu

red

by d

istr

ibut

ed r

epre

sent

atio

ns,

5131

6M

cCLE

LLA

ND

, ST

, JO

HN

, TA

RA

BA

N

SEN

TE

NC

E C

OM

PRE

HE

NSI

ON

AN

D P

DP

5131

7Im

plie

d C

onst

ituen

ts

The

han

dlin

g of

impl

ied

cons

titue

nts

is n

ot a

pro

blem

in th

e m

odel

. It i

squite natural for the m

odel

to le

arn

that

eve

nts involving eating steak

always involve a knife as the instrument. There is no special "in

fere

nce

step

" re

quir

ed to

fill

in th

e kn

ife, T

his

is in

par

t a d

irect

res

ult o

f the

fact

that

ther

e is

no

prio

r st

ipul

atio

n th

at a

par

ticul

ar p

art o

f th

e re

pres

enta

tion

of the sentence corresponds to the in

tern

al r

efle

x of

eac

h pa

rtic

ular

constituent of the sentence, It

's ju

st th

at e

vent

s de

scri

bed

by s

ente

nces

with "

ate

" as

the

verb

and

"steak" as the object always involve knives as

inst

rum

ents

, The

pro

babi

listic

nat

ure

of m

any

impl

ied

cons

titue

nts

is n

ot a

prob

lem

eith

er, b

ecau

se o

f th

e in

here

ntly

gra

ded

natu

re o

f th

e ac

tivat

ion

proc

ess ,

cou

pled

with

the

fact

that

inte

rmed

iate

act

ivat

ion

valu

es d

irect

lyre

flec

t pro

babi

litie

s in

term

edia

te b

etw

een

0 an

d I.

long

er m

akes

any

sen

se a

t all.

Thi

s oc

curs

whe

n on

e re

ache

s th

e w

ord

fell"

in a

sen

tenc

e lik

e th

is o

ne o

f B

ever

s (1

970)

:

21. T

he h

orse

rac

ed p

ast t

he b

arn

fell.

The

que

stio

n is

, why

is it

, on

the

stor

y th

at w

e ar

e te

lling

, tha

t sub

ject

s ar

eno

t at t

his

poin

t abl

e to

rec

over

, bas

ed o

n th

e co

nstr

aint

s on

the

inte

rpre

ta-

tion

impo

sed

by th

e fin

al w

ord?

The

ans

wer

is th

at th

e w

ord

" fel

l" b

y its

elf

may

not

exe

rt a

str

ong

enou

gh in

flue

nce

to r

eorg

anis

e th

e SO

rep

rese

ntat

ion

all o

n its

ow

n. T

his

argu

men

t im

plie

s th

at if

the sentence continues in a way that im

pose

sadditional input favouring the correct in

terp

reta

tion,

the

gard

en- p

ath

effe

ct m

ight

act

ually

be

over

com

e. T

hus

we

expe

ct th

at th

e ' gar

den-

path

effe

ct c

an b

e am

elio

rate

d if

the

read

er ju

st continues to read and the

sent

ence

goe

s on

and

pro

vide

s ad

ditio

nal r

elev

ant c

onst

rain

ts:

22. T

he h

orse

rac

ed p

ast t

he b

arn

fell

into

the

ditc

h.

Obviously, this is a topic for ca

refu

l res

earc

h, r

athe

r th

an th

e ja

ded

intu

ition

s of

theo

retic

ally

bia

sed

psyc

holin

guis

ts; b

ut w

e fe

el th

e ex

ampl

esu

ppor

ts o

ur s

ense

that

eve

n in

suc

h dr

amat

ic c

ases

as

(21)

, we

may

not

real

ly b

e fo

rced

to r

epro

cess

, as

long

as

ther

e is

sub

sequ

ent i

nfor

mat

ion

that allows us to ov

erco

me

the

effe

cts

of w

hat h

as c

ome

earl

ier

in th

ese

nten

ce.

Com

bina

tori

al E

xplo

sion

or

Prem

atur

eC

omm

itmen

tT

he m

odel

avo

ids

com

bina

tori

al e

xplo

sion

by

keep

ing

mul

tiple

alte

rnat

ives

impl

icit

in th

e si

ngle

pat

tern

of

activ

ity o

ver

the

sent

ence

ges

talt.

It a

void

sthe catastrophic side-ef

fect

s of

pre

mat

ure

com

mitm

ent b

ecau

se it

s gr

aded

activ

atio

ns c

an b

e ad

just

ed a

s ea

ch n

ew c

onst

rain

t is

intr

oduc

ed, I

n a

sens

e, i

t doe

s m

ake

com

mitm

ents

as

each

new

con

stitu

ent i

s en

coun

tere

dbu

t the

se a

rc n

ot a

ll-or

-non

e ch

oice

s, b

ut s

impl

y co

ntin

uous

shi

fts in

the

patte

rn o

f act

ivat

ion,

Thu

s co

mm

itmen

ts m

ade

can

be r

ever

sed

with

out

any

back

trac

king

, It i

s tr

ue th

at s

ome

cons

titue

nts

caus

e a

mor

e m

arke

dad

just

men

t of

the

SO r

epre

sent

atio

n th

an o

ther

s, T

hese

mar

ked

adju

st-

men

ts c

an b

e re

late

d to

exp

erim

enta

l dat

a on

rea

ding

tim

es if

we

mak

e th

esi

mpl

e as

sum

ptio

n th

at la

rger

adj

ustm

ents

take

long

er to

mak

e, T

his

assu

mpt

ion

hold

s in

sys

tem

s th

at a

djus

t the

ir a

ctiv

atio

ns c

ontin

uous

ly(McClelland, 1979) rather than in a si

ngle

tim

e-st

ep, W

e vi

ew th

ese

cont

inuo

us s

yste

ms

as m

ore

real

istic

that

the

disc

rete

tim

e-st

ep s

yste

m u

sed

here

; as

with

the

use

of lo

calis

t rep

rese

ntat

ion

, the

use

of

disc

rete

tim

e in

the

illus

trat

ive

exam

ple

mod

el is

sim

ply

a m

atte

r of

gre

ater

trac

tabi

lity,

The

not

ion

that

larg

er c

hang

es in

the

SO

are

ass

ocia

ted

with

long

erre

adin

g tim

es p

rovi

des

a na

tura

l way

of a

ccou

ntin

g fo

r a

lot o

f rea

ding

tim

eda

ta't.

'i' w

hich

a s

low

dow

n in

pro

cess

ing

is o

bser

ved

in o

ne c

ondi

tion

relative to another. In these cases, i

t see

ms unnecessary to invoke the

notio

n of

rep

roce

ssin

g, w

hich

is o

ften

ass

ocia

ted

with

theo

retic

al d

iscu

s-si

ons

of th

ese

effe

cts

in th

e ex

peri

men

tal l

itera

ture

col

oure

d by

the

clas

sica

lframework, But, it does sometimes ha

ppen

that

lang

uage

rea

ders

and

liste

ners

exp

erie

nce

a tr

ue g

arde

n pa

th. B

y a

true

gar

den

path

, we

mea

nth

e st

rong

feel

ing

that

something has gone !Iwry-that the sentence no

The

Pro

blem

of A

cqui

sitio

n

The

use

of g

radu

al c

onne

ctio

n ad

just

men

ts in

the

mod

el h

elps

it o

verc

ome

som

e of

the

prob

lem

s co

nven

tiona

l app

roac

hes

face

in le

arni

ng to

inte

rpre

tse

nten

ces.

Firs

t, th

e st

reng

ths

of c

onst

rain

ts im

pose

d by

var

ious

wor

ds o

nth

e in

terp

reta

tion

proc

ess

are

natu

rally

gra

ded

and

are

brou

ght g

radu

ally

into

bal

ance

by

the

conn

ectio

n ad

just

men

t pro

cess

. Sec

ondl

y, th

e so

lutio

nto

the

boot

stra

ppin

g pr

oble

m e

mer

ges

natu

rally

thro

ugh

the

expo

sure

of

the

mod

el to

the

stat

istic

al p

rope

rtie

s of

an

ense

mbl

e o.

sen

tenc

e-ev

ent

pair

s. h

is tr

ue th

at th

e se

nten

ce "

the

boy

kiss

ed th

e gi

rl"

coul

d m

ap o

n to

the

even

t of

a bo

y ki

ssin

g a

girl

in tw

o di

ffer

ent w

ays;

but

thes

e al

tern

ativ

esar

e fu

rthe

r co

nstr

aine

d by

oth

er s

ente

nces

. Thu

s in

eve

ry s

ente

nce

whe

reth

e su

bjec

t of

the

verb

"ki

ss"

is g

irl,

ther

e is

a g

irl i

n th

e ev

ent a

nd s

he is

the agent.

We

do n

ot w

ish

to s

ugge

st a

t all

that

the

prob

lem

s of

acq

uisi

tion

are

fully

solved by the present model; the sentences and events are highly sim-

plif

ied, and the pre-pa

rsin

g of

sen

tenc

es in

to w

ords

and

con

stitu

ents

,to

geth

er w

ith th

e pr

e-st

ruct

urin

g of

eve

nts

into

rol

e-fi

ller

pair

s ce

rtai

nly

mak

es th

ings

eas

ier

for

the

mod

el.

Our

onl

y cl

aim

is th

at th

e co

nnec

tioni

st le

arni

ng p

roce

dure

we

have

use

d

5131

8M

cCLE

LLA

ND

. ST

. JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

319

does

hav

e so

me

sign

fican

t adv

anta

ges

over

rul

e-le

arni

ng a

ppro

ache

s. A

sno

ted

abov

e, it

rem

ains

for

fur

ther

res

earc

h to

est

ablis

h ho

w m

uch

supp

ort

thes

e pr

oced

ures

req

uire

from

pre

-exi

stin

g st

ruct

ure

and

how

muc

h th

eycan induce from the environment.

Com

pari

son

with

Oth

er C

onne

ctio

nist

Mod

els

As

man

y re

ader

s w

ill d

oubt

less

ly b

e aw

are,

a n

umbe

r of

res

earc

hers

hav

eno

ted

the

appe

al o

f th

e PO

P or

con

nect

ioni

st m

odel

ling

fram

ewor

k fo

rca

ptur

ing

aspe

cts

of la

ngua

ge p

roce

ssin

g, a

nd h

ave

chos

en to

bui

ld m

odel

sw

ithin

this

fra

mew

ork,

Our

app

roac

h ce

rtai

nly

falls

wel

l with

in th

is tr

adi-

tion

, but

diff

ers

from

mos

t of t

he p

revi

ous

wor

k in

one

impo

rtan

t way

,E

arlie

r m

odel

s ha

ve te

nded

to in

corp

orat

e, in

one

way

or

anot

her ,

spe

cifi

cch

arac

teri

stic

s of

con

vent

iona

l mod

els

of la

ngua

ge d

irec

tly in

to th

e in

tern

alre

pres

enta

tions

use

d. O

ur e

arlie

r w

ork

is n

o ex

cept

ion

to th

is r

ule,

For

example. the in

tera

ctiv

e ac

tivat

ion

mod

el o

f vi

sual

wor

d re

cogn

ition

(McC

lella

nd &

Rum

elha

rt, 1

981;

Rum

elha

rt &

McC

lella

nd, 1

982)

inco

r-po

rate

d un

its th

at r

epre

sent

lette

rs a

nd w

ords

; the

TR

AC

E m

odel

of

spee

ch p

erce

ptio

n (E

lman

& M

cCle

lland

, 198

6; M

cCle

lland

& E

lman

1986

) in

corp

orat

es p

hone

mes

and

wor

ds, M

cCle

lland

and

Kaw

amot

o(1

986)

use

sem

antic

feat

ures

suc

h as

"an

imat

e

, "

inan

imat

e, e

tc"

and

gave

ove

r sl

ots

with

in th

e m

odel

to e

ach

of s

ever

al p

ossi

ble

repr

esen

tatio

-na

l rol

es, B

y ta

king

thes

e st

eps-

reif

ying

som

e aspects of conventional

repr

esen

tatio

ns in

side

con

nect

ioni

st n

etw

orks

- these and other models

have

in f

act f

alle

n prey to some of the di

fficu

lties

that

are

face

d by

conv

entio

nal m

odel

s. T

he m

ost s

erio

us p

robl

em is

the

prob

lem

of

com

-bi

nato

rial

exp

losi

on. U

nlik

e co

nven

tiona

l mod

els,

in w

hich

the

prob

lem

of

com

bina

tori

al e

xplo

sion

is o

nly

face

d du

ring

pro

cess

ing,

thes

e co

nnec

tion-

ist m

odel

s ha

ve to

fac

e it

in a

dvan

ce, a

nd c

onst

ruct

net

wor

ks o

f un

its a

ndco

nnec

tions

that

will

allo

w a

ll po

ssib

le in

puts

to b

e pr

oces

sed.

Tw

o ap

proa

ches

hav

e be

en ta

ken

to a

void

this

pro

blem

, The

fir

st is

toex

ploi

t the

idea

of

coar

se c

odin

g, in

whi

ch u

nits

do

not c

orre

spon

d to

spec

ific

rep

rese

ntat

iona

l ent

ities

, but

to c

onju

nctio

ns o

f pr

e-sp

ecif

ied

fea-

ture

s of

pos

sibl

e re

pres

enta

tiona

l ent

ities

, It i

s w

ell e

stab

lishe

d th

at th

is c

anlim

it th

e pr

oble

m in

som

e do

mai

ns, a

nd s

ome

mod

els

have

bee

n su

cces

sful

usin

g th

is a

ppro

ach

(c. f

, Tou

retz

ky &

Hin

ton,

198

8), b

ut w

e ha

ve f

ound

itun

wor

kabl

e fo

r se

nten

ce p

roce

ssin

g (S

t. Jo

hn &

McC

lella

nd, 1

987)

, The

prob

lem

is th

at th

e nu

mbe

r of

feat

ures

that

nee

d to

be

pre-

spec

ified

isun

boun

ded

, as

is th

e nu

mbe

r of

thes

e th

at m

ay n

eed

to b

e co

njoi

ned

tore

pres

ent a

ny p

artic

ular

spe

cifi

c pr

opos

ition

, Aga

in th

e co

mbi

nato

rics

can

be d

evas

tatin

g.T

he o

ther

app

roac

h-th

e on

e th

at is

take

n he

re- i

s to

let c

onne

ctio

nist

lear

ning

pro

cedu

res

do th

e w

ork

of d

eter

min

ing

wha

t the

inte

rnal

rep

re-

sent

atio

n sh

ould

be

like.

Of

cour

se, t

he a

ppro

ach

has

only

bec

ome

poss

ible

sinc

e th

e ad

vent

of

lear

ning

rul

es th

at a

llow

inte

rnal

rep

rese

ntat

ions

to b

efo

rmed

in r

espo

nse

to ta

sk d

eman

ds (

Rum

elha

rt, H

into

n, & Williams,

1986

a). T

his

appr

oach

has

one

ser

ious

dis

adva

ntag

e-le

arni

ng c

an b

e ve

rysl

ow. B

ut it

doe

s ha

ve s

ome

very

impo

rtan

t adv

anta

ges,

It a

llow

s th

ene

twor

k to

lear

n to

form

rep

rese

ntat

ions

that

mak

e ef

fici

ent u

se o

f th

eavailable representational resources, given the task as sp

ecif

ied

by th

ein

puts

that

are

pre

sent

ed a

nd th

e ou

tput

s th

at th

e ne

twor

k is

exp

ecte

d to

gene

rate

in r

espo

nse

to th

ese

inpu

ts, A

num

ber

of r

esea

rche

rs h

ave

been

expl

orin

g th

is a

ppro

ach,

Elm

an (1

989)

, for

exa

mpl

e, has developed a

sim

ple

netw

ork

that

lear

ns to

ant

icip

ate

the

next

ele

men

t of

a se

quen

ce o

fw

ords

, In

so d

oing

, it l

earn

s to

for

m a

n in

tern

al representation of each

wor

d th

at r

efle

cts

the

sequ

entia

l con

stra

ints

in th

e se

quen

ces.

If

the

wor

dsfo

rm s

ente

nces

, and

the

sent

ence

s re

flect

co-

occu

rren

ce c

onst

rain

ts-s

uch

as th

e fa

ct th

at a

cer

tain

ver

b re

quir

es a

n an

imat

e su

bjec

t and

an

inan

imat

eob

ject

- the

rep

rese

ntat

ions

com

e to

ref

lect

the

rele

vant

sem

antic

dis

tinc-

tions

,Ju

st h

ow f

ar th

is k

ind

of a

ppro

ach

can

be ta

ken

rem

ains

to b

e se

en.

Som

e (e

, g. T

oure

tzky

, 198

8) h

ave

sugg

este

d th

at m

ore

stru

ctur

e ne

eds

tobe

bui

lt in

to th

e ne

twor

k, a

nd th

e fa

cts

of le

arni

ng s

peed

do

sugg

est t

hat

som

ethi

ng m

ust b

e do

ne b

efor

e th

e ap

proa

ch c

an b

e su

cces

sful

ly e

xten

ded

to la

rge

corp

uses

of

com

plex

sen

tenc

es (

see

also

Miik

kula

inen

& D

yer

1989

). A

t pre

sent

, it i

s to

o ea

rly

to te

ll ju

st w

hat w

ill b

e th

e m

ost e

ffec

tive

way

to p

rovi

de f

or a

n ex

tens

ion

of th

e ap

proa

ch.

AR

GU

ME

NT

S A

GA

INST

TH

E P

OP

APP

RO

AC

H

Sev

eral

diff

eren

t typ

es o

f arg

umen

ts m

ight

be

give

n in

favo

ur o

f con

vent

io-

nal a

ppro

ache

s an

d ag

ains

t the

app

roac

h th

at w

e ha

ve ta

ken,

Her

e w

e co

n-si

der

thre

e th

at s

eem

par

ticul

arly

cen

tral

. In

all t

hree

cas

es, w

e be

lieve

that

the

argu

men

ts a

re le

ss c

ompe

lling

than

the

prop

onen

ts o

f al

tern

ativ

es h

ave

alle

ged.

.

Sys

tem

atic

ity a

nd P

rodu

ctiv

ity

In th

eir

criti

que

of c

onne

ctio

nist

mod

els,

Fod

or a

nd P

ylys

hyn (1988) point

out t

hat a

n in

here

nt fe

atur

e of

the

conv

entio

nal a

ppro

ach

is th

e fa

ct th

at it

acco

unts

for

the

syst

emat

icity

and

pro

duct

ivity

of

lang

uage

. The

se c

hara

c-te

rist

ics

follo

w d

irec

tly, t

hey

poin

t out

, if

we

assu

me

that

our cognitive

appa

ratu

s m

akes

use

of

a co

mbi

nato

rial

syn

tax

and

sem

antic

s; th

ey a

lso

clai

m th

at th

ese

char

acte

rist

ics

do n

ot f

ollo

w f

rom

the

POP

appr

oach

, LeI

us examine these characteristics,

5132

0M

cCLE

LLA

ND

, ST

, JO

HN

, TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

321

Systematicity refers to the fact th

at if

a s

peak

er can understand a

sent

ence

like

"Jo

hn lo

ves

the

girl

" an

d (l

et u

s say) "

Bill

dis

likes

the

teac

her

, the

n he

can

als

o un

ders

tand

oth

er s

ente

nces

, suc

h as

"Jo

hn lo

ves

the teacher

, "

Bill

dis

likes

Joh

n, e

tc, I

n ot

her

wor

ds, s

ente

nces

are

not

just isolated, un

anal

ysed

who

les

but a

re c

ompo

sed

of p

arts

whi

ch c

an b

ere

com

bine

d to

pro

duce

oth

er s

ente

nces

that

the

spea

ker

will

und

erst

and,

To

test

the

capa

bilit

y of

a m

odel

suc

h as

our

s to

exh

ibit

syst

emat

icity

, we

gene

rate

d a

new

cor

pus,

con

tain

ing

10 p

erso

ns a

nd 1

0 ac

tions

, Eac

h of

the

actio

ns c

ould

be

done

by

any

pers

on to

any

per

son

so th

at th

ere

wer

e a

total of lOOO possible events, Each co

uld

be e

xpre

ssed

in a

n ac

tive

orpa

ssiv

e se

nten

ce f

or a

tota

l of

2000

pos

sibl

e se

nten

ces,

We

trai

ned

the

sam

e ne

twor

k de

scri

bed

abov

e w

ith a

ll bu

t a r

ando

mly

chos

en 2

50 o

f th

e po

ssib

le s

ente

nces

; the

n, a

fter

trai

ning

, we

test

ed it

on

the

rem

aini

ng 2

50 s

ente

nces

, A s

trin

gent

acc

urac

y cr

iteri

on w

as a

dopt

ed:

A s

ente

nce

was

sco

red

corr

ect o

nly

if th

e un

it re

pres

entin

g th

e co

rrec

t

pers

on o

r ac

tion

was

mor

e ac

tive

than

any

oth

er u

nit i

n re

spon

se to

pro

bes

for the actor, a

ctio

n, a

nd p

atie

nt. T

he m

odel

got

97%

of

thes

e no

vel

sent

ence

s co

rrec

t.N

ow o

bvio

usly

this

is b

ut th

e fir

st s

tep

in d

emon

stra

ting

that

con

nect

ion-

ist n

etw

orks

can

exh

ibit

syst

emat

icity

, The

cor

pus

is fi

nite

, and

87,

5% o

f it

was

use

d du

ring

trai

ning

, Nev

erth

eles

s, th

ere

is c

onsi

dera

ble

syst

emat

icity

in th

e m

odel

's p

erfo

rman

ce,

Now

, it m

ight

be

note

d th

at systematicity is not in fact an in

here

ntat

trib

ute

of th

e m

odel

that

we

have

pro

pose

d, I

n fa

ct, i

t is

prob

ably

true

that

the

mod

el c

ould

lear

n to

trea

t sen

tenc

es a

s un

ana

lyse

d w

hole

s, if

infa

ct th

e co

nstit

uent

str

uctu

re o

f se

nten

ces

had

no r

elat

ion

to th

eir

actu

alm

eani

ngs,

Thu

s, w

e co

uld

trai

n th

e m

odel

to p

rodu

ce a

rbitr

ary

answ

ers

topr

obes

giv

en in

con

junc

tion

with

eac

h m

embe

r of

a li

st o

f se

nten

ces,

Lea

rnin

g w

ould

be

slow

er th

an if

the

sent

ence

s ha

d a

syst

emat

ic s

truc

ture

,an

d it

wou

ld r

equi

re m

ore

units

and

con

nect

ions

, but

in p

rinci

ple

this

cou

ldbe done,

The

que

stio

n ar

ises

, the

n, a

s to

why

it is

that

lang

uage

s tu

rn o

ut to

be

syst

emat

ic, O

ne p

ossi

bilit

y is

that

it is

an

inhe

rent

cha

ract

eris

tic o

f hum

anco

gniti

on to

be

syst

emat

ic in

just

this

way

, and

this

is th

e cr

ux o

f Fo

dor

and

Pyly

shyn

s ar

gum

ent.

On

this

vie

w, h

uman

lang

uage

s ar

e sy

stem

atic

be-

cause our minds force them to be that w

ay, d

ue to

thei

r use of a

com

bina

tori

al s

ynta

x an

d se

man

tics,

Ind

eed, Fodor and Pylyshyn argue

that

con

nect

ioni

st m

odel

s ca

nnot

exp

lain

the

syst

emat

icity

of

lang

uage

,because connectionist models do not inherently impose a combinatorial

syntax and semantics,

Our

res

pons

e to

this

arg

umen

t has

two

part

s, F

irst

, we

ques

tion

the

impl

icit

assu

mpt

ion

that

the

expl

anat

ion

for

the

syst

emat

icity

of

lang

uage

mus

t lie

with

inherent characteristics of the m

echa

nism

s of

thou

ght.

Indeed, we would look to the tasks that hu

man

s m

ust p

erfo

rm w

ithlin

guis

tic s

timul

i, an

d th

e ex

peri

ence

s fr

om w

hich

they

lear

n, f

or a

t lea

stso

me

part

of

the explanation, If the task is to form representations

even

ts w

hich

them

selv

es h

ave

a co

mbi

nato

rial

str

uctu

re, t

hen

a co

mpu

ta-

tiona

l mec

hani

sm th

at b

ecom

es s

yste

mat

ic m

ay b

e si

mpl

y di

scov

erin

g th

esy

stem

atic

ity in

the

envi

ronm

ent.

Syst

emat

icity

in th

e en

viro

nmen

t is

prob

abily

onl

y pa

rt o

f th

e st

ory,

It

has

been

the

forc

e of

muc

h w

ork

in li

ngui

stic

s an

d el

sew

here

to a

rgue

that

the

syst

emat

icity

in th

e en

viro

nmen

t is

muc

h le

ss th

an th

e sy

stem

atic

ityth

at is

impo

sed

by th

e hu

man

obs

erve

r. I

ndee

d, w

e su

spec

t tha

t the

re is

som

e tr

uth

to th

is c

laim

,T

his

lead

s us

to o

ur s

econ

d po

int,

whi

ch is

that

in f

act c

onne

ctio

nist

mod

els

do

tend

to im

pose

sys

tem

atic

ity, e

ven

thou

gh th

ey d

o no

t hav

e a

com

bina

tori

al s

ynta

x an

d se

man

tics,

Net

wor

ks d

o no

t, a

s a

mat

ter

of f

act,

sim

ply

mem

oris

e in

divi

dual

inpu

t-ou

tput

pai

rs a

nd tr

eat e

ach

one

as a

nis

olat

ed in

divi

dual

cas

e; th

e ge

nera

lisat

ion

expe

rim

ent j

ust d

escr

ibed

is ju

ston

e of

a v

ery

larg

e nu

mbe

r of

rel

evan

t dem

onst

ratio

ns th

at in

fact

they

do

exhi

bit a

tend

ency

to b

ehav

e sy

stem

atic

ally

, A te

nden

cy to

war

ds b

ehav

ing

syst

emat

ical

ly is

in fa

ct a

cha

ract

eris

tic th

at o

ur m

odel

sha

res

with

mec

han-

ism

s w

hich

sim

ply

stip

ulat

e a

com

bina

tori

al s

ynta

x an

d se

man

tics,

Fro

m h

ere

on, t

he a

rgum

ent s

impl

y ge

ts te

nden

tious

, One

sid

e ca

n cl

aim

that

con

nect

ioni

st m

odel

s do

not

in th

eir

pres

ent f

orm

exhi

bit

enou

ghsy

stem

atic

ity; a

nd w

hile

this

may

be

true

, it p

lace

s an

y in

-principle argu-

men

t aga

inst

sys

tem

atic

ity in

con

nect

ioni

st m

odel

s in

con

side

rabl

e do

ubt.

Or

it m

ight

be

clai

med

that

a te

nden

cy to

beh

ave

syst

emat

ical

ly h

as b

een

snuc

k in

to th

e m

odel

by

som

e sl

ight

of

hand

, It i

s in

fac

t tru

e th

at p

artic

ular

choi

ces

of d

etai

ls o

f ne

twor

k ar

chite

ctur

e do

infJ

uenc

e th

e de

gree

of

syst

emat

icity

; and

inde

ed it

is q

uite

impo

rtan

t to

get a

cle

arer

pic

ture

of

wha

t asp

ects

of

netw

ork

arch

itect

ure

are

cond

uciv

e to

goo

d gc

nera

lisat

ion,

We

do n

ot d

oubt

that

evo

lutio

n m

ay h

ave

shap

ed o

ur c

ogni

tives

truc

ture

sso

as

to m

ake

them

mor

e lik

ely

to b

e ab

le to

act

sys

tem

atic

ally

: but

we

see

no r

easo

n to

sup

pose

that

it h

as d

one

so b

y en

dow

ing

them

in a

dvan

ce w

ithan

exp

licitl

y co

mbi

nato

rial

syn

tax

and

sem

antic

s,W

hat a

bout

pro

duct

ivity

? P

rodu

ctiv

ity is

of c

ours

e in

timat

ely

linke

d to

syst

emat

icity

; it r

efer

s to

the

fact

that

we

can.

und

erst

and

man

y se

nten

ces

that

we

have

not

act

ually

hea

rd b

efor

e, T

he e

xper

imen

t jus

t des

crib

edad

dres

sed

this

poi

nt, t

houg

h ag

ain,

in a

fai

rly

limite

d w

ay, I

t is

gene

rally

assu

med

that

hum

ans

can

com

preh

end

an in

fini

te n

umbe

r of

sen

tenc

es,

whi

le in

our

exp

erim

ent t

he c

orpu

s w

as in

deed

onl

y a

bit l

arge

r th

an th

etr

aini

ng s

et.

Oth

er r

esea

rch

on th

e pr

oduc

tivity

of c

onne

ctio

nist

net

wor

ks is

cur

rent

lyunderway, Servan-

Sch

reib

er. C

leer

eman

s, a

nd M

cCle

lland

(1988) have

show

n th

at a

sim

ple

netw

ork

arch

itect

ure

first

intr

oduc

ed b

y E

lman

(19

88)

5132

2M

cCLe

lLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

323

can

lear

n to

acc

ept a

ll an

d on

ly th

e gr

amm

atic

al to

kens

of a

sim

ple

finite

stat

e la

ngua

ge, B

ecau

se in

the

case

of

this

fin

ite s

tate

lang

uage

the

corp

us is

infi

nite

. we

have

the

firs

t cle

ar in

dica

tion

that

a n

etw

ork

can

lear

n fr

omfi

nite

exp

erie

nce

to p

roce

ss a

n in

fini

te c

orpu

s. T

he handling of long-

dist

ance

dep

ende

ncie

s is

cur

rent

ly u

nder

rat

her

activ

e ex

plor

atio

n (S

erva

n-Sc

hrei

ber

et a

I., 1

988;

Elm

an, 1

989)

, Ext

ensi

on to

com

preh

ensi

on, r

athe

rth

an m

ere

acce

ptan

ce o

f gra

mm

atic

al to

kens

, aw

aits

as

the

next

cha

lleng

e,T

here

is a

n as

pect

of t

he p

rodu

ctiv

ity o

f language that appears to be

bette

r ex

plai

ned

by o

ur c

onne

ctio

nist

app

roac

h th

an b

y co

nven

tiona

lap

proa

ches

, Thi

s is

the

use

of c

onte

xt to

sha

de m

eani

ngs

of c

once

pts

asthey are in

stan

tiate

d in

par

ticul

ar e

vent

s w

hich

may

be

cont

extu

ally

appr

opri

ate,

The

exa

mpl

e of

the

ball

from

McC

lella

nd a

nd K

awam

oto

illus

trat

es th

is. I

n an

othe

r ca

se, they presented th

eir

mod

el w

ith th

ese

nten

ce "

The

dol

l mov

ed",

Thi

s se

nten

ce w

as n

ovel

to th

e m

odel

. Am

ong

the

feat

ures

that

the

mod

el h

ad le

arne

d w

ere

asso

ciat

ed w

ith "

doll"

was

inan

imac

y, H

owev

er, i

n in

terp

retin

g th

is s

ente

nce

the

mod

el "

anim

ated

"th

e do

ll, T

his

is b

ecau

se, i

n al

l of

the

sent

ence

s th

at th

e m

odel

had

bee

ntr

aine

d on

, the

sub

ject

of

the

sent

ence

s of

the

form

"X moved" were

alw

ays

anim

ate,

It s

eem

s to

us

that

this

inte

rpre

tive

liber

ty o

n th

e pa

rt o

fth

e m

odel

is e

ntir

ely

corr

ect a

nd a

ppro

pria

te, a

nd il

lust

rate

s a

prod

uctiv

ityth

at e

xten

ds fa

r be

yond

the

capa

bilit

ies

of c

onve

ntio

nal m

odel

s,

Bey

ond

Com

posi

tiona

lity

We

have

dis

cuss

ed tw

o ou

t of

the

thre

e ch

arac

teri

stic

s Fo

dor

and

Pyly

shyn

(198

8) c

laim

lang

uage

has

that

are

cap

ture

d by

con

vent

iona

l app

roac

hes,

The

thir

d ch

arac

teri

stic

is c

ompo

sitio

nalit

y: T

he id

ea th

at a

wor

d co

ntri-

bute

s th

e sa

me

thin

g to

the

mea

ning

of

all o

f th

e se

nten

ces

in w

hich

itoccurs. In the introduction

, we

criti

cise

d th

e no

tion

of c

ompo

sitio

nalit

y,in

dica

ting

that

in f

act i

t rep

rese

nts

an im

pove

rishe

d vi

ew o

f the

com

pre-

hens

ion

proc

ess,

In o

ur il

lust

rativ

e m

odel

, a w

ord

does

alw

ays

exer

t the

sam

e in

flue

nce

on th

e ne

t inp

ut to

the

firs

t set

of

hidd

en u

nits

in th

eco

mpr

ehen

sion

par

t of

the

mod

el. B

ut, d

ue to

the

non-

linea

ritie

s in

the

hidden units at that layer in the network, and due to the concurrent

infl

uenc

e of

inpu

ts f

rom

con

text

, the

act

ual i

mpa

ct o

f th

e w

ord

can

diff

ergreatly from context to context, The word exerts the same

forc

e on

the

repr

esen

tatio

n at

eac

h oc

curr

ence

, but

this

for

ce is

com

bine

d w

ith th

ose

appl

ied

by c

onte

xt, t

here

by a

llow

ing

for

cont

ext s

ensi

tivity

,It

mig

ht b

e ar

gued

that

the

mod

el is

too

sens

itive

to c

onte

xt, i

n th

at in

fact

it a

llow

s co

ntex

t som

etim

es to

ove

rrid

e th

e co

rrec

t int

erpr

etat

ion

of a

wor

d, T

his

happ

ens,

for

exa

mpl

e, w

ith th

e st

eak

in th

e ex

ampl

e pr

esen

ted

above: After the presentation of

with daintiness,

the activation of

stea

k

prob

ing

for

the

patie

nt is

wea

kene

d, I

n fa

ct, a

t ear

lier

poin

ts in

lear

ning

,

the

mod

el a

ctua

lly activates soup more strongly than steak after

with

dain

tines

s is

pre

sent

ed,

Thi

s be

havi

our

mus

t be

take

n as

an

erro

r, b

ut it

is a

n er

ror

of th

e ki

ndth

at p

eopl

e of

ten

mak

e, F

or e

xam

ple,

Eri

ckso

n an

d M

atts

on (

1981

) as

ked

subj

ects

to a

nsw

er th

e qu

estio

n:

How

man

y an

imal

s of

eac

h ki

nd d

id M

oses

take

on

the

ark?

Mos

t sub

ject

s sa

id tw

o, a

nd n

otic

ed n

othi

ng a

mis

s, e

ven

thou

gh th

ey h

adbe

en w

arne

d to

look

out

for

tric

k se

nten

ces,

Ind

eed

, man

y su

bjec

ts c

ould

not p

inpo

int t

he p

robl

em w

ith th

is s

ente

nce

even

aft

er b

eing

told

that

ther

ew

as s

omet

hing

wro

ng, A

ppar

ently

, the

con

stra

ints

impo

sed

by th

e w

ord

Mos

es

itsel

f ar

e no

t suf

fici

ently

str

ong

to o

verr

ide

thos

e im

pose

d by

the

cont

ext.

Som

e m

ay v

iew

err

ors

of th

e M

oses

type

as

abbe

ratio

ns, t

houg

h th

eef

fect

is e

asy

to p

rodu

ce w

ith o

ther

exa

mpl

es. T

o us

it is

a r

efle

ctio

n of

the

fact

that

the

doct

rine

of c

ompo

sitio

nalit

y m

isre

pres

ents

the

cont

ribu

tions

of words to an understanding of the meanings of se

nten

ces,

In s

um, w

e do

not

see

any

rea

son

to s

uppo

se th

at th

e ob

serv

ed d

egre

e of

syst

emat

icity

, pro

duct

ivity

, and

com

posi

tiona

lity

of h

uman

lang

uage

nee

dbe

attr

ibut

ed to

inhe

rent

str

uctu

ral c

hara

cter

istic

s of

the

kind

that

Fod

oran

d P

ylys

hyn

have

adv

ocat

ed, I

n pa

rt, t

he s

yste

mat

icity

of

the

wor

ld m

aybe to blame; beyond th

is it

is c

lear

that

networks do tend to impose

syst

emat

icity

; pro

duct

ivity

doe

s no

t app

ear

to b

e be

yond

the

pow

er o

f PD

Pm

odel

s lik

e th

e on

e w

e ha

ve c

onsi

dere

d he

re; a

nd th

ere

are

aspe

cts

of th

eex

pres

sive

cap

abili

ty o

f hu

man

lang

uage

that

go

beyo

nd w

hat c

an b

eca

ptur

ed in

a c

ombi

nato

rial

syn

tax

and

sem

antic

s w

hich

see

m n

atur

ally

tofo

llow

fro

m a

PD

P ac

coun

t.

lexi

cal a

nd S

ynta

ctic

Aut

onom

yW

e tu

rn n

ow to

a s

et o

f considerations that arise from psychological

expe

rim

ents

, - w

here

it is

cla

imed

that

at l

east

dur

ing

som

e in

itial

sta

ge o

fpr

oces

sing

, bot

h le

xica

l acc

ess

(i. e. activation of the possible m

eani

ngs

asso

ciat

ed w

ith w

ords

) an

d sy

ntac

tic p

roce

ssin

g (i

, e. a

ssig

ning

atta

chm

ent

rela

tions

bet

wee

n se

nten

ce c

onst

ituen

ts)

are

auto

nom

us p

roce

sses

, The

secl

aim

s ru

n di

rect

ly c

ount

er to

the

basi

c te

nets

of

the

appr

oach

that

we

have

take

n he

re, b

ecau

se th

e ap

proa

ch a

ssum

es th

at th

ese

proc

esse

s ar

e in

ex-

tric

ably

inte

rtw

ined

with

eac

h ot

her

and

with

the

expl

oita

tion

of c

onte

xtua

lin

flue

nces

, Cle

ar e

vide

nce

for

auto

nom

y w

ould

ther

efor

e un

derc

ut o

urap

proa

ch c

ompl

etel

y, S

o le

t us

see ,

wha

t is

the

evid

ence

?

Lex

ical

Acce

ss,

In well-kn

own

expe

rim

ents

(Sw

inne

y, 1

979;

Tan

enha

us, L

eim

an, & Seidenberg, 19

79),

sub

ject

s ha

d to

list

en to

a

5132

4M

cCLE

LLA

ND

. ST

, JO

HN

, TA

RA

BA

N

spok

en te

xl c

onla

inin

g an

am

bigu

ous

wor

d (s

uch

as B

UO

S) a

nd w

ere

prob

ed

for

a le

xica

l dec

isio

n im

med

iate

ly a

fcer

the

offs

el o

f th

e w

ord

wilh

anolher word related

to

eilh

er m

eani

ng o

f Ih

e am

bigu

ity. T

he o

fc-c

iled

resull of such experiments is the finding that decisions

to

words relaled

eilh

er m

eani

ng o

f th

e ambiguity are faster than decisions

10

unre

lale

d

wor

ds, i

ndic

atin

g th

ai b

olh

mea

ning

s ar

e in

itial

ly a

cces

sed;

onl

y la

ter

is I

heam

bigu

ity r

esol

ved

to f

it th

e co

nlex

l, so

that

the

cont

extu

ally

app

ropr

iale

read

ing

is I

he o

nly

one

thai

rem

ains

act

ive,

There are Cwo poinls

to

be m

ade,

The

firs

t is

that

a r

ecen

t mel

a-an

alys

is(S

I. Jo

hn, 1

988)

of

a lo

cal o

f 19

stu

dies

, usi

ng b

oth

lexi

cal d

ecis

ion

and

word naming melhods, reveals that in facl there is a reliable advantage

for

Ihe

conl

extu

ally

app

ropr

iale

rea

ding

, eve

n at

an

imm

edia

te te

st. T

hepa

ltern

exh

ibile

d in

Fig

, 7 f

rom

Ihe

sem

inal

exp

erim

ent o

f Sw

inne

y (1

979)

is e

xem

plar

y of

Ihe

gen

eral

pal

tern

of

Ihe

resu

lls,

The

sec

ond

poin

l is

Ihal

chis paltern is very

close to

wha

t is

foun

d in

asi

mul

alio

n of

the

proc

esse

s of

sel

tling

on

an in

terp

reta

tion

of a

n am

bigu

ous

wor

d in

a P

OP

mod

el

of

Ihe

disa

mbi

gual

ion

proc

ess

(Kaw

amol

o, 1985;

1988

; see

Fig

, 7),

Kaw

amol

os

mod

el d

iffe

rs f

rom

Ihe

illu

slra

live

mod

elde

scri

bed

here

in I

hree

cru

cial

way

s, F

irst,

it u

ses

a co

ntin

uous

, gra

dual

activ

alio

n pr

oces

s, s

o ch

at u

nits

gra

dual

ly s

etlle

inlo

thei

r fi

nal s

late

, rat

her

Ihan

bei

ng I

hrus

t int

o a

stal

e in

a s

ingl

e se

ep. S

econ

dly,

it m

akes

use

of

full

recu

rren

ce in

Ihe

con

necl

ions

am

ong

the

units

, so

Ihal

uni

lS w

ithin

Ihe

sam

e pa

rt o

f th

e sy

stem

fee

d ba

ck o

n ea

ch o

ther

, Thi

rdly

, it d

oes

not

aclu

ally

sim

ulat

e Ih

e fu

ll pr

oces

s of

sen

tenc

e in

lerp

reta

tion,

but

onl

yco

nsid

ers

Ihe

proc

ess

of s

ettli

ng o

n an

inle

rpre

tatio

n of

an

indi

vidu

al w

ord

as a

join

l fun

ctio

n of

con

text

ual a

nd p

hono

logi

cal i

nput

. We

view

Kaw

a-m

oto

s model as an atlempl

to

char

acle

rise

the

fine-

grai

n te

mpo

ral p

roce

s-se

s in

volv

ed in

lexi

cal a

cces

s th

ai a

re m

ore

coar

sely

app

roxi

mat

ed in

Ihe

SO

mod

el.

Now

, Kaw

amol

os

mod

el m

ost c

lear

ly d

oes

nol a

ssum

e Ihallhe process

of a

cces

sing

mea

ning

is a

uton

omou

s; a

s in

the

SO m

odel

, bot

h co

ntex

tual

and

inpu

l-ba

sed

cons

lrai

nts

infl

uenc

e th

e pr

oces

s fr

om th

e st

art.

How

ever

,w

hat h

appe

ns in

Ihe

mod

el is

thai

at f

irst

bot

h of

the

poss

ible

mea

ning

sco

nsis

tenl

wilh

Ihe

inpu

l wor

d ar

e ac

tival

ed. I

e is

onl

y as

the

activ

alio

npr

oces

s co

nlin

ues,

that

one

inte

rpre

latio

n is

gra

dual

ly p

ushe

d ou

t and

the

oche

r co

mes

10

dom

ina

Ie c

ompl

etel

y. T

hus

it ap

pear

s ch

at th

e em

piric

alev

iden

ce is

qui

te s

imila

r to

wha

l sho

uld

be e

xpec

ted

on th

e P

OP

acc

ount

.

I!s

.....

. .....

SEN

TE

NC

E C

OM

PRE

HE

NSI

ON

AN

D P

DP

7-21

U,l

Del

Dr

CO

NT

EX

T -

-VE

RB

5132

5

Autonomous Syntax?

A n

umbe

r of

stu

dies

hav

e be

en r

epor

ted

indi

cat-

ing thai syntactic preferences in

itial

ly d

eter

min

e Ih

e ou

tcom

e of

on-

line

pars

ing

proc

esse

s, s

o ch

at s

ente

nces

in w

hich

the

conl

ext e

vent

ually

re-

quir

es a

n al

lern

ativ

e in

lerp

reta

tion

are

proc

esse

d m

ore slowly than those

in w

hich

the

conl

ent i

s co

nsis

lent

with

the

synt

actic

bia

s. A

var

iety

of

FIG, 7 (a) Data from Swinney (1979); (b) activations of contextually appropriate and

inap

prop

riat

e m

eani

ngs

of th

e am

bigu

ous

wor

d w

ind

(fro

m K

awam

oto

s di

stri

bute

d m

odel

of

ambi

guity

res

olut

ion)

. The

con

text

spe

cifie

s th

at a

ver

b is

exp

ecte

d. a

nd th

e tw

o cu

rves

repr

esen

t act

ivat

ions

of p

atte

rns

corr

espo

ndin

g to

the

cont

extu

ally

app

ropr

iate

mea

ning

(E

for

enci

rcle

) an

d th

e co

ntex

tual

ly in

appr

opri

ate

mea

ning

(D

for

dra

ft).

(a)

is r

epri

nted

from

McC

lella

nd (

1987

) an

d (b

) fr

om K

awam

oto

(198

5),

~..

,.:i

5132

6M

cClE

LLA

ND

, ST

, JO

HN

. TA

RA

BA

N

constructions have been examined in studies

of

this type, One

of

thes

e is

the

redu

ced

rela

tive

cons

truc

tion,

in s

ente

nces

like

(23

) an

d (2

4):

23, T

he a

ctre

ss s

ent t

he fl

ower

s w

as v

ery

plea

sed,

24, T

he f

lori

st s

ent t

he f

low

ers

was

ver

y pl

ease

d,

Ano

ther

is th

e N

-P

P c

onst

ruct

ion,

as

in (

25)

and

(26)

:

25, The spy saw the policeman with the binoculars, but,

, ,

26, The spy saw the policeman with the revolver. but,

, ,

Using the first kind

of

cons

truc

tion.

Ray

ner,

Car

lson

, and

Fra

zier

(19

R3)

foun

d th

at s

ubje

cts

have

dif

ficu

lty p

roce

ssin

g th

e re

duce

d re

lativ

e cl

ause

inboth cases, even though in one

of

the

exam

ples

(th

e ac

tres

s se

nt th

eflowers) semantic constraints are said to favour the idea that the actress

would be the recipient rather than the sender

of

the

flow

ers

as is

req

uire

din

the

redu

ced

rela

tive

inte

rpre

tatio

n,S

uch

a fin

ding

is, i

n ou

r vi

ew, not particularly telling in indicating

whether there is some

initi

al

synt

actic

pro

cess

that

fav

ours

one

inte

rpre

ta-

tion over the other 0

or w

heth

er, a

ltern

ativ

ely,

ther

e is

sim

ply

a st

rong

wei

ght

asso

ciat

ed w

ith th

e sy

ntac

tic p

refe

renc

e to

trea

t a N

-N

seq

uenc

eas

act

or-a

ctio

n-ob

ject

. It c

erta

inly

is the case that the initial part

of

the

sent

ence

(5 -10

SEN

TE

NC

E C

OM

PRE

HE

NSI

ON

AN

D P

DP

5132

7

Tar

aban

& M

cCle

lland

SI'm

uh (

TM

IR

ayne

, Car

rson

& F

raz'

er S

I'muh

(R

CF)

Ave

rage

01

TM

and

RC

F

-20

-30

'00

noun

-fill

err + 1

r + 2

, + 3

Word POOlllon

FIG

,R

eadi

ng ti

me

adva

ntag

e (n

egat

ive

num

bel1

l) o

r di

sadv

anta

ge (

posi

tive

num

bers

) fo

rse

nten

ces

requ

irin

g a

verb

phra

se a

ttach

men

t of

a pr

epos

ition

al p

hras

e co

mpa

red

to m

atch

edse

nten

ces

requ

iring

a n

ounp

hras

e at

tach

men

t. R

ayne

r et

al.

' s (

1983

) st

imul

i are

bia

sed

so th

atsu

bjec

ts e

xpec

t the

PP

to a

ttach

to th

e V

p, T

arab

an a

nd M

cCle

lland

' s s

timul

i lea

d su

bjec

ts to

. exp

ect t

he P

P to

atta

ch to

the

prec

edin

g N

P,

The

act

ress

sen

t the

flo

wer

s, ,

,

is u

nam

bigu

ousl

y in

terp

rete

d by

nat

ive

spea

kers

as indicating that the

actress is the sender not the recipient

of

the

flow

ers;

pla

usib

le c

ontin

ua-

tions

mig

ht in

volv

e a

reci

pien

t (he

rsel

f, p

erha

ps?)

or

anot

her

clau

se, T

hus

it ap

pear

s th

at th

e sy

ntac

tic c

ues

are

sim

ply

over

ridin

g in

this

cas

e, S

imila

rarguments apply to many

of

the

mat

eria

ls u

sed

in th

e su

bseq

uent

stu

dy b

yFerreira and Clifton (1986),

In

the second kind

of

cons

truc

tion.

it w

as fo

und

(Ray

ner

et a

I., 1

983)

that there was an advantage for sentences

of

the form

of

(25)

, in

whi

ch th

epr

epos

ition

al p

hras

e is

ulti

mat

ely

atta

ched

to th

e ve

rbph

rase

, com

pare

d to

sent

ence

s of

the form

of

(26)

, in

whi

ch th

e pr

epos

ition

al p

hras

e is

ulti

-m

atel

y at

tach

ed to

the

noun

phra

se, H

owev

er, a series

of

expe

rim

ents

(Tar

aban

, 198

8; T

arab

an &

McC

lella

nd. 1

988;

199

0) h

as n

ow e

stab

-lis

hed'

~eve

ral i

mpo

rtan

t fin

ding

s re

gard

ing

this

par

ticul

ar c

onst

ruct

ion,

Tar

aban

and

McC

lella

nd (

1988

, exp

erim

ent 1

) established three basic

poin

ts, F

irst

, the

mat

eria

ls u

sed

by R

ayne

r et

a!.

gen

eral

ly h

ada

bias

suc

hthat the part

of

the

sent

ence

pre

cedi

ng th

e di

sam

bigu

atin

g w

ord

(rev

olve

ror

bin

ocul

ars,

in th

is c

ase)

tend

ed to

fav

our

the

VP

atta

chm

ent

of

the

prep

ositi

onal

phr

ase,

Sec

ondl

y, o

ther

mat

eria

ls a

re e

asily

con

stru

cted

inwhich this attachment preference is reversed, Thirdly, studies

of

on-l

ine

proc

essi

ng u

sing

the

wor

d-by

-wor

d re

adin

g ta

sk d

evel

oped

by

Just

, Car

-

pent

er, a

nd W

oolle

y (1

982)

rev

eale

d th

at th

e fi

ndin

g re

port

ed b

y R

ayne

r et

a!. (1983) only holds with the VP-attachment-bi

ased

mat

eria

ls, a

nd is

reversed with the NP-bi

ased

-mat

eria

ls (

Fig,

8),

With

VP-

atta

chm

ent-

bias

ed m

ater

ials

(R

ayne

r et

a!.

' s m

ater

ials

), th

ere

is a

rea

ding

tim

e ad

van-

tage

for

nou

n-fil

lers

that

acc

ords

with

the

VP

atta

chm

ent b

ias,

whi

ch to

tals

abou

t 100

mse

c an

d is

dis

trib

uted

ove

r th

e th

ree

wor

ds f

ollo

win

g th

e no

un-

fille

r, However. with NP-attachment-biased materials (Taraban and

McC

lella

nd' s

sen

tenc

es).

ther

e is

an

appr

oxim

atel

y eq

ual a

nd o

ppos

itepattern; averaging the two types

of

mat

eria

ls. t

here

is v

irtu

ally

no

over

all

advantage for either type

of

atta

chm

ent,

Thu

s, th

e st

udy

indi

cate

s th

atco

nten

t , r

athe

r th

an a

ny g

ener

al s

ynta

ctic

pre

fere

nce,

app

ears

to d

eter

-mine initial attachment preferences in this kind

of

cons

truc

tion,

Another experiment (Taraban & McClelland, 1990) addressed the

sour

ce

of the content-based influences on processing

of the prepositional

phra

se, O

ne p

ossi

bilit

y th

at is

oft

en c

onsi

dere

d is

the

idea

that

the

verb

may

pro

vide

a b

asis

for

exp

ecta

tions

abo

ut p

ossi

ble

argu

men

ts th

at m

ight

influence the course

of

proc

essi

ng; t

hese

exp

ecta

tions

cou

ld s

till b

e a!

tri-

5132

8M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

329

bute

d to

the

wor

king

s of

an

auto

nom

ous

synt

actic

pro

cess

whi

ch n

ever

the-

less

con

sulte

d sy

ntac

tic in

form

atio

n in

the

lexi

con,

In

this

exp

erim

ent

Tar

aban

and

McC

lella

nd d

emon

stra

ted,

how

ever

, tha

t the

con

tent

of t

heob

ject

NP

als

o in

fluen

ced

perf

orm

ance

. For

exa

mpl

e, i

n se

nten

ce (

27),

27. The dictator viewed the masses from the.

, ,

2R, The dictator viewed the petition from the. .

subj

ects

exp

ecte

d a

loca

tive

PP

, atta

chin

g to

the

verb

, ind

icat

ing

the

plac

efr

om w

hich

the

view

ing

was

to o

ccur

; whe

reas

in (

28),

they

exp

ecte

d a

sour

ce o

f th

e pe

titio

n, a

ttach

ing

to th

e ob

ject

NP.

Whe

n th

ese

expe

ctat

ions

wer

e vi

olat

ed, t

here

was

a s

low

dow

n in

pro

cess

ing,

The

exp

erim

ents

by

Tar

aban

and

McC

lella

nd d

emon

stra

te th

at th

eco

nten

t of t

he m

ain

verb

, as

wel

l as

that

of

the

post

-ver

bal o

bjec

t NP

, can

influence on-lin

e pr

oces

sing

dec

isio

ns a

bout

PP

atta

chm

ent.

The

y do

not

address whether or not the content of the

subj

ect

noun

phra

se c

an a

lso

infl

uenc

e on

-lin

e pa

rsin

g de

cisi

ons,

alth

ough

it is

kno

wn

from

Ode

n (1

978)

that

it c

an in

flue

nce

the

choi

ce o

f th

e ul

timat

e at

tach

men

t. H

owev

er, w

ork

repo

rted

by

Tan

enha

us, C

arls

on, a

nd T

rues

wel

l (th

is is

sue)

sug

gest

s th

atthe semantic characteristics of the subject

can

infl

uenc

e on

- lin

e pr

oces

sing

deci

sion

s in

str

uctu

res

sim

ilar

to (

23)

and

(24)

(se

e T

anen

haus

et a

l. fo

rfu

rthe

r de

tails

). T

hus,

it w

ould

app

ear

that

evi

denc

e is

acc

umul

atin

g in

favo

ur o

f th

e vi

ew th

at a

ll pa

rts

of a

sen

tenc

e ca

n influence on-

line

proc

essi

ng d

ecis

ions

abo

ut e

very

oth

er p

art.

Another of Taraban and McClelland'

s (1

988)

exp

erim

ents

con

side

red

the

poss

ibili

ty th

at th

e di

srup

tion

in p

roce

ssin

g th

at is

occ

urri

ng in

thes

ese

nten

ces

is d

ue to

spe

cifi

c ex

pect

atio

ns f

or p

artic

ular

fill

ers

rath

er th

anexpectations concerning the role and/or attachment of the pr

epos

ition

alph

rase

s. T

houg

h a

smal

l eff

ect f

or p

artic

ular

fill

ers

was

fou

nd, t

he la

rges

tef

fect

app

eare

d to

be

due

to v

iola

tions

of e

xpec

tatio

ns fo

r th

e ro

le o

f the

prep

ositi

onal

phr

ase,

Vio

latio

ns o

f ex

pect

ed a

ttach

men

t had

no

furt

her

disr

uptiv

e ef

fect

ove

r an

d ab

ove

that

attr

ibut

able

to th

e in

evita

ble

conc

om-

itant

vio

latio

n of

the

subj

ect's

rol

e ex

pect

atio

ns (

see

Tar

aban

& M

cCle

l-la

nd, 1

98R

, for

det

ails

). T

hese

fin

ding

s ar

e ce

rtai

nly

cons

iste

nt w

ith th

e SO

mod

el, i

n th

at th

ere

is n

o se

para

te r

epre

sent

atio

n of

the

synt

actic

for

m o

f a

sent

ence

; the

re is

, ins

tead

, dire

ct p

roce

ssin

g in

to a

rep

rese

ntat

ion

whi

chca

n be

use

d to

ans

wer

que

stio

ns a

bout

the

role

s of

the

part

icip

ants

in th

eev

ent t

hat i

s de

scrib

ed b

y th

e se

nten

ce.

In s

umm

ary,

the

evid

ence

from

Tar

aban

and

McC

lella

nd' s

PP

atta

ch-

men

t stu

dies

see

ms

cons

iste

nt w

ith th

e vi

ew th

at c

onte

nt c

an in

deed

pla

yarole in setting up expectations for the roles played by

the

obje

cts

ofprepositional phrases, a

nd th

at th

ese

expe

ctat

ions

can

gov

ern

the

initi

alpr

oces

sing

of

thes

e ph

rase

s as

they

are

enc

ount

ered

on-

line

in s

ente

nce

proc

essi

ng. T

anen

haus

et a

l. ' s

find

ings

indi

cate

that

the

effe

cts

of c

onte

nt

are

not r

estr

icte

d to

lexi

cal i

nfor

mat

ion

that

mig

ht b

e st

ored

dire

ctly

with

head

s of

phr

ases

, Tho

ugh

it is

ver

y cl

ear

that

syntax often exerts an

overriding influence, t

here

is n

o reason to suppose on the basis of the

stud

ies

revi

ewed

her

e th

at it

occ

upie

s a

priv

ilege

d or

aut

onom

ous

posi

tion

in th

e in

itial

pro

cess

ing

of s

ente

nces

. Ins

tead

, it a

ppea

rs th

at c

onte

nt a

sw

ell a

s sy

ntax

can

infl

uenc

e in

itial

atta

chm

ent a

nd r

ole

assi

gnm

ent p

refe

r-en

ces,

Further arguments ag

ains

t the

aut

onom

y of

syn

tax

com

e fr

om th

ere

sear

ch o

f C

rain

and

Ste

edm

an (

1985

), A

ltman

n an

d St

eedm

an (1

988)

,an

d A

ltman

n (1

988)

. The

se p

aper

s ar

gue

that

atta

chm

ent d

ecis

ions

can

be

gove

rned

by

refe

rent

ial p

roce

sses

trig

gere

d by

con

text

pre

sent

ed p

rior

toth

e se

nten

ce c

onta

inin

g th

e am

bigu

ity. T

aken

toge

ther

with

Tar

aban

and

McC

lella

nd' s

res

ults

, the

se r

esul

ts h

elp

pain

t a g

ener

al p

ictu

re in

whi

chsy

ntax

is f

ar f

rom

aut

onom

ous.

Altm

ann

and

Stee

dman

(19

88)

poin

t out

that

the

find

ings

on

atta

chm

ent

ambi

guity

res

olut

ion

are

cons

iste

nt w

ith a

vie

w th

ey c

all "

wea

k in

tera

ctiv

-ity

, in which a syntactic m

odul

e co

nstr

ucts

all

poss

ible

par

ses

and

the

cand

idat

e th

at b

est s

atis

fies

all

of th

e co

nstr

aint

s is

sel

ecte

d by

sub

sequ

ent

proc

esse

s se

nsiti

ve to

con

tent

, ref

eren

tial c

oher

ence

, etc

, The

y po

int o

utth

at s

uch

a w

eak

inte

ract

ivity

acc

ount

is p

roba

bly

not d

istin

guis

habl

eem

piri

cally

fro

m p

laus

ible

ver

sion

s of

strongly interactive accounts, in

whi

ch c

once

ptua

l/ref

eren

tial m

odul

es in

the

lang

uage

pro

cess

ing

syst

emin

stru

ct m

odul

es s

peci

alis

ed f

or s

ynta

ctic

pro

cess

ing.

The

vie

w ta

ken

here

goe

s be

yond

eve

n st

rong

ly in

tera

ctiv

e ac

coun

ts, i

npr

opos

ing

that

the

synt

actic

and

con

cept

ual a

spec

ts o

f pr

oces

sing

are

in f

act

inex

tric

ably

inte

rtw

ined

. Per

haps

this

vie

w m

ight

bes

t be

calle

d an

inte

gra-

tive

as o

ppos

ed to

an

inte

ract

ive

acco

unt.

Inte

ract

ivity

sug

gest

s se

para

tesystems that exert simultaneous mutual influence (d. McClel1and, 1987;

Rumelhart & McClelland, 1981), even th

ough

they

con

stru

ct s

epar

ate

repr

esen

tatio

ns o

f di

ffer

ent k

inds

of

info

rmat

ion.

In

the

pres

ent a

ppro

ach,

ther

e is

but

a s

ingl

e in

tegr

ated

sys

tem

in w

hich

syn

tact

ic a

nd o

ther

cons

trai

nts

are

com

bine

d in

the connection weights, to influence the

cons

truc

tion

of a

sin

gle

repr

esen

tatio

n re

flect

ing

the

influ

ence

s of

syn

tact

ic,

sem

antic

, and

lexi

cal c

onst

rain

ts.

Neu

rops

ycho

logi

cal D

isso

ciat

ions

Thi

s in

tegr

ativ

e ap

proa

ch is

act

ually

qui

te d

iffe

rent

fro

m th

e po

sitio

n on

eof us has taken in previous pu

blic

atio

ns (

McC

lella

nd, 1

987)

. We

have

adopted it here, not out of any strong

a priori

com

mitm

ent,

but b

ecau

se it

has

turn

ed o

ut to

wor

k w

ell i

n ca

ptur

ing

the

phen

omen

a co

nsid

ered

in th

ispaper. Indeed

, the

not

ion

that

ther

e is

a s

epar

ate

mod

ule

for

synt

ax is

so

ingr

aine

d in

theo

retic

al tr

eatm

ents

of

lang

uage

pro

cess

ing,

that

it is

dif

-

5133

0M

cCLE

LLA

ND

. ST

, JO

HN

. TA

RA

BA

NSE

NT

EN

r:E

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

331

ficul

t eve

n fo

r us

to b

e fu

lly c

omfo

rtab

le w

ith a

band

onin

g it.

But

, the

succ

esse

s of

the

SG m

odel

in d

ealin

g w

ith s

ome

of th

e ce

ntra

l dif

ficu

lties

faci

ng c

onve

ntio

nal a

ppro

ache

s , c

oupl

ed w

ith th

e fa

ct th

at th

e em

piric

alevidence is beginning to favour at le

ast s

ome

form

of

an in

tera

ctiv

eac

coun

t, m

akes

us

feel

that

it is

wor

thw

hile

to s

ee if

inde

ed th

ere

is a

ny r

eal

basi

s fo

r th

is v

ery

gene

ral a

ccep

tanc

e of

som

e fo

rm o

f m

odul

arity

,In

this

con

nect

ion,

it is

wor

th c

onsi

deri

ng e

vide

nce

from

neu

rops

ycho

l-og

y, b

ecau

se s

ome

of th

e m

ost o

ften-

cite

d ev

iden

ce fo

r th

e vi

ew th

at th

ere

are

sepa

rate

pro

cess

ing

syst

ems

for

synt

actic

and

con

cept

ual i

nfor

mat

ion

com

e fr

om n

euro

psyc

holo

gica

l dis

soci

atio

ns, I

t is

gene

rally

cla

imed

, for

exam

ple

, tha

t Wer

nick

es

apha

sics

hav

e a

gene

ral d

efic

it in

the

com

preh

en-

sion of word and sentence meaning, which in

terf

eres

with

thei

r un

der-

stan

ding

of

all s

ente

nces

reg

ardl

ess

of th

eir

synt

actic

com

plex

ity, w

here

asB

roca

s ap

hasi

cs h

ave

a specific deficit in the ability to m

ake

use

ofsy

ntac

tic in

form

atio

n fo

r co

mpr

ehen

sion

, Suc

h di

ssoc

iatio

ns in

vite

a m

odu-

lari

st a

ppro

ach,

in w

hich

one

par

t of

the

syst

em is

spe

cial

ised

for

the

use

ofco

nten

t inf

orm

atio

n an

d th

e ot

her

for

the

use

of s

ynta

ctic

cue

s in

com

pre-

hens

ion,

Cou

ld s

uch

findi

ngs

poss

ibly

be

cons

iste

nt w

ith th

e fr

amew

ork

considered here

, in

whi

ch s

ynta

ctic

and

con

tent

-bas

ed in

flue

nces

on

pro-

cess

ing

are

inex

tric

ably

inte

rtw

ined

in th

e st

ruct

ure

of th

e la

ngua

ge p

roce

s-si

ng m

echa

nism

?In

fac

t, th

e no

tion

that

the

diff

eren

ce b

etw

een

Wer

nick

es

and

Bro

caaphasics can be characterised in terms of syntax and semantics is being

calle

d in

to q

uest

ion

from

sev

eral

dif

fere

nt v

anta

ge p

oint

s, F

irst

, Milb

erg,

Blu

mst

ein

, and

Dw

oret

zky

(198

8) h

ave

rece

ntly

rep

orte

d th

at b

oth

Wer

-ni

cke

s an

d B

roca

s ap

hasi

cs d

iffe

r fr

om n

orm

als

in le

xica

l acc

ess,

thou

ghth

e di

ffer

ence

s ar

e co

mpl

emen

tary

, Nor

mal

s sh

ow a

gra

ded

decr

emen

t in

prim

ing

as p

rim

es a

re in

crea

sing

ly d

isto

rted

, but Broca

s ap

hasi

cs s

how

prim

ing

only

whe

n th

e pr

ime

is u

ndis

tort

ed, a

nd W

erni

cke

s ap

hasi

cs s

how

prim

ing

over

a w

ider

ran

ge o

f di

stor

tion

than

nor

mal

s, T

his

sugg

ests

that

Wer

nick

es

apha

sics

may

be

suff

erin

g fr

om s

omet

hing

aki

n to

und

ampe

dac

tivat

ion,

whe

reas

Bro

cas

apha

sics

are

suf

feri

ng f

rom

ove

rdam

ping

,O

ther

stu

dies

sug

gest

that

Bro

cas

and

Wer

nick

es

apha

sics

bot

h sh

owco

mpr

ehen

sion

def

icits

, and

that

the

defi

cts

diff

er m

ore

betw

een

apha

sics

who

spe

ak d

iffe

rent

lang

uage

s th

an th

ey d

iffe

r be

twee

n di

ffer

ent t

ypes

of

apha

sics

who

spe

ak th

e sa

me

lang

uage

, For

exa

mpl

e, B

ates

, Frie

deric

i,M

icel

i, an

d W

ulfe

ck (

1985

) st

udie

d gr

oups

of

Bro

cas

apha

sics

, Wer

nick

eap

hasi

cs, a

nd n

orm

al c

ontr

ols

who

wer

e na

tive

spea

kers

of e

ach

of th

ree

diff

eren

t lan

guag

es~E

nglis

h, G

erm

an a

nd I

talia

n, T

hey

foun

d th

at w

ithin

each

lang

uage

, Bro

cas

and

Wer

nick

es

apha

sics

bot

h sh

owed

def

icits

in th

eus

e of

mor

phol

ogic

al c

ues,

and

that

the

degr

ee o

f pr

eser

vatio

n of

the

use

ofth

ese

cues

cor

rela

ted

with

the

exte

nt o

f re

lianc

e on

thes

e cu

es in

the

spea

ker

s la

ngua

ge, I

talia

ns r

ely

muc

h m

ore

on a

gree

men

t and

muc

h le

sson

wor

d or

der

than

Eng

lish

spea

kers

, and

the

Ital

ian

apha

sics

sho

wed

the

leas

t im

pairm

ent i

n th

e us

e of

sub

ject

-ver

b ag

reem

ent t

o m

ark

agen

cy,

whe

reas

Eng

lish

apha

sics

sho

wed

the

grea

test

impa

irm

ent,

Ger

man

isin

term

edia

te b

etw

een

the

two

lang

uage

s in

the

exte

nt o

f nor

mal

rel

ianc

eon word order

vs agreement cues

, and

sho

wed

an

inte

rmed

iate

deg

ree

ofdi

srup

tion

of th

e us

e of

agr

eem

ent w

ith d

amag

e, T

he f

indi

ngs

of th

is s

tudy

are

cons

iste

nt w

ith th

e id

ea th

at b

oth

apha

sic

grou

ps s

how

the

grea

test

defi

cits

in th

e us

e of

cue

s th

at a

re r

elat

ivel

y w

eake

r in

thei

r na

tive

lang

uage

(Bat

es &

Wul

feck

, in

pres

s; M

cDon

ald

and

Mac

Whi

nney

, in

pres

s), a

ndtend to run counter to the notion that Broca

s an

d W

erni

cke

s aphasia

diff

eren

tially

impa

ct m

echa

nism

s sp

ecif

ic to

syn

tact

ic a

nd s

eman

tic a

spec

tsof

com

preh

ensi

on, r

espe

ctiv

ely.

We

do n

ot m

ean

to s

ugge

st th

at th

ere

is n

o ba

sis

at a

ll fo

r th

e id

ea th

atth

ere

may

be

spec

ific

diss

ocia

tions

of a

spec

ts o

f lin

guis

tic k

now

ledg

e th

atca

ll in

to q

uest

ion

the

idea

that

con

tent

and

syn

tact

ic c

onst

rain

ts a

re a

s fu

llyin

tegr

ated

as

they

are

in th

e ap

proa

ch th

at w

e ha

ve ta

ken,

The

re a

rese

vera

l stu

dies

whi

ch s

uppo

rt th

e id

ea th

at th

ere

are

part

icul

ar d

efic

its in

the use of closed-cl

ass

wor

ds th

at a

re r

estr

icte

d to

Bro

cas

and

not t

oW

erni

cke

s ap

hasi

cs, w

hich

hav

e ye

t to

be r

econ

cile

d w

ith th

e ty

pe o

fac

coun

t sug

gest

ed b

y M

ilber

g et

al.

' s f

indi

ngs,

as

wel

l as

othe

r ev

iden

cew

hich

has

bee

n ta

ken

as fa

vour

ing

the

exis

tenc

e of

aut

onom

ous

synt

actic

stru

ctur

es (

Cap

lan

& H

ildeb

rand

t , 1

988)

, Our

onl

y cl

aim

her

e is

that

the

neur

opsy

chol

ogic

al e

vide

nce

is n

ot c

ompl

etel

y cl

ear-

cut,

and

ther

e is

roo

mfo

r a

cons

ider

atio

n of

the

idea

that

ther

e m

ay in

deed

be

a si

ngle

pro

cess

ing

syst

em th

at is

sim

ply

thro

wn

out o

f reg

ulat

ion

in s

light

ly d

iffer

ent w

ays

inB

roca

s an

d W

erni

cke

s ap

hasi

cs, T

he m

odel

we

have

pro

pose

d do

es n

ot o

fco

urse

off

er a

ny in

sigh

t int

o th

is d

iffe

rent

ial d

isru

ptio

n, b

ut th

e m

odel

isco

mpa

tible

with

the

idea

that

ther

e is

a s

ingl

e sy

stem

whi

ch u

ses

synt

ax a

ndcontent together to guide the language co

mpr

ehen

sion

pro

cess

,

FU

TU

RE

DIR

EC

TIO

NS

In th

is p

aper

, we

have

des

crib

ed a

n al

tern

ativ

e to

trad

ition

al m

odel

s of

lang

uage

pro

cess

ing,

We

have

trie

d to

indi

cate

how

this

alte

rnat

ive

may

allo

w u

s to

sol

ve m

any

of th

e pr

oble

ms

faci

ng tr

aditi

onal

app

roac

hes,

and

how

it m

ay p

rovi

de d

iffe

rent

way

s of

con

cept

ualis

ing

basi

c as

pect

s of

the

prob

lem

of

com

preh

ensi

on, W

e ha

ve a

lso indicated that m

any

of th

ear

gum

ents

aga

inst

the

type

of

appr

oach

we

have

take

n ca

n be

cou

nter

ed,

Of

cour

se, t

he fa

cts

are

not a

ll in

, but

giv

en w

hat i

s kn

own

at th

is ti

me

the

appr

oach

see

ms

to u

s to

be

at le

ast a

s vi

able

as

any

othe

r th

at w

e kn

ow o

f.T

he m

odel

we

have

offe

red

is fa

r fr

om th

e fin

al w

ord

, and

man

y pr

oble

ms

5133

2M

cCLE

LLA

ND

. ST

, JO

HN

, TA

RA

BA

NSENTENCE COMPREHENSION AND PDP

5133

3ne

ed to

be

addr

esse

d, O

ur o

nly

goal

has

bee

n to

sug

gest

that

ther

e m

ay b

eso

me

basi

s fo

r op

timis

m th

at f

urth

er d

evel

opm

ent o

f th

e ap

proa

ch m

ight

be s

ucce

ssfu

l,T

here

are

sev

eral

fur

ther

ste

ps th

at n

eed

to b

e ta

ken,

Fir

st, w

e ne

ed to

find

way

s of

impr

ovin

g th

e ra

te o

f lea

rnin

g; a

s th

ings

sta

nd, l

earn

ing

isunduly slow

, esp

ecia

lly g

iven

the

smal

l siz

e of

the

corp

ora

that

we

have

used

in o

ur tr

aini

ng e

xper

imen

ts, S

econ

dly,

we

need

to e

xten

d th

e fr

ame-

wor

k to

for

ce th

e co

nstr

uctio

n of

rep

rese

ntat

ions

that

can

ans

wer

mor

eso

phis

ticat

ed q

uest

ions

than

mer

ely

the

com

plet

ion

of r

ole-

fille

r pa

irs,

As

prev

ious

ly n

oted

, the

rol

e-fil

ler

com

plet

ion

task

that

we

have

use

d he

rehas several limitations; the role-fi

ller

pair

lang

uage

is in

suff

icie

ntly

str

uc-

ture

d, a

nd th

e lo

calis

t rep

rese

ntat

ion

of c

once

pts

lack

s th

e re

lianc

e on

distributed representations which is one of the st

reng

ths

of th

e PD

Pfr

amew

ork.

Thi

rdly

, our

long

- ter

m g

oal i

s to

mov

e in

the

dire

ctio

n of

capturing the influence of broader

, ext

ra-s

ente

ntia

l con

text

on

sent

ence

proc

essi

ng. U

ltim

atel

y, th

e ap

proa

ch w

ill s

tand

or

fall

on it

s ab

ility

toca

ptur

e th

e pe

rvas

ive

influ

ence

s of

thes

e ex

tra-

sent

entia

l fac

tors

.

RE

FER

EN

CE

S

lang

uage

par

. ring

: Psy

chol

ogic

al. c

ompu

tatio

nal , and theoretical perspeclives, C

ambr

idge

:C

ambr

idge

Uni

vers

ity P

ress

,Elman, J, L. (1

988)

, Fin

ding

str

uctu

re in

tim

e, C

RL

Tec

hnic

al R

epor

t 880

\. C

ente

r fo

rR

esea

rch

in L

angu

age.

Uni

vers

ity o

f C

alif

orni

a , S

an D

iego

,E

lman

, J, L

. (19

89),

Rep

rese

ntat

ion

and

stru

ctur

e in

con

nect

ioni

st m

odel

s, C

RL

Tec

hnic

alR

epor

t 890

3, C

ente

r fo

r R

esea

rch

in L

angu

age, Universityof California, San Diego,

Elman, J, L. & McClelland, J, L. (1986), Exploiting the lawful va

riab

ility

in th

e sp

eech

wave. In J, S, Perkell & D, H, Klatt (Eds),

In v

aria

nce

and

vari

abi/i

ly o

f sp

eech

pro

cess

es,

Hill

sdal

e. N

.J,:

Law

renc

e E

rlba

um A

ssoc

iate

s In

c,E

rick

son

. T, D

, & Mattson, M, E, (1

981)

, Fro

m w

ords

to m

eani

ng: A

sem

antic

illu

sion

.Jo

urna

l of V

erb,

II Le

arni

ng a

nd V

erba

l Beh

avio

r . 2

0,

54()

- 55\

.Fe

rrei

ra. F

, & C

lifto

n. C

. (19

86),

The

inde

pend

ence

of

synt

actic

proc

essi

ng,

Journal of

Memory and Language.

25. 3

48- 3

68,

Fillm

ore.

C. (

1968

), T

he c

ase

for

case

, In

E, B

ach

& R

, T, H

arms (Eds),

Uni

vers

als

inlinguistic theory, N

ew Y

ork:

Hol

t, R

ineh

art a

nd W

inst

on,

Fodo

r , J

, A, &

Pyl

yshy

n. Z

. W, (

1988

), C

onne

ctio

nism

and

cog

nitiv

e ar

chite

ctur

e: A

crit

ical

anal

ysis

, C

ogni

tion,

, 3

-7\.

Frazier, L. (1986), Theories of sentence processing, In J, Garfield (Ed,

),

Mod

ular

ity in

knowledge representation and natural language processing,

Cam

brid

ge, M

ass,

: MIT

Pre

ss,

Gle

itman

. L. R

, & W

anne

r, E

, (19

82),

Lan

guag

e ac

quis

ition

: The

sta

te o

f th

e st

ate

of th

ear

t, In

E, W

anne

r &

L. R

, Gleitman (Eds),

Lan

guag

e ac

quis

ition

: The

sta

te o

f th

e ar

/.C

ambr

idge

: Cam

brid

ge U

nive

rsity

Pre

ss,

Hinton. G, E. (1986), Learning distributed representations of concepts,

Proc

eedi

ngs

of th

eEighlh Annual Conference of Ihe Cognitive Science Society,

Amherst, Mass,

Hinton. G, E" McClelland

. J, L

.. &

Rum

elha

rt, D, E, (1986), Distributed representations,

In D, E, Rumelhart. J, L, McClelland, & the PDP Research Group (Eds).

Para

llel

distributed processing: Explorations in the microstructure of cognition,

Vol

, I. C

ambr

idge

.M

ass,

: Bra

dfor

d B

ooks

.Ju

st, M

, A"

Car

pent

er, P

. A,.

& W

oole

y. J

, D, (

1982

), P

arad

igm

s an

d pr

oces

ses

in r

eadi

ngco

mpr

ehen

sion

, Journal of Experimental Psychology: General, III,

228-

238,

Kaw

amot

o. A

, H, (

1985

), D

ynam

ic p

roce

sses

in th

e (r

e)so

lutio

n of

lexical ambiguity,

Doc

tora

l dis

sert

atio

n. D

epar

tmen

t of P

sych

olog

y. B

row

n U

nive

rsity

,K

awam

oto

. A, H

, (19

88),

Dis

trib

uted

repr

esen

tatio

ns o

f am

bigu

ous

wor

ds a

nd th

eir

reso

lu-

tion

in a

con

nect

ioni

st n

etw

ork,

In

S, L

. Sm

all,

G, W

, Cot

trel

l. &

M, K

, Tan

enha

us (

Eds

),L

exic

al a

mbi

guilY

res

olur

ion:

Per

spec

ti,' e

s fr

om p

sych

olin

guis

tics.

neu

rop.

rych

olog

y. a

ndar/ificial intelligence,

San

Mat

eo. C

alil.

: Mor

gan

Kau

fman

n,Marcus. M, p,

(198

0),

A theory of syntaclic recognition for natural language,

Cam

brid

ge.

Mas

s,: M

IT P

ress

,M

cCle

lland

. J, L

. (19

79),

On

the

time

rela

tions

of m

enta

l pro

cess

es: A

n ex

amin

atio

n of

systems of processes in cascade,

Psy

chol

ogic

al R

evie

w, 2

87-3

30,

McC

lella

nd, J

, L. (

1987

), T

he c

ase

for

inte

ract

ion ism in language processing, In M, Col-

theart (Ed,

).

Atte

ntio

n an

d pe

rfor

man

ce X

I/: T

he psychology of reading.

pp, 1-

36,

Lon

don:

Law

renc

e E

rlba

um A

ssoc

iate

s L

td.

McC

lella

nd, J

. L. (

in p

ress

), P

aral

lel d

istributed processing: Implications for cognition and

development, In R, Morris (Ed,

),

Parallel dislribured processing: implicalions for

psychology and neurobiology,

Oxf

ord:

Oxf

ord

Uni

vers

ity P

ress

,McClelland, J, L. & Elman. J. L. (1

986)

, An

arch

itect

ure

for

para

llel p

roce

ssin

g in

spe

ech

reco

gniti

on: T

he T

RA

CE

mod

el, I

n M

, R, S

chro

eder

(E

d,

),

Speech recognition, Basel: S.

Kra

ger

AG

,M

cCle

lland

, J, L

. & K

awam

oto

. A, H

. (19

86).

Mec

hani

sms

of s

ente

nce

proc

essi

ng: A

ssig

n-in

g ro

les

to c

onst

ituen

ts, I

n J,

L. M

cCle

lland

, D, E

, Rum

elha

rt. &

the

PDP

Res

earc

h

Allen. R, B, (1987), Several studies on na

tura

l lan

guag

e an

d ba

ck- p

ropa

gatio

n,

Proc

eedi

ngs

of th

e IE

EE

Fir

st A

nnua

llnte

mal

iOlla

l Con

fere

nce on Neural Networks,

Vol. 11

287-

298,

Bos

ton,

Alle

n, R

, B. (

1988

), S

eque

ntia

l con

nect

ioni

st n

etw

orks

for

ans

wer

ing simple questions

about a microworld,

Proc

eedi

ngs

of I

he T

etlth

Ann

ual C

onfe

renc

e of

Ihe

Cog

niliv

e Sc

ienc

eSo

ciet

y.

pp, 4

89-4

95, M

ontr

eal.

Altmann. G, (1988), Ambiguity. parsing strategies. and computational models,

Lan

guag

eand Cognilive Processes,

3. 7

3-97

,Altmann. G, & Steedman. M, (1

988)

, Int

erac

tion

with

con

text

dur

ing

hum

an s

ente

nce

proc

essi

ng,

Cog

nilio

n. 3

019

1-23

8,B

ates

. E, &

Wul

feck

. B, (

in p

ress

), C

ross

lingu

istic

stu

dies

of

apha

sia,

In

B, M

acW

hinn

ey &

E, Bates (Eds).

The crosslinguislic study of sentence processing.

New

Yor

k: C

ambr

idge

Uni

vers

ity P

ress

,Bates. E,

. Frie

deric

i. A

.. M

icel

i, G

, . & Wulfeck. B, (1985), Sentence comprehension in

apha

sia:

A c

ross

- lin

guis

tic s

tudy

, Man

uscr

ipt.

Uni

vers

ity o

f C

alif

orni

a, S

an D

iego

.Bever. T, G. (1

970)

, The

cog

nitiv

e ba

sis

of, l

ingu

istic

str

uctu

res.

In J

, R, H

ayes

(E

d,Cognition and the development of language,

New

Yor

k: J

ohn

Wile

y,C

apla

n, D

, & H

ildeb

rand

t. N

, (19

88),

Disorders of syntactic comprehension,

Cam

brid

ge,

Mas

s,: M

IT P

ress

,C

hom

sky.

N, (

1988

), L

ectu

re p

rese

nted

at t

he U

nive

rsity

of P

ittsb

urgh

. Fal

l. 19

88,

Clark. H, H, & Clark, E, V, (1977),

Psyc

holo

gy a

nd la

ngua

ge: A

n in

trod

uctio

n to

psy

cho-

lingu

islic

s.

New

Yor

k: H

arco

urt B

race

Jov

anov

ich,

Cra

in, S

, & S

teed

man

, M, J

, (19

85),

On

not b

eing

led

up th

e ga

rden

pat

h: T

he u

se o

f co

ntex

tby

the

psyc

holo

gica

l par

ser.

In

D, D

owty

, L. K

arttu

nen,

& A, Zwicky (Eds).

Nat

ural

5133

4M

cCLE

LLA

ND

. ST

. JO

HN

. TA

RA

BA

NSE

NT

EN

CE

CO

MPR

EH

EN

SIO

N A

ND

PD

P51

335

Group (Eds).

Para

llel d

istr

ibut

ed p

roce

.uin

g: E

xplo

ratio

n.in

th

e m

icro

stru

clllr

e of

cog

ni-

tion.

V

ol. I

I. C

amhr

idge

, Mas

s.: B

radf

ord

Boo

ks.

McC

lella

nd, J

. L..

& R

umel

hart

, D. E

. (19

81).

An

inte

ract

ive

activ

atio

n m

odel

of

cont

ext

effecl~ in letter perception, Part I: An account of basic findings. Psy

chol

ogic

al R

evie

w, 8

11,

375-

407.

McDonald. J. & M

acW

hinn

ey. B

. (in

pre

ss).

Max

imum

like

lihoo

d m

odel

s fo

r se

nten

ceproces~ing. In B. MacWhinney & E. Bates (Eds).

The

cro

sslin

guis

tic s

tudy

of

sent

ence

prou

.uin

K.

New

Yor

k: C

amhr

idge

Uni

vers

ity P

ress

.M

iikku

lain

en. R

. & D

yer,

M. G

. (19

89).

A m

odul

ar n

eura

l net

wor

k ar

chite

ctur

e fo

rse

quen

tial p

arap

hras

ing

of s

crip

t-ba

sed

stor

ies.

Tec

hnic

al R

epor

t UC

LA

-AI-

89-0

2. U

ni-

vcr~

ity o

f C

alif

orni

a. A

rtif

icia

l Int

ellig

ence

Lah

oral

ory,

Com

pute

r Sc

ienc

e D

epar

tmen

t,Lo

~ A

ngel

es.

Milh

erg.

W..

Blu

mst

ein.

S..

& D

wor

etzk

y. B

. (19

88).

Pho

nolo

gica

l pro

cess

ing

and

lexi

cal

access in aphasia. Brain and Language,

, 279

-293

.Oden. G. (1

'!7R

). S

eman

tic c

onst

rain

ts a

nd ju

dged

pre

fere

nce

for

inte

rpre

tatio

ns o

f am

hi-

guous sentences.

Memory and CoKnition,

26-3

7.Rayner. K.

. Car

lson

. M..

& F

razi

er. L

.. (1

983)

. The

inte

ract

ion

of s

ynta

x an

d se

man

tics

during sentence pr

oces

sing

: Eye

movements in the analysis of semantically biased

senl

ence

s.

J(I/Irnal of Verbal Learning and Verbal Behavior,

22. 3

58-3

74Rumelhart. D. E. & McClelland. J.

L..

(198

1). I

nter

activ

e pr

oces

sing

thro

ugh

spre

adin

gactivation. In C. Perfetti & A. Lesgold (Eds),

'nle

ract

ive

prou

. ,se.

in

read

ing.

H

illsd

ale.

: Law

renc

e E

rlha

um A

ssoc

iate

s In

c.Rumelharl. D. E. & McClelland. J.

L..

(198

2). A

n in

tera

ctiv

e ac

tivat

ion

mod

el o

f co

ntex

tef

fect

s in

lette

r pe

rcep

tion.

Par

I II

: The

con

text

ual e

nhan

cem

ent e

ffec

t and

som

e te

sts

and

extensions of the model.

Psy

chol

ogic

al R

evie

w89

. 60-

94.

Rum

elha

rt. D. E.. Hinton. G. E., & Williams. R. J. (19R6a). L

earn

ing

inte

rnal

rep

rese

nta-

tions

by

erro

r pr

opag

atio

n. I

n D

. E. R

umel

hart

, J. L

.. M

cCle

lland

, & th

e PO

P R

esea

rch

Group (Eds),

Par

alle

l dis

trib

uted

pro

uuin

g: E

xplo

ratio

ns

in the microstructure of COKni-

lion

Vol

. I. C

ambr

idge

, Mas

s.: M

IT P

ress

.R

umel

hart

. D. E

.. Sm

olen

sky.

P.. McClelland, J. L.. & Hinton, G

. E. (

1986

b). P

aral

lel

dist

rihu

ted

proc

essi

ng m

odel

s of

sch

emat

a an

d se

quen

tial t

houg

htpr

oces

ses.

In

J. L

.McClelland. D. E. Rumelharl. & the POP Research Group (Eds),'

Parallel distributed

proccuing: Explorations

in the micro..rructure of cognition. V

ol. I

I. C

ambr

idge

, Mas

s.Bradford Books.

St. J

ohn,

M. F

. (1988). Hitting the right pitch: A meta-analysis of the processing of

amhi

guou

s w

ords

in c

onte

xt. M

anus

crip

t.S

t. Jo

hn, M

. F. &

McC

lella

nd, J

. L..

(198

7). R

econ

stru

ctiv

e m

emor

y fo

r se

nten

ces:

A I

'DI'

appr

oach

. Pap

er p

rese

nted

to th

e O

hio

Uni

vers

ity In

fere

nce

Con

fere

nce.

Pr

ocee

ding

s'nference: OUlC

86. U

nive

rsity

of

Ohi

o, A

then

s, O

hio.

St. John. M. F. & McClelland. J. L. (in press). Learning and applying contextual co

nstr

aint

sin sentence comprehension.

Art

ific

ial'n

telli

genc

e.Schank. R. C. (1

981)

. Lan

guag

e an

d m

emor

y. I

n D

. A. N

orm

an (

Ed.

),

Pers

pect

ives

on

cognitivl' science. N

orw

ood,

N. J

.: A

blex

.Se

rvan

-Sch

reih

er,

D.,

Cleeremans, A., & McClelland, J. L. (1

988)

. Enc

odin

g se

quen

tial

stru

ctur

e in

sim

ple

recu

rren

t net

wor

ks. C

MU

-CS-

88-1

83. C

arne

gie

Mel

lon

Uni

vers

ity.

Com

pute

r S

cien

ce D

epar

tmen

t, P

ittsb

urgh

.Sw

inne

y. D

. A. (

1979

). L

exic

al a

cces

s du

ring

sen

tenc

e co

mpr

ehen

sion

: (R

e)co

nsid

erat

ion

ofcontext effects. Journal of Verbal Learning and Verbal Behavior

18, 6

45-6

59.

Tan

enha

us. M

. K.,

Leim

an. J

. M.,

& S

eide

nber

g. M

. S. (

1979

). E

vide

ncef

or m

ultip

le s

tage

sin the processing of ambiguous words in syntactic contexts.

Journal of Verbal Learning

and Verbal Behavior.

'8, 4

27-

440.

Taraban. R. (1

988)

. Con

tent

- bas

ed e

xpec

tatio

ns: O

ne s

ourc

e of

gui

danc

e fo

r sy

ntac

ticat

tach

men

t and

them

atic

rol

e as

sign

men

t in sentence processing. Ph.

D. d

isse

rtat

ion.

Car

negi

e M

ello

n U

nive

rsity

, Pitt

sbur

gh.

Tar

aban

. R. & McClelland, J. L. (1988). Constituent attachment and thematic role assign-

men

t in

sent

ence

pro

cess

ing:

Inn

uenc

es o

f co

nten

t-based expectations.

Jour

nal o

f Mem

orv

and Language,

. 597

- 632.

T

arab

an, R

. & M

cCle

lland

, J. L

. (19

90).

Par

sing

and

com

preh

ensi

on: A

mul

tiple

-con

stra

int

view. In D. Balota, G. B. Flores d'Arcais, & K. Rayner (Eds).

Com

preh

ensi

on p

rOC

l'ues

in

read

ing.

H

illsd

ale,

N.

: Law

renc

e E

rlba

um A

ssoc

iate

s In

c.T

oure

tzky

. D. S

. (19

88).

Con

nect

ioni

sm a

nd P

I' at

tach

men

t. In

D. T

oure

tzky

. G. H

into

n

. &

T. Sejnowski (Eds),

Procl'edings of the

1988

Connectionist Mosels.

San

Mat

eo. C

alif.

:Morgan Kaufmann.

Tou

retz

ky, D

. S. &

Gev

a, S

. (19

87).

A d

istr

ibut

ed connectionist representation for concept

stru

ctur

es. P

aper

pre

sent

ed to

the

9th Annual Conference of the Cognitive Science

Soci

ety.

Sea

ttle.

Tou

retz

ky, D

. S. &

Hin

ton

, G. E

. (19

88).

A d

istr

ibut

ed connectionist production system.

Cog

nitiv

e Sc

i/'nc

e12

, 423

- 466

.va

n G

elde

r, T

. (in

pre

ss).

Com

posi

tiona

lity:

Var

iatio

n on

a classical theme.

Cog

nitiv

eSc

ienc

e.