Prefro

ntal

Cortex

:Delay

-relatedactiv

ity

Spatial

delay

ed-resp

onse

task;F

unah

ashiet

al,1989

Rem

ember

theSRN?(ch

ap6)

Rem

ember

theSRN?(ch

ap6)

thisisagatin

gnetw

ork:contex

tonly

updated

atdiscrete

timep

oints

Sim

ple

SRN

story

isnotflaw

less

•How

ishidden

→“co

py”functio

nim

plem

ented

biologically

?

•Durin

gsettlin

g,co

ntex

tmustbeactiv

elymain

tained

(ongoinghidden

activity

has

noeffect

oncontex

t).

•Assu

mes

allcontex

tisrelev

ant:What

ifdistractin

ginform

ation

presen

tedin

middle

ofseq

uen

ce?Wan

tto

only

hold

onto

relevan

t

contex

t.

•What

ifwan

tto

hold

onto

more

than

onepiece

ofinform

ationin

WM

atatim

e??Orto

selectively

update

onepart

ofWM

while

contin

uing

torobustly

main

tainothers?

•Andwhat

ifthedecisio

nofwheth

erornotto

update

inform

ation

dep

endsoncu

rrently

intern

alWM

state?

Sch

izophren

ia:Im

paired

PFC/BG

Functio

n

•SZPatien

tshav

edefi

citsin

thesam

ecognitiv

etask

sas

those

seenin

patien

tswith

PFCdam

age.

Sch

izophren

ia:Im

paired

PFC/BG

Functio

n

•SZPatien

tshav

edefi

citsin

thesam

ecognitiv

etask

sas

those

seenin

patien

tswith

PFCdam

age.

•ButPFCpatien

tsdon’thav

edelu

sions,hallu

cinatio

ns,p

sych

osis.

Also

someim

agingstu

dies

show

increase

inPFCactiv

ityin

SZ

(e.g.,M

anoach

,

2003;Calico

ttet

al,2003)

Sch

izophren

ia:Im

paired

PFC/BG

Functio

n

•SZPatien

tshav

edefi

citsin

thesam

ecognitiv

etask

sas

those

seenin

patien

tswith

PFCdam

age.

•ButPFCpatien

tsdon’thav

edelu

sions,hallu

cinatio

ns,p

sych

osis.

Also

someim

agingstu

dies

show

increase

inPFCactiv

ityin

SZ

(e.g.,M

anoach

,

2003;Calico

ttet

al,2003)

•Psych

osis

thoughtto

stemfro

mincreased

DA

inBG

(Wein

berg

er,

O’D

onnell,

Grace,etc)

Sch

izophren

ia:Im

paired

PFC/BG

Functio

n

•SZPatien

tshav

edefi

citsin

thesam

ecognitiv

etask

sas

those

seenin

patien

tswith

PFCdam

age.

•ButPFCpatien

tsdon’thav

edelu

sions,hallu

cinatio

ns,p

sych

osis.

Also

someim

agingstu

dies

show

increase

inPFCactiv

ityin

SZ

(e.g.,M

anoach

,

2003;Calico

ttet

al,2003)

•Psych

osis

thoughtto

stemfro

mincreased

DA

inBG

(Wein

berg

er,

O’D

onnell,

Grace,etc)

•Other

diso

rders

with

BG/DA

dysfu

nctio

nare

associated

with

frontal-lik

ecognitiv

edefi

cits(PD,ad

dictio

n,A

DHD,etc);th

esedefi

cits

correlate

w/DA

inBG

(Muller

etal,2000;R

emyet

al,2000)

Widely

Accep

tedRole

ofPrefro

ntal

Cortex

(PFC)

PFChelp

skeep

usontask

,promotes

cognitiv

eflexibility

via

ability

to

rapidly

switch

state(e.g

.,task

switch

ing):

•Robust

main

tenan

ceofneu

ralactiv

ity(w

orkingmem

ory)

Widely

Accep

tedRole

ofPrefro

ntal

Cortex

(PFC)

PFChelp

skeep

usontask

,promotes

cognitiv

eflexibility

via

ability

to

rapidly

switch

state(e.g

.,task

switch

ing):

•Robust

main

tenan

ceofneu

ralactiv

ity(w

orkingmem

ory)

•Flex

ibility

from

adap

tivegatin

g(via

Basal

Gan

glia,B

G)that

switch

es

betw

eenmain

tenan

cean

drap

idupdatin

g

Widely

Accep

tedRole

ofPrefro

ntal

Cortex

(PFC)

PFChelp

skeep

usontask

,promotes

cognitiv

eflexibility

via

ability

to

rapidly

switch

state(e.g

.,task

switch

ing):

•Robust

main

tenan

ceofneu

ralactiv

ity(w

orkingmem

ory)

•Flex

ibility

from

adap

tivegatin

g(via

Basal

Gan

glia,B

G)that

switch

es

betw

eenmain

tenan

cean

drap

idupdatin

g

•Exten

siveinterco

nnectiv

ity,allo

wingtop-downbiasin

goftask

-relevan

t

info

inother

cortical

/su

bcortical

areas.

Insh

ort,P

FCisthe“cen

tralexecu

tive!”

WorkingMem

ory

Dem

ands:Updatin

g&

Main

tenan

ce

Hoch

reiter&

Sch

midhuber,

1997;Brav

er&

Cohen

,1999;Fran

ket

al,2001

Sensory

Input

Working

Mem

ory

Gating

open

a) Update

My

phone # is371−

9624

371−9624

•Workingmem

ory:robust

main

tenan

ceofinform

ation,b

utmustalso

hav

eab

ilityto

berap

idly

updated

—req

uires

gatin

g.

•You’vegotto

know

when

tohold

’em,k

now

when

tofold

’em.

WorkingMem

ory

Dem

ands:Updatin

g&

Main

tenan

ce

Hoch

reiter&

Sch

midhuber,

1997;Brav

er&

Cohen

,1999;Fran

ket

al,2001

Sensory

Input

Working

Mem

ory

Gating

open

a) Update

closed

Should I

Stay or

Should I

Go..?

b) Maintain

My

phone # is371−

9624

371−9624

371−9624

•Workingmem

ory:robust

main

tenan

ceofinform

ation,b

utmustalso

hav

eab

ilityto

berap

idly

updated

—req

uires

gatin

g.

•You’vegotto

know

when

tohold

’em,k

now

when

tofold

’em.

WorkingMem

ory

Dem

ands:Updatin

g&

Main

tenan

ce

Hoch

reiter&

Sch

midhuber,

1997;Brav

er&

Cohen

,1999;Fran

ket

al,2001

Sensory

Input

Working

Mem

ory

Gating

a) Update

openclosed

Should I

Stay or

Should I

Go..?

b) Maintain

867−5309

Jenny, Igot your # ...

open

371−9624

371−9624

867−5309 ...371−

9624

My

phone # is371−

9624

c) Update: O

OP

S!

•Workingmem

ory:robust

main

tenan

ceofinform

ation,b

utmustalso

hav

eab

ilityto

berap

idly

updated

—req

uires

gatin

g.

•You’vegotto

know

when

tohold

’em,k

now

when

tofold

’em.

Butwhocontro

lsthecontro

ller?

BG

dam

age⇒

defi

citsin

motor,learn

ing,m

otiv

ation,w

orkingmem

ory,co

gnitiv

econtro

l

BG

Model

Exten

sionto

WorkingMem

ory

andAtten

tion

•In

themotordomain

,theBG

selectively

facilitatesonecomman

dwhile

suppressin

gothers

(Mink,1996)

BG

Model

Exten

sionto

WorkingMem

ory

andAtten

tion

•In

themotordomain

,theBG

selectively

facilitatesonecomman

dwhile

suppressin

gothers

(Mink,1996)

•In

parallel

circuits,

theBG

may

reinforce

theupdatin

gofPFCworking

mem

ory

represen

tations(A

lexan

der

etal,1986;F

rank,L

oughry

&O’Reilly,

2001).

BG

Model

Exten

sionto

WorkingMem

ory

andAtten

tion

•In

themotordomain

,theBG

selectively

facilitatesonecomman

dwhile

suppressin

gothers

(Mink,1996)

•In

parallel

circuits,

theBG

may

reinforce

theupdatin

gofPFCworking

mem

ory

represen

tations(A

lexan

der

etal,1986;F

rank,L

oughry

&O’Reilly,

2001).

•Dopam

inein

PFCsu

pports

robust

main

tenan

ceover

time(Lew

is&

O’D

onnell,

2000;Durstew

itz&

Seem

ans,2002)

BG

Model

Exten

sionto

WorkingMem

ory

andAtten

tion

•In

themotordomain

,theBG

selectively

facilitatesonecomman

dwhile

suppressin

gothers

(Mink,1996)

•In

parallel

circuits,

theBG

may

reinforce

theupdatin

gofPFCworking

mem

ory

represen

tations(A

lexan

der

etal,1986;F

rank,L

oughry

&O’Reilly,

2001).

•Dopam

inein

PFCsu

pports

robust

main

tenan

ceover

time(Lew

is&

O’D

onnell,

2000;Durstew

itz&

Seem

ans,2002)

•Phasic

DA

bursts

thoughtto

occu

rfortask

-relevan

t(“p

ositiv

e”)

inform

ation,rein

forcin

gBG

updatin

gsig

nals

(O’Reilly

&Fran

k,2006;F

rank

&O’Reilly,2006).

BG

Model

Exten

sionto

WorkingMem

ory

andAtten

tion

•In

themotordomain

,theBG

selectively

facilitatesonecomman

dwhile

suppressin

gothers

(Mink,1996)

•In

parallel

circuits,

theBG

may

reinforce

theupdatin

gofPFCworking

mem

ory

represen

tations(A

lexan

der

etal,1986;F

rank,L

oughry

&O’Reilly,

2001).

•Dopam

inein

PFCsu

pports

robust

main

tenan

ceover

time(Lew

is&

O’D

onnell,

2000;Durstew

itz&

Seem

ans,2002)

•Phasic

DA

bursts

thoughtto

occu

rfortask

-relevan

t(“p

ositiv

e”)

inform

ation,rein

forcin

gBG

updatin

gsig

nals

(O’Reilly

&Fran

k,2006;F

rank

&O’Reilly,2006).

•Tim

ecourse

ofDA

activity

:main

tenan

cein

PFC,u

pdatin

gthru

BG.

BG

Gatin

gofPFCWorkingMem

ory

/Atten

tion

BG

Disin

hibitio

nofPFC

=Gatin

gofWorkingMem

ory

/Atten

tion

GP

e

excitatoryinhibitorym

odulatory

striatum

IndirectD

irectthalam

us

GP

i

prefrontal cortex

D2

NoGo

SN

c

D1 Go

•Base

state:Thalam

usinhibited

=Gate

isclo

sed

PFCrobustly

main

tainsprio

rstates.

BG

Disin

hibitio

nofPFC

=Gatin

gofWorkingMem

ory

/Atten

tion

GP

e

D2

excitatoryinhibitorym

odulatory

striatum

IndirectD

irectthalam

us

GP

i

D1

prefrontal cortex

Go

NoG

o

SN

c

•Base

state:Thalam

usinhibited

=Gate

isclo

sed

PFCrobustly

main

tainsprio

rstates.

•Striatu

mfires:

Thalam

usdisin

hibited

=Gate

open

edPFCrap

idly

updated

tomain

tainnew

inform

ation.

Parallel

Strip

es=Selectiv

eGatin

g

... PF

C

...

Posterior

Cortex

......thalam

us

Striatum

(matrix)

}G

P (tonic act)

excitatoryinhibitory

{

dis−inhib

PFC/BG

loopsform

indep

enden

tstrip

es=selectiv

egatin

g.

BG

gates

flow:su

perfi

cial→

Deep

PFC

Main

tenan

cevia

Thalam

ocortical

loops,B

Gdisin

hibits

Superfi

cialrefl

ectsinputs

andmain

tenan

ceSep

arateMain

tenan

cevs.OutputPFC/BG

stripes

Multip

lestrip

es,competitio

n

Competitio

nin

striatum/GPbetw

eenMain

tvsOutputan

ddiff

stripes

Exam

ple

Task

:Specifi

cWorkingMem

ory

Dem

ands

Targ

et(R):

1=A-X

2=B-Y

1 L

AX

LR

B L

X LL

B L2

RYtim

e

outer loopinner loops

Mustmain

tainouter

loop(1,2)

while

updatin

ginner

loop(A

,X..).

Exam

ple

Task

:Specifi

cWorkingMem

ory

Dem

ands

Targ

et(R):

1=A-X

2=B-Y

1 L

AX

LR

B L

X LL

B L2

RYtim

e

outer loopinner loops

Mustmain

tainouter

loop(1,2)

while

updatin

ginner

loop(A

,X..).

•R

obustmain

tenan

ce(over

interv

eningstim

uli;o

uter

loop).

Exam

ple

Task

:Specifi

cWorkingMem

ory

Dem

ands

Targ

et(R):

1=A-X

2=B-Y

1 L

AX

LR

B L

X LL

B L2

RYtim

e

outer loopinner loops

Mustmain

tainouter

loop(1,2)

while

updatin

ginner

loop(A

,X..).

•R

obustmain

tenan

ce(over

interv

eningstim

uli;o

uter

loop).

•R

apidan

dselective

updatin

g(keep

1,2while

updatin

gA,X..).

Exam

ple

Task

:Specifi

cWorkingMem

ory

Dem

ands

Targ

et(R):

1=A-X

2=B-Y

1 L

AX

LR

B L

X LL

B L2

RYtim

e

outer loopinner loops

Mustmain

tainouter

loop(1,2)

while

updatin

ginner

loop(A

,X..).

•R

obustmain

tenan

ce(over

interv

eningstim

uli;o

uter

loop).

•R

apidan

dselective

updatin

g(keep

1,2while

updatin

gA,X..).

Man

yreal-w

orld

exam

ples:

languag

e,plan

ning...

PFC/BG

Model

of1-2-A

X(Fran

k,L

oughry

&O’Reilly,

2001)

{

11

a)

FC/BG

Model

of1-2-A

X(Fran

k,L

oughry

&O’Reilly,

2001)

{

11

{

C1

a)b)

C

FC/BG

Model

of1-2-A

X(Fran

k,L

oughry

&O’Reilly,

2001)

{

11

{

C1

{

A1

A

a)b)

c)

C

FC/BG

Model

of1-2-A

X(Fran

k,L

oughry

&O’Reilly,

2001)

{

11

{

C1

{

A1

A

{

X1

AR

a)b)

c)d)

C

ButHow

doBG

“Know”What

toUpdate/

Ignore?

ButHow

doBG

“Know”What

toUpdate/

Ignore?

Correct:

Incorrect:

DA

effectsonBG

Updatin

gOfPFC

DA

Burst:

Increased

Go

DA

Dip:Increased

NoGo

SN

r

Posterior C

ortexF

rontal Cortex

GP

e

D2

−D

1

dorsalstriatum

+ Go

NoG

othalam

usV

A,V

L,MD

SN

c

excitatoryinhibitorym

odulatory

SN

r

Posterior C

ortexF

rontal Cortex

GP

e

D2

−D

1

dorsalstriatum

+ Go

NoG

othalam

usV

A,V

L,MD

SN

c

excitatoryinhibitorym

odulatory

•DA

bursts/

dipsmodulate

Govs.NoGofirin

gan

dlearn

ing.

•Burst

=Excitato

ryD1onGo

Dip

=Release

ofD2inhibitio

nonNoGo

Sam

emech

anism

asin

basic

motorcircu

itry!(Fran

k,05;G

erfen,00)

PFC/BG

Model,L

earns12-A

Xan

dother

WM

tasks

(O’Reilly

&Fran

k,2006)

Hidden

(Posterior

Cortex)

DA

XY

Z

AB

C

12

3

Input

LR

Output

SN

c

SN

rThal

XY

Z

AB

C

12

3

XY

Z

AB

C

12

3

XY

Z

AB

C

12

3

XY

Z

AB

C

12

3

PF

C(C

ontext)

PV

LV(C

ritic)

Striatum

(Gating,

Actor)

Go N

oGoG

o NoG

oG

o NoG

oGo N

oGo

Go N

oGoG

o NoG

oG

o NoG

oGo N

oGo

LVi

LVe

PV

i

PV

e

Phonological

LoopTask

pfc

bg

inputs

1a

go a

s

go

a

s

go

1

a

2b

ba

bc

3c

r ab

ca

bc

ab

c

r2

r3

bc

out Phonological Loop T

askS

tore

Recall

Phonological Loop Task(O’Reilly & Frank, 2006)

PBWM LSTM RBP SRNAlgorithm

0

1000

2000

3000

Epo

chs

to C

riter

ion

Phono Loop Training Time

PBWM LSTM RBP SRNAlgorithm

0

25

50

75

100

Gen

eral

izat

ion

Err

or %

Phono Loop Generalization

PBWM = Prefrontal Basal-ganglia Working Memory; LSTM = Long Short Term Memory

(Schmidhuber et al.); RBP = Recurrent BackPropagation; SRN = Simple Recurrent

Network.

Sim

ulatin

gBG

DA

increases

inSZ:

Effects

on12A

XWorkingMem

ory

Perfo

rman

ce

0 50

100

% Networks Solving Task

IntactS

Z (+

BG

DA

)

12AX

Working M

emory P

erformance

200

250

300

350

400

450

Epochs to Criterion

IntactS

Z (+

BG

DA

)

12AX

Working M

emory P

erformance

Accu

racyTrain

ingNeed

edto

Learn

Eviden

ceforBG

gatin

gofPFC

Instru

ctioncu

esig

nals

wheth

ersu

bseq

uen

tyello

winform

ationis

distractin

g(an

dsh

ould

notbesto

redin

workingmem

ory).

McN

ab&

Klin

gberg

(2008),Natu

reNeu

roscien

ce

Eviden

ceforBG

gatin

gofPFC:L

esionpatien

ts

Baier

etal

(2010),Journal

ofNeu

roscien

ce

Eviden

ceforBG

gatin

gofPFC:N

euroim

aging

Distracto

rversu

sno-distracto

rcontrast:

NoGo!

McN

ab&

Klin

gberg

(2008),Natu

reNeu

roscien

ce

Both

BG

andPFCact

correlate

with

WM

capacity

Better

WM

functio

ncorrelates

with

PFC(m

iddle

frontal

gyrus)

andBG

activity

inDist

vs

NoDist

contrast.

Globuspallid

us=outputofBG;stro

nger

→more

NoGo

McN

ab&

Klin

gberg

(2008),Natu

reNeu

roscien

ce

Eviden

ceforBG

gatin

gofPFC:N

euroim

aging

More

GPactiv

ity⇒

Less

“unnecessary

storag

e”ofdistractin

ginfo

inparietal

cortex

(spatial

storag

e;thisparietal

regionissen

sitiveto

mem

ory

load

).

Eviden

ceforBG

gatin

gofPFC:N

euroim

aging

More

GPactiv

ity⇒

Less

“unnecessary

storag

e”ofdistractin

ginfo

inparietal

cortex

(spatial

storag

e;thisparietal

regionissen

sitiveto

mem

ory

load

).

Nosu

chcorrelatio

nin

PFC,su

ggestin

gthat

BG

=filter

(gate).

McN

ab&

Klin

gberg

(2008),Natu

reNeu

roscien

ce

Testin

gtheModel:

AX-C

PTTask

Cohen

etal

(1997);Barch

etal

(2001);Fran

k&

O’Reilly,2006

X R

BX L

time

ALY

A5

74

•targ

etseq

uen

ceisA

-X;p

ressRbutto

n.

•non-targ

etseq

uen

cesA

-Y,B

-X,B

-Y;p

ressLbutto

n.

•ignore

alldistracto

rs(5,7)

Testin

gtheModel:

AX-C

PTTask

Cohen

etal

(1997);Barch

etal

(2001);Fran

k&

O’Reilly,2006

X R

BX L

time

ALY

A5

74

•targ

etseq

uen

ceisA

-X;p

ressRbutto

n.

•non-targ

etseq

uen

cesA

-Y,B

-X,B

-Y;p

ressLbutto

n.

•ignore

alldistracto

rs(5,7)

•Tests

gatin

gan

dmain

tenan

cecomponen

tsofworkingmem

ory

/atten

tion.

B

X

A

X

A

5

X

A

7

4

Y

A

7

5

X

A

5

7

Y

A

X

B

4

5

4

X

WorkingMem

ory

Dem

ands:AX-C

PTTask

•A-X

target

sequen

ceon70%

oftrials.

•B-Y

iscontro

lconditio

n,notdep

enden

tonworkingmem

ory.

•K

eyconditions:

B-X

,A-Y

B-X:↑Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

B Ba) U

pdate B

B-X:↑Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openclosed

B BB

b) Maintain B

a) Update B

5

B-X:↑Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openclosed

ResponseM

otor

B BB

b) Maintain B

a) Update B

c) Prepare N

onTarg R

esp.

B

5

B-X:↑Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openclosed

ResponseM

otor

B BB

b) Maintain B

a) Update B

c) Prepare N

onTarg R

esp.

BX

OK

!

5

B-X:↓Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

B Ba) U

pdate B

B-X:↓Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

B Ba) U

pdate B

open

b) Update 1

5 5

B-X:↓Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openR

esponseM

otor

B Ba) U

pdate B

open

b) Update 1

c) No R

elevant Context.

5 55

B-X:↓Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openR

esponseM

otor

B Ba) U

pdate B

X

open

b) Update 1

c) No R

elevant Context.

OO

PS!

5 55

A-Y:↑Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

A AX

a) Update A

A-Y:↑Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openclosed

A AX

AX

a) Update A

b) Maintain A

5

A-Y:↑Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openclosed

A AX

AX

Xa) U

pdate A

ResponseM

otor

A c) Prepare T

arg Resp

b) Maintain A

5

A-Y:↑Workingmem

ory,

↓perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

openclosed

A AX

AX

Xa) U

pdate A

ResponseM

otor

Y A c) Prepare T

arg Resp

b) Maintain A

OO

PS!

5

A-Y:↓Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

AX

a) Update A

A

A-Y:↓Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

AX

a) Update A

A

open

b) Update 1

5 5

A-Y:↓Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

AX

a) Update A

ResponseM

otor

c) No R

elevant Context:

A

open

b) Update 1

5 55

A-Y:↓Workingmem

ory,

↑perfo

rman

ce

Sensory

Input

Working

Mem

ory

Gating

open

AX

a) Update A

ResponseM

otor

c) No R

elevant Context:

AY

open

b) Update 1

OK

!5 5

5

WorkingMem

ory

Dem

ands:AX-C

PTTask

•M

oreactive

maintenance

oftask-relevantinfo→

more

A-Y

falsealarm

s,lessB

-Xfalse

alarms

SZ:Im

paired

WorkingMem

ory

Barch

etal

(2001)

Short

Long

Delay

-20

-10 0 10 20

WM Context Index

Controls

Schiz

AX

-CP

T W

orking Mem

ory

•WM

contex

tindex

=BX-AYaccu

racy

Psych

opharm

acological

Studies

Fran

k&

O’Reilly,2006

•Double

blin

d,with

in-su

bjects

desig

n(N

=28).

•Cab

ergolin

ean

dHalo

perid

ol:D2ag

onist

andD2an

tagonist

•D2ag

ents:

preferen

tialactio

nin

BG

(Cam

pset

al,1987;Moghad

dam

&Bunney,1990;

Arnsten

etal.,1994;S

eeman

s&

Yan

g,2004)

Workingmem

ory

gatin

gtask

XR

BX

time

AL

1

3Y

2

AL

LL

LL

LL

Rtime

L

L

LL

LL

LL

A3

23

1X

B4

1

Attentional

Shift

BG

Gatin

gofPFCWorkingMem

ory

Fran

ket

al2001,

O’Reilly

&Fran

k2006,

Hazy

etal

2007...

DA

drugeffects

onworkingmem

ory

gatin

g

seeMoustafa

etal

08an

dFran

ket

al07

forsim

ilardrugeffects

inPark

inson’san

dADHD

Analo

gousdrugeffects

onlearn

ingan

dworkingmem

ory

How

domultip

leBG-FCcircu

itsinteract

inmotiv

atedbeh

avior?

•Only

somemain

tained

PFCrep

srelev

antforprocessin

gdurin

ginterm

ediate

stages

ofprocessin

g

•Red

uces

number

ofS-R

map

pingsneed

edto

belearn

edbymotorcircu

it

•Contex

tual

dep

enden

ciesforoutputgatin

g(e.g

.,arithmetic);

seealso

LSTM

(Hoch

reiter&

Sch

midhuber

97)

Hierarch

icalinteractio

nsam

ongBG-FCcircu

its

•BG

gates

frontal

”actions”

(motor,w

orkingmem

ory,co

ntex

t)

•Anterio

rPFCinfluen

ceposterio

rcircu

itsvia

BG

outputgatin

g

•BG-FCcircu

itslearn

reward

probab

ilitiesvia

DA

pred

ictionerro

rsig

nals

Hierarch

icalinteractio

nsin

BG-FCcircu

its:PFCinfluen

cesonBG

learning

Collin

s&

Fran

k2013,P

sych

Rev

;Fran

k&

Bad

re2012

Broad

ersp

eculatio

ns:

Whydoes

motorcontro

ldev

elopso

slowly

inhuman

s??

•Stan

dard

story:infan

tsborn

earlydueto

largehead

,small

birth

canal

•’Fourth

trimester’

Broad

ersp

eculatio

ns:

Whydoes

motorcontro

ldev

elopso

slowly

inhuman

s??

•Stan

dard

story:infan

tsborn

earlydueto

largehead

,small

birth

canal

•’Fourth

trimester’

•But3month

old

infan

tsstill

pretty

incompeten

t(fro

mbab

ycen

ter.com):

Broad

ersp

eculatio

ns:

Whydoes

motorcontro

ldev

elopso

slowly

inhuman

s??

•Stan

dard

story:infan

tsborn

earlydueto

largehead

,small

birth

canal

•’Fourth

trimester’

•But3month

old

infan

tsstill

pretty

incompeten

t(fro

mbab

ycen

ter.com):

“Youno

longerneed

tosupporthis

head.W

henhe’s

onhis

stoma

ch,hecan

lifthis

headand

chest.H

ecan

openand

closehis

hands..”

Broad

ersp

eculatio

ns:

Whydoes

motorcontro

ldev

elopso

slowly

inhuman

s??

•Stan

dard

story:infan

tsborn

earlydueto

largehead

,small

birth

canal

•’Fourth

trimester’

•But3month

old

infan

tsstill

pretty

incompeten

t(fro

mbab

ycen

ter.com):

“Youno

longerneed

tosupporthis

head.W

henhe’s

onhis

stoma

ch,hecan

lifthis

headand

chest.H

ecan

openand

closehis

hands..”

•Hypothesis:

human

brain

iswired

todisco

ver

gen

eralizable

structu

re....which

isinitially

ineffi

cient.

seealso

Gersh

man

etal

2010

Model

mim

icry:C-TSan

dhierarch

icalBG-PFCnetw

ork

•Sparsen

essofcontex

t-PFCconnectiv

itymatrix

islin

ked

toαclu

stering

•Both

models

areap

proxim

ationsofthesam

eprocess:

build

ingTSstru

cture

•fM

RIev

iden

ceforhierarch

icalPFC-BG

mech

anism

sBad

reet

al2010;

Bad

re&

Fran

k2012

Collin

s&

Fran

k2013

Psych

Rev

Hierarch

icallearn

ingan

dclu

stering:First,in

adults

Clusterin

gaffo

rdsfaster

learningwith

inexistin

grule

sets...

Butinitial

(phase

1)clu

steringisineffi

cientan

dslo

w!

Now,in

Infan

ts!

Werch

anet

al,2015;

inprep

Bad

reHierarch

icalLearn

ingTask

Bad

re,Kay

ser&

D’Esp

osito

,2010

Flat

(conjunctiv

e)Hierarch

ical

Learn

ingCurves

Rostro

-caudal

axisan

dhierarch

icalcontro

l

preP

MD:

•2n

dlev

elofhierarch

icalcontro

l(fro

mBad

re,Koech

lin)

•activ

itypred

ictslearn

ingin

2ndlev

elhier

conditio

n,d

eclines

inflat

conditio

n

Striatu

mim

plicated

inhierarch

icalrule

learning

How

doBG-PFCcircu

itsinteract

todisco

ver

hierarch

icalstru

cture?

Hierarch

icalinteractio

nsam

ongBG-FCcircu

its

Applicatio

nto

Bad

retask

•Anterio

rPFCaffects

posterio

rcircu

itsvia

BG

outputgatin

g

•Red

uces

#ofstim

ulus-resp

onse

map

pingsneed

edto

belearn

edin

motorcircu

it.

Applicatio

nto

Bad

retask

Modelin

gBG-PFChierarch

icallearn

ing

01

23

45

67

8E

poch

20 30 40 50 60 70 80

Percent Error

hier_BG

outhier_noD

Am

odhier_B

Gm

ntnohier

Hierarchical T

ask

0100

200300

Trial

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Normalized Max Activation

Hier preP

MD

Flat preP

MD

Model preP

MD

activity by condition

•DA-based

RLin

gatin

gnetw

orksneed

edforrap

idlearn

ingin

Hier

cond

•Asin

fMRIdata,m

odel

preP

MD

activity

decreases

inFlat

relativeto

Hier

•Thisisdueto

reward

pred

ictionerro

rsig

nals

punish

inggatin

gofpreP

MD

when

no

hierarch

icalstru

cture

exists

(staytuned

)

Modelin

gindividual

learners:

abstract

account

•Neu

ralmodel

mak

esplau

sible

linksto

biology,b

utunsu

itable

for

quan

titativefitsto

individual

subject

beh

avior

•Wedev

eloped

abstract

versio

nofhierarch

icalrule

learningusin

g

Bay

esianmixture

ofexperts

(MoE),m

otiv

atedbyneu

ralmodel

Modelin

gindividual

learners:

abstract

account

•Sep

arateexperts

learnP(Rew|R

esp,Shape),P(Rew|R

esp,Orient),etc..

Modelin

gindividual

learners:

abstract

account

•Sep

arateexperts

learnP(Rew|R

esp,Shape),P(Rew|R

esp,Orient),etc..

•Hierarch

icalexperts

separately

learnstatistics

abouteach

dim

ensio

n

contin

gen

tonacan

didate

higher

order

feature

(cf.preP

MD

contex

tforoutputgatin

g).

Modelin

gindividual

learners:

abstract

account

•Sep

arateexperts

learnP(Rew|R

esp,Shape),P(Rew|R

esp,Orient),etc..

•Hierarch

icalexperts

separately

learnstatistics

abouteach

dim

ensio

n

contin

gen

tonacan

didate

higher

order

feature

(cf.preP

MD

contex

tforoutputgatin

g).

•Cred

itassig

nmen

tmech

anism

learnsprobab

ilitythat

eachexpert

contrib

utes

toobserv

edrew

ards

•Atten

tional

mech

anism

selectsam

ongexperts

based

onlearn

ed

probab

ilityofsu

ccessP(Rew|e)∀e.

P(θr,O

|δ1...δ

n)∝

P(δ1...δ

n|θr,O

)P(θr,O

)

weig

hteach

expert

inproportio

nto

learned

probab

ilityofexpert

success

P(Rew|e)∀e.

•Goal:

infer

latenthypotheses

(attentio

nal

wts)

byobserv

ingseq

uen

ceofresp

onses

&rew

ards,m

axim

izinglik

elihoodofch

oices

with

fewfree

param

sforeach

subject

•Param

s:prio

rsto

attendto

eachdim

ensio

n,to

attendto

conjunctio

ns,to

consid

erhierarch

y.3softm

axslo

pes

(motor,w

ithin

experts,b

etween

flat/

hier).

MoEfits

tohuman

s

Exam

ple

attentio

nal

weig

hts:

MoEfitto

human

50100

150200

250300

3500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 1

Trial

Attentional WtsS

ubject118 Hierarchical

50100

150200

250300

3500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 1

Trial

Attentional Wts

Subject118 F

lat

MoEfitto

BG-PFCmodel

50100

150200

250300

350400

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 1

Trial

Attentional Wts

MoE

fit to BG

−P

FC

Net

50100

150200

250300

350400

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 1

Trial

wH

All N

ets: Attention to H

ierarchy

Individual

differen

cesin

attentio

nto

Hierarch

y

50100

150200

250300

3500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 1

Trial

wH

All subjects: A

ttention to Hierarchy

Individual

differen

cesin

attentio

nto

Hierarch

y

50100

150200

250300

3500

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9 1

Trial

wH

All subjects: A

ttention to Hierarchy

Med

iansp

litonAtte

nHvsAtte

nF→

more

preP

MD

inHier

conditio

n

Model-b

asedfM

RI:rew

ardpred

ictionerro

rs

50100

150200

250300

350−

1

−0.8

−0.6

−0.4

−0.2 0

0.2

0.4

0.6

0.8 1

PET

rial

•Rew

ardpred

ictionerro

rstrack

edbyen

tirestriatu

m(cf.

O’D

oherty

etal,

2004)

Pred

ictionerro

rsan

datten

tionto

Hier

vsFlat

experts

PE’spred

ictPPMD

declin

ein

Flat

conditio

n

0100

200300

Trial

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Normalized Max Activation

Hier preP

MD

Flat preP

MD

Model preP

MD

activity by condition

⇒Declin

ein

preP

MD

should

bepred

ictedbysen

sitivity

ofBG

toPE’sasso

ciatedwith

hiearch

icalrule

PE’spred

ictPPMD

declin

ein

Flat

conditio

n

0100

200300

Trial

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Normalized Max Activation

Hier preP

MD

Flat preP

MD

Model preP

MD

activity by condition

⇒Declin

ein

preP

MD

should

bepred

ictedbysen

sitivity

ofBG

toPE’sasso

ciatedwith

hiearch

icalrule

leftcau

date

rightcau

date

Computatio

nal/

Biological

Details

Activ

ationscan

bemain

tained

intw

odifferen

tway

s:

•Recu

rrentexcitatio

n:

•Intrin

sicbistab

ility:selectiv

elyactiv

atedionch

annels.

Problem

with

recurren

t:stro

ngest

activatio

nsat

anypointdeterm

inewhat

ismain

tained

–notnecc.

true:

best stimulus

worst stim

ulus

spikes per second

SN

NM

Computatio

nal/

Biological

Details

Ourproposal:

•Thalam

icdisin

hibitio

nactiv

ateslay

er4FC.

•Converg

ence

oflay

er4an

dcortico

-cortical

2/3projectio

nsoneith

er

2/3or5/

6neu

ronstrig

gers

main

tenan

ceionch

annels.

Other

proposals:

•Thalm

ocortical

loopsthem

selves

driv

emain

tenan

ce

(enoughthalam

icneu

ronsto

specify

what

ismain

tained

?)

•Justrecu

rrentexcitatio

nwith

inFC

(unstab

le,butusefu

lcomplem

entto

ionch

annels).