Learning the value of information in an uncertain world

8/12/2019 Learning the value of information in an uncertain world

1/45

A R T I C L E S

VOLUME 10 [ NUM BER 9 [ SEPTEMBER 2007

NATUR ENE UROSCIENCE11


2/45

A R T I C L E S

VOLUME 10 [ NUMBER 9 [ SEPTEMBER 2007

22NATU RE NEUROSCIENCE


3/45

A R T I C L E S

Learning the va!e "# in#"r$ati"n in an !n%ertain &"r'

Ti$"th( E ) Behren*1+2

+ Mar, - -""ri%h1+ Mar, E -at"n

2. Matthe& / S R!*h&"rth

1+2

O!r 'e%i*i"n* are g!i'e' ( "!t%"$e* that are a**"%iate' &ith 'e%i*i"n* $a'e in the a*t 3"&ever+ the a$"!nt "# in#!en%e

ea%h a*t "!t%"$e ha* "n "!r ne4t 'e%i*i"n re$ain* !n%ear T" en*!re "ti$a 'e%i*i"n5$a,ing+ the &eight given t" 'e%i*i"n

"!t%"$e* *h"!' re#e%t their *aien%e in re'i%ting #!t!re "!t%"$e*+ an' thi* *aien%e *h"!' e $"'!ate'( the v"atiit( "#

the re&ar' envir"n$ent -e *h"& that h!$an *!6e%t* a**e** v"atiit( in an "ti$a $anner an' a'6!*t 'e%i*i"n5$a,ing

a%%"r'ing( Thi* "ti$a e*ti$ate "# v"atiit( i* re#e%te' in the #MRI *igna in the anteri"r %ing!ate %"rte4 ACC8 &hen ea%h

tria "!t%"$e i* "*erve' -hen a ne& ie%e "# in#"r$ati"n i* &itne**e'+ a%tivit( eve* re#e%t it* *aien%e #"r re'i%ting #!t!re

"!t%"$e* /!rther$"re+ variati"n* in thi* ACC *igna a%r"** the "!ati"n re'i%t variati"n* in *!6e%t earning rate* O!r

re*!t* r"vi'e a #"r$a a%%"!nt "# h"& &e &eigh "!r 'i##erent e4erien%e* in g!i'ing "!r #!t!re a%ti"n*

The *tati*ti%* "# the envir"n$ent have een *h"&n t" e4ert "ti$ain#!en%e "n the "rganiati"n an' #!n%ti"n "# er%et!a an' $"t"r

*(*te$*1+2

3"&ever+ higher5eve r"%e**e*+ *!%h a* v"!ntar(

%h"i%e+ have "#ten r"ve' t" e i$$!ne t" *!%h *tati*ti%a

'e*%riti"n In*tea'+ re%ent 'e*%riti"n* "# %h"i%e have e$ha*ie'

it* !nre'i%tae nat!re: -e re"rt interin,e' #in'ing* that

%haenge thi* er*e%tive an' *!gge*t that an e*ti$ate "# a higher5

"r'er *tati*ti%a #eat!re "# the envir"n$ent a##e%t* the &a( that

v"!ntar( 'e%i*i"n* are $a'e

The 'e%i*i"n* that &e $a,e are g!i'e' ( the "!t%"$e* "#

*i$iar 'e%i*i"n* $a'e in the a*t;


4/45

A R T I C L E S

1/MRIBCentre+ Univer*it( "# O4#"r'+ )"hn Ra'%i##e 3"*ita+ O4#"r' O: 9U+ U

2Aeart$ent "# E4eri$enta P*(%h""g(+ Univer*it( "# O4#"r'+ S"!th Par,*

R"a'+ O4#"r' O1 :U+ U C"rre*"n'en%e *h"!' e a''re**e' t" TE)B ehren *D#$ri"4a%!,8

Re%eive' 2: Ma( a%%ete' > )!ne !i*he' "nine > A!g!*t 2007E'"iF 1010:=Gnn19>;

VOLUME 10 [ NUMBER 9 [ SEPTEMBER 2007

;;NATU RE NEUROSCIENCE
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954http://www.nature.com/doifinder/10.1038/nn1954mailto:[email protected]://www.nature.com/doifinder/10.1038/nn1954


5/45

a

b

3763

37 63

k

Vi 1

Vi

ri

yi

ri + 1

yi + 1

/ig!re 1 Pr"aiit(5tra%,ing ta*, a8 E4eri$enta r"%e'!re S!6e%t*

%arrie' "!t a "ne5ar$e' an'it ta*,+ %h""*ing et&een !e an' green "n

thea*i* "# "th the a*t *!%%e** an' the re&ar' a**"%iate' &ith ea%h %""r

(e"& n!$er*8 S!6e%t* atte$te' t" $"ve the re' ar t"&ar' the *iver

ar #"r '10 "r t"&ar' the g"' ar #"r '20 The ar $"ve* a 'i*tan%e

r""rti"na t" the %h"*en re&ar' "n( i# the %h"*en %""r &a* %"rre%t In

thi* in*tan%e+ the *!6e%t %h"*e green+ !t the %"rre%t %h"i%e &a* !e *"

the re'ar re$aine' *tati"nar( 8 Hrahi%a 'e*%riti"n "# the r"aiit(5

tra%,ingr"e$ Arr"&* in'i%ate the 'ire%ti"n "# in#!en%e At ea%h tria i+

'ata (i i*"*erve' !e "r green i* %"rre%t8+ &hi%h i* g"verne' ( r"aiit( r

i

Thi*

r"aiit( %an %hange et&een tria*+ g"verne' ( the v"atiit(+ vi

+ &hi%h %an

it*e# %hange a* the envir"n$ent $"ve* et&een v"atie an' *tae eri"'*8

an' i* g"verne' ( %"ntr" ara$eter , The g"a "# the Ba(e*ian earner i*

t" tra%, the*e ara$eter* thr"!gh the %"!r*e "# the e4eri$ent+ given "n(

the "*erve' 'ata+ (

*!6e%t* "#ten i%,e' the e** i,e( %""r i# it &a* a**"%iate' &ith

a higher re&ar'

/ir*t+ *!6e%t* !n'er&ent 120 tria* &here the r"aiit( "#a

!e "!t%"$e &a* 7>F a *tae envir"n$ent In the *e%"n' ha*e170 tria*8+ re&ar' r"aiitie* *&it%he' et&een =0 !e an'

=0 green ever( :0 "r ;0 tria*F a v"atie envir"n$ent

Thr"!gh"!t the e4eri$ent+ re&ar'* #"r %"rre%t !e re*"n*e*

#

8 &ere *ee%te' ran'"$( et&een 0 an' 100+ an' re&ar'* #"r

%"rre%t green re*"n*e* &ere *et t" 100 < #

8

Thi*+ in t!rn+ re#e%t* h"& !nre'i%tae re%ent "!t%"$e* have

een A hi*t"r( "# *!rri*ing "!t%"$e* &i in%rea*e e*ti$ate'

v"atiit( an' !n%ertaint(+ an' there#"re earning rate /ig!re 2a


6/45

0 v k

0.2

0.170.14

0.11

0.09

0.07

0.06

0.05Stale 1 !olatile 1 !olatile 2 Stale 2

http://ww w .natur e .com/natureneu r oscience

2007 Nature Publishing Groupara$eter* r+ v an' , &hen it get* a ne&

ie%e "# in#"r$ati"n at the "!t%"$e "# ea%h

tria Cr!%ia(+ the !'ate e@!ati"n reie*

"n( "n ara$eter e*ti$ate* #r"$ the

re%e'ing tria+ an' the ate*t tria "!t%"$e

t" 'eter$ine 'e%i*i"n an' earning "n the

ne4t tria S!5e$entar( In#"r$ati"n8

The agent '"e* n"t have t" retain

$e$"rie* "# re%ent "!t5 %"$e* Ath"!gh

the !'ate e@!ati"n* %an "n( e #"r$a(

e4re**e' in r"aii*ti% ter$*+ it i*

!*e#! t" 'e*%rie their ehavi"r in ter$*

"# e##e%tive earning rate* C"ar*e(+ the

earning rate i* 'i%tate' ( the !n%ertaint( "r

varian%e in the e*ti$ate "# re&ar' rate

/ig!re 2 Behavi"r "# Ba(e*ian earner an' h!$an *!6e%t* a8 Margina "*teri"r 'i*tri!ti"n* "n

tra%,ing variae* at three *tage* in the e4eri$ent Le#t+ the 'i*tri!ti"n "n re&ar' r"aiit( an'

v"atiit( Right+ the 'i*tri!ti"n "n v"atiit( an' %"ntr" ara$eter+ , A#ter 120 tria*+ the Ba(e*ian

earner &a* %"n#i'ent that the re&ar' r"aiit( &a* 07>+ the *(*te$ &a* *tae+ an' that thi* *taiit(

&a* !ni,e( t" %hange 8 A#ter a #!rther 1> tria* in &hi%h the re&ar' r"aiit( %hange' t" 02+ the

Ba(e*ian earner &a* !n%ertain a"!t the *tate "# the envir"n$ent Le#t+ high r"aiit( in t&" regi"n*+

either re#e%ting that the envir"n$ent &a* !n%hange'+ "r that the envir"n$ent &a* %hanging an' that

the ne& re&ar' rate &a* "& %8 A#ter a #!rther 2> tria* "# "& re&ar' rate+ the earner ha' re%"vere'

%"n#i'en%e+!t *ti eieve' the *taiit( $ight %hange right8+ en*!ring that it &"!' rea%t #a*ter t"

an( #!t!re %hange in re&ar' rate '8 E4eri$ent I+ the re&ar' *%he'!e an' the Ba(e*ian ara$eter

e*ti$ate*#"r the *tae5#ir*t e4eri$ent are *h"&n Le#t+ the 'a*he' ine *h"&* tr!e re&ar'

r"aiitie* an' the *"i' ine *h"&* the Ba(e*ian5e*ti$ate' re&ar' rate Right+ e*ti$ate "# v"atiit(

thr"!gh the %"!r*e "# the *a$e e4eri$entN"te that &hen v"atiit( i* "&+ the e*ti$ate' re&ar' rate in

e#t8 %hange* itte &ith ea%h tria e8 3!$an ehavi"r Average earning rate* '!ring the *tae an'

v"atieha*e* "# ea%h e4eri$ent *tae5#ir*t an' v"atie5#ir*t+ re*e%tive(8 Re' an' a%, ar* *h"&

the

$ean an' *e$ va!e* #"r the h!$an *!6e%t* "t* *h"& the ehavi"r "# the Ba(e*ian earner
http://www.nature.com/natureneurosciencehttp://www.nature.com/natureneuroscience


7/45

2007 Nature Publishing Grouphttp://ww w .natur e .com/natureneu r oscience

a

b

/ig!re : E4eri$ent II+ %ing!ate a%tivit( re#e%ting e*ti$ate' v"atiit(

a8 C"r"na ( J 2;8 an' *agitta 4 J

+ the ACC $a( have a $"re genera r"e in

rere*enting an' !'ating a%ti"n va!e*22+2K+27

In'ee'+ a#ter e*i"n*

t" the ACC *!%!*+ $a%a@!e* n" "nger !*e $"re than the $"*tre%ent "!t%"$e t" g!i'e ea%h %h"i%e

19

-e %arrie' "!t a *e%"n' e4eri$ent+ !*ing #MRI in 1= *!6e%t* t"

te*t &hether ACC a%tivit( re#e%te' the e*ti$ate "# the

envir"n$ent?* v"atiit( &hen arti%iant* $"nit"re' 'e%i*i"n

"!t%"$e* S!6e%t* %arrie' "!t the *a$e ta*, a* the( 'i' in the

ehavi"ra e4eri$ent Ea%h tria &a* 'ivi'e' int" three ha*e*+

'e%i'e+ interva an' $"nit"r Meth"'*8+ a"&ing !* t" 'i**"%iate

a%tivit( reate' t" v"atiit( in the 'i##erent tria ha*e* I# the ACC

'i##erentia( integrate* in#"r$a5 ti"n #r"$ revi"!* tria* 'een'ing

"n the %!rrent e*ti$ate "# v"atiit(+ ACC a%tivit( *h"!' e

$"'!ate' ( thi* e*ti$ate '!ring the $"nit"r ha*e

The re&ar' envir"n$ent &a* *tae #"r K0 tria* 7> !e8 an'

v"atie #"r K0 tria* =0 *&aing et&een a%ti"n* ever( 20

tria*8 S!6e%t* &ere *it e@!a( int" gr"!* e4erien%ing the

*tae an' v"atie envir"n$ent* #ir*t U*ing the Ba(e*ian earner+

&e %a%!ate' the re'i%te' v"atiit( e*ti$ate at ea%h tria+

'eter$ine' ( the *!6e%t*? "*ervati"n*+ t" !*e a* a regre**"r in

the ana(*i* /ig :a8 -e ana(e' the 'ata !*ing the /MRIB

*"#t&are irar(2=

S!e5 $entar( In#"r$ati"n8 Seven regre**"r*&ere in%!'e' in the ana(*i*F three 'e#ining the ha*e* "# the

e4eri$ent 'e%i'e+ interva an' $"nit"r8+ three 'e#ining

intera%ti"n* et&een the*e ha*e* an' the re'i%te' e*ti$ate "#

v"atiit(+ an' "ne 'e#ining *!6e%t err"r* in the $"nit"r ha*e

A%tivati"n* in the 'e%i'e an' $"nit"r ha*e* "# the tria %"$ri*e

a net&"r, "# regi"n* inv"ve' in 'e%i*i"n5$a,ing29

/ig* 2 an' :a

an' S!e$entar( In#"r$ati"n8 N"ta(+ the 'e%i'e ha*e

a%tivate* %a!'a ACC+ &hi%h $a( e %"$arae t" $a%a@!e r"*tra%ing!ate $"t"r area Thi* area ha* %"nne%ti"n* t" ri$ar( $"t"r

%"rte4 an' *ina %"r':0

+ regi"n* that e4e%!te a%ti"n* a#ter the

'e%i*i"n In %"ntra*t+ the $"nit"r ha*e a%tivate* a r"*tra art "#

ACC Thi* area re*e$e* the regi"n in $a%a@!e that i*

inter%"nne%te' &ith *tr!%t!re* *!%h a* the a$(g'aa+ "rit"#r"nta

%"rte4 an' ventra *triat!$+ &hi%h are i$i%ate' in r"%e**ing

va!e an' re&ar':1

The #MRI 'ata ne%e**ari( %"ntaine'

%"n*i'era( #e&er tria* than the ehavi"ra e4eri$ent

Neverthee**+ the ehavi"ra %hange in earning rate *!rvive' a*

a tren' in*i'e the *%anner

In %"ntra*t+ the $"nit"r v"atiit( intera%ti"n reveae' a

%ir%!$5 *%rie' a%tivati"n in the ACC /ig :8+ the "n( rain

regi"n that *!rvive' thre*h"'ing $a4 J ;2+ at MNI 4 J


8/45

!t n"ne "# the*e %an e4ain the *a$e "rti"n "# the #MRI*igna The ta*, &a* %are#!( %"ntr"e' t" a%%"!nt #"r the

#""&ing"tentia


9/45

%"n#"!n'*F re&ar' attaine' ( the *!6e%t+ *&it%h tria*+

re'i%te' va!e "# the %h"*en "ti"n "!t%"$e *ie "!t%"$e

r"aiit(8+ rea%ti"n ti$e+ re'i%ti"n err"r+ $agnit!'e "#

re'i%ti"n err"r+ re'i%te' re&ar' i,eih""' an' there#"re err"r

i,eih""'2>

8+ err"r tria*2;

+ "%a 1> tria8 varian%e in re&ar'

attaine'+ an' the 'i##er5 en%e in va!e et&een the t&" "ti"n*

re*ente' at the tria The rea%ti"n5ti$e an' va!e5'i##eren%e

regre**"r* %"n*tit!te in'i%e* "# tria 'i##i%!t( A$"ng the*e"tentia %"n#"!n'ing regre**"r*+ there &a* n" %a*e "# a

*igni#i%ant %"rreati"n that %"!' e4ain the #MRI *igna that

&a* attri!te' t" the v"atiit( e*ti$ate /ig ;a8+ an' &hen

a "# the*e %"n#"!n'* are in%!'e' a*

regre**"r*5"#5n"5intere*t in the $"'e+ the e##e%t "# v"atiit(

re$ain* !nt"!%he' /ig ;8

T&" #eat!re* "# the ta*, 'e*ign $a'e it "**ie t" %"ntr" #"r *"

$an( "tentia %"n#"!n'* /ir*t+ *!6e%t* "#ten ign"re' the

$"rer"ae "ti"n+ in #av"r "# the "ti"n &ith higher re&ar'

$agnit!'e S!%h %h"i%e* &ere in'een'ent "# e*ti$ate' v"atiit(

Se%"n'+ the tr!e $a4i$!$5re&ar' i,eih""' &a* *ight( higher in

the v"atie than in the *tae ha*e 0= an' 07>+ re*e%tive(8+*!%h that the average aarent5re&ar' i,eih""' t" the *!6e%t* &a*

e@!a in the t&"ha*e* Ath"!gh+ "n average+ h!$an ehavi"r i*

&e re'i%te' ( a Ba(e*ian earner+ there i* variaiit( in

earning rate* a%r"** the

a 4.0

4.5

5.0

5.5

6.0

6.5

4.0

4.5

5.0

5.5

6.0

6.5

4.0

4.5

5.0

5.5

6.0

6.5

4.0

4.5

5.0

5.5

6.0

6.5

4.0

4.5

5.0

5.5

6.0

6.5

7.0 0

50100

"ewardattai#ed

7.00 0.5 1.0

Stay $0% or swit&' $1% trials

7.0

050

100

(r

edi&ted value 7.00 1)000 2)000

"ea&tio# time 7.01

0 1

(redi&tio# error

4.0

4.0 4.0 4.0 4.0

4.5

5.0

5.5

6.0

6.5

4.5

5.0

5.5

6.0

6.5

4.5

5.0

5.5

6.0

6.5

4.5

5.0

5.5

6.0

6.5

4.5

5.0

5.5

6.0

6.5

7.0

0 0.5 1.0 7.00 0.5 1.0 7.00 0.5 1.0 7.00 1)000 2)000 3)000 7.00 50 100

*a#itue o, -redi&tio# error"eward lieli'ood Error $0% a#d rewarded $1% trials /o&al varia#&e i# reward !alue di,,ere#&e etwee# o-tio#s

E,,e&t sie a&&ou#ti# ,or #o &o#,ou#dsE,,e&t sie a&&ou#ti# ,or all &o#,ou#ds

Trial -rese#ted "es-o#se ut&ome ut&ome

E,,e&t sie $aritrary u#its%

bE,,e&t o, estimated volatility


10/45

c e,,e&t o, volatility -redi&ts lear#i# rate a&ross sue&ts

0.3

orrelatio# $si#ed r2)

0.2

0.1 E,,e&t sie$aritrary

u#its%0 1.00 0

1.00 0

0

0.1

0 5 10 15 20

Time $s%

0.2 Trial -rese#ted "es-o#se ut&ome ut&ome removed

0 5 10 15 20

Time $s%

http://ww w .natur e .com/natureneu r oscience

2007 Nature Publishing Group

/ig!re ; Regi"n5"#5intere*t ana(*i* an' "tentia %"n#"!n'ing #a%t"r* a8 C"rreati"n et&een e*ti$ate' v"atiit( ( a4i*8 an' ten "tentia #a%t"r*that have revi"!*( een r""*e' t" e4ain ACC a%tivit( There &a* n" %a*e "# a *igni#i%ant %"rreati"n that %"!' e4ain the e##e%t "# e*ti$ate'

v"atiit(

8 R"!*tne** t""tentia %"n#"!n'* The ti$e %"!r*e* "# e##e%t *ie* thr"!gh the %"!r*e "# the tria are *h"&n+ #it &ith a genera inear $"'e *ee

Meth"'* an' S!e$entar( In#"r$ati"n8 ata are ta,en #r"$ "%a $a4i$a The a%, ine *h"&* the e##e%t *ie &hen e*ti$ate' v"atiit( &a* in%!'e' a*

a "ne regre**"r The e##e%t "# e*ti$ate' v"atiit( &a* %"n#ine' t" the $"nit"r eri"' "# the ta*, The gra( ine *h"&* the e##e%t *ie &hen ten reate'

%"n#"!n'* that have revi"!*( een th"!ght t" e4ain ACC a%tivit( &ere in%!'e' a*"tentia %"n#"!n'ing regre**"r* in the $"'e N"ne "# the ten

%"n#"!n'* %"!' e4ain the *igna reate' t" v"atiit( %8 V"atiit( reate' a%tivit( in the ACC e4ain* et&een5*!6e%t variati"n in "vera earning rate* A

ti$e *erie* "# %"rreati"n* *igne' r28 et&een the e##e%t *ie in the ACC an' the $ean earning rate #itte' t" *!6e%t ehavi"r "ver "th ha*e* "# the

e4eri$ent i* *h"&n S!6e%t* *h"&ing a greater e##e%t "# v"atiit( in the ACC in the "!t%"$e5$"nit"ringeri"' &ere i,e( t" *h"& a higher average

earning rate in the ehavi"ra 'ata In*ert+ *%atter"t at the ti$e "# the ea, e##e%t "# v"atiit( in the ACC r2

J 027+ P " 001 /5te*t8+ $a4 r2

J 0:28


11/45


E,,e&t sie a&&ou#ti# ,or #o &o#,ou#ds

E,,e&t sie a&&ou#ti# ,or all &o#,ou#ds a#d varia#&e o# r

Trial -rese#ted "es-o#se ut&ome ut&omeremoved


aE,,e&t o, estimated volatility

0 5 10 15 20

Time $s%

E,,e&t sie a&&ou#ti# ,or #o &o#,ou#ds

E,,e&t sie a&&ou#ti# ,or all &o#,ou#ds a#d estimatedvolatility

Trial -rese#ted "es-o#se ut&ome ut&ome


bE,,e&t o, varia#&e o# r

0 5 10 15 20Time $s%

"!ati"n -e te*te' &hether the v"atiit( reate' *igna %hange in

the ACC %"!' re'i%t $ean earning rate* a%r"** in'ivi'!a*

In'ivi'!a* &ith a greater e##e%t "# v"atiit( in the ACC in the $"nit"r

eri"' ha' a higher $ean earning rate+ an' there#"re gave $"re

&eight t" the $"*t re%ent ie%e "# in#"r$ati"n /ig ;%8

The Ba(e*ian 'e*%riti"n *!gge*t* that v"atiit( i* 'ete%te' (

*!6e%t* an' in'!%e* !n%ertaint( in their e*ti$ate "# re&ar'

i,eih""'+ &hi%h 'rive* the earning rate Thi* !n%ertaint( i*

$ea*!re' a* varian%e in the $argina "*teri"r 'i*tri!ti"n "n r

verti%a &i'th "# &hite area* in e#t ane*+ /ig 2a


12/45

/ig!re > E*ti$ate' v"atiit( an' varian%e "n r a8 E##e%t "# v"atiit(

&hen the varian%e "n the e*ti$ate "# re&ar' r"aiit( r &a* in%!'e' a*

a regre**"r a* &e a* the a#"re$enti"ne' %"n#"!n'*8 Thi* varian%e &a*

a %r!%ia in, et&een the v"atiit( e*ti$ate an' the earning rate Ti$e

%"!r*e* arere*ente' a* in /ig!re ; 8 E##e%t "# the varian%e "n the

e*ti$ate "# r In the a*en%e "# e*ti$ate' v"atiit( a* a %"regre**"r+ thi*

varian%e e4aine' the 'ata in a *i$iar &a( t" the v"atiit( e*ti$ate

3"&ever+ &hen the v"atiit( &a* in%!'e' a* a %"regre**"r+ the t&"

e##e%t*"th *!rvive'+ aeit e4aining 'i##erent "rti"n* "# the *igna

i##erent #"r$* "# !n%ertaint(Ne!ra rere*entati"n* "# !n%ertaint( have re%ent( re%eive'

attenti"n /"r e4a$e+ there i* evi'en%e that '"a$inergi% ne!r"n*

in $a%a@!e $"n,e(*:;

an' '"a$in"%etive rain regi"n* in

h!$an*:>

rere*ent the r"aiit( "# a re&ar' "%%!rringThe"reti%a $"'e* have 'ivi'e' !n%ertaint( int" the e4e%te'

!nre'i%taiit( "# a *ti$!!*5"!t%"$e a**"%iati"n+ an' the

!ne4e%te' !n%ertaint( %a!*e' ( %hange* in *!%h %"ntingen%ie*

*i$iar t" the v"atiit( 'riven !n%ertaint( that &e ana(9e

here817

+ an' have *!gge*te' that the*e t&" #"r$* "# !n%ertaint(

*h"!' %"$ine t" 'rive ehavi"r It i* *!gge*te' that thi*

!ne4e%te' !n%ertaint( i* 'riven ( n"reinehrine17

an' it*

intera%ti"n &ith the ACC:K

+ an' that the e4e%te' !n%ertaint( i*

rere*ente' in the %h"inergi% n!%ei17 T" 'ra& %"ntra*t &ith the

v"atiit( reate' a%tivit( in the ACC+ &e there#"re %arrie' "!t a

*e%"n' ana(*i* S!e$entar( In#"r$ati"n8 that in%!'e' the

r"aiit( "# a re&ar' '!ring the interva ha*e a* a regre**"r

Ath"!gh there &ere n" *igni#i%ant a%tivati"n* a#ter %"rre%ti"n*

#"r $!tie %"$ari*"n*+ re'!%ing the thre*h"' ; 2:+ P "

001 !n%"rre%te'8 reveae' a high( #"%a a%tivati"n in a regi"n "#

the rain anat"$i%a( %"n*i*tent &ith the '"a$inergi% ventra

teg$enta area VTA8 MNI 4 J


13/45


ie%e "# in#"r$ati"n #"r re'i%ting #!t!re "!t%"$e*1>+1K

+ an' that

envir"n$enta v"atiit(+ a #a%t"r *een a* eing i$"rtant in

#inan%ia $ar,et*:7

+ i* a 'eter$inant "# *!%h *aien%e Thi* i* a ,e(

e4a$e "# the genera h("the*i* "# Ba(e*ian rea*"ningF $!tie

*"!r%e* "# in#"r$a5 ti"n *h"!' e re%"n%ie' a%%"r'ing t" their

re*e%tive re'i%tive va!e* Thi* h("the*i* ha* revi"!*( een'e$"n*trate' in the %"nte4t "# %"$ining *i$!tane"!* %!e*

1+2

3ere &e *h"& that h!$an* reeate'( %"$ine ri"r an'

*!*e@!ent in#"r$ati"n a* 'ata a%%!5 $!ate "ver ti$e+ even in

the %"nte4t "# %hanging envir"n$enta v"atiit(+ an' there#"re

%hanging reiaiit( "# "ne "r $"re *"!r%e* "# in#"r$ati"n

Re$ar,a(+ e"e "th e*ti$ate an' !*e thi* v"atiit(

ara$eter "ti$a(+ ga!ging the va!e "# ea%h ne& ie%e "#

in#"r$ati"n that the( a%@!ire The #a%t that the v"atiit(

e*ti$ate $"'!ate* the ACC re*"n*e t" ne& ie%e* "# in#"r$ati"n

*!gge*t* that a%tivit( in thi* regi"n $a( in#!en%e *!*e@!ent

earning an' 'e%i*i"n5$a,ing

The ACC i* art "# a 'i*tri!te' ne!ra *(*te$ that i* i$i%ate'

in the rere*entati"n an' !'ating "# 'e%i*i"n va!e*>+7+:2

Pre'i%ti"n err"r *igna* have een #"!n' in '"a$inergi%

regi"n*:=

an' the ventra *triat!$:9

+ an' a%ti"n va!e *igna* have

een #"!n' in the !ta$en;0+;1

+ !t the ACC?**e%ia %"ntri!ti"nha* een !n%ear O!r 'ata *!gge*t that #!%t!ati"n* in ACC a%tivit(

in the !'ate eri"' are reate' t" the e*ti$ate' v"atiit( "# theenvir"n$ent+ an' hen%e t" the earning rate The r"6e%ti"n #r"$ the

ACC t" ventra *triat!$;2

&"!' a"& the earning rate t"

$"'!ate the in#!en%e "# the %!rrentre'i%ti"n err"r "n the ne4t

va!e e*ti$ate=

Ath"!gh there ha* een an e$ha*i* "n the ACC?* r"e in

'ete%ting err"r* an' err"r i,eih""'2;+2>

+ the "*ervati"n that

v"atiit( in#!5 en%e* ACC a%tivit( re*"ve* a n!$er "# 'i*%reant

"*ervati"n* /"r e4a$e+ in *inge5!nit 'ata+ *"$e ACC ne!r"n*

re*"n' t" err"r*+ re&ar'* "r t" "th "!t%"$e* &hen $a%a@!e*#ir*t e4"re their avaiae "ti"n*+ !t a ne!r"n* are e** a%tive

"n%e re&ar' a**"%ia5 ti"n* have een &"r,e' "!t21

3"&ever+ ACC

ne!r"n* %"ntin!e' t" e a%tive &hen re&ar' a**"%iati"n* %hange'

*t"%ha*ti%a( "n ea%h "ti"n;:

The %!rrent ar"a%h a*"

e4ain* &h( ne!r"n* in inter5 %"nne%te' %ing!ate regi"n* %arr(

*igna* that are reate' t" the re%ent average re&ar' rate;:

an' it*

varian%e::

+ ara$eter* that are %"*e( reate' t" th"*e in "!r

$"'e Other re*ear%her* have re"rte' a high ACC BOL *igna

&hen h!$an *!6e%t* *&it%h ta*, *et "r revi*e their e*ti$ate "# the

%!rrent *it!ati"n22+;;

A%%"r'ing t" the %!rrent er5 *e%tive+ ACC

a%tivit( *h"!' in'ee' e greater &hen %ir%!$*tan%e* are %hanging+"r &hen an "!t%"$e i* e*e%ia( in#"r$ative The ACC %ir%!it

ha* a*" een i$i%ate' in *(%hiatri% 'i*ea*e* *!%h a* "*e*5 *ive5

%"$!*ive 'i*"r'er;>

S!%h %"n'iti"n* $a( e %"n%eive' "# a*

'i*"r'er* "# 'e%i*i"n5$a,ing+ in &hi%h the &r"ng in#"r$ati"n i*

given the greate*t &eight

Rather than *tre**ing the rere*entati"n an' !'ating "#

a%ti"n va!e*+ aternative a%%"!nt* "# ACC #!n%ti"n have #"%!*e' "n

*!6e%t ar"!*a;K

an' %hange* in attenti"n %a!*e' ( re*"n*e

%"n#i%t;7

A n!$er "# re%ent *t!'ie*+ h"&ever+ *!gge*t that

re*"n*e %"n#i%t $a( e $e'iate' ( $"re '"r*a regi"n* "# the

$e'ia #r"nta %"rte4;=

Ath"!gh it i* "**ie that the*e

*(%h""gi%a r"%e**e* have *"$e r"e in g!i'ing earning+ there

are *"$e ,e( 'i##eren%e* &ith the *igna inve*tigate' here /ir*t+ in

'e%i*i"n5$a,ing ta*,*+ %"n#i%t an' ar"!*a have ten'e' t" e reate'

t" ACC a%tivit( &hen the *!6e%t i* $a,ing a 'e%i*i"n an' a&aiting

the "!t%"$e;K+;7

In "!r *t!'(+ there i* &i'e5 *rea' ACC a%tivit(

'!ring the*e eri"'*+ !t it '"e* n"t %"rreate &ith the v"atiit(

*igna+ an' there#"re $a( e %"n*i*tent &ith the*e aternative

the"rie* "# ACC #!n%ti"n 3"&ever+ the v"atiit( a##e%t* the ACC

*igna &hen the "!t%"$e i* "*erve'+ &hi%h i* the %r!%ia ti$e #"r

earning Se%"n'+ there are $an( #eat!re* "# the ta*, that are


14/45

e4e%te' t" %a!*e ar"!*a "r %"n#i%t+ !t %ann"t e4ain "!r 'ata

/"r e4a$e+ "!r 'ata %ann"t e e4aine' ( the 'i##i%!t( "# the

tria+ "r ( tria* &hen *!6e%t* ta,e ri*,( 'e%i*i"n* /ig ;8

/!rther$"re+ &e have 'e$"n*trate' that the *e%i#i% ACC re*"n*e

t" v"atiit( ha* a 'ire%t e##e%t "n earning S!6e%t* &ith a higher

re*"n*e t" v"atiit( in the "!t%"$e ha*e have higher average

earning rate* in the ehavi"ra 'ata /ig ;%8 It i* "**ie that

the 'ete%ti"n "# v"atiit( it*e# %a!*e* ar"!*a+ ath"!gh

%"$ari*"n &ith e*i"n 'ata19

*!gge*t* that *!%h ar"!*a *h"!'

have a %entra art in the earning r"%e** T" inve*tigate thi*

"**iiit(+ re*ear%her* *h"!' $ea*!re a!t"n"$i% re*"n*e* in

#!t!re e4eri$ent* &hen "!t%"$e* are "*erve' in %"n'iti"n*

"# 'i##ering v"atiit(

The re*!t* re*ente' here are %"n#ine' t" the !'ate eri"'

/ig* ; an' >8+ an' there#"re '" n"t ne%e**ari( i$i%ate the ACC

in the initia %"$!tati"n "r *t"rage "# v"atiit( "r !n%ertaint(

In thi* *t!'(+ v"atiit( e*ti$ate* varie' $"re *"&( than "&5

#re@!en%( #MRI "*%iati"n* The %r!%ia regre**"r &a*+ there#"re+

the intera%ti"n et&een e*ti$ate' v"atiit( an' the $"nit"r

eri"'+ &hi%h a"&e' !* t" te*t &here the v"atiit( e*ti$ate &a*

!*e' in %a%!ati"n* That $a%a@!e* &ith ACC e*i"n* !*e "n( the

"!t%"$e "# the $"*t re%ent tria t" g!i'e their ne4t 'e%i*i"n i*

%"n*i*tent &ith the i$"rtan%e "# ACC in $e'iating the in#!en%e

"# v"atiit( "n ehavi"r19

It i* n"tae that the Ba(e*ian earner in thi* *t!'( &a* n"t t!ne'

t" the *tr!%t!re "# ta*, %"ntingen%ie* !*e' in the e4eri$ent In

the e4eri$enta r"%e'!re+ the tr!e "!t%"$e r"aiit( %hange'

et&een 'i*%rete eve* In %"ntra*t+ the Ba(e*ian earner a**!$e*

that r"ai5 itie* var( in a %"ntin!"!* #a*hi"n Thi* $"'e &a*

%h"*en t" #it &ith the *!6e%t*?*tate at the "!t*et "# the ta*, -hen

*!6e%t* %arr( "!t the ta*,+ the( are naive t" n"t "n( the ta*,

%"ntingen%ie*+ !t a*" t" an("**ie *tr!%t!re therein 3"&ever+

an aternative &"!' e t" a**!$e *!6e%t* &ere a&are "# the ta*,

*tr!%t!re an' there#"re ai$e' t" "", #"r ar!t 6!$* in re&ar'

rate17+;9

*!%h an a**!$ti"n %an ea*i( e a%e' in the

#ra$e&"r, "# /ig 1 S!e$entar( In#"r$ati"n8 The #a%t that

thi* aternative $"'e+ *!ita( e4ten'e'+ $a,e* re'i%ti"n* "#

*!6e%t ehavi"r that are e@!ivaent t" th"*e "# the %"ntin!"!*

$"'e S!e$entar( Tae 18 'e$"n*trate* that "!r ana(*e*

'" n"t 'een' "n the e4a%t a**!$ti"n* $a'e a"!t the

generative $"'e The 'ete%ti"n "# v"atiit( in an( re&ar'

envir"n$ent a"&* an agent t" a'6!*t it* earning rate &ith"!t

,n"&e'ge "# ta*, *tr!%t!re

There ha* re%ent( een %"n*i'erae intere*t in the rere*entati"n

"# re&ar' e4e%tati"n an' r"aiit( in the rain>+7+1=+27+;0+;9+>0

It i* e%"$ing in%rea*ing( %ear+ h"&ever+ that *evera a*e%t* "#

re&ar' are rere*ente' 'i*tin%t(:>

The re*ent #in'ing* "#

%"rti%a a%tivati"n re#e%ting envir"n$ent v"atiit(+ an' there#"re

!n%ertaint(+ in the re&ar' e*ti$ate "n%e again !n'er*%"re* the

nee' t" rere*ent $an( 'i*tin%t a*e%t* "# an "rgani*$?*e4erien%e"# the re&ar' envir"n$ent in "r'er #"r 'e%i*i"n* t" e $a'e

e##e%tive(

MET3OSE*ti$ating the earning rate #r"$ the *!6e%t 'e%i*i"n* /"r E4eri$ent

1+ *!6e%t* 'e%i'e' et&een !e an' green re%tange* in ea%h tria+

'eter$ine' ( their e4e%tati"n "# the %"rre%t re*!t an' the re&ar'

a**"%iate' &ith ea%h "!t%"$e /ig 18 -e %hara%terie' the *!6e%t*?re*"n*ivene** t" ne& "*ervati"n* at t&" *tage* "# the e4eri$entF &hen

the *!6e%t* *h"!' e*ti$ate the envir"n$ent t" e at it* $"*t *tae+ an'

&hen the( e*ti$ate' it t" e at it* $"*t v"atie /ig 18 -e then #it a

rein#"r%e$ent5earning $"'e t" the *!6e%t*?'e%i*i"n* in ea%h ha*e The

$"'e ha* t&" art*F a Mre'i%t"r?+ &hi%h e*ti$ate* the %!rrent re&ar' rate

given a*t "*ervati"n*+ an' an M*ee%t"r?+ &hi%h generate* a%ti"n* "n the

a*i* "# the*e e*ti$ate*

The re'i%t"r i* in the #"r$ "# a *i$e 'eta5earning r!e= Thi* r!e ha* a

*inge #ree ara$eter+ the earning rate The 'eta5earning r!e=

e*ti$ate* "!t%"$e r"aiitie* !*ing the #""&ing e@!ati"nF


15/45

ri 1

J ri

aei

&here ri1

i* the re'i%te' "!t%"$e r"aiit( #"r the i18th

tria+ ri

i*

there'i%te' "!t%"$e r"aiit( #"r the ith

tria+ eii* the re'i%ti"n err"r at

the ith

tria+ an' a i* the earning rate B( %h""*ing 'i##erent va!e* #"r a+ the

$"'e %an $a,e 'i##erent ar"4i$ati"n* "# the *!6e%t?*"!t%"$er"aiit( e*ti$ate* The *ee%t"r $"'e e4ain* *!6e%t 'e%i*i"n* "n the

a*i* "# the*e e*ti$ate* 3ere+ 'e%i*i"n* are 'eter$ine' ( "th the

e*ti$ate' re&ar' i,eih""'+ ri1

+ an' ( the re&ar' $agnit!'e "n ea%h

"ti"n Oti$a a%ti"n *ee%ti"n &"!' inv"ve %"$!ting the e*ti$ate'

Pa*%aian va!e "!t%"$e *ie "!t%"$e

r"aiit(8 "# ea%h "ti"n a*#""&*F

g!ei1

Jri1

#!ei

1

ggreeni1

JP1 ri1

Q#greeni

1

&here #green+i

an' #!e+i

are the ,n"&n re&ar' *ie* "# ea%h %""r The

"ti$a re*"n*e i* then the %""r &ith the highe*t re'i%te' r"#it 3"&ever+

&e '" n"t $a,e the a**!$ti"n that h!$an *!6e%t* &eigh re&ar' i,eih""'

&ith re&ar' $agnit!'e in thi* "ti$a Pa*%aian #a*hi"n In*tea'+ &e

in%!'e a #ree ara$eter that a"&e' *!6e%t* t" in%rea*e the &eight

"# either re&ar' i,eih""' "r re&ar' $agnit!'e &hen va!ing an "!t%"$e

re*e%tive( rere*enting ri*,5aver*e an' ri*,5r"ne ehavi"r8

S!6e%t* are ta,en t" va!e ea%h "ti"n a%%"r'ing t" the #""&inge@!ati"n*F

g!ei1

J/Pri 1

gQ#!ei

1

ggreeni1 J/P1 ri 1gQ#greeni1

&here #!n%ti"n /r+ g8 i* a *i$e inear tran*#"r$ &ithin the "!n'* "#

0 an' 1F

/PrgQJ$a4R$inRPgPr 0F>Q0F>Q10

an' g J 1+ g " 1 an' g ; 1 i$( "ti$a+ ri*,5r"ne an' ri*,5

aver*e ehavi"r+ re*e%tive(

S!6e%t* &ere then a**!$e' t" generate a%ti"n* *t"%ha*ti%a(+ a%%"r'ing t"

a *ig$"i'a r"aiit( 'i*tri!ti"n #"r e4a$e:9+;9

8F

1

"int+ an' their *tan'ar' err"r* The grah* in /ig* ; an' >+ an' S!e5

$entar( /ig 1 *h"& the*e ti$e *erie* "# e##e%t *i9e* thr"!gh"!t the tria #"r

the regre**"r "# intere*t

N"teF S!e$entar( in#"r$ati"n i* avaiae "n theN a t! re Ne!r"*%ien%e &e*ite

ACNO-LEHMENTSThe a!th"r* &"!' i,e t" than, -at,in* #"r a'vi%e &ith the *t!'( an' the

$an!*%rit Thi* &"r, &a* *!"rte' ( the U Me'i%a Re*ear%h C"!n%i

TB8+ the Engineering an' Ph(*i%a S%ien%e* Re*ear%h C"!n%i M--8+the -e%"$e tr!*t ME-8 an' the R"(a S"%iet( M/SR8

AUT3OR CONTRIBUTIONSA #"!r a!th"r* &ere inv"ve' in generating the h("the*i*+ 'e*igning the

e4eri$ent an' &riting the $an!*%rit -here *e%i#i% r"e* %an e a**igne'FTE)B an' M-- !it the $"'e TE)B a%@!ire' an' ana(e' the'ata ME- *!ie' the ne%e**ar( in%i*ive &it M/SR *!ervi*e' ther"6e%t

COMPETINH INTERESTS STATEMENT

The a!th"r* 'e%are n" %"$eting #inan%ia intere*t*

P!i*he' "nine at httFGG& &&nat!re%"$Gnat!rene!r"*%ien%e

Rerint* an' er$i**i"n* in#"r$ati"n i* avaiae "nine at httFGGng nat!re%"$G

rerin t *an'er$i**i "n*

1 Ern*t+ MO . Ban,*+ MS 3!$an* integrate vi*!a an' hati% in#"r$ati"n in

a *tati*ti%a("ti$a #a*hi"n Nat!re ;1>+ ;29


16/45

"# %"n'iti"ne'+ !t n"t "# !n%"n'iti"ne'+ *ti$!i P*(%h" Rev =7+ >:2>2 19=08

1: i%,in*"n+ A . Ma%,int"*h+ N) Ca**i%a %"n'iti"ning in ani$a* Ann! RevP*(%h"

29+ >=7::;; 20028

1K C"!rvie+ AC+ a&+ N . T"!ret,(+ S Ba(e*ian the"rie* "# %"n'iti"ning in

a %hanging &"r' Tren'* C"gn S%i 10+ 29;9 200;8

2: Ni,i+ 3 . -atanae+M Pre#r"nta an' %ing!ate !nit a%tivit( '!ring ti$ing ehavi"r

in the $"n,e( Brain Re* 171+ 21:


17/45


2> Br"&n+ )- . Braver+ TS Learne' re'i%ti"n* "# err"r i,eih""' in the

anteri"r %ing!ate %"rte4 S%ien%e :07+ 111=8

2K It"+ S+ St!h"rn+ V+ Br"&n+ )- . S%ha+ ) Per#"r$an%e $"nit"ring (the

anteri"r %ing!ate %"rte4 '!ring *a%%a'e %"!nter$an'ing S%ien%e :02+ 1200+ 9=7


18/45

Figure S1: Plots show the effect of volatility at each timepoint when the data is

divided into a) error and rewarded trials and b) switch and stay trials. The effect

of volatility remains even if only error trials, or only rewarded trials, or only

switch trials, or only stay trials are included in the analysis.


19/45

Figure S:

!eural correlates of the value of the decision, during the "#$%"# phase &a), and the

prediction error during the '(!%T( phase. Slices are ta*en at: coronal, y+-,

sagittal, +, aial, /+. a) clusters are shown for a cluster forming threshold of

0+., &p23.34 cluster corrected). b) !o activations were present at corrected

levels. The figure shows effects of prediction error in the presma, and 5entral

tegmental 6rea for 07.8, p23.31 uncorrected. These regions overlap with regions

coding the predicted probability of an outcome during the %!T#569 phase. See

supplementary and main tet


20/45

S!e$entar( Tae 1

M"'e ara$*

ara$**ee%t"r8

Pre'i%ti"n

L"gi,eih"

Ba(e*ian 0 : =7 7K8 51;:2

Ba(e*ian 6!$ 0 : =7 7K8 51;:>

RL 1 : => K98 51>1K

RL : 7 =K 7:8 51;7=


21/45

Learning the value of information in an uncertain world.

Timothy E.J. Behrens, Mark W. Woolrich, Mark E. Walton, Matthew .!.

"ushworth

!u##lementary information.

$escri#tion of the Bayesian learner

3ere &e 'e*%rie a Ba(e*ian ar"a%h t" tra%,ing "!t%"$e r"aiitie* given a a*t

hi*t"r( "# *t"%ha*ti% tria "!t%"$e* in an envir"n$ent &here the*e r"aiitie* $a(

%hange At tria i a ne& "!t%"$e (ii* generate' In the %"nte4t "# thi* e4eri$ent+ (

i

i* a re&ar' either "n !e "r "n green+ &here (ii* "n !e &ith r"aiit( r

i+ an' "n

green &ith r"aiit( 15 ri8 The g"a "# the *!6e%t i* t" e*ti$ate r

i1#r"$ a

revi"!* 'ata ([05i

in "r'er t" %h""*e "ti$a( at the #""&ing tria 3"&ever+ in a

ti$e5var(ing envir"n$ent+ r $a( %hange #r"$ tria5t"5tria It i* ene#i%ia t" the

*!6e%t t" !n'er*tan' h"& r i* %hanging E##e%tive(+ it &i a"& the$ t" ,n"& h"&

e*t t" 'i*%"!nt in#"r$ati"n &ith a**ing tria* /"r e4a$e+ in an envir"n$ent

&here r i* ,n"&n t" e *tae+ a *!6e%t *h"!' %"n*i'er a hi*t"ri%a tria* &ith e@!a

&eight+ in "r'er t" "tain the e*t re'i%ti"n "# r at the #""&ing tria 3"&ever+ in

an envir"n$ent &here r $a( %hange+ re%ent tria* are $"*t i,e( t" e rere*entative

"# the %!rrent *it!ati"n+ *" an "!t%"$e?* re'i%tive "&er i* 'i$ini*he' a* it $"ve*

#!rther int" the a*t

The "ti$ait( "# thi* ar"a%h reie* "n the a**!$ti"n that the r"e$ i*

*"ve' in a Mar,"vian #a*hi"n That i*+ &hen an "!t%"$e i* "*erve'+ the ne&

"!t%"$e r"aiit( 'een'* "n( "n thi* "*erve' "!t%"$e an' "n the revi"!*

"!t%"$e r"aiit(+ !t n"t "n the #! hi*t"r( "# revi"!* "!t%"$e r"aiitie* A

Mar,"vian ar"a%h i* %"$eing *in%e it %an ar*i$"ni"!*( %at!re $an( "# the


22/45

*tati*ti%a reg!aritie* "# the envir"n$ent &ith"!t e4i%it( %"n*i'ering *tate* "#

the re$"te a*t In "ther &"r'*+ the envir"n$ent %an e &e rere*ente' &ith"!t

nee'ing t" *t"re the entire hi*t"r( "# e*ti$ate' re&ar' r"aiitie* an' "!t%"$e*

Un'er thi* Mar,"vian a**!$ti"n+ a 'iagra$ "# the r"aiit(5 tra%,ing r"e$ %an

e *een in Main #ig!re 1+ in the #"r$ "# a generative $"'e In the Mar,"vian *etting+

the %hangeaiit( "# r i* rere*ente' ( ri1

Wri8+ ie the r"aiit( that r &i $"ve

#r"$ the va!e rit" the va!e r

i1"ver the %"!r*e "# a *inge tria Thi* i* a r"aiit(

'i*tri!ti"n "n the ne4t re&ar' rate ri1

+ &hi%h $!*t ie et&een 0 an' 1 The

'i*tri!ti"n i* %entre' "n ri+ en%"'ing the ri"r e4e%tati"n that the re&ar' rate at the

ne4t tria+ ri1

+ 'een'* !"n the %!rrent re&ar' rate+ ri -e rere*ent thi*

'i*tri!ti"n a* a beta distributionF

r i1Wri + v 8 X ri + V 8 18

&here ri'e#ine* the $ean "# the 'i*tri!ti"n an' VYe4v8 'e#ine* the &i'th "# the

'i*tri!ti"n See aen'i4 #"r a 'e#initi"n "# the reara$eteri*e' eta 'i*tri!ti"n a*

!*e' here8 N"te that the e4"nentia i* ta,en *" that integra* "n the &i'th ara$eter

are er#"r$e' in "g *a%e A arge va!e "# v ea'* t" a &i'e 'i*tri!ti"n+ i$(ing r

$a( e e4e%te' t" %hange great( a *$a va!e "# v ea'* t" a narr"& 'i*tri!ti"n+

i$(ing r $a( e e4e%te' t" %hange ver( itte 3en%e+ &e re#er t" v a* the

v"atiit( a* it %"ntr"* the e4tent t" &hi%h the "*erve' "!t%"$e* "# 'e%i*i"n* %an

!'ate the e*ti$ate' re&ar' r"aiit( et&een tria* I# the *!6e%t treat* the

envir"n$ent e4a%t( a* 'e*%rie' a"ve+ the( &i $a,e the a**!$ti"n that the

envir"n$ent i* a&a(* *!*%etie t" the *a$e rate "# %hange+ an' the( &i there#"re

e !nae t" r"#it #r"$ eri"'* "# local *taiit( t" #"r$ $"re a%%!rate e*ti$ate* "#

the re&ar' rate I#+ in #a%t+ the envir"n$ent $a( g" thr"!gh *tae ha*e* an' v"atie


23/45

ha*e*+ the


24/45

*!6e%t $!*t !n'er*tan' thi* %hanging v"atiit( in "r'er t" er#"r$ "ti$a(

Mathe$ati%a(+ &e tra%, the %hanging v"atiit(+ vi+ in the *a$e $anner a* the re&ar'

rate+ ri + ( again a**!$ing a Mar,"vian r"gre**i"n The %hangeaiit( "# v i*

rere*ente' a* vi1

Wvi8 3ere+ the 'i*tri!ti"n '"e* n"t nee' t" e %"n*traine'

et&een 0 an' 1+ *" the #"r$ "# the tran*iti"na 'i*tri!ti"n %an e ta,en t" e

Ha!**ianF

vi1

Wvi+ ,8N v

i+ 8 28

&here Ye4,8 %"ntr"* the rate "# %hange "# v"atiit( Again+ the e4"nent i* ta,en

*" that integra* are er#"r$e' in "g *a%e the nat!ra *a%e #"r a varian%e

ara$eter A arge va!e #"r , ea'* t" a &i'e tran*iti"na 'i*tri!ti"n+ an' &"!'

'e*%rie an envir"n$ent that $"ve* @!i%,( et&een *tae an' v"atie eri"'* a

*$a va!e #"r , ea'* t" a narr"& 'i*tri!ti"n an' &"!' 'e*%rie an envir"n$ent

&h"*e v"atiit( %"!' "n( %hange *"&( In "ther &"r'*+ the ara$eter , rere*ent*

the 'i*tr!*t in the %"n*tan%( "# the envir"n$ent?* v"atiit(

%n alternative forward model.

The #"r&ar' $"'e 'e*%rie' a"ve i* "ti$a #"r tra%,ing 'ri#ting re&ar' rate*

3"&ever+ an aternative $"'e+ i* t" a**!$e that the re&ar' rate re$ain* %"n*tant #"r

a eri"' "# ti$e+ an' then 6!$* t" a ne& !n,n"&n va!e Thi* *e%"n' *%enari" i*+ in

#a%t+ the %a*e in "!r e4eri$ent+ !t *!6e%t* %ann"t ,n"& thi* g"ing int" the ta*,

3ere &e &i a( "!t *!%h a $"'e+ an' ater &e *h"& that the re'i%ti"n* "# the

Ba(e*ian earner are n"t *en*itive t" thi* $"'eing %h"i%e

/"r thi* $"'e+ &e $!*t intr"'!%e a atent variae Si+ &hi%h ta,e* the va!e

1 &hen a 6!$ in re&ar' rate "%%!r*+ an' 0 &hen the re&ar' rate re$ain* *tae

The %"n'iti"na r"aiit( 'i*tri!ti"n "n ri%an n"& e &rittenF


25/45

Zri [ ri[1

8Si Y 0pr

i

W ri[1

+Si

8 Y I.

:

0+18

SiY1

The v"atiit( in thi* $"'e i* ta,en a* the "g

r"aiit( "# &itne**ing a re&ar' rate 6!$ at ea%h

tria 3en%eF

pSi

8 Y 5i

&here+ ViYe4v

i8 -e %an n"& $arginai*e "ver the

atent variae Si+ t" eaveF

priW r

i[1+5

i8 Y 1 [5

i8r

i[ r

i[18 5

i0+18

An' the %"n'iti"na 'i*tri!ti"n "n vii* e4a%t(

a* in the "rigina $"'e a"ve vi1

Wvi+ ,8N

vi+ 8

N"te that the "n( 'i##eren%e et&een the t&" $"'e*

i* there#"re the e4a%t #"r$ "n the %"n'iti"na

'i*tri!ti"n "n vi+ an' that the grahi%a 'e*%riti"n

"# the r"e$ $ain #ig!re 18 i* i'enti%a #"r the t&"

$"'e*

&nferring r from #ast observations.

!ring the e4eri$ent+ the Ba(e*ian earner?*

ta*, i* t" !*e revi"!* 'ata 'e%i*i"n "!t%"$e*8 t"

e*tai*h a eie# a"!t the *tati*ti%* "# the re&ar'

envir"n$ent t" !*e in the ne4t tria The*e !n,n"&n

*tati*ti%* are the "!t%"$e r"aiit(+ r+ the v"atiit(+

v+ an' the rate "# %hange "# v"atiit(+ , It %an e

*h"&n that thi* i* '"ne in the $"*t e##i%ient !nia*e'


26/45

&a( ( !*ing

Ba(e*ian earning 1

Ba(e*ian earning i*

i$e$ente' !*ing

Ba(e* r!eF

-$!n,n"&n* 'ata%

-$'ata

!n,n"&n*%-$!n,n"&n*%

&here+ in thi*

e4eri$ent+ the 'ata

i* the hi*t"r( "#

'e%i*i"n "!t%"$e*

#r"$ a tria* the

!n,n"&n* are the

*tati*ti%* "# the

re&ar' envir"n$ent

r+ v+ an' , an' the

i,eih""'+ 'ata W

!n,n"&n*8+ i* the

generative $"'e

rere*ente'

grahi%a( in $ain

#ig!re 1 Thi*

e@!ati"n %an e !*e'

t" i$e$ent the

Ba(e*ian earner It


27/45

r"vi'e* the $ean* t" !*e the evi'en%e in the 'ata t" !'ate !n,n"&n*8+ theprior

r"aiit( 'i*tri!ti"n* "# "r eie# in8 the !n,n"&n ara$eter*prior t" *eeing the

'ata at ea%h tria+ in "r'er t" "tain -$!n,n"&n* 'ata8+ theposteriorr"aiit(

'i*tri!ti"n* "# "r eie# in8 the !n,n"&n ara$eter* a#ter *eeing the "!t%"$e "# the

tria

Ba(e* r!e %an e !*e' t" &rite '"&n the;ointr"aiit( 'i*tri!ti"n at tria

i "# a the ara$eter* rere*enting the *tati*ti%* "# the re&ar' envir"n$ent thr"!gh

the

%"!r*e "# the e4eri$entF

ipr

[i

+v[i

+ * Wy[i

8 Up7*8p7r1

8p7v1

8py;

W r;

8pr;

W r; Z1

+v;

8pv;

W v; Z1

+ *8

; Y1

&here (\i

i* a 'ata 'e%i*i"n "!t%"$e*8 ! t" an' in%!'ing tria i+ r\I

an' v\I

are the

entire hi*t"r( "# re&ar' rate* an' v"atiitie* at ea%h tria an' , i* the 'i*tr!*t in the

%"n*tan%( "# v"atiit( *ee $ain #ig!re 18 (i+ri an' vi rere*ent the 'ata+ re&ar'

rate an' v"atiit( at a arti%!ar tria+ i N"te that thi* $a,e* !*e "# the Mar,"vian

r"ertie* "# rian' v

iF

pr;

W r[;

+v[;

8 Ypr;

W r; 1

+v;

8

pv;

W v[;

+ *8 Ypv;

W v; 1

+ *8

Marginaiing integrating8 "ver the hi*t"r( "# r an' v revi"!* t" tria i give*

the "*teri"r r"aiit( 'i*tri!ti"n #"r the ara$eter* r+v+ an' , at tria i a*F

ipr

i

+vi

+ * Wy[i

8 U p*8 ZL Z p7r1

8pv1

8.[py;

W r;

8pr;

W r; I1

+v;

8pv;

W v; I1

+ *8dr[i

I1

dv[iI1

; Y1

e< 68

A *i$iar e4re**i"n %an e &ritten '"&n #"r the "*teri"r r"aiit( #"r eie# in8

the ara$eter* r+v+ an' , at tria i1F

i 1pr

i 1

+vi 1

+ * Wy[i 1

8 U p*8 ZL Z p7r1

8pv1

8.[py;

W r;

8pr;

W r; I1

+v;

8pv;

W v; I1

+

*8dr[i

dv[i


28/45

; Y1

I$"rtant(+ thi* %an then e $!%h *i$i#ie' ( *!*tit!ting in the e4re**i"n #"rthe


29/45

"*teri"r r"aiit( #"r eie# in8 the ara$eter* r+v+ an' , at the revi"!* tria i

given in e@!ati"n A8F

pri 1

+vi 1

+ * Wy[i 1

8 Up7yi 1

W ri 1

8

Z [Zp7r

i+v

i+* Wy

[i8pv

i 1W v

i+ *8dv

i

Vpr

i 1W r

i+v

i 18dr

i

e< =8

Thi* e@!ati"n r"vi'e* a &a( "# !'ating "!r eie# in the ara$eter* r+ v+ an'

, "n a tria5(5tria a*i* &ith"!t nee'ing t" *t"re the #! hi*t"r( "# 'e%i*i"n

"!t%"$e* "r the #! hi*t"r( "# *tati*ti%* "# the re&ar' envir"n$ent Thi* %an e *een+

a* n" 'ata "int* revi"!* t" (i1

an' n" ara$eter* revi"!* t" tria i e4i*t in the

e@!ati"n The entire hi*t"r( "# revi"!* re&ar'* i* *!$$arie' in the 6"int

r"aiit( 'i*tri!ti"n ri+v

i+,8 Thi* 'i*tri!ti"n+ the %!rrent eie# in the

ara$eter* r+ v+ an' ,+ i* a the in#"r$ati"n that i* %arrie' t" tria i1

-e er#"r$ the integra* in e@!ati"n B !*ing n!$eri%a integrati"n "n a >5

'i$en*i"na gri' the #ive 'i$en*i"n* rere*ent ri1

+vi1

+ri+v

ian' ,+ &ith integrati"n

"ver t&" "# the$+ rian' v

i8 Thi* eave* a :5'i$en*i"na gri' rere*enting the 6"int

'i*tri!ti"n ri1

+vi1

+,8+ &hi%h i* *t"re' et&een tria* At the eginning "# the

e4eri$ent the 6"int 'i*tri!ti"n i* *et t" e !ni#"r$+ re#e%ting the #a%t that the

*!6e%t* have n" in#"r$ati"n a"!t an( "# the ara$eter* e#"re the e4eri$ent

egin*

In "r'er t" $a,e a 'e%i*i"n at tria i1+ the Ba(e*ian earner re@!ire* an

e*ti$ate "# the re&ar' rate ri1

Thi* e*ti$ate i* the $ean "# the r"aiit(

'i*tri!ti"n "n ri1

The #ina %"$!tati"n #"r the Ba(e*ian earner i* there#"re t"

%"$!te the marginal 'i*tri!ti"n ri18 #r"$ the 6"int r"aiit( 'i*tri!ti"n

ri1

+vi1

+,8 Thi* i* "taine' ( $arginai*ing integrating "ver8 vi1

an' ,F

pri 1

8 Y[ [

pri 1

+vi 1

+*8dvi 1

d*


30/45

The %!rrent e*ti$ate "# the re&ar' rate i* then given ( the $ean "# thi* 'i*tri!ti"nF


31/45

r]i 1

Y# ri 1

8 Y [ ri 1pr

i 18dr

i 1

Again+ the integra* in thi* an' in the revi"!* e@!ati"n are er#"r$e' n!$eri%a(+

thi* ti$e a%r"** a :5'i$en*i"na gri' the three 'i$en*i"n* are ri1

+vi1

an' ,8

E'#eriment () *uman Behaviour

'odel performance:

In "r'er t" #!rther te*t the er#"r$an%e "# the $"'e+ &e %"$are' *!6e%t

er#"r$an%e t" re'i%ti"n* $a'e ( "th the Ba(e*ian earner &ith "th #"r&ar'

$"'e*8 an' ( %a**i%a rein#"r%e$ent5earning $"'e* -e !*e' t&" 'i##erent RL

$"'e* #"r the %"$ari*"n The #ir*t RL18 !*e' "ne earning rate #"r ea%h *!6e%t

The *e%"n' RL28 !*e' *earate earning rate* in ea%h ha*e v"atie an' *tae8 in

ea%h *!6e%t The RL1 $"'e ha* 1= ara$eter* 1 er *!6e%t8+ an' the RL2 $"'e

ha* :K ara$eter* 2 er *!6e%t8 that are #it t" the *!6e%t 'ata In %"ntra*t+ the

Ba(e*ian earner ha* n" ara$eter* that %an e #it t" the *!6e%t 'ata I# the

Ba(e*ian $"'e %an er#"r$ a* &e a* the rein#"r%e$ent5earning $"'e*+ 'e*ite

having n" #ree ara$eter* t" #it t" the *!6e%t 'ata+ it i* *tr"ng evi'en%e #"r the

Ba(e*ian rati"nae #"r *etting the earning rate

On t" "# ea%h "# the*e ^re'i%t"r_ $"'e*+ &e !*e' the ^*ee%t"r_ $"'e

'e*%rie' a"ve+ an' again #it $"'e ara$eter* !*ing Ba(e*ian e*ti$ati"n -e then

$a'e re'i%ti"n* "# *!6e%t 'e%i*i"n* #r"$ ea%h $"'e+ a**!$ing that the $"'e

&"!' %h""*e the a%ti"n &ith the higher va!e at ea%h tria -e %"$are' the*e

re'i%ti"n* again*t *!6e%t 'e%i*i"n* in t&" &a(* Tae 18 /ir*t+ &e %"$are'

again*t *!6e%t 'e%i*i"n* in a tria* 3"&ever+ '!e t" the nat!re "# the ta*,+ $"*t "#

the 'e%i*i"n* $a'e ( the *!6e%t* are trivia+ an' re'i%te' ( an( rea*"nae $"'e

-e there#"re #!rther te*te' the $"'e* ( e4a$ining "n( ^'i##i%!t_ tria* The*e


32/45

tria* are 'e#ine' a* th"*e &here the e4e%te' Pa*%aian va!e* "# the t&" "ti"n* are

*earate' ( e** than > "int* re*!t* in arenthe*e* in tae 18

e*ite the 'i##eren%e in n!$er "# ara$eter*+ the three $"'e* er#"r$ %"$ara(

"ver a tria*+ &ith the Ba(e*ian $"'e $argina( "!t5er#"r$ing the rein#"r%e$ent

earning $"'e* E4a$ining the 'ata #r"$ the 'i##i%!t tria*+ &here %han%e

er#"r$an%e i* >0+ RL1 re'i%t* *!6e%t 'e%i*i"n* at 19 etter than %han%e+ RL2

re'i%t* at 2: etter than %han%e+ an' the Ba(e*ian $"'e re'i%t* at 2K etter

than %han%e C"$aring the "g i,eih""' "# the three $"'e* &ith the #itte'

ara$eter* revea* ehavi"!ra 'ata t" e e*t #it ( the Ba(e*ian $"'e+ #""&e' (

the RL2 $"'e+ again %"n#ir$ing that the 'ata i* e*t re'i%te' ( $"'e* in &hi%h

the earning rate %an %hange

C"$aring the t&" "**ie #"r&ar' $"'e* #"r the Ba(e*ian earner reveae' a ver(

*i$iar eve "# re'i%ti"n "# the *!6e%t er#"r$an%e Thi* i* !n*!rri*ing a*+ "# a

>=0 tria* in the t&" *%he'!e*+ the t&" Ba(e*ian #"r&ar' $"'e* $a'e 'i##erent

re'i%ti"n* "# *!6e%t %h"i%e in "n( 9 tria*

E'#eriment &&) Learning rule related activity in the %++

F'% eperiment timing.

S!6e%t* er#"r$e' the *a$e a*i% ta*, a* in the ehavi"!ra e4eri$ent

'e*%rie' in the $ain te4t Ea%h tria &a* 'ivi'e' int" : ha*e*F $E+&$E -

/seconds 0ittered1) The ha*e &hen *!6e%t* $a'e the 'e%i*i"n The *!6e%t* %"!'

*ee the avaiae "ti"n*+ !t %"!' n"t re*"n' !nti a @!e*ti"n $ar, aeare' "n the

*%reen &2TE"3%L -/ seconds 0ittered1) A#ter $a,ing the 'e%i*i"n+ !t e#"re the

%"rre%t an*&er &a* reveae' M42&T4" 5ss1) S!6e%t* "*erve' the %"rre%t

"!t%"$e "# the tria in the %entre "# the *%reen I# the *!6e%t ha' g!e**e' %"rre%t( at

that tria+ the rie ar $"ve' #"r&ar' ( the 'i*tan%e a**"%iate' &ith that "ti"n


33/45

There &a* an inter5tria interva 56 seconds 0ittered1. T"ta tria ti$e &a* 20* "n

average -e ha' 120 tria*+ giving an average *%an ti$e "# 6!*t "ver ;0 $in!te*

F'% data

/MRI 'ata &ere a%@!ire' in 20 *!6e%t* "n a :T Sie$en* TRIO *%anner

ata &ere e4%!'e' #r"$ "ne *!6e%t '!e t" rai' hea' $"ti"n+ an' #r"$ "ne *!6e%t

a* $ean a%tivati"n eve* "ver the entire rain &ere > *tan'ar' 'eviati"n* #r"$ the

"!ati"n $ean The re$aining 1= *!6e%t* &ere in%!'e' in the ana(*i*

/MRI 'ata &ere a%@!ire' &ith a v"4e re*"!ti"n "# :4:4: $$

:

+ TRY:*+ TEY:0$*+/i angeY=7" The *i%e ange &a* *et t" 1>" an' a "%a 5*hi$ &a* aie' ar"!n'

the "rit"#r"nta %"rte4 t" $ini$ie *igna 'r""!t in thi* regi"n2+ &hi%h ha' een

i$i%ate' in "ther a*e%t* "# 'e%i*i"n5$a,ing in revi"!* *t!'ie* The n!$er "#

v"!$e* a%@!ire' 'een'e' "n the ehavi"!r "# the *!6e%t The $ean n!$er &a*

=:0+ giving a t"ta e4eri$ent ti$e "# ar"!n' ;2 $in!te*

/ie' Ma* &ere a%@!ire' !*ing a '!a e%h" 2 gra'ient e%h" *e@!en%e &ith

e%h"* at >19 an' 7K> $*+ an' reetiti"n ti$e "# ;;;$* ata &ere a%@!ire' "n a

K;4K;4;0 gri'+ &ith a v"4e re*"!ti"n "# :$$ i*"tr"i%

T15&eighte' *tr!%t!ra i$age* &ere a%@!ire' #"r *!6e%t aign$ent !*ing an

MPRAHE *e@!en%e &ith the #""&ing ara$eter*F V"4e re*"!ti"n 14141 $$: "n a

17K41924192 gri'+ E%h" ti$eTE8Y ;>: $*+ Inver*i"n ti$eTI8Y 900 $*+ Reititi"n

ti$e TR8Y 2200 $*


34/45

/MRI ana(*i*

/MRI ana(*i* &a* %arrie' "!t !*ing /MRIB?* S"#t&are irar( /SL:8

!ingle sub0ect #rocessing

Preprocessing

ata &ere rer"%e**e' !*ing /SL 'e#a!t "ti"n*F $"ti"n %"rre%ti"n &a*

aie' !*ing rigi' "'( regi*trati"n t" the %entra v"!$e; Ha!**ian *atia

*$""thing &a* aie' &ith a #! &i'th ha# $a4i$!$ "# >$$ rain $atter &a*

*eg$ente' #r"$ n"n5rain !*ing a $e*h 'e#"r$ati"n ar"a%h> high a** te$"ra

#itering &a* aie' !*ing a Ha!**ian5&eighte' r!nning ine* #iter+ &ith a :'B %!t5

"## "# 100*

'odel estimation

A genera inear $"'e &a* #it in re5&hitene' 'ata *a%e t" a%%"!nt #"r

a!t"%"rreati"n in the /MRI re*i'!a*8K The #""&ing regre**"r* &ere in%!'e' in

the $"'e *ee $ain te4t8F $E+&$E< ti$e* &hen the *!6e%t* &ere $a,ing the

'e%i*i"n*+ e#"re the( $a,e a re*"n*e &2TE"3%L< ti$e* et&een $a,ing a

re*"n*e an' the tr!e "!t%"$e eing 'i*a(e' M42&T4"< : *e%"n' eri"' a#ter

the "!t%"$e ha* een 'i*a(e' "n the *%reen $E+&$E'34L%T&L&T7< 'e%i*i"n

ha*e $"'!ate' ( the v"atiit( e*ti$ate at ea%h tria

&2TE"3%L'34L%T&L&T7< interva ha*e $"'!ate' ( the v"atiit( e*ti$ate at

ea%h tria M42&T4"'34L%T&L&T7< $"nit"r ha*e $"'!ate' ( the v"atiit(

e*ti$ate at ea%h tria E""4"< $"nit"r ha*e "n tria* &hen the *!6e%t %h"*e the

in%"rre%t "ti"n N"te+ #"r a v"atiit( intera%ti"n*+ the $"*t i,e(+ "r modal

va!e &a* !*e' #r"$ the "*teri"r 'i*tri!ti"n "n v"atiit( at ea%h tria

The*e regre**"r* &ere %"nv"ve' &ith the /SL 'e#a!t hae$"'(na$i% re*"n*e

#!n%ti"n Ha$$a #!n%ti"n+ 'ea(YK*+ *tan'ar' 'eviati"n Y:*8+ an' #itere' ( the

*a$e high a** #iter a* the 'ata


35/45

Field %nhomogeneity correction.

EPI 'i*t"rti"n* '!e t" $agneti% #ie' inh"$"geneit( &ere %"rre%te' !*ing the

a%@!ire' #ie'5$a* /ie'5$a* &ere !n&rae' !*ing a %"$$"n5ha*e regi"n

e4an*i"n ar"a%h7+ an' the ara$eter an' varian%e e*ti$ate* #r"$ the $"'e &ere

*atia( *hi#te' a%%"r'ing t" the "%a #ie'

8rou# data #rocessing.

S!6e%t* &ere aigne' t" the MNI1>2 te$ate !*ing a##ine regi*trati"n= A

genera inear $"'e &a* #it t" e*ti$ate the gr"! $ean e##e%t "# the regre**"r*

'e*%rie' a"ve9

%nference: 'ain effects.

/"r the $ain e##e%t* "# $E+&$E an' M42&T4"+ &hi%h &ere e4e%te' t"

have &i'e*rea' a%tivati"n* thr"!gh"!t the rain+ in#eren%e &a* %arrie' "!t !*ing

Ha!**ian ran'"$ #ie' the"r( an' %!*ter5a*e' thre*h"'ing10+ &ith a %!*ter5#"r$ing

thre*h"' "# Y2= an' a #a$i(5&i*e #a*e "*itive rate "# Y00> C!*ter* *!rviving

thre*h"'ing %an e #"!n' in tae 1

%nference: %nteraction effects.

The intera%ti"n et&een $"nit"ring an' v"atiit( &ere e4e%te' t" in'!%e

$"re #"%a a%tivati"n* than the $ain e##e%t* 'e*%rie' a"ve+ an' there#"re n"t t" e

*!itae #"r %!*ter5a*e' thre*h"'ing The( &ere a*" e4e%te' t" ie &ithin the

regi"n "# the anteri"r %ing!ate %"rte4 ACC8 /"r thi* rea*"n+ in#eren%e #"r thi*

intera%ti"n &a* %arrie' "!t !*ing Ha!**ian ran'"$ #ie'5a*e' v"4e5&i*e

thre*h"'ing+ &ith a *$a v"!$e %"rre%ti"n in the regi"n "# the ACC :=00 v"4e*8

Thi* gave a 5thre*h"' "# :> A regi"n "# ;2 v"4e* in the Anteri"r Cing!ate

C"rte4 *!rvive' thi* thre*h"' $a4 Y;2:+ at MNI 4Y5K+(Y2K+Y:; $$8 N"te

that+ a* re'i%te' n" regi"n in the rain "!t*i'e the Anteri"r Cing!ate C"rte4 &"!'

have *!rvive' at thi* thre*h"' eve /"r !r"*e* "# 'i*a(+ $ain #ig!re : i*

ren'ere' &ith a 5thre*h"' "# :1 1K: v"4e* in the ACC *!rvive' at thi* thre*h"'

eve8


36/45

Regi"n "# intere*t ana(*e*

-e er#"r$e' *evera regi"n "# intere*t ana(*e* "n 'ata ta,e #r"$ the regi"n "#

a%tivati"n in the ACC The*e ana(*e* &ere er#"r$e' in "r'er t" e4a$ine the nat!re

"# the BOL *igna #!%t!ati"n*+ an' t" e4%!'e the "**iiit( that the*e #!%t!ati"n*

%"!' e a%%"!nte' #"r ( *evera "tentia %"n#"!n'ing regre**"r*

-e t"", BOL 'ata in ea%h *!6e%t #r"$ a $a*, a%,5r"6e%te' #r"$ the

gr"! ACC a%tivati"n in the MONITOR4VOLATILIT regre**"r -e *earate'

ea%h *!6e%t?* ti$e*erie* int" ea%h tria+ an' re*a$e' ea%h tria t" a '!rati"n "#

20*+ *!%h that the 'e%i*i"n &a* re*ente' at 0*+ the re*"n*e &a* a"&e' at K* an'

the "!t%"$e &a* re*ente' #r"$ 12*51>* The re*a$ing re*"!ti"n &a* 100$*

-e then er#"r$e' a HLM a%r"** tria* at ever( tria ti$e"int in ea%h *!6e%t

in'een'ent( La*t(+ &e %a%!ate' the gr"! average e##e%t *ie* at ea%h ti$e"int+

an' their *tan'ar' err"r* The grah* in $ain #ig!re* ; an' >+ an' *!e$entar(

#ig!re 1 there#"re *h"& a ti$e*erie* "# e##e%t *ie* thr"!gh"!t the tria+ #"r the

regre**"r "# intere*t v"atiit(8

S!e$entar( #ig!re 1 *h"&* the e##e%t "# v"atiit( "n the BOL *igna *earate'

"!t #"r a8 err"r an' re&ar'e' tria* an' 8 *&it%h an' *ta( tria* It %an e *een that

the e##e%t "# v"atiit( i* re*ent in'een'ent "# &hether the tria in @!e*ti"n &a*

re&ar'e'+ an' in'een'ent "# &hether the *!6e%t ha' *&it%he' re*"n*e "n the tria

in @!e*ti"n N"ti%e that in #ig!re 1 an' $ain #ig!re* ;+> the e##e%t "# v"atiit( i*

i$ite' t" the eri"' &hen the *!6e%t i* "*erving the "!t%"$e "# the tria Thi* i* the

%r!%ia eri"' &hen the *!6e%t $!*t !*e their e*ti$ate' v"atiit(+ an' there#"re

!n%ertaint(+ in !'ating their e*ti$ate "# the #!t!re re&ar' i,eih""'


37/45

4ther signals of interest generated by the model.

-e er#"r$e' a *e%"n' ana(*i* "",ing #"r the e##e%t* "# a 'i##erent #"r$ "#

!n%ertaint( the r"aiit( that a tria &"!' e re&ar'e'8 t" %"ntra*t &ith the e##e%t

"# v"atiit( &e ha' area'( #"!n' The %r!%ia ti$e #"r thi* #"r$ "# !n%ertaint( i*

the INTERVAL ha*e &hen the *!6e%t i* a&aiting the "!t%"$e "# the tria Thi* ne&

ana(*i* #""&e' the *a$e r"%e'!re 'e*%rie' a"ve

-hie "!r "rigina e4eri$ent &a* %are#!( 'e*igne' t" %"ntr" #"r an(

"**ie %"n#"!n'ing e##e%t "# v"atiit(+ &e &ere n"t ae t" %"ntr" #"r "**ie

%"n#"!n'ing e##e%t* et&een "ther ara$eter* in "!r $"'e In thi* ana(*i* &e

there#"re in%!'e' "ther regre**"r* that $a( %"n#"!n' &ith re&ar' r"aiit( -e

a*" in%!'e' a re'i%ti"n err"r regre**"r an' "tentia %"n#"!n'*8+ rea*"ning that

rain regi"n* that %"'e #"r the re'i%te' r"aiit( "# a re&ar' &hen the *!6e%t i*

&aiting #"r an "!t%"$e+ &"!' e i'ea( a%e' t" %"'e #"r the err"r "n thi*

re'i%ti"n &hen the "!t%"$e &a* "*erve'

-e there#"re !*e' the #""&ing regre**"r*F

Three regre**"r* %"'ing the ha*e* "# the e4eri$ent

ECIE+INTERVAL+MONITOR three regre**"r* %"'ing their intera%ti"n &ith

v"atiit( the e*ti$ate' r"aiit( "# a re&ar' "n the %h"*en "ti"n '!ring the

ECIE an' INTERVAL ha*e* the va!e r"aiit( 4 "!t%"$e *ie8 "# the

%h"*en "ti"n '!ring the ECIE an' INTERVAL ha*e* the re'i%ti"n err"r

'!ring the MONITOR ha*e the a%t!a re&ar' "taine' '!ring the MONITOR

ha*e


38/45

-e #ir*t n"te that+ a* re'i%te' ( the ROI ana(*i* re*ente' in the $ain

$an!*%rit+ the e##e%t "# v"atiit( &a* !n%hange' ( the in%!*i"n "# the e4tra

regre**"r*

The "n( ne& regre**"r* t" have *igni#i%ant a%tivati"n* either at v"4e5&i*e "r %!*ter

%"rre%te' eve* &ereF

18 The ^va!e_ regre**"r* '!ring the ECIE ha*e *h"&e' *igni#i%ant %!*ter* inF

The ventr"5$e'ia re#r"nta %"rte4 $a4 Y;7;8+ the "*teri"r %ing!ate %"rte4

$a4 Y;;:8+ the e#t "*teri"r in*!ar $a4 Y;=8+ the *!ra$argina g(r!*+

iatera( $a4 Y;28+ the $i''e te$"ra g(r!* iatera( $a4 Y:9>8 an' the

*!eri"r "%%iita "e iatera( $a4 Y:=8 The*e a%tivati"n* %an e *een in

#ig!re S2

28 The ^va!e_ regre**"r '!ring the re'i%ti"n ha*e *h"&* *igni#i%ant e##e%t* in

vari"!* vi*!a an' attenti"n reate' area* in the "%%iita an' arieta "e* -e '" n"t

'etai the*e in'ivi'!a( here

:8 The err"r tria regre**"r *h"&* a *igni#i%ant e##e%t at the SMAGreSMA "!n'ar(

$a4 Y:K+ MNI 4Y5;+(Y0+Y>28

3"&ever+ e%a!*e "# the #ir$ h("the*e* a"!t the %"'ing "# re&ar' r"aiit( in

/i"ri" et a 200:+ &e "",e' #"r the re&ar'5re'i%ti"n *igna '!ring the

INTERVAL ha*e in the '"a$inergi% VTA at a "&er thre*h"' `001

!n%"rre%te'+ 2:8 + an' #"!n' a *igni#i%ant e##e%t $ain #ig!re K8


39/45

-e #!rther rea*"ne' that the *a$e regi"n* that %"'e' the re&ar' re'i%ti"n *igna

'!ring the INTERVAL ha*e+ &"!' e i'ea( "*iti"ne' t" %"'e #"r the re'i%ti"n

err"r *igna '!ring the MONITOR ha*e Thi* &a* the %a*e at the *a$e thre*h"'+

#ig!re S28

The ACC regi"n that %"'e* #"r the v"atiit(5reate' !n%ertaint( *h"&e' n" *igna

reating t" the %"ntra*ting #"r$ "# !n%ertaint(F the r"aiit( "# an "!t%"$e

'!ring the re'i%ti"n ha*e Y0K8

%##endi'

Beta 'i*tri!ti"n

The *tan'ar' #"r$ "# the eta 'i*tri!ti"n i* ara$eteri*e' a* #""&*F

Z 1 . 1

b.+Z8Y

[I Z81 8 [I8[Z8

0 c c 1

Thi* #!n%ti"n i* vai' #"r 4 et&een 0 an' 1+ an' ha* a $ean at d Y

The c

*harne** "# the 'i*tri!ti"n i* 'e#ine' ( the e##e%tive n!$er "# *a$e* 'ra&n

#r"$ a in"$ia 'i*tri!ti"n+ % Y c Thi* %an e ea*i( interrete' in ter$* "#

e*ti$ating the r"aiit( "# thr"&ing a hea' #r"$ a n!$er "# %"in t"**e* The $ean

i* the n!$er "# hea'* 'ivi'e' ( the n!$er "# t"**e*+ the *harne** "r %ertaint(8

"#

the 'i*tri!ti"n i* the n!$er "# t"**e*


40/45

-e reara$eteri*e thi* 'i*tri!ti"n in ter$* "# the e4e%te' re&ar' rate an'

the v"atiit(

r] Yd Y

[

c

v Y "g% 8 Y "g c8

A%tivate' %!*ter* #"r $ain e##e%t* "# ECIE an' MONITOR+ 4+(+ are the MNI1>2

*a%e "%ati"n* "# "%a $a4i$a in the 5*tati*ti%

$E+&$E

C!*ter *ie

V"4e*8

Brain regi"n 4 $$8 ( $$8 $$8 5*%"re

1;=22

Striate an'

e4tra*triat

e vi*!a

%"rte4

:=

5;;

2=

2K

5==

572

59K

590

512

512

52

5=

K09

>9:

>=;

>=;

/!*i#"r$

g(r!*

:;

5;;

5;;

5K2

520

51=

>79

;9

Ceree!$ 5:K 5>0 5:; K29

:=1;

S!eri"r

te$"r

a

K0

KK

5;0

5;K

20

1K

>11

;77


41/45

In*!ar

C"rte4

;K 12 5; >1

/r"nta

Oer%!!$

;2

:=

;2

1=

20

1=

5;

51;

5=

>07

;97

;9K

:2>0

Anteri"r

Cing!ate

C"rte4

K

;

2

5=

20

1K

5=

5=

2K

:0

;=

>2

>71

>2

;9;

;=7

Pre5SMA

SMA

2

;

0

K

1K

5=

K0

>K

K=

>17

>1;

:97

22:> In*!ar%"rte4 5;;

5:=

5:;

5:2

5;

52

1=

1;

2

K

K

10

;=7

;7K

;7;

;K7

/r"nta

Oer%!ar

C"rte4

5;K

52=

=

2;

0

0

;77

;=

H"!*

ai'!*

520

1=

2

0

52

52

;1=

:9

2209 Parieta

"er%!!$

5>;

5;=

5>K

52;

52K

522

1;

12

20

;==

;>9

;:1


42/45

Intraarieta

*!%!*

P"*t5%entra

S!%!*

5:0

5:;

5:=

5>K

5>;

5:;

>2

>;

;2

;;7

;;;

;:K

117: STN =

5=

522

51=

5;

5;

>>1

>07

hi"%a$!*

2; 5:; 5K ;7

Centr"$e'ian

thaa$!*

10

512

520

520

2

0

;K;

:9

9;0 "r*"atera

Pre#r"nta

C"rte4

:;

:0

:K

:2

2=

2=

:0

;K

;=

>;

2K

2;

:;

22

12

;92

;79

;K9

;;

;1;

;>> "r*"atera

re#r"nt

a

5:;

5:2

;K

>K

2;

20

;=:

;>:

2=; P"*teri"r

%ing!a

te

K 52; 2K ;19

1;0 Parieta

"er%!!$

;= 52; 1K ;0K


43/45

M42&T4"

C!*ter *ie

V"4e*8

Brain regi"n 4 $$8 ( $$8 $$8 5*%"re

:;K09 Parahi"%a$a

H(r!*

20 5;= 51; K1:

Striate an'

e4tra*triate

vi*!a an'%"rte4

;K

5;

1=

520

570

572

57;

572

0

K

510

510

K09

K09

>9;

>92

Intraarieta

*!%!*

:K

;2

K

=

5:K

5>2

5>2

5KK

5;K

5>;

>;

;0

;=

;;

>;

>;:

;9>

>12

;11

;=7

S!eri"r

te$"ra *!%!*

>0 5;2 12 >11

Ventra *triat!$ 510

1K

K

K

5=

5;

;7>

:;

Mi'rain 10 520 510 :9>

Me'i"'"r*a

thaa$!*

12 522 12 ;2

3i"%a$!* 22

520

52K

522

510

510

>;>

>;>

Pre$"t"r%"rte4

;;

>2

5;

K

>0

1K

>:1

>01


44/45

"r*"atera

re#r"nta

>2

>0

1;

:K

:;

20

>2>

;;>

17;K Anteri"r

%ing!ate %"rte4

5;

K

;

:;

:K

:2

K

10

2;

;9;

;7;

;;=

Pre5

SMAG*!eri

"r #r"nta

K :; ;; ;=0

>;= /r"nta

"er%!!$

52=

5:2

1K

1;

0

510

;9

;=

In*!ar %"rte4 5:2 1; 510 ;=

;17 Parieta

"er%!!$

5>= 51; 2= ;;>

Ventrare$"t"r 5>; 2 ;2 :>9

2>9 P"*teri"r

%ing!ate %"rte4

= 5:K 2; ;;1

:>K S!eri"r

te$"ra *!%!*

5>K 5>0 = :>:

Re#eren%e*

1 C"4+ R T Pr"aiit(+ #re@!en%(+ an' rea*"nae e4e%tai"n A$ )"!r Ph(*1;+ 151: 19;K8

2 ei%h$ann+ R+ H"tt#rie'+ ) A+ 3!tt"n+ C . T!rner+ R Oti$ie' EPI #"r

#MRI *t!'ie* "# the "rit"#r"nta %"rte4 Ne!r"i$age 19+ ;:05;1 200:8


45/45

: S$ith+ S M et a A'van%e* in #!n%ti"na an' *tr!%t!ra MR i$age ana(*i*an' i$e$entati"n a* /SL Ne!r"i$age 2: S! 1+ S20=519 200;8

; )en,in*"n+ M+ Banni*ter+ P+ Bra'(+ M . S$ith+ S I$r"ve' "ti$iati"n#"r the r"!*t an' a%%!rate inear regi*trati"n an' $"ti"n %"rre%ti"n "# raini$age* Ne!r"i$age 17+ =2>5;1 20028

> S$ith+ S M /a*t r"!*t a!t"$ate' rain e4tra%ti"n 3!$ Brain Ma 17+1;:5>> 20028

K -""ri%h+ M -+ Rie(+ B + Bra'(+ M . S$ith+ S M Te$"ra

a!t"%"rreati"n in !nivariate inear $"'eing "# /MRI 'ata Ne!r"i$age 1;+1:705=K 20018

7 )en,in*"n+ M /a*t+ a!t"$ate'+ N5'i$en*i"na ha*e5!n&raing ag"rith$Magn Re*"n Me' ;9+ 19:57 200:8

= )en,in*"n+ M . S$ith+ S A g"a "ti$i*ati"n $eth"' #"r r"!*t a##ineregi*trati"n "# rain i$age* Me' I$age Ana >+ 1;:5>K 20018

9 -""ri%h+ M -+ Behren*+ T E+ Be%,$ann+ C /+ )en,in*"n+ M . S$ith+ SM M!tieve inear $"'eing #"r /MRI gr"! ana(*i* !*ing Ba(e*ian

in#eren%e Ne!r"i$age 21+ 17:25;7 200;810 -"r*e(+ )+ Evan*+ A C+ Marrett+ S . Neein+ P A three5'i$en*i"na

*tati*ti%a ana(*i* #"r CB/ a%tivati"n *t!'ie* in h!$an rain ) Cere B""'/"& Meta 12+ 90051= 19928

11 !+ A ) . a(an+ P Un%ertaint(+ ne!r"$"'!ati"n+ an' attenti"n Ne!r"n;K+ K=1592 200>8

Documents

Learning the value of information in an uncertain world