COMMENTARY

Of goals and habitsNathaniel D. Daw1

Princeton Neuroscience Institute and Department of Psychology, Princeton University,Princeton, NJ 08544

In the television series TheWire, addicts Bubblesand Johnny regularly engage in bizarre andelaborate schemes to obtain drugs, rangingfrom robbing an ambulance to stealing drugsby lowering a fishing line from a rooftop.That these fictional crimes can be both someticulously planned and yet focused onsuch narrow, shortsighted goals highlightsa gap in our understanding of how thereal brain deploys deliberative vs. automaticmechanisms to make decisions. On a stan-dard account, people can deliberatively eval-uate the consequences of candidate actions,which gives us our flexibility to dream upnovel plans. Alternatively, the brain can crys-tallize repeatedly successful behaviors intohabits: learned reflexes that free up resourcesby executing the behaviors automatically (al-though at the expense of inflexibility and, it isbelieved, underpinning pathological compul-sions). As with most dichotomies, the prob-lem with this view is that the world is not soblack and white. Much as the drug-seekingbehavior of addicts seems not to fit into eithercategory, for healthy behaviors also, neitherof these two sorts of decision making is verypractical on its own. In PNAS, Cushman andMorris suggest a hybrid of these mechanisms,and show behavioral evidence that humansuse it to plan actions (1).The study of Cushman and Morris (1)

draws on recent advances using computa-tional models of learning to make these strat-egies explicit enough that their hallmarkscan be measured in choice behavior. Deci-sions are often modeled as determined byone’s estimates of the rewards expected fromdifferent options. There are many differentmethods to compute these decision variables.Deliberative planning can be formalized byan algorithmic family, called “model-basedreinforcement learning” (2), which evaluatescandidate sequences of actions much like achess computer does, by exhaustively search-ing the “tree” of their future consequences,generated using a learned model of the taskcontingencies (like the rules of chess or themap of a maze) (Fig. 1A).The key feature of habits, in this view, is

that they instead rely on a simpler summaryof the end results of this computation, such

as the overall long-run reward expectedfollowing some action (2). This precomputa-tion gives them both their simplicity and in-flexibility. These summaries do not actuallyneed to be derived from exhaustive compu-tations using a model, but instead can belearned directly—although slowly—“model-free” from experience (3).Cushman and Morris (1) propose a hybrid

of these strategies, in which each system sim-plifies the problem faced by the other (Fig.1B). The hypothesis is that, in a multistepdecision problem, model-free learning selectsa goal or subgoal, then model-based planningfigures out how to get there. If I wanted totravel to Paris, a goal might be the airport; inchess, it might be forking the opponent’squeen. This is a useful division of labor be-cause, in general, finding the best action us-ing a model-based search is too complicatedbecause of the many possible trajectoriesof future action. However, given a singlegoal, finding a good path to it can be moremanageable. Meanwhile, although model-freelearning is computationally simple, it learnsslowly in large, multistep tasks. Narrowing itsattention to a smaller, more abstracted prob-lem—choice between a few candidate goals—lets its simplicity shine.The key to the experiments of Cushman

and Morris (1) is that these strategies learn indifferent ways from experience in trial-and-error decision tasks (4). Indeed, just how be-havior changes following a single, carefullyarranged trial of experience can reveal thingslike whether a goal was updated, and if sohow. The authors (1) harnessed the efficiencyof online data collection to zero-in on theserare informative choices across a large num-ber of subjects.In the studies (1), participants chose be-

tween actions, which led to intermediate sit-uations represented by different colors, andthen to monetary reward. If a choice that ledto the blue intermediate goal was followed bya large pay-off, subjects tended to chooseactions leading to blue again later. Thiswas true even if the later action needed toget to blue was a different or even com-pletely novel one, which suggests that plan-ning to get to blue was model-based. This is

because model-based planning, by evaluat-ing actions in terms of their predicted con-sequences, can generalize across differentways to achieve the same result (5). If Iwant to get to the airport, model-basedlearning can find a route there from any-where, whereas model-free learning canonly repeat previously successful routes.Surprisingly, a rewarded choice for blue

was repeated even if the subjects in the study(1) had failed to achieve the intended bluegoal on the initial choice, so that this rewardwas irrelevant to the true value of the goal.This apparent mistake is exactly what model-free selection predicts, because it assesses thesuccess of a strategy (here, choosing blue) interms of the reward received. Four experiments

$$$ ¢

$$$ ¢

A

B

Fig. 1. (A) A stylized choice problem with a large tree ofpossible future action sequences, posing a difficult searchproblem for model-based planning. (B) Cushman and Morris’(1) hypothesis: model-based planning can be simplified byinitial model-free selection which goal (here, color) to seek.

Author contributions: N.D.D. wrote the paper.

The author declares no conflict of interest.

See companion article on page 13817.

1Email: ndaw@princeton.edu.

www.pnas.org/cgi/doi/10.1073/pnas.1518488112 PNAS | November 10, 2015 | vol. 112 | no. 45 | 13749–13750

COMMEN

TARY

Dow

nloa

ded

by g

uest

on

July

6, 2

020

replicated this pattern and controlled for sev-eral alternative interpretations.This research relates to a broader lit-

erature on the tricks the brain uses tosimplify multistep choice (6). Cushmanand Morris’ (1) hypothesis is an instanceof a set of approaches called hierarchicalreinforcement learning, which decomposea multistep decision problem into a nestedset of choices at different levels of tempo-ral abstraction. There is mounting neuraland behavioral evidence that the brainuses such decomposition (7, 8). However,in principle, such abstraction is indiffer-ent to the learning mechanism: decisionsat any level could be approached bymodel-based or model-free learning (orany combination), with this particular di-vision being just one option. Indeed, a pre-vious study investigating such abstractionwas used to suggest that the top-levelchoices (over goals, in the present termi-nology) were actually model-based ratherthan, as Cushman and Morris suggest,model-free (8). Both might be right: thecurrent experiments (1) do not rule outthe coexistence of other approaches. How-ever, the earlier experiment omitted key con-trols included in the new study, so thoseresults are also compatible with model-freegoal selection.A similar point applies to the lower level

of Cushman and Morris’ (1) model. Severalother researchers have envisioned analogouslower-level choices as essentially model-free: “chunked,” stereotyped behavioral se-quences, like a tennis serve or a dance move.Some evidence suggests that this sort ofmemorization of extended routines may bethe real foundation of habits (9, 10). How-ever, such stereotyped sequences stand incontrast to Cushman and Morris’ (1) flex-ible model-based goal-seeking to find adesired result. For example, in one oftheir experiments, subjects reach the bluegoal by finding groups of numbers thatadd up to 21. Is temporal abstraction par-ticularly helpful for learning new tasks,generalization, and transfer (as Cushmanand Morris’ model would be), or is itmainly about streamlining the controlof well-learned behaviors (as with chunk-ing)? Are these two separate mecha-nisms? Another unresolved issue is howsubgoals themselves are discovered. Theefficiencies of hierarchical control areonly realized given an appropriate (andlimited) set of subgoals: the airport is a

broadly useful subgoal but many other placesare not (11).Another area in which the traditional di-

chotomy between model-based and model-free control has raised many questions isthe systems’ neural substrates. Although brainlesion studies show a dissociation between

The study of Cushmanand Morris draws onrecent advances usingcomputational modelsof learning to makethese strategies explicitenough that their hall-marks can be measuredin choice behavior.areas associated with either sort of control(12), neuroimaging and unit recording stud-ies in intact brains tend not to find such strictseparation (13). Instead, neural correlates ofdecision variables in many parts of the brainshow a mixture of both types of responses (4,14). Hybrid schemes, like that of Cushmanand Morris (1), in which the two mecha-nisms interact toward solving the sameproblem, may help to rationalize these results.It will be particularly interesting to study howthese neural signals behave in the new de-cision tasks, which exercise this interaction.Returning to the hapless Bubbles and

Johnny, this research may point toward theresolution of a longstanding problem inthe study of drug abuse and other disordersof compulsion. It is often argued that the com-pulsive nature of drugs might be rooted in

the inflexible character of habits (15). In-deed, patients with a range of compulsivedisorders show abnormal dominance ofmodel-free over model-based learning ona task similar to that of Cushman andMorris (16), and the neuromodulator dopa-mine (a common target of addictive drugs) isbelieved to drive model-free learning (17).However, although this might explain howaddicts acquire some stereotyped actions in-volved in drug consumption, like raising acigarette to one’s mouth, simply repeatingpreviously rewarded actions wholly fails toexplain many more deliberative and goal-ori-ented drug-seeking behaviors (18), as drama-tized by Bubbles’ bizarre (and presumablywholly novel) fishing escapade. Equally, onthe traditional, dichotomous account,model-based planning should have no interestin attaining a poisonous substance that hap-pens to affect preferences in a separate model-free system. This paradox is resolved if thatsame model-free learning mechanism extendsto choosing goals for the model-based system.Other previously disjoint phenomena of drugabuse, such as craving and attentional biases,may also be seen as further manifestations ofthese abstract habits.Finally, beyond compulsion, researchers

have found signs of dual-system decisionmaking in many diverse situations where weseem to be of two minds, from moral di-lemmas to procrastination to racism. Theinteractions between our more deliberativeand more automatic impulses, of the sortdocumented by Cushman and Morris (1),may therefore ultimately shed light on manyof the most puzzling of human experiences.

1 Cushman F, Morris A (2015) Habitual control of goal selection inhumans. Proc Natl Acad Sci USA 112:13817–13822.2 Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competitionbetween prefrontal and dorsolateral striatal systems for behavioral control.Nat Neurosci 8(12):1704–1711.3 Sutton RS (1988) Learning to predict by the methods of temporaldifferences. Mach Learn 3(1):9–44.4 Daw ND, Gershman SJ, Seymour B, Dayan P, Dolan RJ (2011) Model-based influences on humans’ choices and striatal prediction errors. Neuron69(6):1204–1215.5 Doll BB, Duncan KD, Simon DA, Shohamy D, Daw ND (2015) Model-based choices involve prospective neural activity. Nat Neurosci 18(5):767–772.6 Huys QJ, et al. (2015) Interplay of approximate planning strategies. ProcNatl Acad Sci USA 112(10):3098–3103.7 Botvinick MM, Niv Y, Barto AC (2009) Hierarchically organized behaviorand its neural foundations: A reinforcement learning perspective.Cognition 113(3):262–280.8 Dezfouli A, Balleine BW (2013) Actions, action sequences and habits:Evidence that goal-directed and habitual action control are hierarchicallyorganized. PLOS Comput Biol 9(12):e1003364.9 Dezfouli A, Balleine BW (2012) Habits, action sequences andreinforcement learning. Eur J Neurosci 35(7):1036–1051.

10 Smith KS, Graybiel AM (2013) A dual operator view of habitualbehavior reflecting cortical and striatal dynamics. Neuron 79(2):361–374.11 Solway A, et al. (2014) Optimal behavioral hierarchy. PLOS ComputBiol 10(8):e1003779.12 Balleine BW, O’Doherty JP (2010) Human and rodent homologies inaction control: Corticostriatal determinants of goal-directed and habitualaction. Neuropsychopharmacology 35(1):48–69.13 Doll BB, Simon DA, Daw ND (2012) The ubiquity of model-basedreinforcement learning. Curr Opin Neurobiol 22(6):1075–1081.14 Bromberg-Martin ES, Matsumoto M, Hong S, Hikosaka O (2010) Apallidus-habenula-dopamine pathway signals inferred stimulus values.J Neurophysiol 104(2):1068–1076.15 Everitt BJ, Robbins TW (2005) Neural systems of reinforcement fordrug addiction: From actions to habits to compulsion. Nat Neurosci 8(11):1481–1489.16 Voon V, et al. (2015) Disorders of compulsivity: A common biastowards learning habits. Mol Psychiatry 20(3):345–352.17 Montague PR, Dayan P, Sejnowski TJ (1996) A framework formesencephalic dopamine systems based on predictive Hebbian learning.J Neurosci 16(5):1936–1947.18 Tiffany ST (1990) A cognitive model of drug urges and drug-usebehavior: Role of automatic and nonautomatic processes. Psychol Rev97(2):147–168.

13750 | www.pnas.org/cgi/doi/10.1073/pnas.1518488112 Daw

Dow

nloa

ded

by g

uest

on

July

6, 2

020