Cognitive Reflection and Decision Making

Cognitive Reflection and Decision Making

Shane Frederick

P eople with higher cognitive ability (or IQ) differ from those with lowercognitive ability in a variety of important and unimportant ways. Onaverage, they live longer, earn more, have larger working memories, fasterreaction times and are more susceptible to visual illusions (Jensen, 1998). Despitethe diversity of phenomena related to IQ, few have attempted to understandoreven describeits influences on judgment and decision making. Studies on timepreference, risk preference, probability weighting, ambiguity aversion, endowmenteffects, anchoring and other widely researched topics rarely make any reference tothe possible effects of cognitive abilities (or cognitive traits).

Decision researchers may neglect cognitive ability because they are moreinterested in the average effect of some experimental manipulation. On this view,individual differences (in intelligence or anything else) are regarded as anuisanceas just another source of unexplained variance. Second, most studiesare conducted on college undergraduates, who are widely perceived as fairlyhomogenous. Third, characterizing performance differences on cognitive tasksrequires terms (IQ and aptitudes and such) that many object to because of theirassociation with discriminatory policies. In short, researchers may be reluctant tostudy something they do not find interesting, that is not perceived to vary muchwithin the subject pool conveniently obtained, and that will just get them intotrouble anyway.

But as Lubinski and Humphreys (1997) note, a neglected aspect does notcease to operate because it is neglected, and there is no good reason for ignoringthe possibility that general intelligence or various more specific cognitive abilities areimportant causal determinants of decision making. To provoke interest in this

y Shane Frederick is Assistant Professor of Management Science, Sloan School of Manage-ment, Massachusetts Institute of Technology, Cambridge Massachusetts. His e-mail address [email protected]".

Journal of Economic PerspectivesVolume 19, Number 4Fall 2005Pages 2542

neglected topic, this paper introduces a three-item Cognitive Reflection Test(CRT) as a simple measure of one type of cognitive ability. I will show that CRTscores are predictive of the types of choices that feature prominently in tests ofdecision-making theories, like expected utility theory and prospect theory. Indeed,the relation is sometimes so strong that the preferences themselves effectivelyfunction as expressions of cognitive abilityan empirical fact begging for a theo-retical explanation.

After introducing the CRT, I examine its relations with two important decision-making characteristics: time preference and risk preference. The CRT is thencompared with other measures of cognitive ability or cognitive style, including theWonderlic Personnel Test (WPT), the Need For Cognition scale (NFC) and self-reported SAT and ACT scores. The CRT exhibits considerable difference betweenmen and women, and I discuss how this relates to sex differences in time and riskpreferences. The final section discusses the interpretation of correlations betweencognitive abilities and decision-making characteristics.

The Cognitive Reflection Test (CRT)

Many researchers have emphasized the distinction between two types of cog-nitive processes: those executed quickly with little conscious deliberation and thosethat are slower and more reflective (Epstein, 1994; Sloman, 1996; Chaiken andTrope, 1999; Kahneman and Frederick, 2002). Stanovich and West (2000) calledthese System 1 and System 2 processes, respectively. System 1 processes occurspontaneously and do not require or consume much attention. Recognizing thatthe face of the person entering the classroom belongs to your math teacher involvesSystem 1 processesit occurs instantly and effortlessly and is unaffected by intel-lect, alertness, motivation or the difficulty of the math problem being attempted atthe time. Conversely, finding #19163 to two decimal places without a calculatorinvolves System 2 processesmental operations requiring effort, motivation, con-centration, and the execution of learned rules.1

The problem #19163 allows no role for System 1. No number spontaneouslysprings to mind as a possible answer. Someone with knowledge of an algorithm andthe motivation to execute it can arrive at the exact answer (138.43), but theproblem offers no intuitive solution.

By contrast, consider this problem:

A bat and a ball cost $1.10. The bat costs $1.00 more than the ball.How much does the ball cost? cents

Here, an intuitive answer does spring quickly to mind: 10 cents. But this impul-sive answer is wrong. Anyone who reflects upon it for even a moment would

1 For a discussion of the distinction between System 1 and System 2 in the context of choice heuristics,see Frederick (2002).

26 Journal of Economic Perspectives

recognize that the difference between $1.00 and 10 cents is only 90 cents, not $1.00as the problem stipulates. In this case, catching that error is tantamount to solvingthe problem, since nearly everyone who does not respond 10 cents does, in fact,give the correct response: 5 cents.

In a study conducted at Princeton, which measured time preferences usingboth real and hypothetical rewards, those answering 10 cents were found to besignificantly less patient than those answering 5 cents.2 Motivated by this result,two other problems found to yield impulsive erroneous responses were includedwith the bat and ball problem to form a simple, three-item Cognitive ReflectionTest (CRT), shown in Figure 1. The three items on the CRT are easy in the sensethat their solution is easily understood when explained, yet reaching the correctanswer often requires the suppression of an erroneous answer that springs impul-sively to mind.

The proposition that the three CRT problems generate an incorrect intuitiveanswer is supported by several facts. First, among all the possible wrong answerspeople could give, the posited intuitive answers (10, 100 and 24) dominate. Second,even among those responding correctly, the wrong answer was often consideredfirst, as is apparent from introspection, verbal reports and scribbles in the margin(for example, 10 cents was often crossed out next to 5 cents, but never the otherway around). Third, when asked to judge problem difficulty (by estimating theproportion of other respondents who would correctly solve them), respondents whomissed the problems thought they were easier than the respondents who solvedthem. For example, those who answered 10 cents to the bat and ball problemestimated that 92 percent of people would correctly solve it, whereas those whoanswered 5 cents estimated that only 62 percent would. (Both were consider-able overestimates.) Presumably, the 5 cents people had mentally crossed out10 cents and knew that not everyone would do this, whereas the 10 cents people

2 The bat and ball problem was subsequently used by Nagin and Pogarsky (2003) in a laboratoryexperiment on cheating. When respondents could obtain a $20 reward for correctly answering six triviaquestions, those answering 10 cents were significantly more likely to defy the experimenters request tocomplete the task without looking at the answers.

Figure 1The Cognitive Reflection Test (CRT)

(1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? _____ cents

(2) If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets? _____ minutes

(3) In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake? _____ days

Shane Frederick 27

thought the problem was too easy to miss. Fourth, respondents do much better onanalogous problems that invite more computation. For example, respondents missthe bat and ball problem far more often than they miss the banana and bagelproblem: A banana and a bagel cost 37 cents. The banana costs 13 cents more thanthe bagel. How much does the bagel cost?

The CRT was administered to 3,428 respondents in 35 separate studies over a26-month period beginning in January 2003. Most respondents were undergradu-ates at various universities in the midwest and northeast who were paid $8 tocomplete a 45-minute questionnaire that included the CRT and measures ofvarious decision-making characteristics, like time and risk preferences.3 On thepage on which the CRT appeared, respondents were told only: Below are severalproblems that vary in difficulty. Try to answer as many as you can.

Table 1 shows the mean scores at each location and the percentage answering0, 1, 2 or 3 items correctly. Most of the analyses that follow compare the low group(those who scored 0 out of 3) with the high group (those who scored 3 out of 3).The two intermediate groups (those who scored a 1 or 2) typically fell betweenthe two extreme groups on whatever dependent measure was analyzed. Thus,focusing attention on the two extreme groups simplifies the exposition andanalysis without affecting the conclusions.

Since more of the respondents were college students from selective schools,the two extreme groups that formed the basis for most statistical comparisonswere far more similar in cognitive abilities than two extreme groups formed fromthe general population. Thus, the group differences reported here likely understatethe differences that would have been observed if a more representative sample hadbeen used.

Cognitive Reflection and Time Preferences

The notion that more intelligent people are more patientthat they devalueor discount future rewards lesshas prevailed for some time. For example, in hisNew Principles of Political Economy (1834, pp. 57), Rae writes: The strength of theintellectual powers, giving rise to reasoning and reflective habits. . . brings before usthe future. . . in its legitimate force, and urge the propriety of providing for it.

The widely presumed relation between cognitive ability and patience has beentested in several studies, although rather unsystematically. Melikian (1959) askedchildren from five to twelve years of age to draw a picture of a man, which theycould exchange for either 10 fils (about 3 cents) or for a promissory noteredeemable for 20 fils two days later. Those who opted for the promissory notescored slightly higher on an intelligence test based on an assessment of those

3 There were three exceptions to this: 1) the participants from Carnegie Mellon University completedthe survey as part of class; 2) the 4th of July participants received only a frozen ice cream bar; and 3) theparticipants from the web study were unpaid, although they were entered into a lottery for iPods andother prizes.


drawings.4 Funder and Block (1989) paid 14 year-olds to participate in six exper-imental sessions. For each of the first five sessions, they could choose betweenreceiving $4 or foregoing (investing) their $4 payment for $4.80 in the sixth andfinal session. The teenagers with higher IQs chose to invest more of their money.In a follow-up to an extensive series of experiments investigating the ability ofpreschool children to delay gratification (Mischel, 1974), Shoda, Mischel andPeake (1990) found that the children who had waited longer before succumbing tothe impulse to take an immediately available inferior reward scored higher on theirSATs taken over a decade later. Similarly, Parker and Fischhoff (2005) found thatscores on a vocabulary test taken around age eleven predicted the individualstendency, at around age 18, to prefer a larger later reward over a smaller soonerone (for example, $120 in four weeks to $100 tomorrow). Using small real rewards,Benjamin and Shapiro (2005) found that respondents with higher SAT math scores

4 Given the relatively wide range of ages in this study, it remains unclear whether this relation isattributable to intelligence, per se, or to age, which might correlate with the development of artistic skillor patience or trust or some other specific trait that can be distinguished from cognitive ability.

Table 1CRT Scores, by Location

Locations at which data were collected Mean CRT score

Percentage scoring 0, 1, 2 or 3

N $Low0 1 2

High3

Massachusetts Institute of Technology 2.18 7% 16% 30% 48% 61Princeton University 1.63 18% 27% 28% 26% 121Boston fireworks displaya 1.53 24% 24% 26% 26% 195Carnegie Mellon University 1.51 25% 25% 25% 25% 746Harvard Universityb 1.43 20% 37% 24% 20% 51University of Michigan: Ann Arbor 1.18 31% 33% 23% 14% 1267Web-based studiesc 1.10 39% 25% 22% 13% 525Bowling Green University 0.87 50% 25% 13% 12% 52University of Michigan: Dearborn 0.83 51% 22% 21% 6% 154Michigan State University 0.79 49% 29% 16% 6% 118University of Toledo 0.57 64% 21% 10% 5% 138

Overall 1.24 33% 28% 23% 17% 3428

Notes: a Respondents in this study were people picnicking along the banks of the Charles River prior tothe July 4th fireworks display. Their ages ranged from 15 to 63, with a mean of 24. Many of the youngerparticipants were presumably students at a college in the Boston or Cambridge area. Most completed thesurvey in small groups of friends or family. Although they were requested not to discuss it until everyonein their group had completed it, some may have. (This, presumably, would elevate the CRT scoresrelative to most of the other studies in which participation was more closely supervised.)b The participants in this study were all members of a student choir group, which was predominatelyfemale. Unlike the other locations in which the numbers of men and women were comparable, 42 of 51participants in this study were women.c These were participants in two online studies, consisting of both college students and others whosee-mail addresses were obtained from online retailers.

Cognitive Reflection and Decision Making 29

(or their Chilean equivalent) were more likely to choose a larger later reward overa smaller sooner one (for example, to prefer a postdated check for $5.05 over a$5.00 check that can be immediately cashed). However, Monterosso et al. (2001)found no relation between the IQ of cocaine addicts and their imputed discountrates, and Kirby, Winston and Santiesteban (2005) found no reliable relationbetween students SAT scores and the amount they would bid for a delayedmonetary reward (although they did find that college grade point averages corre-lated positively with those bids).

Collectively, these studies support the view that cognitive ability and timepreference are somehow connected, though they have not generally focused on thetypes of intertemporal decisions over which cognitive ability exerts influence, norexplained why it does so.5 Toward this end, I examined the relation between CRTscores and various items intended to measure different aspects of time prefer-ence. As shown in Table 2, these included several hypothetical choices between animmediate reward and a larger delayed reward (items a through e), an immediatereward and a sequence of delayed rewards (items f through h), a shorter moreimmediate massage and longer more delayed massage (item i) and a smallerimmediate loss or a larger delayed loss (items j and k).6 Item l asked respondentsto state their maximum willingness to pay to have a book shipped overnight ratherthan waiting two weeks. Item m involved real money. Through a series of choices,respondents specified the smallest amount of money in four days that they wouldprefer to $170 in two months, and one of them was selected to actually receive oneof their choices. Items n through q asked respondents to report their impulsivity,procrastination, preoccupation with their future and concerns about inflation onan 11-point scale ranging from 5 (much less than the average person taking thissurvey today) to%5 (much more than the average person taking this survey today).7

Table 2 shows the responses of the low and high CRT groups for each of the17 items. The reported value is either the percentage choosing the patient optionor the mean response. The subscripts are the total number of respondents in thelow and high CRT groups who answered that item. The rightmost column reportsthe level of statistical significance of group differencesthe p-values from a chi-square test (for dichotomous responses) or a t-test (for continuous responses).

Those who scored higher on the CRT were generally more patient; theirdecisions implied lower discount rates. For short-term choices between monetaryrewards, the high CRT group was much more inclined to choose the later larger

5 Shoda, Mischel and Peake (1990) examined preschoolers willingness to wait (for additional marsh-mallows and pretzels and such) under four experimental conditions. They found that patience pre-dicted SAT scores in only one of their four conditionswhen the attractive but inferior reward wasvisually exposed and no distraction technique (such as think fun) was suggested. In the other threeconditions, patient behavior was actually negatively correlated with subsequent SAT scores.6 I assumed that delaying the extraction of a tooth involved a larger delayed loss, because during theintervening two weeks, one will suffer additional toothache pain, or additional disutility from dreadingthe forthcoming extraction pain, and that the only reason for not doing it immediately was that futurepain was discounted relative to immediate pain.7 Among the items in Table 2, men were more patient for items c, k and l, and they worried more aboutinflation. There were no significant differences between men and women for any other item.


reward (see items a and b). However, for choices involving longer horizons (itemsc through h), temporal preferences were weakly related or unrelated to CRT scores.

A tentative explanation for these results is as follows: a thoughtful respondentcan find good reasons for discounting future monetary outcomes at rates exceedingthe prevailing interest ratethe promiser could default, one may be predictablywealthier in the future (with correspondingly diminished marginal utility forfurther wealth gains), interest rates could increase (which increases the opportu-nity cost of foregoing the immediate reward), and inflation could reduce the futurerewards real value (if the stated amounts are interpreted as being denominated innominal units).8 Collectively, these reasons could, for example, justify choosing $9now over $100 in 10 years (item d), even though the implied discount rate of sucha choice (27 percent), exceeds market interest rates. However, such reasons are notsufficiently compelling to justify choosing $3400 this month over $3800 next month(which implies an annual discount rate of 280 percent). Hence, one observesconsiderable differences between CRT groups for choices like those in items a andb, where more careful deliberation or cognitive reflection should argue stronglyin favor of the later larger reward, but negligible differences for many of the otheritems, for which additional reflection would not make such a strong case for thelarger later reward (although one might argue that additional reflection should

8 Frederick, Loewenstein and ODonoghue (2002) offer a detailed and extended discussion of theconceptual dissection of imputed discount rates and discuss many reasons why choices betweenmonetary rewards are problematic for measuring pure time preference.

Table 2Intertemporal Behavior for Low and High CRT Groups(percentage choosing patient option or mean response)

Item Intertemporal Choice or Judgment

CRT group

Stat. Signif.Low High

a $3400 this month or $3800 next month 35%611 60%196 p & 0.0001b $100 now or $140 next year 22%409 37%297 p & 0.0001c $100 now or $1100 in 10 years 47%283 57%208 p & 0.05d $9 now or $100 in 10 years 40%364 46%277 p & 0.10e $40 immediately or $1000 in 10 years 50%135 59%83 n.s.f $100 now or $20 every year for 7 years 28%60 43%28 n.s.g $400 now or $100 every year for 10 years 64%44 72%43 n.s.h $1000 now or $100 every year for 25 years 52%295 49%99 n.s.i 30 min. massage in 2 weeks or 45 min. massage in Nov. 28%272 27%126 n.s.j Lose $1000 this year or lose $2000 next year 78%166 73%86 n.s.k Tooth pulled today or tooth pulled in 2 weeks 59%430 65%242 n.s.l Willingness to pay for overnight shipping of chosen book $4.54150 $2.18163 p & 0.0001m Smallest amount in 4 days preferred to $170 in 2 months $11672 $13382 p & 0.01n How impulsive are you? %1.01110 '0.2147 p & 0.001o How much do you tend to procrastinate? %1.05110 %1.0647 n.s.p How much do you think about your future? %2.49110 %1.6447 p & 0.01q How much do you worry about inflation? '1.16110 %0.1147 p & 0.01

Shane Frederick 31

reveal the wisdom of choosing the delayed 45-minute massage, since one will likelystill be alive, still be stressed and sore, still like massages, and still derive greaterbenefits from longer ones).

It appears that greater cognitive reflection fosters the recognition or appreci-ation of considerations favoring the later larger reward (like the degree to whichthe implied interest rate exceeds the rate offered by the market). However, itremains unclear whether cognitive reflection also influences other determinants ofintertemporal choices (like pure time preference). CRT scores were unrelated topreferences for the massage and tooth-pull items, which were intended as measuresof pure time preference. On the other hand, those in the low CRT group (thecognitively impulsive) were willing to pay significantly more for the overnightshipping of a chosen book (item l), which does seem like an expression of an aspectof pure time preference (the psychological pain of waiting for something desired).

Thus, despite the wide variety of items included to help address this issue,further resolution of the types of psychological characteristics associated withcognitive reflection (and other cognitive abilities) is still required. Toward this goal,respondents in some of the later studies were also asked to report several person-ality characteristics that seemed relevant to intertemporal choices (items n throughq). The self-perceived tendency to procrastinate was unrelated to CRT scores (bothgroups thought that they procrastinate more than their peers). However, the highCRT group perceived themselves to be significantly less impulsive, more concernedabout inflation and (curiously) less preoccupied with their future. The inflationresult supports the idea that the high-scoring groups are more likely to considersuch background factors in their choices between temporally separated monetaryrewards. Its interpretation, however, is ambiguous, since it implies a considerationof future conditions, but would be a justification for choosing the proximatereward.

Cognitive Reflection and Risk Preferences

In the domain of risk preferences, there is no widely shared presumptionabout the influences of cognitive ability and almost no research on the topic.Donkers, Melenberg and van Soest (2001) found that more educated respondentswere more tolerant of risk in hypothetical gambles: for example, they were morelikely to prefer an 80 percent chance of 45 florins (about $23) over a sure 30 florins(about $15). Benjamin and Shapiro (2005) found that students with higher scoreson the math section of the SAT (or its Chilean equivalent) were more likely tochoose according to expected value for real decisions involving small stakes (forexample, they were more likely to prefer a 50 percent chance to win $1.05 over asure 50 cents).

To assess the relation between CRT and risk preferences, I included severalmeasures of risk preferences in my questionnaires, including choices between acertain gain (or loss) and some probability of a larger gain (or loss). For some


items, expected value was maximized by choosing the gamble, and for some it wasmaximized by choosing the certain outcome.

The results are shown in Table 3a. In the domain of gains, the high CRT groupwas more willing to gambleparticularly when the gamble had higher expectedvalue (top panel), but, notably, even when it did not (middle panel). If all five itemsfrom the middle panel of Table 3a are aggregated, the high CRT group gambledsignificantly more often than the low CRT group (31 percent versus 19 percent;!2 $ 8.82; p & 0.01). This suggests that the correlation between cognitive abilityand risk taking in gains is not due solely to a greater disposition to computeexpected value or to adopt that as the choice criterion.9 For items involving losses(lower panel), the high CRT group was less risk seeking; they were more willingaccept a sure loss to avoid playing a gamble with lower (more negative) expectedvalue.

Two pairs of items (d versus o and h versus r) were reflections of one anotherin the domain of gains and losses. Prospect theory predicts that people will be morewilling to take risks to avoid losses than to achieve gains; that respondents willswitch from risk aversion to risk seeking when the valence of a gamble (or pros-pect) changes from positive to negative (Kahneman and Tversky, 1979). Thoughthis is spectacularly true for the low CRT group, who are much more willing togamble in the domain of losses than in the domain of gains, there is no suchreflection effect among the high CRT group, as shown in Table 3b. This resultstarkly shows the importance of considering cognitive ability when evaluating thedescriptive validity of a theory of decision making.10

Is the CRT Just Another IQ test?

Of the 3,428 respondents who completed the three-item CRT, many alsocompleted one or more additional cognitive measures: 921 completed the Won-derlic Personnel Test (WPT)a 12-minute, 50-item test used by the National

9 As expected, the gamble was not popular among either group for any of the anti-expected-valuegambles, since risk aversion and expected value both militate against it. However, any factors favoringthe gamble over the sure thing (for example, valuing the excitement of gambling or dismissing the sureamount as negligibly small) would be more likely to tip preferences in favor of the gamble among thoseless averse to it (the high CRT group, as judged from items a through h). The gambles in items i throughm were designed, in part, to have some chance of being chosen (the sure amounts were small, and theexpected values of the gambles were typically close to the sure amount). Including choices in which thegambles lacked these properties (for example, offering a choice between $4,000 for sure and a 50percent chance of $5000) would be pointless, because nearly everyone would reject the gamble, leavingno response variance to analyze. Item i comes close to illustrating this point.10 Although the descriptive accuracy of expected utility theory markedly improves for respondents withhigher scores, it cannot explain why a 75 percent chance of $200 is frequently rejected in favor of a sure$100, across all levels of cognitive ability, since this is a small fraction of ones wealth, and even a concaveutility function is approximately linear over small changes (Rabin, 2000).


Football League11 and other employers to assess the intellectual abilities of theirprospective hires; 944 completed an 18-item need for cognition scale (NFC),which measures the endorsement of statements like the notion of thinking ab-

11 Pat McInally, a Harvard graduate who later became a punter for the Cincinnati Bengals, was the onlycollege football player to score a perfect 50 out of 50 on the Wonderlica score attained by only oneperson in 30,000. Of the 921 respondents who took it in these studies, the highest score was a 47.

Table 3aRisk Seeking Behavior among Low and High CRT Groups

Item

Percentage choosing riskier option CRT group

Stat. Signif.Certain gains vs. Higher expected value gambles Low High

a $1,000 for sure or a 90% chance of $5,000 52%280 74%225 p & 0.0001b $100 for sure or a 90% chance of $500 56%95 78%92 p & 0.01c $1,000 for sure or a 75% chance of $4,000 37%264 57%102 p & 0.001d $100 for sure or a 75% chance of $200 19%843 38%475 p & 0.0001e $100 for sure or a 75% chance of $150 10%217 34%94 p & 0.0001f $100 for sure or a 50% chance of $300 47%68 75%20 p & 0.05g $500 for sure or a 15% chance of $1,000,000 31%341 60%135 p & 0.0001h $100 for sure or a 3% chance of $7,000 8%139 21%70 p & 0.01

Certain gains vs. Lower expected value gambles Low High

i $100 for sure or a 25% chance of $200 7%68 10%20 n.s.j $100 for sure or a 25% chance of $300 14%137 18%39 n.s.k $5 for sure or a 4% chance of $80 29%84 36%50 n.s.l $5 for sure or a 1% chance of $80 27%37 37%38 n.s.m $60 for sure or a 1% chance of $5000 19%153 32%31 n.s.

Certain losses vs. Lower expected value gambles Low High

n Lose $10 for sure or a 90% chance to lose $50 24%29 6%16 n.s.o Lose $100 for sure or a 75% chance to lose $200 54%339 31%141 p & 0.0001p Lose $100 for sure or a 50% chance to lose $300 61%335 55%109 n.s.q Lose $50 for sure or a 10% chance to lose $800 44%180 23%56 p & 0.01r Lose $100 for sure or a 3% chance to lose $7000 63%68 28%57 p & 0.0001

Table 3bThe Reflection Effect for Low and High CRT Groups

ItemPercentage choosing gamble

in the domain of gains and losses

CRT group

Low High

d $100 for sure or a 75% chance of $200 19%843 38%475o Lose $100 for sure or a 75% chance to lose $200 54%339 31%141h $100 for sure or a 3% chance of $7,000 8%139 21%70r Lose $100 for sure or a 3% chance to lose $7000 63%68 28%57


stractly is appealing to me (Cacioppo, Petty and Kao, 1984). Several hundredrespondents also reported their scores on the Scholastic Achievement Test (SAT)or the American College Test (ACT), the two most common college entranceexaminations.

Table 4 shows the correlations between cognitive measures. The numbersabove the diagonal are the sample sizes from which these correlations were com-puted (the number of surveys that included both measures). For example,152 respondents reported both SAT and ACT scores, and their correlation was 0.77.As expected, all measures correlate positively and significantly with one another.The moderate correlations suggest that all five tests likely reflect common factors,but may also measure distinct characteristics, as they purport to. I have proposedthat the CRT measures cognitive reflectionthe ability or disposition to resistreporting the response that first comes to mind. The need for cognition scale(NFC) is advanced as a measure of someones tendency to engage in and enjoythinking (Cacioppo and Petty, 1982), but relies on self-reports rather than ob-served behavior. The Wonderlic Personnel Test (WPT) is intended to measure apersons general cognitive ability, and the ACT and SAT are described as measuresof academic achievement.

Although the various tests are intended to measure conceptually distinguish-able traits, there are many likely sources of shared variance. For example, thoughthe CRT is intended to measure cognitive reflection, performance is surely aided byreading comprehension and mathematical skills (which the ACT and SAT alsomeasure). Similarly, though NFC and intelligence are distinguishable, the list ofways in which those with high NFC differ from those with low NFC (see Cacioppiet al., 1996) sounds very much like the list one would create if people were sortedon any measure of cognitive ability. Namely, those with higher NFC were found todo better on arithmetic problems, anagrams, trivia tests and college coursework, tobe more knowledgeable, more influenced by the quality of an argument, to recallmore of the information to which they are exposed, to generate more task relevantthoughts and to engage in greater information-processing activity.

The empirical and conceptual overlap between these tests suggests that theywould all predict time and risk preferences and raises the question of their relative

Table 4Correlations Between Cognitive Measures

CRT SAT SATM SATV ACT WPT NFC

CRT 434 434 434 667 921 944SAT .44 434 434 152 276 64SATM .46 .77 434 152 276 64SATV .24 .81 .28 152 276 64ACT .46 .77 .63 .67 466 190WPT .43 .49 .40 .37 .48 276NFC .22 .30 .21 .28 .30 .19

Shane Frederick 35

predictive validities. To assess this issue, I correlated the scores on the variouscognitive measures with composite indices of decision-making characteristicsformed from the time preference items in Table 2 or the risk preference items inTable 3. The composite scores registered the proportion of patient (or risk seek-ing) responses. For example, respondents might have been asked whether theyprefer $3,400 this month or $3,800 next month, whether they would prefer ashorter massage in two weeks or a longer one in November and how much theywould pay for overnight shipping of a book. Respondents who preferred the $3800,the longer later massage and who were willing to pay less than the median personfor express shipping would be coded as patient on all three items and wouldreceive a score of 1. If they were patient on two of the three items, they wouldreceive a score of 0.66, and so on. Thus, the indices are scores ranging from 0 to1, in coarse or fine increments depending on how many questions the respondentanswered.12

As shown in Table 5, the CRT was either the best or second-best predictoracross all four decision-making domains and the only test related to them all. Thus,

12 Composite indices were used to measure respondents general tendencies within a given decision-making domain and to permit aggregation across studies. However, unless respondents receivedidentical items, their scores are not perfectly comparable. This issue is not vital for establishing thepredictive validity of the CRT, because the correlations reflect the pattern plainly observable from theindividual items. However, for the purpose of comparing the cognitive measures, composite indices aremore problematic, because the full battery of cognitive tests was not typically given, and different studiesinvolved different items. For example, at Carnegie Mellon University, respondents answered items b, dand l from Table 2 and items a and d from Table 3. The CRT was the only cognitive measure obtainedfor these respondents. Thus, these particular items will be disproportionately represented in thecomposite decision-making indices with which the CRT is correlated. This problem can be overcome bydoing a pairwise comparison of cognitive measures only for those respondents who were given both.This more painstaking analysis generally confirms the implications of Table 5namely, the differenttests often function similarly, but the CRT is a bit more highly correlated with the characteristics ofinterest.

Table 5Correlations Between Cognitive Measures and Decision-Making Indices

Cognitivemeasure

Intertemporal choice

Choice under uncertainty(Preferences for gambles across domains)

Gains Losses

Preference forpatient option

Expected valuefavors gamble

Expected valuefavors sure gain

Expected valuefavors sure loss

CRT %0.12****3099 %0.22****3150 %0.08**1014 '0.12****1366SAT %0.07387 %0.09368 %0.07149 '0.12*275SATM '0.04387 %0.19***368 %0.05149 '0.11275SATV %0.15**387 '0.03368 %0.06149 '0.08275ACT %0.10*577 %0.14**549 %0.13*367 '0.01358WPT %0.00837 %0.13***904 %0.08287 '0.24****546NFC %0.06755 %0.13****875 %0.03497 '0.00215


for researchers interested in separating people into cognitive groups, the CRT is anattractive test: it involves only three items and can be administered in a minute ortwo, yet its predictive validity equals or exceeds other cognitive tests that involve upto 215 items and take up to 312 hours to complete (or which involve self-reports thatcannot be readily verified).

Sex Differences

Men scored significantly higher than women on the CRT, as shown in Table 6.The difference is not likely due to a biased sampling procedure, because there wereno significant sex differences for any other cognitive measure, except SATmathscores, for which there was a modest difference corresponding to national averages.Nor can it be readily attributed to differences in the attention or effort expendedon the survey, since women scored slightly higher on the Wonderlic test, which wasgiven under identical circumstances (included as part of a 45-minute survey thatrecruited respondents were paid to complete).

It appears, instead, that these items measure something that men have moreof. That something may be mathematical ability or interest, since the CRT itemshave mathematical content, and men generally score higher than women on mathtests (Benbow and Stanley, 1980; Halpern, 1986; Hyde, Fennema and Lamon, 1990;Hedges and Nowell, 1995). However, men score higher than women on the CRT,even controlling for SAT math scores. Furthermore, even if one focuses only onrespondents who gave the wrong answers, men and women differ. Womens mis-takes tend to be of the intuitive variety, whereas men make a wider variety of errors.For example, the women who miss the widgets problem nearly always give theerroneous intuitive answer 100, whereas a modest fraction of the men giveunexpected wrong answers, such as 20 or 500 or 1. For every CRT item (andseveral other similar items used in a longer variant of the test) the ratio ofintuitive mistakes to other mistakes is higher for women than for men. Thus,the data suggest that men are more likely to reflect on their answers and lessinclined to go with their intuitive responses.13

Because men score higher, the high CRT group is two-thirds men, whereasthe low CRT group is two-thirds women. Thus, the differences between CRTgroups may be revealing other male/female differences besides cognitive reflec-tion. To remove this confound, Table 7 presents results split by both sex and CRTscore for selected items, including a heretofore undiscussed item involving thewillingness to pay for a coin flip in which heads pays $100 and tails pays nothing.

Four facts are noteworthy. First, CRT scores are more highly correlated withtime preferences for women than for men; the low and high groups differ more.Second, as suggested by most prior research (Byrnes, Miller and Schafer, 1999,

13 One might draw the opposite conclusion from self-reports. Using the scale described earlier, respon-dents were asked How long do you deliberate before reaching a conclusion? Women reported higherscores than men (1.16 vs. 0.45; t186$ 2.32; p&0.05).


present an overview), women were considerably more risk averse than men, andthis remains true even after controlling for CRT score. Third, for the selected riskitems, CRT is as important as sex. In other words, high-scoring women behavealmost identically to low-scoring men (compare the upper left and lower right cellswithin each of the five items in the lower panel). Fourth, in contrast to the patternobserved for the time preference items, CRT scores are more highly correlated withrisk preferences for men than for women.

The curious finding that CRT scores are more tightly linked with time pref-erences for women than for men, but are more tightly linked with risk preferencesfor men than for women held for the other tests of cognitive ability, as well.Expressed loosely, being smart makes women patient and makes men take morerisks.14 This result was unanticipated and suggests no obvious explanation. The onlyrelated finding of which I am aware is in a study by Shoda, Mischel and Peake(1990), who found that the patience of preschool girls was strongly related to theirsubsequent SAT scores, but the patience of preschool boys was not.

Discussion

The instructions in studies of decision making commonly reassure respon-dents that there are no right or wrong answers. If this line is sincere, it impliesthat researchers will interpret such preferences as they would a choice between

14 This conclusion can also be expressed less loosely. First, when faced with three mathematicalreasoning problems (bat and ball, widgets and lilypads), certain responses that are plausiblyconstrued as manifestations of intelligence (5, 5 and 47) tend to correlate positively with certainother responses that are plausibly construed as expressions of patience (namely, an expressed willing-ness to wait for larger later rewards), and this tendency is more pronounced in women than men.Second, the production of the canonically correct responses tends also to correlate positively withcertain responses that are plausibly construed as expressions of risk tolerance (namely, an expressedwillingness to forego a smaller certain reward in favor of a probabalistic larger one), and this tendencyis more pronounced in men than in women. Third, sex differences in risk seeking and in the degree ofrelation to CRT scores was true only in the domain of gains. For the selected loss items (n through r inTable 3), there were no sex differences.

Table 6Sex Differences in Cognitive Measures

Test Men Women Significance of group difference

CRT 1.47 1.03 p & 0.0001SAT 1334 1324 n.s.SATmath 688 666 p & 0.01SATverbal 646 658 n.s.ACT 26.7 26.3 n.s.Wonderlic 26.2 26.5 n.s.NFC 0.91 0.85 n.s.


apples and orangesas a primitive that neither requires nor permits furtherscrutiny.

However, unlike a preference between apples and oranges, time and riskpreferences are sometimes tied so strongly to measures of cognitive ability that theyeffectively function as such a measure themselves.15 For example, when a choice

15 To encourage respondents to consider each choice carefully, and independently from the otheritems, several filler choices were inserted between the focal items. An analysis of these responsesshows that CRT scores are unrelated to preferences between apples and oranges, Pepsi and Coke, beer

Table 7Results Split by Both CRT and Sex(percentage choosing patient option or mean response)

Intertemporal choice or judgment Sex

CRT group Significanceof groupdifferenceLow High

$3400 this month or $3800 next month Men 39%170 60%84 p&0.01Women 39%252 67%51 p&0.001

$100 this year or $140 next year Men 21%106 34%161 p&0.05Women 25%194 49%70 p&0.001

$100 now or $1100 in 10 years Men 58%88 56%110 n.s.Women 43%186 57%68 p&0.05

$9 now or $100 in 10 years Men 40%123 43%178 n.s.Women 41%229 53%89 p&0.10

Willingness to pay for overnight shipping Men $4.0541 $1.9484 p&0.001of chosen book Women $4.5495 $2.1940 p&0.001

Risky choice or judgment(percentage choosing risky option or meanresponse) Sex

CRT group Significanceof groupdifferenceLow High

$100 for sure or a 75% chance of $200 Men 26%239 43%244 p & 0.0001Women 16%398 29%130 p & 0.01

$500 for sure or a 15% chance of $1,000,000 Men 40%68 80%41 p & 0.0001Women 25%109 38%37 n.s.

$1000 for sure or a 90% chance of $5000 Men 59%103 81%151 p & 0.001Women 46%166 59%65 p & 0.10

$100 for sure or a 3% chance of $7000 Men 6%36 30%44 p & 0.01Women 8%99 8%24 n.s.

Willingness to pay for a coin flip, where Men $13.0054 $20.0059 p & 0.001HEADS pays $100 and TAILS paysnothing.

Women $11.0012 $12.0036 n.s.

Shane Frederick 39

between a sure $500 and a 15 percent chance of $1,000,000 was presented torespondents along with an eight-item version of the CRT, only 25 percent of thosewho missed all eight problems chose the gamble, compared to 82 percent amongthose who solved them all. Should this result be interpreted to mean that choosingthe gamble is the correct response for this item?

The position that some preferences are better than others and that cognitiveability is one indicator of the better preference is not unprecedented. Savage(1954) argued that increased understanding ought to increase the frequency of thetruly normative response; that preferences that initially contradict some norma-tive principle may not survive thorough deliberation (what he termed reflectiveequilibrium).16 Stanovich and West (2000) extended these views, by arguing thatincreased understanding may arise from superior intellect (as well as from ex-tended deliberation or reflection or instruction). In response to those contendingthat judgments commonly labeled as errors or biases are actually equally goodanswers to different interpretations of the question (for example, Hilton, 1995),Stanovich and West argued that if smarter respondents were more likely to givecanonically correct answers, the other answers must not be equally good after all.17

Some, however, reject the notion that a correlation between (some measureof) cognitive ability and some particular response identifies the better response.For example, Sternberg (2000, pp. 697698) argues: [T]o characterize peoplewith high SAT scores as those who should set the norm for what is somehow trueor right seems to be off target. People with high SAT scores have high levels ofcertain kinds of cognitive abilities. They have no monopoly on quality of thinkingand certainly no monopoly on truth.

The prevalence of this view could be directly tested. Respondents could beshown the respective test scores of those who chose the sure $500 and those whochose the 15 percent chance of $1,000,000. If Sternbergs view is widely shared, thismanipulation would have no effect. If, on the other hand, the correlation betweencognitive ability and preference held normative force, making respondents awareof it would cause many of them to choose the gamble.

Of course, the weight that should be placed on the opinions of those withhigher cognitive abilities clearly depends on the type of decision in question. If onewere deciding between a fixed- and variable-interest mortgage, imitating ones

and wine or rap concerts and ballet. However, CRT scores are strongly predictive of the choice betweenPeople magazine and the New Yorker. Among the low CRT group, 67 percent preferred People. Among thehigh CRT group, 64 percent preferred the New Yorker.16 Slovic and Tversky (1974) use an eloquent and entertaining mock debate between Allais and Savageto illustrate opposing views on the related issue of whether the opinions of people who have deliberatedlonger over an issue ought to count more.17 Along similar lines, Bar Hillel (1991, p. 413) comments: Many writers have attempted to defendseemingly erroneous responses by offering interpretations of subjects reasoning that rationalizes theirresponses. Sometimes, however, this charitable approach has been misguided, either because thesubjects are quick to acknowledge their error themselves once it is pointed out to them, or because theinterpretation required to justify the response is even more embarrassing than the error it seeks toexcuse.


brilliant neighbor seems prudent. However, if one were deciding between an appleand an orange, Einsteins preference for apples seems irrelevant.

Thus, a relation between cognitive ability and preference does not, by itself,establish the correct choice for any particular individual. Two individuals withdifferent cognitive abilities may experience outcomes differently, which may war-rant different choices (for example, what magazines to read or movies to attend).But with respect to the example motivating this discussion, one must ask whetherit is plausible that people of differing cognitive abilities experience increments ofwealth as differently as their choices suggest. It seems exceedingly unlikely that thelow CRT group has a marked kink in their utility function around $W % 500,beyond which an extra $999,500 confers little additional benefit. It seems morereasonable, instead, to override the conventional caveat about arguing with tastes(Becker and Stigler, 1977) and conclude that choosing the $500 is the wronganswermuch as 10 cents is the wrong answer in the bat and ball problem.

Whatever stance one adopts on the contentious normative issues of whether apreference can be wrong and whether more reflective people make betterchoices, respondents who score differently on the CRT make different choices, andthis demands some explanation.

y I thank Dan Ariely, Scott Armstrong, Daniel Benjamin, Brett Boshco, Eric Bradlow, CraigFox, Kerri Frederick, Steve Garcia, Timothy Heath, James Hines, Eric Johnson, DanielKahneman, Robyn LeBoeuf, George Loewenstein, Leif Nelson, Nathan Novemsky, GregPogarsky, Drazen Prelec, Daniel Read, Eldar Shafir, Timothy Taylor, Catherine Tucker,Michael Waldman and Jaclyn Zires for comments received on earlier drafts. A special thanksto Steve Garcia, who coordinated most of the surveys generating the data summarized here. Asalways (but particularly in this case), the views expressed or implied are those of the authoralone.

References

Bar-Hillel, Maya. 1991. Commentary on Wol-ford, Taylor, and Beck: The Conjunction Fal-lacy? Memory and Cognition. 19:4, pp. 41214.Becker, Gary and George Stigler. 1977. De

Gustibus Non est Disputandum. American Eco-nomic Review. 67:2, pp. 7690.Benbow, Camilla P. and J. C. Stanley. 1980.

Sex Differences in Mathematical Ability: Fact orArtifact? Science. 210:4475, pp. 1262264.Benjamin, Daniel J. and Jesse M. Shapiro.

2005. Who is Behavioral? Cognitive Abilityand Anomalous Preferences. Working paper,Harvard University.

Byrnes, James P., David C. Miller and WilliamD. Schafer. 1999. Gender Differences in RiskTaking: A Meta-Analysis. Psychological Bulletin.125:3, pp. 36783.Cacioppo, John T. and Richard E. Petty. 1982.

The Need for Cognition. Journal of Personalityand Social Psychology. 42:1, pp. 11631.Cacioppo, John T., Richard E. Petty and

Chuan Feng Kao. 1984. The Efficient Assess-ment of Need for Cognition. Journal of Person-ality Assessment. 48:3, pp. 30607.Cacioppo, John T., Richard E. Petty, Jeffrey

A. Feinstein and W. Blair G. Jarvis. 1996. Dis-


positional Differences in Cognitive Motivation:The Life and Times of Individuals Varying inNeed for Cognition. Psychological Bulletin. 119:2,pp. 197253.Chaiken, Shelly and Yaacov Trope. 1999.

Dual-Process Theories in Social Psychology. NewYork: Guilford Press.Donkers, Bas, Bertrand Melenberg and Arthur

van Soest. 2001. Estimating Risk Attitudes Us-ing Lotteries: A Large Sample Approach. Jour-nal of Risk and Uncertainty. 22:2, pp. 16595.Epstein, Seymour. 1994. Integration of the

Cognitive and Psychodynamic Unconscious.American Psychologist. 49:8, pp. 70924.Frederick, Shane. 2002. Automated Choice

Heuristics, in Heuristics and Biases: The Psychologyof Intuitive Judgment. T. Gilovich, D. Griffin andD. Kahneman, eds. New York: Cambridge Uni-versity Press, pp. 54858.Frederick, Shane, George Loewenstein and

Ted ODonoghue. 2002. Time Discounting andTime Preference: A Critical Review. Journal ofEconomic Literature. 40:2, pp. 351401.Funder, David C. and Jack Block. 1989. The

Role of Ego-Control, Ego-Resiliency, and IQ inDelay of Gratification in Adolescence. Journal ofPersonality and Social Psychology. 57:6, pp. 1041050.Halpern, Diane F. 1986. Sex Differences in Cog-

nitive Abilities. Hillsdale, N.J.: Erlbaum.Hedges, Larry V. and Amy Nowell. 1995. Sex

Differences in Mental Test Scores, Variability,and Numbers of High-Scoring Individuals. Sci-ence. July 7, 269, pp. 4145.Hilton, Denis J. 1995. The Social Context of

Reasoning: Conversational Inference and Ratio-nal Judgment. Psychological Bulletin. September,118, pp. 24871.Hyde, Janet Shibley, Elizabeth Fennema and

Susan J. Lamon. 1990. Gender Differences inMathematics Performance: A Meta-Analysis.Psychological Bulletin. 107:2, pp. 13955.Jensen, Arthur R. 1998. The g Factor: The Science

of Mental Ability. Westport, Conn.: Praeger.Kahneman, Daniel and Shane Frederick. 2002.

Representativeness Revisited: Attribute Substi-tution in Intuitive Judgment, in Heuristics andBiases: The Psychology of Intuitive Judgment. T.Gilovich, D. Griffin and D. Kahneman, eds. NewYork: Cambridge University Press, pp. 4981.Kahneman, Daniel and Amos Tversky. 1979.

Prospect Theory: An Analysis of Decision Un-der Risk. Econometrica. 47:2, pp. 26391.Kirby, Kris N., Gordon C. Winston and Mari-

ana Sentiesteban. 2005. Impatience andGrades: Delay-Discount Rates Correlate Nega-tively with College GPA. Learning and IndividualDifferences. Forthcoming.

Lubinski, David and Lloyd Humphreys. 1997.Incorporating General Intelligence into Epide-miology and the Social Sciences. Intelligence.24:1, pp. 159201.Melikian, Levon. 1959. Preference for De-

layed Reinforcement: An Experimental Studyamong Palestinian Arab Refugee Children. Jour-nal of Social Psychology. 50, pp. 8186.Mischel, Walter. 1974. Processes in Delay of

Gratification, in Advances in Experimental SocialPsychology. L. Berkowitz, ed. San Diego, Calif.:Academic Press, pp. 24992.Monterosso, John, Ronald Ehrman, Kimberly

L. Napier, Charles P. OBrien and Anna RoseChildress. 2001. Three Decision-Making Tasksin Cocaine-Dependent Patients: Do They Mea-sure the Same Construct? Addiction. 96:12,pp. 1825837.Nagin, Daniel S. and Greg Pogarsky. 2003.

An Experimental Investigation of Deterrence:Cheating, Self-Serving Bias, and Impulsivity.Criminology. 41:1, pp. 50127.Parker, Andrew M. and Baruch Fischhoff.

2005. Decision-Making Competence: ExternalValidation through an Individual-DifferencesApproach. Journal of Behavioral Decision Making.18:1, pp. 127.Rabin, Matthew. 2000. Risk Aversion and

Expected-Utility Theory: A Calibration Theo-rem. Econometrica. 68:5, pp. 1281292.Rae, John. 1834. The New Principles of Political

Economy. Reprinted in 1905 as The SociologicalTheory of Capital. New York: Macmillan.Savage, Leonard J. 1954. The Foundations of

Statistics. New York: Wiley.Shoda, Yuichi, Walter Mischel and Philip K.

Peake. 1990. Predicting Adolescent Cognitiveand Self-Regulatory Competencies from Pre-school Delay of Gratification: Identifying Diag-nostic Conditions. Developmental Psychology. 26:6,pp. 97886.Sloman, Steven A. 1996. The Empirical Case

for Two Systems of Reasoning. Psychological Bul-letin. 119:1, pp. 322.Slovic, Paul and Amos Tversky. 1974. Who

Accepts Savages Axiom? Behavioral Science. 19:4,pp. 36873.Stanovich, Keith E. and Richard F. West. 2000.

Individual Differences in Reasoning: Implica-tions for the Rationality Debate? Behavioral andBrain Sciences. 22:5, pp. 645726.Sternberg, Robert J. 2000. The Ability is

not General, and Neither are the Conclu-sions. [Response to K. E. Stanovich and R.F.West.] Behavioral and Brain Sciences. 23:5,pp. 69798.


???????????????????????????????

?? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

?? ??????? ???????? ?????? ??? ??????? ?????? ?????????? ??? ???? ???? ??? ?????????? ????? ????????????????????????????????????????????????????

?? ???????????????????????????????????????????????????????????????????????? ????????? ?????????????????????????????????????????????????????????????????????????????????????

?? ????? ??????? ?????? ?????????????? ????????? ??? ???? ????? ???????? ??? ???? ?????????? ?????????????????????????????????????????????

?? ?????????????????? ?????????????????? ?????????????????? ??????????????? ????? ??????????????????????????? ?????? ?????????? ????????? ??????? ?? ?????? ????? ????????????? ??? ?????????? ?????????????????????????????????????????????????????????

?? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

?? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

?? ????? ??? ????????? ????????? ?????????????? ?????? ?????? ???? ????????? ??????? ??? ????????? ???????????????????????????????????????????????????????????????????????????????

?? ??????? ??????????? ???????????????? ?????? ???????????? ?????????? ??????????? ???? ???????????? ??????????????????????????????????????????????????????????

??? ?????? ??????????? ????? ??????? ?????? ?????????????? ?????????? ?????????? ?? ?????????????? ?????? ?????????????????????????????????????????????

??? ???????? ?? ????????????? ????????? ?? ???????? ????????? ?? ?????????? ???????? ?? ??????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???? ?????????? ?????? ????? ???? ??????????? ??????????? ???? ????????????????? ????????? ????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????? ??????????????????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????

???????????????????????????????????????????????????????????????????????????? ???????????????????????????????????????????????????????????????????????????????????????????????

??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ??????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????? ???????????????????????? ????? ??? ??????? ?????? ???????? ??? ????? ???????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????? ???????????????????????????????????????????

??? ????? ??????????? ?????? ??? ????????? ?????? ?? ????? ???????? ????????????? ?????????? ???? ?????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???? ???????? ?????? ?????????? ??????? ???? ???????????? ??????????????????????????????????? ?????????????? ???? ??????????????? ????????? ??? ????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????? ???????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????? ????????????????????????????? ?????????????? ????????????????? ??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????? ???????????????? ?????????? ??????????? ?????????? ??????????? ???? ???? ??????? ?????????? ????????? ?????????? ?????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????? ??? ????????? ??????? ??????? ?????? ????????? ????? ???? ??????????? ??????? ????????? ??????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????? ?????????? ?????? ???? ??????? ????????? ??????? ??? ?????????????? ??????? ???? ??????? ???????? ????????????????????????????????????????????????

??? ?????????? ????????? ????? ??????? ???????????????? ?????????? ?????? ??????????? ???????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????? ????? ???????? ??????? ???????????????? ???????? ????????????? ?????? ????????? ??????????? ???????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????? ????????????? ???????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????? ???????????? ???????? ?????????????????????????????????????????????????????????????

??? ?????????? ????????? ?????????? ???? ????? ???? ???? ???? ?? ?????? ??? ?? ??????????????????? ???????? ?????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????? ??????? ?????? ????????????????? ??????? ???????????? ????????? ??????? ????????? ?????????????????? ???? ????? ????????? ?????? ?????????? ???????????????? ???? ????????????? ?????????? ???????? ????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????? ?????????? ?????????? ???????? ????????????? ???????? ????????? ???????? ?????? ?? ???????? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????? ????????? ????????? ??????? ????? ?????? ??????? ??? ????????? ?????? ????? ?????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????? ?????????????????????????? ?????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????? ??? ????????? ????????? ??????? ?????? ?????????????? ???? ???????????? ??? ????????? ???????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????????????????????????????????? ????????????????????? ???????? ???????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????? ????????????? ?????????? ????????? ????????? ???????????????????? ?????????????????????????? ???? ????????? ???????????? ???? ??????????? ??? ???????? ??????? ????????????????? ????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????? ??? ?????????? ??????? ??? ?????? ?????? ?????????? ??????? ??? ???? ????????????? ??? ????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????? ??????????? ????? ??? ??????? ???????? ??? ??????? ?????? ??? ???? ???????????? ??? ????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????????????????????????????????????????????????????????????????????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????????? ???????? ???????? ????????? ?????? ??????????????? ????? ??????????? ????????????? ????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????? ??????????????????? ??????? ?????????????????? ????? ???? ??????? ??????????? ???????? ??? ??????????? ???? ????????????? ???????? ??? ?????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????? ?????????? ????????????? ?????????? ???????? ??? ????????????????????????? ????????? ????????? ???????????? ??? ??????????? ?????????? ????????????? ????????? ??????? ???????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ????? ??????? ?????? ?????????????????? ?????? ?????????? ????????? ???? ?????????????? ??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????? ??? ???????? ????????? ??? ????????? ?????? ??????????? ?????????? ????????????? ??????? ????????????????? ??? ???????? ?????????????? ?????????? ??????????? ???????? ???????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????? ????????????????? ???????? ??????? ??????? ??????????? ????? ??? ???????? ?????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

??? ??????????????????????????????????????????????????????????????????????????????????????????????????????? ????? ??? ??????????? ??? ???? ????? ???????? ????????? ???????????? ???? ?????????? ?????????? ???????? ?????????????????????????????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????? ?????????????? ?????? ????????? ?????? ?????? ??????? ??????? ???? ????????? ???? ??????? ???????????????????????????????????????????

???? ?????????????????????? ??????????????????????????????????? ??????????????????????? ????????????????????????????????????

???? ?????????? ????? ?????? ?????? ???????? ????? ?????? ?????? ?????? ??????????? ??? ????????????? ???????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??? ?????? ??? ????? ?????? ???????? ??? ???????? ?????? ????????????? ?????????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????????????????????????????????????????????????? ???????????????????????????? ?? ??????? ??????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????? ???????????? ???????? ???????? ??? ??????????? ????????? ??????? ?????????????????????? ??????????????????

???? ?????? ??????????? ?????? ??????? ???? ???????? ??? ???????????? ????? ??? ???? ????????? ??????????? ??????????????????????????????????????????????????????

???? ?????????????????? ?????????????????????????? ???????????? ??????????????????????? ?????? ??? ????????????? ??????????? ????? ?????????? ??? ????? ??????????? ???? ????????????????????? ????????????????? ???????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????? ??????????????????????????????????????????????????????????????????????????????????????????????

????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????? ??????? ?????? ???????????? ???????? ???? ???? ??????? ????? ?? ????????? ??????????????????????????????????????????????????

???? ??????? ??????????? ?????? ???????????? ?????????? ???????? ???? ?????????? ????????????? ????????? ?????????????????????? ????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????

???? ?????????????? ???????????????????????????????????? ???? ???????????????? ?????????????????????????????????????????????????????????????????????

???? ??????? ??????? ?????? ????? ?? ????????? ???? ?????? ??????? ????????????? ???????? ??? ??????????? ????????????????????????????????????

???? ??????????? ???????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????? ???? ????????? ?????? ??????????? ????? ??????????? ?????? ??????? ??????????? ??? ?????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????? ?????????? ?????? ????????????????? ??? ???????????????????????????? ??? ??????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????? ??????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????? ????????????????????????? ????????? ????????????????? ??????????????????? ??????? ??? ????????? ???

?????????? ????? ?????????????? ????????????? ??? ?????????????? ???????????? ?????????? ???? ????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????? ??? ??? ?????????? ?????? ??????????? ??????????????? ?? ?????????? ????????? ??? ??????????????????????????????????

???? ????? ?????????? ??????? ????????? ????? ?????????? ?????? ????? ?????? ??????????? ???? ????????????????????????? ??????? ??? ??????? ????? ???????? ???? ??????????? ???????????? ???????????? ???????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????? ?????????????????????????????????????????????????????????????????????????????????????????

?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????

???????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????

????????????????????????????????????????????????????????????????????????????????????????? ???? ?????????????? ????????? ??????? ?????? ?????? ????????? ???????? ??? ????? ??? ?????????? ??? ??????????

??? ???????? ?????????????????????? ?????????? ??? ?????? ???????????????? ????????? ??????????????????????????????????????????????????????????????

???? ????? ?????????????????? ???????? ?????? ??????????? ?????????? ??????????? ????? ?? ?????????? ?????? ?????????????????????????????????????????????????????????????

???? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????? ????? ???????? ?????????? ?????? ???? ???????????? ???????? ???????????? ???? ?????????? ????????? ??? ???????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????? ???????? ?????? ??????????? ???????????? ??? ?????????? ???????? ??? ???????????????? ???? ????????????????????????????????????????????????????????????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????? ????????? ????? ?????? ?????? ??????????? ???????? ????????? ???????? ?????? ?????? ??????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ??????? ?????????? ????? ??????? ?????????? ???? ????? ??????? ????????????????? ??????????? ???? ???????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????????? ?????? ??????? ??????????? ????????? ??? ??????? ????????? ??? ??????????? ?????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????? ??? ?????? ??????? ??? ???????????? ?????? ??????????? ????? ???? ?????????? ?????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ?????????? ?????? ????????????????????? ?????????????????????????????? ????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????? ????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????? ???????? ????? ??????? ?????? ??????????? ???????? ???? ???????????? ??? ??????? ?????????????????????????????????????????????????????

???? ????????? ?????? ?????? ?????????????? ??????????? ??????????? ???????? ????????????????????? ??????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ???????????????????????????????????????????????????????????????????????????????????????????????? ????????? ??????? ??????????? ???? ??????????? ??????????? ?????????????????????????????????????????????????????

???? ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ?????????????? ?????????????????????????????????????????????? ?????????? ??? ??????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

???? ????????? ?????? ??????? ?????? ??????? ??????? ???? ??????? ??? ???? ?????????? ????????? ??? ????????????

Documents

Cognitive Reflection and Decision Making