80 vol 27 no 2 february 2014

Baron, J. (2008). Thinking and deciding(4th edn). Cambridge, MA:Cambridge University Press.

Bruine de Bruin, W., Parker, A.M. &Fischhoff, B. (2007). Individualdifferences in adult decision-makingcompetence. Journal of Personalityand Social Psychology, 92, 938–956.

Carroll, J.B. (1993). Human cognitiveabilities: A survey of factor-analyticstudies. Cambridge: Cambridge

University Press.Frederick, S. (2005). Cognitive reflection

and decision making. Journal ofEconomic Perspectives, 19, 25–42.

Kahneman, D. & Tversky, A. (1972).Subjective probability: A judgment ofrepresentativeness. CognitivePsychology, 3, 430–454.

Kahneman, D., & Tversky, A. (1973). Onthe psychology of prediction.Psychological Review, 80, 237–251.

Stanovich, K.E. (2009). What intelligencetests miss: The psychology of rationalthought. New Haven, CT: YaleUniversity Press.

Stanovich, K.E. (2011). Rationality and thereflective mind. New York: OxfordUniversity Press.

Stanovich, K.E. & West, R.F. (1998).Individual differences in rationalthought. Journal of ExperimentalPsychology: General, 127, 161–188.

Stanovich, K.E. & West, R.F. (2000).Individual differences in reasoning:Implications for the rationalitydebate? Behavioral and BrainSciences, 23, 645–726.

Stanovich, K.E. & West, R.F. (2008). Onthe relative independence of thinkingbiases and cognitive ability. Journal ofPersonality and Social Psychology, 94,672–695.

Stanovich, K.E., West, R.F. & Toplak, M.E.

It is a profound historical irony ofthe behavioural sciences that theNobel Prize was awarded forstudies of cognitive characteristics(rational thinking skills) that areentirely missing from the mostwell-known mental assessmentdevice in the behavioral sciences –the intelligence test. Intelligencetests measure important things,but not these – they do not assessthe extent of rational thought. Thismight not be such an omission if itwere the case that intelligence wasan exceptionally strong predictor ofrational thinking. However,research has found that it is amoderate predictor at best and thatsome rational thinking skills can bequite dissociated from intelligence.

In psychology and among the lay publicalike, high scores on intelligence testsare considered a mark of good thinking.

This is not unreasonable. It is now over100 years since Spearman first reported asingle general intelligence factor, known asg or ‘the positive manifold’ – the tendencyfor scores on different cognitive tests tocorrelate. Indeed, it is rare that a cognitiveprocess or phenomenon is found to beindependent of g (Carroll, 1993), so it isreasonable to assume that the construct of general intelligence encompasses mostof cognition.

It is revealing that when critics of IQtests try to argue that such tests fail toassess many essential domains ofpsychological functioning, they often point to non-cognitive domains, including:socio-emotional abilities, motivation,empathy, morality and interpersonal skills.By targeting these non-cognitive domains,these critics indirectly bolster theassumption that IQ tests exhaustivelyencompass the cognitive domain.

IQ ≠ RQOur research group has challenged IQtests much more fundamentally than theaverage critic. Our argument is thatintelligence, as conventionally measured,leaves out critical cognitive domains –domains of thinking itself.

We were led to this conclusionthrough our long-standing interest in theheuristics and biases research programmeinaugurated by Kahneman and Tverskyseveral decades ago (Kahneman &Tversky, 1972, 1973; Tversky &

Kahneman, 1974). In 2002 Kahnemanwon the Nobel Prize in Economics(Tversky died in 1996). A press releasefrom the Royal Swedish Academy ofSciences drew attention to the roots of theaward-winning work in ‘the analysis ofhuman judgment and decision-making bycognitive psychologists’. Kahneman wascited for discovering ‘how humanjudgment may take heuristic shortcuts thatsystematically depart from basic principlesof probability. His work has inspired a newgeneration of researchers in economics andfinance to enrich economic theory usinginsights from cognitive psychology intointrinsic human motivation.’

One reason that the Kahneman andTversky work was so influential was that it addressed deep issues concerning humanrationality. As the Nobel announcementnoted, ‘Kahneman and Tversky discoveredhow judgment under uncertaintysystematically departs from the kind ofrationality postulated in traditionaleconomic theory’. The thinking errorsuncovered by Kahneman and Tversky arethus not trivial errors in a parlour game.Being rational means acting to achieveone’s own life goals using the best meanspossible. To violate the thinking rulesexamined by Kahneman and Tversky hasthe practical consequence that we end upless satisfied with our lives.

The work of Kahneman and Tversky,along with that of many otherinvestigators, has shown how the basicarchitecture of human cognition makes allof us prone to these errors of judgementand decision making. But being prone tothese errors does not mean that we alwaysmake them. Every person, on someoccasions, overrides the tendency to makethese reasoning errors and instead makesthe rational response. More importantly,our research group has shown that thereare systematic differences amongindividuals in the tendency to make errorsof judgement and decision making.

The fact that there are systematicindividual differences in the judgementand decision-making situations studied byKahneman and Tversky means that there

refe

renc

esre

sour

ces

ques

tions

Kahneman, D. (2011). Thinking, fast andslow. New York: Farrar, Straus &Giroux.

Stanovich, K.E. (2009, Nov/Dec). Thethinking that IQ tests miss. ScientificAmerican Mind, 20(6), 34–39.

http://interviewscoertvisser.blogspot.co.uk/2009/11/keith-stanovich.html

Have you ever observed smart peopleacting stupidly?If so, doesn’t that mean that youimplicitly distinguish intelligence fromrationality, even if current cognitiveability tests do not?

ARTI

CLE

What intelligence tests missKeith E. Stanovich and Richard F. West on why we need rationality quotient (RQ)tests as well as IQ tests

are variations in important attributes ofhuman cognition related to rationality –how efficient we are in achieving our goals.It is curious that none of these criticalattributes of human thinking are assessedon IQ tests (or their proxies such asacademic ability tests) given that the typeof ‘good thinking’ studied by Kahnemanand Tversky was deemed worthy of a Nobel Prize. This anomaly is all thestranger given that most laypeople andscientists are prone to think that IQ testsare tests of ‘good thinking’.

What is rationality?To think rationally means taking theappropriate action given one’s goals andbeliefs (instrumental rationality), and

holding beliefs that are commensuratewith available evidence (epistemicrationality). Collectively, the many tasksof the heuristics and biases programme –and the even wider literature in decisionscience – comprise the operationaldefinition of rationality in moderncognitive science (Stanovich, 2011).Psychologists have extensively studiedaspects of instrumental rationality andirrationality and epistemic rationality and irrationality (see box for examples)

In short, we have an extensive and richset of operationalisations for the concept of rationality in modern cognitive science.None of these operational measures areassessed on common IQ tests. Yet people(including scientists) often talk as if theywere. For example, many conceptions ofintelligence define it as involving adaptivedecision making. Adaptive decisionmaking is the quintessence of rationality,but the items used to assess intelligence onwidely accepted tests bear no resemblanceto measures of decision making.

However, there is an important caveathere. Although the tests fail to assessrational thinking directly, it could beargued that the processes that are tappedby IQ tests largely overlap with variation in rational thinking ability. Perhapsintelligence is highly associated withrationality even though tasks tapping the

latter are not assessed directly on the tests.Here is where empirical research comes in,some of which has been generated by ourown research group. We have found thatmany rational thinking tasks showsurprising degrees of dissociation fromcognitive ability in university samples.Myside bias, for example, is virtuallyindependent of intelligence (Stanovich etal., 2013). Individuals with higher IQs in a university sample are no less likely toprocess information from an egocentricperspective than are individuals withrelatively lower IQs. Many classic effectsfrom the heuristics and biases literature –base-rate neglect, framing effects,conjunction effects, anchoring biases, andoutcome bias – are also quite independentof intelligence if run in between-subjectsdesigns (Stanovich & West, 2008).Correlations with intelligence have beenfound (e.g. Bruine de Bruin et al., 2007;Stanovich, 2009; Stanovich & West, 1998,2000) to be roughly (in absolutemagnitude) in the range of .20 to .35 forprobabilistic reasoning tasks and scientificreasoning tasks measuring a variety ofrational principles (covariation detection,hypothesis testing, confirmation bias,disjunctive reasoning, denominatorneglect, and Bayesian reasoning). In fact,even after corrections for reliability andrange restriction, this is a magnitude of

read discuss contribute at www.thepsychologist.org.uk 81

rationality testing

(2013). Myside bias, rationalthinking, and intelligence. CurrentDirections in Psychological Science,22, 259–264.

Tversky, A. & Kahneman, D. (1974).Judgment under uncertainty:Heuristics and biases. Science, 185,1124–1131.

Rational and irrational thinkingInstrumental rationality/irrationalitythe ability to display disjunctive reasoning in decision makingthe tendency to show inconsistent preferences because of framing

effects the default bias substituting gut feelings for difficult evaluationsover-weighting short-term rewards at the expense of long-term well-

beingchoices affected by vivid stimulidecisions affected by irrelevant context

Epistemic rationality/irrationalitythe tendency to show incoherent probability assessmentsoverconfidence in knowledge judgementsignorance of base rates the tendency not to seek to falsify hypotheses the tendency to try to explain chance eventsself-serving personal judgementsevaluating evidence with a myside biasignorance of the alternative hypothesis

82 vol 27 no 2 february 2014

rationality testing

correlation that allows for substantialdiscrepancies between intelligence andrationality. Intelligence is thus noinoculation against many of the sources of irrational thought.

Developing a rationality testIf we want to assess differences in rational thinking, we need to assess thecomponents of rational thought directly,with an RQ (rationality quotient) test.Practically, in terms of the cognitivetechnology now in place, this is doable.There is nothing conceptually ortheoretically preventing us fromdeveloping such a test. We know thetypes of thinking processes that would beassessed by such an instrument, and wehave in hand prototypes of the kinds oftasks that would be used in the domainsof both instrumental rationality andepistemic rationality. Thus, there are nomajor roadblocks preventing thedevelopment of an RQ test. Indeed, this is what our research lab is doing with thehelp of a three-year grant from the JohnTempleton Foundation. Specifically, weare attempting to construct the firstprototype of an assessment instrumentthat will comprehensively measureindividual differences in rational thought.

Rational thought can be partitionedinto fluid and crystallised components byanalogy to the fluid and crystallised formsof intelligence described by theCattell/Horn/Carroll theory of intelligence(Carroll, 1993; see Table 1). Fluidrationality encompasses the process part

of rational thought – the thinkingdispositions of the reflective mind that leadto rational thought and action. Crystallisedrationality encompasses all of theknowledge structures that relate to rational

thought.Unlike the case of fluid

intelligence, fluid rationality is likely to be multifarious –composed of a variety ofdifferent cognitive styles anddispositions. As a multifariousconcept, fluid rationalitycannot be assessed with a single type of item in themanner that the homogeneousRaven Progressive Matrices,for example, provides a goodmeasure of fluid intelligence.

The concept of crystallisedrationality has twosubdivisions, as shown inTable 1. Knowledge structuresthat promote rational thoughtare termed crystallisedfacilitators. Knowledgestructures that impede rationalthought are termedcrystallised inhibitors. Each of these subcategories ofcrystallised rationality is, likefluid rationality, multifarious.

Without learning crystallised facilitators,people will lack declarative knowledge that is necessary in order to act rationally.However, not all crystallised knowledge is helpful – either to attaining our goals(instrumental rationality) or to havingaccurate beliefs (epistemic rationality).Hence the category of crystallisedinhibitors (e.g. astrology) in the table.

Table 1 should not be mistaken for the kind of ‘good thinking styles’ lists thatappear in textbooks on critical thinking. In terms of providing a basis for a systemof rational thinking assessment, it goesconsiderably beyond such lists in anumber of ways. First, many textbookattempts at these lists deal only withaspects of fluid rationality and give shortshrift to the crystallised knowledge basesthat are necessary supports for rationalthought and action. In contrast, ourframework for rationality assessmentemphasises that crystallised knowledgeunderlies much rational responding(crystallised facilitators) and thatcrystallised knowledge can also be thedirect cause of irrational behaviour(crystallised inhibitors).

More importantly, the conceptualcomponents of the fluid characteristics andcrystallised knowledge bases listed in Table1 are each grounded in a task or paradigm

Components of Rationality

Fluid RationalityCrystallised Rationality

Crystallised Facilitators Crystallised Inhibitors

Resistance to miserlyinformation processing

Probabilistic and statisticalreasoning Belief in the paranormal and

in intuition; value placed onungrounded knowledgesources

Absence of irrelevantcontext effects in decisionmaking

Practical numeracy

Risky decision making:maximising expected value Risk knowledge Overreliance on

introspection

Proper knowledgecalibration: avoidingoverconfidence

Knowledge of scientificreasoning

Dysfunctional personalbeliefs

Avoidance of myside bias Financial literacy andeconomic thinking Unrealistic optimism

Openminded/objectivereasoning styles

Prudent attitude toward thefuture

Emotional regulation relatedto reward; sensitivity toemotions

Table 1. Rational thinking skills in the Stanovich & West framework

The tools of rationality represent declarative knowledge that isoften incompletely learned, inaccurate or not acquired at all

from cognitive science. That is, they arenot just potentially measurable, but in facthave been operationalised and measured atleast once in the scientific literature – andin many cases (e.g. context effects indecision making; probabilistic reasoning)they have been studied extensively.

Most of the paradigms that will be usedin our assessment device are therefore wellknown to most cognitive psychologists.For example, there are many paradigmsthat have been used to measure theresistance to ‘miserly informationprocessing’, the first major dimension of fluid rationality in Table 1. In theCognitive Reflection Test, designed byShane Frederick (2005), for example, themost famous item reads: A bat and a ballcost £1.10 in total. The bat costs £1 morethan the ball. How much does the ballcost? When they answer this problem,many people give the first response thatcomes to mind – 10 pence – withoutthinking further and realising that thiscannot be right. The bat would then haveto cost £1.10 and the total cost would thenbe £1.20 rather than the required £1.10.People often do not think deeply enoughto realise their error, and cognitive ability(as measured by IQ) is no guaranteeagainst making the error. Frederick (2005)found that large numbers of highly selectuniversity students at the MIT, Princeton,and Harvard were cognitive misers – theyresponded that the cost was 10 pence,rather than the correct answer… 5 pence.

The cognitive miser tendencyrepresents a processing problem of thehuman brain – it is a problem of fluidrationality. The second broad reason thathumans can be less than rational derivesfrom a content problem – when the toolsof rationality (probabilistic thinking, logic,scientific reasoning) represent declarativeknowledge that is often incompletelylearned, inaccurate or not acquired at all.To illustrate how assigning the correctprobability values to events is a criticalaspect of rational thought, consider thefollowing problem in which both medicalpersonnel and laypersons are often caughtout:

Imagine that the XYZ virus causes a serious disease that occurs in 1 in every1000 people. There is a test to diagnose thedisease that always indicates correctly thata person who has the XYZ virus actuallyhas it. However, the test has a false-positiverate of 5 per cent – the test wronglyindicates that the XYZ virus is present in 5 per cent of the cases where it is not. Nowimagine that we choose a person randomlyand administer the test, and that it yields apositive result (indicates that the person isXYZ-positive). What is the probability thatthe individual actually has the XYZ virus?

The point is not to get the preciseanswer so much as to see whether you arein the right ballpark. The answers of manypeople are not. The most common answergiven is 95 percent. Actually, the correctanswer is approximately 2 per cent! Whyis the answer 2 per cent? Of 1000 people,just one will actually be XYZ-positive. If the other 999 are tested, the test willindicate incorrectly that approximately 50 of them have thevirus (.05 multiplied by999) because of the 5percent false-positiverate. Thus, of the 51patients testing positive,only one (approximately2 per cent) will actuallybe XYZ-positive. Inshort, the base rate is such that the vastmajority of people do not have the virus.This fact, combined with a substantialfalse-positive rate, ensures that, in absolutenumbers, the majority of positive tests willbe of people who do not have the virus.

Rational thinking errors due to suchknowledge gaps can occur in a potentiallylarge set of domains including probabilisticreasoning, causal reasoning, knowledge ofrisks, logic, practical numeracy, financialliteracy, and scientific thinking (theimportance of alternative hypotheses, etc.).

In other publications (e.g. Stanovich,2011) we have provided numerousexamples of tasks like those describedabove that measure each of the rationalthinking concepts in Table 1. Ourframework illustrates the basis for ourposition that there is no conceptual barrierto creating a test of rational thinking.However, this does not mean that it wouldbe logistically easy. Quite the contrary, wehave stressed that both fluid andcrystallised rationality are likely to be more multifarious than their analogousintelligence constructs. Likewise, we arenot claiming that there presently existcomprehensive assessment devices for eachof these components. Indeed, refining andscaling up many of the small-scalelaboratory demonstrations in the literaturewill be a main task of our research. Ourpresent claim is only that, in every case,laboratory tasks that have appeared in thepublished literature give us, at a minimum,a hint at what comprehensive assessment

of the particular component would looklike.

The ability to measure individualdifferences in rational thinking could haveprofound social consequences. In arecently published book (Stanovich, 2011),we showed how shortcomings in each ofthe subcomponents of rational thought hasbeen linked to a real-life outcome ofpractical importance, including: physicians

choosing suboptimal medicaltreatments; people failing toaccurately assess risks in theirenvironment; the misuse ofinformation in legal proceedings;millions of dollars spent onunneeded projects by

government and private industry;parents failing to vaccinate their

children; unnecessary surgery; billions ofdollars wasted on quack medical remedies;and costly financial misjudgements (Baron,2008; Stanovich, 2009).

Likewise, a flurry of recent books fromresearchers such as Dan Ariely, RichardThaler and Cass Sunstein have outlinedpractical, real-life thinking domains wherepeople obtain suboptimal outcomesbecause they make rational thinkingerrors. For example, suboptimalinvestment decisions have been linked to overconfidence, the tendency to over-explain chance events, and to allowemotions to cloud judgement – allcomponents of our rational thinking test.It is critically important to realise thatintelligence has been shown to be aninsufficient inoculation against thesethinking errors and their negativeconsequences.

In summary, we have coherent andwell operationalised concepts of rationalaction and belief formation (for example,the Nobel-winning work of Kahneman and much related research). We also have a coherent and well operationalisedconcept of intelligence. No scientificpurpose is served by fusing these concepts,because they are very different. To thecontrary, scientific progress is made bydifferentiating concepts. We have adecades long history of measuring theintelligence concept. It is high time we put equal energy, as a discipline, into themeasurement of a mental quality that isjust as important – rationality.

read discuss contribute at www.thepsychologist.org.uk 83

rationality testing

Richard F. Westis at James MadisonUniversity, Harrisonburg,Virginia

Keith E. Stanovich is at the Department ofApplied Psychology andHuman Development,University of Torontokeith.stanovich@utoronto.ca

“scientific progressis made bydifferentiatingconcepts”