Stages of Intellectual Developmentand Associated Critical Thinking Skillsin College Students

Robert A. Mines Mines and Associates, P.C., Denver, Colorado

Patricia M . King Department of`College Student Personnel, Bowling Green StateUniversity

Albert B. Hood Division of Counselor Education, University. of Iowa

Phillip K . Wood Department of Psychology, University of Missouri--Columbia

Students who reasoned at higher stages ofreflective judgment also revealed bettercritical thinking skills , suggesting adevelopmental basis for the acquisition ofcritical thinking skills.

Since the early 1970s researchers have proposedvarious descriptions of intellectual developmentin adulthood (e.g., Arlin , 1984; Basseches, 1984;Fischer , 1980 ; Kitchener & King , 1981; Kramer,1989; Labouvie-Vief, 1982 ; Perry , 1981 ; Riegel,1973; Sinnott , 1981). These models have pro-vided global descriptions of adult reasoning andmany insights into the qualitative changes thatcharacterize adult intellectual development. Thecurrent study attempted to identify specific rea-soning skills that are associated with givenstages of the Reflective Judgment model (Kitch-ener & King, 1981 ). Brabeck ( 1984) has notedthat this model has the strongest data base of

Robert A . Mines is president of Mines and Associates, P.C.,and can becontacted at Mines and Associates , P.C., Coun-seling Psychology Services, Denver , CO 80203 . Patricia M.King is an associate professor in the Department of CollegeStudent Personnel and can be contacted at Bowling GreenState University , Bowling Green , OH 43402 . Albert B. Hoodis a professor in the Division of Counselor Education andcan be contacted at the University of Iowa, Iowa City, IA52242 . Phillip K . Wood is an assistant professor in theDepartment of Psychology and can be contacted at the Uni-versity of Missouri , Columbia , MO 65201 . This study wasmade possible through the financial support of the Divisionof Student Services, University of Iowa, and was based onRobert A . Mines's doctoral dissertation . Requests for re-prints should be sent to Dr. Robert A . Mines, Mines andAssociates , P.C., 777 Grant , Suite 203, Denver , CO 80203.

existing models of adult intellectual develop-ment.

The Reflective Judgment Model. The Reflec-tive Judgment model (King, 1985 ; Kitchener,1986; Kitchener & King , 1981; Kitchener, King;Wood, & Davison, 1989) describes a sequenceof changing assumptions about knowledge andhow those assumptions affect the ways a personreasons to a conclusion about problems that donot have verifiable right and wrong . answers.These assumptions are summarized by stage inTable .1. They may be seen in, individuals' re-sponses to the following types of questions:What and how can we know ? How certain canwe be about what we know? How can we con-vincingly defend what we know or believe?Why do people hold different opinions aboutcontroversial issues? Answers to these questionsoffer useful information about students' reason-ing styles , because students ' assumptions aboutknowledge (e.g., what can be known and withwhat degree of certainty) are reflected in thestrategies they use to gain knowledge ; these, inturn, affect the adequacy with which studentscan solve complex and controversial problems.

Prior Research . A rather extensive researchbase exists for this model . Because most of theavailable studies have been reviewed elsewhere(King , Kitchener, Davison , Parker, & Wood,1983 ; King , Kitchener , & Wood, 1985; Kitche-ner, 1986; Kitchener & King , 1990b ; Kitcheneret al., 1989), . only one particularly noteworthystudy is summarized here . Brabeck .( 1983) ex-amined the relationship between reflective judg-

538 Journal of College Student Development / November 1990 / Vol. 31

TABLE 1Reflective Judgment Stages

Through What How MayWhat Can We How Certain Process Can Beliefs Be

Know? Can We Know? We Know? Justified?

1 Reality Absolutely cer-tain

2 True reality Absolutely cer-and false tain and cer-claims tain but not

immediately ,aavailable. .

3 True reality , Absolutely cer-false claims, tain aboutuncertainty some things;

temporarily un-certain aboutothers.

4 While there is No certainty be-a reality, it can cause . of situa-never be tional variablesknown. Knowl- (e.g., time).edge is individuallyidiosyncratic.

By direct obser- Beliefs are a c(i-nation rect reflection

of reality. Noneed. to justifythem.

By direct obser Directobserva-vation and via Lion or via au-what authori- thorities.ties say istrue.

Via' authorities insome areas;

`Via authorities insome areas: -

through our via what feelsown biases right in thewhen knowl- moment whereedge is uncer- knowledge. istain. uncertain.

Via our own and Via idiosyncraticothers ' biases , evaluations ofdata,, and evidence andlogic. unevaluated

beliefs.

5 Personal inter- No certainty ex- Via evidence By rules of in-pretations of cept via per - and rules of quiry for a par-individual real - sonal : per- inquiry appro- ticular context.ities spectives priate for the

within a spe- context.cific context

6 Reality as- Some personal Via personal as-sumed . Evalu certainty about sessment ofated personal beliefs based , arguments andinterpretations . on evaluations data; via eval -

of evidence on uated opinionsdifferent sides of experts.of the ques-tion.

7 Reality is Certainty thatnever "given ." some knowl-Facts and as - edge claimssumptions are better ormay be con - more completestructed into than others al-evaluated though theyknowledge are open toclaims about reevaluation.reality.

Via generalizedrules of in-quiry, personalevaluationsthat applyacross con-texts, evalu-ated views ofexperts.

Via process of As more or less,critical inquiry reasonableor synthesis . conjectures

about reality orthe worldbased on anintegration andevaluation ofdata, evidence,and/or opinion.

Differentiation/Integration

Single categorybelief system:!'What I be-lieve is:.;"

Two categorybelief system;knowledge istrue but someclaims arefalse.

.Three category,belief system;knowledge istrue, someclaims are,false, and others are uncer-tain.

Uncertain-knowl -edge becomes.further differ-entiated infotypes of uncer-tainty and be-comes ..>over-riding sategory, i.e ., ulti-mately .uncertain.

Greater differen-tiation withindomains. Evi -dence inte-grated withinspecific do-mains .

Evidence andopinion can beintegratedacross as wellas within dif-ferent do-mains . Greaterdifferentiation.

Viewpoint con-structed by ab -stracting orsynthesizingacross as wellas within dif-ferent do-mains.

Note . From Kitchener , K.S. (1986). The reflective judgment model:. Characteristics, evidence and measurement(pp. 78-79). In R.A. Mines & K .S. Kitchener (Eds.), Adult cognitive development: Methods and models (pp. 76-91). New York : Praeger. Reprinted by permission.

Journal of College Student Development / November 1990 / Vol. 31 539

ment stage and critical thinking. She matchedfemale students at four educational -levels (fromhigh school seniors to master ' s level graduates)on high and low extremes , of critical thinkingscores using the Watson -Glaser Critical Think-ing Appraisal (Forms A and B ) and thenadmin-istered the Reflective Judgment Interview (RJl).She found a moderate (r=.40, p< 001) correla-tion between - the two measures . Despite. the,matching , she also found significant .differencesin reflective judgment scores between educa-tional levels , with more educationally a,dvancedstudents scoring higher . High- level criticalthinkers also outscored low-level critical think-eraon the RJI. She concluded that "the development of reflective judgment is separate from,and involves something other , than the acquisi-tion of these skills" (p. 32)and; suggested thatcritical thinking skills maybe necessary but notsufficient for: development of reflective judg-ment (see also Brabeck & Wood , 1990).

The Current Study. In this study , a more fine-grained' approach was used to examine the rela-tionships between reflective judgment and stan-dardized critical thinking tests . We focused onthe specific skills constituting these measuresrather than use the more global overall score.This approach allowed us to determine whethercomponent critical thinking skills are present atsome reflective judgment stages and not others,and how important certain critical thinking skillsare to the complex problem -solving abilities thatare reflected _ in the more advanced stages ofreflective judgment.

METHOD

Sample

The sample for this investigation was' composedof 100 students at a large midwestern university:20 freshmen, 40 seniors , and 40 graduate stu-dents, (second year or beyond). (The groups werealso balanced by gender and area of study [eithermathematical or social sciences ]; results for thiscompanion' study are reported in King, Wood,& Mines, 1990.)

The purpose of selecting students in this man-ner was to obtain a group of individuals' whowould represent a broad range of critical think-ing skills and reflective judgment stages. Twiceas many students were selected at the senior andgraduate levels , as compared with the freshman

level, to increase the likelihood of including stu-dents who would score at the middle and upperstages of the .Reflective Judgment model. Stu-dents were contacted through courses,, posters,advertisements,' and departmental census listsuntil the cells were filled. Students were paid$ 10 each to participate in the study.

Instruments

Reflective Judgment Interview (RJI),The RJIconsists of four intellectual problems . and a se-ries of standardized probe questions and uses asemistructured format; it takes about 1 hour tocomplete . Each problem consists of two contra-dictory points of view on an intellectual issue;:respondents are asked to explain and defendtheir responses to these points of view. A certi-fied interviewer presents the dilemmas (usuallyin random order) and asks the probe questions,following up for clarity when needed. Tran-scripts of the interviews are then independentlyanalyzed by certified raters using the ReflectiveJudgment Scoring Rules (Kitchener & King,1985). Acceptable interrater reliability andagreement rates, as well as internal consistencylevels, have been found in previous studies (seeMines, 1982; Schmidt & Davison, 1981, for re-views).

Watson-Glaser Critical Thinking Appraisal(WGCTA). The WGCTA is a power test de-signed to assess abilities thought to be importantin critical thinking . The test contains 100 itemsand is usually completed in about 50 minutes.Watson and Glaser (1964)-list five subtests: (a)inference, the ability to discriminate among de-grees of truth and falsity of inference from givendata; (b) recognition of assumptions, the abilityto recognize unstated assumptions or presuppo-sitions that are taken for granted in given state-ments or assertions ; (c) deduction, the ability toreason deductively: from given statements orpremises, to recognize the relation of implica-tion between propositions , or to determine whatmay seem to be an implication or a necessaryinference from given premises is indeed such;(d) interpretation , the ability to weigh evidenceand distinguish between generalizations fromgiven data that are or are not warranted beyonda reasonable doubt; and (e) evaluation of argu-ments, the ability to distinguish between argu-ments that are strong and relevant and those thatare weak or irrelevant to a particular questionor issue . Split-half reliabilities of the subtests

540 Journal of College Student Development / November 1990 / Vol. 31

for Form Zm range from .40 to .55 (Watson &Glaser, 1964). These moderately low reliabili-ties likely reflect the brevity of some of thesubtests. Internal consistency reliabilities`are notreported in the manual,

Cornell Critical Thinking Test (CCTT). TheCCTT, Form Z, was used in this study to assessthe hypothetico-deductive critical thinking .pro-cess. The CCTT is -described as having a 50minute time limit, although its authors (Ennis &Millman , 1971) suggest that it may be used as°a power test , which was done in this study. FormZ consists of 52 items in seven sections , as fol-lows: (a) deduction, determining whether"astatement follows from premises in material thatis emotionally=loaded;(b) detecting fallaciouslyambiguous arguments (circularity , nonsupport-ing emotive language , oversimplification of al-ternatives); (c) judging the .reliability of infor-mation and authenticity of=sources ; (d) judgingwhether or not a hypothesisor-generalization is-warranted; (e) choosing useful hypothesis-test-ing predictions 'when planning experiments; (f)assumption-finding, identifying a definition thatbest expresses another person's usage of a term;and (g) assumption-finding, identifying a state-ment that fills a gap in a deductive 'argument.Ennis and Millman (1971) have reported KR-20reliability indexes ranging from.61, ;to .67, forForm Z; no subtest reliabilities are reported. TheCCTT also has short subscales , which may con-tribute to lower subtest reliabilities.

i

Procedure

The two written critical thinking tests (CCTTand WGCTA) were administered in a group set-ting . Each of these tests took 45 to 55 minutesto complete , and the test order was counterbal-anced by a student during the testing session.The RJI was administered individually by oneof two interviewers in a private office. Refer-

in the transcriptions to educational levelencesand gender were deleted prior to rating.

Permission was secured from all participantsto obtain their American College Test (ACT),Scholastic Aptitude Test (SAT), or Graduate Re-cord Examination (GRE) scores from institu-tional records as measures of academic aptitude;this was of interest given the possibility that anyobtained differences in critical thinking and in-tellectual development might be the result ofdifferential levels of academic aptitude that areassociated with different educational levels.

I

Where necessary, students' SAT or GRE scoreswere converted to ACT composite score equiv-alents.

RESULTS

Psychometric Information

For the RJI, interrater reliability (the degree ofconsistency between the two raters ) was .97,calculated using a Pearson 's r correlation . Inter-rater agreement (the proportion of time the tworaters assigned the same scores) was .90. Fourinternal consistency reliabilities (coefficient al-phas) are reported next for each instrument, bothoverall and by educational level (these are listedin `order: freshmen , seniors, and then graduatestudents). For the RJI, the alphas •were . 89 (over-all), .69, .77, and .84. The corresponding alphasfor the WGCTA were .82 (overall), .49, ,74, and.72. For the CCTT, alpha levels of .70,(overall),.00, .49, and .62,were obtained.

Educational Level Differences

The means and standard deviations of the fourmeasures are given by educational level in Table2. (The RJI mean scores correspond to stages:a mean score of 4.0 indicates that the students'assumptions about knowledge and justificationof beliefs reflect Stage 4 assumptions as de-scribed in Table .1.) Without exception, themeans of each of the measures were ordered inincreasing magnitude across educational, levels.

The RJI scores for the seniors are comparableto those that have been reported for other sam-ples of college -seniors The RJI scores for thefreshmen and graduate students are somewhatlower than those that have been reported forother samples (Kitchener & King , 1990b). Forthe WGCTA, the freshmen in this study scoredalmost a standard deviation lower than did anorm group of liberal arts freshmen (M=70.2)reported by, :Watson and Glaser (1964). Theseniors ' scores were very close to the mean of74.4 that was reported for a sample of 200 seniorwomen . Watson and Glaser report no graduatenorms . Ennis and Millman (1971) have reportedCCTT norms from two postsecondary samples,one group of college students and one group ofgraduate students . The mean scores betweenthese two groups did not differ and were com-parable to the scores obtained here from thefreshman sample.

Journal of College Student Development / November 1990 / Vol. 31 541

TABLE 2RJI, WGCTA , CCTT, and ACT Means and Standard Deviations by Educational. Level

RJI WGCTA CCTT ACT'

Educational Level M SD M SD M SD M SD

Freshman 3.31 .37 61.55 6.71 27.35 3.33 19.40 3.70Senior 4.08 .56 78.87 8.27 35.57 5.20 '26.37. 4.50Graduate 4.76 78 " 82.40 7.79 39.02 5.28 28.83 2.70

Note . RJI = Reflective Judgment Interview ; WGCTA = Watson -Glaser Critical Thinking Appraisal ; CCTT = Cor-nell Critical Thinking Test; ACT = American College Test,'ACT composite score.

A series of ANCQVAs; one for each measure,was run to see whether or not scores differedby educational level and to examine concur-rently the role of academic aptitude on thesescores: Using the ACT composite score as thecovariate, significant differences in scores werefound for all three measures between the groupsafter linearly adjusting for the. effects- o_ f aca-demic aptitude: F(2, 96)=52.11, p<.001 for theRJI; F(2, 96)=8.86, p<.01 for the WGCTA; F(2,

96)=6.76 p=.02 for theCCTT. In other words,the educational level differences reported inTable 2 cannot be attributed to academic apti-tude.

Table 3 reports the means and standard devi-ations of the WGCTA and CCTT subtest scores,listed by. reflective judgment stage. For both theWGCTA and CCTT subtests, the, highly consis-tent. pattern of increasing subtest scores acrossreflective judgment stages was reported. On the

TABLE 3Means and Standard Deviations of the WGCTA and CCTT Subtests by Reflective

Judgment Stage

Reflective Judgment Stage3 (n=19) 4 (n=40) 5 (n=30) 6 (n=9) 7 (n=1)

Subtests M SD M SD M SD M SD M SD'

WGCTAInference 9.05 2.34 12.71 2.88 13.13 1.94 14.33 1.58 15.00Recognition of as

sumptions 11.26 2.70 13.10 1.96 13.37 1.56 14.00 1.12 15.00Deduction : ? 16.84 2.12 21.12 3.23 22.27 2.27 22.67 1.94 23.00Interpretation 15.00 3.04 19.71 3.40 22.03 1.65 21.33 2.34 23.00Evaluation of ar-

guments 10.37 1.74 10.95 2.01 11.47 2.05 11.56 2.18 14.00CCTT

Does statementfollow frompremise? .47 .43 .80 .03 .51 .16 .56 .59 0.00

Detecting ambigu-ous arguments 4.37 1.50 6.39 2.40 7.77 1.77 8.56 1.59 8.00

Judging reliabilityof information 1.84 1.12 2.54 1.19 2.90 0.71 3.00 0.87 4.00

Is hypothesis orgeneralizationwarranted? 8.32 1.11 9.78 1.51 9.87 1.61 9.78 1.92 12.00

Making predictions 1.84 1.26 2.19 1.12 2.47 1.04 2.67 0.87 4.00Determining defini-

tions 1.84 0.69 2.73 1.07 3.10 0.66 2.67 0.87 4.00Identifying as-

sumptions 2.89 1.29 3.73 1.55 4.23 1.17 3.89 1.69 4.00

Note. See Table 2 Note.' Insufficient data to compute standard deviations.

542 Journal of College Student Development / November 1990 / Vol. 31

WGCTA, an examination of subtest scores be-tween. adjacent reflective judgment stages re-veals that 19.-out- of 20 comparisons formed aperfect. Guttman- scale.: For the CCTT, 28 of 32comparisons followed this pattern. Of the fiveinconsistencies, four involved slight mean scorereversals between Stages 5 and 6. This patternisconsistent with the theoretical assumptions ofthe. Reflective Judgment model.. That is, if thecritical thinking. skills measured 3 by these twotests are associated with progressively higherstages of reflective judgment; an improvementin the complexity.- of critical thinking skillsshould be exhibited.. If the critical thinking sub-test scores had exhibited nonlinear patterns, fur=ther analysis:would have been impossible to in-terpret, and the, use of discriminant analysis(discussed below) would have been inappropri-ate.

Relationships Between Measures

Pearson product moment correlations were runbetween all pairs of the three measures. TheWGCTA and CCTT correlated highly with eachother (r=.71, -p<A 1) and with the ACT score(r=.59 and .62,p<.01). The correlation betweeneach of these measures and the RJI was moder-ate (r=.46, p<.01), When the effects of academicability were removed, the resulting partial -cor-relation between the WGCTA and CCTT was.54 (p<.01); the correlation between these mea-sures and the RJI decreased to .27 (p<.05).

The structural relationship between criticalthinking subtest scores and reflective judgmentstage was investigated by means of a discrimi-nant analysis. This procedure is designed to dis-tinguish between two or more groups (in thiscase, between students who score at differentreflective judgment stages) by assigning weights

to variables (here, the critical : thinking skills)and then linearly . combining the discriminatingvariables to make the groups as distinct as pos-sible.

First , a global discriminant analysis (based oniesidualized scores ) was conducted across alleducational ,levels; with statistically significantresults , x2(12, 100)=76.9, p<.001 . This indicatesthat critical - thinking'scores can be used to pre-dict reflective ' judgment stage at a rate greaterthanchance : Subsequent analyses were also runby educational level; none of these analysesachieved statistical significance.

The discriminant analysis yielded four vari-ables that significantly distinguished betweenreflective judgment stages. These variables,along ' with their respective standardized discrim-inant function coefficients , were as follows:WGCTA, interpretation (.52); CCTT, detect-ing ambiguous arguments (-30); WGCTA, de-duction (-.28);and WGCTA, inference (-.21).Of these ` theinterpretation subtest was by farthe most potent contributor to the function.',

The utility of the discriminant function wastested by classifying each of the students into agiven reflective judgment stage on the basis oftheir discriminant function scores. The percent-age of correct classifications was compared withthe probability of being assigned to a given re-flective judgment stage based on the distributionof scores on which the student's overall reflec-tive judgment scorewas based. The results ofthe classification procedure are reported in Table4. The discriminant function correctly classified50% of the students (e.g., Stage 3 , with Stage3). Percentages of correct classification by stagewere 74 (Stage 3), 46 (Stage 4), 57 (Stage 5)and 0 (Stage 6). Scores assigned to adjacentstages (e.g., Stage 3 with Stage 4) account for

TABLE 4Classification Effectiveness of the Discriminant Function for Reflective Judgment

Stages 3-6

Actual Group Predicted Group Membership (by Stage)Membership

by Stage n 3 (%) 4(%) 5(%) 6(%)

3 19 14(74) 5(26) 0(0) 04 41 9(22) 19 (46) 12 (29) 1 (2)5 30 0 13 (43) 17(57) 06 10 0 4(40) 6 (60) 0Total 100 23 41 35

1

Percent of cases correctly classified: 50%

Journal of College. Student Development / November 1990 / Vol. 31 543

100% of the Stage 3 assignments, 98% for Stage4, 100% for Stage 5,. and 60% for . Stage .6.Although these are not all correct classifications;they show the clustering of classificationsaround given stages . Based on the square of thecanonical correlation, this model accounted fdr50% of the variance, of reflective judgment stagescores.

A comparison of the classificationof eachstage with that which would" have occurred bychance yielded; another indication of;the effec-tiveness of the discriminant function. As shownin Table 4, the discriminant function correctlyclassified 74% of the students who scored atStage 3 , as .compared with 19% if they were'assigned by chance.. None of the other stageswere classified as. effectively. Students whoscored at Stage 4 were correctly classified 46%of the time,"above the chance level of 41%.Stage S was the only other stage in which stu-dents were classified at an accuracy level thatwas well above its corresponding chance prob-ability (57% versus 30%). The discriminationfunction classified students who scored at Stages3 and' Sat only two stages (the actual one andone adjacent stage). Four students who scoredat Stage 6 and one who scored at Stage 4 wereclassified at stages that were up to two stagesdiscrepant from the correct stage; in these cases,the model was not accurate in classifying stagemembership.

Four follow-up discriminant. Analyses werethen run to determine whether or not a differentpattern of critical thinking skills might distin-guish Stage 3 reasoning from that characterizingthe more advanced stages. The first comparisons

were between. Stage 3. and Stages 5,:and 6._Tl eresults of these. analyses are reported in Table5. Three of the. four variables that' were. signifscant on the global analysis were also significanthere; the exception was theWGCTA ihferencevariable, which was 'not significant for, eithercomparison. With only two exceptions, the samesubtests were significant for both sets of comeparisons. Furthermore, the" coefficients =were" either very similar or stronger when Stage3, scoreswere compared with Stage 6 scores (with" oneexception: the WGCTA°i.nference variable).Variations in= the patterns are" also apparent. Forexample, the interpretation subtest;"which hadacoefficient of -.52 in the "overall analysis wascomparably high (a^-.55) for the Stage-31 versus5 comparison, but it was tied<"for the. lowestweighted variable (-30) "for the Stage=3, versus6 comparison. A classification effectiveness of100% was achieved for these two discriminantanalyses; the' probabilities "of :arriving at theseclassifications by chance were 19%, 30%, and10% for Stages 3, ' 5, and 6 (see Table 6). Dif-ferent critical thinking skills seem to be associ-ated with the assumptions of the three reflectivejudgment stages compared here:

DISCUSSION

Educational 'Level DifferencesWithout exception, the overall scores for eachmeasure increased across the three educationallevels. In each case, the more educationally ad-vanced students scored higher; than did" theircounterparts at.earlier educational levels. De-spite their higher RJI scores, .however, the col-

TABLE 5Discriminant Function for Reflective Judgment Stages 3 and 5, and 3 and 6

Standardized Discriminant FunctionCoefficient

Variable Stages 3 and 5

Detecting ambiguous arguments (CCTT) -.22Does statement follow from premise? (CCTT) -.24Recognition of assumptions (WGCTA) -.25Determining definitions (CCTT) -.32Judging reliability of information (CCTT) -.34Identifying assumptions (CCTT) -.35Deduction (WGCTA) -.36Is hypothesis or generalization warranted? (CCTT)Interpretation (WGCTA) -.55

Note. See Table 2 Note.

. Stages 3 and 6

-.49

-.51-.30-.74-.33-.68- .44

;-.30

544 Journal of College Student Development I November 1990 / Vol. 31

I

TABLE 6Classification Effectiveness of the Discriminant Function

for Reflective Judgment Stages

Stages 3 and 5 PriorPredicted Group (Stage)

MembershipActual Group n Probability 3(0/6) 5 (%)

3 19 .39 19(100) 05

Ungrouped Cases3051

.61 013 (26)

30(100)38 (74)

Percentage of Cases Correctly Classified: 100%

Stages 3 and 6 PriorPredicted Group (Stage).

MembershipActual Group n Probability 3 (%) .6(%)

3 19 .66 19 (100) 06 10 .34 0 10 (100)

Ungrouped Cases 71 18 (25) 53 (75)Percentage of Cases Correctly Classified: 100%

lege and graduate students did not reflect theskills of critical thinking that are commonly as-sociated with the intended outcomes of highereducation discussed in several recent nationalreports (Association of American Colleges,1985; Garrison, 1984; National Institute of Ed-ucation, 1984). (See King et al., 1990, for a moredetailed discussion of this point.) The acquisi-tion of higher order cognitive skills has beenshown (Fischer & Kenny, 1986; Fischer & Pipp,1984) to be related to environmental opportuni-ties to learn and practice one's reasoning skills.In such environments, skills are modeled andtaught, and students are given opportunities-topractice and receive feedback about their suc-cess in applying new skills. The college studentsin this sample may not have had sufficient op-portunities to refine their thinking skills (a dif-ficult undertaking in a large university withmany large classes), or they may not have hadthe self-confidence to take advantage of suchopportunities.

Academic Ability

It is noteworthy that ACT scores also increasedby educational level. Nevertheless , academicability did not statistically account for educa-tional level differences in the three measures ofreasoning .;;In other words, it seems that the de-velopment of these skills is more strongly re-lated to students ' educational experiences thanto their academic aptitude at the time they en-tered college . Educators attempting to teach rea-soning skills to college and graduate -students

may find this 'reassuring. Cross-sectional' re-search designs such as this one offer a prelimi-nary basis for this conclusion; evidence fromlongitudinal studies, however, would provide astronger data base from which to examine ques-tions regarding the development of these sets ofskills over time. Furthermore, it should be notedthat in the analyses regarding the effect of aca-demic ability (the ANCOVAs), the ACT scoresof all participants were covaried out, and therelationships remained significant. It may be thatthe effect of academic ability is different foreach educational level (or for each educationallevel by gender combination). Examining thispossibility, however, would require a largersample than was feasible in this study.

Relationships Between Measures

A major finding of this study was that studentswho reason using the assumptions of the higherstages of reflective judgment demonstrate bettercritical thinking skills than do those who uselower stage assumptions. The near-perfect or-derings of subtest scores across reflective judg=ment stages (Table 3)offer preliminary supportfor the argument that there, is a developmentalbasis for the acquisition of critical thinkingskills.

Other results clarify some of the cognitiveskills necessary to reason at the various stagesof reflective judgment. The critical thinkingskills that distinguished between reflective judg-ment stages were the following: (a) interpreta-tion,weighing evidence and identifying gener-

Journal of College Student Development / November 1990 / Vol. 31 545

alizations that are warranted beyond a reason-able doubt; (b) detecting fallaciously ambiguousarguments (e.g., evaluating arguments to deter-mine if they violate laws of a valid argument);(c) deduction, reasoning deductively from prem-ises to conclusions; and (d) inference, analyzingthe degree of accuracy of inferences drawn fromgiven statements. Specifically, the studentsholding Stage 3 assumptions showed less mas-tery of these skills than did their counterpartswho held more advanced assumptions. Master-ing these skills may be a necessary prerequisitefor: continued intellectual development throughthe reflective judgment stages; evidence of thesecentral critical skills is certainly consistent withthe reasoning that is characteristic of the moreadvanced stages of reflective judgment.

The four skills identified in the discriminantanalyses, are theoretically consistent with thecharacteristics of higher stage reflective judg-ment reasoning. Two of these skills. (inferenceand interpretation) seem particularly -well-matched to the assumptions,of Stages 5, 6, and7. For example, the interpretation subscale in-volves weighing evidence and identifying gen-eralizations that are warranted" beyond a reason-able doubt. Using a reasoning style that doesthis explicitly is a major hallmark of the upperstages. A major characteristic of Stage 4xeason-ing, by contrast, is that evidence is used incon-sistently to support a point of view, and evidenceis not assumed to entail a conclusion.

Implications

Student affairs practitioners as well as facultymembers have many, opportunities to help stu-dents examine their assumptions about what andhow they claim to know. Furthermore, they havemany opportunities to create environments ex-pressly designed to teach critical thinking skills,environments that include many opportunitiesto practice and receive feedback about theseskills. If in fact there is a developmental basisfor the acquisition of critical thinking skills, asthis study suggests , then those who create andwork in such learning environments would bewell-advised to attend to the developmentalcharacteristics of the students they attempt toserve and teach. For example, students who holddifferent reflective judgment assumptions trans-late these into different expectations for thelearning environment, and as a consequence,perceive different challenges and supports in

their educational tasks (e.g., see Kitchener &King, 1990a). Trying to teach critical thinkingwithout taking these factors into account wouldnot only contradict our knowledge of develop-mental processes but would probably also resultin less effective practice (Strange & King, 1990)and less success in achieving the central educa-tional goal of teaching students to reason criti-cally.

REFERENCES

Arlin, P. K. (1984). Adolescent and adult thought: A struc-tural interpretation. In M. L. Commons, F. A. Richards,& C. Amnon (Eds.), Beyond formal operations: Late ad-olescent and adult cognitive development (pp. 258-27 1).New York: Praeger.

Association of American Colleges. (1985). Integrity in thecollege curriculum : A report to the academic community:The findings and recommendations of the project on re-defining the meaning and purpose of baccalaureate de-grees. Washington, DC: Association of American Col-leges.

Basseches , M. (1984). Dialectical thinking and adult devel-opment. Norwood, CA: Ablex.

Brabeck, M. (1983). Critical thinking skills and reflectivejudgment development:. Redefining the aims of highereducation. Journal of Applied Developmental Psychol-ogy, 4, 23-34.

Brabeck, M. (1984). Longitudinal studies of intellectual de-velopment during adulthood: Theoretical and researchmodels. Journal of Research and Development in Edu-cation , 17, 12-27.

Brabeck, M„ & Wood, P. (1990). Cross-sectional and lon-gitudinal evidence for differences between well -struc-tured and ill-structured problem solving abilities. In M.L. Commons, C. Armon, L. Kohlberg, F. A. Richards, T.A. Grotzer, & J. D. Sinnott (Eds.), Adult development:Models and methods in the study of adolescent and adultthought (Vol. 2, pp. 133-146). New York: Praeger.-

Ennis, R. H., & Millman , J. (197 1).,Cornell critical thinkingrest manual . Urbana, IL: The University of Illinois, Crit-ical Thinking Project.

Fischer, K. W. (1980). A theory of cognitive development:The control and construction of hierarchies, of skills, Psy-chological Review, 87, 477-531.

Fischer, K. W., & Kenny, S. L. (1986). Environmental con-ditions for discontinuities in the development of abstrac-tions. In R. A. Mines & K. S. Kitchener (Eds.), Adultcognitive development: Methods and models (pp. 57-75).New York: Praeger.

Fischer, K. W., & Pipp, S. L. (1984). Processes of cognitivedevelopment: Optimal level and skill acquisition. In RJ.Sternberg (Ed.), Mechanisms of cognitive development(pp. 45-80). New York: (W. H.) Freeman.

Gamson, Z. (1984). Liberating education. San Fran cisco:Jossey-Bass.

King, P. M. (1985). Choice-making in young Adulthood: Adevelopmental double-bind. Counseling and Human De-velopment, 18(Whole no. 3).

King , P. M., Kitchener, K. S., Davison, M. L., Parker, C.A., & Wood, P. K. (1983). The justification of beliefs in

546 Journal of College Student Development / November 1990'/ Vol. 31

young adults: A longitudinal study. Human Development,25, 106-116.

King, P. M., Kitchener , K.S., & Wood, P . K. (1985). The

development of intellect and character : A longitudinalsequential study of intellectual and moral development inyoung adults . Moral Education Forum, 10. 1-13

King,. P . M., Wood, P. K., & Mines, R. A. (1990). Criticalthinking among college and graduate students . The Re-view of Higher Education , 13, 167-186.

Kitchener, K. S. (1986 ). The reflective judgment model:Characteristics, evidence and measurement . In R. - A.Mines & K. S. Kitchener, (Eds.), Adult cognitive devel-opment: Methods and models (pp. 76-91). New York:Praeger.

Kitchener , K. S., & King , P. M. (1981 ). Reflective judgment:Concepts of justification and their relationship to age andeducation. Journal ofAppliedDevelopmental Psychology,2, 89-116.

Kitchener , K. S., & King, P. M. (1985 ). Reflective judgment

scoring rules . Unpublished manuscript, Bowling Green

State University and University of Denver.Kitchener, K. S., & King , P. M. (1990a). The reflective

judgment model :, Transforming assumptions about know-ing. In J . Mezirow (Ed.), Fostering critical reflection in

adulthood: A guide to transformative and emanicaptorylearning (pp. 159-176). San Francisco : Jossey-Bass.

Kitchener , K. S., & King, P. M. (1990b). The reflectivejudgment model : Ten years of research . In M. L. Com-mons, C. Armon, L. Kohlberg, F. A. Richards, R. A.Grotzer, & J. Sinnott (Eds.), Adult development : Modelsand methods in the study of adolescent and adult thought(Vol. 2, pp . 63-78). New York: Praeger.

Kitchener, K. S., King , P. M., Wood , P. K., & Davison, M.L. (1989). Sequentiality and consistency in the develop-ment of reflective judgment : A six-year longitudinalstudy . Journal of Applied Developmental Psychology, 10,73-95.

.Kramer, D.,(1989). Development of an awareness of con-tradiction across the life span and the question of postfor-

mal operations . In M. L. Commons, J. D. Sinnott, F. A.Richards , & C.Armon (E&), Adult development, Vol. J:

Comparisons and applications of developmental models

(pp. 133-159).New York: Praeger.Labouvie-Vief, G. (1982). Dynamic development and ma-

ture autonomy : A theoretical prologue. Human Develop-ment, 25, 161-191.

Mines, R. (1982). Student development assessment tech-niques. In G. R. Hanson (Ed.), New directions for studentservices.- .Measuring student development (No. 20, 6592).San Francisco : Jossey-Bass..

National Institute of Education . ( 1984). Involvement inlearning: Realizing the potential of American higher ed-ucation (Stock No. 065-00000213-2). Washington, DC:U.S. Government Printing Office.

Perry , W. G., Jr. ( 1981). Cognitive and ethical growth : Themaking of meaning. In A. Chickering & Associates .(Eds.),The modern American college (pp. 76- 116). San Fran-cisco : Jossey-Bass. '

Riegel , K. F: (1973). Dialectic operations : The final periodof cognitive development . Human Development , 16, 346-370.

Schmidt , J. A., '& Davison , M. L. (1981 ). Does collegematter? Reflective judgment : How students tackle thetough questions . Moral Education Forum , 03), 2-14.

Sinnott , J. (1981 ). The theory of relativity : A metatheoryfor development? Human Development , 24, 293-311.

Strange, C. C., & King, P. M. (1990 ). The professional'(Ed.),practice of student development . In D. Creamer

College student development : Theory and practice in the1990s (Vol. II, pp . 9-24). Alexandria , VA: ACPA MediaPublications #49.

Watson , G., & Glaser , E. M. (1964.). Watson -Glaser criticalthinking appraisal manual . New York : Harcourt ,: Brace,and World.

Journal of College Student DevelopmentI November 1990 1 Vol. 31 547