Pilot Administration of the Critical Thinking Assessment Test (CAT)

Phil Dunwoody, Kathy Westcott , Jerry Kruse

Carlee Ranalli, Jim Roney, Kim Roth, Peter Rothstein, Jim Tuten

Juniata CollegeHuntingdon, PA

Overview of CATThe focus of the CAT is to…

“assess and promote the improvement of critical thinking and real-world problem solving skills.”

Engage faculty in the assessment of critical thinking

Be reliable and valid

Be administered in 1 hour

Overview of CATAdministration of CAT

Short essay answer test with a variety of “real-world” problems

Questions focus on interpreting information and reaching valid conclusionso 1) There is an increase in campers getting sick. Students

evaluate causal claims based on the evidence provided.o 2) A family hiking scenario is described along with the pros

and cons of various water filters. Students are prompted to select the water filter that meets the family’s needs.

Prompts are ordered from vague to specific

Overview of CATReliability and Validity of CAT

Correlates with the SAT (r=0.52) and the California Critical Thinking Skills Test (r=0.65)

NSSE correlationso (2a) Memorizing facts, ideas, or methods from your courses and

readings so you can repeat them in pretty much the same form. (negative relationship) -.341*

o (3b) Number of books read on your own (not assigned) for personal enjoyment or academic enrichment. .277 **

o (11e) Thinking critically and analytically &(11m) Solving complex real-world problems.244 **

Faculty scoredRubric ensures reliable coding of responsesClosing the loop

General Education Assessment

Goals:1)Develop communication skills2)________higher order thinking skills

IC, CA, upper-level FISHN3)Develop knowledge about and ways of understanding the world

CAT AdministrationOne section each:

o CWS o Psychology Research o Quantitative Methods

Pre/post (using local codes), early and late in semester

$200/year and $6/test

We are the graderso Aim for one grader per 10 testso Transparent: No proprietary institutional scoring (CLA)

CAT Report (below) and Excel file w/raw data

CAT Results

CAT Results

CAT Question Profile

CAT Score Breakdown

Grader Comments and Discussion

Critical Thinking Assessment Test (CAT)

Face validity“looks like a reasonable measure…”

Inexpensive$6 / test vs. Collegiate Learning Assessment (CLA)

SensitivityFr – to – Sr, effects within a course, no ceiling or floor effects

Culturally fairno effects from gender, race, or ethnic origin

“Reasonable to Moderate” correlation to other measures of critical thinking (CAAP and CCTST) and academic performance (ACT/SAT)

Negative correlation with question 2a on the NSSE,Do the student's courses primarily focus on rote retention of information?

We almost all agree that Critical Thinking is the answer“With all the controversy over the college curriculum, it is impressive to find faculty members agreeing almost unanimously that teaching students to think critically is the principal aim of undergraduate education.”

Derek Bok, Our Underachieving Colleges: A Candid Look At How Much Students Learn and Why They Should Be Learning More, Princeton, New Jersey: Princeton University Press, 2006.

A definition to “critically analyze”

http://www.tntech.edu/cat/home/