FACULTY USE OF PRIMARY TRAIT ANALYSIS TO ASSESS CREATIVE THINKING

Sarah Murnen, Kenyon College

Barbara Andereck, Ohio Wesleyan University

(Nancy Grace, College of Wooster)

FUNDED BY THE TEAGLE FOUNDATION

Grant to the “Ohio-5” Colleges administered by the College of Wooster 2006-2009

“Creativity and Critical Thinking: Assessing the Foundations of a Liberal Arts Education”

Organized by Lori Bettison-Varga, former Associate Provost at the College of Wooster, and then Nancy Grace, Professor of English at the College of Wooster

http://www.wooster.edu/teagle

How Do You Define Creativity?

STARTING OUR PROJECT

Year 1: Assembled groups of faculty across four colleges

in the Ohio-5 consortium of colleges to talk about evaluating creative and critical thinking among students

Developed survey for students and faculty, asked about traits defining critical and creative thinking

Idea Generation

Curiosity

Imagination

Reasoning by metaphor & analogy

Invention

Synthesis, integration & combination

Abstraction & simplification

Elaboration

Complexity

Evaluation & assessment

Reasoning through logic

Interpretation

Categorization & classification

Creative Thinking Critical Thinking

Awareness of Environment Divergent

thinking

Openness to Novelty

Description

Identification

Playfulness

Analysis

How Can We Best Assess Creative Thinking in

Students?

PRIMARY TRAIT ANALYSIS

• Identify the “primary traits” – essential or central components of the discipline (both knowledge and skills) to be learned by the student

• Build a scale for scoring the student’s performance on the trait

• Evaluate the student’s performance against those criteria

PRIMARY TRAIT ANALYSIS

Resources:

Presentation by Dr. Douglas Eder on “Primary Trait Analysis”

Presentation by Barbara Walvoord on the use of rubrics for effective grading

Walvoord, B. E., P& Anderson, J. A. (1998). Effective Grading: A tool for learning and assessment. Jossey

Bass: San Francisco, CA.

STAGES IN CONSTRUCTING A RUBRIC

From: Stevens & Levi (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishing: Sterling, Virginia.

Reflecting –

• What do we want for our students and from our students? (Learning objectives)

• Why did we create the assignment?• What happened the last time we used the

assignment? • What are our expectations for the

assignment?

STAGES IN CONSTRUCTING A RUBRIC

Listing – Focus on particular details of the assignment and what specific learning objectives we hope to see completed (Sometimes helps to imagine the best and the worst performance on the assignment)

Grouping and Labeling the goals together - Organize the results of our reflections in first two stages, grouping similar expectations together in what will probably become the dimensions of the rubric

Application – Transfer groupings to a rubric grid; define assessment categories

GROUP DISCUSSION RUBRICHTTP://WWW.MASHELL.COM/~PARR5/TECHNO/GROUP.HTML

Measures Emerging (1), Competent (2), and Exemplary (3) Performance on Each Trait

LISTENING

1. Recognizes and responds to others speaking.

2. Uses and practices listening processes regularly.

3. Habitually uses listening processes.

NON-VERBAL COMMUNICATION Eye contact, gestures, posture, facial expression, voice.

1. Comprehends some information from non-verbal cues.

2. Draws accurate conclusions from body language and facial expressions.

3. Able to recognize and use subtle non-verbal communication cues.

CO-OPERATION

1. Sometimes shows ability to wait to give appropriate verbal / non-verbal responses.

2. Usually shows ability to wait to give appropriate verbal / non-verbal responses.

3. Habitually shows ability to wait with openness and awareness to give appropriate verbal / non-verbal responses.

PARTICIPATION Tells thoughts, feelings, ideas so others understand.

1. Rarely talks during the discussion or talk is off the subject. Offers few ideas to the discussion.

2. Shares freely and explains with details. Makes connections to what others say.

3. Talk inspires others. Supports and leads others in discussion.

DEVELOPING RUBRICS THROUGH PRIMARY TRAIT ANALYSIS

Year 1: Faculty developed rubrics to measure

creative and critical thinking for their courses or departments

Year 2:Taught other faculty to develop and use

rubrics, and examined the rubrics they created along with student performance data from the rubrics

Developed a “generic” rubric using PTA designed to measure creative and critical thinking

“GENERIC RUBRIC” CRITICAL AND CREATIVE THINKING TRAITS

1. Elements of Argumentation a. Explanation b. Analysis c. Evaluation d. Interpretation e. Logic

2. Domain and Disciplinary Knowledge

3. Synthesis and Connections 4. Abstract Thinking 5. Complexity of Thought 6. Ideas Generated

a. Fluencyb. Flexibility

“GENERIC RUBRIC” CRITICAL AND CREATIVE THINKING TRAITS

7. Completeness/Coherence 8. Elegance 9. Divergent Thinking10. Novelty or Uniqueness

a. Germinalb. Originalc. Transformational

11. Engagement12. Risk Taking

“GENERIC RUBRIC” CRITICAL AND CREATIVE THINKING RATING SCALE DESCRIPTIONS

6 = Trait is evident to an exceptionally high degree5 = Trait is evident to a high degree4 = Trait is evident to an intermediate degree3 = Trait is somewhat evident2 = Trait is barely evident1 = Trait is not evident

What Assignments/Pedagogies Promote

Creative Thinking?

FACULTY USE GENERIC RUBRIC

Year 3: Faculty across many disciplines measured

student performance using the generic rubric in 31 classes (N = 444 students) across three of the participating colleges (College of Wooster, COW; Kenyon College, KC; and Ohio Wesleyan University, OWU)

YEAR 3: KEY FINDINGS

Creative and critical thinking can be assessed in students, and we found improvement of performance on many creative and critical thinking traits among the students we studied (N = 444 students across 31 courses).

The traits measured most frequently across all disciplines were complete and original measured in 90% of students, followed by knowledge measured in 85% of students. The traits least likely to be measured included abstract, logic, and novel. There were some differences across disciplines in the traits that were emphasized; for example, faculty teaching in interdisciplinary courses were more likely to examine complexity, and they shared with fine arts faculty a greater concern for engagement and risk.  

YEAR 3: KEY FINDINGS In the longitudinal data measuring change in

student performance across a semester, the traits where the most change was evident included complete, elegant, knowledge, and engaged; and the most change occurred among students in interdisciplinary courses.

In the cross-sectional data comparing less expert students to more expert, the most reliable differences were on the traits of analyze, logic, knowledge, and explain. There was no significant difference between the groups’ performance on abstract, fluent, flexible, elegant, divergent, novel, and risk.

Participating faculty who filled out an on-line survey indicated that they had discussed the use of the rubric with colleagues, and that the rubric helped them learn more about creative and critical thinking among their students

TRAITS MEASURED IN CLASSES BY DISCIPLINE – ALL DATA

 

ALL FA HUM NSCI SOSC INTExplain 0.63 0.64 0.89 0.87 1.00 0.12Analyze 0.54 0.44 0.89 0.87 0.72 0.12Evaluate 0.44 0.00 0.54 0.73 1.00 0.12Integrate 0.43 0.75 0.74 0.56 0.28 0.12Logic 0.40 0.44 0.12 0.71 0.72 0.12Knowledge 0.85 1.00 0.60 0.84 0.72 0.99Synthesis 0.53 0.79 0.48 0.58 1.00 0.12Abstract 0.38 0.00 0.12 0.71 1.00 0.12Complex 0.62 0.00 0.63 0.58 0.55 1.00Fluent 0.63 0.45 0.74 0.58 0.72 0.65Flexible 0.45 0.21 0.89 0.28 0.45 0.47Complete 0.90 0.89 0.63 0.86 1.00 1.00Elegant 0.58 0.89 0.12 0.28 0.56 0.81Divergent 0.52 0.21 0.48 0.14 0.72 0.81Novel 0.40 0.25 0.74 0.69 0.45 0.12Original 0.90 0.79 0.89 0.73 1.00 1.00Transform 0.74 0.68 0.88 0.56 0.72 0.81Engaged 0.72 0.89 0.12 0.56 0.72 1.00Risk 0.59 0.75 0.48 0.14 0.28 1.00_____________________________________________________________________N Students 444 73 65 85 82 139N Classes 31 5 6 6 8 6_____________________________________________________________________ Note: High responses are noted in bold, low in italics.Number indicates proportion of students by discipline tested on the particular trait.

AVERAGE PERFORMANCE ON TRAITS ACROSS TIME – LONGITUDINAL DATA

 Trait M time 1 M time 2 tvalueExplain 3.85 4.52 7.26***Analyze 3.76 4.73 7.84***Evaluate 3.28 4.27 8.86***Integrate 3.67 4.51 5.86***Logic 3.76 4.78 6.85***Knowledge3.40 4.35 10.63***Synthesis 3.77 4.63 8.50***Abstract 3.53 4.58 9.76***Complex 3.27 4.14 9.60***Fluent 3.56 4.39 9.16***Flexible 2.85 3.82 8.56***Complete 3.56 4.41 10.98***Elegant 3.03 4.05 10.83***Divergent 2.53 3.61 9.00***Novel 3.19 4.22 5.78***Original 3.19 4.01 9.40***Transform 2.23 3.31 9.22***Engaged 3.56 4.48 10.58***Risk 2.80 3.55 7.60***

_____________________________________________________________

*** indicates t value is significant at p<.001 revealing change in mean across time

Note: High responses are noted in bold, low in italics.

CROSS-SECTIONAL DATAScores on the traits could range from 1 to 6, with a higher score indicating better performance.

Differences between groups were analyzed with independent sample t-tests.

Trait M Less Exp M More Exp Difference?Explain 3.99 4.71 4.35***Analyze 3.64 4.65 5.86***Evaluate 4.18 4.70 2.51*Integrate 3.47 4.19 3.72***Logic 3.87 4.80 5.28***Knowledge 3.37 4.01 4.39***Synthesis 3.67 4.61 4.32***Abstract 4.26 4.56 nsComplex 3.15 3.71 3.04**Fluent 4.45 3.93 nsFlexible 3.92 3.59 nsComplete 3.60 4.16 3.52**Elegant 3.04 3.23 nsDivergent 2.87 3.21 nsNovel 3.38 3.67 nsOriginal 2.91 3.58 3.30***Transformation 2.30 3.02 3.34***Engaged 3.81 4.29 2.59**Risk 2.34 2.53 ns

Note: ns indicates the t value is not significant; indicates significant at p<.05, **p<.01, ***p<.001

YEAR 3 INTERVIEWS PARTICIPATING FACULTY Twenty-two faculty who employed the

“generic” rubric to measure creative and critical thinking among the students were interviewed about their experience

Interviewers asked questions about the specific assignment used and whether it

promoted creative thinking, other assignments and pedagogies that might

promote (or inhibit) creative thinking, aspects of the discipline that might promote

(or inhibit) creative thinking, aspects of the college environment that might

be important

YEAR 3 INTERVIEWS PARTICIPATING FACULTY

Most of the faculty who participated chose an assignment they had used previously, and did not develop a new assignment for the grant.

Many of these assignments seemed designed to stimulate critical thinking more than creative thinking, but using a rubric that emphasized creative thinking increased the focus on these traits and likely increased the amount of creativity students showed.

In general, most people thought their assignments did promote creative thinking in students (and this was validated by the quantitative data gathered from the rubrics). However, the assignments people used for their data collection were not necessarily the most creative assignments the faculty had used or could imagine using.

FACULTY INTERVIEWS YEAR 3: VALUE OF “ACTIVE LEARNING TECHNIQUES”

Take devil’s advocate position

Use problem solving

Students debate ideas

Use small group discussion

“Hands-on” science activities

FACULTY INTERVIEWS YEAR 3: VALUE OF “ACTIVE LEARNING TECHNIQUES”

One natural scientist spoke about student experience with creating their own experiments versus using “canned” labs. She said, “I didn’t know when I started doing this in my upper-level comparative class whether they would like it.... The experiments don’t usually work, because experiments usually don’t work, period. And, so, I would always say next year I am going to write labs. By the end of the class students come to my office to tell me emphatically, they write it on their evaluations, do not change the lab. This is what we want to do. This is what scientists do. I am always surprised by that… they always like it that way. Even if they have to put in more time. Evenings, weekends to feed their animals, take measurements…. the students like it better, it is better for them… thinking of something that has not been done but is doable is hard and you do have to be creative to do it.”

FACULTY INTERVIEWS YEAR 3: VALUE OF “TAKING THE PERSPECTIVE OF OTHERS”

Similarly, another historian said, “I ask them a question that is surprising. It might be something like, ‘What does it smell like in 14th century Florence?’ A question we have not talked about at all, but they have done a great deal of reading from primary sources about markets in Florence, or about demography, or about domestic spaces and ask them to think about that evidence in some new sort of way and to extrapolate about the lived realities of 14th century people about things they know from records.”

FACULTY INTERVIEWS YEAR 3: VALUE OF “TAKING THE PERSPECTIVE OF OTHERS”

A couple of other professors also mentioned types of “role playing.” One humanities professor said, “In my journalism class they do mock press conferences. They are given a topic – political topic - that is not a scandal. They have to draw on real-world knowledge, using realistic people who would appear at a press conference. Have to generate the story, present the information, and the rest of the class asks questions as reporters and they write the news story as journalists. Then the student presenting has to field the questions. They think outside, but within the box as well. They have to be active. I go back to the definition of [creativity as] ‘new but useful.’ “

FACULTY INTERVIEWS YEAR 3: VALUE OF “STUDENT CHOICE”

  One professor of an interdisciplinary

course talked about the creation of a student-led senior colloquium. She said, “...second semester senior year giving seniors a term paper is just a death sentence. So I designed the senior colloquium as a way to get them to be more creative and to flow a little bit more with the material, rather than a research paper. The students design some of the colloquium itself, they get to pick the topic, they get to pick the readings and they get to determine the project that they will do collectively…. So for instance this year they did a magazine and I think they liked the opportunity to explore the genre of this sort of women’s magazine at the same time trying to make some sorts of statements about women and feminist issues that are important to them.”

FACULTY INTERVIEWS YEAR 3: VALUE OF “PEER EVALUATION, INFLUENCE”

  Many faculty talked about the importance of

students engaging with their work, of “owning” their work. Faculty thought that to the extent that their peers would see their work, students would care more about it. The professor who had her students use “Moodle” to discuss their ideas publicly found that there was healthy competition among the students to present their unique voice which led to very creative expression.

FACULTY INTERVIEWS YEAR 3: VALUE OF STUDENTS MAKING CONNECTIONS OUTSIDE THE CLASSROOM

A social science professor who has students do work outside the classroom said, “If they are out in the community, if they are working with a group of people, I think it gives them a way to think about an issue in a much bigger way… I mean you can look at the school violence issue, you can look at a bunch of material about school violence and have good book knowledge of it and the way that boys and girls interact in high school, but when you go to a football game and you watch it happen, you see these little kids pushing each other around and you see the bullies… it makes them come back with different questions, different ways to think about what they read.”

FACULTY INTEVIEWS, YEAR 3: VALUE OF “INTERDISCIPLINARITY”

At Ohio Wesleyan the “National Colloquium” is a way to promote interdisciplinary thinking and discussion of issues outside of the classroom. One of the professors brought this up and said, “There’s a different topic every year, a

major national or international topic, and they bring in speakers and we try to bring tie in other events on campus and stage a play that focuses on that topic. Students will receive partial credit for attending the lectures or the performances or participating in activities which have something to do with the topic of the National Colloquium and one year it was about the arts. One of my colleagues raised the topic of having the National Colloquium for the arts and students could get other class credits for attending art openings or jazz band performance…. Making it part of the curriculum kind of forces the students to take advantage of things that are available to them… but that kind of thing is what I think educational institutions could do.” 

Where does Creativity Happen on Campus?

SUMMARY YEAR 2 SURVEY DATA

  Surveys were administered to 260 first-year students, 375 senior students, and 147 faculty across the four colleges during the 2007-2008 academic year to assess perceptions and experiences. Out of a wealth of information yielded by this research, a few interesting items are described below.

1. There seem to be more opportunities for critical than creative thinking in classes.

2. Class projects are evaluated by students as a means to engage in both critical and creative thinking.

3. Our campus environments were rated favorably with respect to the potential for creative thinking, although faculty rated “idea time” and “risk taking” as somewhat limited in their experience.

SUMMARY YEAR 2 SURVEY DATA4. Women students reported more experience with both

creative and critical thinking and more positive attitudes about creativity compared to men students.

5. When asked to write about their most creative experiences, 61% of faculty mentioned their own research. Teaching and collaboration with students were each mentioned by 32% of the faculty.

6. When asked about barriers to creative thinking on campus, the most common response of senior students was that class pedagogy was a barrier (40% of students mentioned this); while faculty most often wrote about time as a barrier (18% of faculty).

7. A diverse campus environment (in terms of class, race, sex, sexual orientation, and national origin) is perceived to facilitate creative thinking.

YEAR 2 SURVEY DATA: FACULTY AND STUDENT PERCEPTIONS OF CREATIVE ENVIRONMENT CHARACTERISTICS

Participants rated the extent to which each characteristic existed on their campus: 1 = not at all, 7 = extensively

Faculty 1st Yr Seniors Item5.02 5.50 5.31* Challenge 5.08 5.50 4.94* Freedom 4.69 5.30 4.90* Idea Support 4.45 5.28 4.88* Trust/Openness 4.65 5.11 4.89* Dynamism/Liveliness4.50 5.28 5.05* Playfulness/Humor 4.72 5.23 5.32* Debate 3.83 4.79 4.36* Risk Taking3.73 4.80 4.45* Idea Time 4.14 4.28 4.56* Conflict 4.67 5.40 4.99* Supportive Environment4.86 5.00 5.04 Working in groups4.58 5.14 4.98* Active models of creative thinking 4.88 5.22 5.10 Assignments encourage independent

problem-solving & risk-taking

*indicates significant difference between groups

YEAR 2 SURVEY: WHERE DOES CREATIVE THINKING HAPPEN ON CAMPUS, ACCORDING TO STUDENTS?

Students rated how much each event involved creative thinking, rating 1 = low to 5 = high

CREATIVE THINKING

First Year Seniors

Engaging in a hobby 4.18 4.13Completing projects for class 3.74 3.78Writing papers for classes 3.74 3.68Extra-curricular or co-curricular activity 3.45 3.55Interacting w/ students outside of class 3.51 3.52Cultural events on campus 3.19 3.18Interacting w/ faculty outside class 3.02 3.14Interacting w/ students, faculty in class 3.02 3.04Listening to speakers on campus 2.78 2.74Communicating w/ peers internet 3.03 2.73*Reading material for class 2.83 2.61*“Surfing” the internet 2.63 2.59

*indicates significant difference between groups

STUDENT IDEAS ON THE ENCOURAGEMENT OF CREATIVE THINKING, FOCUS GROUPS YEAR 2 • Professors allow students to give input (open-minded professors);

discussion-based classes• Bringing in outside artists, bands, speakers, poets to “stir the pot”• Open-ended assignments in classes• Opportunities to connect work in different courses• Classes with real-world applications • Need more time• Student-themed housing• Classes where professors challenge students “to the edge”• Comfortable spaces where students can come in and “do their own

thing”• Lunch tables with faculty• Diversification and diversity requirements encourage students to

learn something new• Classes that require internships, service learning (first-year

seminars with these components were praised)• Study abroad • Senior research project (projects in general); • Fewer required courses would mean more freedom for students;

Any way for students to propose their own classes?

FACULTY IDEAS ON THE ENCOURAGEMENT OF CREATIVE THINKING, FOCUS GROUPS YEAR 2 • General Comments.

• Spaces for faculty to get together and discuss pedagogy

• Fifty minute time periods are not long enough• Student-controlled spaces, events, organizations• Need to find time to live “the life of the mind”• Service learning that involves many members of

the community• Senior seminars; Independent research project,

summer science programs, senior projects;• Teaching abroad with students from your institution• Encourage diversity, appreciation for

internationalism• Taking students to conferences, on field trips, etc. • Provide faculty with opportunities to do things

differently – team-teach, time to do projects• Make interdisciplinary work easier, e.g., break down

departmental autonomy• Allow faculty to remain “learners” as much as

possible

FACULTY IDEAS ON THE ENCOURAGEMENT OF CREATIVE THINKING, FOCUS GROUPS YEAR 2 •  In the classroom.

• Guest speakers• Use of Moodle and other forms of technology (that

encourage engagement)• A final project with several options to the traditional

paper (assignments encouraging active engagement)

• Provide opportunities for students to interact with the “real world”

• Setting up projects, assignments where outcome is unclear

• Assignments that require “vulnerability” – students have to show something about themselves

FACULTY INTERVIEWS, YEAR 3: VALUE OF ASSESSMENT

“Doing the rubrics – I didn’t know how that would work, I had my doubts two years ago. I actually started using a rubric and I find it easier to grade consistently …. I really like it and use it now for all my labs, it helps my grading. If students would look at them they could realize exactly what they aren’t doing well. I give this back to students with written comments, and the ones who look at it could improve their labs which is helpful…. I found it helpful and I was totally shocked. Someone said you can’t do it with lab reports – I think you can – I showed that you could.”

FACULTY INTERVIEW YEAR 3: QUOTE

“The challenge is to encourage faculty members to be experimental in the ways they imagine student work, without abandoning key commitments to core educational goals having to do with having students express themselves clearly in argument to also say that the ways in which you might go about cultivating a sense of the discipline should go beyond, at least experiment with going beyond, writing 5 or 25 page papers. How you encourage faculty members to do that I am not sure. I suppose one way of doing that is giving them inspirational models. Make sure people within their departments and across disciplines talk to each other about whatever works, whatever seems to gain traction in this area.”

“Expectations of what a classroom is supposed to be like are changing. Using the classroom as a place to do presentations, as a place to do small group work, as a place to do debates… use the classroom as a theatre some days, making a place where we do lecture some days… making it a place where when they are getting ready to go to my class they are wondering what are we going to do today? Mixing it up, building some excitement. It can ‘t be like that everyday, but…. “

FACULTY INTERVIEWS YEAR 3: Changing Classroom

AAC&U VALUE AND OHIO 5 TEAGLE

VALUE TEAGLE

15 general rubrics 1 general rubric

Developmental scale Evaluative scale

Wide range of institutions Liberal arts colleges

Adaptable Adaptable

Useful for assessment Useful for assessment

www.aacu.org/value/rubrics www.wooster.edu/teagle