Critical Thinking AssessmentFebruary 4, 2010

Jack BautschVerna SwanljungBrian PalmerBarbara Goldner

Aryana Bates Deanna Li

Tracy FurutaniCesily Crowser

Ann Murkowski

To decide upon a writing prompt was an arduous process. We decided as a group that it needed to be as open-ended as possible. This is what we came up with.

Write a short essay addressing the following:

Barack Obama recently became President of the United States. The cartoon can have many meanings.

1. What does the cartoon say to you?

2. Does it leave anything out?

3. Which classes that you have taken here at North help you to think about your answer?

For your information: The words listed in this cartoon are the names of some important people and events in United States history.

I think it would have been more useful for assessing critical thinking if we had added one more question that asked them to show the evidence.

We learned that next time that if we want to evaluate something we have to be specific and ask for it.

As far as the rubric goes, it is clear in my mind, at least, that this is just a model for us that has been vetted by the AACU. It isn’t some pie in the sky that we are all going to use to evaluate our assignments.

We decided to try this tool, see where it fails, and then tweak the definitions in a way that works for us. We decided to see how we can make it work.

The prompt question is, “What does this cartoon say to you?” That requires a relationship with what is going on inside the illustration.

I am not so sure that many people have assimilated and can relate to this mountain of names, so in some respects this prompt is exclusionary. It puts some people at a disadvantage.

Based on my group’s experience and the other group’s scores, it seems to me the scores we found are remarkably low.

I can see either of two conclusions: either our students are not getting the learning outcome, or our assessment is flawed.

One of the rubric dimensions concerned considering other perspectives. But if the prompt asks, “What does the cartoon say to you?”, you are not going to tap into that.

If we are going to do this again, we need to ask questions that will reveal the answers we are wishing to assess.

Our group felt the same. The rubric did not match the task at all.

We had some people who answered the question really well, but if you are taking the rubric as written, they were falling short.

We are working with Civic Engagement now, starting anew. We have a similar rubric from the AACU, so now the task is to create a curriculum around that rubric.

That is so different from where this is. This starts with the curriculum we have and looking to find what critical thinking we can find.

When I decide what I want my students to know from this class and I write a test, I have this rubric in my mind.

If they cannot do these basic things, then they most likely will not pass my class.

Beyond that basic, of course there is more to know. It is good that they know more.

From there I construct my tests based upon what I see.

This rubric on critical thinking is a good rubric, if we can find the right questions that can address young people from Running Start and people from different countries.

The assessment must be unbiased without an assumption of prior knowledge and where everyone can talk from a personal viewpoint.

I was talking to Deanna before we started. Our math students have to do critical thinking, so how would we apply this rubric?

I could not think of a way to do it. I felt like a duck out of water.

I don’t understand the rubric, yet I have to evaluate someone else’s work. It was not a very good feeling.

You and I come from a similar perspective, and I do teach critical thinking. I can see assessing some of my assignments with this rubric.

But using this rubric for this cartoon was a problem. I am not used to reading and evaluating this kind of thing.

We were so lucky to have Aryana, someone with a social sciences perspective. We had to spend time together to understand how Aryana looks for evidence in this type of work.

The second problem was that the papers were pretty poor. Do we say they are all poor or do we put it in the context?

They were given this setting with this amount of time. Do we take that into account, or do we just say that we do not see evidence of satisfactory critical thinking?

We are seeing that this rubric is not structured from a mathematics and sciences perspective. For me it did make sense to use it on a piece of writing that had been done in a very short period of time.

It is easier to use from a particular disciplinary perspective, for sure.

I am feeling better about what we are doing in the next set of rubrics where we are doing it inside our classes, where it is more meaningful to me and more meaningful to my students.

For the Information Literacy and Civic Engagement Essential Learning Outcomes groups of faculty are working together starting from the rubric. We are finding places in our classes where we can assess one or several of the items.

We are starting from the rubric to line up assignments that are germane to our curriculum. We are from different disciplines, but we all teach Information Literacy. We are sharing how we make that clear and how our students demonstrate that competency in our different classes.

This approach seems much more relevant. I am working on the Civic Engagement outcome. I know I really want to do this, but as a parent educator how do I work on it? Where do I start?

Then I thought, “Oh! Kindergarten readiness.” I teach parents how to be involved in the school and advocate for their child.

It was interesting to take the rubric and ask myself where it fits, as opposed to using a given rubric to assess something outside of a given context.

Applying the Critical Thinking rubric to this cartoon task left me feeling bad. How can I give these students a zero when they didn’t even know what we were looking for?

I know these names that are in the cartoon. I have a relationship with these names. But some people didn’t know; they thought it was somebody standing with a Priest or a Judge.

It seems to me many people got stuck on the fact that they didn’t understand or have a relationship with the message. When we ask them how it touches them in some sort of way, nobody had the courage to say, “This doesn’t touch me at all.” That would have made an interesting essay.

In our group we had an international student who admitted she didn’t know what it was about or the context, but nevertheless she was able to illustrate that she could take something that was foreign and think about it.

That was a demonstration of critical thinking, talking about what she did and did not know in an intelligent way.

I think the cartoon lacks clarity for many people. The question, “Does it leave anything out?” implies an understanding of what should be in here. If it is not clear or logical, you are not going to be able to answer the question.

When you were developing this assessment what was the first question you started with that you said was more sophisticated?

We started with a text-based question and threw that out in favor of an image-based question. The major reason was that we didn’t want to place a bias against students who were reading in a second language and who only had an hour to read, comprehend and respond.

So we moved from very specific to super open-ended.

I understand why we use this instrument on a broad sample of students. I understand the model is changing to having assessments of a particular outcome occur within a class. My question is, why was this format chosen?

Well, we had a good advocate for collecting something from across the campus. It wasn’t what I would have chosen. I am much more prone to go with the current approach of embedding the assessment within a class.

In this one there was no reason for a student to do a good job on it. I am going to give them $6 worth of coffee coupons whether they write their name and leave or write the best essay.

I have trouble with this model, but it is a model that people do use.

But I hear you say that there is such a mismatch between the prompt and the rubric in this case that we really don’t know much about what students can do. Or am I wrong?

This data doesn’t say anything to me. What I need is a standard deviation, and I need it to correlate to how many classes they have had where critical thinking was an outcome.

Without that correlation we don’t know much. Then we have the additional problem of the lack of synergy between the rubric and the prompt.

It didn’t help us see what we wanted to see, which was whether or not our students are gaining something by coming to North Seattle.

The two students who did the best had either a bachelors degree or prior college experience.

I know the students to whom you refer who listed all the courses they had taken.

I would have expected a closer correlation between that experience and the result that was evident.

So we are destroying their ability to critically think.Laughter

Cesily and I found that we were interpreting the rubric two different ways. We found that I was looking on the first dimension and Cesily was looking on the fourth dimension.

That discussion changed our scores. If we are going to apply rubrics across disciplines, there needs to be more training, or we all need Aryana in our group.

At first I couldn’t get my brain around how to score this. It wasn’t until I had read them all, and we had our little session with Aryana, that I understood.

I would recommend you for all the groups.

Maybe the lesson out of this is that if a rubric seems more natural to a particular discipline, maybe we need a representative from that discipline to help normalize the scoring for the rest of the group.

A question: if you say you teach critical thinking in the sciences and math, do you think this rubric applies or needs modification?

I think the rubric is quite nice. I can see asking my students to interpret a data set or think critically about experimental design and apply this rubric.

That is the kind of evidence I would have available to me. I am not accustomed to assessing this kind of evidence or recognizing what to look for. I would be comfortable using it on an experimental design.

The Mountaintop cartoon is outside my discipline. It is not where I am used to looking for evidence of critical thinking.

Computer science students are working on an end product. These things on the rubric are happening, but they don’t necessarily have to explain it. It is built into the final answer.

I don’t know that I would break what they do into these specific pieces.

I am much more digital. This is too analog for me.

I have a question. In programming, is it really critical thinking that is happening, or is it the ability to recognize an algorithm that is appropriate?

Is this a rubric that you would even test a computer science student on?

Is it critical thinking or is it problem solving? Maybe I am in the problem solving realm and not in critical thinking. I don’t know. It could be.

What do you see as our next step?

Using the approach we are taking in Information Literacy and Civic Engagement obviously.

But as far as critical thinking is concerned, we have to remake our instrument, so it enables us to get information that will be useful.

So we are going to find faculty that say they are teaching critical thinking. But do you think faculty would be able to use this rubric on an assignment in the classroom? After massaging it as necessary, of course.

I think that massaging is very important. I need to put it into my words so that I am clear about what I am looking for.

The dimensions are there, but it needs the terminology of my class with things specific to the assignment.

And then you can show it to your students ahead of time so they know what you are looking for.

I don’t know if it was really clear that this was an early effort at collecting systemic data.

Yes. This was a first for us in terms of going beyond a college-wide survey into collecting student work across the campus outside of classroom boundaries.

Critical Thinking AssessmentFebruary 4, 2010

Jack BautschVerna SwanljungBrian PalmerBarbara Goldner

Aryana Bates Deanna Li

Tracy FurutaniCesily Crowser

Ann Murkowski