1

Improving Student Writing and Thinking through Assessment

General Education and Assessment: Creating Shared Responsibility for Learning Across the Curriculum

Association of American Colleges and Universities (AAC&U)Network for Academic Renewal Conference

Atlanta, GA • February 17, 2005Teresa L. Flateby

University of South Florida

2

General Education Assessment

Evolution 2000

Problem Areas-- Writing skills-- Liberal Arts Mathematics

Common Outcomes -- General Education Dimensions - Contemporary Issues

Intellectual Development

2003 – 2004Planned General Education Revisions

3

General Education Assessment

Evolution (continued)

In ProcessCommon for all Undergraduates

-- Mathematics

-- Natural Sciences

-- Social and Behavioral Sciences

-- Fine Arts and Humanities

-- Human and Cultural Diversity within the Global Context

-- English Composition

4

General Education Assessment

Targeted Student Outcomes• Writing Achievement• Cognitive Level Attainment• Intellectual Development• Contemporary Issues Understanding• Liberal Arts Mathematics

Achievement

5

Assessment Tools / Strategies Depend on

Outcomes• What you intend to do with results

– Determine proficiency– Use for program improvement

6

Evaluating Writing Assessment Process

• Initially assessed writing with a holistic method

• Realized writing was weaker than desired

• But – impossible to determine specific areas of weakness

7

To address this assessment concern…

An analytic scale was introduced to assess writing and cognitive levels

reached.

Keep in mind assessment purposes.

8

CLAQWA’s Development

Developed:•For use by faculty whose discipline is not English

– to provide consistent assessment of students’ writing

– to encourage student feedback•To be flexible

– identify skills appropriate for the assignment

– provide different weighting for skills•To attend to cognitive levels when developing assignments and to assess levels reached

.

9

Cognitive Level and Quality of

Writing Assessment (CLAQWA) Scale

10

Cognitive Scale

11Based on Madaus, 1973.

Analysis

Evaluation/Synthesis

Application

Comprehension

Knowledge

Branching of Bloom’s Taxonomy of Educational Objectives: Cognitive Domain

12

Importance of Cognitive Level Targeted in

Assignments

• Writing suffers when unprepared• Clear expectations for students and

instructors / administrators• Better understanding of results

Why?

13

First Assignment:USF's General Education Advisory Committee and the Undergraduate Council are interested in the elements you value in the classroom. These Committees are composed of university faculty, the USF Assessment Office, and undergraduate students.

In a 2-3 page essay, describe a course that would represent the ideal learning experience for you. Please be as specific and concrete as possible about what this course would include; use as much detail as you think is necessary to present clearly this ideal situation. For example, you might want to discuss what the content or subject matter would be, what the teacher would be like, your responsibilities as a student, the evaluation process that would be used, and so on. Please explain why you feel the specific course aspects you discuss are ideal for you.

Essay Assignment: FYC – ENC 1101

14

Essay Assignment: Capstone Course

The University is interested in the elements you value in the classroom. To address these, you have been given the opportunity to assist with developing and teaching a university course.

DirectionsDesign the ideal (discipline) college level course including one or more of these dimensions: values/ethics, ethnicity, gender, environmental issues, international and global perspectives.

15

In a well-written, typed 2-4 page essay, please describe your course and provide a title. Explain why the elements you have focused on are important.

The General Education Advisory Committee and the Undergraduate Council appreciate your ideas and will consider them in their assessment of the university’s General Education curriculum.

Essay Assignment: Capstone Course (continued)

16

Your essay should address the following questions:

1. What subject matter and method(s) of instruction will you include in your course?

2. What type of learning will occur? (Examples: learning facts, analyzing concepts, applying skills or ideas to new situations, creating solutions to complex problems)

3. What would you do as the teacher and what would you expect your students to do in return?

4. What types of interactions will occur among members of the class?

5. What do you expect the students to get out of the course and how would you know (assess) what was accomplished?

Essay Assignment: Capstone Course (continued)

17

Verbs to Use for Constructing Prompts to Elicit Specific Cognitive Levels in Student

Writing

This list suggest verbs intended to guide the development of the prompt. Using these words in writing prompts will help communicate the expected cognitive level to be achieved by students.

Level 4: Analysis, Synthesis, Evaluationdiscriminate plan weighinfer organize evaluatecompare generatecombinecontrast appraiseconcludecreate critique supportdesign judge

Level 3: Applicationdetermine use an approach apply a principlechart develop solve a problemimplement choose an appropriate relateprepare procedure demonstrate

Level 2: Comprehensiondescribe estimateillustrategeneralize classify give an example ofparaphrase explain state in your own wordssummarize predict translate

Level 1: Knowledgedefine list nameidentify match reportlabel recall selectstate recite

18

Level 5: Transitional words, phrases, sentences and paragraphs (coherence devices) smoothly connect the paper’s elements, ideas and/or details allowing the reader to follow the writer’s points effortlessly.

Level 4: Coherence devices are rarely missing and do not impact the reader’s understanding.

Level 3: Coherence devices are appropriate when present, but effort is required to connect the writer’s points.

Level 2: Coherence devices are attempted, but are ineffective.

Level 1: Coherence devices are absent or inappropriate.

Trait: Coherence Devices

19

Level 5: Details develop the main idea and provide supporting statements, evidence, or examples to explain or persuade effectively.

Level 4: Details support the purpose and main idea of the paper with adequate clarity, depth and accuracy.

Level 3: Details are related to the purpose and main idea of the paper but do not provide sufficient depth or clarity or accuracy to explain or persuade effectively.

Level 2: Details are loosely related to the purpose of the paper but are lacking clarity, depth or accuracy.

Level 1: Details do not support the purpose or main idea of the paper.

Trait: Quality of Details

20

Level 5: The paper clearly has and maintains a main idea throughout.

Level 4: The main idea is clear, although a rare occasional extemporaneous element is introduced.

Level 3: The paper has a main idea, but additional unrelated ideas distract the reader.

Level 2: The main idea is not maintained or it is unclear.

Level 1: The paper lacks a main idea and appears to reflect the writer’s “stream of consciousness.

Trait: Main Idea

21

Assessment Participants

Assessment Activity: Participants:General Education Assessment Faculty, Administrators, Students Advisory Committee

CLAQWA Writing Scoring Team Faculty, TA’s (undergraduate & graduate)

MID Scoring Team Faculty

Cognitive Level Scoring Team Faculty

Individual Student Participants Honors College Students (Essay & Questionnaire Content Analysis)

22

Analytic Writing Skills Ranked

• Sentence construction varies appropriately.• Point of view is consistent.• A main idea is present and maintained.• Ideas are comprehensible.• All assignment requirements are fulfilled.• Appropriate audience(s) are consistently

addressed.• Purpose is clear and specific.• Details are sufficient in quantity to develop

main idea.• Word choice is accurate.• Grammar and mechanics.• Opening supports the main idea.• Coherence devices are present and

appropriate.• Details are sufficient in quality to develop

main idea.• Paragraphs support unity.• Reasoning supports the main idea.• Closing supports the main idea. (Lowest skills related to

reasoning.)

23

Cognitive Level Distribution

24

• Emphasis on Critical Thinking

-- Push for smaller classes

-- Funds for TAs and undergraduate assistants

• Focus on Process Writing

-- Emphasis on feedback and revision

-- Expansion of writing lab to center

-- Changes in composition program

• Center on Learning/Student outcomes rather than faculty content centered

General Education Curricular ChangesResulting from Assessment Data

25

Discoveries

• Value in an institution assessing writing rather than an agency (ETS/ACT)

• Benefits of analytic scoring over holistic scoring

• Efficiency of essay assignments– Qualitative and quantitative data– Writing, cognitive levels, intellectual development– Data regarding general education dimensions– Areas of strength/weaknesses in curriculum– Conversations about quality writing

26

Discoveries (continued)

• Students’ degree of pedagogical savvy• Writing skill and intellectual

development moderately correlated• Both faculty and administrative

support necessary for assessment to be successful

• Faculty development process

http://isis.fastmail.usf.edu/ugrads/eandt/

http://usfweb2.usf.edu/ugrads/writingsurvey/