AssessingAssessingStudent Learning OutcomesStudent Learning Outcomes

ininWriting CoursesWriting Courses

Lisa Marchand

Cosumnes River College

Sacramento, California

ACCJC Accreditation ACCJC Accreditation StandardsStandards

California Community Colleges are required to establish “well-documented, ongoing processes for evaluating” effectiveness at all levels of the institution.

The foundation of this evidence must be data collected and analyzed in a consistent way, over time.

By the Year 2012By the Year 2012Engage in an ongoing dialog

about student learning Have a well documented process

for evaluating Student Learning Outcomes

Show tangible evidence that your program is continuously moving toward improved student learning.

Think about your own Think about your own classes or program…classes or program…

How do you assess students writing skills?

Are Student Learning Outcomes in place?

Are they measurable?How are they measured?

Take five minutes to talk it over!

Cosumnes River CollegeCosumnes River College

Questions:How well can students demonstrate

that they have met the Student Learning Outcomes for writing courses that they have passed?

How well can students who have placed into a level through the assessment process demonstrate the skills specified in the prerequisite course’s SLO’s?

Existing DiagnosticsExisting Diagnostics

Not specifically aligned to course SLO’s (Either too general or too specific)

Mining data from these diagnostics would be too difficult

ConcernsConcerns

We wanted to assess SLO skill sets but we didn’t want to have these assessments affect the students’ grades.

We didn’t want to have the SLO assessments be used to evaluate the teachers in any way.

Assessing SLO’s at the end of a course wouldn’t provide the best information on students who entered through the placement process.

Our Goal: Our Goal: Sustainable Continuous Quality Sustainable Continuous Quality

ImprovementImprovement

Analyze Data and Discuss Observations

Measure Student Learning Outcome

s

Make Adjustments

Ramping UpRamping Up

Develop student learning outcomes

Define metrics to measure SLO’s

Experiment with various measures and discuss observations

Creating Departmental Creating Departmental Diagnostic Writing Diagnostic Writing

AssessmentsAssessments

Measure expressive writing skillsA holistically scored writing sample

Measure meta-linguistic abilitiesAn objective multiple choice

assessment

Analyze SLO’s to Identify Analyze SLO’s to Identify MetricsMetricsMeasurable elements of writing

include:◦Syntactic structures and variety◦Accuracy of verb tense form and use◦Appropriate word forms◦Range of vocabulary◦Rhetorical patterns◦Development◦Use of cohesive devices◦Other

……include measures of meta include measures of meta skillsskillsMeta-linguistic skills are included in

SLO’sExamples:

◦Identify X◦Label X◦Analyze X◦Recognize X◦Differentiate between X and Y

An objective instrument can provide discrete measures of each of these

Build a Scoring GuideBuild a Scoring GuideCRC chose an ETS style holistic

scoring guide model with 5 levels:

◦Insufficiently prepared = 1◦Under-prepared = 2◦Fairly prepared = 3◦Well prepared for the level = 4◦Demonstrates skill above level = 5

……write descriptors for each write descriptors for each levellevelStart with the SLO’s – write a

descriptive narrative to match expressive skills they describe

This will be the score 4 – “well prepared for the course”

To write descriptors for scores 1 – 3 and for score 5, examine the sample papers already marked by faculty

Select A Writing PromptSelect A Writing PromptFor the lowest levels, we chose a

picture and guiding questions:“Look at the picture. Who is the man? Where is he? What happened? How does he feel? Why does he feel that way? Where is he going? What will happen next? Write the story!”

For intermediate SLO’s and beyond, we selected this prompt:

“Some people like to live in big cities, others do not. Tell where you prefer to live and explain why.”

Create an Objective Create an Objective InstrumentInstrumentAlign assessment items directly to SLOMake more than one item to measure

each skill…vary the contextsUse multiple choice format & SCANTRON©

Check to see that the terms used are consistent with what students have encountered in classes and texts

Collaborate to check for validity, bias, and clarity

Scrutinize the ProcessScrutinize the Process

Validity CheckValidity CheckAccording to Test Better, Teach Better

by W. James Popham, content-related evidence of validity can be determined by relying on “human judgement”

A team of expert colleagues agrees that each test item measures the skill it was designed to reflect.

The team must also agree that the test items are accessible and free of bias.

Check for ClarityCheck for ClarityConduct item-by-item study of

the testBe sure that the language level

matches the textsBe sure that vocabulary is

appropriate and standardBe sure the content doesn’t

unfairly affect a student’s ability to respond—it is not biased.

Check for BiasCheck for BiasContent Bias

◦Gender◦Culture◦Experiential

Linguistic Ambiguity◦Vague wording◦Confusing grammar◦Low-frequency vocabulary

Difficulty

Create a Score Tally in Create a Score Tally in ExcelExcel

student id

SLO's 1 - 4

Holistic Score

SLO #1 Evidenc

e of Plannin

g

SLO #2 Logic and

Development

SLO #4 Syntax

SLO #3 Mechan

ics

Identifying

Sentence

Parts SLO #5 Items 1-7

(7Q's)

Identifying

Parts of Speech SLO #6 Items

8-14 (7 Q's)

Verb Tense Forms SLO #7 Items 15-20 (6

Q's)

Verb Tenses

in Questions SLO

#2 Items

21-24 (4 Q's)

475549 4 2 3 2 2 5 4 1 2485646 2 2 2 1 2 4 4 4 3922766 3 2 2 2 2 4 4 6 4987991 3 2 2 2 1 5 6 5 31020104 3 2 3 3 2 2 3 3 21036656 2 2 1 1 1 4 3 4 2

……includeincludeColumns for

◦Student ID #’s◦Holistic Writing Score◦Scores for component skills

evaluation◦Scores from each section of the

objective assessment

Analyze DataAnalyze DataCollect descriptive statisticsMake correlationsFocus on evidence of student learningProvide data sets to all teachers for

use in self-studies and program reviewUse the assessments over several

semesters to enable longitudinal study of a significant sample of student work

Continue the DialogContinue the DialogUse collective observations to suggest

and implement changes to ◦Curriculum◦Methods◦Materials

Select appropriate data to support resource requests like facilities and FTE

Have faculty experienced in the process mentor and norm their colleagues

Other than ACCJC Standards, Other than ACCJC Standards,

Why Assess SLO’s?Why Assess SLO’s?

Find out what you’re “teaching” that the students aren’t learning.

Move faculty dialogue away from anecdotal discussion

Confirm intuition (or not)Gather substantive support for

resource requests

Example SLO’sExample SLO’sSLO 1: Identify and apply basic essay components

SLO 2: Employ the writing process

SLO 3: Interpret and respond to writing prompts

SLO 4: Develop paragraphs appropriate to rhetorical mode

SLO 5: Identify and correct grammatical errors through editing

SLO 6: Write simple, compound, and complex sentences with correct punctuation

SLO 7: Identify and correctly use grammar structures and word forms

Example of an SLO with Example of an SLO with MetricsMetrics

SLO 7: IDENTIFY AND USE GRAMMAR STRUCTURES

AND WORD FORMS

identify and use all basic verb tenses with very rare errors in form (simple and progressive aspect of present, past and future)

identify and use some perfect verb tenses with occasional error in form and usage (present perfect, present perfect progressive, past perfect)

identify and use singular, plural and non-count forms and appropriate quantifiers and articles without errors in form

identify parts of speech and use correct word forms in writing

W 50 SLO Attainment Fall W 50 SLO Attainment Fall ‘09‘09

W 50 SLO#7 W 50 SLO#7 IDENTIFY AND USE GRAMMAR IDENTIFY AND USE GRAMMAR

STRUCTURES AND WORD FORMS STRUCTURES AND WORD FORMS

0

1

2

3

4

5

6

7

8

9

10

Holistic Score Grammar Correcting Errors

Total Possible

Average

Best Score

Lowest Score

Convert to a Comparable Convert to a Comparable FormFormDo some simple arithmetic:

◦Divide the total possible into 100 100÷5 = 20

◦Multiply the answer by the average score 20 x 2.18 = 43.6%

W 50 SLO#7 W 50 SLO#7 IDENTIFY AND USE GRAMMAR IDENTIFY AND USE GRAMMAR

STRUCTURES AND WORD FORMS STRUCTURES AND WORD FORMS

0

10

20

30

40

50

60

70

80

Holistic Score Grammar in Writing

Correcting Errors on objective test

Student Success

Student Success

Observations from the Observations from the DataDataNeither the students’ ability to

correct grammar errors on the objective assessment nor the teachers’ evaluation of students’ grammar in the writing sample were significantly correlated with their success on the holistically graded writing sample.

There was no significant correlation between students writing sample and the objective assessment.

Significant CorrelationsSignificant CorrelationsThe students holistic writing

scores were directly correlated with the score the rater gave them on their essay organization.

The students holistic writing scores were directly correlated with the score the rater gave them on their essay development.

Thank you for your Thank you for your participation!participation!