Why Assessment Results Are Hard to Use

(and what to do about it)

Hi…I’m the Assessment Director

The Theory

What Could Go Wrong?

Look familiar?

Assessment Design

Choosing an Achievement Scale

Pick a scale that corresponds to the educational progress we expect to see.

Example:

0 = Remedial work

1 = Fresh / Soph level work

2 = Jr / Sr level work

3 = What we expect of our graduates

Subjective measurements

Use authentic data that already exists Sacrifices reliability for validity

Create subjective ratings in context Sacrifices reliability for validity

Assess what you observe, not just what you teach! Recovers some reliability

For complex goals like general education:

www.coker.edu/assessment

Survey Attitudes and Behaviors

CIRP, BCSSE, Student satisfaction inventory Link student ID numbers to other indicators These can be easily and productively

outsourced

Analysis and Reporting

Some Tools

ANOVA Logistic Regression and ROC curves

http://statpages.org/logistic.html Pivot tables

Longitudinal Analysis

Use data you already have

Grades FAFSA, SAT, ACT items Demographic data ePortfolio statistics or work Library circulation statistics

Analysis: Example

Writing by Class and Library Use

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Fall 05 Fall 04 Fall 03 Fall 02

Start Term

Avg

. W

riti

ng

Sco

re

> Library

< Library

You don’t have to average

Measurement requires: Units Ability to aggregate

What we actually do is estimation Without measurement we shouldn’t average

Averages

Proportions

Min / max

Major Code Class Student ID Last First ObservationsArt (4) Senior xxxxxxx Tatiana Tolstoy 7

Min Max AvgAnalytical Thinking Remedial Jr/Sr 1.00Creative Thinking Remedial Fr/So 0.80Writing Effectiveness Fr/So Jr/Sr 1.17Speaking Effectiveness Remedial Jr/Sr 1.00

Confidence Intervals

Longitudinal Writing Effectiveness in Day

0.00

0.50

1.00

1.50

2.00

2.50

2003-4 2004-5 2005-6 2006-7

Academic Year

Avg

. S

core AF F

AF M

W F

W M

Portfolio analysis: epic FAIL!

Grammar Scores

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

5.00

07 06 05 04

Entering Year

Ave

rag

e S

core

Day

Evening

Analysis: Example

Proportions

Day Analytical Min Rating

0%

20%

40%

60%

80%

100%

2003-4 2004-5 2005-6 2006-7

Jr-Sr

Fr-Soph

Remedial

Conclusions

Design it yourself Use a sensible longitudinal scale Combine with other data Avoid averages

Last Requests?

highered.blogspot.com