A National Online Essay Evaluation Service: Valid, Reliable, & Cost Effective Writing Assessment Leslie C. Perelman Leslie C. Perelman Program in Writing

A National Online Essay A National Online Essay Evaluation Service:Evaluation Service:Valid, Reliable, & Cost Valid, Reliable, & Cost Effective Writing Effective Writing AssessmentAssessment

Leslie C. Perelman Leslie C. Perelman <[email protected]><[email protected]>

Program in Writing and Humanistic Program in Writing and Humanistic StudiesStudies

Massachusetts Institute of Massachusetts Institute of TechnologyTechnology

TestWriteTestWrite™™ (a. k. a. (a. k. a. iMOAT)iMOAT) Enhancing the best way of Enhancing the best way of

assessing writingassessing writing Enthusiastic reception from both Enthusiastic reception from both

students & facultystudents & faculty Cost effectiveCost effective Reliable scoring and deliveryReliable scoring and delivery

MotivationsMotivations

Existing modes of placement Existing modes of placement inadequateinadequate Standardized testsStandardized tests Timed impromptuTimed impromptu PortfolioPortfolio

Students write on computers Students write on computers not papernot paper

The Common The Common PracticePractice

Student walks into Student walks into large room and large room and writes with pen and writes with pen and paper on small deskpaper on small desk

Student reads, at Student reads, at most, a two or three most, a two or three paragraph text for paragraph text for each essayeach essay

Student writes one Student writes one or two essays on or two essays on specific questionsspecific questions

Handwritten essays Handwritten essays are scored (usually are scored (usually by two readers)by two readers)

Score is sent on Score is sent on paper to student and paper to student and / or advisor/ or advisor

Student works at Student works at homehome

Student views Student views readings of any readings of any length onlinelength online

Student accesses Student accesses questions onlinequestions online

Student writes one Student writes one or two essays at or two essays at home within a home within a specified period specified period using a computer.using a computer.

Essays are scored on Essays are scored on either printed copy either printed copy or on screen by or on screen by readers in real or readers in real or virtual grading virtual grading sessionssessions

Student and advisor Student and advisor access both score access both score and text of essays.and text of essays.

TestWriteTestWrite

A Better Way to A Better Way to Assess WritingAssess Writing

Developed by Experts in Developed by Experts in Writing Assessment at Five Writing Assessment at Five Diverse UniversitiesDiverse Universities

With TestWriteWith TestWrite Students Write Students Write

This WayThis Way Not This WayNot This Way

Intelligent ReadingsIntelligent Readings

Give Students Time Give Students Time to Think & Writeto Think & Write

Assess the Writing Assess the Writing ProcessProcess

More Time (and More Time (and savings) May Allow for savings) May Allow for Two EssaysTwo Essays “ “if you want to give real weight to if you want to give real weight to

[student] essays, you must secure two [student] essays, you must secure two test essays from each student with test essays from each student with some separation in time as well as in some separation in time as well as in topic, and the least possible separation topic, and the least possible separation in time as well as in topic, and the in time as well as in topic, and the least possible separation in time is to least possible separation in time is to have one test essay written in the have one test essay written in the morning and the other in afternoon of morning and the other in afternoon of the same day.”the same day.”

-- Paul Diederich of ETS-- Paul Diederich of ETS

The founder of modern The founder of modern writing writing assessmentassessment

TestWrite High-Level WorkflowTestWrite High-Level Workflow

Create Class

Administrator Student Reader

Create Test

Upload Students

View Readings & Questions

Write & Submit Essay

[Write Comments]

Assign Readers

Finalize Scores

Publish Results

Reporting

Register for Exam

View Results [& Comments]

Read & Score Essays


Create Class


Admin - Create ClassAdmin - Create Class

Admin - Create ClassAdmin - Create Class

Admin - Create Test GroupAdmin - Create Test Group

Admin - Create Test GroupAdmin - Create Test Group


Create Class


Create Test

Admin - Create Test - DefineAdmin - Create Test - Define

Admin - Create Test – Set DatesAdmin - Create Test – Set Dates

Admin - Create Test – Enter ReadingsAdmin - Create Test – Enter Readings



Admin - Create Test – Enter QuestionsAdmin - Create Test – Enter Questions


Create Class


Create Test

Upload Students

Admin - Upload StudentsAdmin - Upload Students





Create Class


Create Test

Upload Students Register for Exam

Student - LoginStudent - Login

Student - RegistrationStudent - Registration





Create Class


Create Test

Upload Students


Register for Exam

Student – View ReadingsStudent – View Readings

Student – View ReadingsStudent – View Readings

Student – View AssignmentStudent – View Assignment

Student – View AssignmentStudent – View Assignment


Create Class


Create Test

Upload Students



Register for Exam

Student – Submit EssayStudent – Submit Essay

Student – Submit EssayStudent – Submit Essay

Student - Confirmation EmailStudent - Confirmation Email


Create Class


Create Test

Upload Students



Assign Readers

Register for Exam

Admin – Assign ReadersAdmin – Assign Readers

Admin – Assign ReadersAdmin – Assign Readers


Create Class


Create Test

Upload Students



Assign Readers

Register for Exam

Read & Score Essays

Reader – Get Essay to ReadReader – Get Essay to Read

Reader – Get Essay to ReadReader – Get Essay to Read

Reader – Read EssayReader – Read Essay

Reader – Grade EssayReader – Grade Essay

Reader – Second Reader, Second ReadReader – Second Reader, Second Read


Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Register for Exam

Read & Score Essays

Admin – Resolve Split ScoreAdmin – Resolve Split Score



Admin – Real-time Reader ReportingAdmin – Real-time Reader Reporting





Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Register for Exam

Read & Score Essays

Admin – Define Result CategoriesAdmin – Define Result Categories



Admin – Set Test CutsAdmin – Set Test Cuts

Admin – Send to ResultsAdmin – Send to Results

Admin – Finalize and PublishAdmin – Finalize and Publish


Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Register for Exam

View Results

Read & Score Essays

Student – View Results (not yet Student – View Results (not yet available)available)

Student – View ResultsStudent – View Results




Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Reporting

Register for Exam

View Results

Read & Score Essays

Reporting – Download ResultsReporting – Download Results




Reporting – Download Results (Excel)Reporting – Download Results (Excel)


Create Class


Create Test

Upload Students



Assign Readers

Finalize Scores

Publish Results

Reporting

Register for Exam

View Results

Read & Score Essays

Architectural OverviewArchitectural Overview

Web Server MS Windows 2000 / IIS

MS ASP.NETC#

Database MS SQL/2000

InternetInternet

Administrator

Student

Reader

TestWriteTestWrite FeaturesFeatures Automates all components Automates all components

of best practices in writing of best practices in writing assessmentassessment Focusing on writing processFocusing on writing process Allowing students modern Allowing students modern

toolstools Online holistic scoring Online holistic scoring

sessions with real time sessions with real time quality control measuresquality control measures

Digital Archives of Digital Archives of EssaysEssays For program assessmentFor program assessment

For portfoliosFor portfolios

For institutional researchFor institutional research

Two Perpetual Two Perpetual QuestionsQuestions What about cheating?What about cheating?

Will all students have Will all students have access?access?

Reponses & ResultsReponses & Results

Feedback from Feedback from StudentsStudents ““I am from a small town in I am from a small town in

Missouri. My writing essay was Missouri. My writing essay was easy for me because I enjoy easy for me because I enjoy thinking about these things, such thinking about these things, such as philosophy and the future. I as philosophy and the future. I went out to my cabin and I just went out to my cabin and I just sat around the campfire the sat around the campfire the whole time, just enjoying the whole time, just enjoying the silence and writing whatever silence and writing whatever came into my head. The next day came into my head. The next day I typed it up.”I typed it up.” (DePaul) (DePaul)

Feedback from StudentsFeedback from Students

““Taking the online writing Taking the online writing tests, I found, gave me the tests, I found, gave me the opportunity to leave the opportunity to leave the information when I needed information when I needed time to think and come back to time to think and come back to it when I was ready. The it when I was ready. The freedom also allowed me to freedom also allowed me to revise my work and not feel revise my work and not feel though my work was rushed or though my work was rushed or inhibited in any way.”inhibited in any way.” (DePaul) (DePaul)

Benefits for Writing Benefits for Writing FacultyFaculty Allows essays and questions to Allows essays and questions to

reflect the university’s and reflect the university’s and program’s specific educational program’s specific educational needs, priorities, and goalsneeds, priorities, and goals

Maintains control of all aspects of Maintains control of all aspects of evaluation process within the evaluation process within the university and writing programuniversity and writing program

Online grading sessions maintain Online grading sessions maintain “staff consensus building ” benefit “staff consensus building ” benefit while potentially reducing costswhile potentially reducing costs

Reflects and measures the writing Reflects and measures the writing process used and valued in collegeprocess used and valued in college

Encourages sharing of “best Encourages sharing of “best practices” among schools using the practices” among schools using the systemsystem

ReliabilityReliability

Reliability of Reliability of AssessmentAssessment ““Faculty involved in English Faculty involved in English

placement strongly feel that placement strongly feel that the Online represents much the Online represents much better the kind of writing better the kind of writing students will do in our courses, students will do in our courses, and, so far, it is working and, so far, it is working equally well for a wide range equally well for a wide range of students from honors to of students from honors to very low developmental.”very low developmental.” (U. (U. of Cincinnati)of Cincinnati)

Reliability of Assessment at Reliability of Assessment at MIT:MIT:Fall 2002 Survey of Instructors Fall 2002 Survey of Instructors Teaching Introductory Writing Teaching Introductory Writing ClassesClasses 1st Semester Students

in Introductory Writing Classes N = 147

1%12%

34%53%

1 2 3 4

1=Should not 1=Should not be in classbe in class

4=Definitely should4=Definitely shouldBe in classBe in class

Reliability of Assessment at Reliability of Assessment at MIT:MIT:Fall 2002 Survey of Faculty Fall 2002 Survey of Faculty TeachingTeachingWriting-Intensive ClassesWriting-Intensive Classes1st Semester Students in Writing-Intensive

Humanities Classes N=125

0%10%20%

30%40%50%60%70%

80%90%

100%

1 2 3 4Faculty Responses 1=Should not have been in class; 4 = appropriate writing

skills for class

Pass

Subject Req

Reliability of Assessment: Reliability of Assessment: Reduction in Complaints and Reduction in Complaints and Appeals at MIT 1992-2003Appeals at MIT 1992-2003

Appeals & Complaints by Students about Scores on MIT Placement Test

0

10

20

30

40

50

60

70

80

90

19

92

19

93

19

95

19

96

19

97

19

98

19

99

20

00

20

01

20

02

20

03

Year

# o

f S

tud

en

ts

SavingsSavings

Greater Efficiency in Greater Efficiency in Scheduling and Staffing Scheduling and Staffing ClassesClasses

Saving Money: Reliability & Saving Money: Reliability & Timeliness of Assessment Timeliness of Assessment Reduce Number of Sections Reduce Number of Sections NeededNeeded

Accuracy of Online Evaluation = Accuracy of Online Evaluation = Fewer SectionsFewer Sections LSU (4,000 1LSU (4,000 1stst Year Students) is Year Students) is

scheduling 4 fewer sections per scheduling 4 fewer sections per year because of fewer students year because of fewer students being misplacedbeing misplaced

Extrapolate to UC’s nine Extrapolate to UC’s nine undergraduate campusesundergraduate campuses

Save Money: Typed texts (even Save Money: Typed texts (even if they are longer) cost less to if they are longer) cost less to gradegrade At MIT At MIT (@$25 / hr.)(@$25 / hr.)

the average the average handwritten handwritten text costs text costs $7.15 to be $7.15 to be scored by two scored by two readersreaders

the average the average typed text typed text costs $5.40 to costs $5.40 to be scored by be scored by two readerstwo readers $0.00

$1.00

$2.00

$3.00

$4.00

$5.00

$6.00

$7.00

$8.00

Handwritten Typed

$7.15

$5.40

Save Money: Save Money: Automatically Automatically AccommodateAccommodateStudents with Students with DisabilitiesDisabilities

Save Money: Save Money: Eliminate Data EntryEliminate Data Entry

Save Money: Eliminate Save Money: Eliminate PaperworkPaperwork

Save Money: Avoid Save Money: Avoid Hotel Expenses for Hotel Expenses for ReadersReaders

Save Money: Eliminate Save Money: Eliminate Travel ExpensesTravel Expenses

Save Money: Reduce Save Money: Reduce Expense of Feeding Expense of Feeding ReadersReaders

OptionsOptions

Three ChoicesThree Choices

UC joins current consortiumUC joins current consortium

UC collaborates in further UC collaborates in further development and development and implementation through a implementation through a non-profit corporationnon-profit corporation

We all expand by partnering We all expand by partnering with for-profit companywith for-profit company

Staying SmallStaying Small

BenefitsBenefits Keep what we are doingKeep what we are doing Maintain controlMaintain control

CostsCosts Problem with reliabilityProblem with reliability Expansion difficultExpansion difficult Future uncertainFuture uncertain

Non-ProfitNon-Profit

BenefitsBenefits Maintain controlMaintain control Potential for growthPotential for growth Low economic costLow economic cost

CostsCosts Too small for economies of scaleToo small for economies of scale Build organization that actively Build organization that actively

functions for only a few months functions for only a few months a yeara year

Ensuring reliability still a Ensuring reliability still a problemproblem

PartneringPartnering

BenefitsBenefits Professional reliabilityProfessional reliability Economies of scaleEconomies of scale Limitless expansionLimitless expansion

CostsCosts More expensiveMore expensive Some loss of controlSome loss of control Possible culture conflictsPossible culture conflicts

A Possible PartnerA Possible Partner

Sapient | Higher Education Sapient | Higher Education ExperienceExperience Harvard UniversityHarvard University

Active relationship since 1997Active relationship since 1997 Multiple administrative initiatives including:Multiple administrative initiatives including:

ERPERP Technology StrategyTechnology Strategy Grants ManagementGrants Management Telecommunications BillingTelecommunications Billing Asset/Facility ManagementAsset/Facility Management

MITMIT Primary partner for OpenCourseWare digital publishing Primary partner for OpenCourseWare digital publishing

initiativeinitiative Program managementProgram management Technology implementationTechnology implementation Business process outsourcingBusiness process outsourcing Application management servicesApplication management services

Open University, UKOpen University, UK Enterprise eLearning FusionEnterprise eLearning Fusion Enterprise Architecture Planning and ImplementationEnterprise Architecture Planning and Implementation Customer Relationship ManagementCustomer Relationship Management

Sapient | Some Support Sapient | Some Support ClientsClients

JanusReduced costs and eliminated bottlenecks through a 24x7 application that processes millions of dollars of online transactions a year

NissanReduced turnaround time for issue resolution to 30 minutes and drove record volumes of unique visits

Royal Mail Achieved cost and process efficiencies in the development and deployment of new applications and work packages

Opodo24x7 support and uptime for Europe's leading travel portal with access to 400 airlines, 30,000 hotel properties, and a fleet of over 750,000 hire cars worldwide, as well as travel insurance

MITProvide 24x7 support for the workflow application that feeds the MIT OCW web site.

Lloyd’s of London24x7 uptime, significantly reduced costs, and transitioned to new releases without impacting the business

http://www.nissanusa.com/0,8848,,00.html

http://web.mit.edu/site/aboutsite.html

TestWrite Management TestWrite Management TeamTeam Leslie C. Perelman, Ph.D.Leslie C. Perelman, Ph.D.

Associate Dean, Undergraduate Associate Dean, Undergraduate EducationEducation

Director of Writing Across the Director of Writing Across the Curriculum, MITCurriculum, MIT

Chris FelknorChris Felknor Former System Architect, Former System Architect,

Staples.comStaples.com Glenn P. StrehleGlenn P. Strehle

Principal, Campus Business AdvisorsPrincipal, Campus Business Advisors Former CFO, MITFormer CFO, MIT

Frank AndryauskasFrank Andryauskas Former CIO, Staples Inc.Former CIO, Staples Inc.

FinallyFinally

A more valid and reliable A more valid and reliable assessmentassessment

Greater faculty and student Greater faculty and student satisfactionsatisfaction

Significant reduction in costsSignificant reduction in costs Proven and reliable platformProven and reliable platform

UC System is Taking UC System is Taking Some Hard HitsSome Hard Hits Most of these cuts are Most of these cuts are

threatening one of the threatening one of the world’s greatest educational world’s greatest educational experiencesexperiences

TestWrite offers the rare TestWrite offers the rare opportunity for Win – Winopportunity for Win – Win Significant reduction in costsSignificant reduction in costs Better educational practiceBetter educational practice

Documents

A National Online Essay Evaluation Service: Valid, Reliable, & Cost Effective Writing Assessment Leslie C. Perelman Leslie C. Perelman Program in Writing