Department of Humanities

College of Sciences and Liberal Arts

Writing Program Assessment at

New Jersey Institute of New Jersey Institute of TechnologyTechnology

Carol Siri Johnson

Associate Professor and Chair

Outline of Presentation

• Personal Introduction

• New Jersey Institute of Technology

• ABET

• Writing / Technical Writing Assessment

• Using Online Portfolios

• Using Analytic Scoring

• Examples of Data Analysis Results

Personal Introduction

• Mount Holyoke College

• English Ph.D., but few jobs

• New – Technical Writing Field

• Tenure-track Position at NJIT

• Chair, Research EA and TC History

My Assessment History

• Knew nothing about it

• Disliked standardized tests

• Was unconvinced of its value

• Then I saw a best-paper reading

• The Power of Assessment

Assessment at NJIT

• Department of Humanities active in assessment for ~20 years

• Previous Chair, NJCBSPT

• Norbert Elliot and ETS

• Course / Program Assessment

• Analytic Online Portfolio Assessment

On a Scale – Norbert Elliot

What is Assessment?

●Giving a student a grade?

●End-of-semester course evaluations?

●Faculty review of portfolios?

●Student surveys?

●The chair’s opinion?

What is Assessment?

●For Teachers, GRADES

●For Departments, COMMON EXAMS

●For Administration, COURSE EVALUATIONS

●For Learning, OUTCOMES – stating and reaching curricular goals in Courses and Programs

Assessment Can Be:

• Program assessment

• Curricular assessment

• Method of automatic updating

• Method of teaching collaboration

• Gathering statistics for multiple purposes.

ABET 2000- Accreditation Board for Engineering and Technology

ABET’s 8 General Criteria:

#4 Continuous Improvement

“The results of these evaluations must be systematically utilized as input for the continuous improvement of the program.”

Outcomes Assessment Cycle

NJIT Core Curriculum – GURs(General University Requirements)

● English Composition – 6 credits

● Cultural History – 6 credits

● Humanities Electives – 6 credits

● Senior Seminar – 3 credits

GURs Outcomes Assessment

●English Composition – every semester

●Cultural History – every other spring

●Humanities Electives – every other fall

●Senior Seminar – every semester

Assessment Basics – Validity

• The issue of “validity” means we are measuring what we want to measure; the test is appropriate to the subject

• The problem with writing assessment has been using multiple choice tests only – they alone are not valid as a measure of writing

• Multiple measures are often the best

Assessment Basics – “Associative” Validity

Correlating Multiple Inputs:• Machine-scored essay scores• Portfolio scores• Grade in course• Student GPA

Assessment Basics – Reliability

Can we be sure that the measurement, if taken again, would yield the same results?

•Inter-reader Reliability (scored by two separate readers, discrepancies resolved by a third)

•Content Reliability (portfolio contains work from entire semester)

Two Important Steps:

•Decide on and define goals or Decide on and define goals or criteriacriteria

•Develop method to assess Develop method to assess outcomes (goals/criteria)outcomes (goals/criteria)

Large Scale Writing Assessment Initiatives

●1950s ETS – Holistic Scoring

●1970s California (Edward White)

●1980s New York (CUNY)

●1990s New Jersey (NJCBSPT)

●2000s NJIT – Program Assessment

Example: Technical CommunicationAnalytic Online Portfolio Assessment

• Established Criteria by “Online Established Criteria by “Online Delphi” – email exchanges by facultyDelphi” – email exchanges by faculty

• 2004 – Failed effort (network down)2004 – Failed effort (network down)

• 2005-2009 – Successful Analytic 2005-2009 – Successful Analytic Online Portfolio Assessment, every Online Portfolio Assessment, every semestersemester

• 2009-Present – once per year2009-Present – once per year

Technical Communication Cycle

Using Online Portfolios

Procedure

1. Have Students Make Online Portfolios with Examples of the Main Assignments

2. Collect URLs3. Select Random Sampling (~60-100)4. All Instructors Meet5. ~1 Hour Calibration with Sample Portfolios where

instructors discuss scoring processes6. ~3 Hours Scoring7. Assessment Director Conducts Necessary

Adjudications8. Enter Scores into Excel or SPSS Dataset

Old Humanities Rubric

Old Score Sheets

Revised Score Sheet

Sample Data Collection

A Community of Assessment

• Shared Instructional Content

• Self-Evaluation and Norming

• Curricular Goals Evaluation

• Continual Improvement (ABET)

• Interesting Statistics