GENERAL  EDUCATION  COURSE  REVIEW    

ENG  1013,  Composition  II  

Fall  2013  Response  to  Fall  2012  Assessment  Report  and  Findings  

Due  to  anecdotal  evidence  of  problems  and  complaints  from  students  and  instructors  in  2011,  four  specific  areas  were  targeted  for  assessment  in  Fall  2012/Spring  2013.      

I.  Clear  communication  of  goals  and  student  learning  outcomes.    Led  by  Dr.  Marcus  Tribbett,  Interim  Director  of  Composition.    

Prior  to  2010  Composition  I  and  II  syllabi  did  not  contain  learning  outcomes  and  often  didn’t  contain  clear  grading  standards,  percentage  breakdown  for  assignments,  or  course  calendars  including  assignments  and  major  due  dates.    In  2010  the  Dean  of  Humanities  and  Social  Sciences  instituted  the  following  requirements  for  all  syllabi:  

Inclusion  (in  Outcomes/Objectives)  of  language  of  the  specific  General  Education  goal:  Communicating  Effectively  

Inclusion  of  required  texts  and  materials  Inclusion  of  clear  grading  policies:    percentage  breakdown  for  each  assignment  and  

grading  scale  Inclusion  of  a  course  calendar  with  schedule  of  major  due  dates  

In  fall  2012,  Dr.  Tribbett  reviewed  all  of  the  syllabi  submitted  for  Composition  II  to  determine  compliance  with  the  standard  requirements.  

Findings:    Most  syllabi  were  in  compliance  with  the  requirements  but  some  needed  revision.    A  few  (five  or  six)  were  missing  the  General  Education  Goal  for  Communicating  Effectively  (some  had  the  goal  but  in  different  words).    Some  did  not  list  course  requirements  clearly,  have  a  grading  scale,  or  have  any  calendar  for  assignments.  

Actions:    Individually  corresponded  with  instructors  requesting  appropriate  changes  in  syllabi  and  resubmission  to  HSS.  

Conclusions  and  proposed  future  actions:    Most  instructors  began  incorporating  the  agreed-­‐upon  goal  and  met  university  and  department  requirements  for  consistency.    The  Interim  Director  presented  these  findings  to  Comp  II  instructors  in  a  meeting  that  took  place  in  spring  2013  and  stressed  the  importance  of  having  clear  grading  policies  and  calendars.    Future  syllabi  will  be  systematically  reviewed  for  statement  of  specific  learning  outcomes  and  assessment  method  as  well  as  inclusion  of  the  General  Education  goal.  

Institutional  Response  and  Implementation:  Prior  to  the  beginning  and  during  the  first  week  of  the  fall  2013  semester,  the  Department  of  English  and  Philosophy  and  the  Dean  of  Humanities  

and  Social  Sciences  sent  several  detailed  emails  to  faculty  regarding  syllabus  requirements.  The  emails  included  written  instructions  as  well  as  a  sample  general  Humanities  and  Social  Sciences  syllabus.  Such  requirements  were  also  reiterated  at  the  Composition  pre-­‐semester  workshop.  A  sample  Composition  II  syllabi  was  distributed  to  faculty  and  also  posted  to  Composition  Instructor  Network,  a  new  online,  interactive  resource  and  instructional  material  repository  for  ASU’s  Composition  faculty  developed  by  the  Director  of  the  Writing  Program,  Dr.  Kristi  Costello.  During  the  fall  2013  semester,  Dr.  Costello  will  create  a  common  Composition  II  syllabus,  complete  with  assignments,  deadlines,  and  policies,  for  optional  use  by  new  Composition  faculty.  Several  new  and  returning  faculty  have  shown  interest  in  the  new  syllabus.    

Fall  2013  Composition  II  syllabi  were  reviewed  for  inclusion  of  general  education  goals,  clear  grading  standards,  percentage  breakdown  for  assignments,  and  course  calendars  including  assignments  and  major  due  dates.  Similar  to  Dr.  Tribbett’s  2012  conclusions,  the  majority  of  fall  2013  syllabi  were  in  compliance  with  ASU’s  requirements.  Dr.  Costello  has  been  individually  corresponding  with  and  mentoring  instructors  whose  syllabi  were  not  in  compliance.  

II.  Grade  Inflation  

In  response  to  a  perception  among  some  English  faculty  and  others  across  the  university  that  grades  in  Composition  II  have  become  inflated  in  recent  years,  data  was  obtained  from  ASU’s  Office  of  Institutional  Research  containing  five  years  of  course  grades  semester  by  semester.      

 

 

Findings:    As  is  apparent  from  the  data  above,  the  skewing  of  grades  toward  “A,”  already  troubling  at  39.2  percent  in  2008,  increased  by  19.4%  percent  to  46.8  percent  in  2012.      This  data  does  indeed  indicate  substantial  grade  inflation  in  the  past  five  years,  with  yearly  increases  in  the  occurrence  of  “A”  grades  and  reductions  of  almost  all  other  grades  over  the  same  time  period.    While  there  are  a  significant  number  of  Honors  and  Concurrent  High  School  sections  of  Composition  II  over  the  five  year  period,  which  would  be  expected  to  produce  a  higher  number  of  “A”  grades  in  those  sections,  the  relative  number  of  Honors,  Concurrent,  and  regular  Composition  II  sections  remained  fairly  constant  over  the  five  years.    Thus,  the  number  of  Honors  and  Concurrent  sections  could  explain  some  level  of  grade  inflation  but  not  the  19.4  percent  increase  in  the  occurrence  of  “A”  grades  over  this  five  year  period.  

Actions:    The  Interim  Director  of  Composition  discussed  grading  and  did  some  norming  exercises  in  the  Fall  2012  workshop,  but  clearly  more  work  needs  to  be  done  in  this  area  to  stress  the  usefulness  of  rubric  grading  for  improved  consistency  across  the  Composition  Program.  The  

Prepared  by  the  Office  of  Institutional  R esearch  &  P lanningThis  information  reflects  what  is  in  the  B anner  S ystem  as  of  the  date  and  time  of  this  report.    R un  Date:  01/28/2013 ;  R un  T ime  1:37  pm.

Assessment of Student CoursesBased on Grades Earned in ENG 1013 - Composition IIArkansas State University-Jonesboro, 2008 - 2012

Term Subject Course # Course Title A AU B C D F FN I W Total A AU B C D F FN I W200810 ENG 1013 COMPOSITION II 520 0 345 216 49 77 0 0 89 1,296 40.1% -- 26.6% 16.7% 3.8% 5.9% -- -- 6.9%200830 ENG 1013 COMPOSITION II 1 0 6 8 1 2 0 0 0 18 5.6% -- 33.3% 44.4% 5.6% 11.1% -- -- --200840 ENG 1013 COMPOSITION II 24 0 10 15 2 4 0 0 4 59 40.7% -- 16.9% 25.4% 3.4% 6.8% -- -- 6.8%200860 ENG 1013 COMPOSITION II 169 0 96 53 16 22 32 0 59 447 37.8% -- 21.5% 11.9% 3.6% 4.9% 7.2% -- 13.2%

Total 2008 714 0 457 292 68 105 32 0 152 1,820 39.2% -- 25.1% 16.0% 3.7% 5.8% 1.8% -- 8.4%

200910 ENG 1013 COMPOSITION II 514 0 275 127 22 38 23 0 78 1,077 47.7% -- 25.5% 11.8% 2.0% 3.5% 2.1% -- 7.2%200930 ENG 1013 COMPOSITION II 38 0 15 7 1 2 0 0 3 66 57.6% -- 22.7% 10.6% 1.5% 3.0% -- -- 4.5%200960 ENG 1013 COMPOSITION II 156 0 115 99 28 42 18 0 40 498 31.3% -- 23.1% 19.9% 5.6% 8.4% 3.6% -- 8.0%

Total 2009 708 0 405 233 51 82 41 0 121 1,641 43.1% -- 24.7% 14.2% 3.1% 5.0% 2.5% -- 7.4%

201010 ENG 1013 COMPOSITION II 484 0 294 135 50 61 10 0 96 1,130 42.8% -- 26.0% 11.9% 4.4% 5.4% 0.9% -- 8.5%201030 ENG 1013 COMPOSITION II 46 0 22 8 10 4 1 0 11 102 45.1% -- 21.6% 7.8% 9.8% 3.9% 1.0% -- 10.8%201060 ENG 1013 COMPOSITION II 192 0 90 72 17 44 16 0 61 492 39.0% -- 18.3% 14.6% 3.5% 8.9% 3.3% -- 12.4%

Total 2010 722 0 406 215 77 109 27 0 168 1,724 41.9% -- 23.5% 12.5% 4.5% 6.3% 1.6% -- 9.7%

201110 ENG 1013 COMPOSITION II 483 1 278 153 30 64 21 0 95 1,125 42.9% 0.1% 24.7% 13.6% 2.7% 5.7% 1.9% -- 8.4%201130 ENG 1013 COMPOSITION II 49 0 15 13 5 1 1 0 3 87 56.3% -- 17.2% 14.9% 5.7% 1.1% 1.1% -- 3.4%201160 ENG 1013 COMPOSITION II 216 0 127 92 27 18 6 0 48 534 40.4% -- 23.8% 17.2% 5.1% 3.4% 1.1% -- 9.0%

Total 2011 748 1 420 258 62 83 28 0 146 1,746 42.8% 0.1% 24.1% 14.8% 3.6% 4.8% 1.6% -- 8.4%

201210 ENG 1013 COMPOSITION II 472 0 239 137 24 48 24 0 67 1,011 46.7% -- 23.6% 13.6% 2.4% 4.7% 2.4% -- 6.6%201230 ENG 1013 COMPOSITION II 70 0 24 13 1 3 1 1 8 121 57.9% -- 19.8% 10.7% 0.8% 2.5% 0.8% 0.8% 6.6%201260 ENG 1013 COMPOSITION II 234 0 110 76 21 27 14 3 40 525 44.6% -- 21.0% 14.5% 4.0% 5.1% 2.7% 0.6% 7.6%

Total 2012 776 0 373 226 46 78 39 4 115 1,657 46.8% -- 22.5% 13.6% 2.8% 4.7% 2.4% 0.2% 6.9%

Percent of Total GradesNumber of Total Grades

Interim  Director  also  researched  scholarly  literature  and  listservs  for  writing  program  administrators  to  determine  whether  Grammarly,  a  program  currently  available  to  all  ASU  faculty,  could  be  useful  in  helping  faculty  objectively  measure  the  level  to  which  essays  meet  the  learning  outcome  of  demonstrating  “proficiency  in  standard  American  English.”  

Proposed  Future  Actions:      The  Composition  Committee  will  work  on  developing  and  approving  a  standard  grading  rubric  for  Composition  I  and  II  essays.  Grade-­‐norming  will  be  more  greatly  emphasized  in  the  fall  2013  Composition  Workshop.    

Institutional  Response  and  Implementation:  A  new  assessment,  one  which  took  a  representative  sample  of  argumentative  essays  (common  assignment)  and  jury  assessed  them  was  conducted  in  spring  2013.    Their  ratings  of  these  essays  were  compared  to  the  grades  the  same  essays  received  in  the  course.  (See  below  for  more  information  regarding  this  study.)  

In  fall  2013,  Dr.  Costello  slightly  adapted  the  rubric  used  in  the  spring  2012  assessment,  developed  by  faculty  who  took  part  in  the  2013  composition  assessment,  to  include  corresponding  grades  to  be  associated  with  the  level  of  mastery  of  the  three  areas  of  import  (also  slightly  revised  to  reflect  best  practices):  Content/Thesis,  Organization  and  Structure,  Style  and  Mechanics.  Dr.  Costello  is  optimistic  that  pairing  grades  with  textual  priorities  will  help  instructors  ensure  that  they  are  assigning  grades  appropriate  to  the  quality  of  work  being  assessed.  Dr.  Costello  distributed  the  rubric  to  faculty  at  the  one-­‐day  training  and  orientation  for  Composition  Link  faculty  and  at  the  general  Composition  pre-­‐semester  workshop.  This  rubric  has  also  been  posted  to  Composition  Instructor  Network  for  instructors’  convenience.    

At  the  general  Composition  pre-­‐semester  workshop,  Dr.  Costello  led  a  grade-­‐norming  session,  initiated  a  discussion  regarding  what  ASU  composition  faculty  value  in  student  writing,  shared  scholarship  on  best  practices  in  composition  grading,  and  strongly  encouraged  instructors  to  conduct  norming  sessions  in  their  own  classrooms.  Recognizing  that  some  instructors’  reluctance  to  assign  quality-­‐based  grades  is  due  to  their  concerns  regarding  student  retention  and  confidence,  Dr.  Costello  hopes  that,  in  line  with  Composition  scholarship,  engaging  in  grade-­‐norming  in  the  composition  classroom  will  help  create  a  common  discourse  for  class  discussions  of  writing  and  facilitate  students’  understanding  of  the  standards  expected  of  student  writers  at  ASU.  As  of  fall  2013,  new  composition  faculty  are  required  to  have  at  least  fifty  percent  of  the  final  grade  in  their  course  based  on  the  quality  of  student  writing,  as  opposed  to  effort,  participation,  completion,  and  attendance  (criteria  heavily  weighted  by  current  composition  faculty.).  This  will  be  required  of  all  faculty  beginning  fall  2014.  Conversations  about  grading  and  grade  inflation  will  be  continued  by  the  Composition  Committee  whose  first  charge  is  to  revise  the  common  rubric  and  thus,  create  community  grading  standards  to  be  endorsed  by  the  ASU  Writing  Program,  required  for  new  faculty,  and  encouraged  of  all  faculty  spring  2014.  The  committee  will  then  decide  whether  or  not  the  rubric  will  become  required  for  all  composition  faculty.    

The  committee  will  also  generate  common  course  objectives  and  requirements  for  a  range  of  numbers  of  papers  and  paper  lengths,  which  should  help  ensure  a  more  consistent  level  of  rigor  across  Composition  II  sections,  to  be  piloted  spring  14  and  implemented  fall  2014.  The  committee  will  also  consider  other  strategies  for  improving  grade  inflation  such  as  holistic  scoring,  multi-­‐reader  grading,  and  portfolio  systems.      

 III.  No  Systematic  Observation  of  Teaching  

Since  the  Composition  Program  has  been  without  a  director  for  several  years,  there  has  been  no  systematic  observation  of  teaching  and  the  Composition  Committee,  comprised  of  interested  faculty  regularly  involved  in  teaching  composition,  had  not  met  for  several  years.  

Findings:    The  Interim  Director  learned  anecdotally  and  through  student  complaints  that  some  composition  instructors  were  not  holding  scheduled  class  meetings  but  were  instead  using  “Blackboard  Fridays”  or  “Blackboard  Thursdays”  to  substitute  regularly  for  class  meetings,  even  though  these  courses  were  listed  as  Traditional  instructional  method  in  the  Class  Schedule.      While  Dr.  Tribbett  was  also  getting  reports  of  excellent  quality  instruction  going  on  in  many  Comp  II  classes,  he  had  reason  to  believe  that  the  quality  of  classroom  instruction  was  inconsistent.    He  therefore  determined  that  we  needed  to  assess  classroom  teaching  through  observation.    

Actions:      

• The  new  Chair  of  English  &  Philosophy  temporarily  banned  the  use  of  “Blackboard  Fridays”  except  in  cases  where  individual  instructors  came  to  her  and  made  a  case  for  the  quality  of  particular  instructional  practices  to  substitute  for  face-­‐to-­‐face  class  time.      

• Faculty  will  conduct  classroom  observations  of  Composition  II  instructors  of  all  ranks  (tenured  to  adjunct)  during  Spring  2013,  having  completed  observations  of  Composition  I  instructors  during  Fall  2012.  

• Revived  Composition  Committee  and  tasked  all  eight  members  with  classroom  observations  taking  academic  rank  into  account  so  that  no  pre-­‐tenure  or  non-­‐tenured  faculty  were  reviewing  tenured  faculty.    Only  tenured  faculty  observed  other  tenured  faculty  to  protect  pre-­‐  and  non-­‐tenured  faculty  from  any  potential  ill  will.      

• Developed  a  draft  rubric  for  observation  with  four  categories  and  a  scale  (see  attached)  despite  faculty  argument  for  narrative-­‐only  comments  

• Shared  preliminary  findings  of  observations  with  Comp  instructors  in  a  meeting  prior  to  Spring  2013  semester  

Proposed  Future  Actions:    Composition  Committee  will  continue  to  conduct  observations  and  will  reconvene  to  discuss  observation  results.  The  committee  will  also  plan  how  to  address  findings  with  eye  toward  making  best  practices  more  widespread  in  the  program  through  teaching  workshops,  training  sessions,  peer  observations,  mentoring,  shared  assignment  databank,  etc.    New  Director  of  Writing  Program  will  work  with  Composition  

Committee  and  get  their  input  on  improving  program-­‐wide  class  instruction  and  on  developing  best  practices  for  Web-­‐Assisted  and  online  classes.  

Institutional  Response  and  Implementation:  The  Chair  of  English  &  Philosophy,  Dr.  Janelle  Collins  has  continued  the  ban  on  “Blackboard  Fridays,”  except  in  a  couple  of  cases  in  which  individual  instructors  made  a  case  for  the  quality  of  particular  instructional  practices  to  substitute  for  face-­‐to-­‐face  class  time.  Faculty  conducted  classroom  observations  of  Composition  II  instructors  of  all  ranks  (tenured  to  adjunct)  will  continue  fall  2013.  All  Composition  II  faculty  will  be  observed  each  academic  year.  Similar  to  last  year,  the  Composition  Committee  will  be  tasked  with  classroom  observations  taking  academic  rank  into  account  so  that  no  pre-­‐tenure  or  non-­‐tenured  faculty  are  reviewing  tenured  faculty.    Only  tenured  faculty  will  observe  other  tenured  faculty  to  protect  pre-­‐  and  non-­‐tenured  faculty  from  any  potential  ill  will.  For  consistency,  observing  faculty  will  utilize  the  same  observation  rubric.  Findings  will  be  shared  with  faculty  spring  2014.    

In  addition  to  continuing  faculty  observations,  several  additional  steps  (explained  in  more  detail  above)  are  being  explored  and  implemented  to  create  a  better  and  consistent  standard  of  quality  in  our  Composition  II  courses:  Composition  Instructor  Network  (which  a  variety  of  assignments,  activities,  readings,  and  other  resources),  a  community  grading  rubric,  common  course  objectives,  improved  training  and  workshops,  and  consistent  longitudinal  assessment.  Already,  the  Composition  instructors  have  been  offered  two  technology-­‐related  pedagogical  workshops  and  an  extended  pre-­‐semester  workshop.    

IV.  No  documentated  use  of  student  learning  data  specifically  related  to  the  General  Education  goal  for  communicating  effectively.  Although  there  have  been  efforts  over  the  years  to  assess  this  goal  using  standardized  tests  and  collection  of  sample  papers,  the  results  were  never  widely  disseminated  or  discussed  in  order  to  lead  to  improved  student  learning.    The  Interim  Director  of  Composition  focused  on  assessment  of  Composition  I  during  Fall  2012  and  Composition  II  during  Spring  2013.  

 Assessment:  The  Interim  Director  collected  one  argumentative  paper  using  sources  from  a  representative  sample  of  Composition  II  courses  and  assessed  them  by  a  jury  of  trained  composition  assessment  raters  using  a  rubric.    The  essays  were  assessed  for  thesis,  coherence,  use  of  sources,  and  grammar/usage.    

ENG  1003,  Composition  II  

Spring  2013  Assessment  Plan  

Needed:    Direct  assessment  data  on  whether  Composition  II  courses  are  meeting  objectives  in  terms  of  measurable  student  outcomes.      Fall  2012  assessment  data  did  show  significant  growth  in  both  the  category  of  competency  with  thesis/coherence  and  also  in  high  competency,  but  as  the  findings  discuss,  the  assignment  needs  to  be  more  standardized  and  the  assessment  needs  

to  be  normed  to  give  the  Composition  Program  more  useful  information.    In  addition,  while  there  is  anecdotal  information  (such  as  the  energetic  faculty  listserv  discussion  Summer  2012)  that  faculty  university-­‐wide  feel  that  our  students  aren’t  proficient  in  standard  American  English,  we  don’t  have  solid  data  to  help  us  analyze  the  extent  of  the  problem  or  if  the  problem  is  worse  among  particular  groups  of  students.      

Assessment:    For  this  assessment,  the  Interim  Director  of  Composition  gathered  the  last  essay  assignment  from  sections  of  Composition  II  representing  as  nearly  as  possible  the  proportions  of  kinds  of  students  (developing,  regular,  Honors)  and  kinds  of  instructors  (tenured/pre-­‐tenure,  non-­‐tenure  track  contract  instructors,  and  temporary  or  part-­‐time  adjuncts).    The  essays  collected  represented  an  end-­‐of-­‐term  argumentative  essay  assignment  (a  position  essay)  common  in  Composition  II.    Student  names  were  removed  but  the  actual  grade  assigned  to  each  essay  was  kept  in  a  separate  file  by  the  Interim  Director  for  comparison  purposes  later  and  the  essays  were  then  coded  to  indicate  the  type  of  section  (remedial,  regular,  Honors)  as  well  as  the  type  of  instructor.  

Instructors  representing  the  range  of  ranks  teaching  Composition  II  classes  were  chosen  for  training  during  spring  2013  and  May  Interim  2013.    These  instructors  met  with  the  Interim  Director,  Henry  Torres,  and  Josie  Welsh  twice  during  spring  2013  to  discuss  the  task,  potential  problems,  what  kinds  of  instructions  need  to  go  out  to  instructors  participating  in  the  assessment,  etc.    These  instructors  looked  at  how  Blackboard  Learn  could  be  used  as  a  platform  for  assessing  these  student  essays.  

After  the  essays  were  collected  at  the  end  of  April  2013,  the  instructors  received  training  from  Henry  Torres  and  ITTC  in  assessment  rating.    This  included  a  norming  session  to  determine  the  level  of  inter-­‐rater  reliability.    Following  the  mid-­‐May  training  session,  the  instructors  analyzed  all  the  student  artifacts  as  homework.  

The  essays  were  analyzed  using  the  Grammarly  tool  available  in  Blackboard.    This  tool  is  available  to  all  ASU  faculty  but  the  Composition  faculty  in  particular  feel  that  it  is  not  reliable  or  useful.      The  Composition  faculty  analyzed  the  first  twelve  lines  of  the  second  page  of  each  student  artifact  for  grammar  and  usage  using  a  rubric,  identifying  and  categorizing  not  only  the  level  of  overall  competence  but  also  individual  areas  of  competence  such  as  subject-­‐verb  agreement,  comma  usage,  and  spelling  and  numbers  of  significant  errors.    Each  artifact  was  analyzed  by  at  least  two  ASU  instructors.    The  Grammarly  analysis  of  usage  and  grammar  was  then  compared  to  the  ASU  faculty  analysis.    The  goal  was  not  only  to  establish  data  showing  the  extent  of  our  student’s  competence  in  standard  American  English  at  the  end  of  Composition  II  (proficiency  in  standard  American  English  is  a  key  part  of  the  General  Education  learning  outcome),  but  also  the  extent  to  which  Grammarly  might  or  might  not  be  a  useful  tool  for  Composition  instructors.    

In  addition,  the  essays  were  also  analyzed  with  a  rubric  to  determine  the  level  of  competency  >75%  and  the  level  of  high  competency  >90%  for  thesis  and  coherence  measured  separately.    

Again,  each  student  essay  was  analyzed  by  at  least  two  ASU  instructors  in  an  attempt  to  produce  more  comparable  and  reliable  findings  than  were  achieved  in  fall  2012.    Thesis  and  coherence  are  central  to  the  first  element  of  the  General  Education  outcome  for  Composition  II,  that  students  can  “construct  and  deliver  a  well-­‐organized”  (emphasis  added)  oral  or  written  presentation.    

The  instructors  then  returned  to  the  ITTC  as  a  group  to  compile  their  report  of  their  findings  concerning  the  level  of  competence  Composition  II  students  achieved  in  their  end-­‐of-­‐term  argumentative  essays  in  the  areas  of  thesis,  coherence,  and  grammar/usage.    At  the  conclusion  of  their  work,  the  instructors  completing  the  training  received  certificates  as  composition  assessment  raters.      

Composition Assessment – Spring 2013

NUMBER OF ARTIFACTS GRADED – COMP I = 189; COMP II = 392 NUMBER OF STUDENTS – COMP I = 95; COMP II = 196 ASSIGNMENT – End-of-Course Paper TOOL – DEPARTMENTAL RUBRIC; Adapted from The University of Nebraska at Kearney 2010. Permission granted.

TRAINING HELD FOR GRADERS? YES GRADERS: 8 Faculty – including adjunct faculty, instructors, and professors

RESULTS – THESIS

Thesis: thesis statement and thesis development

4—The writer articulates the thesis clearly and presents cogent evidence in favor of his or her argument in every paragraph.

3—The writer states the thesis reasonably clearly—the reader does not need to guess or even to infer the paper’s thesis—and supports the argument with solid evidence and reasons. In one or two spots the evidence seems flimsy, or the argument tendentious, but overall the writer presents a careful, sound, and convincing argument.

2—The writer states one thesis but ends up arguing two or more. The argument seems rushed or perfunctory, and the evidence that the writer presents to support his/her claims is inadequate.

1—There is more than one thesis or none at all. The writer often substitutes textual summary for argumentation. S/he presents opinions rather than evidence and reasons for his/her claims, often signaled by such phrases as “I think” and “I believe.”

CONSISTENCY BETWEEN GRADERS – Do raters grade in a similar fashion? Reliability statistic should be at least .7. Raters had some trouble agreeing: Rater1 and Rater2 = .38 Rater3 and Rater4 = .63 Rater5 and Rater6 = .54 Rater7 and Rater8 = .43 Recommendation: Continue to train graders for this category; use sample artifacts

EFFECTIVENESS OF TOOL: How Well is Rubric Working? – Pretty well.

Distribution of scores (green) look different from the random distribution (orange) that one would expect if random ratings were assigned to the essays:

Actual match rate between two raters on a given artifact (yellow) was higher than the match rate one would expect by chance (orange). Percentages reflect differences in actual and random match rates.

SCORES: Summarize the ratings of the artifacts - THESIS scores from COMP II essays were statistically significantly higher than scores for COMP I essays. Many final COMP I essays contained no thesis. Are the authors of such essays passing the course? Comp I – 45% scored 3 (proficient) or higher; 40% stuck at 2; 15% received 1s

Comp II - 57% scored 3 (proficient) or higher; 31% stuck at 2; 12% received 1s

RESULTS – ORGANIZATION

Organization and Coherence:

4—Every paragraph is in its proper place, and the transitions are smooth. The argument builds methodically toward a conclusion. In addition, the sentences within each paragraph are well-articulated; there is a topic sentence where needed, followed by evidence and reasons for the claim laid out in the topic sentence. Each paragraph supports the thesis and fits integrally into the paper as a whole.

3—The paper is well organized, but the transitions do not have the finesse of those in a “4” paper. The reader can see the “rivets” that hold the paper together. Claims in paragraphs are developed but sentences could flow together more smoothly or be revised to make the point more tightly.

2—The paper has a discernible order, but it might benefit from re-ordering or reorganizing some paragraphs. Claims may lack sufficient development. The transitions are rough, some quotations are “dropped” into the text without warning or explanation, and the essay does not build organically toward a conclusion.

1—There is little discernible organization within the paragraphs in the essay as a whole and/or claims are undeveloped.

CONSISTENCY BETWEEN GRADERS: Do raters grade in a similar fashion?

Reliability statistic should be at least .7. Raters had less trouble agreeing: Rater1 and Rater2 = .54 Rater3 and Rater4 = .62 Rater5 and Rater6 = .51 Rater7 and Rater8 = .67 Recommendation: Continue to train graders for this category; use sample artifacts EFFECTIVENESS OF TOOL: How Well is Rubric Working? – Pretty well. Distribution of scores (green) look different from the random distribution (orange) that one would expect if random ratings had been assigned to the essays. 1 and 4 aren’t applied very often; however, one would expect few ratings of 1 for COMP II students:

Actual match rate between two raters on a given artifact (yellow) was higher than the match rate one would expect by chance (orange). Percentages reflect differences in actual and random match rates. Lower difference scores probably reflect little use of 1:

A grade of 4 is rarely assigned.

SCORES: Summarize the ratings of the artifacts:

Comp I – 40% scored 3 (proficient) or higher; 52% stuck at 2; 8% received 1s

Comp II - 51% scored 3 (proficient) or higher; 41% stuck at 2; 8% received 1s

Similar to findings from THESIS, COMP II essay scores were significantly higher than COMP I essays in ORGANIZATION; however, for both courses, essays received higher marks for thesis than they received for organization.

RESULTS – GRAMMAR

Grammar and Usage:

4—Grammar, spelling, and mechanics are nearly perfect. The language is clear, concise, and engaging. Sentence structures and lengths are varied with excellent command of subordinating and coordinating strategies used to create compound-complex sentence structures. The word choice is apt and precise, not overblown, clichéd, or too flowery.

3—There may be a few minor grammatical errors, but on the whole the paper is grammatically clean and correct. The grammatical and spelling errors are limited to difficult issues. The language is clear, though not elegant, economical but not quite succinct. Sentence structures show some variation and complexity. Word choice lacks the crisp appropriateness of a “4” essay.

2—There are multiple grammatical errors, but they are neither so pervasive as to slow down the reader nor so serious that the reader cannot understand what the writer is trying to say. The writer uses more words than necessary to convey the point, perhaps to pad the essay. Though most sentences are readable, little variation in sentence structure

occurs beyond some compounds formed with conjunctions; repetitive sentence patterns create “choppiness.” The writer occasionally lapses into cliché. The essay seems written in phrases, without attention to precise meaning of words or recognition of redundancy. Some words may be misused, some used in almost the right sense.

1—Sloppy grammatical, spelling, and mechanical mistakes litter the piece, interrupt the flow of reading, and make comprehension difficult. Sentences, even when grammatically correct, rarely if ever vary from simple SVO structures.

CONSISTENCY BETWEEN GRADERS – Do raters grade in a similar fashion? Reliability statistic should be at least .7. Raters had less trouble agreeing Rater1 and Rater2 = .6 Rater3 and Rater4 = .6 Rater5 and Rater6 = .47 Rater7 and Rater8 = .7

Recommendation: Continue to train graders for this category; use sample artifacts

EFFECTIVENESS OF TOOL: How Well is Rubric Working? – Pretty well.

Distribution of scores looks different from the random distribution (orange) that one would expect from random ratings assigned to the essays. Grades of 1 and 4 are not assigned often; however, one would expect few ratings of 1 for COMP II students:

Actual match rate between two raters on a given artifact (yellow) was higher than the match rate one would expect by chance (orange), except for category 4 because there were few ratings of 4. Percentages reflect differences in actual and random match rates:

Faculty should discuss whether they wish to keep 4 or make finer distinctions between 2 and 3 such that a less rigorous 4 emerges as a scoring option. Program should consider taking a close look at the percentage of 2’s and 3’s for the learning outcome called GRAMMAR for Comp I and Comp II:

SCORES: Summarize the ratings of the artifacts:

Comp I - 41.5% scored 3 (proficient) or higher; 42% were stuck at 2: 16.5% were rated 1

Comp II - 52% scored 3 (proficient) or higher; 43% were stuck at 2; 5% were rated 1

Similar to findings from ORGANIZATION, COMP II ratings for GRAMMAR were significantly higher for COMP II essays than for COMP I essays. Additionally, many final essays were graded as incoherent sentences. Are the authors of those essays passing the course?

FEEDBACK TO RATERS

Below are charts that present each rater’s grading tendencies compared with ratings given by the rater with whom they were paired. Individuals were told they could contact Dr. Josie Welsh, Director of Assessment, ABI 315, 972-2989 to confirm unique identifiers. Codes were only given only to the individual rater.

0 means you assigned the same score as your paired rater’s score

1 means you assigned one more point to an artifact than your fellow rater assigned.

-1 means you assigned one fewer point to an artifact than your fellow rater assigned.

Rater One:

Rater Two:

Rater Three:

Rater Four:

Rater Five:

Rater Six:

Rater Seven:

Rater Eight:

Feedback to Raters: Overall Average Ratings Given to Essays

Descriptive Statistics

Measure Rater M SD N

THESIS POINTS One 2.56 .55 75

Two 2.26 .95 74

Three 2.72 .90 76

Four 2.17 .79 76

Five 2.86 .67 70

Six 2.88 .90 69

Seven 2.26 .97 70

Eight 2.49 .91 71

Total 2.52 .88 581

ORGANIZATION POINTS

One 2.52 .64 75

Two 2.18 .83 74

Three 2.62 .89 76

Four 2.18 .76 76

Five 2.79 .61 70

Six 2.74 .70 69

Seven 2.36 .90 70

Eight 2.59 .77 71

Total 2.49 .80 581

GRAMMAR POINTS One 2.53 .64 75

Two 2.50 .82 74

Three 2.46 .93 76

Four 2.34 .62 76

Five 2.66 .61 70

Six 2.38 .81 69

Seven 2.53 .79 70

Eight 2.32 .79 71

Total 2.46 .76 581

TOTAL POINTS One 7.61 1.55 75

Two 6.93 2.24 74

Three 7.80 2.53 76

Four 6.70 2.02 76

Five 8.30 1.64 70

Six 8.00 2.14 69

Seven 7.14 2.49 70

Eight 7.41 2.11 71

Total 7.48 2.16 581

Findings:  Conclusions  reached  from  this  study  led  the  assessment  team  to  reach  the  following  conclusions:    

• Scores  improve  from  Comp  I  to  Comp  II  • Data  should  be  reviewed  by  group  to  inform  decisions  • Rubric  should  be  used,  and  people  should  be  trained  to  use  it  • Perhaps  upper  level  classes  should  use  it  to  reach  “4”  rating  

 Proposed  Future  Actions:  All  of  this  data  will  be  presented  to  the  Composition  Committee,  the  incoming  Composition  Director,  and  the  English  faculty  and  Composition  instructors,  to  help  shape  and  improve  the  teaching  of  composition  at  Arkansas  State  University.  

Institutional  Response  and  Implementation:  The  data  was  presented  to  the  Composition  English  faculty  and  Composition  instructors  at  the  first  Department  meeting.  The  data  was  also  recently  shared  with  the  new  Writing  Program  Director,  Dr.  Costello  (August  2013)  who  is  in  the  process  of  developing  practices  that  address  the  deficiencies  noted  in  the  assessment  (as  discussed  above:  common  rubric,  grade-­‐norming  among  instructors  and  in  the  classroom,  etc.)  and  an  assessment  project  that  will  assess  future  improvement  in  ASU’s  Composition  I  and  II.  

 

Composition  I  and  II  Fall  2012-­‐2013  Assessment  Summary The  Composition  Program  at  Arkansas  State  University  has  been  without  a  director  for  several  years,  so  while  some  artifacts  have  been  gathered,  these  have  not  been  analyzed  and  systematic  assessment  has  not  occurred  prior  to  Fall  2012.    In  Fall  2011  the  Department  of  English  &  Philosophy,  realizing  this  problem,  obtained  permission  from  the  then-­‐interim  provost  to  search  for  a  twelve-­‐month  director  of  composition,  with  fifty  percent  teaching  responsibilities  fall  and  spring  (two  courses  each  fall  and  two  each  spring)  and  fifty  percent  administrative  responsibilities  to  help  build  a  stronger  writing  program.  This  new  director  of  composition  has  been  tasked  with  bringing  greater  consistency  and  effectiveness  to  first-­‐year  writing  courses  and  supervising  graduate  students  who  staff  the  Learning  Center  Writing  Lab.