Machine Scoring Brief 4.5.16

8/18/2019 Machine Scoring Brief 4.5.16

1/8

April 5, 2016

Too Many Unanswered Questions about Machine Scoring of the Common

Core Exams

By Leonie Haimson, Executive Director, Class Size Matters and Co-chair, Parent Coalition for

Student Privacy

The Common Core PARCC and SBAC exams have begun in 26 saes a!ross he

!ounr"# A!!ording o Polii!o, $o hirds o% sudens& $rien responses in 'man"(

PARCC saes $ill be s!ored b" !ompuers, $ih onl" en per!en o% hose responses

hen !he!)ed b" hand# Tha means a suden has a lile more han *6 per!en

!han!e o% having heir essa"s read b" an a!ual human being# As $e shall see, onl"

50 per!en o% sudens $ho a)e he SBAC exam are going o have heir $riing read

and s!ored b" humans#

A%er a sharp drop+o, PARCC saes !urrenl" in!lude Colorado, -isri! o%

Columbia, .llinois, /ar"land, e$ erse", e$ /exi!o, and Rhode .sland# .n

3ouisiana and /assa!huses, he Bureau o% .ndian 4du!aion and -eparmen o%

-e%ense s!hools also are also pari!ipaing o var"ing exens#

hi!h poses he 7uesion, in $hi!h o% hese PARCC saes $ill he vas ma8ori" o%

43A ess be s!ored b" ma!hines9 And $here is he eviden!e ha auomaed

s!oring is eiher valid or reliable9 . and m" !olleagues in he Paren Coaliion %or

Suden Priva!" did a lile digging and $e %ound ou a lile more, bu no enough o

ans$er eiher o% hese !rii!al 7uesions#

e dis!overed he %ollo$ing passages %rom he PARCC :hio !onra! alhough :hio

has no$ pulled ou o% PARCC be!ause o% huge e!hnologi!al gli!hes and he

1

http://www.politico.com/tipsheets/morning-education/2016/03/more-relief-for-former-corinthian-students-213412http://www.parcconline.org/about/states/coloradohttp://www.parcconline.org/about/states/district-of-columbiahttp://www.parcconline.org/about/states/district-of-columbiahttp://www.parcconline.org/about/states/illinoishttp://www.parcconline.org/about/states/marylandhttp://www.parcconline.org/about/states/new-jerseyhttp://www.parcconline.org/about/states/new-mexicohttp://www.parcconline.org/about/states/rhode-islandhttp://www.parcconline.org/about/states/louisianahttp://www.parcconline.org/about/states/massachusettshttp://procure.ohio.gov/pdf/CSP903215_Redlndprpsl.pdfhttp://www.cleveland.com/metro/index.ssf/2015/06/ohio_dumps_the_parcc_common_core_tests_after_woeful_first_year.htmlhttp://www.parcconline.org/about/states/coloradohttp://www.parcconline.org/about/states/district-of-columbiahttp://www.parcconline.org/about/states/district-of-columbiahttp://www.parcconline.org/about/states/illinoishttp://www.parcconline.org/about/states/marylandhttp://www.parcconline.org/about/states/new-jerseyhttp://www.parcconline.org/about/states/new-mexicohttp://www.parcconline.org/about/states/rhode-islandhttp://www.parcconline.org/about/states/louisianahttp://www.parcconline.org/about/states/massachusettshttp://procure.ohio.gov/pdf/CSP903215_Redlndprpsl.pdfhttp://www.cleveland.com/metro/index.ssf/2015/06/ohio_dumps_the_parcc_common_core_tests_after_woeful_first_year.htmlhttp://www.politico.com/tipsheets/morning-education/2016/03/more-relief-for-former-corinthian-students-213412


2/8

Colorado !onra! issill in %or!e;

e also %ound his passage in he 4xe!uive Summar" $rien b" Pearson, he main

!onra!or o PARCC;

Handscorin costs ma!e u" nearly #$ "ercent of the costs of the P%&CC

assessments' (e are eaer to conduct the automated scorin e)cacy study

later this year, in coordination *ith E+S and P%&CC, *ith the "lan to "hase in

automated scorin einnin *ith the s"rin administration of the rst

o"erational year'

.n anoher se!ion o% he !onra!, his 'e

auomaed s!oring o be !arried ou b" Pearson menioned $as menioned, $ih

spe!i=! deadlines aa!hed;

./' %utomated Scorin0 P%&CC ac!no*ledes the "otential advantaes of

automated scorin 12%3 Scorin45 to "romote e)ciency associated *ith

scorin of student constructed res"onses, *hich other*ise re6uire human

scorin' +he automated scorin "hase-in "lan is incor"orated in the ase

contract, as detailed in +ale 7-/ aove and in the &edlined Pro"osal'

Ho*ever, Contractor shall conduct an e)cacy study, and P%&CC shall revie*

contractors Proof-of-conce"t study results, and "rovide Contractor *ith

authorization to utilize %3 Scorin as s"ecied herein' Contractor shall

"rovide the follo*in deliverales, as s"ecied in Section 8'B'. of the

&edlined Pro"osal0

a' Proof-of concent research desin

2

https://www.cde.state.co.us/cdereval/pearsonamend3https://www.cde.state.co.us/cdereval/pearsonamend3


3/8

i' &e"ort of results of "roof-of-conce"t study, *hich shall e "rovided

"ursuant to the follo*in 9ey Milestones and re6uirements0 Proof-of-

conce"t research desin a""roved 1to e mutually determined "rior to

aseline of "ro:ect schedule'

ii' Contractor *ill "rovide "roof-of-conce"t;e)cacy study re"ort to

P%&< /$;/#;/=iii' P%&CC "rovides nal a""roval to "roceeded >sic? ith automated

scorin "hase-in "lan ased on results of e)cacy study0 /$;./;/=

iv' %ny states electin to o"t out of "hase-in "lan and use human

scorin for EL% online res"onses, notied Contractor no later than

//;/=;/=< sined areement must e in "lace y /@;./;/=

v' %ny modication of the 9ey Milestone due dates, or chane to the

"hase in "lan, shall re6uire a Sco"e Chane "ursuant to the terms of

the Contract'

So $here is his proo%+o%+!on!ep>e

A!!ording o a Polii!o sor" %rom ovember 201?, hough Pearson $as supposed

o deliver he sud" b" mid+:!ober 201?, he !ompan" missed ha deadline#

As des!ribed above, he deliver" o% he sud" $as supposed o rigger he =nal

approval o% he PARCC !onsorium o go ahead $ih auomaed s!oring b" :!ober

*1, 201?# .% he deadline $as missed his $as supposed o re7uire a 'S!ope

Change( o he erms#

ha happened $hen Pearson didn& deliver he !onra!9 A!!ording o Polii!o,

+here have een no sco"e chanes to this "ortion of the contract, accordin

to Larry Behrens, a s"o!esman for the Ae* Mexico Pulic Education

De"artment, *hich oversees the contract' So *heres the Pearson study

P%&CC s"o!esman David Connerty-Marin told Mornin Education its ein

revised ut he declined to say *ho had as!ed for the revisions or *hat

they entail' Pearson *ouldnt ans*er any 6uestions on the su:ect, referrin

them all to P%&CC'

And what of the provision that PARCC provide “nal approval” ofthe "hase-in of automated radin Connerty-Marin *ouldnt ans*er6uestions aout *hether a vote has already ta!en "lace or *ill e held in thefuture' He at rst told Mornin Education that P%&CC states are currently2conductin studies4 on the e)cacy of usin com"uter alorithms' But helater ac!no*leded that states arent doin their o*n studies< theyre relyinon the Pearson re"ort' Mornin Education contacted all the states usinP%&CC tests to as! if they had made a decision on automated radin' nlyColorado and D'C' re"lied< oth said no decision had een made'

*

http://www.politico.com/tipsheets/morning-education/2014/11/robo-grading-the-common-core-recession-hinders-college-completion-vergara-of-the-midwest-girls-shun-nonfiction-212543http://www.politico.com/tipsheets/morning-education/2014/11/robo-grading-the-common-core-recession-hinders-college-completion-vergara-of-the-midwest-girls-shun-nonfiction-212543


4/8

There is an alernaive lised in he PARCC !onra!s, !alled he '@uman S!oring:pion( $hi!h $ould !os an addiional *#50 per suden his "ear, rising o morehan 5 per suden in "ears hree and %our nearl" a 10 in!rease in pri!e '

:n /ar!h 2, 2015, PARCC pu ou a bullein en!ouraging saes o pari!ipae in

heir spring =eld esing 'to otain item level data to su""ort test construction for

@$/#F/G and ather res"onses to train the automated 9+ >a""arently Pearsons

9no*lede %nalysis +echnoloies 1 KAT 5 ? scorin enine# '

So 8us a lile more han a "ear ago, Pearson $as sill 'raining( is s"sem hrough

he PARCC =eld esing program $hi!h does no lend !on=den!e o is proven

e

So $as his 'proo% o% !on!ep( or e

he Polii!o ari!le $as published in ovember 201?9 And i% so, $ha did i sa"9 :n

he PARCC $ebsie, here is a lis o% repors and sudies ha $ere !ompleed,

in!luding his one;

Automated Scoring Proof of Concept Study +he "ur"ose of the study

*as to evaluate *hether machine scorin can e used in scorin of Prose

Constructed &es"onse 1PC&5 in P%&CC EL%;Literacy assessments'

e =lled ou an online %orm o as) %or he sud", bu have no "e re!eived a

response#

An re!en pie!e b" Pearson on Ari=!ial .nelligen!e or A. on his issue !on!ludes,

'3f *e are ultimately successful, %3Ed *ill also contriute a "ro"ortionate res"onse

to the most sinicant social challene that %3 has already rouht F the steady

re"lacement of :os and occu"ations *ith clever alorithms and roots'4

ha abou he s!oring mehod used b" he Smarer Balan!ed exam, $hi!h is beinggiven in 1D saes his "ear, in!ludingCali%ornia, Conne!i!u, -ela$are, @a$aii, .daho, .o$a, /onana, evada, e$@ampshire, orh -a)oa, :regon, Souh -a)oa, Eermon, ashingon, esEirginia and "oming, plus he FS Eirgin .slands, as $ell as o some exen in/i!higan#

.n an ari!le daed /ar!h 15, 201*, Smarer Balan!ed repored ha he !onsoriumhad rereaed %rom is original plans o s$i!h enirel" o auomaed s!oring;

'Smarer Balan!ed has actually already scaled ac! its "lans for radin*ritin *ith machines ecause articial intellience technoloy has notdevelo"ed as 6uic!ly as it had once ho"ed' 3n @$/$, *hen it *as startin todevelo" the ne* Common Core exams for its @= memer states, the rou"*anted to use machines to rade /$$ "ercent of the *ritin'

?

http://parcc.pearson.com/bulletins/2015/03/02/parcc-administrative-bulletin.htmlhttp://www.parcconline.org/assessments/test-design/researchhttps://www.pearson.com/content/dam/corporate/global/pearson-dot-com/files/innovation/Intelligence-Unleashed-Publication.pdfhttp://blogs.edweek.org/edweek/curriculum/2016/03/Reach_of_PARCC_Smarter_Balanced_drops_sharply_in_2015-16.html?intc=main-mpsmvshttps://www.insidehighered.com/news/2013/03/15/professors-odds-machine-graded-essayshttp://parcc.pearson.com/bulletins/2015/03/02/parcc-administrative-bulletin.htmlhttp://www.parcconline.org/assessments/test-design/researchhttps://www.pearson.com/content/dam/corporate/global/pearson-dot-com/files/innovation/Intelligence-Unleashed-Publication.pdfhttp://blogs.edweek.org/edweek/curriculum/2016/03/Reach_of_PARCC_Smarter_Balanced_drops_sharply_in_2015-16.html?intc=main-mpsmvshttps://www.insidehighered.com/news/2013/03/15/professors-odds-machine-graded-essays


5/8

2ur initial estimates *ere assumin *e could do everythin y machine, ut *eve chaned that,4 said Iac6ueline 9in, a director at Smarter Balanced'+he technoloy hasnt moved ahead as fast as *e thouht,4 9in said'

Ge here is an ex!erp %rom he Conne!i!u agreemen $ih A.R, he primar"!onra!or o% he Smarer Balan!ed exam;

AHR44/4T B4T44 T@4 C:4CT.CFT STAT4 B:AR- :I 4-FCAT.: A- T@4 A/4R.CA .ST.TFT4S

I:R R4S4ARC@

i# -uring Gear 1, A.R shall hand+s!ore 100 per!en o% he responses and dohuman se!ond double+blind s!orings 15 per!en o% he ime# A.R shall useAri=!ial .nelligen!e A. o !ondu! a se!ond s!ore o% he remaining J5per!en o% he responses#

ii# .n Gear 2, A.R shall use A. o s!ore 100 per!en o% he responses and shallhand+s!ore 50 per!en on a se!ond s!oring#

iii# Ior Gear * and be"ond, A.R shall use A. o s!ore 100 per!en o% he

responses shall hand+s!ore25 Ksi!L per!en on a se!ond s!oring#

So given ha i is no$ "ear $o, onl" hal% o% he sudens in Conne!i!u $ill have

he !han!e %or a human o read heir responses#

This A.R agreemen also re%eren!es a validi" sud" ha $as supposed o have been

provided o saes !on!erning he P4H A. KAri=!ial .nelligen!eL s!oring s"sem o

be used b" SBAC; 'a re"ort containin multi"le measures of human;%3 areement,

includin0 "ercent "erfect areement, "ercent ad:acent, "ercent "erfect J ad:acent,

and 6uadratic *eihted !a""a(#

A!!ording o expers, he purpose o% he se!ond human s!oring is o !on=rm hea!!ura!" o% ma!hine s!oring# ihou ma)ing publi! he degree o $hi!h boh are!orrelaed, here is no eviden!e ha he ma!hines are able o !ome !lose orepli!aing human s!ores#

The P4H A. s!oring s"sem else$here is des!ribed as 'an auomaed s!oring

e!hnolog" M pur!hased b" /easuremen .n!orporaed in 2002#( :n anoher P4H

sie, i sa"s; ' %lthouh factors li!e creativity are eyond the sco"e of com"uterized

assessment, these "rorams can still e used in the classroom to "rovide timely,

uniased feedac! that allo*s students to im"rove their *ritin s!ills and

"rociency #(

hi!h brings up he obvious poin $asn& he Common Core supposed o

en!ourage !reaivi" and !rii!al hin)ing9 And he Common Core aligned exams

supposed o assess hese s)ills9 .s here an" eviden!e ha ma!hines !an do

eiher9 As %ar as one !an deermine, he ans$er is no#

3as "ear, 3es Perelman, $ho $as in !harge o% /.T&s riing program, $roe an

opinion pie!e %or he Boson Hlobe# Perelman esed ou anoher auomaed s!oring

5

http://www.studentprivacymatters.org/wp-content/uploads/2016/04/Air-CT-Agreement.pdfhttp://www.pegwriting.com/abouthttp://www.pegwriting.com/blog/de-myth-ifying-ai-scoring-myth-3http://www.pegwriting.com/blog/de-myth-ifying-ai-scoring-myth-3http://www.bostonglobe.com/opinion/2014/04/30/standardized-test-robo-graders-flunk/xYxc4fJPzDr42wlK6HETpO/story.htmlhttp://www.studentprivacymatters.org/wp-content/uploads/2016/04/Air-CT-Agreement.pdfhttp://www.pegwriting.com/abouthttp://www.pegwriting.com/blog/de-myth-ifying-ai-scoring-myth-3http://www.pegwriting.com/blog/de-myth-ifying-ai-scoring-myth-3http://www.bostonglobe.com/opinion/2014/04/30/standardized-test-robo-graders-flunk/xYxc4fJPzDr42wlK6HETpO/story.html


6/8

s"sem, .nelli/eri!, ha !ould no disinguish essa"s $ih meaning%ul !oheren

prose %rom nonsense, and ha high mar)s o gibberish, su!h as his;

2%ccordin to "rofessor of theory of !no*lede Leon +rots!y, "rivacy is the

most fundamental re"ort of human!ind' &adiation on advocates to an orator

transmits amma rays of "arsimony to im"lode'

Fnable o anal"Ne meaning, narraive, or argumen, auomaed s!oring insead

relies on lengh, grammar, and measures o% absruse vo!abular" o do assess prose#

Perelman had as)ed o es he Pearson KAT s"sem being used b" PARCC, bu $as

denied a!!ess o heir robo+grader# @e !on!luded;

3f P%&CC does not insist that Pearson allo* researchers access to its roo-

rader and release all ra* numerical data on the scorin, then Massachusetts

should *ithdra* from the consortium' Ao "harmaceutical com"any is allo*ed

to conduct medical tests in secret or deny leitimate investiators access'

+he KD% and inde"endent investiators are al*ays involved' 3ndeed, even

toasters have more oversiht than hih sta!es educational tests

A paper daed /ar!h 201* %rom he 4du!aional Tesing Servi!e one o% he SBAC

sub+!onra!ors !on!luded;

Current automated essay-scorin systems cannot directly assess some of themore conitively demandin as"ects of *ritin "rociency, such as audiencea*areness, arumentation, critical thin!in, and creativity' % related*ea!ness of automated scorin is that these systems could "otentially e

mani"ulated y test ta!ers see!in an unfair advantae' Examinees may, forexam"le, use com"licated *ords, use formulaic ut loically incoherentlanuae, or articially increase the lenth of the essay to try and im"rovetheir scores'

A!!ording o a re!en ari!le, Ilorida plans o use A.R&s AuoS!ore o grade essa"s

on is sae$ide examsO and Fah has reporedl" used auomaed s!oring sin!e

2010# As o% /ar!h 2015, e$ erse" had no "e made up is mind $heher o adop

auomaed s!oring %or he PARCC exams o be given his spring#

3as "ear, he -eparmen o% 4du!aor dire!or o% assessmens e @auger $as

7uoed as sa"ing he sae had no de!ided ho$ 7ui!)l" o adop !ompuer s!oring;

2+he state *ill consider the o"tion if the automated scorin "roves to e

accurate and cost eective' But the state understands the "erce"tion that

automated scorin may not e as eective as ein raded y hand(e

*ould not o full automated scorin *ithout havin some information for us

to elieve that actually it does :ust as ood of a :o as human scores'4

6

https://www.ets.org/Media/Research/pdf/RD_Connections_21.pdfhttp://stateimpact.npr.org/florida/2014/03/25/why-computer-scored-essays-could-eliminate-the-need-for-writing-tests/http://www.nj.com/education/2015/03/who_is_grading_the_parcc_tests.htmlhttps://www.ets.org/Media/Research/pdf/RD_Connections_21.pdfhttp://stateimpact.npr.org/florida/2014/03/25/why-computer-scored-essays-could-eliminate-the-need-for-writing-tests/http://www.nj.com/education/2015/03/who_is_grading_the_parcc_tests.html


7/8

Hiven all he unresolved issues, oda" paren leaders and advo!aes %rom man"

saes, in!luding Paren Coaliion %or Suden Priva!", Parens A!ross Ameri!a,

IairTes and e$or) %or Publi! 4du!aion, sen a leer o heir Sae 4du!aion

Chie%s, demanding ans$ers o he %ollo$ing 7uesions;

1+ ha per!enage o% he 43A exams in our sae are being s!ored b" ma!hineshis "ear, and ho$ man" o% hese exams $ill hen be re+s!ored b" a human

being9

2+ ha happens i% he ma!hine s!ore varies signi=!anl" %rom he s!ore given

b" he human being9

*+ ill parens have he opporuni" o learn $heher heir !hildren&s 43A exam

$as s!ored b" a human being or a ma!hine9

?+ ill "ou provide he 'proo% o% !on!ep( or e

in he !onra!s as aesing o he validi" and reliabili" o% he ma!hine+

s!oring mehod being used9

5+ ill "ou provide an" independen resear!h ha provides eviden!e o% he

reliabili" o% his mehod, and pre%erabl" sudies published in peer+revie$ed

8ournals9

e en!ourage parens o send heir o$n leers o heir Sae 4du!aion !hie%s, as

$ell as sae and lo!al s!hool o

com"uterized scorin of student essays'4 3t also cites research ndins and

iliora"hy on the issue Heres a scholarly criti6ue y Les Perelman of the

claims made y "ro"onents of machine scorin, *ho cite a @$/@ com"etition

s"onsored y the He*lett Koundation' Heres a AN +imes column aout the

controversy'

D

http://www.studentprivacymatters.org/our-letter-to-the-education-commissioners-in-the-parcc-and-sbac-states/http://nycpublicschoolparents.blogspot.com/2014/01/what-privacy-protections-are-there-when.htmlhttp://nycpublicschoolparents.blogspot.com/2015/05/continuing-mystery-where-is-smarter.htmlhttp://http/humanreaders.org/petition/index.phphttp://humanreaders.org/petition/results.phphttp://humanreaders.org/petition/research_findings.htmhttp://humanreaders.org/petition/works_cited.htmhttp://www.journalofwritingassessment.org/article.php?article=69http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html?_r=0http://www.studentprivacymatters.org/our-letter-to-the-education-commissioners-in-the-parcc-and-sbac-states/http://nycpublicschoolparents.blogspot.com/2014/01/what-privacy-protections-are-there-when.htmlhttp://nycpublicschoolparents.blogspot.com/2015/05/continuing-mystery-where-is-smarter.htmlhttp://http/humanreaders.org/petition/index.phphttp://humanreaders.org/petition/results.phphttp://humanreaders.org/petition/research_findings.htmhttp://humanreaders.org/petition/works_cited.htmhttp://www.journalofwritingassessment.org/article.php?article=69http://www.nytimes.com/2012/04/23/education/robo-readers-used-to-grade-test-essays.html?_r=0


8/8

Other readings:

Benne, Rand" 4llio# The !hanging naure o% edu!aional assessmen# &evie* of

&esearch in Education *Q#1 2015; *D0+?0D#

Perelman, 3# 'Consru! Ealidi", 3engh, S!ore, And Time .n @olisi!all" Hradedriing Assessmens; The Case Agains Auomaed 4ssa" S!oring#( 3nternational

%dvances in (ritin &esearch0 Cultures, Places, Measures' 4d# Charles BaNerman,Chris -ean, essi!a 4arl", aren 3uns%ord, SuNie ull, Paul Rogers, and AmandaSansell# Colorado; The AC Clearinghouse and Parlor Press, 2012# 121+1*2#

Perelman, 3# '/ass+/ar)e riing Assessmens as Bullshi#( (ritin %ssessment inthe @/st Century0 Essays in Honor of Ed*ard M' (hite# 4d# orber 4llio and 3esPerelman# @ampon Press, 2012# ?25+?*D#Perelman, 3# hen he sae o% he ar& is !ouning $ords#( %ssessin (ritin 21

201?; 10?+111#

J
https://www.researchgate.net/profile/Randy_Bennett/publication/273064367_Randy_Elliot_Bennett_The_Changing_Nature_of_Educational_Assessment_Review_of_Research_in_Education_March_2015_39_370-407_doi10.31020091732X14554179/links/552ac7710cf2e089a3aa100d.pdfhttp://wac.colostate.edu/books/wrab2011/chapter7.pdfhttp://wac.colostate.edu/books/wrab2011/chapter7.pdfhttp://wac.colostate.edu/books/wrab2011/#contacthttp://wac.colostate.edu/books/wrab2011/#contacthttp://wac.colostate.edu/books/wrab2011/#contacthttp://lesperelman.com/wp-content/uploads/2015/09/perelman_bullshit.pdfhttps://www.researchgate.net/publication/263093530_When_the_state_of_the_art_is_counting_wordshttps://www.researchgate.net/profile/Randy_Bennett/publication/273064367_Randy_Elliot_Bennett_The_Changing_Nature_of_Educational_Assessment_Review_of_Research_in_Education_March_2015_39_370-407_doi10.31020091732X14554179/links/552ac7710cf2e089a3aa100d.pdfhttp://wac.colostate.edu/books/wrab2011/chapter7.pdfhttp://wac.colostate.edu/books/wrab2011/chapter7.pdfhttp://wac.colostate.edu/books/wrab2011/#contacthttp://wac.colostate.edu/books/wrab2011/#contacthttp://wac.colostate.edu/books/wrab2011/#contacthttp://lesperelman.com/wp-content/uploads/2015/09/perelman_bullshit.pdfhttps://www.researchgate.net/publication/263093530_When_the_state_of_the_art_is_counting_words

Documents

Machine Scoring Brief 4.5.16