View
225
Download
0
Category
Preview:
Citation preview
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
1/76
With Support From the William and Flora Hewlett Foundation
TECHNOLOGY COUNTS 2003
Pencils Down:Technologys Answer
To Testing
American Educations Newspaper of Record Volume XXII Number 35 May 8, 2003 2003 Editorial Projects in Education / $6.00
EDUCATIONWEEK
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
2/76
Techs AnswerTo TestingSchool districts and states are turning to computer-
based assessment for quicker results and to meet
new federal accountability requirements.
Legal Twists,Digital TurnsThe No Child Left Behind Act of 2001 is having the
unusual effect of both encouraging and discouraging
schools use of computerized testing.
12 State Initiatives
15 A Test-Takers Perspective
A Question
Of DirectionComputer adaptive testing, which adjusts questionslevel of difficulty based on how well a student
answers them, is the source of much debate.
Prepping for theBig TestStudents turn to test-preparation Web sites
to help them prepare for high-stakes state
assessments and college-entrance exams.
Spec. Ed. TechSparks IdeasTesting tools for children with disabilities, such as
digital video cameras and computer text readers,
are attracting mainstream attention.
The TeachersNew TestThe use of computerized quizzes and exams for
classroom testing is on the rise as teachers look
for more efficient ways to evaluate students.
MarketingTo the TestTraditional test publishers and start-up businesses
have developed a host of technology-based testing
programs in the hope of cornering this new market.
Essay GradingGoes DigitalSoftware that evaluates student essays has its
share of critics, but some teachers say it has made
it easier to add more writing exercises.
Tracking TechTrendsStudent access to the Internet continues to improve,
but schools are spending only a small percentage
of technology dollars on professional development.
50 Analyzing the Tech Effect
State DataTables
54 Access to Technology
58 Capacity to Use Technology
60 Use of Technology
92 Sources and Notes
State of theStatesSnapshots of what each state is doing in the area
of computer-based testing, and how the states
are using technology to improve schools.
Index toAdvertisers
Contents8
11 39
43
54
62
94
17
23
27
31
35
COVER BY LAURA BAKER AND ALLISON SHELLEY/PHOTO ILLUSTRATIONS BY ALLISON SHELLEY
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
3/76
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
TECHNOLOGY COUNTS 2003
Pencils Down:Technologys
Answer to Testing
EDITOR & PUBLISHER
Virginia B. Edwards
MANAGING EDITOR
Gregory Chronister
PROJECT EDITORKevin Bushweller
PROJECT DIRECTOR
Ronald A. Skinner
EDITORIAL PRODUCTION CHIEF
Darcia Harris Bowman
RESEARCH ASSOCIATES
Susan E. Ansell
Melissa McCabe
Jennifer Park
Lisa N. Staresina
CONTRIBUTING WRITERS
Rhea R. Borja
Michelle Galley
Lisa Fine Goldstein
Kathleen Kennedy Manzo
Lynn Olson
Andrew Trotter
Mark Walsh
DESIGN DIRECTOR
Laura Baker
ASSOCIATE DESIGN D IRECTOR
Gina Tomko
DESIGN ASSOCIATE
Alyson Salon
PHOTO EDITOR
Allison Shelley
Technology Counts 2003was produced with support
from the William and Flora
Hewlett Foundation.
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
4/76
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
EDITORIAL & BUSINESS OFFICES:
6935 Arlington Road, Suite 100, Bethesda, MD 20814-5233
(301) 280-3100 E-mail address: ew@epe.org
FAX: Editorial (301) 280-3200 Business (301) 280-3250
EDUCATION WEEK is available on the World Wide Web atwww.edweek.org.
For New Subscriptions, Subscription Problems,
and Changes of Address:
EDUCATION WEEK, Post Office Box 2083, Marion, OH 43305
1-800-728-2790 E-mail address: jschwar@epe.org
EDITOR & PUBLISHER Virginia B.Edwards
MANAGING EDITOR Gregory Chronister
DEPUTY MANAGING EDITOR Karen Diegmueller
SENIOR EDITORS Caroline Hendrie, Lynn Olson (Special Projects),M. Sandra Reeves (Commentary)
ASSISTANT MANAGING EDITORS
Ann Bradley, Kevin Bushweller, Robert C.Johnston, Ben Wear
CONTRIBUTING EDITORS Mary-Ellen Phelps Deil y, Sandra Graziano, Scott W.Wright
ASSOCIATE EDITORS Jeff Archer, Kathleen Kennedy Manzo, Debra Viadero, M ark Walsh
ASSISTANT EDITORS Catherine Gewertz, David J. Hoff, Linda Jacobson,Bess Keller, Mary Ann Zehr
ASSISTANT COMMENTARY EDITOR Amy Conrad
STAFF WRITERS Julie Blair, Rhea R. Borja, Darcia Harris Bowman, Sean Cavanagh,Michelle R. Davis, Michelle Galley, John Gehring, Catherine Gewertz, Lisa Fine Goldstein,
Karla Scoon Reid, Alan Richard, Erik W.Robelen, Joetta L.S ack, Andrew Trotter
EDITORIALASSISTANT Marianne D.Hurst
EDITORIALINTERN Hattie Brown
DIRECTOR OF SPECIAL RESEARCH PROJECTS Kathryn M.Doherty
RESEARCH DIRECTOR Ronald A. Skinner
RESEARCH ASSOCIATES Susan E.Ansell, Melissa McCabe, Jennifer Park, Lisa N. Staresina
CONTRIBUTING WRITERS Rich Shea, Samantha Stainburn, Mark Toner
DESIGN DIRECTOR Laura Baker
ASSISTANT DESIGN DIRECTOR Gina Tomko
DESIGN ASSOCIATE Alyson Salon
PHOTO EDITOR Allison Shelley
ASSISTANT PHOTO EDITOR James W.Prichard
SYSTEMS MANAGER Hunter McCleary
ASSISTANT SYSTEMS MANAGER Bill Moyer
DIRECTOR OF NEW MEDIA Jeanne McCann
SENIOR ONLINE EDITOR Anthony Rebora
ONLINE EDITOR Rachell Deutsch
ASSISTANT ONLINE EDITOR Craig Stone
ONLINE INTERNS Patrick Byrnett, Alex Kobulsky, Michele Owens
LIBRARY DIRECTOR Kathryn Dorko
ASSISTANT LIBRARIAN Claire Guimbert
GENERAL MANAGER Michele J.Givens
CHIEF FINANCIAL OFFICER Eric W.Arnold
ACCOUNTING MANAGER Janice Brunson
GENERAL LEDGER ACCOUNTANT Kiki Aboaba
ACCOUNTS RECEIVABLE Barbara M.Guinn
ACCOUNTS PAYABLE Sage Wicinski
ONLINE BUSINESS MANAGER Angela Lunter
MARKETING DIRECTOR Karen Creedon Dunn
MARKETING ASSOCIATE Cheryl Staab
CIRCULATION DIRECTOR Patrick Haller
ASSOCIATE CIRCULATION MANAGER Christopher Klipple
CIRCULATION ASSOCIATE Alicia Osgood
FULFILLMENT MANAGER Laurie Breathnach
CUSTOMER SERVICE REPRESENTATIVE Joel I.Schwartz
PRODUCTION MANAGER Melayne Cohen (301) 280-3176, production@epe.org
PRODUCTION ASSISTANT Armando Calonje Jr.
ASSOCIATE PUBLISHER Sam Bonfante, Advertising Sales(301) 280-3100
SENIOR ADVERTISING SALES EXECUTIVE Irving Carpenter
REGIONAL ADVERTISING MANAGERS Julie Fagan, Norlie Lin
MIDWEST ADVERTISING MANAGER Brian McGoldrick
WESTERN ADVERTISING MANAGER Sharon Makowka
ADVERTISING SALES ASSOCIATE Jennifer S. Haley
ADVERTISING ASSISTANT Carolina Calonje
CLASSIFIED MANAGER Titilayo Ellis
CLASSIFIED ACCOUNT REPRESENTATIVE Sean McLaughlin
CLASSIFIED ADVERTISING ASSISTANT Rose Wilson
EXECUTIVE ASSISTANT Shaiy E.Knowles
RECEPTIONIST Jackie Briceo
ADMINISTRATIVE ASSISTANTS Catherine Carroll, Maria Shaffer
EDUCATION WEEK (ISSN 0277-4232) is published 43 times per year (weekly on Wednesdays)
except for December 18, and December 25, 2002, and January 1, June 25, July 2, July 16, July23, July 30, August 13, August 20, and August 27, 2003, and issues will also be publishedThursday, January 9, 2003, and Thursday, May 8, 2003, by Editorial Projects in Education Inc.Editorial and advertising-sales offices are located at 6935 Arlington Road, Suite 100, Bethesda,MD 20814-5233;(301) 280-3100.Subscriptions: U.S.:$79.94 for one year (43 issues).Canada:$135.94, Foreign $208.94.Orders must be paid in U.S.dollars. Periodicals postage paid atBethesda, MD, and at additional mailing offices.POSTMASTER: Send address changes toEDUCATION WEEK, PO Box 2083, Marion, OH 43306.Volume XXII, Issue 35 . (USPS#599-870)
Back issues are available at $3.50 each, except the May 9, 2002 (Technology Counts) issue at $6.00each, and the January 10, 2002 (Quality Counts) issue at $10.00 each.Price includes first-classpostage and handling. Send order to:B ack Issues, EDUCATION WEEK, 6935 Arlington Road,Suite 100, Bethesda, MD 20814-5233.
Copies of the newspaper on microfilm can be ordered from Bell & Howell Information & Learning,Periodicals, PO Box 1346, Ann Arbor, MI., 48106-1346 .
EDUCATIONWEEK
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
5/76
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
Education and technology forces have converged
this year to vault computer-based testing into
the headlines, raising important questions
about whether this new mode of assessment is more
useful than traditional paper-and-pencil exams.
To begin with, the increased testing requirements
imposed by the No Child Left Behind Act of 2001a
far-reaching overhaul of federal education policy signed
into law by President Bush in January 2002have set
schools scrambling to find more efficient ways to
assess academic skills and get children ready for high-
stakes state exams. Unlike traditional standardized
tests on paper, which can take weeks or even months to
score and return to schools, computer-based assess-
ments can provide almost immediate feedback. That is
arguably one of the biggest draws for educators.
Already, 12 states and the District of Columbia have
a computerized exam or a pilot project under way to eval-
uate the effectiveness of computer-based testing, accord-
ing to a new Education Week survey of state depart-
ments of education.All of these testsexcept for one in
North Carolina and the District of Columbia examare
administered via the Internet. In five states, officials re-
port that computerized testing was designed to partially
meet requirements under the new federal law.
Eventually, experts predict, technology could change
the face of testing itself, enabling states to mesh the use
of tests for instructional and accountability purposes.
Youve got the potential that technology could be a
solution, says Wesley D. Bruce, the director of school
assessment for the Indiana Department of Education,
but there is, right now, just a huge set of issues.
Chief among them is a simple question: Do schools
have enough computers to test children in this new
manner? The answer in many places is no. And with
most states struggling with budget deficits, its unlikely
they are going to use their limited resources to fill
that void.
Yet researchers point out that computer-based test-
ing has the potential to be far cheaper than its printed
counterpart.
Richard Swartz, a senior research director at the Edu-
cational Testing Service, in Princeton, N.J., estimates that
the actual costs of putting a test online and building a cus-
tomized scoring model are comparable to those of develop-
ing a good paper-and-pencil exam.Once the tests are im-
plemented, he adds, the difference in scoring costs is
enormously in favor of the computer.
Still, other problems with computerized assessment
have emerged.
Techs AnswerTo TestingSchools turn to computerized
exams to meet new demands
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
6/76
One prickly issue involves the use of what is calledadaptive testing, in which the computer adjusts thelevel of difficulty of questions based on how well a stu-dent is answering them. Proponents of this form oftesting argue that it provides a more individualized
and accurate assessment of a students ability.But the No Child Left Behind law, a revision of the
Elementary and Secondary Education Act that puts ahigher premium than ever on schools accountabilityfor student achievement, continues to mandate thatstates measure student performance against the ex-
pectations for a students grade level.With adaptive testing, a 7th grader, for instance,
might be bumped up to questions at the 8th gradelevelor dropped down to the 6th grade level. As aconsequence, debate is growing about whether adap-tive testing can meet the purposes of the federal law,and if it doesnt, how the technology should be modi-fied to meet the requirements.
To give educators a head start on understanding
computer-based testing, Technology Counts 2003thesixth edition ofEducation Weeks annual report on ed-ucational technology in the 50 states and the Districtof Columbiaexamines these new developments
from a host of angles, beginning with an analysis ofthe impact of the No Child Left Behind law. Surpris-ingly, perhaps, the story points out that the law ishaving the effect of both encouraging and discourag-ing the use of computerized assessments.
As another part of this years focus on computer-based testing, Technology Counts 2003 takes a closelook at adaptive testing, with analysis from propo-nents and critics, and a description of how it works.The upshot of the adaptive-testing debate is that de-velopers of such assessments are worried that theymay be left out of what could be the greatest precol-legiate testing boom in history.
Computerized assessment may turn out to have itsbiggest impact in the area of online test preparation,observers of the field say. Last year, for instance, morethan 200,000 students in 60 countries signed up forthe Princeton Reviews online demonstrations of suchtests as the SAT and state exit exams. TechnologyCounts 2003 tracks the online test prep trend.As educators face the new federal requirement to
test all 3rd through 8th graders annually in readingand mathematics, states are experimenting with new
ways of using technology to evaluate the abilities ofspecial education students. Testing experts say thatwhat educators learn from how to tailor assessmentsto the needs of special education students could alsoshape how they test other students, who may havemore subtle individual needs. This years report ex-amines those lessons.
Technology Counts 2003 also includes a story aboutteachers who are using computer-based testing togive classroom quizzes and tests, an examination ofthe benefits and drawbacks of essay-grading soft-ware, an analysis of the growing business of com-puter-based testing, and a look at national trends ineducational technology.
Snapshots of the steps each state has taken to usecomputer-based testingor simply to use educationaltechnology more effectivelyare also included in thereport, as are data tables with state-by-state statis-
tics on technology use in schools.We hope youll find information here that will help
you understand computer-based testing and its evolv-ing role in education. THE EDITORS
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
10
Techs
Answer
to
Testing
"There is ...justa huge set of
issues" with thisnew way of testing.
WESLEY D. BRUCE, Director of School Assessment, Indiana Department of Education
COMPUTER-BASED STATE TESTS
Twelve states and the District of Columbia are administering computer-based state exams in 2002-03.
Six of these tests are pilots. Georgias online testing was temporarily suspended for spring 2003. Florida
and Oklahoma are planning pilots for 2003-04.
SOURCE: Education Weeksurvey of state departments of education, 2003
State has a computer-based state test (13)
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
7/76
Educat ion Week ' s Techno logy Counts/May 8,20 03
Computerized testing
feels the impact of
No Child Left Behind
ByBy Lynn Olson
South Dakota officials announced this past
winter that they were making the states
much-touted online testing program
voluntary for districts, and instead
requiring new paper-and-pencil
tests to meet the require-
ments of the No Child
Left BehindAct of 2001.
Meanwhile, officials in Idaho are
forging ahead with a statewide online
testing system, which they have modified
to meet the requirements of the federal law.
The contradictory moves illustrate the push
and pull that the law, a wide-ranging revision of
the Elementary and Secondary Education Act,
could have on the growth of technology-based
assessments in K-12 schools. While many
experts predict the law could lead to a
moderate slowing-down of computer-
based testing at the state level, they
anticipate a boom in low-stakes, tech-
nology-based assessments at the dis-
trict level to help students prepare for the
state exams.
I think No Child Left Behind poses an in-
teresting situation, because it could be both an
impediment and an impetus to the use of technology-
Legal Twists,
DigitalTurns
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
8/76
based tests, says Randy Bennett, an expert on tech-nology and assessment at the Educational TestingService in Princeton, N.J.
Technology can affect virtually every aspect ofassessment, from test design and administration to
the scoring and reporting of results. Advocates arguethat technology could make it easier for states anddistricts to meet some of the federal laws require-ments, by providing for cheaper and more efficienttest delivery, a quicker turnaround of scores, and theability to analyze and report test data in new ways.
Technological innovations also could make statetests more accessible to special populations of stu-dents, such as those with disabilities or limited flu-ency in English, who must be included in state test-ing systems under the federal legislation.
Eventually, experts predict, technology could changethe face of testing itself, enabling states to mesh the useof tests for instructional and accountability purposes.
Youve got the potential that technology could be asolution, says Wesley D. Bruce, the director of schoolassessment for the Indiana Department of Education,but there is, right now, just a huge set of issues.
An Immediate Concern
One of the most immediate issues for states thathave not yet ventured into computer-based testing iscost, particularly given ballooning state deficits.
Many states dont believe that theyre going to beable to meet the federal government mandate for as-sessments under No Child Left Behind with thefunds that the act is providing, Bennett of the ETSsays. So, given that, to then go out and try to buildan assessment system thats going to require evenmore upfront investment is just not going to be veryattractive to anyone.
In addition,Bennett notes, the federal laws aggressivetimelinesstates must administer reading and mathtests in grades 3-8 and at least once in high school by
2005-06work against the multiyear ramp-up re-quired to do high-stakes testing on a computer.
For that reason, I think its going to slow thingsdown from the point of technology-based testing, hesays, at least on the accountability end.
Not everyone shares that view.Scott Elliot, the chief operating officer for Vantage
Learninga for-profit company based in Yardley, Pa.,that provides online testing and scoring services for bothprecollegiate and higher educationsays his companyexpects to give some 17 million tests online this year.Although its not atypical for a state to spend $8 to
$10 per student to administer a paper-and-penciltest, he estimates the same test could be given on acomputer for $5 to $6 a student. And while the cost ofhuman scorers for essay and open-ended questionskeeps rising, the cost of automated scoring for suchquestions will probably stay the same or decrease.
While Elliot initially was concerned that the weakeconomy would deter states from venturing into tech-nology-based assessment, he now argues otherwise. Ijust think its going to break wide open in the nextfew months with state interest, he asserts. Demand,
he reasons, will be driven largely by the need to pro-duce test results faster, cheaper, and more efficiently.
Im getting more interest than ever before, Elliotsays, and I think it makes sense.
Richard Swartz, a senior research director at theETS, estimates that the actual costs of putting a testonline and building a customized scoring model arecomparable to those of developing a good paper-and-pencil exam.
But, he adds, once the tests are implemented, thedifference in scoring costs is enormously in favor ofthe computer, particularly when it comes to the com-puterized scoring of essays and open-ended ques-tionsan area in which both the ETS and Vantagehave worked extensively.
Last year, Indiana used the ETSs automated-scoringtechnologies for an online pilot of an end-of-courseEnglish exam for high school students, which in-
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
12
L
egal
Tw
ists,
Dig
ital
Tur
ns
Name of statecomputer-based test
Subject(s)tested3
State InitiativesTwelve states and the District of Columbia are administering computer-based assessments in 2002-03. Six of the tests are pilots. Of the states
with computer-based tests this year, five have reported that they designed the exams to partially meet federal requirements under the No Child Left
Behind Act of 2001, which requires testing of students in English and math in grades 3-8 and once in high school. Georgias online testing was
suspended for spring 2003 because of problems with test security. Florida and Oklahoma have planned computer-based testing pilots for the
2003-04 school year.
5
12
4
2
V
E, M
E, M
E, M
E, M
M
E, M, H, S,
E, M
T
E, M
E, M
E, M
E, M, S
E, M, H, S
8-12
5, 6, 8, 9
2
2-10
8-12
7
4, 5, 7, 8, 10-12
6-12
8
3-8, 10
3-12
2-12
3-12
high school
Vocational Student Competency Testing Program
Delaware Student Testing Program1
Measure of Academic Progress1
Idaho Standards Achievement Test
Indiana End-of-Course Assessments1
Kansas Mathematics Assessment1
Kentucky Core Content Assessment1, 2
Maryland Functional Reading/Math Tests
Computer Skills Test
North Carolina Computerized Adaptive Testing System1, 2
Technology Enhanced Student Assessment
Dakota Assessment of Content Standards
Utah Performance Assessment System for Students
Standards of Learning End-of-Course Test
Arkansas
Delaware
District of Columbia
Idaho
Indiana
Kansas
Kentucky
Maryland
North Carolina
Oregon
South Dakota
Utah
Virginia
U.S.
Grade(s) in whichcomputer-based
tests are available
All districtsand/or schoolsare required to
participate incomputer-based
testing
Test isadministered
over the Internet
Test is adaptive(questions
change based
on prior answers)
1Testing is in the pilot phase.
2Test serves as an accommodation option for students with disabilities to take the state assessments. Test is only available for students with disabilities.
3E-English/language arts, M-Mathematics, H-Social Studies/history,
S-Science, T-Technology, V-Vocational studies.4Any district or school chosen by the state for the pilot phase of testing must participate. However, there is no statewide participation requirement. State does not count in U.S. total.
SOURCE: Education Weeksurvey of state departments of education, 2003
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
9/76
cluded both essay and short-answer questions.We compared the cost of administering paper-and-
pencil tests and scoring all of the open-ended re-sponses by hand to administering the tests on com-puter and scoring everything with computer, says
Swartz, and the computer-administered and com-puter-scored version cost about a quarter of what thepaper-and-pencil version costs.
Whats more, students received the results in a mat-ter of days, rather than months.
The ETS official says more states and districts arenow open to technological solutions to their testingneeds, where we never saw that before.
Before, it was strictly, We want paper-and-pencil,Swartz says. Now, theyre saying, If youve got a tech-nology solution to propose, feel free.
Tight budgets definitely have put a crimp in stateplans, however.
Indiana had hoped to create an online item bank forteachers to craft classroom assessments linked to statestandards. But the planwhich would have cost about$800,000 a year for two years to generate a deepenough pool of test questionsis on hold.
We certainly havent given up on that, says Bruce,thestate testing director,but its going to be slightly delayed.
Adapting to Roadblocks
In Oregon, work on an online writing assessment hascome to a halt because of budget cuts. Schools alreadycan opt to administer state reading and math tests ingrades 3, 5, 8, and 10 either electronically or on paperthrough the Technology Enhanced Student Assess-ment program, or TESA, and the state will pilot addi-tional online tests in other grades this spring.
Were able to implement tests at new grades morequickly through TESAbecause essentially all the infra-structure is right there, says Bill Auty, the associatesuperintendent for the state education departmentsoffice of assessment and evaluation.
But he adds: Were fortunate that we started whenwe did. We could not start this up now. Our state fundsfor assessment have been cut considerably.
States that have ventured into online testing have al-ready confronted one roadblock under the federal law:
a mandate that states measure student performanceagainst the expectations for a students grade level.
In 2002, that requirement forced Idaho officials to mod-ify their plans for an online adaptive testing system, de-vised by the Portland, Ore.-based Northwest EvaluationAssociation.The system would have permitted studentsto take tests harder or easier than their actual grade lev-els, based on their ability to answer an initial set of testitems. As a compromise, state officials agreed to give allstudents a uniform, grade-level examination in additionto a more individualized set of test questions that couldprovide better diagnostic information.
Then, in January, South Dakota officials announcedthat they were making their states adaptive, onlinetesting programunder contract to the Scantron Corp.of Irvine, Calif.voluntary for districts and instead re-quiring new paper-and-pencil tests to meet the federalrequirements.
Wade Pogany, the director of education services forthe state education department, says the fact that theonline teststhe Dakota Assessment of Content Stan-dards, or DACSwere adaptive was not the primaryreason for moving to a new testing program.
I dont ever want to give the impression that SouthDakota does not like computer-adaptive tests, he says.Were a big supporter of that.But there are some issueswith the new legislation that caused us to look at a fixed-form test in relation to our standards, including the re-quirement that students be tested at grade level.
South Dakota, Pogany adds, is trying to use the bestof both worlds by continuing to finance the DACS fordistricts that want to use it, while coming up with anew set of exams that will allow the state to measurestudent performance against both state standards andnational norms.
Those experiences have clearly caused states to think
Educat ion Week ' s Techno logy Counts/May 8,2 003
13
Test was designedto meet testingrequirements
under the federalNo Child Left Behind
law
L
egal
Tw
ists,
Dig
ital
Tur
ns
Test providesdiagnosticinformationabout thetest-taker
State uses anoutside vendor
to designand/or
administerthe test
State administerscomputer-based
tests withspecial
accommodationsfor ELL or
disabled students
Test includesopen-ended questions
Answers toopen-endedquestions are
machine-graded
Test includesmultimedia features
(sound orvideo clips)
Arkansas
Delaware
District of Columbia
Idaho
Indiana
Kansas
Kentucky
Maryland
North Carolina
Oregon
South Dakota
Utah
Virginia
U.S.
1
4
5
2
6
10
5
8
5Tests are both machine-graded and graded by hand.
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
10/76
twice about venturing into adaptive, online assess-ments. Oregon had planned to add an adaptive fea-ture to its Web-based assessment program this schoolyear, but delayed it in part because of budget cutsand development work.
We will implement that next year, says Auty, theassociate superintendent for assessment and evalua-tion. Its something were looking into.
He adds, though: We want to be very clear, giventhe controversy, were talking about adaptive testingin the grade level.
Others argue that adaptive testing could meet therequirements under the federal law, but that it wouldtake more extended discussion and explanation fromtesting experts.
A Major, Major Issue
A far bigger impediment for state testing systems isthe infrastructure needed to deliver a secure com-puter-based exam to thousands of students under thesame conditions at the same time.
The major, major issue is crystal clear, and thats theinfrastructure, argues David J. Harmon,the director oftesting for the Georgia Department of Education.
Georgia had been planning to give districts the op-tion of offering the states Criterion-Referenced Com-petency Tests in grades 1-8 in a paper-and-pencil orWeb-delivered format this spring. But the state wasforced to suspend online testing this year after discov-ering that some 270 actual test questions were pub-licly available on an Internet site for students, parents,
and teachers. The state had developed the Web-basedtest-item bank to help schools prepare for the exams.
Kathy Cox, the state superintendent of education,says a miniscule amount of school districts evenwanted that online option, about half a dozen out of
181 districts.But in Oregon, where tests are available online,
Auty estimates about one-third of schools use theTechnology Enhanced Student Assessment programelectronically.
In Virginia, where districts can choose to offer mosthigh school end-of-course tests online or on paper,Mark J.Schaefermeyer, the associate director of Web-based assessment in the state department of educa-tion, estimates that up to 85 of 135 districts willchoose the Web-based version this spring.
Heres the challenge: High-stakes tests, such asthose in Virginia, which are used to make key deci-sions about individual students or schools, must begiven under secure conditions so that there are fewopportunities for cheating.Theyre typically adminis-tered during a limited time window. And they aresupposed to be given under the same conditions toevery student.
So how many computers does it take? asks Har-mon. I think thats the biggest problem.
In addition to the upfront and ongoing costs of buy-ing and maintaining hardware and software systems,computers and Internet connections dont alwaysfunction dependably. And testing sessions may be in-terrupted or proceed so slowly that the conditions in-terfere with student performance.
Capacity is the number-one issue; compatibility isprobably number two; and keeping your lines open isnumber three, says Bruce of Indiana.
This year, Indiana plans to pilot both end-of-courseEnglish and algebra exams exclusively online for
high school students. One of the things we ran intoin one of the largest high schools in the state wherethey wanted to do this, Bruce says, is that in orderto give the algebra test, they had to close down fourcomputer labs for two weeks. It took that capacity.
Eventually, Georgias Harmon predicts, the solutionmay lie with small, handheld, wireless devices thatlook like GameBoys, but with larger screens. Givensomething like that, he says, then there will be verysubstantial savings compared with the printing,shipping, and storage costs of a traditional exam.
The Wrong Question
On the measurement side, the biggest challenge forstate testing programs is the comparability of testresults. To compare performance fairly, each student
should be tested under the same conditions. If thereare any variations, they shouldnt affect performance.
But Bennett of the ETS points out that, in reality,the equipment used in statewide computerized-test-ing programs often varies from one school to the next,and sometimes even from one machine to the nextwithin the same building. Researchers dont yet knowhow to adjust for such variations in scoring results.
In states such as Virginia, where some students willtake a test online while others take a paper-and-pen-cil version, policymakers need assurances that nei-ther group is particularly advantaged or disadvan-taged by the mode of delivery.A study by researchers at Boston College, for in-
stance, found that students accustomed to writingon computers scored higher on test questions an-swered electronically than on those answered byhand, while those who were used to writing in long-
hand scored worse than similar students who tookthe exam on paper.Although several test ing companies report they
have conducted comparability studies, few have beenpublished. Moreover, observes Wayne Ostler, the di-rector of eMeasurement Services for Pearson Educa-tional Measurement, based in Iowa City, Iowa, youreally have to perform comparability studies on everyprogram. You cant just say, Oh, I tested math in Vir-ginia, and it turned out to be comparable, thereforeeverything is comparable.
But Elliot of Vantage Learning suggests the con-cerns are off target. When the car was introduced,can you imagine them conducting a car-horse compa-rability study? he says.Its the wrong question. Thecar is coming. Ive got news for you. If theyre notcomparable, so be it.
Not surprisingly, though, such concerns are leading
some states to focus their initial online efforts on low-stakes examsones that arent used to rate schoolsor decide students futures.
Low-Stakes Testing Boom
In contrast to some of the reticence about givinghigh-stakes tests electronically, experts predicta boom in the use of technology for instructionaland diagnostic purposes spurred by the new federallegislation.
What I think No Child Left Behind may speed upis the development of low-stakes, computer-basedtests that are designed to help kids do better on thehigh-stakes ones,Bennett of the Educational TestingService says. I think thats where were really goingto see a lot of movement.
In December 2002, Texas unveiled the first phase ofthe Texas Math Diagnostic System, a roughly $2 million,online practice assessment program for grades 4-8.Thisspring, the state is adding a 3,400-question item bankthat teachers can use to come up with their own tests,quizzes, and homework assignments.And it is translat-ing the test items in grades 4-6 into Spanish.
Texas also is working to add open-ended questions to
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
14
Legal
Tw
ists,
Dig
ital
Tur
ns
"How many computers
doesit take? I thinkthats the biggestproblem."
DAVID J. HARMON, Director of Testing, Georgia Department of Education
Continued on Page 16
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
11/76
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
By Jennifer Park
A
n informal survey of students about Oregons online test-ing system shows they find computer-based testing fasterand more enjoyable than the paper-and-pencil variety, and
they report feeling that they perform better on computer-ized assessments than on traditional tests.
While some states are just beginning to contemplate the possi-bility of computerized testing, Oregon piloted a program in spring2001 that is now being expanded across the state.The system de-livers the Oregon Statewide Assessment over the Internet.
The main goal of the Technology Enhanced Student Assessment,or TESA, program is to provide test resultsquickly, so teachers can more effectively deter-mine how well students are meeting state stan-dards. The program also aims to offer more flexi-bility: Teachers can administer the online testswhenever they deem appropriate, whether to anindividual student, a group of students, or an en-tire class.
To assess the pilot phase of the online testingprogram, Oregon state education officials surveyed740 3rd graders and 730 high school students from
around the state about their experiences takingthe state assessment online compared with takingthe hard-copy version. About 2,500 students hadtaken the online tests. State officials also held de-briefing sessions and hired evaluators to conductinterviews with teachers.
Third graders were especially positive aboutWeb-based testing. Seventy-nine percent of thosepupils surveyed believed they had done theirbest work on the computer-based reading test,compared with a little more than 5 percent of3rd graders who felt they either had done betteron paper or were frustrated using the computer-ized test.
Results were similar for the online math test.Onthat assessment, more than 94 percent of the 3rdgraders said they felt that they had done theirbest work on the computer test or that they had
done equally well on both versions. Only 6 percentbelieved they had performed better on paper orhad felt frustrated with the computer-based exam.A majority of the 3rd graders enjoyed taking the
Web-based version more than they did the papertest: Sixty-two percent reported that the onlinereading test was easier to use and more enjoyablethan the paper test, and 58 percent said the sameabout the Web-based test in mathematics.
Not as Positive
High school students were slightly less positiveabout their performance on the Internet-basedtests.
On the reading test, 78 percent of the highschool students believed they had either donebetter on the computerized version or equallywell on the Web and paper tests. About 22 per-cent of the high schoolers felt they had eitherdone better on paper or were frustrated using thecomputer-based exams.
In math, about 76 percent of the high school students either feltthat they had done their best work on the computerized test orthat they had done equally well on both versions. Fewer than aquarter of the high school test-takers felt they had done better onthe paper exam or were frustrated using the Web assessment.
Most of the high school students reported finding that the Inter-net version had taken less time or the same amount of time as thepaper test.
But a considerable number of the students disagreed. Thirty-onepercent of the high school students tested said the Web version ofthe reading test had taken more time than the paper test. Thesame percentage reported that the computerized math test hadtaken longer to complete than the paper test.
Thirty-seven percent of the high school students rated the com-puterized version of the reading test easier to use and more enjoy-able to take than the paper version; 38 percent found the Web-based math test easier to use and more enjoyable to take.
Twenty percent of the high schoolers surveyed in Oregon saidthat the Web-based reading assessment was harder to use thanthe paper-and-pencil test.
A Test-Takers PerspectiveSurvey asks Oregon students
what they think about the states
computerized assessments
15
Legal
Tw
ists,
Dig
ital
Tur
ns
COMPUTER VS. PAPER TESTING
Third graders responded positively about their performance on a Web-based test,
compared with a paper-and-pencil version of the Oregon state reading exam. While
high school students were less positive, most still felt they either did better on the
Web-based test or equally well on the Web and paper versions of the test.
SOURCE: Oregon Department of Education, Technology Enhanced Student Assessment (TESA), student survey, 2001
A majority of 3rd graders found the online version of the Oregon state math test
easier to use than the paper-and-pencil version. Thirty-eight percent of high school
students found the Web version easier to use and more enjoyable.
3rd Grade Math High School Math
Web test was easier to use and more enjoyable
About the same as paper-and-pencil test
It was harder than paper-and-pencil test
More difficult at first, then easier
0 20 40 60 80 100
Percent of students
I feel sure I did my
best work on the Web
I did equally well
on Web and paper
I didnt do as well
as I did on paper
I was too frustrated
using the Web
79
39
40
16
16
6
4
23rd grade reading
High school reading
15%
58%
22%
5%
38%
39%
17%
6%
NOTE: Percentages do not add up to 100 because of rounding.
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
12/76
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
16
L
egal
Tw
ists,
Dig
ital
Tur
ns
the system, as well as an adaptive-testing component.And in Florida, teachers can use the FCAT Ex-
plorer, a free, Web-based program that provides a
series of test prompts and skills packages designedaround the states academic-content standards,which guide the Florida Comprehensive Assess-ment Test.
We are currently getting about 11.5 million hits aday, says policy consultant Don Griesheimer, and Ithink the company thats hosting this has said wehave about 4,000 simultaneous users a second. Soits taken off big guns here in the state of Florida.
Other states have offered schools access to commer-cially developed assessment-and-diagnostic packagesthrough state Web portals.Indeed, the greatest activityis taking place at the district level, where companiesare offering schools a range of products to help monitorstudent progress and provide teachers with instantfeedback on what to do next.
This indirect market, generated by the No ChildLeft Behind Act, I see as the real impetus for how
technology will be used in schools, says Michael T.Nesterak, the director of product management forCTB/McGraw-Hill, one of the countrys biggest com-mercial test-makers.
Untapped Power
Despite such inroads, most observers agree thatthe potential power of technology to redesign as-sessment remains largely untapped.
The powers of the computer have not been fullyused yet in any of the state testing programs, saysNeil Kingston of Measured Progress, a Dover, N.H.-based company that has worked on online testingprograms in Georgia and Utah.
Most computer-based programs at the state level re-main exclusively multiple-choice.Thats not surprising,
observers say. The reality is we have to start whereschools and districts are, says John E. Laramy, thepresident of Riverside Publishing, one of the nationslargest test companies.He believes that in time, youwill see companies introducing more innovative, inter-active assessments.
But with the federal laws emphasis on providingmore tests, at more grade levels, faster and moreefficiently, no one expects that breakthrough any-time soon.
No Child Left Behind might be pushing states,says Doris Redfield, a technology expert at the Ap-palachian Education Laboratory in Charleston,W.Va., but its pushing them in the direction thattheyre already going of getting quicker results andensuring that their assessments are aligned withtheir standards.
Is it really pushing them to create assessments
that are more true-to-life or out of the ordinary?Redfield wonders. I dont know that it is.At least one observer, Ed Roeber, the vice presi-
dent of external relations for Measured Progress,worries that the current emphasis on cheaper,faster, better is leading states to eliminate or re-duce the number of open-ended responses in theirtesting programs.After an initial burst of steam,experts admit,the ex-
pansion of computer-based testing at the state level ishappening more slowly than they had expected.
Its kind of frustrating, says Greg Nadeau, the di-rector of the U.S.Open e-Learning Consortium,a col-laboration among 14 states for sharing online educa-tional tools. The economic contraction is taking someof the wind out of the sails of the [information tech-nology] industry and of state and local and federalprojects.
It takes years to develop online testing,Pogany ofSouth Dakota says. So you cant come in and say:OK.We have a new test. Put it online. Do it.
Until I know that we have a solid and reliabletest, he adds of South Dakotas new assessmentsystem, were not going to put kids at a computer.Now, will that happen next year? I dont know. It willhappen when were ready, and were not going to doit before.
Continued from Page 14
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
13/76
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
C
omputer adaptive testing is used to test re-
cruits to the U.S. military, for licensing nurses
and computer technicians, for entrance tests
to graduate school, and for a popular place-
ment test used by community collegesbut not for aca-
demic testing in all but a handful of K-12 schools.
Most notably, computer adaptive testing has been
left out of nearly all the large-scale testing programs
that states are ramping up to meet the requirements
of the federal No Child Left Behind Act of 2001.
A prime reason:The U.S.Department of Educa-
tion interprets the laws test-driven accountability rules
as excluding so-called out-of-level testing. Federal offi-
cials have said the adaptive tests are not grade-level
tests, a requirement of the law.
Psychometricians regard that decision as humor-
ous, Robert Dolan, a testing expert at the nonprofit
Center for Applied Special Technology in Wakefield,
Mass., says of the departments stance.
Adaptive tests deliver harder or easier items, depending
on how well the individual test-taker is doing.They are con-
sidered out-of-level because the difficulty range could in-
clude skills and content offered in higher and lower grades.
Dolan and other test experts concede states may have
reason to say no to computer
adaptive testing, because of cost, uneven
technology levels in schools, and even educators
unfamiliarity with the methodbut not because of
grade-level testing.
The span of [test item] difficulty from easiest to
A QuestionOfDirection
ByBy Andrew Trotter
Adaptive testing puts federal
officials and experts at odds
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
14/76
hardest is entirely under the control of the test devel-oper,says Tim Davey, the senior research director of theEducational Testing Service, based in Princeton, N.J.
Some experts say adaptive tests give schools a bet-ter return on the time and money devoted to test-
ingincluding more accurate measurement of theproficiency of students who are above and below av-erage, and speedier access to the test results.
But Education Department officials say their handsare tied. The regulations are very clear in saying all stu-dents have to be held to the same standard as the foun-dation for school accountability,says Sue Rigney, an ed-ucation specialist in the department.The focus here isvery explicitly on the grade level the state has defined.
Federal officials worry that out-of-level testing mightlead to lower expectations for below-average students.
They also note that states are free to use computer-adaptive tests outside the accountability purposes ofthe No Child Left Behind law, which requires yearlyassessments in reading and mathematics of studentsin grades 3-8.
But the upshot, for now, is that computer adaptivetests are left out of the federal law, along with the pub-
lic attention and federal money for test developmentthat come with it.And the developers of adaptive testsfeel they are missing out on what may be the greatestprecollegiate testing boom in history.
Made Us a Pariah
[The Education Departments] decision made us apariah, says Allan L. Olson, the president of theNorthwest Evaluation Association, a nonprofit test-ing organization in Portland, Ore.The group was de-veloping a computer adaptive test for Idahos assess-ment when the department ruled its method out justover a year ago.
Federal officials gave the same message to SouthDakota and Oregon. South Dakota subsequentlymade voluntary its once-required computer adaptive
test, and has adopted a conventional paper-and-pen-cil test for its statewide program. Oregon has post-poned for a year the addition of a computer adaptivefeature to its online test.
I think the [departments] interpretation in the caseof South Dakota was based on a sort of misunder-standing of what adaptive testing does, says Davey ofthe ETS. He says computer adaptive tests typicallyspan more than a single grade levela diagnostic ben-
efitbut they dont have to, and in any case, grade-level information is recorded for each test item.
Researchers express puzzlement because the federalgovernment has been deeply involved in the develop-ment of computer adaptive testing,starting with sem-inal research at the U.S. Office of Naval Research inthe 1970s and 1980s. A decade later, Education De-partment grants paid for new computer adaptive read-ing tests in foreign languages,and department officialslauded the methods potential for school-based testing.
David J. Weiss, one of the original leaders of theNavy research, says there is no reason why com-puter adaptive testing is not appropriate for K-12.
Now the director of the psychometric-methods pro-gram at the University of Minnesota, Twin Cities,Weiss notes that a study of children who took suchtests in Oregon for several years produced beautifuldata on improvements in math and reading.
Federal officials say they would consider the useof a computer adaptive test if it tested within thegrade level.
But other test experts say the federal governmentis right to be wary of computer adaptive testing.
The technology is not ready for prime time, con-tends Robert A. Schaeffer, the public education direc-tor for the National Center for Fair & Opening Testing,or FairTest, a Cambridge,Mass.-based advocacy groupthat opposed the No Child Left Behind Act because ofits testing mandates.
He says the computer adaptive version of the Grad-uate Record Examination launched at ETS testingcenters in 1994 was initially flawed because it had apool of test items that was too small, and there wereinsufficient facilities for the number of test-takers.
ETS spokesman Tom Ewing acknowledges thoseproblems occurred but says they were quickly re-
solved through enlarging the pool of questions andimproving test scheduling.
But Schaeffer warns that schools could face arougher transition, considering their budget limita-
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
18
A
Question
of
Direction
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
15/76
tions and the high stakes involved in testing.W. James Popham, a professor emeritus and educa-
tional testing authority at the University of Califor-nia, Los Angeles, says the theoretical accuracy ofcomputer adaptive testing does not necessarily trans-
late into reality: Even though [such testing] makesmeasurement types quite merry, they can play gameswith numbers and it doesnt help kids.
Popham, a former president of the American Educa-tional Research Association, contends that the testingtechnology is opaqueto the public and policymakers.
He says federal officials may believe the testingmethod could introduce loopholes into the education law.
They fear educational con artists who have histori-cally frustrated congressional attempts to safeguard dis-advantaged youngsters, Popham says, referring to edu-cators who wish to avoid accountability. The fear is,theyll pull a fast one and downgrade expectations.
Zeroing In on Skills
But proponents of adaptive, computer-based testing
fear that schools may wait decades for access to amajor improvement over conventional, linear stan-dardized tests, which present each student with thesame set of test items.
The logic of the new tests is that of a coach pitching to ayoung batter: If the youngster is missing,the coach easesup a little; if not, he increases the challenge. Sooner orlater, the coach zeroes in on the batters skill level.
Some testing experts argue that the adjustment im-proves test accuracy.
In paper-and-pencil tests, items tend to be groupedaround average kids.Those in the tails of distributionwe dont get as much information about those kids,saysMichael L. Nering, the senior psychometrician at Mea-sured Progress,a testing company in Dover,N.H.
The great thing about adaptive testing is that it hasequal precision, meaning the results are accurate at allproficiency levels, says Nering, who helped design two
state assessments and developed computer adaptivetests for ACT Inc. No matter what your ability is,
whether youre really smart or not, the test will stop ad-ministering items when equal precision is reached.
By contrast, most of the items on conventionaltestson paper or computerare aimed at the aver-age student in the target population.
If Im a very low-performing student, there may beonly two or three items on the [conventional] test thatare appropriate to my level of performance, Davey ofthe ETS says, adding that the same is true for high-performing students.
Inside the IRT
Computer adaptive tests often use the same types ofquestions as conventional tests, though with adjust-ments for display on a screen. Other features are dis-tinctive, such as the order of items being irreversible.Students are not allowed to recheck or change answers.
This one-way street is necessary because of theprocess that takes place after each answer: A centralcomputer recalculates the test-takers ability level,then selects the next item, based on the individuals
success to that point.As the student completes more items, the computer
tracks the statistical accuracy of the score until a setaccuracy level is reached. Then the test moves to an-other skill or content area. Reaching that level mayrequire fewer items if the student answers with con-sistent proficiencyor many more items, if the stu-dent answers inconsistently.
Adaptive testing doesnt waste the examinees time byasking questions that were already pretty sure we knowhow the student is going to answer, says Davey.
To make the crucial decisions about which items topresent, the test is outfitted with an item responsetheory modelessentially its brains and the part ofthe system that some critics consider opaque.
The IRT model governs the interaction between thetest-taker and the test items. It weighs the studentsrecord of right and wrong answers against several
known characteristics of the test itemssuch as diffi-culty, the ability to discriminate between higher- and
Educat ion Week ' s Techno logy Counts/May 8,2 003
20
A
Question
of
Direction
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
16/76
lower-ability students, the degree to which guessingmay succeed,and coverage of academic content.
By solving the complex algorithm written into theIRT model, the computer determines which test itemshould be presented to the student next.
Test developers concede that IRT models are un-fathomable to lay people and even challenge the in-tellects of experts unfamiliar with a given test.
Schaeffer of FairTest calls the IRT model the pig ina poke that makes computer adaptive testing hardfor policymakers to accept.
Who knows what the algorithm is for test deliv-ery? he asks. You have to accept the test manufac-turers claims about whether the test is equivalentfor each student.
Scott Elliot, the chief executive officer of VantageLearning, a major maker of computer-based tests lo-cated in Yardley, Pa., says, There are many technicalnuances under the IRT; some differences [betweenIRTs] are sort of like religion.
Davey of the ETS agrees that the IRT resists attemptsto explain it, but adds that the apparent simplicity ofconventional testing is based largely on oversimplifi-
cation of how paper testing typically is done.In fact, he says, virtually identical IRT models are
used with some conventional state tests to ensurethat the same score in different years represents ap-proximately the same proficiency level on the testavital issue for accountability.
Breaking With the Past
Because of technology hurdles and spotty acceptanceof computer adaptive testing, experts generally predictthat the field will struggle for the next five or 10 years,but that schools will eventually turn to the approach.
Davey believes educators will be persuaded by thegreater amount of diagnostic information the tests pro-duce from fewer school days spent testing.
Thats not to overlook other formidable problems
that computer-based testing poses for schoolsno-tably, the difficulty of providing technology that is re-liable and consistent for all students, so the playingfield is kept level. The tests must be delivered over arobust infrastructure to avoid processing and com-munications delays that would leave students wait-ing for their next test items.
Computer adaptive tests also require larger banksof test items than conventional tests do. Yet theadaptive method gives items a longer useful life be-cause its harder for test-takers to predict whichitems they will encounter.
Finally, adaptive tests are subject to some of thesame well-documented problems as other standard-ized tests, such as cultural biases, says FairTestsSchaeffer. Automating test items that are used in-appropriately, in many ways makes matters worseyou add technical problems and dissemination-of-in-
formation problems, he says.Referring to the ETS adaptive Graduate Record Ex-
amination, he adds, The GRE, in spite of all the hoopla,is the same lame questions put out using a hidden algo-rithm, instead of linearly on a sheet of paper.
Ewing of the ETScounters that its test items are whatthe graduate deans have said are the math and verbalskills that they want students to be able to handle.
Meanwhile, researchers are working on new kindsof adaptations that could be applied in computeradaptive testsincluding presenting items usingmultimedia or computer simulations and catering toan individuals preferred learning style. Already,some tests present items in different languages.
Those changes highlight another potential pitfall.Today, policymakers insist on having new testsdemonstrate comparability with old tests, a taskthat Davey says becomes vastly more difficult astesting methods change.
Benefiting from many promising innovations willrequire letting go of comparability, Davey maintains.
Its like when we moved away from strictly essaysand constructed-response items 100 years ago andintroduced multiple-choice items, he says. For testsbased on simulations, theres no paper-and-pencilequivalent anymore.You have to make a clean breakwith the past.
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
21
A
Question
of
Direction
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
17/76
Educat ion Week ' s Techno logy Counts/May 8,20 03
A
s test-preparation materials leap off the
printed page and onto the Web, an increas-
ing number of states and districts are
turning to online test-prep programs
to help raise student scores on high-stakes
assessments, Advanced Placement tests,
and college-entrance exams.
California, Florida, and Massa-
chusetts are among the states
signing up for Web-based remedial
offerings and other academic help,
spurred on by stiffer state and federal
accountability requirements.
Soon, experts predict, most states and districts
will use online test prep because of its flexibility, its
reasonable cost, and some promising, though very
preliminary, results on its effects.
This is definitely not a fadyoull see Web-based
assessments broadly adopted in the next several years,
says Adam J. Newman, the vice president of research for
Eduventures Inc., a Boston-based research firm that
tracks the K-12 market.
Print will never disappear, but institutions will have
more cost-effective solutions with the Web.Also, the ability
for educators to
customize their assessments
will really enhance what they already do
[for students].
The many companies that have jumped into the test-
prep market range from the well-established, New York
City-based Princeton Review to small, aggressive
upstarts such as Test University, also in New York, and
the Washington-based Smarthinking Inc.
The money has followed. The Princeton Reviews
PreppingforThe BigTestStudents turn to the Web to get
ready for high-stakes exams
ByBy Rhea R. Borja
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
18/76
K-12 revenue shot up 43 percent last year, boostedin large part by a $2.5 million boom in online test-prep subscriptions and other supplemental programs,according to the Heller Reports, an educationalnewsletter and market-research publisher based in
New York.Last year, more than 200,000 students in 60 coun-
tries signed up for the Princeton Reviews onlinedemonstrations of such tests as the SAT and state exitexams, according to Drew Deutsch, an assistant vicepresident of the company.
Our online test prep is growing by leaps andbounds, he says.
Not for Everybody
Eduventures Newman and other experts caution,though, that online test preparation is only as good asa students motivation to use it and a teachers abilityto analyze the data gleaned from it. Web-based testprep is a tool to improve learning, they emphasize, nota magic bullet.
Teachers need to know how to use this resource andapply it to their classroom pedagogy, Newman says.Otherwise, its useless.
Online test prep isnt for everybody, adds BurckSmith, the chief executive officer of Smarthinking,which serves 29,000 high school and college studentsin at least 15 states. He says students who use themethod must be highly self-motivated or have teachersmonitoring them. He found that out with students whohad failed their state assessments the first time anddidnt seem especially impelled on their own to im-prove their scores.
With any online test-prep program, youre not goingto see a lot of success unless theres someone theremaking sure the students are doing what theyresupposed to be doing, Smith says. Motivation is ahuge factor.
And critics of online test preparation wonder whetherits approach of drill and repetition is supplanting morelively, creative teacher-led instruction.Are the studentstaking online test prep getting only the chunks of infor-mation they need to know to improve their scores, in-
stead of a more balanced education?Web-based, online computerized coaching pretty
much focuses on the basics, so the kids pretty muchlose out on everything else, says Robert A. Schaeffer,the public education director for the National Centerfor Fair & Open Testing, or FairTest, a test-reformadvocacy group based in Cambridge, Mass. What istested is what is taught.
Its a Conversation
But supporters of online test prep point out that theprograms can be customized with lessons and practicetests to fit each students academic needs, can providea wealth of student data, and can be used anytimeinclass during the day, and at home at night and onweekends.
The Princeton Reviews Homeroom.com, for example,has a 130,000-question bank for grades 3-12, which dis-tricts and states can mine to produce online practice testsand courses specifically aligned to state exams.Its onlineSAT-prep program offers 185 interactive, multimedialessons and drills, and an online coach to answer ques-tions 24 hours a day.
Test Universitys online SATcourse has interactive micro-courses covering 145 basic skills, 18 practice tests, a2,000-word vocabulary lab, and two full-length SAT testswith immediate feedback. TestU also features lessonscovering 55 test-taking strategies.
Smarthinking, meanwhile, uses e-tutors, or online in-structors, and digital whiteboards to give students moreintensive tutoring help. Students can communicate withe-tutors in real time, as well as submit questions or as-signments to the instructors online and receive feedback
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
24
Prepping
for
the
Big
Test
COMPUTER-BASED PRACTICE
Several states are using technology to help their students prepare for state tests by offering computer-based
practice exams. These practice exams range from tutorials that mainly serve to guide students through an
online testing format to those that test students knowledge of content related to state standards.
SOURCE: Education Weeksurvey of state departments of education, 2003
Computer-based practice exams (12)
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
19/76
within 24 hours,the company says.The virtual whiteboards act much like real chalk-
boards: A student can write out or draw a multisteptrigonometry problem using an online toolbox of math-ematical symbols. The e-tutor sees the students work,
and can comment on the whiteboard and help correctany missteps.
The 12,000-student Whitfield County school systemin rural northern Georgia uses Smarthinking in partto help its 164 Spanish-speaking, English-as-a-second-language students pass the Georgia Graduation Test.For two hours a day, ESL-trained, Spanish-speakingmath tutors answer students questions online, saysLorijo Calhoun, the districts ESL instructor.
Its a conversation, she says. Its literally backand forth.
In Marylands 107,400-student Baltimore Countysystem, teenagers preparing for the PSAT, the SAT, orthe ACT can log on to TestUs Web site and get extrahelp on their vocabulary, for example, or more in-depthinformation on algebraic algorithms.
Students first take a 30-minute diagnostic pretest, orbrainscan, that TestU uses as a baseline for the on-line curriculum. If a student does poorly on grammarbut well on comprehension, for example, TestU offersmore intensive work on the former. Like many otherschool districts and states, the Baltimore County sys-tem is making TestU available to all students, but es-pecially pupils from low-income families.
We want to ensure equity of access, says BarbaraDezmon, the assistant to the superintendent of theBaltimore County district, which does not include thecity of Baltimore. It started as an effort to assistschools that had significant minority student popu-lations, significant poor populations, and it spreadfrom there.
Diane Young, an assistant principal at the 1,350-stu-dent Eastern Technical High School in Baltimore
County, says that online test preparation is more help-ful than the schools traditional SAT-prep class becauseinstruction is customized for each studentsomethinga teacher with 25 students cannot do.
It allows a teacher to say to a student, Youre hav-ing problems with fractions, and I cant spend anymore time in class, so go on TestU and work on it, andIll check your answers, she says.
I Felt More Prepared
But online test prep is not a substitute for a teacher,Young and other educators stress. Instead, its an-other tool that can enhance the effectiveness of in-class instruction.
Glenn Finnerty, a test-prep teacher at the 3,200-stu-dent Timber Creek High School in Orlando, Fla., sayshis students use both a workbook and individualizedonline mathematics and reading courses by TestU tohelp them improve their scores on the Florida Com-prehensive Assessment Test, or FCAT.
During class, Finnerty walks around his room andmonitors students as they work on the computercourse and practice tests. He can also view a variety ofcomputer reports that analyze student performanceand participation rates.
One is a group diagnostic report, which aggregatesand analyzes students responses to test questions,while a group learning report shows the skills andstrategies studied on a course, and evaluates the stu-dents quiz performance. A group performance report
shows student participation and performance rates,the frequency of tests taken, the average percent ofquestions answered correctly, and the distribution ofstudent responses to each question.
The combination of in-class instruction, onlinecoursework, and teacher monitoring is essential in try-ing to help low-scoring and perhaps poorly motivatedstudents improve their test scores, Finnerty says.
Its a complement to what we already have with ourbook work, he says. If teachers just say to the kids,Go at it, and leave them at the computers by them-selves, thats not good.
One of his students, 15-year-old Davis Ho, recentlytook the FCAT, and says that online test prep did itsjob. In fact, he says, many of the questions in his test-prep program were similar to the questions on thestate assessment.
I felt more confident, more prepared, says the 9th
grader.Ho says that in some ways, online test preparation
is more helpful than a teacher because it allows him tomove at his own pace. You can go back and review thestuff, he says. You can take your time and read it overand over again without having to frustrate anybody.
He and others suggest that the main difficulty inusing Web-based test prep is the tendency of comput-ers to freeze up. Technical glitches greatly hamperedTimber Creek Highs ability to use the TestU programlast year, school staff members say, but emphasizethere are fewer such problems this year.
Ho replies that while that may be so, glitches arestill not uncommon. Sometimes, the program kicksme off, he says.
Often, glitches occur because of a miscommunicationbetween TestU and the school districts filtering andsecurity software, says Nicole Marshall, TestUs vice
president for educational research. Thats what hap-pened to Timber Creek.
So county technology administrators readjusted thesettings in the security software to accommodateTestU. If the school is still experiencing glitches, itcould be that a lot of students are using the programand other technology at peak times, slowing down thesystem, Marshall adds.
Buying in Bulk
Web-based test prep tends to be cheaper for schooldistricts than face-to-face preparation classes are.Costs vary widely, but they fall well under the hun-dreds of dollars a test-prep course would cost an indi-vidual student.
Massachusetts, for example, paid $200,000 to the
Princeton Review this school year to help up to 80,000high school students statewide prepare for the Massa-chussetts Comprehensive Assessment System,or MCAS.
On the west coast, the University of California sys-tem signed a three-year, $125,000 contract with Num-ber2.com, a division of Xap Corp., an education in-formation-management-systems company based inLos Angeles. Number2.com provides online SAT andACT preparation for all California high school studentsfor free.
Research on the results of online test preparation issparse, but preliminary results of one analysis showonline test prep can be one factor that helps improvestudent test scores.
For instance,75 percent of high school students who hadpreviously failed the Texas Assessment of Academic Skillsin the 211,000-student Houston school district improvedtheir TAAS reading scores by 29 percent on average in apilot test-prep program, according to TestU, whose ser-vices the Houston district uses.
Jo Beth Harris, the director of Houstons VirtualSchool, who administers the online test-prep program,says that last summer 50 seniors who previously failedthe TAAS spent time with the online test-prep programfor two hours a day.
The result the second time they took the TAAS?They all passed, Harris says.
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
26
Prepping
for
the
Big
Test
"You can take yourtime...withouthaving to frustrateanybody."
DAVIS HO, 15, Timber Creek High School, Orlando, Fla.
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
20/76
Educat ion Week ' s Techno logy Counts/May 8,20 03
I
ndiana special education student Marvin
Stuller couldnt believe the boy reading
aloud on the TV set was him.xxxxxxxxxx
His teacher had videotaped
him at the beginning of 5th grade and at
the midway point.The before and
after scenes of the then-12-year-
old with speech difficulties were
dramatic: His reading level had
risen,and his voice had even lowered
and deepened by the later video.
Mom, I really improved, he told
his mother when he got home that night,
recalls his mother, Jenny Stuller of Elkhart,
Ind. She was impressed with how the video gave
her an intimate glimpse into her sons academic progress.
As educators face new federal requirements to test all
students annually in reading and mathematics in grades
3-8, states are experimenting with new ways of using
technology to evaluate the abilities of special education
students. Testing experts say that what educators learn
from tailoring assessments to the needs of special educa-
tion students could shape how they test regular students,
who have more subtle individual needs.
Special education students are likely to be the ca-
naries in the coal mine as educators experiment with
new technological assessments, says David Rose, the co-
executive director of the Center for Applied Special Tech-
nology, or CAST. The nonprofit group,based in Wakefield,
Mass., seeks to expand educational opportunities for stu-
dents with disabilities through the use of technology.
[Special education students] are the first to experience
ByBy Lisa Fine Goldstein
Spec Ed TechSparks IdeasTesting tools for children with disabilities
attract mainstream attention
. .
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
21/76
an oxygen deficit if a test doesnt work, Rose ex-plains. Its like when you test-drive a new car: Youdo it on rough terrain. You take it out to where it willbe tested.
Testing in the Margins
Many districts and an increasing number of statesare ramping up to launch computerized assessments,which experts say can be more efficient, use less labor,and provide instant scoring. The tests will offer thepotential for more sophisticated accommodations,such as the use of spell-checkers or audio featuresthat read questions aloud.
There was a time when schools would try to deteras many special education students from taking thetests as possibletell them to stay home on test dayso their [schoolwide] scores wouldnt be lowered,saysRobert Dolan, a senior research analyst at CAST.Now, all of those students have to be included, sostates want to do whatever they can to help themscore as high as possible.
At the same time, those students diff icul ties onsuch tests could bring to light broader problems:whether the tests are really evaluating the right ma-terial; if administration of the tests is biased; andwhether the technology presents more problems thanit overcomes.
For example, if a student is uncomfortable in frontof a camera, the videotaping approach may be coun-terproductive. Or if a student is a poor typist, com-puterized assessments might be more difficult thanpaper-and-pencil versions.
The [testing] revolution begins in the margins,Rose says. Special education students are being in-creasingly pushed toward the general education cur-riculum. How they adjust will teach us a lot about as-sessments.
In Indiana, for instance, educators use electronicportfolios to measure the progress of students with
disabilities. To do that, teachers collect videotape, au-diotape, and written materials on their students and
then scan that information onto CD-ROMs. Teachersreview the electronic portfolios to determine if childrenare ready to advance to the next grade.
Many general educators have looked on enviouslyat those alternative assessment methods, and some
have even adopted the practices themselves, saysRobert Marra, the state director of special educationfor the Indiana Department of Education.
General education teachers who have seen thetapes want to be able to do this for their children,Marra says.There are some who are using the video-tapes and other techniques already.
Whats most important, Marra says, is that theportfolios are helping teachers prepare students withdisabilities for regular assessments. The teachershave feedback right away on how a student is pro-gressing. Unlike with conventional testing methods,there is no lag time while a test is being scored.
The electronic portfolios also cut down on reamsof paperwork, one of the chief complaints of specialeducators, because the digital records help teachersdocument information on how a student is progress-ing as required by the federal Individuals with Dis-
abilities Education Act, Marra says.Whats more, teachers can use the electronic portfo-
lios to carry out IDEA-required obligations to give par-ents evidence of a childs academic progress.Teacherscan simply download the work from a computer ande-mail files to parents.
In the past, teachers and parents may have as-sumed that students with disabilities werent ableto take assessments, because they performed badlyon classroom tests that measure standards. But elec-tronic portfolios have changed that impression,Marra says.
Seeing proof of a childs abilities helps teachers andparents feel more comfortable with the prospect ofthe student being tested. Or, in some cases, a portfo-lio can be used in place of testing to show a studenthas met the standards for graduation.
If a parent says, I dont think my child can do that,
a teacher can turn around and show it to them onvideotape, Marra says. Also, the parents could use it
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
28
Spec.
Ed.
Tech
Sparks
Idea
s
SPECIAL PREPARATIONS
Although special education teachers are more likely than regular teachers to report that students use computers
in their classes, those same teachers, on average, are less likely to report that they have had professional
development on how to use computers in the classroom. New special education teachers feel less prepared
to use computers for teaching than do other novice teachers in their first year of teaching.
100
80
60
40
20
0
Percentofteachers
79
Teachers whose
students
use computers
in their classes
Teachers who have had
professional development
on the use of computers
for instruction in the
past 12 months
Novice teachers who
felt well or very well
prepared to use
computers for instruction
in their first year
Special education teachers
Non-special-education teachers
68 67
71
37
43
SOURCE: Education Weekanalysis of U.S. Department of Education, National Center for Education Statistics, Schools and Staffing Survey,1999-2000
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
22/76
to videotape their children at home doing somethingthe teacher didnt know the student could do.
It works both ways, he says.
World of Glitches
With states preparing to test so many more special ed-ucation students who would otherwise have been ex-empt from the process, some states are already comingup with innovative technological accommodations.
Oregon is devising a test that would allow hearing-impaired students to use American Sign Language.By clicking on either English or ASL, the studentscould choose to read the problems in English text orsee a pair of hands signing the questionsor evensplit the screen with both English and ASL.
The same concept could potentially be used for, say,Spanish-speaking students for whom English is asecond language.
Oregon is also working on a version of the test thatwould allow certain students with disabilities to hearthe questions read aloud by the computer. In some
cases, that feature could benefit students with read-ing disabilities or visual impairments.
Though the state has no plans to offer the option tostudents without disabilities, Pat Almond, the as-sessment project director, says one day the audio fea-ture could be used on tests taken by all students.Sheemphasizes that would not be the case, of course, ifthe test were designed to assess reading ability.
The prospect of having an audio capability on testsfor all students isnt financially out of sight, Almondadds, because the technology to do so is becomingmore widely available and affordable.
Meanwhile, Massachusetts has prepared versionsof its state assessments for students with disabilitiesthat would allow the use of text readers.About 1,000 Massachusetts students with reading
disabilities or visual impairments use the text read-ers for their classroom work; the accommodation is
allowed in their individualized education plans, orIEPs, which are required by federal law. Only thosestudents would be eligible to take large-scale assess-ments using the text readers.
Careful piloting of such programs is essential, saysDaniel J.Wiener,the states assessment coordinator forspecial populations. For example, text readers have atough time presenting math problems that includeGreek symbols,or equations within parentheses.
Theres just a world of glitches out there, Wienercautions. Its expensive to test these students. Youdont want to go down the wrong path.
Because of the high cost and other potential prob-lems, he doesnt foresee that text readers will ever beused by general education students.And, he says, it would not be appropriate to pro-
vide use of an electronic reader to a s tudent whodoes not have a disability that prevents him or her
from learning to read without use of a reader.
Show What They Know
For states gearing up to offer assessments online, theact of transferring a paper-and-pencil test to a comput-erized format would be like designing a building with nowheelchair ramps, wide hallways, or elevators, saysDolan ofCAST. Undoubtedly, retrofitting the traditionaltests would prove expensive and complicated,he says.
Still, Oregon is developing computerized tests inwhich teachers would set individualized parametersfor special education students, according to GeraldTindal, the director of behavioral research and teach-ing at the University of Oregons college of education.For example, students with disabilities who are poorreaders could take a math test with the reading leveladjusted to their own needs.
It seems to Tindal that such a strategy could oneday be used with all students to make sure theirknowledge of academic content isnt held hostage toan unrelated skill.
It would be great if we could increase the opportu-nity for students to show what they know, Tindalsays. Its an interesting time to proceed in the con-text of high-stakes testing.
Educat ion Week ' s Te chno l o g y C oun t s/May 8 , 2003
30
Spec.
Ed.
Tech
Sparks
Idea
s
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
23/76
Educat ion Week ' s Techno logy Counts/May 8,20 03
S
tudents in Carole Givens U.S. history class last
year took a quiz at the beginning of every
period.But they didnt line up at the
pencil sharpener to get ready. In-
stead,they fired up their laptops.
Givens, a 34-year teaching veteran who
teaches at the 1,500-student Varina High
School in suburban Richmond,Va., used online
quizzes as a way of reviewing each unit with her
students. A computer graded the quizzes and gave
her a running total on how each student was performing.
It was an incredible feature, says Givens, who knew
right away if she needed to review a lesson, based on how
her students had performed on the latest quiz.
Computerized testing is starting to catch on in class-
rooms across the country as teachers look for more efficient
ways to evaluate students. Some say their use of computer-
based quizzes or tests has saved them untold hours that
would otherwise have been spent grading by hand.
Yet a host of barriers exist that could prevent many
teachers from going down the path to computerized tests.
And even the teachers who are already using such tests
say the method has its limitations.
To begin with, many teachers dont know how to use
software to give tests and quizzes, says Rosemary Skeele,
an education professor at Seton Hall University, in South
Orange, N.J., who trains teachers to use technology in
their classrooms.
Educators also point out that computerized exams
could underestimate the knowledge of some students who
ByBy Michelle Galley
TheTeachersNewTestComputerized exams finding a niche
in classroom assessment
7/29/2019 Pencils Down - Technology's Answers to Testing.pdf
24/76
are not particularly skilled at using computers.Beyond those concerns, Skeele says most class-
rooms have only one or two computers, which is notenough to give every student a test. And in manycases, the computers that teachers use are so old that
they cannot support new test-writing programs.The systems in K-12 schools are just not sophisti-
cated enough, Skeele says.Those problems are not issues for Givens. Still, she
doesnt use computers to give regular exams.Computer access is not the reason: Givens district,
Virginias 42,000-student Henrico County publ icschools, supplies each student and teacher with his orher own laptop.
Training is not a problem, either: Each middle andhigh school in the district has a technology-supportperson on staff to train and assist teachers. This year,Givens fills that position
Recommended