Vol. 69, N. 4 pp. 401494,0200{3ot 5 ctol tChildren.

Altrnaate Assesm:ent;s: inRadnand Mat: Develoment andVlidation fr Sudent WithSig ficn t Disabilities.

GERALD TiNDAL

MARInL;EMCDON;:ALD

MARICfKTEOgSCO

AARON GLASGOW

PAT ALM4OND:

LUNDY CAAWFORDUJniesvity a olord

KEITH KIOLLrNBECK

Springfild&looDiszJricr, Oregogn

AoS*RAT Given the mandates ofMLE to in studnts with disabilities in lare-scal assess-ments, maststtes hae eiher adpd alternte standard or dled alternate assessments. In eithercase, it is u to undestand the students' erfmnance relative to the primary assessment program.And in both rses,the tehnical adqucy isgenera ssume ratier than spealy documented. Inthis sudy, we deveopda seriesfstanadie tass that can be cosidere as part of the same con-struct as oeationalied in the primarylargesca asseshent prgrqaM. We then arled studentperjrmance to ascertain reib ility and initialva id t In rei and h, teachers were trainedto administer the tasks andjud peofrrmance, provi system h i ctnal and evaluativeuses. The resus suprt the ial quy thalr e assessment.T 0_;:he statutory provisions on gen- ate accommodations and modifications in admin-

ea fstate- and districtwide assess- istration, if necessary; and (2) As appropriate themi ents from the IDEA State or LEA-(A) develops guidelines for theAmendments of 1997 included participation of children with disabilities in alter-the provision that "(1) Children nat assessfents for those chiliren who cannot

with disabilities are induded in rlgeeral aState and participate in State and districtwide assessmentdistrietwid assessment programs, with appropri- programs; (B) develops alternate assessments; and

cOpinalCi lrnn

:: : ::: :: S :::: :

4801

(C) beginning not later than Jtdly 1, 2000,t n cn-

ducts the alternate assessments" (§300.138; PartB) utilizing statistically sound regular and alter-

nate assessments (9300.139).While IDEA '97 stipulates participation,

the regulationls do nor provide "specific dfircirion

to states abouit what an alternate assessment is,what it should look like, or how it should bescored or reported, nor does it specify the type ornumber of alternate assessmnicrt participants"(Thompson & T1hurlow, 2000, p. 1). Accordingto Thompson andt Thurlow, a majority of states'alternate assessments are tied to their academiccontent standards and are designed as substitutesto the states' standard large-scale tests. Thompsonand Thiurlow also report that the inost commonalternate assessment approach uses portfolios re-flecting the collection of evidence as functionalindicators on the studelnt's progress toward statestandards. However, not all states have chosen totuse a portfolio-base as their alteinate assessment,and this study describes the results from a noni-portfolio approach.

First, we describe how one state developedits statewide alternate assessmnent system and thenpresent results from an initial validation study.This; state's alternate statewide assessment svstemin reading and math is considered aan extension of

the standard assessment ratther than an alternateassessment, primarily because of the tight rela-tionship betweetn it and the standard assessment.Thie measures are designed to reflect a develop-mental progression of skills wittin a continuumof the same constrtict. I'he purpose of the systemis to produce clear and meaningful data that canbe prescriptively used within the individualizededucation program (IEP) rather tban simply de-scriptrively applied to reporting perforimance aspart of an accountability svstemt. It also is to beu1sed: in conjunction with the general edutCation

assessmetnt system so that performnance on one is

related to performance on the other.

482

U ti lizing Deto's (1985) propositions, the

measures were designed to evaluate student

progress, interventions, and programs in the samemanner that the statewide assessment is used.Most important, the measures were designed to

be systemically.,valid (set Frederiksen & Collins,1 989) as a direct mneasutre of the constructs of in-terest and sensitive to the effects of instruction.Students' skills have been expressed in terms of

performance on tasks that can provide instruc-tional feedback. Ideally, systemic change in reach-

ing would positively affect perfoirance.

AL2TERN ATE ASSlESSMENT

SYSTEMS AND THE CANONS

OF VALIDITY

It is important that all large-scale assessments, in-cluding alternative assessments, reflect the tradi-tional canons of measurement standards, likecontent validity, concurrent criterion-reilated: va-lidity, and predictive criterion-related validity.Also, measurement systems should reffect Mes-

sick's (1988,1 ]994) four facets of validiry: (a) con-struct validity, (b) values implication6s, (c)

relevance and utility, and (d) social consequences,

which refer to hoth the justification f.or and thefunction of the test. Messick's validity criteria areused to characterize the quality of an assess-ibentwhether the test scores: accurately reflectthe knowledge, skills,' and/or abilities the test is

intended to measure. "Special validity dispensa-tions" (Messick, 1995. p. 5-8) are unwarranted tolegitimize all the various alternate assessment ap-proaches, and there should be no diffrentiationof validity standards betveen newer types of as-sessments and other types of older, traditionaltests.

Accordingy, evaluations of all tests need touse similar evidential and consequential validitycriteria because validity, reliability, comparability,

and fairness are in effect "social valuesjthathiavemeaning and force outside of assessismentwhereverevaluative judgnments and decisions are made"(Messick, 1994, p. 13). Basic to good assessmentIs the notion that results represent importantknowledge and/or capabilities, broader thaan the

specific tasks that happen to be chosen for assess-meint. Test performance must generalize to a

5'urorner 2003

larger domain of knowledge and/or skills and thusenable ug to make accurate inferences about stu-

dent' capabilities and accomplishments.Ain important component of Messick's con-

ception of validity is construct underrepresenta-tion, and&construct irrelevant variance. The formerterm refesJ6to assessments that poorly reflect tthedepth andbreadth of knowled and skills, andthe latter term reflects assessments that include

the confbunding of skills and knowledge otherihan the constucti of interest. In this assessmIent

we attermpt toiadeuately reflect functional skills

of importance whilie avoiding the influenkcet of

other skillsithat are not, part of the construct. We,therefore, present af series of measures to CApture

performance on Icmompont Skills 'instead of try-

ingto assess$:complex knowledge or skills that

would use ta singl, o lexitask. Many Students

with significant disabilities need carefully con-

structd tasksi that c capture a range of behav-

iors (Fuchs et al., 1994). In our system, these

coimponent skills are assessed using tasks that

elicit specific behaviors. We believe that complex

performances often.reflect construct undetrepre-

sentation hecause few tasks are used tto make

staterments about a student's proficiency in an aca-

demic area. WOdelso reason that complex perfir-mances migth entail construct irrelevant variance,

given:i the confluence of ma:ny separate skills

needed foi anysingle performance.In these exteinded measures, replicahility

and generalizability also mrust be considered,

not only in detettriining the boundaries of score

meaning as it pertains to the relvance, utilityl

and social consequences of interpretations, but

also as it relates to the conisistency of score

meaning. Low scores on the extended rneasures

must:occuribecause of the lack of student's com-

petence, niot: $because part of the :focal construct

is absent or because the assessment "contains

somnething irrelevant that interferes with the af-

fkectd students' .idemonstration of competence"

(Messick, 1994, p. 21). Finaliy, the extended

measures must be constrnct-driven rather than

task-drivenr. Wedefine tasks within a tniverse in

which a systematic sampling plan can be used to

cteate alternate forms. In this vwy, we adopti an

assessment systemi to documeint student perfor-

mance an.Ld i :progress with both' ditensions ori-

ented toward the same construtlct.

RESEARCH ON PORTFOLIOS

AS AN ALTER?NATE ASSESSMENT

As stated earlier, many states have implemented

alternate assessment systems that do not directlymeasure contstructs of the standards for which

their general education assessmrent system has

been: created. And in this process, many states also

have. relied On portfolio assessments, which have

been difficult to itmplemenet with any level of

technical adequacy. Any assertions proclaimed by

portfolio assessment have been rtantamount to a

claim of construct validity and need to be sup-

:ported by empirical evidence of construct valid-

iWty (Messick, 1994, p. 21). The major problem

manry be that while portfolios contain relevant

products, they. typically have ignored the process

and therefore have remained ill defined as mea-

sure:ment constructs.

Research shows that portfolios as part of alargescale assessment program have posed signifi-

?cant validity problems. Vermont's early experi-mients with portfolios and Arizona's recent forayinto portfolios both ended disastrously. Bothstates suffered from poor rater reliability problemsthat prevented tie public release of the assessmentresults (Koret, McCaffrey, Klein, Bell, & Stecher,1:993).:: :

ONE STATE'S EXTENDED

ASSEStSMENTS IN READING

A:ND MATHEMATICS

As noted earlier, in this large-scale assessment pro-gram, alternate academic assessments are consid-ered an extension of the standard assessment. Thegenesis :of this .syst:em is curricuIlum-based mea-suretment (CBM.X The measures require multiplealternate forms using CBM technology drawn onaRfoundation of time series data. The technical ad-

483Excephd~ra Ck"4dr

equacy and psychometritc properties of CBM havebeen extensively represented in the professionalli.teratutire o afexp:erime-ntal investigations (e.g.,Bradley-Kiug, Shapiro>, LIutz, & DuPaul, 19998;Fuchs, Fuchs, Hamlett, & Stecher, 1990; Fuchs etal., 1994; Hintze & Shapiro, 1997; Hovell, Fox,& Morehead, 1993; Marston, 1989; Nolet &McLaugh]in, 1997; :1'indal, 1998- Wesson &King, 1992; Yell, Deno, & Marston, 1992).

According to Deno (198 5), CBM is an as-semfblage of , procedures: for devising: assessmentsby sampling thie domaini in question, administer-ing and scoriqg those assessmients, and us;ing thedata to help rnake anid evaluate instructional deciLsions. CBM quantifies student performnance inthe basic academic sktill areas and can be used asani index of student progress over tiie because itis sensitive to a student's academic growrh (Fuchs,Fuchs, Bishop, & Hlamlett, 1992; Hlartman &Fuller, 1997; Swain & Allinder, 1996). It also cancapture both the range and depth of studenftachievement (Nolet, 1992).

An important compolent of this study wasthe development of generalized academic skill 'in-dicatorsthatf would not be tiedto a ispecific cur-ricultumt as reported by4 mtian y CBM' researchers(Bradley-Klug et al, 1998; Hartran & Fuller,1997; Hintze & Shapiro, 1997; Tindal, Flick, &Cole, 1993). In vindication of general outcomemeasures u sinig CBM technology, Mehtens anedClarizio (1993) declared that education suiffersfrom a faulty assumption that all assessmentstasks need to be sampled directly from some in-structional curriculum to be tho t ofas valid.In the encd, we wanted to avoid tising measurestied to a specific curriculum which would limit

generalizabiliry and, thus, limit score inferetnces tothe larger domain (Tindal et al., 1993).

E XTEN DED mEAsUREsI

S U BTESTS

IBased upon CBM reseairch, the extended mea-sures were designed to assess skills' that are m,oreunitary in the construct they purport to measureinstead of measuring complex, mutiskill perfor-mnmces. We developed the tasks to reflect moresingular dimensionts using the logic of Kane,Crooks, and Cohen (1999): First links are estab-

fished between obscrvations and observed scores(requiring adequate control of administration andscoring procdu-res); ithentinfierences canii be mrtadebetween the'observed score and the tuniverse ofscores (providing generalizabiliry); finally, infer-ences can be made to tie range of possible taskswithirn the sathe construot (allowing extrapola-tion).

For us, this specificity avoide construct ir-relevant variance. For example, instead of mathmfultip1l-choie tests that required reading as wellas math skill to perform successfiully, th extendedmeasures sampled mnat skills albne. Or, insteadof math open-endiedd measutres that required skillin nmathematics and writing, the measUres assessedmnath skills only. In both computation' and open-ended problemis, the measures utilized Iultiple

forms of access (pointing, speak in1g, arn writing).Similar requirements wvere made of reading anfdwriting measures.

We also wanted to avoid conistruct under-representatiot by having many different taskswithin each of oulr constructs that sampled differ-ent components of each academic skill. Specifi-cally, the reading subtests ranged from naming orpointing to pictures, to blending sounds, to read-ing sentences, to retelling stories as listening com-prehension. The math subtest extended fronmcopyinIg numbers, to tellingjtime, to math c:on-cepts like discrifiinating differences to mixedcomptitationls.

METHOD

PAR TI CI PA NTS

The study took place in one istate in the PacificNorthwest. Thirty-six school dlistricts within 11Educatiorlfl Service Districts from all regions ofthe state were included.

W'achers. A sample of 131 teachers partici-pated in this stutdy.. Teachers were eected to par-ticipate by district administrators and regionalrepresentatives who were chosern to supervisetraining, test adininstration, and data collectionthroughout the study.

Snedents. Each teacher assessed 1 to 5 sttu-dents. A total of 437 students in Gres Kinder-garten through postsecondary d school were

484Sunopier 20003

TABLE IDisabilties OfParticipantsPrimary Disability

Mcntal retardationAutism

Specific learning disabilitiesOrthopedic impairmentOther health impairmentsSpeech or language impairmentsVisual impairments (or blindness)

Traumatic brain inju:ry

n %

249 57

73 17

16

25

21

4

65

12 3

5 1

3

Lniotional disturbance 2 .5Hearing impairments (or deafness) 1 .2Unknown 30 7

selected using the following three criteria: (a) stu-dent was exempt from the standard statewide as-sessment; (b) student had been d oiagnsed with amoderate to severe disability; and (c) student par-ticipated in a functional daily living skills curricu-lutm. Students' average age was 15 years with aminimum of 5 and a maximum of 2].

All 437 students received special educationservices. Sevenity-five of the 437 students were un-able to be assessed due to extremely low academicskills, resuilting in 112 students in K-3 (26/% of thesample), 77 students in Grades 4-6 (180/o of thesample), 83 students in Grades 7-8 (19% of thesample), 54 students in Grades 9-10 (12% of thesample), and 41 in Grades 11-12 (9% of the sam-ple); 32 students were postsecondary, and 38 stu-dents had no grade reported. Forty-one percent ofthe students were female and 56% were male. Eth-nic make-up included: 80% White, 2% AfricanAmerican, 1% Asian, I% American Indian, 1%Multi-Racial, and 8% Unknown. See Table 1 fordisability information of participating students.

MEA S URES

This report includes only data from reading andmath, the two areas with multiple-choice tests inthe standard assessment program. and most dis-similar to the extended measures. From each con-tent domain, we specified tasks that measuredknowledge of "basic skills" and applications. Wedefined a range of tasks that were sensitive to sub-

tie differenices in skill levels but also allowed a va-riety of response options, particularly given therange of disabilities within the population.HIence, items were included that could be adrmin-istered or responded to using a variety of formats(see Scoring section that follows). We developedtasks using the logic of CBM and as logical pre-requisites for success in meeting the standards atthe first benchmark. During the process of imple-mentation, new state standards were adopted forstudents in Grades K-2 and have been used toback-map these measures.

For example, in reading, phonemic aware-ness is now a standard in Kindergarten with out-comes such as listening to words spoken andidentifying the beginning and ending sounds, seg-menting single-syllable spoken words, orallyblenditng two to three spoken sounds into recog-nizable words, and so forth. Other standards ad-dress recognition of sight words, and learning,using and understanding new vocabulary. Inmath, Kindergarten students are now expected toread, write, order, and identify whole numbersless than 100. We have several tasks that assessthis standard. Other standards also can be applied(dealing with concepts, m-oney, counting, adding,etc.) to the tasks in this alternate assessment.

We used two pilot studies to guide frame-work specification and test development, eachconducted with students who had moderate tosevere cognitive and physical disabilities. 'I'he

Exceptitmal Children 4ftS

intent of thei first pilot study was to~ determine an

apropriate ranIge of test content, to detail thetype an fomats of Test itms an to identif

the number of tasks to be included in each subdo-

main. The second pilot studywas conduted to

refinie the procedue usedIfo adminstein and

scoring the test. Particular cnideration was

giveni to developing: a conisolidated, y~et compre-

henisive, system of test adminhistration and scoring

that facilitated a Controlled andmininmally dis-

tracting test administration environmient. Ulti-

mately, these pilot studies provided us with

valuable evidnce''regadin itemn sets that couild

maintain theirintegrit even ifatraeamns

trationt and response formis were needed due to

individual student requirements (needs). The re-

sult of the Pilot studies was a refiementr Of a Pre-

assessment interve and a basic skills assessment

consistig of readin,wiigadmthc po

nients. A discussio'o theseopnnsflos

n eadng studlents prorse hog hfollowinigtasks: a Namring Pictures (b) Namring

or Pointing to Letters, (c)Blendin ~Sounids,kdReading Wrds, (e ReadIng Names,6 (f) Reading

Sentenices, (g) Readintg Text rally., and: (h)

Reeln torties as R ead ing or Listening.IC.omTpre-

hesion. I mathematics,~ studenlts proressed

through:: the followinfg tasks: (a) Naming N:um-

bers, (b) Pointing to Numbers (c opying

Numnbers,(d Counting on: Dictatrion, e rtn

Nqumerals, (f) Writig Nmerals, (g): NamingShps, ()Tling Time, (i)amin :and Count

Ing Money, ()Mnpltoof Objects uisingMat Concepts )Mandpu(k)tioqp to_~ I bt

content areas, Ithe skills are anch:Ored:to newly de-

vlped, state standards establishd for stdensi

Grades K-2.

PROCLOUES

Al f h dmtinistration~ and scorinig:procedure

wereiidude in atesting kitthat wasdeveloped

by the a uithors., The~: tasks.werex administered: slo

tha-t the most:basic skill wa pente frt, it

each successive task presented on: texbasis: of in-

creasing:'difficult or7coomlexity.AdmWIinistrin:. The~ test materias in-

chiedal o th iem rquied4fo boh6theh ad-

minnistration. and. scoringof th tstan werecompied ino a hree-ing inder. In additioni,

the folders .were arrange6d by dbimain, with each

directlionsl, admin6istration scripts,arepresenta-dtioni of each stuidentak,lsorin instuctos

and suetrcr hes h folde als 0con-

taineid testing materiafs :as well as student re-,

sponse formis. In addiftion, th : foldeCm contained~

standardiz.ed ~p.rocedures: :that provided th tesadinsrao witrules o determining, basa

and ceiling levels, as well as, options, fo:r docu-

mntigi floarmtchanges.Scoing Weprovided flexibifity in scoring

to avoid ITirrlvant variance frto:mtw 6sourCes.First, teachers could avoid:Admin'~rr tak if

they bhelived he suden could SUCCessfullycoplt alitems in the jas:k:~They simply

marked the task NA-P foir Not Administered-

Proficeiet. This featurealoe tecers to foIcustheir Assessment on the key level:of'difficultymobst relevant to the student, a: particual Im-

portat itaspett cosdewhereting stdet

who oten exhibit ohritreigbhvosSecond teachers could::avoid adminimstering tasks:

toa stud ent Who is def. Inour:Sequeneotsks, student col e diistered thoeta

appare laer nd efected: mnore complex pr

frmances: revealn avneskill aniece'e a

highr sorewitout: exhU,ibting prevous. compoI-nentf performance efletin indvidUalskills. In.

this:eXamfpke, a: stud"ent who isdefcud: read

pssae and retell stories to receive: a very high

scor,ewithout ever having taken prevousc -

ponient s kil Tasks.Each :task was scored: bbth quantitatively

and ualtatvel. Te qantitatie scoring al-7

lwdfrSmall: units of progress to be d6cu-

mcntaed ie.letters: and soun d s correct in,

rdig or: correct digits innmath) Foreach sub-w

dmain,: he quantitative sc0ores Aeliten trans-

formd ito ualtative scores~ torfec I lbal

estimate of, prfoane Stdents:in a graduate

special edcation program were trainted4 'to eiw

the prtocols and mfake aglba jugentabout

proficienc (see Tbe )

Our oalforeachtraimnin :session was:to ensurethat participants :were eCxtremely famlia with

tes mteias,procentin their administration

~and ompetent: in:scoring theacademic behav-

406 Swn~~~~~~~~~~~~0mea200-406

TA8LE 2Fwrnspk of e Erend,-d Reacli g ai-nd Math &wg Sheet

Eash task il ioiatesds oin f 1- rtyln T wits sthe nuier .,f psfoinnt possible paresri'.s soilownig t-isZ k A- nsu.I are the studcs on thrse rublric whstre the nuss pusimsr :s.

T ask I. Nainie Nuijers (1(s)s | 0 _ Po~~~~~~inting Tti Numbrs (S) 0 V5

T.jsk L Naming Picrurtc Pwnsr TAsk 2. ' .- tiimh (8) quEtrsSlustinig Tn ivVscmre=sI 6 NasPisisg i Task 5. Namsing Shapes tflu 1

Poinpig To Shlape (8): triis'rask 4. WritiissNui:t aiNi : Am )

t' 2. NAmia4- kf:rs P(lfttfAig : * : :Ike Li ir(16) N ask 3I (sus1; S inag Oil t S;.1

1.sIk 3. Reafing Words Poins ing Hsk-Bs isissbers in aT * (- I >Tsk, h. Mnanl ipulating Math parse 2.5

t ask 4. PhtiniS legmstatic Lettcr S£SiIIIS T-mb 6. Tellinrg Tim; (11 N)(22) 2 s5 TLk 7a. NainIld4g Coisns ansd Bills NLirn in 3

Tas B S. lending I Bf aiding a8Sort& (22) Salrinds

3 riii3lSt3Mi8 O .5

2Task '7b. G ot uing Menxy i12)

: ... :

t .S (5. rixascug 'NZanes i'f3

I ask 7 Reading Stnettanss (14)

Task S. Rv.adissg Fret Orally(4pra wvith ('WP Z)

Tas&k 10. RerAissg Swrnie:sl.ist;ning ssmprt-AcrsiounLevel I -22, lAwl- 2 44,IUVd I i- 6

'Task 9. Reelling Swories: ReadingCoissspreltesisalrnLCel I - 22, Level 2 = 44,Cxvct 3 -; 66

SrsrniceReading

---Sz s_- r s- _-

OrTA Re,adingF;Tuienta

1 IsstcraingCopr"be sd

Cssssspseliessd

3.5

4

4.5

5

Task #c-d. iManipiil.ttstig withMaill

(Sucejxs (6)

(on)a. Ad&bstsin fPscusb. SWnracsstime FsLmT

- s ac7N

'I'ask IO, Mixca: Addin1C. MoZtip(18)

C Cosmspustation

operatitntWith

mltani,fillaxiv,es

Sysnallis.

operatioens(timed math

k"s)

erSIgh. Sti,sracttng (I i) ilvngd. &isiding als

mans

pcsatisisisaddlinbtr;actjgbrpling.

d disiding

4

4.5

iors assessed. The agenda for each training sessionwas arranged according to academic skill areas,with the Reading, Writing, and Math tests intro-duced and modeled separately. We first provided athorough description of test materials, thenshowed a videotape to model test administrationand scoring, and finally allocated timne for teach-ers to practice administering the test to each otherwhile trainers observed. A sample notebook oftesting materials was provided to teachers. Thishands-on approach ensured teachers' familiaritywith all of the materials.

We believe teachers left the training wellprepared. 'rotal time for each training session wasapproximately 6 hr wiith somne variations in timedepending on the size of each group, the qites-tions asked during training, and participants'prior knowledge related to administration of basicskills tests. The videotape provided a clear modelof how test conditions or test format might be al-tered to capture as much academic behavior aspossible fromr each participating student. Fourdifferent students, each with varying types of dis-ability and levels of proficiency were shown on

Evepti,onal Children4

I __- -W ___ 2_

4S7

the, videotape, as well as fourf diffteirkn test admin-

istrators, each With his or -her owni testing style.

Fial,pacticing test administratio. witha.pee

affor,ded teachers ~the opportunity. to:bcm

comforal wih heflsharP n maiua-~

tives included in the sudent packets, as wel as,

learn how to man4ipulate. test materil .wh, ithon

hadwhile, scoring stud en-t responses: with' the

other hanid.We condlucted four trainn se$ssons en-

compassing geographiclly diverse: regi ons. All~

training was conduted in-March, with sessionsbeing ateddb rusrnigfro 12 to 42

teachers.Regional represent!tives :who' hd been

tritned prior to our: workshops were available: to

answer any questin ha ecer 4ftsrthi

training, whenf they were actuallyadministnering

and scoring tests lin thelir buidns Theywr

also responisible forensring, tha test:rmaterils

were completed And re turned to tersacsite.

Perfrmance was~ analyae Stparately for i rdIng

And mnath academic6 skill areas tin two ways: (a

using quantitative indices of performancel forsub-tasks, nd (b) sing Summryjd:gemensf over-

tall a prfiinc (e Tb 2freaInganIt

raigseets respectiel).Th totalroup of 0 Stu--

dents In readinwa36 and 39 stdet in

math.

in readinig, the same grouips of students were an-

alyxd to ascertaini inital vialidity. The largest

group were students~ with miental retaid4tibn (n

249) follwed by 73 students with auitism, 25

stuidents' with otpeic impairments, 21 stu-

dentsl with other ~healthipimnslnstudnts ithspecficlearning :disaibilities. .~The

and Finally, theI hgest: reading performaace:was

frstudent wilth speciflic learning diab'itltes.The iffeence beweenstudnts ithmental

retardation anid with:autismn were::less:distin-

guishable: in: the early task tha thy were in th

late6r ones. :With namning letters, bllending

sounds, and reading words, both :grodups of stu-

dents appearebd to have at least some miniimal

proficiency.:: HoWever Y:in reading contiguous

tex, ~more giroup diffreces appeared in their

prfrAne In4 the- most de.veloped tasks, wvhich

IIinvolve edngtx n rltint the m:eaning

of ~the story students froff all three groups were

relatively poor perfobrmers. In may ~of the mndi-_

vidual subtas~ks,~ co:nsiderable :variation ~can be

seen, reflecting the fact that some students per-form very well andother viery poorly. The stani-

datd deviatioin frequently~ is: greater than the

Trained students with mnAste's degrees (8

total studlents ,with: two pairs Ieach ~in -readitig andmiath) prvdedindividuialjugetofac

studjenit's overall peformnce taking into ac-

count ithe range of tasks. completed' an;d the lev-r

els of performance~ on eCach, one. Al :studets bad

been partof one of our regional training. Thie re-

liability of ratings vwa qulite high-~ Fully half of

al h ugmet were in coplteagreement

and the other half were within onte4half point.

Only 17 judgments of: t he 362 excteded: one

point, wi'th 161 ibeing exac marchesj 162 being

Off by one-half podint, And 21 off by :one point.Por eah of the: ratings the number. of, studeats

scoring at thAt leve i's reported. The actual dis-tribution o0f scores(fo both judge)wabimIodal

anid plaryurric with a large numnber anid per-

cenitage of ~students rated 25. Pars ofudges

r4ate U*lrenumbe of 4students~ low on the: scale

as well as high.The,highestlevell of peforac

reflecting the mIost toImplex conistruct: (comfpre-

henision) ~was notabl l6 ow with few ~stuidenIts

We tested A tota of 359 studenits ~in miath, wt

the maJority of the suetshavn etal retar-

and: studet wtlarigdsblie( 16)

The oter two prominn grut,pG)s::were- students

with orthope dic impairment: andb Othe health nim-pairmens Whl tdnt ihohr dsab ilities

weretested,j werepor o6nlyont thesexthree grouIpsin:ourpiniia validtin

We fundstuentsdisribtedthemselves

differentl according to: thieirprimar dis,abflity~

trndents :with :autism::perfo.rmed the &oest~ fol

lowetd bysudents with mnental etardation, and

488 ~~~~~~~~~~~~~~~~~~Sammer2003408

TABLE 3Reading fvr itudents Wit/7 Mental Retardation (MIR), Autsm, and Lea rnie Disabilitries (LD)

Task and Number eflItem.r M SD 1I SD M SD

Task 1: Naming Letters (16)

'l'ask 1: Pointinig to T esters (8)

Task 2: Nansiing Pictures (16)

I'ask 2: Pointing to Pkictuies (8)

Task 3: Blending Sounds (22)

Task 4: Words (I 6)

Task 4: Pointing to Words (a)

'Task 5: Names (min. of 91

Task 5: Names (nimm. of 9)

Task 6: Sentence, (1/4)

Task 7: Text Orally, Kevin's Stoty (cw)

'Task 7: Read 'I'ext, Kevini's Story (iw)

Task 7: Read Text, Sue & Peg (cw)

T'ask 7: Read 'c'ext, Suie & Peg (iw)

Task 7: Read Text, TV (cw)

Task 7: Read 'Fexr, TV (iw)

Task 8: Read Comprehend, Kevin's story

Task R: Read Comprehend, Sue & Peg

Task 8: Read Comprehensions, TV

Task 9: Listen Comprehend, Kevin's srory

Task 9: Listen Conspeehend, Sue & Peg

Task 9: l isten Comprehend, TV

Test total points student carned

Average of Qualitative Score-Judge 1

Average ol'Qualitative Sroie- Jusdge 2

A46te, cw=correctly read words, iw=ilinoirr.ly reid wordsMR (n-=249),Auitisrin l=73). LD (n=16)

then those with specific learning disabilities per-formed the highest. Students with mental retarda-tion, completed about half the items correctly,whereas those with specific learning disabilitieswere completely correct in their responses in alltasks except the computation problems, an area inwhich students from all disabilities performedquite poorly. Students with autism performed notthat much differently than those with mental re-tardation in the early tasks (naming, copying,counting, and writing nuinerals) but did differ inmost of the later tasks. Again, a large amount ofvariation is present, reflecting considerable diver-sity among the students on many of the tasks (seeT'able 5). As in reading, for many of the tasks the

standard deviation is greater than the mean.

The data on the qualitative judgments re-flected consistency in overall value as well as inthe distribution of scores. Of the 359 ratings, 95were exact matches, another 135 were within one-half point, 89 were within one point, and 31 werewithin one and a half points. Only 9 of the totalsamnple were 2 or more points apart. For each ofthe ratings, the number of students scoring at thatlevel also is reported. For both juudges, the distrib-utions were slightly negatively skewed with a largegroup of students achieving values of nearly 5points and exceeding 5 points. With the last taskinvolving symbolic problems, however, a largedrop-off occurred with few students performingat that level (see Table 6).

Exeeptional Children

11.30

i .31

11.31

10.60

6.82

1.02

6.07

14.11

5.97

9.55

1.91

10.00

1.22

11.89

(737

1.44

1 * 70

0.91

2,11

1.25

0.36

63.44

2.67

2.86

9.89

1.63

.18

1.88

7.66

5.80

1.43

5.48

12.61

5.04

6.27

1.18

8.16

1.16

5.32

f.21

1.00

1.59

0.2-,7

1.23

0.45

0.48

52.13

2.28

7..40

6.49

2.66

5..39

2.60

9.18

6.97

2.20

6.90

6.51

5.98

23.57

5.09

29.31

1 117

36.01

1.60

3.49

4.85

3.32

3.62

4,13

1.65

39.93

1.66

1.74

7'.14

3.08

6.42

3.00

9.17

6.80

2.49

6.92

7.25

6.10

16.21

3.37

22.54

8.02

21.84

0.99

3.26

'1.84

1.55

2.79

2.04

2.90

41.25

1.72

1.79

15.88

100

14.00

0.00

20.00

14.88

0.00

10.50

13.19

12.94

32.50

2.81

15.691

0.38

11.06

0.00

6.25

4.19

1.25

0.00

3.94

0.00

104.81

4.6.3

4.75

0.50

2.73

2.13

0.00

3.14

2.53

0.00

5.72

4,37

3.00

26.69

4.94

38.44

1 02

30.26

0.00

5.59

8.29

2.98i

0.00

8.71

0.00

24.01

0.*7

0.88

====

489

TABLE 4 : I : :

ScorVe X2H DitT~tosfrTwaJ4tsi Eatin~Raii::jodg I Rativg Cnt

0.0 I 27

0.5 35: : it:0f23 :i

JW2&4*0

0.5

1.U

40 1.5

40 2.04

29 2.5

23 3.0

23 3.525 4.0

32 4.5

40 0

1 I8 5-5 0:: 0 : :t: 0;0I 0 60 ;000 t~~~~~C) 0 0:

1 .

3 6 Mi i::::0362 t:;:ff : 0:t T tal

TAKE 5

Mek orS e~zoe McudRaw uton( I?, ear, ndLarin

aask 1: Pointinig to Nw bes (8) 16

fask 2: Copying N nibetr (24)

'Iask 3: Counting ot Dictation (16)

a.sk 4: Writing NuMervs (I18)

T'ask 5: Naniinig Sbaes (8)

Task 6.:'tlling1'rilem (12)

T<sik 74:Narming Coins (8)

TaW k7a: Poiuting to Coins (4)

1.isk- 7b:Cou 4in Money (12)

J 'asks :gab: Me1anipulire ,Mathi (.on.epwts (6)o

T 1Sks&hd: f6.Manipulu a th Concets ()

abisk 9a: Computation (22)

Task gb: Colmputation (16)

le t oal ons t d i a

T eart ood poiinsx lts posaih k i:; 0i: : ::;

T ear p erc- ent corrf e ct:0 0 0 0 :::Tt:::tt:tttt:t ; F t :08:t: t

Ai eoflQii.iiitacive & ore JUdge 5

Atvs: g of(2 Q alkativt Sco Jud-tg: 6

Arote: MR kni-249) Autism 73), LDI) (z 10)

12-35

11.13

16.24

101.34:

II 101.0700

4.69

4.16

4.A0

3.33

4,13

02.40008 79

2.89

50.53

3.4)7

: 3.57:

bi4li vi (LDSD m 2D M SD

:.29 10.6g 77:7 16.00 000

2.59 157 3.415 o000 0.00

10f.4( 0 01254 0::;$ 101.0120000 t$ 23.$56i00 0 .0:073

6.1 955 7.0 ~ 1604 0.00

0000008.0300 i 9.090 :8589 I7.t69 0.70)

221 4.21 2.72 6.31 1.20

4.35 3.23 3.95 .81 : 3 58

3.14 3.75 3.42 7.9 0T25

lbZ (0.64 1.33 0.25 1.00

4.56 2.02 3.80 9.88 2,.5

L2U2 3-21 2.51 .000 0m00

18.80 1.14 1.93 4.13 1 A.4

9.25 56 5) 8.07 20.44 2.7ll

t0005.21 152 3 63A3 10.56 5 5.39

51.18 6925 501.13: 14756 13.42

.38.71 14.500 47.62, 164.25 1.00

1000t.67: ;0 t 2.98 1 89 5.53 0.39

4 0Su2mIer. 2003

1.5

: 2.0

2.5

3.0

3.5

405

5.0

5.56.0

iissing

ITotA

Co00unt:

26

35

35

27

23

22

17

355

26

40

5

8

362

ti:;:iR::::::i:::: ::: ::;;:::;:;:;:;:Q:;ia-:4:;:::::;::;:i#:::::::::#:::::: ::: :::::: :::::::::!:::::::::::: !::::!:!:: ::::::!::!::: :::

-- ; ; ; : ; # ; F ; ; L L ; ; : ; S ; ; ;

:::::::::: ::

.....! :# f .:::::::: i:: i::::: :i:! !: :E: P:

..... I..!!:!::: S ::!r?

iA:g:::E:#:4:! :::;:;:4::i;::::::::i: ::: ::: :::i:: ::

490

DISCUSSION

The main findings from this study focus on scal-ing performance on reading and math tasks thatis consistent with disability and establishing relia-bility of rating performance on a qualitative scale.We found students distributed themselves on ourperformance tasks in roughly the manner onewould expect given their disability. For example,students with mental retardation performed lesswell than students with learning disabilities. Thisfinding may reflect the fact that students withmental retardation spend mLuch less time in acad-emically focused classroom settings than studentswith learning disabilities. Probably a more impor-tant lesson learned is that many students with sig-nificant disabilities are learning the symbolsystems of our language. Therefore, their instruc-tional programs need to provide them more directinstruction of these skills and our measurementsystems need to reflect performance and progressin their learning.

We have argued for the use of well-stan-dardized tasks that (a) fit the conistructs of basicskills (reading and math in this article), (b) arealigned with the standards used to form the large-scale assessments, and (c) have documented tech-nical adequacy. The initial data we have reportedfocus primarily upon reliability and construct va-lidity: Students performed on a set of progres-sively difficult tasks to the point that it was cleartheir performance could generalize to a larger do-main and eventually be connected to the stan-dards themselves. We focused on two dimensionsof construct validity in particular: construct un-derrepresentation and construct irrelevant vari-ance (Messick, 1994).

Using a progression of tasks, we attemptedto sample complex behaviors that did not uncder-represent the construct; and by using well-de-signed sampling plans and task formats, weattempted to avoid many other nuisance influ-ences on performnance. Finally, using the logic ofKane et al. (1999) in constructing tasks, wewanted to generalize from observations and ob-served scores to a universe of scores. With partialscoring and trained testers, we arranged the tasksinto a progression, allowing us to infer the level ofa student's skill within a construct like reading ormath, providing us a means to extrapolate. All of

these components are likely important in makinginferences from observed scores to generalizationsand inferences in targeted measurement domainsused to document performancte proficiencies.

Our approach contrasts with the traditionalalternate assessinents based on portfolios in struc-turing the domain and the format of the assess-ment administration and scoring. Given thereliability problems reported by Koretz et al.(1993), we focused on products, trying to rmini-mize the problems with process. Furthermore, inthe development of the tasks, as well as their ad-ministration and scoring, we emphasized explicit-ness, resulting in relatively consistent judgmentsin performance across raters. It is likely that therelatively high reliability of our outcomes is be-cause of this explicitness. At the same time, arange of administration formats was availablewhile controlling the content that was sampled.For example, in reading, students unable to pro-nounce the letter or word were allowed to pointto it (the letter or word) after it had been read tothem. For a student with a speech-language imn-pediment, this flexihility allowed us to documentwhether or not the student 'understood" thegrapheme-phoneme relationship.

In the eid, we argue for the assessment ofstudents with significant disabilities so that per-formance and progress can be documented ontasks reflecting coinstruct validity. However, theassessments must be related to the standards, notunderrepresent the skill, aind reflect minimal irrel-evant variance. Although it makes little sense toadminister the standard test, the standards them-selves are important for them. Students with sig-nificant disabbilities are learning to read andcompute. Tfhey, therefore, should be given appro-priate assessments that can document this perfor-mance. Furthermore, this assessment should besensitive to their instructional level, so that arange of tasks are present allowing them to per-forrm now as well as 'grow" into more complexperformances later, as they receive more instruc-tion. This range of tasks should allow teachers tosample their performance formatively so they cansee them progress We believe the assessments wehave described in this article provide the rneansfor accomplishing this outcome and can be usedto complement large-scale assessment systems.

Exceptiona1 Chiklren 49t

1.5

-.5

3.0

3,5

4.0

&4:e.lu'mArW200492:

working on the development of systems for pro-fessional development. Whether we inicorporatethis training into our preservice programs orembed it in staff developmnent for practicingteachers, any successful use of the system dependson qualified teachers. Specific strategies need tobe available for helping teachers understand howto ensure student's performance is well docu-mented, to utse the information diagnostically inbuilding instructional programs, and to evaluateand report on student outcomes.

These three issues (a part-whole focus onbehavior, multiple measures that establish rela-tionships, and training for decision making) allconverge to determine success at the systems level.The practical implications are that large-scale as-sessment programs need to explicitly address allthree issues to successfully support teachers. Al-though we have reported on one component, it isclear that in this state, much more developmentneeds to take place for the system to effectivelyserve teachers and students. A long-term focus isneeded, therefore, in which state departments al-locate their resources for such supports. Then thefield can more appropriately devote attention notjust to simple participation of students with sig-nificant disabilities in large-scale testing pro-grams, but to the improvement of theirproficiencies and development of effective pro-grams.

RE FE R EN C ES

Bradley-Klug, K. L., Shapiro, E. S., Lutz, J. G., & Du-Paul, G. J. (1998). Evaluation of oral reading rate as acurriculum-based measure within literature-based cur-riculum. Journal of School Psycholog 36, 183-197.

Deno, S. L. (1985). Curriculum-based measurement:The emerging alternative. Exceptional Children, 52,219-232.

Frederiksen, J. R., & Collins, A. (1989). A systems ap-proach to educational testing. Fducational Researcher,

18(9), 27-32.

Fuchs, L. S., Fuchs, D., Bishop, N., & Hamlett, C. L.(1992). Classwide decision-making strategies with cur-riculum-based measurement. Diagnostique, 18(1), 39-

Fuchs, L. S., Fuchs, D., I lamlett, C. L., & Stecher, PM. (1990). 'IThe role of skills analysis in curriculum-based measurement in math. School Psychology Review,19(l), 6-22.

Fuchs, L. S., Fuchs, D., Hamlett, C. L., Thompsoti, A.,Roberts, PR 1., Kubek, P., & Stecher, P. M. (1994).Technical features of a niathematics concepts and appli-catiolns curriculum-based measurement system. Diag-nostique, 19(4), 23-49.

Hartman, J. M., & Fuller, M. L. (1997). The develop-ment of curriculunm-based measurement norms in liter-ature-based dassroonis. [Journal of School Psychology 35,377-389.

Hintze, J. M., & Shapiro, E. S. (1997). Currictulum-basedmeasurement and literature-based reading: Is curriculurn-based measurement meering the needs of changing read-ing curricula? Journal ofSchoolPsychologo 35, 351-375.

Howell, K. W, Fox, S. L., & Morehead, M. K. (1993).('urriczlum-b asedassessment. Teachingand decision mak-ing (2nd ed.). Pacific Grove, CA: Brooks/Cole.-

Individuals with Disabilities Education Act Amend-nients of 1997, 20 U.S.C. § 1400 et. seq. (ERIC Docu-mnent Reproduction Service No. ED 412 721)

Kane, M., Crooks, T., & Cohen, A. (1999). Validatingmeasures of performance. Educational Measurement: Is-sues and Practice, 1842), 5-17,

Koretz, D., McCaffrey, D., Klein, S., Bell, R., &Stecher, B. (1993). 7Xe reliability of'scoresfirom the 1992Vermont portfolio assessment program. (CSE TechnicalReport 355.) Los Angeles: National Center for Re-searc on iEvaluation, Sta rs and Student Testing.

Marston, D. B. (1989). A curriculum-based rneasure-ment approach to assessment: What it is and why do it.In M. R. Shinn (Ed.), Curriculum-based measurement:Assessing special children (pp. 18-78). New York: Guil-ford.'

Mehrens, W A., & Cl-arizio, H. F. (1993). Curriculum-based measurement: Consequential and psychometricconsiderations. Psychology in the Schools, 30, 241-254.

Messick, S. (1988). The once and future issues of valid-ity: Assessing the meaning and consequences of mea-surement. In H. Wainer & H. I. Braun (Eds.), Testvalidity (pp. 89-103). Hillsdale, NJ: Erlbauni.

Messick, S. (1994). The interplay of evidence and con-sequences in the validity of petformance assessments.Educatonal Researcher, 2,3(2), 13-23.

Fxceptional Children 493

Messiek, S. (1995). Standards CFofvidty :and th valid

ityof standards in erraneassessment. EducationlwMAsureyneit: "oes andPracice, i44) 5-.&

Nolet V. (1 992)~. Classroom-based -neasurenent andportfoli assessmnt. Diagostiqu, 18(),5 26.

Nols ~V., & McAugln4.(99) iigCBM to

xpoeaconsequential baSis for the~ validity of a

statewide performan"ce assessment. Dignlue, 22(3),

Swin, K. D.M Ainder, R. M. (19%). The effects of

repeated Treading .on twlo types of CBM: CbomputermazLe and oral reaing "ih. secon-grad tdents it

learing disabiltides. DiagOssi0de 2WM) 51-66.

Thompon S. &r Ihro,M.(200:0). State altern-ate

assessmients:, Status of I[DEA alternate assmn e

quiremenits take effect (SnhssRpr 5.M'nneapolis~, MN. Univrsty f M'nnsoa, National

center on Edueiat onal Outcomes.

Tind-al G. (1998). AMoa'd fior:i unesadin rA ta sk

comarailiy i acommdatd rstng.. Wshington,

Tindal G., Flick, DI, &Cole, C. (1993). Tleeffiect of

curriculum: on inferncs fradn peformance andImprovement:D Dignsiqe*!(),:6 6941.

Wesson, C. L,, &King R. P(1992):. The. role of cur-

riculum-~based measuremet i prfoio asessmnt

Yel M.i L., Deno, S. L., & ~MarsonMD B. (1992).

Diagnostq*ue.8(1),99-112.

AOUT T E *49 AUTHO idRIS

GER AlLi T INDALA (EC 20), Proessor, College of Edcton, University of Oeo. uee

POATl ALMOND (CEC #20), EvaluatonSpeial-

ist, Oregonl Deppartment Of Education, Salem.MARILVE ICo0NA.LD, Research AsIsistan,Be

havioral' Research~ and Teaching, Uniyerit of

Oregon, Egene. ~LIDY CRAWFORP (CEC

# 11,Assistant Profes~sor, University of CIl

orado at Colorado Springs. MARICK TEDDISCO

(CEC: O regont Federation), Research: As'sociate

AARON GLAAGOW, TeC'hnoogy Consultalt,Univerity ofOregon Euee ET OLLIN-

BECK CEC #1111), Di~~.1rector of Programis

Spr ingfield: School District, Oregon.~

Adesalcorresp)otl&ence to Gerald Tindal, Be-Isaviora Researh an Tleachig 24V1Coeg of

Educarion~ University of O.regon, Eugene, OR

:97403-5262.E-mail: geraldtda rk w ing. uopre gon.ed

Preparation Of this: document was supported in

prby th ceo ~ EUa 1Offic of Special Euato adReha-

bilitative: Services (OSP) grat award number

H324D000063-0E 1.Oinions expressed herein do0

noti necessar Ily reflect the posi tion or policy ofOSEP, andno offlicilal edoreet by OSEP

shul,b inferred

Manuscript received May 2002; accepted Omtber

* oioderbookSlil eeecd fin thsj Ina,pes

call 24 hrs!36 5 das I 800-BROOKS-NOW(26566); o 172-781040;ovsitemn

te: Web4 athtp:/ wbooksNow.com/Excep,-

tional :Child,renhtm.~ Use Visa~ M.C AMEX

Discover. or Send check lor: money.order + $405

S& $2.50 eah: ddi ite ) to: C ickat, 400

Moris Avenue, Log Branch, NJ 07740;: 1-32-

728-14 orFAX 1-732-728-17080.

494 l~~~~~~~~~~~~~~~~~~~~~~~Surnmm 2,003494

COPYRIGHT INFORMATION

TITLE: Alternate Assessments in Reading and Math:Development and Validation of Students With SignificantDisabilities

SOURCE: Except Child 69 no4 Summ 2003WN: 0319603306006

The magazine publisher is the copyright holder of this article and itis reproduced with permission. Further reproduction of this article inviolation of the copyright is prohibited. To contact the publisher:http://www.cec.sped.org/

Copyright 1982-2003 The H.W. Wilson Company. All rights reserved.