Webb Et Al 2005 Web Alignment Tool (WAT) Training Manual 2.1 Draft 091205

7/23/2019 Webb Et Al 2005 Web Alignment Tool (WAT) Training Manual 2.1 Draft 091205

1/184

Web Alignment Tool (WAT)

Training Manual

Norman L. Webb

Research Team:

Meredith Alt

Robert Ely

Brian es!erman

"uly# $%%&

ersion '.'

Council of Chief Wisconsin Center forState School Officers Education Research


2/184

For questions regarding this Training Manual or about the Alignment Tool, pleasecontact

!orman "# Webb, $ro%ect "eader

nl&ebb'&isc#edu()*+(-+.*/

The de0elopment of the Web Alignment Tool and related &or1 &as supported b2a grant from the 3# S# 4epartment of Education, !o# S-(*A)-))55, to the

O1lahoma 4epartment of Education &ith a subcontract to the Council of ChiefState School Officers and the Wisconsin Center for Education Research,

3ni0ersit2 of Wisconsin, for the period from April 5., ))-, throughSeptember -), ))6, and a grant from the !ational Science Foundation, !o#

E7R )--..6, to the Wisconsin Center for Education Research, 3ni0ersit2 ofWisconsin, for the period from October 5, )), to September -), ))6# The

products, findings, opinions, and conclusions presented are those of thede0elopers and authors and do not necessaril2 reflect the 0ie&s of thesupporting agencies#
mailto:[email protected]:[email protected]


3/184

Table o ontents

*. *ntroduction#################################################################################################################5

**. The Web Alignment Tool and the Alignment +rocess############################################/ A# 8roup "eaders####################################################################################################9 5# Registering &ith the Web Alignment Tool :WAT;##################################5) # Entering Assessments into the S2stem###################################################5- -# Adding State Standards to the S2stem :or "e0els to Each Ob%ecti0e in the State Standard###########-/ .# Reaching Consensus on 4O> "e0els for Ob%ecti0es#############################-9 6# Coding State Assessment =tems##############################################################.) (# 4ebriefing &ith 2our 8roup###################################################################.-

***. ,e!th-o-no/ledge Le0els##################################################################################.6 A# Mathematics 4O> "e0els###############################################################################.6 E@amples Applied to Ob%ecti0es and Assessment =tems#############################.(

?# Science 4O> "e0els########################################################################################6/ E@amples Applied to Ob%ecti0es and Assessment =tems#############################69C# Reading 4O> "e0els######################################################################################/) E@amples Applied to Ob%ecti0es and Assessment =tems#############################/

*. +re!aring or an Alignment 1tudy########################################################################*6A# $reliminar2 $reparations for an Alignment Stud2###########################################*6?# Sample Forms for the Alignment Stud2#########################################################9

. Re!orts####################################################################################################################5)6 A# $roducing Computer Reports#########################################################################5)6

?# =nterpreting Computer Reports######################################################################55) C# Writing a Report that Summaries and =nterprets the Alignment Results#### #55/

*. The , Alignment Tool######################################################################################56 A# C4 Alignment Tool Setup###############################################################################56 ?# 4o&nloading and 3ploading Standards#########################################################5/


4/184

ontact 2s####################################################################################################################5*

Reerences#####################################################################################################################59

A!!endi3 A

English "anguage "earner :E""; Alignment $rocess and Criteria "inguistic 4ifficult2+"e0el Matri@

A!!endi3 B

Sample WAT Tables for Anal2sis


5/184

*. *ntroduction

Question: What is the purpose of this Training Manual?

Answer: This Training Manual is meant to a; prepare program administrators toconduct an alignment stud2 using the Web Alignment Tool :WAT; and b;familiarie alignment stud2 participants &ith the alignment process andthe WAT#

At the end of training, pro%ect staff members should feel comfortable &ith

the general principles of alignment ho& participants register &ith the WAT

ho& participants enter standards and assessments into the WAT ho& participants code assessment items and standards using depth+of+1no&ledge

:4O>; le0els generating alignment reports anal2ing reports and interpreting WebbBs fi0e alignment criteria training alignment stud2 participants to use the WAT organiing and planning an alignment stud2 using this s2stem

Question: How is this Training Manual organized?

Answer: The Manual is organied so thatprogram administrators, Group Leaders,and reviewerscan alluse it in different &a2s, to familiarie themsel0es&ith &hat the2 need to 1no

=f 2ou are a Dou should read

+rogram Administrator

organiing and conductingan alignment stud2

The entire Training Manual

Re0ie/er

coding assessment itemsand standards in onesub%ect area

+art *# +art **.B# and +art ***

4rou! Leader

training and leading agroup of re0ie&ers throughthe alignment process inone sub%ect area

+art *# +art **# and +art ***

5


6/184

Question: How do I use the CD Alignment Tool?

Answer: =f 2ou are using the C4 0ersion of the Alignment Tool rather than the WebAlignment Tool :WAT;, 2ou &ill find important information in +art *ofthe Training Manual# This section &ill tell 2ou ho& to installandset upthe C4 Alignment Tool# =t &ill also gi0e 2ou information about creating alocal area net&or1 of computers for conducting an alignment stud2#

3sing the C4 Alignment Tool is essentiall2 no different from using theWeb Alignment Tool, aside from installation# Thus, the instructions foundin +art ** of this Manual appl2 for allusers of the Alignment Tool#

Question: What is alignment?

Answer: The term alignmentdescribes the degree to /hich e3!ectations andassessments are in agreement and ser0e in con5unction /ith one another to guide the

system to/ard students learning /hat they are e3!ected to 6no/ and do# 4efined inthis &a2, alignment is a qualit2 of the relationship bet&een e@pectations and assessments,not an attribute of an2 one of these t&o s2stem components#It describes the matchbetween expectations and assessment that can be legitimately improved by changingeither student expectations or the assessments.As a relationship bet&een t&o or mores2stem components, alignment is determined on the basis of multiple criteria described indetail in a !ational =nstitute for Science Education :!=SE; research monograph, Criteriafor Alignment of Expectations and Assessments in Mathematics and cience Education:Webb, 599/;# Fi0e general guidelines for alignment are found in the monograph

5# The alignment process must focus on the content or sub%ect area and its attributes#Science alignment could loo1 0er2 different from alignment for language arts#

# Alignment should be articulated across grades and agesthat is, alignment occurs notonl2 &ithin a grade, but also bet&een grades# For e@ample, the gro&th in 1no&ledge o0ergrades e@pressed b2 the mathematics standards should be similar to the progression of1no&ledge measured b2 assessments o0er the grades#

-# Alignment must address issues of equit2 and fairness# This can pertain to specialeducation or culturall2 rele0ant content#

.# The alignment process should address pedagogical implications that might arise#

6# Alignment should be reasonable &ith respect to the resources needed to attain suchalignment#


7/184

Question: Why is alignment important?

Answer: The alignment of e@pectations for student learning &ith assessments formeasuring studentsB attainment of these e@pectations is an essential attribute for aneffecti0e standards+based education s2stem# Without adequate alignment bet&een

standards and assessments, teachers and students become confused about educationale@pectations# Among other things, this can result in learning e@pectations being lo&eredfor some students &hile being raised for others, creating potential inequities# For thesereasons,alignment between curriculum standards and assessments is now re!uired underthe provisions of several federal education statutes.

Alignment has some other implications for educators and administrators# $articipating inan alignment anal2sis leads to increasing a&areness of the t2pe of 1no&ledge and depthof 1no&ledge that can be displa2ed or demanded in 0arious content areas, standards, andassessment items# Furthermore, the results of an alignment anal2sis &ill not onl2 helpdetermine the qualit2 of the alignment, but also pro0ide direction for ho& the district and

state education personnel can refine standards andor identif2 more appropriateassessment items#

Question: What are the alignment riteria used in this alignment system?

Answer: !orman WebbBs alignment s2stem is one of se0eral alignment s2stemsde0eloped during the last decade :Webb, 599/;# =n this s2stem, there are fi0e criteria foralignment bet&een standards and assessments# These criteria, and their associated cutoff

0alues, are described more technicall2 in +art of this Manual# ategorical oncurrence +++ This criterion measures the e@tent to &hich the

same or consistent categories of content appear in the standards and theassessments# The criterion is met for a gi0en standard if there are more than fi0eassessment items targeting that standard#

,e!th-o-no/ledge onsistency+++ This criterion measures the degree to&hich the 1no&ledge elicited from students on the assessment is as comple@&ithin the conte@t area as &hat students are e@pected to 1no& and do as stated inthe standards# The criterion is met if more than half of targeted ob%ecti0es are hit

b2 items of the appropriate comple@it2#

Range-o-no/ledge orres!ondence+++ This criterion determines &hether thespan of 1no&ledge e@pected of students on the basis of a standard corresponds tothe span of 1no&ledge that students need in order to correctl2 ans&er thecorresponding assessment itemsacti0ities# The criterion is met for a gi0enstandard if more than half of the ob%ecti0es that fall under that standard aretargeted b2 assessment items#

-


8/184

Balance o Re!resentation+++ This criterion measures &hether ob%ecti0es thatfall under a specific standard are gi0en relati0el2 equal emphasis on theassessment#

1ource o hallenge+++ This criterion is met if the primar2 difficult2 of the

assessment items is significantl2 related to studentsB 1no&ledge and s1ill in thecontent area as represented in the standards#

The WAT allo&s 2ou to perform more than %ust Webb alignment studies# For e@ample,&ith the help of 8ar2 Coo1 the WAT has been adapted to perform English "anguage"earners :E""; alignment studies# Dou ma2 also use a different set of cogniti0e le0els&ith a greater number or fe&er le0els than the four depth+of+1no&ledge le0els Webbde0eloped# The follo&ing parts of the Manual e@plain ho

Question: What does an alignment study loo! li!e? What atually happens?

Answer: This Manual has been designed to ans&er these questionsto gi0e adetailed e@planation of &hat happens during an alignment stud2# A summar2 of the mainphases of an alignment stud2 follo&s#

First, &hen a state education department decides to use this s2stem to e0aluate thealignment bet&een its curriculum standards and assessments, it must determine &hichsub"ect areasand &hichgrade levelsto align# Each sub%ect area, li1e mathematics, &illbe studied b2 one or t&o groups of re0ie&ers, depending on ho& man2 grades are to bestudied and ho& man2 da2s are allocated to the process#

For the sa1e of reliabilit2, each group should ha0e at least fi0e re0ie&ers, one of &hom isthe Group Leader# The reviewersshould be administrators, teachers, and content e@pertsfamiliar &ith the general content and e@pectations for the grades that the group &ill bere0ie&ing# Re0ie&ers from outside the state should be brought into the process to addperspecti0e and 0alidit2 to the stud2#

For each grade le0el, the primar2 role of arevieweris5# To %udge the depth+of+1no&ledge le0el of the content ob%ecti0es under each

standardG# To %udge the depth+of+1no&ledge le0el of each assessment itemG-# To identif2 the primar2 and up to t&o secondar2 ob%ecti0es to &hich each item

correspondsG and,.# To %udge &hether there is a source+of+challenge issue &ith an2 items#

The re0ie&ers record all of this information using the Web Alignment Tool :WAT;, &hichthen aggregates the data and calculates &hether or not the fi0e alignment criteria are metfor each standard and the assessment# A detailed description of the alignment process and

.


9/184

the WAT follo&s in +art **# The depth+of+1no&ledge :4O>; le0el definitions ande@amples for the content areas of mathematics, reading, language arts, science, and socialstudies can be found in +art ***of this Manual#

Along &ith the duties of a re0ie&er, the additional duties of aGroup Leaderare

5# To register each group member &ith the WATG# To enter into the WAT the number and t2pes of assessment items for each gradeBs

assessment:s;G-# To enter the stateBs standards and ob%ecti0es into the WATG.# To train the re0ie&ers in using the WAT and to familiarie them &ith the depth+of+

1no&ledge :4O>; le0els for their content areaG6# To lead the groupBs consensus process for determining the 4O> le0els of the

standards and ob%ecti0es for each grade le0el, and to enter this information intothe WAT#

Finall2, the primar2 duties of theprogram administratorsorganiing the alignment stud2are

5# To schedule and organie the alignment &or1shop :+art *;, and# # To &rite the final report of the results of the stud2 :+art ;#

6


10/184

(


11/184

**. The Web Alignment Tool and the Alignment +rocess

The purpose of this section is to familiarie program administrators, 8roup

"eaders, and re0ie&ers &ith the Web Alignment Tool and the alignment process#

1ection Adescribes theres!onsibilities o the 4rou! Leaders#

1ection Bdescribes the res!onsibilities o the re0ie/ers.

=M$ORTA!T Since a group leader is also a re0ie&er, 8roup "eaders should becomefamiliar &ith both sectionsH

Question: What e"atly is the Alignment Tool?

Answer: The Alignment Tool is a computer tool that automates the process ofaligning state education standards &ith standardied tests# The tool is a0ailable in t&oforms the Web Alignment Tool :WAT;, &hich is accessed 0ia the =nternet, and a C40ersion of the WAT##ecause most people will use the Internet application$ this %rainingManual uses the terms &A% and 'Alignment %ool interchangeably$( unless referring

specifically to the C).

The Web Alignment Tool :WAT; is an =nternet application that allo&s state and

district program administrators to automate the process of gauging alignmentbet&een standards and assessments# =t can be accessed at the follo&ing Web site

http&&&cer#&isc#edu&at

The , ersiono the WAToffers all of the same functions as the WAT &ithoutrequiring an =nternet connection# =nstead, the C4 is do&nloaded to a computerhard dri0e, and computers are net&or1ed together to share data# For moreinformation on ho& to install the C4, refer to +art *, The C4 Alignment Tool#Once installed, the , 0ersion o the WAT operates in e@actl2 the same &a2 asthe Web site# References to the WAT in this Training Manual thus appl2 to both

the Web+based and C4 0ersion of the Alignment Tool#

Question: How does the WAT measure alignment?

Answer: This tool ma1es it possible to gauge the alignment bet&een the standardsand the assessment on the basis of the fi0e criteria outlined in the introduction

/
http://www.wcer.wisc.edu/wathttp://www.wcer.wisc.edu/wat


12/184

categorical concurrence, depth+of+1no&ledge consistenc2, range+of+1no&ledgecorrespondence, balance of representation, and source of challenge# The WAT records allof the information about assessments and standards acquired during the alignmentprocess, producing statistics and alignment reports on each of the fi0e criteria# :Stud2participants ma2 0ie& these reports immediatel2 after finishing coding items for a gi0en

grade b2 simpl2 entering the Re!ortslin1 at the top of the WAT screen# +art of theManual pro0ides detailed information about ho& to interpret and use these computerreports#;

*


13/184

#etion A: $roup %eaders

What a 4rou! Leader Must ,o in the Alignment +rocess7

Beforethe

alignmentseminar

5# Register your grou!&ith the Web Alignment Tool#

# Enter inormation about the assessment(s) 2ou &ill be stud2ing&ith the WAT#

-# Enter the state standards into the WAT :or 0erif2 that the2 arealread2 in the s2stem;#

.# reate studiesin the WAT b2 pairing assessments and standards#

Duringthe

alignmentseminar

6# Train your re0ie/ers (+hase *) on 2our content areaBs 4epth+of+>no&ledge :4O>; "e0els and on ho& to register &ith the WAT#

II(# 8acilitate the consensus !rocess for each grade# This is &henre0ie&ers come to agreement on the 4O> le0el of each ob%ecti0e#

/# Train your re0ie/ers (+hase **)on ho& to code assessment items#

II*# Enter the consensus ,9 0alues for each ob%ecti0e in the statestandards as the leaderBs :hence, the groupBs; 0alues#

II9# Lead a debrieing !rocess about each gradeBs alignment stud2, after2our re0ie&ers ha0e finished coding items for that assessment#

I=n addition to the 8roup "eader responsibilities outlined abo0e, 8roup "eaders &ill alsoparticipate in the ob%ecti0e coding process :&hich occurs bet&een Steps 6 and (; and the item

coding process :&hich occurs bet&een Steps * and 9;# These processes are outlined in thedirections for re0ie&ers later in this section#

IISteps 5, , -, ., 6, and / onl2 need to be performed once b2 the 8roup "eader# Steps (, *, and 9must be completed once for each stud2 that 2our group performs#

9


14/184

#teps to &e ompleted &efore the day of the Alignment Institute

#tep ': (egistering with the We& Alignment Tool )WAT*

The follo&ing information applies to both the Web and C4 0ersions of the WebAlignment Tool# The first step in using the WAT is registering 2our group &ith thes2stem# This is done at the same time the 8roup "eader registers &ith the group# Whilethe 8roup "eader must register separatel2 from the other re0ie&ers, he or she &ill also beconsidered a re0ie&er in the alignment stud2# $art of the tas1 in registering a group isdetermining ho& the group &ill be constituted# There are se0eral options for organiing

groups

5# A 8roup "eader ma2 help facilitate all contentareas across all grades.

:E@ample One 8roup "eader for .th, *th, and 5)thgrade mathematics,science, and readinglanguage arts, etc#;

# A 8roup "eader ma2 help facilitate eachcontentarea and each grade.

:E@ample One 8roup "eader for .thgrade scienceG one 8roup "eader for *thgrade scienceG one 8roup "eader for 5)thgrade scienceG one 8roup "eader for .thgrade social studies, etc#;#

-# A 8roup "eader ma2 help facilitate each contentarea across all grades.

:E@ample One 8roup "eader for .th, *th, and 5)thgrade mathematicsG another8roup "eader for .th, *th, and 5)thgrade readinglanguage arts, etc#

We recommend that groups be formed according to Option -# 3nder this scenario, there0ie&ers are dealing &ith standards and items from the same content area, instead of&ith multiple re0ie&ers for each grade# This practice lends itself to more consistentalignment results# =t is an option that also presents greater opportunities for future 0erticalalignment bet&een grades#

5)


15/184

To register the group and the 8roup "eader

5# 8o to the Web Alignment Tool homepage# J=f 2ou are using the C4, enter theC4 into 2our computer# The WAT screen should appear &ith a lin1 for

*nstallation *nstructionsat the bottom# Clic1 this and follo& the instructionsto set up the C4 Alignment Tool# See +art *of this Training Manual formore detailed instructions#K

# Clic1 on the Loginlin1 at the top of the screen# The screen that &ill load ispictured in Fig# #5

Fig# #5

-# When the login page loads, clic1 on the 4rou! Registration +agelin1 nearthe bottom of the screen# Fill out the required group information :name, emailaddress, etc#; and ma1e a note of the user name and pass&ord 2ou ha0eselected# An e@ample of this screen then appears#

!ote The information the 8roup "eader enters on theregistration page does not limit in an2 &a2 the t2pes ofstudies that can be conducted b2 the 8roup "eader# Fore@ample, 2ou ma2 register for grades 5, , - and still dostudies for grades (, /, and *# The same is true &ith state,content, etc#

55


16/184

The 4rou! Registration +age

Fig# #

When 2ou finish registering as a 8roup "eader, go to the bottom of the screen and select

1ubmit#

On the ne@t screen, the computer tool &ill gi0e 2ou agroup I) number, &hich 2oushould note and share &ith re0ie&ers in the group# ou and the other re0ie/ers /illneed this number e0ery time you log onto the system.Dou should also &rite do&n2our usernameandpasswordG 2ou &ill need these later as &ell#

Once 2ou ha0e registered &ith the WAT, the registration information 2ou ha0e sa0ed &illbe recorded, e0en if something goes &rong &hile 2ou are entering information# =f a

s2stem error appears, 2ou ma2 al&a2s log off and then log on again# Dou &ill not lose2our registration information b2 logging off, closing the bro&ser &indo&, or shuttingdo&n 2our computer#

5


17/184

#tep +: ,ntering Assessments into the #ystem

=n order to enter an assessment into the WAT, the 8roup "eader needs the follo&ing

information about the assessment

7o& man2 total items are on the assessmentL or, =f the test has numeroussections, &hat are the sections called and ho& are the2 numberedL

=f the test has numerous sections, are the2 numbered sequentiall2L

Which items, if an2, should notbe included in the alignment anal2sisL :Forinstance, items that are being field+tested and do not count to&ard student scoresshould not be coded b2 re0ie&ers, or considered in the alignment data#;

What are the point 0alues of each assessment itemL

Once the rele0ant information about each assessment is obtained from the stud2coordinators, the 8roup "eader ma2 enter the assessments into the WAT# Dou ma2 eitherenter assessment information into the s2stem no& :during the group registration process;,or later, from the 8roup "eaderBs Administrator Start $age# =f 2ou are doing this duringthe group registration process, the follo&ing screen &ill appear#

5-


18/184

Fig##-

=f 2ou are entering assessments for the first time, the initial step is to scroll to the bottomof this screen and select Add Ne/ Assessment# =f 2ou belie0e that 2our assessment hasalread2 been entered into the s2stem, 2ou ma2 also use the other search features on the

screen to search for the assessment#

When the ne@t screen appears, the follo&ing information about the ne& assessmentshould be entered into the s2stem

Assessment Title State Sub%ect 8rade

Course 4istrict Source of Assessment Form :5, unless 2ou are anal2ing multiple forms for that grade; !umber of =tems Test Administration ?egins Test Administration Ends

5.


19/184

After all of the assessment information is inserted, clic1 on Add to Listto add theassessment into the s2stem# After the assessment is added, information about theassessment 2ou %ust entered should appear at the bottom of the screen# !o& 2ou ma2change the assessment information on the top half of the screen to that of a ne&assessment, adding ne& assessments to the Web Alignment Tool from this screen#

=f there is specific information 2ou &ant to customie for a particular assessment, 2ouma2 also, at this point, ma1e such changes# Clic1 Editne@t to the assessment title and2ou &ill reach the screen pictured in Fig# #.# From here, 2ou can enter specificinformation about the assessment items#

Fig# #.

a; If an item on the test is worth more than * point, 2ou should change the &eight ofthat item accordingl2#

b;If an item is not meant to be included in the analysis

, because it is a field test itemor for some other reason, chec1 the Missing;bo@ne@t to that item# When are0ie&er gets to this item, he or she &ill see a prompt telling himher to s1ip it#

c; If the test has multiple sections that are numbered differently, do not tr2 to changethe item numbers along the left side of the assessment information screen# Thesenumbers cannot be altered# =nstead, 2ou should use the ommentbo@es to ma1ethe sectional numbering clear to the re0ie&ers, in order for re0ie&ers to 1no&

56


20/184

ho& the numbering in their test boo1let corresponds to the computer numbering:see Fig# #.;# The comments 2ou ma1e here &ill appear under the item numberon the =tem Coding screen for each re0ie&er :see Fig# #5/;# For e@ample,suppose 2our assessment has t&o sections


21/184

The 4rou! Leader Administration +age

To get to this page, simpl2 log in as a 8roup "eader# =f 2ou are alread2 logged in as a8roup "eader, simpl2 clic1


22/184

#tep -: Adding #tate #tandards to the #ystem )or .erifying #tandards*

!ote State standards should be entered into the s2stem at leastt&o da2s before re0ie&ers are to complete the process in order

to assure there is adequate timeH

To enter standards into the WAT, the 8roup "eader must first 1no& ho& the stud2Bsadministrators &ant the stateBs student e@pectations to be classified# %he Alignment %oolallows for up to three levels.4ifferent states ha0e different names for these 0ariousle0els, including strands,N benchmar1s,N competencies,N etc# For the sa1e ofconsistenc2 in the Alignment Tool, &e call the three le0els 1tandard, 4oal, and9b5ecti0e# =f 2ou emplo2 onl2 t&o le0els, these &ill be called 1tandardand 9b5ecti0e#

1tandardis the most general# =t ma2 be a broad statement of student acti0ities,

such as Students read for understanding,N or it ma2 simpl2 be a contentclassification li1e 8eometr2#N The data are reported out at this le0el# =n other&ords, the computer reports for a particular grade gi0e information about thealignment of each standard&ith the appropriate assessment#

4oalis the middle le0el of specificit2# Each standard is composed of goals thatma2 in0ol0e smaller topic areas or more precise student acti0ities# Man2 states donot ha0e three le0els, so this le0el ma2 be omitted &hen entering standards intothe WAT#

9b5ecti0eis the most specific le0el# Each goal is composed of ob%ecti0es, &hich

specif2 particular 1inds of acti0ities or s1ills :e#g#, Read and identif2 t2pes ofpoetr2 and the use of in0ersion, rh2me, and rh2thm,N or Con0ert units &ithin ameasurement s2stem#N;# When re0ie&ers are coding items, the2 &ill matchassessment items &ith one or more ob%ecti0es, if possible# Re0ie&ers areinstructed to match an item &ith a 8oal or a Standard only ifnone of theob%ecti0es target the item#

When 2ou determine ho& the state standards should be classified, 2ou ma2 enter thestandards into the WAT# Dou &ill enter standards from the 8roup "eader AdministratorStart $age# ?ut, first, chec1 the 8roup "eader Administration $age to ensure that 2our

stateBs standards ha0e not alread2 been entered into the WAT#

5*


23/184

Alternate 1tandard 1tructures

Dour standards ma2 be categoried according to multiple dimensions# Fore@ample, man2 English "anguage "earners :E""; standards ha0e aStandard8oalOb%ecti0e structure, as well asa $roficienc2

"e0elOb%ecti0e structure# Dou ma2 &ish to produce alignment resultsaccording to both of these structures#

Ma1e sure &hen 2ou are entering standards on this screen that 2ou enteronl2 the primar2 Standard8oalOb%ecti0e structure of the standards#Once 2ou ha0e created studies that use these standards, 2ou &ill get achance to enter alternate standard configurations for the standards#

See #tep /: Creating #tudiesfor information about ho& to enter thisalternati0e standard structure# =f 2ou &ould li1e more information aboutconducting an E"" stud2, see Appendi@ ?#

The 8roup "eader Administration $age lists all standards that ha0e been entered for 2ourstate# ?2 clic1ing Editne@t to the title of the standard, 2ou ma2 see &hether the standardsare those 2ou &ant to use andor &hether the2 are up to date#

To enter a ne& standard, clic1 Enter Ne/ 1tandardin the middle of the 8roup "eaderAdministration $age#

!ote if 2ou are using the C4 0ersion of the WAT, it is possible that 2our stateBsstandards ha0e been entered into the Web 0ersion of the WAT# =f so, 2ou ma2 be able

to upload 2our standards from the Web# See +art *of the Training Manual forinstructions about uploading and do&nloading standards from the WAT#

At the ne& screen :Fig# #(;, 2ou should enter the name, state, sub%ect, and grade le0el ofthe standard, and then clic1 on reate 1tandard#

Fig# #(

59


24/184

The ne@t screen is &here 2ou &ill begin entering standards, and the lo&er half of thescreen sho&s the basic outline format that reflects, and allo&s 2ou to differentiatebet&een, the le0els of 2our stateBs standards#

The 8roup "eaderBs ne@t step is to manuall2 enter the standards, ob%ecti0e b2 ob%ecti0e,

or to upload them from a sa0ed file.=nstructions for uploading standards from a sa0ed filecan be found on p# # =f 2ou can easil2 get the standards into an E@cel spreadsheet, thisoption ma2 sa0e 2ou time#

To enter the standards manuall2, go to the section of the ne& screen that reads Ro& toinsert belo& of#N Dou &ill use the drop+do&n bo@ to number 2our first standard, after&hich ne& standards and ob%ecti0es &ill be placed in relation to e@isting standards andob%ecti0es# This screen is pictured in Fig# #/#

To see some sample standards in another bro&ser &indo&, select1am!le 1tandard 1tructure# =f 2ou &ould li1e to e@periment &ith

these sample standards in order to become more comfortable &ithediting standards, select Add 1am!le 1tandard 1tructure#

Fig# #/

)


25/184

To enter a first standard, Select 5 from the drop+do&n menu# =n the space for "e0el, enter5, and in the space for 4escription, enter the description of the first standard :i#e#,


26/184

Fig# #5)

The outline format that begins to ta1e shape as 2ou enter standards should sho& cleardifferentiation bet&een standards, ob%ecti0es, and goals through its indentations# Eachindented number in the outline indicates a suble0el# =f 2ou &ish to ma1e changes toindent something that does not appear to be listed as a suble0el :&hen it should be;, usethe *ncrease *ndentbutton to tell the computer to indent the ob%ecti0e and recognie it asa suble0el# Dou can also ,ecrease *ndentif something is indented and identified as asuble0el that should not be#

For instance, in Fig# #55, Ob%ecti0e 5#5 is not indented to indicate a suble0el, andStandard is indented in such a &a2 that it appears to be an ob%ecti0e rather than astandard#

Fig# #55

?2 using the *ncrease *ndentbutton to change Ob%ecti0e 5#5, and the ,ecrease *ndentbutton to change Standard , 2ou ma2 indicate the appropriate le0els as sho&n in Fig##5#


27/184

Fig# #5

=f at an2 point 2ou &ish to edit or delete a standard, goal, or ob%ecti0e 2ou ha0e alread2entered into the WAT, clic1 on the Editor ,eletebutton ne@t to the appropriate ob%ecti0e#The subsequent screen &ill allo& 2ou to edit information 2ou ha0e entered, or &ill clarif2&hether 2ou &ant to delete the ob%ecti0e# When finished ma1ing changes, clic1 on 1a0e#

The e@ample and description pro0ided abo0e is for a two+level system:StandardOb%ecti0e;# =f 2our state has a three+level system:Standard8oalOb%ecti0e;,then 2ou &ill add an e@tra indentation &hen 2ou add goals and ob%ecti0es# For instance,

to add the first goal pictured in Fig# #5-, 2ou &ould enter 5 for the Ro& to =nsert ?elo&ofN and 5#5 for the "e0el#N After the outline adds the ob%ecti0e, 2ou indent to sho& ane& le0el#

To add the first ob%ecti0e under the first goal in Fig# #5-, 2ou &ould enter 5#5 for theRo& to =nsert ?elo& ofN and 5#5#a for the "e0el#N Dou &ould then indent the ob%ecti0eto distinguish it from the goal#

Fig# #5-

,ploading tandards from a aved -ile

=f 2ou clic1 on E@cel File+Standard 3pload *nstructions, the follo&ing &ill appear in apop+up &indo

The E@cel file needs to be in the correct format in order to be uploaded to the WATproperl2# The name of the file does not matter, but the name of the &or1sheet the data are

-


28/184

in does matter# The &or1sheet needs to be titled Sheet5#N When 2ou create a ne& E@celspreadsheet, b2 default there are three &or1sheets titled Sheet5, Sheet, and Sheet-#Sheet5 needs to minimall2 contain t&o columns# The first column should be titled"e0elN and contain the le0el names# "e0el names are limited to 5) characters and shouldcontain no spaces# The second column should be titled 4escriptionN and contain the

description or 0erbiage of the ob%ecti0e# After creating the document, 2ou should ha0et&o columns titled "e0elN and 4escriptionN and, as 2ou mo0e do&n the ro&s, 2oushould ha0e a "e0el name and Ob%ecti0e description in each ro Cells &ith missing te@tma2 inappropriatel2 import the ob%ecti0e# Also, there should be no merged ro&s orcolumns &here one column spans t&o ro&s, or one ro& spans t&o columns# Thefollo&ing is an e@ample of &hat the &or1sheet should loo1 li1e#

After uploading the E@cel file, 2ou &ill still need to set up the structure of the standard#For e@ample, "e0el 5#5 needs to be under "e0el 5 and "e0els 5#5#a and 5#5#b need to beunder 5#5# To do this, clic1 on the =ncrease =ndentN lin1 b2 "e0el 5#5 once to place itunder "e0el 5# To place 5#5#a, and 5#5#b under 5#5, clic1 =ncrease =ndentN t&ice for eachob%ecti0e#

Dou can ha0e the structure set up &hen 2ou unload the E@cel file b2 adding the thirdoptional column =ndent#N To place the ob%ecti0e in the correct hierarch2, place a number,5 through ., in the =ndentN column# The follo&ing illustrates &hat the &or1sheet shouldloo1 li1e#

.


29/184

6


30/184

#tep /: Creating #tudies

After standards and assessments are entered into the s2stem, there is one eas2 step

remaining before the alignment stud2 can begin# Dou must create studies b2 pairing eachassessment &ith the appropriate grade+le0el standards# To do this, log on as a groupleader, &hich &ill ta1e 2ou to the 4rou! Leader Administration +age# !o& go to thebottom of this page and clic1 on the lin1 Enter Ne/ 1tudy# This &ill ta1e 2ou to thescreen sho&n in Fig# #5.#

Fig# #5.

On this screen, select 2our state and sub%ect and clic1 1earch Assessmentsfor alist of assessments to choose from#

!e@t, select 2our state and sub%ect and clic1 1earch 1tandardsfor a list ofstandards to choose from#

!o&, select one assessment and one standard :i#e#, a set of 8rade "e0elStandards;, and 2ou &ill create a stud2 to measure the alignment bet&een theassessment and the corresponding grade+le0el standards#

Clic1 reate 1tudy# Dou ha0e no& created a stud2 for 2our re0ie&ers tocomplete for one grade le0el#

!ote =f 2our re0ie&ers &ill be performing an alignment process on more than oneassessment and standard, 2ou &ill need to create additional studies# Dou ma2 do soin the same &a2, entering reate 1tudyfor each ne& alignment stud2, from thisscreen#

When 2ou are finished creating studies, clic1 ,one# Dou ha0e no& completed thecomputer steps required to prepare for an alignment stud2#

(


31/184

,diting #tudy Details

After 2ou ha0e created a stud2, 2ou ma2 go to the 8roup "eader Administration $age

:see p# 5/; and edit the 1tudy ,etailsfor that stud2# This enables 2ou to customie thestud2 in se0eral &a2s#

Fig# #565# Dou ma2 change the number of )/ Levelsto a number other than .# =f 2ou choose touse a different set of cogniti0e le0els, be sure to inform 2our re0ie&ers of this choice andtrain them appropriatel2H

# Dou ma2 ma1e the stud2 a Gary Coo0 ELL Alignment tudyrather than a1orman&ebb Alignment tudy# This &ill allo& 2ou to aggregate the data for the stud2 across t&odimensions instead of %ust one# To ma1e use of this feature, 2ou must enter an AlternateStandard Configuration and assign Sub 8roups in the bo@ belo

ELL Alignment tudies

Often a stateBs English "anguage "earners :E""; standards &ill ha0e t&o dimensionsinstead of one# An2 ob%ecti0e &ill be listed as falling under a particular contente@pectation :standard; and falling under a particular language proficienc2 le0el#Therefore, the alignment anal2sis should be performed according to both dimensions#

/


32/184

4oes each standard meet the four alignment criteria :Categorical Concurrence, 4epth+of+>no&ledge Consistenc2, etc#;L 4oes each proficienc2 le0el meet the four alignmentcriteriaL

Dou need to use this screen for an2 stud2 &here 2ou &ant to anal2e alignment results

along t&o or more dimensions# For each configuration in &hich 2our standards ha0eother than the primar2 configuration, 2ou must select Alternate 1tandardoniguration#

=n the bo@ pro0ided, t2pe the namethat 2ou &ish to use for the alternateconfiguration# Clic1 Add.

The ne& standard configuration should appear belo Clic1 Edit.

On the ne@t screen, there is a place &here 2ou can add the names of the alternatestandardsN that 2ou are using# For e@ample, if 2our alternate configuration is

$roficienc2 "e0els, 2our alternate standards ma2 be called ?eginning,N=ntermediate,N and Ad0anced#N Enter the name of each standard, and clic1 Add.=t should appear belo& on 2our list# =n a moment, 2ou &ill get a chance tocategorie the e@isting ob%ecti0es under these alternate standards#

Clic1 ,one&hen 2ou ha0e entered the alternate standard titles# This &ill bring2ou bac1 to the 1tudy ,etailspage#

Clic1 on Assign 1ub 4rou!s# This &ill bring up a page &ith a list of thestandards and ob%ecti0es# For each ob%ecti0e, 2ou no& ha0e the chance tocategorie it under the alternate standard that it should fall under# For e@ample,2ou can categorie Ob%ecti0e =#A#- :&hich is alread2 listed under Standard = and8oal A for the purposes of the regular standard configuration; as falling under the=ntermediateN proficienc2 le0el#

Clic1 1a0eafter 2ou are finished categoriing the standards under the alternatestandard structure#

=f 2ou ma1e a mista1e along the &a2, 2ou ma2 edit or delete an2 alternatestandard configuration from the 1tudy ,etailspage#

A!!endi3 Bof this manual includes much additional information about conducting anE"" alignment stud2# This includes a slightl2 different set of alignment criteria that ma2be used, as &ell as information about ho& to interpret the alignment results according tothe different standard configurations#

4et +a!er oding 8orms. There are t&o lin1s that allo& 2ou to print out paper codingforms for the stud2Bs assessment items and for the stud2Bs standards# These forms are

*


33/184

e@tremel2 useful for re0ie&ers to ha0e on hand during the alignment stud2 as bac1upcopies# See +art *of this Manual for other sample forms for 2our stud2##teps to omplete the day of )during* the Alignment Institute

#tep 0: Training (e1iewers )2hase I*

The pre0ious four steps should all be completed before the Alignment =nstitute begins#The &or1shop &ill open &ith a &elcome and introduction b2 the state staff members &hoare organiing the stud2# After this, the re0ie&ers &ill split into sub%ect+area groups, and2ou &ill train them on (a)the depth+of+1no&ledge :4O>; le0els for their sub%ect area and(b)ho& to register &ith the WAT#

a; 4O> "e0els Training

4efinitions for the 4O> le0els for the different sub%ect areas can be found in +art ***ofthis Manual# Re0ie&ers should be gi0en copies of the definitions at this time and,together, should go through these definitions carefull2# Also included in +art *** of theManual are sample ob%ecti0es and items coded to the described 4O> le0els# The sampleob%ecti0es and items ma2 be reproduced and discussed to help re0ie&ers gain a greatersense of the application of the 4O> le0els to specific items#

b; Registering &ith the WAT

Dou should first sho& the re0ie&ers ho& to register &ith 2our group :2ou &ill need 2ourgroup number here;# Then 2ou must pro0ide orientation on ho& to select a stud2 andcomplete +art *:coding ob%ecti0es;# =nstructions for coding ob%ecti0es can be found inthe follo&ing section of the Manual#

9


34/184

#tep 3: 4ailitating the $roup Consensus 2roess

The purpose of this Step is to reach consensus about the 4O> le0els of the ob%ecti0es#

This requires the follo&ing

A# Re0ie&ers individuallyenter 4O> 0alues for each of the ob%ecti0es in the appropriate grade le0el#

?# The 8roup "eader prints out the re0ie&ersB indi0idual 4O> 0alues#

C# The group comes to consensus about the 4O> le0el of each ob%ecti0e#

A# Re0ie&ers indi0iduall2 enter 4O> 0alues for each of the ob%ecti0es in the appropriate grade le0el#

The steps for entering these 4O> 0alues are described in Section ? of this part of theManual :+art **;# For this part, 2ou should log in as a re0ie&er, and notas a 8roup"eader# :"ater, &hen it is time to enter the group consensus 4O> 0alues, 2ou &ill log inas a 8roup "eader# ?ut at this point, the 4O> 0alues 2ou enter are not treated differentl2

from the 0alues entered b2 the other re0ie&ers#;

Fig# #5(

-)


35/184

?# The 8roup "eader prints out all the re0ie&ersB indi0idual 4O> 0alues#

At an2 time during the 4O> entr2 process, 2ou ma2 chec1 on the progress of the otherre0ie&ers b2 clic1ing on the lin1 1ho/ 4rou! Members> Results# The screen sho&n in

Fig# #5/ &ill appear# Dou should !rint outthe screen that sho&s the 4O> le0els enteredb2 each of the re0ie&ers# Dou &ill need this for the group consensus process#

This same screen can be arri0ed at b2 going to the 8roup "eader Administration page:the screen that appears immediatel2 after 8roup "eaderBs login; and clic1ing on 1tudy,etailsfor the appropriate stud2#

Fig# #5/

-5


36/184

C# The group comes to consensus about the 4O> le0el of each ob%ecti0e#

At this point, gather the re0ie&ers around a table# Dou should ma1e sure each of them

has a cop2 of the standards for the grade 2ou are re0ie&ing, and a cop2 of the 4O>+le0el definitions# The re0ie&ers &ill be referring to the

definitions in the upcoming discussion#

=n addition, 2ou should ha0e a print+out of

the 4O> le0els assigned b2 the re0ie&ers to that gradeBs ob%ecti0es#

Dou no& lead the group in a discussion that &ill bring about consensus on the 4O>

le0els for these ob%ecti0es# For each ob%ecti0e, the re0ie&ers must come to agreementabout &hat the most appropriate 4O> le0el is for that ob%ecti0e# Ma1e sure that 2ourecord these group consensus 0alues2ou &ill need to enter them into the WAT later#There is no one &a2 to lead this discussion process, but the table on the following pagepro0ides suggestions &e ha0e found to be helpful in the past# ?e sure to remind re0ie&ersof the guidelines found in 1ection Bbelo& for coding 4O> le0els to ob%ecti0es#

-


37/184

Tips for 4ailitating the Consensus 2roess

5# Read each ob%ecti0e aloud before discussing it#

# As 2ou go through the ob%ecti0es, acti0el2 solicit comments from all re0ie&ers#$a2 special attention to ma1ing sure that the re0ie&ers from &ithin the state feelin0ol0ed# :!ot e0er2 re0ie&er needs to address e0er2 ob%ecti0e, but ma1e surethat e0er2one is included in the process#;

-# 3se 2our print+out to call on people &ho coded 4O> le0els differentl2 from thecoding of other members of the group, and as1 them to e@plain &h2 the2 codedthe ob%ecti0e to the particular 4O> le0el# ?e sure the2 use the 4O> definitionsto %ustif2 their ans&ers#

.# Once t&o re0ie&ers ha0e described ho& the2 ha0e coded an ob%ecti0edifferentl2, as1 a third re0ie&er to highlight the differences bet&een these t&ointerpretations#

6# Restate and summarie to re0ie&ers 2our interpretation of &hat the re0ie&ersha0e agreed on and &hat the2 ha0e disagreed on#

(# =f there is a difference in interpretation of the ob%ecti0eBs terminologyorexpectations, appeal to a re0ie&er &ith e@perience in teaching that grade le0el&ith these standards to discern ho& the stateBs teachers might be interpreting theob%ecti0e#

/# As1 if an2one, through other re0ie&ersB e@planations, no& &ants to change his orher mind about their original coding#

*# =f the 0ie&points on the 4O> le0el of an ob%ecti0e are di0ided, point to the mostli1el2 s1ills or content 1no&ledge required in the ob%ecti0e, not the more e@tremepossibilities the ob%ecti0e might allo& for#

9# As the facilitator, tr2 not to dominate the consensus process# E0en if 2ou ha0estrong feelings about the 4O> le0el of an ob%ecti0e, &ait to see if otherre0ie&ers highlight 2our point#

--


38/184

#tep 5: Training (e1iewers )2hase II*

JAs in the case of Step 6, this step onl2 needs to be ta1en once, rather than for each stud2 2our groupperforms#K

a; Coding sample items

!o& that 2our group has completed its first consensus process, members should befamiliar &ith the standards for that grade# $ic1 four or fi0e items on the assessment andha0e the re0ie&ers code them :indi0iduall2, &ith pencil and paper, not using the WAT;according to)/ level,primary2secondary ob"ectives, andsource of challenge# Whenthe2 are finished, go through these together to ma1e sure people understand and agree onthe process of coding# $ic1 these sample items from different parts of the test in order toget some range of sub%ect matter and item st2le#

?e sure to emphasie the distinction bet&een Source of Challenge and !otes# Re0ie&ersoften forget to loo1 for specific source+of+challenge issues unless the2 are reminded of it#

b; Coding items using the WAT

4etails about ho& to code assessment items using the WAT are included in 1ection Bofthis part :+art **; of the Training Manual#

#tep 6: ,ntering the $roup Consensus D78 .alues

Once 2ou ha0e the group consensus 4O> 0alues for the ob%ecti0es, 2ou must enter theminto the WAT# This step ta1es 0er2 little time and ma2 be carried out &hile the re0ie&ersare coding the assessment items for the appropriate grade le0el#

To enter the results of the group consensus, the 8roup "eader must log in as a GroupLeader$and then go to Re0ie/and +art *# Enter 1a0eat the bottom of each page as 2ou

enter the group consensus 0alues#When 2ou are finished, proceed to +art **of the re0ie& process coding the assessmentitems# The steps for this process are described in 1ection Bof this part of the Manual#

-.


39/184

#tep 9: %eading the De&riefing 2roess

After re0ie&ers ha0e coded the items on the initial assessment, the 8roup "eader should

facilitate a group discussion of the process# This discussion is important in the effort toma1e sure that re0ie&ers are interpreting the assessment based on the defined depth+of+1no&ledge le0els# =n addition, the ans&ers gi0en b2 re0ie&ers to the debriefing questionsma2 be referred to in the reports, if the2 e@plain an aspect of the assessment or theprocess that is &orth2 of note# The follo&ing questions :included as a printable Appendi@in Section =le0els; 2ou e@pected on the competenc2L =f not, &hat competenc2 &as not assessed at anappropriate depth+of+1no&ledge le0el and ho& should the assessment of the competenc2be impro0edL

C# Compared to competencies and assessments in other grades, did the competencies andassessment at this grade sho& an appropriate progression in content and depth of

1no&ledgeL

4# What is 2our general opinion of the alignment bet&een the competencies andassessment

i# $erfect alignmentii# Acceptable alignmentiii# !eeds slight impro0ementi0# !eeds ma%or impro0ement0# !ot aligned in an2 &a2L

E# Other commentsLAfter you have finished leading the debriefing process$ you have finished your

responsibilities as Group Leader. &e than0 you for your participation. -or information

on how to interpret or create reports$ the steps involved in planning an alignment

seminar$ or using the C) version of the tool$ turn toParts IV, V$ or VIof this %rainingManual.

-6


40/184

#etion : (e1iewers

What a Re0ie/er Must ,o in the Alignment +rocess

5# +artici!ate in trainingb2 2our 8roup "eader on depth+of+1no&ledge le0els andho& to use the Web Alignment Tool :WAT;#

# Register /ith your grou!on the WAT#

-# For each grade,assign de!th-o-6no/ledge (,9) 0alues to each ob5ecti0e inthe state standard#

.# With the facilitation of the 8roup "eader, discuss &ith the other re0ie&ers ho&

2ou coded the ob%ecti0es in order toreach a consensus on the 4O> le0els of theob%ecti0es#

6# ode the assessment itemsfor each grade# For each item, assign an appropriate4O> le0el and select the ob%ecti0e:s; that the item targets#

(# ,ebrie /ith your 4rou! Leaderfor each grade#

#tep ': Training on D78 %e1els and the WAT

Dour 8roup "eader &ill offer directions on using the Web Alignment Tool :WAT; and&ill train 2ou on the depth+of+1no&ledge :4O>; le0els# The second phase of thistraining, on ho& to code assessment items, &ill come after 2ou participate in 2our firstconsensus process &ith the other re0ie&ers# 4etails on using the WAT and 4O> le0elsare presented on the follo&ing pages#

-(


41/184

#tep +: (egistering as a (e1iewer

3eviewerscan register &ith the WAT b2 going to the Loginpage

:&&&cer#&isc#edu&at; and entering the follo&ing information

8roup =4 number on the first screen :obtained from 8roup "eader; 8roup Title Start 4ate the da2 the Alignment =nstitute begins End 4ate the da2 the =nstitute ends State Content Area Mathematics, "anguage Arts, Science, Social Sciences 4istrict :optional; Re0ie&erBs 8rade Co0erage

!ote This information entered b2 indi0idual re0ie&ers does not limit in an2 &a2 thet2pes of studies that can be conducted b2 this group# For e@ample, the Start 4ate andEnd 4ate do not limit the time during &hich studies are conducted# =n addition, thecontent area does not limit the studiesG 2ou ma2 register the group as a Mathematicsgroup and still do studies &ith "anguage Arts# The same is true &ith other elements,such as state, grade co0erage, etc#;

#tep -: Assigning D78 .alues to ,ah 7&;eti1e in the #tate #tandard

3nderstanding and 3sing the 4epth+of+>no&ledge :4O>; "e0els

=nterpreting and assigning 4O> le0els both to ob%ecti0es &ithin standards and toassessment items is an essential requirement of alignment anal2sis#?efore beginning the re0ie& process, 2ou should be adequatel2 trained to identif2,understand, and appl2 the different 4O> le0els for items and ob%ecti0es &ithin theircontent area#

4O> "e0el Title of "e0el

5 Recall S1ills and Concepts- Strategic Thin1ing. E@tended Thin1ing

-/


42/184

)etailed definitions$ explanations$ and examplesfor the 4O> le0els in mathematics,science, and readinglanguage arts are pro0ided in +art ***# After de0eloping a strongunderstanding of the different 4O> le0els, 2our tas1 for each stud2 is to assign a 4O>le0el to each ob%ecti0e &ithin the grade le0el of that stud2# The follo&ing guidelines are

helpful &hen considering &hich 4O> le0el to assign an ob%ecti0e The 4O> le0el of an ob%ecti0e should be the le0el of &or1 students are most

commonl2 required to perform at that grade le0el to successfull2 demonstratetheir attainment of the ob%ecti0e#

The 4O> le0el of an ob%ecti0e should reflect the complexity of the ob%ecti0e,rather than its difficulty# The 4O> le0el describes the 1ind of thin1ing in0ol0ed ina tas1, not the li1elihood that the tas1 &ill be completed correctl2#

=n assigning a 4O> le0el to an ob%ecti0e, thin1 about the complete domain of

items that &ould be appropriate for measuring the ob%ecti0e# =dentif2 the depth+of+1no&ledge le0el of the most common of these items#

=f there is a question regarding &hich of t&o le0els an ob%ecti0e addresses, such as"e0el 5 or "e0el , or "e0el or "e0el -, it is usuall2 appropriate to select thehigher of the t&o le0els#

The team of re0ie&ers should reach consensus on the 4O> le0el for eachob%ecti0e before coding an2 items for that grade le0el#

3sing the WAT to Assign 4O> le0els of ob%ecti0es use the follo&ing steps

'# Clic1 on the Loginlin1 at the top of the first screen and log in &ith username and group =4#

$# Once 2ou are logged in, clic1 on the Re0ie/lin1 at the top of the screen#

?# On the follo&ing screen, clic1 on +art *:coding ob%ecti0es;#

@# At this point, a ne& page &ill appear &here 2ou must select a standard to re0ieThere are t&o &a2s to select the standard 2ou &ant to re0ie&

a# Search for the standard b2 state and content area# After 2ou select the state andcontent area and clic1 on 1earch, a table appears, and 2ou select the standard 2ou&ish to re0ie

-*


43/184

b# Select the standard from the table at the bottom of the page b2 clic1ing 1electin the column to the right of the desired standard# This table is a list of standards2our group has &or1ed &ith before#

After 2ou select the standard for re0ie&, 2ou are directed to a ne& page to actuall2

assign the 4O> 0alue to the ob%ecti0e# When 2ou are finished &ith all of the ob%ecti0es on a page, clic1 on 1a0e at the bottom of the page# This mo0es 2ou to the ne@t page of ob%ecti0es#

Fig# #5*

#tep /: (eahing Consensus on D78 %e1els for 7&;eti1es

As part of determining the degree to &hich assessment items are aligned &ithstandardsob%ecti0es, the team of re0ie&ers must read each ob%ecti0e and reach groupconsensus on the appropriate 4O> le0el for the ob%ecti0es# This is an integral part of theanal2sis and pertains to both the Web and C4 0ersions of the tool#

With the 8roup "eader facilitating, each team of re0ie&ers &ill participate in theconsensus process b2 discussing, as a group, the depth+of+1no&ledge 0alues the2assigned to each of the ob%ecti0es# This process is an opportunit2 for all re0ie&ers toe@plain &h2 the2 assigned a particular 4O> le0el and to demonstrate ho& it applies tothe ob%ecti0e# The 8roup "eader &ill record and enter the results of the consensus processas the groupBs results for the state standardsob%ecti0es#

-9


44/184

#tep 0: Coding #tate Assessment Items

After reaching consensus on the 4O> le0els for the ob%ecti0es, the ne@t step for the teamof re0ie&ers is to assess the comple@it2 of thestate assessment itemsfor the particulargrade and to match the items&ith the state standards that most closel2 relate to the items#This is done b2 coding each assessment item using the 4O> definitions used to code theob%ecti0es, and then determining &hich standardob%ecti0e reflects the s1ill being testedb2 each assessment item#

The first time the re0ie&ers underta1e this process, the2 &ill be gi0en additional trainingat this point# The 8roup "eader &ill select four or fi0e items from the assessment andha0e the re0ie&ers code them indi0iduall2, according to)/ Level$ 4rimary2econdaryb"ective %argeted$and ource of Challenge# Then the re0ie&ers &ill discuss their

results#

Coding Assessment =tems

After discussing the sample items, re0ie&ers can proceed to code the assessment items#As &ith the practice re0ie&, re0ie&ers are allo&ed to identif2 one assessment item ascorresponding to up to three ob%ecti0esone primar2 hit :ob%ecti0e; and up to t&osecondar2 hits# Re0ie&ers can onl2 code one 4O> le0el to each assessment item, e0en ifthe item corresponded to more than one ob%ecti0e# 4uring this time, re0ie&ers shouldalso identif2sources of challenge:for each item; and record an21otes#

The follo&ing directions to re0ie&ers are for +art ** of the re0ie& process, during &hichre0ie&ers assign 4O> le0els to the assessment items

5# "og in as a re0ie&er and clic1 on the Re0ie/lin1 at the top of the pageG &hen the page loads, clic1 on +art **#

# After clic1ing on +art **$a page &ill appear that pro0ides instructions oncoding the assessment items# After reading the instructions, clic1 on ontinue.

-# A page &ill load &ith a table that contains all of the assessments the 8roup "eader assigned to the group :see Fig# #59;# Select the particular assessment 2ou &ish to code#

.)


45/184

Fig# #59

.# The ne@t page &ill be the page on &hich the coding information is entered# The re0ie&er is as1ed to enter the 4O> le0el, primar2 ob%ecti0e, and up to t&o secondar2 ob%ecti0es, as &ell as an2 source+of+challenge issues or notes for each assessment item# =f re0ie&ers prefer not to use their paper cop2 of the standards, the2 ma2 also select the lin1 entitled +o!u! 1tandardfor a list of the state standardsob%ecti0es# Fig# #), &hich sho&s an e@ample of the assessment item re0ie& :coding; page, follo&s#

Fig# #)

When 2ou are finished coding an item, clic1 1a0eto mo0e to the ne@t item in the

assessment# Dou ma2 page through e@isting coded assessment items b2 clic1ing on the +re0

*temand Ne3t *temlin1s, to&ard the bottom of the page# Dou ma2 also go bac1 to completed assessment items b2 clic1ing on the drop+

do&n bo@ ne@t to the 4o To *temlin1 and selecting the item number 2ou &ant#

.5


46/184

Once 2ou select the appropriate item number, clic1 on the 4o To *temlin1 and2ou &ill be ta1en to that item#

*n!ut 1ummary Tablesho&s all of the item data 2ou ha0e entered into thes2stem#

=f 2ou clic1 Auto 8ill With +re0ious 1elections, the data that 2ou entered for the

pre0ious item &ill immediatel2 appear on the screen for the current item#

=f 2ou &ould li1e hints about ho& to sa0e time coding items, clic1 on the *temoding


47/184

#tep 3: De&riefing with your $roup

After 2ou ha0e finished coding each assessment, 2our 8roup "eader &ill facilitate adebriefing discussion# This discussion &ill allo& re0ie&ers to offer more specificcomments about their impressions of the alignment bet&een the assessment and thestandards# The state coordinators &ill use these comments to gain perspecti0e on thenumerical alignment data#

,pon completion of the debriefing process$ you have completed your responsibilities as areviewer6 &e than0 you for your participation. If you are interested in learning how to

interpret or create reports$ the steps involved in planning an alignment seminar$ or using

the C) version of the tool$ turn toParts IV, V,or VIof this %raining Manual.

.-


48/184

..


49/184

***. ,e!th-o-no/ledge-Le0els

#etion A< Mathematis D78 %e1elsLevel * 73ecall8includes the recall of information such as a fact, definition, term,

or a simple procedure, as &ell as performing a simple algorithm or appl2ing a formula#That is, in mathematics, a one+step, &ell defined, and straight algorithmic procedureshould be included at this lo&est le0el# Other 1e2 &ords that signif2 "e0el 5 includeidentif2,N recall,N recognie,N use,N and measure#N e2&ords thatgenerall2 distinguish a "e0el item include classif2,N organie,N Nestimate,N ma1eobser0ations,N collect and displa2 data,N and compare data#N These actions impl2 morethan one step# For e@ample, to compare data requires first identif2ing characteristics ofob%ects or phenomena and then grouping or ordering the ob%ects# Some action 0erbs, suchas e@plain,N describe,N or interpret,N could be classified at different le0els dependingon the ob%ect of the action# For e@ample, interpreting information from a simple graph, orreading information from the graph, also are at "e0el # =nterpreting information from a

comple@ graph that requires some decisions on &hat features of the graph need to beconsidered and ho& information from the graph can be aggregated is at "e0el -# "e0el acti0ities are not limited onl2 to number s1ills, but ma2 in0ol0e 0isualiation s1ills andprobabilit2 s1ills# Other "e0el acti0ities include noticing or describing non+tri0ialpatterns, e@plaining the purpose and use of e@perimental proceduresG carr2ing oute@perimental proceduresG ma1ing obser0ations and collecting dataG classif2ing,organiing, and comparing dataG and organiing and displa2ing data in tables, graphs, andcharts#

Level : 7trategic %hin0ing8requires reasoning, planning, using e0idence, and ahigher le0el of thin1ing than the pre0ious t&o le0els# =n most instances, requiring

students to e@plain their thin1ing is at "e0el -# Acti0ities that require students to ma1econ%ectures are also at this le0el# The cogniti0e demands at "e0el - are comple@ andabstract# The comple@it2 does not result from the fact that there are multiple ans&ers, apossibilit2 for both "e0els 5 and , but because the tas1 requires more demandingreasoning# An acti0it2, ho&e0er, that has more than one possible ans&er and requiresstudents to %ustif2 the response the2 gi0e &ould most li1el2 be at "e0el -#

.6


50/184

Other "e0el - acti0ities include dra&ing conclusions from obser0ationsG citing e0idenceand de0eloping a logical argument for conceptsG e@plaining phenomena in terms ofconceptsG and deciding &hich concepts to appl2 in order to sol0e a comple@ problem#

Level ; 7Extended %hin0ing8requires comple@ reasoning, planning, de0eloping,

and thin1ing, most li1el2 o0er an e@tended period of time# The e@tended time period isnot a distinguishing factor if the required &or1 is onl2 repetiti0e and does not requireappl2ing significant conceptual understanding and higher+order thin1ing# For e@ample, ifa student has to ta1e the &ater temperature from a ri0er each da2 for a month and thenconstruct a graph, this &ould be classified as a "e0el # 7o&e0er, if the student is toconduct a ri0er stud2 that requires ta1ing into consideration a number of 0ariables, this&ould be a "e0el .# At "e0el ., the cogniti0e demands of the tas1 should be high and the&or1 should be 0er2 comple@# Students should be required to ma1e se0eral connectionsrelate ideas withinthe content area or amongcontent areasand ha0e to select oneapproach among man2 alternati0es on ho& the situation should be sol0ed, in order to beat this highest le0el# "e0el . acti0ities include designing andconducting e@periments and

pro%ectsG de0eloping and pro0ing con%ectures, ma1ing connections bet&een a finding andrelated concepts and phenomenaG combining and s2nthesiing ideas into ne& conceptsGand critiquing e@perimental designs#

E3am!les A!!lied to 9b5ecti0es and Assessment *tems

Sample Mathematics Ob%ecti0es

3se the mathematics 4O> le0els on the pre0ious pages to determine the 4O> le0els forthe follo&ing fi0e sample ob%ecti0es# When 2ou are finished, turn the page to see &hether2ou agree &ith the &a2 wecoded these ob%ecti0esH Then tr2 using the 4O> le0els on the56 sample mathematics items that follo

Ob%ecti0e 5# Read, &rite, and compare decimals in scientific notation#

Ob%ecti0e # :8rade *; Sol0e t&o+step linear equations and inequalities in one 0ariableo0er the rational numbers, interpret the solution or solutions in the conte@t from &hichthe2 arose, and 0erif2 the reasonableness of results#

Ob%ecti0e -# :8rade *, from the !EA$ Mathematics Frame&or1; 4esign a statisticale@periment to stud2 a problem and communicate the outcomes#

Ob%ecti0e .# Compute &ith numbers :that is, add, subtract, multipl2, di0ide;#

Ob%ecti0e 6# Construct t&o+dimensional patterns for three+dimensional models, such asc2linders and cones#

.(


51/184

4O> "e0els of the Sample Mathematics Ob%ecti0es

Ob%ecti0e 5# This ob%ecti0e is an e@ample of "e0el 5# The highest demand for studentsto successfull2 meet this e@pectation requires them to use recall and use a routine method

to con0ert a decimal to scientific notation#Ob%ecti0e # This ob%ecti0e is an e@ample of "e0el -# The e@pectation e@pressed in thisob%ecti0e is that students &ill not onl2 sol0e a t&o+step linear equation, but &ill alsointerpret the solution and 0erif2 the results# This &ill require students to do somereasoning in order to interpret the solution and could be fairl2 comple@, depending on theconte@t# =f students &ere onl2 required to sol0e linear equations and 0erif2 solutions, thenthe e@pectation &ould be "e0el #

Ob%ecti0e -# To plan a statistical e@periment, a student must define the problem andde0elop a procedure for sol0ing it# This requires that the student identif2 the correct

statistical model, appl2 the model to data, and communicate the outcome of the selectedmodel# The student must interpret findings and ma1e reasonable and rationed inferencesfrom obtained data# This represents comple@, multistep reasoning and reflects a "e0el .tas1#

Ob%ecti0e .# This ob%ecti0e requires students to conduct basic calculations# This is"e0el 5 because it in0ol0es routine processing and in0ol0es a one+step process#

Ob%ecti0e 6# This ob%ecti0e is an e@ample of "e0el # Although recogniing anddra&ing a t&o+dimensional pattern, or a regular c2linder, is e@pected to be routine :"e0el5;, building a three+dimensional model &ould not be as routine# =t &ould require at least

t&o steps first, recogniing the shape and, second, dra&ing a t&o+dimensional ob%ect toreflect the shape in three dimensions#

Sample Mathematics Assessment =tems

!o& tr2 coding some sample assessment items using the Mathematics 4O> "e0els#Sample items for three different grade le0els# After 2ou are finished coding these, readour Ans&ersN on the page that follo&s#

./


52/184

JThe follo&ing fi0e items are from 8rade . mathematics assessmentsK

')

$)

Sam, Terr2, and >im each o&n some baseball cards that Ted is &illing totrade them for# 7ere is &hat the2 are &orth

SamBs cards TetsuoBs cards >imBs cards

?ret ?oone )#*)

Samm2 Sosa 5#-) Rand2ohnson

)#/)

Andres8alarraga )#.) 8regMaddu@ 5#)) ?arr2 ?onds #-)Mar1 Mc8uire 5#6

)

Ted &ill trade his Ale@ Rodrigue card for (#/6 &orth of cards# What isthe best trade that Sam, Tetsuo, and >im can ma1e for TedBs Ale@Rodrigue cardL

What trade could Sam, Tetsuo, and >im offer Ted that &ould be the mostfair bet&een Sam, Tetsuo, and >imL

E@plain 2our thin1ing and sho& all 2our &or1#?)

.*


53/184

@)

Thin1 carefull2 about the follo&ing question# Write a complete ans&er# Dou ma2 use dra&ings,&ords, and numbers to e@plain 2our ans&er# ?e sure to sho& all of 2our &or1#"aura &anted to enter the number *-/6 into her calculator# ?2 mista1e, she entered thenumber */6# Without clearing the calculator, ho& could she correct her mista1eL

Without clearing the calculator, ho& could she correct her mista1e another &a2L

&)

?ased on the map abo0e, about ho& man2 miles is the shortest route from Oa1dale to FentonLA; 5))?; /)C; .)4; )

.9


54/184

JThe follo&ing fi0e items are from 8rade * assessmentsK

)

)

C)

6)

From an2 0erte@ of a .+sided pol2gon, 5 diagonal can be dra&n#From an2 0erte@ of a 6+sided pol2gon, diagonals can be dra&n#From an2 0erte@ of a (+sided pol2gon, - diagonals can be dra&n#From an2 0erte@ of a /+sided pol2gon, . diagonals can be dra&n#7o& man2 diagonals can be dra&n from an2 0erte@ of a )+sided pol2gonL


55/184

D)

The school ne&spaper conducted a sur0e2 about &hich ingredient &asmost preferred as a pia topping# This graph appeared in the ne&spaperarticle#

What information &ould best help 2ou determine the number of peoplesur0e2ed &ho preferred sausageL

A number of people sur0e2ed and t2pe of sur0e2 usedB t2pe of sur0e2 used and ages of people sur0e2ed percent 0alues sho&n on chart and number of people sur0e2ed, ages of people sur0e2ed and percent 0alues sho&n on chart

'%)

"oo1 at the dra&ing# The numbers alongside each column and ro& are the total of the0alues of the s2mbols &ithin each column and ro What should replace the questionmar1L

A. -B. 6. *,. -)E. -

65


56/184

JThe follo&ing fi0e items are from 8rade 55 assessmentsK

'') Which of the follo&ing is !OT true for an2 0alue ofxL

A x < x= < x>

B x> < x < x=

x= < x < x>

, x < x> < x=

E @U V @ V @

'$) $la2ers A and ? are pla2ing a game# On a table bet&een them is a stac1 of npennies# First, $la2er A remo0es either one or t&o pennies from the stac1# Then $la2er ?remo0es either one or t&o pennies from the stac1# The2 alternate in this &a2 until nopennies remain# The loser is the pla2er &ho remo0es the last penn2 from the stac1#

=f the2 start the game &ith 6 pennies in the stac1, ho& man2 pennies should$la2er A ta1e from the stac1 on her first turnL Wh2L

=f the game starts &ith / pennies in the stac1, &ould 2ou rather be $la2er A or ?LWh2L

For &hat 0alues of n, if an2, is it best to be pla2er ALFor &hat 0alues of n, if an2, is it best to be pla2er ?L

E@plain and %ustif2 2our ans&ers#

6


57/184

'?)

'@)

One plan for a state income ta@ requires those persons &ith income of 5),))) or less to pa2no ta@ and those persons &ith income greater than 5),))) to pa2 a ta@ of ( percent onl2 onthe part of their income that e@ceeds 5),)))#

A personXs effecti0e ta@ rate is defined as the percent of total income that is paid in ta@#

?ased on this definition, could an2 personXs effecti0e ta@ rate be 6 percentL Could it be (percentL E@plain 2our ans&er# =nclude e@amples if necessar2 to %ustif2 2our conclusions#

'&)

? a2b @ c2d

=f ) V a V b V c V d in the equation abo0e, then the greatest increase in &ould resultfrom adding 5 to the 0alue of &hich 0ariableL

(A)a(B)b()c(,)d(E)%here is not enough information to 0now for certain.

6-


58/184

4O> "e0els for the Sample Mathematics Assessment =tems

8rade . =tems

5; "e0el # The choices offered indicate that this item is intended to identif2students &ho &ould simpl2 subtract 9 minus 5 to get an *# More than one step is requiredhere# The students must first recognie the difference bet&een a#m# and p#m# and ma1esome decisions about ho& to ma1e this into a subtraction problem, then do thesubtraction#

; "e0el .# This is a comple@ open+ended problem requiring students to ma1ese0eral connections and appl2 one approach among man2#N =t requires the students toplan and organie, and to &eigh solutions based on different 1inds of criteria# Studentsshould be allo&ed at least ) minutes for this problem, &hich is an e@tended period oftime for a test item#

-; "e0el 5# Students onl2 need to be able to recognie e0en numbers#

.; "e0el -# An acti0it2 that has more than one possible ans&er and requiresstudents to %ustif2 the response the2 gi0e &ould most li1el2 be a "e0el -#N Since there aremultiple possible approaches to this problem, the student must ma1e strategic decisionsabout ho& to proceed, &hich is more cogniti0el2 comple@ than simpl2 appl2ing a setprocedure or s1ill#

6; "e0el 5# This measurement item requires no anal2sis of the map itself, since theroute in question is a straight line# =f the line &ere not straight, this item &ould require

estimation and perhaps e0en calculation, ma1ing it "e0el or -#

8rade * =tems

(; "e0el 5# The first thing to note is that this is not reall2 a geometr2 item# Rather, itsimpl2 requires students to notice an eas2, routine pattern# 4O> le0els are difficult toassign for man2 pattern+recognition problems, because the2 depend on ho& routine thepattern is# This particular pattern is immediatel2 recogniable and requires no processing,but a more comple@ pattern could ma1e this a "e0el or e0en "e0el - item#

/; "e0el # This item is included in order to contrast it &ith the pre0ious item#$attern recognition is required, but the non+routine nature of this pattern brings the itemup to a higher 4O> le0el# Some anal2sis and generaliation is required in order tounderstand and e@tend this pattern#

*; "e0el # There are a number of different concepts and procedures that can beused for this problem, rather than an ob0ious, simple one# Students must not onl2 be ableto identifydifferent representations of rational numbers :"e0el 5;, but also to manipulate

6.


59/184

and comparethese representations :"e0el ;# This means that numerous interdependentand non+tri0ial steps are in0ol0ed here# 7o&e0er, this does not require an2 con%ecturing,planning, abstracting, or e@plaining, so it is not "e0el -#

This item demonstrates the importance of 1no&ing &hether calculators are

allo&ed on the e@aminations or not# =f a calculator &ere allo&ed on this e@amination, this&ould clearl2 be "e0el 5, instead of "e0el #

9; "e0el # This is an e@ample of ho& a problem that is multiple choice can reduceits 4O> le0el# =f the multiple choices &ere remo0ed here and the problem &ere left open+ended, it &ould be "e0el -# ?ut here the student need onl2 &eigh the options against oneanother, easil2 discarding t2pe of sur0e2 usedN and ages of people sur0e2edN asboge2s# So the2 can easil2 determine that C is immediatel2 better than A or 4, &ithoute0en ha0ing to thin1 anal2ticall2 or creati0el2 about &h2 percent 0alues sho&n ornumber of people sur0e2ed &ould be important information to 1no

5); "e0el -# This item can be approached through a number of 0iable strategiespattern recognition, guess+and+chec1, algebra, etc# This freedom means that the studentmust ma1e choices and assumptions# Furthermore, no matter &hat strateg2 she emplo2s,she must 1eep trac1 of a comple@ logical chain# The multiple choices pro0ided do notma1e this tas1 an2 less comple@#

8rade 55 =tems

55; "e0el -# This is another e@ample of an item that is at "e0el - &ithout being open+ended# This item requires generaliation, reasoning, and h2pothesis testing, in0ol0ing

some creati0it2 in choosing e@amples that test the h2potheses#5; "e0el .# This problem requires students to form game strategies, create data,notice number patterns, and %ustif2 ho& and &h2 those patterns arise# =t in0ol0esinducti0e, deducti0e, and strategic reasoning o0er an e@tended period of time, perhaps -)minutes# This ma2 e0en be a problem best done in pairs or groups &ithin a testingen0ironment#

5-; "e0el # This item is not "e0el 5 because it is not routine, nor does it focus on amemoried definition or procedure# =n fact, it in0ol0es numerous steps, because itrequires students to compare se0eral different pairs of shapes before arri0ing at the

correct ans&er# For these reasons, man2 spatial reasoning items are "e0el #!ote that this ma2 be a coded as a source+of+challenge item, because choice C

seems to be dra&n in a misleading &a2#

66


60/184

5.; "e0el -# This item gi0es the student a ne& definition and as1s her to use it as abasis for reasoning# =n order to ascertain &hether the student reall2 understands theas2mptotic beha0ior that ma1es a (Y effecti0e rate impossible, this item mustbe open+ended# This is &h2 most "e0el - items are open+response items, because the comple@it2of thin1ing the2 require the students to displa2 could not be displa2ed using the multiple

choice option#56; "e0el -# =f a multiple+choice item is "e0el -, often it is because the multiplechoices do not constrain or guide the possible solutions# The choices here allo& for allpossible responses to this item, including the response that the problem cannot be sol0ed#This gi0es such an item the character of an open+ended item, e0en though it is not one#

6(


61/184

#etion < #iene D78 %e1els

$lease note that, in the sciences, 1no&ledgeN can refer both to content1no&ledge and 1no&ledge of scientific processes# This distinction in the meaning of1no&ledge is consistent &ith the1ational cience Education tandards :!SES;, &hichterms Science as =nquir2N as its first Content Standard#

Level * 73ecall and 3eproduction8requires the recall of information, such as afact, definition, term, or a simple procedure, as &ell as performance of a simple scienceprocess or procedure# "e0el 5 onl2 requires students to demonstrate a rote response, use a&ell+1no&n formula, follo& a set procedure :li1e a recipe;, or perform a clearl2 definedseries of steps# A simpleN procedure is &ell defined and t2picall2 in0ol0es onl2 one step#e2&ords that generall2 distinguish a "e0el item include classif2,N organie,N Nestimate,N ma1e obser0ations,N collect and displa2data,N and compare data#N These actions impl2 more than one ste!# For e@ample, tocompare data requires first identif2ing characteristics of the ob%ects or phenomena andthen grouping or ordering the ob%ects# "e0el acti0ities include ma1ing obser0ations and

collecting dataG classif2ing, organiing, and comparing dataG and organiing anddispla2ing data in tables, graphs, and charts# Some action 0erbs, such as e@plain,Ndescribe,N or interpret,N could be classified at different 4O> le0els, depending on thecomple@it2 of the action# For e@ample, interpreting information from a simple graph,requiring reading information from the graph, is a "e0el # An item that requiresinterpretation from a comple@ graph, such as ma1ing decisions regarding features of thegraph that need to be considered and ho& information from the graph can be aggregated,is at "e0el -# Some e@amples that represent, but do not constitute all of, "e0el

6/


62/184

performance, are Specif2 and e@plain the relationship bet&een facts, terms, properties, or 0ariables# 4escribe and e@plain e@amples and non+e@amples of science concepts# Select a procedure according to specified criteria and perform it# Formulate a routine problem, gi0en data and conditions# Organie, represent, and interpret data#

Level : 7trategic %hin0ing8requires reasoning, planning, using e0idence, and ahigher le0el of thin1ing than the pre0ious t&o le0els# The cogniti0e demands at "e0el -are comple@ and abstract# The comple@it2 does not result onl2 from the fact that therecould be multiple ans&ers, a possibilit2 for both "e0els 5 and , but because the multi+step tas1 requires more demanding reasoning# =n most instances, requiring students toe@plain their thin1ing is at "e0el -G requiring a 0er2 simple e@planation or a &ord or t&oshould be at "e0el

Documents

Webb Et Al 2005 Web Alignment Tool (WAT) Training Manual 2.1 Draft 091205