Understanding Assessment Mark Leal Director of Assessment Alaska Department of Education and Early...

Understanding Assessment

Mark LealDirector of AssessmentAlaska Department of Education and Early Developmentmark_leal@eed.state.ak.usOffice phone number- 465-8691

STANDARDS-BASED REFORMSTANDARDS-BASED REFORM

Content and Performance Content and Performance

StandardsStandardsStateState clear expectations clear expectations

Standards-based Standards-based teststestsTest Test StandardsStandards

Adjust instructionAdjust instruction to achieve standards

RemediationRemediationstandards-based instruction

Standards-based Standards-based

instructioninstructionTeach Teach StandardsStandards

EXPECTATIONS FOR STANDARDS-BASEDASSESSMENT SYSTEMS

Rigorous Rigorous standardsstandards

TestsTests based on and aligned with based on and aligned with standardsstandards

Fair, valid, reliable Fair, valid, reliable teststests and and assessment assessment systemsystem

Systems that include Systems that include ALLALL students students

Results reported to public/stakeholdersResults reported to public/stakeholders

STANDARDS-BASEDSTANDARDS-BASEDASSESSMENT ASSESSMENT SYSTEMSSYSTEMS

Individual studentsIndividual students

reportreportresultsresults

forfor

comparecompareresultsresults

by subgroup forby subgroup for

StatewideStatewide

DistrictsDistricts

SchoolsSchoolsDisability status

English proficiency

Low-income status

Migrant status

Gender/Race/Ethnicity

USING ASSESSMENT ASSESSMENT SYSTEMSSYSTEMS TO MAKE DECISIONS

ASSESSMENT SYSTEMS(Possible Components)

Norm-referenced testsNorm-referenced tests

compares student performance to norm group

Standards-based testsStandards-based tests

compares student performance to standard

Blended assessment systemsBlended assessment systems

combines norm-referenced and standards-based assessments

STANDARDS-BASED TESTSSTANDARDS-BASED TESTS

Also known as criterion referenced testsAlso known as criterion referenced tests compares performance to standards or criteriacompares performance to standards or criteria

Sixmile Creek is a comfortable distance from Anchorage. However it is not a comfortable river. Sixmile has similar character to many Washington and Oregon whitewater runs: lush forest, deep canyons, and tremendous rapids. The river is easily accessible from Seward Highway. Sixmile Creek has three distinct sections. The Upper Canyon rates a Class IV. The toughest stretch is the lower canyon, where the river rampages along at a constant Class V for two killer miles.

The Wildest One!Difficulty: Class V Season: May thru September Trip Length: Half-day to one day

Whitewater Classifications

Most experts use a I to VI rating system for whitewater difficulty. Here’s a quick rundown of what to expect from each classification:

I- Small rapids with low waves and no obstructions

II- More frequent rapids with few or no obstructions

III- Waves up to four feet, with some maneuvering around obstructions

IV- Very Difficult, extended rapids. Scouting is necessary.

V- Long and violent rapids. Large waves that are unavoidable. Complex course. Scouting is a must.

VI- Maximum difficulty. Involves serious risk of life.

TRIAL USFSA JUDGING FORMAdult Gold Free Skating TRIAL

Candidates’ Name _________________________________________USFSA # ____________________________

OFFICIAL JUDGES

TRIAL COMMENTS

Technical Merit

Presentation

Total Pass Retry Judge-on charge overwrite total in ink

Judge-in-Charge Signature_____________________________________________________ USFSA # ______________________Duration: Not to exceed 2 minutes, 40 seconds to the music of the skater’s choicePassing Average 3.0Candidate must give a strong performance and demonstrate a command of the following required elements showing very good form and carriage, strong true edges, effortless flow and preciseness to steps.JUMPS: ____Axel

____Lutz____Two different additional single jumps

________________________________

SPINS: ____Two different solo spins (minimum of 4 revolutions)

____One of the following: Split, Stag,

Host Club___________________________________________Date _______________________________

Falling Leaf, or Half Loop____One jump combination consisting of two

(2) different single jumps (no turnor change of foot between jumps)

________________ ________________

____One combination spin with at least one change offoot (minimum 4 revolutions in position)

STEPS: ____One step sequence – either straight line or circular) ____Connecting moves consisting of spirals, strong edges

good use of music and full utilization of ice surface

Extra elements may be added without penalty. Only two of the required elements may be retried, if necessaryTrial Judge’s Signature _______________________________________USFSA # _________________________Trial Judge’s Club ___________________________________Current Judging Status_______________________

SkaterTechnical

Merit Mark

Presentation Mark

Total of Marks

Ordinal Value

Amber 2.7 2.7 5.4 2

Jill 2.8 2.7 5.5 1

Tori 2.5 2.6 5.1 3

Haley 2.3 2.3 4.6 5

Nikki 2.4 2.4 4.8 4

STANDARDS-BASED TESTSSTANDARDS-BASED TESTScompares student performance to standardscompares student performance to standards

Results report levels of proficiency Advanced or exemplaryexemplary Proficient or masterymastery Below proficient or basicbasic Not-proficient or novicenovice

Items look more like classroom work

Multiple-choice Open-ended Extended Constructed-response and essay Short Constructed-response

StandardsStandards-Based TestsUSEUSE

Monitor mastery of Monitor mastery of standardsstandardsMeasure performanceMeasure performanceSample depth of Sample depth of student knowledge & student knowledge & skillsskillsAccurate with validity Accurate with validity & reliability& reliability

MISUSEMISUSEAs sole indicator of As sole indicator of learninglearning

As stand-alone As stand-alone evidence of a evidence of a school’s successschool’s success

As accurate without As accurate without validity & reliabilityvalidity & reliability

1115

75

66

1419

0 1

0102030405060708090

100

Novice NeedsImprovement

Proficient Advanced

District

State

NORM-REFERENCED NORM-REFERENCED TESTSTESTS

compares performance (data) to norm group.

Reference Population: Standard Normal Curve

95th5th

50th Percentile

Body Mass Index- Norm Referenced Measure

Boys: 2 to 20 years

BMI BMI

BMI BMI

Indicators of Nutritional StatusIndicators of Nutritional Status

<5th percentileStunting/shortness length or height-for-age

Head circumference-for-age

<5th percentile

<5th percentile>95th percentile

Underweight weight-for-length BMI-for-age

NORM-REFERENCED TESTSNORM-REFERENCED TESTS compares student performance to norm group

Results are reported asResults are reported as Standard scoreStandard score

Percentile scorePercentile score

Grade equivalentGrade equivalent

Half performed betterHalf performed better

Half performed worseHalf performed worse

Referenced to 50Referenced to 50thth %ile %ile

Norm-Referenced TestsUSE

To measure group To measure group

performanceperformance

To compare to norm groupTo compare to norm group

To measure subgroups To measure subgroups

represented in norm grouprepresented in norm group

As reliable & validAs reliable & valid

MISUSE

To measure individual To measure individual

student performancestudent performance

To compare to specific To compare to specific

standards through standards through

general domainsgeneral domains

To measure To measure

excluded/unaccounted excluded/unaccounted

for subgroupsfor subgroups

As modified without As modified without

new new

validation/reliabilityvalidation/reliability

05

101520253035404550

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

Majority

Hispanic

Black

Norm-Referenced Tests Does the NRT measure OUR studentsOUR students?

Does the score tell if the standardstandard is MET?

Do questions match OUR standardsOUR standards?

Did OUR experts match items/tasks to standardsstandards?

Are “alignment” studies specific to OUR OUR standardsstandards?

Does the NRT measure ALL students against OUR OUR standardsstandards?

BLENDED ASSESSMENTSBLENDED ASSESSMENTS

combines norm-referenced and standards-based testscombines norm-referenced and standards-based tests

Norm-referenced tests provide Comparisons to national normsComparisons to national norms

High levels of technical qualityHigh levels of technical quality

Familiarity and confidenceFamiliarity and confidence

Standards-based tests provide Comparisons to standardsComparisons to standards

Feedback on learning what is intendedFeedback on learning what is intended

Data useful to classroom teaching and Data useful to classroom teaching and learninglearning

Developmental Profile

Benchmark Tests

High School Graduation Qualifying Exam

Terra Nova CAT/6

National Assessment

of Educational

Progress

September 1 to

November 1

March

4, 5 & 6

October

28, 29 & 30

March

4, 5 & 6

March

4 - 21

January 27 to

March 7

K X K

1 X# 1

2 2

3 X 3

4 X X 4

5 X 5

6 X 6

7 X 7

8 X X 8

9 X 9

10 X 10

11 R R 11

12 R R 12

AlaskaLarge-Scale Assessment Matrix

LEGAL ISSUESTechnical quality of results

Validity and reliability of results Accuracy of results Appropriate use of results

Stakes for schools Stakes for students and teachers Legal issues

Rewards and sanctions Opportunity to learn Accounting for/Accommodating AllAll

Changes to the HSGQE

Framework for Continuous Renewal

SB 133

Committee Actions and Recommendations

Affirmed the Appropriateness of the Performance Standards

Reviewed bank of test questions Suggested that a number of questions be

changed or eliminated.

Suggested that the test be refocused

Suggested that certain portions of the test be revised.

Refocused HSGQE

Examination focusing on assessing minimum competencies in essential skills

Data used to insure that all students who receive a diploma in the state of Alaska can demonstrate minimum competencies in essential skills

Revised HSGQEMore specific direction in items and questions According to the article… Give a reason and list two examples

Format of Test Reading or charts on facing pages Writing check list

Varied Reading PassagesLength of TestReal World Focus and Application

Process for Development of Refocused and Revised HSGQE

New Test Item Maps Constructed

New Test Forms Generated based on new item maps Spring 2002 (Refocused)

New passing scores established August 2002

HSGQE Test Blueprint

ReadingMultiple choice

Short Response

Extended Response

Initial Test -Totals 40 13 1

R4.1 Use context clues 9 0 0R4.4 Summarize Information; Make connections 5 2 1R4.5 Support Main Idea; critique arguments 13 1 0R4.6 Read and apply multi-step directions 4 1 0R4.7 Analyze literary conventions & techniques 4 1 0R4.8 Analyze narrative elements 1 3 0R4.9 Make and support assertions 2 2 0R4.10 Analyze and evaluate themes 2 3 0

Refocused Test -Totals 35 15 0

R4.1 Use context clues 4 0 0R4.4 Summarize Information; Make connections 12 2 0R4.5 Support Main Idea; critique arguments 7 3 0R4.6 Read and Apply multi-step directions 4 3 0R4.9 Make and support assertions 4 4 0R4.10 Analyze and evaluate themes 4 3 0

Percent of Emphasis

100%

12%19%19%8%11%10%8%12%

100%

6%24%19%16%19%16%

HSGQE Test Blueprint

WritingMultiple choice

Short Response

Extended Response

Percent of Emphasis

Initial Test Totals 40 5 1 100%

W4.1/2 Write compositions /Demonstrate elements of discourse 0 4 1 32%W4.3 Use conventional English 18 1 0 35%W4.4 Revise Writing 22 0 0 32%

Refocused Test Totals 26 8 1 100%

W4.1/2 Write compositions / Demonstrate elements of discourse 0 4 1 54%W4.3 Use conventional English 14 2 0 27%W4.4 Revise Writing 12 2 0 19%

Percent of Emphasis

100%

10%14%14%21%21%20%

HSGQE Test Blueprint

MathematicsMultiple choice

Short Response

Extended Response

Intial Test Totals 52 10 2

A1: Numeration 6 1 0A2: Measurement 7 2 0A3: Estimation and Computation 7 2 0A4: Functions and Relationships 11 3 0A5: Geometry 11 1 1A6: Statistics and Probability 10 1 1

Refocused Test Totals 50 6 2

A1: Numeration 12 1 0A2: Measurement 10 2 0A3: Estimation and Computation 10 2 0A4: Functions and Relationships 6 0 1A5: Geometry 5 1 0A6: Statistics and Probability 7 0 1

100%

20%20%20%14%10%16%

What does this mean for the classroom?

HSGQE Refocused to assess minimum competencies of essential skills

Teachers need to continue teaching all of the standards to all of the students

Student performance needs to be evaluated using multiple assessments

Alaska State Assessment and TerraNova-CAT/6

Second EditionCopyright 2000K-12 Basic Skills Achievement BatteryThree assessment models Multiple Assessments Complete Battery Survey

TerraNova-CAT/6 Complete Battery Plus

Selected Response Items Subtests Reading/Language Arts* Mathematics* Science Social Studies Vocabulary* Language Mechanics* Computation* Spelling

TerraNova-CAT/6 Content

Objective Structure Developed across levels Thematic approach in

reading/language Real life context Model for CSSA item

Participation Guidelines

New booklets will be sent to schools by early December. Indicates 2002/2003 on cover

Will include changes specified by SB 133 Optional assessment program

Test SecurityBoth NRT, HSGQE and Benchmark Tests must be administered by school personnel at a school test center. Correspondence programs must

arrange to have tests administered by school personnel at school test centers.

Test materials must be stored in a secure manner

Test Security-continued

All test questions are confidential-no unauthorized access- TerraNova/CAT,HSGQE,Benchmark

HSGQE Associate test coordinator

Alaska Certification required Proctor ratio 1:30

Alaska Certification required Cannot be teachers who, at time of testing have

the student in class (in subject tested)

Duties of School Personnel

Ensure confidentiality of test questionsResponsible for test administration Inventory and track materials Control distribution, storage,

administration and collection of tests Ensure that no test is copied in any

manner

Duties of School Personnel- continued

District Test Coordinator-HSGQE Assure that each HSGQE test booklet

and test administration manual is accounted for from the time the materials arrive at the district office until the time they are returned to the test publisher.

Ensure adequate supervision of students taking the HSGQE

Duties of School Personnel- continued

School Test Center Code tests according to directions Inventory and track materials Securely store tests before and after use Control distribution of tests Ensure that:

no test is copied examinees only use reference materials specified

in directions no student receive a copy of test or learns

questions in advance Examinees do not exchange information Examinee’s answer is not altered after testing is

completed

Results returned to parents/students

HSGQE within 15 days of receiving or before

end of the school year.

Benchmark Exams within 20 days of receiving or before

end of the school year.

Norm Referenced Test In a timely manner

Results distributed to teachers

Class and individual results to each teacher for the teachers classes

Intent is that teachers have assessment data for students in their classes

ConfidentialityIndividual student results are confidentialFamily Educational Rights and Privacy Act (FERPA) Scores can be shared with school and

other education officials to complete their duties

May not be publicly reported without parent consent Fewer than 5 students tested/3 in any cell

Issues in Reporting Data

Confidentiality Family Educational Rights and Privacy Act

Technical Issues Small Schools

Release of Information Results not reported if fewer that 5 students

tested or fewer than 3 students are reported in any cell.

Protocol for Reporting ResultsTwo Categories of Achievement

Complete results will be reported as long as at least 3 students are reported in any cell. If either proficiency level contains 0,1, or 2 students the number of students will be eliminated and the percentage of students at each achievement level will be reported as a range. In reporting a range of performance the following rules will apply.  NOTE: No results will be reported if less than 5 students are tested.

Number of Achievement Level ReportedStudents Tested

5 60% or more proficient (or not proficient)40% or fewer not proficient (or proficient)

 8 75% or more proficient (or not proficient)

25% or fewer not proficient (or proficient) 

10 80% or more proficient (or not proficient)20% or fewer not proficient (or proficient)

 20 90% or more proficient (or not proficient)

10% or fewer not proficient (or proficient) 

40 95% or more proficient (or not proficient) (or more) 5% or fewer not proficient (or proficient) 

Reporting Protocol

Making effective use ofthe DATA

Provide results to parents, students and teachers School wide Individual

Devise ways to provide easy access to assessment data to teachers Include as much information as possible

Designation API

Mining the DATALook beyond the mean Disaggregate data to determine the effect of

program on various groups Quartiles, item analysis API Scores

Look at trends over time

Work with staff to determine what you can learn from the data how you will react to the data

Using the Results to Make Using the Results to Make DecisionsDecisions

About school improvement planningschool improvement planning

About policy changes and program evaluation

About rewardsrewards and sanctionssanctions

Use the results to:focus instructiontarget student interventionidentify needsdevelop programsevaluate programs and initiativesguide Staff Development activitiesset goals for improvement

Examining the Data to Set Goals

Weakest results Domain scores = reading?

Specific standards = use comprehension strategies?

Specific skills within the standard = paraphrase key points?

Demographics Population impact = low socio-economic majority?

Opportunity to learn = poor attendance rates?

SCHOOL IMPROVEMENT PLAN Selecting successful instructional interventions

For the area of concern For the age, grade, and skill level of the students For the demographics of the schoolFor long term effectiveness

Study and training Support through coaching, observation, and feedback Follow-up for ongoing training and reflection Monitoring through supervision and results

Specific Reasonable

Time lines and assignment

Planning staff development

Data CollectionImportance of Proper Coding Demographic Data Program data District or school data

Importance of Accurate and timely Data SubmissionsImportance of Verification of Data

Data CollectionChanges for 2002/2003

Increase in OASIS information Special Education Fields Summer Collection

Changes to Common Core Data-October 2002 Reporting Teacher Assignments Certification

Implementation of Unique Student Identification Number

Questions, Comments, Concerns