Embed Size (px)
Citation preview
Alternate Assessment Alignments:
What Best Practices and Lessons
Learned Have Taught Us
June 28, 2017
10:30 AM
PRESENTERS:
Dr. Jan Sheinker President Sheinker Educational Services, Inc.
Dr. Hillary Michaels Director of Educational Research and EvaluationHumRRO
Nannette PenceAssessment Specialist Alabama Department of Education
MODERATOR:
Dr. Jan BarthState Solutions ManagerData Recognition Corporation
Presenters
2
Overview
This presentation will highlight
• Creating acceptable extended standards and alternate achievement level
descriptors from state content standards
• Acceptable alignment evidence for alternate assessments
• Contracting with an assessment vendor for alternate assessment
• Requesting necessary processes for an informative and effective alternate
alignment study
3
Overview
This presentation will discuss Alabama’s experiences
• Developing new extended standards and descriptors based on the state
content standards
• Transitioning to performance-based assessments from portfolio-based
assessments
• Selecting an alignment contractor for their AAA Alignment Study
• Using alignment results
4
Sheinker Educational Services, Inc.
Dr. Jan Sheinker
President
Sheinker Educational Services, Inc.
5
1) Creating extended standards (ES’s) and alternate achievement
level descriptors (AALDs) from state content standards
Extended standards emerge from the content standards
Developed AFTER content standards are final
Developed in similar process
Alternate achievement standards emerge from the extended standards
• Initially developed with the extended standards
• Revised when cut scores are set
Sheinker Educational Services, Inc.
6
Creating extended standards linked to state content standards
• Clearly link to the standards for all – Can you see the content?
• Standardize meaning – Is the meaning clear to all users?
• Create consistency in expected performance – Are the expectations comparable for all students taking the AA-AALD?
• Accommodate diverse disabilities – Can students perform the same content in different ways?
• Ground alternate assessments – Are the ES’s clear enough for test developers to create precise items?
Sheinker Educational Services, Inc.
7
Creating alternate achievement level descriptors aligned with
extended standards
• Ideally draft with ES’s – Are stakeholders involved at every step?
• Clearly align with ES’s – Is proficient performance aligned?
• Describe performance at, above, and below ES’s - Does achievement escalate from grade to grade and achievement level to achievement level?
• Review and finalize with cut scores – Do final AALDs describe performance consistent with actual achievement within each achievement level?
Sheinker Educational Services, Inc.
8
2) Collecting acceptable alignment evidence
Evaluate:
• Linkage between the ES’s and the State Content Standards
• Alignment between the ES’s and AALDs
• Alignment of items with ES’s and proficient AALDs for level of
cognitive complexity
• Alignment of items for coverage of all achievement levels
Sheinker Educational Services, Inc.
9
Evaluate the alignment of the State’s Theory of
Action for Reduction of Breadth and Cognitive
Complexity against the AA-AALDs:
• Alignment of each assessment with the emphasis of the ES’s and AALD’s
• Alignment of the balance of representation of items on each assessment with the test blueprints and specifications
Sheinker Educational Services, Inc.
10
3) Contract with the assessment vendor should specify
• Every step in the process – Do not assume!
• Expectations for compliance with Standards for Educational and Psychological Testing and Peer Review Guidance
• Expectations for coverage, balance, and range of items to be developed for all achievement levels
• The dimensions of alignment that items and assembled assessments must meet
• Consequences for failure to meet these requirements
• Expectations that assessment vendor will address any deficiencies identified by independent alignment studies
Sheinker Educational Services, Inc.
11
4) Processes and protocol that is necessary for an
informative and effective alternate alignment study
Contractor independent of the assessment vendor that provides:
• Comprehensive information about strengths and weaknesses of each overall assessment and each item within it
• Specific information about strategies for addressing weaknesses and maintaining strengths over time
Sheinker Educational Services, Inc.
12
Independent, expert team including members who are
• Knowledgeable of students with significant cognitive disabilities
• Knowledgeable of content being measured
• Knowledgeable of most current accommodations for this population and their impact on content
A process robust in evaluating assessments against the State’s
Theory of Action for:
• Item alignment, quality, challenge, and fairness
• Judging balance and range of coverage across achievement levels
Sheinker Educational Services, Inc.
13
HumRRO
Hillary Michaels
Director of Educational Research
and Evaluation
HumRRO
14
Processes and Protocols
• Based on Links for Academic Learning (LAL)
• Adapt to meet state needs
• Adapt to meet content complexity, such as NGSS
• Supplement with other indicators
o Webb-like metrics, such as
Items Represent Intended Content
Items Represent Intended Categories
o Review of documentation
o Evidence of quality of fairness
o Comparisons with internal system articulation of linkage levels15
Links for Academic Learning Criteria
• Academic (not often used anymore)
• Age Appropriate
• Standards Fidelity
• a. Content Centrality
• b. Performance Centrality
• Content Differentiation
• Achievement
16
HumRRO Criteria
• Content Coverage
– Items Represent Intended Content
– Items Represent Intended Categories
– Item DOK Represent Test Specifications
– Item Sufficiency for Category Reporting
• Performance Accuracy
– HumRRO: Survey questions on quality and fairness
17
Supplemented Ratings Under Content Coverage
Webb based, but diverges in key ways that include:
• Instructs reviewers to assign an overall degree of alignment rating
• Uses student assessment data to examine content categories
• Adjusts the Balance of Knowledge representation to account for designed differences in emphasis in the test specifications
• Interprets results on a continuum that indicates areas of strength and weakness
18
Supplemented Ratings
HQAP based
• Review of documentation
• Item quality
• Item fairness
• Equity
19
General Panelist Procedures
• Review standards in grade span and determine consensus DOK for
standards and extended standards
• Evaluate the relationship between the extended standards and the
general content standards
• Evaluate the extended standards regarding grade-level appropriateness
and accessibility to different disability groups
• Evaluate the extended standards grade-level progression in ELA and
math to ensure that the extended standards are increasing in depth and
breadth as grade increases
• Verify or identify an item’s DOK and other features such as math
practice, science cross cutting concepts, 21st century skills
20
General Panelist Procedures
• Verify or identify the item’s designated content standard
• Verify or identify the item’s designated reporting category
• Calibration and re-calibration of raters
• If multiple standards are identified for an item, determine the holistic
alignment rating
• Evaluate items across grades, holistically, to determine progression of
content areas in the items in ELA and math
• Review the ability of the alternate assessment system to provide an
opportunity for students to show learning of grade-level academic
content
21
Included in Report
• Multiple sets of alignment statistics generated
• Results of all criteria
• Provide flags by item set
• Key findings and recommendations
22
Application of Thresholds
.
Age
Appropriate
Performance
Centrality
Content
DifferentiationAchievement
M3 5 out of 7; 3
M4
M5
Note: Green boxes indicate 90% threshold meet.
Are
ta
sk
s a
me
na
ble
to
ac
co
mm
od
ati
on
s o
r
su
pp
ort
s?
Is
th
e ite
m s
et
tas
k
co
mp
ara
ble
to
th
e
co
mp
lex
ity
in
th
e e
xte
nd
ed
sta
nd
ard
s?
Do
ta
sk
s w
ith
in a
n ite
m s
et
inc
rea
se
in
co
mp
lex
ity
?
Stu
de
nt
ac
hie
ve
me
nt
de
mo
ns
tra
tes
le
arn
ing
.
Are
ta
sk
s a
cc
es
sib
le t
o
dif
fere
nt
dis
ab
ilit
y g
rou
ps
?
Performance Accuracy
Content/
Grade
Is t
he
co
nte
nt
of
the
ite
m
se
t ta
sk
s a
ge
ap
pro
pri
ate
?
23
Application of Thresholds
24
Item
Alignment
Represent
Intended
Categories
M3 1 out of 3
M4 87% 3 out of 4
M5
Note: Green boxes indicate 90% threshold meet.
Content Coverage
Content/
Grade
Do
ite
m s
et
tas
ks
re
fle
ct
the
ran
ge
of
DO
K in
th
e
sta
nd
ard
s?
Do
pa
ne
lis
ts a
gre
e w
ith
DO
K?
Do
pa
ne
lis
ts a
gre
e w
ith
Vo
lum
e o
f In
form
ati
on
?
Do
pa
ne
lis
ts a
gre
e w
ith
Vo
ca
bu
lary
?
Do
pa
ne
lis
ts a
gre
e w
ith
Co
nte
xt?
Task ComplexityA
re ite
m s
et
tas
ks
fu
lly
alig
ne
d w
ith
th
e s
tan
da
ds
?
Do
ite
m s
et
tas
ks
ad
eq
ua
tely
re
pre
se
nt
rep
ort
ing
ca
teg
ori
es
?
Validity Evidence
Refer to test development documents– Used more tradition test development
– Used principled-based design
– Look for consistency with TOA
Under more traditional test development– Focus on blueprint
– Focus criteria
Under principled-based design– Include consideration of model
– Review of consistency among elements, (e.g., item specifications, blueprints, items, tasks, scoring criteria)
25
Sufficiency for Category Reporting
• Use student data
• Examine
– Internal consistency
– Factor structure
26
Nannette Pence
Assessment Specialist
Alabama Department of Education
Alabama Department of Education
27
1) Development of new extended standards and descriptors
around the state content standards:
• Alabama Special Education Services (SES), Student Assessment and vendor work as partners (team) in development of the Alabama Alternate Assessment (AAA);
• Alabama College and Career Ready Standards (CCRS) were finalized and approved before the development of the Alabama Extended Standards;
• Research of several state alternate assessment processes including Dynamic Learning Maps (DLM) and National Center and State Collaborative (NCSC) on how their extended standards were developed from Common Core Standards were discussed;
Alabama Department of Education
28
Alabama Department of Education
1) Development of new extended standards and descriptors
around the state content standards:• ALSDE selected a process that allowed a committee of state educator
experts to determine the essence of the standards and how the extended standards link to the regular standards based on the content;
• Educator experts from varied backgrounds and content areas were chosen as representatives of all regions (8 state board districts) in Alabama including rural and urban schools to serve on committees throughout the different processes;
o Alabama Extended Standards development
o Achievement-level descriptors development
standard setting (assured the descriptors described performance consistent with actual achievement within each achievement level).
29
2) Development of a new performance-based alternate
assessment:
ALSDE transitioned from a portfolio assessment to a performance-based assessment for the following reasons:
• research (national trends)
• consistent measures of student work
• appropriate validity and reliability data
• federal requirements
• more instructional time
Alabama Department of Education
30
2) Development of a new performance-based alternate assessment
ALSDE prepared a change of scope (within original pricing) with contractor that transitioned from a portfolio assessment to performance-based (multiple choice) in September of 2016
• item design and review
• content and bias review
• training materials
• standard setting
• operational pilot Spring 2017
• operational administration Spring 2018
However, the trending (un-fake news) is that Alabama will be releasing a Response for Bid (RFB) which will start the process all over again ….
Alabama Department of Education
31
3) Selection of an independent alternate assessment alignment study/and uses of the results:
ALSDE prepared a scope change with current contractor
• logistics/payment
• independent national alignment experts
• Alabama education experts
Alignment finding and recommendations for improvement were reported to ALSDE
Due to anticipated changes in the Alabama assessment program, ALSDE may not use
the alignment recommendations and findings to improve the current alternate
assessment, but will view the findings and recommendations as lessons learned as
development of new extended standards and a new alternate assessment work begins.
Alabama Department of Education
32
Alternate Assessment:
State by State Peer Review Findings
To date, around 13 states have received Substantially Meets Requirement with their Alternate Assessments in the Federal Peer Review Process
• Colorado, Illinois, Iowa, Kansas, Minnesota, Missouri, New Jersey, North Dakota, Texas, Utah, Vermont, West Virginia and Wisconsin
• Visit Website with State by State Outcome Lettershttp://www2.ed.gov/admins/lead/account/nclbfinalassess/index.html
33
Lessons Learned
• AA standards must have evidence of acceptable linkage to state regular standards
• Must have acceptable evidence of rigorous alignment of the alternate assessment with the AALDs.
• Must assure acceptable technical documentation is same for alternate as it is for regular Assessment
Rigorous Alignment Studies
Strong Standard Setting Report
High Quality Technical Reports
• Reporting requirements must be the same as the regular assessment
• Overall, Portfolio Assessment seems more difficult to get approved than Performance Task Assessment; for example, more difficult to prove consistency of content/forms, technical documentation, etc. 34
Questions
35
