Upload
amberlynn-barrett
View
217
Download
0
Tags:
Embed Size (px)
Citation preview
Who are the Students in Who are the Students in Alternate and Modified Alternate and Modified Achievement Standards Achievement Standards
Assessments?Assessments?Jacqueline F. Kearns, Ed.D., NAAC
Martha Thurlow, Ph.D., NCEO
Elizabeth Towles-Reeves, Ph.D., NAAC
OSEP Project Directors’ Conference
July 22, 2008
Part IPart I
Alternate AchievementAlternate Achievement
Standards Standards AssessmentsAssessments
TopicsTopics
1.1. Alternate Achievement Standards Alternate Achievement Standards Assessments and the Validity EvaluationAssessments and the Validity Evaluation
2.2. Current Research from NAAC-Who are Current Research from NAAC-Who are the Students in Alternate Achievement the Students in Alternate Achievement Standards Assessments?Standards Assessments?
3.3. Implications for the Validity EvaluationImplications for the Validity Evaluation
OBSERVATION
INTERPRETATION
COGNITIONStudent PopulationAcademic contentTheory of Learning
Assessment SystemTest DevelopmentAdministration Scoring
ReportingAlignmentItem Analysis & DIF/BiasMeasurement errorScaling and Equating Standard Setting
VALIDITY EVALUATIONEmpirical evidence
Theory & logic (argument)Consequential features
The Assessment Triangle & Validity EvaluationMarion & Pellegrino (2006)
Cognition Vertex Validity Cognition Vertex Validity QuestionsQuestions
1)1) Is the assessment appropriate for Is the assessment appropriate for the students for whom it was the students for whom it was intended? intended?
2)2) Is the assessment being Is the assessment being administered to the appropriate administered to the appropriate students?students?
Both are important for the validity Both are important for the validity evaluationevaluation
Issues in Teaching/Assessing Students in Alternate Achievement
Standards Assessments Varied levels of symbolic communication Attention to salient features of stimuli Memory Limited motor response repertoire Generalization Self-Regulation Meta-cognition Skill Synthesis Sensory Deficits Special Health Care Needs
Kleinert, H., Browder, D., Towles-Reeves, E. (in press).Kleinert, H., Browder, D., Towles-Reeves, E. (in press). Models of Models of cognition for students with significant cognitive disabilities: cognition for students with significant cognitive disabilities: Implications for assessment. Implications for assessment. Review of Educational Research.Review of Educational Research.
Learner Characteristics Learner Characteristics Demographic VariablesDemographic Variables
Learner Characteristics (all on a continuum of skills): Expressive Language Receptive Language Vision Hearing Motor Engagement Health Issues/Attendance Reading Mathematics Use of an Augmentative Communication System
(dichotomous variable)
MethodologyMethodology
Seven partner states chose to participate during the 2006-2007 school year. States 1, 2, 3, 4, 5, and 6:
gathered data in the administration process for their alternate achievement standards assessment (i.e., bubble sheet, paper/pencil version of the LCI, etc.)
State 7: gathered data using Zoomerang, an online survey
package.
States & LCI Response States & LCI Response RatesRates
State Geography Participation Rate Sample N Response Rate
State 1 North East 0.96% 2793 100%
State 2 Mid West 1.17% 2216 100%
State 3 East 1.14% 3595 75%
State 4 North East 0.99% 722 93%
State 5 South East 0.70% 2134 87%
State 6 East 0.76% 468 91%
State 7 West 0.94% 219 47%
Expressive LanguageExpressive Language
Receptive LanguageReceptive Language
Use of Augmentative Use of Augmentative Communication SystemsCommunication Systems
ReadingReading
MathematicsMathematics
Expressive Language Across Grade Bands
Reading Across Grade Bands
Mathematics Across Grade Bands
Who are the Kids in Alternate Achievement Standards
Assessments? Represent ~1% or less of the total assessed population All disability categories were represented but primarily 3
emerge, Mental Retardation Multiple Disabilities Autism
Highly varied levels of expressive/receptive language use Most students in the population use symbolic communication Level of symbolic language use does not significantly change
across grade-bands The majority of students do not use AAC Most of the population read basic sight words and solve
simple math problems with a calculator. Changes in skill progression in reading and math across
grade bands most likely due to identification of students rather than teaching and learning
Cognition Vertex: Cognition Vertex: Validity Evaluation Essential Validity Evaluation Essential
QuestionsQuestions Who is the population being assessed?Who is the population being assessed? How do we document and monitor the How do we document and monitor the
population?population? What do we know about how they learn (theory of What do we know about how they learn (theory of
learning) academic content?learning) academic content? What do our assessment results tell us about how What do our assessment results tell us about how
the population is learning academic content?the population is learning academic content? Are our data about the population and theory of Are our data about the population and theory of
learning learning consistentconsistent with student performances with student performances on the assessment?on the assessment?
If not, what assumptions are challenged?If not, what assumptions are challenged? What adjustments should be made?What adjustments should be made?
ParticipationParticipation Theory of LearningTheory of Learning Student PerformanceStudent Performance
Alternate Achievement Standards Alternate Achievement Standards AssessmentsAssessments
ReferencesReferences Kleinert, H., Browder, D., Towles-Reeves, E. (in Kleinert, H., Browder, D., Towles-Reeves, E. (in
press). Models of cognition for students with significant press). Models of cognition for students with significant cognitive disabilities: Implications for assessment. cognitive disabilities: Implications for assessment. Review Review of Educational Research.of Educational Research.
Marion, S., & Pellegrino, J. (2006). A validity framework for Marion, S., & Pellegrino, J. (2006). A validity framework for evaluating the technical quality of alternate assessments. evaluating the technical quality of alternate assessments. Educational Measurement: Issues and Practices, 25Educational Measurement: Issues and Practices, 25(4), 47-(4), 47-57.57.
Additional ResourceAdditional Resource Towles-Reeves, E., Kearns, J., Kleinert, H., & Kleinert, J. Towles-Reeves, E., Kearns, J., Kleinert, H., & Kleinert, J.
(2008, May 12). An analysis of the learning characteristics (2008, May 12). An analysis of the learning characteristics of students taking alternate assessments based on alternate of students taking alternate assessments based on alternate achievement standards. achievement standards. Journal of Special Education. Journal of Special Education. Retrieved June 2, 2008, from Retrieved June 2, 2008, from http://sed.sagepub.com/cgi/rapidpdf/0022466907313451v1. .
Part IIPart II
Modified AchievementModified Achievement
Standards Standards AssessmentsAssessments
Topics
1.Research and Regulation Advice
2.Current Practice
3.GSEG Project Work
Why Start from the Student?
National Research Council – Pellegrino, Chudowsky, & Glaser, (2001) – Knowing What Students Know
Luecht (2007) - good assessment design would apply human factors engineering principles by developing cognitive maps and cognitive construct models
Pellegrino (2007) - Principled Assessment Design process that started with a clear student model as the basis for an evidence model that would, in turn, serve as the basis for a task model.
Why Start from the Student?
Mislevy and Haertel (2007) - central point of agreement of these models is the necessity of first developing a good understanding of how people do or fail to do what is to be measured. Then tasks can be developed that let us observe what people do so we are able to make inferences that are more fully supported by clearer evidence.
Assessment as a Process of Reasoning from Evidence
Cognition
– model of how students represent knowledge & develop competence in the domain
Observations– tasks or situations that allow one
to observe students’ performance
Interpretation– method for making sense of the
data
observation
interpretation
cognitionMust be
coordinated!
The Assessment Triangle
Cognition Vertex Validity Questions
1) Is the assessment appropriate for the students for whom it was intended?
2) Is the assessment being administered to the appropriate students?
Both are important for the validity evaluation
April 9, 2007 Regulations Address Who the Students Are
Preamble: The final regulations intentionally do not prescribe which students with disabilities are eligible to be assessed based on modified academic achievement standards; that is the determination of a student’s IEP Team, which includes the student’s parents, based on criteria developed by the State as part of the State’s guidelines for IEP Teams
April 9, 2007 Regulations Address Who the Students Are
The student’s progress to date in response to appropriate instruction . . ., is such that, even if significant growth occurs, the IEP team is reasonably certain that the student will not achieve grade-level proficiency within the year covered by the student’s IEP.
Section 200.1(e)(2)(ii)
April 9, 2007 Regulations Address Who the Students Are
Inform IEP teams that students eligible to be assessed based on alternate or modified academic achievement standards may be from any of the disability categories listed in the IDEA.
Section 200.1(e)(2)(ii)
From Cortiella (2007), Learning Opportunities for Your Child Through Alternate Assessments – Alternate Assessments based on Modified Achievement Standards
Current Practice
• In 2007, 5 states had assessments that they believed to be an AA-MAS before the April, 2007 regulation release – see Lazarus, Thurlow, Christensen, & Cormier (2007)
• Update study now being conducted – have pulled information on eligibility for AA-MAS
Eligibility Criteria in States
In 2008, 10 states had assessments that they believed to be an AA-MAS
California North Carolina
Connecticut North Dakota
Kansas Oklahoma
Louisiana Texas
Maryland Virginia
Number of States With Selected Eligibility Criteria
Format of Criteria# of
States
Descriptions (bullets, chart, written description) 10
Flowchart/Decision Tree 4
Checklist 5
2008 Study
Number of States With Selected Eligibility Criteria
Disability Status Criteria# of
States
Student has IEP 10
Not based on disability category label 5
Not due only to ELL designation or being on 504 plan 3
Not due to being identified as having a significant cognitive disability
2
Student may be in any of the disability categories 6
2008 Study
Number of States With Selected Eligibility Criteria
Classroom Learning/Teaching Criteria# of
States
Student not progressing at rate expected to reach grade level proficiency within school year covered by IEP
8
Student is learning grade-level content 6
Not due to receiving instruction based on extended or alternate standards or being eligible to take AA-AAS
6
Not based on attendance (extended/excessive absence) 5
Student receives specialized instruction 3
2008 Study
Number of States With Selected Eligibility Criteria
Classroom Learning/Teaching Criteria – cont. # of
States
Student requires differentiated content for classroom assessment
3
Student needs accommodations during classroom instruction
2
Student’s classroom achievement and performance significantly below grade-level peers
2
Student consistently requires instruction in pre-requisite skills to the grade-level indicators
2
2008 Study
Number of States With Selected Eligibility Criteria
Classroom Learning/Teaching Criteria – cont.# of
States
Not based on placement setting 2
Not based on amount of time in general or special education services
2
2008 Study
Number of States With Selected Eligibility Criteria
Previous Performance Criteria# of
States
Student passed or failed AA-AAS or other large-scale tests
5
Student cannot demonstrate knowledge on regular assessment even with accommodations
3
Student has been tested on multiple, valid, objective measures over time
3
Student’s previous performance on multiple measures is considered
3
2008 Study
Number of States With Selected Eligibility Criteria
IEP Goals & Other# of
States
IEP includes goals that are based on grade-level content standards
4
Not due to social, cultural, language, economic, or environmental factors
2
2008 Study
Identifying Accommodations for AA-MAS
• State Approaches• Accommodations issues for
regular assessment• Integration of “accommodations”
and universal design principles into the regular assessment first, then in the design of the AA-MAS
State Approaches – Accommodations Incorporated into AA-MAS Design
Accommodation No. of States
Fewer items/page 3
Larger font size 3
Calculator 2
Breaks as needed 2
Key text underlined/bolded 1
State 1 (long time) Reading = 19%Math = 17%
State 2 (long time) Reading = 10.5% Math = 9.9%
AA-MAS Participation Rates (2006-07)
State 3 (long time) Reading = 23%Math = 21%
State 4 (newer) Reading = 31%Math = 29%
Rates based on # students with IEPs
State 1 (long time) Reading = 24%Math = 33%
State 2 (long time) Reading = 4.3% Math = 2.6%
AA-MAS Rates Proficient (2006-07)
State 3 (long time) No Data No Data
State 4 (newer) Reading = 52%Math = 54%
Rates based on # students with IEPs
Alternate Achievement Standards Alternate Achievement Standards Assessments:Assessments: Consideration of students with high Consideration of students with high
reading and math abilitiesreading and math abilities Assessment design for a highly varied Assessment design for a highly varied
populationpopulation Considering symbolic language useConsidering symbolic language use Skill progressions in reading and mathSkill progressions in reading and math
ConsiderationsConsiderations
ConsiderationsConsiderations
Modified Achievement Standards Modified Achievement Standards Assessments:Assessments: Moving from student characteristics to an Moving from student characteristics to an
assessment based on grade-level content, but with assessment based on grade-level content, but with modified achievement standardsmodified achievement standards
Clearly defining the relationships among the Clearly defining the relationships among the general assessment, the AA-AAS, the AA-MAS, and general assessment, the AA-AAS, the AA-MAS, and the AA-GLAS, if one existsthe AA-GLAS, if one exists
Separating instructional issues from assessment Separating instructional issues from assessment issuesissues
Providing training and assistance for good Providing training and assistance for good decisions about who needs which assessmentdecisions about who needs which assessment
Contact InformationContact Information
Jacqueline Kearns, Ed.D.Jacqueline Kearns, Ed.D. 1 Quality Street, Suite 7221 Quality Street, Suite 722 Lexington, Kentucky 40507Lexington, Kentucky 40507 859-257-7672859-257-7672 859-323-1838859-323-1838 [email protected]
u
Elizabeth Towles-Reeves, Ph.D.Elizabeth Towles-Reeves, Ph.D. 1 Quality Street, Suite 7221 Quality Street, Suite 722 Lexington, Kentucky 40507Lexington, Kentucky 40507 859-257-7672859-257-7672 859-323-1838859-323-1838 [email protected]
http://www.naacpartners.org/
Contact InformationContact Information
Martha Thurlow, Ph.D.Martha Thurlow, Ph.D. 207 Pattee Hall207 Pattee Hall 150 Pillsbury Drive SE150 Pillsbury Drive SE Minneapolis, MN 55455Minneapolis, MN 55455 612-624-4826612-624-4826 [email protected]
www.nceo.info