Virginia Modified Achievement Standards Test (VMAST

2011 National Conference on Student Assessment Orlando, Florida

Virginia Modified Achievement Standards Test (VMAST):

Lessons Learned

Shelley Loving-Ryder Virginia Department of Education

June 20, 2011

2011 National Conference on Student Assessment Orlando, Florida

VMAST: Overview

•  GSEG to develop assessment for Grade 8 reading and mathematics

•  Adds supports and simplifications recommended by educators to existing online test items

2011 National Conference on Student Assessment Orlando, Florida

•  Conducted focus groups and online surveys with teachers and special education directors on learning characteristics and support/simplifications used with identified students in instruction

•  In fall 2008 and spring 2009 conducted 2 pilots using cognitive interviews with students from urban and rural school districts who were identified as likely VMAST participants

- 3 -

Activities to Date

2011 National Conference on Student Assessment Orlando, Florida

Field Test •  Field test with approximately 500 students in

May-June, 2010 for VMAST grade 8 reading and mathematics.

•  Students participating in field test were students with disabilities who met the participation criteria for VMAST

•  47 field test students were administered unsupported items first and were then administered the supported items

2011 National Conference on Student Assessment Orlando, Florida

Results: Reliability

Subject N-Count Alpha VMAST

Alpha SOL

Math 235 0.85 0.89 - 0.93

Reading 257 0.84 0.88 – 0.90

2011 National Conference on Student Assessment Orlando, Florida

Results of the Mathematics Field Test

2011 National Conference on Student Assessment Orlando, Florida

Math Supports and Simplifications •  Provided hint boxes with formulas,

strategies, and reminders, •  Color coded important information •  Provided online manipulatives •  Simplified numbers •  Simplified language in stems •  Added or altered graphics •  Reduced answer options from 4 to 3

2011 National Conference on Student Assessment Orlando, Florida

Overall Comparison of VMAST vs SOL for Grade 8 Mathematics

VMAST is 40 MC items (vs 50 on the SOL test) measuring: •  RC1: Number and Number Sense; •  RC2: Computation and Estimation; •  RC3: Measurement and Geometry; •  RC4: Probability and Statistics; and •  RC5: Patterns, Functions and Algebra.

Test N Average Percent Correct

Std Dev Min Max

VMAST 47 59 17.52 8 95 SOL 47 48 15.28 20 80

2011 National Conference on Student Assessment Orlando, Florida

Mathematics Item-Level Analyses

•  Change in p-values for supported items •  5 items had an increase in p-value

from .20 to .34 •  21 items had an increase in p-value

from.10 to .19 •  8 items either had no change in p-value

or the increase was less than .10 •  6 items had a decrease in p-value

2011 National Conference on Student Assessment Orlando, Florida

Item-Level Analyses: Overall Results

•  P-Value •  Ranged from

.34 to .90 •  Mean of .65

•  Point biserial •  Ranged from

.11 to .59

2011 National Conference on Student Assessment Orlando, Florida

Recap of Mathematics Results •  Reliability and Validity look good •  Students have higher percent correct on VMAST

than on SOL •  No atypical patterns in gender/ethnicity results •  Nice range in item difficulty and item

discrimination •  Some supported items became more difficult

•  Consider number of supports added •  Were supports used by students?

2011 National Conference on Student Assessment Orlando, Florida

Results of the Reading Field Test

2011 National Conference on Student Assessment Orlando, Florida

Reading Supports and Simplifications •  Shortened reading passages •  Excerpted relevant sections of reading passages

and presented them with item •  Provided hint boxes with reminder to return to

passage when item dealt with passage as whole •  Used graphic organizers to present passage

information •  Simplified language in stems •  Highlighted key words •  Reduced answer options from 4 to 3

2011 National Conference on Student Assessment Orlando, Florida

Overall Comparison of VMAST vs SOL for Grade 8 Reading

VMAST is 36 MC items (vs 45 on the SOL test) measuring:

•  RC1: Use word analysis strategies and information resources; and •  RC2: Demonstrate comprehension of printed materials.

Test N Average Percent Correct

Std Dev Min Max

VMAST 47 60 17.93 22 94

SOL 47 44 16.76 11 84

2011 National Conference on Student Assessment Orlando, Florida

Item-Level Analyses: Overall Results

•  P-Value •  Ranged from

.29 to .90 •  Mean of .62

•  Point Biserial •  Ranged from .22 to .57

2011 National Conference on Student Assessment Orlando, Florida

Reading Item-Level Analyses

•  Impact of supports and simplifications on reading VMAST items – 10 items had an increase in p-value

from .20-.38 – 16 items had an increase in p-value

from .10-.19 – 10 items had p-values that were either

unchanged or had an increase of less than .10

2011 National Conference on Student Assessment Orlando, Florida

Recap of Reading Results

•  Reliability and Validity look good •  Students have higher percent correct on

VMAST than on SOL •  No atypical patterns in gender/ethnicity results •  Nice range in item difficulty and item

discrimination •  Passage type may play a role in effectiveness

of the supports and simplifications

2011 National Conference on Student Assessment Orlando, Florida

Lessons Learned: Effective Supports

•  Hint boxes •  Simplifying language •  Simplifying presentation of item •  Drag and drop tool to fill in equations or

expressions •  Color coding when used judiciously •  Highlighting •  Excerpted text •  Reminder to return to passage when item

addresses passage as a whole

2011 National Conference on Student Assessment Orlando, Florida

Lessons Learned Supports are most helpful when they are consistently displayed across items •  Hint boxes always in the same location •  Components that can be dragged and

dropped identified consistently •  Same “icon” to activate animation

- 19 -

2011 National Conference on Student Assessment Orlando, Florida

Lessons Learned •  Must be judicious in use of supports:

“oversupporting” can be confusing to students

•  Supports and simplifications not helpful if students don’t know content

•  Providing practice with items is crucial. Practice items and guides for teachers may be found at www.doe.virginia.gov/testing/alternative_assessments/vmast_va_mod_achievement_stds_test/practice_items/index.shtml

2011 National Conference on Student Assessment Orlando, Florida

Next Steps •  Moving toward implementation of VMAST in

reading and mathematics in grades 3-8, Algebra I and end-of-course reading

•  Field tested mathematics in spring 2011 •  First operational administration of

mathematics in spring 2012 •  Field test reading in spring 2012 •  First operational administration of reading in

spring 2013

2011 National Conference on Student Assessment Orlando, Florida

Application to Next Generation Assessments

•  Continue to study the characteristics of students to ensure that participation criteria are clear and appropriate

•  Use principles of universal design in item development •  Reduce language load •  Simplify presentation of items

•  Maximize the use of technology to develop additional supports and simplifications

2011 National Conference on Student Assessment Orlando, Florida

Contact Information:

Shelley Loving-Ryder Virginia Department of Education

student_assessment@doe.virginia.gov (804) 225-2102