Virginia Modified Achievement Standards Test (VMAST

2011 National Conference on Student Assessment Orlando, Florida

Virginia Modified Achievement Standards Test (VMAST):

Lessons Learned

Shelley Loving-Ryder Virginia Department of Education

June 20, 2011


VMAST: Overview

•  GSEG to develop assessment for Grade 8 reading and mathematics

•  Adds supports and simplifications recommended by educators to existing online test items


•  Conducted focus groups and online surveys with teachers and special education directors on learning characteristics and support/simplifications used with identified students in instruction

•  In fall 2008 and spring 2009 conducted 2 pilots using cognitive interviews with students from urban and rural school districts who were identified as likely VMAST participants

- 3 -

Activities to Date


Field Test •  Field test with approximately 500 students in

May-June, 2010 for VMAST grade 8 reading and mathematics.

•  Students participating in field test were students with disabilities who met the participation criteria for VMAST

•  47 field test students were administered unsupported items first and were then administered the supported items


Results: Reliability

Subject N-Count Alpha VMAST

Alpha SOL

Math 235 0.85 0.89 - 0.93

Reading 257 0.84 0.88 – 0.90


Results of the Mathematics Field Test


Math Supports and Simplifications •  Provided hint boxes with formulas,

strategies, and reminders, •  Color coded important information •  Provided online manipulatives •  Simplified numbers •  Simplified language in stems •  Added or altered graphics •  Reduced answer options from 4 to 3


Overall Comparison of VMAST vs SOL for Grade 8 Mathematics

VMAST is 40 MC items (vs 50 on the SOL test) measuring: •  RC1: Number and Number Sense; •  RC2: Computation and Estimation; •  RC3: Measurement and Geometry; •  RC4: Probability and Statistics; and •  RC5: Patterns, Functions and Algebra.

Test N Average Percent Correct

Std Dev Min Max

VMAST 47 59 17.52 8 95 SOL 47 48 15.28 20 80


Mathematics Item-Level Analyses

•  Change in p-values for supported items •  5 items had an increase in p-value

from .20 to .34 •  21 items had an increase in p-value

from.10 to .19 •  8 items either had no change in p-value

or the increase was less than .10 •  6 items had a decrease in p-value


Item-Level Analyses: Overall Results

•  P-Value •  Ranged from

.34 to .90 •  Mean of .65

•  Point biserial •  Ranged from

.11 to .59


Recap of Mathematics Results •  Reliability and Validity look good •  Students have higher percent correct on VMAST

than on SOL •  No atypical patterns in gender/ethnicity results •  Nice range in item difficulty and item

discrimination •  Some supported items became more difficult

•  Consider number of supports added •  Were supports used by students?


Results of the Reading Field Test


Reading Supports and Simplifications •  Shortened reading passages •  Excerpted relevant sections of reading passages

and presented them with item •  Provided hint boxes with reminder to return to

passage when item dealt with passage as whole •  Used graphic organizers to present passage

information •  Simplified language in stems •  Highlighted key words •  Reduced answer options from 4 to 3


Overall Comparison of VMAST vs SOL for Grade 8 Reading

VMAST is 36 MC items (vs 45 on the SOL test) measuring:

•  RC1: Use word analysis strategies and information resources; and •  RC2: Demonstrate comprehension of printed materials.

Test N Average Percent Correct

Std Dev Min Max

VMAST 47 60 17.93 22 94

SOL 47 44 16.76 11 84


Item-Level Analyses: Overall Results

•  P-Value •  Ranged from

.29 to .90 •  Mean of .62

•  Point Biserial •  Ranged from .22 to .57


Reading Item-Level Analyses

•  Impact of supports and simplifications on reading VMAST items – 10 items had an increase in p-value

from .20-.38 – 16 items had an increase in p-value

from .10-.19 – 10 items had p-values that were either

unchanged or had an increase of less than .10


Recap of Reading Results

•  Reliability and Validity look good •  Students have higher percent correct on

VMAST than on SOL •  No atypical patterns in gender/ethnicity results •  Nice range in item difficulty and item

discrimination •  Passage type may play a role in effectiveness

of the supports and simplifications


Lessons Learned: Effective Supports

•  Hint boxes •  Simplifying language •  Simplifying presentation of item •  Drag and drop tool to fill in equations or

expressions •  Color coding when used judiciously •  Highlighting •  Excerpted text •  Reminder to return to passage when item

addresses passage as a whole


Lessons Learned Supports are most helpful when they are consistently displayed across items •  Hint boxes always in the same location •  Components that can be dragged and

dropped identified consistently •  Same “icon” to activate animation

- 19 -


Lessons Learned •  Must be judicious in use of supports:

“oversupporting” can be confusing to students

•  Supports and simplifications not helpful if students don’t know content

•  Providing practice with items is crucial. Practice items and guides for teachers may be found at www.doe.virginia.gov/testing/alternative_assessments/vmast_va_mod_achievement_stds_test/practice_items/index.shtml


Next Steps •  Moving toward implementation of VMAST in

reading and mathematics in grades 3-8, Algebra I and end-of-course reading

•  Field tested mathematics in spring 2011 •  First operational administration of

mathematics in spring 2012 •  Field test reading in spring 2012 •  First operational administration of reading in

spring 2013


Application to Next Generation Assessments

•  Continue to study the characteristics of students to ensure that participation criteria are clear and appropriate

•  Use principles of universal design in item development •  Reduce language load •  Simplify presentation of items

•  Maximize the use of technology to develop additional supports and simplifications


Contact Information:

Shelley Loving-Ryder Virginia Department of Education

[email protected] (804) 225-2102

Documents

Virginia Modified Achievement Standards Test (VMAST