42510011 0010 1010 1101 0001 0100 1011

Making MAP More MeaningfulWinter 2008

David Dreher, Project Coordinator

Office of Accountability

Highline Public Schools

4251

0011 0010 1010 1101 0001 0100 1011

Overview

• Recap of Predicting 2008 WASL

• Examining 2008 WASL Predictions

• Moving Forward to 2009

4251

0011 0010 1010 1101 0001 0100 1011

RECAP WERA Spring 08 Presentation

• Predictions Released November 2007– A “best guess” about each student’s

performance on the upcoming WASL based on prior MAP and/or WASL performance

– Intended Uses• Provide building staff with a level of risk for not

meeting WASL standard.

• School- and District- level 2008 WASL “forecasts”

– Theory: Putting MAP scores in context with WASL scores will make MAP more meaningful.

4251

0011 0010 1010 1101 0001 0100 1011

Example of Projection and Prediction7th Grade Student in Reading

MAP Scores

Winter 2006 Spring 2007 Fall 2007 Highest MAP

210 212 216 216

Expected MAP Growth

3

WASL 2007Projected MAP

Spring 2008

392 219

Predicted

Prediction Model 2008 WASL

Score 2008 WASL

Range

MAP and WASL 399 392 - 406

MAP Only 402 395 - 409

WASL Only 397 390 - 404

4251

0011 0010 1010 1101 0001 0100 1011

WASL Prediction Range

• Constructed using the SEM values reported in the 2001 WASL Technical Reports.

• Predicted Range = Predicted WASL Score +/- SEM

Grade Level SEM – Reading SEM – Math

3, 4, 5, 6 7 12

7,8 7 16

10 10 12

4251

0011 0010 1010 1101 0001 0100 1011

Interpreting Predictions• If the prediction range is:

– Entirely below 400 (ex.: 380-396): student has less than a 20% chance on the WASL this spring unless we accelerate their learning.

– Straddles 400 (ex.: 396-410): student has basically a coin-flip chance on the WASL, even if their prediction is above 400.

– Entirely above 400 (ex.: 408-424): student has more than an 80% chance on the WASL in the spring, IF they continue to progress.

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

NWEA’s MAP/WASL Alignment StudyReleased January 2008

ReadingFall Testing Window

Grade NWEA “Meets Standard” Cut ScoreHPS Accountability

“Strategic” RIT Range

3 187 182-190

4 190 189-197

5 199 199-207

6 209 206-214

7 216 210-218

8 215 213-221

9 214 212-220

10 213 210-218

4251

0011 0010 1010 1101 0001 0100 1011

NWEA’s MAP/WASL Alignment StudyReleased January 2008

MathematicsFall Testing Window

Grade NWEA “Meets Standard” Cut Score Strategic RIT Range

3 189 183-191

4 202 198-206

5 211 207-215

6 221 219-227

7 228 223-231

8 233 229-237

9 236 233-241

10 239 237-245

4251

0011 0010 1010 1101 0001 0100 1011Spring 2008: WASL happened . . .

Late Summer 2009: WASL results arrive!

4251

0011 0010 1010 1101 0001 0100 1011

Examining the Predictions

• Reliability Analysis – Repeated the “Backward Look”

analysis– “Within Group Look”

• Analysis of “Exceptional” Performances– Predicted Level Analysis

4251

0011 0010 1010 1101 0001 0100 1011

GradeWASL 2008

“Backward Look”(%)

WASL 2007“Backward Look”

(%)

4 80.1 87.3

5 93.8 86.8

6 92.9 86.8

7 85.1 89.8

8 97.5 89.0

10 96.1 86.9

“Backward Look”: Math

% = Actual Met / Predicted to Meet

Predicted to Meet = Predicted WASL score of 400 or better.

4251

0011 0010 1010 1101 0001 0100 1011

Grade

WASL 2008“Backward Look”

(%)

WASL 2007“Backward Look”

(%)

4 93.9 83.1

5 113.3 81.8

6 105.2 83.1

7 85.2 82.5

8 68.9 81.0

10 94.3 87.7

“Backward Look”: Reading

% = Actual Met / Predicted to Meet

Predicted to Meet = Predicted WASL score of 400 or better.

4251

0011 0010 1010 1101 0001 0100 1011

Benchmark(%)

Strategic(%)

Intensive(%)

Grade Estimated >80% Estimated ~50% Estimated <20%

4 85.2 40.3 2.5

5 93.1 47.3 6.7

6 87.8 56.4 8.9

7 83.1 34.4 2.5

8 94.8 51.5 5.4

10 91.6 51.1 6.5

“Within Group”: Math

% = Actual Met Within Group / Total Number In Group

4251

0011 0010 1010 1101 0001 0100 1011Benchmark

(%)Strategic

(%)Intensive

(%)

Grade Estimated >80% Estimated ~50% Estimated <20%

4 89.3 45.1 8.7

5 91.9 62.0 17.1

6 88.9 60.8 16.5

7 84.2 45.3 10.0

8 86.7 50.5 12.6

10 92.2 57.7 17.2

“Within Group”: Reading

% = Actual Met Within Group / Total Number In Group

4251

0011 0010 1010 1101 0001 0100 1011

Questions/Comments

• Procedures for making predictions

• Results of reliability analyses

• What about our theory behind doing this?– “Putting MAP scores in context with WASL

scores will make MAP more meaningful.”

4251

0011 0010 1010 1101 0001 0100 1011

“Meaningful”: Depends on Who You Ask

• My experience talking with the people who work directly with the kids suggests that the strength of our ability to assess the risk level of their students doesn’t impress them.– “You don’t need a weatherman to tell

you which way the wind blows” Bob Dylan

4251

0011 0010 1010 1101 0001 0100 1011

What would be more “Meaningful”?

• Information that would help determine whether things done to help kids pass the WASL worked

Data Users

Principals, Coaches, Teachers

Data Creators

Office Of Accountability

4251

0011 0010 1010 1101 0001 0100 1011

But . . .I don’t really know what schools are doing to try to help kids pass the WASL. . .

So . . .how can I find out?

4251

0011 0010 1010 1101 0001 0100 1011

“Exceptional” Performance Analysis

• Please see handout

• Objective: Start conversations that would increase the flow of information from data users back to us in Accountability

• What are your observations of the data?

4251

0011 0010 1010 1101 0001 0100 1011

Expectations for “Above Level”

• Students were receiving interventions designed to address skills/knowledge deficits

• Students were receiving interventions designed to familiarize them with WASL format

• Students benefited from actions taken by the school to improve the WASL testing environment

• Your ideas?

4251

0011 0010 1010 1101 0001 0100 1011

Expectations for “Below Level”

• Students were ELL or SPED• Students were chronically absent or highly mobile

• Students did not take the WASL seriously

• Your ideas?

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

Moving Forward

• Predictions simplified: Use BSI designations only

• Raising awareness and understanding of NWEA’s Alignment Study

• Increase understanding of NWEA goals and how to interpret goal-level results

• Investigate the possible use of MAP data in evaluation of interventions, initiatives, and programs

4251

0011 0010 1010 1101 0001 0100 1011

Continue to Solicit Input

Data Users

Principals, Coaches, Teachers

Data Creators

Office Of Accountability

42510011 0010 1010 1101 0001 0100 1011

Contact Information

David Dreher, Project Coordinator

Office of Accountability

Highline Public Schools

www.hsd401.org

206-433-2334

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

What they said . . .

Expected Responses Unexpected Responses

Sch RespSupplement/Content

Supplement/Format

Test Environ

School/ Class Environ

Perception/ Understanding of MAP

Did Not Answer the Question

BEV X X     X X  

CED              

DES X       X    

HAZ X           X

MID X X          

MOU X X X        

MCM              

SHO              

WCH X X X        

               

CAS X ? ?        

CHI X ? ?        

               

GLO X         X  

4251

0011 0010 1010 1101 0001 0100 1011

4251

0011 0010 1010 1101 0001 0100 1011

What is MAP• Measures of Academic Progress

– Developed by the Northwest Evaluation Association – Norm-referenced assessment – Computerized and adaptive– Performance is reported as a RIT score

• The RIT Scale – Uses individual item difficulty values to estimate

student achievement – A RIT score has the same meaning regardless of grade

level– Equal interval scale

• Highline Public Schools– Three testing windows per year (Fall, Winter, Spring)– Test students in the areas of math and reading– Test students in grades 3-10

4251

0011 0010 1010 1101 0001 0100 1011

The Needs of the Data User• Building staff were saying things

like . . .– “How can we use MAP data to help us

make decisions?” – “How do MAP and WASL

performance compare?”– “I want to know what a student’s

history is with MAP.”– “What is a RIT score?”– “Giving me a RIT score is like telling

me the temperature in Celsius!”

4251

0011 0010 1010 1101 0001 0100 1011

Making The Predictions• Snooped and found the best

indicators of WASL success

• Applied linear regression models to generate WASL scores for each student

• Examined the predicted WASL scores

4251

0011 0010 1010 1101 0001 0100 1011

Projecting MAP to Spring

• For the models with “Projected MAP” as one of the factors individual student performance on MAP in the Spring of 2008 was projected. – The amount of expected growth added to

a student’s Highest MAP score came from NWEA’s Growth Study

4251

0011 0010 1010 1101 0001 0100 1011

Snooping (Reading)R-Values

WASL 2007 Reading Scale vs.

Grade

3 4 5 6 7 8 10

WASL Reading 2006   0.730 0.769 0.764 0.745 0.755  

WASL Reading 2005       0.727      

WASL Reading 2004             0.723

MAP-R Spring 2007 0.792 0.778 0.799 0.774 0.744 0.726 0.755

MAP-R Winter 2007 0.782 0.782 0.796 0.767 0.780 0.711 0.786

MAP-R Fall 2006 0.769 0.750 0.793 0.768 0.768 0.755 0.806

High MAP-Read (F06, W07, S07) 0.804 0.779 0.816 0.784 0.791 0.746 0.804

High MAP-Read + High MAP-Math 0.814            

High MAP-Read +WASL06 Read scale   0.803 0.835 0.824 0.816  0.780  

High MAP-Read + WASL 04_Rscale             0.787

4251

0011 0010 1010 1101 0001 0100 1011

Snooping (Math)R-values

WASL 2007 Math vs.Grade

3 4 5 6 7 8 10

WASL Math 2006   0.801 0.840 0.875 0.873 0.890  

WASL Math 2005       0.603      

WASL Math 2004             0.856

MAP-Math Spring 2007 0.817 0.862 0.862 0.865 0.888 0.896 0.828

MAP-Math Winter 2007 0.809 0.851 0.860 0.863 0.883 0.906 0.856

MAP-Math Fall 2006 0.794 0.817 0.848 0.833 0.879 0.907 0.865

High MAP-Math (F06, W07, S07) 0.832 0.877 0.879 0.878 0.902 0.915 0.879

High MAP-Math + High MAP-Read 0.846            

High MAP-Math + WASL06 Math scale   0.892 0.896 0.910 0.921 0.930  

Highest MAP-M + WASL04 Math scale             0.908

4251

0011 0010 1010 1101 0001 0100 1011

What we learned by snooping. . .• Correlations were generally good.

– Reading R-value range: 0.711 - 0.835– Math R-value range: 0.603 - 0.921

• Correlations in math were stronger than in reading.

• “Highest MAP” consistently correlated better than any single MAP score.

• Correlations were generally strongest when Highest MAP and WASL 2006 factors were combined.

4251

0011 0010 1010 1101 0001 0100 1011

Regression ModelsFor students with both MAP and 2006 WASL scores (~95%)

WASL 2007 = b0 + b1*Highest MAP + b2*WASL 2006

For students that only had MAP score(s) (~3%)WASL 2007 = b0 + b1*Highest MAP

For students that only had WASL 2006 score (~2%)

WASL 2007 = b0 + b1*WASL 2006

Where: Highest MAP = The student’s highest score on MAP

from the Fall 2006, Winter 2007, or Spring 2007 windows.

Typically Spring 2007.

WASL 2006 = The student’s raw score from the 2006 WASL Spring testing.

4251

0011 0010 1010 1101 0001 0100 1011

Prediction ModelsFor students with both MAP and 2007 WASL scores

WASL 2008 = b0 + b1*Projected MAP + b2*WASL 2007

For students with only MAP score(s)WASL 2008 = b0 + b1*Projected MAP

For students with only WASL 2007 score

WASL 2008 = b0 + b1* WASL 2007

Where: Projected MAP = Projected Spring 2008 MAP score based

on the student’s highest score on MAP from the Winter 2007, Spring 2007 or Fall 2008 windows.

WASL 2007 = The student’s raw score from the 2007 WASL Spring testing.