Including a detailed description of the Colorado Growth Model

1

OverviewReview Growth Model OptionsDetails on Colorado Growth ModelMajor technical questions

2

Basic DefinitionsStatus describes the academic performance

of a student or group (a collection of students) at a single point in time.

Growth describes the academic performance of a student or group (a collection of students) over two or more time points.

A growth model is a collection of definitions, calculations, and rules that quantifies student performance over two or more time points and supports interpretations about students, their classrooms, their educators, and their schools.

3

School-level “Growth” MeasuresYear to Year changes in percent met

safe harborGrade to Grade changes in percent met or

average scoresHow are my fifth graders doing compared to

my fourth gradersCohort growth over time in percent met or

average scoresHow has my 2009-10 third grade class

performed overtime

4

Growth ModelsA growth model is a collection of definitions,

calculations, and rules that quantifies student performance over two or more time points and supports interpretations about students, their classrooms, their educators, and their schools.

Most growth models follow individual student growth, and these are the subject of discussion today.

5

Common Growth ModelsGain ScoresTrajectoryCategoricalResidual GainProjectionStudent Growth PercentilesMultivariate

6

Key Questions Growth ModelsWhat interpretation does the growth model

support Formative? School accountability? Teacher evaluation?

Program Evaluation?What is the statistical foundation?

Gain-based, enhanced status, multivariateWhat are required data features?

Vertical scale? Articulated cut scores? Large datasets? Factors beyond test scores?

What interpretations does the model support? Does aggregation at the school level make sense?

Does the model include growth to standard?What are common misinterpretations and threats

to validity?7

Gain ScoresDefinition: Calculates the gain in a student’s

test score from year to year. Example: 5th grade math score – 4th grade math score

Requires: vertical scale.Strengths: very easy to calculate, very easy to

aggregatedWeaknesses: may not be very reliable, easier to

manipulate, poor vertical scale lessens value.School measure: average gainsOregon data: Gains vary by previous

performance levels, see next slide.8

Gain Scores – Oregon Data

9

Growth From Grade 5

-10

-5

0

5

10

15

20

25

200 210 220 230 240 250

Growth to Grade 6 Growth to Grade 7 Growth to Grade 8

Trajectory ModelDefinition: If a student continues on this path,

where will she be in the future (Oregon’s current model)

Requires: vertical scale useful, but not requiredStrengths: Simple to computeWeaknesses: assumes students make same

growth year after year, targets for student already meeting may be difficult to determine

School measure: percent of students “on track”Oregon data: see next slide.

10

Percent of students “On Track”

11

Categorical ModelDefinition: evaluates students transitioning

between performance levelsRequires: cut scores, benefits from articulated

cut scoresStrengths: useful when vertical scale not

available Weaknesses: coarse measure of student progressSchool measure: percent on track or weighted

index of students improving their performance levels

Oregon data: none12

Residual Gain ModelDefinition: student gains are measured against

“expected gains.” This is an “enhanced status model.”

Requires: state level data, model to predict expected gains, does NOT require vertical scaling

Strengths: value added measure, statistically robust - many statistics can be derived from the measure (median, standard deviation, school variability)

Weaknesses: does not directly measure growth, growth to standard is an add-on

School measure: average residual gains.Oregon data: gains are based on the mean gain,

conditioned on the prior test score. See next slide.

13

Residual Growth versus Status

14

Projection ModelDefinition: uses a regression model to predict future

scores. (Oregon’s federal growth model pilot submission was a hybrid of this model and a trajectory model)

Requires: linear regression model, state level data. Strengths: empirically grounded: uses historic data

to help predict current student’s performance, can use data from multiple subjects, supports growth to standard

Weaknesses: assumptions, such a linear growth, are often untenable.

School measure: average predicted score, percent of students “on track”

Oregon data: I’ve looked at versions of this, but don’t have pretty data to present.

15

Student Growth PercentilesDefinition: student growth compared to other

students with similar score histories, expressed as a percentile. It is an “enhanced status model”.

Requires: state data, large data sets, does NOT require vertical scale

Strengths: includes growth to standard algorithm, is value added

Weaknesses: complex modelSchool measure: median or mean growth percentile.Oregon data: see next slide and later in this

presentation.

16

Growth Percentiles in Oregon

17

The Colorado Growth ModelBasic AssumptionsSample DataAdequate GrowthMajor technical decisions

18

Quick DescriptionStudents with two consecutive years of test

scores are provided with a growth percentile.Student growth is compared to “academic

peers”, who are students with similar score histories.

A student growth percentile of 60 means the student “grew” as much or more than 60 percent of students with similar test score histories.

Various percentiles of growth can be projected into the future to determine “Adequate Growth Percentiles”

19

GlossaryAcademic Peers – students with the same or

similar test scores in the past.Student Growth Percentile (SGP) – the

growth of the student as compared to her academic peers

Adequate Growth Percentile (AGP) – the growth percentile that needs to be sustained in order for the student to meet or still meet standard in 3 years.

20

Growth Percentiles in Oregon

21

Sample SGP Output

22

Prior Grade

Current

Grade

Prior RIT

Current RIT

Growth

SGP AGP

5th 6th 195 200 5 16 80

5th 6th 209 208 -1 8 72

5th 6th 209 214 5 34 64

5th 6th 209 218 9 61 58

5th 6th 209 222 13 85 53

5th 6th 209 226 17 96 48

5th 6th 227 232 5 57 27

Student Growth ReportStudent reports can include the following:Student test historyCurrent Growth percentile – is the growth

low, average, or highScores that correspond to 1st,35th, 65th and

99th percentile (for comparison)Projections into the future for various growth

percentiles

23

Sample Student Report

24

Aggregations at School LevelMedian SGP by subgroup

This is the recommendation by the SGP author, though others recommend mean

Median AGP by subgroupThis is the recommendation of the SGP author,

though others recommend meanSubgroups Making Adequate Growth

Comparison of the median SGP and the median AGP.

Percent meeting AGPThis is not in the proposed school ratings model.

25

Draft School Level Growth Report

26

Median SGP Median AGP

Subject

Subgroup2011-

122012-

13Combine

d2011-

122012-

13Combine

d

MathAll

Students45 51 47 34 37 36

Math LEP 51 54 53 56 61 58

Reading

All Students

55 52 53 26 28 27

Reading

LEP 45 48 47 65 63 64

Web-based reporting

27

Now the ComplicationsSGPs -- The model can take into account

more than two years of data for each student

By using more than two years of data we may get a better estimate of the student’s growth.

See the next slide for examples

28

SGPs – Higher Order Growth

29

Current Grade

Back 2 years RIT

Prior RIT

Current RIT

Growth

SGP Adjustment

7th 225 231 6 47

7th 206 225 231 6 77 +30

7th 211 225 231 6 69 +22

7th 216 225 231 6 57 +10

7th 221 225 231 6 45 -2

7th 226 225 231 6 33 -14

7th 231 225 231 6 17 -30

This data show how 3 years of test scores can affect growth percentiles.

Focus and Priority SchoolsNew “rating” system that will identify focus

and priority schools. Growth is a very strong factor in the rating.

30

Draft Focus/Priority Identification Weighting

Elem/Middle

High

Achievement 25% 20%

Growth 50% 20%

Subgroup Growth 25% 10%

Graduation -- 35%

Subgroup Graduation

-- 15%

Details on the waiver: http://www.ode.state.or.us/go/nextgen

Details on priority, focus, and model schools: http://www.ode.state.or.us/search/page/?id=3742

Priority, Focus and Model School Detail Sheets: http://www.ode.state.or.us/search/page/?id=3749

Policy and Technical Manual: http://www.ode.state.or.us/wma/policy/accountability/nextgen2012/nextgenaccountabilitymanual2012.pdf

Jon WiensEmail: jon.wiens@state.or.us

Contact Information and Links

31