Upload
tahir
View
21
Download
0
Embed Size (px)
DESCRIPTION
New York State Growth Model for Educator Evaluation 2011–12. July 2012 PRESENTATION as of 7/9/12. Today’s Agenda. Background The What, Why, and How of Growth Models and Measures Using Growth Measures for Educator Evaluation What Data Will Be Available and When?. Background. - PowerPoint PPT Presentation
Citation preview
www.engageNY.orgwww.engageNY.org1
New York State Growth Model for Educator Evaluation
2011–12
July 2012
PRESENTATIONas of 7/9/12
www.engageNY.orgwww.engageNY.org2 www.engageNY.orgwww.engageNY.org2
Today’s Agenda
Background The What, Why, and How of Growth Models and
Measures Using Growth Measures for Educator Evaluation What Data Will Be Available and When?
www.engageNY.orgwww.engageNY.org3
Background
www.engageNY.orgwww.engageNY.org4 www.engageNY.orgwww.engageNY.org4
Evaluating Educator Effectiveness
www.engageNY.orgwww.engageNY.org5
The What, Why, and How of Growth Models and Measures
www.engageNY.orgwww.engageNY.org6 www.engageNY.orgwww.engageNY.org6
By the End of This Section…. You should be able to:
–Explain why the state is measuring student growth and not achievement
–Describe how the state is measuring growth compared to similar students
–Define a student growth percentile and mean growth percentile
www.engageNY.orgwww.engageNY.org7 www.engageNY.orgwww.engageNY.org7
0
200
400
600
800
Student AStudent BStudent CStudent DStudent E
Ms. Smith
Prior Performance
0
200
400
600
800
Student AStudent BStudent CStudent DStudent E
Ms. Jones
Prior Performance
Prior Year Performance for Students in Two Teachers’ Classrooms
─ Proficiency
www.engageNY.orgwww.engageNY.org8 www.engageNY.orgwww.engageNY.org8
0
200
400
600
800
Student AStudent BStudent CStudent DStudent E
Ms. Smith
Prior Performance Current Performance
0
200
400
600
800
Student AStudent BStudent CStudent DStudent E
Ms. Jones
Prior Performance Current Performance
Current Year Performance of Same Students
─ Proficiency
www.engageNY.orgwww.engageNY.org9 www.engageNY.orgwww.engageNY.org9
Prior and Current Year Performance for Ms. Smith’s Students
Ms. Smith’s Class
Prior Score Current Score
Student A 450 510
Student B 470 500
Student C 480 525
Student D 500 550
Student E 600 650
www.engageNY.orgwww.engageNY.org10 www.engageNY.orgwww.engageNY.org10
2011 2012
Student A
450
High SGPs
Low SGPs
Student A’s Current Year Performance Compared to “Similar” Students
If we compare student A’s current score to other students who had the same prior score (450), we can measure her growth
relative to other students. We
describe her growth as a “student
growth percentile” (SGP). Student A’s SGP is the result of a statistical model and in this example is 45,
meaning she performed better in
the current year than 45% of similar
students.
www.engageNY.orgwww.engageNY.org11 www.engageNY.orgwww.engageNY.org11
Comparing Performance of “Similar” Students
Prior Year Score
Given any prior score, we see a range of
current year scores, which give us SGPs of 1 to 99.
www.engageNY.orgwww.engageNY.org12 www.engageNY.orgwww.engageNY.org12
SGPs for Ms. Smith’s Students
Ms. Smith’s Class
Prior Score
Current Score
SGP
Student A 450 510 45
Student B 470 500 40
Student C 480 525 70
Student D 500 550 60
Student E 600 650 40
www.engageNY.orgwww.engageNY.org13 www.engageNY.orgwww.engageNY.org13
Student Growth Percentiles: True or False?
1. A student with an SGP of 50 performed better than 50% of similar students.
2. A student with an SGP of 80 must be proficient.
3. A student with an SGP of 20 grew less than a student with an SGP of 60.
4. The highest SGP that a student can receive is 99.
5. A student with an SGP of 80 grew twice as much as a student with an SGP of 40.
www.engageNY.orgwww.engageNY.org14 www.engageNY.orgwww.engageNY.org14
From Student Growth to Teachers and Principals
Ms. Smith’s Class
SGP
Student A
45
Student B
40
Student C
70
Student D
60
Student E
40
To measure teacher performance, we find the mean growth percentile (MGP) for his or her students. To find an educator’s mean growth percentile, take the average of SGPs in the classroom. In this case:
Step 1: 45+40+70+60+40=255
Step 2. 255/5=51
Ms. Smith’s mean growth percentile (MGP) is 51, meaning on average her students performed better than 51% of similar students.
A principal’s performance is measured by finding the mean growth percentile for all students in the school.
www.engageNY.orgwww.engageNY.org15 www.engageNY.orgwww.engageNY.org15
Which Students Count in a Teacher’s or Principal’s MGP for 2011–12?
Student has valid test
scores for at least 2011–12 and 2010–11
Student has valid test
scores for at least 2011–12 and 2010–11
Student scores do not
count for 2011–12
Student scores do not
count for 2011–12
Yes
Student meets continuous enrollment
standard for 2011–12
Student meets continuous enrollment
standard for 2011–12
No
Student growth is
attributed to the teacher
and the school
Student growth is
attributed to the teacher
and the school
Yes
No
Expected for 2012–13: students weighted by
duration of instructional linkage
Expected for 2012–13: students weighted by
duration of instructional linkage
www.engageNY.orgwww.engageNY.org16 www.engageNY.orgwww.engageNY.org16
From Student Growth to Teachers and Principals
In order for an educator to receive a growth score, he or she must have a minimum sample size of 16 student scores in ELA or mathematics across all grades taught.
Examples:
–A teacher has a self-contained classroom with 8 students who take the 4th grade ELA and math assessments; this teacher would then have 16 student scores contributing to his or her growth score.
–A teacher has a class with 12 students in varied grades (4th, 5th, 6th) who take the ELA and math assessments for their respective enrolled grade level; this teacher would then have 24 student scores contributing to his or her growth score.
If an educator does not have 16 student scores, he or she will not receive a growth score from the state and will not receive information in the reporting system.
–Educators likely to have fewer than 16 scores should use student learning objectives (SLOs).
www.engageNY.orgwww.engageNY.org17 www.engageNY.orgwww.engageNY.org17
MGPs and Statistical Confidence
8787
Confidence Range
Upper
Limit
Lower
Limit
MGP
• NYSED will provide a 95% confidence range, meaning we can be 95% confident that an educator’s “true” MGP lies within that range. Upper and lower limits of MGPs will also be provided.
• An educator’s confidence range depends on a number of factors, including the number of student scores in their MGP and the variability of student performance in the classroom.
www.engageNY.orgwww.engageNY.org18 www.engageNY.orgwww.engageNY.org18
Pause and Reflect: Mean Growth Percentiles
We talked about:–How to find a mean growth percentile (MGP)–How to interpret an MGP–What students are counted in an MGP–How many student scores are needed to provide an MGP–How a measure of statistical confidence (upper and lower
limits of a 95% confidence range) will be provided with MGPs and why
www.engageNY.orgwww.engageNY.org19 www.engageNY.orgwww.engageNY.org19
Expanding the Definition of “Similar” Students
So far we have been talking about “similar” students as those with the same prior year assessment score
We will now add two additional features to the conversation:
Two additional years of prior assessment scores– Remember—a student MUST have current year and prior year
assessment score to be included
Student-level factors–Economic disadvantage (ED)–Students with disabilities (SWDs)–English language learners (ELLs)
www.engageNY.orgwww.engageNY.org20 www.engageNY.orgwww.engageNY.org20
Adjustments for Three Student-Level Factors in Measuring Student Growth
Student
performance
Teacher
Instruction
Other factors(12–13) Economic
disadvantageLanguage
proficiency
Disability
www.engageNY.orgwww.engageNY.org21 www.engageNY.orgwww.engageNY.org21
2011 2012
Student A
450
High SGPs
Low SGPs
Student A’s Current Year Performance Compared to “Similar” Students
If we compare student A’s current score to other students who had the same prior score (450), we can measure his or her growth relative to
other students. We describe that growth as a student growth percentile (SGP).
Student A’s SGP is the result of a statistical
model and in this example is 45,
meaning student A performed better in
the current year than 45% of similar
students.
www.engageNY.orgwww.engageNY.org22 www.engageNY.orgwww.engageNY.org22
2011 2012
Student A
450
High SGPs
Low SGPs
Expanding the Definition of “Similar” Students to Include Economically Disadvantaged—An Example
Now if student A is economically
disadvantaged, we compare student A’s current score to other students who had the same prior score (450) AND who are also
economically disadvantaged. In
this new comparison group,
we see that student A now has
an SGP of 48.
www.engageNY.orgwww.engageNY.org23 www.engageNY.orgwww.engageNY.org23
Further Information on Including Student Characteristics in the Growth Model
The following slides were developed using sample data from 2010–2011.
–The “combined” MGPs on the charts have been calculated at the educator level (combining all grades and subjects).
–Not all districts provided data linked to teachers for grades 4–8 ELA/Math in 2010–11.
www.engageNY.orgwww.engageNY.org24 www.engageNY.orgwww.engageNY.org24
Teacher MGPs after Accounting for Economic Disadvantage
Taking student-level characteristics into account helps ensure educators with many students with those characteristics have a fair chance to achieve high or low MGPs. For example, note that for teachers with any percent of economically disadvantaged students, teacher MGPs range from 1 to 99.
NOTE: Beta results using available 2010–11 data.
www.engageNY.orgwww.engageNY.org25 www.engageNY.orgwww.engageNY.org25
Teacher MGPs after Accounting for SWD
NOTE: Beta results using available 2010–2011 data.
www.engageNY.orgwww.engageNY.org26 www.engageNY.orgwww.engageNY.org26
Teacher MGPs after Accounting for ELL
Percent of ELL Students in Class
NOTE: Beta results using available 2010–2011 data.
www.engageNY.orgwww.engageNY.org27 www.engageNY.orgwww.engageNY.org27
“Similar” Students: A Summary
“Similar” Student Characteristics
Unadjusted Mean Growth Percentiles
Adjusted Mean Growth Percentiles
Up to Three Years of Prior Achievement
Up to Three Years of Prior Achievement
English Language Learner (ELL) Status
Students with Disabilities (SWD) Status
Economic Disadvantage
Reported to Educators Reported to Educators
Used for Evaluation
www.engageNY.orgwww.engageNY.org28 www.engageNY.orgwww.engageNY.org28
One Last Feature of the Growth Model….
All tests contain measurement error,
with greater uncertainty for
highest and lowest achieving students
All tests contain measurement error,
with greater uncertainty for
highest and lowest achieving students
The New York growth model accounts for measurement error in computing student growth percentiles.
www.engageNY.orgwww.engageNY.org29 www.engageNY.orgwww.engageNY.org29
State Growth Model Summary
Growth model for 2011–12 only for grades 4–8 ELA/Math for teachers and principals
www.engageNY.orgwww.engageNY.org30 www.engageNY.orgwww.engageNY.org30
By the End of This Section…. You should be able to:
–Explain why the state is measuring student growth and not achievement
–Describe how the state is measuring growth compared to similar students
–Define a student growth percentile and mean growth percentile
www.engageNY.orgwww.engageNY.org31
Using Growth Measures for Educator Evaluation
www.engageNY.orgwww.engageNY.org32 www.engageNY.orgwww.engageNY.org32
By the End of This Section…. You should be able to:
–Explain how growth ratings and scores will be obtained, using illustrative data
www.engageNY.orgwww.engageNY.org33 www.engageNY.orgwww.engageNY.org33
Growth Ratings and Score Ranges
Growth Rating Description Growth Score Range
(2011–12)
Highly Effective
Well above state average for similar students
18–20
Effective Results meet state average for similar students
9–17
Developing Below state average for similar students
3–8
Ineffective Well below state average for similar students
0–2
The growth scores and ratings are based on an educator’s combined MGP.
www.engageNY.orgwww.engageNY.org34 www.engageNY.orgwww.engageNY.org34
Distribution of 2010–11 Teacher-Level MGPs
MGP
1 50 99
NOTE: Beta results using available 2010–2011 data.
For
illustrative
purposes
only
0.0
1.0
2.0
3.0
4N
umbe
r of
Tea
cher
s
Distribution of Teacher Growth ScoresDistribution of Mean Student Growth Percentiles (Teacher Level)
Per
cen
t o
f M
GP
s
www.engageNY.orgwww.engageNY.org35 www.engageNY.orgwww.engageNY.org35
MGPs and Statistical Confidence
8787
Confidence Range
Upper
Limit
Lower
Limit
MGP
• NYSED will provide a 95% confidence range, meaning we can be 95% confident that an educator’s “true” MGP lies within that range. Upper and lower limits of MGPs will also be provided.
• An educator’s confidence range depends on a number of factors, including the number of student scores included in his or her MGP and the variability of student performance in the classroom.
www.engageNY.orgwww.engageNY.org36 www.engageNY.orgwww.engageNY.org36
HEDI Classification Approach for Teachers (using 2010–11 sample data)
Effective requires MGPs within 1 standard deviation of the average MGP of 51.
–MGPs between 40 and 61 will earn Effective ratings.
Well Above Average (Highly Effective) requires–MGP of 62 or higher–AND confidence range above 51. (If not, rating is
Effective.)
Well Below Average (Ineffective) requires–MGP of 39 or lower–AND confidence range must be less than 51. (If not,
rating is Developing.)
www.engageNY.orgwww.engageNY.org37 www.engageNY.orgwww.engageNY.org37
Yes
Yes
Yes
No
Yes
From MGPs to Growth Ratings: Teachers
Mean Growth
Percentile ≥62
Lower Limit > 51
Highly Effective: Results are well
above state average for
similar students
Mean Growth
Percentile ≤39
Upper Limit < 51
Ineffective: Results are well
below state average for
similar students
Developing: Results are below state average for similar students
No Effective: Results equal
state average for similar students
Mean Growth Percentile Confidence Range Growth Rating
Mean Growth
percentile 40–61
YesAnyYes
www.engageNY.orgwww.engageNY.org38 www.engageNY.orgwww.engageNY.org38
HEDI Classification Approach for Principals (using 2010–11 sample data)
Same methodology as for Teachers. Slightly different cut scores.
Effective requires MGPs within 1 standard deviation of the average MGP of 50.
–MGPs between 43 and 57 will earn Effective ratings.
Well Above Average (Highly Effective) requires:–MGP of 58 or higher–AND confidence range above 50. (If not, rating is
Effective.)
Well Below Average (Ineffective) requires–MGP of 42 or lower–AND Confidence Range must be less than 50. (If not,
rating is Developing.)
www.engageNY.orgwww.engageNY.org39 www.engageNY.orgwww.engageNY.org39
Yes
Yes
Yes
No
Yes
From MGPs to Growth Ratings: Principals
Mean Growth
Percentile ≥ 58
Lower Limit > 50
Highly Effective: Results are well
above state average for
similar students
Mean Growth
Percentile ≤ 42
Upper Limit < 50
Ineffective: Results are well
below state average for
similar students
Developing: Results are below state average for similar students
No Effective: Results equal
state average for similar students
Mean Growth Percentile Confidence Range Growth Rating
Mean Growth
percentile 43–57
YesAnyYes
www.engageNY.orgwww.engageNY.org40 www.engageNY.orgwww.engageNY.org40
Illustrating Possible Teacher Growth Ratings
MGP 1 MGP 99
Well Below
Average
(39)
Well Below
Average
(39)
Average
(51)
Average
(51)
Well Above
Average
(62)
Well Above
Average
(62)
MGP
MGP
MGP
MGP
MGP
MGP
MGP
www.engageNY.orgwww.engageNY.org41 www.engageNY.orgwww.engageNY.org41
Illustrating Possible Teacher Growth Ratings
MGP 1 MGP 99
Well Below
Average
(39)
Well Below
Average
(39)
Average
(51)
Average
(51)
Well Above
Average
(62)
Well Above
Average
(62)
MGP
MGP
MGP
MGP
MGP
MGP
MGP
www.engageNY.orgwww.engageNY.org42 www.engageNY.orgwww.engageNY.org42
Illustrating Possible Teacher Growth Ratings
Ineffective
Developing
Highly Effective
Effective
MGP 1 MGP 99
Well Below
Average
(39)
Well Below
Average
(39)
Average
(51)
Average
(51)
Well Above
Average
(62)
Well Above
Average
(62)
MGP
MGP
MGP
MGP
MGP
Developing
www.engageNY.orgwww.engageNY.org43 www.engageNY.orgwww.engageNY.org43
Illustrating Possible Teacher Growth Ratings
Effective
Effective
MGP 1 MGP 99
Well Below
Average
(39)
Well Below
Average
(39)
Average
(51)
Average
(51)
Well Above
Average
(62)
Well Above
Average
(62)
MGP
MGP
www.engageNY.orgwww.engageNY.org44 www.engageNY.orgwww.engageNY.org44
Illustrative Results: Teachers(Using 2010–11 sample data)
Rating &Points (2011–12 )
Number of Teacher MGPs
Percent of Teacher MGPs
Highly Effective18–20
1618 7%
Effective9–17
16,681 76%
Developing3–8
2015 10%
Ineffective0–2
1419 7%
Points available within each HEDI category will be assigned based on educator MGP
www.engageNY.orgwww.engageNY.org45 www.engageNY.orgwww.engageNY.org45
Illustrative Results: Principals(Using 2010–11 sample data)
Rating &Points (2011–12)
Number of Principal MGPs
Percent of Principal MGPs
Highly Effective18–20
345 10%
Effective9–17
2696 75%
Developing3–8
318 9%
Ineffective0–2
241 7%
Points available within each HEDI category will be assigned based on educator MGP
www.engageNY.orgwww.engageNY.org46 www.engageNY.orgwww.engageNY.org46
By the End of This Section…. You should be able to:
–Explain how growth ratings and scores are obtained
www.engageNY.orgwww.engageNY.org47
What Data Will Be Available and When
www.engageNY.orgwww.engageNY.org48 www.engageNY.orgwww.engageNY.org48
Data — What to Expect When
Growth scores
provided to
districts
Mid-July
Test scores finalized
and teacher linkage
data final submission
Early fallMid-August
Online reporting
system available
www.engageNY.orgwww.engageNY.org49 www.engageNY.orgwww.engageNY.org49
Data — What to Expect in AugustData Elements (for teachers and schools) Unadjusted mean growth percentiles (Unadjusted MGPs) Adjusted mean growth percentiles (Adjusted MGPs and upper and lower
limits based on confidence range for these adjusted MGPs) Percent of students above the State median: this will be provided at the
teacher and school level, and can be used as a local measure in APPR Number of student scores included Growth rating (HEDI) Growth score (0–20)
Breakdowns (by teacher and school) MGPs by subject, grade, and overall (not HEDI)
– Can be used with SLOs as part of the Comparable Measures or Locally Selected Subcomponent
Overall MGPs for subgroups — ELL, SWD, Economic Disadvantage, High- and Low-Achieving
– Subgroup scores will not be included on reports if there are fewer than 16 student scores
www.engageNY.orgwww.engageNY.org50 www.engageNY.orgwww.engageNY.org50
One Teacher’s Information — August
Number of
Student Scores
Percent of Students Above
the State Median
Unadjusted MGP
Adjusted MGP
Adjusted MGP
Growth Rating
Growth ScoreLower Limit Upper Limit
Jane Smith 56 60 70 75 65 85 Highly
Effective 18
Number of
Student Scores
Percent of Students Above
the State Median
Unadjusted MGP
Adjusted MGP
Adjusted MGP
Lower Limit Upper Limit
Jane Smith
Math
Math Grade 4 28 70 75 78 65 88
ELA
ELA Grade 4 28 50 65 68 55 79
Students with disabilities 4 * * * * *
English language learners 0 * * * * *
Economically disadvantaged 2 * * * * *
Low achieving (Level 1) 4 * * * * *
High achieving (Level 4) 4 * * * * *
www.engageNY.orgwww.engageNY.org51 www.engageNY.orgwww.engageNY.org51
Number of Students
Percent of Students Above
the MedianUnadjusted
MGPAdjusted
MGP
Adjusted MGP
Lower Limit Upper Limit
Jane Smith
Math
Math Grade 4 28 70 75 78 65 88
ELA
ELA Grade 4 28 50 65 68 55 79
Students with disabilities 4 * * * * *
English language learners 0 * * * * *
Economically disadvantaged 2 * * * * *
Low achieving 4 * * * * *
High achieving 4 * * * * *
Adjusted MGP: 75
Adjusted MGP: 75
Number of Students
Percent of Students Above
the MedianUnadjusted
MGPAdjusted
MGP
Adjusted MGP
Growth Rating
Growth ScoreLower Limit Upper Limit
Jane Smith 56 60 70 75 65 85Highly
Effective 18
2 SWD students, 0 ELL students, 1 econ
disadvantaged student, 2 high- and
2 low-achieving students
2 SWD students, 0 ELL students, 1 econ
disadvantaged student, 2 high- and
2 low-achieving students
Math number of
student scores:
28
Math number of
student scores:
28
ELA number of
student scores:
28
ELA number of
student scores:
28
Total number of
student scores:
56
Total number of
student scores:
56
Upper and lower limits of
adjusted MGP: 65 and 85
Upper and lower limits of
adjusted MGP: 65 and 85
Growth rating of highly
effective and growth score of
18
Growth rating of highly
effective and growth score of
18
60 percent of students above
the State Median
60 percent of students above
the State Median
Unadjusted MGP: 70 Unadjusted MGP: 70
One Teacher’s Information — August
Adjusted MGPs by subject—can be used in an SLO for the Comparable
Measures subcomponent
Adjusted MGPs by subject—can be used in an SLO for the Comparable
Measures subcomponent
No scores reported here since fewer
than 16 student scores in a sub-
group
No scores reported here since fewer
than 16 student scores in a sub-
group
Unadjusted MGPs by subject
Unadjusted MGPs by subject
www.engageNY.orgwww.engageNY.org52 www.engageNY.orgwww.engageNY.org52
Scavenger Hunt and Quiz1. What is Ms. Smith’s overall
adjusted MGP?2. What are the upper and lower
confidence limits for Ms. Smith’s overall MGP and what do they represent?
3. How many scores are included from Ms. Smith’s class for ELA?
4. What is the adjusted MGP for Ms. Smith’s class in ELA?
5. How do Ms. Smith’s high-achieving students compare to her low-achieving students in terms of growth?
6. What score is Ms. Smith’s growth rating based on?
www.engageNY.orgwww.engageNY.org53 www.engageNY.orgwww.engageNY.org53
Definitions SGP (student growth percentile): the result of a
statistical model that calculates each student’s change in achievement between two or more points in time on a state assessment or other comparable measure and compares each student’s performance to that of similarly achieving students
Similar students: students with the same prior test scores, ELL, SWD, and economic disadvantage status
ELLs: English language learners SWD: students with disabilities Economic disadvantage: a student who participates in,
or whose family participates in, economic assistance programs such as the Free- or Reduced-price Lunch Programs (FRPL), Social Security Insurance (SSI), Food Stamps, Foster Care and others
www.engageNY.orgwww.engageNY.org54 www.engageNY.orgwww.engageNY.org54
Definitions High-achieving, low-achieving: defined by the
performance of students based on prior year state assessment scores (i.e., Level 1 = low-achieving, Level 4 = high-achieving)
MGP (mean growth percentile): the average of the student growth percentiles attributed to a given educator
“Unadjusted” MGP: an MGP based on SGPs that have NOT accounted for ELL, SWD, and economic disadvantage status
“Adjusted” MGP: an MGP based on SGPs that HAVE accounted for ELL, SWD, and economic disadvantage status
Growth rating: HEDI rating based on growth Growth score: growth subcomponent points from 0–20
www.engageNY.orgwww.engageNY.org55 www.engageNY.orgwww.engageNY.org55
Definitions Measurement error: uncertainty in test scores due to
sampling of content and other factors Standard error: a measure of the statistical uncertainty
surrounding a score Standard deviation: a measure that shows the spread
of scores around the mean Upper/lower limit: highest and lowest possible MGP
taking statistical confidence into account Confidence range: range of MGPs within which we
have a given level of statistical confidence that the true MGP falls (95% statistical confidence level used for state growth measure)
www.engageNY.orgwww.engageNY.org56 www.engageNY.orgwww.engageNY.org56