Upload
malachi-hood
View
24
Download
0
Embed Size (px)
DESCRIPTION
How Much Value is Added? An Evaluation Plan for the Achievement Challenge Pilot Project. What’s wrong?. Teachers affect student performance, however… Policy problem General & specific teacher shortages Measuring teacher effectiveness Providing incentives to teachers Need - PowerPoint PPT Presentation
Citation preview
How Much Value is Added? An How Much Value is Added? An
Evaluation Plan for the Achievement Evaluation Plan for the Achievement Challenge Pilot ProjectChallenge Pilot Project
Little Rock Merit Pay Year 1 Evaluation
2
What’s wrong? Teachers affect student performance, however… Policy problem
General & specific teacher shortages Measuring teacher effectiveness Providing incentives to teachers
Need System to recruit, retain, and reward high quality
individuals in the teaching field
Little Rock Merit Pay Year 1 Evaluation
3
Can current system rectify the problems? Status Quo:
Single Salary System Based on tenure and degree
Arguments for single system: Fair Simple (critics call it a “breathing bonus”) Status quo
Concerns: Lacks extrinsic reward for innovation, creativity, hard work Lacks extrinsic reward for innovation Does not encourage or reward outcomes Does not recruit, retain, or reward effective teachers
If status quo isn’t working, what alternatives do we have?
Little Rock Merit Pay Year 1 Evaluation
4
Policy Solutions “Lump Sums” (recruit and retain)
Often in the form of lump increases Intuitively lacks motivation to work harder
Differential Pay (recruit and retain) Hard-to-staff schools Specific subjects Disadvantaged students
Merit Pay (rewards) Teacher characteristics Teacher behavior Student performance gains
Literature: Johnson, 2000; Lazear, 1996; Murnane & Cohen, 1986
Little Rock Merit Pay Year 1 Evaluation
5
Merit Pay Literature Supporters believe performance improves:
Innovation Work harder Salary satisfaction
Opponents believe performance decreases Counter-productive competition Degraded work environment Focus on high-performing students
Evidence: Very few evaluations Policy questions:
Effects of merit pay programs on student performance? Effects of merit pay programs on teachers?
Little Rock Merit Pay Year 1 Evaluation
6
Possible Policy Implications
Possible options: Improves student achievement, and teachers like
program: Improves student achievement, but teachers
dislike program Does not improve student achievement, but
teachers like program; Does not improve student achievement, and
teachers dislike program.
Little Rock Merit Pay Year 1 Evaluation
7
Achievement Challenge Pilot Project (ACPP)
Program Goals: Increase student performance Reward effective teachers Make positive influences to school culture Ultimately, recruits, retains, and rewards effective
teachers 5 elementary schools in Little Rock School District Financial rewards based on student performance
payouts computed as NCE gains between fall and spring tests (SAT-9; SAT-10) Meadowcliff payouts per student gain Wakefield payouts based on class average gains
Little Rock Merit Pay Year 1 Evaluation
8
ACPP Addresses Literature ConcernsTable 1: Payouts for Wakefield for 2006-07
Employee Type / Position0-4%
Growth5-9%
Growth10-14% Growth
15%+ Growth
Maximum Payout
Principal $2,500 $5,000 $7,500 $10,000 $10,000
Teacher (Grades 4 -5) $50 $100 $200 $400 $11,200
Teacher (Grades 1-3) $50 $100 $200 $400 $10,000
Teacher (Kindergarten) $50 $100 $200 $400 $8,000
Coach $1,250 $2,500 $3,750 $5,000 $5,000
Specialist; Spec. Ed. $1,000 $2,000 $3,000 $4,000 $4,000
Music Teacher $1,000 $2,000 $3,000 $4,000 $4,000
Physical Examiner $500 $1,000 $1,500 $2,000 $2,000
Aide $250 $500 $750 $1,000 $1,000
Secretary & Custodian $125 $250 $375 $500 $500
Little Rock Merit Pay Year 1 Evaluation
9
“Observable” School Characteristics
School Name
Enrollment, 2005-06
% Free/ Reduced Lunch, 2005-06
% Black, 2005-06
2-Year % Proficient,
Math, 2003, 2004
2-Year % Proficient, Literacy,
2003, 2004
Meadowcliff 270 90% 80% 45.8 49.4Wakefield 365 94% 75% 47.2 54.0Baseline 202 96% 97% 54.9 59.2Chicot 367 89% 75% 37.3 44.1Franklin 281 95% 96% 25.0 45.3Treatment 635 93% 78% 46.5 51.8Control 850 93% 82% 38.1 48.4
Table 2: Demographic Characteristics of Schools in 2005-06 ACPP Evaluation
Little Rock Merit Pay Year 1 Evaluation
10
Research Question 1: Student Effects Question:
What is the impact of the ACPP on the math performance of students?
Method: Student level fixed-effects regression model Data provided by the Little Rock School District
Test scores Stanford Achievement Test-9 (2003; 2004) Iowa Test of Basic Skills (2005; 2006)
Reduces “gaming effect”
Demographic data Race, Poverty (FRL), Gender, Age
Little Rock Merit Pay Year 1 Evaluation
11
Methods: Data – Why Math Only?
SubjectCohorts, 2005-06 ITBS, 2006
ITBS, 2005
SAT-9, 2004
SAT-9, 2003
Reading 1 Voc K: Voc
2 Total 1: Total
3 Total 2: Total
4 R Comp; Total 3: R Comp 2: Total
5 R Comp; Total 4: R Comp 2: Total
Math 1 Total K: Total
2 Total 1: Total
3 Total 2: Total
4 Total 3: Total 2: Total
5 Total 4: Total 2: Total
Table 3: Summary of Tests by Grade and Year for Fall 2006 Report
ITBS 2005, Language subtest not administered to Grade 4 & 5
Little Rock Merit Pay Year 1 Evaluation
12
Methods: Analytic Strategy
Regression Student-level Individual Fixed effects
Compares the difference in test scores for treatment students to the difference in test scores for control students
This model only applies to 4th and 5th grade students because they are the only students who possess pre-gains (2002-03 or 2003-04 to 2004-05) post-gains (2004-05 to 2005-06) Meadowcliff removed – no pre-gain scores
Little Rock Merit Pay Year 1 Evaluation
13
RQ1: Study Sample CharacteristicsTreatment
(n = 132)Comparison
(n = 334)Total
(n = 466)
Schools 1 3 4
Grade Level, 2005-06
Grade 4 55 166 221
Grade 5 77 168 245
Race
% African-American 78.8% 83.5% 83.2%
% Caucasian 4.5% 4.8% 4.7%
% Hispanic 15.2% 8.9% 10.7%
% Other 1.5% 2.4% 2.2%
Free and Reduced Lunch Status
% Free or Reduced 90.8% 92.1% 91.7%
% Full Pay 9.2% 7.9% 8.3%
Little Rock Merit Pay Year 1 Evaluation
14
Teacher Effects
What are the attitudes regarding merit pay of ACPP teachers compared to those of teachers in the comparison schools?
Little Rock Merit Pay Year 1 Evaluation
15
Teacher Survey
Advantages
Innovation Work harder Salary satisfaction
Disadvantages Counter-productive competition Degraded work environment Focus on high-performing students
Teacher effectiveness
Little Rock Merit Pay Year 1 Evaluation
16
Policy Implications & Conclusions ACPP improves student performance
Student performance increased 3.5 NCE points
Teachers support the ACPP Significantly more satisfied with ACPP than single salary
system Believe the program did not lead to counterproductive
competition Believe the school environment is more positive with ACPP Teachers believe ACPP has positive impacts for students
Based on student performance increases and teacher opinions, program should be expanded to other elementary schools.
Little Rock Merit Pay Year 1 Evaluation
17
Limitations & Policy Concerns Receptivity is a factor
Teacher support may be vital to program success Limited sample of teachers (58 treatment) Limited sample of students (132 treatment)
All from same school Only two grades used
Funding $225,000 / school
Little Rock Merit Pay Year 1 Evaluation
18
Survey Practice & Questions