Productive Investments in Pernambuco
Paying Teachers to Perform:The Effects of Bonus Pay in
Pernambuco, Brazil Barbara Bruns Lead Education Economist, Latin
America and The Caribbean RegionWorld Bank
Claudio FerrazProfessor of EconomicsPontifical Catholic
University (PUC)Rio de Janeiro, Brasil
Jessica Rodriguez, Tassia Cruz, Vitor Pereira
OPCS PSP GET PREM Knowledge and Learning SeriesWashington, DC,
May 8 2012
Teacher Bonus Pay (Pay for Performance)Theory of action Global
evidence
Context for Bonus pay program in Pernambuco
Results to date
Next Steps
OverviewTeacher Pay for performance: theory of action
(1)Accountability strategy: strengthens incentives/pressures for
teachers to produce key results (growing use in OECD, MICs, 15
programs in Brasil since 2007)
Theory of action: Bonus pay linked to student learning results
can Create short term stimulus for teachers to exert more effort
and focus on learning results Create long-term effects on teacher
selection (by making profession more attractive to high
performers)
Logical policy in context of: Consensus about the desired
outcome:student learning gains are key education result for growth
and inequality reductionBetter measures of students learning
Difficulty monitoring the desired teacher behavior directly
(effort and effectiveness in producing learning)
Perception that teacher salaries are a problemLow and
undifferentiated pay making it difficult to attract talent into
teachingNo links between salary and performance (91% of salary
variance in PE explained by yrs on job)
Increasingly common in OECD (US) and MICs (Brazil, Chile, Est.
de Mexico PROEBB, Ser Maestro and NL Premio al Merito Docente)
Annual rewards using two main modelsReward individual
teachersFor classroom level learning improvements - outcomeFor
teacher behaviors (eg, attendance or skills) - inputReward schools
For school-level learning improvements - outcomeFor observable
behaviors input
Performance incentives strongest if individual teachers are
rewarded for their students learning progress, but hard to
doRequires testing all grades and subjects twice a year
Most systems adopt school-based bonuses3Teacher Pay for
performance: theory of action (2)Design
Most common form is school-based bonus (individual bonus hard to
implement) rewarding year on year learning gains
Relatively expensive (10-20% of wage bill) compared to other
accountability reforms such as information and SBM (but not
compared to across-the-board wage increases)
Highly controversial
Corruptible: Extensive evidence of gaming and cheating on high
stakes tests in US contextDebate on whether teacher effort is the
binding constraint? Or teacher capacity? Little evidence yet on
longer-term selection effects
Increasingly common in OECD (US) and MICs (Brazil, Chile, Est.
de Mexico PROEBB, Ser Maestro and NL Premio al Merito Docente)
Annual rewards using two main modelsReward individual
teachersFor classroom level learning improvements - outcomeFor
teacher behaviors (eg, attendance or skills) - inputReward schools
For school-level learning improvements - outcomeFor observable
behaviors input
Performance incentives strongest if individual teachers are
rewarded for their students learning progress, but hard to
doRequires testing all grades and subjects twice a year
Most systems adopt school-based bonuses4Teacher Pay for
Performance: Global evidence Seven rigorous evaluations outside US
bonus pay raised student learning outcomes 0.13-0.27 SD (highest
for individual bonus in India)
Latest US evidence is different: even significant individual
bonuses had no impact on student results (Nashville, NYC)
In several developing country cases bonus did not lower teacher
absence (puzzling)
Most consistent pathway in JPAL studies (very low income
settings) was extra teacher effort out of school
5Teacher Pay for Performance: Global evidence Research frontier
now is to explain these results:
Context (low/high accountability?) Neal, 2011
Design features?
Controllability (noise in performance measure + locality)Size of
bonusCoverage/predictability of bonus
Deepen understanding of how incentives change teacher behavior
inside the classroom Positive: stimulate teacher effort (higher
attendance, more preparation)Negative: cheating, teaching to
test6Teacher Pay for Performance: Global evidenceCountryPredicted
Strength, Core Design (bonus size, predictability, noise in
performance measure)Average Bonus Value (% MW)Observed Effect Size
(Max.)India: Andhra Pradesh: test scores(RCT)4.7 (indiv)4.3
(group)3636.27.16 Rajasthan: attendance (RCT)4.7
(indiv)30.17Israel: Individual bonus based on student test results
(RD)4.3 (indiv)30014% higher pass rates, 10% higher test
scoresIsrael: School bonus based on multiple student performance
measures (tests, graduation rates, credits taken) (RD)2.7
(group)
40.13Kenya: school bonus, test scores (RCT)443.14Brazil:
Pernambuco school bonus (DD) 3180.44Chile: SNED school
bonus340.127Teacher Pay for Performance: Global evidence
PEPE8
Bonus pay program in Pernambuco Brazil
Context for Bonus Pay in Pernambuco, Brasil 27 States and 5,000+
municipalities deliver basic education
But federal govt. has put in place: funding equalization and
high quality national results measurement system IDEBTest
scoresPass rates
Pernambuco:950 State schools1 million studentsLarge schools
(1,000 students and 30-40 teachers/per school)Many teachers work
two jobs (state and/or municipal or private school
Weak salary incentives for teachers (unified salary scale 90% of
salary explained by age, exp, and educationWeak sanctions and
supervision (no dismissal for poor performance or high absence)
Limited results focus:Teachers late or absentUnused school
libraries and computer labs
Pernambuco Educational Context
Pernambuco Educational ContextIn 2007, Pernambuco had some of
the worst education outcomes in BrazilIn 2008, the state government
introduced a performance pay system for teachers: the Bonus de
Desempenho Escolar (BDE)
Reward for achieving 50% or more of annual school targets,
modeled on IDEB 12
Targets sent to School Principals13June 2008Bonus Program
announcedAug 2008Targets sent to schoolsNov 2008SAEPE examJune 2009
Bonus paidAug 2009New targets Nov 2009SAEPE/ Prova Brasil examsMay
2010Bonus PaidAugust 2010 New targetsNov 2010 2nd round classroom
observationsDec 2010 SAEPE examApril 2011 Bonus paidProgram
ImplementationNov 20091st round classroom observationsQuestion 1:
Did the bonus work? Did it improve test scores and pass rates?
(more than they would have improved w/o the bonus)Question 2: How
did it work?How did schools respond? What strategies? Change in
strategies over time?Did teacher behavior change in response to
incentives?Did effects vary by student, teacher, and school
characteristics?Were there negative effects? (eg. gaming,
cheating)Question 3: How do schools respond to non-linear
performance targets and threshold? Did not winning the bonus
encourage or discourage effort in subsequent year?Research method:
Difference in differences analysis plus classroom observation and
directors interviews Pernambuco Teacher Bonus: Research
questions15Pernambuco School BonusProgram Results
16State IDEPE Index Results, 9th grade
17State IDEPE Index Results, 11th grade (high school)
18Strategy: Difference-in-differencesCan answer the question:
Did the Pernambuco state schools improve more than they otherwise
would have after the introduction of the bonus?Method: exploit rich
national and state-level performance data for the period pre- and
post bonus to compare trends in:Pernambuco state schools with other
Northeast states
Impact Evaluation: Did the bonus work?
19Pre-trend on test scores Portuguese 5th grade
20Pre-trend on test scores Math 5th grade
21Pre-trend on test scores Portuguese 9th grade
22Pre-trend on test scores Math 9th grade
23Impact Evaluation: Did the bonus work?24Post-bonus: PE state
schools vs. other NE states Effects on test scores Portuguese 5th
grade
25Post-bonus: PE state schools vs. other NE states Effects on
test scores Math 5th grade
26Post-bonus: PE state schools vs. other NE states Effects on
test scores Portuguese 9th grade
27Prova Brasil: Pernambuco state vs. Northeast
5th grade9th grade28Increase 10% in the pass rate
Pass rate: Pernambuco vs. Other Northeast States29Increase in
test scores for 9th grade, both for Portuguese (.36 SD) and Math.
(0.41 SD) using the difference-in-differences between Pernambuco
and other NE statesSmall effect on 5th grade math (.12 SD) and none
for 5th grade language minor Increases in approximately 10 to 12%
on pass rates for 6th to 8th gradesNo effects on pass rates for 1st
to 5th grades (but very few state schools offer these
grades)Overall Bonus Impact30Why? Secretariat believes reason is
less state school influence on 5th grade student performance Fewer
lower primary (5th grade) enrollments in state schools (officially,
this level is decentralized to municipalities) State policies have
focused more resources on upper primary and secondary schoolState
policies such as bonus may have cumulative results (although
results presented are short-term)
Overall Bonus Impact31Evaluation design: Classroom observations
in a panel of 1800 classrooms in 300 schools Can answer questions:
Do teachers behave differently in schools that achieve/dont achieve
the bonus?Did teacher behavior change over time in Pernambuco state
schools?Are changes in teacher classroom practice correlated with
results? Surveys/interviews with school directors, teachers and
regional supervisorsCan answer questions: What strategies do
schools use to try to reach targets?How do these strategies
correlate with bonus results?
Research Question #2: HOW did the bonus work?32
Bonus Pay and Teacher Classroom practice in Pernambuco
Results: Classroom Observation data translated.
33Analysis of Panel of 220 Schools in Pernambuco 2009 and 2010
(after introduction of school bonus program)
Slight increase in instructional time and decline in time spent
on classroom managementbut increase in time off task (social
interaction w/students) and no change in teacher absence from the
class
Increase in teacher effectiveness keeping ALL students
engaged(share of time entire class engaged with teacher) Small
Group: 2 to 5 studentsLarge Group: More than 5 studentsIncrease in
share of time teacher involved the entire class36Research Question
3: How do different targeting mechanisms affect bonus
incentives?37
Pernambuco school bonus: First round targets had sharp
discontinuities(8th grade Portuguese)
Pernambuco school bonus: Second round targets(eliminated
discontinuities)39
Pernambuco school bonus: Third round targets(linear)40
.but maintained non-linearity in the bonus distributionLosing
the bonus by small margin seemed to create incentives for
improvement compared to winning the bonus by small margin in year
1But effects heterogeneous based on school size and specific
targets Non-linear bonus assignment, at least close to
discontinuity, does not seem to induce a discouragement effectMuch
scope for further analysis of impact of target-setting
mechanismsPreliminary (and incomplete) results42Unlike US results,
bonus pay can stimulate improvement in key school results within
1-2 yearsPernambuco results to date even most conservative
estimates show a significant positive impact in the upper
gradesSummary: what are we learning from Pernambuco?43Summary: What
are we learning from Pernambuco?
PEPEPE44Early evidence that it is possible to observe
differences in classroom dynamics between improving and stagnant
schools, and system-wide progress over time (Hopefully) also some
insights into design questions in setting up target-based bonus
programsLinear or nonlinear targets?Linear or nonlinear bonus
assignment? (everyone gets it or only those that pass a threshold
for improvement Sao Paulo vs PE)Reward level of performance,
value-added, or both? Should schools that remain at the top be
rewarded (Sao Paulo vs PE)?Should targets be multi-year?
Renegotiated annually? Summary: what are we learning from
Pernambuco?45What strategies do schools most commonly employ in
response to bonus incentives? (2012 qualitative survey and
study)How does school size (free-rider potential) and school social
capital affect bonus attainment? (2012 analysis of existing survey
data)Does classroom-level improvement continue over time? Does it
correlate with bonus? (panel of observations continuing in 2012 and
2013)Do perverse impacts arise over time (as agents become more
familiar with program and potential gaming strategies)? (unique set
of high stakes and low stakes test data)How to design effective
bonus pay programs? (Comparative study of 3 yr. results in
different Brazilian programs MG, PE, SP, RJ 2012/13)Size of bonus?
Design of targets? Reward level of performance, value-added, or
both? Multi-year targets or renegotiated annually? Summary: next
phase of Brazil teacher incentives research46
Muito Obrigados!
Barbara Bruns [email protected] Ferraz
[email protected]
Claudio Ferraz 47
