Teacher Professional Development and its Effects on Students:
Evidence from the [email protected] Program in Italy
Gianluca Argentin Aline Pennisi
Daniele VidoniGiovanni AbbiatiAndrea Caputo
The problem
Weakness of Italian students in international assessments on mathematics and science
(i.e. IEA, TIMSS and OECD PISA).
North South gap achievement.
North/South gap increasing over grades(difference from national average in the percentage of correct answers in grades 2,5,6 and 8 – source: SNV)
-5,0
-4,0
-3,0
-2,0
-1,0
0,0
1,0
2,0
3,0
4,0
5,0
2 5 6 8 2 5 6 8 2 5 6 8
North Centre South
difference from national average
The opportunity
Thanks to the EU funding, there has been a boost in initiatives
to help schools and teachers to improve student achievements.
(PON Istruzione 2007-2013I-3-FSE-2009-2)
Target regions of PONEU funding.
One of the solutions
A professional development program, called [email protected], was offered to
tenured math teachers in lower/upper secondary schools.
Enrolment on voluntary basis at school/teacher levels.
The questions
Does [email protected] work?
Does it increase students’ math achievment?
Does it change teachers’ way of teaching?
We are tryng to get an answer using a rigorous method,
a Randomized Control Trial.
To our knwoledge, this is an absolute first in the Italian school system
How did we make randomization acceptable to Italian school
authorities?
Delaying by one year the PD of the control group,
instead of excluding it completely.
The [email protected] program in the Italian context
The [email protected] program:• math applied to daily life problems
• is based on a mixture of formal lectures and on-line mentoring;
• offers a huge repository ofscripts for math lessons;
• it lasts one entire school year;• it requires to be implemented in
classes (at least 4 units); • it promotes teacher community.
[email protected] seems promising, according to the literature on
professional development[Garet et al 2001; Desimone et al 2002]:
• content focused;• extended in duration;
• active learning processes; • implemented directly in classes;
• based on peer collaboration.
[email protected] seems promising also looking at the Italian teachers:
• the oldest worldwide [OECD 2007];• the majority did not have any
specific training in teaching;• the majority of math teachers
did not graduate in math/physics.
Moreover, during their career, the Italian teachers:• rarely attend PD;
• are not assessed at all;• do not have any salary differentiation
based on merit;• even do not have feedbacks
about their job [Talis 2008].
The Randomized Control Trial
The RCT – 1st year
1. Teachers applied for the PD through their school
2. The school must send at least 2 teachers
3. Schools were randomly assigned to treatment in the current year or to
delayed treatment next year
The estimate of the effect of the PD is the difference between T & C
Effect = T - C
For the first year, we have a classical experiment
Randomization
September 2009
Sent to the treatment
Control group
Schools 125 50Teachers 473 193
We checked the equivalence on an unusually wide set of characteristics at schools/teachers/students levels.
The equivalence is guaranteed.We found only minor differences,
anyway controlled in our estimates.
The outcomes
Students: • math test scores on the
standardized National Assessment• attitude toward math
Teachers: •self reported teaching behaviours•attitudes toward math teaching
Data collectionNovember 2009/January 2010
CATI survey pre-intervention on teachers’ attitudes
May 2010
Standardized math tests + questionnaires on students (background & attitude vs math)
+ data from teachers’ logs
December 2010CATI survey post-intervention on teachers’ attitudes and evaluation of the experience.
Intervention
November 2009 – May 2010
TEACHERS Sent to the treatment
(473)
Control group(193)
Lost – no data available
79 27
Compliers 156 166Non compliers 238 0
STUDENTS 7.692 3.372
Only 39% of teachers are compliers ….
Drop out Attended only lectures Attended courseAttended course + classroom activities with students
Effects estimations
ITTComparison of sent to treatment
and control group
ATTComparison of actually treated
and control group (Instrumental Variable regression)
The short term effects
1st year for studentsbeginning of 2nd year for teachers
Effects on students math achievment
No effect.
Math achievment scaled to an average of 500 and standard deviation of 100 for the 7th grade.
Descriptive statistics
Effect estimates and
standard errors
(OLS and IV regression)
Treatment Control
ITT ATT
Math score (mean) 493 496 1,8 4,8 (4,8) (12,3)
Effects on students attitudes
Some slight effects:• more interested in maths (1 item out of 4);
• feeling more time costraints;• less frequently attributing academic failure to
chance or to bad luck;• more anxious during the assessment;
• more frequently skipping at least one item during the assessment.
Development of a perfectionist attitude?
Estimated effects on teachersIncrease:
• more exchange with colleagues• more frequent lessons based
on group activities• production of didactic material
Decrease:• use of the school textbook
• mnemonic approach to math learning• perceived self efficacy in
making students work in groups
Further data collection on teachers
May-June 2012
Good reponse rate: 90%Teachers who answered all the surveys: 85%
Survey on•Attitudes
•Instructional practices•Use of Matabel 2 years after of the experiment
Further data collection on students
May 2011-May 2012
National Students Assessment
Measure of math performance
All previous classes (6/7th grade, now 7/8th on 2011, 6th grades now 8th grades on 2012),
but not all students (dropping out, failures)
Further step
Merging data at student individual level,obtaining a panel and assessing the
PON [email protected] effect on the increase in math achievement across school years
Long term effects estimation on teachers
Concluding...
• it is possible to run a RCT evaluation also in the Italian school system;
• first year results do not show any effect of [email protected] on the main outcome...
• ... but we found promising effects on teachers and students attitudes
• second and third year estimation will be crucial to evaluate the effectiveness of the
program
Thank you!
Further steps
…in Year 2 many (56%) of the former controls are treated
Computing the difference TY1&2 - TY2
we will get the effect of one additional year of exposure to the treatment.
If we want the effect of 2 years of exposure to the new method
We have to compute the difference TY1&2 - CY1
The non-testable assumption is that there is no cohort effect.
Additional analysis
1. Effects on the score distribution(quantile regression)
2. Subgroup analyses(effects’ heterogeneity)
Eterogeneous effects by teachers age
But too much uncertainty...
Probability of being full complier
Compliance is associated mostly with individual factors, such as:o Age (50-55 year old, - 16 perc.
points; over-55, - 22 perc. points)o Previous training experience (+11 perc.
points)o Personal motivation to enroll (-25 perc.
points if forced to enroll)
(Binary logistic regression models)
2. Data collection (3rd year)
3. Data collection on a brand new cohort of teachers
Dimensions Value
controls ITT ATT
Attitudes towards math
5 items factor (std score) -0,05 +0,05 +0,12
4 items factor (std score) -0,05 +0,05 +0,12
In math I’m good (1-4 points scale)
+2,78 +0,05** +0,12**
Curriculum pace
We proceeded even if some classmates did not understand the topic (1-4 points scale)
+1,55 +0,07*** +0,17**
Causal Attributions
Attribution of failures to bad luck (0-6 points scale factor)
+0,19 -0,04*** -0,09***
Test Anxiety
4 items factor (std score) -0,04 +0,05* +0,13*
I was so nervous I could not find the answers (1-4 points scale)
+1,89 +0,06*** +0,16***
Effects on students’ attitudes