Upload
xavier-pereira
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
The Great Divide: fixed vs. random effects in an education context
Claire Crawford with Paul Clarke, Fiona Steele & Anna Vignoles
and funding from ESRC ALSPAC Large Grant
Introduction I
• Our strand concerned with determinants of educational achievement
• Number of substantive research questions:– Impact of SEN– Impact of school size– Joint determination of cognitive and non-cognitive
skills (ECM agenda)
Introduction II
• Thinking about appropriate models– Pupils clustered within schools → hierarchical models
• Two popular choices: fixed and random effects• Which approach is best in which context?
– Idea is always to move closer to a causal interpretation
• Choice of model: – Often driven by discipline tradition– May depend on whether primary interest is pupil or school
characteristics
Outline of talk
• Why SEN?• Fixed and random effects models in the
context of our empirical question• Data and results• Tentative conclusions
Introduction to SEN
• One in four Y6 pupils in England identified as SEN– With statement (more severe): 3.7%– Without statement (less severe): 22.3%
• SEN label means different things in different schools and for different pupils– Maximum means special school or full time teaching
assistant (i.e. additional resources)– Minimum means close monitoring or annual review– Recognition that SEN is not a “treatment”
Why adjust for school effects?
• Want to estimate causal effect of SEN on pupil attainment no matter what school they attend
• Need to adjust for school differences in SEN labelling– e.g. children with moderate difficulties more likely to be
labelled SEN in a high achieving school than in a low achieving school (Keslair et al, 2008; Ofsted, 2004)
– May also be differences due to unobserved factors
• Hierarchical models can account for such differences– Fixed or random school effects?
Fixed effects vs. random effects
• Long debate:– Economists tend to use FE models– Educationalists tend to use RE/multi-level models
• But choice must be context and data specific
Basic model
issisis euXy 10
• FE: us is school dummy variable coefficient
• RE: us is school level residual– Additional assumption required: E [us|Xis] = 0
• That is, no correlation between unobserved school characteristics and observed pupil characteristics
• Both: both models assume: E [eis|Xis] = 0– That is, no correlation between unobserved pupil
characteristics and observed pupil characteristics
Relationship between FE, RE and OLS
issisis euXy 10
)()(1 iisiisiis eeXXyy
)()(1 iisiisiis eeXXyy
FE:
RE:
Where:22 /1
11
euS
How to choose between FE and RE
• Very important to consider sources of bias:– Is RE assumption (i.e. E [us|Xis] = 0) likely to hold?
• Other issues:– Number of clusters– Sample size within clusters– Rich vs. sparse covariates – Whether variation is within or between clusters
• What is the real world consequence of choosing the wrong model?
Sources of selection
• Probability of being SEN may depend on:– Observed school characteristics
• e.g. ability distribution, FSM distribution
– Unobserved school characteristics• e.g. values/motivation of SEN coordinator
– Observed pupil characteristics• e.g. prior ability, FSM status
– Unobserved pupil characteristics• e.g. education values and/or motivation of parents
Intuition I
• If probability of being labelled SEN depends ONLY on observed school characteristics:– e.g. schools with high FSM/low achieving intake
are more or less likely to label a child SEN
• Random effects appropriate as RE assumption holds (i.e. unobserved school effects are not correlated with probability of being SEN)
Intuition 2
• If probability of being labelled SEN also depends on unobserved school characteristics:– e.g. SEN coordinate tries to label as many kids SEN as
possible, because they attract additional resources;• Random effects inappropriate as RE assumption
fails (i.e. unobserved school effects are correlated with probability of being SEN)
• FE accounts for these unobserved school characteristics, so is more appropriate– Identifies impact of SEN on attainment within schools
rather than between schools
Intuition 3
• If probability of being labelled SEN depends on unobserved pupil/parent characteristics: – e.g. some parents may push harder for the label
and accompanying additional resources;– alternatively, some parents may not countenance
the idea of their kid being labelled SEN• Neither FE nor RE will address the
endogeneity problem:– Need to resort to other methods, e.g. IV
Other considerations
• RE model may be favoured over FE where:– Number of clusters is large
• e.g. ALSPAC vs. NPD
– Most variation is between clusters• e.g. UK (between) vs. Sweden (within)
– Have rich covariates
Can tests help?
• Hausman test: – Commonly used to test the RE assumption
• i.e. E [us|Xis] = 0
– But really testing for differences between FE and RE coefficients
• Over-interpretation, as coefficients could be different due to other forms of model misspecification and sample size considerations (Fielding, 2004)
– Test also assumes: E [πis|Xis] = 0
Data
• Avon Longitudinal Study of Parents and Children (ALSPAC)– Recruited pregnant women in Avon with due
dates between April 1991 and December 1992– Followed these mothers and their children over
time, collecting a wealth of information:• Family background (including education, income, etc)• Medical and genetic information• Clinic testing of cognitive and non-cognitive skills• Linked to National Pupil Database
Looking at SEN in ALSPAC
• Why is ALSPAC good for looking at this issue?– Availability of many usually unobserved individual
and school characteristics:• e.g. enjoyment of school, education values of parents,
headteacher tenure
– In particular:• IQ (measured by clinicians)• Good measures of non-cognitive skills (including
behavioural difficulties) reported by parents/teachers
Descriptive statistics• 18% of sample are SEN at age 10
Individual characteristics School characteristics
Standardised KS1 APS -0.104** Independent school -0.102**
IQ (age 8) -0.003** % eligible for FSM -0.002**
SDQ (age 7) 0.012** H’teacher tenure: 1-2 yrs -0.044**
Mum high qual vocational -0.028* H’teacher tenure: 3-9 yrs -0.046**
Mum high qual O-level -0.021 H’teacher tenure: 10+ yrs -0.031
Mum high qual A-level -0.033*
Mum high qual degree -0.019 Observations 5,615Notes: relationship between selected individual and school characteristics and SEN status. Omitted categories are: mum’s highest qualification is CSE level; head teacher tenure < 1 year.
Impact of SEN: full model
• OLS, RE and FE don’t give qualitatively different answers to question of impact of SEN on KS2 APS– Hausman test suggests no difference between FE and RE
OLS FE RESEN -0.514**
[0.046]-0.494** [0.025]
-0.511** [0.025]
Std KS1 APS 0.399** [0.041]
0.445** [0.012]
0.407** [0.011]
Observations 5,615Note: model also controls for vast array of other individual and school characteristics (where appropriate).
Impact of SEN: NPD only
• Again OLS, RE and FE don’t give qualitatively different coefficients on SEN– But global Hausman test suggests FE and RE are NOT equivalent– May be because there is correlation between SEN and unobserved
individual characteristics?• SEN coefficients about 0.1 SDs higher than in full model
OLS FE RE
SEN -0.610** [0.060]
-0.581** [0.026]
-0.597** [0.026]
Std KS1 APS 0.537** [0.044]
0.580** [0.011]
0.560** [0.011]
Observations 5,615Note: model also controls for limited other individual and school characteristics (where appropriate).
Impact of SEN: girls only
• Despite halving sample size, OLS, RE and FE again don’t give qualitatively different coefficients on SEN– But Hausman test suggests FE and RE are NOT equivalent
OLS FE RE
SEN -0.606** [0.066]
-0.624** [0.043]
-0.606** [0.042]
Std KS1 APS 0.377** [0.038]
0.402** [0.017]
0.377** [0.016]
Observations 2,741Note: model also controls for vast array of other individual and school characteristics (where appropriate).
Summary
• SEN appears to be strongly negatively with progress between KS1 and KS2– SEN pupils score around 0.5 SDs lower
• Choice of model does not seem to matter here– OLS, FE and RE all give qualitatively similar results– Suggests correlation between probability of being SEN and
unobserved school characteristics is not important – But doesn’t mean we don’t have to worry about selection
on unobserved individual characteristics
Still to come . . .
• More detailed investigation of conditions under which FE and RE are equivalent– Simulation study
• Do effects of SEN differ across schools?
Tentative conclusions
• Approach each problem with agnostic view on model– Should be determined by theory and data, not tradition
• In reality, choice may not make very much difference– Can our results be generalised?
• Different questions? Different data?
• Worth remembering that neither FE nor RE deals with correlation between observed and unobserved individual characteristics