View
213
Download
1
Category
Preview:
Citation preview
Major Trends in QLD State School Student Results[Socioeconomic Background and Student Mobility]
Dr Roland Simons (DETA) Performance Monitoring and Reporting Branch
Purpose
Fundamental / Induction– What measures do we have?– What do International and National results indicate?– How advanced are Qld systems?
Advanced– What are Qld state results indicating?– How relevant is international and national research to Qld?– Evidence Based Practice
• List of related research.• Opportunity to discuss implications,
– [strategic] [policy] [operations]
Outline
• Recent Advances in Measurement in Qld
• Relationships with Student Achievement– Socio Economic Position (SEP)– Mobility
• Questions
Recent advances• This research is only possible because of
recent advances / developments– Unique Student ID– Collection of parental SEP data – linked to student
• Trends– Data integration– Data specificity– Data diversity– New Measures (e.g., Value Add)
This is helping us to• Investigate relevance of international / national research
• Test theoretical propositions / assumptions
• Investigate major trends across and within the state
• Disseminate information / knowledge for Departmental use
• Reaffirm existing practices
• Respond to National agenda
• Develop an “Evidence Base” for practice
Today’s presentation
• Has arisen from specific methodological work that PMRB has been doing and we have extracted the key findings because they also inform us about QLD specific relationships
• Limitations with data presented– It is largely a sample rather than the full population
of students (but the numbers are large enough to warrant attention)
– Typically focusing only on a small segment of time (e.g., 2002 to 2005)
When talking about student performance it is important to note = international and national research to date suggests…..
The single largest Predictor of Student
Performance is…..
Student ability…
Decomposing influences generally has resulted in
Pisg= ƒ{X:S}+ε
E.g., 2005 year 7 numeracyStudent IRSED (5.6%)School IRSED (6.7%)
Pisg= ƒ{X:S}+ε
e.g.,• Resourcing• Teacher Quality• Curriculum• Culture
e.g.,• SEP• Mobility• Family• Isolation
Performance
Unknowns (ε)
Schooling (S)
Characteristics (X)
e.g.,• Ability• Cultural
Pisg= ƒ{X:S}+ε
e.g.,• SEP• Mobility• Family• Isolation
3 5 7 TestPerformance
Unknowns (ε)
Schooling (S)
Proxy Measures (X)
Socio-Economic Position (SEP)
• Exploratory study using emerging information
• Context, Hypotheses, Sample, Results
• Implications for the future of SEP and achievement measurement and reporting
Introduction
• SEP is one of the most commonly included variables in studies of student achievement
• General consensus is that it is relevant to the experience of education and therefore related to performance.
• Studies both local and international have implicated it with early, middle, tertiary education and beyond.
What is SEP?“the relative position of a family or individual in a social
structure, based on their access to scarce and valued resources such as education, wealth, and prestige” (Marks, McMillan, Jones, & Ainley, 2000, p.9)
• SES – employment status, occupational status, education and income/wealth
• Social class
• Disadvantage
…..of parents/caregivers!!!
Research Advances
• … has matured somewhat from pure methodological debate thanks to international data sources (e.g, PISA, TIMMS).
• ….. countries do differ….. But a relationship remains
• What do these studies tell us?– different levels of effectiveness, and/or– different base levels of equity variation (i.e., learning
contexts)
PISAReading Literacy 2003
PISA
Research Advances• It is likely that the states also differ because of either:
– different levels of effectiveness, and/or– different base levels of equity variation (i.e., learning
contexts)
• Qld is likely to have a particular interest in the relationship…..
• Recent NATSEM paper on child social exclusion is yet another reminder of regional distributions of disadvantage that our education system responds to
Link to Achievement
• Student aspirations, • self-esteem, • attitude to education, • ability/disability, • parental aspirations, • parental attitudes, • home education
resources, • exposure to literature,
• teacher attitudes, • curriculum delivery, • peer group attitudes, • school physical
condition, • class sizes, • quality of teaching and
materials, • community
characteristics…
Direct / Indirect
International, National and State Research
International• Gary Marks, et al.,: 30 countries (2006)
National and State• Ken Rowe: All states (2006)
• Phillip Holmes-Smith: ACT, NSW, QLD, SA, NT (2006)
• Stephen Lamb: VIC (2004)
Studies largely at different levels
• International/National
• State
• School
• Individual Ultimate level of conceptual accuracy
Reporting
High level research established generallyresilient and enduring relationship but not berelevant to specific jurisdictions
Phillip Holmes-SmithHolmes-Smith, P. (2006). School Socio-Economic Density and its Effect on School Performance. Report for the New South Wales Department of Education and Training. School Research Evaluation and Measurement Services, 1-39.
” … while individual characteristics are poor predictors of individual performance, school average student characteristics are very strong predictors of school average performance.”
He later concluded: “..school average student characteristics
(particularly socioeconomic indicators) are very strong predictors of school average performance.”
(2006, p.2).
Individual versus School Level
Achievement
SES
Achievement
SES
School Level Regression (n=1,000+)
Hypothetical Regression Graphs
Individual Level Regression (n=200,000+)
School Averages Exaggerate the Effects of SES!
Achievement
SES
State Level Regression (n=1)
Hypothetical Regression Graph
State Level
Research findings - QLD
SchoolKen Rowe – [~20.8% - 21.6%] – Reading / Maths / Science PISA sample
Phillip Holmes-Smith – [~32.4% - 47.2%]– Literacy / Numeracy QLD
IndividualKen Rowe – [~7.6% - 8.8%] – Reading / Maths / Science PISA sample
Phillip Holmes-Smith – [~2.8% VIC] – Literacy / Numeracy VIC
Our ResearchABS – School IRSED measure
• ABS (SEIFA) – geographically coded based on 2001 census
• Relatively scores (mean 1000, sd 100)
• Qld uses ABS IRSED data (combines income, education, occupation & employment, housing status, English fluency)…. Calculated at CD (collection district level) and then averaged to a best fit approach for postcodes and schools.
Parental Data (each student)
• 2005-2006 introduction of parental data collection, collecting data and building databases continues
• Post code (student IRSED), parental occupation, parental education.
Collection of Parental Education / Occupation
• Now, under National guidelines, state is collecting Parental data
• Occupation– Snr Management– Other business managers– Trades, skilled staff– Operators, assistants, labourers– Not working
• Education– Yr9– Yr10– Yr11– Yr12– Non-school qualification– Cert I to IV– Diploma– Degree or higher
Argument 1
Parental data is more relevant than geographically coded data because of the amount of error implicit in geographically coded SEP data
Therefore should be a better prediction of student achievement than IRSED
Argument 2IRSED data and parental data should start
to approximate one another at higher levels of aggregation (e.g., school level). Assuming patterns of socio-economic mobility would not confound the value of the older census data (2001).
Therefore there should be no added benefit for parental data at the school level over and above school IRSED
Socio-economic Mobility
LowSEP
Region
Scenario 1 – Net increase in SEP, people don’t move
2001 2007
HighSEP
Region
TransformsCENSUS DATAGOES OUT OF
DATE
Scenario 2 – Geographic SEP is same, people move
HighSEP
Region
2001 2007
LowSEP
Region
People Transfer CENSUS DATAREMAINSCURRENT
Argument 3
Social mobility is not high.
Therefore IRSED data based on 2001 will not demonstrate a decline in its relationship to student achievement over time.
Argument 4
The new trend toward growth or gain scores has led some to argue that gain scores effectively account for SEP…
…and therefore student gain scores should be free of SEP effects.
How SEP might be controlled via gain scores
Student A Student B
Same Gain
e.g., Year 3 to Year 5
Stu
den
t A
chie
vem
ent
Yr 5
Yr 3
High SEP Low SEP
BUT, this is also possible
Student A Student B
e.g., Year 3 to Year 5
Stu
den
t A
chie
vem
ent
Yr 5
Yr 3
High SEP Low SEP
Data used
• N=291,705• 2003-2005, Reading, Writing, Numeracy• 48.7% female, 51.2% male• 5.8% Indigenous, 1.1% TSI, 0.6% both• 63.9% metropolitan, 11.0% provincial city,
20.7% rural, 4.4% remote• 1,104 schools (2-1,031 students)• About 34% had parental data…of some sort
(10% could not be categorised leaving 24% with usable parental data)
Parental Occupation n %
Category 1. Senior management in large business organisation, government administration and defense, and qualified professionals 13962 4.80%
Category 2. Other business managers, arts/media/sportspersons and associate professionals 16681 5.70%
Category 3. Tradesmen/women, clerks and skilled office, sales and service staff 18469 6.30%
Category 4. Machine operators, hospitality staff, assistants, labourers and related workers 14570 5.00%
Category 5. Not in paid work in last 12 months 5799 2.00%
Not stated / unknown / missing 222224 76.20%
Total 291705 100.00%
Parental Occupation/Employment
Please note: Highlighted categories were removed from the analyses
Parental EducationParental Education n %
Category 1. Bachelor degree or above 14046 4.80%
Category 2. Advanced diploma/Diploma 9393 3.20%
Category 3. Certificate I to IV (including trade certificate) 23244 8.00%
Category 4. No non-school qualification 187 0.10%
Category 5. Year 12 or equivalent 7343 2.50%
Category 6. Year 11 or equivalent 2944 1.00%
Category 7. Year 10 or equivalent 9133 3.10%
Category 8. Year 9 or equivalent or below 2223 0.80%
Not stated / unknown / missing 223192 76.50%
Total 291705 100.00%
Please note: Highlighted categories were removed from the analyses
IRSED• Student Postcode IRSED. IRSED values
calculated using student postcode collection district averages indicated a range from 472.0 to 1,177.9, with an average of 980.7 and standard deviation of 88.2. Of the sample 274,628 students were allocated an IRSED value based on the postcode listed in their address at the time of their most recent test.
• When categorized into four IRSED groups the sample indicated 93,702 (34.1%) of students were of the high SEP group and 61,559 (22.4%) were of the low SEP group
• School IRSED. IRSED values calculated using school CD coverage averages indicated a range from 472.0 to 1,141.5, with an overall average of 978.8 and standard deviation of 67.7. Of the sample 291,413 students were allocated an IRSED value based on the school that they were attending at the time of their most recent test.
• When categorized into four IRSED groups the sample indicated 60,140 (20.6%) of students were of the high SEP group and 43,655 (15.0%) were of the low SEP group
One measure of disadvantage
ATSI. Indigeneity has been argued to be relevant to SEP as a proxy measure for disadvantage separate from the issue of socioeconomic status.
Outcome measures
• Student achievement scores (means)– Student– School– mid 80% of scores
• Raw gain scores
• Standardised gain scores (mean 0, sd 1)
Initial Results
High Mid-High Mid-Low Low Total% % % % %
High 46.4 38.8 12.2 2.6 100Mid-High 12.1 50.1 30.9 7 100Mid-Low 5.9 35.4 44.7 14 100Low 3.4 18.4 34.6 43.6 100
Total 20.7 36.3 27.9 15 100
N=274,350
School
Student
Spearman rank coefficient
Student IRSED School IRSEDParent
Occupation Parent Education
Student IRSED
School IRSED .365**
Parent Occupation .269** .249**
Parent Education .272** .207** -..625**
ATSI -.161** -.125** -.127** -.133**** = p<0.01
Individual student achievement
• Graphs of relationship
Standardised Student Achievement by School IRSED
400
450
500
550
600
650
700
750
800
850
900
Low Mid-Low Mid-High High
School IRSED
Sta
nd
ard
ised
Stu
den
t A
chie
vem
ent
Sco
res
yr 2003 grade 3 Reading
yr 2003 grade 3 Numeracy
yr 2003 grade 3 Writing
yr 2003 grade 5 Reading
yr 2003 grade 5 Numeracy
yr 2003 grade 5 Writing
yr 2003 grade 7 Reading
yr 2003 grade 7 Numeracy
yr 2003 grade 7 Writing
yr 2005 grade 3 Reading
yr 2005 grade 3 Numeracy
yr 2005 grade 3 Writing
yr 2005 grade 5 Reading
yr 2005 grade 5 Numeracy
yr 2005 grade 5 Writing
yr 2005 grade 7 Reading
yr 2005 grade 7 Numeracy
yr 2005 grade 7 Writing
Caution: Research Sample Averages not full state data (n=68,513)
Standardised Student Achievement by Student IRSED
400
450
500
550
600
650
700
750
800
850
900
Low Mid-Low Mid-High High
Student IRSED
Sta
nd
ard
ised
Stu
den
t A
chie
vem
ent
Sco
res
yr 2003 grade 3 Reading
yr 2003 grade 3 Numeracy
yr 2003 grade 3 Writing
yr 2003 grade 5 Reading
yr 2003 grade 5 Numeracy
yr 2003 grade 5 Writing
yr 2003 grade 7 Reading
yr 2003 grade 7 Numeracy
yr 2003 grade 7 Writing
yr 2005 grade 3 Reading
yr 2005 grade 3 Numeracy
yr 2005 grade 3 Writing
yr 2005 grade 5 Reading
yr 2005 grade 5 Numeracy
yr 2005 grade 5 Writing
yr 2005 grade 7 Reading
yr 2005 grade 7 Numeracy
yr 2005 grade 7 Writing
Caution: Research Sample Averages not full state data (n=68,513)
Standardised Student Achievement by Parental Education Level
400
450
500
550
600
650
700
750
800
850
900
yr 9 yr 10 yr 11 yr 12 cert I - 4 Diploma Bachelor
Parental Education Level
Sta
nd
ard
ised
Stu
den
t A
chie
vem
ent
grade 3 Reading
grade 3 Numeracy
grade 3 Writing
grade 5 Reading
grade 5 Numeracy
grade 5 Writing
grade 7 Reading
grade 7 Numeracy
grade 7 Writing
grade 3 Reading
grade 3 Numeracy
grade 3 Writing
grade 5 Reading
grade 5 Numeracy
grade 5 Writing
grade 7 Reading
grade 7 Numeracy
grade 7 Writing
Caution: Research Sample Averages not full state data (n=68,513)
Standardised Student Achievement by Parental Occupation
400
450
500
550
600
650
700
750
800
850
900
Category 5 Category 4 Category 3 Category 2 Category 1
Parental Occupation
Sta
nd
ard
ised
Stu
den
t A
chie
vem
ent
Sco
res
yr 2003 grade 3 Reading
yr 2003 grade 3 Numeracy
yr 2003 grade 3 Writing
yr 2003 grade 5 Reading
yr 2003 grade 5 Numeracy
yr 2003 grade 5 Writing
yr 2003 grade 7 Reading
yr 2003 grade 7 Numeracy
yr 2003 grade 7 Writing
yr 2005 grade 3 Reading
yr 2005 grade 3 Numeracy
yr 2005 grade 3 Writing
yr 2005 grade 5 Reading
yr 2005 grade 5 Numeracy
yr 2005 grade 5 Writing
yr 2005 grade 7 Reading
yr 2005 grade 7 Numeracy
yr 2005 grade 7 Writing
Caution: Research Sample Averages not full state data (n=68,513)
Argument 1Individual – Stepwise regressions
Year Grade Test
Order of entry %
O rder of entry %
O rder of entry %
O rder of entry % Total %
2003 3 Reading 1 7.4 2 1.5 3 0.9 4 0.3 10.1
2003 3 Numeracy 1 6 2 1.6 3 1 4 0.3 8.9
2003 3 Writing 1 4 2 1.1 4 0.2 3 0.8 6.1
2003 5 Reading 1 7.7 2 1.6 4 0.4 3 1 10.7
2003 5 Numeracy 1 6.6 3 1.2 4 0.4 2 1.8 10
2003 5 Writing 1 3.8 2 1 4 0.2 3 0.5 5.5
2005 3 Reading 1 7.3 2 1.7 4 0.4 3 0.9 10.3
2005 3 Numeracy 1 6.5 2 1.9 4 0.3 3 0.9 9.6
2005 3 Writing 1 4.7 2 1.4 4 0.2 3 0.6 6.9
2005 5 Reading 1 7.7 2 1.8 3 1.2 4 0.5 11.2
2005 5 Numeracy 1 7.7 2 2 4 1.2 3 0.5 11.4
2005 5 Writing 1 5.1 3 0.9 4 0.4 2 1.4 7.8
2005 7 Reading 1 8.5 2 1.7 3 1.2 4 0.5 11.9
2005 7 Numeracy 1 7.5 2 1.8 3 1.3 4 0.5 11.1
2005 7 Writing 1 4.5 2 1.1 3 0.6 4 0.3 6.5
Parental Education Parental O ccupation IRSED School IRSED Student
ATSI at step 3
Caution: Research Sample Averages not full state data (n=68,513)
Argument 2School – Stepwise Regressions
Year Grade Test Total
O rder of entry %
O rder of entry %
Order of entry %
O rder of entry % Total %
2003 3 Reading 3 1.5 4 0.4 1 19.5 2 3.3 24.7
2003 3 Numeracy 4 0.3 3 0.9 1 16.7 2 2.3 20.2
2003 3 Writing 3 0.8 4 0.2 1 12.6 2 2.5 16.1
2003 5 Reading 3 0.4 4 1.8 1 21.8 2 4.3 28.3
2003 5 Numeracy 3 1.4 4 0.4 1 18.5 2 4 24.3
2003 5 Writing 3 1.1 4 0.3 1 14 2 2.2 17.6
2005 3 Reading 3 1.5 4 0.4 1 20 2 4.1 26
2005 3 Numeracy 3 1 4 0.4 1 13.7 2 2.7 17.8
2005 3 Writing 3 0.9 4 0.3 1 13.4 2 2.9 17.5
2005 5 Reading 3 1.5 4 0.5 1 23.2 2 4.1 29.3
2005 5 Numeracy 3 1.4 4 0.5 1 20.2 2 4.2 26.3
2005 5 Writing 3 1.1 4 0.3 1 17.3 2 3.6 22.3
2005 7 Reading 3 1.8 4 0.4 1 22.7 2 4 28.9
2005 7 Numeracy 3 1.2 4 0.3 1 18.1 2 3 22.6
2005 7 Writing 3 1.5 4 0.4 1 13.1 2 3.1 18.1
Parental Education Parental O ccupation IRSED School IRSED Student
ATSI at step 3
Caution: Research Sample Averages not full state data (n=68,513)
Argument 3IRSED over time (school level)
2003 2004 2005 Trend% % % over time
3 Numeracy 16.7 14.8 13.7 down3 Writing 13.1 11.7 12.6 u-shaped3 Reading 19.6 18.6 19.1 flat/u-shaped5 Numeracy 18.3 18.5 19.9 up5 Writing 14 13.5 17.3 flat/j-shaped5 Reading 21.5 23.1 22.5 flat/n-shaped7 Numeracy 20 18.6 17.7 down7 Writing 16.5 13.9 13.5 down/l-shaped7 Reading 21.7 22.3 21.6 flat/n-shaped
Grade Test
Caution: Research Sample Averages not full state data (n=68,513)
Argument 4SEP and gain scores
Measure Grade TestO rder of
entry %O rder of
entry %O rder of
entry %O rder of
entry % Total %
Z-score 3 to 5 Reading 2 0.3 3 0.1 1 12.8 4 0 13.1Z-score 3 to 5 Numeracy 2 0.2 3 0 1 12.5 - - 12.7Z-score 3 to 5 Writing 2 0.4 3 0.1 1 10.6 4 0 11.1Z-score 5 to 7 Reading 2 0.3 3 0 1 12.2 - - 12.5Z-score 5 to 7 Numeracy 2 0.1 3 0 1 4.5 - - 4.6Z-score 5 to 7 Writing 2 0.3 3 0.1 1 13.9 4 0 14.3
IRSED School IRSED StudentParental EducationParental
Occupation
ATSI at step 1 for raw scores & step 2 for standardised
Caution: Research Sample Averages not full state data (n=68,513)
Special issues tested
• Mid 80% of student achievement scores tested against the full range of scores - error at the extremes
• Metro only versus all areas – social mobility
• Missing data – no special variation by achievement or regional zones
• Scaling types: Probabilities versus multi-level versus binomial categorisations…..
• All suggest similar profiles
Arguments for SEP• Parental data would be better at predicting
individual student achievement – supported.
• Parental data would not be better at predicting school achievement – supported.
• IRSED does not appear to be influenced by social mobility – supported.
• SEP is not relevant to study of growth/gain scores (value add) – not supported.
Implications• Developmental - Parental data is superior to census data
at the individual level but apart from a conceptually accurate understanding we can’t use this data for individual reporting…. It is useful at an academic level.
• School level + reporting – only marginal incremental value is suggested by parental measures. Trading pragmatics with accuracy with regard to student achievement.
• Earlier criticisms leveled at school based IRSED measures do not seem to hold for school and higher level reporting purposes and that issues of instability such as socio-economic mobility were not suggested by the data (limitation = restricted range).
More Implications
• SEP is almost as relevant to the notion of gain-scores as it is to mean scores.
• National consistency in SEP measurement and usage is going to be important or else we will revert to methodological debates again.
• Correlations for Qld seem to be on par with previously reported research both nationally and internationally.
The path VERY well traveled!
Some would say that we have known this for a long time…. (e.g., 1960s)
Only recently is empirical testing becoming possible on such a large scale.
We are trying many things – collection of parental data has made much of this possible….
Student Mobility
• Similar methodological debates• Similar history of debate• Similar lack of empirical data• EQ ID allows for some tracking• 2003 to 2005• Broad classifications of mobility only• 8% to 12% prediction at student level• Relationship to Student IRSED (-.16**)
Results support literature
For whatever reason, students experiencing high mobility perform less well on tests.
Some reasoning offered in the literature
-Disruption of home environment-Disruption of schooling (e.g., curriculum flow, class time lost)-Disruption of peer groups-Increased stress and anxiety-Change of teacher
200
450
700
950
1200
2005 yr7 Numeracy 2005 yr7 Reading 2005 yr7 Writing
Max
Min
Avg
Mobility
SEP
1sd
1sd
The Best Predictor of Performance is Student Ability
In Perspective
Pisg= ƒ{X:S}+ε
e.g.,• Resourcing• Teacher Quality• Curriculum• Culture
e.g.,• SEP• Mobility• Family• Isolation
Performance
Unknowns (ε)
Schooling (S)
Characteristics (X)
e.g.,• Ability• Cultural
Correlation is not causation(e.g., SEP could be a proxy for something else)
SEP
Achievement
Mobility
e.g., Health…
Not likely to be that simple
Correlation is not causation(e.g., our measures are only proxy measures for real
phenomena)
SEP
Achievement
Mobility
e.g., Health, Community Service, Housing, Educational Continuity, Parenting, Home Stability, Peer Group, ……
AbsenteeismATSI SEP
And with all the error implicit in measurement we could even doubt this too if the research internationally and nationally were not converging on similar relationships
Questions / Share your views
Recommended