Major Trends in QLD State School Student Results [Socioeconomic Background and Student Mobility] Dr Roland Simons (DETA) Performance Monitoring and Reporting

Major Trends in QLD State School Student Results[Socioeconomic Background and Student Mobility]

Dr Roland Simons (DETA) Performance Monitoring and Reporting Branch

Purpose

Fundamental / Induction– What measures do we have?– What do International and National results indicate?– How advanced are Qld systems?

Advanced– What are Qld state results indicating?– How relevant is international and national research to Qld?– Evidence Based Practice

• List of related research.• Opportunity to discuss implications,

– [strategic] [policy] [operations]

Outline

• Recent Advances in Measurement in Qld

• Relationships with Student Achievement– Socio Economic Position (SEP)– Mobility

• Questions

Recent advances• This research is only possible because of

recent advances / developments– Unique Student ID– Collection of parental SEP data – linked to student

• Trends– Data integration– Data specificity– Data diversity– New Measures (e.g., Value Add)

This is helping us to• Investigate relevance of international / national research

• Test theoretical propositions / assumptions

• Investigate major trends across and within the state

• Disseminate information / knowledge for Departmental use

• Reaffirm existing practices

• Respond to National agenda

• Develop an “Evidence Base” for practice

Today’s presentation

• Has arisen from specific methodological work that PMRB has been doing and we have extracted the key findings because they also inform us about QLD specific relationships

• Limitations with data presented– It is largely a sample rather than the full population

of students (but the numbers are large enough to warrant attention)

– Typically focusing only on a small segment of time (e.g., 2002 to 2005)

When talking about student performance it is important to note = international and national research to date suggests…..

The single largest Predictor of Student

Performance is…..

Student ability…

Decomposing influences generally has resulted in

Pisg= ƒ{X:S}+ε

E.g., 2005 year 7 numeracyStudent IRSED (5.6%)School IRSED (6.7%)

Pisg= ƒ{X:S}+ε

e.g.,• Resourcing• Teacher Quality• Curriculum• Culture

e.g.,• SEP• Mobility• Family• Isolation

Performance

Unknowns (ε)

Schooling (S)

Characteristics (X)

e.g.,• Ability• Cultural

Pisg= ƒ{X:S}+ε


3 5 7 TestPerformance

Unknowns (ε)

Schooling (S)

Proxy Measures (X)

Socio-Economic Position (SEP)

• Exploratory study using emerging information

• Context, Hypotheses, Sample, Results

• Implications for the future of SEP and achievement measurement and reporting

Introduction

• SEP is one of the most commonly included variables in studies of student achievement

• General consensus is that it is relevant to the experience of education and therefore related to performance.

• Studies both local and international have implicated it with early, middle, tertiary education and beyond.

What is SEP?“the relative position of a family or individual in a social

structure, based on their access to scarce and valued resources such as education, wealth, and prestige” (Marks, McMillan, Jones, & Ainley, 2000, p.9)

• SES – employment status, occupational status, education and income/wealth

• Social class

• Disadvantage

…..of parents/caregivers!!!

Research Advances

• … has matured somewhat from pure methodological debate thanks to international data sources (e.g, PISA, TIMMS).

• ….. countries do differ….. But a relationship remains

• What do these studies tell us?– different levels of effectiveness, and/or– different base levels of equity variation (i.e., learning

contexts)

PISAReading Literacy 2003

PISA

Research Advances• It is likely that the states also differ because of either:

– different levels of effectiveness, and/or– different base levels of equity variation (i.e., learning

contexts)

• Qld is likely to have a particular interest in the relationship…..

• Recent NATSEM paper on child social exclusion is yet another reminder of regional distributions of disadvantage that our education system responds to

Link to Achievement

• Student aspirations, • self-esteem, • attitude to education, • ability/disability, • parental aspirations, • parental attitudes, • home education

resources, • exposure to literature,

• teacher attitudes, • curriculum delivery, • peer group attitudes, • school physical

condition, • class sizes, • quality of teaching and

materials, • community

characteristics…

Direct / Indirect

International, National and State Research

International• Gary Marks, et al.,: 30 countries (2006)

National and State• Ken Rowe: All states (2006)

• Phillip Holmes-Smith: ACT, NSW, QLD, SA, NT (2006)

• Stephen Lamb: VIC (2004)

Studies largely at different levels

• International/National

• State

• School

• Individual Ultimate level of conceptual accuracy

Reporting

High level research established generallyresilient and enduring relationship but not berelevant to specific jurisdictions

Phillip Holmes-SmithHolmes-Smith, P. (2006). School Socio-Economic Density and its Effect on School Performance. Report for the New South Wales Department of Education and Training. School Research Evaluation and Measurement Services, 1-39.

” … while individual characteristics are poor predictors of individual performance, school average student characteristics are very strong predictors of school average performance.”

He later concluded: “..school average student characteristics

(particularly socioeconomic indicators) are very strong predictors of school average performance.”

(2006, p.2).

Individual versus School Level

Achievement

SES

Achievement

SES

School Level Regression (n=1,000+)

Hypothetical Regression Graphs

Individual Level Regression (n=200,000+)

School Averages Exaggerate the Effects of SES!

Achievement

SES

State Level Regression (n=1)

Hypothetical Regression Graph

State Level

Research findings - QLD

SchoolKen Rowe – [~20.8% - 21.6%] – Reading / Maths / Science PISA sample

Phillip Holmes-Smith – [~32.4% - 47.2%]– Literacy / Numeracy QLD

IndividualKen Rowe – [~7.6% - 8.8%] – Reading / Maths / Science PISA sample

Phillip Holmes-Smith – [~2.8% VIC] – Literacy / Numeracy VIC

Our ResearchABS – School IRSED measure

• ABS (SEIFA) – geographically coded based on 2001 census

• Relatively scores (mean 1000, sd 100)

• Qld uses ABS IRSED data (combines income, education, occupation & employment, housing status, English fluency)…. Calculated at CD (collection district level) and then averaged to a best fit approach for postcodes and schools.

Parental Data (each student)

• 2005-2006 introduction of parental data collection, collecting data and building databases continues

• Post code (student IRSED), parental occupation, parental education.

Collection of Parental Education / Occupation

• Now, under National guidelines, state is collecting Parental data

• Occupation– Snr Management– Other business managers– Trades, skilled staff– Operators, assistants, labourers– Not working

• Education– Yr9– Yr10– Yr11– Yr12– Non-school qualification– Cert I to IV– Diploma– Degree or higher

Argument 1

Parental data is more relevant than geographically coded data because of the amount of error implicit in geographically coded SEP data

Therefore should be a better prediction of student achievement than IRSED

Argument 2IRSED data and parental data should start

to approximate one another at higher levels of aggregation (e.g., school level). Assuming patterns of socio-economic mobility would not confound the value of the older census data (2001).

Therefore there should be no added benefit for parental data at the school level over and above school IRSED

Socio-economic Mobility

LowSEP

Region

Scenario 1 – Net increase in SEP, people don’t move

2001 2007

HighSEP

Region

TransformsCENSUS DATAGOES OUT OF

DATE

Scenario 2 – Geographic SEP is same, people move

HighSEP

Region

2001 2007

LowSEP

Region

People Transfer CENSUS DATAREMAINSCURRENT

Argument 3

Social mobility is not high.

Therefore IRSED data based on 2001 will not demonstrate a decline in its relationship to student achievement over time.

Argument 4

The new trend toward growth or gain scores has led some to argue that gain scores effectively account for SEP…

…and therefore student gain scores should be free of SEP effects.

How SEP might be controlled via gain scores

Student A Student B

Same Gain

e.g., Year 3 to Year 5

Stu

den

t A

chie

vem

ent

Yr 5

Yr 3

High SEP Low SEP

BUT, this is also possible

Student A Student B

e.g., Year 3 to Year 5

Stu

den

t A

chie

vem

ent

Yr 5

Yr 3

High SEP Low SEP

Data used

• N=291,705• 2003-2005, Reading, Writing, Numeracy• 48.7% female, 51.2% male• 5.8% Indigenous, 1.1% TSI, 0.6% both• 63.9% metropolitan, 11.0% provincial city,

20.7% rural, 4.4% remote• 1,104 schools (2-1,031 students)• About 34% had parental data…of some sort

(10% could not be categorised leaving 24% with usable parental data)

Parental Occupation n %

Category 1. Senior management in large business organisation, government administration and defense, and qualified professionals 13962 4.80%

Category 2. Other business managers, arts/media/sportspersons and associate professionals 16681 5.70%

Category 3. Tradesmen/women, clerks and skilled office, sales and service staff 18469 6.30%

Category 4. Machine operators, hospitality staff, assistants, labourers and related workers 14570 5.00%

Category 5. Not in paid work in last 12 months 5799 2.00%

Not stated / unknown / missing 222224 76.20%

Total 291705 100.00%

Parental Occupation/Employment

Please note: Highlighted categories were removed from the analyses

Parental EducationParental Education n %

Category 1. Bachelor degree or above 14046 4.80%

Category 2. Advanced diploma/Diploma 9393 3.20%

Category 3. Certificate I to IV (including trade certificate) 23244 8.00%

Category 4. No non-school qualification 187 0.10%

Category 5. Year 12 or equivalent 7343 2.50%



Category 8. Year 9 or equivalent or below 2223 0.80%

Not stated / unknown / missing 223192 76.50%

Total 291705 100.00%

Please note: Highlighted categories were removed from the analyses

IRSED• Student Postcode IRSED. IRSED values

calculated using student postcode collection district averages indicated a range from 472.0 to 1,177.9, with an average of 980.7 and standard deviation of 88.2. Of the sample 274,628 students were allocated an IRSED value based on the postcode listed in their address at the time of their most recent test.

• When categorized into four IRSED groups the sample indicated 93,702 (34.1%) of students were of the high SEP group and 61,559 (22.4%) were of the low SEP group

• School IRSED. IRSED values calculated using school CD coverage averages indicated a range from 472.0 to 1,141.5, with an overall average of 978.8 and standard deviation of 67.7. Of the sample 291,413 students were allocated an IRSED value based on the school that they were attending at the time of their most recent test.

• When categorized into four IRSED groups the sample indicated 60,140 (20.6%) of students were of the high SEP group and 43,655 (15.0%) were of the low SEP group

One measure of disadvantage

ATSI. Indigeneity has been argued to be relevant to SEP as a proxy measure for disadvantage separate from the issue of socioeconomic status.

Outcome measures

• Student achievement scores (means)– Student– School– mid 80% of scores

• Raw gain scores

• Standardised gain scores (mean 0, sd 1)

Initial Results

High Mid-High Mid-Low Low Total% % % % %

High 46.4 38.8 12.2 2.6 100Mid-High 12.1 50.1 30.9 7 100Mid-Low 5.9 35.4 44.7 14 100Low 3.4 18.4 34.6 43.6 100

Total 20.7 36.3 27.9 15 100

N=274,350

School

Student

Spearman rank coefficient

Student IRSED School IRSEDParent

Occupation Parent Education

Student IRSED

School IRSED .365**

Parent Occupation .269** .249**

Parent Education .272** .207** -..625**

ATSI -.161** -.125** -.127** -.133**** = p<0.01

Individual student achievement

• Graphs of relationship

Standardised Student Achievement by School IRSED

400

450

500

550

600

650

700

750

800

850

900

Low Mid-Low Mid-High High

School IRSED

Sta

nd

ard

ised

Stu

den

t A

chie

vem

ent

Sco

res

yr 2003 grade 3 Reading

yr 2003 grade 3 Numeracy

yr 2003 grade 3 Writing
















Caution: Research Sample Averages not full state data (n=68,513)

Standardised Student Achievement by Student IRSED

400

450

500

550

600

650

700

750

800

850

900

Low Mid-Low Mid-High High

Student IRSED

Sta

nd

ard

ised

Stu

den

t A

chie

vem

ent

Sco

res




















Standardised Student Achievement by Parental Education Level

400

450

500

550

600

650

700

750

800

850

900

yr 9 yr 10 yr 11 yr 12 cert I - 4 Diploma Bachelor

Parental Education Level

Sta

nd

ard

ised

Stu

den

t A

chie

vem

ent

grade 3 Reading

grade 3 Numeracy

grade 3 Writing

grade 5 Reading

grade 5 Numeracy

grade 5 Writing

grade 7 Reading

grade 7 Numeracy

grade 7 Writing

grade 3 Reading

grade 3 Numeracy

grade 3 Writing

grade 5 Reading

grade 5 Numeracy

grade 5 Writing

grade 7 Reading

grade 7 Numeracy

grade 7 Writing


Standardised Student Achievement by Parental Occupation

400

450

500

550

600

650

700

750

800

850

900

Category 5 Category 4 Category 3 Category 2 Category 1

Parental Occupation

Sta

nd

ard

ised

Stu

den

t A

chie

vem

ent

Sco

res




















Argument 1Individual – Stepwise regressions

Year Grade Test

Order of entry %

O rder of entry %

O rder of entry %

O rder of entry % Total %

2003 3 Reading 1 7.4 2 1.5 3 0.9 4 0.3 10.1

2003 3 Numeracy 1 6 2 1.6 3 1 4 0.3 8.9

2003 3 Writing 1 4 2 1.1 4 0.2 3 0.8 6.1

2003 5 Reading 1 7.7 2 1.6 4 0.4 3 1 10.7

2003 5 Numeracy 1 6.6 3 1.2 4 0.4 2 1.8 10

2003 5 Writing 1 3.8 2 1 4 0.2 3 0.5 5.5

2005 3 Reading 1 7.3 2 1.7 4 0.4 3 0.9 10.3

2005 3 Numeracy 1 6.5 2 1.9 4 0.3 3 0.9 9.6

2005 3 Writing 1 4.7 2 1.4 4 0.2 3 0.6 6.9

2005 5 Reading 1 7.7 2 1.8 3 1.2 4 0.5 11.2

2005 5 Numeracy 1 7.7 2 2 4 1.2 3 0.5 11.4

2005 5 Writing 1 5.1 3 0.9 4 0.4 2 1.4 7.8

2005 7 Reading 1 8.5 2 1.7 3 1.2 4 0.5 11.9

2005 7 Numeracy 1 7.5 2 1.8 3 1.3 4 0.5 11.1

2005 7 Writing 1 4.5 2 1.1 3 0.6 4 0.3 6.5

Parental Education Parental O ccupation IRSED School IRSED Student

ATSI at step 3


Argument 2School – Stepwise Regressions

Year Grade Test Total

O rder of entry %

O rder of entry %

Order of entry %

O rder of entry % Total %

2003 3 Reading 3 1.5 4 0.4 1 19.5 2 3.3 24.7

2003 3 Numeracy 4 0.3 3 0.9 1 16.7 2 2.3 20.2

2003 3 Writing 3 0.8 4 0.2 1 12.6 2 2.5 16.1

2003 5 Reading 3 0.4 4 1.8 1 21.8 2 4.3 28.3

2003 5 Numeracy 3 1.4 4 0.4 1 18.5 2 4 24.3

2003 5 Writing 3 1.1 4 0.3 1 14 2 2.2 17.6

2005 3 Reading 3 1.5 4 0.4 1 20 2 4.1 26

2005 3 Numeracy 3 1 4 0.4 1 13.7 2 2.7 17.8

2005 3 Writing 3 0.9 4 0.3 1 13.4 2 2.9 17.5

2005 5 Reading 3 1.5 4 0.5 1 23.2 2 4.1 29.3

2005 5 Numeracy 3 1.4 4 0.5 1 20.2 2 4.2 26.3

2005 5 Writing 3 1.1 4 0.3 1 17.3 2 3.6 22.3

2005 7 Reading 3 1.8 4 0.4 1 22.7 2 4 28.9

2005 7 Numeracy 3 1.2 4 0.3 1 18.1 2 3 22.6

2005 7 Writing 3 1.5 4 0.4 1 13.1 2 3.1 18.1

Parental Education Parental O ccupation IRSED School IRSED Student

ATSI at step 3


Argument 3IRSED over time (school level)

2003 2004 2005 Trend% % % over time

3 Numeracy 16.7 14.8 13.7 down3 Writing 13.1 11.7 12.6 u-shaped3 Reading 19.6 18.6 19.1 flat/u-shaped5 Numeracy 18.3 18.5 19.9 up5 Writing 14 13.5 17.3 flat/j-shaped5 Reading 21.5 23.1 22.5 flat/n-shaped7 Numeracy 20 18.6 17.7 down7 Writing 16.5 13.9 13.5 down/l-shaped7 Reading 21.7 22.3 21.6 flat/n-shaped

Grade Test


Argument 4SEP and gain scores

Measure Grade TestO rder of

entry %O rder of

entry %O rder of

entry %O rder of

entry % Total %

Z-score 3 to 5 Reading 2 0.3 3 0.1 1 12.8 4 0 13.1Z-score 3 to 5 Numeracy 2 0.2 3 0 1 12.5 - - 12.7Z-score 3 to 5 Writing 2 0.4 3 0.1 1 10.6 4 0 11.1Z-score 5 to 7 Reading 2 0.3 3 0 1 12.2 - - 12.5Z-score 5 to 7 Numeracy 2 0.1 3 0 1 4.5 - - 4.6Z-score 5 to 7 Writing 2 0.3 3 0.1 1 13.9 4 0 14.3

IRSED School IRSED StudentParental EducationParental

Occupation

ATSI at step 1 for raw scores & step 2 for standardised


Special issues tested

• Mid 80% of student achievement scores tested against the full range of scores - error at the extremes

• Metro only versus all areas – social mobility

• Missing data – no special variation by achievement or regional zones

• Scaling types: Probabilities versus multi-level versus binomial categorisations…..

• All suggest similar profiles

Arguments for SEP• Parental data would be better at predicting

individual student achievement – supported.

• Parental data would not be better at predicting school achievement – supported.

• IRSED does not appear to be influenced by social mobility – supported.

• SEP is not relevant to study of growth/gain scores (value add) – not supported.

Implications• Developmental - Parental data is superior to census data

at the individual level but apart from a conceptually accurate understanding we can’t use this data for individual reporting…. It is useful at an academic level.

• School level + reporting – only marginal incremental value is suggested by parental measures. Trading pragmatics with accuracy with regard to student achievement.

• Earlier criticisms leveled at school based IRSED measures do not seem to hold for school and higher level reporting purposes and that issues of instability such as socio-economic mobility were not suggested by the data (limitation = restricted range).

More Implications

• SEP is almost as relevant to the notion of gain-scores as it is to mean scores.

• National consistency in SEP measurement and usage is going to be important or else we will revert to methodological debates again.

• Correlations for Qld seem to be on par with previously reported research both nationally and internationally.

The path VERY well traveled!

Some would say that we have known this for a long time…. (e.g., 1960s)

Only recently is empirical testing becoming possible on such a large scale.

We are trying many things – collection of parental data has made much of this possible….

Student Mobility

• Similar methodological debates• Similar history of debate• Similar lack of empirical data• EQ ID allows for some tracking• 2003 to 2005• Broad classifications of mobility only• 8% to 12% prediction at student level• Relationship to Student IRSED (-.16**)

Results support literature

For whatever reason, students experiencing high mobility perform less well on tests.

Some reasoning offered in the literature

-Disruption of home environment-Disruption of schooling (e.g., curriculum flow, class time lost)-Disruption of peer groups-Increased stress and anxiety-Change of teacher

200

450

700

950

1200

2005 yr7 Numeracy 2005 yr7 Reading 2005 yr7 Writing

Max

Min

Avg

Mobility

SEP

1sd

1sd

The Best Predictor of Performance is Student Ability

In Perspective

Pisg= ƒ{X:S}+ε

e.g.,• Resourcing• Teacher Quality• Curriculum• Culture


Performance

Unknowns (ε)

Schooling (S)

Characteristics (X)

e.g.,• Ability• Cultural

Correlation is not causation(e.g., SEP could be a proxy for something else)

SEP

Achievement

Mobility

e.g., Health…

Not likely to be that simple

Correlation is not causation(e.g., our measures are only proxy measures for real

phenomena)

SEP

Achievement

Mobility

e.g., Health, Community Service, Housing, Educational Continuity, Parenting, Home Stability, Peer Group, ……

AbsenteeismATSI SEP

And with all the error implicit in measurement we could even doubt this too if the research internationally and nationally were not converging on similar relationships

Questions / Share your views

Documents

Major Trends in QLD State School Student Results [Socioeconomic Background and Student Mobility] Dr Roland Simons (DETA) Performance Monitoring and Reporting