Measures of School Effectiveness

8/8/2019 Measures of School Effectiveness

1/34

Prof Sally Thomas

Value Added Measures of School

Effectiveness:Examples from England of Methods,Uses and Limitations


2/34

Plan

Why is educational quality and school effectiveness important? Methods used to measure school effectiveness and improvement

What are value added measures?

Examples from Lancashire LEA How data and evidence can be used by different stakeholders for

different purposes


3/34

Why is Educational Quality important?

For individuals achieving their own economic, social and cultural objectives

Increased lifetime salary Improved health Longer life

For society lower crime/conflict, promotes responsible, active, productive

citizenship, equity, increased economic growth

Moreover there is increasing influence from international legislation

promoting education quality and the principles of relevance, equity,

rights eg 1990 United Nations Convention on the Rights of the Child; 1990

Education for All; 2000 Millennium Development Goals and Dakar framework.


4/34

Why do we need to evaluate education quality?

Accreditation - to formally regulate desired levels ofquality of educational outcomes and provisions

Accountability - to hold education systemsaccountable for their functioning and performance andsupport democracy in education

Improvement - as a mechanism to stimulateimprovement in education and organisational learning

Also enhanced evaluation processes are required

alongside decentralisation policies

Scheerens, Glas & Thomas (2003)


5/34

Measuring educational quality and

effectiveness some issues to consider:

From whose perspective is quality/effectiveness judged? Which area of activity within an organisation determines quality

/effectiveness?

At what level of the organisation is quality/effectiveness analysed? How is quality/effectiveness defined in terms of time? What data are used to form an opinion of quality/effectiveness? What standards or measures are used in order to make quality

/effectiveness judgements?

(adapted from J. Scheerens, 1992, Effective Schooling: Research, theory

and practice, London: Cassell)


6/34

Defining Educational Quality

depends on the selection of relevant elements, the

assessment of the character of these elements and the

weighting given to their relative importance. The

assessment of quality is thus complex and value laden

OECD: schools and quality an international report (1989) pg 27


7/34

7 Cultural and Political relativity

Effectiveness is not a neutral term. Defining the

effectiveness of a particular school always

requires choices among competing

values.criteria of effectiveness will be the

subject of political debate.

(Firestone 1991:2)(Firestone, W.A. Intro. Chapter 1 in JR Bliss, WA Firestone & CE Richards (eds)Rethinking Effective Schools: Research and practice, Englewood Cliffs, New

Jersey: Prentice Hall)


8/34

Defining Educational Quality focus on student outcomes

Two principles characterize most attempts to define quality in

education:

(1) The first identifies learners cognitive development as the majorexplicit objective of all education systems. Accordingly, the

success with which systems achieve this is one indicator of their

quality.

(2) The second emphasizes educations role in promoting valuesand attitudes of responsible citizenship and in nurturing

creative and emotional development. The achievement of

these objectives is more difficult to assess and compare across

countries.

UNESCO (2004) Education for All (EFA) Global Monitoring Report 2005: The Quality

Imperative. Page 17.


9/34

Every Child Matters Agenda

The Childrens Act 2004requires schools to work withother professionals to ensure 5key outcomes for all children:

Being healthyStaying safeEnjoying and achievingMaking a positivecontribution

Economic well being


10/34

What has research told us?

The central focus of SER indicates a belief in the potency ofsocial institutions

the idea that schools matter, that schools do have major effectsupon childrens development and that, to put it simply, schools

do make a difference(Reynolds & Creemers, 1990, page 1)

SER has presented evidence on:

Size and extent of school effects

Characteristics that promote better student outcomes

Influences of context on outcomes

Processes of institutional change

Long term impact of schools & schooling on life chances

Sammons 2007


11/34

Importance of Student Outcomes

For us the touchstone criteria to be applied to all educationalmatters concern whether children learn more or lessbecause of the policy or practice

Reynolds 1997

An effective school is one in which students progress furtherthan might be expected from consideration of its intake

Mortimore 1991

Thus School Effectiveness Research seeks to identify theValue Added by schools to student outcomes


12/34

Value Added Measures Context in England

From early 1990s political attention and market driven reforms have sought toincrease the accountability of schools in the UK.

Since 1992 schools' raw examination performance has been published annually.These league tables have been widely criticised as unfair to schools withdisadvantaged intakes.

Other reforms in the last 20+ years include a new national inspection system(OFSTED), a national curriculum and national testing via standard assessmenttasks and examinations.

In this context school effectiveness and improvement research has been given afar greater emphasis than it enjoyed previously.

For example, the government has now introduced contextualised value addedmeasures of student progress. Also the idea of intelligent accountability hasbeen introduced via A new Relationship with Schools (DFES, 2004) and thishas resulted in school self-evaluation becoming a key feature of the national

inspection system (Ofsted, 2005).


13/34

Sir Ron Dearings Report toSecretary of State for Education (1993)

Without a value added dimension, the obvious

basis for judgement is that higher scores

represent better practice and lower scoresworse. This could lead to unwarranted

complacency on the part of some schools ... and,

conversely, to despair on the part of others ...


14/34

What Is Meant by Value Added?

The rationale for the value added technique is that:

Raw results describe the grades that students haveobtained

Raw results do not describe how well a school orcollege has performed (eg see Popham, 1999)

Value added results describe how effective a schoolor college is in promoting students' achievement


15/34

Summary of English School System:

Assessment, progression and qualifications of pupils

Promotion to the next year/key stage is automatic and does not depend on theresults of assessment.

There are currently statutory assessment arrangements in the first year ofprimary school and towards the end of Key Stages 1, 2 3 and 4 (at ages, 5, 7,11, 14 and 16 years). These arrangements include teacher assessment and, atthe end of Key Stages 2 and 4, externally set and marked or moderated tests.

QCA serves as regulatory authorities for the qualifications taken at the end ofcompulsory education (Key stage 4). The majority of pupils take GeneralCertificate of Secondary Education examinations (GCSEs) in a range of singlegeneral or vocational subjects. Assessment schemes vary, but always includeexternally set and externally marked assessment; there may also be internallymarked and externally moderated assessment. Assessment may include oral

and practical assessment as well as written examinations.


16/34

What Is Meant by Value Added?

Value added is a measure of the relative progress made by pupils in a school

over a particular period of time (usually from entry to the school until publicexaminations in the case of secondary schools, or over particular years or

curriculum stages in primary schools) in comparison to pupils in others

schools in the same sample.

It compares outcomes after adjusting for varying intake achievementand reflects the relative boost a school gives to a pupils previous levelof attainment in comparison to similar pupils in other schools.

The concept of value added is, therefore, both an indicator of a schoolseffectiveness and a tool for head teachers and their staff to use to

analyse the extent to which they have effectively raised pupil

achievement.

However, it is not a magic wand. It has real limitations, which need tobe well understood.


17/34

Value Added Technique can be applied

across any phase of education

Primary Secondary Post 16

The common aim of these studies is to look at factors outside

the control of the school or college (such as students' prior

attainment, gender, ethnicity and social class) that may have an

impact on assessment or examination results, and where

appropriate, to control for these factors in the analysis.


18/34


19/34

Methodology: Multi-Level Modelling

Methodological advances have facilitated the development of value added measures of

school effectiveness. These include Multi-level modelling techniques which are

considered the most accurate and flexible tools for examining the hierarchical nature of

pupil attainment data (Goldstein, 1995).

In order to describe the complex reality that constitutes educational systems we require

modelling tools that involve a comparable level of complexity' (Goldstein, 1998, p2)

Essentially these methods are a generalised form of multiple regression.

This approach allows the statistical analysis to compare different models and separateout the effect of the school experience on individual pupil outcomes (what pupils

achieve) and the extent to which pupil intake characteristics (prior attainment, socio

-economic background) affect pupil outcomes.

Therefore, accurate baseline information about pupils' prior attainment is crucial tocalculate the value added component.

Value added measures can also be fine-tuned using additional background informationabout pupils, such as their gender, ethnicity and social class.

Using this approach, the residual or value added scorefor each school in the sample can be calculated.


20/34

Multilevel Modeling - Recent Advances

Multilevel Meta-Analysis (Hox & De Leeuw, 2003) enablesbetter estimates of the size and variation in educational effects

for a range of outcomes, phases of education and contexts

Alternative methods of Assessing School Effects Specialissue of School Effectiveness & School Improvement Journal- Sammons & Luyten (2009) Eg regression discontinuity

Multiple membership and cross-classified models, forexample to examine the impact of neighbourhood, family and

school grouping on pupil performance (Leckie, 2009)


21/34

The Lancashire Value Added Project

The Lancashire value added project was set up in 1992 and has involved over

130 secondary schools and over 140,000 pupils.

The project aims to provide an innovative system of school evaluation andself-evaluation via the feedback of student performance, attitude and other

data such as teacher and parent attitudes.

Crucially, the evaluation process is not intended for external accountabilitypurposes, rather a confidential tool for internal accountability and school

improvement.

Since the early beginnings, the project has expanded to incorporate anumber of different types of value added evaluation feedback and themethodology employs state of the art statistical techniques such as

multilevel modelling.


22/34

The Data Employed for Lancashire Analysis (1993-2006)

Outcome measures:

Total GCSE Score (excluding GNVQs)

Total GCSE Score (including GNVQs)

5 Best GCSE score

English

Mathematics

Science

(Calculated using GCSE grades treated numerically, A* = 8, A = 7, B = 6, C = 5,D = 4, E = 3, F = 2, G = 1, Other = 0)

Are used to create a total of 48 separate value added measures including for

each outcome measure:

separate measures for each year and a three year rolling average separate measures for all pupils and pupils grouped according to their previous

attainment in three bands

Additional 28 separate VA measures from 7 Extra Outcomes (from 2004):

IT, Arts, Religious Studies, Modern Foreign Language, Sport/PE Studies, D&Tand Humanities


23/34

The Data Employed for Lancashire Analysis (1993-2006)

Variables controlled for in value added analysis:

Verbal CAT sub-testQuantitative CAT sub-testNon-verbal CAT sub-testAge in monthsMale or FemaleEntitlement to free school mealsMobility - attending more than one secondary schoolEthnicity using DfE categoriesSpecial Education Needs


24/34


25/34


26/34


27/34


28/34

Definitions and measures of school effectiveness as well as the

outcomes of school effectiveness research are dependent on the

following methodological issues:

Sample of schools examined is it representative? Choice of outcome measures Adequate control of differences in school intakes to ensure that 'like is

compared with like'

Methodology (more or less sophisticated) Timescale to take account of stability and consistency in schools' effects

from year to year

Consideration of differential school effectiveness Consideration of pupil mobility and other grouping effects (regions,

neighbourhoods, family)

See Scheerens,1992; Teddlie & Reynolds, 2000; Townsend (Ed), 2007; Sammons, et al. 1995


29/34

Innovation in School EvaluationLancashire VAP: The Case Study School

I think it was really fairly quickly that we began to realise

what a powerful tool that we had got. For first of all, I

suppose selfishly, I was their head and for me as amanagement tool it was the one thing I would never give

up, because it has enabled me to understand so many things

better than I have understood before, and to deal with them,

in a more dynamic and positive way

(Head teacher)

(Thomas & Smees 2003)


30/34

Limitations of Value Added Methodology

Importance of statistical significance and confidence intervalswhen making comparisons between schools and departments.

Data accuracy and measurement error. Validity of student outcome measures. Validity of background factors. (e.g. How well can we control for

factors outside the control of the school, such as additionalprivate tuition?)

Retrospective nature of the data. Finally, are we measuring what we value or valuing what we

measure? . it is important to emphasise that value addedmeasures of student progress are not the only criteria that should

be employed to judge school effectiveness, other more

qualitative measures of school processes (eg quality of teaching)

are also vital for this purpose.


31/34

Uses of Value Added Data

Research

overall trends in pupil and school performance regional/international comparisons case studies of more/less effective schools linking effectiveness measures to improvement

School Self-Evaluation

providing teachers with information that will allow them to reflect on, evaluateand improve their educational practice; departmental and/or teachereffectiveness versus institutional effectiveness and the implications for

institution wide policies; differential effectiveness for different groups of pupils

(eg boys/girls, high/low attainers) and implications for equal opportunities.

External Evaluationproviding school inspectors with information about relative school performance

that can be used alongside their own judgements and other evidence of qualityand standards

Accountabilitypublication of value added school league tablesparental choice


32/34

32 League Tables and Target Setting

The publication of league tables in relation to

examination results, whether national or local, can

improve the effectiveness of an individual school

by stimulating a useful process of analysis and

action which leads to a genuine raising of

educational standards.

Brown (1998:33)


33/34

33 But

Alternatively, pressure to move up the league

tables can trigger teacher stress and short-term

anti-educational solutions, like rejection of poor

attenders or low-attaining pupils by their nearest

schools or changing to an examination board that

sets less challenging questions and gives higher

grades for a given level of performance.

Brown (1998:33)


34/34

Conclusions

The importance of taking account of background factors and prior attainment

using appropriate value added models in order to estimate the influence of theschool is clear. The evidence points to significant differences between

schools effectiveness and the importance of schools utilising value added

information as part of a framework for self-evaluation, alongside other data or

evidence (eg pupil, teacher and parent questionnaires).

However, the evidence also shows that effectiveness is best seen as a feature

that is outcome and time specific. Therefore judgements about schools need to

address at least five key questions:

Effective in promoting which outcomes? Effective over what period of time?

Effective for whom? Effective for which curriculum stage? Effective in what educational policy or regional context?

Documents

Measures of School Effectiveness