Performance Reporting in Education · Performance Reporting in Education iii Table of contents Executive summary iv Highlights of the 2012 Churchill study tour iv Major learnings

2012 Churchill Fellowship report Michelle Wheeler

Performance Reporting in EducationWinston Churchill Memorial Trust of Australia

Report by Michelle Wheeler

2012 Churchill Fellow

Performance Reporting in Education ii

International approaches to comparative performance reporting in education

I understand that the Churchill Trust may publish this Report, either in hard copy or on theinternet or both, and consent to such publication.

I indemnify the Churchill Trust against any loss, costs or damages it may suffer arising out ofany claim or proceedings made against the Trust in respect of or arising out of the publication ofany Report submitted to the Trust and which the Trust places on a website for access over theinternet.

I also warrant that my Final Report is original and does not infringe the copyright of any person,or contain anything which is, or the incorporation of which into the Final Report is actionablefor defamation, a breach of any privacy law or obligation, breach of confidence, contempt ofcourt, passing-off or contravention of any other private right or of any law.

Michelle Wheeler

15 February 2013

Contact details

Performance Reporting in Education iii

Table of contentsExecutive summary iv

Highlights of the 2012 Churchill study tour ivMajor learnings iv

Overview vBackground vSummary of key learnings vRecommendations vii

1. Program and acknowledgements 11.1 Theoretical framework for the study 11.2 Program 11.3 Acknowledgements 4

2. Performance reporting and accountability in England 52.1 The government policy framework 62.2 School providers 82.3 Commentators on the system of performance accountability 14

3. Comparing performance across countries 193.1 Performance reporting for OECD countries 193.2 Performance reporting for the European Union 263.3 Performance reporting for developing countries 29

4 Challenges of indicator and data development 324.1 Performance reporting for the UK Department for Education 324.2 International comparisons 33

Conclusions and recommendations 36Conclusions 36Recommendations 37

References and further reading 38

Appendix A. Logic diagram 40

Michelle Wheeler 2012 Churchill FellowDirector, Education, COAG Reform CouncilTel: 02 8229 7380 Report February 2013

Executive summaryPerformance Reporting in Education iv

Executive summaryDuring the last two decades, governments around the world have implemented systems formeasuring and reporting publicly on outcomes, as a way of improving services. The purpose ofthis research is to investigate the connection between performance reporting and improvingschooling outcomes and to identify the challenges of complex reporting environments. Thefocus is on comparative performance reporting in multi-jurisdictional contexts.

Highlights of the 2012 Churchill study tour Discussing with world-leading researchers the diverse work of the Organisation for

Economic and Cultural Development in performance reporting and education policy.

Meeting people from across the wide range of stakeholders engaged in school education inEngland—schools, education providers, government bodies, academics and think tanks.

Visiting schools in England and appreciating the singular focus on improving outcomes forstudents and the role that performance information plays in this.

Learning how analysts in the European Union organisations address the measurement issuesacross very diverse member states.

Major learnings

The learnings from this research are two-fold: identification of similarities and differences withthe Australian approach, and identification of emerging trends for further exploration andconsideration for framing improvements in Australia.

There is a mixture of competition and collaboration in approaches to striving for highperforming schools, systems or countries.

- Competition comes about through the public reporting of schooling outcomes and thecomparisons made possible from a unified approach to measurement, along withsystems for describing or ranking varying levels of achievement.

- Collaborative approaches are generally taken to identifying ‘what works’ to deliver highperformance. This is a necessary approach where individual jurisdictions are responsibleand accountable for education delivery and outcomes.

Transparency in reporting by governments is achieved through the widespread availabilityof data on education. Tools for analyzing this data are an important feature where there areadvanced systems of performance reporting.

Increased availability and use of comparative data has led to more evidence-basedapproaches to reforms aimed at driving improvement for all students. However, data byitself cannot deliver improvements, it must be supported by widespread implementation ofevidence-based best practice.

Participation in relevant forums will provide opportunities to discuss the research during 2013.Importantly, major benefits will be delivered through the daily application of the experiencesand sources of information to the performance reporting role I perform here in Australia.

Michelle Wheeler 2012 Churchill Fellow Report February 2013

OverviewPerformance Reporting in Education

OverviewBackground

Performance reporting in education has served many diverse purposes over the decades.Accountability to the public is one of its purposes and is not a new phenomenon. QueenVictoria established the concept of external inspections of schools for 'public accountability forpublic funds' and to ensure 'the safety of children'. (Reported by Ceri Morgan, Her Majesty’sInspector, Ofsted).

Of recent time, there has been growing interest in performance reporting as a management tool.Performance-based accountably systems link financial or other incentives to measuredperformance, as a means of improving services to the public (Stecher, B., 2010). In education,this improvement is largely measured by improvements in outcomes for students, for schools,for particular groups in society and for a nation as a whole. Improved educational outcomes arefurther linked to overall economic and social prosperity of a nation and global rankings.

Australia’s public performance reporting, a key element of Council of Australian Governments(COAG) reform agenda, is promoted as part of an overall performance management system thatfocuses both on public accountability and driving improvements in outcomes. Can it achieveboth? Does performance reporting actually lead to improvements in schooling outcomes? Thatis the question this research is addressing. By looking outside Australia, at systems in othercountries and at the work of organisations using the comparative model to drive improvement, Iwas seeking insights into the uses and limitations of performance reporting as a vehicle to driveimprovement in schooling outcomes.

My study tour comprised visits to institutions and individuals in England, France, Belgium andLuxembourg during October and November 2012. The full program is in Chapter 2.

The following report is structured around three elements:

1. School level improvement and accountability in England

2. Comparisons of schooling outcomes across countries

3. Measuring performance—the challenges of indicator development and measurement

Summary of key learnings

On school improvement and accountability

As Australia moves towards mandating school improvement plans tied to its funding regime, itis timely to reflect on the English experience with performance reporting, accountability andschool improvement.

The English accountability framework is complex and dynamic. It appears complexity anddiversity dominate over simplicity and allowing schools and systems to ‘bed down’ reforms.



The existence of an independent external inspection system is fundamental to leveragingmarket pressures and competition between schools, driving school improvement. Thisfeature is not part of the school education system in Australia, although forms of externalinspection exist in most states.

Collaboration and competition can co-exist in devolved systems with strong accountabilityframeworks, as long as sufficient support and incentives are provided for collaboration.

There is substantial evidence to support school autonomy as a factor in drivingimprovement in outcomes. Governance by boards of local citizens is fundamental to the UKschool reforms, and there seems to be less scrutiny of the effectiveness of non-executiveboards and their connection with school effectiveness.

Opportunities for ‘gaming’ the system need to be guarded against, both in design andreview of the accountability system.

Healthy and critical debate can be fostered through the marshalling of the resources(intellectual and creative) of research bodies, academics, peak bodies and other stakeholdersusing new media genres and methods of consultation.

On comparative performance reporting across countries

Australia can learn from the experience of comparative reporting from organisations workingwith independent jurisdictions responsible for education and training delivery and outcomes.The Australian federation of states and territories provides an in-country comparativeframework, and, like independent nations, the states are responsible and accountable fordelivery and outcomes in education.

In general, measurement issues can be overcome and the driving principle for indicatorselection and reporting at system level should be its capacity to impact on outcomes.

An efficiency agenda is now part of international comparisons. A sole focus on outcomesignores a key element of policy reform and progress. Efficiency is the missing link inAustralia’s high-level performance reporting framework through COAG.

There is an overall harmonisation of policy approaches to educational accountability andimprovement, at the international level. At the same time, there are fundamental differencesbetween countries. For example, the balance of power between central and local authoritiesvaries across countries, and only a few countries attach high stakes to the results of externalassessments, for example the threat of shutdown or reconstitution for underperformingschools (Canada, the United Kingdom and the United States). The majority of OECDcountries publish assessment and/or evaluation results, for the purposes of transparency andto drive improvement.

Targets and benchmarks are not universally the mechanism for improvement. Reportingagainst high level targets across developing countries has shown disappointing results.Targets and benchmarks are best renewed at frequent intervals, achieving renewedcommitment to their purpose. For example, the European Commission renewed five of itsbenchmarks for 2020, with more ambitious country-specific targets for headline indicators.



On indicator and data development

Organisations engaged in performance reporting experience similar challenges with indicatorand data development issues as experienced in Australia.

There should not be an over concern with the technical issues surrounding the reporting ofsurvey data and associated error. As data collections improve, the capacity to makedefinitive findings will improve.

Investment in systems to manage data remains a priority for most governments and willsupport accountability, improvement and timeliness issues.

Measuring progress in education sits within a wider agenda of measuring well-being andsocietal progress. These approaches are now at the forefront of national and internationalstatistical and political agendas and it is our interests to continue to participate in thesedevelopments.

Recommendations

Australian agencies and individuals should continue to monitor and engage in collaborativeprojects with overseas-based performance and accountability bodies to ensure the latest researchand experiences inform our approach.

In particular, there should be action in three critical areas.

Establishing a framework for the ongoing review of indicators and their further developmentdriven by a principle of relevance to 21st century learning environments and what we knowabout the nature of learning.

Investing in technology and systems to improve our capacity for timely performancereporting and efficient management and development of key collections.

Developing more systematic approaches to collaboration to increase the uptake of bestpractice and approaches to lifting performance at school and system level.


Performance Reporting in Education 1

1. Program and acknowledgementsThe itinerary for the study tour was designed to cover major aspects of performanceaccountability systems in countries and agencies with significant experience in comparativeperformance reporting in education. I was interested in exploring issues with both the designersand implementers of systems and seeing the impact in terms of policy implementation inschools and on outcomes.

1.1 Theoretical framework for the studyThe theoretical framework for the research in this study is set out in a logic diagram atAppendix A. The framework was a useful ‘tool’ to guide my discussion with researchers,administrators, school personnel and analysts. All those interviewed used education measures orindicators in some way:

for the purposes of public accountability

to improve the way we measure and what we measure so it has a closer link to desiredoutcomes,

to inform learning processes

to drive improvement processes.

All of those interviewed were dedicated to the goal of better outcomes for individuals, forschools and or society. Each of the interviewees could identify and describe their use ofperformance measures and how their roles contributed to the shared goal of ‘better outcomes’.

The itinerary was divided into three distinct segments:

School level accountability and improvement in England—from visits to governmentpolicy and regulatory bodies, through to third party change agents, school providers,schools, think tanks, researchers and academics.

Comparisons across countries of schooling outcomes—OECD, UNESCO, EuropeanCommission

Measuring performance—challenges of indicator development and measurement—UKDepartment for Education, Ofsted, OECD, Eurostat.

The following report is structured around these three elements.

1.2 Program



Dates Organisation Name and position

22 – 26

October

UK Department forEducation

Matthew Young Deputy Director for Assessment andAccountability Division

DfE Paul Roberts Manager, School Performance Data

DfE Neil Lawson Manager, Performance Tables

DfE Benedict Coffin Assessment Division

DfE Audrey Brown Deputy Director, EducationStandards and Research

Consultant to DfE Craig Lawson Consultant, RAISEonline

Ofsted Ceri Morgan Her Majesty’s Inspector

The Sutton Trust Dr Lee Elliot Major Director, Development and Policy

Royal Society of Arts Alison Critchley Executive Director, RSA Academies

CfBT Education Trust Tony McAleavy Director, Education

29 Oct –

1 Nov

Kings College London Dr Becky Francis Professor of Education and SocialJustice

City of London School Noeleen Murphy Director of Studies

Institute of Education

University of London

Prof ChrisHusbands

Director and member Commission onAcademies

Innovation Unit David Albury Director

12-16November

Australian delegation to the

OECD

Margaret Pearce

Mark Unwin

Minister-Counsellor, Education

Adviser, Education and Employment

OECD AndreasSchleicher

Deputy Director and Special Advisoron Education Policy to SecretaryGeneral

OECD Richard Yelland Head of Division, Policy Advice andImplementation

OECD, Statistics Marco MiraD’Ecole

Head of Division, ProgressMeasurement

UNESCO Manos Antoninis Senior Policy Analyst, EFAmonitoring report

OECD Michael Davidson Head of Division, EducationDirectorate

OECD Dirk Van Damme Head of Division, Centre forEducational Research and Innovation(CERI)



Dates Organisation Name and position

OECD Paulo SantiagoClare Shewbridge

Senior Analysts, Early Childhood andSchools Division

OECD David InstanceMarco Kools

Centre for Educational Research andInnovation

OECD Beatriz Pont Senior Analyst, PolicyImplementation Division

International Institute ofEducational Planning

Suzanne GrantLewis

Deputy Director

IIEP Demus Makuwa Head of SACMEQ

IIEP Muriel Poisson Program specialist

IIEPKahlil Mahshi

Director

19 -23November

Australian-European Forumon Education and Training

Shelagh Wittleston Education Manager, AustralianEmbassy, AEI International

Flemish Ministry ofEducation

Gaby Hostens (former) Director-General, EducationDepartment, Flemish Community ofBelgium

European Commission Paul Holdsworth, Head of Sector, School PolicyCooperation

European Commission Dr Graham Wilkie Policy officer,Education and Culture

Eurostat Sylvain Jouhette, Education and Health, lifelongsurveys

Eurostat Ana MariaMartinez Palou,

Education and Health, administrativecollections

26-29

November

Achievement for All Karen Iles Senior Regional Lead (South ofEngland

Haven School, Sussex Sandra Trathen Deputy Head Teacher

King Solomon AcademySchool

Venessa Willems Head teacher – primary

Lilian Bayliss TechnologySchool

Gary Phillips Head teacher

Cambridge University Prof Peter Gronn Head of Faculty of Education

RAND Europe Michael Frearson Research Leader Education Skills



1.3 Acknowledgements

This overseas study and report have been made possible by the generosity of the WinstonChurchill Memorial Trust in Australia. I would like to thank the National Office for theirongoing support and advice and the NSW committee for recommending me and supporting myapplication so enthusiastically.

I would like to acknowledge and thank all the educational leaders I met and their organisationsfor the generous spirit in which they shared their time, their research and their expertise tosupport my study of the systems that underpin improvement in student learning outcomes.

I would like to particularly acknowledge a number of Australian educators for their assistance:

Professor Richard Sweet, who generously assisted me planning my program of visits at theOECD by providing recommendations and introductions

Robin Shreeve from Skills Australia (Workplace Productivity Agency) who provided mewith leads and recommendations in the UK

Margaret Pearce and Mark Unwin from the Australian delegation to the OECD in Paris

Shelagh Whittleston and Tom Holdsworth from the Australian Embassy in Brussels andAustralian Education International.

A special thank you to my workplace colleagues and executive at the COAG Reform Councilwho granted me leave to undertake the overseas study and who continue to show interest in thefindings of my research. In particular, I single out for special mention two people whosupported my application as referees and assisted through the planning stages of my itinerary:

Mary Ann O’Loughlin, Executive Council and Head of Secretariat

Suzanne Varrall, (former) Director of the Centre of Excellence.



2. Performance reporting andaccountability in EnglandThere is a long history of reform and change in UK school education. In particular, Englandrepresents a rich case study for examining the use of performance reporting at various levels andfrom various perspectives. Although derived from common approaches, Northern Ireland,Wales and Scotland have diverged in their approaches and were not included in this study tour.

England’s school reforms have focused on school autonomy and public accountability. Theevolution of ‘academies’, schools operating under semi-autonomous governance arrangements,is the cornerstone of the reforms. In 2012, there were 2000 academies, mostly secondaryschools.

As a general observation, the unit of improvement in England is the school and this is where themajor accountability systems lie. The school education sector is characterised by widedistribution of legal authority and funding sources, multiple regulators and accountabilities.There are government and non-government bodies running schools or clusters of schools,providing interventions in schools, researching, commenting on and supplying schools withvarious services and products. So there is a decentralisation of influence in the sector anddivergent views on reform and the performance management systems that accompany them.

My itinerary allowed me to visit a good cross section of these participants as outlined below.

Type of organisation Name and location

Government agencies UK Department for Education

Office for Standards in Education

School providers CfBT Education Trust

Royal Society of Arts

Ark Schools

Achievement for all 3A

Schools Haven School, Sussex

City of London School, City of London

King Solomon Academy, North Westminster

Lillian Baylis Technology High School, Kennington

University faculties/Institutes Institute of Education, University of London

Cambridge University, Faculty of Education

Research bodies Sutton Trust

Rand Europe - Rand Education

Researchers/commentators Dr Becky Francis (various appointments)

Mr David Albury, Innovation Unit



2.1 The government policy framework

The performance-based accountability system for schools is driven by three main elements:

performance tables for schools

the setting of minimum performance benchmarks on key indicators, known as ‘floor levels’

school inspections.

The government has an overriding agenda on transparency of government processes andoutcomes, and the range of information and indicators publicly available keeps expanding. Dataand reports are published and freely available on the internet, allowing comparisons ofperformance at school level and aggregated levels.

There is no overriding system-wide evaluation framework for school reforms, althoughelements of the system are subject to review and evaluation, which are also publicly available.

Performance tablesPerformance tables are the main vehicle for performance reporting. The policy goal is to makedata driven decisions at every level, however, there is an emphasis on the school as the unit forchange/improvement and therefore reporting.

The stated goal of the performance tables is to:

‘enable parents, governors and others to find information which allows them to assesslevels of attainment and progress of pupils and students in a particular school orcollege and to compare that with other schools and with national averages’..(DfE, July, 2012)

The statutory assessment regime forms the basis of the performance reporting framework. (Seewww.education.gov.uk). Statutory assessments occur at the conclusion of key stages ofschooling and the number of tests has expanded each year under the current government. Onlycertain tests are officially in ‘the accountability regime’, others are supportingstudent/teacher/parent intervention. For example, the Phonic Screening Check at age 6 (years 1or 2) is a standardised test looking at a single skill, phonic decoding, and provides feedback toteachers and parents and is also captured in a national report. It is not used in schoolaccountability.

Indicators in the performance tables keep expanding in number each year. Tables includeinformation about academic results and school attendance, and data on school workforce andschool spending. The 2012 Tables include new indicators on attainment and progress measuresfor certain groups, such as non-mobile pupils (a good measure of the value-add of a school),boys and girls separately, pupils with English as an additional language, pupils from low socio-economic backgrounds.

Reporting uses both benchmarking—proportions of students reaching set levels ofachievement—and value-added measures, which look at cohorts of students over time and trackprogress.



Reporting is mostly at school level, as local authorities are no longer as relevant a unit ofanalysis with the advent of new governance models for schools. However, performance tablescan still be used at school, local area and national level.

There are examples of more complex measures in the accountability regime. ‘Narrowing thegap’ indicators are calculated and the gap monitored is between students from low socio-economic backgrounds and others. The proxy measure used is Free School Meals eligibility—this has replaced an earlier 16-factor index (like the Australian ICSEA index).

A ‘like schools’ model has been considered, but to date, the government has not includedcontextual factors in improvement measures—getting ‘all students over the line’ is the priority.

Statistical testing is a big part of the reporting and calculations are transparent. Change overtime is tested for statistical significance and is mostly year-on-year or a rolling average.

RAISEonline is a secondary vehicle for performance reporting but it is not part of theaccountability regime. It is an online tool marketed to schools using performance data and theoutcomes of Ofsted reports, all in one. There are many layers of analysis. Statistical significancetesting is used for change analysis, which is mostly year-on-year. All data are administrativecollections and there is no interface with survey data.

School inspections

The Office for Standards in Education, Children’s Services and Skills (Ofsted) inspects schools(among other regulatory and inspection functions) using a continuous improvement framework,involving school self-evaluation and the data available through statutory assessment and theschool performance tables, from RAISEonline. In this sense the system is well coordinated.

From1983 there have been moves toward more transparent accountability and Ofsted reportswere required to be public. Prior to this it was a collegial quality assurance developmentapproach. The current public accountability framework includes the rating of schools accordingto a four grade classification: Outstanding; Good; Requires improvement; Inadequate—hasserious weaknesses, or requires special measures.

Reports are used widely by schools, parents, researchers, the media and the government, foraction against underperforming schools.

Ofsted also has a good practice series, to promote transfer of good practice in schools. It is notwell moderated as schools nominate programs and they are put on the website if there is anevidence base cited.

Rewards and sanctions

There are no overt incentives for high performance or improvement. The Ofsted inspections,which use a grading approach, drive a public accountability agenda and influence both choice ofparents and the market advantage of schools, where competition exists to attract students.

Sanctions may include the government closing a school, requiring it to become an academy, orrequiring it to form part of a chain, if it is at the underperforming end of the scale. This has



generally brought extra money to a school (until recently), greater freedoms with staffing andoperational matters, and extra scrutiny.

There are examples of gaming behaviour in the reporting of indicators. Various policiesintroduced for good policy reasons may now be gamed in the performance regime. 1

2.2 School providers

The categories of schools and range of providers in England is numerous. The majority ofprimary schools are still operated by local authorities, however over 50% of secondary schoolsare independently governed by boards of governors or managed within ‘chains’ or federationsof providers, usually charities. Academies, which are publicly funded independent schools, freefrom local authority and national government control, were initially intended as a strategy toaddress failing schools. Under the current government, high performing schools are opting tobecome academies and the program of opting for local control is growing rapidly. Outstandingschools are also no longer required to have routine Ofsted inspections.

I visited a range of new and established providers of education services, including chains, and aselection of schools operating under the most successful or prominent models.

Achievement for all schools (3As)

Achievement for All (3As) is a national charity that supports schools to improve the achievementof young people. It does not directly operate schools. Schools opt into the program, which issupported and funded by government, as a school improvement strategy

The Achievement for All framework is a continuous improvement strategy aimed at improvingpupils’ progress, parental engagement, pupil attendance and behaviour, peer relationships,participation in extra-curricular activities and access to future opportunities for pupils. It wasoriginally designed for students with special education needs and is based on evidence of ‘whatworks’. It has been extensively evaluated with impressive results.

Achievement for All has been shown to have a profound impact on the outcomes of pupils withspecial education needs and on school improvement. Like many ‘whole-of-school’ approaches it

1 For example one indicator for schools is the proportion of students with five A-C GSCEs. The General Certificate ofSecondary Education (GSCE) is an academic qualification awarded in a specified subject, by students aged 14-16 insecondary education in England. Schools can use the early entry provisions for GSCEs intended for gifted students to'bank' a C for a student who may have been capable of getting an A if they did the whole course. The students are toldthey can do the exam again later and they may or may not. Changes in cohort are not taken into account for thisindicator.



is focused on improving the attainment and wider outcomes of all children, but particularly the20 per cent of the school population identified as disabled or with special educational needs.

Key features of the program include:

rigorous tracking of children’s progress in English and mathematics with intervention whenpupils fall behind

frequent structured conversations on educational outcomes between the teacher and theparent

a structured approach to addressing what is getting in the way of learning, such as bullying,persistent absence or poor social skills.

These approaches are led at senior leadership and head teacher level.

I visited the Haven School in Sussex, which is a Church of England/Methodist primary schoolof 244 students using the 3A (Achievement for All) system to improve student outcomes. Anexplicit goal is to improve its overall Ofsted rating. Its profile is one of average disadvantage onmeasures of socio-economic status (SES) but it has attracted more students than average withspecial education needs and disabilities and many students have had interrupted or little pre-school education. It attracts additional funding through various loadings to government fundingand, although class sizes are large by Australian primary school standards (30-32), each classhas a teacher and an assistant and sometimes up to two individual aides for students who qualifyfor one-on-one support.

It has a strong ethos and systems for learning through assessment which tracks student progressand uses 3A type strategies across the school to maintain and lift pupil attainment. A favourableOfsted inspection in 2010 has judged Haven School to be capable of being rated ‘outstanding’ atthe next inspection and there is strong evidence of success in the attainment and progressindicators. The quality of leadership and management was rated as outstanding on most criteriaand this appears to be the success factor that will pull this school from poor to averageperformance, through to good, then outstanding, in a five year timeframe.

ARK schools

ARK Schools is an education charity operating a network of 25 to 30 academy schools, invarious stages of establishment. The network comprises both new-start and transition schoolsthat have become ARK academies. The aim is to create “outstanding schools that give everyARK pupil the opportunity to go to university or pursue the career of their choice”.

To achieve this, ARK schools prioritize six principles: high expectations; exemplary behaviour;excellent teaching; depth before breadth; more time for learning; knowing every child.

I visited the King Solomon Academy in North Westminster. King Solomon Academy is a start-up school under the Ark banner. It moved into a low SES area into a closed Westminster school.There are many innovations operating within the school, supported by a loosening ofgovernment policy, but high expectations is the overriding principle.



“Our goal is to provide all pupils who attend KSA with the choice at 18 years of age togo on to university and begin a successful career.”(http://www.kingsolomonacademy.org/Welcome)

King Solomon Academy uses prominently displayed slogans to inspire students

This goal is reflected in many aspects of the school. Most obvious are the marketing andbranding, the names of the classrooms are prestige UK universities, and the approach ofteachers to all aspects of learning. Some key success factors were identified by the principal,Vanessa Willems and observed during my visit.

A long school day of 8:30 to 4:30, even in reception (ages 4-5).

Every child has the aspiration of attending university...even if their parents are sceptical, asthere are families at the school with multi generational unemployment.

Phonics for reading (government policy) and children receive short spurts in designatedPHONICS rooms more than once a day.

Good behaviour and silence is revered.

There are no class size issues, this is not seen as important with up to 4 adults in anyclassroom.

Teacher conditions are the same as other schools but teachers ‘buy into’ a 7am start and 6pm finish, when they apply to join the school. (Recent adjustments have been made toshorten the school week).

Teacher appraisal is weekly and involves student feedback.

There is a large number of senior executive staff to support learning programs and teacherdevelopment and the systems required to support intensive and complex planning.

There are targets for every child and for classes as a group.

The school uses the Sutton Trust progress model (see below).

The school is impressive in its presentation and the achievement of students. It has not yetoperated through to the graduation of a cohort of students to judge it on its own goal of everystudent transitioning to university. It was very unclear what happened to and what goals are set



for students with special needs, who are not turned away from the school, but for whom theaspiration of university may not be in their best interests.

Royal Society of Arts – academy schools

The RSA is an organisation, with a strong history of education initiatives. It is a recent entrantto the sector of school providers. The RSA's first sponsored Academy, the RSA Academy inTipton, opened in September 2008, and in 2011 was rated by Ofsted as good with outstandingcapacity for further improvement. The aim of RSA Academies is to develop a mutuallysupportive family of schools, which use innovative practice to raise standards and improveoutcomes for all children. Schools joining the RSA Family of Academies place an emphasis onincreasing social mobility by ensuring equality of opportunity, challenging discrimination, andnarrowing the attainment gap to enable all young people to realise their potential.

I visited the head office of RSA and met with Alison Critchley, the Director of the AcademiesProgram.

RSA has three schools and an interesting partnership model whereby high performing schoolsare assisting struggling schools. RSA is not a bureaucracy and is working on a different modelto the other chains. Collaboration and school-to-school improvement are the principles. It is alsousing the extensive network of RSA fellows to provide networks to work experience forstudents and other benefits.

They are strong users of RAISEonline, the online tool for performance monitoring. The purposeof RSA visits to schools as sponsors is to identify best practice for sharing. The ideal modelthey are working towards is a network of 2 to 3 secondary schools and 4 to 5 primary schools.The schools set targets with their own governing bodies using the DfE indicators. RSA islooking at expanding the framework to include indicators from surveys of students and parentsatisfaction and staff surveys.

I visited Lillian Baylis Technology School, which is currently under consideration to come intothe RSA family of academies. Also known as LBTS, it is a secondary school for students aged11 to 16, located in Kennington London. From 2012 it has offered ‘sixth form’ for Years 12 and13 (17-18 years). The school has been rated good with outstanding features by Ofsted at its mostrecent inspection.

The College specialises in technology and has similar offerings to Australian schools whichoffer extensive vocational education and training, including partnerships with workplaces andcolleges of further and technical education. Two thirds of the students are boys but genderbalance in enrolments is improving. Three quarters of students are eligible for free school meals,a very high proportion. There is a high proportion of students who are refugees or asylumseekers. The principal, Gary Phillips, reports that just under half of students in school requireEnglish Language support and about half have some form of learning difficulty or disability.These proportions are much larger than in most schools. The vast majority of students are ofminority ethnic heritage, with the largest groups being Black African, Black Caribbean, andPortuguese. Over fifty languages are spoken in students' homes.



Lillian Bayliss Technology School – Kennington, London

The strengths of the school lie in school leadership and innovation in curriculum andconnections with the community, including employers. There is not a great reliance on studenttracking and performance assessment and reporting beyond that required from the statutoryregime, which in England is substantial.

CfBT Education Trust, Reading

The Center for British Teachers started in the 1960s as a training organisation for teachers doingEnglish as a second language qualifications. It is now known as CfBT and has a substantialbusiness in education system reform projects internationally.

CfBT has 14 academies comprising sponsored academies, converter academies and freeschools. Each has a governing body with differing CfBT membership.

CfBT is accountable to a board of trustees like any charity. For reporting it uses the standardmetrics for English schools. To do so it employs in-house statisticians. CfBT sets schoolspecific improvement targets and also uses the national benchmarks the 'floor levels'. Twoschools are rated as outstanding—Oxford Spires and St Marks.



Independent Schools

The independent sector in England is small, accounting for only 9% of enrolments in Englandand 18% in London.

I visited the City of London School, an independent day school for boys aged 10 to 18. Itdescribes itself as being fully independent of government subsidy but is located rent-free onland owned by the City of London and the city councillors make up the majority of thegoverning board. It follows the mandated parts of the national curriculum, which are veryminimal.

In terms of results, it is in the top 30 schools nationally. The curriculum is broad andemphasizes a classic education in the liberal arts sense. For example, boys at CLS are able totake five modern and classical languages at GCSE. 90+% of leavers go to their first or secondchoice, Russell Group university. Consistently, about one fifth of the year goes to Oxbridge,according to the Director of Studies, Noeleen Murphy.

All students undergo extensive testing and it is highly competitive to enrol at age 11 and 13.

There are no disadvantaged students and therefore interest in the performance tables and theleading indicators is minimal, from parents and authorities alike.

In terms of performance measurement and accountability, there is a strong focus on testing.Testing to get in and then baseline testing done by the University of Durham, Centre forEvaluation and Testing under contract, from which the school does its own value-addedanalysis. The view is the benchmark indicators used by the DfE relate to low levels of baselineperformance—5 units at GSCEs at A to C, when this school has every pupil at A or A * for all10 to 14 units.



CLS compares itself to other independent schools and participates in the newspaper publishedleague tables as a measure of performance at school completion. A school can opt out, butCLBS does not because they perform so well. All students are in the top quintile. Schools withother such high performance are Westminster and Eton.

2.3 Commentators on the system of performance accountability

The English school sector is characterized by the number and range of active stakeholdersinvolved. Philanthropists, research bodies, established and new charitable trusts, all have a handin it, along with government bureaucracies, local authorities, and local governance bodiescomprised of parents, school executives and local prominent community members.

As in Australia, within the English education sector, a focus on external assessment foraccountability and improvement is often criticised for being a narrow blunt measure of studentachievement. Further, in the high stakes environment created by government-imposed sanctionsfor ‘underperformance’, the testing is said to narrow the curriculum and impact on time fordeeper and broader teaching and learning.

Those in support of external comparative assessment argue that the use of comparative testingdata, linked to sanctions for underperformance, is required to build school leaders, teacher andschool responsibility for improved student learning outcomes. The strategy becomesproblematic when the school, its leader and teachers, are overly focused on the externalmeasures of student performance and ignore the other drivers of student performance. Inparticular, the debate is vigorous around educationally disadvantaged students and communitiesand the policies being implemented to secure ‘social mobility’ through education.

I met with a range of these key influencers and critics of education reform.

University faculties/Institutes:

Professor Chris Husbands, Director, Institute of Education, University of London

Professor Peter Gronn, Head of Faculty of Education, Cambridge University

Research bodies/think tanks:

Dr Lee Elliot Major, Director, the Sutton Trust

Dr Michael Frearson, Research leader, education skills, RAND Europe, located inCambridge

Researchers/commentators:

Dr Becky Francis, Professor of Education and Social Justice, Kings College London andDirector, Pearsons Think Tank

David Albury, Innovation Unit



On the accountability framework

Criticisms of the accountability framework include:

it only measures cognitive outcomes

it is purely a competitive model not collaborative, except where collaboration may becoerced

evaluating academies as a group, which have outstanding schools and failing schools, meansthe averages will be average, regression to the mean in operation

it has introduced perverse incentives which disadvantage low achieving students andschools.

Professor Husbands is an active commentator on education reform and a member of theinfluential Academies Commission.

His overall view is that England has established a satisfactory accountability framework but thattransparency is not sufficient. There are some schools that have got the ‘data drivingimprovement’ approach integrated as part of routine school practice. Using high level technicalanalysis, regression analyses, analysis of the residuals, etc. some schools have developedsophisticated approaches to performance monitoring and tracking to inform local schoolimprovement strategies. The Government’s framework has provided opportunities for a marketfor third party brokers to assist schools in public reporting and this is conducive to gaming ofresults in the face of incentives, which are not aligned with school improvement goals.

He cited the Campbell2 principle at work in UK education:

"The more any quantitative social indicator is used for social decision-making, themore subject it will be to corruption pressures and the more apt it will be to distort andcorrupt the social processes it is intended to monitor."

2 Campbell's law was published in 1976 by Donald Campbell, a social science researcher and the author of works on researchmethodology. The principle of Campbell's law is sometimes used to point out the negative consequences of high stakes testing inU.S. classrooms. Campbell also states in this principle is that "achievement tests may well be valuable indicators of general schoolachievement under conditions of normal teaching aimed at general competence. But when test scores become the goal of theteaching process, they both lose their value as indicators of educational status and distort the educational process in undesirableways.”



The main foyer at the Institute of Education, central London

Professor Becky Francis writes influentially about accountability and school reform in the UKand social justice. (See for example her monograph on improving ‘satisfactory’ schools).

Professor Francis characterises the English school reforms as marketisation of the schoolingsystem with leanings towards social exclusion through competition. Policy vacuums have beencreated through the principle of free market economics and the losers are poor kids, kids withspecial needs and some school communities.

Her view is that the accountability framework—performance tables, Ofsted reports, floor targets(minimum performance benchmarks)—drives the market and introduces incentives anddisincentives, mainly around unwanted kids and low performers. If a school is operating close tothe floor level, that is 35%, then it may seek to exclude students likely to jeopardise theachievement of the target. She cited public test cases around academies refusing to take studentswith special needs. The lower court judged that these schools were operating like privateschools and therefore had the right to exclude families. The higher court overturned this rulingand the schools were directed to take the students,.

From some perspectives, the system is now driven by the single indicator—the 5 A*-Cindicator3. It used to be relatively easy to achieve the proportion required to reach the ‘floorlevel’ signifying adequate performance, due to two factors. Before the Blair government, theindicator did not include the required English and Maths subjects, so schools were includingless rigorous courses. This led to a proliferation of so called ‘dumbed-down’ courses, whichwere also consuming much of the school week of weaker students at age 13 or 14. There wasalso the notion of equivalence to GSCEs, which has since been has abolished. Equivalents

3 The indicator is the proportion of students at the end of Key Stage 4, achieving 5 or more GSCEs, graded from C toA*, or equivalents, including English and Maths. The ‘floor level’ is set at less than 35%, increasing to 40% in 2013 and50% by 2015.



allowed less rigorous courses to be equivalised to GSCEs and each one had been worth up tothree or four equivalent units. All this had the effect of excluding poorer kids and lowerachieving kids from curriculum options, leading to poor outcomes.

Professor Francis supports the development of the English Baccalaureate—a new qualificationbased on a pattern of study across the liberal arts and sciences—but has some reservations aboutthe pattern of study being too narrow. It introduces more rigour and again is a policy reformintroduced by the development of the indicators as a policy lever, rather than mandating anyrequired patterns of study. This is the market approach.

Professor Francis’ report on satisfactory schools has been influential in Ofsted’s decision torelabel these schools as ‘requiring improvement’. The Government had focused a lot on failingschools through the academy developments and outstanding schools through Ofsted reforms(reducing the need for inspections) but done little for the 70% of schools in between. Thisrelabelling puts a focus back on schools previously rated as satisfactory, to lift theirperformance.

On school reform and school autonomy

There has been some strongly contested evaluation of the 2006 to 2012 reforms for schoolautonomy. Academies have made some differences to some schools but there have beenfailures, with schools surrounded by high achieving selective schools in the wrong locationfailing to attract cohorts of students to lift school performance.

Stakeholders argue that Government policies aimed at schools learning from each other andforming partnerships can work for academies, which are part of chains. But this collaboration isnot available for the vast majority of government-maintained schools, which are in competitionfor high achieving students and survival as a satisfactory school. The goal of becoming a 'selfimproving' system can't be realised in such a market system.

The Commission on Academies, a self-appointed review of the academy developments inEngland, has worked to identify the success factors and challenges of the school autonomymovement, epitomised by school academies. Their most recent report and dialogues (videopresentation on You Tube) identified the following:

collaboration, not just autonomy leads to school improvement,

schools themselves should assist other schools—school-to-school collaboration not topdown

teaching and learning are the main game, with a focus on ongoing teacher development

competitive environment and collaboration can co-exist

there is a need to set standards for collaboration and curriculum areas like school sport

we should also measure leadership in terms of contribution of the leader to the system

there is a need to review and reinvigorate the role of local authorities in schooling as theirformer roles are being residualised

peak bodies should become more active in the debate on school improvement



social inclusion has to be in the remit of all academies not just some—stamp out exclusionof low performers

parity of admissions and funding needs Government policy implementation

greater accountability to parents and the community through better performance by schoolgovernors, through better training and advertising of Chair posts, to get the right people

high standards for sponsors—chains running schools—and reporting on their performance

a stronger role for Government in financial accountability.

On the impact of accountability and school reform on social inclusion

The Sutton Trust, a private philanthropic organisation contributing to research, advocacy andworking directly with schools and school leaders, was instrumental in the adoption of the term‘Social Mobility’ in education and its wide uptake, including by Government. Social mobility isconcerned with making social background less of a determinant of school outcomes.

The current priority for the Sutton Trust is to effectively mobilise the knowledge about ‘whatworks’ for students from disadvantaged backgrounds. Dr Lee Major’s view is that we knowwhat works for school improvement and for disadvantaged schools and students; it is the uptakeand implementation that should be the current research and advocacy agenda. The Sutton TrustTeaching and Learning Toolkit, which is being promoted to schools through workshops, isfilling a gap vacated by the Department for Education, as the current Government is hands offwith its school autonomy drive. Although it is schools that do the work, they still need resourcesand support and the toolkit has been hugely successful.

The Sutton Trust is actively against putting a school improvement tool, such as the toolkit in theaccountability framework. This is seen as counter – productive and does not lend itself toteacher development or incentivising but to a 'tick the box' mentality.

Faculty of Education, Cambridge University, student courtyard and garden



3. Comparing performance acrosscountries

Performance reporting using indicators that allow comparisons between countries is a well-established function of a number of high profile international organisations.

The organisations I chose to visit publish both education indicators and produce policy researchand best practice in education, drawing on the evidence base informed by the data.

The education indicators are mainly used in comparative reporting to drive improvement in theperformance of national education systems. The setting of national goals or targets andreporting against them is increasingly forming part of the performance reporting regime.

The policy research work that flows from, or is supported by, international comparisons ofindicators is substantial and cannot be described in full in this report. I have described someinteresting work in each case and will be using the other resources as follow up material.

The organisations I visited are:

the Organisation for Economic and Co-operative Development (Paris)

the European Commission - Education and Culture Directorate (Brussels)

UNESCO (Paris)

- Education for All Monitoring group

- International Institute for Educational Planning.

Data development and technical issues are discussed in Chapter 4, including my visit to Eurostatin Luxembourg.

3.1 Performance reporting for OECD countries

The OECD makes a significant contribution to comparative performance reporting in educationand the policy learnings that flow from identifying high performing education systems. Thiswork is guided by its 50th anniversary vision, Better Policy for Better Lives, and is underpinnedby a third pillar, Better Measures.

Well known for its flagship international testing the Programme for International StudentAssessment (PISA), the OECD reports on a suite of international surveys, guided by two keyprinciples: policy relevance and comparability across countries.

The major surveys are:

PISA—the Programme of International Student Assessment—tests 15 year olds on reading,mathematical and scientific literacy

TALIS—the Teaching and Learning International Survey—covers elements from initialteacher education, to mentoring, to professional development, appraisal and feedback toteaching practices in the classroom



PIACC—the Programme for the International Assessment of Adult Competencies—asurvey of adults aged 16–65 years old assessing their literacy and numeracy skills and theirability to solve problems in technology-rich environments

HELO—Higher Education Learning Outcomes—is the new higher education survey to testwhat graduates know and can do. It was partly developed due to the over-reliance onuniversity rankings and research effort to the ignorance of student leaning outcomes.

A related arm of reporting is Education at a Glance, an annual publication that reports a rangeof internationally comparable data on education. It is a compendium of education statistics thatserves as the foundation for other work drawn from collections in 34 member countries andeight non-member participants.

In general, the measurement approach at the OECD is that indicators must be relevant andsupporting data must be of high quality. For relevance, indicators must have:

face validity

be easily understood with an unambiguous interpretation

be amenable to policy changes

provide possible disaggregations.

Chateau de la Muette – HQ of the OECD

An example of comparative reporting from PISA is below.



How proficient are students in reading? Percentage of students at the different levels ofreading proficiency

Notes: Countries are ranked in descending order of the percentage of students at Levels 2, 3, 4, 5 and 6.

Source: OECD PISA 2009 database, Table I.2.1.

Targets and benchmarks are not part of the comparative framework at the OECD. Whenqueried, I was advised it is not part of their ‘culture’, which is built on collaboration betweencountries and letting comparisons per se drive performance improvement. The terms ‘highestand lowest performers’ are used and function as quasi benchmarks.

100 80 60 40 20 0 20 40 60 80 100

KyrgyzstanAzerbaijan

PanamaPeru

QatarKazakhstan

AlbaniaIndonesiaArgentina

TunisiaBrazil

MontenegroJordan

ColombiaTrinidad and Tobago

ThailandUruguayBulgaria

RomaniaMexicoSerbia

Dubai (UAE)Chile

AustriaRussian Federation

IsraelLuxembourg

TurkeyLithuania

Czech RepublicCroatia

Slovak RepublicGreece

SloveniaItaly

FranceSpain

GermanyUnited Kingdom

BelgiumPortugal

United StatesLatvia

HungarySwedenIrelandIceland

SwitzerlandLiechtenstein

Chinese TaipeiDenmark

PolandNorway

Macao-ChinaNew ZealandNetherlands

AustraliaJapan

EstoniaSingapore

CanadaHong Kong-China

FinlandKorea

Shanghai-China

%

Students at Level 1aor below

Students at Level 2or above



Current and future priorities in performance measurement

Comparative assessment, as the OECD calls it, gives “an idea on where countries should go anda sense of the pace of change, the progress that can be made by countries in certain timeframes”(www.oecd.org/education).

PISA is a gold standard international test using a survey. It is single point in time reporting, withgreat reliability and quality. The focus is on quality and good public understanding of it. Eventhough there is criticism that PISA is a single year of testing of a survey sample, its uses andvalue are supported widely.

According to Andreas Schleicher, a leading international researcher and speaker oneducational measurement, the driving principle for reporting on education indicators is theimpact it can have on educational outcomes. He believes that good comparative data has greaterinfluence over policy than anything else.

All potential developments are examined through this lens of impact on outcomes.

PISA remains the flagship program for the OECD and if possible new indicators anddevelopments are tied into the international infrastructure already invested in PISA rather thannew surveys or collections. The OECD Education directorate is developing new areas inresponse to priorities:

PIACC is the adult learning and skills test, which has incorporated technology and digitalcompetences. Its first report is in 2013.

PISA continues to be developed. The digital literacy component is new and was reported onin 2012. In 2015 they will include collaborative problem solving into the testing.

For system and school level, they will be trying to look at the quality of teaching,internationally. It is considered possible but difficult to measure. The Gates FoundationMeasures of Effective Teaching provides a starting point for this work. Work on acommercial product, PISA for Schools, is also progressing.

According to my interviewees, there is always pressure for inclusion of new indicators. Ingeneral, the OECD avoids what M Schleicher calls ‘faddish measures’, which may reflectcurrent operational pressures in systems, such as bullying or student satisfaction. New indicatorshave to be ongoing, cyclical, and conceptually good. For example, the directorate is looking atindicators for educational mobility, building on the UK and the EU’s interest in and policy focuson social mobility. Financial literacy is also of interest post the global financial crisis.

The future of education indicators must reflect the 21st century model of learning. There will bea move away from reporting accredited learning and qualifications as the outcomes of educationthat focus on the means of production. The future framework should incorporate other thancognitive measures and move to social outcomes reporting and non-cognitive skills. Anothernew area is collaborative problem solving.



Reporting on efficiency in education systems is a new priority for the OECD, as historically ithas spent a lot of time on quality and equity. (Efficiency is a missing plank in Australia’snational performance reporting framework.)

Impact on policy and innovation in education

Historically the OECD’s approach to its mandate has been structurally separated into aquantitative and a qualitative side. The culture change being driven now is to coordinate thesethings together as one process. The quantitative side informs the qualitative reviews, leading topolicy formulations, leading to change in practice and outcomes.

The policy advice function of the OECD brings together all major international tests andreviews and provides advice to relevant committees from an education perspective.

The policy implementation function involves:

thematic reviews, which look at a particular policy area across countries

consultancies for country reviews, usually at the invitation of governments to advise onpolicy and implementation

international summits, for example the International Summits on the Teaching Profession:Preparing Teachers and Developing School Leaders.

Structurally, within the OECD, the Center for Education Research and Innovation (CERI), ledby Director Dirk Van Damme, conducts large-scale inter-country policy reviews and researchinto emerging issues in education, driven by an innovation agenda. The research agenda isdriven by what is relevant for economic and social development of countries.

Dr Van Damme reflected on the changes over time to indicators and suggested that the currentthinking was to incorporate the social outcomes of education. There is, in fact, a redefinition ofeducational outcomes as social outcomes, which sit alongside and contribute to economicoutcomes. They present like a basket of indicators not an integrated set. Whilst elsewhere in theOECD the work on the Healthy Life Index has conceptualised education as part of a largerbasket of drivers of healthy outcomes (see discussion below).

The Steiglitz Commission on Measurement of Economic and Social Progress was veryinfluential in setting the agenda in measurement of social and economic outcomes.

Work on innovation in education is linked to the OECD-wide Innovation Strategy. This hasfocused on the innovative learning environment. It is a collaborative model of research.

In terms of new horizons, the CERI work is moving towards measuring non-cognitive skills e.g.perseverance, resilience, personality traits. They are at the framework stage, and next they willlook for evidence in data collections and research.

Two ground-breaking research projects led by David Instance are:

Schooling for Tomorrow, which developed scenarios for a 10 year timeframe



Innovation in Learning Environments, which commenced in 2010 with research and thepublication ' The Nature of Learning'. This led to the Seven Principles, ratified anddeveloped at the International Conference on School Effectiveness.

An example of the take-up of this work is in British Columbia where they are using this workfor their own innovation strategy.

Other CERI projects which demonstrate the link between research, policy and indicators are:

Technology and education - New Millennium Learners, includes mainstream indicators onuse of technology

Connected Minds - provided the research base for what is now the digital reading literacytest in PISA.

As a metaphor, CERI is the incubator where the research work is done that produces a new orrevised education indicator for international comparative reporting.

OECD researchers also work at the level of raising political awareness and buy-in to theirfindings. The ‘Policy Outlook' series will be used to promote these innovation projects to thepolitical level of awareness.

Reporting for accountability and to drive improvement

A major review that Australia has participated in is the Review of Evaluation and AssessmentSystems led by Paul Santiago and Clare Shewbridge. The country-specific reviews in this areahave been done in 15 countries out of 26 member countries. Other countries provide anoverview of their own situation. This will be distilled into a final report during 2013.

The review findings relate to systems within countries for evaluation and assessment of studentoutcomes at student, school and system level. The reports paint an interesting picture ofdiversity in assessment and reporting practices. Some highlights outlined by Santiago andShewbridge include:

Australia’s systems are highly regarded (see Australian country report) for coherence acrossthe levels of the education system.

This contrasts to Norway, which does not have a national overview.

Some countries are data driven, and they limit their activity to the supply and publication ofthe data/indicators.

New Zealand has rejected national testing at the primary level and their focus is on teacherdevelopment. They conduct moderation of testing at a national level only and thedepartment makes available a non-mandatory assessment tool which teachers can use.

Finland conducts sampling testing rather than census testing as in Australia. They focus on‘learning to learn’ skills.

For the Flemish system in Belgium, there is complete agreement there will be no universaltesting and sampling is used in national testing. There are no incentives built into the testingregime and collaboration is the underlying principle. Assessment tools are made available,but there is no requirement to use them.



Denmark does not have an evaluation culture at all, online tools are made available but theyare optional. There is some national testing and local authorities have the large role inassessment and evaluation.

Across the countries reviewed, there has been some overall harmonisation of policy approachesto assessment, but there has not been a focus on an inspectorate approach at the school level, asin the UK. New Zealand has made the connection between the two approaches, internal versusexternal review. It has an external review office but it is a collaborative model with schools.

An OECD principle for participation in OECD programs is that capacity has to be in evidence.Some countries don't have good systems even if they participate in PISA, for example, theydon't collect data on SES background, or place of residence.

Reporting for transparency

Education at a Glance serves a transparency purpose. The document largely meets the needs ofthe media. It includes all the caveats with the data. It is as timely as it can be, as it is dependenton the data collection cycles and individual countries.

The main transparency strategy for the OECD is to put data in the public domain, but it issecondary source data not micro data and therefore has more limited capacity for analysis.

The OECD is looking at further transparency steps, like dumping all the data on the web (theUK model), however there are mixed views on this.



3.2 Performance reporting for the European Union

The European Commission’s role in education is to

“To reinforce and promote lifelong learning, linguistic and cultural diversity,mobility and the engagement of European citizens, in particular the young.”

(http://ec.europa.eu/education)

EU Member States and the European Commission adopted a high level strategy in 2009 forEuropean cooperation in education and training called Education and Training 2020.

Under Education and Training 2020, four strategic objectives have been agreed and a number ofEU benchmarks have been set, of which two – concerning early school leaving and tertiaryeducational attainment – have been taken up by Europe 2020 as headline targets.

The Education and Culture Directorate has oversight of major policy, performance monitoringand associated programs.

Central Brussels – home to the European Commission and public homage to the brussel sprout



Current and future priorities in performance measurement

These objectives and benchmarks provide the framework for reporting on Member States'progress in relation to education and training policy goals.

Education & Training 2020 Strategic Objectives

Making lifelong learning and mobility a reality

Improving the quality and efficiency of education and training

Promoting equity, social cohesion and active citizenship

Enhancing creativity and innovation, including entrepreneurship, at all levels of education and training.

Seven Education & Training 2020 Benchmarks

At least 95% of children between the age of four and the age for starting compulsory primary educationshould participate in early childhood education

The share of 15-year-olds with insufficient abilities in reading, mathematics and science should be lessthan 15%

The share of early leavers from education and training should be less than 10%

The share of 30 to 34-year-olds with tertiary educational attainment should be at least 40%

An average of at least 15% of adults should participate in lifelong learning

At least 20 % of higher education graduates and 6% of 18-34 year-olds with an initial VET qualificationshould have had a period of study or training abroad

The share of employed graduates (20-34 year-olds) having left education and training no more thanthree years before the reference year should be at least 82%

Source: European Commission (2012) Education and training in Europe 2020 at www.ec.europa.ed/education

Benchmarks on early leavers and tertiary attainment form headline education targets of theEurope 2020 strategy. An eighth benchmark, covering foreign language skills is currently beingdeveloped, with a view to adoption by the Council in 2013. Sixteen core indicators formonitoring progress towards the objectives support EU benchmarks.

Reporting on progress towards benchmarks is through the report Education Monitor, which iscomparative and provides assessment of achievements and where progress is not sufficient.

The European Council and the Commission also publish a joint report on the overall situationevery two years. Using data from both the progress reports and national reports, this takes amore strategic view, assessing developments across national education systems and delivering aseries of key messages.

The targets are non-binding and aspirational. For the first time in 2012 country specific targetshave been set to recognize the diversity and inequity between member states on these indicators.

Measurement issues are managed by EuroStat the commission’s statistical body based inLuxembourg (see chapter 4).



Impact on policy and innovation in education

The Education and Culture directorate is always mindful that EU Member States are responsiblefor the organisation and content of their own education and training systems. The Commissionuses what they call the ‘Open Method of Coordination’ in areas such as education, where thereis no direct control over member activity. National efforts are supported in two main ways.

Funding programs, such as the Comenius program, which invests in projects to promoteschool exchanges, school development, education of school staff, school assistantships.

The Commission works closely with national policy-makers to help them develop theirschool education policies and systems. It gathers and shares information and analysis andencourages the exchange of good policy practices.

Current policy priorities are:

Key competences for a rapidly changing knowledge society. This includes increasing levelsof literacy and numeracy and modernising curricula, learning materials and pupilassessment.

High-quality learning for every student. This includes improving equity in schoolsystems, making pre-school education more widely available, reducing early school leavingand improving support within mainstream schooling for students with special needs.(Special needs students have a distinct European agency and are not included).

Improving support for teachers, school leaders and teacher educators, through moreeffective recruitment and selection and better-quality professional education.

I had the opportunity while in Brussels to observe the policy dialogue process in action. I was anobserver at the Australian-European policy forum on Education and Innovation, co-hosted bythe Australian Ambassador to the EU and the Deputy Director-General of the Education andCulture Directorate at the European Commission.

The Australian – EU Policy Forum on Education and Innovation 19 November 2012



3.3 Comparative performance reporting for developing countries

UNESCO - Education For All Global Monitoring report

The Education for All (EFA) movement is a global commitment to provide quality basiceducation for all children, youth and adults. At the World Education forum in Dakar, in 2000,164 governments pledged to achieve EFA and identified six goals to be met by 2015.

UNESCO has a mandate to coordinate reporting against the six goals and also sponsors policydialogues, monitoring, advocacy, mobilization of funding, and capacity development.

Education for All Goals

Goal 1 Expanding and improving comprehensive early childhood care and education,especially for the most vulnerable and disadvantaged children.

Goal 2 Ensuring that by 2015 all children, particularly girls, children in difficultcircumstances and those belonging to ethnic minorities, have access to, andcomplete, free and compulsory primary education of good quality.

Goal 3 Ensuring that the learning needs of all young people and adults are met throughequitable access to appropriate learning and life-skills programmes.

Goal 4 Achieving a 50 per cent improvement in levels of adult literacy by 2015, especiallyfor women, and equitable access to basic and continuing education for all adults.

Goal 5 Eliminating gender disparities in primary and secondary education by 2005, andachieving gender equality in education by 2015, with a focus on ensuring girls’ fulland equal access to and achievement in basic education of good quality.

Goal 6 Improving all aspects of the quality of education and ensuring excellence of all sothat recognized and measurable learning outcomes are achieved by all, especially inliteracy, numeracy and essential life skills.

Source: www.unesco.org

UNESCO uses targets to support the goals in the EFA framework. However, benchmarks(external aspirational targets that represent performance in OECD countries or ‘best in world’benchmarks) are not used in the EFA reporting yet, due to cultural and economic factors.

UNESCO leads the EFA monitoring process through its Institute for Statistics and the EFAGlobal Monitoring Report:

The Institute for Statistics, based in Montreal, is responsible for collecting data and statisticsfor the Education for All goals and for the Millennium Development Goals that relate toeducation. It is the primary source for cross-nationally comparable statistics on education,science and technology, culture, and communication for more than 200 countries.

The EFA Global Monitoring Report is an annual publication that monitors the progress ofcountries and identifies global challenges in order to reach the EFA goals by the year 2015.



Headquarters of UNESCO, Paris

An example of the style of comparative reporting is below:

There is a high level of emphasis on communications and both the website and publications areimpressive. The achievements focus on key messages and media coverage of performancereports is extensive.

For time series reporting the presentation is not a sophisticated analysis, it simply presents thedata. Background papers do get more sophisticated and these are made available on the web.















On measures of growth, the high performers are the BRICs—Brazil, Russia, India, and China—so there is interest in these countries as ‘best improvers’ and the policy implementation thatunderpins this progress. But overall, the focus remains on inequity within countries.

Impact on policy in education and innovation

A single report is issued covering progress on the goals and the thematic review chosen for theyear. The thematic review is the primary policy implementation instrument and is based onreviews of the literature, commissioned papers, workshops and consultation. In 2012, the themewas youth and work and the resulting work reflects evidence-based research on best practice.

The 2013 report will cover innovation in Education. The 2013 thematic review will be relookingat the goals, as 2015 is approaching. The agenda is to make education the priority it deserves inthe review of the Millennium Goals, which is substantially aid focused.

Reporting for transparency

Only some countries in the UNESCO family are pursuing a transparency agenda, but there isgrowing recognition that it is international best practice.

The International Institute for Educational Planning

UNESCO created the International Institute for Educational Planning (IIEP) in 1963. It issupported by grants from UNESCO and by voluntary contributions from Member States andothers.

The IIEP mission is to help countries plan and manage education systems. They have a largerole in training of education planners but are also involved in performance measurement andsystems for accountability. The goal is to get each country to develop a five-year plan. It maytake 1.5 years to develop a plan and in some countries there is little or no data.

There are a number of interesting projects underway both in performance reporting and policyimplementation to improve performance.

SACMEQ - Southern and Eastern Africa Consortium for Monitoring Educational Quality

SACMEQ is a collaborative network of 15 ministries of education. It has produced a number ofreports on reading, mathematics and HIV/AIDS education. National testing, which allowscomparisons across countries, is still in its infancy in many countries. The reporting projects, asthey are called, use a standardized survey tool and result in policy recommendations, which arenon-bonding on countries. The current priorities are teacher quality and literacy and numeracy.

Portal Project - Global Outlook on Learning

This is a clearinghouse concept that synthesises available research. It tries to be neutral and isadvisory. It also has ‘controversy pages’ for contested areas - I like this idea. Entry to the portalwill be via a multilingual glossary.



4 Challenges of indicator and datadevelopmentThe public agencies I visited invest, to different degrees, in indicator development andimproving the quality of data collections to support their approaches to performancemeasurement and analysis.

I took the opportunity to explore the approaches to various data challenges we experience herein Australia.

4.1 The UK Department for Education

The Department for Education does all data collection with no involvement of the nationalstatistics body, although an outsourced officer provides a quality overview of the collections.

Timeliness is cited as an issue in performance reporting in the UK transparency andaccountability regime. In the case of statutory assessments, testing is done in May and resultsare not available till September to schools and validated final results in December for publicreporting.

A major project is underway to modernise data management arrangements and improve enquiryand publication services across the accountability and school performance data. Theimprovement project is called the School Performance Data Programme, under the leadership ofPaul Roberts. It is a business change project based on improving the IT system to deliver thegovernment’s agenda for transparency and multiple users.

The future vision is for better access for all data users and powerful analysis, as defined by theneeds of the user. It is firmly driven by a transparency agenda, and not concerned with limitingor agreeing on valid indicators and their policy relevance. All users will be able to access anddefine their own queries and policy questions. The main features are data warehouse servicesand portal services which will be out-sourced from government, with adequate security controls.

In terms of data development priorities that Australia is pursuing:

the UK has relied on data matching and data linkage in the past, but it is now consideredvery expensive and very ‘clunky’

the new streamlined model is based on making data accessible rather than doing analysisfor users

a unique pupil number exists, but it is not being properly administered and the currentgovernment does not intend to further develop it, as part of a privacy agenda

data linkage development is now mainly confined to whole of government initiatives forvulnerable children.



4.2 International comparisons

Agencies involved in the compilation of internationally comparable statistics on educationoperate using the Internatioanl Standard Classification of Education (ICSED) which providesthe basis for comparing different education systems and a standard terminology.

OECD

The OECD has a dedicated Statistics Directorate supporting a high level advisory committee.Most recently they have shifted in focus from advocacy of data to data driving policy. The focusof change came about through the Steilitz Commission and its report on measuring social andeconomic progress.

Choice of indicators

The OECD is comfortable using proxies where data is not available and prepared to mix yearsof data for different countries. Publishing ‘dirty data’ in reports is considered acceptable and isseen as a strategy for improving the underpinning collection. The OECD highlights theindicators built on dirty data with a 'development ' tag.

New indicators are included if they are conceptually well linked to the framework, it driveseverything. A new indicator has to be of enduring value, and the technical issues of datacollection are secondary.

Future of indicators

At the OECD, there is considerable investment in looking beyond single indicators to a basketof indicators or indices. As education is being positioned as an economic and social indicatorunderpining individual and country well-being and productivity, it makes sense to look atmulti-faceted measures.

An important OECD project is the Better Life Index, led by M Marco Mira d'Ercole, in chargeof Progress Measurement in the directorate. It considers a basket of indicators across threeinteracting domains—quality of life (including education and skills); material living conditions;and sustainability of well-being over time. The associated online tool allows indicators to beweighted by the individual in the construction of the index, which feeds back into the workbeing done to further develop the index.

Similar work is being done in Britain under a national well-being initiative of the Office ofNational Statistics.

Process for developing new indicators

Early childhood is a development priority for Education at a Glance. Early childhood reportinghas focused on access indicators to date and now has to go to outputs and outcomes. Achallenge for this sector is that there are two distinct approaches in the world for these earlyyears, which makes the issue of comparability problematic. The Nordic approach focuses onsocial growth of the whole person and rejects the measurement of early cognitive development



and the US/ Aus approach concentrates on early literacy. So inclusion of an indicator has to dealwith the underlying concept being measured.

Another new area is financial literacy, which is of increasing interest post the GFC crisis.

The process the OECD uses is collaborative, driven by agreed principles about the quality ofdata. So they start with an expert advisory panel, they get together and define the scope of thedomain to be tested or measured. Then work is done to define it, work out the relationships, andwork out the framework components. They then engage in an item review process, field-test itfor congruence with the framework and cultural appropriateness.

There is a four-year process for framework development. Field-testing for the new area offinancial literacy is in 2014.

The 2013 agenda is for the OECD to develop a good understanding of the policy drivers of highperformers. Although this is not the frame for benchmarks, it is implied in the comparisons.

Data development issues

OECD data specialists believe most measurement issues can be resolved. They subscribe toflexibility in their reporting and are transparent in the use of ‘dirty data’ and proxies foroutcome indicators.

Although there is interest in administrative collections in education, the OECD supports the useof surveys as efficient and they can be high quality. For example, household income surveys areused worldwide and therefore are an ideal vehicle for comparative assessment.

Statistical analysis

The OECD do not generally have access to micro data and therefore do not report standarderrors, but consider this would be best practice. Progress over time is reported on time lines.

Timeliness is not an issue they are overly concerned about. They just use the latest data and gowith the collection. There is little concern about different countries having different time setsbeing compared.

The trend for the future will be on rolling sampling and reporting in volumes. Online reportingis the future but there are still some countries not ready for that so they are running concurrentsystems with paper based and computer based. For PISA 2015, the domain of assessment andproblem solving will be incorporated. Only countries that find it relevant and can afford it areincluded in these new domains.

European Commission reporting

Data collection for European Commission reporting is managed by Eurostat, the statisticaloffice of the European Communities. There is joint UNESCO/OECD/EUROSTAT datacollection on education statistics to provide comparable data on education systems.



Data is collected under the protocols of the ‘Open Method of Coordination’ and there istherefore no access to micro data. Data is collected by questionniare using common definitionsand criteria for the control of data. As countries are at different stages of development of theirdata management systems, comparability can at times be contested.

Choice of indicators

Indicators are chosen based on consensus and that which can be measured through theagreeements on data collection. This is enshrined in legislation and supported by regulation, soit is a very formal process wthout the flexibility evident at the OECD.

There has been some movement in ensuring coverage of the whole European Union whereparticipation in Education at a Glance had been voluntary up till 2011 and now is mandatory.

Statistical analysis

As there is no access to the mico data, no standard errors are reported and analysis is thereforelimited.

UNESCO – EFA reporting

The source of the data is the Institute of Education Statistics in Montreal working withEurostat.. The countries provide the data via a common template and there is no access to themicro data. It does not report on error for this reason.

Where the collection is done by others and there is a partnership they have access to the microdata e.g. UNICEF survey, main interest is in maths and some early childhood data.



Conclusions and recommendationsConclusions

On school improvement and accountability

As Australia moves towards mandating school improvement plans tied to government fundingregimes, it is timely to reflect on the English experience with performance reporting,accountability and school improvement.

While the accountability framework for English schools is complex and dynamic, it hasbeen supported by significant investment in technology and training. More time is requiredto allow schools and systems to ‘bed down’ reforms.

Collaboration and competition can co-exist in devolved systems with strong accountabilityframeworks, as long as sufficient support and incentives are provided for collaboration.

Opportunities for ‘gaming’ the system need to be guarded against, both in design andreview of the accountability system.

On comparative performance reporting across countries

Comparative performance reporting is part of the Australian architecture for intergovernmentalrelations and accountability in Australia. The experience of international organisations such asthe OECD, the European Commission and UNESCO, and the associated protocols developedaround such reporting, provide insights for governments in Australia.

In general, the driving principle for indicator selection and reporting at system level shouldbe the capacity to impact on educational outcomes.

Indicators of efficiency are the missing link in Australia’s high-level performance reportingframework through COAG.

There is substantial benefit in planning a forward program of indicator development as theknowledge about ‘what works’ and the evidence underpinning high performance educationsystems is determined and disseminated.

There is limited evidence for improving outcomes through the use of data alone—it needs tobe placed in a framework of improvement through the uptake of practices delivering highperformance.

Targets and benchmarks are not universally supported as effective mechanisms forimprovement. A process of renewal at frequent intervals, achieving renewed commitment totheir purpose, is recommended.



On the challenges of indicator development and measurement

Organisations engaged in performance reporting for accountability and improvement experiencesimilar challenges as Australia with indicator and data development issues.

There should not be an over concern with the technical issues surrounding the reporting ofsurvey data and associated error. As data collections improve, the capacity to makedefinitive findings will improve.

Investment in systems to manage data remains a priority for most governments and willsupport accountability, improvement and timeliness issues.

Measuring progress in education sits within a wider agenda of measuring well-being andsocietal progress. New approaches to multi-faceted indicators are now at the forefront ofnational and international statistical and political agendas and it is in our interests tocontinue to participate in these developments

Recommendations

Australian agencies and individuals should continue to monitor and engage in collaborativeprojects with overseas-based performance and accountability bodies to ensure the latest researchand experiences inform our approach.

In particular, there should be action in three critical areas.

Establishing a framework for the ongoing review of indicators and their further developmentdriven by a principle of relevance to 21st century learning environments and what we knowabout the nature of learning.

Investing in technology and systems to improve our capacity for timely performancereporting and efficient management and development of key collections.

Developing more systematic approaches to collaboration to increase the uptake of bestpractice and approaches to lifting performance at school and system levels.



References and further readingAfA Achievement for all Framework Programme at http://www.afa3as.org.uk/programmeaccessed on 13/02/13

Anderson, Jo Anne (2005) Accountability in Education. Booklet No.1 Education Policy Series.International Institute of Education Planning. UNESCO

Ark Ark Schools:Building Better Schools at http://www.arkschools.org/ accessed on 13/02/13

Center for Educational Research and Innovation, OECD. Schooling for Tomorrow Series(various) see Think Scenarios, Rethink Education (2006), OECD, July.

Center for Educational Research and Innovation, (2012) Connected Minds: Technology andToday’s Learners, OECD Publishing, July.

Department for Education (2012) Statement of Intent: Performance Tables July 2012 accessedon 17/12/12/ at http://www.education.gov.uk/schools/performance/

European Commission (2012) 2012 Education and Training Monitor. November

European Commission (2012) Rethinking Education and Supporting the Teaching Profession.

European Commission (2012) Europe 2020. Various publications.http://ec.europa.eu/europe2020/europe-2020-in-a-nutshell/index_en.htm

Frances, R. (2012) (Un)Satisfactory? Enhancing Life Chances by Improving 'Satisfactory'Schools. Royal Society of Arts at http://www.thersa.org/action-research-centre/education/social-justice/satisfactory-schools accessed on 13/02/13

Bill and Melinda Gates Foundation Measures of Effective Teaching athttp://www.gatesfoundation.org/united-states/Pages/measures-of-effective-teaching-fact-sheet.aspx accessed on 13/02/13

Masters, G. (2012) Measuring and Rewarding School Improvement. ACER, Aprilhttp://www.acer.edu.au/documents

OECD How’s Life: Better Life Index Index at http://www.oecdbetterlifeindex.org accessed on15 Feb 2013.

OECD (2010) The Nature of Learning: Using Research to Inspire Practice. CERI, OECDPublishing, September

OECD (2012) Education at a Glance 2012, OECD Publishing.http://www.oecd.org/edu/highlights.pdf

OECD (2012) Preparing Teachers and Developing School Leaders for the 21st Century -Lessons from around the world (Background Report for the International Summit on theTeaching Profession) at www.oecd.org/site



Royal Society of Arts RSA Family of Academies at http://www.thersa.org/action-research-centre/education/family-of-academies accessed on 13/02/13

Stiglitz, J.E., Sen, A., Fitoussi, JP.,(2009) Report of the Commission on the Measurement ofEconomic Performance and Social Progress. At www.stiglitz-sen-fitoussi.fr accessed on13/02/13

Stecher, B et al. (2010) Toward a Culture of Consequences. Performance-Based AccountabilitySystems for Public Services RAND Corporation

Appendix A. Logic diagramPerformance Reporting in Education 40

Appendix A. Logic diagram





Appendix A. Logic diagramPerformance Reporting in Education xli

Documents

Performance Reporting in Education · Performance Reporting in Education iii Table of contents Executive summary iv Highlights of the 2012 Churchill study tour iv Major learnings