The role of assessment data in Austria

© Schreiner, February 2015

Education Governance: The Role of Data in Austria

Education Governance:

The Role of Assessment Data

in Austria

Governing Complex Education Systems

Tallinn,12-13 February 2015

Dr. Claudia Schreiner

Director

BIFIE – Federal Institute for Education Research,

Innovation & Development of the Austrian School System

Salzburg, Austria



The Austrian Education System



What data are collected

and for what purpose?

System monitoring: international

assessments

OECD-PISA

IEA-PIRLS(IEA-TIMSS)






assessments

Quality development in schools:

national standards assessments;

census with focus on the school

level, but includes also system level

data for monitoring purposes

National Standards Assessments






assessments

Quality development in schools:

national standards assessments;

census with focus on the school

level, but includes also system level

data for monitoring purposes

Self evaluation tools: centrally

provided test sets for self

evaluation; voluntary for teacher

use with no central data collection

7

6

5

3

9



National Standards AssessmentsMain Features



Starting point: Educational Standards

Intended outcomes defined through mandatory

Performance Standards in 2009

In German and Mathematics (grade 4) and German,

Mathematics and English (grade 8)

Baseline tests 2009 (grade 8) und 2010 (grade 4)

– samples of about 10.000 students per grade

National Standards Assessments as a census since

2012 with feedback to all schools and teachers as a basis

for quality development in schools

One assessment per year – one full cycle takes 5 years



• National Report (1)

• Reports for Provinces (9)

• feedback to the regional school supervision

• school reports (1400 – 3100; online)

• feedback to teachers (~ 4900 per grade; online)

• individual student feedback (~75000 per grade; online)

system

monitoring

Target Groups for Feedback

quality

development

in schools



• Census: every school and every teacher gets individual

results to raise local responsibility to work with the data

• Focus on instruction outcomes: students cannot be

identified in the teacher and school reports (are included

as individual data points, but without reference to names or

IDs)

• Quality assured external feedback through central data

collection, coding, processing and analysis

Main Purpose:

Quality Development in Schools



Possibilities of comparison

in school/teacher feedback

Performance of school/teaching

group in comparison to …

direct comparisons … all Austrian students/schools

fair comparisons … schools/teaching groups working

under similar conditions

criterion-referenced

comparisons

… a centrally defined criterion

(proficiency levels)

= educational standards in the

form of achievement outcomes



FEEDBACKfrom national standards

assessments

Competency-oriented teaching through evidence

based, regional school development

• professional self concept

• attitude towards evaluation

• expertise

• readiness to cooperate

school principal

reflecting on

the results

acting on the

resultsabsorbing the

feedback

“new” school inspection with coaching

– agreement of objectives and

development plans

feedback

facilitators

systems of development external support

teacher training and development

school characteristics• climate of evaluation und cooperation

• engagement / involvement of parents

• culture of open feedback

• innovative orientation

Wiesner & Schreiner, 2014 based on Helmke & Hosenfeld, 2006



Our experiences so far …



Evaluation of the feedback system and

experiences

positive feedback on content, structure, scope and comprehensibility of the reports

mostly open communication in schools

many schools managed to derive measures within the first 6 months

most difficult phase is the transition from feedback to action.



Main challenges and success factors

One main challenge for teachers is accepting the results as an outcome of their teaching and use them in order to reflect upon their teaching.

This data system will only make a real change if the results are used by schools and are handled in a reflective way.

How the data is handled by other players, is crucial for the acceptance by schools and teachers. Trust is a decisive factor!

There are different players involved in the process, who need to collaborate in a productive way.



Résumé

A comprehensive assessment and feedback system has been introduced in a complex system which lacks a strong tradition in evaluation.

The first round of assessments in central subjects as a census in grades 4 and 8 has started 2012.

The feedback system generally gets positive feedback concerning the organisation of the assessment, the feedback reports and the willingness of school staff to engage in working on the basis of the results.

Translating the information of the feedback reports into specific measures is being found difficult and needs special support – and requires a long learning process.



Thank you!

Dr. Claudia Schreiner

BIFIE – Federal Institute for

Education Research,

Innovation & Development

Salzburg, Austria

www.bifie.at



Aus: Schulbericht

(Rückmeldung an

die Schulleitung

und Schulpartner)

MathematikStreuung an der

Schule



Aus: Schulbericht

(Rückmeldung an

die Schulleitung

und Schulpartner)

MathematikErgebnis nach

Geschlecht



Mathematikkompetenz im fairen

VergleichÄhnliche Rahmenbedingungen in

Bezug auf:

Merkmale des Schulstandorts

(Gemeindegröße, Urbanisierungs-

grad, Entfernung zur nächsten

AHS)

Merkmale der Schule

(Schulgröße, Schulart, Privatschule,

Anteil ausgenommener

Schüler/innen)

Merkmale der Schülerschaft

(Anteil an Mädchen/Burschen,

Anteil an Kindern mit nichtdeutscher

Erstsprache, Anteil an Kindern mit

Migrationshintergrund, mittlerer

Sozialstatus)



Mathematikkompetenz im fairen

Vergleich



Mathematikkompetenz nach Kompetenzstufen



Aus: Schulbericht

(Rückmeldung an

die Schulleitung

und Schulpartner)

Mathematik:Verteilung auf die

Kompetenzstufen



Rückmeldungaus der Überprüfung der

Bildungsstandards

Kompetenzorientiertes Unterrichten durch

datengestützte, regionale Schulentwicklung durch flächendeckende Erhebung und Rückmeldung

• Professionelles Selbstverständnis

• Akzeptanz von Evaluation

• Vorwissen/Expertise

• Kooperationsbereitschaft

Schulleitung

• Suche nach

Erklärungen

• Analyse der

Ergebnisse

Reflexion

• Verbesserung

von Unterricht

• Schul-

entwicklung

AktionRezeption

• Verständnis

• Erkennen von

Stärken und

Schwächen

Schulqualität Allgemeinbildung (SQA) -

Zielvereinbarung und Entwicklungspläne

Rückmelde-

moderatoren

Entwicklungsberatung in Schulen (u.a. EBIS)

Lehreraus- und -weiterbildung

SchulischeRahmenbedingungen• Evaluations- und Kooperationsklima

• gelebte Schulpartnerschaft

• Feedbackkultur

• Innovative Orientierung

Modell Wiesner/Schreiner © 2014



Rückmeldemoderation

• In allen Bundesländern wurde ein Pool an RMMs

ausgebildet.

• Ausbildung in Kooperation zwischen BIFIE und Päd.

Hochschulen

• Grundausbildung: 2 * 2 Präsenztage

• Einsatztraining (jährlich): 1 Tag je Fach/Schulstufe

• RMMs sollen den Prozess der Rezeption unterstützen

• Umfang: max. 2 Kontakte

• Gespräch mit Schulleitung (fixer Bestandteil)

• ggf. zweiter Besuch für Arbeit mit gesamtem Kollegium, Fachteam

oder den betroffenen Lehrer/innen



Rückmeldungaus der Überprüfung der

Bildungsstandards

Kompetenzorientiertes Unterrichten durch

datengestützte, regionale Schulentwicklung durch flächendeckende Erhebung und Rückmeldung

• Professionelles Selbstverständnis

• Akzeptanz von Evaluation

• Vorwissen/Expertise

• Kooperationsbereitschaft

Schulleitung

• Suche nach

Erklärungen

• Analyse der

Ergebnisse

Reflexion

• Verbesserung

von Unterricht

• Schul-

entwicklung

AktionRezeption

• Verständnis

• Erkennen von

Stärken und

Schwächen

Schulqualität Allgemeinbildung (SQA) -

Zielvereinbarung und Entwicklungspläne

Rückmelde-

moderatoren

Entwicklungsberatung in Schulen (u.a. EBIS)

Lehreraus- und -weiterbildung

SchulischeRahmenbedingungen• Evaluations- und Kooperationsklima

• gelebte Schulpartnerschaft

• Feedbackkultur

• Innovative Orientierung

Modell Wiesner/Schreiner © 2014



What data are collected and for what purpose?

System monitoring: international assessments



The Austrian School System –

international Assessments

OECD-PISAin M – R – S3-year cycle

IEA-PIRLS (& TIMSS)in R – M – S

4/5-year cycle





Quality development in schools: national standards

assessments; census with focus on the school level, but

includes also system level data for monitoring purposes




data for quality development in schools

National Standards Assessment in German – Mathematics

National Standards Assessmentin German – Mathematics – English





Quality development in schools: national standards

assessments; census with focus on the school level, but

includes also system level data for monitoring purposes

Self evaluation tools: centrally provided test sets for self

evaluation; voluntary for teacher use with no central data

collection




self evaluation tools

Informal competency assessment 3in German – Mathematics

Informal competency assessment 6In German – Mathematics – English

Informal competency assessment 7in German – Mathematics – English – Science

competency check 9in Mathematics, English (German will follow)

competency check 5Mathematics in trial (German will follow)



Domain-orientated Test Design

The cycle is now being changed into a 5-year cycle with only one assessment per year.



School report: overview of content

Information about school results:

compared to AUT

compared to the defined goals (in the regulation)

compared to other schools with similar basicconditions („fair comparison“)

Areas of conflict:

Educational Standards apply to all students – and

therefore to all schools

Conditions of schools vary widely



Evaluation of the feedback reports and

procedures for schools and teachers (1)

positive feedback on content, structure, scope and comprehensibility of the reports

communication:

in most schools open communication amongst teachers

with the whole teaching body, the subject matter groups or bilaterally with individual teachers

but low interest by parent representatives

~ 40 % of headmasters asked for feedback facilitators, in most cases in order to validate their own interpretation of the school report



Evaluation of the feedback reports and

procedure for schools and teachers (2)

The use of feedback reports:

Many schools initiated measures in response to feedback reports within the first 6 months.

Action was mostly taken with the purpose of quality development (in contrast to quality assurance).

Most measures derived concerned the development of subject matter teaching and didactics.

Headmasters described their teachers mostly as ranging from positive to playing an active role in taking action.



Educational Standards –

What is special about the Austrian Model?

School-specific development of instruction

Mandatory educational standards

Survey of all students and schools

Feedback to all students, teachers, schools

Feedback facilitators („Rückmeldemoderation“)

Stimulation of quality development

= Austrian Model

Strong linkage of monitoring and practical school andinstructional development

Strengthening of schools, school management and regional quality development



Different players and roles

Government & education ministry: determination of goals (educational standards) and recipient of nation-wide results

Schools and teachers: recipients of external feedback and responsible for quality development

School authorities: support school development work and check on it (possible role conflict)

BIFIE: responsible for assessment, data collection, processing and feedback



Herzlichen Dank

für Ihre

Aufmerksamkeit!

www.bifie.at



OECD-PISA

IEA-PIRLS(IEA-TIMSS)

International Assessments: system monitoring

National Standards Assessments

National Standards Assessments: quality development in schools

7

X Self evaluation tools and competency checks in grades X

6

5

3

9

What data are collected,

at what points, and for what purpose?