10 things policymakers should know about learning … · OECD EMPLOYER BRAND Playbook 1 10 things policymakers should know about learning goals and assessment Andreas Schleicher St

OECD EMPLOYER BRAND Playbook

1

10 things policymakers should know about learning goals and assessment

Andreas Schleicher St. Petersburg, May 2014

2 2 Benchmark globally

Lesson 1 In a global economy, the benchmark for educational success is no longer solely

improvement by national standards, but the best performing education systems internationally

A world of change – higher education

0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

1995

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)

Tertiary-type A graduation rate (%)

Cost

per

stu

dent

Graduate supply


0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

1995

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)

Tertiary-type A graduation rate (%)

Cost

per

stu

dent

Graduate supply

United States


0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2000

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)

Tertiary-type A graduation rate

United Kingdom


0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2001

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2002

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2003

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2004

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2005

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2006

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2007

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2008

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2009

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)



0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2010

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)


Iceland

Poland

UK


0

5,000

10,000

15,000

20,000

25,000

30,000

0 10 20 30 40 50 60 70

2010

Expen

diture

per

stu

den

t at

ter

tiar

y le

vel (U

SD)


US

17 17 Don’t sacrifice validity gains for efficiency gains

Lesson 2

The kinds of things that are easy to teach and test are also easy to digitise,

automate and outsource

35

40

45

50

55

60

65

70

1960 1970 1980 1990 2000 2006 2009

Routine manual

Nonroutine manual

Routine cognitive

Nonroutine analytic

Nonroutine interpersonal

Mean task input in percentiles of 1960 task distribution

18 Changes in the demand for skills Trends in different tasks in occupations (United States)

Source: Autor, David H. and Brendan M. Price. 2013. "The Changing Task Composition of the US Labor Market: An Update of Autor, Levy, and Murnane (2003)." MIT Mimeograph, June.

19 19 Monitor excellence and equity jointly

Lesson 3

The false choice between equity and excellence

Singapore

Hong Kong-China Chinese Taipei Korea

Macao-China Japan Liechtenstein Switzerland

Netherlands Estonia Finland Canada Poland Belgium Germany Viet Nam

Austria Australia Ireland Slovenia Denmark New Zealand

Czech Republic France United Kingdom Iceland Latvia Luxembourg Norway

Portugal Italy Spain Russian Fed. Slovak Republic United States Lithuania Sweden Hungary Croatia

Israel

Greece Serbia Turkey

Romania Bulgaria U.A.E. Kazakhstan Thailand

Chile Malaysia Mexico

410

420

430

440

450

460

470

480

490

500

510

520

530

540

550

560

570

580Mean score

High mathematics performance

Low mathematics performance

… Shanghai-China performs above this line (613)

… 12 countries perform below this line

Average performance of 15-year-olds in

Mathematics Fig I.2.13

Socially equitable distribution of learning

opportunities

High mathematics performance

Low mathematics performance

Average performance of 15-year-olds in

mathematics

Strong socio-economic impact on student

performance

Singapore

Hong Kong-China Chinese Taipei Korea

Macao-China Japan Liechtenstein Switzerland

Netherlands Estonia Finland Canada Poland Belgium Germany Viet Nam

Austria Australia Ireland Slovenia Denmark New Zealand

Czech Republic France United Kingdom Iceland Latvia Luxembourg Norway

Portugal Italy Spain Russian Fed. Slovak Republic United States Lithuania Sweden Hungary Croatia

Israel

Greece Serbia Turkey

Romania Bulgaria U.A.E. Kazakhstan Thailand

Chile Malaysia Mexico

Australia Austria

Belgium Canada

Chile

Czech Rep. Denmark

Estonia

Finland

France

Germany

Greece

Hungary

Iceland Ireland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US

Singapore

Hong Kong-China Chinese Taipei

Macao-China Liechtenstein

Viet Nam

Latvia

Russian Fed. Lithuania

Croatia

Serbia Romania

Bulgaria United Arab Emirates Kazakhstan

Thailand Malaysia

02468101214161820222426

2012 Shanghai-China


opportunities


performance

Australia Austria

Belgium Canada

Chile

Czech Rep. Denmark

Estonia

Finland

France

Germany

Greece

Hungary

Iceland Ireland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US

Australia Austria Belgium Canada Chile Czech Rep. Denmark Estonia Finland France Germany Greece Hungary Iceland Ireland Israel Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Rep. Slovenia Spain Sweden Switzerland Turkey UK US

2012


opportunities


performance

Australia Austria

Belgium Canada

Chile

Czech Rep. Denmark

Estonia

Finland

France

Germany

Greece

Hungary

Iceland Ireland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US


25 25 Spend wisely to make a difference

Lesson 4

Not more money but better spending choices make a difference

-10

-5

0

5

10

15

Port

ugal

Spai

n

Swit

zerl

and

Belg

ium

Kore

a

Luxe

mbo

urg

Germ

any

Gree

ce

Japa

n

Aus

tral

ia

Uni

ted

King

dom

New

Zea

land

Fran

ce

Net

herl

ands

Den

mar

k

Ital

y

Aus

tria

Czec

h Re

publ

ic

Hun

gary

Nor

way

Icel

and

Irel

and

Mex

ico

Finl

and

Swed

en

Uni

ted

Stat

es

Pola

nd

Slov

ak R

epub

lic

Salary as % of GDP/capita Instruction time 1/teaching time 1/class size

Contribution of various factors to upper secondary teacher compensation costs, per student as a percentage of GDP per capita (2004)

Percentage points

Difference with OECD average

Australia Austria

Belgium Canada

Chile

Czech Rep. Denmark

Estonia

Finland

France

Germany

Greece

Hungary

Iceland Ireland

Israel

Italy

Japan

Korea

Luxembourg

Mexico

Netherlands

New Zealand

Norway

Poland

Portugal

Slovak Rep.

Slovenia

Spain Sweden

Switzerland

Turkey

UK

US


Singapore

Shanghai

Singapore

2003 - 2012

28 28 Keep track of ‘added value’

Lesson 5 The country where students go to class matters more than what social class students come from

School performance and socio-economic background: Brazil 29

Advantage PISA Index of socio-economic background Disadvantage

Stu

dent

per

form

ance

Brazil

30

Vietnam

Desempeño del colegio y nivel socioeconómico

Ventaja Índice de PISA de nivel socioeconómico Desventaja

Des

empe

ño d

e lo

s es

tudi

ante

s

School performance and socio-economic background: Brazil 31

Advantage PISA Index of socio-economic background Disadvantage

Stu

dent

per

form

ance

Brazil

32 32 Resilience in education PISA performance by decile of social background

300

325

350

375

400

425

450

475

500

525

550

575

600

625

650

675

Mex

ico

Chile

Gre

ece

Norw

aySw

eden

Icel

and

Isra

elItal

yU

nited

Sta

tes

Spai

nD

enm

ark

Luxe

mbourg

Aust

ralia

Irel

and

United

Kin

gdom

Hungar

yCan

ada

Finla

nd

Aust

ria

Turk

eyLi

echte

nst

ein

Cze

ch R

epublic

Esto

nia

Port

ugal

Slove

nia

Slova

k Rep

ublic

New

Zea

land

Ger

man

yN

ether

lands

Fran

ceSw

itze

rlan

dPola

nd

Bel

giu

mJa

pan

Mac

ao-C

hin

aH

ong K

ong-C

hin

aKore

aSi

ngap

ore

Chin

ese

Taip

eiSh

anghai

-Chin

a

Source: PISA 2012

33

33 Provide a data-rich school environment to combat inequities

National and school

data

School-based

strategies for

analysis

Data for supporting

learning

34 34 Le

sson

s fro

m h

igh

perfo

rmer

s

Catching up with the top-performers

Low impact on outcomes

High impact on outcomes

Low feasibility High feasibility

Money pits

Must haves

Low hanging fruits

Quick wins

35 35 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins

Commitment to universal achievement

Gateways, instructional systems

Capacity at point of delivery

Incentive structures and accountability

Resources where they yield most

A learning system Coherence

36 36 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 6: Believe that all children can achieve Universal educational standards and personalization as

the approach to heterogeneity in the student body… … as opposed to a belief that students have different

destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity

Clear articulation who is responsible for ensuring student success and to whom

37 37 Le

sson

s fro

m h

igh

perfo

rmer

s

37

37 High expectations for all students

United States

Poland

Hong Kong-China

Brazil

New Zealand

Greece

Uruguay

United Kingdom

Estonia Finland

Albania

Croatia

Latvia

Slovak Republic Luxembourg

Germany

Lithuania

Austria

Czech Republic

Chinese Taipei

France Thailand

Japan

Turkey Sweden

Hungary Australia

Israel

Canada

Ireland Bulgaria

Jordan

Chile

Macao-China

U.A.E.

Belgium Netherlands

Spain

Argentina

Indonesia

Denmark

Kazakhstan

Peru

Costa Rica

Switzerland

Montenegro

Tunisia

Iceland

Slovenia

Qatar

Singapore

Portugal

Norway

Colombia

Malaysia

Mexico

Liechtenstein

Korea

Serbia

Russian Fed.

Romania

Viet Nam

Italy

Shanghai-China

R² = 0.36

300

350

400

450

500

550

600

650

-0.60 -0.40 -0.20 0.00 0.20 0.40 0.60 0.80 1.00 1.20

Mea

n m

athe

mat

ics

perf

orm

ance

PISA mean index of mathematics self-efficacy

OEC

D a

vera

ge

Countries where students have stronger beliefs in their abilities perform better in mathematics

Source: PISA 2012

Perceived self-responsibility for failure in mathematics

Percentage of students who reported "agree" or "strongly agree" with the following statements:

0 20 40 60 80 100

I’m not very good at solving mathematics problems

My teacher did not explain the concepts wellthis week

This week I made bad guesses on the quiz

Sometimes the course material is too hard

The teacher did not get students interested inthe material

Sometimes I am just unlucky

%

Russian Federation United States OECD average

Fig III.3.6 38

39

39

39 A continuum of support

Make learning central, encourage engagement and responsibility

Be acutely sensitive to individual differences

Provide continual assessment with formative feedback

Be demanding for every student

Ensure that students feel valued and included and learning is collaborative

40 40 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 7: Have clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems Well established delivery chain through which

curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved)

High level of metacognitive content of instruction …

41 41 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 8: Build capacity at the point of delivery Attracting, developing and retaining high quality

teachers and school leaders and a work organisation in which they can use their potential

Instructional leadership and human resource management in schools

Keeping teaching an attractive profession System-wide career development …

42 42 Le

sson

s fro

m h

igh

perfo

rmer

s

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

3

Luxe

mbo

urg

Jord

anTh

aila

ndTu

rkey

Sha

ngha

i-Chi

naIs

rael

Col

ombi

aP

eru

Chi

leN

ethe

rland

sM

exic

oG

erm

any

Vie

t Nam

Rus

sian

Fed

.U

rugu

ayN

orw

ayK

azak

hsta

nIn

done

sia

Bel

gium Ita

lyM

alay

sia

Aus

tralia

Bra

zil

Icel

and

U.A

.E.

Sin

gapo

reN

ew Z

eala

ndK

orea

Sw

itzer

land

Est

onia

Mac

ao-C

hina

Cos

ta R

ica

OE

CD

ave

rage

Sw

eden

Arg

entin

aTu

nisi

aA

ustri

aQ

atar

Irela

ndC

hine

se T

aipe

iFr

ance

Den

mar

kU

nite

d K

ingd

omH

ong

Kon

g-C

hina

Alb

ania

Japa

nC

anad

aS

lova

k R

epub

licLa

tvia

Gre

ece

Uni

ted

Sta

tes

Cze

ch R

epub

licC

roat

iaFi

nlan

dM

onte

negr

oR

oman

iaH

unga

ryLi

thua

nia

Slo

veni

aS

pain

Ser

bia

Por

tuga

lB

ulga

riaP

olan

d

Mea

n in

dex

Mean index Top quarter of this index Bottom quarter of this index

Teacher shortage Fig IV.3.5

43

43

43 Prepare for work in disadvantaged schools

Preparation

Prepare teachers for

work in disadvantage

Provide mentoring in disadvantage

Improve working

conditions

Career and financial

incentives

• Reinforce initial teacher training including curriculum content for disadvantage

• Strengthening diagnostic capacity • Include practical field experience

• Both new and experienced teachers benefit

• Pedagogical and relational strategies

44 44 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 9: Align autonomy with accountability Aligned incentive structures

For students How gateways affect the strength, direction, clarity and nature of the

incentives operating on students at each stage of their education Degree to which students have incentives to take tough courses and study hard Opportunity costs for staying in school and performing well

For teachers Make innovations in pedagogy and/or organisation Improve their own performance

and the performance of their colleagues Pursue professional development opportunities

that lead to stronger pedagogical practices

A balance between vertical and lateral accountability Effective instruments to manage and share knowledge and spread

innovation – communication within the system and with stakeholders around it

A capable centre with authority and legitimacy to act

45 45 Le

sson

s fro

m h

igh

perfo

rmer

s

45

45 Align autonomy with accountability

The question is not how many charter schools you have but how you enable every teacher to assume charter-like autonomy

46 46 Le

sson

s fro

m h

igh

perfo

rmer

s

46

46

Hong Kong-China

Brazil

Uruguay

Albania

Croatia

Latvia

Lithuania

Chinese Taipei

Thailand Bulgaria

Jordan

Macao-China

UAE Argentina

Indonesia

Kazakhstan

Peru

Costa Rica Tunisia

Qatar

Singapore

Colombia

Malaysia

Serbia

Romania

Viet Nam

Shanghai-China

USA

Poland

New Zealand

Greece

UK

Estonia

Finland

Slovak Rep.

Luxembourg

Germany Austria

Czech Rep.

France

Japan

Turkey

Sweden

Hungary Australia

Israel

Canada

Chile

Belgium Netherlands

Spain Denmark

Switzerland

Iceland

Slovenia Portugal

Norway

Korea

Italy

R² = 0.13

300

350

400

450

500

550

600

650

-1.5 -1 -0.5 0 0.5 1 1.5

Mat

hem

atic

s pe

rfor

man

ce (s

core

poi

nts)

Index of school responsibility for curriculum and assessment (index points)

Countries that grant schools autonomy over curricula and assessments tend to perform better in mathematics

Source: PISA 2012

Schools with more autonomy perform better than schools with less autonomy in systems with more collaboration

Teachers don't participate inmanagement

Teachers participate inmanagement455

460

465

470

475

480

485

Less school autonomy

More school autonomy

Score points

School autonomy for resource allocation x System's level of teachers participating in school management Across all participating countries and economies

Fig IV.1.17

Schools with more autonomy perform better than schools with less autonomy in systems with more accountability arrangements

School data not public

School data public464

466

468

470

472

474

476

478



Score points

School autonomy for curriculum and assessment x system's level of posting achievement data publicly

Fig IV.1.16

No standardisedmath policy

Standardised mathpolicy455

460

465

470

475

480

485



Schools with more autonomy perform better than schools with less autonomy in systems with standardised math policies

Score points

School autonomy for curriculum and assessment x system's extent of implementing a standardised math policy (e.g. curriculum and instructional materials)

Fig IV.1.16

0 20 40 60 80 100

Written specification of the school's curriculum andeducational goals

Written specification of student-performance standards

Systematic recording of data, including teacher andstudent attendance and graduation rates, test results…

Internal evaluation/self-evaluation

External evaluation

Written feedback from students (e.g. regarding lessons,teachers or resources)

Teacher mentoring

Regular consultation with one or more experts over aperiod of at least six months with the aim of improving…

Implementation of a standardised policy for mathematics

%

Percentage of students in schools whose principal reported that their schools have the following for quality assurance and improvement:

Singapore OECD average

Quality assurance and school improvement Fig IV.4.14 50

51

51

51 Square school choice with equity

Financial incentives

for schools

Assistance for disadvantaged

parents

Controlled choice

Financial incentives

Inform parents

Foster collaboration

among teachers and

schools

Use student and school

assessments

52 52 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







Lesson 10: Invest resources where they can make most of a difference Alignment of resources with key challenges (e.g.

attracting the most talented teachers to the most challenging classrooms)

Effective spending choices that prioritise high quality teachers over smaller classes

53 53 Align the resources with the challenges

Hong Kong-China

Brazil Uruguay

Croatia

Latvia

Chinese Taipei

Thailand Bulgaria

Jordan

Macao-China

UAE Argentina

Indonesia Kazakhstan

Peru

Costa Rica Montenegro

Tunisia

Qatar

Singapore

Colombia

Malaysia Serbia

Romania

Viet Nam

Shanghai-China

USA

Poland

New Zealand

Greece

UK

Estonia

Finland Slovak Rep.

Luxembourg

Germany Austria France

Japan

Turkey Sweden Hungary Australia Israel

Canada Ireland

Chile

Belgium

Spain Denmark

Switzerland

Iceland

Slovenia

Portugal Norway

Mexico

Korea

Italy

R² = 0.19

300

350

400

450

500

550

600

650

700

-0.500.511.5

Mat

hem

atic

s pe

rfor

man

ce (s

core

poi

nts)

Equity in resource allocation (index points)

Greater equity Less equity

Adjusted by per capita GDP

Countries with better performance in mathematics tend to allocate educational resources more equitably

Source: PISA 2012

54 54 Adequate resources to address disadvantage

Disadvantaged schools reported more teacher shortage

Advantaged schools reported more teacher shortage

-0.5

-0.3

-0.1

0.1

0.3

0.5

0.7

0.9

1.1

1.3

1.5

Kor

eaEs

toni

aIs

rael

Latv

iaSl

oven

iaIta

lyPo

land

Sing

apor

eA

rgen

tina

Net

herla

nds

Port

ugal

Col

ombi

aFr

ance

Finl

and

Tuni

sia

Mac

ao-C

hina

Spai

nG

reec

eSw

itzer

land

Nor

way

Rus

sian

Fed

.Ja

pan

Aus

tria

Mon

tene

gro

Cro

atia

Can

ada

OEC

D a

vera

geG

erm

any

Den

mar

kH

unga

ryU

nite

d K

ingd

omLu

xem

bour

gH

ong

Kon

g-C

hina

Bel

gium

Icel

and

Viet

Nam

Irela

ndU

nite

d St

ates

Chi

leC

zech

Rep

ublic

Serb

iaTu

rkey

Mex

ico

Indo

nesi

aU

rugu

aySh

angh

ai-C

hina

Slov

ak R

epub

licSw

eden

Bra

zil

New

Zea

land

Aus

tral

iaC

hine

se T

aipe

i

Mea

n in

dex

diffe

renc

e

Difference between socio-economically disadvantaged and socio-economically advantaged schools

A shortage of qualified teachers is more of concern in disadvantaged schools

55

55

55 Reduce tracking and grade repetition

Both vertical and horizontal stratification hurt equity

56 56 Le

sson

s fro

m h

igh

perfo

rmer

s




Money pits

Must haves

Low hanging fruits

Quick wins







A final thought Alignment of policies

across all aspects of the system Coherence of policies

over sustained periods of time Consistency of implementation Fidelity of implementation

(without excessive control)

CAN

57 57 Le

sson

s fro

m h

igh

perfo

rmer

s

57

57 Thank you

Find out more about our work at www.oecd.org – All publications – The complete micro-level database

Email: [email protected] Twitter: SchleicherEDU

and remember: Without data, you are just another person with an opinion

Documents

10 things policymakers should know about learning … · OECD EMPLOYER BRAND Playbook 1 10 things policymakers should know about learning goals and assessment Andreas Schleicher St