Scottish Learning Festival

International StudiesTheir potential for the practitioner

Presenters

Pete Whitehouse – Head of schools analytical services unit

Audrey MacDougall – Principal Researcher: schools

An analysts view

International studies undertaken by the Scottish Government

• PISA – Programme for International Student Assessment

• PIRLS – Progress in International Reading Literacy Study

• TIMSS - Trends in International Mathematics and Science Study

The Assessment Quadrant

Source: circular no. 02 June 2005 Qualifications, Assessment and Curriculum Division

Why do we do these studies?

• Complements SSA and other national activities

• Contributes to right-hand side of assessment quadrant - external/summative

• Feedback can also contribute to external/formative

• Complements Scotland’s AifL agenda

International study activity Timetable 2006-08

2006

2007

2008

• PISA 2006 main study testing • PIRLS 2006 main study testing

• TIMSS 2007 main study testing• PIRLS 2006 reporting• PISA 2006 reporting

• PISA 2009 Field trial

PIRLS 2006

• 40 countries and 5 Canadian provinces took part

• 150 primary schools with one P5 class per school

• Pupils aged 9 and 10 (P5) • Testing took place in Spring 2006• Results due Nov 2007

PISA 2006

• 57 countries took part including all OECD countries

• Main test domain was science. Reading and maths minor domains

• 15 year old pupils, 110 schools with 35 pupils per school

• Testing took place in spring 2006• Report due in Dec 2007

TIMSS 2007

• More than 60 educational systems participate• Data in mathematics and science at fourth and

eighth grades (P5 and S2)• 150 schools with one class per school• Testing took place in Spring 2007• Results due December 2008

What can each study tell us?

• Attainment• Attitudes, aspirations, ambitions• Impact of socio-economic and background

factors• Information about the learning environment• The curriculum

AttainmentA number of countries perform significantly better than Scotland

PISA 2003

Science Maths Reading

Finland, Japan, Korea Finland, Korea, Netherlands Canada, Finland, Korea

TIMSS 2003 (Science)Singapore, Chinese Taipei, Rep of Korea, Hong Kong, Estonia, Japan, England, Hungary, Netherlands, Australia, Sweden, Slovenia, Lithuania

PIRLS 2001 (Reading)Sweden, Netherlands, England, Bulgaria, Latvia, Lithuania, Hungary, United States, Italy, Germany, Czech Republic

Attitudes, aspirations, ambitions

PISA• Interest and enjoyment• Motivation• Sense of belonging• Self-belief and confidence• Anxieties• Learning strategies• Behaviour and effort

TIMSS• Aspirations• Behaviour• Confidence• Liking school• Learning strategies

Self-confidence in learning science (S2) - TIMSS 2003

High Medium Low

Country % Mean score

% Mean score

% Mean score

International average

48 490 38 445 13 430

**Scotland ** 59 539 28 481 14 459

Norway 60 512 30 475 10 456

New Zealand 41 548 41 509 19 489

Japan 20 595 46 551 34 529

Anxiety in PISA 2003

-2.00

-1.50

-1.00

-0.50

0.00

0.50

1.00

1.50

2.00

All students Bottomquarter

Top quarter Females Males

Ind

ex m

easu

re

Canada

Finland

Japan

Korea

Netherlands

Scotland

OECD Average

Socio-economic and background factors

PISA• Socio-economic index of occupational status• Index of economic, social and cultural statusTIMSS• Home study and cultural resources• Parental background and home environmentPIRLS 2001• Parental involvement• Literacy resources at home

Impact of socio-economic factors PISA 2003

0.002.004.006.008.00

10.0012.0014.0016.0018.0020.00

Str

eng

th o

f im

pac

t o

f so

cio

-ec

on

om

ic f

acto

rs

470.00

480.00

490.00

500.00

510.00

520.00

530.00

540.00

550.00

Mea

n m

ath

s sc

ore

Economic factors

Maths score

Impact of socio-economic factors PISA 2003

500

505

510

515

520

525

530

535

540

545

550

11 11 12 14 17 18 19

Strength of socio economic relationship

Mea

n m

ath

s sc

ore

Scotland

N’lands

Canada

Finland

Japan

Korea

OECD average

Having a computer at home TIMSS 2003

0 20 40 60 80 100

Scotland

Japan

New Zealand

Norway

Co

un

try

Percentage

Have computer

No computer

497 445

524 489

558 529

516 475

Having a desk at home TIMSS 2003

0 20 40 60 80 100

Scotland

Japan

New Zealand

Norway

Co

un

trie

s

Percentage

Have deskNo desk

496

525 489

554 526

517 489

Early Home Literacy Activities PIRLS 2001

Countries High EHLA

Medium EHLA

Low EHLA

% students

Average scores

% students

Average scores

% students

Average scores

Scotland 82 546 16 529 2 -

England 83 578 14 546 3 513

Sweden 41 572 45 561 14 548

Norway 47 518 41 493 13 474

N’lands 55 566 37 561 8 555

Parents discuss child’s classroom reading work with him/her

PIRLS 2001Countries Almost

every day

Once or twice a week

Once or twice a month

% students

Average scores

% students

Average scores

% students

Average scores

Scotland 44 529 41 552 12 558

England 32 562 43 573 20 580

Sweden 23 551 40 563 29 573

Norway 23 494 42 503 25 509

N’lands 23 555 41 560 28 572

The learning environment

PISA 2003School climate

School policies and practices

Resources invested in education

TIMSS 2003Demographics

Home/school involvement

School resources

School climate

Classroom characteristics

PIRLS 2001DemographicsHome/school involvementSchool resourcesClassroom characteristicsAbsenteeism, bullying and safety

Student views of teacher support PISA 2003

Country Scotland %

Finland%

Korea% Canada%

Netherlands%

The teacher shows an interest in every student's learning

65 54 58 63 49

The teacher gives extra help when students need it

78 77 56 80 66

The teacher helps students with their learning

84 86 79 86 49

The teacher continues teaching until the students understand

74 61 40 71 60

The teacher gives students an opportunity to express opinions

53 62 49 62 54

Impact of teacher support PISA 2003

Emphasis on homework TIMSS 2003 science

0% 20% 40% 60% 80% 100%

Int average

Scotland

Korea

Japan

Netherlands

Australia

Sweden

Lithuania

Co

un

try

% of students

High

Medium

Low

Student experiences of incidences in school

PIRLS 2001

0

5

10

15

20

25

30

35

40

45

Country

Per

cen

tag

e o

f st

ud

ents

Stolen

Bullied

Hit/hurt

The curriculum

TIMSS 2003• Instructional time on maths and science• Emphasis on approaches and processes in the science

curriculum• Assessment

PIRLS 2001• Instructional time on reading• Emphasis on approaches and processes in the reading

curriculum• Assessment

Emphasis in the science curriculum TIMSS 2003

Nature of science

Integrating science

Impact on society

Impact of human on environment

Experience of different ethnic groups

Scotland N Y Y Y N

Korea N N Y Y NN

Japan YY Y Y Y NN

Netherlands N Y YY YY N

Australia YY N Y Y N

Sweden YY N N YY YY

Lithuania YY Y Y YY N

Methods of assessment PIRLS 2001

0

10

20

30

40

50

60

70

80

90

100

Multi-choicequestions

Shortquestions

Longquestions

Listen toreading aloud

Oralquestioning

Oral summary

Type of question

% o

f s

tud

en

ts

Int average

Scotland

England

Sweden

New Zealand

Norway

School level reporting

• Has not been previously undertaken as sample sizes too small for reliable attainment data

• Some elements of school level reporting may be introduced for further studies for some elements of the survey

How your school might conform to the overall findings for Scotland?

Detailed results

PISA

http://www.pisa.oecd.org/pages/0,3417,en_32252351_32236130_1_1_1_1_1,00.htmlPIRLS

http://pirls.bc.edu/pirls2006/index.htmlTIMSS

http://www.timss.org/