What do international assessments measure: PISA

Raymond J. Adams

Washington DC, May 30 2008

This paper is intended to promote the exchange of ideas among researchers and policy makers.  The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.

2

What is PISA?

Programme for International Student Assessment

Funded by the Organisation for Cooperation and Development (OECD)

International data collection has been managed and led by ACER since its inception

3

Developing PISA 1995: OECD countries sought

• comparative assessment of yield of education systems

• a basis for defining educational standards cross-nationally

1997: OECD countries adopted strategy and financial framework to...• monitor a broad range of curricular and cross-

curricular outcomes...…within a comparative framework established

collaboratively by countries 1998: Contract let

4

Participation 2000

• OECD 28, non-OECD 4• PISA+, 11 non-OECD 170,000 students

2003• OECD 30, non-OECD 12 250,000 students

2006• OECD 30, non-OECD 27• 14365 schools• 398,750 students (representing a

cohort of over 22 million) 2009

• OECD 30, non-OECD 35 (plus Scotland)

5Plus: Dubai, Kazakhstan, Moldova

6

PISA survey cycle

2000

Reading literacy

Mathematical literacy

Scientific literacy

Background information

7

PISA survey cycle

2000

Reading literacy

Mathematical literacy

Scientific literacy

Cross Curricular Comp.

Reading habits & attitudes

Background information

eg: self-regulation, metacognitive

eg: Reading Activities, motivation, interest, engagement

8

PISA survey cycle

2000 2003

Reading literacy

Mathematical literacy

Scientific literacy

Cross Curricular Comp.

Habits & attitudes Reading Maths

Problem solving

Background information

9

PISA survey cycle

2000 2003 2006

Reading literacy

Mathematical literacy

Scientifi c literacy

Cross Curricular Comp.

Habits & attitudes Reading Maths Science

Problem solving

Computer-Based Science

Background information

10

PISA survey cycle

2000 2003 2006 2009

Reading literacy

Mathematical literacy

Scientific literacy

Cross Curricular Comp.

Habits & attitudes Reading Maths Science Reading

Problem solving

Computer-Based Science Reading

Background information

11

Separating policy and science

Two consortia commissioned to run PISA 2009

Core A

Australian Council for Educational Research

Unit for the Analysis of Educational Systems and Practices. University of Liege, Belgium

Westat, USA

National Institute for Educational Research, Japan

The German Institute for International Educational Research

cApStAn Linguistic Quality Control

12

Separating policy and science

Core B• Netherlands National Institute for

Educational Measurement (CITO)

•University of Jyvaskyla•University of Twente•French Ministry of Education

• Sets policy objectives and priorities

• Sets budget• Monitors adherence to policy

objectives• Guides analysis and reporting of

results• Establishes quality standards• Adjudicates on breaches of

quality standards

Consortia are commissioned to run PISA

15

Innovations in PISA International, intergovernmental

ownership• primary focus on public policy issues• sustained commitment• collaborative development

Age not grade-based target population Substance

• not common denominator of national curricula

• not whether students have learned• whether they can use what they have

learned• criterion-based reporting

16

PISA’s Literacy Orientation Projective (rather than reflective)

approach to instrument development

Not specifically related to curriculum• It is about authentic application of

what is learned• In contrast to assessments that are

like instructional exercises A broadening, not narrowing notion

17

Literacy Orientation

Domain definitions include two components:• each domain defined, in terms of

knowledge and skills needed in adult life, not merely in terms of mastery of the school curriculum.

• emphasis is placed on the mastery of processes, the understanding of concepts and the ability to function in various situations related to real life.

18

Basic Methodology Target population: 15-year-olds in school

• Students are spread over classes and grades Target sample sizes of 5250

• 150 schools of 35 students• 50 schools required for national sub-entities

Rotated two-hour written tests• 13 booklets• 3 domains (two minor and one major)• About 200 items in total• About 11 responses per item in each school• Students respond to an average of 13 minor

domain items and 30 major domain items• Some students are not assessed in minor

domains

19

Basic Methodology Student questionnaires School questionnaires Parent questionnaires No teacher questionnaires

• Students from multiple classrooms

• Focus is not limited to taught subjects

20

Focus of the Results Yield for well-defined populations and

domains• Reading, Mathematics, and Science every three

years• Sub-domains every nine years

National sub-group performance Profiles encompassing broad domain

definitions• Literacy orientation, habits, behaviours, attitudes

Trends Focus on comparisons of equity

• Eg Gender, minority status, socio-economic status Relations between national structures

outcomes• Eg Tracking, Assessment, funding and Decision

making policies

21

Shall I slice the pizza into four or eight pieces?

Make it four ! I could never eat eight !