CIES 2016Vancouver, Canada

9 March 2016

From assessment to actions: Impact of student assessment data on educational policy reform for

sustainable future

Learning for All: Using assessment data for policy and planning in Asia

Laura Paviot and Mioko Saito, IIEP

Since 1994 Collaboration with SACMEQ

1989-1994NLSA Zimbabwe

2000-2003NLSA Viet Nam

Since 2014Learning Portal, e-Forum

Since 2014Collaboration with

NEQMAP & SEA-PLM

Since 2010Collaboration with PASEC

Background of this panel

1969 / 1978 / 1989 Seminars on QoE

2004 Policy Forum on C-N Ass’t

2016 MOOC

Since 1991 Specialized Training Course

1969 / 1978 / 1989 Seminars on QoE

Since 2014Collaboration with

NEQMAP & SEA-PLM

Since 2010Collaboration with PASEC

Presentation Outline

• Trends in learning assessments• Researchers vs. policy-makers • “Enlightenment model” as research utilization• Categories of actions from assessment

evidence• Illustrations: Vietnam and South Korea• MOOC survey

Frequency of National Assessments: Pre and Post Dakar

Source: UNESCO GMR (2015)

Learning Assessment Capacity Index (LACI)

2010-2015

Source: UIS (2016)

National, primary, 2010-2015 National, secondary, 2010-2015Trend data for primary & secondary, 2010-2015International assessment, 2010-2015Regional assessment, 2010-2015

Two “communities”: Policy Makers vs. Researchers

Policy Makers ResearchersProblem Definition

Deal with complex real world social problems

Develop questions from theoretical frameworks

Culture Targets, ways and means Generalization and explanation

“what” results “how” and “why” resultsUse procedural and legal jargon

Use restricted technical senses

Role & Accountability

Risk-avoidance New and unexpected

Time frame Quick solutions Need time – internal quality criteria

Source: Cross et al (2000)

Policy research cycle

Interpretation and reporting

Datacollection

and analysis

Specific research questions

Programme implementation

Source: Saito (1999)

Enlightenment Model• Major way with research effect on policy• Role of researchers and educational planners

as “information brokers”• Research ‘enlightens’ policy makers• Findings made available by different channels• Sensitize and inform public opinion• Provide orientation for reflection• Ideals and evidence with ‘percolating’ effect

Source: Weiss (1979); Postlethwaite (2001); Ross et al (2006)

Enlightened policy research cycle

Policy reform and agenda for action

Consultation and debate

Policy suggestions

Interpretation and reporting

Datacollection

and analysis

Specific research questions

General policy concerns

Programme implementation

Source: Saito (1999)

Mass media,

journals, conference

Sensitize public

Orientation to expand

different reflections

Feed debate

Research Knowledge

Categories of ‘change’ after assessments• General analyses of education system• Inputs to a general review of policy• Preparation of a reform programme• Dissemination and discussion of results with stake-holders• Reform of the intended curriculum• Reform of the implemented curriculum• Improvement and reallocation of resources• Monitoring of the effects of an intervention• Helping donors identify aspects of the education system • Improvement of learning achievement

Source: Postlethwaite and Kellaghan (2008)

Vietnam• Assessments

– National assessment, Grade 5, 2001, 2007, 2011– Expanded to lower and upper secondary levels since 2011– Participation in PISA 2012 and PASEC 2013

• Findings– Disparity (location, SES, ethnicity)

• Research Utilization and Actions– Until 2010 no data usage on policy– Targeted resource allocation for disadvantaged population– Curriculum and textbook reform to develop cognitive and non-

cognitive skills– Re-orient teaching methods to apply knowledge in “real life”

situation

Source: Le (2014)

South Korea• Assessments

– Since 1995, TIMSS, PISA, ICILS to establish benchmark– National assessment since 2008 to identify underachievers

• Findings– Top cognitive results and bottom affective results in ILSA– Disparity (gender, location) in NLSA

• Research Utilization and Actions– Recommendations from ILSA rarely used– ‘Targeted programmes’ based on NLSA results– Research competition on ideas on using ILSA & NLSA

results by policy makers

Source: Cho (2014)

IIEP MOOC Survey

• February 2016 IIEP MOOC on Learning Assessments

• 2,700 registered (F 53%; A&P 36%); 1,000 active; • 295 responses (F 54%; A&P 36%) as of 8 March• Questions on:

– (i) how well informed on results; – (ii) opportunities to participate; and – (iii) perceived changes in inputs, policy review,

reforms, etc.

Preliminary Results on IIEP Survey• Much more informed on public examinations and NLSA

(rather than ILSA and RLSA)• Much more informed about performance than on its

enabling variables• Information sources – (1) Official documents; (2)

Observation; (3) Informal conversation• Opportunity for involvement – (1) questionnaire

construction; (2) item writing• Perceived changes after assessments – (1) learning

improvement; (2) general analyses• Policy-research process as extra work as ‘leaders’; not

teachers and school heads

Conclusion from country illustrations

• Possible to observe:– How research was used– Category of action

• Illustration of “enlightenment model”– Dissemination via different channels– Active participation of actors

• Resulted in the improvement of student achievement