Cautions About Inferences From International Assessments

Kadriye Ercikan University of British Columbia

This discussion is based on a paper co-authored with Wolff-Michael Roth and Mustafa Asil, in press in the Teachers College Record

Key Inferences From International Assessments

Compare country performances for identifying the countries with the best education systems

Generate insights about effective policy and practice strategies that are associated with higher learning outcomes.

Some Examples

TIMMS video study data (e.g., Shimizu, 1999)

Germany’s efforts to make many system level changes as a reaction to the “PISA Shock” (Ertl, 2006)

England’s importing of mathematics teachers from China as a reaction to the country’s low rank in the most recent PISA mathematics assessment (FlorCruz, 2014)

Validity of Inferences

Do country rankings really reflect the quality of education in different countries?

What are the fallacies of simply looking to higher performing countries to identify strategies for improving learning in our own countries?

Focus

Describe limitations of using country rankings as indicators of quality of education

Discuss limitations of using correlates of higher performance in high ranking countries as a way of identifying strategies for improving education in our home countries

Sequence of of Inferences

Inference 1: Identifying countries with the most effective education

Implied Propositions

Inference 2: Identifying effective policies or practices responsible for high ranks

Implied propositions

Inference 3: Emulating policies and practices will lead to improvement in education

Implied Propositions

Limitations in Identifying Countries with Most Effective Education Systems

There are several sources of incomparability in international assessments that create inaccuracies in comparing and ranking countries

Overall country performance is one of many indicators of education systems

Multiple indicators such as school drop out rates, school climate, student and teacher behavior, and students’ perceptions of benefits of schooling additional criteria to consider

Focusing on performance at the level of entire countries may obscure important within-country differences in rankings for subgroups, such as gender or ethnic groups

Graduation Rate Rank versus Reading Score Rank

Comparison of gender groups within the five countries on PISA 2009

Conclusions

Do higher ranks indicate better education systems?

Comparability

Multi-dimensionality of quality of education

Should we look toward other jurisdictions to copy strategies and practices for the purpose of changing education in another jurisdiction?

Variations in achievement are likely due to much more complex interrelations of cultural, societal, and educational factors.

Taking this or that student variable and education practice in one context and trying to replicate it somewhere else may not result in the aspired to learning outcomes from the original jurisdiction.