Web viewEvery few years, international organizations conduct and publish results from international large-scale assessments (ILSAs) of student achievement

Compare Globally, Interpret Locally:

International Assessments and News Media in Israel

Oren Pizmony-LevyTeachers College, Columbia University

ABSTRACTScholars have documented the global spread of international large-scale assessments of student achievement (ILSAs), such as the Trends in International Mathematics and Science Study (TIMSS), and the Programme for International Student Assessment (PISA). Research to date has focused on the influence of national and world characteristics on the likelihood of a country to participate in these assessments. This article investigates the role of local public discourse on the institutionalization of ILSAs “on the ground.” To do so, I use a sample of articles (n=97) from two newspapers in Israel. The analyses reveal three key patterns. First, public discourse about ILSAs developed gradually, and not immediately after the release of the first few reports. Second, policy makers are more active in the public discourse about ILSAs than other actors (e.g., students, teachers, parents, and education scholars). Third, public discourse about ILSAs constructed an achievement crisis that calls for change in policy/practice. These patterns facilitated the institutionalization of ILSAs in the Israeli context, beyond the contribution of transitional organizations.

Total word count: 8,525

Key words: globalization, international large-scale assessments, public discourse, news media

Compare Globally, Interpret Locally | Pizmony-Levy

Compare Globally, Interpret Locally: International Large-Scale Assessments of Student

Achievement and News Media in Israel

INTRODUCTION

Every few years, international organizations conduct and publish results from

international large-scale assessments (ILSAs) of student achievement. The International

Association for the Evaluation of Educational Achievement (IEA) coordinates the Trends in

International Mathematics and Science Study (TIMSS) and the Progress in International Reading

Literacy Study (PIRLS). TIMSS is administered every four years to students in 4th and 8th grades,

whereas PIRLS is administered every five years to students in 4th grade. In addition, the

Organisation for Economic Co-operation and Development (OECD) coordinates the Programme

for International Student Assessment (PISA), which is administered every three years to 15

years-old students. These ILSAs draw on data from large number of participants in many

countries/economies worldwide. More than half a million students in 65 countries/economies

took part in the most recent cycle of PISA (2012).

ILSAs are considered to be an advanced scientific and technical endeavor (National

Research Council, 2002). They involve complicated sampling techniques of schools and

students, development of reliable questionnaires and instruments, translation and adaptation to

various languages. Further, ILSAs are based on state-of-the-art statistical methods aimed to

produce comparable and valid scores (e.g., Item-Response Theory [IRT]). Results from each

cycle of ILSAs are reported in thick and dense volumes that include numerous charts, graphs,

and tables as well as technical notes.

2


Nevertheless, shortly after the release of a new ILSA report, the news media begins with

frenzied attention. For some time, newspapers worldwide carry articles highlighting the

standings of nations in international ranking, and commenting on the implications of these

results. Following the release of PISA 2012, for example, the New York Times published an

article titled “American 15-year-olds lag, mainly in math, on international standardized test”

(Rich, 2013), and the Israel-based Ha’aretz published an article titled “Israeli student

achievement does not leave much room for optimism” (Kashti, 2013). These articles challenge

educational systems and schools to implement changes and mobilize stakeholders to improve the

quality of education.

In addition to describing, benchmarking, and monitoring education systems, ILSAs are

intended to enrich the public discussion about education (Howie and Plomp, 2005). Although

both IEA and OECD make the results of ILSAs available to the public for free (digital versions

of the reports are available on the organizations’ websites), they are not necessarily accessible to

the public due to their scientific/technical language and jargon. Given this, and the fact that

people are more likely to seek news and information from the news media, it is important to

understand how the media “filter and translate scientific information to the public” (Epstein,

1996:22). Moreover, leaders of IEA and OECD view public visibility of ILSAs as “key factor in

shaping the public policy debate (Wagemaker, 2011[2004]: 259; see also Wagemaker, 2013) and

argue that “the media had a key role in raising awareness of policy implications of the PISA

research results […]” (Schleicher, 2006: 267).

This article will focus on two research questions: (a) What role the news media play in

informing the public about the results of ILSAs and in framing these results as meaningful

indicators of educational quality? (b) Which actors participate in the translation, interpretation

3


and filtering of results of ILSAs? Or which actors are granted “speech acts” or “claim making” in

the public discourse on ILSAs? To investigate these questions in a systematic fashion, I draw on

content analyses of articles published between 1996 and 2014 in two Israeli newspapers –

Ha’aretz and Yediot Ahronoth – on ILSAs (sample size is 97). Using data from both newspapers,

I am able to investigate the ways in which quality-elitist (Ha’aretz) and mass-popular (Yediot

Ahronoth) newspapers engage with ILSAs.

In so doing, this article speaks to global and transnational sociology, specifically World

Society theory that examines institutionalization of global rationalized models and scripts

(Meyer, Boli, Thomas & Ramirez, 1997). To date, scholars have focused on institutionalization

in terms of organizational practices that result from top-down processes. David H. Kamens and

Connie L. McNeely (2010), for example, examined the institutionalization of ILSAs by looking

at the immense growth in the number of countries participating in these assessments worldwide.

However, we know less about the incorporation of these organizational practices into the local

context and discourse (Phillips, Lawrence & Hardy, 2004). Public discourse on ILSAs provides a

good window to the “friction” (Tsing, 2005) between transnational organizations (i.e., IEA and

OECD) and local actors who interact with these organizations beyond organizational practice of

participation in ILSAs. In other words, analysis of the public discourse on ILSAs could shed

light on the social process of institutionalization “on the ground.”

BACKGROUND

Theoretical Framework

World Society theory holds that nation-states enact policy scripts diffused and legitimized by a

network of international organizations (Meyer et al., 1997). This theory, which is an extended

4


version of Neo-Institutionalism, emerged as a response to the postwar rise in actor-centered

theories of modern societies (Meyer, 2010). Scholars writing within this framework have

observed the dramatic rise of national and supranational non-governmental organizations (NGOs

and INGOs) as well as international governmental organizations, such as the United Nations and

its affiliated organizations (IGOs; Boli & Thomas, 1999). These organizations promote global

scripts (models) for nation-states on “how to be better actors in light of general principles”

(Meyer, 2010: 7). These global scripts, as World Society theory posits, are transmitted to nation-

states through various mechanisms (e.g., intergovernmental conferences, professional

associations, international standards, and epistemic communities), and over time nation-states

adopt them.

The majority of World Society research has relied on cross-national time-series datasets,

using nation-states, organizations, and other artifacts as the unit of analysis. In the field of

education, we find research on the worldwide diffusion of mass schooling and the enormous

increases in enrollment in primary and secondary education (Meyer, Ramirez & Soysal, 1992),

and in post-secondary and higher education (Schofer & Meyer, 2005). Other examples include

research on the worldwide diffusion of curricular models (Meyer, Kamens & Benavot, 1992),

international schools (Suarez, Ramirez & Koo, 2009), and pedagogical movements such as

human rights education (Meyer, Bromley, & Ramirez, 2010) and environmental education

(Bromley, Meyer & Ramirez, 2011; Pizmony-Levy, 2011). More recently, scholars have

examined the emergence of national and international assessments of student achievement as a

new global policy script (Kamens 2015; Kamens & Benavot, 2011; Kamens & McNeely, 2010;

Pizmony-Levy, 2013).

5


One major criticism of World Society theory is that it is overlooking local contexts:

discourses and actors. Scholars argue that local discourse and actors shape whether or not global

scripts become reality (see Anderson-Levitt 2003, 2012). Further, while World Society scholars

assume that global scripts are legitimized at the transnational level, they neglect the social

process of legitimacy that might take place “on the ground.” Drawing on literature in social

psychology and organizations, Johnson, Dowd and Ridgeway (2006) argue that legitimacy

processes, which are important to the survival and success of social objects, begin with local

validation. Legitimacy, they argue, is dependent on consensus among actors in the local arena as

well as links between new social objects and existing, widely accepted culture. In this paper, I

examine what happens to a global script – ILSAs – once it “lands” in a local setting.

Literature Review

Following the immense expansion of ILSAs, scholars have begun to explore public

discourse about these assessments. Schmidt (2003) provides a broad overview of reactions to

PISA 2000 in eight countries – Canada, France, the Netherlands, the Russian Federation,

Sweden, Switzerland, the United Kingdom, and the United States. Michelle Stack (2007)

examines U.S. media coverage of the 1999 TIMSS and both 2000 and 2003 PISA results. She

finds that the media endorses these assessments, with little critique, as valid indictors of system

performance. Further, Stack argues that the media interpreted the results of these assessments in

concert with business and political elites. Takayama (2008, 2009) shows how conservatives and

progressives in Japan interpreted and appropriated results from PISA to support their political,

economic, and cultural needs. He argues that shifts in the political and social conditions in Japan

affected media responses to the Japanese performance in PISA 2000 and PISA 2003 and thus

6


created biases in the report. Xavier Pons (2012), drawing on six national case studies (Belgium,

France, Hungary, Portugal, Romania, and Scotland), find that media coverage of PISA focuses

on the ranking itself more than anything else. He demonstrates how local actors – including

policy makers and teachers unions – use PISA results for political instrumentalization. He also

points to missing voice of local scholars in the interpretation of PISA results. Dixon and her

associates (2013) compare reactions to ILSAs and biased coverage in four countries (Britain,

Finland, France, and Germany). They illustrate how media coverage includes negative reactions

to performance in ILSAs across countries, including Finland that performs high in these

assessments.

While important, extant studies have methodological and conceptual limitations. For

instance, Stack’s (2007) does not examine whether and how media coverage of ILSAs changed

over the years, while Dixon et al. (2013) overlooks the role of different local actors in the

construction of public discourse about ILSAs. Takayama (2008, 2009) and Pons (2012) examine

the ways in which local actors engage with ILSAs, but they focus on newspaper articles that

cover the release of new PISA reports. Finally, most studies draw heavily on qualitative analysis

of small samples of newspaper articles, thereby limiting the generalizability of the findings.

THE CASE: ISRAEL

Israel offers a fruitful case study for exploring public discourse on ILSAs. Despite its relative

newness among the world’s states (independent since 1948), Israel has developed rapidly and is

considered a Western industrialized democracy with a very high level of human development.

Israel’s Human Development Index in 2015 is .888 and it is among the top-twenty countries with

respect to this measure (UNDP, 2015). As such, the education system in Israel is well-developed.

7


Public spending on education in Israel is 5.9 percent of the gross domestic product (slightly more

than in the United States 5.6 percent). The average years of schooling is 11.9, and the average

expected years of schooling is 15.7 years (in the United States these figures are 13.3 years and

16.8 years, respectively).

Israel is among the early participants in ILSAs, and its engagement began with the pilot

study, titled Educational Achievements of Thirteen-Year-Olds in Twelve Countries (Foshay et al.

1962). Therefore it is no surprise that Israel joined the new cycle of studies that was launched in

the mid-1990s. Because this article examines public discourse about ILSAs, it is important to

review the performance of Israeli students, as reported in the official reports of TIMSS, PIRLS,

and PISA (Figure 1). These assessments suggest that the performance of Israeli students is close

to the average of participating countries. The results vary by domain (mathematics, science and

reading literacy) and the year in which assessments were conducted. Results also vary by the

method of presentation (average score versus location in ranking tables). This is important

because fluctuation in the average scores is a result of changes in student achievement, whereas

fluctuation in the ranking is affected by changes in student achievement in other countries and by

changes in the composition of countries in the ranking tables (e.g., countries joining or leaving

the ranking tables).

Figure 1 – About Here

The overall performance of Israeli students in mathematics and science has not changed

since the mid-1990s. Specifically, TIMSS reports show that the average achievement scores did

not change significantly between 1999 and 2003, but declined significantly between 2003 and

2007. PISA reports further confirm that achievement in mathematics and science did not change

8


between 2000 and 2009. Although the average achievement did not change over time, the

location of Israel in the ranking tables has been fluctuating.

As for reading literacy, international reports show improvement in the overall

performance of Israeli students since the early 2000s. PIRLS show statistically insignificant

improvement between 2001 and 2006 in the average achievement scores in reading literacy. This

pattern is further supported by international reports of PISA that show significant improvement

between 2000 and 2009. Yet, the location of Israel in the ranking tables, however, has decreased

over time.

These mixed results provide local actors with ample opportunities to engage with ILSAs.

On the one hand, drawing on the average achievement scores, one could characterize the Israeli

educational system as mediocre, but stable in terms of performance. On the other hand, drawing

on the ranking tables, one could characterize the Israeli educational system as going downward

with regard to performance. Feniger, Livneh and Yogev (2012) argue that the low achievement

of Israeli students in ILSAs were to be expected, given the economic and demographic

characteristics of Israel. They conclude that the public debate in Israel regarding the low

achievement in ILSAs “has been magnified out of all proportion.” This article systematically

analyzes the public discourse in Israel and the actors involved in its creation.

DATA AND METHODS

Data Collection

In order to examine the public discourse about ILSAs in Israel, I constructed an original

dataset of newspaper articles published between 1996 and 2010. I used the Bet Ariela Catalog for

Daily Newspapers (the Catalog), which is part of the Israeli Center for Digital Information

9


Services, to identify relevant newspaper articles. The Catalog contains four news sources in

Hebrew, and it is the only one to cover publications from the early-1990s. The Catalog does not

include all news articles; rather it includes a sample of news articles that Bet Ariela’s librarians

deemed important or meaningful. I use all the news articles referenced in the Catalog. I searched

for articles that were published in two newspapers in Israel: Yediot Ahronoth and Ha'aretz. These

newspapers represent different journalistic style and cater to different audience. Yediot Ahronoth,

the most popular newspaper in Israel, is a daily tabloid and is considered to be the “people's

newspaper” aimed at mass consumption and favors simplistic, dramatic coverage. In contrast,

Ha'aretz is a quality-elitist broadsheet that targets a smaller, more educated audience who expect

more depth in their news coverage. In many ways, Yediot Ahronoth is comparable to USA

Today, whereas Ha'aretz is comparable to The New York Times. Using both newspapers,

therefore, provides a broad and inclusive perspective to the public discourse in Israel.

The search protocol includes the following key-words: “international studies,”

“international tests,” and “international achievement tests.” In addition, I searched for the full

title and acronym of each ILSAs (in English and in Hebrew): “Third International Mathematics

and Science Study” (TIMSS), “Trends in International Mathematics and Science Study”

(TIMSS), “Progress in International Reading Literacy Study” (PIRLS), and “Program for

International Student Assessment” (PISA). This search protocol yielded a dataset with 97

newspaper articles. More than three-fourths of the articles (78 percent) were published in

Ha’aretz and less than one-fourth of the articles (22 percent) were published in Yediot Ahronoth.

The dataset is limited in two ways. First, because the Catalog began indexing newspaper

articles in 1993, public discourse about early ILSAs is neither indexed nor accessible. Second,

because the Catalog is restricted to news articles published in Hebrew, the analysis to follow

10


applies to public discourse among the Jewish majority in Israel more than it applies to public

discourse among the Arab minority in Israel who predominantly speak Arabic. Nevertheless, for

the purpose of this paper, the dataset provides an opportunity to explore public discourse in a

critical time point when the field of ILSAs is dramatically expanding worldwide.

Coding

In order to analyze the structure and content of the public discourse, I developed a detailed code

book. The code book was informed by open coding and theoretical coding. Open coding began

by reading the articles in the dataset and immersing myself in issues and happening in the data.

This preliminary reading resulted in a list of questions and ideas about the development of public

discourse about international assessments in Israel. Then, I began to identify themes and

categories that repeated across articles. For example, I found several articles where achievement

of Israeli students in ILSAs was discussed in the context of national security/defense. This

became a category titled “transnational practices and local logics.” I found articles that draw a

comparison between achievement of Israeli students in ILSAs in the 1960s and in the past

decade (a comparison that is not advanced by international organizations). This also became a

category titled “extending the global message.” I found articles that included critical comments

about ILSAs in general and in Israel specifically. This became a small, but telling, category titled

“critiques.”

The next step involved theoretical coding. For example, I used Steiner-Khamsi’s

typology (2003) for political reaction to ILSAs (i.e., scandalization, glorification, and

indifference). I searched for articles that used different linguistics and visual techniques to shame

– or scandalize – the Israeli government/society for the low achievement in ILSAs. Inspired by

11


the literature on transnational transfer of educational policy (Steiner-Khamsi, 2004), I searched

for articles that discuss and/or call for policy reaction to results in ILSAs. Following research on

media coverage of contentious issues (Ferree, Gamson, Gerhards, and Rucht, 2002), I identified

a list of speakers (individuals and their organizational affiliation and roles) who were given

“speech acts” in the articles. The list includes seven groups: Ministry of Education (e.g.,

Minister, Executive Director, and Chief Scientist), politicians (e.g., Knesset Members), ILSAs

practitioners (e.g. national research coordinator), scholars in academy, social-movements

organizations, parents and students, educators (e.g., teachers and head teachers), and others. For

each articles I counted how many types of speakers were mentioned.

The final dataset includes three additional background variables: year of publication,

location of article (1=news section, 2=feature articles, editorials, opinions, or letters to the

editor), and objective of article (1=covers the release of new ILSAs report or focus on ILSAs, 2=

refers or cites ILSAs as part of a larger news story). All coding sheets were entered to data

management software.

12


RESULTS

Descriptive Patterns

I begin by presenting descriptive statistics of the public discourse. Figure 1 shows the usage of

ILSAs-related search terms in two daily newspapers in Israel during the period under

investigation. The number of newspaper articles increased over the years, especially after 2003.

Between 1996 and 2002, the volume of articles covering ILSAs was small, with an average of

1.8 articles per year. Between 2003 and 2010, however, the volume of articles covering ILSAs

accelerated, with an average of 10.4 articles per year. Another measure for this growth is the

ratio between number of ILSAs reports released and the number of articles in each period. In the

first period (1996-2002) two ILASs reports were released and the ratio is 6.5 articles/per ILSA

report, whereas in the second period (2003-2010) seven ILASs reports were released and the

ratio is 11.8 articles/per ILSA report. It is clear from Figure 1 that the public discourse about

ILSAs was not automatically diffused to Israel, but rather it took several years (or cycles of

ILSAs) before it evolved in the Israeli context.

Figure 2 – about here

Almost all of the newspaper articles in the sample are original. Only two articles are

translated from a foreign source (both published in Ha’aretz). The first article, which is titled

“Who’s top?” and was originally published in The Economist in 1997, is devoted to TIMSS 1995

(then the Third International Mathematics and Science Study). After brief overview of the

ranking tables, the article describes the political context of TIMSS and the growing interest and

role of governments in international assessments. The second article, which is titled “The last

time you used algebra was in...” and was originally published in the New-York Times in 2004,

offers much more critical perspective on the meaning and implications of international

13


assessments. The author argues that while American students are doing poorly in these

assessments, the United States is still a superpower in different domains related to mathematics

and science.

Figure 2 also shows an association between the release of ILSAs reports and the usage of

ILSAs-related search terms in newspaper articles. The number of newspaper articles is higher in

years when new ILSAs reports are released than in other years. This pattern is confirmed in an

independent t-test (M1=9.0, M2=3.4, p<.05). The public discourse about ILSAs is somewhat

dependent on the release of new ILSAs reports. Yet, it is important to note that while more than

half of the articles (54 percent) were published in the news section of the newspaper, the rest (46

percent) were published in other sections of the newspaper (e.g., daily magazine, editorial,

opinions, and letters to the editor). In other words, the public discourse about ILSAs is reactive

to transnational developments and expansive with regard to the contexts in which articles are

published.

Further, less than half of the articles (43 percent) actually cover the release of new ILSAs

reports or focus explicitly on ILSAs. These articles often include general information about the

organization that conducted the assessment, details about the domain and the target population, a

visual representation of the results, and key-points from press release of the Ministry of

Education. The following titles exemplify the orientation of these articles: “Study: level of eighth

graders in mathematics similar to the international average” (Ha’aretz 1996) and “Failed:

Evidence of poverty, international study finds Israeli students are ranked 31st out of 41 in reading

literacy, mathematics and science” (Yediot Ahronoth 2003). Other articles in this group include

interviews with key figures in organizations conducting ILSAs, critical comments about the

implications of international assessments, and commentary about the implementation of these

14


assessments. For example, the article “The world is an educational laboratory” (Ha’aretz 2004) is

devoted to an interview with Dr. Ina Mullis, Co-Director of the TIMSS and PIRLS International

Study Center. This article directly corresponds with the original idea behind ILSAs as expressed

in the First International Mathematics Study (Husén, 1967). Although Dr. Mullis situates TIMSS

within the broader context of IEA research and refers to the idea of learning from differences,

she offers a new interpretation that emphasizes the importance of learning from ranking tables as

a way to reduce differences: “The world is an educational laboratory, and we are trying to look at

the countries with the best performance and learn from them so that others can learn and emulate

them.”

Majority of the articles (57 percent), however, only cite and refer to ILSAs reports. These

references are made by the author of the article as well as by interviewees who participate in the

article. For example, the article “One, two and we failed” (Ha’aretz 2003 ID16) presented the

story of the Israeli Foundation for Mathematical Achievement for All (IFMAA), a local non-

governmental organization. IFMAA was established in 2002 by a group of scholars committed to

improving and revising the mathematics curriculum in Israel. Throughout the article,

interviewees keep referring to ILSAs as the context and the main motivation behind the creation

of IFMAA. Another example is an article titled “Money yes, education no” (Yediot Ahronoth

2003), which was published on the occasion of the International Children's Rights Day. In this

article, the authors contrast the high governmental investment in education and the low return on

this investment as reflected in student achievement. Among other interviewees, the authors bring

quotes from members of the Citizens for Education Committee (local non-governmental

organization) citing the poor achievement of Israeli students in ILSAs.

15


Voice in Public Discourse

Although ILSAs pertain to various stakeholders, not every stakeholder has an equal chance to

have a voice in the public discourse about these assessments. Because the interpretation and

translation of these assessments is done by actors that participate in the public discourse, it is

important to examine who are these actors. Figure 3 shows the representation of different

stakeholders in the news articles. Majority of the articles (83.5 percent) included at least one

speaker. The most common participants in the public discourse about ILSAs in Israel are the

Ministry of Education and scholars in the academy. Slightly more than two-fifths of the articles

(41.2 percent) include speakers from the Ministry of Education, and about two-fifths of the

articles (39.2 percent) include scholars affiliated with academic institutions. Other common

participants are individuals involved with the planning and implementation of ILSAs. Slightly

more than one-fifth of the articles (22.7 percent) include ILSAs practitioners. Interestingly, other

important and relevant stakeholders are represented less frequently in the public discourse about

ILSAs. For example, less than one-fifth of the articles (16.5 percent) include teachers or head

teachers, and one-eighth of the articles (12.4 percent) bring the perspective of parents and

students.


The representation of different stakeholders in the public discourse about ILSAs varies

across the years, as illustrated in Figure 4. Between 1996 and 2002, when the public discourse

was small in volume (n=13), more than half of the articles include scholars affiliated with

academic institutions (53.8 percent). This figure, however, dropped to less than two-fifths (36.9

percent) between 2003 and 2010. In earlier news articles, scholars were the source of information

about ILSAs. For example, in the article “Survey: The achievement level of 8th graders in

16


mathematics is similar to the international average” (Ha’aretz 1996 ID200) the author indicates

that “results were reported by the President of The Academic College of Tel Aviv-Yafo,

Professor Elazar Kochva.” Interestingly, this article was published before the official press

release of TIMSS 1995. And in another article “Not good in numbers” (Yediot Ahronoth 1997

ID2) is based on a lecture by Professor David Chen, Faculty of Education at Tel-Aviv

University.


Except for the Ministry of Education, all other stakeholders saw a meaningful growth in

their representation in news articles about the about ILSAs. The representation of teachers and

head teachers, for example, grew from 7.7 percent to 17.9 percent of the articles. Also, the

representation of ILSAs practitioners grew from 7.7 percent to 25.0 percent of the articles.

Between 2003 and 2010 new stakeholders joined the public discourse about ILSAs: politicians,

parents and students, and social-movements organizations. These patterns suggest that ILSAs

became a legitimate and useful tool to make claims about education and schools in Israel.

Making ILSAs Taken-For-Granted

Local newspapers and stakeholders in the education domain contributed to the further

institutionalization of ILSAs in three interconnected ways. First, they embedded ILSAs in the

Israeli context by discussing these assessments according to dominant logics in Israel (i.e., the

logic of national security and defense). Second, they constructed ILSAs as reaction-worthy by

dramatizing the results and their implications, and by taking different actions in order to improve

the location of Israel in the international ranking tables. Third, they presented structured and

coherent discourse, with limited questions and critique of ILSAs themselves.

17


In the following paragraphs I demonstrate these processes.

Embedding ILSAs in the Israeli Context

The public discourse about ILSAs embraced a critical tone and framed the performance of Israeli

students as an ongoing national crisis. Newspaper articles focused on the international “horse

race” and often overlooked more meaningful comparisons based on the average achievement

scores. This critical tone is reflected articles headlines: “Not good in numbers” (Yediot 1998

ID2), “Israel repeats a grade” (Yediot Ahronoth 2007 ID157), and “Failing” (Ha’aretz 2010

ID112). Out of the articles covering the release of new international assessments, three-fifths (60

percent) had a headline with a critical and negative tone. Other articles had a neutral headline (34

percent) or a positive headline (6 percent).

In addition to the international “horse race,” newspaper articles also paid attention to

domestic inequalities in student achievement. Slightly more than one-third of the articles (35

percent) point not only to differences between countries, but also to differences within Israel

based on students’ social location: gender, ethnicity (Jewish versus Arabs), socioeconomic

background, and sector (Jewish secular sector versus Jewish religious sector). For example, one

headline indicated that “In Israel: World's largest gap between boys and girls in eighth grade

mathematics” (Ha’aretz 1998 ID3) and another headline indicated that “Barely enough:

Alarming international test, low achievement in all subjects, gaps between rich and poor”

(Yediot Ahronoth 2010 ID159).

The significance of the Palestinian-Israeli conflict, coupled with the central role that the

military play within the Israeli culture and society, is implicated in the public discourse about

ILSAs. Local actors in Israel used the discourse of national security and national defense in three

18


different ways. First, local actors argue that low performance in international assessments will

gradually translate into national security threat. The following excerpt suggests that low

performance in ILSAs will negatively affect the development of future cadre of scientists and

innovative technologists. In turn, this will have impact on the national capacity to develop

necessary weapons and intelligence systems:

“Israel was ranked in 39th place out of 53 countries […] This gives a hint of the nation's future ability to absorb, utilize and develop new technologies that are the primary ingredients underlying an economy's ability to grow, and of its future ability to operate and invent the advanced weapons and intelligence systems necessary for maintaining the existence of a country with our national security problems" (Ha’aretz 2005 ID56)

Second, local actors argue that the negative consequences of low performance in

ILSAs are more dangerous than the consequences of military conflict and terror. The

quotes in the following excerpt are especially telling because both speakers are affiliated

with the left-wing of the political spectrum in Israel. Although these speakers’ political

agenda emphasizes issues of peace and demilitarization of Israeli society, their statements

connect low performance in ILSAs with national defense/security discourse:

“Knesset Member Yossi Sarid, former Minister of Education and member of the Knesset Education Committee said: ‘the level of the Israeli student is a real existential threat, and Israel's ranking in education is more dangerous than all types of terrorism together’. […] Knesset Member Shimon Peres said that the harsh and severe report is prejudicial to the security of Israel” (Ha’aretz 2003 ID23)

Third, local actors contrast the centrality of defense/security issues in public agenda with

the marginality of education in the public agenda. They also compare the high priority and

resources allocated to defense/security policy with the relatively low priority and resources

dedicated to education policy. The following excerpts exemplify this pattern:

The Executive Director of the Ministry of Education: “This is a national problem that the Prime Minister needs to decide on. Enough talk about national security

19


and wars. In the name of the threat of war, they are streaming budgets to defense items. It comes on the expense of our children's education. Only decision to make education a national priority will get us out of the mess” (Yediot Ahronoth 2003 ID26)

Constructing ILSAs as Reaction-Worthy

The public discourse about ILSAs constructed these assessments as reaction-worthy by further

scandalizing and dramatizing the results. Scandalization took place by comparing between

achievement of Israeli students in ILSAs in the 1960s and in the past decade. Almost one-tenth

of the articles (9 percent) framed the results to suggest that over the years the performance of

Israeli students in ILSAs eroded. Quotes such as “Who is to blame for the fact that Israeli

children fell from first place in the same test in 1964 to 28th, 25 years later?” (Ha’aretz 2003

ID16) and “Forty years ago Israel was the first in the world in an international test in

mathematics. Twenty years ago, Israel fell to the sixth place and in 1999 - was ranked 28th”

(Ha’aretz 2003 ID19) exemplify this theme. This frame was employed by various actors,

including the Ministry of Education, scholars in the academy, and journalists. Importantly, this

frame was not advanced by international organizations. Nor was this frame accurate, as

illustrated in the article “The myth behind the excellence in mathematics: The assumption that

Israel was once an educational superpower in mathematics is unfounded” (Ha’aretz 2007 ID50).

In an opinion column, the author argues that early ILSAs were based on a non-representative

sample of students in Israel, which excluded Arab schools and many Jewish schools serving new

immigrants. He concludes:

“The importance of international comparative studies for the local education system should not be minimized. However, for the discussion to be to the point and informed, it must be based on examined facts. The time has come to bid farewell to the myth that once, many years ago, when we were young and beautiful and innocent, we were a light unto the nations in mathematics education.”

20


Further, scandalization took place by comparing the performance of Israeli students with

developing countries that are located in the periphery of the world system. By doing so, local

actors are able to manipulate public perception of performance in ILSAs by shaming the Israeli

government/society. This approach is reflected in the following excerpts in which the author

links between performance in ILSAs and location in the global chain of production. Countries in

which the educational system is ranked low international assessments are destined to be

exporters of cheap textiles and/or labor:

“[Israel’s ranking is] below every industrialized country participating in the study and below other countries much poorer than Israel, including Thailand and Romania from which we import laborers. Even the children of Malaysia, a country so poor that its textile workers are willing to work for miniscule wages that no Israeli would be willing to accept, have better grades than us” (Ha’aretz 2001 ID98).

“We ranked below all of the Western countries and below additional countries such as Malaysia, Thailand and Romania who provide us – still, though apparently not forever – with cheap textiles and labor” (Ha’aretz 2006 ID86).

Dramatization of the results took place by presenting manipulated visuals that emphasize

the low performance of Israeli students in ILSAs. Slightly less than one-third (29 percent) of the

articles included visual representation of results from ILSAs. Most of the visuals (20 out of 27)

presented some form of an international ranking table. Often, these visuals articles included

visuals with a partial list of participating countries that create a misleading impression that Israel

is at the bottom of the ranking table. For example, while 41 countries participated in TIMSS

1995, the visual that accompanied a news article in Yediot included only 20 countries. Given the

ranking of Israel (17th), this visual creates the impression that Israel is located closer to the

bottom of the list rather than to the middle of the list. This practice was used in different time

points and across assessments, as illustrated in Figures 5A and 5B.

21



The response of local actors, especially the Ministry of Education, to ILASs also

contributed to the construction of these assessments as reaction-worthy and thus to their

institutionalization. After all, responses suggest that ILSAs are meaningful, important and thus

deserve attention. I found two types of responses to ILSAs in Israel: (1) system-oriented changes,

and (2) ILSAs-oriented changes.

System-oriented changes. Following the release of ILSAs reports, newspaper articles

presented a wide range of initiatives by the Ministry of Education. These initiatives include

revision of national curriculum, establishment of system-wide standards, revision of textbooks,

and the establishment of the National Task Force for the Advancement of Education in Israel.

The following excerpt demonstrates this type of policy actions:

“The Ministry of Education announced yesterday the establishment of a committee to examine ways to improve the teaching of mathematics in schools [...] The timing is not coincidental. The release of Israeli student achievement in international assessments of mathematics and science is scheduled for tomorrow” (Ha’aretz 2000 ID137).

ILSAs-oriented changes. In several cases, newspaper articles presented a set of initiatives

that were intended to improve Israel’s location in the international ranking tables. In contrast to

system-oriented changes, these actions are less concerned with teaching and learning processes.

These initiatives include short-term preparation of students in cohorts that are scheduled to be

assessed by ILSA, and increasing students’ motivation to invest energy and time when

participating in ILSAs. In an article titled “The solution of the Ministry of Education for the poor

results in international knowledge tests: introducing a sense of national mission to the students”

(Ha’aretz 2009 ID115) the author reports on a memo sent by the Ministry of Education to

22


students in which participation in ILSAs is situated in the context of national pride and

international competition:

“Dear Student, not every day you are given the opportunity to represent our country in an international study. This time it happens! […] All that is expected from you is serious attention to any question, and your effort to provide the best answer. Consistent, serious and responsible answers will allow obtaining a true picture of the achievements.”

Additional analysis shows that ILSAs-oriented changes of students was endorsed in times

when the Minister of Education was affiliated with the Likud Party, which is a right-wing

leaning party (Limor Livnat, 2001-2006 and Gideon Sa’ar, 2009-2013). This activity, however,

was criticized and rejected in times when the Minister of Education was affiliated with the Labor

Party, which is a left-wing leaning party (Professor Yael Tamir, 2006-2009). This pattern may

suggest that ideological orientation play a role in the way in which the Ministry of Education

engages with ILSAs.

Presenting Structured and Coherent Discourse

Overall, the public discourse about ILSAs in Israel was receptive. In many newspaper articles,

journalists and speakers engaged with ILSAs in un-critical fashion, thus constructing a coherent

discourse that contributed to the institutionalization of ILSAs. The lack of critical perspective is

evident in two articles that feature interviews with leaders of international organizations that

conduct ILSAs. Both articles recognize the immense power of ILSAs and professionals, but fail

to offer any critique whether this power is appropriate in a democracy. For example, in an

interview, Dr. Ina Mulis, Co-Director of the TIMSS and PIRLS International Study Center,

explains the process in which assessments are developed and the role of experts from different

countries in this process (Ha’aretz 2004 ID94). She also indicates the power of assessments and

their impact on sovereign countries:

23


“Curricula are determined, of course, by each state individually, but the uniformity of tests requires matching the content [of the curricula] to questions […] In the 1980s [experts] decided to focus on the core issues, reading literacy, which is the basic proficiency needed for attaining knowledge, mathematics and science, which are the means for understanding the subjects that will contribute to growth in the economy, welfare and health. In the 1990s there were several series of consultations, which formed a consensus among experts about the level of knowledge required of the students”

Further, several articles pointed to the importance of continued participation in ILSAs. In

one article, for example, the Chief Scientist of the Ministry of education said “Not participating

in these tests [assessments] will kill the messenger […] I do not imagine that Israel decide not to

participate in international studies, even if it costs a quarter million" (Ha’aretz 2003 ID16). And

in another article, the Executive Director of the Ministry of Education reiterated this sentiment:

“Not participating in these tests [assessments] will cause severe damage to the image of Israel”

(Ha’aretz 2006 ID45). The latter excerpt echoes the notion of normative factor that is

emphasized in World Society Theory.

Nonetheless, I found several articles that challenge ILSAs and the public discourse that

has evolved around them. In a letter to the editor, “A student’s witness,” the author provides a

glimpse to students’ engagement (or lack of) with ILSAs:

“Results from tests, such as PISA, show the Israeli students are lazy, ignorant and stupid. As a former student who experienced these tests, I would like to argue that these tests do not represent the ability of students. The system works as follow: teachers explain the tests are [important] for the pride of Israel and will rank students from different countries worldwide. […] The thought that goes through the mind of the average high school student is this: this test is not affecting my report card. I do not mind the honor of Israeli students and Israeli pride worldwide. […] If I finish the test quickly, I can get out of here fast. I do not like tests. I marked answers randomly and left the classroom. I spent the next two hours reading a book. So did many of my friends. Is it really possible to take these tests as an indication of something?” (Ha’aretz 2009 ID300)

In this letter, the author describes the main reason for students’ disengagement with

international assessments and teachers’ efforts to motivate students (i.e., national pride).

24


Further he argues that instead of spending time on assessments (that often measure

achievement in literacy) students choose to spend their time on actual reading. The fact

that this critical perspective is limited to the opinion section and is not echoed in the news

section of the newspaper highlights the broad acceptance of international assessments in

the public discourse.

25


CONCLUSION

By analyzing a sample of newspaper articles published in two Israeli newspapers between 1996

and 2010, this study sheds light on the local reception of ILSAs, such as TIMSS, PIRLS, and

PISA. Rather than assuming that transnational developments (e.g., the creation of new cycles of

assessments) and country participation are sufficient indications for the legitimacy and

institutionalization of ILSAs, I analyze the ways in which news media and local actors enhance

the taken-for-grantedness of these assessments.

The descriptive part of this study finds that the public discourse about ILSAs in Israel

evolved over the years and increased dramatically in 2003, soon after the release of PISA 2002.

Although it is not possible to discern whether the discourse took-off because of PISA and its

affiliation with the OECD or due to the accumulation of social indicators about the education

system in Israel (i.e., two rounds of TIMSS), it is clear that after 2003 the public discourse

became more vibrant. Overall, the discourse includes not only coverage of new ILSAs report, but

also numerous references to and citations of ILSAs reports.

The participation of various stakeholders in the public discourse varies across the years.

In the first period (1996-2002), scholars in the academy dominated the discourse as they were the

main brokers of information about ILSAs. In the second period (2003-2010), however, scholars

in the academy were less central in the public discourse and most often provided explanations for

the results and/or commented about the possible implications of the results. This pattern echoes

similar developments at the transnational level. In other work (Pizmony-Levy et al., 2014), I

show how the transnational field of ILSAs emerged in the late-1950s from the collaborative

work of scholars in the academy (among them Benjamin Bloom and Torsten Husén). In the late-

1990s, however, the field was transformed into inter-governmental collaboration.

26


The expansion of the public discourse about ILSAs in Israel created an opportunity for

new local actors to use these assessments as a tool to advance claims. The Ministry of Education

capitalized ILSAs results to support their request to break teachers’ strike. The Israeli Teachers

Union supported questionable pedagogical initiatives (that involved salary increases) as a way to

improve the location of Israel in the international ranking table. And other local actors leveraged

ILSAs results to justify the creation of common core curriculum across all schools.

The qualitative part of this study shows how the news media and local actors enhanced

the legitimacy and institutionalization of ILSAs in Israel. This cultural work necessitates three

simultaneous tasks or processes. The first task involves the “indigenization” of ILSAs in the

local context. Local actors did so by highlighting achievement gaps between social categories

that are well established in Israel (e.g., ethnicity and religiosity). That is, rather than focusing

solely on the international “horse race” and thus “flattening” differences within Israel, public

discourse paid attention to differences and inequalities between groups. Local actors also

employed jargon taken from public discourse on national security/defense to “make sense” of

ILSAs results and their implications. The second task involves the construction of ILSAs as

reaction-worthy rather as something to ignore. This task was accomplished by dramatizing and

scandalizing the results, and by taking actions aimed to affect the education system and future

results of ILSAs. The third task involves the creation of coherent public discourse that accepts

and supports ILSAs. This task was accomplished by marginalizing critiques and questions

regarding the efficacy and accuracy of these assessments.

These findings suggest some interesting directions for future research. First, will these

discursive patterns vary by countries? This question is beyond the scope of this study, but it is

important to consider that Israel is a relatively small polity. It may be that in larger and more

27


complex polities – for example: the United States, Canada, and Germany – the discourse will

evolve differently. A comparative study is required could show how country-level characteristics

affect public discourse about ILSAs. Second, does public discourse matter for public opinion

toward education and schooling? This study might provide some interesting results that would

complement this article. Scholars have documented decrease in public trust in the education

system in the United States (Loveless, 1997; Smith, 2008); they conclude that this trend might be

related to the release of ILSAs reports. Yet, this hypothesis was not tested. Third, how do local

actors – Ministry of Education, politicians, ILSAs practitioners, scholars in academy, social-

movements organizations, parents and students, educators, and journalists – view their role in

disseminating ILSAs reports to the public?

This study of public discourse about ILSAs in Israel increases our understanding of what

happens to global rationalized models after countries adopt them. This study complements Neo-

Institutionalism (i.e., World Society Theory) by exploring the social process “on the ground” and

beyond formal commitment to global rationalized models. Findings suggest that public discourse

(i.e., ILSAs) emerge over time and not immediately after a country adopt global rationalized

models. This study enriches scholarship in sociology of education by exploring the politics

beyond large-scale datasets.

28


REFERENCES

Dixon, R., Arndt, C., Mullers, M., Vakkuri, J., Engblom‐Pelkkala, K., & Hood, C. (2013). A lever for improvement or a magnet for blame? Press and political responses to international educational rankings in four EU countries. Public Administration, 91(2), 484-505.

Epstein, S. (1996). Impure science: AIDS, activism, and the politics of knowledge. University of California Press.

Feniger, Y., Livneh, I., & Yogev, A. (2012). Globalisation and the politics of international tests: the case of Israel. Comparative Education, 48(3), 323-335.

Ferree, M. M. (2002). Shaping abortion discourse: Democracy and the public sphere in Germany and the United States. Cambridge University Press.

Foshay, A. W. (1962). Educational Achievements of Thirteen-year Olds in Twelve Countries: Results of an International Research Project, 1959-1961. UNESCO.

Gamoran, A., & Porter, A. C. (Eds.). (2002). Methodological advances in cross-national surveys of educational achievement. National Academies Press.

Howie, S., & Plomp, T. (2005). International comparative studies of education and large-scale change. In International handbook of educational policy (pp. 75-99). Springer Netherlands.

Kamens, D. H., & McNeely, C. L. (2010). Globalization and the growth of international educational testing and national assessment. Comparative Education Review, 54(1), 5-25.

Meyer, J. W., Boli, J., Thomas, G. M., & Ramirez, F. O. (1997). World society and the nation-state. American Journal of sociology, 103(1), 144-181.

Phillips, N., Lawrence, T. B., & Hardy, C. (2004). Discourse and institutions. Academy of management review, 29(4), 635-652.

Pizmony-Levy, O. (2011). Bridging the global and local in understanding curricula scripts: The case of environmental education. Comparative Education Review, 55(4), 600-633.

Pizmony-Levy, O., Harvey, J., H. Schmidt, W., Noonan, R., Engel, L., J. Feuer, M., ... & Torney-Purta, J. (2014). On the merits of, and myths about, international assessments. Quality Assurance in Education, 22(4), 319-338.

Schleicher, A. (2006). Chapter 14: How can international organizations work with the media to manage the results of cross-national studies. In: Cross-National Studies of the Quality of Education: Planning Their Design and Managing Their Impact (pp. 265-275). International Institute for Educational Planning (IIEP) UNESCO.

29


Stack, M. (2007). Representing School Success and Failure: media coverage of international tests. Policy Futures in Education, 5(1), 100-110.

Steiner-Khamsi, G. (2003). The politics of league tables. Journal of Social Science Education, 2(1).

Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387-407.

Tsing, A. L. (2005). Friction: An ethnography of global connection. Princeton University Press.

Wagemaker, H. (2011[2004]).

Wagemaker, H. (2013). International Large-Scale Assessments: From Research to Policy. Handbook of International Large-Scale Assessment: Background, Technical Issues, and Methods of Data Analysis (pp.11-33).

30


Figure 1: Performance of Israeli Students in TIMSS, PIRLS, and PISA 1995-2009

Study Mathematics Science Reading

Ranking Score Ranking Score Ranking Score

TIMSS 1995a 21/41 522 23/41 524 -- --

TIMSS 1999 29/39 466 27/39 468 -- --

TIMSS 2003 19/46 496 23/46 488 -- --

TIMSS 2007 24/49 463* 25/49 468* -- --

PIRLS 2001 -- -- -- -- 23/35 509

PIRLS 2006 -- -- -- -- 31/40 512

PISA 2002 [Reading]b 31/41 433 33/41 434 30/41 452

PISA 2006 [Science] 40/57 442 39/57 454 40/57 439

PISA 2009 [Reading] 41/64 447 41/64 455 36/64 479*

Notes:* Significant difference between this year and previous year a In TIMSS 1995, Israel sampled students in both fourth and eighth grade. In subsequent

assessments, however, Israel sampled only students in the eighth grade. Due to unapproved sampling procedures in TIMSS 1995, comparison between this assessment and later assessments is not available in official publications of IEA.

b Similar to ten other non-OECD countries, Israel joined the first wave of PISA in 2002.

31


Figure 2: Number of Articles Published, by Year

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 20100

2

4

6

8

10

12

14

16

18

20

0

1

12

3

0

4

21

19

9

3

5

17

5

12

14

Press Release Total

Num

ber o

f Arti

cles

32


Figure 3: Representation of Stakeholders in Newspaper Articles

33


Others

No Speaker

Social-Movements

Parents and Students

Politicians

Teachers and Head Teachers

ILSAs

Academy

Ministry of Education

22.7%

16.5%

6.2%

12.4%

13.4%

16.5%

22.7%

39.2%

41.2%

Figure 4: Representation of Stakeholders in Newspaper Articles, by Time Period

34


Others

No Speaker

Social-Movements

Parents and Students

Politicians

Teachers and Head Teachers

ILSAs

Academy

Ministry of Education

23.1%

15.4%

0.0%

0.0%

0.0%

7.7%

7.7%

53.8%

38.5%

22.6%

16.7%

7.1%

14.3%

15.5%

17.9%

25.0%

36.9%

41.7%

2003 to 2010 (n=87) 1996 to 2002 (n=13)

35


Figure 5: Visual Representation of ILSAs Item A: TIMSS 1995 Item B: TIMSS 2007

36


Appendix Table 1: Ministers of Education in IsraelBegin End Minister of Education Political Party Leaning

June 1993 June 1996 Amnon Rubinstein Meretz Left

June 1996 January 1998 Zevulun Hammer Mafdal Right

February 1998 June 1999 Yitzhak Levy Mafdal Right

June 1999 June 2000 Yossi Sarid Meretz Left

June 2000 March 2001 Ehud Barak Labor Left

March 2001 January 2006 Limor Livnat Likud Right

January 2006 March 2006 Meir Shitrit Kadima Center

March 2006 March 2009 Yuli Tamir Labor Left

March 2009 March 2013 Gideon Sa'ar Likud Right

37

Documents

Web viewEvery few years, international organizations conduct and publish results from international large-scale assessments (ILSAs) of student achievement