Measuring Public Perceptions of International ... · Measuring Public Perceptions of International Organizations’ Legitimacy Bril e Anderson ETH Zurich ECPR Joint Sessions Workshop

Measuring Public Perceptions of International Organizations’

Legitimacy

Brile Anderson ∗

ETH Zurich

ECPR Joint Sessions Workshop 2015

March 22, 2015

Abstract

Present techniques to measure public perceptions’ of international organizations’ legitimacy areinsufficient. Measurements either intertwine theories of legitimation into the measure itself or includebyproducts of legitimacy (i.e. obedience) as the measure. The former method assumes the legitimationtheories are valid while the latter presumes that legitimacy is a necessary condition for obedience.The measures that avoid these shortcomings tend to be single-item proxies, which highlight only anisolated feature of legitimacy. Therefore, I propose a new measure of legitimacy using a latent modellingapproach. Thus far, I have conducted a pilot test of the measure on two international organizations,the World Trade Organization and the governance framework under the United Nations FrameworkConvention on Climate Change. I use item response theory to test the behaviour of items (i.e. surveyquestions) as well as to assess their contribution to the measure. Future analyses will use item responsetheory with covariates to account for differences in item functioning between participants (e.g. politicalideology, sex) in aggregation. From these results, I plan to revise the set of items and subsequently,retest the measure.

1 Introduction

Legitimacy is pertinent to the maintenance of international organizations’ authority. Securing authority,as international organizations seek to do in a variety of domains from trade to health, is difficult. Theycan neither coerce states into recognising their authority nor is it alway in the interests of states to do so;therefore, according to Hurd (1999) and Koppell (2008), legitimacy is paramount. Legitimacy implies thatthe relevant stakeholders believe that an international organization ought to be the designated authority.

Legitimacy is, however, a nebulous concept and before proceeding, it is important to clarify its meaning.As Suchman (1995) states, ”Legitimacy is often invoked instead of described and described instead ofdefined,” (295). And of those who do define legitimacy, there is a tendency to further modify the termthus contributing confusion. Etymologically, legitimacy dates to Thucydides’ History of the PeloponnesianWar (Zelditch Jr 2001)); however despite millennia of usage, there is no consensus on its meaning.

∗This project is funded by the ERC Advanced Grant “Sources of Legitimacy in Global Environmental Governance”(Grant: 295456, PI: Thomas Bernauer) and supported by ETH Zurich. I am grateful to Aya Kachi, Zorzeta Bakaki, ThomasBernauer, Oliver Bragger, Lisa Dellmuth, Robert Gampfer, Steffen Mohrenberg, Quynh Nguyen, Jonas Tallberg, Erik Voeten,Stefan Wehrli, Joel Westra, and Michael Zurn for their useful comments and their technical support. The project benefitedfrom two excellent research assistants, Frederic Hans and Irina Shaymerdenova.

1

Contemporary definitions can be broadly classified as either absolutist or relativist. An example of theformer is Buchanan and Keohane (2006) who define ”normative” legitimacy as, ”an institution has theright to rule,” (see also Bodansky 1999). Relativist definitions, in contrast, define ”political” legitimacy asthe ”acceptance of and justification of shared rule by a community ... where an actor or institution makesa claim to have the right to govern and be obeyed plus a community who is subject to the decisions of theauthority, and actors intersubjectively hold the belief that the claim is justified (Bernstein and Cashore2007),” (see also Suchman 1995; Bodansky 1999; Hurd 1999; Steffek 2003; Dellmuth and Tallberg 2014).The distinction is whether an institution is legitimate or actors perceive it as legitimate. This begs thequestion: who are the relevant actors?

For decades, international organizations sought to legitimise themselves to states. Yet, it seems that thereis a shift occurring in the public consciousness starting in 1999 with protests against the WTO in Seattle1

up to the present with the construction of 193 snowmen in Davos at the 2015 World Economic Forum.2

This has prompted international organization scholars to investigate public perceptions of internationalorganizations’ legitimacy (see Dellmuth and Tallberg 2014; Bernauer and Gampfer 2013).

Yet, present techniques to measure public perceptions of international organizations’ legitimacy and ofinstitutions, in general, are limited. Measurements on institutional legitimacy either intertwine theoriesof legitimation into the measure itself or include byproducts of legitimacy (i.e. obedience) as the mea-sure. The former assumes the legitimation theories are valid while the latter presumes that legitimacy isnecessary condition for obedience. The measures that do not fall prey to these shortcomings tend to besingle-item proxies, which highlight only an isolated feature of legitimacy. It is possible these single-itemproxies might be appropriate yet this is an empirical question. The next section outlines these in detail.

Section 2 further details these limitations. I subsequently detail the conceptualization of the presentmeasure, which uses a latent modelling approach. Goertz (2006) stresses concepts are multidimensionaland multilevel. To properly measure a concept, he advocates conceptualising its three levels. The first orbasic level is the concept itself, in our case, legitimacy. Below this is the secondary level which identifiesthe key dimension or sub-constructs of a concept. This is the ontological level where a researcher identifieswhat the concept truly means. The lowest level is the indicator level that operationalises the dimensionsin the secondary level. From this, I create a series of questions that measure the theorised attributesof legitimacy. After which, I use item response theory to identify the behaviour of indicators and theircontribution to measurement. The final section outlines future directions.

2 Literature Review: Problems of Existing Measures

International organization scholars are not the first to tackle measuring institutional legitimacy. Otherfields in political science investigate the public’s legitimacy perceptions of the police (Tyler 2004; Tylerand Wakslak 2004; Hinds and Murphy 2007), national governments (Fraser 1974; Weatherford 1992; Gilley2006, 2012), courts (Gibson et al. 1998, 2010), even the European Central Bank (Eijffinger and Cruijsen2007). Even fields outside of political science like organization management (Suchman 1995) and sociology(Tyler 2001) examine employees’ legitimacy perceptions of an organisation’s and individuals’ perceptionsof other individuals’ legitimacy. Across disciplines and subfields, measurement techniques suffer fromthree limitations. First, they either intertwine theories of legitimation in the measurement of legitimacy,second, use byproducts of legitimacy as the measure, and/or rely on single-item proxies. Each of these is

1http://www.seattle.gov/cityarchives/exhibits-and-education/digital-document-libraries/

world-trade-organization-protests-in-seattle [Accessed: 04-03-2015]2http://www.swissinfo.ch/eng/davos-digest/41224580 [Accessed: 17-03-2015]

2

http://www.seattle.gov/cityarchives/exhibits-and-education/digital-document-libraries/world-trade-organization-protests-in-seattle

http://www.seattle.gov/cityarchives/exhibits-and-education/digital-document-libraries/world-trade-organization-protests-in-seattle

http://www.swissinfo.ch/eng/davos-digest/41224580

addressed in turn.

Intertwining theories of legitimation into measurements automatically presumes that the legitimationtheory is true. Legitimation, by definition, is the process whereby an institution becomes legitimate.Legitimation can occur in the input or output of an institution (Scharpf 1999). Input refers to the proce-dural processes that legitimate an institution like democratization (Bernauer and Gampfer 2013; Dohertyand Wolak 2012; Moravcsik 2004; Kweit and Kweit 2007; Dryzek and Stevenson 2011; Dombrowski 2010;Scholte 2011)) or equity in procedure (Tyler 2001). Output, in contrast, refers to the favourability anddistribution of outcomes as a legitimation process. In a causal framework, the legitimation tactic, be itinput or output, is the independent variable and the legitimacy perception, the dependent variable. Yet,a multitude of scholars use the legitimation tactic, the independent variable, as a proxy for legitimacy, thedependent variable without any empirical proof of their causal link (see Fraser 1974; Booth and Seligson2005; Bernauer and Gampfer 2013; Persson et al. 2013). For example, Weatherford (1992) starts with thefoundation that governments permitting participation (i.e. elections), based on majority rule, accountableand so on are legitimate. This is an input argument that procedure matters for legitimacy perceptions.However, Weatherford subsequently designs a legitimacy measure that then includes accountability andrepresentation in the conceptualization. As mentioned, the fallacy is not that Weatherford claims thatdemocratic features (e.g. accountability, representation) enhance legitimacy, but that he subsequentlyuses the independent variable as a proxy for the dependent variable.

Other measures use byproducts of legitimacy, namely, obedience, as the measure itself. Conceptually, if anindividual perceives an entity to be legitimate, then this individual should obey the authority. However,the inverse is not true. Even if an entity is obeyed by individuals, it does not imply it is perceived aslegitimate. For example, obedience can result from fear. This is the key distinction between authorityand legitimacy. Authority is an institutionalisation of power. Legitimacy implies that its subjects alsothink it ought to be the authority. A few measures only use obedience as their measure for legitimacy(see Tyler 2001) thereby missing this subtlety.

Other authors avoiding these conceptual flaws conduct their empirical investigations with single-itemproxies of legitimacy perceptions like confidence as a proxy for the United Nation’s legitimacy (Dellmuthand Tallberg 2014) or trust to measure European Central Bank’s legitimacy (Eijffinger and Cruijsen 2007).These are related to legitimacy yet none captures the concept in isolation. As Goertz (2006) states, ”Thecore attributes of a concept constitute a theory of ontology of the phenomenon under consideration. Todevelop a concept is more than providing a definition, it is deciding what is important about an entity.”Thus, these measures misplace emphasis on only one part of legitimacy. It it possible that single-itemproxies might be suitable; yet, this needs to be empirically shown. For the above reasons, it is necessaryto create a new measure of legitimacy.

3 Theoretical Framework

Goertz (2006) stresses concepts are multidimensional and multilevel. To properly measure a concept,he advocates conceptualising its three levels. The first or basic level is the concept itself, in our case,legitimacy. Below this is the secondary level which identifies the key dimension or sub-constructs of aconcept. This is the ontological level where a researcher identifies what the concept truly means. Thelowest level is the indicator level that operationalises each of the dimensions in the secondary level.

To start, it is necessary to define the concept. Legitimacy, as I define it, is the perception that the actionsof an international organization are desirable and appropriate within some socially constructed system of

3

norms and values making subjects willing to substitute the international organisation’s decisions for theirown and state’s evaluation of the situation. This is an amalgamation of definitions from the relativiststream.

From this, I identify two sub-constructs, social affinity, the perception that the actions of an internationalorganization are desirable and appropriate within some socially constructed system of norms and valuesand obedience, making subjects willing to substitute the international organisation’s decisions for their ownand state’s evaluation of the situation. Neither is a sufficient condition for legitimacy; both are necessary.Table 1 outlines previous measurements that elude to similar sub-constructs and indicators. Typically,previous measures include one sub-construct or the other. In measurements that use latent-modelling,authors include legitimation indicators (see Muller 1970; Weatherford 1992; Gilley 2006, 2012).

Table 1: Previous Measurements Incorporating Social Affinity or ObedienceApproach: ♦: Latent Variable Modelling, �: Single-Item

Author Institution Approach Social Affinity Obedience

Muller (1970) Supreme Court,Congress

♦ (1) Legitimating Ideology(2) Overt Support Sub-Constructs

Overt Support Sub-Construct

Fraser (1974) National Govern-ment, State Govern-ment, University ofKentucky

� (1) What do you think oughtto be the purpose ...? (2) Onthe whole, how well do youthink ... has fulfilled thispurpose over the years?

-

Weatherford(1992)

Political System ♦ Would you say that, ”I amproud of many things aboutour form of government”

-

Booth andSeligson(2005)

Political System ♦ (1) Existence of PoliticalCommunity Sub-Construct(2) Support for Core RegimePrinciples

-

Gilley (2006,2012)

Political System ♦ - Acts of Consent Sub-Construct

Levi et al.(2009)

Police, Courts, TaxDepartment

♦ - The [police/court/tax de-partment] always has theright to make people [obeythe law/ decisions peopleabide by/pay taxes].

Dellmuthand Tallberg(2014)

United Nations � Confidence in the UnitedNations

-

The obedience sub-construct will differ from those listed in Table 1. For an international organisation,it is not an individual who obeys and subjugates their own judgement, but that an individual wouldlike deference from their state. For example, in Gilley (2006, 2012), their operationlization of the Actsof Consent sub-construct included: election turnout, tax payments, military recruitment. These areacts that an individual does to demonstrate obedience to the national government. In the supranationallevel, an individual would like their state to follow the recommendations and policies of an internationalorganization. The present measure will need to capture this nuance when formulating indicators.

4

4 Preliminary Survey Items and Data Collection

Table 2 lists the indicators for the two sub-constructs. There is a minimum of three indicators for each at-tribute as recommended by Little (2009). Table 2 lists possible indicators with potential responses rangingfrom disagree, mildly disagree, agree, and mildly agree. The answer categories are based off respondentfeedback from a pre-pilot. The pre-pilot included a dichotomous answer category (agree/disagree) andthe Likert scale above. Respondents receiving the binary categorization in the pre-pilot felt forced intostronger responses then desired. A middle-category and ”Don’t Know” were intentionally excluded toforce respondents to form an opinion. If respondents are answering randomly, I will be able to identifythis by the item characteristics curves (i.e. item response theory) in the analysis.

Table 2: List of itemsAnswer categories: Agree, Mildly Agree, Mildly Disagree, Disagree

RS: Reverse Scaled

Item # Content Social Affinity Obedience

1 The principles of this international organization match mine. ♦

5 I feel that this international organization is working towards abetter life for people.

♦

6 I sympathize with the purpose of this international organization. ♦

7 The morals of this international organization are incompatiblewith my own. (RS)

♦

13 I believe this international organization is necessary. ♦

14 The purpose of this international organization is pointless. (RS) ♦

9 My country should follow the recommendations by this inter-national organization, even when the reasons for their decisionsare not clear.

♦

4 My country should abide by the recommendations of this inter-national organization even if it is inconvenient.

♦

18 I think my country should follow the proposals of the interna-tional organization, even when my country disagrees, because itis this international organization that proposed the policy.

♦

5 Empirical Investigation on Measurement Models Using Item Re-sponse Theory

In order to verify this theoretical statement, the following five questions need to be answered by theempirical analysis.

1. Do the 11 items actually manifest a single latent construct (e.g., legitimacy)?

2. If so, is this latent construct really legitimacy belief or topic preference? (Content validity)

3. Empirically, how do the 11 items contribute to measuring legitimacy beliefs?

4. Do some items correlate more than others; i.e. do the data confirm our two theory-drivensubconstructs: “social affinity” and “obedience”?

5. Do individual-specific characteristics affect the importance of each item in discriminatingbetween legitimacy beliefs? (Demographics and general environmental and trade attitudes.)

5

This section outlines my empirical approach to answer these questions. I start with a brief descriptionof the pilot study followed by an explanation of the method and present the relevant methodology foranswering questions listed above.

5.1 Data Collection: Pilot Survey Design

I conducted two pilot surveys on January 20, 2015 in the United States. Participants were recruitedusing the online crowdsourcing platform, Amazon Mechanical Turks (AMT, www.mturk.com).3 Afterrecruiting participants through AMT, the surveys were implemented using an online platform called Uni-park (Unipark 2012). I tested the measure on two international organizations, World Trade Organization(WTO) and the environmental governance framework under the United Nations Framework Conventionon Climate Change (UNFCCC).

In each of the surveys, participants read a brief information pamphlet on either the WTO or UNFCCCfollowed by three questions to ensure participants’ comprehension of the procedure and purpose of theinternational organization (see Appendix A).4 Participants are allowed to proceed to the main surveyonly after answering three questions correctly.

Following the comprehension questions were the 11 items in the legitimacy measure listed in the previoussection. In order to minimize systematic effects of item order on responses, the 11 questions were shownin randomized order across participants.

After the 11 measurement questions, I included two questions to enhance the content validity.5 Thefirst question asks participants: In this survey, all of the questions can be considered to be part of abroader concept in political science. Of the concepts listed below, which do you think best incorporates theprevious questions? The possible answers included: legitimacy, accountability, transparency, participation,environment (in the UNFCCC survey)/trade (in the WTO survey), and performance. The presumptionbeing that if participants place the highest frequency on legitimacy or the second highest frequency afterthe topic dimension that the measure might actually be measuring what I intend to. The second questiondirectly asks participants whether the international organization (i.e. WTO or UNFCCC) is legitimate.Note that none of the 11 survey items includes the word legitimacy. Our intention is to predict whether aparticipant would find an institution legitimate from our measure, and then validate their response withthis direct question. However, this has certain limitations because it presumes that an individual knowswhat legitimacy means and interpretation is the same across respondents. I discuss some more issues

3There has been some debate on the quality of samples from online surveys. An increasing number of meta-studies showthat the quality of opt-in online surveys is as reliable as that of traditional phone- and mail-based surveys. For instance,Ansolabehere and Schaffner (2014) compare three modes of surveys in the US—an opt-in internet survey, a telephone surveyvia Random Digital Dialing (RDD), and a mail-in survey. Many of the variables within this questionnaire were verifiablewith the government data and they conclude that carefully executed opt-in online surveys are as accurate as a traditionaltelephone survey. Thus I are fairly confident about the mode of survey I employ in this study.

4Clearly there are pros and cons in presenting such an information sheet. The main concern is the risk of (unintentionally)priming participants for certain characteristics of the organization. The organizational characteristic that is unintentionallyemphasized might affect one’s legitimacy belief. However, there is also a risk associated with not presenting such an infor-mation sheet, the risk of no response or random response due to lack of knowledge. This issue—lack of knowledge—wasraised by multiple participants in the comment section of the pre-pilot online survey fielded in December 2014, where theorganizational information was not provided at all. I determined that the risk of random responses is more harmful than therisk of priming in our research, and opted for the use of information pamphlets. I took precautions to maintain neutrality ofthe pamphlet content. In the future surveys, I plan on using short informational video clips, in order to reduce respondents’cognitive burden.

5A measure is content valid when it measures what one claims to measure. Establishing content validity is, however, oneof the biggest hurdles that I am facing. After I rework the questions from the pilot, I plan to validate them with a panel ofexperts on legitimacy.

6

related to content validity in Section 5.3.

Finally as a way to control for participants’ predisposition to the two topical areas, I included a measureof environmental concern (Franzen and Vogl 2013) for the UNFCCC as well as a question on climateskepticism. For the WTO survey, we included a question on trade preferences. There is a possibility thatparticipants’ responses to the measure are influenced by two latent characteristics, participants’ legitimacybeliefs and their topic preference. I will come back to this issue of the uni- vs. multidimensionalityassumption in next Section 5.2. These topic preference questions were followed by a series of controlquestions on participants: sex, age, education, income, political ideology, and zip code.

5.2 Unidimensionality of the Measure

Do the 11 items actually manifest a single latent construct (e.g., legitimacy)?

The unidimensionality assumption suggests that the correlation among survey items included in themeasurement model should be explained by a single latent factor (e.g. “legitimacy”). In other words, ifthis assumption is violated, then it is likely that the chosen survey items manifest more than one underlyinglatent concepts. I use a principle component analysis (PCA) as described in Abdi and Williams (2010).PCA is primarily used as a data reduction technique. The researcher can take a series of correlatedvariables and identify a series of uncorrelated components using orthogonal transformation. The firstcomponent describes the greatest possible variation in the dataset and each successive component describesthe greatest possible variation under the condition that is orthogonal to the previous components.

5.3 Content Validity

If so, is this latent construct really legitimacy belief or topic preference?

A measure is content valid when it measures what one claims to measure. As Carmines and Zeller (1979)put it succinctly, “[c]ontent validity, however, provides no method or procedure to determine the extentto which this goal is achieved in practice” (p.26), meaning that we cannot evaluate it concretely viastatistical tools. This is due to the inevitable fact that “in measuring most concepts in social sciences,it is impossible to sample content...” “[r]ather one formulates a set of items that is intended to reflect acontent of a given theoretical concept.” (p.22) It can be a different story for example in medical researchwhere the predicting power of a new measure can be statistically evaluated based on observed behavioror symptoms that reveal hidden traits.

That said, for the purpose of this study, it is important to minimize the risk of content invalidity. Surveyliterature recommends qualitative expert reviews based on relevant literatures and/or based on expertpanels as methods for establishing content validity (Rubio et al. 2003). So far my “expert reviews” relysolely on the literature mentioned earlier in the paper. My strategy was to list items covering the entirerelevant content domain.6

In addition, I added three weaker instruments for enhancing content validity within our survey design.First, we conducted the surveys on multiple international organizations. We propose that if the behaviorof the items is similar between the two international organizations that belong to distinct topical domainsthen it is possible to claim the latent dimension underneath is more likely legitimacy than the topic.

6For future iterations of the study, I plan on have scholars with relative expertise review the measure.

7

As mentioned in the survey design, I also included the following “which−concept” question;

In this survey, all of the questions can be considered to be part of a broader concept in polit-ical science. Of the concepts listed below, which do you think best incorporates the previousquestions?

—legitimacy, accountability, transparency, participation, environment (in the UNFCCC sur-vey)/trade (in the WTO survey), and performance.

If “legitimacy” is among the predominant choices among participants in both surveys then it is indicativethat the items I include coincide with what is typically though of as legitimacy beliefs. This would berather encouraging result, given that none of the 11 items contains the word legitimacy. I also expect thatparticipants might choose the topical domain—environment or trade. If both legitimacy and a topicaldomain attracts more votes then it is indicative (though not deterministic) that our measure is centeredaround the intended content. However, if only the topical words (trade and environment) attracts largervotes but legitimacy does not, then it indicates that our survey items are not capturing legitimacy beliefs.I see this procedure as a relatively conservative test of construct validity in that all the other words—accountability, performance, participation and transparency—are also related concepts often used in thebroader literature of international organisation democratization and legitimation.

Finally, as mentioned in the survey design, I included a direct question asking whether respondents foundthe institutions legitimate. My intention is to predict which participants will find an entity legitimatebased on our aggregated legitimacy measure and validate it with their response from this direct question.

5.4 Evaluating the Relational Structure among Sub-Constructs and Observed Items

Do some items correlate more than others; i.e. do the data confirm our four theory-drivensub-constructs: “social affinity” and “obedience”?

To determine whether there are two latent sub-constructs. I employ Exploratory Factor Analysis (EFA).In EFA, fitting procedures estimate the factor loadings for every factor on each variable (Fabrigar et al.1999). This will validate whether the particular items actual load on theorised constructs. Among severaltests that are commonly used in the literature, I use the Kaiser criterion (Kaiser 1966) to determinethe number of factors. The Kaiser criterion suggests that the number of latent factors is the number ofeigenvalues (of the item correlation matrix) that are greater than one. I will use an oblique rotation sinceI expect there to be correlation between factors.

5.5 Item Contribution to the Latent Construct

Empirically, how do the 11 items contribute to measuring legitimacy beliefs?

Do individual-specific characteristics affect the importance of each item in shaping theirlegitimacy beliefs? (Demographics and general environmental and trade attitudes.)

In order to answer this question, I employ a latent variable modeling approach to evaluate the relationshipbetween survey items and the latent construct, legitimacy beliefs. Latent variable models are a class ofstatistical models, which relate a set of observable variables to a set of latent variables. When the empiricalanalysis is based on a theoretically driven structure among observed items and latent constructs, structural

8

equation modeling (SEM) and item response theory (IRT) are most commonly used (see Glockner-Ristand Hoijtink 2003). Thus far, I analyzed the pilot data using IRT.

Item Response Theory

IRT models the probability that an individual opts for a particular response to an item given their levelof the latent trait, θ. Since item responses in the legitimacy measure are polytomous and categorical,we apply Samejima’s Graded Response Model (GRM), which approximates parameters using maximumlikelihood estimation (Samejima 1997).

P (Xij = xij |θi) = P ∗xij (θi)− P

∗xij+1

(θi), (1)

where P ∗xij (θi) = P (Xij ≥ xij |θi) = e

Daj(θi−bxj )

1+eDaj(θi−bxj )

.

Xij is the response of individual, i, on item j. For each item j, there is one discrimination-parameter, aj , andM-1 difficulty-parameters, bm, where M is equal to the number of answer categories. The discriminationparameter describes the strength of an item’s ability to differentiate between individuals with varying levelsof the latent trait. The difficulty parameters are the points on the latent scale where the probability ofselecting the answer category is 0.50.

Item information curves (IIC) and item characteristic curves (ICC) identify which items help discriminatebetween different levels of legitimacy beliefs. The IIC indicates the range of θ where an item is best atdiscriminating among individuals. The shape of an IIC is determined by the discrimination and difficultyparameters. Ideally, items should be well spread out across the continuum and the higher the peak thegreater differentiation an item allows between levels of the latent trait. An item with a flat IIC could beexcluded from the measure since it fails to distinguish between an individual who finds an internationalorganization illegitimate or legitimate. ICC indicates the probability that an individual with a given levelof the latent trait selects a given answer category. As in the item information function, it is ideal if themaximum probabilities of selecting given answer categories vary across levels of latent trait. The utilityof IIF depends on the fit of the ICC to the test data. If the fit of ICCs to the data is poor, then the itemstatistics and the IIC will be misleading.

The behavior, i.e. ICC and IIC, can vary with covariates. This is known as differences in item functioning(DIF). I will employ a technique developed by Choi et al. (2011) to identify DIF. The technique useslogistic regression to detect DIF combined with Monte Carlo simulation approach to derive empiricalcriteria for DIF statistics and effect size measures. I will do this for political ideology, age, education,topic preference, and sex.

6 Preliminary Results from the Pilot Surveys

This section reports preliminary results from the pilot surveys.

6.1 Respondent characteristics

The WTO and UNFCCC data contain 480 responses after data cleaning (see Appendix B for the datacleaning process). Both respondent groups are, on average, slightly more male and around 30 or 32 years

9

old. The average respondent has completed 2-year college education or has some college experience. Themedian household income falls into the USD 30,000-39,000 range in the UNFCCC survey (2 participantsdeclined to respond) and USD 40,000-49,000 range in the WTO survey (1 participant declined to re-spond)7, and has the “slightly liberal” political view. More detailed descriptive statistics for the samplescan be found in Appendix C.

With regard to respondents’ general attitudes toward the environmental issue, I first asked two kindsof questions. One is to ask whether they think human activities cause global warming. About 84% ofrespondents responded that they believe in anthropogenic causes of global warming. The other is a setof four items that ask about their willingness to take pro-environmental behavior. I aggregated the fourquestions in a simple linear additive form Franzen and Vogl 2013 and rescaled the aggregated variable suchthat it ranges from 0 to 1. The mean score is 0.59, suggesting that respondents are on average slightly,but not much, more willing to behave pro-environmentally than the middle score. In the WTO survey, Iasked a single item question identifying trade protectionist attitude. The respondents’ trade protectionistattitudes are almost evenly distributed across favoring (129), opposing (173) and Don’t Know (178). Moredetailed descriptive statistics for these variables can be found in Appendix D.

6.2 Item Contribution

Do the 11 items manifest a single construct?

Do the 11 items manifest two sub-constructs, i.e., obedience and social affinity?

First, the Cronbach’s alpha values, which indicates the degree of internal consistency among the surveyitems, are 0.90 and 0.93 for the WTO and UNFCCC cases respectively. The values are well above theconventional acceptance level of 0.6 (George and Mallery 2003), suggesting that the initial 11 survey itemsare conceptually clustered around a single latent variable. This is supported further by the PCA, whichreveals one principle component in the WTO and UNFCCC. This is suggestive of one latent constructexplaining the maximum variation in the dataset; however, it is unclear, at this point, whether this latentconstruct is legitimacy.

The EFA identifies two factors in the WTO and UNFCCC data. Figure 1 illustrates the factor loadingsof each item on the two factors. I observe that Items 1 to 6 load on one factor, theorised as social affinity,and Items 7 to 9 load on another factor. This is suggestive that the theorised relationship between itemis correct.

6.3 Behavior of Items

Empirically, which of the 11 items contribute more than others in measuring legitimacy beliefs?

Initially, I fit a so-called constrained and unconstrained version of the graded response model (GRM) forthe WTO and UNFCCC data. As explained in Section 5.5, the GRM model evaluate the performanceof each item via three model parameters: ability, discrimination and difficulty. The constrained versionassumes equal discrimination parameters across all items and answer categories, meaning that we obtaina single estimate for the discrimination parameter for the entire model. The unconstrained version, on the

7We are slightly surprised that the no-response rate for the income question is extremely low compared to typical surveysfielded in the US.

10

(a) WTO (b) UNFCCC

Figure 1: Exploratory Factor Analysis

other had, allows the discrimination parameter of each item to vary and we obtain 18 separate estimates.A likelihood ratio test indicates the unconstrained model fits the UNFCCC and WTO data better thanthe constrained models.8 Thus, the rest of the report is based on the unconstrained GRM specification.

First, I examine whether each item is capable of distinguishing an individual’s latent trait associated withhis or her legitimacy belief. One way to test this item capacity is via item characteristic curves (ICCs).ICC plots the probability for a person to select each of the all possible response categories within anitem. Take an example of Item 1—The principles of this organization match mine. If a person findsthe organization legitimate (i.e. if the person has a high legitimacy trait on this organization), what isthe probability that this person should choose the response category “Agree”? In theory, the probabilityshould be higher than that exhibited by a respondent of a low legitimacy trait. This is what ICC plots.Therefore, there are four ICC plots for each item—for “Agree”, “Mildly agree”, “Mildly disagree”, and“Disagree”. On the contrary, if an item shows that a person with low legitimacy trait exhibits a highprobability of choosing “Agree”, then it is a fair indication that our survey item does not perform as wetheorized 9

Figure 3 displays the ICCs for WTO and Figure 2 for the UNFCCC. The curves well distinguish theaverage probably of choosing theoretically right categories based on one’s trait. This is the desiredbehavior.

In order to assess to what degree each item contributes to the latent measure of legitimacy beliefs, onecan look at inter information curves (IICs). Figure 4a and Figure 4b plot the IICs for the 11 items for theWTO and the UNFCCC, respectively. The graphs indicate how well each item performs (i.e. how muchinformation it contains) in determining the level of legitimacy belief given different levels of individualability. The plots provide an initial assessment of the behavior of items across organizations.

Items 4 and 8 in the IICs of WTO and UNFCCC are relatively flat providing approximately 10 percentof the total information for both models. According Rizopoulos (2006), this is grounds to exclude these

8Likelihood ratio test. UNFCCC: the test score = 209.36 with 8 degrees of freedom. P-value <0.001. WTO: the testscore =94.99 with 8 degrees of freedom. P-value <0.001

9Note that we rescaled the answer categories so respondents with high legitimacy beliefs are more likely to ”Agree” withitems.

11

Figure 2: UNFCCC

items from the measure. Encouragingly, the pattern is relatively similar between the two internationalorganizations. The contribution of Item 4 - The morals of this international organization are incompatiblewith my own - might be low since it was reverse scaled. It is questionable whether reverse scaled itemsfoster participant attention or inadvertently confuse respondents (see Schriesheim and Hill 1981). Item8 - My country should abide by the recommendations of this international organization even if it isinconvenient - uses stronger language then the two other items measuring the obedience, sub-construct.The other two items mentioned ”following” while Item 8 used ”abide”.

Preliminary results in differences in item functioning (DIF) reveal that DIF based on political ideology,age, and sex are minimal. No items behave differently between sexes or political ideology (i.e. self-identified liberal versus conservatives). Item 3 in the UNFCCC data behaves differently between youngand old participants while Items 6 and 9 behave differently in WTO data between participants. Thedifference between ages in the UNFCCC seems negligible while the differences in WTO data suggest thatlegitimacy scores need to be altered between age groups. I plan to test for DIF between protectionism,environmental concern, and education.

12

Figure 3: WTO

(a) WTO Item Information Curves (b) UNFCCC Item Information Curves

13

Figure 5: Which–Concept Question

6.4 Content Validity: Is this legitimacy?

With the strategy of exploiting multiple surveys on multiple international organizations, the preliminaryanalysis demonstrates very similar behavior of the included survey items, indicating that the measureindeed taps into a common concept across multiple topic domains. One can see in the similar shapesof Figure 4a and Figure 4b in particular that, the items behavior similarly between the two surveyscontrolling for the personal ability level. This is an encouraging result; however, the equivalence of itembehavior between the two organizations still needs to be statistically validated.10

Interestingly, for the which−concept question in the WTO and UNFCCC surveys, participants respondedwith the highest frequencies to the subject (i.e. trade or environment) and legitimacy. This providesa slight indication that the questions tapped into the desired concept (Figure 5). Additionally, I ampleased to see almost exactly the same response pattern between the two organizations. This means thatthe perceived meaning of our survey items do not shift across topics, which is a necessary condition forconstructing a more general measure of “legitimacy”.

Finally, I have yet to try and predict which participants will find an entity legitimate based on ouraggregated legitimacy measure and validate it with their response from the direct question asking whetherthey found the institution legitimate.11

7 Future Directions

With the present data, I need to continue exploring DIF as well as try to predict participants’ overtlegitimacy beliefs. After which, I plan to revise Items 4 and 8. I will modify the wording of Item 4 to thefollow the normal direction of the scale and soften the language in Item 8. With the revised set of items,I plan to send these to a panel of experts to review whether these measure the intended sub-constructsas desired. I am considering examining the measure beyond international organizations in general. I will

10Work in progress.11Work in progress. Using the direct questions, we see that 84% and 90% of respondents found the respective institution

legitimate.

14

rerun the experiment on the WTO and UNFCCC as well as other institutions. I can subsequently checkwhether the behavior of items is consistent and produce the final measure.

References

Abdi, H. and Williams, L. J. (2010). Principal component analysis. Wiley Interdisciplinary Reviews:Computational Statistics, 2(4):433–459.

Ansolabehere, S. and Schaffner, B. F. (2014). Does survey mode still matter? findings from a 2010multi-mode comparison. Political Analysis, doi: 10.1093/pan/mpt025.

Bernauer, T. and Gampfer, R. (2013). Effects of civil society involvement on popular legitimacy of globalenvironmental governance. Global Environmental Change, 23(2):439–449.

Bernstein, S. and Cashore, B. (2007). Can non-state global governance be legitimate? an analyticalframework. Regulation & Governance, 1(4):347–371.

Bodansky, D. (1999). The legitimacy of international governance: a coming challenge for internationalenvironmental law? American Journal of International Law, pages 596–624.

Booth, J. A. and Seligson, M. A. (2005). Political legitimacy and participation in costa rica: Evidence ofarena shopping. Political Research Quarterly, 58(4):537–550.

Buchanan, A. and Keohane, R. O. (2006). The legitimacy of global governance institutions. Ethics &international affairs, 20(4):405–437.

Carmines, E. G. and Zeller, R. A. (1979). Reliability and validity assessment, volume 17. Sage Publications.

Choi, S. W., Gibbons, L. E., and Crane, P. K. (2011). Lordif: An r package for detecting differentialitem functioning using iterative hybrid ordinal logistic regression/item response theory and monte carlosimulations. Journal of Statistical Software, 39(8):1.

Dellmuth, L. M. and Tallberg, J. (2014). The social legitimacy of international organizations: Interest rep-resentation, institutional performance, and confidence extrapolation in the united nations. Forthcomingin Review of International Studies.

Doherty, D. and Wolak, J. (2012). When do the ends justify the means? evaluating procedural fairness.Political Behavior, 34(2):301–323.

Dombrowski, K. (2010). Filling the gap? an analysis of non-governmental organizations responses toparticipation and representation deficits in global climate governance. International environmentalagreements: politics, law and economics, 10(4):397–416.

Dryzek, J. S. and Stevenson, H. (2011). Global democracy and earth system governance. EcologicalEconomics, 70(11):1865–1874.

Eijffinger, S. C. and Cruijsen, C. v. d. (2007). The economic impact of central bank transparency: asurvey.

Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., and Strahan, E. J. (1999). Evaluating the use ofexploratory factor analysis in psychological research. Psychological methods, 4(3):272.

Franzen, A. and Vogl, D. (2013). Two decades of measuring environmental attitudes: A comparativeanalysis of 33 countries. Global Environmental Change, 23(5):1001–1008.

Fraser, J. (1974). Validating a measure of national political legitimacy. American Journal of PoliticalScience, pages 117–134.

15

George, D. and Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0update 4th ed. Allyn & Bacon.

Gibson, J. L., Caldeira, G. A., and Baird, V. A. (1998). On the legitimacy of national high courts.American Political Science Review, 92(02):343–358.

Gibson, J. L., Gottfried, J. A., Carpini, M. X. D., and Jamieson, K. H. (2010). The effects of judicialcampaign activity on the legitimacy of courts: A survey-based experiment. Political Research Quarterly,page 1065912910370684.

Gilley, B. (2006). The meaning and measure of state legitimacy: Results for 72 countries. EuropeanJournal of Political Research, 45(3):499–525.

Gilley, B. (2012). State legitimacy: An updated dataset for 52 countries. European Journal of PoliticalResearch, 51(5):693–699.

Glockner-Rist, A. and Hoijtink, H. (2003). The best of both worlds: Factor analysis of dichotomousdata using item response theory and structural equation modeling. Structural Equation Modeling,10(4):544–565.

Goertz, G. (2006). Social science concepts: a user’s guide. Princeton University Press.

Hinds, L. and Murphy, K. (2007). Public satisfaction with police: Using procedural justice to improvepolice legitimacy. Australian & New Zealand Journal of Criminology, 40(1):27–42.

Hurd, I. (1999). Legitimacy and authority in international politics. International Organization,53(02):379–408.

Kaiser, H. F. (1966). The application of electronic computers to factor analysis. Educational and psycho-logical measurement, 20:141–151.

Koppell, J. G. (2008). Global governance organizations: Legitimacy and authority in conflict. Journal ofPublic Administration Research and Theory, 18(2):177–203.

Kweit, M. G. and Kweit, R. W. (2007). Participation, perception of participation, and citizen support.American Politics Research, 35(3):407–425.

Levi, M., Sacks, A., and Tyler, T. (2009). Conceptualizing legitimacy, measuring legitimating beliefs.American Behavioral Scientist, 53(3):354–375.

Little, T. D. (2009). Longitudinal structural equation modeling. Guilford Press.

Moravcsik, A. (2004). Is there a ”democratic deficit” in world politics? a framework for analysis. Gov-ernment and opposition, 39(2):336–363.

Muller, E. N. (1970). Correlates and consequences of beliefs in the legitimacy of regime structures.Midwest Journal of Political Science, pages 392–412.

Persson, M., Esaiasson, P., and Gilljam, M. (2013). The effects of direct voting and deliberation onlegitimacy beliefs: an experimental study of small group decision-making. European Political ScienceReview, 5(03):381–399.

Rizopoulos, D. (2006). ltm: An r package for latent variable modeling and item response theory analyses.Journal of statistical software, 17(5):1–25.

Rubio, D. M., Berg-Weger, M., and Lee, S. E. (2003). Objectifying content validity: Conducting a contentvalidity study in social work research. Social Work Research, 27(2):94–104.

16

Samejima, F. (1997). Graded response model. In Handbook of modern item response theory, pages 85–100.Springer.

Scharpf, F. W. (1999). Governing in Europe: effective and democratic? Oxford University Press.

Scholte, J. A. (2011). Towards greater legitimacy in global governance. Review of international politicaleconomy, 18(1):110–120.

Schriesheim, C. A. and Hill, K. D. (1981). Controlling acquiescence response bias by item reversals: Theeffect on questionnaire validity. Educational and psychological measurement, 41(4):1101–1114.

Steffek, J. (2003). The legitimation of international governance: a discourse approach. European Journalof International Relations, 9(2):249–275.

Suchman, M. C. (1995). Managing legitimacy: Strategic and institutional approaches. Academy ofmanagement review, 20(3):571–610.

Tyler, T. R. (2001). A psychological perspective on the legitimacy of institutions and authorities. In Jost,J. T. and Major, B., editors, Oxford Handbook of Innovation. Cambridge University Press.

Tyler, T. R. (2004). Enhancing police legitimacy. The annals of the American academy of political andsocial science, 593(1):84–99.

Tyler, T. R. and Wakslak, C. J. (2004). Profiling and police legitimacy: Procedural justice, attributionsof motive, and acceptance of police authority*. Criminology, 42(2):253–282.

Unipark (2012). Unipark-online survey software for universities. http://www.unipark.com.

Weatherford, M. S. (1992). Measuring political legitimacy. American Political Science Review, 86(01):149–166.

Zelditch Jr, M. (2001). Theories of legitimacy. In Jost, J. T. and Major, B., editors, Oxford Handbook ofInnovation. Cambridge University Press.

A Questionnaire

The following is the survey questionnaire used in our pilot study.

First for the UNFCCC;

17

Print versionPrint version

QuestionnaireQuestionnaire

11 Part One: Consent Part One: Consent

Please read the following statement carefully. If you choose to participate, please click on the button next to “I have read and understood the consent form and agree to participate in this survey.” If youchoose not to participate, please click on the button next to "I do not wish to participate".

This survey is a component of a larger research project at ETH Zürich. The objective of the research project is to better understand personal opinions concerning international politics. This survey isadministered by Brilé Anderson and Dr. Aya Kachi and is for scientific purposes only. It has no commercial or government-related purpose. The duration of the survey is about 15 minutes.

There are no known risks for you if you decide to participate in this survey, nor will you experience any costs when participating in the survey. The information you provide will help us understand opinionsconcerning international politics. This survey is anonymous. The information you provide in this survey will not be stored or used in any way that could reveal your personal identity.

If you have any questions or concerns about completing this survey, you may contact either Brilé Anderson at [email protected] or Dr. Aya Kachi at [email protected].

The ETH Zurich Ethics Review Commission has reviewed and approved this project. If you have any concerns about your rights in this survey, please contact us at [email protected] or RaffaelIturrizaga from the ETH Zurich Ethics Review Commission at [email protected] or +41 44 632 2354 with reference to its decision EK 2012-N-41.

I have read and understood the consent form and agree to participate in this survey.

I do not wish to participate.

2.12.1 Consented Consented

We appreciate your participation! The next page shows an information pamphlet about an environmental international organization. Please read carefully. After finishing the pamphlet, please find the "continue" button at thebottom of the page.

Thank you!

2.22.2 Part Two: Information Part Two: Information

2.32.3 Part Three: Comprehension Part Three: Comprehension

We like to ask you a few questions related to the previous page. Please note that you will not be able to proceed with the questionnaire until all questions on this page are answered correctly.At conferences, member countries to the UNFCCC negotiate agreements on

Climate

Health

Aid and Development

Migration

In the climate governance framework, decisions are made by

Consensus of all participating countries

Majority of all participating countries

Rotating council of member countries

Board of directors comprised of member countries and NGOs

The UNFCCC and associated framework is a type of international organization.

Yes

No

2.4.1.12.4.1.1 Block No. 1 Block No. 1

Do you agree, mildly agree, mildly disagree, or disagree with the following statements?

There are no right or wrong answers. We are interested in your opinions.

Agree Mildly agree Mildly disagree Disagree

The principles of this internationalorganization match mine.

Print version http://ww3.unipark.de/www/print_survey.php?syid=478051...

1 of 3 1/23/15 1:40 PM

18

I do not trust this internationalorganization.

This international organization shouldcontinue to make decisions in the future.

My country should abide by the decisionsmade at this international organizationeven if it is inconvenient.

I feel that this international organizationis working towards a better life forpeople.

I sympathize with the purpose of thisinternational organization.

2.4.1.22.4.1.2 Block No. 2 Block No. 2




The morals of this internationalorganization are incompatible with myown.

I feel there is insufficient expertise andknowledge in this internationalorganization.

My country should follow therecommendations by the internationalorganization, even when the reasons fortheir decisions are not clear.

Certain countries will be favoured by thisinternational organization over others.

The international organization’sprocedure for making decisions is unfair.

All countries will be treated fairly by thisinternational organization.

2.4.1.32.4.1.3 Block No. 3 Block No. 3




I believe this international organization isnecessary.

The purpose of this internationalorganization is pointless.

This international organization has theright to make decisions.

I believe the existence of thisinternational organization is anembarrassment to society.

Regardless of convenience, othercountries should follow the decisions ofthis international organization.

I think my country should follow theproposals of this internationalorganization, even when my countrydisagrees, because it is the internationalorganization that proposed the policy.

2.52.5 Part Four-II: Validity Questions Part Four-II: Validity Questions

In this survey, all of the questions can be considered to be part of a broader concept in political science. Of the concepts listed below, which do you think best incorporates the previous questions?

Legitimacy

Accountability

Transparency

Participation

Environment

Performance

2.62.6 Validity Question: Part II Validity Question: Part II

Do you think the environmental international organization presented at the beginning is legitimate?

Yes

No

2.72.7 Part Five: Topic Part Five: Topic

Do you think human activities such as burning fossil fuels are causing an increase in the Earth’s temperature?

Human activities are causing an increase in global temperatures.

Temperatures are increasing but not as a result of human activities.

Temperatures are not increasing.

Very willing Mildly willing Mildly unwilling Unwilling

I do what is right for the environment,even when it costs more money or takesmore time.

How willing would you be to accept cutsin your stand of living in order to protectthe environment?

How willing would you be to pay muchhigher prices in order to protect theenvironment?

How willing would you be to pay highertaxes in order to protect theenvironment?

2.82.8 Part Six: Control Part Six: Control

Are you male or female?

Male

Female

How old are you?

What is your current occupation? If more than one job, please indicate the job you work most in.

No paid employment

Responsible for ordinary shopping and looking after the home, or without any current occupation, not working

Student

Unemployed, temporarily not working

Retired or unable to work through illness

Other, please specify

Self-employed

Farmer (has own farm)

Fisherman


2 of 3 1/23/15 1:40 PM

19

Professional (lawyer, medical practitioner, accountant, architect, etc.)

Owner of a shop, craftsmen, other self-employed person

Business proprietors, owner (full or partner) of a company

Employed

Employed professional (employed doctor, lawyer, accountant, architect)

General management, director or top management (managing directors, director general, other director)

Middle management, other management (department head, junior manager, teacher, technician)

Employed position, working mainly at a desk

Employed position, not at a desk but travelling (salesmen, driver, etc.)

Employed position, not at a desk, but in a service job (hospital, restaurant, police, fireman, etc.)

Foreman or supervisor

Member of armed forces, security personnel

Skilled manual worker

Agricultural worker

Other (unskilled) manual worker, helper


2.92.9 Control Part II Control Part II

What is the highest level of education you have completed? Please select from the drop down menu.

--- please select ---Nursery school to 8th GradeHigh school, or the equivalentSome college credit, 2-year college completedCollege degree (e.g., B.A., B.S.)Master’s degreeDoctorate degree

What was you annual household income in 2014?

--- please select ---Less than $10,000$10,000 to $19,999$20,000 to $29,999$30,000 to $39,999$40,000 to $49,999$50,000 to $59,999$60,000 to $69,999$70,000 to $79,999$80,000 to $89,999$90,000 to $99,999$100,000 to $149,999$150,000 or more

What is your zip code?

This is optional!

2.102.10 Political views Political views

Which of the following categories best describes your political views?

Would you say that you are:

Strongly liberal

Liberal

Slightly liberal

Middle of the road

Slightly conservative

Conservative

Strongly conservative

Other, please specify:

2.112.11 Comments Comments

Please leave any feedback or comments below.

2.122.12 Final (Consent) Final (Consent)

DO NOT FORGET to copy the code you see below and paste it into the corresponding field on the Mechanical Turk HIT page. This is necessary so that we can approve the HIT!

#u_passwd#

This is a randomly generated code to confirm you have completed the survey. Please copy and paste it into the corresponding field in the Mechanical Turk HIT to have your HIT approved. After you have donethat, we recommend that you close this browser window.

Should you have problems with having your HIT approved or receiving your payment via Mechanical Turk, please look for a HIT called "ETH DeSciL Trouble Ticket", where you will receive further instructions.

Thank you very much for participating!

33 Final page (No Consent) Final page (No Consent)

Please return to Mechanical Turk. Thank you.


3 of 3 1/23/15 1:40 PM

Next for the WTO;

20

Print versionPrint version

QuestionnaireQuestionnaire

11 Part One: Consent Part One: Consent

Please read the following statement carefully. If you choose to participate, please click on the button next to “I have read and understood the consent form and agree to participate in this survey.” If youchoose not to participate, please click on the button next to "I do not wish to participate".

This survey is a component of a larger research project at ETH Zürich. The objective of the research project is to better understand personal opinions concerning international politics. This survey isadministered by Brilé Anderson and Dr. Aya Kachi and is for scientific purposes only. It has no commercial or government-related purpose. The duration of the survey is about 15 minutes.

There are no known risks for you if you decide to participate in this survey, nor will you experience any costs when participating in the survey. The information you provide will help us understand opinionsconcerning international politics. This survey is anonymous. The information you provide in this survey will not be stored or used in any way that could reveal your personal identity.

If you have any questions or concerns about completing this survey, you may contact either Brilé Anderson at [email protected] or Dr. Aya Kachi at [email protected].

The ETH Zurich Ethics Review Commission has reviewed and approved this project. If you have any concerns about your rights in this survey, please contact us at [email protected] or RaffaelIturrizaga from the ETH Zurich Ethics Review Commission at [email protected] or +41 44 632 2354 with reference to its decision EK 2012-N-41.

I have read and understood the consent form and agree to participate in this survey.

I do not wish to participate.

2.12.1 Consented Consented

We appreciate your participation! The next page shows an information pamphlet on the World Trade Organization also known as the WTO. Please read carefully. After finishing the pamphlet, please find the "continue" button at thebottom of the page.

Thank you!

2.22.2 Part Two: Information Part Two: Information

2.32.3 Part Three: Comprehension Part Three: Comprehension

We would like to ask you a few questions related to the previous page. Please note that you will not be able to proceed with the questionnaire until all questions on this page are answered correctly.Member countries in the WTO negotiate agreements on

Trade

Education

Terrorism

Cyber security

In the WTO framework, decisions are made by

Consensus of all participating countries

Majority of all participating countries

Rotating council of member countries

Board of directors comprised of member countries and NGOs

The WTO is a type of international organization.

Yes

No

2.4.1.12.4.1.1 Block No. 1 Block No. 1




The principles of this internationalorganization match mine.


1 of 3 2/6/15 5:41 PM

21

I do not trust this internationalorganization.

The international organization shouldcontinue to make decisions in the future.

My country should abide by the decisionsmade at this international organizationeven if it is inconvenient.

I feel that this international organizationis working towards a better life forpeople.

I sympathize with the purpose of thisinternational organization.

2.4.1.22.4.1.2 Block No. 2 Block No. 2




The morals of this internationalorganization are incompatible with myown.

I feel there is insufficient expertise andknowledge in this internationalorganization.

My country should follow therecommendations by this internationalorganization, even when the reasons fortheir decisions are not clear.

Certain countries will be favoured by thisinternational organization over others.

This international organization’sprocedure for making decisions is unfair.

All countries will be treated fairly by thisinternational organization.

2.4.1.32.4.1.3 Block No. 3 Block No. 3




I believe this international organization isnecessary.

The purpose of this internationalorganization is pointless.

This international organization has theright to make decisions.

I believe the existence of thisinternational organization is anembarrassment to society.

Regardless of convenience, othercountries should follow the decisions ofthis international organization.

I think my country should follow theproposals of the internationalorganization, even when my countrydisagrees, because it is this internationalorganization that proposed the policy.

2.52.5 Part Four-II: Validity Questions Part Four-II: Validity Questions

In this survey, all of the questions can be considered to be part of a broader concept in political science. Of the concepts listed below, which do you think best incorporates the previous questions?

Legitimacy

Accountability

Transparency

Participation

Trade

Performance

2.62.6 Validity II Validity II

Do you think the WTO legitimate?

Yes

No

2.72.7 Part Five: Topic Part Five: Topic

Some people have suggested placing new limits on foreign imports in order to protect American jobs. Others say that such limits would raise consumer prices and hurt American exports. Do you favour oroppose placing new limits on imports, or have you not thought much about this?

Favour

Oppose

Don't know

2.82.8 Part Six: Control Part Six: Control

Are you male or female?

Male

Female

How old are you?

What is your current occupation? If more than one job, please indicate the job you work most in.

No paid employment

Responsible for ordinary shopping and looking after the home, or without any current occupation, not working

Student

Unemployed, temporarily not working

Retired or unable to work through illness


Self-employed

Farmer (has own farm)

Fisherman

Professional (lawyer, medical practitioner, accountant, architect, etc.)

Owner of a shop, craftsmen, other self-employed person

Business proprietors, owner (full or partner) of a company

Employed

Employed professional (employed doctor, lawyer, accountant, architect)

General management, director or top management (managing directors, director general, other director)

Middle management, other management (department head, junior manager, teacher, technician)

Employed position, working mainly at a desk

Employed position, not at a desk but travelling (salesmen, driver, etc.)

Employed position, not at a desk, but in a service job (hospital, restaurant, police, fireman, etc.)


2 of 3 2/6/15 5:41 PM

22

Foreman or supervisor

Member of armed forces, security personnel

Skilled manual worker

Agricultural worker

Other (unskilled) manual worker, helper


2.92.9 Control II Control II

What was you annual household income in 2014?

--- please select ---Less than $10,000$10,000 to $19,999$20,000 to $29,999$30,000 to $39,999$40,000 to $49,999$50,000 to $59,999$60,000 to $69,999$70,000 to $79,999$80,000 to $89,999$90,000 to $99,999$100,000 to $149,999$150,000 or more

What is the highest level of education you have completed? Please select from the drop down menu.

--- please select ---Nursery school to 8th GradeHigh school, or the equivalentSome college credit, 2-year college completedCollege degree (e.g., B.A., B.S.)Master’s degreeDoctorate degree

What is your zip code?

This is optional!

2.102.10 Political views Political views

Which of the following categories best describes your political views?

Would you say that you are:

Strongly liberal

Liberal

Slightly liberal

Middle of the road

Slightly conservative

Conservative

Strongly conservative

Other, please specify:

2.112.11 Comments Comments

Please leave any feedback or comments below.

2.122.12 Final page Final page

DO NOT FORGET to copy the code you see below and paste it into the corresponding field on the Mechanical Turk HIT page. This is necessary so that we can approve the HIT!

#u_passwd#

This is a randomly generated code to confirm you have completed the survey. Please copy and paste it into the corresponding field in the Mechanical Turk HIT to have your HIT approved. After you have donethat, we recommend that you close this browser window.

Should you have problems with having your HIT approved or receiving your payment via Mechanical Turk, please look for a HIT called "ETH DeSciL Trouble Ticket", where you will receive further instructions.

Thank you very much for participating!

33 Final page Final page

Please return to Mechanical Turk. Thank you.


3 of 3 2/6/15 5:41 PM

23

B Data cleaning procedure

After cleaning the raw survey data, we retained 480 cases for the UNFCCC and the WTO each. (It is acoincidence that we ended up with the same size of observations for both organizations.) The followingis our data cleaning procedure.

Among those 512 participants for the UNFCCC and 509 participants for the WTO who agreed on the sur-vey consent statement, we dropped participants who dropped out of the survey after seeing the informationpamphlet, or after not answering correctly to the three factual questions about the given organization.Then we further dropped cases that went through the survey unreasonably fast. We used the followingcut-off criterion, which we consider conservative. The median survey duration was 6 minutes. Whenthe author (knowing the contents of the survey) carefully read and respond to each questions, it takesabout 124 seconds to complete the survey after reading he pamphlet. This might sound rather short butone should note that each survey item measuring legitimacy is very concise. We decided to cut off caseswhose duration was less than 120 seconds. We further dropped participants who did not answer all theenvironmental or trade attitude questions. The total became 480 for both organizations.

C Demographic characteristics of our pilot survey participants

This section reports our survey participants’ demographic characteristics. Our pilot survey includesgender, age, occupation, education (highest education attained), household income, and political ideologyon a liberal-conservative scale.

UNFCCC (N = 480)

Gender

Male: 286, Female: 194

WTO (N = 480)

Gender

Male: 246 , Female: 233

24

Age

Range 18 - 69, Median 30.0

Age

Range 19 - 82, Median 32.0

Occupation

The five most frequent occupations.(1) Employed position, but not at a desk - 104(21.67%)(2) Student - 52 (10.83%)(3) Unemployed, temporarily not working - 50(10.42%)(4) Employed professional - 49 (10.2%)(5) Owner of a shop, craftsmen, other self-employed person - 37 (7.7%)Total of the five = 292 (60.8%).

Occupation

The five most frequent occupations.(1) Employed position, working mainly at adesk - 95 (20%)(2) Middle management, other management -60 (12%)(3) Employed Professional - 49 (11%)(4) Students - 44 participants (9%)(5) Farmer (has own farm) - 37 participants(8%)

Education Education

25

Household income

Median: USD 30,000-39,000

Household income

Median: USD 40,000-49,000

Political ideology

Median: Slightly liberal

Political ideology

Median: Slightly liberal

26

D Participants’ predisposition to the two topical areas

This section reports our survey participants’ predispositions to the two topical areas, the environmentand international trade.

UNFCCC (N = 480)

Skepticism

Q: Do you think human activities such as burning fossil fuels are causing an increase in the Earthstemperature?

[1 Human activities are causing an increase in global temperatures. 2 Temperatures are increasing butnot as a result of human activities. 3 Temperatures are not increasing.]

“Anthropogenic”: 83.8%

General environmental attitude

Q1: I do what is right for the environment, even when it costs more money or takes more time.

[1 Very willing 2 Mildly willing 3 Mildly unwilling 4 Unwilling]

Q2: How willing would you be to accept cuts in your stand of living in order to protect the environment?

[Same as above]

Q3: How willing would you be to pay much higher prices in order to protect the environment?

[Same as above]

Q4: How willing would you be to pay higher taxes in order to protect the environment?

[Same as above]

27

We aggregated the above four questions regarding respondents’ general environmental attitudes in a simplelinear additive form (Franz and Vogen 2013) and created a variable named “environmental attitude”.Cronbach’s alpha is 0.89, suggesting a very high level of internal consistency among the four items. Thefinal variable is rescaled such that it ranges from 0 to 1 and lower scores indicate lower willingness forpro-environmental behavior. Figure 6 shows the frequency. The mean score is 0.59, suggesting thatrespondents are on average slightly, but not much, more willing to behave pro-environmentally than themiddle score. However, it is also noteworthy that there are two spikes at both ends of the distribution.

Figure 6: Frequency: Environmental attitude

WTO (N = 480)

Protectionism

Q: Some people have suggested placing new limits on foreign imports in order to protect American jobs.Others say that such limits would raise consumer prices and hurt American exports. Do you favour oroppose placing new limits on imports, or have you not thought much about this?

[1 Favour 2 Oppose 3 Don’t know]

Our respondents’ trade protectionist attitudes are almost evenly distributed across favoring, opposingand no opinion.

28

Figure 7: Trade protectionist attitutde

29

Documents

Measuring Public Perceptions of International ... · Measuring Public Perceptions of International Organizations’ Legitimacy Bril e Anderson ETH Zurich ECPR Joint Sessions Workshop