Towards Greater Objectivity in Governance Measurement: Second

Generation Citizen-Centric Governance Indicators

Draft

By Maksym Ivanyna, Joint Vienna Institute, Michigan State University,

and Anwar Shah, Bookings Institution, World Bank and SWUFE, China∗

July 06, 2015

Abstract Most widely used governance indicators (e.g. the World Bank’s Worldwide

Governance Indicators) lack information on how citizens evaluate their governments and suffer

from time and cross-country inconsistencies in their assessment methodologies. The first

generation citizen-centric governance indicators by Ivanyna and Shah (2011) attempted to

overcome this and other deficiencies of the WGIs by providing a framework for comparative

assessment of governance quality across countries and over time and by using data from World

Value Surveys to capture citizens’ perceptions of governance environment and outcomes in their

own countries. The citizen based evaluations while addressing the voids of experts’ purely

subjective perspectives, however, are not immune from systemic bias such as indoctrination,

intimidation, and critical citizenship. This paper presents methods for testing and correcting these

biases of public opinion surveys and develops a second generation citizen-centric governance

indicators that incorporate these adjustments. In doing so, it develops more reliable indicators of

governance quality across the globe. The correction procedures presented here would be helpful

in correcting for systemic biases in other measures that use public opinion survey data.

∗ Comments are welcome and may please be addressed to: mivanyna@jvi.org and Shah.anwar@gmail.com

1. Introduction

Governance quality assessments are now being used to judge various political regimes, conduct

development dialogue, allocate external assistance and to influence foreign direct investment.

In view of these uses, a comparative assessment of governance quality across countries and

benchmarking country performance over time is of growing interest to policy makers, citizens

and scholars alike. However, available worldwide governance indicators are in many aspects

inadequate for use in judging development effectiveness or in allocating external assistance

equitably based upon an objective criteria. 1

One of the most important limitations common to all available composite indexes of governance

is that they do not capture how citizens perceive the governance environment and outcomes in

their own countries. Most of the indexes are either solely based on external armchair experts’

evaluations or mix them with the citizens’ evaluations, with usually much smaller weight given

to the latter. For example, the Worldwide Governance Indicators (WGI), which are perhaps the

most widely used governance measures nowadays, in 2013 only 8 out of 150 representative

datapoints used by these indicators were based on public opinion surveys. If we simply average

these 8 datapoints and compare the resulting country ranking with that of the WGI itself (average

of six dimensions), the mean absolute percentile difference would be 24, i.e. with 215 countries

covered, an average country would be expected to change its ranking by about 50 positions up or

down. It suggests that in most countries, citizens’ assessment of governance quality is at various

with so-called international experts.

To overcome these deficiencies, Ivanyna and Shah (2011) developed a uniform and consistent

global framework for measuring governance based on citizens’ evaluations, the so-called citizen-

centric governance indicators (CGIs). They implemented this framework by using the first five

waves of the World Values Surveys, which combine decent geographical coverage with

acceptable number of governance-related questions.

This paper attempts to improve the measurement methodology used by Ivanyna and Shah (2011)

and to present up to date CGIs using the sixth wave of the WVS, conducted in 2010-2014. The

1 See, for example, Arndt 2008, Arndt and Oman 2006, Kurtz and Schrank 2007, Iqbal and Shah 2008, Langbein

and Knack 2008, Schrank and Kurtz 2008

paper streamlines the aggregation procedure and selects questions to ensure maximum

comparability over time. As a result, citizen-centric governance indicators are computed for 100

countries over six waves of the survey from 1980 to 2014, on average 2.5 times per each country

– 259 country-year measurements.

Public opinion surveys contains useful and unique information for measuring governance. But it

can also be subject to systemic biases, stemming for example from indoctrination in mass-media

or government’s oppression under totalitarian regimes, as ordinary people may be afraid to

answer questions about their governments truthfully for fear of reprisals. On the contrary in open

societies, people may have too high expectations of their government failing which they may

pass harsh judgments on their performance – the so-called phenomenon of “critical citizenship”

developed by Norris (1999). The presence of these biases may limit the usefulness of CGIs

comparisons across countries and through time. Good news is that in carefully constructed public

opinion surveys these biases can be corrected.

This paper develops and implements a method to correct public opinions about governance for

the systemic biases. This is done by by isolating these biases by separating personal (objective)

and government-related (subjective) questions in the surveys. Both groups of questions measure

the concept of governance, but only the latter one is likely to be subject to the above-mentioned

biases.

The paper is organized as follows. Section 2 briefly summarizes a citizen-centric conceptual

framework on measuring governance quality, presented first in Ivanyna and Shah (2011). Section

3 presents an empirical framework, data sources and aggregation techniques. Section 4 presents

the resulting CGIs without the adjustment. Finally, Section 5 discusses the correction for

systemic biases in public opinion, and the resulting adjusted CGIs.

2. Citizen-centric governance indicators: Conceptual framework

This section is a short summary of the corresponding part in Ivanyna and Shah (2011). Here, we

define “governance” as the norms, traditions and institutions by which power and authority in a

country is exercised—including the institutions of participation and accountability in governance

and mechanisms of citizens’ voice and exit and norms and networks of civic engagement; the

constitutional-legal framework and the nature of accountability relationships among citizens and

governments; the process by which governments are selected, monitored, held accountable and

renewed or replaced; and the legitimacy, credibility and efficacy of the institutions that govern

political, economic, cultural and social interactions among citizens themselves and their

governments (see also Huther and Shah, 1998). Note that this definition encompasses both the

governance environment (quality of institutions and processes) as well as governance outcomes.

A stylized view of the public interest can be characterized by four dimensions of governance

institutions, processes and outcomes.

• Responsive Governance. The fundamental task of governing is to promote and pursue

collective interest while respecting formal (rule of law) and informal norms. This is done

by government creating an enabling environment to do the right things – that is it

promotes and delivers services consistent with citizen preferences. Further, the

government carries out only the tasks that it is authorized to do, that is it follows the

compact authorized by citizens at large.

• Fair (equitable) Governance. For peace, order and good government, the government

mediates conflicting interests, is focused on consensus building and inclusiveness and

ensures a sense of participation by all and protection of the poor, minorities and

disadvantaged members of the society.

• Responsible Governance. The government does it right i.e. governmental authority is

carried out following due process with integrity (absence of corruption), with fiscal

prudence, with concern for providing the best value for money and with a view to earning

trust of the people.

• Accountable Governance. Citizens can hold the government to account for all its actions.

This requires that the government lets sunshine in on its operations and works to

strengthen voice and exit options for principals. It also means that government truly

respects the role of countervailing formal and informal institutions of accountability in

governance.

Given the focus on governance outcomes, Table 1 presents some preliminary ideas for discussion

on how to operationalize these concepts in individual country assessments.

Table 1 Governance outcomes and relevant considerations

Governance outcome Relevant considerations

Responsive governance - public services consistent with citizen preferences;

- direct possibly interactive democracy;

- safety of life, liberty and property;

- peace, order, rule of law;

- freedom of choice and expression;

- improvements in economic and social outcomes;

- improvements in quantity, quality and access of public services;

- improvements in quality of life;

Fair governance - fulfillment of citizens’ values and expectations in relation to participation, social justice, and due process;

- access of the poor, minorities and disadvantaged groups to basic public services;

- non-discriminatory laws and enforcement;

- egalitarian income distribution;

- equal opportunity for all;

Responsible governance - open, transparent and prudent economic, fiscal and financial management;

- working better and costing less;

- ensuring integrity of its operations;

- earning trust;

- managing risks;

- competitive service delivery;

- focus on results;

Accountable governance - justice-able rights and due process

- access to justice, information;

- judicial integrity and independence;

- effective legislature and civil society oversight;

- recall of officials and rollbacks of program possible;

- effective limits to government intervention;

- effective restraints to special interest capture.

Source: Shah (2008).

The above simple framework captures most aspects of governance outcomes especially those

relevant for development policy dialogue and can serve as a useful starting point for a consensus

framework to be developed. Once a consensus framework is developed then one needs to focus

on only a few key indicators that represent citizens’ evaluations and could be measurable with

some degree of confidence in most countries of the world and could be defended for their

transparency and reasonable degree of comparability and objectivity.2

Implementation of the above framework requires a worldwide survey with uniform questionnaire

honing on the four dimensions of governance identified above across countries. Such a survey is

yet to be developed. In the following section, we develop rough indexes of governance quality

based upon the survey data, which is already available.

3. Citizen-centric Governance: Empirical Framework

We use World Values Surveys (WVS) as our source of the survey data. It provides a reasonable

compromise between consistency with our conceptual framework and coverage of countries and

years. WVS publishes somewhat dated information (with the time lag of 2-3 years after actual

survey was taken), and only few questions from this survey are relevant for governance

assessments (since the survey is mainly about cultural values, not governance). However, WVS

provides quite comprehensive geographical coverage (100 countries with all major economies

included) combined with acceptable time coverage (six waves, from 1981 to 2014) and its

questionnaire is disclosed.

Appendix Table A1 presents the questions from the WVS that we use to measure governance.

Overall, we pick 15 questions. For quite a few sub-criteria from Table 1 no survey questions are

available. In principle, there are more governance-related questions in the surveys that we could

use, but they are not asked in most waves or in most countries. The questions we pick are asked

in at least 75% of all surveys in all waves, and in at least 90% of the surveys in fifth and sixth

wave. At least 10 sub-criteria from Table 1 can be covered by the questions with sufficient

representation throughout years and countries. The average coverage over all questions and

countries is 93% in all six waves, and 98% in the last two waves. In addition, we drop the

countries for which the coverage is less than 50% - seven out of 239 country-year surveys.

2 See (Andrews and Shah 2005) for details and relevant indicators of an approach that emphasizes citizen-

centric governance and (Shah and Shah 2006) for citizen-centered local governance and relevant indicators

The WVS provides raw survey data, so the observations can be sorted by gender, income,

education of a respondent, as well as by sub national administrative unit or other characteristics.

Then corresponding “adjusted” CGIs can be constructed.

3.1 Aggregation: Empirical model of governance

The underlying assumption of our empirical investigation is that the quality of governance

institutions and processes directly affects governance outcomes. Thus, the quality of governance

should correlate positively with the answers of survey respondents. At the same time, the

answers are random variables, which are subject to personal errors, including potential systemic

biases:

(1)

where is the index of a country, is the index of a respondent (total number of

respondents changes from country to country), and is the index of a question in a survey

(thus of a particular governance outcome). is the answer on question of the respondent

in the country . Each response is normalized to be between 0 and 1, with 0 being the worst

answer, and 1 being the best answer. is the quality of governance in the country , which

obviously does not depend neither on concrete respondent, nor on specific question. Finally,

is the random error, which is assumed to be independently normally

distributed with mean µik and the variance , and both may depend on country and specific

question. If µik is zero then provides an unbiased estimate of gi. This is assumption that we

make further in this section and in Section 4. If µik is not zero then the systemic bias in public

opinion is present, and it has to be corrected. We deal with the correction in Section 5.

Given the assumption of µik=0, the most efficient, unbiased, and consistent estimator for the

governance in country is just the sample mean of weighted averages of citizens’ responses.

The estimator for the governance’s variance is adjusted sample variation:

, (2)

,ijkijkiijkiijk sggs εε −=⇒+=

Mi ..1= iNj ..1=

Kk ..1=

ijks k j

i

ig i

),(~ 2..

ikik

di

ijk N σµε

ik2σ

ijks

i

∑ ∑∑∑∑= === =

−−

==i ii N

j

N

jijk

iijk

i

K

kk

N

j

K

kiijkk

ii s

Ns

Nwgsw

Ng

1

2

11

2

1 1)1(

11)r(av,1ˆ

where the weights w for each question are chosen to minimize the variance of governance

indicator. Roughly speaking, questions with smaller variance of the measurement error ε should

get bigger weight. Since σ’s are not observed, the eventual choice of weights is effectively

arbitrary. We take parsimonious and the most comprehensible approach, and assign equal weight

to each “covered” sub-criterion in Table 1, with the exception of few questions, which seem to

be relatively more far-reaching in assessment of governance (i.e. “satisfaction with life in

general” is clearly more comprehensive than “satisfaction with health”). Such questions get

bigger weight. All weight sum up to one.

4. Citizen-centric governance indicators without adjustment

The maps of the citizen-centric governance indicators for wave 5 (2005-2009) and wave 6 (2010-

2014) of the survey are presented in Figure 1. The surveys were taken in 57 and 59 countries

accordingly. The countries are grouped into four quartiles: the darker is the color the higher is

the CGI. Table 2 presents summary statistics of CGIs over all six waves.

Table 2 CGIs unadjusted: Summary statistics

Wave N Mean S.d. 10th perc. 90th perc

1 (1981-1984) 10 0.61 0.05 0.55 0.68

2( 1989-1993) 17 0.55 0.06 0.47 0.62

3 (1994-1998) 51 0.54 0.07 0.44 0.62

4 (1999-2004) 38 0.57 0.08 0.47 0.67

5 (2005-2009) 57 0.57 0.07 0.47 0.67

6 (2010-2014) 59 0.58 0.07 0.48 0.67

Table 2 shows that quality of governance in 80% of the countries ranges roughly between 0.44

and 0.67 in all six waves, on the [0,1] scale. The mean CGI remains practically unchanged

(except between waves 3 and 4), although the sample of countries changes with each wave.3

3 Ivanyna and Shah (2011) show that in overlapping samples of countries CGIs improved statistically significantly

from wave 3 to wave 4, and did not change from wave 4 to wave 5.

Several observations on individual countries emerge from Figure 1. As expected, most developed

countries (especially Scandinavian countries, Switzerland, Canada, New Zealand) demonstrate

stable and good performance. At the same time, a number of developing countries are also

among the top-performers. East Asian governments (especially, Vietnam, China) get particularly

high plaudits from their citizens. Governance quality is also rated well by the citizens of Ghana,

South Africa, and Turkey. This may appear surprising to a reader in OECD countries, but can be

justified as these governments have an excellent track record of improving economic and social

outcomes for their residents and hence captured in their citizen’s evaluations. On the other side,

countries in Central and Eastern Europe and Latin America consistently get the lowest scores

from their citizens.

Figure 1 Citizen-centric governance indicators (data source WVS, waves 5 and 6)

Figure 2 shows the relationship between the citizen-centric governance indicators and the UN’s

Human Development Index. In general the correlation is positive, but the fit, is very bad.4 Many

countries, which are ranked low by the HDI, get high scores from the CGI. Similar picture

emerges if we compare CGI with other proxies of quality of life or goverance: GDP per capita,

perceived levels of corruption, etc. One reason for this is that public opinion contains

information about governance institutions and outcomes, which is not present in other measures.

Another potential reason could be that public opinion is subject to systematic biases. We explore

the latter in the next section.

4 We exclude SSA countries from the figure for the representation purposes. These countries make the fit even

worse, although the point estimate of the slope of the fitted line is still positive.

Figure 2 Citizen-centric governance indicators vs. Human Development Index, wave 6

5. Removing systematic biases from public opinion

While CGIs may contain information about governance, which is not present in other measures,

public opinion may also suffer from various kinds of systematic biases.5 If this is the case then in

(1) the mean of the random error µik is not equal to zero, so one cannot extract gi from the

citizens’ responses by simply taking a weighted average. The systemic biases distort the real

picture of citizens’ assessment of governance, and in particular make cross-country comparison

misleading. In this section we discuss and implement alternate methods to correct for these

biases.

5 See Bertrand et al. 2001, Heath et al. 2005, King et al. 2004, MacKerron 2012, Olken 2009 for the discussion on

subjective measurements

Public opinion, in particular on the government-related issues, may suffer from at least three

systemic biases. First is the “intimidation” effect, when people are afraid to express their true,

usually negative, opinion about their government, because they think they could be punished for

that. The second bias is the “indoctrination” effect, when mass media in a country praise or

criticize the government disproportionately, so that it distorts public opinion. The third factor,

which may bias public opinion, is the degree of citizen activism and perceived role of

government in a country. In particular, (Norris 1999) argues about the emergence in the 70s in

developed countries of a class of so called “critical citizens” – people, who raise the bar for

government performance to unrealistic levels and become vocal critics once those high

expectations are not met in practice. Adjusting for this “excessive” criticism as well as for

intimidation and indoctrination, is vital for valid cross-country comparison. Below we suggest

how to do it within the WVS framework, and then we discuss other potential biases.

Intimidation, indoctrination and “critical citizenship” affect respondents' answers together with

the quality of governance. We can write down the random error εijk as a sum of the biases and the

true random error, and rewrite (1):

(3)

where similarly to the notation in Section 3.1, is a response of individual in country on a

question , is the quality of governance in country , are the degrees of

intimidation, indoctrination and critical citizenship of individual in country (independent of

question k), and is citizen-, country- and question-specific random error. Its mean is zero if

there are no other systemic biases in the public opinion. and are the coefficients of our

interest. We assume that they can vary by country, as well as by question (f.e. respondents may

be more intimidated to answer some of the questions than others).

The estimation of and is not possible from (3), since we do not observe . However,

the problem can be resolved if we note, or rather assume, that for some questions (call them

“objective”) the effect of “intimidation”, “indoctrination” or “critical citizenship” is likely to be

very low to zero, and for some the effect is likely strong. For instance, when an individual is

asked about the satisfaction with her/his health, it is likely that she/he will not be intimidated to

,_ ijkijikijikijikiijk citcrindings ζµηγ ++++=

ijks j i

k ig i i ji ji j c ic ri n di n _,,

j i

ijkζ

ikik ηγ , ikµ

ikik ηγ , ikµ ig

tell the truth. At the same time, questions like “Do you have confidence in your government?”

may invite all above mentioned biases. Therefore, we can isolate the effect of biases on the

individual responses by differentiating (diffij) between “subjective” and “objective” questions.

Suppose of K questions 1st to K1th are subjective, and the rest are objective. Their corresponding

weights in the CGI are and , see (2). Then for the subjective

questions:

(4)

For the objective questions biases are assumed to be absent:

(5)

Taking a weighted difference between (4) and (5) we can get rid of gi on the right hand side and

get an expression of only observables and random error:

(6)

where γ’, η’, and μ’ are the original coefficients from (3), multiplied by wsub. Expression (6) can

be estimated directly using OLS provided we have corresponding data on indoctrination,

intimidation and critical citizenship. The estimated coefficients can then be used to derive the

true gi from the citizens’ responses (adjusted CGIs):

(7)

We assume the following questions (governance outcomes) to be independent from the

biaseffects:6

6 Not all questions used in the estimation of g must be used in the identification of biases. Questions in doubt (e.g. if

it is hard to say whether the question is objective or subjective) can simply be left out. The bias identification

procedure described in (4)-(6) still carries through as long as there are at least some questions in both groups.

∑ == 1

1

K

k ksub ww ∑ ==

K

Kk kobj ww1

.)_(11

11∑∑==

++++=K

kijkkijiijiijiisub

K

kijkk wcitcrindingwsw ζµηγ

.11

∑∑==

+=K

Kkijkkiobj

K

Kkijkk wgwsw ζ

,'_' '''

11 1

1

ijkijiijiijii

K

Kkijk

obj

subK

kijkkij citcrindins

ww

swdiff ζµηγα ++++=−= ∑∑+==

)_ˆˆˆ(1 '''

1 1ijiijiiji

N

i

K

kijkk

ii citcrindinsw

Ng

i

µηγ ++−= ∑∑= =

- O1. How satisfied are you with the financial situation of your household?

- O2. All things considered, how satisfied are you with your life as a whole these days?

- O3. All in all, how would you describe your state of health today?

- O4. Taking all things together would you say you are [happy, unhappy]?

The following questions, however may invite systemic biases:

- S1. How much confidence do you have in the government?

- S2. How much confidence do you have in the parliament?

- S3. How much confidence do you have in the press?

- S4. How much confidence do you have in the television media?

- S5. How much confidence do you have in the courts?

- S6. How much confidence do you have in the police?

- S7. How much confidence do you have in the armed forces?

- S8. How much confidence do you have in the civil services?

- S9. How much respect is there for individual human rights nowadays in the country?

- S10. How proud are you of your nationality?

- S11. Would you fight for your country?

We use two steps to estimate and - effects of intimidation, indoctrination and “critical

citizenship”. First, we test for indoctrination ( ) on an individual response level within each

country. Then we use country-level regressions to identify effects of intimidation and critical

citizenship. We cannot use individual response level, as we do not have proxies for personal

intimidation or “critical citizenship”.

Adjustment of CGI (7) is less straightforward though as one needs to decide whether the questions in doubt contain

bias and to what extent.

ii ηγ , iµ

iη

5.1 Testing for indoctrination

To identify the indoctrination effect we test whether the individual subjective-objective questions

difference (diffij) depends on the exposure to the mass-media – press, TV and internet. If there is

a statistically significant relationship it means the that mass-media systematically skews citizens’

perceptions/confidence about the government relative to citizens’ perceptions about their own

financial situation, health and life overall, which in turn depend on the governance environment.

To measure individual exposure to the mass media we use the corresponding questions in the

WVS. For the wave 6 of the survey the question is: “'People learn what is going on in this

country and the world from various sources. For each of the following sources, please indicate

whether you use it to obtain information daily, weekly, monthly, less than monthly or never”.

The sources are “daily newspaper” for press exposure, “TV news” for TV exposure, “internet”

for internet exposure. All answers are rescaled to vary from 0 (never) to 1 (daily). For the wave 5

of the survey the mass media related questions are more rudimentary, and hence less suited to

measure the exposure: “Did you watch TV/ read newspapers / read news in internet during the

last week?”. Hence, the exposure variables are dummies: 0 means “No”, 1 means “Yes”.

Nevertheless, we test for indoctrination in both waves. The more people watch TV or read

newspaper / internet news the more they are exposed to possible indoctrination (or excessive

criticism of mass-media).

The statistical summary of all mass media exposure variables is presented in Table 3. The

television remains the dominant source of information for an average person in the world.

Internet is still lagging behind the traditional mass media, even in 2010-2014, but its role is

increasing rapidly (an increase from 0.28 in wave 5 to 0.45 in wave 6), while that of press and

TV remains stagnant.

Table 3 Mass media exposure variables: Summary statistics

Variable N Mean S.d. 10th perc. 90th perc

Wave 5:

Press exposure 78256 0.55 0.48 0 1

TV exposure 75854 0.88 0.33 0 1

Internet exposure 74231 0.28 0.45 0 1

Wave 6:

Press exposure 83109 0.57 0.39 0 1

TV exposure 81063 0.88 0.25 0.5 1

Internet exposure 82696 0.45 0.45 0 1

The exact econometric specification that we use is the following:

(7)

where is a set of individual demographic variables: 1 if unemployed, 1 if student, 1 if

retired, 1 if female, education (1 to 7), relative income (1 to 10), age, 1 if married, interest in

politics (0 – not at all interested to 1 – very interested), and political preferences (1 – left to 10 -

right). These variables are likely to be correlated both with diff and with the exposure to mass

media, so they need to be included in the regression.

The linear regression (7) is run separately by each country-wave – 50 regressions for wave 6, and

46 regressions for wave 5. The summary of coefficients for wave 6 regressions is presented in

Table 4. Magnitudes of the coefficients by countries are presented in Table 5.

First point to note from the tables is that the TV seems to be the primary mean for the

indoctrination and government appraisal in the world. The mass media bias on TV seems to be

present (statistically significant) in 34 countries out of 50, which we tested. Only in one country

out of 34, the TV media is actually too critical about the government. The average coefficient on

the TV exposure is 0.03, and it can go as high as 0.11. With on average 88% respondents

watching the TV news daily the indoctrination could increase CGIs by as much as 9.7 percentage

points – a magnitude large enough to move a median country to as low as 10th percentile.

Table 4 Effect of indoctrination: Summary statistics for wave 6

Variable N Mean S.d. (mean) Min Max

,'' 321 ijkijiijiijiijiiij demogrinetpresstvdiff εϑηηηα +++++=

ijdemogr

Press exposure 50 0.004 0.012 -0.053 0.053

TV exposure 50 0.03 0.019 -0.032 0.105

Internet exposure 50 -0.016 0.011 -0.085 0.02

Indoctrination adjustment in

CGI, total

50 0.022 0.027 -0.035 0.092

Table 5 Mass media bias in public opinion: Magnitude of coefficients by country, 2010-2014

Indoctrination, magnitude of coefficients (

)

Press exposure TV exposure Internet exposure

0.04-0.11 EGY'13, EST'11 LBY'14, TWN’12, CHL'11, RUS'11, SWE'11, COL'12, BLR'11, ARM'11, KOR'10, POL'12, DZA'13, KGZ'11, JPN'10, NZL'11, BHR'14,

TUN'13

0.02-0.04 KAZ'11, ROM'12, WBG'13, LBN'13, AUS'12, IRQ'12, ZAF'13, SWE'11, LBY'14, UKR'11, AZE'11, BLR'11

IRQ'12, LBN'13, TUR'11, RWA'12, NLD'12, MYS'12, CYP'11, MEX'12, AUS'12, DEU'13, UZB'11, USA'11, EST'11, TTO'11, KAZ'11,

ZAF'13, IND'14

PAK'12

-0.02-0.02 30 countries 16 countries 29 countries

-0.04 - -0.02 IND'14, PAK'12, TUN'13 PAK'12 TUR'11, CHL'11, BRA'14, UZB'11, ARM'11, PHL'12, CYP'11, EST'11, YEM'14,

ZWE'12, KOR'10, MYS'12, RUS'11, UKR'11, ROM'12

-0.09 - -0.04 TTO'11, RWA'12, THA'13 BHR'14, BLR'11, THA'13, DZA'13, KAZ'11,

Note: The country-year pairs for which the indoctrination effect was estimated: ARM'11 AUS'12 AZE'11 BHR'14 BLR'11 BRA'14 CHL'11 COL'12 CYP'11 DEU'13 DZA'13 ECU'13 EGY'13 EST'11 GHA'12 IND'14 IRQ'12 JPN'10 KAZ'11 KGZ'11 KOR'10 LBN'13 LBY'14 MEX'12 MYS'12 NGA'11 NLD'12 NZL'11 PAK'12 PER'12 PHL'12 POL'12 ROM'12 RUS'11 RWA'12 SVN'11 SWE'11 THA'13 TTO'11 TUN'13 TUR'11 TWN’12 UKR'11 URY'11 USA'11 UZB'11 WBG'13 YEM'14 ZAF'13 ZWE'12. Coefficients in the category “-0.02-0.02” are all statistically insignificant at 10% level and less.

While newspapers seem to be mostly neutral or sympathetic to the government, but to a much

lesser degree than the television, internet news follow a completely opposite pattern. In 40% of

iii 321 ,, ηηη

the countries in our sample internet news seem to bias the public opinion about the government

downwards. In 60% of the countries internet coverage is neutral, which is much better than for

the TV. The average magnitude of the coefficient on the internet exposure is -0.02, and it can go

as low as -0.09. In some countries overly “critical” internet news seem to provide a balance to

overly “favorable” TV news: countries like Bahrain, Belarus, Kazakhstan, Algeria are among the

“leaders” in the indoctrination by the TV, and at the same time their internet news segments are

one of the world’s most critical.

The internet does not balance out the TV’s indoctrination bias completely. In absolute terms the

coefficients on the internet exposure are on average smaller than the coefficients on the TV

exposure. In addition, people still rely more on the TV than on the internet as their source of

news. As a result, the average indoctrination bias in the CGIs (last line in Table 4) is positive –

0.02, which is around a third of CGIs’ standard deviation. The bias can be as high as 0.09

(Tunisia).

The second noteworthy point is that even though a lot of developing countries, especially those

in Middle East, Europe and Central Asia, seem to be indoctrinated, the mass media bias is also

present in many developed countries – e.g. in Japan, Sweden, New Zealand, USA, Germany,

Netherlands, Australia. Given that the mass media in these countries are unlikely to be captured

by the government, it signifies presence of some sort of ideological or media ownership biases in

the news coverage in these countries. For most of the countries a good practice to get a balanced

view on the government is to diversify sources of information.

5.1 Testing for intimidation and critical citizenship

We do not observe intimidation and “critical citizenship” at the individual respondent level, so

we test for the presence of these biases in a cross-country regression. As a proxy for the

intimidation we use the average score of the country in the “Freedom in the World” index – an

annual publication of the Freedom House, where political and civil rights of the citizens are

assessed. The index varies from 1 (most free) to 7 (least free). Countries rated 1 to 2.5 are

considered “free”, and countries rated 5.5 to 7 are considered “Not Free”. The expectation is that

in more oppressed countries respondents’ public assessment of their governments is more

favorable than their true assessment.

As for the “critical citizenship”, we follow (Norris, 1999) in her definition of a “critical citizen”,

and define the country to be in the stage of “critical citizenship” if it had been classified “Free”

by the Freedom House for at least ten years before the survey was conducted (long period of

stable democracy), and the real GDP per capita in this country (in 2005 prices) was more than 10

thousands US dollars (wealthy population). Most OECD countries enter the group.

Our econometric specification is the following:

(8)

where is the “Freedom in the World” index, is the “critical citizenship”

dummy, diffi is the subjective-objective questions difference, averaged over all residents of

country i and after the adjustment for the indoctrination.

The estimation results are presented in Table 6. We find that both freedom of the county and its

status of “critical citizenship” are statistically significant in explaining biases of subjective

questions in the WVS surveys. The directions of the effects are as expected. One score up in the

Freedom House ranking (which means country becomes less free) does make people more

cautious in answering government-related questions in a public opinion survey, and consequently

overpraise their governments by 0.013 points. From the other side, residents of the countries,

which are in a stage of “critical citizenship”, do have significantly less confidence in their

governments then they should have had. If not hyper “critical”, the residents of these countries

would have given their governments a score 0.039 points higher. The total CGI adjustment for

both biases varies from -0.09 in Uzbekistan to 0.026 in most OECD countries.

Table 6 Effect of indoctrination and "critical citizenship"

Dependent variable - diff

Coef.

freedom 0.013***

,_ iiii citcrfreedomdiff εµγα +++=

ifreedom icitcr _

(0.004)

cr_cit -0.039**

(0.017)

R-squared 0.22

Number of obs 142

Note: *** - significant at less than 1% level ** - significant at 5% level. Method of estimation – OLS. Standard errors in parentheses.

5.3 Adjusted CGIs

Figure 5 CGIs, wave 6, adjusted for indoctrination, intimidation and critical citizenship

Figures 5 and 6 report the adjusted CGIs in 2010-2014 and their comparison with the Human

Development Index. First thing to note is that the adjusted CGIs fit HDI much closer than the

non-adjusted CGIs,7 the relationship becomes statistically significant. Same goes with other

measures of quality of life and governance. Western European countries, U.S., New Zealand, and

Australia are now in upper half of the ranking. Latin American countries somewhat improve

their rankings. Yet, even after the adjustment CGIs still seem to contain information, which is

not captured by other governance and life quality indexes. The scores of East Asian countries,

Uzbekistan, Ghana go down, but these countries still remain in the upper half of the country

ranking. Apparently, there are other reasons for some governments to score so high in the public

opinion polls. In case of East Asia the main of them is probably last decade’s stable economic

growth and development in the region (as it is argued for China by Wang 2005). At the same

time, poor economic performance, political conflicts and corruption in the 90s (and for many

countries up until today) in Central and Eastern European countries keep the scores of the

governments in this regions quite low (although Poland shows significant progress in the last

wave).

7 We still exclude SSA countries from the figure. Including them reduces the fit, but the slope estimate is still

positive.

Figure 6 Adjusted CGIs vs. Human Development Index, wave 6

5.4 Robustness checks, limitations and further research

Our results are robust to a number of checks. First, we change sets of objective and subjective

questions.8 Second, we try different sets of auxiliary control variables in specification (7). Third,

we try different definitions of “critical citizenship” in (8).9 None of the checks leads to a

qualitative changes in the results. Set of countries with significant indoctrination effect remains

almost unchanged. Coefficients on intimidation and “critical citizenship” in (8) remain

significant. Magnitudes of coefficients statistically do not differ from our main specification.

8 In particular, we exclude questions S6-S11 from the list of subjective questions, O2 and O4 from the list of

objective questions.

9 E.g. five years of being “free” and GDP per capita = 5000, or 15 years of being “free” and GDP per capita =

15000.

Even though robust to a number of robustness checks, our procedure of public opinion

adjustment is subject to certain limitations.

First, it rests on the assumption that some governance-related questions are objective, i.e. they

are not subject to systematic biases that we consider. If this is not the case than the objective-

subjective questions differentiation would not capture the full extent of the biases, only its lower

boundary. For example, if in country X people are indoctrinated to overpraise their government,

and they are also led to think too favorably about their personal financial situation and health,

our procedure will only correct for the difference between the two biases. It is reassuring though

that our results do not change much after we rebalance the sets of objective and subjective

questions to exclude the most “doubtful” ones (e.g. “Are you proud of your nationality?”).

Additionally, even if we underestimate the magnitudes of the biases, their signs are likely to be

estimated correctly. It is quite plausible to assume that the questions which we deem “objective”

contain less bias than the questions which we deem “subjective”.

The second limitation of our procedure is that it may not correct for all the biases, which affect

public opinion. For historical, cultural, religious or other reasons respondents may have different

perception of the same governance effort. These differences can translate to a systematic bias at a

country level, which invalidates the cross-country comparison. If the bias appears in both

“objective” and “subjective” questions, then it is particularly hard to identify. We believe,

however, that provided we know the nature of the bias, it is almost always possible to construct a

public opinion survey so that this bias can be later adjusted for.10 This can often be done even

with surveys, like WVS, which were not initially designed to measure governance.

One potential alternative way to adjust for biases is to contrast respondents’ perception with

objective data.11 For example, excessive optimism or pessimism could be identified by looking

at, say, respondents’ perception of their health or crime situation in their neighborhood, and

comparing them with the actual situation. In WVS we do not observe the state of health at a

respondent level, which would be desirable, but we can aggregate the responses at a country

level and contrast them with the country-specific objective data. As a demonstration, Figure 7

10 King et al 2004, King et al 2007, MacKerron 2012 have similar conclusions

11 Approach is similar to Jürges 2007, Olken 2009

shows the average response to a question “How would you describe your state of health today?”

vs. life expectancy, both at a country level. 12 We do find positive correlation between the two:

countries where people live longer rate their state of health higher. Yet, we also find significant

heterogeneity between the countries. For example, in Pakistan in 2012 most people rated their

state of health as good or very good, score of 0.77, whereas in Japan in 2010 the score was more

than 10 percentage points lower. The difference in life expectancy between the two countries is

18 years. It implies that Pakistan’s people are 15% too optimistic relative to the objective

measure of health, while Japan’s people are 16% too pessimistic. In general, the most pessimistic

regions are Central and Eastern Europe and developed countries in East Asia. South Asia and

Middle East seem to be the most optimistic. Interestingly, 80% of the countries in top-20 by the

suicide rate,13 for which we have data, are also in our 25th percentile by the pessimism.

Assuming people apply the same degree of optimism to all other governance-related questions,

CGIs can be adjusted accordingly. For example, Pakistan’s score in 2012 goes down from 0.56

to 0.47 (35th percentile to 70th). Japan’s score in 2010 goes up from 0.52 to 0.61 (46th to 1st

percentile). Our main conclusion carries through though: adjusted CGIs are positively correlated

with quality of life indicators , but they still contain a significant amount of unique information.

12 We exclude Sub-Saharan African countries from the figure because of the unusually high scores these countries

have. Apparently, questions about health carry a certain stigma in these countries, which prompts people to severely

overreport it. We do include these countries in the regression analysis, and add the corresponding regional dummy.

13 According to the World Health Organization. Accessed from:

http://en.wikipedia.org/wiki/List_of_countries_by_suicide_rate (May 3, 2015)

Figure 3 Perception of health (WVS) vs. life expectancy, 2004-2014

6. Conclusions

Listening to the opinions of local residents (nationals) through public opinion surveys is critical

to measuring and monitoring governance as demonstrated by Ivanyna and Shah (2011). This

paper attempts to refine the methodology of such assessment by proposing correction methods to

mitigate systemic biases in opinion surveys as well to update results using recently released data

from Sixth Wave (2010-2014) of World Values Surveys.

The paper identifies and corrects for biases in governance assessment arising from

indoctrination, intimidation, and critical citizenship.. The paper concludes that such adjustments

would invite greater confidence in governance indicators and the resulting measures of citizen

centered governance offer better cross-country and time series comparisons.

A major limitation of the approach advocated in this paper arises from the non-availability of

worldwide survey on governance quality using a uniform questionnaire across countries and over

time. While the WVS is an excellent open source of information, with reasonable geographic,

time and conceptual coverage, the proper measurement of governance requires a specifically

designed survey with stratified random sampling employing a uniform questionnaire across

countries and over time. Differentiating “objective” (personal) from “subjective” (trust)

questions about the government in such survey would enable a researcher to test for the presence

of systemic biases and carry out appropriate corrections. Such corrections could be further

facilitated with comparison of responses with objective data on quality of life indicators from

other sources. This paper has taken first step in achieving greater objectivity in governance

assessment. Much further work lies ahead in establishing full confidence in the validity of

governance indicators for comparative and benchmarking purposes.

REFERENCES

Andrews , M. and A. Shah (2005), “Citizen-Centered Governance: A New Approach to Public

Sector Reform”, in A. Shah, eds., Public Expenditure Analysis, Washington, DC: World Bank,

Chapter 6:153-82.

Arndt, Christiane (2008), “The Politics of Governance Ratings”, International Public

Management Journal 11(3): 275-297

Arndt, C. and C. Oman (2006), Uses and Abuses of Governance Indicators, Development Center

Studies, OECD.

Bertrand, M. and S. Mullainathan (2001), “Do people mean what they say? Implications for

subjective survey data”, American Economic Review 91 (2): 67–72.

Heath, Anthony, Stephen Fisher and Shawna Smith, “The globalization of public opinion

research” (2005), Annual Review of Political Science 8: 297-333

Huther, J. and A. Shah (1998), “Applying a Simple Measure of Good Governance to the Debate

on Fiscal Decentralization”, Policy Research Working Paper 1894, World Bank, Washington,

DC.

Iqbal, K. and A. Shah (2008), “Truth in Advertisement: How do governance indicators stack

up?”, unpublished paper, World Bank Institute, Washington, DC , available at

http;//www.worldbank.org/wbi/publicfinance.

Ivanyna, Maksym and Anwar Shah (2011). Citizen-centric Governance Indicators: Measuring

and Monitoring Governance by Listening to the People . CESifo Forum, 1/2011: 59-71

Jürges, Hendrik (2007), “True health vs. response styles: exploring cross-country differences in

self-reported health”, Health Economics 16(2): 163-178

King, Gary, Christopher Murray, Joshua Salomon and Ajay Tandon (2004), “Enhancing the

validity and cross-cultural comparability of measurement in survey research”, American Political

Science Review 97(4): 567-583

King, Gary and Jonathan Wand (2007), “Comparing incomparable survey responses: Evaluating

and selecting anchoring vignettes”, Political Analysis 15(1): 46-66

Kurtz, M. and A. Schrank (2007), “Growth and Governance: Models, Measures and

Mechanism”, The Journal of Politics 69(2), 538-54.

Langbein, L. and S. Knack (2008), “The Worldwide Governance Indicators and Tautology”,

Policy Research Working Paper no. 4669, July 2008, World Bank, Washington, DC.

MacKerron, George (2012), “Happiness economics from 35000 feet”, Journal of Economic

Surveys 26(4): 705-735

Norris, P. (1999), Critical Citizens: Global Support for Democratic Governance, New York:

Oxford University Press.

Olken, Benjamin (2009), “Corruption perceptions vs. corruption reality”, Journal of Public

Economics 93(7-8): 950-964

Shah, A. (2008), “Demanding to be Served: On Holding Government to Account for Service

Delivery”. In J. de Jong and G. Rizvi , eds., The State of the Access, Washington, DC:

Brookings Institution Press.

Shah, A., and S. Shah (2006), “The New Vision of Local Governance and the Evolving Role of

Local Governments”, in A. Shah, eds., Local Governance in Developing Countries, Washington,

DC: World Bank, Chapter 1: 1-46.

Schrank, A. and M. Kurtz (2008), “Conceptualizing and Measuring Institutions: A View from

Political Science”, unpublished paper, Department of Political Science, Ohio State University

Wang, Z. (2005), “Before emergence of Critical Citizens: Economic Development and Political

Trust in China”, International Review of Sociology 15(1), 155-71.

(WBI) World Bank Insitute (2008), “Worldwide Governance Indicators”,

(info.worldbank.org/governance/wgi/index.asp)

(WVS) World Values Survey (2008), “Official Datafile v.20081015”, World Values Survey

Association (www.worldvaluessurvey.org), aggregate file producer: ASEP/JDS, Madrid.

APPENDICES

TableA1 Governance outcomes: weights and questions assigned

Code Governance criteria

Questions assigned Weight Coverage, %, all waves

Coverage, %, 5th and 6th wave

A Responsive governance

1 safety of life, order, rule of law

How much confidence do you have in police? 0.035 95.8 98.3

3 improvements in economic and social outcomes

How satisfied are you with the financial situation of your household?

0.15 98.3 99.2

4 improvements in quality of life: general

All things considered, how satisfied are you with your life as a whole these days?

0.1 99.2 100

5 improvements in quality of life: health

All in all, how would you describe your state of health today?

0.07 97.5 100

6 peace How much confidence do you have in armed forces?

0.035 94.6 95.7

7 inmprovements in quality of life: happiness

Taking all things together would you say you are [happy, unhappy]?

0.1 99.6 100

B Fair governance

1 social justice, respect for human rights

How much respect is there for individual human rights nowadays in the country?

0.08 76.5 94

2 government represents the whole country

How proud are you to be your nationality? 0.035 98.7 99.1

3 government represents the whole country

Would you fight for your country? 0.035 94.4 99.1

C Responsible governance

1A earning trust: executive branch

How much confidence do you have in government?

0.075 87.9 98.3

1B earning trust: legislative branch

How much confidence do you have in parliament?

0.075 95.4 97.4

2 earning trust: general

How much confidence do you have in civil services?

0.07 96.2 98.3

D Accountable governance

1A access to information, independent mass media - press

How much confidence do you have in press? 0.035 97.1 99.1

1B access to information, independent mass media - television

How much confidence do you have in television?

0.035 89.1 100

2 judicial integrity and independence

How much confidence do you have in courts? 0.07 79.9 96.6

Note: The data source for all questions is World Values Surveys (WVS). Questions are common to all six waves of the survey.

Table A2 CGI, raw and adjusted, waves 5 and 6