Measurement and Concept Formation

Meredith Rolfe

Centre for Corporate Reputation, Saïd Business School

Nuffield College, University of Oxford

In physical science the first essential step in the direction of learning any subject is to find

principles of numerical reckoning and practicable methods for measuring some

quality connected with it. I often say that when you can measure what you are speaking

about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in

numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of

knowledge, but you have scarcely in your thoughts advanced to the state of Science,

whatever the matter may be.

Baron William Thomson Kelvin, Popular Lectures and Addresses 1:73 (originally: Lecture to the Institution of Civil Engineers, 3 May 1883)

Theoretical System

Steps in a Measurement Process

1. Identify theoretical concepts

2. Choose one or more indicators/variables1. Brainstorm possible variables

2. Choose operational rule or definition

3. Level of measurement

4. Measurement technique

3. Assess Measurement Reliability & Validity

Theoretical concepts related to states:democracyregime typeeconomic growthstate capacity

Individualsincomeauthoritarianismpolitical interestsocial class

Step 1) Identify theoretical

constructs or concepts

There have been many authorities who have asserted that the basis of science

lies in counting or measuring, i.e. in the use of mathematics. Neither

counting nor measuring can however be the most fundamental processes in our study of the material universe—

before you can do either to any purpose you must first select what you

propose to count or measure, which presupposes a classification.

Roy Albert Crowson, Classification and Biology (1970)

Step 2) Variables: operational rules

to turn observations into data

Democracyfair electionsGNP/capita

State capacitysize of militaryexpenditures

Violence, war or military conflictDeaths (civilian or otherwise?)expenditures

Possible indicators of democracy

Procedural (free and contested elections)Multi-partyCampaign financingBalance of powersUniversal suffrageOne person, one voteCollective deliberationAccountability of elected officialsProperty rightsCitizens have a voice in governmenteconomic outcomes

Variables:Levels of measurement

Nominal Algeria, Russia (country)gender or ethnicity of respondentRevolution, civil war (0/1)

Ordinaldemocracy in 7 categorieswar

Intervalagree/disagree 5 point scale

Ratio (Continuous)income, length of war

Discrete “Variables”

WAR

PEACE

CONFLICT

Variables:Measurement techniques

Counting (words, elections, and so on)Sources: government statistics, newspapers, archivesIncludes: basic content analysisIncludes: Is there a senate? Are there two large parties?

Asking peopleSurvey questions

Fixed formatOpen-response

Interviews (elite or random sample)Implicit attitudes (psychology, often experimental)

Variables: Measurement Techniques

(cont.)Observing people

Discourse analysis

Content/semantic analysis

Participant observation

Coding of gestures/speech (psychology)

InterpretiveCompare measurement techniques/sources – why are they different?

Semiotic practices (observing people)

Step 3) Assess Measurement

Reliability & Validity

Reliable measures are…

consistent (repeated measurement gives similar results for the same case)

inter-coder reliability (qualitative data)

unbiased (right answer, on average)

precise/low error (close to the right answer)

Reliable Measures

Consistent

Step 3) Assess Measurement

Reliability & Validity

Reliable measures are…

consistent (repeated measurement gives similar results for the same case)

inter-coder reliability (qualitative data)

unbiased (right answer, on average)

precise/low error (close to the right answer)

Reliable Measures

ConsistentConsistentUnbiased

Step 3) Assess Measurement

Reliability & Validity

Reliable measures are…

consistent (repeated measurement gives similar results for the same case)

inter-coder reliability (qualitative data)

unbiased (right answer, on average)

precise/low error (close to the right answer)

Reliable Measures

ConsistentConsistentUnbiased

ConsistentUnbiased

Low Variance

Valid measures are…

Content or substance of measure captures construct

face validity – ask an expert

“what are you thinking about” probes

manipulation checks (experiments)

Criterion-related validityPredictive: Correct relationship with some predicted outcome variable

Decision validity: Useful or pragmatic applications

Valid Measures are… (cont.)

Construct validityConvergent validity: Related to other theoretically related variables/indicators (correlations, alpha scores, factor analysis)• Example: GNP/capita, tax revenue/capita

– Discriminate validity: empirically discriminates between related constructs (correlations, factor analysis)• Example: poverty, education

Though he may not always recognise his bondage, modern man lives under a tyranny of numbers.

Nicholas EberstadtThe Tyranny of Numbers; Mismeasurement and Misrule (1995)

Debates

• Can we measure everything?

• Should we measure observable behaviors or social meaning?

• Do concepts need to be better defined or better measured?

Debates: Can you measure

everything?

• Measurement people – yes, in theory (Kerlinger – but it is just a game, play it well)

• Sartori – no. Example: party system.

Debates: Observation versus

meaning• Example: voting in authoritarian regimes

versus democratic

• Example: survey responses in Africa versus United States

• Example: organization membership across social class

• Solution: contingent/contextual measures, more complete specification 

Debates: concept definition versus

concept measurement

Defining concepts

versus

Measuring concepts

Go, wondrous creature, mount where

science guides.

Go, measure earth, weigh air, and state

the tides;

Instruct the planets in what orbs to run,

Correct old Time, and regulate the sun;

Go, teach Eternal Wisdom how to rule,

Then drop into thyself and be a fool.

Alexander PopeQuoted in James Wood Dictionary of Quotations from Ancient and Modern, English and Foreign Sources (1893), 125.

Graph based measures:Social Networks and DistancesMeasurement goes beyond one dimension!

Who buys political books?

Source: Krebs, orgnet.com

Sou

rce:

Ron

ald

Ingl

ehar

t an

d C

hris

tian

Wel

zel,

Mod

erni

zatio

n, C

ultu

ral C

hang

e an

d D

emoc

racy

New

Yor

k: C

ambr

idg

Uni

vers

ity P

ress

, 20

05:

page

63.