PAIRED COMPARISONS - WU (Wirtschaftsuniversität Wien)statmath.wu.ac.at/~hatz/PcP_Web/LinkedDocuments/ResEval... · 2003-11-05 · PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser)

PAIRED COMPARISONS (Dittrich, Hatzinger, Katzenbeisser) – WU Wien – 6.11.2003 1

PAIRED COMPARISONS

MODELS AND APPLICATIONS Regina Dittrich Reinhold Hatzinger Walter Katzenbeisser


PAIRED COMPARISONS (PC) a method of data collection where individuals are asked to judge a number of different pairs of objects, taken from a larger set of J objects.

for each comparison between two objects j and k , the individual can respond j preferred to k

k preferred to j aim is to rank objects into a preference order – obtain an overall ranking of the objects common in marketing (food tasting,..), social and management sciences, …


THE BASIC BRADLEY TERRY MODEL (BT MODEL) for the comparison ( )jk of object j to object k : we observe jkN , the number of times where j is preferred to k we observe kjN , the number of times where k is preferred to j ( )jkn is the number of times this comparison was performed, ie, jk kjN N+

let jkp be the probability that j is preferred to k in comparison ( )jk the BT-model is

jπ are a set of so-called worth parameters

(positive, constrained to sum to one)

jjk

j kp

ππ π=+


THE LOG-LINEAR FORM OF THE BRADLEY TERRY MODEL the BT-Model can be formulated as a log-linear model following the usual Multinomial / Poisson - equivalence. the expected value jkm of jkN is given as ( )jk jk jkm n p=

j kjjk

j k k j j kp

π πππ π π π π π= =+ +

then our basic paired comparison model (PC Model) is

jλ are the object parameters

( )jkα are nuisance parameters

this model formulation is feasible for further extensions

( )ln jk jk j km α λ λ= + −


EXTENSIONS TO THE BASIC PC MODEL - RESPONSES

ALLOW FOR TIES (TRICHOTOMOUS RESPONSES, NOMINAL)

for each comparison between two objects j and k , the response can be

j preferred to k no preference k preferred to j

ORDINAL COMPARISONS (POLYTOMOUS RESPONSES)

for each comparison between two objects j and k , the response can be

strong for j moderate for j … … strong for k


EXTENSIONS TO THE BASIC PC MODEL – STRUCTURE

POSITION EFFECTS

Is there an effect due to the ordering of the presentation of objects? Does it make a difference ?

MANCHESTER UNITED – INTER MILAN (playing in Manchester)

INTER MILAN – MANCHESTER UNITED (playing in Milan) introduce additional parameter(s) for position effects


EXTENSIONS TO THE BASIC PC MODEL – COVARIATES

OBJECT COVARIATES ( Dittrich, Hatzinger, Katzenbeisser, J. Royal Statistical Society, C, 1998 )

to model the objects by a few characteristics

jqx covariate for characteristicq of object j qβ effect of characteristic q

c

CATEGORICAL SUBJECT COVARIATES ( Dittrich, Hatzinger, Katzenbeisser, J. Royal Statistical Society, C, 1998 )

CONTINOUS SUBJECT COVARIATES

one contingency table for each subject

SMOOTHED SUBJECT COVARIATES (GAMS) (Francis, Dittrich, Hatzinger, Penn, J. Royal Statistical Society, C, 2002)

1

QOj q jq

qxλ β

==∑


CATEGORICAL SUBJECT COVARIATES ( Dittrich, Hatzinger, Katzenbeisser, J. Royal Statistical Society, C, 1998 )

Are the preference orderings different for different groups of subjects? For one categorical subject covariate we now have

| ( )ln ( ) ( )j j k kO O S O O SSjk s jk s s j js k ksm α λ λ λ λ λ= + + + − +

where

jOjλ object parameter

jO Sjsλ interaction parameter between object j and subject category s Ssλ fixing the margin for category s of covariate S (nuisance) ( )jk sα nuisance parameters


EXAMPLE: UNIVERSITY PREFERENCES CEMS – exchange programme students of the WU can study abroad visiting one of currently 17 CEMS universities aim of the study:

• preference orderings of students for different locations • identify reasons for these preferences

data:

• PC-responses allowing for ties about their choices of 6 selected CEMS universities for the semester abroad (London, Paris, Milan, Barcelona, St.Gall, Stockholm)

• several covariates (e.g., gender, working status, language abilities, …)


Main results: • in general London by far the most popular place • depending on language abilities according places move up • full-time working students prefer Latin universities

0.0

0.1

0.2

0.3

0.4

0.5

ST

MI

SGBA

PA

LO

STOCKHOLM

ST.GALL

MILANBARCELONA

LONDON

PARIS

STOCKHOLM

ST.GALL

LONDON

PARIS

MILAN

BARCELONA

ALL FRENCH FULLTIME JOB


CONTINOUS SUBJECT COVARIATES ( Francis, Dittrich, Hatzinger, Penn, J. Royal Statistical Society, C, 2002 )

we need to extend the model on individual level

FIRST STEP: we model ,i jλ by 1

,

Q

j jq qii jq

xλ βλ=

= +∑

for each object j there is a separate set of parameters β describing the effect of the subject covariates on that object SECOND STEP: add smooth non-linear subject effects

11, ( )

M

jq qi j

Q

j jq qiq M

iqf x xλλ β

= = += + +∑ ∑

, ,,( ) ,ln i jk i j i jk i kp λα λ= + −


EXAMPLE: VALUE ORIENTATION IN EUROPE

Questions come from Inglehart(1990) ’Cultural shift in Advanced Industrial Society’ to assess postmaterialist and materialist values.

4 Inglehart Items: post-materialist items:

‘more say in government’ ‘protect freedom of speech’

materialist items ’maintain order in the nation’ ‘fight rising prices’ several sociological hypotheses: e.g., “Younger Europeans have been raised in relative prosperity, and are shifting towards postmaterial values” Data from ISSP (International Social Survey Program) 1993 • Social survey carried out each year since 1984 in 21 countries.

Same questions in each country • we analysed five European countries and look at the Inglehart items

important to sociologists – collected in every modern major social survey


MATERIALISM / POSTMATERIALISM: AGE EFFECT FOR SELECTED COUNTRIES

20 30 40 50 60 70 80 900.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9All countries

age

ORDER

SAYPRICES

SPEECH

20 30 40 50 60 70 80 90

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9Poland

age

ORDER

SAY

PRICES

SPEECH

20 30 40 50 60 70 80 900.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9West Germany

age

ORDER

SAY

PRICES

SPEECH

20 30 40 50 60 70 80 90

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9Great Britain

age

ORDER

SAYPRICES

SPEECH


VARIOUS DATA STRUCTURES CAN BE ANALYSED USING PC MODELS by transforming them into paired comparison data

RANKINGS

respondent is asked to rank a set of J objects - can be converted to a set of paired comparison data

consider the ranking (3, 1, 2, 4, 5) for objects (A,B,C,D,E), where a low rank means a higher preference:

example: (A,B) ⇔ (3,1) implies that B is preferred (A,C) ⇔ (3,2) implies that C is preferred, …

this correspondence was used to analyse preferences for Austrian newspapers ( Dittrich, Katzenbeisser, Reisinger, OR Spectrum, 2000 )


PARTIAL RANKINGS ( Francis, Dittrich, Hatzinger, Penn, J. Royal Statistical Society, C, 2002 )

investigation of value orientation in Europe in ISSP respondents are asked to state highest and next highest priority - rank only a subset of 4 items (objects) e.g. (ORDER, SAY, PRICES, SPEECH) with response (1, na , 2, na ) leads to the following paired comparisons (ORDER, SAY) ⇔ (1, na) implies that ORDER is preferred to SAY (ORDER, PRICE) ⇔ (1, 2) implies that ORDER is preferred to PRICE … (SAY, SPEECH ⇔ (na,na) response treated as missing

RESPONSES TO ITEMS ON A LIKERT SCALE ( Dittrich, Hatzinger, Katzenbeisser; forthcoming)

CONTINOUS RESPONSES

( Francis, Soothill, Dittrich, Brit.J.Criminol, 2001)


MODELLING DEPENDENCIES IN PAIRED COMPARISON DATA - A LOG-LINEAR APPROACH

( Dittrich, Hatzinger, Katzenbeisser, Comp.Stat.&Data Analysis, 2002 ) main assumptions so far was the independence between comparisons we assume that dependencies between responses may arise from repeated evaluation of the same objects in paired comparisons e.g., comparing (object j with object k ) and (object j with object l )

the assessment of object j might be similar in both comparisons in general the result of paired comparisons can be represented by a

response pattern vector: , , ........, ..........,(12) (13) (1 ), ( 1, )( )J J JY Y Y Y −=Y where for dichotomous responses

if object is preferred over

if object is preferred over

( )

1

1

jkjk

j k

O OY

O O

−=


from

( )( 1)j kj j

jkj k k j j k k

P Y cπ ππ π

π π π π π π π= = = =+ +

the model of independence for a certain response pattern is

where ∆ is a normalizing constant

to model dependencies we include terms of the form ( ),( ) ( ) ( )exp{ }jk jl jk jly yθ , where pairs of comparisons have one object in common.

the parameter ( ),( )jk jlθ can be interpreted as proportional to the log-odds ratio of

( )jkY and ( )jlY given the rest (conditional odds ratio) this is similar to the approach taken by Cox (1972) in modelling multivariate binary data

( )

( )jky

j

kj kP

π∆ π<

= = ∏Y y


RECENT WORK AND OUTLOOK

MODELLING REPEATED PAIRED COMPARISONS ( Dittrich, Francis, Katzenbeisser (2003) Proceedings of 18th IWSM )

comparisons of the same objects by the same judges are made on more than one occasion British Household Panel Study: follow-up of concerns about topical issues (unemployment, ozone layer, decline in moral standards)

A PAIRED COMPARISON APPROACH FOR THE ANALYSIS OF LIKERT SCALE DATA

( Dittrich, Hatzinger, Katzenbeisser; forthcoming )

Likert scale responses are transformed into ordinal paired comparison responses - log linear version of the adjacent categories models for ordinal paired comparisons Motives of students to obtain a doctoral degree at the WU: importance of motives for various groups of students


A LOG LINEAR REPRESENTATION FOR DEPENDENT MULTIVARIATE COMPARISONS

judges are asked to compare the objects on more than one attribute • dependencies between the decisions of the judges • association structures between the attributes

location of Austrian party leaders in twodimensional preference space (social and economic competence)

RANDOM EFFECTS – LATENT CLASS PC MODELS ( Francis, Dittrich (2000) Proceedings of 5th IC Logic and Methodology )

nonparametric ML estimation of subject effects aims of social work ranked by first year social work students from Australia, Canada, UK, and USA

dichotomous

INDEPENDENCE MODEL

(productmultinomial)

paired comparisons

OBJECT COVARIATES

categoricalmetric

SUBJECT COVARIATES

FIXED EFFECTScategorical

metricsmoothed

(GAMS)

SUBJECT COVARIATES

RANDOM EFFECTScategorical

metric

UNDECIDED/CATEGORY

EFFECT

POSITION EFFECT

LOCATION OF OBJECTS ON

PREFERENCE SCALE

STRUCTURE OBJECTS SUBJECTSRESPONSE STRUCTURE

OBSERVED DATA

OBSERVATION

DESIGN

CROSS-SECTIONAL

MULTI-DIMENSIONAL

REPEATED MEASURES

PERMUTATION OF PRESENTATION

OF OBJECTS

MODEL PARAMETERSRESPONSES

paired comparisons

with ties

ordinal paired comparisons

rankings

partial rankings

responses on Likert scale

trichotomous(nominal)

polytomous(ordinal)

DEPENDENCE MODEL

(loglinear, comparison patterns,transitive-

intransitive)

PAIRED COMPARISON MODELS OVERVIEW