View
1
Download
0
Category
Preview:
Citation preview
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
1
Multi-criteria Analysis for Impact Assessment
The Maximum Likelihood Approach
Michaela Saisana
michaela.saisana@jrc.ec.europa.eu
European Commission
Joint Research Centre
Econometrics and Applied Statistics Unit
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
2
Outline
• Applications of MCA and CBA
• Cost Benefit Analysis (+ limitations)
• Roots of MCA in Social Choice Theory
• 5 methods (Relative majority, Condorcet, Borda, Successive eliminations, Median ranking)
• Limitations of the Weighted Sum (most common approach)
• Weights as importance coefficients (BA and AHP)
• MCA: Maximum likelihood approach (steps, suitability)
• Sensitivity Analysis of MCA result
• Conclusion
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
3
Some Decision or Evaluation Problems • Locating a new plant
• Human resources management
• Evaluating projects
• Selecting an investment strategy
• Electricity production planning
• Regional planning
• Evaluation of urban waste management systems
• Environmental applications
• Health Risk Prediction
• Systemic Risk Assessment ( JRC collaboration with the European Systemic Risk
Board – European Central Bank)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
4
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
5
Basic steps of cost-benefit analysis (CBA)
1. Determine if CBA is worth doing
2. Identify objectives and policy alternatives
3. Determine stakeholders
4. Identify costs and benefits of each alternative
5. Sort into measurable and non-measurable costs and benefits
6. Estimate costs and benefits that can be measured in monetary terms
7. Conduct sensitivity analysis
8. Compare costs-benefits across alternatives
9. Adjust for non-measurable costs and benefits(?)
10. Make a decision
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
6
Cost-benefit guidelines
• UK Department of the Treasury, Appraisal and Evaluation in Central Government (The Green
Book), London:2002, http://www.hmtreasury.gov.uk/data_greenbook_index.htm
• NZ Treasury guidelines
www.treasury.govt.nz/publications/guidance/planning/costbenefitanalysis>
• Australian Government, Office of Best Practice Regulation,
http://www.finance.gov.au/obpr/cost-benefit-analysis.html (see especially Handbook of Cost-
Benefit Analysis, and Best Practice Regulation Handbook)
• Queensland Government, Department of Infrastructure and Planning, Cost Benefit Analysis,
www.dip.qld.gov.au/resources/guideline/project-assurance-framework/pafcost-benefit-
analysis.pdf
• Government of Western Australia, Department of Treasury and Finance, 2005, Project
Evaluation Guidelines,
www.dtf.wa.gov.au/cms/uploadedFiles/project_evaluation_guidelines_2002.pdf
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
7
Limitations of CBA
• Results often highly sensitive to specific assumptions, such as discount rate
• Difficult to balance non-quantifiable costs/benefits against quantifiable ones
• Anthropocentric in its underlying social vision
How much is life, education (literacy), welfare, health, ecological
sustainability, employment (business confidence) worthy?
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
8
Multi Criteria Analysis (MCA) - Definition
“Multi Criteria Analysis is a decision-making tool, developed for
complex multi-criteria problems that include quantitative and/or
qualitative aspects of the problem in the decision making process.”
(Center for International Forestry Research, CIFOR, 1999)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
9
MCA - Steps
1. Establish the decision context
2. Identify the performance criteria and the options
3. Describe/rate the performance of each option against the criteria
4. Assign weights across criteria
5. Combine the information to obtain a ranking of the options
6. Examine the results and review
7. Conduct sensitivity analysis
8. Final decisions
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
10
MCA - Performance matrix
Criterion 1
(/20)
Criterion 2
(rating)
Criterion 3
(qual.)
Criterion 4
(Y/N) …
Action 1 20 135 G Yes …
Action 2 9 156 B Yes …
Action 3 15 129 VG No …
Action 4 9 146 VB No …
Action 5 7 121 G Yes …
… … … … … …
Criteria should not be dependant on each other and not redundant (to avoid
double counting)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
11
MCA - Performance matrix
• Who decides the ratings?
MCA very flexible wrt who gets a say in either the criteria or rating the options:
Democratic decision-making - all members of the decision-making body, or each
organizational branch/unit, independently allowed to rate options
Panel of experts asked to make judgments; can use different panel to judge different
criteria
Consensus model - decision-making body ‘thrash it out’
Stakeholder inclusion
Different groups can rate options on different criteria
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
12
MCA - Result
The outcome of MCA can be used to:
• Identify a single, most-preferred option
• Rank options
• Short-list a limited number of options for subsequent detailed appraisal
through other methods such as CBA
• Distinguish acceptable from unacceptable options
• Combine different options based on relative strengths
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
13
Social Choice Theory
Problem:
• A group of voters have to select a candidate among a group of candidates (election)
• Each voter has a personal ranking of the candidates according to his/her
preferences
• Which candidate must be elected?
What is the «best» voting procedure?
Analogy with multi-criteria analysis:
• Candidates actions
• Voters criteria
Best interest of society
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
14
Social Choice Theory
Social choice theory methods would be ideally suited for assessing
multiple options through multiple criteria … and were already available
between the end of the XIII and the XV century, …
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
15
1. Ramon Llull (ca. 1232 – ca. 1315) proposed first what would then become known as the
method of Condorcet.
2. Nicolas de Condorcet, (1743 –1794) His „Sketch for a Historical Picture of the Progress of the
Human Spirit (1795)‟ can be considered as an ideological foundation for evidence based policy
(modernity at its best!).
3. Nicholas of Kues (1401 – 1464), also referred to as Nicolaus Cusanus and Nicholas of Cusa
developed what would later be known as the method of Borda.
4. Jean-Charles, chevalier de Borda (1733 – 1799) developed the Borda count.
1 2 3 4
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
16
Five methods (among many others)
1. Relative majority
2. Condorcet
3. Borda
4. Successive eliminations
5. Median ranking
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
17
3 candidates: Adam, Brian, Carlos
11 voters
10 voters
9 voters
A B C
B C B
C A A
A 11
B 10
C 9
Adam is elected
30 voters:
Method 1 : Relative majority
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
18
3 candidates: Adam, Brian, Carlos
11 voters
10 voters
9 voters
A B C
B C B
C A A
A 11
B 10
C 9
Adam is elected
30 voters:
Method 1 : Relative majority
Problem: B and C preferred to
A by a majority of voters!
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
19
11 voters
10 voters
9 voters
A B C
B C B
C A A Brian is elected
B preferred to A 19
votes
B preferred to C 21
votes
C preferred to A 19
votes
3 candidates: Adam, Brian, Carlos
30 voters:
Method 2 : Condorcet
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
20
4 voters
3 voters
2 voters
A B C
B C A
C A B
A preferred to B 6
votes
B preferred to C 7
votes
C preferred to A 5
votes
Method 2 : Condorcet
3 candidates: Adam, Brian, Carlos
9 voters: Problem: Nobody is elected!
(cycle)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
21 39 x 2 + 31 x 1
Points
2
1
0
30 voters
29 voters
10 voters
10 voters
1 voter
1 voter
A C C B A B
C A B A B C
B B A C C A
Scores
A 101
B 33
C 109
31 x 2 + 39 x 1
11 x 2 + 11 x 1
Carlos is elected!
Method 3 : Borda
3 candidates: Adam, Brian, Carlos
81 voters:
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
22
3 voters
2 voters
2 voters
C B A
B A D
A D C
D C B
Points
3
2
1
0
Scores
A 13
B 12
C 11
D 6
Ranking
A
B
C
D
Adam is elected
Method 3 : Borda
4 candidates: Adam, Brian, Carlos, David
7 voters:
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
23
3 voters
2 voters
2 voters
C B A
B A C
A C B
Points
2
1
0
Scores
A 6
B 7
C 8
Ranking
C
B
A
Carlos is elected
Method 3 : Borda
4 candidates: Adam, Brian, Carlos, David
7 voters:
Problem: Fully Dependant on
irrelevant alternatives (easy to
manipulate)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
24
Method 4 : Successive eliminations
6 voters
4 voters
1 voters
A C C
C A B
B B A
3 candidates: Adam, Brian, Carlos
11 voters:
Ranking
A
C
B
A tour-wise procedure, whereby the
worst candidate (most voted in the
last position) is eliminated
progressively until one is left.
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
25
Method 5 : Median ranking
6 voters
4 voters
1 voters
A C C
C A B
B B A
3 candidates: Adam, Brian, Carlos
11 voters:
A: 11111122223
B:23333333333
C:11111222222
•Ranking of candidates for each voter
•Median rank for each candidate across
voters
Ranking
A
C
B
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
26
5 candidates: Adam, Brian, Carlos, David, Edison
8 voters
7 voters
4 voters
4 voters
2 voters
A B E D C
C D C E E
D C D B D
B E B C B
E A A A A
25 voters: Relative majority Adam elected
Condorcet: Carlos elected
Borda:
David elected
Successive eliminations:
Edison elected
Median ranking:
Carlos elected
?
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
27
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
28
Kenneth Arrow
(Nobel prize in economy, 1972)
Impossibility theorem (1952):
With at least 2 voters and 3 candidates, it is impossible to build a voting procedure that
simultaneously satisfies the 5 following properties:
• Non-dictatorship
• Universality
• Independence with respect to third parties
• Monotonicity
• Non-imposition
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
29
Most common approach: Weighted sum
V(a) = V(b) = V(c) = 50
Problems: 1) Fully compensatory (elimination of conflicts)
weights
I1
(50%)
I2
(50%)
a 90 10
b 10 90
c 50 50
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
30
Most common approach: Weighted sum
V(a) = V(b) = 55, V(c) = 50
Problems: 2) Does not encourage improvement in the weak dimensions
weights
I1
(50%)
I2
(50%)
a 100 10
b 20 90
c 50 50
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
31
Most common approach: Weighted sum
Problems: 3) Weights are used as if they were importance coefficients while they are
trade off coefficients
Y = 0.5 ×X1+ 0.5 ×X2
R12 = 0.08, R2
2 = 0.83, corr(X1, X2) =−0.151, V(x1) = 116, V(x2) = 614, V(y) = 162
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
32
Most common approach: Weighted sum
• Weighted sum approach only possible under special circumstances (eg standardized
variables, uniform covariance matrix…)
• Hence we need to move away from weighted sums …
Effective weights are compared with nominal weights to ensure coherence
between the two.
[Paolo Paruolo, Michaela Saisana, Andrea Saltelli, 2013, Ratings and
rankings: Voodoo or Science?, J. R. Statist. Soc. A, 176 (3), 609-634]
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
33
MCA: Maximum likelihood Approach Features:
• no impact of outliers;
• no need for data normalisation;
• no need for uniform covariance matrix;
• no need to attach monetary value and use of both continuous and categorical variables;
• no use of any linear or multiplicative formula;
• use of the weights attached to the indicators as importance coefficients;
• a compromise between conflicting opinions;
• reasonably resistant to manipulation;
• produces a ranking that is statistically optimal (anonymous, neutral, Pareto optimal, satisfies reinforcement
and local independence of irrelevant alternatives)
[Kemeny (1959), Young and Levenglick (1978)] Led to: Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
34
MCA - Performance matrix
Criterion 1
(20%)
Criterion 2
(30%)
Criterion 3
(20%)
Criterion 4
(30%)
Action 1 20 135 G Yes
Action 2 9 156 B Yes
Action 3 15 129 VG No
Action 4 9 146 VB No
Action 5 7 121 G Yes
• Criteria should not be dependant on each other and not redundant (to
avoid double counting)
Where do
weights come
from?
(…next couple
of slides)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
35
43
32
25
29
38
33
38
34
28
39
26
36
39
32
2931
33
36 35
30
34
38
33
29
33 33 33 3335
32
0
10
20
30
40
50
60
70
Co
nsu
mp
tion
Acc
ess
Sta
bili
ty
Nu
triti
on
Qu
alit
y
Qu
alit
y
Ava
ilab
ility
Acc
ess
Sta
tus
Acc
ess
Qu
alit
y
To
ilet
Fa
cilit
ies
Wa
ste
Ma
na
ge
me
nt
Pra
ctic
es
Qu
alit
y
Fa
cilit
ies
En
erg
y
Qu
alit
y
Ava
ilab
ility
Acc
ess
Te
nu
re
Qu
alit
y
Inp
uts
Ski
lls
Se
rvic
es
Ass
ets
Exp
osu
re
Co
pin
g a
bili
ty
Re
cove
ry a
bili
ty
Fo
od
Ed
uca
tion
He
alth
care
Education Farm
Assets
Exposure &
Resilience to
Shocks
Gender
Equality
In 4 dimensions of poverty, the average expert
weight is similar to equal weighting Tiredness
in filling in the questionnaire on weights??
Weights based on Budget
Allocation (42 experts)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
36
USING PAIRWISE COMPARISONS, THE
RELATIVE IMPORTANCE
OF ONE CRITERION OVER ANOTHER CAN BE
EXPRESSED
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME
1 2 3 4 5 6 7 8 9
Patents vs. x Royalties x
x Patents vs. Internet x
x Patents vs. Technology exports x
x Patents vs. Telephones x
x Patents vs. Electricity x
Patents vs. x Schooling years x
Patents vs. x University Students x
x Royalties vs. Internet x
Royalties vs. x Technology exports x
x Royalties vs. Telephones x
x Royalties vs. Electricity x
Royalties vs. x Schooling years x
Royalties vs. x University Students x
Internet vs. x Technology exports x
x Internet vs. Telephones x
x Internet vs. Electricity x
Internet vs. x Schooling years x
Internet vs. x University Students x
x Technology exports vs. Telephones x
x Technology exports vs. Electricity x
Technology exports vs. x Schooling years x
Technology exports vs. x University Students x
x Telephones vs. Electricity x
Telephones vs. x Schooling years x
Telephones vs. x University Students x
Electricity vs. x Schooling years x
Electricity vs. x University Students x
x Schooling years vs. University Students x
Which Indicator Do You Feel Is More Important? To What Degree?
Questionnaire
Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.
Patents 1 1/3 5 4 3 9 1/6 1/8
Royalties 3 1 3 1/4 5 9 1/3 1/4
Internet 1/5 1/3 1 1/6 2 2 1/7 1/6
Tech.Exports 1/4 4 6 1 5 9 1/4 1/5
Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9
Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9
Schooling 6 3 7 4 9 9 1 2
University St. 8 4 6 5 9 9 1/2 1
solve for the
Eigenvector
Patents 0.109
Royalties 0.103
Internet hosts 0.029
Tech exports 0.117
Telephones 0.030
Electricity 0.014
Schooling 0.301
University st. 0.297
Weights
Inconsistency
17.4 %
Weights based on Analytic Hierarchy Process
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
37
USING PAIRWISE COMPARISONS, THE
RELATIVE IMPORTANCE
OF ONE CRITERION OVER ANOTHER CAN BE
EXPRESSED
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME
Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.
Patents 1 1/3 5 4 3 9 1/6 1/8
Royalties 3 1 3 1/4 5 9 1/3 1/4
Internet 1/5 1/3 1 1/6 2 2 1/7 1/6
Tech.Exports 1/4 4 6 1 5 9 1/4 1/5
Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9
Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9
Schooling 6 3 7 4 9 9 1 2
University St. 8 4 6 5 9 9 1/2 1
Weights based on Analytic Hierarchy Process
P=5I
R=3I
We expect:
P > R
Expert said:
R > P (R=3P)
Inconsistency
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
38
Performance
matrix
Criterion
1
Criterion
2
Criterion
3
Criterion
4
Criterion
5
Weights 10% 20% 10% 30% 30%
Option A 50 0.6 400 0.6 4000
Option B 70 0.3 500 0.7 5000
Option C 90 0.4 600 0.4 3000
Step 1 - Input matrix to the multicriteria analysis
Example: Three options need to be ranked according to five criteria. The importance
of the criteria is reflected in the respective weights.
MCA: Maximum likelihood Approach
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
39
Step 2 – Options are compared pairwise
For each comparison, e.g. option A versus option B, all the weights corresponding to the criteria
that favour A versus B are added up (abbreviated as AB). In this case AB gets the weight of
Criterion 2 only (=0.2). The comparison BA gets the sum of the weights of the remaining
criteria: 1, 3, 4, 5 (=0.8). For n options, there are n (n-1) comparisons to be made. All the values
from the pairwise comparisons are entered in a so called outranking matrix.
Outranking
matrix Option A
Option B
Option C
Option A 0 0.2 0.8
Option B 0.8 0 0.6
Option C 0.2 0.4 0
MCA: Maximum likelihood Approach
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
40
Step 3 – Calculate support for all permutations and select the maximum
• All 3! (=6) permutations of the options are considered and the support score for each ranking is
calculated.
• ABC has a support of 1.6 (=0.2+0.8+0.6), which is the sum of elements above the diagonal in the
outranking matrix.
• Support scores for all six rankings:
ABC= 1.6 |ACB=1.4 | BAC=2.2 | BCA=1.6 | CAB=0.8 | CBA=1.4
• The ranking selected is the one with the maximum likelihood score: BAC
Outranking
matrix Option A
Option B
Option C
Option A 0 0.2 0.8
Option B 0.8 0 0.6
Option C 0.2 0.4 0
MCA: Maximum likelihood Approach
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
41
How to shake coupled stairs How coupled stairs are shaken in most of
available literature
MCA: Maximum likelihood Approach
Important to assess sensitivity of results to the weights
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
42
Frequency matrix – Sensitivity of the final ranking to the assumptions (e.g.
weights)
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
43
• The main limitation of this method is the difficulty in computing the ranking when
the number of options grows (e.g. 100).
• For 10 options 10 = 3,628,800 permutations …still trivial for today’s PCs
• To solve this NP-hard problem when the number of options is very large there are
plenty of numerical algorithms (JRC works on them!)
MCA: Maximum Likelihood Approach
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
44
Concluding: How to use MCA in your work
1. Decide on the criteria that you want to use in your evaluation;
2. Identify appropriate indicators for each of the criteria (more than one indicator for each
criteria is OK);
3. Score the alternatives on each criterion based on their performance on that criterion;
4. Determine the weights of all the criteria (use for instance AHP);
5. Calculate the overall ranking of the alternatives using Maximum Likelihood;
6. Examine the results: try to explain why some options turn out to be the better than
others;
7. Do a Sensitivity Analysis: what happens if you assign other weights to the criteria? Does it
affect the overall results?
8. Make a final decision
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
45
“
[Peyton Young, 2002, Optimal Voting Rules,
Journal of Economic Perspectives 9:51-64]
Peyton Young Professor Emeritus, Research
Professor in Economics,
Johns Hopkins University
The more important issue is whether the (maximum
likelihood) method is intuitively easy to grasp, and
whether it improves on methods currently in use. On both
these counts I think that the answer is affirmative, and
I predict that the time will come when it is considered
a standard tool for political and group decision
making.
Michaela Saisana
4th Impact Assessment Course
JRC, Ispra, 9-10 December 2013
46
More reading at
http://composite-indicators.jrc.ec.europa.eu
Recommended