Upload
veronica-waters
View
229
Download
8
Tags:
Embed Size (px)
Citation preview
S ystemsAnalysis LaboratoryHelsinki University of Technology
Can We Avoid Biases in Environmental
Decision Analysis ?
Raimo P. Hämäläinen
Helsinki University of Technology
Systems Analysis Laboratory
www.paijanne.hut.fi
S ystemsAnalysis LaboratoryHelsinki University of Technology
Structure of the presentation
• Background & decision analysis interviews
• Goals of the study
• Case: Regulation of Lake Päijänne
• Splitting bias & swapping of levels
• Description of the experiment
• Results of the experiment
• Conclusions ?
S ystemsAnalysis LaboratoryHelsinki University of Technology
Environmental decision analysis
• Parliamentary nuclear power decision(Hämäläinen et. al)
• Decision analysis interviews(Marttunen & Hämäläinen)
• Spontaneous decision conferencing in nuclear emergency management(Hämäläinen & Sinkko)
S ystemsAnalysis LaboratoryHelsinki University of Technology
Cognitive biases
• Splitting bias– attribute receives more weight if it is split
– origins: subjects give rank information only (Pöyhönen & Hämäläinen)
– Not observable in hierarchical weighting
S ystemsAnalysis LaboratoryHelsinki University of Technology
Decision analysis interviews
• Opinions of large groups of people traditionally collected through questionnaires
• Decision analysis interviews may provide a more reliable way to collect these opinions
• Idea:– one value tree for all = common terminology
– emphasis on finding the viewpoints of different stakeholder groups
– interactive, computer supported
S ystemsAnalysis LaboratoryHelsinki University of Technology
Research interest
• Existence of biases in a real case
• Can biases can be avoided through training and proper instructing ?
• Identify what can go wrong in the Lake Päijänne case
• Compare the well trained university students’ and spontaneous stakeholders’ responses
S ystemsAnalysis LaboratoryHelsinki University of Technology
The Lake Päijänne case
• Regulation started 1964
• Main aims were to improve hydroelectricity production and to reduce damages caused by flooding
• Environmental values & increase in free time
• need for an improved regulation policy
S ystemsAnalysis LaboratoryHelsinki University of Technology
Splitting bias
• When an attribute is split, the weight it receives increases
Attribute 1
Attribute 2
Attribute 3
Attribute 1
Attribute 2
Attribute 3
subattribute 1b
Subattribute 1a
0.4
0.3
0.3
0.3
0.3
0.1
0.3
0.4
S ystemsAnalysis LaboratoryHelsinki University of Technology
Swapping of levels
• Does the order of the levels affect the resulting weights?
• Important question in environmental decision analysis:– stakeholder groups may vary regionally
• Not studied before
S ystemsAnalysis LaboratoryHelsinki University of Technology
Example of swapping of levels
Lake Päijänne
River Kymijoki
Attribute 3
Attribute 3
Attribute 2
Attribute 1
Attribute 2
Attribute 1
Attribute 3
Attribute 2
Attribute 1
River Kymijoki
Lake Päijänne
River Kymijoki
Lake Päijänne
River Kymijoki
Lake Päijänne
S ystemsAnalysis LaboratoryHelsinki University of Technology
Earlier experiments on biases
• Structure of the decision model affects the results
• Previous experiments typically:– subjects: university students
– problems: artificial
– results: taken from group averages
• Lake Päijänne-case: a real problem with real stakeholders
S ystemsAnalysis LaboratoryHelsinki University of Technology
Important new features
• Realistic case
• Decision analysis interviews instead of passive decision support or survey
• Interactive computer support (resulting weights shown immediately)
• Instructions and training before the weighting
S ystemsAnalysis LaboratoryHelsinki University of Technology
Subjects:
• University students attending a course on decision analysis (N = 30)– held during a tutorial session, not mandatory
• Habitants of Asikkala (N = 40)– 3 groups of students
– 1 group of adults (volunteers)
• 3 experts from the Finnish Environment Institute & 2 summer residence owners
S ystemsAnalysis LaboratoryHelsinki University of Technology
Experimental setting
• Weighting done with the SWING method using a tailored Excel interface
• Subjects entered the numbers themselves, two assistants were present to help
• Resulting weights shown as bars
• Order of value trees partly randomized
S ystemsAnalysis LaboratoryHelsinki University of Technology
Sessions
• A short introduction to: – Lake Päijänne case
– value trees & weighting
– different structures of the value tree
• In HUT the avoidance of biases was emphasized more
• Duration: 60 - 90 minutes
S ystemsAnalysis LaboratoryHelsinki University of Technology
SWING method
• Easy to use
• Attribute ranges clearly presented
• Idea:– choose the attribute you would first like to
move to its best level
– assign it 100 points
– assign other attributes points less than 100 in respect to the first attribute
S ystemsAnalysis LaboratoryHelsinki University of Technology
Flat-weighting
Muu talous ???
VesivoimaVesivoima
Muu talous
Ympäristö
Talous
Rantojen käytettävyys
Virkistyskalastus
Kalojen lisääntyminen
Rantakasvillisuus
Lahtienumpeenkasvu
Virkistys
Luonto
Tulvat, maatalous jateollisuus
Tulvat, loma-asutus
Vesiliikenne
Ammattikalastus
S ystemsAnalysis LaboratoryHelsinki University of Technology
Upper level weights:
Muu talous ???
VesivoimaVesivoima
Muu talous
Ympäristö
Talous
Rantojen käytettävyys
Virkistyskalastus
Kalojen lisääntyminen
Rantakasvillisuus
Lahtienumpeenkasvu
Virkistys
Luonto
Tulvat, maatalous jateollisuus
Tulvat, loma-asutus
Vesiliikenne
Ammattikalastus
S ystemsAnalysis LaboratoryHelsinki University of Technology
ENV5-tree:
Luonto
Virkistys
Ympäristö
Talous
Rantojen käytettävyys
Virkistyskalastus
Kalojen lisääntyminen
Rantakasvillisuus
Lahtienumpeenkasvu
S ystemsAnalysis LaboratoryHelsinki University of Technology
ENV2-tree:
Luonto
Virkistys
Ympäristö
Talous
S ystemsAnalysis LaboratoryHelsinki University of Technology
EC5-tree:
Muu talous ???
VesivoimaVesivoima
Muu talous
Ympäristö
TalousTulvat, maatalous ja
teollisuus
Tulvat, loma-asutus
Vesiliikenne
Ammattikalastus
S ystemsAnalysis LaboratoryHelsinki University of Technology
EC2-tree:
Muu talous ???
Vesivoima
Muu talous
TalousMuu talous ???
Vesivoima
Muu talous
Ympäristö
Talous
S ystemsAnalysis LaboratoryHelsinki University of Technology
Swapping of levels:
Muu talous ???
Kymijoki ja muut
Päijänne
Rantakasvillisuus
Tulvavahingot
Kymijoki ja muut
Päijänne
Muu talous ???
Rantakasvillisuus
Tulvavahingot
Kymijoki ja muut
Päijänne
Rantakasvillisuus
Tulvavahingot
S ystemsAnalysis LaboratoryHelsinki University of Technology
Flat weights vs. upper level weights
• Both in group averages and in results of individuals the total weights for the environment and economy were similar with both methods
• One explanation: symmetric value tree
S ystemsAnalysis LaboratoryHelsinki University of Technology
Splitting bias0
.00
.30
.50
.81
.0
ASIKKALA HUT
Means of Wenv
GROUP
Wenv
TREE
EC2EC5ENV2ENV5
S ystemsAnalysis LaboratoryHelsinki University of Technology
A typical resident in Asikkala
0
0.2
0.4
0.6
0.8
1ENVIRONMENT ECONOMY
5 1 5 2 1 1 5 1 1 1 5 2
S ystemsAnalysis LaboratoryHelsinki University of Technology
Example from HUT(one of the best ones)
0
0.2
0.4
0.6
0.8
1ENVIRONMENT ECONOMY
5 1 5 2 1 1 5 1 1 1 5 2
S ystemsAnalysis LaboratoryHelsinki University of Technology
Why even weights ?
• Some students: none of the attributes seemed to be important
• Asikkala: all of the attributes were important
even weights for all attributes
S ystemsAnalysis LaboratoryHelsinki University of Technology
What caused the bias ?
• Similar points for
all attributes in one branch
regardless of the structure
of the value tree
100 80
80100 7070 70 50
100 80 70
S ystemsAnalysis LaboratoryHelsinki University of Technology
Effect of instructions
• Students had good instructions– only some had bias in their results
• In the spontaneous stakeholders’ sessions the information load was too high and thus the instructions were not adopted as well– nearly all had systematically consistent bias
S ystemsAnalysis LaboratoryHelsinki University of Technology
n
nenv
econ
n
nenv
econ
w
wenv
econ
w
wenv
econ
STUDENTS STAKEHOLDERS
Adjusted / not adjusted weights
S ystemsAnalysis LaboratoryHelsinki University of Technology
Examples
01234567
0 1 2 3 4 5
01234567
0 1 2 3 4 5
STUDENTS STAKEHOLDERS
w
w
env
econ
w
w
env
econ
n
nenv
econ
n
nenv
econ
S ystemsAnalysis LaboratoryHelsinki University of Technology
Observation
• The students and the experts from FEI could nearly avoid the splitting bias– good background education + instructions did
reduce the bias
• What did the students think? - Arithmetics or real avoidance of biases
S ystemsAnalysis LaboratoryHelsinki University of Technology
Avoiding the splitting bias ?
• Good instruction can eliminate it
• When the economical attributes were split, the magnitude of the bias was slightly larger
• Graphical feedback did not eliminate
• Hierarchical weighting
S ystemsAnalysis LaboratoryHelsinki University of Technology
Swapping of attribute levels
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
If the order of the levels would not affect the weigts, the pairs of weights should be equal (as in the first picture)
S ystemsAnalysis LaboratoryHelsinki University of Technology
Conclusions about swapping of levels ?
• Only a few had clearly differing weights with the two trees
• No systematic pattern was found
• Less differences residents of Asikkala and students than with the splitting bias
• A simple scale lead to similar weights with both trees (100, 70 for example)
• Neither tree gained clear support
S ystemsAnalysis LaboratoryHelsinki University of Technology
Solutions to reduce biases ?
• Hierarchical weighting
• Models should be tested on real decision makers
• Interactiveness of weighting (= possibility to return to change the points given earlier )
• Well balanced trees
S ystemsAnalysis LaboratoryHelsinki University of Technology
Other observations in Asikkala
• Concept of weight seemed to be difficult for most subjects in Asikkala
• Information load was high
• Facilitators role becomes important when the DM’s are uncertain
S ystemsAnalysis LaboratoryHelsinki University of Technology
Problems related to the Lake Päijänne case
• Current regulation policy cannot be improved very significantly – no big differences between the alternatives
– unrealistic hopes and false information are probably larger problems than the regulation itself
• ‘money is not money’– strong feelings against the power companies
and regulation (shape of value function ?)
S ystemsAnalysis LaboratoryHelsinki University of Technology
Suggestions for future research
• Hierarchical weighting
• Encouragement to reconsider and readjust the statements iterate
• Decision Analyst must supervise!
S ystemsAnalysis LaboratoryHelsinki University of Technology
R.P. Hämäläinen, E. Kettunen, M. Marttunen and H. Ehtamo: Evaluating a framework for multi-stakeholder decision support in water resources management, Group Decision and Negotiation, 2001. (to appear)
M. Pöyhönen, Hans C.J. Vrolijk and R.P. Hämäläinen: Behavioral and procedural consequences of structural variation in value trees. European Journal of Operational Research, 2001. (to appear)
M. Pöyhönen and R.P. Hämäläinen: There is hope in attribute weighting, Journal of Information Systems and Operational Research (INFOR), vol. 38, no. 3, Aug. 2000, pp. 272-282. Abstract
R.P. Hämäläinen, M. Lindstedt and K. Sinkko: Multi-attribute risk analysis in nuclear emergency management, Risk Analysis, Vol. 20, No 4, 2000, pp. 455-467.
M. Pöyhönen and R.P. Hämäläinen: Notes on the weighting biases in value trees, Journal of Behavioral Decision Making, Vol. 11, 1998, pp. 139-150.
Susanna Alaja: Structuring effects in environmental decision models, Helsinki University of Technology, Systems Analysis Laboratory, Theses, 1998.
References
S ystemsAnalysis LaboratoryHelsinki University of Technology
M. Pöyhönen, R.P. Hämäläinen and A. A. Salo: An experiment on the numerical modeling of verbal ratio statements, Journal of Multi-Criteria Decision Analysis, Vol. 6, 1997, pp. 1-10.
R.P. Hämäläinen and M. Pöyhönen: On-line group decision support by preference programming intraffic planning, Group Decision and Negotiation, Vol. 5, 1996, pp.485-50.
M. Marttunen and R.P. Hämäläinen: Decision analysis interviews in environmental impact assessment, European Journal of Operational Research, Vol. 87, No. 3, 1995, pp. 551-563.
R.P. Hämäläinen, A.A. Salo and K. Pöysti: Observations about consensus seeking in a multiple criteria environment, in: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences, Vol. IV, 1991, IEEE Computer Society Press, Hawaii, pp. 190-198.
R.P. Hämäläinen: Computer assisted energy policy analysis in the parliament of Finland, Interfaces, Vol. 18, No. 4, 1988, pp. 12-23. Also in: Case and Readings in Management Science, 2nd edition, M. Render, R.M. Stair Jr. and I. Greenberg (eds.), Allyn & Bacon, Massachusetts 1990 pp. 278-288.