Can We Avoid Biases in Environmental Decision Analysis ?

S ystemsAnalysis LaboratoryHelsinki University of Technology

Can We Avoid Biases in Environmental

Decision Analysis ?

Raimo P. Hämäläinen

Helsinki University of Technology

Systems Analysis Laboratory

[email protected]

www.paijanne.hut.fi


Structure of the presentation

• Background & decision analysis interviews

• Goals of the study

• Case: Regulation of Lake Päijänne

• Splitting bias & swapping of levels

• Description of the experiment

• Results of the experiment

• Conclusions ?


Environmental decision analysis

• Parliamentary nuclear power decision(Hämäläinen et. al)

• Decision analysis interviews(Marttunen & Hämäläinen)

• Spontaneous decision conferencing in nuclear emergency management(Hämäläinen & Sinkko)


Cognitive biases

• Splitting bias– attribute receives more weight if it is split

– origins: subjects give rank information only (Pöyhönen & Hämäläinen)

– Not observable in hierarchical weighting


Decision analysis interviews

• Opinions of large groups of people traditionally collected through questionnaires

• Decision analysis interviews may provide a more reliable way to collect these opinions

• Idea:– one value tree for all = common terminology

– emphasis on finding the viewpoints of different stakeholder groups

– interactive, computer supported


Research interest

• Existence of biases in a real case

• Can biases can be avoided through training and proper instructing ?

• Identify what can go wrong in the Lake Päijänne case

• Compare the well trained university students’ and spontaneous stakeholders’ responses


The Lake Päijänne case

• Regulation started 1964

• Main aims were to improve hydroelectricity production and to reduce damages caused by flooding

• Environmental values & increase in free time

• need for an improved regulation policy


Splitting bias

• When an attribute is split, the weight it receives increases

Attribute 1

Attribute 2

Attribute 3

Attribute 1

Attribute 2

Attribute 3

subattribute 1b

Subattribute 1a

0.4

0.3

0.3

0.3

0.3

0.1

0.3

0.4


Swapping of levels

• Does the order of the levels affect the resulting weights?

• Important question in environmental decision analysis:– stakeholder groups may vary regionally

• Not studied before


Example of swapping of levels

Lake Päijänne

River Kymijoki

Attribute 3

Attribute 3

Attribute 2

Attribute 1

Attribute 2

Attribute 1

Attribute 3

Attribute 2

Attribute 1

River Kymijoki

Lake Päijänne

River Kymijoki

Lake Päijänne

River Kymijoki

Lake Päijänne


Earlier experiments on biases

• Structure of the decision model affects the results

• Previous experiments typically:– subjects: university students

– problems: artificial

– results: taken from group averages

• Lake Päijänne-case: a real problem with real stakeholders


Important new features

• Realistic case

• Decision analysis interviews instead of passive decision support or survey

• Interactive computer support (resulting weights shown immediately)

• Instructions and training before the weighting


Subjects:

• University students attending a course on decision analysis (N = 30)– held during a tutorial session, not mandatory

• Habitants of Asikkala (N = 40)– 3 groups of students

– 1 group of adults (volunteers)

• 3 experts from the Finnish Environment Institute & 2 summer residence owners


Experimental setting

• Weighting done with the SWING method using a tailored Excel interface

• Subjects entered the numbers themselves, two assistants were present to help

• Resulting weights shown as bars

• Order of value trees partly randomized


Sessions

• A short introduction to: – Lake Päijänne case

– value trees & weighting

– different structures of the value tree

• In HUT the avoidance of biases was emphasized more

• Duration: 60 - 90 minutes


SWING method

• Easy to use

• Attribute ranges clearly presented

• Idea:– choose the attribute you would first like to

move to its best level

– assign it 100 points

– assign other attributes points less than 100 in respect to the first attribute


Flat-weighting

Muu talous ???

VesivoimaVesivoima

Muu talous

Ympäristö

Talous

Rantojen käytettävyys

Virkistyskalastus

Kalojen lisääntyminen

Rantakasvillisuus

Lahtienumpeenkasvu

Virkistys

Luonto

Tulvat, maatalous jateollisuus

Tulvat, loma-asutus

Vesiliikenne

Ammattikalastus


Upper level weights:

Muu talous ???

VesivoimaVesivoima

Muu talous

Ympäristö

Talous


Virkistyskalastus


Rantakasvillisuus

Lahtienumpeenkasvu

Virkistys

Luonto

Tulvat, maatalous jateollisuus

Tulvat, loma-asutus

Vesiliikenne

Ammattikalastus


ENV5-tree:

Luonto

Virkistys

Ympäristö

Talous


Virkistyskalastus


Rantakasvillisuus

Lahtienumpeenkasvu


ENV2-tree:

Luonto

Virkistys

Ympäristö

Talous


EC5-tree:

Muu talous ???

VesivoimaVesivoima

Muu talous

Ympäristö

TalousTulvat, maatalous ja

teollisuus

Tulvat, loma-asutus

Vesiliikenne

Ammattikalastus


EC2-tree:

Muu talous ???

Vesivoima

Muu talous

TalousMuu talous ???

Vesivoima

Muu talous

Ympäristö

Talous


Swapping of levels:

Muu talous ???

Kymijoki ja muut

Päijänne

Rantakasvillisuus

Tulvavahingot

Kymijoki ja muut

Päijänne

Muu talous ???

Rantakasvillisuus

Tulvavahingot

Kymijoki ja muut

Päijänne

Rantakasvillisuus

Tulvavahingot


Flat weights vs. upper level weights

• Both in group averages and in results of individuals the total weights for the environment and economy were similar with both methods

• One explanation: symmetric value tree


Splitting bias0

.00

.30

.50

.81

.0

ASIKKALA HUT

Means of Wenv

GROUP

Wenv

TREE

EC2EC5ENV2ENV5


A typical resident in Asikkala

0

0.2

0.4

0.6

0.8

1ENVIRONMENT ECONOMY

5 1 5 2 1 1 5 1 1 1 5 2


Example from HUT(one of the best ones)

0

0.2

0.4

0.6

0.8

1ENVIRONMENT ECONOMY

5 1 5 2 1 1 5 1 1 1 5 2


Why even weights ?

• Some students: none of the attributes seemed to be important

• Asikkala: all of the attributes were important

even weights for all attributes


What caused the bias ?

• Similar points for

all attributes in one branch

regardless of the structure

of the value tree

100 80

80100 7070 70 50

100 80 70


Effect of instructions

• Students had good instructions– only some had bias in their results

• In the spontaneous stakeholders’ sessions the information load was too high and thus the instructions were not adopted as well– nearly all had systematically consistent bias


n

nenv

econ

n

nenv

econ

w

wenv

econ

w

wenv

econ

STUDENTS STAKEHOLDERS

Adjusted / not adjusted weights


Examples

01234567

0 1 2 3 4 5

01234567

0 1 2 3 4 5

STUDENTS STAKEHOLDERS

w

w

env

econ

w

w

env

econ

n

nenv

econ

n

nenv

econ


Observation

• The students and the experts from FEI could nearly avoid the splitting bias– good background education + instructions did

reduce the bias

• What did the students think? - Arithmetics or real avoidance of biases


Avoiding the splitting bias ?

• Good instruction can eliminate it

• When the economical attributes were split, the magnitude of the bias was slightly larger

• Graphical feedback did not eliminate

• Hierarchical weighting


Swapping of attribute levels

0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

0.6

0.8

1

If the order of the levels would not affect the weigts, the pairs of weights should be equal (as in the first picture)


Conclusions about swapping of levels ?

• Only a few had clearly differing weights with the two trees

• No systematic pattern was found

• Less differences residents of Asikkala and students than with the splitting bias

• A simple scale lead to similar weights with both trees (100, 70 for example)

• Neither tree gained clear support


Solutions to reduce biases ?


• Models should be tested on real decision makers

• Interactiveness of weighting (= possibility to return to change the points given earlier )

• Well balanced trees


Other observations in Asikkala

• Concept of weight seemed to be difficult for most subjects in Asikkala

• Information load was high

• Facilitators role becomes important when the DM’s are uncertain


Problems related to the Lake Päijänne case

• Current regulation policy cannot be improved very significantly – no big differences between the alternatives

– unrealistic hopes and false information are probably larger problems than the regulation itself

• ‘money is not money’– strong feelings against the power companies

and regulation (shape of value function ?)


Suggestions for future research


• Encouragement to reconsider and readjust the statements iterate

• Decision Analyst must supervise!


R.P. Hämäläinen, E. Kettunen, M. Marttunen and H. Ehtamo: Evaluating a framework for multi-stakeholder decision support in water resources management, Group Decision and Negotiation, 2001. (to appear)

M. Pöyhönen, Hans C.J. Vrolijk and R.P. Hämäläinen: Behavioral and procedural consequences of structural variation in value trees. European Journal of Operational Research, 2001. (to appear)

M. Pöyhönen and R.P. Hämäläinen: There is hope in attribute weighting, Journal of Information Systems and Operational Research (INFOR), vol. 38, no. 3, Aug. 2000, pp. 272-282. Abstract

R.P. Hämäläinen, M. Lindstedt and K. Sinkko: Multi-attribute risk analysis in nuclear emergency management, Risk Analysis, Vol. 20, No 4, 2000, pp. 455-467.

M. Pöyhönen and R.P. Hämäläinen: Notes on the weighting biases in value trees, Journal of Behavioral Decision Making, Vol. 11, 1998, pp. 139-150.

Susanna Alaja: Structuring effects in environmental decision models, Helsinki University of Technology, Systems Analysis Laboratory, Theses, 1998.

References


M. Pöyhönen, R.P. Hämäläinen and A. A. Salo: An experiment on the numerical modeling of verbal ratio statements, Journal of Multi-Criteria Decision Analysis, Vol. 6, 1997, pp. 1-10.

R.P. Hämäläinen and M. Pöyhönen: On-line group decision support by preference programming intraffic planning, Group Decision and Negotiation, Vol. 5, 1996, pp.485-50.

M. Marttunen and R.P. Hämäläinen: Decision analysis interviews in environmental impact assessment, European Journal of Operational Research, Vol. 87, No. 3, 1995, pp. 551-563.

R.P. Hämäläinen, A.A. Salo and K. Pöysti: Observations about consensus seeking in a multiple criteria environment, in: Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences, Vol. IV, 1991, IEEE Computer Society Press, Hawaii, pp. 190-198.

R.P. Hämäläinen: Computer assisted energy policy analysis in the parliament of Finland, Interfaces, Vol. 18, No. 4, 1988, pp. 12-23. Also in: Case and Readings in Management Science, 2nd edition, M. Render, R.M. Stair Jr. and I. Greenberg (eds.), Allyn & Bacon, Massachusetts 1990 pp. 278-288.

Documents

Can We Avoid Biases in Environmental Decision Analysis ?