Decision Theory (CIV6540 - Machine Learning for Civil ...€¦ · Intro Utility theory Utility & Loss Functions Value of Information Summary Decision Theory (CIV6540 - Machine Learning

Intro Utility theory Utility & Loss Functions Value of Information Summary

Decision Theory(CIV6540 - Machine Learning for Civil Engineers)

Professor: James-A. Goulet

Departement des genies civil, geologique et des mines

Polytechnique Montreal

Chapter 14 – Goulet (2018).Probabilistic Machine Learning for Civil Engineers.

Chapter 16 – Russell, S. and Norvig, P. (1995).Artificial Intelligence, A modern approach. Prentice-Hall.

Professor: J-A. Goulet Polytechnique Montreal

Decision Theory | V1.2 | Machine Learning for Civil Engineers 1 / 26


Context

Making rational decisions - Fire alarm

t

0.0

5

0.95

Pr = 0.99

Pr = 0.01

t − 1 min

0.99

0.01

1

Pr = 0.7

Pr = 0.3

t + 1 min

0.99

1

0.01

0.0

50.95

Pr = 0.1

Pr = 0.9

1

10

10

0.01

E[U] =3.7

E[U] =7.0

t + 1

Pr = 0.1

Pr = 0.9

E[U] =9

E[U] =1




Context

Making rational decisions - Soil contamination

We have 1 m3 of soil form an industrial site. What should we do?

Pr = 0.9

Pr = 0.1

0$

100$

10K

$

100$

E[$| ] = 0$× 0.9 + 10K$× 0.1 = 1K$

E[$| ] = 100$× 0.9 + 100$× 0.1 = 100$

Optimal action




Nomenclature

Nomenclature

A = {a1, a2, · · · , aA} A set of possible actions

x ∈ Z or ∈ R An outcome in a set of possible states

Pr(x) Probability of a state x

U(ai , x) Utility given a state x and an action ai

L(ai , x) ≡ −U(ai , x) Loss given a state x and an action ai




Nomenclature

Soil contamination example

ai ∈ { , } ≡ {0, 1}

x ∈ { , } ≡ {0, 1}

Pr(x) = [0.9, 0.1]ᵀ

U(ai , x) = U[

, ,, ,

]≡[

0$ −10K$−100$ −100$

]

L(ai , x) = L[

, ,, ,

]≡[

0$ 10K$100$ 100$

]




Rational decisions

Rational decisions → Expected utility maximization

The perceived benefit of an outcome xi given an action ai ismeasured by the expected utility or expected loss

E[U(a)] =∑X

i=1 U(a, xi ) · Pr(xi )

E[L(a)] =∑X

i=1 L(a, xi ) · Pr(xi )

The optimal action a∗ is the one that maximizes the expectedutility or minimizes the expected loss

a∗ = arg maxa

E[U(a)] = arg mina

E[L(a)]




Module #9 Outline

Intro

Utility theoryUtility & LossFunctionsValue ofInformation

Topics organization

Probabilitytheory

{1 Revision probability & linear algebra

2 Probability distributions −4 −2 0 2 40

0.2

0.4 µ µ+σµ−σ

x−4 −2 0 2 40

0.5

1µ µ+σµ−σ

x

F X(x)

f X(x)

MachineLearning

Basics

0 Introduction

3 Bayesian Estimation p(A|B) =p(B|A)p(A)

p(B)

4 MCMC sampling & Newton −5 0 5−5

0

5

θ1

θ 2

−50

5

−50

50.00

0.05

0.10

θ1θ2

s = 1000

Data-drivenmethods

5 Regression [ ] cl+

Pato

gens

[ppm

]

6 Classification0 0.2 0.4 0.6 0.8 10

10

x

f X|D

(x|d

)

d=0 d=1 d=2 d=3 d=4 d=5

0 0.2 0.4 0.6 0.8 10

0.5

1

P(D

S)

S={4,5} S={2,3} S={0,1}

7 State-space model for time-series

Model drivenmethods

{8 Model Calibration −2

02

−20

2

y1y2

−2 0 2

−2

0

2

y1

y 2

DecisionMaking

{9 Decision Theory

E[U] = 7.0

E[U] = 3.7

10 AI & Sequential decision problems




Section Outline

Utility theory2.1 Lotteries2.2 Axioms of utility theory




Lotteries

Nomenclature for ordering preferences

A lottery: Li = [{p1, x1}, {p2, x2}, · · · , {pX, xX}]

L = [{1.0, , }{0.0, , }]L = [{0.9, , }{0.1, , }]

A decision maker

Li � Lj prefers Li over Lj

Li ∼ Lj is indifferent between Li and Lj

Li � Lj prefers Li over Lj or is indifferent




Axioms of utility theory

Axioms of utility theory

What is defining a rational behaviour?

Orderability: Exactly one of (Li � Lj), (Lj � Li ), (Li ∼ Lj) holds

Transitivity: if (Li � Lj) ∧ (Lj � Lk), then (Li � Lk)

Continuity: if (Li � Lj � Lk), then ∃p : [{p, Li}; {1− p, Lk}] ∼ Lj

Substitutability:if (Li ∼ Lj), then [{p, Li}; {1− p, Lk}] ∼ [{p, Lj}; {1− p, Lk}]

Monotonicity:if Li � Lj , then (p ≥ q ⇔ [{p, Li}; {1− p, Lj}] � [{q, Li}; {1− q, Lj}])

Decomposability: ...no fun in gambling




Section Outline

Utility & Loss Functions3.1 Utility3.2 Non-linear utility functions3.3 Utility and Loss functions U(v) & L(v)3.4 E[L(v(ai ,X ))] and risk aversion




Utility

Axioms → utility

Existence of a utility function:

U(Li ) ≥ U(Lj) ⇔ Li � Lj

U(Li ) = U(Lj) ⇔ Li ∼ Lj

Expected utility of a lottery:

E[U([{p1, x1}, {p2, x2}, · · · , {pX, xX}])] =X∑

i=1

piU(xi )




Non-linear utility functions

Do you want to take the lottery?

L = [{12 , + }, {1

2 , - }]L = [{1,+0$}]

Which lottery do you choose? Why?

E[$(L )] = 12 ×+200$ + 1

2 ×−100$ = +50$

E[$(L )] = 0$

Are you being irrational?For individuals U($) and L($) are non-linear...




Utility and Loss functions U(v) & L(v)

Risk aversion and utility functions U(v)U(v): An utility function weight monetary value (v) as a functionof risk aversion/propension(± 1$ not the same effet if you have 1$ or 1M$)

0 0.2 0.4 0.6 0.8 1v

0

1

Util

ity,

(v)

Risk averseRisk neutralRisk seeking

People/organizationsare risk averse

U(v) = vk

k > 1 Risk seekingk = 1 Neutral0 < k < 1 Risk averse





Risk aversion and loss functions L(v)L(v): A loss function weight monetary value (v) as a function ofrisk aversion/propension

0 0.2 0.4 0.6 0.8 1v

0

1

Loss

, (

v)


People/organizationsare risk averse

L(v) = vk

k > 1 Risk aversek = 1 Neutral0 < k < 1 Risk seeking





Attitude toward risks

0 0.2 0.4 0.6 0.8 1v

0

1

Loss

, (

v)


L(v) = vk


A neutral attitude toward risks minimizes the expected costsover a multiple decisions

I Insurance compagnies: neutral attitude toward risksI Insured people: risk averse; they pay a premium not to be in

a risk neutral position(i.e. expected costs are higher over multiple decisions)





Expected Loss

0 0.2 0.4 0.6 0.8 1v

0

1

Loss

, (

v)


Value → Loss:

Value x = x =a = v( , ) v( , )a = v( , ) v( , )

→Loss x = x = E[L(v(a,X ))]

a = L(v( , )) L(v( , )) E[L(v( ,X ))]a = L(v( , )) L(v( , )) E[L(v( ,X ))]




E[L(v(ai , X ))] and risk aversion

E[L(v(ai ,X ))] and risk aversion (ex. discrete) [ ]

0 0.5 10

0.5

1

E[v(a

1,X

)]=

E[v(a

2,X

)]v(ai, x)

p(v|a

i,x)

Action a1Action a2

0 10

1

v(ai, x)

Loss

,L(v)

01

E[L(v(a1, X)] = E[L(v(a2, X)]

p(L(v)|ai, x)

Risk perception ¬neutral: E[L(v(a1,X ))] 6= E[L(v(a2,X ))]



[CIV ML/Decision/PCgA discrete.m]



E[L(v(ai ,X ))] and risk aversion (ex. discrete) [ ]

0 0.5 10

0.5

1

E[v(a

1,X

)]=

E[v(a

2,X

)]v(ai, x)

p(v|a

i,x)

Action a1Action a2

0 10

1

v(ai, x)

Loss

,L(v)

01

E[L(v(a1, X)]

E[L(v(a2, X)]

p(L(v)|ai, x)




[CIV ML/Decision/PCgA discrete.m]



E[L(v(ai ,X ))] and risk aversion (ex. continuous) [ ]

0 0.5 1

E[v(a

1,X

)]=

E[v(a

2,X

)]

v(ai, x)

f(v|a

i,x)

Action a1Action a2

0 10

1

v(ai, x)

Loss

,L(v)

0

E[L(v(a1, X))] = E[L(v(a2, X))]

f (L(v)|ai, x)




[CIV ML/Decision/PCgA continuous.m]



E[L(v(ai ,X ))] and risk aversion (ex. continuous) [ ]

0 0.5 1

E[v(a

1,X

)]=

E[v(a

2,X

)]

v(ai, x)

f(v|a

i,x)

Action a1Action a2

0 10

1

v(ai, x)

Loss

,L(v)

0E[L(v(a1, X))]

E[L(v(a2, X))]

f (L(v)|ai, x)




[CIV ML/Decision/PCgA continuous.m]


Section Outline

Value of Information4.1 Value of perfect information4.2 Value of imperfect information4.3 Exemples




Value of perfect information

Expected utility of collecting informationIn cases where the value of a state x is imperfectly known,onepossible action is to collect information about X .

E[U(a∗)] = maxa

X∑i=1

U(a, xi ) · Pr(xi )

U(a∗, x = y) = maxa

U(a, x = y)

Because y has not been observed yet, we must consider allpossibilities Y = Xi according to their probability

E[U(a∗)] =X∑

i=1

maxa

[U(a, xi )] · Pr(xi )


VPI (y) = E[U(a∗)]− E[U(a∗)] ≥ 0





Soil contamination exampleL(a, x) x = x =

a = 100$ 100$a = 0$ 10K$

Current expected costs conditional on actions

E[L( ,X )] = 0$× 0.9 + 10K$× 0.1 = 1K$

E[L( ,X )] = 100$× 0.9 + 100$× 0.1 = 100$ = E[L(a∗,X )]

Expected costs conditional on perfect information

E[L(a∗)] =∑X

i=1 mina

(L(a, xi )) · Pr(xi )

= 0$× 0.9︸︷︷︸y=x=

+ 100$× 0.1︸︷︷︸y=x=

= 10$


VPI (y) = E[L(a∗)]− E[L(a∗)] = 90$Professor: J-A. Goulet Polytechnique Montreal




Value of information

The value of information represents how much you are willing topay for an information.

What if the information is not perfect?




Value of imperfect information

Value of imperfect information

Discrete state case

E[U(a∗)] =X∑

i=1

X∑j=1

maxa

(U(a, yi )) · Pr(yi |xj) · Pr(xj)

Continuous state case

E[U(a∗)] =

∫ ∫maxa

(U(a, y)) · f (y |x) · f (x)dydx




Exemples

Soil contamination example - Discrete caseL(a, x) x = x =

a = 100$ 100$a = 0$ 10K$

Pr(y = | ) = 1

Pr(y = | ) = 0.95

Current expected costs E[L(a∗,X )] = 100$

Expected costs conditional on imperfect information

E[L(a∗)] =∑X

i=1

∑Xj=1 min

a(L(a, yi )) · Pr(yi |xj) · Pr(xj)

= 0$× 0.9︸︷︷︸x=y=

+ ( 100$× 0.95︸︷︷︸y=

+ 10K$× 0.05︸︷︷︸y=

)× 0.1

︸︷︷︸x== 59.5$

Value of information

VOI (y) = E[L(a∗)]− E[L(a∗)] = 40.5$




Summary

Rational Decision:

Choose the action a∗i which minimize the expected lossL(a, x) or maximizes the expected cost U(a, x)

a∗i = arg minai

E[L(ai , x)] = arg maxai

E[U(ai , x)]

L(v(a, x)) & U(v(a, x)): Subjective weight on value as afunction of the attitude toward risks(± 1$ not the same effect if you have 1$ or 1M$)

0 0.2 0.4 0.6 0.8 1v

0

1

Loss

, (

v)


L(v) = vk


Value of information:

Value you should be willing to pay for information

VOI (y) = E[U(a∗)]− E[U(a∗)] ≥ 0

Value of perfect information:

E[U(a∗)] =X∑

i=1

maxa

[U(a, xi )] · Pr(xi )

Value of imperfect information:

E[U(a∗)] =X∑

i=1

X∑j=1

maxa

(U(a, y)) · Pr(y|xj ) · Pr(xj )



Documents

Decision Theory (CIV6540 - Machine Learning for Civil ...€¦ · Intro Utility theory Utility & Loss Functions Value of Information Summary Decision Theory (CIV6540 - Machine Learning