CS 589 Information Risk Management 23 January 2007

CS 589Information Risk Management

23 January 2007

Today’s Discussion

• Start with risk

• Discuss types of information risk

• Start with systematic, modeling-based framework for assessing alternatives when risks are known

• Continue with the hard part – specification of risk when risks are unknown

Next Week

• Discuss specification of risks using probability distributions

• Discuss incorporation of this information into a decision tree

• Discuss ways to apply these techniques to Information Risk scenarios

After Next Week

• Discuss the Expected Utility decision criterion

• Discuss Multiple Objectives and Expected Value and Expected Utility

• Discuss Applications in Information Risk Analysis and Management

References for Today

• Clemen, R. L. and T. Reilly, Making Hard Decisions. Duxbury, 2001.

• Gaffney Jr., J. E., J. W. Ulvila, “Evaluation of Intrusion Detectors: A Decision Theory Approach”, Proceedings of the IEEE Symposium on Security and Privacy. 2001.

Risk

• ???

• Chance of something bad happening?

• Having something bad happen?

• Anything else?

Risk

• The probability of an event occurring combined with the consequences of that event

• Just about everything is risky

• How do we actually measure risk?

Risk vs Uncertainty

• Uncertainty – We don’t know what the key variables are– We don’t know how they relate to alternatives

• Risk– Specify probability distributions– Connect them with alternatives

• One goal: Uncertainty Risk via Modeling

Thinking About Risk

• Probabilities and Outcomes

• Which is riskier?– Living near a large power generation station– International flight– Driving to Albuquerque

• We have to define factors, events, outcomes, and associated probabilities

Dealing with Risk

• Define Risk

• Assess Risk

• Define Alternatives for Handling the Risk

• Evaluate Alternatives

• Evaluate your Evaluation Model

• Sensitivity Analysis

• Implementation

Evaluation

• Choosing among Alternatives

• Should be Evaluated on the same dimension(s)– Expected Value– Expected Utility– Value at Risk (VAR)– Multiple criteria

• Measurement of Alternatives on criteria dimensions is key – and another modeling issue

Sensitivity Analysis

• Checking on the evaluation of each alternative by varying individual variables

• Find the variable(s) that have the largest impact(s) on the ordering of alternatives

• Goal: robust solutions

Visual Representation

• Influence Diagrams– Connect factors, events– Help us define risks– Decomposition

• Decision Trees– Ordering of decisions, risky events– Easy to see and present – and solve

Visual Representations

• Squares denote Decisions

• Circles denote Risks

• Influence Diagrams – arcs connect decision and risk (aka chance) nodes

• Decision Trees – decision and chance nodes are sequentially ordered from left to right

A Very Simple Example

• Coin Flip Game

• Decisions: Play/No Play

• Risks: Heads/Tails

• Outcomes Must be Specified

50.0% 0.5

0 0

TRUE Chance

0 0

50.0% 0.5

0 0

Decision

0

FALSE 0

0 0

Coin Flip

Play

No Play

Heads

Tails

Coin Flip Game Decision Tree With $0 Outcomes

If All Outcomes are $0

• We are Indifferent between Play and No Play based on the Expected Value criterion

• We Prefer Play to No Play if

E(Play) > E(No Play)

• Which means that the sum of the outcomes (if we have a fair coin) must be positive

• Generally, Play if

0 TTHH OOOp

What if we can play twice?

• Sequential decision – we see the result of the first coin flip, and decide to continue

• This leads to the notion of Strategies – we can make a plan contingent upon resolution of risks that are resolved between decision nodes

• Everything is still based on Expected Value

50.0% 0.25

0 0

TRUE Chance

0 0

50.0% 0.25

0 0

50.0% Decision

0 0

FALSE 0

0 0

TRUE Chance

0 0

50.0% 0.25

0 0

TRUE Chance

0 0

50.0% 0.25

0 0

50.0% Decision

0 0

FALSE 0

0 0

Decision

0

FALSE 0

0 0

Coin Flip

Play

No Play

Heads

Tails

Play

No Play

Heads

Tails

Play

No Play

Heads

Tails

Suppose

• O(H) = $10, O(T) = -$7

• p(H) = p(T) = .5 (Fair coin)

• We can easily see that we would choose to Play in the one-game case

• What about the 2-game case?

p(H) 0.5 50.0% 0.25

O(H) 10 10 20

O(T) -7 TRUE Chance

0 11.5

50.0% 0.25

-7 3

50.0% Decision

10 11.5

FALSE 0

0 10

TRUE Chance

0 3

50.0% 0.25

10 3

TRUE Chance

0 -5.5

50.0% 0.25

-7 -14

50.0% Decision

-7 -5.5

FALSE 0

0 -7

Decision

3

FALSE 0

0 0

Coin Flip

Play

No Play

Heads

Tails

Play

No Play

Heads

Tails

Play

No Play

Heads

Tails

Strategy

• It’s pretty simple – keep playing

• Would you really do this?

• Do you believe this?

• Why or why not??

Simple Example

• Suppose we are assessing two alternative intrusion detection systems.

• What’s the problem?

• What are the key risks for this decision?

• What are the decisions?

• What are the outcomes?

• How would we measure the outcomes?

• What is the decision criterion?

Key Point

• The optimal choice will be the one that is associated with the best expected criterion value – such as expected total cost

• This will be determined by how we define the outcomes – in terms of total costs – and probabilities

• When we roll back a decision tree, we assume that the downstream decision is the best one

Expected Value

• Random Variable with possible discrete

outcomes

X~

nxxx ,...,, 21

n

iii xxpXE

1

~

Choose System Intrusion

Detection

Respond?

Payoff

Example

System 1

System 2

Detection

No Detection

Respond

No Response

Respond

No Response

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Respond

No Response

Intrusion

No Intrusion

Intrusion

No Intrusion

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Outcome 5

Outcome 6

Outcome 7

Outcome 8

Outcome 9

Outcome 10

Outcome 11

Outcome 12

Outcome 13

Outcome 14

Outcome 15

Outcome 16

Example

System 1

System 2

Detection

No Detection

Respond

No Response

Respond

No Response

Intrusion

No Intrusion

Intrusion

No Intrusion

Intrusion

No Intrusion

Intrusion

No Intrusion

Detection

No Detection

Respond

No Response

Respond

No Response

Intrusion

No Intrusion

Intrusion

No Intrusion

Intrusion

No Intrusion

Intrusion

No Intrusion

What do we need to know?

• Probabilities– P(Detection|An Intrusion) P(D|I)– Associated Info– P(I)– And, finally, P(I|D)

• Outcomes– Individually, these will not be stochastic – for now– They will still lead to an expectation for each

decision node

Conditional Probability

• P(D|I) and P(D| Not I)

• P(Not D|I) and P(Not D|Not I)

• Where would we get this information?

• What about P(I)?

Bayes Rule – Simple Version

)()|(

)()|()(

)()(|

|

IPIDP

IPIDPDP

DPIPIDP

DIP

Interpretation

• Two types of Accuracy

• Two types of Error

)|()|(

)|()|(

IDPIDP

IDPIDP

Solving the Tree

• Establish the Outcomes

• Compute the Probabilities – the conditionals on the endpoints and others

• Find Expected Values and roll back the tree

P(I) 0.3 0.953177258 0.285

P(D|I) 0.95 0 0

P(D not|I not) 0.98 TRUE Chance

C(No Intrusion) 0 0 -0.468227425

C(Intrusion) 100 0.046822742 0.014

C(False Positive) 10 -10 -10

0.299 Decision

0 -0.468227425 0.953177258 0

-100 -100

FALSE Chance

0 -95.31772575

0.046822742 0

0 0

TRUE Chance 0 -1.64

0.021398003 0

0 0

FALSE Chance

0 -9.786019971

0.978601997 0

-10 -10

0.701 Decision 0 -2.139800285

0.021398003 0.015

-100 -100

TRUE Chance

0 -2.139800285

0.978601997 0.686

0 0

System 1

Detection

No Detection

Respond

No Response

Intrusion

No Intrusion

Respond

No Response

Intrusion

No Intrusion

Intrusion

No Intrusion

Intrusion

No Intrusion

Sensitivity Analysis

• What are the strategies given the numbers we used in the example?

• What are the key variables?

• How would we assess the base-case outcome of this example?

Different Conditional Information

• What if we don’t know P(D|I)?

• We can flip the tree according to what we do know

• Outcomes should remain the same

• And the decision should remain the same

Another Way – Info Dependent

)()|(

)()|()(

)()()|(

)|(

DPDIP

DPDIPIP

IPDPDIP

IDP

Modeling

• Decisions, chance events

• Probability distributions for chance events– Lack of data Bayesian methods– Expert(s)– Lots of data Distribution model(s)

• Outcomes– Financial, if possible– Multiple measures/criteria/attributes

Decision Situation

• In the context of Firm or Organization Goals, Objectives, Strategies

• A complete understanding should lead to a 1-2 sentence Problem Definition– Could be risk-centered– Could be oriented toward larger info issues

• Problem Definition should drive the selection of Alternatives and, to some degree, how they are evaluated

Information Business Issues

• Integrity and reliability of information stored and used in systems

• Preserve privacy and confidentiality

• Enhance availability of other information systems

Risk Management

• Process of defining and measuring or assessing risk and developing strategies to mitigate or minimize the risk

• Defining and assessing– Data driven– Other sources

• Developing strategies– Done in context of objectives, goals