Week 3 – Assessing Risk

Risk Analysis Process Technical & systematic process Examine events Focus on causes, not symptoms Determine interrelationships Document impact in terms of

probability & consequence

Analysis Phase May actually start during identification

process Availability of experts As a natural by product of interviewing, etc.

Define likelihood or probability ratings Define ratings for severity of

consequence Establish assessment matrix

RM Execution Phases

Assessment Phase Primary objective – Identify &

analyze program risks To control the most critical risks Id factors that contribute most to

achieving desired results Id factors to use in setting cost,

schedule, and performance objectives Problem definition stage

Assessment Process Basis of most RM actions Quality of the assessment

determines the effectiveness of the RM program

Tools are available but no one tool has all the answers

Definitizes probability & consequence of potential events

Identify Risk Drivers Compile potential risk events Describe in detail to understand

significance & causes Are events that have significant

impact to program Adverse Consequence Significant Opportunity

Address Root Sources of Risk Assess processes vs acceptable best

practices Consider cost, schedule & technical

impacts Use Willoughby templates in DoD 4245.7

Templates describe an area of risk Specify technical methods for risk reduction Correlate with acquisition phases & milestones

Primarily applicable during development

Willoughby Templates

Risks in Acquisition Process Templates address risks by

common DoD program elements Discussion of risk Outline for reducing risk Relates to program phase timeline

Industrial process for design, test & production of low risk products

Process-Oriented Assessment Program critical technical

processes Evaluate program baselines

Against current & historical data Critical paths Process constraints Critical inputs from outside program

Process Metrics Track process of developing,

building & introducing the system Meeting established milestones Variances from baselines

Earned value Parametric comparisons

Details of critical path, constrained process items

Dependencies beyond program scope

Product Focused Assessment Address risk by program output

Utilize WBS breakout Constraints of master schedule Interfaces with other

programs/products Use independently or in

conjunction with process analysis techniques

Product Metrics Track development of the product Measures of effectiveness &

performance Progress in meeting requirements Test & analysis results Ability to produce & deliver Availability of resources Comparison to past experience

Areas for Assessment

Cost Assessment Use probability distributions

Define distribution & range by WBS element

Use Monte Carlo simulation to assess & aggregate

Utilize expert opinion Address performance and schedule

causes of cost risk

Ranges & Distributions

Risk Element Min Most Likely MaxSimulated

Value RationaleScrap/Rework (2,000) 10,000 12,500 10,774 Min: potential underrun in scrap/rework rates(Firm Fixed Most Likely: Expect current rates to increase 1%Price negotiated) Max: Exceed scrap/rework rates by 25%

Schedule Delay -24000 0 24000 (13,364) Min: complete 1 week early(Level of effort Most Likely: Meet negotiated scheduleLabor) Max: Exceed negotiated schedule by 1 week

Meet Performance 0 2000 20000 3,263 Min: no add'l testingReqmts (new Most Likely: 1 day add'l testingtechnology) Max: 10 days add'l testing

Total Impact: 672

Distributions

StatisticsForecast: Total Cost Impact

Summary:Display Range is from -30,000 to 50,000 Entire Range is from -20,864 to 47,999 After 10,000 Trials, the Std. Error of the Mean is 112

Statistics: ValueTrials 10000Mean 14,069Median 13,993Mode ---Standard Deviation 11,164Variance 124,630,124Skewness 0.00Kurtosis 2.68Coeff. of Variability 0.79Range Minimum -20,864Range Maximum 47,999Range Width 68,863Mean Std. Error 111.64

Percentile Value0% -20,864

10% -62820% 4,35130% 8,03540% 11,20350% 13,99360% 16,96670% 20,18180% 23,76290% 28,667

100% 47,999

Schedule Assessment Extension of Critical Path Method

Define duration ranges & distributions for scheduled activities

Use analytical techniques to identify schedule drivers

Address external schedule impacts Assess probability & magnitude of

an overrun

Schedule AssessmentTask Min

Most Likely Max

Simulated Value Rationale

AA 90 100 120 115.5 Min: 10% challenge of program estimates

AB 72 80 85 72.9 Most Likely: program estimate

AC 45 50 60 56.6 Max: worst case scenario

AD 67.5 75 95 79.4

Total A 305 324.3

Cost-Schedule Containment Chart

Modeling & Simulation Physical, mathematical, or logical

representation of system or process Implementation of a model over

time Use data or expert opinion to select

PDF (probability density function) Preferred for assessing cost or

schedule risk

Which Distribution to Use?

Which Distribution to Use?

More Modeling & Simulation As Virtual Prototyping

Replica of a system flow Duplication of a physical product Representation of a process flow

May be the only way to verify or validate design or process, or assess risk

Before Using any Model Verify

Functions as designed Validate

Represents the system it models Accredit

Is acceptable for the special purpose

ExerciseBy study group, identify how you would verify,

validate, and accredit these simulation methods Group 1 – Virtual simulation

Physical & electrical system representation Ex: Built-in training

Group 2 – Constructive simulation Represents the system & its usage Ex: Mock-up

Group 3 – Live simulation Uses real operators & equipment Ex: Operational tests

You have __ minutes for this exercise. Be prepared to discuss your results.

Establish Rating Criteria From empirical data if possible

Else, define rigorous qualitative measures

Significance based on expert opinion Polling program & industry experts

Variance from best practices Accepted rating definitions

Best Practices example Low

Little or no anticipated impact Normal mgmt attention should control at

acceptable level Medium

May cause some impact Special action & attention may be required

High Likely to cause significant impact Significant additional action & attention

would be required

Concurrency Impact Overlap between program phases From combining phases / activities Schedule adequacy Assess with best practices or historical

dataOverlap in DT&E, ProductionDT&E % Complete Concurrency Risk

>67% Low33% - 67% Moderate

<33% High0% Very High

Developing Measurement Scales

Qualitative analysis Ordinal scales Defines a relative relationship

Quantitative analysis Numerical methods Calibrated ordinal or cardinal scales May be linear or nonlinear

Qualitative Scales Levels defined by experts

Criteria used coordinated with PM Early definition avoids bias

Reflect relative relationship between risk /consequence levels Absolute value on scale not known Not valid for mathematical

manipulation

Ordinal scales Generally reflect ranked data Difference between scale values is

unknown, and not necessarily constant

Misleading if scale is defined numerically

Mathematical operations: Are at best meaningless At worst: misleading

Quantitative Scales Reflect measurable relationship

between risk /consequence levels Cardinal or validated ordinal scales Valid for mathematical manipulation

Tendency to use for calculating a ‘risk value’ Empirical data Simulation & decision analysis results

Qualitative vs. Quantitative Depends on several factors

Information available Nature & phase of program Available resources – personnel,

schedule, budget Availability of experts

Generally qualitative at first, then quantitative as needed or feasible

Probability Ratings Use empirical data if available Otherwise, use expert opinion, etc. Important to know if scales are

ordinalProbability Level

a Remote <10%

b Unlikely 10% to 35%

c Likely 35% to 65%

d Highly Likely 65% to 90%

e Near Certainty >90%

What is the Likelihood the Risk Event Will Happen

Consequence Ratings Define for technical, schedule and

cost

Assessment Matrix Define overall risk ratings

Prioritization

Multi-Voting Technique Each team member receives votes

equal to ½ the number of risks Team members vote for risk items

they think have the highest priority Risks are ranked according to the

vote

Benefits v. Biases ?

Borda Ranking Method Ranks risk by criticality based on

identified criteria Rank by impact of consequence Rank on probability of occurrence

Borda count used to rank risk by criticality

Impact Frequencies

Risk # Conseq Prob C.rank P.rank Borda Cnt Borda Rnk Per Matrix1 Critical 60%2 Serious 100%3 Serious 90%4 Critical 60%5 MInor 40%

Conseq C.freq C.rank Probaility P.freq P.rankCritical 2 91-100% 1Serious 2 61-90% 1Moderate 41-60% 2Minor 1 11-40% 1Negligible 0-10%

Rank Consequence, Probability

Risk # Conseq Prob C.rank P.rank Borda Cnt Borda Rnk Per Matrix1 Critical 60%2 Serious 100%3 Serious 90%4 Critical 60%5 MInor 40%

Conseq C.freq C.rank Probaility P.freq P.rankCritical 2 1.5 91-100% 1 1Serious 2 3.5 61-90% 1 2Moderate 41-60% 2 3.5Minor 1 5 11-40% 1 5Negligible 0-10%

C, P-rank = 1/2 [ 2 (# risks @ higher levels) +1 + # risks @ current level ]

Determine Borda Count

Risk # Conseq Prob C.rank P.rank Borda Cnt Borda Rnk Per Matrix1 Critical 60% 1.5 3.5 52 Serious 100% 3.5 1 5.53 Serious 90% 3.5 2 4.54 Critical 60% 1.5 3.5 55 MInor 40% 5 5 0

for N total risks, Borda Count = ( N - C.rank) + ( N - P.rank)

Highest count is most critical

Borda v. Matrix RankRisk # Conseq Prob C.rank P.rank Borda Cnt Borda Rnk Per Matrix

1 Critical 60% 1.5 3.5 5 1 H2 Serious 100% 4 1 5.5 0 H3 Serious 90% 4 2 4.5 3 M4 Critical 60% 1.5 3.5 5 1 H5 MInor 40% 4 5 0 4 L

Borda Rank = # risks with a higher Borda count

Risk Matrix

P \ C Negligible Minor Moderate Serious Critical0-10% L L L M M

11-40% L L M M H41-60% L M M M H61-90% M M M M H91-100% M H H H H

Assessment Documentation Goal: communicate to customer,

program management, team Define aggregation criteria

Voting method Summary level

Process, e.g. WBS break out Area of risk – cost, schedule, performance By criticality

Impact & Rating Criteria

Risk Aggregation

Summarize Rank Frequency

L Minimum impact; minimum oversight requiredM Some disruption; may need additional attentionH Unacceptable; major disruption likely; priority attention required

P \ C Negligible Minor Moderate Serious Critical0-10%

11-40% 141-60% 261-90% 1

91-100% 1

Order Risks by Borda Count

Risk # Conseq Prob C.rank P.rank Borda Cnt Borda Rnk Per Matrix2 Serious 100% 4 1 5.5 0 H1 Critical 60% 1.5 3.5 5 1 H4 Critical 60% 1.5 3.5 5 1 H3 Serious 90% 4 2 4.5 3 M5 MInor 40% 4 5 0 4 L

Aggregating Results Define reporting format that

communicates the best Group by phase, product, WBS, … Order by color, Borda count, …

Information provided Description, relationship to

requirements Action required Risk owner, …

Aggregation Results

Common Failures in RM Process Definition phase too focused on activities

Need detail on motives, timelines, resources Unclear relationships or motives

Between organizations, analysis methods, models

In identifying sources of uncertainty, risk, consequences

Addressing commonality between issues Links, interdependencies

Document RM process flow to clarify

Next Time: Risk Handling, Monitoring Read: Risk Management Guide

section 5.7

Project – Part IISubmit a paper copy of results with your final exam.

1. Explain how to map risks on your class project into the integrated master schedule you prepared for the mid-term exam and

Provide a cost-schedule containment chart with a description of the steps you took along with intermediate calculations you made to construct the cost-schedule containment chart,

Or, use MS Project (or similar tool) to calculate optimistic, expected, and pessimistic project cost and schedule estimates.

Modifications to your mid-term schedule and task loadings are allowed. (12 pts)

2. Provide a comprehensive trade study for your class project that makes use of program performance measures (technical, cost, and schedule) for decision criteria. Explain how risk considerations were included in your trade study. (12 pts)

3. Provide a risk management process flowchart suitable for presentation that has been tailored for the projected life cycle of your class project. (12 pts)

4. Provide briefing charts (in addition to #3) suitable for orientation and training of project personnel on the procedures in which risk management would be conducted on your project. (14 pts)

Mid-term Closed book, closed notes Turn Part I of your project in with

your exam paper You have 90 minutes for exam. Any questions?