Upload
fay-terry
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Levers of Control along with other
Management Models&
Risk Management
May 04
John Driessnack
Outline
• Tools for the Tool Box – how broad is your box!• Initial thoughts…take a broad view• Management Models – various articles/books • Robert Simons – Levers of Control • Driessnack Framework View
•Risk Management … Overview of Theory and Models• Sources on the WEB … lots of help/ ARQ articles • DoD Policy … more emphasis, but can avoid!• Summary• Possible Discussions …
• DAU workshop…simple approach for IPTs• Implementation….details on F-18 Risk Program • RISK inside EV – need for RPI and TPI• Tools (databases) …
Management 1974
Peter F Drucker• The FORD vis SLOAN (GM) Story…”Management is needed not only because the job is too big
for any one man to do himself, but because managing an enterprise is something essentially different from managing one’s own property” pg 384
• “Above all, disagreement is needed to stimulate the imagination. One may not need imagination to find the one right solution to a problem. But then this is of value only in mathematics. In all matter of true uncertainty such as the executive deals with – whether his sphere be political, economic, social, or military – one needs creative solutions which create a new situation. And this means that one needs imagination – a new and different way to perceiving and understanding.” pg 473
• “…the effective decision-maker compares effort and risk of action to risk of inaction. There is no formula for the right decision here. But the guidelines are so clear…act if on balance the benefits greatly outweigh cost and risk; and act or do not act; but do not “hedge” or compromise.” pg 476
• “As a specific discipline, management has its own basic problems, its own specific approaches, its own distinct concerns. A manger who understand the discipline of management will still be an effective – and may even be a first-rate – manager with no more than minimum competence in managerial skills and tools. A man who knows only the skills and techniques, without understanding the fundamentals of management, is not a manger ; he is, at best a technician.
• Management is a practice rather than a science. In this, it is comparable to medicine, law and engineering. It is not knowledge but performance. Furthermore, it is not the application of common sense, or leadership, let alone financial manipulation. Its practice is based both on knowledge and on responsibility.”
Delusions of SuccessHow Optimism Undermines Executive’s Decisions
By Lovallo and Kahneman; HBR July 03
• Lists numerous examples of failures in Industry• Reject “rational risks in uncertain situations”• Propose over optimism from cognitive biases
• errors in way mind processes information• organizational pressures
•Problems• Anchoring – initial plan accentuate the positive • Competitor Neglect – underestimation of negative events• Organizational Pressure – internal competition big incentive to accentuate positives in forecasts
• Optimism in Its Place – a distinction between• functions and positions that
• involve decision making• that promote or guide action
Space Systems Development Growth Analysis5
SBIRS High Quantitative Framework (A Point in Time)
1996 Estimate
$3.2B1997
President’sBudget$2.6B
MPC $2.4B Award
$2.1B1
2
3
4
5
6
Bill
ions
of
Th
en Y
ear
Dol
lars
SBIRS High EMD CostsEstimate, PB, Bid, Award and EAC TY$
Note: PB - President’s Budget; MPC - Most Probable Cost; EAC - Estimate at Completion
(1) $6.2B from 29 March 2001 SBIRS High Program Office Estimate (POE). Includes 3600, 3020 and SPO Cost funds (satellites 3,4, & 5)
Other Sources: ABIDES; Source Selection Document; BAH analysis
ContractorBid
$1.6B
Mar 2002POE
Scope$1.3B
Variance$3.3B
Total = $6.2B
Growth Attributed to
Variance
Added Scope
S&V
• REQ GEN/TRANS
• JET/SEIT
• RISK
• BASELINE RESTRUCTURE
• OTHER ECPS
• ESTIMATING ERRORS
•INCOMPLETE INFORMATION
• PRICE INCREASES
•CHANGES FROM JET/SEIT
• CONTRACTOR SCHEDULE CHANGES
• RISK ESTIMATION
Why Good Projects Fail Anyway
By Matta and Ashkenas; HBR Sept 03
• Focus on “execution risks” and neglect;• “white space risk” – unknowns• “integration risk” – disparate activities won’t come together
• Suggest a “rapid-results initiative”…spirals!!
• Closing paragraph: “Attempting to achieve complex goals in fast-moving and unpredictable environments is humbling. Few leaders and few organizations have figured out how to do it consistently. … Managers expect they will be able to identify, plan for, and influence all the variables and players in advance, but they can’t. Nobody is that smart or has that clear a crystal ball. They can, however, create an ongoing process of learning and discovery, challenging the people close to the action to produce results – and unleashing the organization's collective knowledge an creativity in pursuit of discovery and achievement.”
Why Hard-Nosed Executives Should Care About
Management TheoryBy Christensen & Raynor; HBR Sept 03
• Understand more than jacket blurbs
• Need to “articulate a theory of causation”
• “Good theories are circumstance contingent”• “Under what circumstances will the use of this theory lead to failure?”
• Recommendations:• Go beyond description, explain what works under what circumstances• … fallacy of “all companies in all situations”• “Correlations that masquerade as causation…”
What Really Works…4+2By Nohria, Joyce, Roberson; HBR July 03
• Findings surprising, most tools no direct causal relationship. What matters is strong grasp of basics.
• Excel in primary management practices…the 4• Strategy – Devise and maintain a clearly stated focus• Execution – Develop/maintain flawless operational execution• Culture – Develop/maintain a performance-oriented culture• Structure – Build/maintain a fast, flexible, flat organization
• Secondary management practice…the plus 2• Talent – Winners hold onto talented employees/develop more• Innovation – Turns out innovative products/service and anticipates disruptive events rather than reacting • Leadership – Choosing great chief executives can raise performance significantly• Mergers and Partnerships – Internally generate growth is essential, but master of mergers and acq can also be winners
Driessnack
Contract Cost Track
$M (TY)
* 3 PIPs & 96 RS* 38 SRS & 1 TIP* 7 SRS & 15 GRS
(LRIP)
50
100
200
150
0
ContractSCP
OriginalContract
CAIV TODAYPre-CAIV
* 3 PIPs & 96 RS* 38 SRS & 1 TIP* 7 SRS & 15 GRS
(LRIP)
* 3 PIPs & 150 RS** 41 SRS & 1 TIP
** 7 SRS & 15 GRS (LRIP)
* 3 PIPs & 150 RS
$172M * $116M
$205M
$85M
$136M
* 3 PIPs & 150 RS* 41 SRS& 1 TIP
* 7 SRS & 15 GRS (LRIP)
$13M Award Fee
$19M O&M Options
$19M (41SRS & 1 TIP)
$65M ECPs Cost Growth
Scope Changes #
$85M
$17M Award Fee
$19M O&M Options
$19M (38SRS & 1 TIP)
$85M
$19M O&M Options
$12M Award Fee
$30M O&M Options
$14M**
$65M ECPs
Cost Growth Scope Changes
$85M
$17M Award Fee
$19M O&M Options
$19M (38SRS & 1 TIP)
$8M VAC
$232M$245M
$19M ORD Requirements
# $65M ECP Changes; 50% overrun, 50% scope change consisting of S/W growth, PMO and NR engineering
$29M Award Fee
Driessnack
Schedule History (as briefed on 3 May to DAE)
285 days
405 days
J F M A M J J A S O N D1998
J F M A M J J A S O N D1999
J F M A M J J A S O N D2000
S O N D1997
J F M A M J J A S O ND2001
XPS Impact
TRANSMIT S/WIAT & C
SYSTEM TEST / TSATMOT & E IOC & MS III
(OBJECTIVE)
WAHIAWA NORFOLK
SHIP INSTALLTIME
2002
J FM A
2 MONTHMARGIN
AS OF 2/00
4/99
AS OF PDR
GOVT CONFIDENCE TESTADDED (GREEN)
210 days
210 days
CONTRACT AWARDAS OF11/97
SIGONELLA
Driessnack
-16
-14
-12
-10
-8
-6
-4
-2
0
2
Jan Apr Jul Oct Jan Apr Jul Oct Jan
Cost "A Leg"
Schedule
Cost "B Leg"
GBS EVMS EvolutionCumulative Dollar Variance
Cu
m V
aria
nce
($M
)
CAIV Mid-CourseCorrection
B Leg WBSA Leg WBS
As of Jun 00 Rebaseline
$-5M CV
$-5.5M SV
98 99 00
Delay PACOM
(3 Months)
ATR Delayed
(3 Months)
-$13.1M A Leg CV
C Leg WBSUpdateEVM
Simons, 94; Figure 1
Simons, 94; Table 1
Simons, 94; Table 4
Simons - Levers of ControlAn integrated approach is required
BusinessStrategy
Risks to Be Avoided
CoreValues
Strategic Uncertainties
CriticalPerformance
Variables
- Positive- mindfulness- process - innovation
BELIEF SYSTEM(Perspective - commitmenton grand purpose)
- Negative- mindless- outcomes- efficiency
BOUNDARY SYSTEM(Position - stakingout the territory)
DIAGNOSTIC CONTROL SYSTEMS(Plan - getting the Job done)
INTERACTIVE CONTROL SYSTEM(Patterns in Action - positioning for tomorrow)
Defines the Domain
Internal Controls
Driessnack
DoD Weapons AcquisitionPM Framework Measures
Costs, Schedule, Performance
Product Process
Defined - Stability Repeatable Accurate
Technology Challenge People Tools
Training WBS/EV Experience Integrated
Management and Leadership – over Periods
Pro
du
cer - Po
litics
Driessnack
Driessnack Framework View(visual – not really 5x3x4)
Product
Producer
Process
People
Politics
Cost Sched Perf
Belief Boundary DiagnosticInteractive
Visual Representation
What Period – look over time(time phasing different for each program)
Assess the • Capability Maturity• Risks/Opportunities
Capability Maturity ModelGeneric view of levels
• Level 1: Initial/Performed– Simply performing processes in a manner that is often ad hoc– Competence/heroics of individuals doing the work drive activity
• Level 2: Managed/Repeatable– Processes managed, they are planned, performed, monitored,
and controlled for individual projects and groups
• Level 3: Defined– Defines processes and tailors them based on organization set of
standards. Deviations beyond allowed tailoring is documented
• Level 4: Quantitatively Managed– Control processes using statistical and other quantitative
techniques. Process performance is understood
• Level 5: Optimizing– Continually improved based on understanding of variations
Outline
• Tools for the Tool Box – how broad is your box!• Initial thoughts…take a broad view• Management Models – various articles/books • Robert Simons – Levers of Control • Driessnack Framework View
•Risk Management … Overview of Theory and Models• Sources on the WEB … lots of help/ ARQ articles • DoD Policy … more emphasis, but can avoid!• Summary• Possible Discussions …
• DAU workshop…simple approach for IPTs• Implementation….details on F-18 Risk Program • RISK inside EV – need for RPI and TPI• Tools (databases) …
Driessnack
Acquisition Review QuarterlySpecial Edition on Risk - Spring 2003
• Copy in Print or DAU web page (www.dau.mil)• Editors – LtCol John Driessnack & Noel Dickover
• My two cents- quotes from article
• …implementation of risk management is often shallow and not well integrated with other program management tools• Transparency relative to program status is going to increase as each service expands its information systems, …• The emphasis on risk, realism, and knowledge-based acquisition has the potential to truly transform the way we view acquisition programs.• … move beyond standard program baselines with point estimates…transparency into the risks in a program by exposing the range in the estimate, given the risks involved, …
•
Driessnack
Acquisition Review QuarterlySpecial Edition on Risk - Spring 2003
• Article from PMCoP Risk Community Members
• 2nd – Research…“there was no correlation found between cost and schedule length. In addition, we did not find the anticipated connection between cost growth and schedule growth…”• 3rd – Overview that asks you questions about Managing Risk in a Program Office Environment • 4th – Seven specific issues relative to contracts and risk program • 5th – An approach that breaks out the business (project) and performance (product) side of decision supports• 6th – Industry view with PMI PM Body of Knowledge (PMBOK) and defense extension - comparisons• 7th –Normalize your measures with an Index to Measure a System’s Performance Risk • 8th – Army’s Health Hazard Assessment Program – one of many views of a program
Driessnack
Driessnack
Driessnack
Break Out One Step Added Implementation
Driessnack
Driessnack
DAU PM Tool Kit Feb 2002
Driessnack
Proposed Risk Management ProcessRisk CoP Community Effort
Prioritized List
AvoidTransferAssume
Quantify Some
Qualify All
Identify
Select how to Handle
Assess ListMonitor
status toplan
Mitigate
Plan for eachProgram byPhase/Evolution
Implementation through IPPD
PeriodicUpdate
Raw List
Integrate Into Program Plans& Control Toolsas necessary
Update asneeded
Risk/Opportunity Analysis
Negative
CriticalSeriousModerateMarginalMinorLimited Value
Some Value
Moderate Value
High Value
Very
High Value
Positive -- Consequence --
Remote
Unlikely
Likely
Highly Likely
Near Certainty
Pro
bab
ilit
y
Need Unknown and Effect
Driessnack
Some Analysis Usingthe Triangular Distribution
Average (expected) cost = (low + most likely + high) / 3(70 + 100 + 180) / 3 = 350 / 3 = 116.7
RelativeLikelihoodofOccurring
Possible Element Costs
Expected Cost = 116.7
70 100 180
Driessnack
Logic of Probabilistic Branch
Test Unit
Start: 8/10/02 ID: 5
Finish: 9/3/02 Dur: 25 d
Res:
FIXIT
Milestone Date: Tue 9/3/02
ID: 6
Retest
Milestone Date: Tue 9/3/02
ID: 7
Finish
Milestone Date: Tue 9/3/02
ID: 8
Succeed 70% Branch
Fail 30% Branch
New Activities
Driessnack
Driessnack
Driessnack
DoD “old” PolicySelected References to RISK
• DoDD 5000.1, The Defense Acquisition System– 4.5. Effective Management….tailor considering risk
• DoD 5000.2-R, Mandatory Procedures for MDAPs/MAIS – Numerous references to RISK....management and mitigation
– 1.2.4.2 Risk reduction in source selection criteria
– 1.4.3.3.2 Cost Estimates include assessment of RISK
– 2.3, 2.5, 2.9 Acquisition Strategy …reduce System-Level risk to acceptable levels…industry bear risks
- 5.2.3.4.3…establish a risk management process
– 7.4 Exit Criteria
• DoDD 5000.4, OSD CAIG – The CAIG Chair report … include quantitative assessments of risk…
• DoD 5000.4-M, Cost Analysis Guidance and Procedures– Para 1.E.1.2, … Subsystem Description address risk issues
– Para 1.E.2.0, Risk..PM assess & plan to address/reduce
Driessnack
DoD 5000.2-R Exit Criteria
“Phased-specific exit criteria normally track progress in important technical, schedule, or management risk areas. Unless waived or modified by the MDA, exit criteria must be substantially satisfied for the program to continue with additional activities within an acquisition phase or to proceed into the next acquisition phase, depending on the decision with which they are associated. Exit criteria shall not be part of the APB and are not intended to repeat or replace APB requirements or the entrance criteria specified in DoD Instruction 5000.2. They shall not cause program deviations. The DAES (see paragraph C7.15.3. and Appendix 1) shall report the status of exit criteria.”(5000.2-R, Para 7.4)
New 5000.2 - no mention of risk in criteria
Driessnack
Program Baselines
Threshold Objective
Performance
Cost
Schedule
Minimum acceptable level which will meet user needRDT&E, MilCon, Proc & AUPC, PAUC
Objective + 10%
6 Months beyond objective date (3 months for ACAT IA)
Cost effective increment in operational capability above threshold
Planned cost to meet program objectives
Planned event dates to meet program objectives
Does not represent the RISK on a ProgramNo baseline for, or tracking of Risk
New 5000.2 – no suggested 10% or 6 months
Driessnack
DoD “new” Policy References to RISK – 22 to 22!
• 5000.1– 4.3.1 Flexibility …no one best way– 4.3.2 Responsiveness …time phased capability – 4.3.4 Discipline …program goals for minimum number…– 4.5 Streamlined and Effective Management …decentralize to maximum extent
practicable– E1.5 Cost Realism…proposal that are realistic– E1.6 Cost Sharing…undue risk is not imposed (contractor)– E1.14 Knowledge-Based Acquisition…
• Tech, Integration, and manufacturing risk reduced– E1.21 Program Stability …realistic program schedules.– E1.27 Systems Engineering approach
• 5000.2 – 19 references to Risk
Risk and Realistic in enough paragraphs?
DoD Acquisition Guide ..Draft Risk Areas
5.3.12.2 ESOH Risk Management5.5.2 Management Activities5.5.2.1 Risk Management
7.5 Human Systems Integration (HIS) Strategy, Risk, and Risk Mitigation
9.4 Acquisition Protection Strategy for PMs9.4.4 Risk Management
11 Decisions, Assessments, and Periodic Reporting • The review associated with each decision point typically
addresses program progress and risk, affordability, program trade-offs, acquisition strategy updates, and the development of exit criteria for the next phase or effort.
12 Program Management Activity12.4 Risk Management
PROGRAM SUCCESS PROBABILITY SUMMARY
Program Success(2)
Program Requirements (3)
Program Execution
Contract Earned Value Metrics (3)
Program “Fit” in Capability Vision (2)
Program Parameter Status (3)
DoD Vision (2)
Transformation (2)
Interoperability (3)
Army Vision (4)
Current Force (4)
Stryker ForceTesting Status (2)
Program Risk Assessment (5)
Contractor Performance (2)
Program Resources
Budget
Contractor Health (2)
Manning
Program Advocacy
OSD (2)
Joint Staff (2)
War Fighter (4)
Army Secretariat
Congressional
Industry (3)
Fixed Price Performance (3)
Other
Program “Smart Charts”
Program Scope Evolution
Sustainability RiskAssessment (3)
Joint (3)
Technical Maturity (3)
Legends:Colors: G: On Track, No/Minor Issues Y: On Track, Significant Issues R: Off Track, Major Issues Gray: Not Rated/Not Applicable Trends: Up Arrow: Situation Improving (number): Situation Stable
(for # Reporting Periods) Down Arrow: Situation Deteriorating
PEOPEOXXXXXX COL, PM Date of Review: dd mmm yy
ProgramProgramAcronymAcronym
ACAT XXACAT XX
INTERNAL FACTORS/METRICS EXTERNAL FACTORS/METRICS
Program Life Cycle Phase: ___________
Future ForceInternational (3)
Force Management Risk
Definition: Challenge of sustaining personnel,
infrastructure and equipment
Risk Mitigation Examples
• Manage careers and rotations
• Modernize infrastructure and facilities
• Training, spares and overall readiness
Future Challenges Risk Definition: Challenge of dissuading, deterring,
defeating longer-term threats Risk Mitigation Examples
Experiment with new concepts, capabilities and organizational designs
Investing in transformational capabilities for portions of the force
Foster a spirit of innovation and risk taking
Operational Risk Definition: Challenge of deterring or defeating
near-term threats Risk Mitigation Examples
Plan and prosecute war on terror Elevate role of homeland defense Develop forward deterrence posture Enhance operational capabilities with Allies
Balanced Scorecard Approach(Proposed DoD Scorecard Areas)
Institutional Risk Definition: Challenge of improving efficiency
represented by unresponsive processes, long decision cycles, segmented information, etc.
Risk Mitigation Examples Modernize financial management systems and
approaches Acquisition excellence initiatives Improve planning and resource allocation
Driessnack
Intro Summary
• Lots of information….
• Several models to follow…don’t be constrained• think about how to incorporate “opportunity” • how to integrate across broad framework
• Get off point solutions…understand “range”• you pick the confidence level…not the technician
• Policy is not specific…may be changing
• “Realistic” is the new buzz word…use it !
Outline
• Tools for the Tool Box – how broad is your box!• Initial thoughts…take a broad view• Management Models – various articles/books • Robert Simons – Levers of Control • Defense Acquisition View
•Risk Management … Overview of Theory and Models• Sources on the WEB … lots of help/ ARQ articles • DoD Policy … more emphasis, but can avoid!• Summary• Possible Discussions …
• DAU workshop…simple approach for IPTs• Implementation….details on F-18 Risk Program • RISK inside EV• Tools (databases) …
Driessnack
Parking Lot Some questions
• Does your program have/had Risk Management Plan ?• Contractor plan or PROGRAM plan ? • Are you following the plan? On all activities?• Do you have a Risk Board/Panel/Group ?
• How often does it meet ? Does it have influence ?• How many risks are you tracking ?• Do you quantify your risks?
• How integrated is your risk process with other tools• Do you quantify risk in your…
• Cost/Budget Estimates … Schedule/Network?• Requirements/Technical requirements matrix
• Do you force issues into your risk process?• What percentage of issues were foreseen?
RISK IDENTIFICATION:Output-risk events
Use the yellow cards. Write one risk per card. Use complete sentencesDo not discuss with your team…yet ( take about 20 Minutes)
WORK HERE
1. I dentify Clearly Describe Risk Event
Work Package #________
2. Analyze
Very Likely
Not Likely
Big I mpact Little I mpact
Risk Priority # ________
Risk Areas People_____ Process_____ Technology_____
1
2
3
4
Long Term ______ Short Term _____ Program Phase(s) and/ or Expected Date(s) of risk fi rst occurrence ___________________________________
I nternal Control___ _ External Control_______ Are the causes of the risk event within or outside the control of program team?
WORK HERE
Also check if The risk event Is PEOPLEPROCESS ORTECHNOLOGY
Driessnack
Driessnack
Driessnack
UNCLASSIFIED
UNCLASSIFIED
48
— ILS Software— Supportability Engineering— Training— Peculiar Support Equipment— Contractor Logistics Support
B
B
B
B
B
A
A
A
A
A
— Launcher Software— Launcher Transporter/
Missile Round Pallet
BB
AA
THAAD 02T-1036.02a
High- State-of-the-art research is required, and failure is likely and will cause seriousdisruption of program schedule and/or degradation of system performance even with special attention from contractor and close government monitoring
Moderate- Major design changes in hardware/software may be required with moderateincrease in complexity. If failure occurs, it will cause disruption in scheduleand/or degradation in performance. Special attention from contractor andclose government monitoring can overcome the risk.
Low- Existing hardware is available and/or proven technology application can overcome risk. Failure is unlikely to cause disruption in program schedule or degradation in system performance. Normal efforts from contractor/government can overcome risk.
20 Oct 2003
= Before Mitigation
= After Mitigation
B
A
THAAD Development Program Risk Assessment (U)
Risk reduced from moderate to low since contract award
THAAD System
Weapons Systems Engineering Integration Team (WSEIT)
System Test & Evaluation
C2/BM Missile
— System Engineering— Integration & Test— Software Management— Battle Management— Operations Management— Communications— System Support— Operator System Interface— Embedded Training— C2/BM Hardware— C2/BM Segment I&T
Environment (BSITE)
ILS
Launcher
Program Mgt
RadarB A
B
B
B
B
B
B
B
B
B
B
B
B
A
A
A
A
A
A
A
A
A
A
A
A
B A
AB
AB
AB
B A
B A AB
B A
— Product Test Equipment— Mission Software— Lethality— Missile Fore-Body— Missile Mid-Body— Range Safety Equipment— Ordnance Initiation System/
Flight Termination System— Seeker— Mission Computer— Inertial Measurement Unit— Two Axis Rate Sensor— Interstage & Booster Intr— Flare & Aft Skirt— Canister & MR Assembly— Booster Motor— Thrust Vector Actuator— Divert & Attitude Control System— Communications Systems— Power System & Interconnects— Instrumentation & Telemetry— Systems Engineering
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
— Radar Software— Antenna Equipment— T/R Module— Electronics & Shelters— Systems Engineering
B
B
B
B
B
A
A
A
A
A
THAAD 03T-1300.88
Driessnack
RISK ASSESSMENT Sub-process: Identification
• What Are the Risk Events in the Program?– Known / Knowns … Known / Unknowns– Unknown / Unknowns (uncertainty….)
• Where Are Risk Events Located in the Program? – Technology, Design, T&E, M&S, Cost, Funding, Schedule
• The IPPD process and empowerment…but measure!
– Program Office Structure – Functional or Product or both– Oversight and Politics…MDA/Space making major changes!– US or World Markets – IT (hot/cold cycles)
• Examine Sources/Areas of Risk – Tactical and STRATEGIC (inside/outside PM Office)– Product or Process (predicts the product problems)
Don’t narrow vision of your Risk Program
Driessnack
Understand External Risk In each decision system
• Requirements Generation – JCS/Service Chief Oversight• Requirements Generation System (CJCSI 3170.01B)
• Track beyond your KPP/thresholds• What is the changing environment – Crusader!
• Interoperability and Supportability of National Security Systems, and Information Technology Systems
• Worry about the “other guy”…two in “interoperability”
• PPBS Process – SECDEF Oversight and Politics• There really is no book! A Political “Free” Market!
• Defense Acquisition – MDA Oversight• DoD 5000….but this is only the formal rules• Who are the players…what are their agendas
Driessnack
Technology Readiness Levels (TRL) DoD 5000.2-R (Appendix F)
Technology Readiness Levels (TRL) DoD 5000.2-R (Appendix F)
• TRL #1 - Basic principles observed & reported• TRL #2 - Technology concept and/or application
formulated• TRL #3 - Analytical & experimental critical function
and/or characteristic proof of concept• TRL #4 - Component and/or breadboard validation in
laboratory environment.• TRL #5 - Component and/or breadboard validation in
relevant environment• TRL #6 - System/subsystem model or prototype
demonstration in a relevant environment• TRL #7 - System prototype demonstration in an
operational environment• TRL #8 - Actual system completed and qualified
through test & demonstration• TRL #9 - Actual system proven through successful
mission operations
(IOT&E during LRIP)
(System Demonstration)
(ATD/ACTD/ System Integration)
(basic research)
(applied research)
(advanced technologydevelopment)
(Component Advanced Development)
Driessnack
PlannedProfile
AchievedTo Date
Tolerance Band
Current Estimate
Threshold
Milestones
PlannedValue
VariationTECHPARAMVALUEe.g.,VehicleWt (lb)
TIMEVerPDRSFREqns
10
5
15
Technical Performance Measurement
Dr Falk Chart - modified
Normalize and track on index (like SPI & CPI)
Driessnack
Driessnack’s Thoughts…why EV evolved for 35 plus years
• Tracks programs on THEIR baseline, not rubrics• Rubrics – such as OSD O&E rates – drive behaviors
• Want program managed to particular baseline• Normalize the metric
• EV CPI/SPI track to the plan• TPMs that are normalized (see ARQ article)• O&E rates could be normalized to plan• ….almost anything can be put into an index
• Index (normalization) allows comparisons
• Each program has unique path of unique events• Need to characterize the uncertainty….
Driessnack
Driessnack’s Thoughts…why EV evolved for 35 plus years
• AT&L Staff working on next CARS (DAES/SAR) system• What data is needed…what process is needed…?
• EV has method of validating the overall process• Improve trust in the data reported…know detail exists • Reputation dependent on validation…real impacts• Need to incorporate RISKs into baselines…ranges!
• Industry builds tools to comply with criteria…• Need new criteria that drives industry to develop tools…incorporate Risk plus schedule into wInsight• Other tools manufactures do the same….
Driessnack
• Overall goal - integration of tools• WUG …
• Integrate Risk into EV develop RPI & TPI• CS Solutions - Risk+ schedule inside wInsight
• DoD baselines and tracks programs…risk neutral!
• Discuss of Risk • What is taught….some program examples• Various Management Tools not integrated
• General Proposal…beyond risk reserves• Discussion….• Discussion on RISK CoP…DAU WEB pages
Outline
Driessnack
Most trying to integrate with other tools…schedule, cost, TPMs, etc
Driessnack
- RIGHT PARTNERS (CPAR)- RISK MANAGEMENT PLANS/TEMPLATES- TPMs- EARNED VALUE ANALYSIS- COMMUNICATION
Insight not Oversight- TESTING AND MORE TESTING
…..ALL COMBINE TO YIELD AN INTEGRATED APPROACH TO DEVELOPING AN AFFORDABLE FAMILY OF …..
RISK REDUCTION TOOLS
Chart from Major DoD Program
Driessnack
- RIGHT PARTNERS (CPAR)- RISK MANAGEMENT PLANS/TEMPLATES- TPMs- EARNED VALUE ANALYSIS- COMMUNICATION
Insight not Oversight- TESTING AND MORE TESTING
…..ALL COMBINE TO YIELD AN INTEGRATED APPROACH TO DEVELOPING AN AFFORDABLE FAMILY OF …..
RISK REDUCTION TOOLS
Chart from Major DoD Program
Programs understand
they must integrate
Driessnack
Supporting Processes
• Quality management
• Contract acquisition
• Contract execution
Risk Identification• Change in threat• Technology
introduction• Design cost• Funding• Schedule• Field problems• Pollution
prevention initiatives
• Spare replacement• Environmental
compliance directives
Risklevel
Retirerisk?
Onplan
?
Monitor lowand retired risks
Risk MitigationPlanning/Implementation
Input Process Output
Statuschange
?
Perform riskassessment
analysis
No
Yes
Yes
• Review Analysis• Assess impact• Assign levels
System Wide
……. Risk Management Process from …. Risk Management Process from …
Med, High
Low
No
No
Yes
Track progress
Implement risk plan
Plan risk mitigations
• Change specification• Re-evaluate impact• Revise mitigations
Replan
Risk Management Board (RMB)
RiskAssessment
• Risk forms• Watchlist• Mitigation
Plans• Data Book
Driessnack
Supporting Processes
• Quality management
• Contract acquisition
• Contract execution
Risk Identification• Change in threat• Technology
introduction• Design cost• Funding• Schedule• Field problems• Pollution
prevention initiatives
• Spare replacement• Environmental
compliance directives
Risklevel
Retirerisk?
Onplan
?
Monitor lowand retired risks
Risk MitigationPlanning/Implementation
Input Process Output
Statuschange
?
Perform riskassessment
analysis
No
Yes
Yes
• Review Analysis• Assess impact• Assign levels
System Wide
……. Risk Management Process from …. Risk Management Process from …
Med, High
Low
No
No
Yes
Track progress
Implement risk plan
Plan risk mitigations
• Change specification• Re-evaluate impact• Revise mitigations
Replan
Risk Management Board (RMB)
RiskAssessment
• Risk forms• Watchlist• Mitigation
Plans• Data Book
Most Program have separate
Risk and EV processes
Driessnack
Risk Management Products Overview
Risk Management BoardRisk Management Board
Risk FormsRisk Item Description: Very brief description of the risk item. Be sure to use words like“Potential for…” instead of implying that risk is actually occurring (unless it really is).
Product/Task: Examples: Maintain WS Reliability, Maintain Availability of XX assets.
Responsible Individual/IPT: Firstname Lastname/Level 3 IPT Date of Evaluation: M/D/Y
Performance Measures: Examples: Voltage at GPR vs. Age, Subsystem Reliability vs. Age.
T: TechnicalConsequence if Risk I tem Occurs :
High - What will happen if risk item occurs & why it is High (or Mediumor Low) in terms of technical risk
Probability of Risk Item Occurring:Low - What is the probability of it this consequence happening & why itis Low (or High or Medium).
S: ScheduleConsequence if Risk I tem Occurs :
High - What will happen if risk item occurs & why it is High (or Mediumor Low) in terms of schedul risk
Probability of Risk Item Occurring:Low - What is the probability of it this consequence happening & why itis Low (or High or Medium).
C: CostConsequence if Risk I tem Occurs :
High - What will happen if risk item occurs & why it is High (or Mediumor Low) in terms of cost risk
Probability of Risk Item Occurring:Low - What is the probability of it this consequence happening & why itis Low (or High or Medium).
Mitigation Plan:• Description 1• Description 2• Description 3
Conseque
nce
Probability of Risk Item Occurring
H M L
H
M
L6
4
9
7
52 1
8
3
TSC
Risk ID #MM & PKRisk Assessment Summary
Mitigation PlansID Task Name Duration Start
1 Task 1 14.5 days 12/13/99
2 Task 2 4 days 12/13/99
3 Milestone A 1 day 12/14/99
4 Task 3 7 days 12/15/99
5 Task 4 10.5 days 12/17/99
6 Task 5 9 days 12/17/99
7 Task 6 5 days 12/20/99
8 Milestone B 1 day 12/21/99
9 Task 7 3 days 12/27/99
10 Task 8 7 days 12/17/99
11 Task 9 4 days 12/27/99
12 Task 10 4 days 12/28/99
13 Milestone C 1 day 12/31/99
12/14
12/21
12/31
S S M T W T F S S M T W T F S S M T W T F SW3 W2 W1
Risk Quantification
23David Lindblad / TRW 05/16/2000 4:13 PM
Risk Quantification Sample
* Readiness is measured by Availability (Alert Readiness Rate: ARR and Operational Readiness Rate: ORR,Reliability, Accuracy, and Survivability (Hardness).
Readiness Projections
WeaponSystem
Readiness*Measure
Assessed at Component, Subsystem, and System levelsNotional Data
Fiscal Year
Acceptable Minimum
With Mitigation
Without Mitigation
WatchlistChange
RiskID #
WSRisk
Factor/Opened
MitigStatus
LMCFundingStatus
Risk Item Description
New S98 PK1
9/9/99Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).
M: W to GF: W to G
S99 MM2
9/9/99Green Green
Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).
M: R to Y Gr99 Both3
9/9/99Yellow
SCGreen
Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).
F: R to Y Pr99 PK4
9/9/99Green Yellow
Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).
2 to 4 Gu99 MM 4Red
TGreen
Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).
Closed R99 Both 6Very short title/phrase. If xxxxxx then, xxxxx (or equivalent impact statement).
Watchlist SummariesWatchlist Count Summary
IPT 1 2 3 4 5 6 7 8 9 TotalSEIT # # # # # # # # # ##Ground # # # # # # # # # ##Propulsion # # # # # # # # # ##Guidance # # # # # # # # # ##RS/RV # # # # # # # # # ##Total ## ## ## ## ## ## ## ## ## ###
Risk Factor
M itig atio nS tatu s 1 2 3 4 T o ta lG reen # # # # ##Y ello w # # # # ##
R ed # # # # ##T o ta l ## ## ## ## ###
R isk F acto r
M itig a tio n S ta tu s S u m m ary
M itig a tio nS ta tu s 1 2 3 4 T o ta lG reen # # # # ##Y ello w # # # # ##
R ed # # # # ##T o ta l ## ## ## ## ###
R isk F acto r
F u n d in g S ta tu s S u m m ary
C h a n g eR is kID #
W SR is k
F a c to r/O p e n e d
M itigS ta tu s
M itigF u n d in g
S ta tu sR is k Ite m D e s c rip tio n
N e w S 9 8 P K1
9 /9 /9 9V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).
M : W to GF : W to G
S 9 9 M M2
9 /9 /9 9G re e n G re e n
V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).
M : R to Y G r9 9 B o th3
9 /9 /9 9Y e llo w
S CG re e n
V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).
F : R to Y P r9 9 P K4
9 /9 /9 9G re e n Y e llo w
V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).
2 to 4 G u 9 9 M M 4R e d
TG re e n
V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).
C lo s e d R 9 9 B o th 6V e ry s h o rt t it le /p h ra s e . If xxxxxx th e n , xxxxx (o r e q u iva le n t im p a c t s ta te m e n t).
W a tc h lis t C h a n g e S u m m a ry
Most Program have separate
Risk and EV processes
Driessnack
Most Program have separate
Risk and EV processes
Driessnack
• Overall goal - integration of tools• WUG …
• Integrate Risk into EV develop RPI• CS Solutions - Risk+ schedule inside wInsight
• DoD baselines and tracks programs…risk neutral!
• Discuss of Risk • What is taught….some program examples• Various Management Tools not integrated
• General Proposal…beyond risk reserves• Discussion….• Discussion on RISK CoP…DAU WEB pages
Outline
Driessnack
-16
-14
-12
-10
-8
-6
-4
-2
0
2
Jan Apr Jul Oct Jan Apr Jul Oct Jan
Cost "A Leg"
Schedule
Cost "B Leg"
GBS EVMS EvolutionCumulative Dollar Variance
Cu
m V
aria
nce
($M
)
CAIV Mid-CourseCorrection
B Leg WBSA Leg WBS
As of Jun 00 Rebaseline
$-5M CV
$-5.5M SV
98 99 00
Delay PACOM
(3 Months)
ATR Delayed
(3 Months)
-$13.1M A Leg CV
C Leg WBSUpdateEVM
Driessnack
-16
-14
-12
-10
-8
-6
-4
-2
0
2
Jan Apr Jul Oct Jan Apr Jul Oct Jan
Cost "A Leg"
Schedule
Cost "B Leg"
GBS EVMS EvolutionCumulative Dollar Variance
Cu
m V
aria
nce
($M
)
CAIV Mid-CourseCorrection
B Leg WBSA Leg WBS
As of Jun 00 Rebaseline
$-5M CV
$-5.5M SV
98 99 00
Delay PACOM
(3 Months)
ATR Delayed
(3 Months)
-$13.1M A Leg CV
C Leg WBSUpdateEVM
As PM I knew I would continue to rebaseline and develop C, D, E legsUsed Critical Chain – separate EV
baseline
Management Reserve
ACWPC
$
TimeNow
BCWSC
Completion Date
Schedule VarianceC
Cost VarianceC
PROJECTED
SLIPPAGE
OverBudget
EACContract at a Glance
BCWPC
ScheduleSlip
PMB
BAC
Driessnack
Driessnack
Driessnack
Driessnack
Driessnack
Driessnack
PRODUCTION
MILESTONES
DOCUMENTATION
EMD
12/93
MS II
4/00
LRIP DAB
9/01
LRIP 2
6/02
LRIP 3
7/03
MS III/FRP
F/A-18 TEMP
APB/ASR
JTEMP SINGLE MIDS ORD
11/95 5/96 3/029/0110/00 6/01
NAVYORD
SHIPTEMP
NOW
2/00OA
5/03IOC
2/01OA
10/02
Pilot Prod IOT&EAward DEL FRP IOC
SHIPS
ARMY
F-16
12/99INTEGRATION
LUT IOT&E1/01
7/011ST FLIGHT
6/02
F-15LVT (3)
9/989/96 10/99 3/00 6/00 2/01
9/02LVT(2)
FRP/IOC
INTEGRATION TECHEVAL OPEVAL
F/A-18
OA 11/00
OPEVALTECHEVAL
FY06PLATFORM INTEGRATION & TEST
3/94 2/98 1/99 6/00
MIDSCO 1st DELIVERY CDT&EMIDSCO ENDS SE&I
7/00
PROD READINESS
1/0011/011ST US DELIVERIES
8/97
PROD LONG LEAD & NRE
6/00 12/00
DLS/VIASAT EUROMIDS
3/031ST EU DELIVERIES
4/02
2004200320022001200019991998FY
DT6/02
DT 3/03
MIDS Program Overview
1/991ST FLIGHT
2/98
DT 3/01
6/02 10/02 1/04 6/04
Del Compl
11/03
FOE II IOT&E
Driessnack
COBRA JUDY (O-IPT CHARTS MS A)
Risk Assessment
L
Functional Area Risk
Technology
Design and Engineering
Human Systems
Support
Cost1
Schedule
Programmatics2
L
L
H
M
L
M
1High risk at current funding levels2Requires Acquisition Agent designation
Driessnack
MUOS Risk Assessment
Risk: CAD Contract award delayed
Mitigation Plans:
– Source selection process– Well defined selection criteria– Four versions of draft RFP– MS A scheduled for Aug 02
x
Schedule Risk #1
Risk: IOC of July 2008
Mitigation Plans:
– EVMS– Contract Incentives– Design off-ramps– Need funding stability
x
Schedule Risk #2
Risk: Cost estimates inaccurate
Mitigation Plans:
– Better system definition– Fund to CAIG estimate at MS B– Cross-check PO model– CAD Ktr bottoms-up estimate
x
Cost Risk #1
Risk: MUOS Cost Control
Mitigation Plans:
– CAIV– EVMS– Multi-year procurement– Incentives in SD&D
x
Cost Risk #2
Risk: Software development
Mitigation Plans:
– Software development plan– SEI Level III certification– OPTEVFOR EOA– Independent assessment
x
Technical Risk #1
Risk: System Integration
Mitigation Plans:
– JTRS MOA– Modeling/simulation– CAD demonstrations– Interface Control IPTs
x
Technical Risk #2
Driessnack
IPPD ENVIRONMENTEmpowerment
“ Decision making should be driven to the lowest possible level commensurate with risk. Resources should be allocated to levels consistent with risk assessment authority, responsibility & ability of people.
– The team should be given the authority, responsibility, & resources to manage its product & its risk commensurate with the team’s capabilities.
– The authority of team members needs to be defined & understood by the individual team members.
– The team should accept responsibility & be held accountable for the results of its efforts.”
IPPD Guide,” 5 Feb 96 - OUSD(DTSE&E)
You get what your measure
Driessnack
Critcal Risk ProcessesRisk Management Guide 2001
Driessnack
What is the Likelihood the Risk will happen?
Level Chance of Occurrence Your Approach and Processes
a
b
c
d
e
Not Likely:10% chance
Low Likelihood:25% chance
Moderate:50% chance
Highly Likely:75% chance
Near Certainty:90% chance
…Will effectively avoid this risk based on standard practices
…Have usually avoided this type of risk with minimal oversight in similar cases
…May avoid this risk, but workarounds will be required
…Cannot avoid this risk with standard practices, but a different approach may work
…Cannot avoid this risk with standard practices, probably not able to mitigate
Given the risk is realized, what would be the magnitude of the impact?
Level Technical Performance Schedule Impact
1Negligible
Small performance shortfall inspecific technical area; overall system performance unaffected
Minimal schedule slip but able to meet need dates w/o additional resources. Critical path unaffected
2Marginal
3Moderate
4Critical
5Catastrophic
Con
seq
uen
ce
Addition resources required to meetneed dates, Critical path unaffected
Minor performance shortfall inspecific technical area; overallsystem performance below goal but w/in acceptable limits
Mod performance shortfall inspecific technical area; overallsystem performance below goal& possible below acceptable limits
Minor schedule slip; will missneed date. Critical pathunaffected
Overall system performancebelow acceptable limits
Overall system performanceunacceptable to the degree thatthe ship is undeliverable
Major schedule slip. Program criticalpath affected (<1 month)
Major schedule slip. Program criticalpath affected (>1 month)
CostMillions
Costincrease <1
Costincrease 1-6
Costincrease 6-20
Cost increase
21-50
Costincrease >50
Li k
eli h
ood
Consequence
Lik
elih
o od
1 2 3 4 5
e
d
c
b
a
HIGH--Major program disruption,immediate priority management actionrequired.
MODERATE--Moderate disruption,possible management action required
LOW--Minimum impact
Questions about Risk Management?
Call Systems Engineering - Risk Management,Dept. E47
Steve Waddell, NNS 757-688-3760
CVN 77 Program Risk Assessment
Driessnack
NSSN Risk Assessment ProcessNSSN Risk Assessment ProcessQuestions about Risk
Management?
Call a Member of the Process Integration Team for Risk.
1 Minimal or No Impact Minimal or No Impact Minimal or No Impact None
2 Acceptable with Some Additional Resources Required ; < 5%Some Impact
Reduction in Margin Able to Meet Need Dates
3 Acceptable with Minor Slip in Key Milestone; 5 - 7% Moderate Impact
Significant Reduction Not Able to Meet Need Datesin Margin
4 Acceptable, No Major Slip in Key Milestone > 7 - 10% Major Impact
Remaining Margin or Critical Path Impacted
5 Unacceptable Can’t Achieve Key Team or > 10% Unacceptable
Major Program Milestone
CONSEQUENCE:Given The Risk is Realized, What is the Magnitude of the Impact?
NSSN Risk Process Card
February 1996
RISK ASSESSMENT
HIGH - Unacceptable. Major disruption likely. Different approach required. Priority management attention required.
MODERATE - Some disruption. Different approach may be required. Additional management attention may be needed.
LOW - Minimum impact. Minimum oversight needed to ensure risk remains low.
a Remote
b Unlikely
c Likely
d Highly Likely
e Near Certainty
Level What Is The Likelihood The Risk Will Happen?
LIKELIHOOD:
Level Technical Schedule Cost Impact on Other Teams
Performance
and/or and/or and/or
edcba
1 2 3 4 5
Lik
elih
ood
Consequence
ASSESSMENT GUIDE
Safety Reqt’s Exceeds Spec
Family of Systems
Safety of Crew in Cab
PRS #2 Availability
Reloader Redesign
Safety impacts Design
11
22
45
42
15
Insufficient Spares
Turret Servo Maturity
DT Duration too short
High Failure Rates
Moderate
642
18 32
28
12
15452232
18
60.6
1
Potential Severity of Consequence (Cf)
0.5
Low
High
•
•
•
•
••
•
•
•
•
0.1 0.2 0.3 0.4 0.6 0.7 0.8 0.9 10
0.1
0.2
0.3
0.4
0.5
0.7
0.8
0.9
0
11
28
12
Pro
bab
ility
of
Occ
urr
ence
(P
f)
CLAWS Top Risks
Driessnack