Quantitative Prediction and Improvement of Program
Execution – A New Paradigm
Dr. Shawn Rahmani Boeing Defense, Space/Security
Huntington Beach, California
NDIA 16th Annual SE Conference October, 2013
See attached Boeing’s Assignment of Copyright. The work of authorship referred to herein is a work-made-for hire by one or more employees of The Boeing Company
Agenda
• Problem
• Approach
• Process
• Examples Products
• Recommended Practices and Lessons Learned
• Summary
2 Copyright © 2013 Boeing. All rights reserved.
3
Problem
Col Dave Madden: “. . SW metrics are too rear view mirror - want projections into the future that we can measure ourselves against . . .”
3 Copyright © 2013 Boeing. All rights reserved.
4
$1 M 24 months Based on X ESLOC
Problem (Software Example)
Proposal Estimation Activities
Program Planning
Award ATP
$0.9 M (allocated) 22 months (Based on Y ESLOC??)
Value of the Proposal BOE is ignored/discarded after the Program is awarded
Negotiation/ Budget/Schedule Updates
ESLOC Prod
Cost Cost ESLOC Prod
Copyright © 2013 Boeing. All rights reserved.
Approach: Integrated Model-based PM
Project Characteristics,
WBS, Size (I)
Total Cost and Schedule
(O)
Execution Plan, Cost, Staffing & Work Product
Profiles
(I/O)
Risk Reduction Defects,
Containment (I/O)
Technical Performance-
Cost, Schedule, EVM Data
(I/O) Model
Traditional: Estimating
Integrated Model-based PM Approach
5 Copyright © 2013 Boeing. All rights reserved.
Control
Measure
Predict
Proposal Execution
Estimate /Plan
0
100
200
300
400
500
600
700
800
900
1000
1100
1-M
ar
1-A
pr
1-M
ay
1-J
un
1-J
ul
1-A
ug
1-S
ep
1-O
ct
1-N
ov
1-D
ec
1-J
an
1-F
eb
1-M
ar
1-A
pr
1-M
ay
1-J
un
1-J
ul
1-A
ug
1-S
ep
1-O
ct
1-N
ov
1-D
ec
1-J
an
1-F
eb
1-M
ar
1-A
pr
1-M
ay
1-J
un
1-J
ul
1-A
ug
1-S
ep
1-O
ct
1-N
ov
1-D
ec
1-J
an
1-F
eb
1-M
ar
1-A
pr
1-M
ay
1-J
un
1-J
ul
1-A
ug
BTU 26c SW Tester
BTU 26b SW Coder
BTU 26a SW Designer
BTU 25a SW Requirement Analyst
BTU 25b SW Planner/Management
BTU 24 SW Configuration Management
Estimate /Plan
Detailed Staffing by BTU*
Technical/EAC Predictions
WP** Execution Profiles
What If Analyses
-50% 0% 50% 100%
10. Memory Constraints
9. Tar Sys Exper/Compl
8. Development System Volatility
7. Real Time Code
6. Special Display Requirements
5. Quality Assurance Level
4. Test Level
3. Target System Volatility
2. Requirements Volatility (Change)
1. Specification Level - Reliability
Rank Aerial Refueling Control Computer (ARCC) CSCI build 1
Identified Risks
Model
Metrics collection Periodic model-based performance prediction, risk assessment, and recommended corrections
Model-based estimate & plan and project validation Integrated SW risk mitigation plan, measure and control
A close-loop control, full lifecycle approach, for model-based, software project management
Repeat or
Re-plan
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2/1
2/2
01
2
3/1
2/2
01
2
4/1
2/2
01
2
5/1
2/2
01
2
6/1
2/2
01
2
7/1
2/2
01
2
8/1
2/2
01
2
9/1
2/2
01
2
10
/12
/20
12
11
/12
/20
12
12
/12
/20
12
1/1
2/2
01
3
2/1
2/2
01
3
3/1
2/2
01
3
4/1
2/2
01
3
5/1
2/2
01
3
6/1
2/2
01
3
7/1
2/2
01
3
8/1
2/2
01
3
9/1
2/2
01
3
10
/12
/20
13
11
/12
/20
13
12
/12
/20
13
1/1
2/2
01
4
2/1
2/2
01
4
3/1
2/2
01
4
4/1
2/2
01
4
5/1
2/2
01
4
6/1
2/2
01
4
7/1
2/2
01
4
8/1
2/2
01
4
SWRR SRR PDR CDR C&UT CI&TFQT SI&T
SW Reqts (SRS)
SW Test (STP, STD, Scripts, STR)
SW Planning
SW CM
SW Code & UT (EKSLOC)
SW Design (SDD)
Track Progress by WP** Complete and
BTU* Charging
Cost Volume/Basis of Estimate (BOE)
6
Process
*BTU - Basic Task Unit, standard task based cost accounting structure **WP – Work Product Copyright © 2013 Boeing. All rights reserved.
Advanced SW Estimating Toolset
(ASET)
• Master Input/Parse • Calibration Tool • CER Catalog • MS Project Templates
Traditional Inputs • Size • Environmental
Characteristics* • Historical Data**
Advanced Inputs
• Feature Roadmap (agile)
• Team structure
Inputs
Traditional Outputs • Effort (hours) • Schedule (calendar months)
Advanced Outputs
• Staffing Profiles • Work Product (Feature)
Plans • MS Project Plans • EAC Projections • Defect Predictions
Outputs
* Environmental Characteristics include personnel, development / target / integration environments, and program constraints ** Historical Data includes size, effort, schedule, and productivity and Cost Estimating Relationships (CERs)
Improvements
7
Before Current
Copyright © 2013 Boeing. All rights reserved.
INPUT Iterate
INPUT As needed
INPUT Periodic
2
PSRx Master SLOC Input Sheet
SEER-SEM & PPMC
PSRx
Define Project Model - Type - Parameters - WBS (CSCIs, CSCs …) - Sizes (builds)
Project Estimate(s) - Cost - Schedule - Defects
Proposal Execution
Plan & Measure Project - Metrics / Measurements - Planned & Actuals
Project Health Assessment
Reports
Analyze “What-if’s” to assess any
Corrective Actions
Adjust Model Parameters … Rebaseline(s)
Project Estimate - Cost - Schedule ( - Defects )
Tools
Main Activities
Reports etc
Add’l Tools
SEER-SEM & PPMC
Analyze “What-if’s”
to assess Estimate
Adjust Model Parameters
Tool Master SLOC Input Sheet
Model
Tool
Define Project Model - Type - Parameters - WBS (CSCIs, CSCs …) - Sizes (builds)
Project Estimate(s) - Cost - Schedule - Defects
Proposal Execution
Plan & Measure Project - Metrics / Measurements - Planned & Actuals
Project Health Assessment
Reports
Analyze “What-if’s” to assess any
Corrective Actions
Adjust Model Parameters … Rebaseline(s)
Project Estimate - Cost - Schedule ( - Defects )
Tools
Main Activities
Reports etc
Add’l Tools
Model
Analyze “What-if’s”
to assess Estimate
Adjust Model Parameters
INPUT Iterate
INPUT As needed
INPUT Periodic
1
Process Cycle
8 Copyright © 2013 Boeing. All rights reserved.
Capabilities and Products Actual & Predicted Cost/Schedule
versus Baseline Estimate
SPI & CPI (recent & cum)
Summary Stoplight
Staffing Profiles - Default & Constrained
Technical Metrics/Analysis Including Productivity by Work-Product and Phase
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
3/1
1/2
01
1
4/1
1/2
01
1
5/1
1/2
01
1
6/1
1/2
01
1
7/1
1/2
01
1
8/1
1/2
01
1
9/1
1/2
01
1
10
/11
/20
11
11
/11
/20
11
12
/11
/20
11
1/1
1/2
01
2
2/1
1/2
01
2
3/1
1/2
01
2
4/1
1/2
01
2
5/1
1/2
01
2
6/1
1/2
01
2
7/1
1/2
01
2
8/1
1/2
01
2
9/1
1/2
01
2
10
/11
/20
12
11
/11
/20
12
12
/11
/20
12
1/1
1/2
01
3
2/1
1/2
01
3
3/1
1/2
01
3
4/1
1/2
01
3
5/1
1/2
01
3
6/1
1/2
01
3
7/1
1/2
01
3
8/1
1/2
01
3
9/1
1/2
01
3
10
/11
/20
13
11
/11
/20
13
12
/11
/20
13
1/1
1/2
01
4
2/1
1/2
01
4
3/1
1/2
01
4
4/1
1/2
01
4
5/1
1/2
01
4
6/1
1/2
01
4
7/1
1/2
01
4
8/1
1/2
01
4
9/1
1/2
01
4
SRR SWRR PDR CDR C&UT CI&T FQT S IT&E
Work- Product Execution Profiles
-50% 0% 50% 100%
10. Memory Constraints
9. Tar Sys Exper/Compl
8. Development System Volatility
7. Real Time Code
6. Special Display Requirements
5. Quality Assurance Level
4. Test Level
3. Target System Volatility
2. Requirements Volatility (Change)
1. Specification Level - Reliability
Rank Aerial Refueling Control Computer (ARCC) CSCI build 1
A constrained model to track actuals against plan to assess impact of not meeting plan. An unconstrained model for independent analysis and what if trades
Independent Assessments
Program Driven Assessment
Periodic Actuals %complete hours etc
Program Model
Baseline Plan
Program Model Adjusted Prediction Assessment
Recalibrated Model
Relaxed Constraints
Model
Revised Model
Variations between Shadow and Nominal
Models
etc etc Models
Adjusted Parameters
Model
Replica Recommendations etc
Program Model Setup Reflects Program Characteristics
Off-Plan Trigger
Shadow Model
Nominal Model
Model Settings parameters calibration etc
Imposed Constraints milestones staffing-profile etc
Model
Risk Identification and Analysis
Proposal Support, What-if Scenario &
Independent Assessment
0
5
10
15
20
25
Mar
ch 1
, 20
11
Ap
ril 1
, 20
11
May
1, 2
01
1
Jun
e 1
, 20
11
July
1, 2
01
1
Au
gust
1, 2
01
1
Sep
tem
be
r 1
, 20
11
Oct
ob
er
1, 2
01
1
No
vem
be
r 1
, 20
11
De
cem
be
r 1
, 20
11
Jan
uar
y 1
, 20
12
Feb
ruar
y 1
, 20
12
Mar
ch 1
, 20
12
Ap
ril 1
, 20
12
May
1, 2
01
2
Jun
e 1
, 20
12
July
1, 2
01
2
Au
gust
1, 2
01
2
Sep
tem
be
r 1
, 20
12
Oct
ob
er
1, 2
01
2
No
vem
be
r 1
, 20
12
De
cem
be
r 1
, 20
12
Jan
uar
y 1
, 20
13
Feb
ruar
y 1
, 20
13
Mar
ch 1
, 20
13
Ap
ril 1
, 20
13
May
1, 2
01
3
Jun
e 1
, 20
13
July
1, 2
01
3
Au
gust
1, 2
01
3
Sep
tem
be
r 1
, 20
13
Oct
ob
er
1, 2
01
3
No
vem
be
r 1
, 20
13
De
cem
be
r 1
, 20
13
Jan
uar
y 1
, 20
14
Feb
ruar
y 1
, 20
14
Mar
ch 1
, 20
14
Ap
ril 1
, 20
14
May
1, 2
01
4
Jun
e 1
, 20
14
July
1, 2
01
4
Au
gust
1, 2
01
4
Sep
tem
be
r 1
, 20
14
BTU 26c SW Tester
BTU 26b SW Coder
BTU 26a SW Designer
BTU 25a SW Req
BTU 25b SW Planner/ManagementBTU 24 SW CM
Staffing ProfileDetailed Plans
Actual Defects versus Baseline Estimate
1 2 4
8
6
5 7
3
9
Copyright © 2013 Boeing. All rights reserved.
Sample Product
10 10
0
5
10
15
20
25
Mar
ch 1
, 20
11
Ap
ril 1
, 20
11
May
1, 2
01
1
Jun
e 1
, 20
11
July
1, 2
01
1
Au
gust
1, 2
01
1
Sep
tem
be
r 1
, 20
11
Oct
ob
er
1, 2
01
1
No
vem
be
r 1
, 20
11
De
cem
be
r 1
, 20
11
Jan
uar
y 1
, 20
12
Feb
ruar
y 1
, 20
12
Mar
ch 1
, 20
12
Ap
ril 1
, 20
12
May
1, 2
01
2
Jun
e 1
, 20
12
July
1, 2
01
2
Au
gust
1, 2
01
2
Sep
tem
be
r 1
, 20
12
Oct
ob
er
1, 2
01
2
No
vem
be
r 1
, 20
12
De
cem
be
r 1
, 20
12
Jan
uar
y 1
, 20
13
Feb
ruar
y 1
, 20
13
Mar
ch 1
, 20
13
Ap
ril 1
, 20
13
May
1, 2
01
3
Jun
e 1
, 20
13
July
1, 2
01
3
Au
gust
1, 2
01
3
Sep
tem
be
r 1
, 20
13
Oct
ob
er
1, 2
01
3
No
vem
be
r 1
, 20
13
De
cem
be
r 1
, 20
13
Jan
uar
y 1
, 20
14
Feb
ruar
y 1
, 20
14
Mar
ch 1
, 20
14
Ap
ril 1
, 20
14
May
1, 2
01
4
Jun
e 1
, 20
14
July
1, 2
01
4
Au
gust
1, 2
01
4
Sep
tem
be
r 1
, 20
14
BTU 26c SW Tester
BTU 26b SW Coder
BTU 26a SW Designer
BTU 25a SW Req
BTU 25b SW Planner/ManagementBTU 24 SW CM
Staffing Profile
Copyright © 2013 Boeing. All rights reserved.
BOEING PROPRIETARY
0
19
38
57
76
95
114
9-12 1-13 5-13 9-13 1-14 5-14 9-14 1-15 5-15 9-15
Months
% of Baseline Plan
Baseline EAC: 61155 Hours; 39.65 Months
Earned ValueActual SpentBaseline Plan
Forecast End Date: 1/12/2016Forecasting Method: Cum Performance
Time Now: 3/01/2014
Baseline Plan
Large Program - ADHRAs of 3/01/2014 -
Time Now
11
Execution Phase – Prediction Example
Case: Slow progress • Observations
• Reduced performance (green solid) • $ burn rate continues per plan (red
solid) • Projections (EAC)
• Late to 100% done (green dotted) • Over budget (red dotted)
Projected
EAC
Based on actual performance data the model predicts cost overrun
and late delivery
‘What if’ analysis can help determine path back to within budget Staff level
Personnel Characteristics
Development Environment
Target Environment
Copyright © 2013 Boeing. All rights reserved.
Recommended Practice/ Lessons Learned -1
12
• Process used in proposal phase creating
representative model of project and plans
applicable throughout lifecycle
• Lesson: retrofitting during execution is
cumbersome and costly
• Metrics Plans – Ensure proper information is
available
• Lesson: metric plans may not support
desired performance management
Copyright © 2013 Boeing. All rights reserved.
Recommended Practice/ Lessons Learned -2
13
• Process should analyze metrics and cost data
ensuring:
• Cost charging alignment to WBS and use of
BTUs are correct and support EVM analysis
• Metrics are as planned and provide accurate
assessment of program progress
• Accurate and efficient historical data
collection
• Lesson: actual metric plan and cost
charging vary from planned approach and
may not provide needed information Copyright © 2013 Boeing. All rights reserved.
Recommended Practice/ Lessons Learned -3
14
Process automation improves time and effort
to produce plans, collect/analyze EVM, and
prepare proposals
Core SMEs support estimating and proposal
development and mentoring of project
personnel
Lesson: planning and monitoring is a time
consuming activity for management
resources, especially when doing re-plans
during execution
Copyright © 2013 Boeing. All rights reserved.
15
Summary
Demonstrated Program Executability Improvements
• Integrated model-based project estimating, planning,
predicting, tracking and management
• Carries value of proposal BOE through program
execution
• Capability applied to software projects, expandable to
SE
• Best if used in proposal phase and continued during
contract execution
• Value proposition with metrics collection and process
automation
Copyright © 2013 Boeing. All rights reserved.
Backup Information
Abstract
Quantitative Prediction and Improvement of Program Execution – A New Paradigm By: Dr. Shawn Rahmani Boeing Defense, Space/Security Huntington Beach, California Flawless execution of programs is crucial for the US Government and its contractors. This presentation focuses on a “predictive methodology” that uses program data to provide leading indicators about the health of the program execution throughout the development cycle. While the methodology can be applied to an entire program, the initial focus has been on the “software” portion of a program (developed internally or by a supplier). Software implements complex system functionalities such as network operations and communications, system automation, control and autonomy, information assurance, data manipulation and management, and user interface capabilities. A number of DOD programs have experienced software related problems, including inaccurate and incorrect estimation of the software effort, lack of adequate and quantifiable definition of the planned work, and insufficient visibility and inaccurate measurement of the work performed. While methods based on Earned Value Measurement (EVM) have been in place for some time, they have proved to be insufficient for addressing the above problem. Specifically, the following contribute to the problem: Lack of integrated leading indicators (based on the product size, cost, schedule, quality, and staffing requirements) to predict technical success of the software activities within a program. Lack of value-added measures that provide predictive program insight while supporting lean improvements. Lack of ability to effectively package and communicate the predicted program impact and recommended solutions to decision makers across the life cycle. This presentation is based on the work done at Boeing to address the above issues. It covers: A set of predictive metrics for software engineering, consistent with Government’s predictive metrics for Probability of Program Success (PoPS) and Predictive System/Software Measurements (PSM). A set of raw software measurements that can be used for estimation and measurement of detailed work planned and performed on government contracts. A technical process that defines how software estimation, measurement, prediction, and control should be handled. Extensibility of current metrics (e.g., EVM, quality, risk). Information on how the process contributes to First Time Quality and Lean Program Management/Execution. Information about the tool that automates the collection of the data and implementation of the process. Use of the process and tool on trial projects and pilot programs Lessons learned and future plan.
17 Copyright © 2013 Boeing. All rights reserved.
ASSIGNMENT OF COPYRIGHT Date of Presentation(s): October 14, 2013 Copyright Form To: National Defense Industrial Association (NDIA) 16th Annual Systems Engineering Conference Publication Title: Quantitative Prediction and Improvement of Program Execution – A New Paradigm Author(s): Shawn Rahmani At least one, but possibly not all, of the authors of the Work is an employee of The Boeing Company or one of its wholly-owned subsidiaries (“BOEING”) preparing the Work within the scope of his or her employment. The Boeing employees in question are not authorized to assign or license Boeing’s rights therein, or otherwise bind Boeing. However, subject to the limitations set forth below, Boeing is willing to assign its copyright in the Work to Publisher. Notwithstanding any assignment or transfer to the Publisher, or any other terms of this agreement, the rights granted by Boeing to Publisher are limited as follows: (i) any rights granted by Boeing to the Publisher are limited to the work-made-for-hire rights Boeing enjoys in the Work; (ii) Boeing makes no representation or warranty of any kind to the Publisher or any other person or entity regarding the Work, the information contained therein, or any related copyright; and (iii) Boeing retains a non-exclusive, perpetual, worldwide, royalty-free right, without restriction or limitation, to use, reproduce, publicly distribute, display, and perform and make derivative works from the Work, and to permit others to do so. Candice Jordan Contracts Specialist Intellectual Property Management