Process Synchronization and Stabilization

© Copyright 2007. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.1

Process Synchronization and Stabilization

February 2007

Rick SelbyHead of Software ProductsNorthrop Grumman Space [email protected], 310-813-5570

Adjunct Professor of Computer ScienceUniversity of Southern [email protected]


“Process Owners” Define, Monitor, and Improve Products & Processes Using Metric-Driven Analyses What do managers control at different levels of an

organization?

Executives – Define, monitor, and improve: Vision, values, high-level policies, financial, etc.

“Process owners” – Define, monitor, and improve: Processes using metric-driven analyses to improve

products and services

Project managers (or “product owners”) – Define, monitor, and improve:

Products and services


Six Sigma Projects Decrease Process or Product Defects by Reducing Variances and Shifting Means

Spread of variationtoo wide compared

to specifications

Spread of variationtoo wide compared

to specifications

Spread of variationnarrow compared to

specifications

Spread of variationnarrow compared to

specifications

ImprovementImprovement

Delivery Time

Too Early Too Late

Delivery Time

Too Early Too Late

DefectsDefects

ImprovementImprovement

Delivery Time

Too Early Too Late

Delivery Time

Too Early Too Late

DefectsDefects

Six Sigma projects typically focus on decreasing process or product non-conformances (“defects”) through reducing variances and shifting means of process performance or product quality metrics

Projects decrease the variability of process performance or product quality metrics to improve predictability (such as smaller gaps in plans vs. actuals), efficiency (such as shorter cycletimes), and effectiveness (such as fewer defects)

Six Sigma gets it name because it states a goal of achieving no worse than 3.4 defects per million opportunities, which is six process sigma (roughly analogous to standard deviations) from the mean (99.9997% accuracy)

Commonly use DMAIC approach: Define, Measure, Analyze, Improve, and Control


Control Charts Identify Behavior Outside of Expected Control Limit Boundaries

Exception processvariation

LowerControl

Limit

UpperControl

Limit

ProcessAverage

Exception processvariation

Actual PerformanceData from Project XYZ

Project XYZ(Defect Density for Process X)


Control Limits Define “Voice of the Process” and Specification Limits Define “Voice of the Customer”

UCL

LCL

USL

LSL

Control limits represent “voice of the process”: Upper control limit (UCL) and lower control limit (LCL) Derived statistically using process or product data Help determine whether the process is stable

Specification limits represent “voice of the customer”: Upper specification limit (USL) and lower specification limit (LSL) Represent goals, requirements, or targets Help determine whether the process is capable


Data-Driven Statistical Analyses Identify Trends, Outliers, and Process Improvements for Defects

• Six Sigma Project Introduced New Peer Review Process• Provided Training on Process

• New Web-Based Peer Review Tool • Provided Training on Tool

These defects are action items resulting from peer reviews of software code and unit testing plans and results.

Control chart of metric data from example Six Sigma projects focusing on fault (or defect) density in peer reviews of software components

Data from 10 systems


Data-Driven Statistical Analyses Identify Trends, Outliers, and Process Improvements for Cycletimes

5004003002001000

300

200

100

0

-100

-200

Day

s P

ast D

ue

dateopen

Closure PerformanceI Chart for Action Item

1

1

1

1

111

1

111111111

111

1

1

11

1

1

1

1

1111111111111111111111111 11

111

11

1111111111111

Mean=-19.71UCL=1.631

LCL=-41.05

after April 1, 2004actions opened

Spec Limit

• Web-based tracking tool deployed• Action tracking process started

Specification limit defines goal for action item cycletime closure.

All actions are from SW Requirements Reviews.

Chart plots deltas between action item due date and closure date.

Positive values indicate actions closed after their due date.

Negative values indicate actions closed before their due date.

Two-sample t-test confirms that data after April 1, 2004 is statistically different than data preceding this date (α < 0.05)

Control chart of metric data from example Six Sigma projects focusing on action item closure cycletime in peer reviews of software components

Data from one system


Data-Driven Statistical Analyses Identify Trends, Outliers, and Process Improvements for Cycletimes

Control chart of metric data from example Six Sigma projects focusing on change request closure cycletime for software components

Data from one system

• Series of process improvements instituted and new control limits calculated

Statistically significant improvement in process performance


Synchronize-and-Stabilize Timeline and Milestones: 12-36 Months from “Milestone 0” to Manufacturing

Documents and Intermediate Activities

Milestone I release

Milestone II release

Project plan approval

Milestone 0

Schedule complete

Milestone III release

Code complete

Zero bug release

Release to manufacturing

Visual freeze

Plan

ning

3-12

mon

ths

Stab

iliza

tion

3-8

mon

ths

Feature complete

Vision statement

Specification document

Prototypes

Testing strategySchedule

Project review

Implementation plan

Specification review

Postmortem document

Internal testing

Beta testingBuffer time

Buffer time

Major Reviews

MilestonesTimeline

Subp

roje

ct II

ISubp

roje

ct II

Subp

roje

ct I

Design feasibility studies

OptimizationsTesting and debuggingOptimizationsTesting and debugging

(Ship date)

6-12

mon

ths

Dev

elop

men

t• Integration • Testing and debugging

2-5 weeks

• Buffer time2-5 weeks

6-10 weeks • Code and optimizations • Testing and debugging • Feature stabilization

Development Subproject 2-4 months

(1/3 of all features)

Phases

Dev

elop

men

t

6-16

mon

ths


Incremental Software Builds Deliver Early Capabilities and Accelerate Integration and TestFigure 4.3-4. JIMO Incremental Software Builds

We provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05

Data Server Unit (DSU) Builds

Science Computer Unit (SCU) Builds

Flight Computer Unit (FCU) Builds

Note: Science Computer builds for common software only (no instrument software included)

FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13

Prelim Exec and C&DH Software


FCU2

FCU3

FCU4

FCU5

FCU6

FCU7

Final Exec and C&DH Software

Science Computer Interface

Reactor Controller Interface

AACS (includes autonomous navigation)

Thermal and Power Control

Configuration and Fault Protection

SCU1

Final Exec and C&DH SoftwareSCU2

DSU1

DSU3


DSU2 Final Exec and C&DH Software

Data Server Unique Software

Ground Analysis Software (GAS) Computer BuildsGAS1Preliminary Ground Analysis Software

GAS2Final Ground Analysis Software

A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154

JPL/NGC, Prelim. Hardware/Software IntegrationJPL/NGC, Final Hardware/Software IntegrationJPL, Mission Module Integration

JPL, Prelim. Hardware/Software IntegrationJPL, Final Hardware/Software Integration

NR, Reactor Controller IntegrationNGC, AACS Validation on SMTBNGC, TCS/EPS Validation on SSTBNGC, Fault Protection S/WValidation on SSTB

NGC, Prelim. Hardware/Software IntegrationNGC, Final Hardware/Software IntegrationNGC, HCR Integration on SMTBJPL, Prelim. Integration into Ground System

JPL, Final Integration into Ground System

1 Requirements2 Preliminary Design3 Detailed Design4 Code and Unit Test/Software

Integration5 Verification and Validation

Legend: N is defined as follows:541 32=

2 53 41=

542 31=

PrototypeActivity

NGC

N Performer of Activity NJPL

Role/activity shared by JPL and NGC

Design Agent

P

Figure 4.3-4. JIMO Incremental Software BuildsWe provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1


DSU1

DSU3






A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154









2 53 41=

542 31=

PrototypeActivity

NGC



Design Agent

P

We provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1


DSU1

DSU3






A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154









2 53 41=

542 31=

PrototypeActivity

NGC



Design Agent

P

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1

Final Exec and C&DH SoftwareSCU2 Final Exec and C&DH SoftwareSCU2

DSU1

DSU3


DSU2 Final Exec and C&DH SoftwareDSU2 Final Exec and C&DH Software




A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154









2 53 41=

542 31=

PrototypeActivity

NGC



Design Agent

P

PowerPower


Analyses of Software Defect Injection and Detection Phases Reveals Distributions and Gaps

Software Defect Injection and Detection Phases

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

System Development Phase

Defe

cts

(cum

ul. %

)Defects Injected (cumul. %) Defects Detected (cumul. %)

Cumulative distribution of software defect injection and detection phases based on using peer reviews across 12 system development phases

3418 defects, 731 peer reviews, 14 systems, 2.67 years


Analyses of Software Requirements Shows Leading Indicators for Implementation Scope

Data from 14 NASA systems Ratio of implementation size to software requirements has 81:1 average and 35:1

median; Excluding system #14, the ratio has 46:1 average and 33:1 median Ratio of software requirements to system requirements has 6:1 average

Ratio of Implementation Size to Requirements

0

50100150200250300

350400450500550

1 2 3 4 5 6 7 8 9 10 11 12 13 14

System

Sour

ce-L

ines

-of-C

ode

/

Req

uire

men

tsSize / Requirements Average (ex. #14) Average + 2 std. (ex. #14)


Five-Phase Process Defines Structured Approach for Six Sigma Improvement Projects

Phase Exit Criteria Duration

Define Project scope defined, business case established, and stakeholders engaged

5 weeks

Measure Metrics defined, data collected, and data validated 3 weeksAnalyze Processes and products analyzed, root cause analyses

completed, and sources of variation understood 3 weeks

Improve Potential solutions identified, recommended solution piloted, and improvements documented using data

5 weeks

Control Ongoing monitoring using statistical methods such as control charts, supported by special cause and common cause analysis of violations of control limits and specification limits

Ongoing

Total 16 weeks + Control Phase

Six Sigma implementation approach can span 16 weeks and include tollgate reviews for define, measure, analyze, improve, and control phases

Tollgate reviews provide checkpoints for progress, evaluation, and feedback

1Define1

Define

2Measure

2Measure

3Analyze

3Analyze

4Improve

4Improve

5Control

5Control

1Define1

Define

2Measure

2Measure

3Analyze

3Analyze

4Improve

4Improve

5Control

5Control


Synergistic Strategies Help Enable Large-Scale Software System Development and Management

Modeling

Evaluations & Feedback

Analysis

Synthesis

Infrastructure & Techniques

Processes & Architectures

Analysis Capabilities

Requirements & Opportunities

Models, Relationships & Feedback

Research focus: Large-scale, mission-critical embedded software systems Research themes: Early lifecycle, system perspective, frequent design cycles,

multi-artifact integration, scalable modelware

Flexible lifecycle process models, extensible system architectures, and pro-active development guidance mechanisms

Infrastructure and techniques for system modeling, analysis, and simulation

System modeling, evaluation, tradeoff, and prediction using simulations and empirical studies