Systems and Software Design Principles for Large-Scale ... · Example large-scale mission-critical embedded software system ... with 3-15 Builds per Program ... Embedded software

© Copyright 2001-2008. Richard W. Selby and Northrop Grumman Corporation. All rights reserved.0

October 2008

Rick SelbyHead of Software ProductsNorthrop Grumman Space Technology310-813-5570, [email protected]

Adjunct Professor of Computer ScienceUniversity of Southern California

Systems and Software Design Principles for Large-Scale Mission-Critical Embedded

Products from Aerospace and Financial Problem Domains


Research Investigates Systems and Software Synthesis, Analysis, and Modeling Principles

Overview

Systems and software engineering strategies, principles, benefits, and tradeoffs

Example large-scale mission-critical embedded software system

Investigations of synthesis, analysis, and modeling principles

Synthesis: Lifecycle models

Synthesis: System architectures

Analysis: Reuse analysis

Analysis: Structure analysis

Modeling: Defect detection techniques

Modeling: Measurement and prediction

Conclusions and future work


Research Investigates Systems and Software Engineering Principles, Benefits, and Tradeoffs

Lifecycle models: Frequent synchronized design cycles and system releases

System architectures: Layered system architectures containing embedded meta-language programs and interpreters

Reuse analysis: Reconfigurable component-driven development

Structure analysis: Inter-component connectivity analysis

Defect detection techniques: Disciplined team-based peer reviews

Measurement and prediction: Automated measurement-driven analysis infrastructure using predictive models

SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G

Organization of and parallelization within large-scale projects

Rapid feedback and innovation

Visibility into stabilization and handoffs

User-customizability

Multi-platform portability

Automated testing

Sustainable multi-project reuse

Lower component defect rates

Lower component development effort


Lower component defect correction effort


Early lifecycle defect detection

Low out-of-phase defect rates

High return-on-investment for prevention

Early identification of high defect or high effort components

Statistical process control

Pro-active process guidance

Enables?

PRINCIPLES BENEFITS and TRADEOFFS

Enables?

Enables?

Enables?

Enables?

Enables?



Overview












Embedded software for

Advanced robotic spacecraft platforms

High-bandwidth satellite payloads

High-power laser systems

Emphasis on both system management and payload software

Reusable, reconfigurable software architectures and components

Languages: O-O to C to asm

CMMI Level 5 for Software in February 2004; ISO/AS9100; Six Sigma

High-reliability, long-life, real-time embedded software systems

Organizational Charter Focuses on Embedded Software Products

Prometheus / JIMO

Software Peer ReviewSoftware Development Lab

Restricted

JWSTNPOESS EOS Aqua/Aura Chandra

Airborne Laser

Software Analysis

Software Process Flow for Each Build, with 3-15 Builds per Program

GeoLITE AEHF MTHEL


Prometheus Spacecraft Supports Jupiter JIMO Mission over 9 to 14 Year Duration


Spacecraft Docking Adapter

Aerothermal Protection

Power Module

Heat Rejection

Electric

Propulsion

Spacecraft

Bus and

Processors

Prometheus Spacecraft for JIMO and Related Missions Enables Data-Intensive Science Spacecraft configuration PB1

58m length

36,375kg launch mass

5 processors, excluding redundancy

250mbps transfer, 500gbit storage, 10mbps downlink

Gas cooled power with 200kW Brayton output

Stows in 5m diameter fairing

Stowed Spacecraft


Embedded softwareimplements functions for commands & telemetry, subsystem algorithms, instrument support, data management, and fault protection

Size of on-board software growing to accelerate data processing and increase science yield

Software “adds value” to mission by enabling post-delivery changes to expand capabilities and overcome hardware failures

Architecture Defines 5 Processors: Flight, Science, Data, Power Generation, and Power Distribution

Using an EOS-proven approach, our architecture provides high bandwidth for science data by using a separate server to offload throughput-intensive processing and communication from the flight computer and science computer.

Figure 2.4.1-5. JIMO Software Architecture

04S01176-2-111h_154

A. ARCHITECTURE

Command and Data Handling (C&DH) subsystem

ReactorInstrumentand Control

MissionModule

ReactorModule

Spacecraft Module

Instrument Control

Status &Sensor

Data

Control

Status/Commands

Prioritized/Merged Data

Ground CMDS,Loads

CommandsEng. DataCommandsEng. Data

Engineering andScience Data

Status

Flight ComputerUnit (FCU)

FCU Software

JPL/NGC

Science ComputerUnit (SCU)

SCU SoftwareJPL/NGC (including instrument software JPL-GFP)

ReactorControllerSoftware(NR-GFP)

Spacecraft Subsystems

Embedded Software

Computer/Processor

Other Interfacing Hardware

PCS/PCAD

PCAD Elec.

PCADControllerSoftware(HS-Sub.)

Commands

Science and Engineering Data

Requested Data/Status

HighCapacityRecorder

(HCR)

RCS

TCS

COMM

AACS

HRS

EPS

Data ServerUnit (DSU)

DSU Software

JPL/NGC

Instruments

Instrument Status, and Science Data

Requested Data

Enhances ability to accommodate new requirements, accommodate in-flight contingencies and enhancements

Wide margins for size and throughput, asynchronous tasking for non-real-time functions, large margins for data busses

Allowance for growth

Reduces development and test risk, reduces spacecraft integration complexity, allows timely and robust response to dynamic development and operational contingencies

Table driven designs, architectural simplicity, deterministic behavior, common software across processors, common interfaces, designed-in accommodation of in-flight changes

Software Maintenance

Standard interfaces ease integration and maximize reuse

Use of 1553 and 1394 bussesStandard BusInterfaces

Offloads science and flight computers and isolates science from spacecraft health and safety preventing SCU software faults from affecting the health and safety of the spacecraft

Dedicated data server with high speed link to High Capacity Recorder (HCR)

Science Data Management

Enhances spacecraft safety, distributes size and throughput, provides manageable work allocations to team members

Separate processors for reactor control, high-speed power conversion, spacecraft control, science control and analysis, mission data management

Partitioned Functionality

Enhances spacecraft safety, allows timely independent critical safing actions, provides centralized orchestration of normal and contingency configurations and operations and isolates faults

System level autonomy and fault protection centralized, lower level autonomy distributed to appropriate control point

Autonomous Operations and Fault Protection

BenefitsDesign ApproachSoftware Architecture

Drivers

Enhances ability to accommodate new requirements, accommodate in-flight contingencies and enhancements

Wide margins for size and throughput, asynchronous tasking for non-real-time functions, large margins for data busses

Allowance for growth

Reduces development and test risk, reduces spacecraft integration complexity, allows timely and robust response to dynamic development and operational contingencies

Table driven designs, architectural simplicity, deterministic behavior, common software across processors, common interfaces, designed-in accommodation of in-flight changes

Software Maintenance

Standard interfaces ease integration and maximize reuse

Use of 1553 and 1394 bussesStandard BusInterfaces

Offloads science and flight computers and isolates science from spacecraft health and safety preventing SCU software faults from affecting the health and safety of the spacecraft

Dedicated data server with high speed link to High Capacity Recorder (HCR)

Science Data Management

Enhances spacecraft safety, distributes size and throughput, provides manageable work allocations to team members

Separate processors for reactor control, high-speed power conversion, spacecraft control, science control and analysis, mission data management

Partitioned Functionality

Enhances spacecraft safety, allows timely independent critical safing actions, provides centralized orchestration of normal and contingency configurations and operations and isolates faults

System level autonomy and fault protection centralized, lower level autonomy distributed to appropriate control point

Autonomous Operations and Fault Protection

BenefitsDesign ApproachSoftware Architecture

Drivers

Ground System

GroundComputers

GroundAnalysisSoftware

(JPL/NGC)

Telemetry

EMS

B. SOFTWARE ARCHITECTURE DRIVERS

1553

High-Speed Serial

1553 and 1394

Status,

Data

Sta

tus/C

om

ma

nd

s

Power

Power

Power



Overview



















SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G

Organization of and parallelization within large-scale projects

Rapid feedback and innovation

Visibility into stabilization and handoffs

Enables?

PRINCIPLES BENEFITS and TRADEOFFS

SCALING DIMENSIONS

Teams1 >1

1

>

1P

roje

cts

X


Incremental Software Builds Deliver Early Capabilities and Accelerate Integration and TestFigure 4.3-4. JIMO Incremental Software Builds

We provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05

Data Server Unit (DSU) Builds

Science Computer Unit (SCU) Builds

Flight Computer Unit (FCU) Builds

Note: Science Computer builds for common software only (no instrument software included)

FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13

Prelim Exec and C&DH Software


FCU2

FCU3

FCU4

FCU5

FCU6

FCU7

Final Exec and C&DH Software

Science Computer Interface

Reactor Controller Interface

AACS (includes autonomous navigation)

Thermal and Power Control

Configuration and Fault Protection

SCU1

Final Exec and C&DH SoftwareSCU2

DSU1

DSU3


DSU2 Final Exec and C&DH Software

Data Server Unique Software

Ground Analysis Software (GAS) Computer Builds

GAS1Preliminary Ground Analysis Software

GAS2Final Ground Analysis Software

A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154

JPL/NGC, Prelim. Hardware/Software Integration

JPL/NGC, Final Hardware/Software Integration

JPL, Mission Module Integration

JPL, Prelim. Hardware/Software Integration

JPL, Final Hardware/Software Integration

NR, Reactor Controller Integration

NGC, AACS Validation on SMTB

NGC, TCS/EPS Validation on SSTB

NGC, Fault Protection S/WValidation on SSTB

NGC, Prelim. Hardware/Software Integration

NGC, Final Hardware/Software Integration

NGC, HCR Integration on SMTB

JPL, Prelim. Integration into Ground System

JPL, Final Integration into Ground System

1 Requirements2 Preliminary Design3 Detailed Design4 Code and Unit Test/Software

Integration5 Verification and Validation

Legend: N is defined as follows:

541 32=

2 53 41=

542 31=

Prototype

ActivityNGC

N Performer of Activity N

JPL

Role/activity shared by JPL and NGC

Design Agent

P

Figure 4.3-4. JIMO Incremental Software BuildsWe provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1


DSU1

DSU3







A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154


















541 32=

2 53 41=

542 31=

Prototype

ActivityNGC


JPL


Design Agent

P

We provide incremental software deliveries support integration and test activities and synchronize with JPL, Hamilton Sundstrand, and Naval Reactors to facilitate teaming, reduce risk, and enhance mission assurance.

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1


DSU1

DSU3







A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154


















541 32=

2 53 41=

542 31=

Prototype

ActivityNGC


JPL


Design Agent

P

Delivered to, Usage

201320122011201020092008200720062005CY 2004

PMSR1/05





FCU1

ATP11/04

SM PDR6/08

SM CDR8/10

BUS I&T8/12

SM AI&T8/13



FCU2

FCU3

FCU4

FCU5

FCU6

FCU7







SCU1

Final Exec and C&DH SoftwareSCU2 Final Exec and C&DH SoftwareSCU2

DSU1

DSU3


DSU2 Final Exec and C&DH SoftwareDSU2 Final Exec and C&DH Software





A B C D

P

P

P

P

P

P

P

P

P

04S01176-4-108f_154


















541 32=

2 53 41=

542 31=

Prototype

ActivityNGC


JPL


Design Agent

P

PowerPower


WBS: 4.0 Spacecraft IPTRisk Owner: McWhorter, Larry Printed: 14 Dec 2005

Risk: CEV-252 - Flight Software Requirements Management

1 : Conduct BM1 2 : Estimate software requirements scope (preliminary)

3 : Establish software control board (preliminary) 4 : Release SDP (with spec tree, change process)

5 : Complete RTOS lab evaluation; Validate capabilities using sim 6 : Estimate software requirements scope (final)

7 : Implement system development process flow models 8 : Spacecraft/etc users define/val/sim use-cases (I/F, funct, off-nom, ...)

9 : Finalize/val/sim IFC1 requirements: Infrastructure SW 10 : Baseline allocation of SW requirements to IFCs with growth/correction

11 : Establish software control board (final) 12 : Conduct SwRR

13 : Finalize/val/sim IFC2 requirements: Inter-module & inter-subsystem I/F 14 : Initial end-to-end architecture model

15 : Finalize/val/sim IFC3 requirements: Subsystems major functions 16 : Finalize/val/sim IFC4 requirements: Nominal operations

17 : Deliver IFC3: Subsystems major functions 18 : Finalize/val/sim IFC5 requirements: Subsystems off-nominal operations

19 : Finalize/val/sim IFC6 requirements: System off-nominal operations 20 : Deliver IFC7: No new capabilities; Only corrections; 1st mission ready

Ris

k S

core

26

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

1

2

3

4 5 6

7

8

9 10

11

12

13

14

15 16

17

18 19 20

2005 2006 2007 2008 2009 2010

SDR

Moderate

High

Low

Projects Define Risk Mitigation “Burn Down” Charts with Specific Tasks and Exit Criteria

Last Updated: 03-October-05

CSRR PDR

Last Updated: 14-Dec-05

Events

CDR

TRL4

TRL5

TRL6

TRL7

WBS: 4.0 Spacecraft IPTRisk Owner: Wood, Doug Printed: 15 Nov 2005

Risk: CEV-252 - Flight Software Requirements Management

Planned Risk Level Planned (Solid=Linked, Hollow =Unlinked) Control Points

Actual Risk Level Completed Completed

Ris

k S

core

26

25

24

23

22

21

20

19

18

17

16

15

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0

Spacecraft/etc users define/val/sim use-cases (I/F, funct, off-nom, ...)

Conduct Sw RR

Finalize/val/sim IFC4 requirements: Nominal operations

Deliver IFC3: Subsystems major functions

2005 2006 2007 2008 2009 2010

Exit/Success Criteria:

1. BM1 complete; customer concurs with approach

2. Software requirements scope estimated (preliminary).

3. Software control board established (preliminary); change control process established.

4. SDP released. Spec tree defined.

5. RTOS lab evaluation completed. Capabilities validated using sim.

6. Software requirements scope estimated (final)

7. System development process flow models implemented.

8. Spacecraft/subsystems/etc. users define use cases (for I/Fs, functions, nominal ops, off-nominal ops, etc.)

completed.

Validated using models/sim.

9. Finalize IFC1 requirements: Infrastructure SW completed. Validated requirements using models/sim.

10. Baseline allocation of SW requirements to IFCs with growth/correction/deficiency completed.

11. Software control board (final) established

12. SwRR conducted. NASA customer agrees with software requirements.

13. Finalize IFC2 requirements: Inter-module & inter-subsystem I/Fs completed. Validated requirements using

models/sim.

14. Initial end-to-end architecture model completed.

15. Finalize IFC3 requirements: Subsystems major functions completed. Validated requirements using

models/sim.

16. Finalize IFC4 requirements: Nominal operations completed. Validated requirements using models/sim.

17. Deliver IFC3: Subsystems major functions completed. Validated capabilities using sim.

18. Finalize IFC5 requirements: Subsystems off-nominal operations completed. Validated requirements using

models/sim.

19. Finalize IFC5 requirements: Subsystems off-nominal operations completed. Validated requirements using

models/sim.

20. Deliver IFC7: No new capabilities; Only system I&T corrections completed. SW complete for 1st mission.









SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G

User-customizability

Multi-platform portability

Automated testing

PRINCIPLES

BENEFITS and TRADEOFFS

Enables?

SCALING DIMENSIONS

Teams1 >1

1

>

1P

roje

cts

X


Common Requirements Enable Software Product Lines and Layered Architectures Across Projects

Legend

Mission-specific

Common across projects

Deg

ree

of

fun

cti

on

ali

ty

80

%

2

0%

Mission-Specific

requirements

Mission-Specific

requirements

Mission-Specific

requirements

Mission-Specific

requirements

Mission-

Specific HW

Processor &

I/O Interfaces

Mission-Specific SW

Table-driven configurations,

commands

Project 1

Common SW

Functions for

Protocols, Fault

management,

Attitude, Power,

Thermal, etc.

Real-time

operating

system

Mission-

Specific HW

Processor &

I/O Interfaces

Project 2

Common SW

Functions for

Protocols, Fault

management,

Attitude, Power,

Thermal, etc.

Real-time

operating

system

Mission-

Specific HW

Processor &

I/O Interfaces

Project 3

Common SW

Functions for

Protocols, Fault

management,

Attitude, Power,

Thermal, etc.

Real-time

operating

system

Mission-

Specific HW

Processor &

I/O Interfaces

Project 4

Common SW

Functions for

Protocols, Fault

management,

Attitude, Power,

Thermal, etc.

Real-time

operating

system

Mission-Specific SW


commands

Mission-Specific SW


commands

Mission-Specific SW


commands


Architecture Uses Simple Task Structure, Deterministic Processing, and Predictable Timeline

Fli

gh

t P

roc

es

so

r (C

TC

)

Minor Cycle

Tasks *

High Rate

Tasks *

Background Task

T

M

H

D

A

T

L

M

M

e

m

L

D

H

C

D

C

M

D

O

B

F

M

S

C

S

M

C

&

D

H

F

M

T

M

H

D

A

T

L

M

M

e

m

L

D

H

C

D

C

M

D

O

B

F

M

S

C

S

M

C

&

D

H

F

M

B

S

T

Background Task

B

S

T

B

S

T

B

S

T

B

S

T

B

S

T

B

S

T

B

S

T

S

U

S

S

U

S

S

U

S

S

U

S

S

U

S

S

U

S

S

U

S

S

U

S

* Not to scale

Three-task structure: 32ms task (high rate), 128ms (minor cycle), and background task

Minor cycle serves as the main workhorse task that executes commands, formats telemetry, and handles fault protection

Minor cycle command processor reads active command sequences and executes individual deterministic commands

>50% margins at system delivery for processor, memory, storage, and bus









SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G

Sustainable multi-project reuse



PRINCIPLES


Enables?

SCALING DIMENSIONS

Teams1 >1

1

>

1P

roje

cts

X


32% of Software Components are Either Reused or Modified from Previous Systems

Data from 25 NASA systems

Component origins: 68.0% new development, 4.6% major revision, 10.3% slight revision, and 17.1% complete reuse without revision

0.00

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

90.00

100.00

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Project index (sorted order)

Perc

en

tag

e o

f m

od

ule

s (

%)

Modules reused with no, slight, or major revision

Modules reused with no or slight revision

Modules reused with no revision


Analyses of Component-Based Software Reuse Shows Favorable Trends for Decreasing Faults

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Module origin

Fau

lts p

er

mo

du

le (

ave.)

Mean 1.28 1.18 0.58 0.02 0.85

Std. dev. 2.88 1.81 1.20 0.17 2.29

New

developmentMajor revision Slight revision Complete reuse All


Overall difference is statistically significant ( < .0001). Number of components (or modules) in each category is: 1629, 205, 300, 820, and 2954, respectively.









SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G


Lower component defect correction effort


PRINCIPLES


Enables?

SCALING DIMENSIONS

Teams1 >1

1

>

1P

roje

cts

X


Analyses of Software Architectures Shows Fault Trends for Component Interactions


5469 components analyzed and categorized by quintiles

Fault-Proneness for Component Interactions

0.000

0.250

0.500

0.750

1.000

1.250

1.500

1.750

Interactions per Component

Fau

lts p

er

Co

mp

on

en

t (a

vera

ge)

Average 0.191 0.280 0.438 0.789 1.524 0.598

Std. Dev. 0.745 0.875 1.358 2.048 3.048 1.809

N 1566 867 1128 920 988 5469

1st Quintile 2nd Quintile 3rd Quintile 4th Quintile 5th Quintile All

Absolute


Analyses of Software Architectures Shows Fault Trends for Component Interaction Relative Factors

Relative Factors for Component Interactions

0.850.92

1.00

1.25

1.44

0.74

0.931.00

1.26 1.28

0.53

0.66

1.001.09

2.11

0.00

0.50

1.00

1.50

2.00

2.50

1st Quintile 2nd Quintile 3rd Quintile 4th Quintile 5th Quintile

Interactions per KSLOC per Component

Dep

en

den

t V

ari

ab

le p

er

KS

LO

C p

er

Co

mp

on

en

t (r

ela

tive f

acto

r)Faults (ave.) FaultCorrectionEffort (ave.) DevelopmentEffort (ave.)


5469 components analyzed and categorized by quintilesAbsolute norm-norm


Analyses of Software Requirements Shows Leading Indicators for Implementation Scope and Priorities


Ratio of implementation size to software requirements has 81:1 average and 35:1 median; Excluding system #14, the ratio has 46:1 average and 33:1 median

Ratio of software requirements to system requirements has 6:1 average

Ratio of Implementation Size to Requirements

0

50

100

150

200

250

300

350

400

450

500

550

1 2 3 4 5 6 7 8 9 10 11 12 13 14

System

So

urc

e-L

ine

s-o

f-C

od

e /

Re

qu

ire

men

tsSize / Requirements Average (ex. #14) Average + 2 std. (ex. #14)









SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G

Early lifecycle defect detection

Low out-of-phase defect rates

High return-on-investment for prevention

PRINCIPLES


Enables?

SCALING DIMENSIONS

Teams1 >1

1

>

1P

roje

cts

X


Software Defect Injection Phase

0.0%2.3%

49.1%

9.0%12.1% 11.7%

2.5% 1.9%

10.6%

0.7% 0.1% 0.0%0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

Propos

al

Extern

al R

eq. Sourc

e

Require

ments

Prelim

inary

Desi

gn

Deta

iled D

esig

n

Code

Unit

Test

SW In

tegra

tion T

est

SW V

erificatio

n

Support to

I&T

Main

tenance

Opera

tions

System Development Phase

De

fec

ts (

%)

Analyses of Software Defect Injection Phases Reveals Distributions

Distribution of software defect injection phases based on using peer reviews across 12 system development phases

3418 defects, 731 peer reviews, 14 systems, 2.67 years

49% of defects injected during requirements phase


Analyses of Software Defect Injection and Detection Phases Reveal Distributions and Gaps

Software Defect Injection and Detection Phases

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

Propo

sal

Exter

nal R

eq. S

ourc

e

Req

uire

men

ts

Prelim

inar

y Des

ign

Det

aile

d D

esig

n

Cod

e

Uni

t Tes

t

SW In

tegr

atio

n Te

st

SW V

erifica

tion

Suppo

rt to

I&T

Mai

nten

ance

Ope

ratio

ns

System Development Phase

De

fec

ts (

cu

mu

l. %

)

Defects Injected (cumul. %) Defects Detected (cumul. %)

Cumulative distribution of software defect injection and detection phases based on using peer reviews across 12 system development phases

3418 defects, 731 peer reviews, 14 systems, 2.67 years

50% defects injected by requirements, 70% by detailed design; Gap shows leakage


Web-Based Workflow Tools and Infrastructure Support Software Process Flow

Peer

reviews

Process

flow

Action

items

Exit

checklist









SY

NT

HE

SIS

AN

ALY

SIS

MO

DE

LIN

G

Early identification of high defect or high effort components

Statistical process control

Pro-active process guidance

PRINCIPLES


Enables?

SCALING DIMENSIONS

Teams1 >1

1

>

1P

roje

cts

X


Data-Driven Statistical Analyses Identify Trends, Outliers, and Process Improvements for Defects

• Six Sigma Project Introduced

New Peer Review Process

• Provided Training on Process

• New Web-Based Peer Review

Tool

• Provided Training on Tool

These defects are action items

resulting from peer reviews of

software code and unit testing

plans and results.

Control chart of metric data from example Six Sigma projects focusing on fault (or defect) density in peer reviews of software components

Process improvements decreased variances and decreased means

Data from 10 systems

27


Focus on high-payoff areas: the 80:20 rule

Generate decision trees or networks automatically

Scalable to large systems

Leverage previous experience and calibrate to new environments

Integrate measurements from processes, products, projects, teams, and organizations

Measurement-Driven Decision Models (Trees, Networks) Predict High-Risk Software Components

Example:Metric-A

Metric-DMetric-CMetric-B_

__+

0-3

+_

>120-12

4-5 6-10 >10

Metric-E

_+

>1500-150

High Real-time Non-real

timeLow or

medium

“+” : Classified as likely to have property P (e.g., high integration faults)

“-” : Classified as unlikely to have property P


Predictive Models Identify Leading Indicators of High-Fault and High-Effort Components

Model Accuracies and Tradeoffs

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60 70 80 90 100

Consistency (%)

Co

mp

lete

ness (

%)

All

Trees

Networks

Faults

Effort

Design

Code&Test

Faults:Trees

Faults:Networks

Effort:Trees

Effort:Networks

Design:Trees

Design:Networks

Code&Test:Trees

Code&Test:Networks

Data from 16 NASA systems. 1920 model variations.

Consistency is 100% minus percent false positives. Completeness is 100% minus percent false negatives.

Model Accuracies and Tradeoffs


Interactive Metric Dashboards Provide Framework for Visibility, Flexibility, Integration, Automation

Requirements

0

5

10

15

20

25

30

Jun-04 Jul-04 Aug-04 Sep-04

Plan Actual

LCL UCL

Reuse

0%

10%

20%

30%

40%

50%

60%

70%

Proposal SSR PDR CDR

Plan Actual

LCL UCL

Technology Infusion

0

1

2

3

4

5

6

7


Plan Actual

LCL UCL

Progress

0

50

100

150

200

250


Plan Actual

LCL UCL

Cycletime

0

5

10

15

20

25


Plan Actual

LCL UCL

Deliveries

0

5

10

15

20

25


Plan Actual

LCL UCL

Pre-Delivery Defects

0

50

100

150

200

250


Plan Actual

LCL UCL

Post-Delivery Defects

0

2

4

6

8

10

12

14


Plan Actual

LCL UCL

Contact Contact

ContactContactContact

Contact

Data Data

DataDataData

Data

Outliers Outliers

OutliersOutliersOutliers

Outliers

Help Help

HelpHelpHelp

Contact

Data

Outliers

Help

HelpContact

Data

Outliers

Help

Up Down Level 1 AllUp Down Level 1 All

Up Down Level 1 All

Up Down Level 1 All

Up Down Level 1 All Up Down Level 1 All Up Down Level 1 All

Up Down Level 1 All

DASHBOARD Metrics: Organization: Project: Manager: Contact: Status:

Development ABC Products Division XYZ System FirstName LastName [email protected] x12345 10/1/2004

Interactive metric dashboards incorporate a variety of information and features to help developers and managers characterize progress, identify outliers, compare alternatives, evaluate risks, and predict outcomes



Overview












Define “Grand Challenges” Problems for Systems and Software Engineering

Example from Computing (2004)

Source: http://www.ukcrc.org.uk/gcresearch.pdf


Software Engineering Book Captures Best Practices for Economics, Quality, Process, Risk Management

Richard W. Selby, Editor, Software Engineering: Barry W. Boehm’s Lifetime Contributions to Software Development, Management, and Research, IEEE Computer Society and John Wiley & Sons: New York, May 2007, ISBN 9780-4701-48730.

[email protected]

Documents

Systems and Software Design Principles for Large-Scale ... · Example large-scale mission-critical embedded software system ... with 3-15 Builds per Program ... Embedded software