USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items Barry Boehm, USC-CSSE February 14, 2007

USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items

Barry Boehm, USC-CSSE

February 14, 2007

http://csse.usc.edu/BoehmsTop10/

[email protected]

2

Outline: Top-3 SIS Risks Workshop

Working group guidelines Risk survey results and survey update(?) The top three risks

– Architecture complexity; system quality tradeoffs

– Requirements volatility; rapid change

– Acquisition and contracting process mismatches Architecture complexity and system quality tradeoffs

– Architecture complexity phenomenology

– Nature of system quality

– Quality tradeoff perspectives

3

Working Group Guidelines

Product: briefing, preferably with notes

Topics should include:– Most critical success factors in each area

– Current best practices for addressing them

– Areas for further research Rated 0-10 on value and difficulty of research

4

Research Topics: Agile Methods

0

5

10

0 5 10

Difficulty of Research

Valu

e o

f R

esearc

h

1a

1b

2

3

4

5

6

7

8

9

10

11

12

1. Relationship between plan driven and agilitya. For individualsb. For organizations

2. Differences between agile and plan driven outcomes

3. Effect of Gurus4. Mismatches between development

approach and acquisition practices5. How do you measure quality in an agile

environment?6. Data collection; agile experience base7. Team of teams8. Agile Development and Evolutionary

Prototyping9. Shared Code and/or module ownership10. Architecture: when, how much, how to

express11. Lack of user consensus12. Dynamic Homegrounds

5

SIS Risk Survey 2006: Statistics

Number of Surveys: 25 Average Experience: ~28 years (6 years – 51 years) Area Distribution:

– Software: 20– Systems: 17– Hardware: 0

Business Domain Distribution:– Aerospace: 18– Software Infrastructure: 5– Business: 4– Telecom: 3– Others: Secure Apps (1); Safety Critical Apps (1); C4ISR (1)

6

Risk Survey 2006: Nominees

Acquisition and contracting process mismatches Architecture complexity; quality tradeoffs Budget and schedule constraints COTS and other independently evolving systems Customer-developer-user team cohesion Migration complexity Personnel shortfalls Process maturity Requirements mismatch Requirements volatility; rapid change Technology maturity User interface mismatch

7

USC CSSE Top 10 Systems and Software Engineering Risk Items

130.5142

109.5

77

115.5

75.5

99

46.83

100.33

131.66

58.564.67

0

20

40

60

80

100

120

140

160

Risk Areas

Su

m o

f R

anks

Acq & Cont process mismatches

Arch complexity; quality tradeoffs

Budget & Schedule

COTS

Customer-developer-user

Migration Complexity

Personnel Shortfalls

Process Maturity

Requirements Mismatch

Requirements Volatility

Technology Maturity

User Interface Mismatch

Risk Survey 2006 Results

8

SIS Risk Grouping

# Risk Item ΣRanks

1 Architecture complexity, quality tradeoffs 142

2 Requirements volatility 131.66

3 Acquisition and contracting process mismatches 130.54 Customer-developer-user 115.5

5 Budget and schedule 109.5

6 Requirements mismatch 100.33

7 Personnel shortfalls 99

8 COTS 77

9 Migration complexity 75.5

10 User interface mismatch 64.67

11 Technology maturity 58.5

12 Process maturity 46.83

9

Survey 2007: Early Statistics

Number or Surveys: 41 Average Experience: ~27 years (6 years – 51 years) Area Distribution:

– Software: 33– Systems: 34– Hardware: 0

Business Domain Distribution:– Aerospace: 32– Software Infrastructure: 7– Business: 6– Telecom: 5– Others: Secure Apps (1); Safety Critical Apps (1); C4ISR (1);

Network and Protocols (1); Defense (1); Program and Risk Management (1)

10

Survey Results: 2006-2007

USC CSSE Top 10 Systems and Software Engineeiring Risk Items 2007

241.8

284.7

238.5

188.4206.2

133.8

188.65

109.31

205.93

265.06

158.03

113.2

0

50

100

150

200

250

300

Risk Areas

Su

m o

f R

anks

Acq & Cont process mismatches

Arch complexity; quality tradeoffs

Budget & Schedule

COTS

Customer-developer-user

Migration Complexity

Personnel Shortfalls

Process Maturity

Requirements Mismatch

Requirements Volatility

Technology Maturity

User Interface Mismatch

11

SIS Risk Grouping 2006-2007

# Risk ItemPrevious Rank

ΣRanks

1. Architecture complexity, quality tradeoffs ↔ 1 284.7

2. Requirements volatility ↔ 2 265.06

3.Acquisition and contracting process mismatches

↔ 3 241.8

4. Budget and schedule ↑ 5 238.5

5. Customer-developer-user ↓ 4 206.2

6. Requirements mismatch ↔ 6 205.93

7. Personnel shortfalls ↔ 7 188.65

8. COTS ↔ 8 188.4

9. Technology maturity ↑ 11 158.03

10. Migration complexity ↓ 9 133.8

11. User interface mismatch ↓ 10 113.2

12. Process maturity ↔ 12 109.31

12









13

SIS Architecture Complexity: Future Combat Systems

14

Requirements Volatility: Ripple Effects of Changes- Breadth, Depth, and Length

Platform N

• • • Platform 1

Infra

C4ISR

Command and ControlSituation AssessmentInfo FusionSensor Data ManagementSensor Data IntegrationSensorsSensor Components:

2008 2010 2012 2014 2016

…1.0 2.0 3.0 4.0 5.0

Breadth

Length

Depth

DOTMLPF

Legend: DOTMLPF Doctrine, Organization,

Training, Materiel, Leadership, Personnel, Facilities

C4ISR Command, Control, Communications, Computers,

Intelligence, Surveillance, and Reconnaissance

15

Average Change Processing Time: 2 Systems of Systems

Average workdays to process changes

0

20

40

60

80

100

120

140

160

WithinGroups

AcrossGroups

ContractMods

16

Acquisition/Contracting Mismatches: Fitness Landscapes

Role of Fitness Landscapes in Complex Adaptive Systems (CAS)– S. Kauffman, At Home in the Universe, Oxford University

Press, 1995 CSoS Acquisition Challenges

– B. Boehm, “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), 2006, pp. 1-19.

A Candidate Three-Agent Acquisition Fitness Landscape– D. Reifer and B. Boehm, “Providing Incentives for Spiral

Development: An Award Fee Plan”, Defense Acquisition Review 13(1), 2006, pp. 63-79.

17

Role of Fitness Landscapes in CAS

Incentive structures for local behavior Induce global behavior via adaptation to change

Fitness Landscape

Uniform RandomSurvival-Related

Global Result Gridlock Chaos Edge of Chaos

Acronym

(Metaphor)

OWHITS

(Ostriches with Heads in the Sand)

TRAW

(Turkeys Running Around Wild)

NOSUFAS

(No One-Size-Uniformly-Fits-All

Solutions

Acquisition Example

MIL-STD-1521B Waterfall, Fixed

Price, Build-to-Spec

Recursive Acquisition Reform,

Total Systems Performance Responsibility

Candidate for Discussion:

3-Agent Model

18

Complex Systems Acquisition Challenges

ObjectiveCandidate Solution

Example Challenges

Avoid Obsolescence

Plan-Driven Rapid Development

4-Hour House Inflexible

Adapt to Rapid Change

Agile MethodsExtreme

ProgrammingUnscalable; Buggy

Releases

Assure Resilience

Independent V&V Formal Methods Expensive

19

Candidate Approach: 3-Agent Model

Agent Objective Agent ApproachFitness Landscape/ Incentive Criteria*

Build Current Increment

Rapid, Stable, Schedule-As-Independent Variable (SAIV),

Build to Specs and Plans

Meet Milestones/Exercise SAIV; Deliver on Time;

Collaboration with Other Agents

Assure ResilienceIntegrated, Independent

Verification and Validation

Priority-Weighted Identification of Risks and Concerns;

Collaboration with Other Agents

Prepare for Build of Next Increment

Observe, Orient, Decide on Proof-Carrying Rebaselined

Specs and Plans

Risk/Opportunity Management; Rebasline Proof Thoroughness; Collaboration with Other Agents

20

Risk-Driven Scalable Spiral Model:Increment View

Increment N Baseline

Future Increment Baselines Rapid Change

High Assurance

Agile Rebaselining for Future Increments

Short, Stabilized Development of Increment N

V&V of Increment N

Increment N Transition/O&M

Current V&V

Short Development Increments

Future V&V

Stable Development Increments

Continuous V&V

Concerns Artifacts

Deferrals Foreseeable Change (Plan)

Resources Resources

Increment N Baseline

Future Increment Baselines Rapid Change

High Assurance

Agile Rebaselining for

Short, Stabilized Development of Increment N

V&V of Increment N

Increment N Transition/O&M

Current V&V

Short Development Increments

Future V&V

Stable Development Increments

Continuous V&V

Concerns Artifacts

Deferrals Foreseeable Change (Plan)

Resources Resources

Unforseeable Change (Adapt)

21









22

Larger Systems Need More Architecting: COCOMO II Analysis

0

10

20

30

40

50

60

70

80

90

100

0 10 20 30 40 50 60

Percent of Time Added for Architecture and Risk Resolution

Per

cen

t of T

ime

Ad

ded

to O

vera

ll S

ched

ule

Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution

Added Schedule Devoted to Rework(COCOMO II RESL factor)

Total % Added Schedule

10000KSLOC

100 KSLOC

10 KSLOC

Sweet Spot

Sweet Spot Drivers:

Rapid Change: leftward

High Assurance: rightward

23

Architecture-Breakers are the Biggest Source of Rework

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

% of Software Problem Reports (SPR’s)

TRW Project A373 SPR’s

TRW Project B1005 SPR’s

% ofCosttoFixSPR’s

Major Rework Sources:Off-Nominal Architecture-BreakersA - Network FailoverB - Extra-Long Messages

24

Best Architecture is a Discontinuous Function of Quality Level

$100M

$50M

Arch. A:Custommany cache processors

Arch. B:ModifiedClient-Server

1 2 3 4 5

Response Time (sec)

Original Spec After Prototyping

25

The Nature of Quality: Participant Survey

Which figure best symbolizes quality improvement?

Holistic Approach

Lean Approach

Analytic Approach

Preoccupation with Booze and Sex

30

There is No Universal Quality-Value Metric

Different stakeholders rely on different value attributes– Protection: safety, security, privacy– Robustness: reliability, availability, survivability– Quality of Service: performance, accuracy, ease of use– Adaptability: evolvability, interoperability– Affordability: cost, schedule, reusability

Value attributes continue to tier down– Performance: response time, resource consumption (CPU,

memory, comm.) Value attributes are scenario-dependent

– 5 seconds normal response time; 2 seconds in crisis Value attributes often conflict

– Most often with performance and affordability

31

Overview of Stakeholder/Value Dependencies

AttributesStakeholders

**

*

** ** **

**

** **

**

**

**

***

*

*

**

* *

*

Prote

ction

Robus

tnes

s

Quality

of S

ervic

e

Adapt

abilit

y

Afford

abilit

y

Developers, AcquirersMission Controllers, Administrators

Info. ConsumersInfo. Brokers

Info. Suppliers, Dependents

Strength of direct dependency on value attribute

**- Critical ; *-Significant; blank-insignificant or indirect

32

Implications for Quality Engineering

There is no universal quality metric to optimize

Need to identify system’s success-critical stakeholders– And their quality priorities

Need to balance satisfaction of stakeholder dependencies– Stakeholder win-win negotiation

– Quality attribute tradeoff analysis

Need value-of-quality models, methods, and tools

33

Tradeoffs Among Cost, Schedule, and Reliability: COCOMO IIWant 10K hour MTBF within $5.5M, 20 months

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50

Development Time (Months)

Co

st

($M

)

(VL, 1)

(L, 10)

(N, 300)

(H, 10K)

(VH, 300K)

-- Cost/Schedule/RELY:

“pick any two” points

(RELY, MTBF (hours))

•For 100-KSLOC set of features•Can “pick all three” with 77-KSLOC set of features

34

Agenda : Wednesday, Feb 14

8:15 – 10:00 am: Architecture Complexity and Quality Tradeoffs; Elliot Axelband (RAND), Chair– Overview, Issues and Approaches; Barry Boehm (USC)– From Dependable Architectures To Dependable Systems; Nenad Medvidovic (USC)– Architecture Tradeoff Analysis: Towards a Disciplined Approach to Balancing Quality Requirements;

Azad Madni (Intelligent Systems Technology)

10:00 – 10:30 am: Break

10:30 am – 12:30 pm: Requirements Volatility; George Friedman (USC), Chair– Process Synchronization and Stabilization; Rick Selby, Northrop Grumman– Disciplined Agility; Rich Turner (SSCI)– Using Anchor Point Milestones; Tom Schroeder, BAE Systems

12:30 – 1:30 pm: Lunch

1:30 – 3:30 pm Acquisition and Contracting Mismatches; Rick Selby (NGC), Chair– Acquisition Assessment Analyses; Kristen Baldwin (OSD/AT&T/S&SE)– Commercial Acquisition Practices; Stan Rifkin (Master Systems Inc.)– Space Program Acquisition: Systems Engineering & Programmatic Improvements; Marilee Wheaton

(Aerospace Corporation)

3:30 – 4:00 pm: Break

4:00 – 5:00 pm: General Discussion: Working Group Formation; Barry Boehm, Chair

Documents

USC CSSE Workshop Overview: Top 3 Software-Intensive Systems Risk Items Barry Boehm, USC-CSSE February 14, 2007