ilities Tradespace Workshop Summary

Barry Boehm, Supannika Koolmanojwong USC-CSSE ARR 20 March 14, 2013

03-12-2013 1

Importance of ility TradeoffsMajor source of DoD system overruns

• System ilities have systemwide impact– System elements generally just have local impact

• ilities often exhibit asymptotic behavior– Watch out for the knee of the curve

• Best architecture is a discontinuous function of ility level– “Build it quickly, tune or fix it later” highly risky– Large system example below

03-12-2013 2

Importance of Cyber-Physical Systems Major gap in tradespace analysis capabilities• Current ERS, DARPA tradespace research focused on

physical system tradeoffs– Range, payload, size, weight, lethality, power and fuel

consumption, communications bandwidth, etc.– Some focus on physical modularity, composability

• Current cyber tradespace research focused on software, computing, human factors tradeoffs– security, safety, interoperability, usability, flexibility,

adaptability, dependability, response time, throughput, etc.

• Gaps in capabilities for co-design of hardware and software, integration of tradespace analyses

03-12-2013 3

iTAP Startup Results• Top-level ility hierarchy view

– Relation to JCIDS combat command user view– Draft survey to compare with acquirer, developer, supporter views

• Affordability means-ends framework view– Quantification via COCOMO, COSYSMO, CORADMO models

• Architecture-strategy synergies and conflicts views– Conflicts with other capabilities

• Change-Oriented Views: Incremental Commitment– The Cones of Uncertainty– MIT Epoch-Era Approach– Agile SE Schedule Acceleration Model

• Domain-Oriented Views– Ground: Wayne State, Georgia Tech– Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia

03-12-2013 4

SERC Value-Based ilities HierarchyBased on ISO/IEC 9126, 25030; JCIDS; previous SERC research

• Individual ilities– Quality of Service: Performance, Accuracy, Usability, Scalability, Versatility– Resource Utilization: Cost, Duration, Personnel, Scarce Quantities (size,

weight, energy, …)– Protection: Safety, Security, Privacy– Robustness: Reliability, Availablilty, Maintainability– Flexibility: Modifiability, Tailorability/Extendability, Adaptability– Composability: Interoperability/Portability, Openness/Standards Compliance,

Service-Orientation

• Composite ilities– Comprehensiveness/Suitability: all of the above– Dependability: Quality of Service, Protection, Robustness– Resilience: Protection, Robustness, Flexibility– Affordability: Quality of Service, Resource Utilization

03-12-2013 5

Prioritized JCIDS ilitiesUser View by Combatant Commands: Top priority first

• Intelligence, Surveillance, and Reconnaissance– Comprehensive Persistent Survivable Integrated Timely Credible Adaptable Innovative

• Command and Control (note emphasis on Usability aspects)– Interoperability Understanding Timeliness Accessibility Simplicity Completeness Agility

Accuracy Relevance Robustness Operational Trust

• Logistics: Supply– Responsiveness Sustainability Flexibility Survivability Attainability Economy Simplicity

• Logistics: Maintenance– Sustainability Responsiveness Attainability Flexibility Economy Survivability Simplicity

• Net-Centric: Information Transport– Accessible Capacity Accurate Timely Throughput Expeditionary Latency

03-12-2013 6

703-12-2013

Legacy System Repurposing

Eliminate Tasks

Eliminate Scrap, Rework

Staffing, Incentivizing, Teambuilding

Kaizen (continuous improvement)

Work and Oversight StreamliningCollaboration Technology

Early Risk and Defect Elimination

Modularity Around Sources of ChangeIncremental, Evolutionary Development

Risk-Based Prototyping

Satisficing vs. Optimizing PerformanceValue-Based Capability Prioritization

Composable Components,Services, COTS

Affordability Improvements and Tradeoffs

Get the Best from People

Make Tasks More Efficient

Simplify Products (KISS)

Reuse Components

Facilities, Support Services

Tools and Automation

Lean and Agile Methods

Evidence-Based Decision Gates

Domain Engineering and Architecture

Task AutomationModel-Based Product Generation

Value-Based, Agile Process Maturity

Means-Ends Framework: Affordability

Reduce Operations, Support Costs

Streamline Supply ChainDesign for Maintainability, EvolvabilityAutomate Operations Elements

Anticipate, Prepare for ChangeValue- and Architecture-Based Tradeoffs and Balancing

03-12-2013 8

USC: COCOMO II-Based Tradeoff AnalysisBetter, Cheaper, Faster: Pick Any Two?

Slider-based equalizer version being developed

0

1

2

3

4

5

6

7

8

9

0 10 20 30 40 50

Development Time (Months)

Cos

t ($M

)

(VL, 1)

(L, 10)

(N, 300)

(H, 10K)

(VH, 300K)

-- Cost/Schedule/RELY:

“pick any two” points

(RELY, MTBF (hours))

•For 100-KSLOC set of features•Can “pick all three” with 77-KSLOC set of features

iTAP Startup Results• Top-level ility hierarchy view– Relation to JCIDS combat command user view– Draft survey to compare with acquirer, developer, supporter views

• Affordability means-ends framework view– Quantification via COCOMO, COSYSMO, CORADMO models

• Architecture-strategy synergies and conflicts views– Conflicts with other capabilities

• Change-Oriented Views: Incremental Commitment– The Cones of Uncertainty– MIT Epoch-Era Approach– Agile SE Schedule Acceleration Model

• Domain-Oriented Views– Ground: Wayne State, Georgia Tech– Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia

03-12-2013 9

Architecture-Based Attribute Trades: Flexibility Example (RT-18a)

Flexibility Arch. Strategy Synergies Conflicts

High module cohesion; Low module coupling

InteroperabilityReliability

High Performance via Tight coupling

Service-oriented architecture Composability, Usability, Testability High Performance via Tight coupling

Autonomous adaptive systems Affordability via task automation; Response time

Excess autonomy reduces human Controllability

Modularization around sources of change

Interoperability, Usability, Reliability, Availability

Extra time on critical path of Rapid Fielding

Multi-layered architecture Reliability, Availability Lower Performance due to layer traversal overhead

Many built-in options, entry points Functionality, Accessibility Reduced Usability via options proliferation; harder to Secure

User programmability Usability, Mission Effectiveness Full programmability causes Reliability, Safety, Security risks

Spare/expandable capacity Performance, Reliability Added cost

Product line architecture, reusable components

Cost, Schedule, Reliability Some loss of performance vs. optimized stovepipes

03-12-2013 10

11

MIT: ilities in Tradespace ExplorationBased on Lean Aerospace, DARPA research

For this plot, Ĉ=C∞

More changeable(ie including flexible, adaptable, scalable

and modifiable)

Colored by outdegree

Enabling Construct: Tradespace Networks Changeability

Survivability

Value RobustnessEnabling Construct: Epochs and Eras

Set of Metrics

03-12-2013

WSU: Versatility Factors and Physical OrganizationComponents that Can be in Different Positions or Orientations

Isolated or Separated Compartments

Running Gear

Chassis

Turret

Sight Weapon

suspension

drive

drivedriveMass & Structure Properties•Mass •Angular moments•Imbalances*•Load bearing wall strength•Deck surface area•Interior volumes**•Interior surface areas**

*Angular moments of the CG about axes of rotation** By crew station and compartment

03-12-2013 12

Workshop objectives and approach • Workshop objectives

– Identify interested collaborators and data– Identify user needs for better reasoning about ility tradeoffs

and affordability– Identify improved approaches for cyber-physical system co-

design

• Workshop approach– Ask participants about their current and likely future

challenges and research needs• For ilities and their tradeoffs• For cyber-physical co-design

– Prioritize research with respect to strength of need, difficulty 03-12-2013 13

Workshop approach

• Ask participants about their current and likely future challenges and research needs– For -ilities and their tradeoffs– For cyber-physical co-design

• Prioritize research with respect to strength of need, difficulty

Participants• Shawn Rahmani• Gary Hafen• Winsor Brown• J.D. Baker• Ed Colbert• Thammanoon K.• Peter Suk• Rachchabhorn W.• Sue K.

• Qi Li• Lori Vaughan• Qing Wang• Jing Du• Liming Zhu• Da Yang• Lee Osterweil• Barry Boehm

Current and likely future challenges and research needs for -ilities and their tradeoffs• Simulation model to address performance, usability model, comparing -ilities• Define hierarchy, top factors• Quantification of value of each –ilities, matrices • Contribution of each –ilities (H/M/L. option1/2) • Prototype for a short life vs requirements in terms of ilitlities• Scale the tradeoff, scale up? • Affordability (prioritizing issues, not only low cost)• Inter-dependency of -ilities • Pair-wise comparison tool (lockheed)• Issue of subjective tradeoff study • Modeling and trade study, matrices of architecture and –ilities• Healthcare group (right distribution of data vs performance, privacy control/security)• Usability vs design• Parametric diagram • Agile architecture development – tradeoff, architecture your asset & product• How to specify requirements in a measurable way• Security, accessibility, QoS • Scalability – esp. diseconomy of scale, e.g. mobile network• Reliability - 5 9s system in banking sector• Macro view & micro view; complimentary process modeling (agent, task, resources) optimize the resources & others to optimize the choices,

a tool that guide resource allocation (TWINS)• Statistical mechanic to understand the process & relationship to COCOMO• How to deploy, data /design selection; tradeoff guideline / tool for system implementation• Architectural style tradeoff; compare & normalize & confidence level; validation tool &Sensitivity analysis tool to improve confidence level• Fault tolerance; process view supporting tradeoff analysis• How many (functional ) requirements is enough; size, # per release , customized development vs maintenance/enhancement • unstated non-functional requirements• Estimation – over/under-estimation• Adaptability & flexibility – adapt to emergent behavior/requirements – not enough representative users • How to help decision makers – can we perform as stated in the proposal• Cyber vs physical; dual cone of uncertainty; • Guidance /model ; how to do the trade/ how to weigh, checklist ; human process part • Process for continuous delivery, how to prioritize requirements and assign them into each release based on revenue (considering dependency

& process adoption) • Using checklist – definition of Done(value, cost, quality)• Scenario generators• Product vs program characteristics - executability

Prioritization resultTools -ilities

•Models & Simulations (21)• Multilevel

•Architecting (17)•Prioritization (12)•Metrics/ Matrices (8)•Scenario generators (8)•Checklists (7)•Parametric diagrams (2)

•Affordability (14) •Timeliness (12)•Reliability / Fault Tolerance (11)

• Safety•Scalability (10)•Security / Privacy (10)•Performance (5)•Usability (4)•Accessibility (3)•Adaptability (3)•Flexibility (1)

Afternoon Agenda

• Exploring high score -ilities factors– Exploring degree of difficulties

• Exploring high score tools• Exploring cyber-physical co-design methods

Affordability• Definition = Effectiveness & Cost (INCOSE)• Fixed effectiveness & try to reduce cost• Define cost (Total cost of ownership, life cycle cost)• Cost = including operational & support cost? System cost

(e.g. cheap to build to maintain but not cheap to operate)

• What is the biggest gap that is not covered when doing affordability analysis? – Technical debt

• As long as you are aware of your debt, give you leeway• Technical debt identification tool

– HW SW integration

Timeliness• Define timeliness

– Meet deadline subject to definition of effectiveness (timebox – discard low priority items to meet schedule; agile rebaselining)

– Version control issues– Conflict in reusing items

Reliability / Fault Tolerance /Safety• Standard assurance issue• Define “safe”• Reliability = impact of defect (loss of life/properties/ $)• Dependent on other systems

– Cloud – no full control • How can your system tolerate other systems?• Not equal (reliable <> fault tolerance <> safety)• Acceptable levels• A tool to provide analysis of balancing these factors • KPP – key performance parameter – no single number• Estimating certification cost & schedule• Level of testing

Scalability• Scalability of Product• Architecture evaluation tool

– Something like static source code analysis tool• Modeling tool• Network traffic & overhead• Communication mechanism• Co-dependent systems (with internet connection? )• Diversity

– Versions or platforms• Horizontal vs Vertical (scale out vs scale up)• Consistency among diversity/nodes• Timing • How to upgrade

Security / Privacy• Acceptable levels / level of assurance• Security vs complexity tradeoff (also with reliability, availability)• multiple independent level of security• Certification cost & schedule• Scope of certification (privacy)• Lifetime of data (dynamic data)• Anti-tamper • communication security• Scaling• How to show feasibility evidence• How to justify being assured (how the entire thing is developed)• Proof of correctness • Domain dependent

Prioritization resultTools -ilities

•Models & Simulations (21)• Multilevel

•Architecting (17)•Prioritization (12)•Metrics/ Matrices (8)•Scenario generators (8)•Checklists (7)•Parametric diagrams (2)

•Affordability (14) •Timeliness (12)•Reliability / Fault Tolerance (11)

• Safety•Scalability (10)•Security / Privacy (10)•Performance (5)•Usability (4)•Accessibility (3)•Adaptability (3)•Flexibility (1)

Models & Simulations (Multilevel)

• Accuracy / fidelity• VV&A (verification, Validation & accreditation)• Scalability• Modeling environment (jungle/desert)• Modeling language • Visualization• Test cases, usage(scope of use), underlying

assumption

Architecting

• Language (dependent)• Visualization (static & dynamic)• Executable architecture• Analyzable• Multiple views (like DODAF), integration of views• Generation of the system from the architecture• Architecture pattern, reuse

Prioritization

• Multiple stakeholder value proposition, criteria analysis

• Tradeoff• Interdependencies• Cost & schedule• Value estimation• Environment (scenario)