National Defense Industrial Association Implementing

National Defense Industrial Association

Implementing Continuous Iterative Development and Acquisition

Executive SummaryNDIA Systems Engineering Division

in partnership with INCOSE and PSM

22-Apr-2019

22-Apr-2019 NDIA Continuous Iterative Development and Sustainment WG 1

Background

Defense Science Board (DSB) released a report in Feb-2018 containing seven recommendations regarding software design and acquisition. Section 868 of NDAA 2019 mandates implementation of these recommendations within 18 months.

The Defense Innovation Board (DIB) Software Acquisition and Practices (SWAP) study group has also provided many insightful and largely compatible recommendations.

NDIA, INCOSE and PSM support the DSB and DIB concepts and the opportunities they offer to DoD and the defense industry.

• NDIA offers the recommendations herein to ASD(A&S) and ASD(R&E) representing an “industry perspective” on path forward.

• NDIA appreciates the opportunity to partner with DoD on implementation.


DSB SW Task Force Recommendations

22-Apr-2019 3

1. Software Factory – A key evaluation criteria in the source selection process should be efficacy of the offeror’s software factory.

2. Continuous Iterative Development – DoD and defense industrial base partners should adopt continuous iterative development best practices for software, including through sustainment.

3. Risk Reduction and Metrics for New Programs – For all new programs, starting immediately, implement best practices in formal program acquisition strategies (multiple vendors and down-selects, modernized cost and schedule measures, status estimation framework)

4. Current and Legacy Programs in Development, Production, and Sustainment – for ongoing development programs, PMs/PEOs should plan transition to a software factory and continuous iterative development.

5. Workforce – The U.S. Government does not have modern software development expertise in its program offices or the broader functional acquisition workforce. This requires Congressional engagement and significant investment immediately.

6. Software is Immortal: Software Sustainment – RFPs should specify the basic elements of the software framework supporting the software factory… reflected in source selection criteria

7. IV&V for Machine Learning – Machine learning is an increasingly important component of a broad range of defense systems, including autonomous systems, and will further complicate the challenges of software acquisition.

The NDIA working group developed consensus recommendations responding to each of the 7 DSB findings:

• Assumptions

• Picture of Success (End State)

• Current State

• Description

• Obstacles

• Path Forward

This briefing is an executive summary of those recommendations. Detailed report provided separately.

NDIA Continuous Iterative Development and Sustainment WG

Framing Assumptions

Continuous iterative development (CID) methods have cross-functional implications. The scope includes not just SOFTWARE but also SYSTEMS ENGINEERING and supporting disciplines.

Software Factories include people, processes, and tools – not just a tool chain.

Funding and contracts must be aligned to support implementation and/or migration to SW factories with life cycle sustainment.

A collaborative approach to Intellectual Property (IP) across the entire acquisition life cycle will be developed that meets both Government and Supplier needs.

A business case can be made for the effective deployment and maintenance of integrated tool chains to build capability throughout the life of the system.

Traditional waterfall-based processes, tools, and measures are generally not well suited to CID.

A skilled SW-informed workforce cadre is available or can be developed across functions(e.g., software, acquisition, PMs, sustainment).


Cross-cutting assumptions. Refer to the separate detailed report with assumptions specific to each DSB recommendation area.

DSB #1: Software FactoryNDIA WG Recommendations

Picture of Success (end state)

People • Qualified factory workforce • Continuous learning(relentless improvement, pipeline feedback)

Process • Integrated PMB• Metrics• Predictability

• Digital Blueprint / Play Book • Ontology, Nomenclature• Secure Supply Chain• Relentless Improvement

Tools • Platform agnostic tool chain• Adaptable to change• Test automation at all levels

• Model-based SW validation vs. architecture

• Red team / Blue team factory Security integrated into factory workflows (DevSecOps)

Click thumbnails to zoom

Recommendations for Path Forward:


PMB: Performance Measurement Baseline

DSB #2: Continuous Iterative Development (MVP)NDIA WG Recommendations


Government / Contractor Interface

Contracting • New programs defined by solution intent (CV-1)• Contracts defined by evolutionary viability products (MVP/NVP)

Funding • Contract funding structure supports seamless capability evolution

Stakeholders • Active engagement in CID lifecycle

Design • Guided by MOSA

IP • Government access to source code with negotiated IP protections

Program Execution

People • Multi-discipline agile execution includes aligned milestones• Direct user/developer interaction informs design (product owner)

Process • Early SE ➢ SW sequencing, refactoring, tools, environments

Tools • Test automation accelerates delivery (rapid release, deployment)

Procurements based on iterative development of releases according to product capability thresholds




CID: Continuous Iterative DevelopmentFOC: Final Operating CapabilityIOC: Initial Operating CapabilityIP: Intellectual Property

MOSA: Modular Open Systems ArchitectureMVP: Minimally Viable Product NVP: Next Viable Product

DSB #3a: Risk Reduction (Competitive Prototyping)NDIA WG Recommendations


Competition • Business case: win-win partnership, common goals, acquisition/support strategy

• Objective downselect evaluation criteria (RFP L&M) and feedback• Open architecture on critical components

Contracts • IP agreement negotiated, sustained across the life cycle• Funding and contracts aligned to support factory migration

Metrics • Continuous improvement, SMART measures against objectives• Risk-based decision making

Resources • Funding, staffing, tools, environments to support multiple teams



IP: Intellectual PropertyRFP: Request for ProposalSMART: Specific, Measurable, Achievable, Relevant, Time bound

Competitive prototyping can help in many situations, but does not solve all acquisition problems.


DSB #3b: Measures for CIDNDIA WG Recommendations


Consensus frameworks

• Objectives first - measures aligned and tailored from information needs, goals and constraints, at program and enterprise levels

Modernized measures

• Migration toward consensus alternatives to traditional waterfall and phase-based SW measures (LOC, EVM, milestones, …)

• Derived from SW factory processes, automated by toolchain• Basis for measuring cost and schedule vs. plan

History-based estimates

• Repositories collect performance-based measures (e.g., WBS, staff, cost, productivity) supporting future comparisons, basis of estimates, proposals, and program monitoring



CID: Continuous Iterative DevelopmentEVMS: Earned Value Management SystemLOC: Lines of CodeWBS: Work Breakdown Structure

Measures for CID should be aligned with information needs and constraints, at program and enterprise levels


DSB #4: Transition for Current and Legacy ProgramsNDIA WG Recommendations

Click thumbnails to zoom22-Apr-2019 NDIA Continuous Iterative Development and Sustainment WG 9

COTS: Commercial Off the ShelfFOSS: Free Open Source SoftwareGOTS: Government Off the Shelf


People • Skill assessment for gap analysis • Skilled capable workforce for transition on legacy programs

Process • Business case for transition• Playbooks and Blue Prints for

legacy code transition• Assessment of supply chain and

SW pedigree (FOSS, COTS, GOTS)

• Risk adjusted product backlog• Strategies for incrementally

building up test automation

Tools • Tools to generate legacy ‘as-built’ documentation and models for legacy code base

Defense Science Board, Design and Acquisition of Software for Defense Systems, Feb 2018

See also: Defense Innovation Board SWAP Study Report: Supplementary Documents, Appendix B.6 Sustainment / Modernization Subgroup Report


https://www.acq.osd.mil/dsb/reports/2010s/DSB_SWA_Report_FINALdelivered2-21-2018.pdf

https://media.defense.gov/2019/Feb/22/2002092142/-1/-1/0/DIB-SWAP_STUDYREPORT%5bDRAFT%5dSUPPLEMENTARYDOCS.PDF

https://media.defense.gov/2019/Feb/22/2002092142/-1/-1/0/DIB-SWAP_STUDYREPORT%5bDRAFT%5dSUPPLEMENTARYDOCS.PDF

DSB #5: WorkforceNDIA WG Recommendations



CDRL: Contract Data Requirements ListCID: Continuous Iterative DevelopmentDAU: Defense Acquisition UniversityIPT: Integrated Product TeamPM: Program ManagerPMO: Program Management Office


Education and Training

• DAU curriculum for DevSecOps and modern SW-centric systems

• Community of practice platforms

• Training across career fields(PM, sustainment, acquisition)

• Aligned with current/future development and recruiting needs

StakeholderEngagement

• Collaborative government / industry partnerships

• Trained experienced industry partners and supply chain

• Consensus measurement framework

• Multi-discipline CID support teams (CDRLs, events, milestones)

Staffing • Increased hiring, retention, training for acquisition experts

• Recruiting pipeline for SW experts

• Dedicated workforce funding and coaches across services

• PMO IPTs for modern SW practices

• Continuous Iterative Software Development, Acquisition and Sustainment

• DevSecOps• Competency Models• Practical Experience (Boot Camp)


DSB #6: Sustainment (Software Is Immortal)NDIA WG Recommendations




Resources • Availability and support of a trained proficient workforce• Organic DoD software infrastructure, incentives, funding• Collaborative IP strategy throughout the life cycle, using a

“work shared sustainment” approach

Contracting Language

• Contracts specify elements of framework supporting SW factory• Policies and guidance validated by workshops, pilots

Sustainment Ecosystems

• Understanding of current and future organizational ecosystems to ensure effective transfer of SW factories


DSB #7: IV&V for Machine LearningNDIA WG Recommendations




Consensus ML IV&V Framework

• Model-based inference engine considering full system context• Risk-based methodology supporting T&E needs, linked to ML model

failures early in system development process• Mitigation throughout system design, development, sustainment

Open Data Sets

• High data quality, quantity, availability, and traceability• Data repository accessible to government and industry• Governance model for availability, level playing field, innovation • New repository data continuously collected and published

Perpetual Updates

• Continuous ML model updates – evolution at speed of relevance• Continuous V&V methods sensing changes from models, environment• Performance/accuracy aligned with changing environment, threats

IV&V: Independent Verification & ValidationML: Machine LearningT&E: Test and Evaluation

T&E is a full lifecycle activity focused on mitigating risk of failing to meet operational needs

Perpetual Upgrades


SummaryThe NDIA WG provides an industry perspective on picture of success, current state, obstacles and path forward for each DSB recommendation

DSB Recommendation NDIA “Path Forward” recommendations

#1 – Software Factory 14 Contracting, funding, incentives, methods, security, supply chain, and measures

#2 – Continuous Iterative Development

3 Pilots and continuous improvement

#3 – Risk Reduction & Metrics

10 Acquisition strategy, competitive prototyping, culture, workforce, IP, and measures

#4 – Legacy Systems 5 Assessments, supply chain, methods, tools, and modeling

#5 – WorkforceDevelopment

3 Competency models, workforce assessment, workforce management, and training

#6 – Sustainment 2 Contracting and industry-government transfer of sustainment responsibilities

#7 – Machine Learning 5 Risk, research, CONOPs, ML data, and Software Factory interactions

Details of each topic and recommendation are provided in the separate report.


Acknowledgments

Joseph Elm L3 Technologies Firas Glaiel Raytheon Virginia Perkins MDAGeoff Draper Harris Lemonte Green MDA Mike Phillips SEIJames Belford USAF STSC Brian Hann SAIC Geoff Pierce NRODawn Beyer Lockheed Martin Stephen Henry DAU Marilyn Pineda Lockheed MartinBarry Boehm USC Paul Janusz US Army RDEC Garry Roedler Lockheed Martin

Sean Brady DAU Suzette Johnson Northrop Grumman Heather Romero Raytheon

Kevin Chapman Harris Cheryl Jones US Army CCDC Armaments Gene Rosenbluth Northrop Grumman

Yann Chazal Renault Geethesh Kukkala SAIC Larri Rosser Raytheon

David Chesebrough NDIA Richard Kutter USAF Dan Strickland MDA

Chris Collins DAU John MacCarthy Univ. of Maryland James Thompson OUSD(R&E) retired

Mark Cornwell OUSD(R&E) Phyllis Marbach INCOSE Steve Verga Harris

Truc DeSa Lockheed Martin Jason McDonald Harris Ketchiozo Wandji NAVAIR

James Doswell US Army ARDEC Mike McLendon SEI Allison Weigel Toray

Rick Dove Paradigm Shift Jenna Meyers HQDA ASA FM Beth Wilson retired

Jim Duffy Raytheon Jeffrey Mueller DAU / USAF Erik Wylie MDA

Robert Epps retired Kenneth Nidiffer SEI Hasan Yasar SEI/CERT

Mark Ginese DAU John Norton Raytheon Robin Yeman Lockheed Martin


The NDIA Systems Engineering Division and its partners, INCOSE and PSM, appreciate the opportunity to provide an industry perspective for advancing the use of iterative methods in defense software acquisition.

The defense industrial base embraces the opportunities offered by the DSB and DIB recommendations and looks forward to supporting the Department of Defense with implementation.

NDIA Continuous Iterative Development and Sustainment Working Group:

Backup

Supporting Content (Hidden Slides)

Excerpts of NDIA Recommendations by DSB Finding

(see separate briefing package for full details)
