Revitalization of Systems Engineering

Revitalization of Systems Engineering

Kristen BaldwinActing Director, Systems and Software Engineering

Office of the Deputy Under Secretary of Defense(Acquisition and Technology)

presented to:

NDIA Manufacturing Division Meeting August 13, 2008

2

USD, AcquisitionTechnology & Logistics

DUSD, Acquisition &Technology

AT&L OrganizationAT&L Organization

IndustrialPrograms

Defense Procurementand Acquisition Policy

Defense ContractManagement Agency

Small BusinessPrograms

Dir, Joint AdvancedConcepts

Dir, Systems andSoftware Engineering

Dir, PortfolioSystems Acquisition

Defense AcquisitionUniversity

3

Briefing OutlineBriefing Outline

Where is DoD Systems Engineering now?SE Revitalization StatusAnalysis of FailureKey near-term initiatives

Where does SE need to be?Strategic DirectionEnhanced SE

2003 201520102008

Institutionalize SERevitalizeSE

KeyInitiatives

4

SSE FunctionsSSE Functions

Acquisition Program SupportDecision Support to Senior DoD Leadership

Mentoring Acquisition Programs

Technical Reviews of Key Program Documentation

Policy, Guidance, Best Practices

Acquisition Workforce Education and TrainingSE, Test, Production Quality and Manufacturing, Software

Systemic Root Cause Analysis

“Discipline” Expertise Risk CMMI Modeling/Simulation Reliability Software Assurance/Cyber Architectures Energy Comm/Networking Test/Eval Safety System of Systems

5

Revitalizing DoD SERevitalizing DoD SEIssued DoD-wide SE policy – focused effort on up front, sound technical planningIssued guidance on SE, test and evaluation (T&E) and softwareRevised SE & T&E curricula at Defense Acquisition University Established SE Forum—senior-level focus across DoDInstituted system-level assessments in support of OSD major acquisition program oversight role

Executed 83 reviews since March 2004

Integrated, software, system assurance, energy, M&S and system of systems into SE revitalization effortsEstablished closer working relationships with industry and academia

Prescribing Good Practices Is Not Enough…

6

Analysis of Program FailureAnalysis of Program Failure

Acquisition cost growth over 11 years (SAR data FY 1995–2005):

Estimation changes: $201B

Engineering changes: $147B

Schedule changes: $70B

Over the past 10 years, DoD systems have experienced a 33% cost growth due to “RDT&E mistakes”

DoD IOT&E results, FY2001-2007

35 systems; mix of ACAT II, 1C, 1D across 3 Services

57% of programs found to be “Not Suitable or Effective” ● Approx. 49% were deemed “Not Suitable” or partially NS

● Approx. 34% were deemed “Not Effective” or partially NE

7

Analysis of Program FailureAnalysis of Program FailureSpecific Causes of Program Failure

(Systemic Root Cause Analysis findings)

We do not start programs right Insufficient requirements analysis and definition at program initiation Lack of rigorous SE approach Optimistic/realistic reliability growth – not a priority during development Inadequate software architectures, design, development discipline, and organizational competencies

We do not manage programs right Insufficient trade space Insufficient risk management Inadequate IMP, IMS, EVMS Most programs lack quantifiable entrance/exit criteria Maturing “suitability” (e.g., RAM) is not always a priority

Key Initiatives2008-2009

Key Initiatives2008-2009

9

SSE FY08/09 Thrust AreasSSE FY08/09 Thrust AreasSystemic Root Cause Analysis – the collection and analysis of systemic program performance issues, determination of root causes, and development of corrective action

System Assurance – the reduction of vulnerability to malicious intent in our systems, considering the full spectrum security of information, technology and hardware/software components

Software Engineering Competency – the focus on software engineering as a critical element of complex systems acquisition, and strategic initiatives to ensure future Defense software demands can be met by government and industry

Systems of Systems – augmenting acquisition and engineering practices to better plan, develop, and manage interdependent systems of systems

Early Systems Engineering – the institution of technical management and engineering practices prior to program initiation (MS B) in order to enable risk informed, balanced acquisition and budgeting decisions

10

SSE FY08/09 Thrust Areas (cont’d)SSE FY08/09 Thrust Areas (cont’d)

Revitalizing Developmental Test & Evaluation – comprehensive strategy for early T&E involvement and integrated testing to manage technical risks across the acquisition system, sustaining systems and capabilities, and rebuilding government T&E expertise.

System Engineering Research UARC – establishment of a system engineering research program

Energy – reducing DoD energy consumption across our operations, to include force development, deployment, and support

Reliability – ensuring rigor, discipline and an integrated approach in systems engineering practices and T&E strategies for improving the reliability of systems

Safety – integrating safety process advances into Department acquisition policies

Modeling and Simulation – ensuring rigor in M&S policy and guidance to advance the state-of-practice; execute acquisition M&S Master Plan

11

Manufacturing Readiness LevelsManufacturing Readiness Levels

MRL/EMRL Background: MRLs documented by a “body of knowledge”, the MRA Deskbook and

piloted in selected Air Force acquisition programs EMRLs used as exit criteria on ACAT ID programs since 2002

ODDR&E and OSD SSE collaborating to: Transition from EMRL to robust MRL body of knowledge Align metrics with new DoD life cycle management framework Leverage acquisition milestones and SE technical reviews to assess

manufacturing maturity Draft policy to integrate manufacturing into acquisition decisions

Doubtful that it will go into the pending DoDI 5000.02 update

ACTION: OSD SSE has requested the NDIA Engineering and Manufacturing Committees to: Review MRL criteria and its time phasing across the acquisition lifecycle Better define production “relevant” and “representative” environments Assess use of MRAs/Production Readiness Reviews on supplier base Provide comments on the MRA Deskbook

12

Manufacturing Readiness Levels (cont.)Manufacturing Readiness Levels (cont.)

Goals:Increase emphasis on manufacturing throughout the acquisition lifecycle Ensure considerations of manufacturing and producibility through established service mechanisms

Revised DoDI 5000.02 mandates consideration of manufacturing maturity during each acquisition phase and at each milestone

Areas to continue to work:Appropriate assessment of manufacturing at MS A and MS BEnsuring manufacturing assessments are not stand-aloneUpdate engineering training & guidance accordingly

13

Integration Readiness LevelsIntegration Readiness Levels

Background: Systemic analysis has shown integration issues in ~ 45% of PSRs

Contributes to cost and schedule issuesDUSD(A&T) requested SSE study and develop integration readiness levels for use in acquisitionSSE initiating a collaborative effort to develop quantifiable integration metrics

Product: Draft set of IRLs in 120 days

ACTION:Identify existing IRL activities/products and interest to support comprehensive study

14

Human Systems IntegrationHuman Systems Integration

Hardware

Software Human

Total System

Approach

Human Systems Integration (HSI). The interdisciplinary technical and management processes for integrating human considerations within and across all system elements; an essential enabler to systems engineering practice.

15

Recent HSI Guidance and DirectionRecent HSI Guidance and Direction

H.R. 1585 National Defense Authorization Act for FY 2008: Section 231Directs the Secretary to: (1) coordinate and manage human systems integration activities throughout DOD acquisition programs; and (2) designate a senior DOD official to be responsible for such effort.

USD(AT&L) memorandum, Apr 3, 2008Designates DUSD(A&T) as the senior official responsible for coordination and management of HSI activities throughout DoD acquisition programs, with DUSD(S&T), Director, Biological Systems as the co-lead for S&T

FY09 HASC language directs the USD(AT&L) to develop a comprehensive plan for funding and implementing HSI through all phases of science, research, and acquisition. Report due by March 15, 2009

ACTION: We are soliciting input from DoD and Industry partners to help respond to Congressional Direction. Specifically:

What are key HSI activities, initiatives, capabilities that exist todayWhere gaps or issues exist

Our Challenge: Institutionalization

2010-2015

Our Challenge: Institutionalization

2010-2015

17

SSE Strategic DirectionSSE Strategic Direction

Cut cycle time to field capability by embedding systems engineering engagement with requirements and early phase acquisition

Change engineering culture, both vertically and horizontally, across Government and industry

Implement Human Capital Strategy to assure DoD workforce competency

Advance Engineering State-of-Practice through research transition

Implement comprehensive solution for system assurance/program protection/cyber security

18

FormalProgram

Start

Improve Knowledge through Technical FoundationImprove Knowledge through Technical Foundation

Systems Engineering is effective when it informs, and is informed by, other Acquisition process owners

Un

ce

rta

inty

Agreement to pursue a materiel solution

Material Solution Analysis Technology Development

Selection of a

preferred solution

System Level Specs Preliminary

Design CompletedDesign

AoA

Business Decisions

Engineering Support

PDR

CDR

Preferred System Concept

Preferred System

Analysis

Technology Maturation

And Prototyping

MDD MS A

MS B

National Research Council

“Pre-Milestone A and Early-

Phase Systems

Engineering”Jan 2008

Questions?Questions?

20

Systems Engineering Research (SER) University Affiliated Research Center (UARC)Systems Engineering Research (SER) University Affiliated Research Center (UARC)

Overview: SER UARC implementation is intended to continue efforts to advance practice of SE across DoD enterprise

Approved by the DDR&E, October 2007Committed funding of $10M over 5 years

Mission: Research and analyze advanced and emerging systems engineering practices and relevant technologies to address the full spectrum of DoD systems across the Department, from capability areas, enterprise systems, systems of systems, and interoperability down to subsystems and configuration items with the goal of ensuring consistency and systems engineering excellence throughout the acquisition cycle.Status: Request for Proposal nearing completion

All domestic institutions with engineering programs notified (>700)Draft RFP reviewed by over 30 universities

Key Dates (planned): RFP release – May 2008Contract award – July/August 2008

21

Initial SE Research TasksInitial SE Research Tasks

Examine and recommend areas for advancing current SE methods, processes, and tools as they are applied across the DoD acquisition life cycle focusing on three different development environments: individual weapons systems, SoS, and network-centric systems

Characterize SE effectiveness within the context of DoD acquisition and identify methods of measurement suitable for application to project execution organizations (e.g., defense contractors), program management organizations (e.g., project managers and program executive offices), and oversight organizations (e.g., OUSD(AT&L))

22

SoS SE Guide (SEG) v 1.0 RevisionSoS SE Guide (SEG) v 1.0 Revision

Draft SoS SEG version 0.9 released December 2006

March – September 2007 -- ‘Pilot Phase’

Structured review of version 0.9 with SoS SE practitioners and researchers

Results offer ‘boots on the ground’ basis for guide revision

Draft SoS SEG version 1.0 released for comment December 10, 2007

Available at http://www.acq.osd.mil/sse/ssa/guidance.html

Formal comment acceptance closed on February 14, 2008

~600 comments were adjudicated by April 1

Final release of version 1.0 by June 2008

23

Orchestrating upgrades

to SoS

Orchestrating upgrades

to SoS

Orchestrating upgrades

to SoS

Multiple, possibly concurrent increments

Assessing SoS

Performance

Monitoring & Assessing

Changes

Translating Capability Objectives

Developing & Evolving

SoSArchitecture

UnderstandingSystems &

relationships

SoS

Systems

View of SoS Systems EngineeringView of SoS Systems Engineering

Addressing new requirements

& options

Addressing new requirements

& options

Addressingrequirements

& solution options

24

DoD Software Core CompetencyDoD Software Core Competency

DUSD(A&T) vision for software engineering (SWE) leadership and world class capabilityIssue: Software Growth

Software Requirements Growth: 80% of system functionality [CSIS]

Software Size Growth: 20 million ESLOC [CSIS]

Software Overruns: 50% of SW efforts [Standish Group]

Issue: SE and SW have not been well integrated on projectsTop Software Issues and Project Areas

Requirements SE/SW Integration

Human Capital

SW Sustainment SW Testing SW Quality Metrics

SW Estimation

25 25

System AssuranceSystem Assurance

We continue to be concerned with assurance of our critical DoD assets:

Critical informationCritical technologies Critical systems

Observations: Increasing numbers of network attacks (internal and external to DoD)Broader attack space

Trends that exacerbate our concerns:Globalization of our contracts, expanding the number of international participants in our system developments Complex contracting arrangements that further decrease transparency below prime, and visibility into individual components

These trends increase the opportunity for access to our critical assets and for tampering

26

System Assurance for the Program Manager

System Assurance for the Program Manager

Program Manager

TrustedFoundry

ProgramProtection

InformationAssurance

CenterFor

AssuredSoftware

SoftwareProtectionInitiative

Anti-Tamper

SoftwareAssurance

ConfigurationManager

SafetyEngineer

QualityEngineer

ReliabilityEngineer Systems

Engineer

Lack of focused guidance to counter the threat

27

Raise the bar:Awareness - Knowledge of the supply chain

- Who has access to our critical assets

Protection - Protect critical assets through security practices- Design our systems for assurance

Acquisition Path ForwardAcquisition Path Forward

Create a framework to integrate multiple security policies and oversight

Leverage Program Protection requirement for all acquisition programs as set by DoDD 5200.39 policy

Integrate all assurance oversight, planning, and risk mitigation activity at the system level

Develop Guidance on Engineering for System AssuranceGuidebook on Engineering for Assurance for program managers/engineers

Defines how assurance can be incorporated into system engineering and design:

● e.g. Isolation, Redundancy, Quality, and Fault Analysis

28

“We believe we have gone too far in reducing Systems Engineering capability … we need to re-build.”

Honorable Secretary England, 7 November 2007, PEOSYSCOM Conference Keynote

Leadership Direction for SELeadership Direction for SE

AT&L Source Document Objectives:

Capabilities: Balance and rationalize requirements; Review key capability areas to seek greater efficiency; Evaluate mission capability gaps against technology opportunities

Programs: Accurately price programs and insist schedule and budget reflect realistic pricing, recognizing technical and integration risks; Build prototypes and conduct experiments to provide options; Arm the Program Manager with tools

People: Play an appropriate stewardship role for the science and engineering community

Strategic Objectives identified by the SSE Forum:Enhance SE pre-MS B to ensure programs succeedTransform Architecture to support Systems Engineering EnterpriseEstablish a Human Capital Strategy for SEAssess needs for additional SE Tools

USD(AT&L)Direction

DoD SELeaders

Deputy Secretaryof Defense

29

SSE FY10-15 Program Outcomes (1/2)SSE FY10-15 Program Outcomes (1/2)

Create enhanced Systems Engineering capability (pre-MS A and between MS A&B)

Provide SSE resources (support, tools) for Technology Development phase

Integrate SSE with Requirements, Programming and Management

Support pre-program and capability analyses (AoAs)

Establish AT&L matrix support to CPMs and SoSs

Develop a Human Capital Strategy for System and Software Engineering (SSE)

Advance Systems Engineering state of practice through research by leveraging and transitioning UARC recommendations

Revitalize Developmental Test and Evaluation to support faster fielding and reliability growth testing

30

SSE FY10-15 Program Outcomes (2/2)SSE FY10-15 Program Outcomes (2/2)

Transition and promulgate SSE tools, practices

Institutionalize PSRs, SEPs, Software, Reliability and T&E practice, etc. among Services, Agencies, Industry Transition Partners

Transform lessons learned to predictive measures

Transparent data, correlate program findings with external databases (e.g., DAMIR, PoPs, PA&E)

Implement corrective actions

Value Energy in acquisition processes

Institutionalize energy in Milestone reviews and portfolio management

Ensure fuel demand considerations in AOA and all program analysis

Integrate assurance and security practice into SE

Transition from FY08/09 cyber and system assurance pilot activities