Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6

The CMMI Modelsin

Oversight of Space Flight Software Development

Liz StrassnerLead, Software Process & Process Management Group

David RetherfordSr. Software and Systems Engineer/ERC

Spacecraft Software Engineering BranchNASA/JSC

Used with permission

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Agenda

Introduction Selection of CMMI Models for Spacecraft Software Engineering Branch NASA Surveillance Strategies Project Orion Surveillance Strategies Orion Software Oversight

CMMI Maturity Level 2 Rating on Oversight Who are we What we did to earn CMMI Level 2 Trials & Tribulations of Software Oversight What we are doing now

2/22/20102

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Spacecraft Software Engineering & CMMI Models

We evaluated the three CMMI models (CMMI-ACQ, CMMI-SVC, & CMMI-DEV) for applicability to our work CMMI-SVC was discovered to be completely non-applicable

IT support help desk; Pizza delivery service Oversight of a contract doesn’t meet the intent of “service” in this model

CMMI-ACQ was reviewed for applicability in a SCAMPI B NASA/JSC organizational structure caused some goals to be completely out of

scope of the organization We were fully compliant with goals and practices that were in scope of the organization

The model does not allow goals to be out of scope

CMMI-DEV Software development work was obviously in scope Software oversight is also in scope, but needs to be looked at from a systems

engineering perspective Software oversight on a 30 year project is never-ending so needs to be broken

into smaller projects

2/22/20103

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Selection of CMMI Model & Projects

Based on our experience with the SCAMPI B, we chose to go for a CMMI Maturity Level 2 rating under the CMMI-DEV model

Selecting projects was an interesting experience Software Development Projects Many active software development projects to select from A long history of flight software development projects from the former

Flight Software Branch that was merged into Spacecraft Software Engineering Branch

Software Oversight Role is significantly different than in the Shuttle or ISS Programs Lead appraiser insisted that we have at least one project in this area

because of the large focus in the organization on oversight Chose to create sub-projects out of milestone reviews Software Requirements Review for SCAMPI B “System PDR & Software Spiral Review" for SCAMPI A

2/22/20104

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6NASA Surveillance Strategies (From COTR Refresher Training 2-18-2008)

Basic strategies: insight, oversight, or hybrid Insight: relies on performance requirements & metrics. Use a

minimum set of product/process data to give adequate visibility into the integrity of the product/process. Contractor assumes more responsibility & accountability Government steps back, evaluates deliverables & existing contractor

processes—not day-to-day close monitoring Appropriate strategy for contracts with little cost risk, clearly-defined

products/services, & confidence in proven contractor performance Oversight: rely on NASA-imposed product specification’s & process

controls, MIL standards, mandatory inspection points, etc. Intrusive monitoring at contractor’s plant, & in-line NASA involvement in contract work. Appropriate strategy when NASA is assuming the liability for

performance or quality; when we determine oversight is necessary to mitigate performance risk; or when the contractor is unproven

2/22/20105

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Project Orion Surveillance

The Orion Contract as a whole uses the hybrid approach to surveillance with pre-declared oversight in high risk areas Building a space vehicle is high risk so the contract is Cost Plus Award

Fee (CPAF) type Very extensive DRD requirements for insight Software was pre-declared as a major risk item by the Agency, so

oversight of software was planned and scheduled Joint NASA-Contractor Risk Board

Insight for Software All of the NPR 7150.2 chapter 5 requirements exist as individual DRDs CMMI Maturity Level 3 was levied on all software development

organizations

2/22/20106

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Software Oversight of Orion

Oversight for Software NASA co-chairs all boards and panels Every Contractor functional area lead has a NASA counterpart NASA participates in ALL software working groups NASA has signature authority on all major software deliveries including

the Software Development Plan NASA has an internal software test environment NPR 7150.2 NASA Software Engineering levied on contract in its

entirety NASA Standard 7009(I) Models & Simulations Standard levied on

contract in its entirety NASA Software Assurance and Safety Standards levied on contract in a

“cut and paste” method into the contract (DRD CEV-S-001) Constellation Software documents also levied by the “cut and paste”

method or by reference to specific sections or paragraphs of the CxPdocument

2/22/20107

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6

Spacecraft Software Engineering Branch & CMMI Level 2

2/22/20108

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Spacecraft Software Engineering Branch

Spacecraft Software Engineering Branch

Process & Process Management Group Test & Verification Group GFE Group Technology Group Systems Engineering Group

Orion Flight Software System Manager

Orion Recon System ManagerOrion Vehicle Systems Management Manager

Altair Lead

2/22/20109

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Orion Software Team Interfaces

2/22/201010

Orion Software

Team

Prime Contractor

Subcontractor

Orion Project Office

Constellation

Subcontractor

Subcontractor

Subcontractor

SubcontractorSubcontractor

Customer - Provider

Stakeholder – Provider Rep

Provider - Customer

Provider - CustomerCustomer Rep -Provider

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6NASA Orion Software Team Organization

Orion Software Manger is part of the Orion Project Office Everyone below that level is matrixed out of Engineering organizations

at JSC, Ames, GRC, and LaRC Frank Delgado/ER6 is the Flight Software System Manager and leads

the multicenter flight software team Neil Townsend/ER6 is the Reconfiguration System Manager and leads

the multicenter software process and process management team The Spacecraft Software Engineering Branch and the Orion Software

Team are approximately the same size and have a large overlap.

2/22/201011

Orion Software Team

Spacecraft Software

Engineering Branch

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Year One – Baby Steps

Organization was very small Only function was Orion oversight Responsible for Orion Vehicle System Management (Class A) Responsible for in-house flight software development environment for

all Orion software oversight (Class D & E) Interviewed all employees to find out how they did their work Wrote processes based on the “how they did their work” Assumed no higher level organizational processes affected oversight

At a conceptual level, we did not have any problem mapping the CMMI Specific Goals and Practices to processes

Late in the year, Orion Software Manager function moved here in addition to Vehicle System Manager and the branch grew

2/22/201012

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Year Two – Chaos Reigns

Former Flight Software Branch merged into the branch as one of many steps JSC Engineering took to focus software Brought in several GFE Flight Software Development projects, most in

maintenance and operations phase already CMMI Level 2 compliant Consolidated all JSC Orion software oversight personnel Brought in methodology for meeting Generic Goals and Practices useful

for all projects GFE Projects must follow extensive higher level organizational

processes New Branch Chief assigned Perfect storm Policies and processes for oversight crashed into GFE flight software

policies & processes Previous branch-only view had to be extended to reflect division and

directorate infrastructure New branch chief had new goals and expectations

2/22/201013

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Year Two – Coming out of Chaos

CMMI Consultation Bill Pierce provided advise on how to structure projects for oversight and

methods to apply CMMI practices to an oversight project Selected projects: 1) Class A development, 2) Class D

development/acquisition, 3) Class A-E oversight The former Intelligent Systems Branch was merged into Spacecraft

Software Engineering Branch Primarily Class E research and technology software

SCAMPI B Newly acquired projects from the recent merger were excluded Uncovered a problem with PPQA Discovered that we still had disconnects in applicability of division and

directorate policies and processes Determined that our SwRR Oversight project was mis-scoped

Appraiser direction was to do better for PDR

2/22/201014

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Year 3 – Success!

Solved PPQA problem temporarily by returning to original resources Orion PDR slipped significantly, so best we could do was “plan” for

PDR without any action past that Appraiser decided to merge our SwRR Project and our start of the PDR

Project into a single “Software Oversight” project for the purpose of the SCAMPI A

Allowed us to show that we learned from SwRR and incorporated those lessons into the PDR planning activities

Since all our development projects were new, we had to pull in an old (CMMI L2 rated) project that was in maintenance to ensure end of lifecycle coverage This end of lifecycle coverage would be a problem for any new

organization

2/22/201015

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6

Trials & Tribulations of planningSoftware Oversight

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Orion Software Spiral Review – Overview

Meet diverse set of goals & requirements Orion project SW Management Plan (CxP 72099) Need to plan component-level PDR activities for LM SW Spiral

delivery Follow general flow of Orion system PDR (CxP 72212)

Spacecraft Software Engineering Branch Demonstrate an oversight project meeting CMMI ML 2

Agency Software engineering requirements (NPR 7150.2) Systems/project requirements (NPR 7123.1)

CMMI Dev 1.2 Demonstrate that oversight project can meet ML 2 SPs & GPs

2/22/201017

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Software Oversight – Changes in Attitude

Requires an orthogonal change of view of Requirements Configuration Management Evaluation & assessment vs. technical development

Requirements Technical requirements of contractor product (e.g. spacecraft) differ from

oversight As part of project tasks may influence or evaluate technical requirements Actual oversight project requirement is to review products (e.g. SRS) ,

generate comments Separation of concerns Technical requirement development vs. oversight activities

2/22/201018

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6SW PDR Project Plan Impact Sources

2/22/201019

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Orion PDR Flow – High Level Overview

PDR Plan & Kickoff•PDR Ground Rules & POD•PDR Review Process•PDR RID Process

Subsystem Design Reviews (SSDR)•Subsystem Specifications & IRDs•Subsystem Design Data Books•Subsystem Specific Design & Requirements Review Presentations

System & Module Review•Subsystem Rollups•System Specifications and Design Review Presentations

System PDR•RID Process•RID Screening, Review, & Dispositions•PDR Pre-board & Board

Software PDR•Specification & Design Tech/Peer Review Process•SW Artifact Release and RID Process•SW RID Screening, Review, & Disposition•SW PDR Pre-board & Board

2/22/201020

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Oversight Project – CMMI Model

CMMI Maturity Level 2 process areas for oversight 7 PA’s – PP, PMC, REQM, CM, PPQA, MA, SAM

CMMI Maturity Level 2 process area emphasis on managing project Planning project, process, and products Monitor & control project

2/22/201021

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Spiral Review Project – CMMI Process Areas

Project Planning (PP) Develop project life cycle and

process Document in project plan

Project Monitoring and Control (PMC) Track team participation in

contractor reviews Team assessment of reviews

Requirements Management (REQM) Screen/review requests from

Project Office or Branch Management

Product & Process Quality Assurance (PPQA) Evaluate team process

compliance Evaluate team assessment

products Configuration Management

(CM) NASA ICE/Windchill area for

document storage Measurement and Analysis

(MA) Metrics for effort and effort per

review/document Supplier Agreement

Management (SAM) Contractors part of Orion SW

team Use division process for

engineering service support

2/22/201022

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Basic Software Oversight Requirements

Not the same as the product technical requirements What ‘we’ must do to demonstrate Contractor development of software technical baseline Accomplished proper oversight of contractor Assure adequacy of contractor efforts and products

As an oversight team we must Review contractor technical efforts (requirements & design) Generate comments/RIDs Attend technical and peer reviews Evaluate software product quality and maturity Within known (agreed to) constraints

Basically – our requirements are: To provide a set of software engineering services to Orion project Engineering involvement and assessment

2/22/201023

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Software PDR Oversight Lifecycle

Not traditional SW lifecycle – no actual SW being produced Formulate and initiate the project/process (Inception) Execute the plan/process (Execution) Finish the project & baseline (Closeout)

Product/artifact data Assessment/evaluation vs. SRS/IRD

Configuration Management Oversight project artifacts, not contractor technical artifacts

2/22/201024

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Oversight Phases

InceptionDefine project lifecycle and

organizationDefine review process(es) and

boardsDefine NASA technical and

management rolesGenerate project planAnd away we go…

ExecutionMonitor contractor software

development effortsReview artifacts, generate

comments, attend reviews, attend disposition and solution meetings

CloseoutFormal review releaseRID inputsRID review/screeningRID dispositionsSW Spiral Pre-boardSW Spiral boardArtifact update/re-release

2/22/201025

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Process Area Considerations

PPQA No software artifacts (e.g. SRS, SDD,

source code, etc.) Process and assessment report

focused Use outside organization – provide

independent assessment

M&ANo technical measures (e.g.

SLOCs, etc.)Effort/product based Hrs/review No. comments

Assessments Completion Quality

REQM Oversight requirements NOT the

same as technical product Activity and evaluation/assessment

based “The software shall do this …..” vs.

“The project shall participate in technical reviews.”

Control/manage between Orion SW Mgr & Av&SW Project Office

PPProject “within a project”Predetermined schedule Orion project/contractor level

2/22/201026

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6SW Spiral Review – CMMI Appraisal Nuggets

Double vision syndromeEasy to get cross-eyed over

apparent identical areas between LM and NASA oversight project CMMI model vs. NASA

engineering focusEx.: CM – whose CM? LM

Project Link vs. NASA ICEEx.: Requirements management

– Oversight project requirements vs. contractor project technical requirements

Project within a projectScope focus very tight for

oversight project – milestone based

Other work did not stand still

KISSKeep oversight plan, processes,

and tasks simpleSimple data/configuration

management method Only controlling documents

Reasonable/simple plan and process description results in Effective CMMI Maturity Level

2 compliance

2/22/201027

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Typical Review Process

Perform Tech Review

SFM/SWEProduct Owner

Disposition Product Comments

Perform Peer Review

Product Comments

Initial attempt to use SE process to describe & define process oversightEx.: Use case diagram to represent project process for technical and peer

reviewsEach use case may breakdown and be described in project plan for each

specific processValue of approach is still being evaluated (may refine for next version &

evaluate for improvement)2/22/2010

28

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6

Next Steps

2/22/201029

Spa

cecr

aft S

oftw

are

Eng

inee

ring

Bra

nch/

ER

6Next Steps

New Project – Software Process Improvement Project Among other things, earn CMMI Maturity Level 3 by April 2011 Requires major culture change to implement OT, OPF, & OPD

Orion Software PDR Project Continues to evolve Creating checklists to further refine measures from reviews Expect to learn enough to provide guidance to development projects on Planning and scheduling peer reviews Planning and scheduling milestone reviews

Expect confusion in implementing PI, VAL, VER, and TS because our products are brain-power related rather than code

2/22/201030