Quantitatively Measured Process Improvements at Northrop Grumman IT

Craig HollenbachNorthrop Grumman IT

Agenda

Northrop Grumman IT Overview 2002 SCAMPI AppraisalSample Project Data

• Inventory Tracking System (ITS)• AIT• JPATS• SIGS

Conclusions

Northrop Grumman Overview

Northrop Grumman• $26 B in revenue; 120,000 employees; 50 states; 25 countries

Information Technology (IT) Sector• $4 B in sales; 22,000 employees; 48 states; 15 countries

Defense Enterprise Solutions (DES) Business Unit• $548 M in sales; 2,900 employees, 23 states, 3 countries

DES provides enterprise-wide technology solutions to the Defense marketplace

Major Applications:

Logicon LIS

Litton TASC

Logicon LTS

L5 Litton PRC

(to other units)

DES

2001 20021999 2000

Logicon LATL3

LDES L5

(to other units)

L3

CMMI

(to other units)

Logicon LISS L3

ENABLER

LIEB

SPII

DES Maturity Pedigree

2002 CMMI Approach

Background• Kent’s quote about problems at beginning of 2002

Personnel & Teams• PA Process Owners • DES Organizational Units (e.g., EPG, training, procurement)• High Maturity Process Area Teams, composed of project

representatives (L4WG, L5WG, MO, DPWG, TCMSIG)Approach

• DES Organizational Improvements– CMMI Process Gap Analysis– Built Umbrella processes for legacy orgs

• DES Project Improvements– Assigned support reps to assist project personnel– Project representatives participated on high maturity process area teams

2002 SCAMPI Appraisal

SCAMPI appraisal led by independent SEI-certified appraisers in December 2002 determined that DES achieved• CMMI-SE/SW maturity level 5• CMMI-SE/SW capability level 5 in PMC, IPM, TS, and VER• SW-CMM maturity level 5

DES works with other IT Business Units to transfer our process improvement experience throughout the sector

Inventory Tracking System (ITS)

Inventory Tracking System

Project Description:USAF/AFMC/MSG Inventory Tracking System (ITS)

ModernizationA 3.5-year, $11M Firm-Fixed Price project with a development

staff of approximately 15 membersDevelopment Team uses SEI Personal Software Process (PSP)Implemented CMMI Level 5 quantitative management

processes to dramatically improve the cost, schedule, and delivered quality of the software

Currently in preparation for 1st contractual customer driven test cycle

Contractual Quality Goal is to deliver no known severity 1-3 defects (1-Critical, 2-Urgent, 3-Routine).

ITS Critical + Urgent Defect Density

Quantitative Management Plan Goal: 5/KLOC for Critical + Urgent

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.025

1/9/

02

1/22

/02

2/5/

02

2/13

/02

3/5/

02

3/28

/02

5/10

/02

6/6/

02

6/17

/02

6/25

/02

3/29

/02

6/3/

02

7/11

/02

7/26

/02

8/5/

02

8/8/

02

8/14

/02

7/2/

02

8/20

/02

9/6/

02

9/16

/02

9/25

/02

9/27

/02

10/4

/02

10/2

2/02

10/2

4/02

10/2

8/02

10/3

1/02

11/8

/02

8/29

/02

10/2

8/02

10/3

1/02

11/1

2/02

11/1

4/02

11/1

5/02

X

DDr DDr Mean UNPLX LNPLX U2S L2S U1.5S L1.5S

5.3 4.9 3.2

1.31.4

Peer Review - Builds 1 - 5

KLOC = Thousand Lines of Code

Peer Review Defect Density (Critical + Urgent cont.)

DP 2 DP 3

2.1

3.93.5

6.16.6

0

1

2

3

4

5

6

7

1 2 3 4 5

Build

Defe

ct D

ensi

ty (D

efec

ts/K

LOC)

DP 1

Cost Variance by Build

-46%

11%21%

41%

-15%

-60%

-40%

-20%

0%

20%

40%

60%

1 2 3 4 5

Build

Cost

Var

ianc

eDP 2 DP 3DP 1

Schedule Variance by Build

36%

49%

38%

-1%1%

-10%

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5

Build

Sche

dule

Var

ianc

e

DP 2 DP 3DP 1

Return on Investment – Construction Phase

Hours invested: 124• Team training: 48• Conducting DP Cycles: 76

Defects avoided: If the Defect Density had remained at 6.6 (Build 1), we would have injected 110 more defects.

Hours saved: At an estimated cost of 15 hours per defect this equals 1650 hours.

Return: • Hours:1650/124 = 1330%• Customer satisfaction: Priceless! – “The contractor has

always provided products and services with less defects that industry standards. Most have been provided with no defects. Personnel have been used that show a complete understanding of their subject area and are able to convey this information in a highly professional manor.”

Inventory Tracking System – Test Phase

12

Need to understand Total Defect Density in Peer Review in Construction Phase to relate to Defect Density in Test Phase• DP Cycles had an effect on Total defect density also

– Build 1 Total defect density = 21.6 defects/KLOC– Build 5 Total defect density = 13 defects/KLOC

• Total Defect Density for Construction Phase = 19 defects/KLOC• Total Defect Density for Testing to Date = 4.5 defects/KLOC

0

5

10

15

20

Construction Test

Defect Density

400% Reduction

Inventory Tracking System – Test Phase

Management Goals in Test are being exceeded! Critical/Urgent Defect Total Defects

DDt Unit .5/KLOC 2/KLOCDDt .25/KLOC 1/KLOC(all internal integration cycles)

ITS Test Defects By Test CycleActual Defect Density by KLOC

12

0

20

40

60

80

100

120

140

160

180

Unit Test Build .01 -0.6

Build 0.7 Build 0.75 Build 0.80(Informal)

Critical + Urgent Total

.46

.2

.056

.33

1.4

.18.065

.233.32

.123

DP Cycle

0100

200300400

500600700

800900

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Project Month

Defe

cts

Disc

over

ed

Theoretical Curve Actual Values

Defect Discovery Rayleigh CurveAll Test Defects

Feb 03 Prediction = 14.55 defectsFeb 03 Actual = 15 defects .14 defects per K LOC/month

Inventory Tracking System – Test Phase

Quantitative Management Plan Goals: DDs – Defect Discovery 1/KLOC

.21

.065

When the Total Defect Discovery rate falls under 1 defect per KLOC per month the project manager and test lead have enough confidence to stop test cycle.

0100

200300

400500

600700

800900

0 10 20 30

Project Month

Defe

cts

Disc

over

ed

Theoretical Curve Actual Values

Prediction = 1.44 defectsActual = 1 defect (last week of May)

Defect Discovery Rayleigh CurveBuild .75 Test Defects

Results

CMMI Quantitative Management & Defect Prevention Cycles have a huge return on Investment in the Construction Phase.• Specific results from the first coding cycle to the fifth are:

Critical / Urgent defect density reduced by 68%, Cost Variance improved from -59% to +39%, and Schedule Variance improved from +26% to +49%.

This return has a significant effect on the Test Phase• Where most projects have the highest defect detection rate in

Test, ITS has its lowest defect detection rate. Latent defect analysis estimates delivering a defect density of between 0.35 and 0.7; a total of 20 to 40.

Understanding the quality of the product allows for better management decisions and with highly satisfied customers.

Automated Identification Technology (AIT)

AIT Document Defect Data QM

Began collecting data in Feb ‘02 as part of DP Cycle • Process improvement techniques per DP Cycle identified

– Use of CM controlled Templates for documents enforced for authors– Type classification identified for defects: technical/non-technical– Documentation Input Defect Report checklist completed

Identify number of pages each document Identify document types Identify defects as technical/non-technical

– Management and personnel awareness of data collection and purposes• Six months data had defect rate vary from 4.9% to 11.6%, with one

outlier higherProcess Improvement implementation

• Resulted in re-evaluation of upper and lower limits• Increased personnel and management focus on data• Data for last 12 months has defect rate vary from 0.4% to 5.9%

Document Input Defect X Control Chart

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

55%

60%

Feb-

02

Mar

-02

Apr-

02

May

-02

Jun-

02

Jul-0

2

Aug-

02

Sep-

02

Oct

-02

Nov

-02

Dec

-02

Jan-

03

Feb-

03

Mar

-03

Apr-

03

May

-03

Jun-

03

Jul-0

3

X

% Defects % Defects Mean UNPLX LNPLX U2S

L2S U1.5S L1.5S Upper Spec LimitNorthrop GrummanInformation Technology Company Sensitive / Company Proprietary -- Subject to Nondisclosure

Document defect data Feb ’02 – Jul ‘03

JPATS TIMS

JPATS Challenge

JPATS Build 1.05 (June 2002)• The first build to be released after progressing from

“Development” to the “Maintenance” phase of the program.– Builds 1.01 – 1.04 were internal builds, not released for customer

verification

• Used “development”–style processes for fixing STRs • Failed 9/30 (30%) of the on-site STR verification tests with

customer witnesses. – NOTE: STRs Fixed vs. STRs Accepted is a measure that is quantitatively

measured by the JPATS Program

As a Result … Kicked off JPATS DP Cycle #1• GOAL: Reduce the STR verification failures to < 5% for JPATS

builds 1.0.6 and 1.07

DP Cycle Findings

Root causes included• Lack of “maintenance”-style processes (e.g. streamlined for

dealing with many (~30-130) STRs/build)• Lack of “maintenance”-style build planning & tracking

~40 Countermeasures identified• Many top-tier countermeasures focused on

improving/updating our STR build processes– Most of these were approved for action by the sponsor

DP Cycle Improvements

Actions• JPATS updated/developed the following processes

specifically for the Contractor Logistics Support (CLS = maintenance) phase

– BP 100, CLS Software Build Process– BP 200, Define a Build – BP 300, Plan and Track a Build– BP 400, Develop a Build

UT 100, Unit Test Procedure PR 100, Peer Review Procedure (Code) RT 100, Regression Test Procedure

– BP 500, Deploy a Build

• JPATS developed a build planning and tracking matrix called the “STR Big Board” to track all the elements required by the process per STR across all STRs

DP Cycle Effectiveness

Build 1.06 (July 2002)• 0% STR verification failure rate (0/11)• Goal of verification failure rate < 5% was met

Build 1.07 (Oct 2002)• 2.7% STR verification failure rate (3/113)• Goal of < 5% verification failure rate was still met

Subsequent builds have continued to perform well• See next slide showing current JPATS QM measure for STR

Verification

DP Cycle Effectiveness

STRs Projected Fixed vs Actual Fixed: (Actual - Projected) / Projected

-60%

-40%

-20%

0%

20%

40%

1.05 1.06 1.07 1.08 1.1 1.2

Build #

% D

evia

tion

DDrc Goal X-bar UCL LCLNorthrop GrummanInformation Technology Company Sensitive / Company Proprietary -- Subject to Nondisclosure

Problem Occurred

DP Cycle Subsequent Results

Synthetic Imagery Graphical System (SIGS)

SIGS Schedule Performance

Goal: SPI (X bar) of 85% in the 1st third of each PoP, 90% in the 2nd third, and 95% in the last third

Actual: 92.1% over multiple PoPs; 88.3% at the end of the 1st PoP; 88.7% at the end of the 2nd; and 96.8% at the end of the last PoP

Highlights: Cost Performance (CPI) was 96.8% over the same period; Award Fee average was 99% over the same period

SPIm X Chart for SIGS

50%

60%

70%

80%

90%

100%

110%

120%

130%

Date

Perc

ent (

%)

SPIm X bar UNPL LNPL Wt. Ave.

SIGS Schedule Performance (Cont’d)

Situation: O&M project undertaking a major redesign of the system over multiple years using new technology

At the beginning: unfamiliar technology meant that schedule estimates had large uncertainty since there was no available historical data to support the basis of estimate

Process changes: introduced Earned Value (EV) tracking combined with statistical process control (SPC) techniques allowed better monitoring of progress against the plans and identifying when there are special causes of variation

Improvement: by closely tracking the actual effort required to complete the earlier activities, we were able to feed that back into the estimates for the later activities and thus able to produce schedules with less uncertainty

Conclusions