Quantitative Methods in Defense and National Security 2010

Ms. Anne TwistDeputy Director, Resource Analysis

Office of the Under Secretary of Defense (AT&L)25 May 2010

Acquisition Cost Growth – The Stats Matter!

2

JSF (F-35)BMDS

SSN 774DDG 51 Destroyer

C-17AF-22 Raptor

V-22 (OSPREY)F/A-18E/F

CVN 21TRIDENT II MISSILE

Joint MRAPP-8A

PATRIOT/MEADS CAPChem Demil-CMA

CH-53K ProgramBLACK HAWK UH-60MAMRAAM (AIM-120)

FMTVDDG 1000 Destroyer

JTRS GMRE-2D AHE

LPD 17 CLASSWIN-T INCREMENT 3

EFV (Formerly AAAV)Stryker

C-130J HerculesSBIRS High

BAMSMH-60R

Global HawkCH-47F

LONGBOW APACHEAdvanced EHF

USMC H-1 UPGRADESREAPER

EA-18GPATRIOT PAC-3

AB3AMF JTRS

Navstar GPSChem Demil-ACWA

JLENSMH-60S

JASSMC-5 RERP

MUOSTOMAHAWK(R/UGM-109E)

LHA 6SM-6

C-130 AMPGMLRS

IAMDJDAMJPATS

JTRS HMSER/MP UAS

JSOWWIN-T Increment 2CECATIRCM/CMWS

GPS-IIIAFAB-TJHSV

FBCB2WIN-T Increment 1

LITTORAL COMBAT SHIPAIM-9X

WGSPREDATORInc 1 E-IBCT

VTUAVMIDS

ExcaliburIDECM

NMTHIMARS

JCALUH, UH-72A Lakota

JTRS NEDAGM-88E AARGM

COBRA JUDY REPLACEMENTNASRMS

B-2 RMPMP-RTIPC-5 AMP

EA-6B ICAP IIIGBS

JPALSSBSS BLOCK 10

B-2 EHFASIP

LAIRCM

-100 -50 0 50 100 150 200 250 300Investment Funding (TY$B)

Sunk Cost Remaining Cost (FY11+)

MDAP Portfolio Tornado Chart

We have a bad behaving population of MDAPs; the magnitude of the JSF program makes portfolio-wide statistics meaningless. Measuring portfolio

cost growth and other similar statistics is Bad Math.

Measuring Cost Growth:GAO Method vs. AT&L Method

The Stats Matter!

4

March 31, 2009: Development costs for the Pentagon's major weapons systems soared last year, helping drive overruns that are "staggering," the Government Accountability Office said in a report released yesterday. Overall, the cost overruns associated with the military's major weapons …”total near $300 billion, and the average program delay has stretched from 21 to 22 months,”

The figures reflect a weapons development and procurement system that is woefully broken… “Pentagon planners don't do a good enough job of analyzing those requirements to understand whether they have the technologies and designs to build to them," GAO analyst Michael J. Sullivan said.

Doomsday Press Reports

April 1, 2008: The Government Accountability Office found that 95 major systems have exceeded their original budgets by a total of $295 billion, bringing their total cost to $1.6 trillion, and are delivered almost two years late on average. In addition, none of the systems that the GAO looked at had met all of the standards for best management practices during their development stages.

Auditors said the Defense Department showed few signs of improvement since the GAO began issuing its annual assessments of selected weapons systems six years ago. "It's not getting any better by any means," said Michael Sullivan, director of the GAO's acquisition and sourcing team. "It's taking longer and costing more."

Measuring Cost Growth:GAO Method vs. AT&L Method

The Stats Matter!

• GAO Methodology• Single Program Example: DDG 51• Oldest 10 programs• MDAP Portfolio

--Mr. John J. Young, former USD(AT&L), 2009

“The new GAO report continues to sensationalize the assessed cost growth of $296 billion on 96 programs. This number has been cited by many people as a condemnation of the defense procurement process. I have analyzed the components of this GAO number, and I would suggest that the number is misleading, out-of-date, and largely irrelevant to the current management of DoD programs.”

GAO Methodology• GAO publishes an annual report on cost growth

– Summarizes all Major Defense Acquisition Programs (MDAPs) – highlight is their “Total Cost Growth” calculation

– The FY10 report did not give a total cost growth figure• GAO defines cost growth as the change in total

program acquisition costs from the first original estimate to the current estimate

• Problems with GAO’s methodology: 1) Procurement of additional quantities and/or required capability

counts as cost growth2) Poor early performers can never recover, even if they have been

performing well for many years3) Differing “portfolios” compared on an annual basis4) Pre-Milestone B estimates used (i.e., before the program defined)5) Acquisition painted with broad brush as if all programs are broken

6

7

GAO Methodology• Quantity growth and age of the programs greatly

inflate cost growth• The fine print….

– Past sins are never forgiven: Many programs have been in the portfolio for more than 15 years and continue to carry huge cost growth figures even if they have been performing well for years

– Quantity Changes are Leadership Decision to Shape the Force: Procurement of additional quantities and/or required capability is included in GAO’s cost growth calculation, often making more desired programs appear to be broken

• These two factors consistently had impact on program cost, and therefore we must isolate these factors to really understand what drives cost growth

Focus on Measurement: Cost growth is an issue that must be controlled; however, the amount of cost growth has been significantly overstated.

8

DDG 51 Example

The DDG 51 is a multi-mission guided missile destroyer designed to operate offensively and defensively, independently, or as units of Carrier Strike Groups (CSG), Expeditionary Strike Groups (ESG), and Missile Defense Action Groups in multi-threat environments that include air, surface, and subsurface threats.  These ships will respond to Low Intensity Conflict/Coastal and Littoral Offshore Warfare (LIC/CALOW) scenarios as well as open ocean conflict providing or augmenting power projection, forward presence requirements, and escort operations at sea.

9

GAO Method: DDG 51

• GAO Method simply subtracts the Original estimate from the Current estimate to calculate the total program cost growth

• DDG 51 had a MS B in February, 1988. At that time, only 23 units were planned to be produced. The Dec 2009 SAR estimate shows that DoD now plans to procure 71 units

• The increase in unit cost for DDG 51 is relatively small; however it is incurring a huge cost growth penalty for its increase in quantity

• The increase in quantity was a leadership decision driven by the fact that this is a successful program – according to GAO, this is one of the worst performers in the inventory!

PAUC = Program Acquisition Unit Cost

Program Original Estimate to Current MS B Date

Original Estimate ($B)

Current Estimate ($B)

Cost Growth (GAO Method) ($B)

DDG 51 Destroyer Feb 1988 27.6 93.4 65.8

AT&L Method: DDG 51

10

• First, we eliminate historical bias by comparing the Current Estimate to the 5 year prior estimate

• Second, we separate cost growth due to changes in unit cost from cost growth strictly due to quantity increases

• The difference between GAO Method and AT&L Method is significant in many programs – for DDG 51 it is about a $61.5B difference

• We used this process for about 100 MDAPs, and saw that when quantity is considered, Actual Cost Growth becomes very concentrated among a few programs

PAUC = Program Acquisition Unit Cost

Program 5 Year Estimate to Current MS B Date

DEC04 Quantity

DEC04 PAUC ($B)

Current Quantity

Current PAUC ($B)

Unit Cost Growth ($B)

Quantity Cost Growth ($B)

Actual Cost Growth (ARA Method) ($B)

DDG 51 Destroyer Feb 1988 62 1.2 71 1.3 4.3 11.8 4.3

11

Oldest 10 Programs: GAO Method

• If we replicate the GAO Method using Dec 2009 SARs, the 10 oldest programs (out of a total of 98) account for $102.2B of cost growth

All data are in BY$10 Billions

Index Older Acquisition ProgramsFirst Original

APB

First Original Estimate

Dec 09 SAR

Estimate

GAO Cost Growth

($B) from MS B

1 TRIDENT II (D-5) MISSILE Jul 1987 50.4 50.9 0.52 DDG 51 Destroyer Feb 1988 27.6 93.4 65.83 V-22 OSPREY Feb 1988 31.4 55.5 24.14 FMTV Oct 1988 10.2 21.1 10.95 AMRAAM Dec 1988 16.4 23.8 7.46 C-17A Dec 1988 50.1 81.5 31.47 F-22 Raptor Feb 1992 89.0 76.6 -12.48 F/A-18E/F Jun 1992 79.7 54.1 -25.69 JSOW Baseline/BLU-108 Jun 1992 2.8 2.2 -0.610 CVN-77 Jun 1993 6.1 6.8 0.7

Sub-total ($B) 363.7 465.9 102.2

12

Oldest 10 Programs: AT&L Method

• From earlier slide, GAO method shows that the 10 oldest programs account for $102.2B of cost growth

• When we account for age of the program and quantity increases, the AT&L actual cost growth for these same 10 programs is $8.2B!

All data are in BY$10 Billions

Index Older Acquisition ProgramsFirst Original

APB

First Original Estimate

Dec 09 SAR

Estimate

GAO Cost Growth

($B) from MS B

AT&L Cost Growth

($B) last 5 years

1 TRIDENT II (D-5) MISSILE Jul 1987 50.4 50.9 0.5 2.12 DDG 51 Destroyer Feb 1988 27.6 93.4 65.8 4.33 V-22 OSPREY Feb 1988 31.4 55.5 24.1 1.84 FMTV Oct 1988 10.2 21.1 10.9 3.45 AMRAAM Dec 1988 16.4 23.8 7.4 2.36 C-17A Dec 1988 50.1 81.5 31.4 -5.87 F-22 Raptor Feb 1992 89.0 76.6 -12.4 2.08 F/A-18E/F Jun 1992 79.7 54.1 -25.6 -1.79 JSOW Baseline/BLU-108 Jun 1992 2.8 2.2 -0.6 0.010 CVN-77 Jun 1993 6.1 6.8 0.7 -0.3

Sub-total ($B) 363.7 465.9 102.2 8.2

-40

-20

0

20

40

60

80

100

1985 1990 1995 2000 2005 2010 2015

Portf

olio

Cos

t Gro

wth

$B

Milestone B Date

JSF$281B

C17A $82B

SSN 774$81B

V-22$56B

FMTV $21B

Trident II$51B

F-22$77B

F/A-18$54B

DDG 1000$20B

DDG 51$93B

MRAP$38B

Apache$14B

Blackhawk$22B

AMRAAM$24B

C-130J$15B

CH-47F$13B

13

Total Program Cost is represented by

the size of the circle

GAO Method: FY10 MDAP Portfolio

Using GAO Method, cost growth is concentrated in JSF and several very old programs, many of which have been performing well for some time.

-40

-20

0

20

40

60

80

100

1985 1990 1995 2000 2005 2010 2015

Portf

olio

Cos

t Gro

wth

$B

Milestone B Date

JSF$281B

C17A$82B

SSN 774$81B

Trident II$51B

F-22 $77B

F/A-18$54B

DDG 1000$20B

DDG 51$93B MRAP

$38B

CVN 21$34B

GMLRS$6B

14

Total Program Cost is represented by

the size of the circle

AT&L Method: FY10 MDAP Portfolio

Using AT&L Method, it is obvious that JSF and a small set of troubled programs account for the bulk of cost growth.

The Stats Matter!

15

Cost Growth or Capability Trade-off?

• There are numerous methods we can use to classify cost growth, but the real issue is determining what capability can be supplied to the warfighter in a given period of time

• Sometimes we are willing to “pay,” or incur cost growth for capability– Immediate warfighting needs – Necessary quantity or technology increase – Flexibility from numerous simultaneous acquisition programs– Efficient production schedules

• Other times cost growth is not willingly incurred– Poor initial cost estimates– Poor program management– Unrealistic expectations– Premature technology– Redundant capability

• Reducing the capability lost to the warfighter as a result of unplanned cost growth should be our primary focus!

16

F-22 Funding Example

The F-22 combines stealth, speed, maneuverability, integrated avionics, multi-role precision attack, and improved supportability.  It is a critical component of the Global Strike concept of operations with unmatched air-to-air and air-to-ground capabilities.  The Raptor ensures air dominance, rapidly and at great distances, by countering and defeating threats, including those attacking friendly forces and those attempting to deny access by our forces.

17

83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 '00 '01 '02 '03

FY85$M

F-22 RDT&E Program Funding

Total R&D Program CostRelative to Original Baseline

SAR Date

'04 '05 '06 '07

-40%-20%

0%20%40%60%80%

Dec-84

Dec-86

Dec-88

Dec-90

Dec-92

Dec-94

Dec-96

Dec-98

Dec-00

Dec-02

'080

500

1000

1500

2000

2500

3000

3500

4000

18

83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 '00 '01 '02 '03

FY85$M

F-22 RDT&E Program Funding

Total R&D Program CostRelative to Original Baseline

SAR Date

'04 '05 '06 '07

-40%-20%

0%20%40%60%80%

Dec-84

Dec-86

Dec-88

Dec-90

Dec-92

Dec-94

Dec-96

Dec-98

Dec-00

Dec-02

'080

500

1000

1500

2000

2500

3000

3500

4000

Original Estimate

Current Estimate

MS B: Feb 1992Original Unit Cost: $137MCurrent Unit Cost: $408M

19

Acquisition Reform Initiatives

• Weapon Systems Acquisition Reform Act (WSARA) of 2009

• MS A and MS B Certifications• IMROVE Acquisition Act: HR 5013• QDR & FY11 Budget• Acquisition Workforce Reform• Tactical Acquisition Reform• Strategic Acquisition Reform

20

WSARA Funding Requirements

• OSD(CAPE) responsible for developing guidance governing cost estimation and cost analysis processes, including confidence levels for cost estimates and conducting independent estimates/analyses for all MDAP and MAIS for which USD(AT&L) is the Milestone Decision Authority (MDA)

• D,CAPE and the Secretaries of Military Departments must disclose the confidence level used in cost estimates for MDAPs/MAISs

• If less than 80 percent, include such disclosure in any documentation approving a cost estimate, and in the next Selected Acquisition Report (SAR) or quarterly report

21

Funding to 80% Confidence Level

• Previously, MDAPs were funded so it was equally likely that actual costs will be higher or lower than the estimate

• Problems with funding to the 80% confidence level:– What is the underlying distribution?– Money into the program is more than we expect to spend -

is this a self-fulfilling prophecy?

• MDAP spending in 2011 is $77B – what would it be if we had to fund to 80% confidence level?

22

Ongoing Studies

• Numerous studies on program stability and portfolio cost growth have been done

• Common themes:– Funding Stability– PM Tenure– Accurate cost estimates– Requirements Creep

• AT&L is aggressively addressing cost growth:– Formation of a Value Team, headed by the PDUSD(AT&L)– MDAP Transparency Study to identify program costs in the financial

& acquisition databases

23

Conclusions

• When controlled for quantity and time, cost growth becomes a much more manageable problem for the Department

• Not all cost growth is bad. There are smart and necessary capability increases to platforms that result from deliberate & intelligent decision making for the benefit of the warfighter and National Security

• Large cost growth in individual acquisition programs is the exception, not the rule

• The Department continues to implement many policies & procedures to help control cost growth – results indicate they are working!

• The Stats Matter!