IMTEC-89-18 Space Defense: Management and Technical … · 2020. 9. 1. · Aerospace Defense Command’s ability to provide space surveillance and satellite attack warning ... monitor

Information Management and Technology Division

B-203028

April 20, 1989

The Honorable John P. Murtha Chairman, Subcommittee on Defense Committee on Appropriations House of Representatives

Dear Mr. Chairman:

In response to your predecessor’s request and subsequent discussions with your office, we assessed how well the Air Force has managed the Space Defense Operations Center modernization effort. The replacement system will be an integral part of the North American Aerospace Defense Command’s ability to provide space surveillance and satellite attack warning information to United States leaders. We found that the modernization effort has been marked by Air Force management problems, is significantly behind schedule and over budget, and has not met established requirements.

We are sending copies of this report to the Secretary of Defense; the Secretary of the Air Force; the Chairmen, House and Senate Committees on Armed Services; the Chairman, Senate Committee on Appropriations; the Director, Office of Management and Budget; and to other interested parties.

Sincerely yours,

Ralph V. Carlone Assistant Comptroller General

Ekecutive Summ~

Purpose The North American Aerospace Defense Command (NORAD), a binational command supported by the United States Space Command, is responsible for notifying United States and Canadian leaders that North America is under air, missile, or space attack. The former Chairman, Subcommittee on Defense, House Committee on Appropriations asked GAO to evaluate the Air Force’s efforts to modernize United States Space Command’s space surveillance and attack assessment subsystem-the Space Defense Operations Center (SPADOC). In response to this request, GAO primarily evaluated (1) whether Air Force management of this program has been effective and (2) the difficulties in meeting program requirements.

Background Cheyenne Mountain Air Force Station in Colorado Springs, Colorado houses the communications and data processing subsystems supporting the Integrated Tactical Warning and Attack Assessment (ITW/AA) system for the United States Space Command, which supports NORAD. SPADOC is intended to be a data processing and communication center that can monitor and maintain orbital information on up to 10,000 man-made objects in space, provide timely warning of any threat or attack, and protect satellites by identifying the need for satellite maneuvers.

The on-going SPADOC acquisition is divided into three evolutionary blocks (A, B, and C). Block A, which entered full scale development in 1983, is to provide the hardware and software to automate the assessment of foreign activities (such as a nuclear detonation) that might affect U.S. satellites. It also is intended to alert national decision makers of actual or potential threats or attacks, and plan and coordinate countermeasures.

Block B, which entered full scale development in 1986, before block A was completed, is to improve space surveillance by making orbital position predictions on about 400 satellites of particular interest to the Department of Defense. Block C, which has not yet been funded, is to complete the automated capability needed to consolidate United States Space Command’s space defense data processing functions.

The Air Force plans to spend about $437 million, up from its original $290 million estimate, to develop SPADOC. And although the entire acquisition was supposed to be installed and operating by June 1988, the Air Force now estimates the system will not be completed until fiscal year

Page 2 GAO/&l’l’EG89-18 Operations Center Acquisition Delayed

Executive Summary

1994. In September 1988, the Congress directed that all NORAD modernization programs, including SPADOC, be placed under Defense Acquisition Board oversight.

Results in Brief The SPADOC program has been marked by management problems, unreal- ized expectations, and program delays. The Air Force has invested over $235 million in a system that is now more than 4 years behind schedule and far from meeting its required operational capability.

Throughout 5 years of development, technical contractors warned the Air Force that the program would have difficulty achieving its requirements. The Air Force continued to press forward with the program despite these warnings, consistently deferring problem resolution to later development phases.

The Air Force accepted block A in April 1988, even though it did not satisfy most requirements for mission performance, The Air Force hoped the deficiencies would be resolved in block B. However, in the same month, the Air Force disapproved the block B design because it was not expected to meet performance requirements.

At the root of SPADOC’S technical problems is the Air Force’s attempt to achieve controlled mode security.1 Software development tasks designed to achieve this form of multilevel security are time-consuming, technically demanding, and still undergoing much research and development. In SPADOC’S case, functions such as notifying national decision makers that a satellite is under attack take as much as four times longer to complete than required.

Principal Findings

Problems Surfaced Early in Block A Development

Concerns about block A began to surface shortly after development began in April 1983. Mitre Corporation, the Air Force’s engineering support contractor, repeatedly told the Air Force that it was concerned about the adequacy of SPADO& security design, about inadequacies in a model used by the contractor to predict performance, and about overall

‘In the SPADOC system specification, “controlled mode” is defined as a type of multilevel security in which the system controls users access to and receipt of different levels of classified data.

Page 3 GAO/IMIIX-89-18 Operations Center Acquisition Delayed

Executive Summary

software design quality. At critical design review in March 1984, system design should have been complete but was not, However, the Air Force decided to continue development based on the incomplete design because it believed it was less risky than delaying development. This began a continuing pattern of approving key project milestones although requirements had not been satisfied and deferring problem resolution.

Correcting Block A Deficiencies Deferred to Block B

The Air Force accepted block A almost 3 years late, but even given this extra development time, the system still could not achieve controlled mode security and only met 7 of 23 required mission functions stated in the contract. The Air Force acknowledged the problems the contractor was having meeting the security requirement in September 1984 and deferred the security requirement to block B, In March 1988, the Air Force deferred meeting specific block A performance requirements to block B because the Air Force believed that even with fine tuning the system would not perform as required.

Problems Identified in When the Air Force began developing block B in June 1986, it knew that

Block A Are Continuing in many of the block A problems could seriously impair block B develop-

Block B ment. Although a different system performance model was being used, the Air Force was aware of concerns that the new model also could not accurately predict system performance. Further, the system design had not shown it could meet the performance and controlled mode security requirements. This inability of the system design to meet the requirements finally led to the Air Force’s disapproval of the block B design in April 1988, following critical design review and after almost 2 years of development.

The contractor has been allowed to continue developing the block B system, even though the Air Force has not approved the hardware configuration, software approach, or system design. As with block A, continuing on this track only serves to increase program costs and risks with no discernable benefit.

Page 4 GAO/IMTEC89-18 Operations Center Acquisition Delayed

ExecutiveSummary

Recommendations to GAO recommends that the Secretary of Defense halt block B development

the Secretary of Defense

until the Air Force has complied with congressional requirements to submit NORAD modernization programs, including SPADoC, to Defense Acqui- sition Board review. During that review, the Secretary of the Air Force should specifically submit to the board:

. Recommendations on the ultimate disposition of the SPADOC block A system, including an evaluation of the capabilities and deficiencies of the block A system as accepted and recommendations on how block A should be used based on careful analysis of costs incurred and benefits derived.

l Plans for the future SPADOC system including a thorough analysis of the requirements for SPADOC and the feasibility of satisfying those requirements, in particular controlled mode security.

GAO is making other recommendations to the Secretary of Defense in chapter 5.

Recommendations to GAO recommends that the Congress withhold further funding for the

the Congress SPADOC 4 acquisition until the Defense Acquisition Board has submitted and the Secretary of Defense has approved program plans meeting the objectives described in GAO'S above recommendations.

Agency Comments The Department of Defense agreed with most of the information presented in the report, but did not concur with several of the findings and conclusions and disagreed with GAO'S recommendations to withhold funding and halt development. Defense stated it has effectively managed the program and adequately assessed the program’s risks. How- ever, GAO'S report provides numerous examples where Air Force management did not resolve but continually deferred problem resolution. Further, GAO did not find evidence that the risk involved in achieving critical requirements was assessed. Finally, GAO'S recommendations to withhold funding and halt development are designed to minimize future cost growth and schedule delays and give the Air Force an opportunity to resolve complex technical problems confronting the SPADOC 4 program. Chapter 5 contains GAO'S evaluation of Defense’s comments.

Page 6 GAO/JMTEC89-18 Operations Center Acquisition Delayed

Contents

Executive Sun-u-nary 2

Cha.pter 1 8

Introduction The ITW/AA System 8 How SPADOC Fits Into the ITW/AA System 9 Organizations Involved With SPADOC 10 Objectives, Scope, and Methodology 11

Cha,pter 2 13

SPA.DOC 4 Is Not Yet SPADOC Performance Expectations 13

Operational and Is Security 13 Block A Is Not Yet Operational 14

Significantly Over The Air Force Rejected Block B’s Design 14

Cost and Behind Block C Is Planned but Not Yet Funded 15

Schedule SPADOC 4 Acquisition Costs Have Risen and Milestones 16

Have Been Delayed

Cha,pter 3 18

Air Force Deferral of Potential Development Risks Were Not Formally Assessed 19

Block A Problems During Requirement Setting

Resulted in a Marginal Problems Began to Surface Early in Block A’s Design 19

Phase

Sysrtem The Block A Development Phase Was Plagued With Technical Uncertainties and Schedule Delays

Operational Testing Further Highlighted Block A System Performance Shortfalls

21

25

The Air Force Accepted a Marginal Block A System 27

Cha.pter 4 28 Block B’s Design and Block B Development Was Risky Because of Incomplete 28

Development Has Block A Software

Proceeded Much Like Block B’s Performance Prediction Model, Like the Block A 29

Model, Is Deficient

Block A’s Disagreement Remains Over Whether the Security Requirement and QPRs Can Be Simultaneously Achieved

30

Block B Development Is Continuing Without Agreement on Whether Additional Computing Capability Is Needed

30

The Air Force Disapproved the Block B Design 31

Page 6 GAO/IMTEG89-18 Operations Center Acquisition Delayed

Contents

Chapter 5 Conclusions and Recommendations

Recommendations to the Secretary of Defense Recommendation to the Congress Agency Comments and Our Evaluation

33 34 35 35

Appendixes Appendix I: Current ITW/AA System 42 Appendix II: Modernized ITW/AA System 43 Appendix III: Comments From the Department of Defense 44 Appendix IV: Major Contributors to This Report 60

Tables Table 2.1: SPADOC 4 Cost Estimates 16 Table 2.2: SPADOC 4 Completion Dates 16

Abbreviations

AFOTEC Air Force Operational Test and Evaluation Center GAO General Accounting Office IBM International Business Machines IMTEC Information Management and Technology Division ITw/AA Integrated Tactical Warning and Attack Assessment NORAD North American Aerospace Defense Command QPR Quantitative Performance Requirement SPADOC Space Defense Operations Center


Chapter 1

Introduction

The Space Defense Operations Center (SPADE) is a centralized command and control facility being developed to support the surveillance of all man-made objects in space and to assess attacks on U.S. satellites. SPADOC is a major component of the United States Space Command’s Inte- grated Tactical Warning and Attack Assessment (ITW/AA) system- formerly called TW/AA. The ITW/AA system provides the U.S. Space Command and the North American Aerospace Defense Command (NOFLAD) with timely warning and assessments of any attack on North America, including attacks on satellites.

NORAD is a binational military command consisting of U.S. and Canadian personnel. American and Canadian leaders rely on NORAD to provide surveillance of North American airspace, and warn of bomber and missile attacks. NORAD is supported by the United States Space Command, a uni- fied command made up of three components: Air Force Space Command, Naval Space Command, and the United States Army Space Command, which oversee U.S. missile warning and space surveillance. The Air Force Space Command, as a component of U.S. Space Command, provides the majority of ground and space-based systems, equipment, and personnel that enable KORAD and U.S. Space Command to perform their missions. The Commander-in-Chief of U.S. Space Command also serves as the Commander-in-Chief of NORAD.

The ITW/AA System The command and control center for the ITW/AA system is the Cheyenne Mountain Air Force Station in Colorado Springs, Colorado. This facility houses data processing and communications equipment supporting the warning and assessment missions of U.S. Space Command and NORAD. The ITW/AA system, which the Air Force calls a “system of systems,” consists of air defense, space defense, and missile warning subsystems, as well as communication links, systems for correlating information, and display terminals.

Because of evolving mission requirements and threats, the ITW/AA system is constantly changing. The current modernization will upgrade the existing system, which consists of hardware and software predomi- nantly dating from the mid-1970s. Appendix I illustrates the current system, which is comprised of five major subsystems: (1) the Communi- cation System Segment, (2) Space Defense Command and Control Sys- tem, (3) NORAD Computer System, (4) Mission Essential Back-up System, and (5) the Intelligence Data Handling System. Appendix II shows the system as it will look when modernized by replacement subsystems, specifically with the Communication System Segment Replacement, the

Page 8 GAO/lMTEC89-18 Operations Center Acquisition Delayed

Chapter 1 Introduction

Command Center Processing and Display System Replacement, the Survivable Communications Integration System, Granite Sentry, and SPADOC 4. SPADOC 4 is the fourth upgrade to the SPADOC program designed to replace SPADOC 3 and the Space Surveillance Center, parts of the Space Defense Command and Control System. Although each of these modernization programs is required to interact with one or more of the others, all the programs are being developed, acquired, and installed separately.

How SPADOC Fits Into The Air Force initiated the SPADOC program to be a single command

the ITW/AA System center for all command, control, and communications, and data processing functions for space defense activities. The current system receives observations on satellites and other objects in space from radar and optical tracking sensors worldwide, processes this information to update satellite orbits, and upon request, provides satellite orbit and threat information to satellite owners and operators.’ Currently, much of the tracking information on man-made objects in space-from large satellites to screwdrivers and gloves left in space during space shuttle missions -and the information used to protect satellites from threats and attacks is processed manually within the Space Surveillance Center. A modernized SPADOC is intended to automate these functions and provide up-to-date satellite status information to other subsystems in Cheyenne Mountain, such as US. Space Command’s Space Command Center, and to national decision makers when the need arises. (See app. II.) SFADOC is being implemented in phases. As discussed below, SPADOC phases 1 through 3 are complete, and phase 4, the current modernization effort, is under development. SPADOC 4 is also being built in phases called SPADOC

4 blocks A, B, and C. Initially, the SPADOC 4 program was scheduled for completion by June 1988 at a cost of $289.6 million. However, due to development problems and schedule delays, the Air Force now anticipates the program will be completed in fiscal year 1994 at a cost of $437 million.

The SPADOC Mission and The SPADOC mission is to monitor space activities, inform decision mak-

Program Evolution ers of a threat or attack, and help protect satellites. The “monitor and inform” mission includes space surveillance, detecting and identifying threats to U.S. and allied space systems, and disseminating information to key decision makers. The “protect” mission includes assisting U.S.

‘A satellite owner is the agency that funded or built the satellite. A satellite operator is the agency that ensures the satellite is in its proper orbit, its orientation is correct, and that it is operating correctly.

?age 9 GAO/JMTEGWlS Operations Center Acquisition Delayed


satellite owners and operators to improve the survivability of their space systems by identifying and coordinating actions such as satellite maneuvers to avoid threats,

The SPADOC program began in 1979 and has evolved through three phases to date. SPADOC 1, completed in 1979, provided a location for SPADOC operations in Cheyenne Mountain, and established formal procedures and message formats2 for communicating with satellite owners and operators. SPADOC 2, completed in 1981, added a manual subsystem to manage the data base that provides information on thousands of objects in space. SPADOC 3, completed in 1983, added dedicated communication lines between the system and satellite owners and operators. The current SPADOC 3 functions as the command and control center for Chey- enne Mountain space defense operations.

SPADOC 4, the last SPAIKE modernization phase, will add data processing equipment to (1) computerize the existing, manually maintained, space object data base; (2) monitor and assess additional space activities (such as lasers and the effects of nuclear detonations); and (3) expand the ability to perform several space defense activities concurrently. Because SPADOC 4 will not perform all functions currently performed in the Space Surveillance Center until block C is complete, the Space Surveillance Center and SPADOC 4 will operate concurrently using the same space object data base until at least fiscal year 1994.

Organizations Involved With SPADOC

U.S. Space Command is the user of the SPADOC system, which is being acquired by Air Force Systems Command’s Electronic Systems Division. Air Force Space Command, a component of US. Space Command, will manage and maintain SPADOC and provide U.S. Space Command with personnel and equipment to accomplish its space defense mission. Ford Aerospace and Communications Corporation (hereafter referred to as Ford) is the SPADOC prime contractor and International Business Machines (IBM) is the major hardware subcontractor. Mitre Corporation is providing engineering support to Electronic Systems Division for SPADOC. Logicon, Incorporated is Electronic Systems Division’s independent validation and verification contractor for SPADOC, and is responsible for independently evaluating the system to determine whether it will satisfy mission and operational requirements.

2A message format defies the structure and organization of the data transmitted from the sensors, or radar, to computer subsystems.

Page 10 GAO/~G89-18 Operations Center Acquisition Delayed


Objectives, Scope, and This report is the second of three reports in response to a request from

Methodology the former Chairman, Subcommittee on Defense, House Committee on Appropriations that we assess the Air Force’s management and system integration of computer modernization programs for NORAD’S Cheyenne Mountain Complex.,’ This report assesses the Air Force’s progress in acquiring blocks A and B of the SPADOC 4 program (block C is planned but unfunded). Specifically, we evaluated (1) why block A does not meet Air Force requirements and is significantly behind schedule, (2) whether block A’s problems are likely to be corrected in block B, and (3) whether Air Force management of the program has’been effective.

To determine why block A does not meet Air Force requirements, we reviewed system requirement documents, contracts (including system specifications), test reports, and other program documentation, and discussed these documents and related information with program officials from the Air Force’s Electronic Systems Division, Air Force Space Com- mand, Ford, Mitre Corporation, and Logicon, Incorporated. We also discussed the relative importance of SPADOC system requirements with Air Force Space Command and U.S. Space Command officials. We compared test results with system requirements and specifications to determine the extent and causes of performance shortfalls, and discussed this information with Air Force and contractor officials. We also determined the extent and causes of schedule delays by reviewing program documentation and discussing this information with program officials.

To determine whether block B is likely to correct block A’s development problems, we reviewed system requirements and specifications, system design documentation, and other program documentation, and discussed the design and changes made or being considered with Air Force and contractor officials. We evaluated documentation pertaining to a model developed by the contractor to predict the system’s performance and discussed the model’s capabilities and deficiencies with Air Force and contractor officials.

To evaluate Air Force program management effectiveness, we reviewed information contained in contract and program files, minutes from program management reviews and working group meetings, and correspon- dence between the Air Force and the contractors. We interviewed Air Force and contractor officials to clarify the information contained in the

“Our first report, Attack Warning: NOfL4D’s Communications System Segment Replacement Should be Reassessed (Gfl that rendered that subsystem unable to meet established requirements. Our third report will address NORAD’s overall management trf the systems integration procf3ss.

Page 11 GAO/IMTEGWlS Operations Center Acquisition Delayed

Chapter I Introduction

documents reviewed, and to obtain more information about Air Force and contractor management of the SPADOC program.

We conducted our work at Air Force Headquarters and Air Force Sys- tems Command in Washington, D.C.; Air Force Systems Command’s Elec- tronic Systems Division at Hanscom Air Force Base, Massachusetts; US, Space Command and Air Force Space Command at Peterson Air Force Base, Colorado; Ford Aerospace and Communications Corporation and Logicon, Incorporated at Colorado Springs, Colorado; and Mitre Corpora- tion, Bedford, Massachusetts. Our work was conducted from February through September 1988. Information has been updated through November 1988. Our work was performed in accordance with generally accepted government auditing standards.


Chapter 2

SPmC 4 Is Not Yet Operational and Is Significantly Over Cost and Behind Schedule

SPADOC 4 did not become operational in June 1988 as originally planned and is significantly over cost and behind schedule* Block A was accepted by the Air Force in April 1988 although it does not meet all contractually specified requirements and is not yet operational.’ Correcting block A deficiencies was deferred to block B. Block B has been under development since 1986, but its design was rejected by the Air Force in April 1988 because it too was not expected to meet contractually specified requirements. A review of another design is scheduled for April 1989. While the Air Force has not yet obtained congressional funding for block C development, through August 1988 it has spent $235.8 million developing blocks A and B. Under the Air Force’s best scenario, the program will become operational at least 6 years late and $147 million over the original budget.

SPADOC Performance The Air Force has established certain performance requirements (called

Expectations quantitative performance requirements or QPRs) as the primary means for measuring system performance. QPRS specify how quickly the system should process incoming data and generate the required output messages. According to Air Force Space Command officials, the most important of these QPHS are 23 that are designated “mission related.” These mission related QPKs include specifications for system performance when:

. identifying and assessing threats from orbital interceptors, nuclear detonations, electronic warfare, and lasers; and

l disseminating timely information about those threats to officials and organizations in the space community, such as the NORAD command post and satellite owners or operators.

Security The SPADOC contract specified that block A operate in “controlled mode,” or be capable of evolving to this mode of operation. A system operating in controlled mode is intended to assure that users cleared at secret, confidential, or unclassified levels can access only the information to which they are entitled, that is, a system operating in controlled mode will not permit an unclassified user to access secret data.

‘The Air Force signed a DD Form 250 (contract acceptance form) accepting the system on behalf of the government. This form requires the program manager to certify that the system meets contract requirements. The system becomes operational after it successfully completes operational test and evaluation.

Page 13 GAO/lMTEG89-18 Operations Center Acquisition Delayed

Chapter 2 SPADOC 4 Is Not Yet Operational and Is Signiticantly Over Cost and Behind Schedule

Block 4 Is Not Yet Operational

Although block A has been in development since April 1983, it can not perform 14 of 23 required mission functions quickly enough to satisfy contractually specified quantitative performance requirements.2 Some of these functions, such as notifying national decision makers that a satellite is under attack, take as much as four times longer to complete than specified. Additionally, the system does not automatically ensure that confidential or secret information will not be sent to unauthorized satellite owners and operators. This function is done manually under the existing system and will continue to be done manually with block A.

The Air Force concluded that these difficulties would have an impact on SPADOC’S completion schedule, and would have to be resolved during block B development. In September 1984, the Air Force deferred meeting the controlled mode security requirement to block B. In April 1988, according to the Vice Commander, Electronic Systems Division, the Air Force deferred meeting the quantitative performance requirements to block B and accepted the block A system to “get some minimal and marginal capability into Air Force Space Command’s hands.”

When block A was accepted, 295 other deficiencies and enhancements had been identified by the Air Force, Mitre, and Logicon. Ford has agreed to correct 90 deficiencies that Electronic Systems Division identified as significant and as needing correction before the system can become operational. No plans have been made to address the 205 enhancements because Electronic Systems Division does not consider them to be contract requirements. Testing was conducted between Sep- tember and December 1988 to verify that the 90 deficiencies were corrected. Air Force Space Command is supposed to finish operational testing to establish block A’s operational capability in March 1989. The Air Force currently anticipates declaring block A operational in April 1989.

The Air Force Rejected Additional space surveillance functions are to be automated in block B.

Block B’s Design Block B is to establish a fully automated space object data base that can catalog at least 10,000 objects, including the 7,000 objects currentlv

Y

maintained in the Space Surveillance Center. Block I3 is to maintain and update orbital data and make orbital position predictions on about 400 satellites of particular interest to the Department of Defense. Block B is

‘The numerical values for the quantitative performance requirements are classified, and therefore not provided in this report.

Page 14 GAO/lMTEGS9-18 Operations Center Acquisition Delayed

Chapter 2 SPADOC 4 Is Not Yet Operational and Is Significantly Over Cost and Behind Schedule

also expected to process up to 100,000 observations of satellite positions per day, double the 50,000 observations normally processed currently.

In June 1986, the Air Force modified the SPADOC 4 contract to include block B, with initial operational capability scheduled for January 1989. In December 1987, Ford presented its block B design for Air Force approval. The design incorporated proposed design changes to overcome block A performance problems. Ford proposed that the computer hardware be upgraded, and many system requirements be reduced because, in Ford’s opinion, the contract requirement to concurrently meet the QPRS and controlled mode security specified in the contract could not be met, even with upgraded equipment. In April 1988, after almost 2 years of development, the Air Force rejected Ford’s block B design proposal, stating that it was not expected to meet the QPRS. The Air Force emphasized to Ford that it is required to deliver a system design that will meet all operational requirements.

In August 1988, Ford agreed to redesign block B in another attempt to meet all SPAWC operational requirements. Another design review has been scheduled for April 1989 to evaluate Ford’s revised block B design proposal.

Block C Is Planned but Block C is intended to complete the transition from the existing Space

Not Yet Funded Surveillance Center system and provide the automated capability needed to manage U.S. Space Command’s space defense mission. Block C is to maintain and update orbital data, and make orbital position predictions on at least 10,000 potential objects in the space object data base. When block C is completed, the Space Surveillance Center will be retired and its functions will be performed by SPADOC 4.

Under current Air Force schedules for modernizing the ITW/AA system, block C is to be operational in 1994 to complete the transition from the existing system. As of February 1989, the Air Force had not obtained congressional funding nor made specific plans to award a contract to design and develop block C.


Chapter 2 SPADOC 4 Is Not Yet Operational and Is SigkfbntIy Over Cost and Behind ScheduIe

SPADOC 4 Acquisition The estimated cost to acquire SPADOC 4’s three blocks has risen from

Costs Have Risen and $289.6 million in May 1982 to $437 million, as of March 1988, an increase of about $147 million. The differences in these estimates, bro-

Milestones Have Been ken down by appropriation, are shown in table 2.1,

Delayed

Appropriation Research, Development, Test and Evaluation

Other Procurement

Operations and Maintenance Military Construction .- Total

Estimate as of: May 1982 March 1988 .-

$106.7 $315.8 .._ 151 7 114.0

31 2 5.5

$289.6’

1.7

$437.0

In February 1989, the Air Force said it was re-estimating SPADOC’S cost in preparation for the fiscal year 1991 budget request. Preliminary Air Force estimates indicate the cost to complete the SPADOC 4 program may increase to $446 million.

In addition to increasing costs, the scheduled completion of SPADOC 4’s three blocks has been delayed significantly. A comparison of the original and current completion dates for each block is shown in table 2.2.

table 2.2: SPADOC 4 Completion Dates Scheduled completion

SPADOC 4 As of May 1982 Current ..- Block A December 1984 April 1989 “._ Block B December 1986 March 1990 Block C June 1988 fiscal 1994 year

SPADOC'S original cost estimates did not meet the expenditure thresholds that would have required Defense-level oversight throughout development. As cost estimates rose above these thresholds early in the SPADOC contract, Defense and the Air Force did not reconsider the original decision to manage the program without formal Defense oversight. In Sep- tember 1988, the Congress directed that all of NOR&S modernization

Page 16 GAO/IM’IEW39-18 Operations Center Acquisition Delayed

Chapter 2 SPADOC 4 Is Not Yet Operational and Is Sigui&antly Over Cost aud Behind Schedule

programs, including SPADOC, be consolidated and placed under the oversight of the Defense Acquisition Board.:’ Congress further required that c

the board conduct a management review of the consolidated program 1 during fiscal year 1989 and report the results to the Congress. ? t

‘)Making Appropriations for the Department of Defense, House of Representatives Conference Report, 1

Report No. 100-1002, (100th Cung., 2nd Fess., Sept. 28, 1988). 1

Page 17 GAO/IMTJ3X418 Operations Center Acquisition Delayed r t

Chapter 3

Air Force Deferral of Block A Problems Resulted in a Marginal System

From the outset of the block A acquisition, the Air Force was aware that achieving some block A requirements would be risky, and because of this, the program needed close management oversight. However, the Air Force did not adequately assess and mitigate early risks and later allowed the program to continue without resolving identified problems or providing adequate program management. During design and development, the Air Force was frequently alerted by its technical support contractors of their concerns about the progress of the security design, the adequacy of the SPADOC performance prediction model, overall software design and quality, and the possibility that the system might not meet performance requirements. Nevertheless, the Air Force allowed Ford to continue developing block A without solving known problems. Operational testing verified significant block A problems, yet the Air Force accepted the block A system and deferred unmet performance requirements to block R.

The Department of Defense recognizes the need to manage information systems development and acquisition by (1) defining discrete system development phases, (2) requiring certain activities to take place in each phase, (3) establishing time frames for each phase, and (4) requiring management reviews and approvals of each phase to ensure that systems being acquired will satisfy mission needs in a cost effective man- ner. The phases that SI’AIXK block A proceeded through can generally be described as requirements setting and concept development, design, development, test, and acceptance. Ideally, during these phases (1) requirements are identified and prioritized, and concepts proposed to achieve the requirements are assessed to determine their risk and feasibility; (2) a specific system design is identified and assessed to assure that it can satisfy performance requirements; (3) the system design and computer programs are developed and integrated; (4) the system under- goes development and operational testing to assure that it meets contract specifications; and (5) the system is accepted for operational use, However, the Air Force permitted block A to pass through each of these phases without resolving critical concerns at established decision points.

Page 18 GAO/IMTEC-841S Operations Center Acquisition Delayed

Chapter 3 Air Force Deferral of Block A Problems Resulted in a Marginal System

Potential Development The requirement setting and concept development phase is supposed to

Risks Were Not Formally Assessed Dur:ing Requirement Setting

identify and prioritize users’ functional requirements and evaluate alternative proposals for addressing these requirements. During this phase, trade-off analyses are conducted to clarify and refine the functional requirements and to assess the functional and technical feasibility of attaining them, in order to reduce program risk and costs. Techniques such as modeling and simulation may be used in evaluating alternative concepts, particularly when extensive software development may be required.

The Air Force developed detailed requirements for block A, given U.S. Space Command’s space defense mission. Two major requirements for block A were that the system operate in controlled mode and that it satisfy quantitative performance requirements.

The security level required for block A had not been achieved by any system in 1983, when the SPADOC contract was awarded. Software development tasks designed to achieve this form of multilevel security are time-consuming, technically demanding, and still undergoing much research and development. Controlled mode security creates a much larger processing load, which slows system performance. Both Mitre and the Air Force recognized that attempting to implement controlled mode security would be risky and one of the two contractors competing for the SPADOC 4 development contract stated that satisfying the security requirement would put the SPADOC 4 acquisition at risk. However, the Air Force did not formally study the requirements for controlled mode operation to determine its achievability or its impact on system performance before proceeding into the design phase.

Problems Began to The purpose of the design phase is to translate functional requirements

Surface Early in Block and system concepts into a detailed design and to validate the selected d esign.

A’s Design Phase Modeling and simulation techniques are used to refine require-

ments and complete the system design. The Department of Defense requires critical design reviews to formally review the detailed design to ensure that the system will satisfy performance requirements.

As early as August 1983, only 4 months after contract award, Mitre raised concerns about Ford’s approach toward controlled mode security,



and in October 1983, questioned a performance prediction model1 that Ford developed for predicting the performance of the system Ford was designing. Despite these indications of development problems, the Air Force chose to continue beyond design and into the system development phase.

Security Problems Surf aced Early

In August 1983, Mitre pointed out that Ford’s system design did not meet security requirements. Specifically, Ford had not included provi- sions in the system for storing certain levels of classified data (e.g., unclassified, confidential, or secret). Further, in February 1984, Mitre stated in a letter to Electronic Systems Division that Ford’s approach to protecting security-relevant software from unauthorized modification was inadequate and did not comply with the contract. In a March 1984 letter to Electronic Systems Division, Mitre reemphasized its concerns about the lack of protection given to prevent unauthorized modifications to security software.

Validity of Performance Model Questioned

The block A system specifications required Ford to provide a system performance model to help guide SPADOC 4 design and development. Using different potential work loads, the model was to predict how well alternative system designs could perform message handling, data base update and retrieval, computations, and other operational functions. During block A’s design, Ford’s model predicted that the proposed block A system design would meet all requirements, including security and QPRS. However, as early as October 1983, Mitre told Electronic Systems Division that the model did not accurately reflect Ford’s system.

Significant Concerns Identified During Design Review

A critical design review is supposed to establish the integrity of a sys- tern design by requiring the contractor to present a complete design and demonstrate that it meets performance criteria before proceeding into software coding. A primary product of the critical design review is specific documentation to be used by computer programmers during software coding. At the conclusion of block A’s critical design review in March 1984, over 350 design problems had been identified and changes were still being made to the block A design. Following this review, Mitre

‘A model, in this context, is a complex computer program that simulates how a system behaves. If the simulation is accurate, the model can be used to predict how the system will behave under different conditions.

Page 20 GAO/lMTEGS9-18 Operations Center Acquisition Delayed


informed Electronic Systems Division that the risk of not completing block A within established time frames had increased.

Mitre informed the Air Force following the critical design review that block A’s risk had increased because of Ford’s changing software design, incomplete software development controls (lack of detailed coding guidance>, and an unrealistic software development schedule. Two software units were incomplete-one controlling the SPADCC functions related to the 23 mission-related QPRS and the other controlling operator interactions with the system. In many cases, the design Ford presented at the design review differed significantly from earlier versions presented to the Air Force. Also, according to Mitre, Ford’s proposed controls did not contain sufficient detail on how code should be written. This guidance was therefore inadequate for programmers to begin coding. Further- more, Ford planned to accelerate the production of code to a level 45 percent faster than had been achieved on a previous, less complex, Ford software development effort, leading Mitre to question the reasonable- ness of Ford’s software development schedule. In addition, Logicon reported that Ford’s test documentation failed to identify specific test approaches, particularly for critical functional areas and the QPRS.

In all, over 350 concerns were identified at the critical design review. They provided early indications of system design immaturity and portended the development problems experienced during block A. How- ever, despite the deficiencies identified at the critical design review, the Air Force decided that the risk of postponing further development until design issues could be resolved was greater than the risk of proceeding with an admittedly incomplete design. As a result, the Air Force allowed Ford to enter into the development phase, without having resolved the 350 concerns identified during the critical design review. This began a pattern of deferring problem resolution that has continued to date.

The Block A Development Phase Was Plagued With Technical Uncertainties and Schedule Delays

During the development phase, the computer software programs needed to perform the required functions are written and integrated into the system design. At the conclusion of system development, component and system level tests are conducted to validate that system performance meets development specifications.

Mitre and Logicon raised concerns regarding Ford’s approach to block A throughout the development phase, which began in 1984. These concerns involved the quality and timeliness of software development, the

Page 21 GAO/tMTEGW-18 Operations Center Acquisition Delayed


implementation of security requirements, the adequacy of Ford’s performance prediction model, and block A system performance. However, notwithstanding clear indications during development that block A would not meet performance requirements, the Air Force permitted Ford to continue with its development and, in 1986, begin the block B effort+

Softvvare Development Fe1 Behind Schedule and Was of Questionable Quality

1 Logicon and Mitre continually raised concerns to the Air Force about the timeliness and quality of Ford’s software development. Following the critical design review in March 1984, Mitre predicted a &month delay in the block A schedule primarily because Ford’s design was incomplete and still changing. By the end of September 1984, Mitre predicted the delay would grow to 8 to 13 months due to software development problems.

According to Logicon, Ford’s software development documentation showed that many internal deadlines had been missed. Early tests were repeatedly postponed, and planning for software tests proceeded slower than expected, Logicon said.

Logicon analyzed Ford’s software periodically throughout block A development and expressed concerns to Electronic Systems Division about Ford’s software quality. For example, in December 1985, Logicon identified 8.94 errors per thousand lines of code in one software unit, which Ford supposedly had tested successfully. Logicon compared this error rate to the 1 to 2 errors per thousand lines of code typical in analyses of other programs they had conducted. Logicon concluded that although Ford’s quality assurance and software configuration management procedures were satisfactory, they were apparently not being enforced. As late as March 1987, Logicon was still reporting concerns about the number of errors found in Ford’s tested block A software. Fur- ther, since Logicon checked only part of the software, and Ford was required to correct only the specific errors identified, Logicon told the Air Force that it was likely that the completed block A software contained a large number of undetected errors, subjecting the system to a high risk of poor performance or inconsistent results.

According to Logicon’s July 1987 final block A report, Ford’s test schedule had little built-in margin to accommodate delays in software deliv- eries or required retests; therefore, block A’s test program posed a



significant schedule risk. Logicon also noted that some software development controls were relaxed to try to meet schedules, and that, if controls had been maintained, it is likely that the software would have had fewer errors. Logicon concluded that the high error rate in block A software led to problems integrating the software, and that it is likely that the block A code still contains undetected errors that could lead to similar integration problems during block B development.

Security Requirement Throughout block A development, Mitre raised concerns that Ford’s

Achievability Questioned security design would not meet contractual requirements. As previously discussed, Mitre considered Ford’s approach for isolating security software and for protecting security labels to be security design problems. Mitre was concerned that Ford’s design would not achieve the controlled mode requirements.

In September 1984, Air Force Space Command acknowledged the problems Ford was having meeting the security requirements. The Command changed the requirement for block A to operate in controlled mode, instead substituting a requirement that the system operate at system high secret level.’ Air Force Space Command agreed that operating the system at system high secret level was acceptable for block A because the message volume would be low enough that outgoing messages could be manually reviewed before release. However, the requirement for controlled mode security in block B was retained because, according to the Air Force, the volume of outgoing messages is expected to increase dra- matically under block B, making a manual security review of each message impractical. Although block A was no longer required to achieve controlled mode, the software needed to achieve this capability had to be retained in the block A system because it would become part of block B when completed.

Ford, Mitre, and Logicon independently informed the Air Force that the design features implemented to achieve controlled mode security are a primary cause of the degraded system performance, although no one has measured the extent of degradation. Ford estimated the performance degradation to be 20 to 50 percent, by analyzing a functional string’ of block A code. Mitre estimated, through a similar analysis, a

“A system is operating at system high secret level when, although it may contain inform&ion at the unclassified, confidential, and secret levels, all information is protected as if it were secret and all personnel with access to the system must have a secret clearance.

‘]A functional string of code is a series of software modules that performs a particular funct,ion



degradation of as much as 25 percent, According to the Mitre SPADOG project leader, Mitre did not conduct any further analyses because it believed that Ford was going to undertake such a study. Ford, however, stated that Electronic Systems Division has never told it to assess the risks or effects of controlled mode security. Therefore, the actual effect on system performance of continuing to strive for controlled mode security remains unknown. However, according to Logicon, if less stringent security requirements had been specified, it is likely that a simpler system design could have been developed and that this design might also have exhibited fewer performance problems.

Concerns Were Raised About the Adequacy of Ford’s Performance Prediction Model

During block A’s development, Mitre and Logicon continued to raise con- terns that Ford’s model was predicting inconsistent results and questioned whether it could accurately predict performance. In March 1984, Mitre evaluated the model’s assumptions and reported to the Air Force that it could not validate the model’s credibility. Logicon’s July 1987 final report on block A stated that the assumptions underlying the model and its results have never been validated. Logicon reported that the model’s fidelity (detail) may have been sufficient to discriminate among broad design alternatives, but it produced insufficient detail to accurately predict actual system performance. Logicon recommended that more work be done to validate the model and increase the fidelity by modeling system features more accurately. Even though Mitre and Logicon raised significant questions about the adequacy and accuracy of Ford’s model, we found no indication that the Electronic Systems Divi- sion’s program manager took any action to correct the model.

Performance Problems Persisted Through the Development Phase

Mitre informed the Air Force as early as May 1984 that block A might not achieve the required performance levels. However, it was not until summer 1986 that Mitre observed early system testing and confirmed that the system would have difficulty meeting the QPRS. No quantitative measurements were collected to assess system performance during these tests, so Mitre proposed an evaluation plan to identify ways to improve system performance.

In September 1986, Mitre and Ford began tests to identify ways to improve those computer processing functions that caused the system to perform so slowly. Mitre and Ford determined that performance could be improved by: (1) adding eight megabytes of system memory (increasing from 16 to 24 megabytes) to improve overall system processing speed; (2) revising software-controlling operator interactions with the



system and moving this from SPADX’S main processor to a peripheral processor to improve system response to operator commands; and (3) improving the efficiency of software controlling access to the data base. Although some of these changes resulted in faster system performance, it was still below that required to meet the QPRS,

Although significant development problems still had not been resolved, the Air Force determined that the block A system development tests and subsequent deficiency corrections had reduced the block A errors to a “manageable level.” As a result, the Air Force decided to proceed with Air Force Operational Test and Evaluation Center (ARYFEC)~ testing in Air Force Space Command’s test facility in June 1987.

Operational Testing Further Highlighted Block A System Performance Shortfalls

Operational testing of a completed system is conducted to ensure that the system meets the user’s functional requirements and is ready for operational use. Tests conducted by AITTEC and Air Force Space Com- mand only confirmed what the Air Force had been told repeatedly since 1984, that block A could not meet performance requirements. AFOTEC found that block A “failed to achieve an operational capability,” and concluded that long term efforts would be needed to achieve that capability.

AFUI’EC Tests Identified ANTEC’S testing was conducted to determine the readiness of block A to

Significant Operational support operations, and was structured to address critical operational

and Performance Problems issues including: system responsiveness, validity of system output, and the level of operational capability provided by block A compared to SPALIOC 3. Formal testing was conducted over a 3-day period; however, AFDTEC continued to collect data for the next 3 months. AFWEC’S final report, issued in December 1987, contained the following observations:

l System responsiveness was not adequate; it could be overloaded during low levels of activity. For example, block A was unable to handle day-today message loads. As a result, it lost 10 percent of incoming message traffic under routine conditions, and was too slow to meet the system’s mission.

4AFWEC was the agency responsible for planning, directing, and conducting initial operational test and evaluation for SPADOC 4 and providing an independent evaluation of the operational readiness of the system.

Page 26 GAO/IMTJXSD-18 Operations Center Acquisition Delayed

Chapter 3 Air Force Deferral of Block A Problems Resulted iu a Marginal System

. System output for six of the eight astrodynamic programs5 was not suf- ficiently accurate to meet user requirements. In addition, the system’s ability to monitor and inform decision makers on events in space was hampered by poor message composition and handling, inaccurate message routing, and inconsistencies and inaccuracies in the data base.

l Block A did not perform any mission critical functions significantly better than SPADOC 3. For example, critical warning messages could not be sent to users significantly faster than SWDW 3. In addition, the attempt to achieve controlled mode security impeded many system operations and did not help classify data entered into the system.

Performance Problems ConGnued During Air Force Space Command - ‘I’est:ing

Beginning in January 1988, Air Force Space Command began testing block A in Cheyenne Mountain using the same test procedures used by AFVFEC. Primarily, this testing was to verify that Ford had corrected the deficiencies previously identified by AFOTEC. However, not all identified deficiencies had been corrected and testing was suspended in February 1988 to allow Ford to make additional corrections. Mitre reported on some of these early test activities.

According to Mitre reports of Air Force Space Command’s test, block A had serious performance problems. For example, Mitre observed a con- tinuous 48-hour test of block A in which messages were to be received from but not transmitted to external agencies through Cheyenne Moun- tain’s communications center. As a result of these tests, Mitre reported that the system was so slow at several points that it was virtually impossible to interface with it through operator consoles. For example, one request to retrieve a single message, which should have taken less than 10 seconds, took over 18 minutes to execute. Mitre also reported that at one point it appeared that two operator consoles were “locked up.” In fact, the systems weren’t locked; the response took over 1 hour because the system was busy sorting out messages containing errors as well as recording system performance monitoring and evaluations called for under controlled mode. Mitre attributed the slow response primarily to (1) heavy message loads, (2) the system’s inability to handle backlog- ged messages from the communications center, and (3) recording performance monitoring and evaluations on tape.

Although many of these problems have been corrected over the past 2 years and Ford continues to work on fixing others, the system’s performance remains slower than required. According to Air Force Space

‘Mathematical formulas uwd tu calculate orbital information.



Command, the astrodynamic programs’ accuracy has been corrected, the system’s ability to handle the required message load has been improved but not totally corrected, and overall performance has improved but still does not meet requirements. Initial operational test and evaluation was resumed in February 1989 to verify Ford’s corrections. This testing was completed in March 1989. Air Force Space Command plans to issue a final test report by May 1989.

The Air Force On the basis of system performance demonstrated in Ford’s tests, the

Accepted a Marginal Air Force deferred achieving the QPRS from block A to block B and accepted block A on April 21, 1988. In the Ford demonstration test,

Block A System block A met 7 of the 23 mission QPRS. The time needed to perform the required functions for the QPRS that were not met exceeded the required values by as little as 5 percent to as much as 600 percent. In a March 1988 letter to Electronic Systems Division, Air Force Space Command stated that, given block A test results and numerous discussions with Electronic Systems Division, it was clear that Ford could not meet the QPRS soon. Air Force Space Command made it clear, however, that deferral of the block A requirements was not intended to reduce or relax block B contractual performance requirements and that the specified QPRs must be achieved in block B.

U.S. Space Command says block A is only “marginally acceptable,” however it offers a “definite improvement” over SPADOC 3 in the areas of nuclear event threat analysis, multiple threat processing, directed energy threat analysis, and overall satellite data management. SPADOC 3 either cannot perform these functions, or the operators must perform them manually. For these reasons, U.S. Space Command is prepared to declare the system operational at the conclusion of testing.

page27 GAO/RvITJSG89-18 Operations Center Acquisition Delayed

Block B’s Design and Development Has Fbeeeded Much Like Block A’s

In June 1986, the Air Force modified the SPADOC 4 contract with Ford to develop block B. The Air Force allowed block B’s design and development to continue for more than 2 years even though (1) block B was built on unstable’ block A software; (2) strong indications existed that the performance prediction model, while improved, was still deficient; (3) Ford and the Air Force remained at a standoff about whether the controlled mode security requirement and the QPR requirements could simultaneously be achieved; and (4) Ford and the Air Force could not agree on whether or how to increase block B’s computing capability to achieve the QPRS. In April 1988, with 70 percent of the block B software written, the Air Force disapproved Ford’s block B design, stating that it could not meet the QPRS required by the contract. Ford has been revising the design since August 1988, and a second critical design review has been scheduled for April 1989. Additionally, block B software programs were about 80 percent complete as of August 1, 1988; Mitre estimates that some of this software will have to be rewritten to be compatible with the new design.

Block B Development When Ford began developing block B software, it was still making sig-

Was Risky Because of nificant changes to the block A software. Since the block B software would incorporate the ever changing block A software, the engineering

Incomplete Block A support contractor repeatedly warned the Air Force that developing

Software block B software posed serious risks. The contractor was concerned that block B, when completed, would include unstable block A software, which would make it difficult to integrate the two blocks into a total system.

As early as January 1986 (before block B development began in June 1986), Mitre identified the need for planning to assure that block B would not be built on unstable block A software. Mitre informed the Air Force, in March 1986, that going far with the block B design before completing block A software would constitute a risk to system design and development. In June 1986, Mitre again raised concerns about merging unstable block A software with the software to be developed in block B. Mitre continued to state its concern to the Air Force in monthly project activity reports. Finally, as recently as March 1988, Mitre again told the project manager of its continuing concern about integrating block A and

‘Unstable software is a term applied to software that is unpredictable, that may or may not perform as expected, or may not produce consistent results when run against a known set of operating conditions.


Chapter 4 Block B’s Design and Development Has Proceeded Much Like Block A’s

block B software. Notwithstanding these persistent concerns by its engineering support contractor, the Air Force allowed Ford to continue developing block B, even though the block A software was unstable.

Block B’s Performance Ford developed a new performance prediction model for block B. How-

Prediction Model, Like ever, although the model was improved, Mitre and Logicon repeatedly raised concerns to the Air Force about the new model’s fidelity (detail)

the Block A Model, Is and validity (reliability). While the Air Force was aware of these con-

Deficient terns with the block B performance prediction model, it continued to allow Ford to use the model to predict the relative performance of alternative system designs and to make critical decisions based on data provided by the model.

The improvements Ford made to the model in block B increased fidelity somewhat. For example, to increase fidelity, Ford added details such as the effects of operating system overhead and paging on system performance-significant effects that were not included in the block A model. Ford also attempted to validate the model by comparing the actual performance of one functional string of code from block A to the performance for that string as predicted by the model.

However, these improvements did not overcome Mitre’s and Logicon’s concerns about the reliability of the model’s results. Mitre, in a February 1988 project activity report, told the Air Force that the model was substantially better than the block A model, but still contained a number of deficiencies. According to Mitre, the model did not provide enough detail on the operators’ interaction with the system or enough detail about how the system accesses the data base. Further, Ford had not incorporated changes made to block A into the block B model. According to Mitre, these deficiencies were enough to make model results unreliable. Logicon, in its July 1987 block A final report, stated that while more attention was being given in block B to the performance prediction model, no assurances existed that its results would be substantially more reliable. Logicon staff who assessed the block B model maintained that, as with the block A model, fidelity and validation continued to be major concerns. Because of doubt cast on the model, the Air Force still does not have reasonable assurance that the system, if built to the current design, will meet critical specifications.

Page 29 GAO/IMTEC-89-18 Operations Center Acquisition Delayed

Chapter 4 Block B’s Design and Development Has Proceeded Much Lie Block A’s

Disa,greement Remains Over Whether the Secwity Requirement and 1QPRs Can Be Simultaneously Achieved

Block B Development Is Continuing Without Agreement on Whether Additional Computing Capability Is Nleeded

Both the Air Force and Ford have known since July 1987 that the design used to achieve controlled mode security requirements slowed system performance, but neither party has assessed the actual extent of the degradation. Further, throughout block B development, the Air Force and Ford have not established whether a system can be built that operates in controlled mode and meets the QPRS.

Since block B development began, Ford has maintained that it can not simultaneously achieve the controlled mode security requirement and the QPRS, because in Ford’s opinion, the processing time it takes to perform the security function degrades system performance by as much as 50 percent. The Air Force, on the other hand, has maintained that a controlled mode computer system is needed to reduce the risk of security breaches in block B and that all QPRs are needed (with the exception of two that are being re-evaluated) to achieve specific SPADOC mission requirements. The Air Force contends the contract requires Ford to produce a system that operates in controlled mode security and meets the QPRS. However, as discussed in chapter 3, neither Ford, Mitre, or Logicon has accurately demonstrated the degree to which the security design actually affects system performance. In the absence of this information, further decisions on the block B design will, at best, be based on conflict- ing opinions.

As it did throughout block A, and up to critical design review in the block B acquisition, the Air Force allowed development to continue without resolving critical issues. According to Ford, it repeatedly told the Air Force that it could not meet block B requirements without increased computing capability. However, block 13 development proceeded without agreement between the Air Force and Ford on whether increased block B computing capability was needed, and if so, who would pay for it.

At the February 1987 block B preliminary design review, Ford presented a design consisting of four IBM model 3083 processors, along with a proposal to upgrade these machines to IBM model 3081s. The IBM 3081 machine contains two central processing units, as opposed to the single- processor IBM 3083. According to Ford, this increase from one to two central processors per machine would almost double the computing capability and substantially enhance system performance. Ford believed that the additional computing capacity would enable it to meet the contract requirements. Ford contends that it told Electronic Systems Divi- sion in February 1987 and again in April 1987 that it could not meet all

Page 30 GAO/JMTJ3GS9-18 Operations Center Acquisition Delayed

Chapter 4 Block B’s Design and Development Has Proceeded Much Lie Block A’s

the SPADOC contract requirements using the IBM 3083 machines, and that they should be upgraded to more powerful model 3081 machines.

In May 1987, Electronic Systems Division approved Ford’s preliminary design of four IBM 3083s, and Ford continued to develop block l3 under an assumption that its proposal to upgrade the computers would be accepted and implemented. The Air Force was not convinced that insufficient computing capacity was the reason that the QPRS could not be met, nor did the Air Force agree that the contract required it to pay for upgraded computers. The Air Force contended that Ford had not conclu- sively shown that additional computing capacity would solve the SPADOC 4 performance shortfalls. The Air Force contended that Ford’s system design was deficient, and that Ford’s software approach was inefficient. Additionally, the Air Force contended that the contract requires Ford to pay for any increased computing capacity needed to meet contract requirements.

Ford states that it again notified Electronic Systems Division, both before and during the December 1987 critical design review, that it would not be able to meet the performance requirements without the upgraded computers. Accordingly, Ford’s block B design presented at the critical design review was based on the upgraded computers (IBM model 3081) and on a November 1987 Ford proposal to relax the QPRS by increasing the time specified to perform the required functions because Ford could not meet the original requirements.

The Air Force Disapproved the Block B Design

On April 1, 1988, the Air Force declared Ford’s block B design unaccept- able and disapproved it. Ford’s design, presented at critical design review in December 1987, was based on two assumptions: (1) that the government would pay to upgrade the computers to IBM 3081s and (2) that the government would approve Ford’s proposal to relax the QPRS. Electronic Systems Division rejected both assumptions. According to Electronic Systems Division, Ford is responsible for delivering a system that meets all contractual requirements and that if upgrading the computers is required, Ford must bear the cost of the upgrade. Additionally, after Air Force Space Command validated all but 2 of the 23 mission related QPRS, Electronic Systems Division disapproved Ford’s request to relax the QPRS because such a reduction would result in an operationally unsuitable system. The Air Force instructed Ford to propose an alternative system design that would meet contract requirements. Ford has

Page 31 GAO/IMTEGI(s-18 Operations Center Acquisition Delayed

Chapter 4 Block B’s Design and Development Has Proceeded Much Like Block A’s

presented an alternative design to the Air Force, which is to be reviewed in April 1989. Questions of whether the system will meet performance specifications and, if so, whether the government will have to pay additional costs have yet to be decided.

Page 32 GAO/IMTEGW18 Operations Center Acquisition Delayed

5 Chapter

Conclusions and Recommendations

The Air Force has spent over $235 million for a system that is now more than 3 years behind schedule and does not perform as required. Given the lack of progress that Air Force has made on SPADOC to date and the severity of the problems that remain, we question whether the system, as currently designed and developed, can meet its required operational capability and whether the Air Force’s cost and schedule estimates for attempting to do so are realistic.

There are two primary causes of s~moc's problems. First, the program is highly complex and technically risky. Defense has put forth considerable effort in recent years to develop multilevel secure systems, yet we know of no mission critical controlled mode system similar to SPADOC that has become operational. The problems in building such systems involve both the complexity in writing the software (the development tasks required are time consuming, technically demanding, and still the object of active research) and the difficulty in attaining satisfactory system performance, given the extra processing needed to run software with extensive security functions built into it. This latter problem is especially acute when systems, such as SPADOC, must perform quickly to provide timely warning of threat or attack.

Second, the Air Force did not prudently manage the SPADOC effort, given the technical difficulties and risks involved. System requirements were not adequately analyzed at the outset to identify which were most difficult to satisfy and posed the greatest risk to project success, and management strategies were not formulated and executed to accommodate these risks effectively. In particular, the Air Force did not formally evaluate its requirements for both controlled mode security and high system performance to determine whether these were concurrently achievable. Furthermore, when problems occurred in meeting these requirements and questions were raised about the validity of the model being used to predict system performance and the adequacy of the system design, Air Force did not take effective corrective action. The Air Force had numerous opportunities to suspend development until problems were addressed and resolved, but it did not. Instead, Air Force continued com- mitting resources without resolving underlying technical problems, hop- ing that difficult, fundamental problems would somehow be resolved in later phases of the program.

As a result of the technical challenges and ineffective management that have characterized the SPADOC program, the Air Force is now faced with a dilemma. It has accepted and paid for a system that is only marginally



useful, does not meet most contractually specified performance requirements, and is not yet operational but which, according to U.S. Space Command, when operational will offer some functional improvement over the current, primarily manual system. Given that the Air Force now owns a system that is only marginally useful, it must decide how to use it cost effectively.

Further, the Air Force must ensure that later phases of the SPADOC program avoid the pitfalls that have hampered the effort to date. However, the Air Force does not appear to be doing so. Ford has been allowed to continue developing the block B system, even though the Air Force has not approved the hardware configuration, software approach, or system design. As with block A, continuing on this track only serves to increase program costs and risks with no discernable benefit.

Reclommendations to Due to the mission critical nature of the SPADOC project, its high cost, its

the Secretary of Defense

developmental difficulties, and its history of ineffective Air Force management, we recommend that the Secretary of Defense halt block B development until the Air Force has complied with congressional requirements to submit NORAD modernization programs, including SPADOC, to Defense Acquisition Board review. During that review, the Secretary of the Air Force should specifically submit to the board:

l Recommendations on the ultimate disposition of the SPADOC block A system. If the Secretary recommends continuing to use block A as an interim improvement over the current, primarily manual system, these plans should include (1) an evaluation of the capabilities and deficiencies of the block A system as accepted; (2) an assessment of the incremental costs and benefits of changes and modifications required to make the system fully operational; and (3) recommendations on how block A should be used based on careful analysis of costs incurred and benefits derived.

l Plans for the future SPADOC system. These plans should include a thorough analysis of the requirements for SPADOC and the feasibility of satisfying those requirements, in particular controlled mode security. Plans should also include identification and analysis of alternative technical and contractual approaches to meeting the requirements, and well- founded estimates of costs and benefits of the alternative approaches,



Recommendation to the Congress

We recommend that the Congress withhold further funding for the SPADOC 4 acquisition until the Defense Acquisition Board has submitted and the Secretary of Defense has approved program plans meeting the objectives described in our above recommendations.

Agency Comments and The Department of Defense agreed with most of the information pre-

Our Evaluation sented in the report but did not fully concur with several of the findings and conclusions. The Department disagreed with two of the four recommendations. Defense acknowledged that the SPADOC 4 program has experienced many technical problems and uncertainties and that SPADOC 4 does not meet all performance specifications, is over cost, and behind schedule. However, Defense stated that the Air Force has taken the necessary actions to ensure that the later phases of the SPADOC 4 program will avoid the pitfalls that have hampered the program to date. Defense’s comments can be consolidated into two major areas: (1) whether the Air Force adequately assessed the risks of achieving the technical requirements, and (2) whether the Air Force effectively managed the SPADOC program.

Assessment of Potential Development Risks

Defense stated in its comments that the Air Force spent 2 years developing the requirements for the SPADOC 4 system. These requirements were based on U.S. Space Command’s need for a system to operate at multiple security levels (controlled mode security) and to perform certain critical tasks within specified time periods (stated as quantitative performance requirements). Defense stated that an additional 2 years and $12 million were invested in concept definition studies by two contractors-Martin Marietta Corporation and Ford Aerospace Corporation. According to Defense, these contractors, with technical oversight by Mitre Corpora- tion, developed the system requirements, assessed the risks of these requirements, and stated in their best and final offers that a system meeting both controlled mode security and quantitative performance requirements was achievable. Finally, Defense’s comments suggest that because another system-the Honeywell MULTICS system-had been accredited and operating for several years with controlled mode security requirements similar to those specified for SPADCK 4, that development risk was minimal.

Our review of SPADOC 4 source selection documentation revealed that meeting both the controlled mode security and quantitative performance requirements was not as clearly achievable as Defense’s comments would indicate. While both contractors’ best and final offers just prior to

Page 35 GAO/IMTECSY-18 Operations Center Acquisition Delayed


contract award indicated that most SPADOC 4 requirements could be achieved, significant concerns about the achievement of the multilevel security requirements raised by both contractors throughout concept development should have put the Air Force on notice as to the risk of this undertaking.

Martin Marietta made it clear in its initial security trade-off analysis that there had been little success in achieving controlled mode security and that the SPADOC 4 acquisition need not be put at risk when other viable alternatives were available. In a subsequent design proposal, Martin Marietta proposed that security limitations be identified and a security analysis be undertaken, Further, Ford’s initial design proposal identified hardware and software limitations, and exceptions to the security requirements. Because both contractors identified limitations, neither proposal was an unqualified endorsement that the security requirements could be met. The initial concerns raised by both concept definition contractors and the limitations subsequently identified in Martin Marietta’s later design should have put the Air Force on notice that an independent assessment of the achievability of the security requirement was needed. However, none was performed,

Further, Defense did not provide evidence to support its statement that the Honeywell MULTICS system’s controlled mode security requirements are very similar to those specified for SPADOC 4. On the contrary, our analysis of the Honeywell MULTICS system shows that its secure operating capabilities are not similar to those capabilities required for SPAMH: 4. MULTICS is a stand alone time-shared system with no electronic connection to other computer systems, and thus, all classified information is protected within the system. However, because SPADOC is connected to other computer systems (some with lower levels of security than SPADW) it must be able to prevent compromise of classified data. Further, MULTICS is not a real-time system. Therefore MULTICS provides a security capability but does not have to simultaneously meet stringent performance requirements. Meeting both performance and stringent security requirements has been a major technical problem in SPADOC, a problem that MULTICS does not have to overcome. Because of these significant differences between the Honeywell MULTICS and the SPADOC 4 systems, the Air Force’s comparison is not valid.

In commenting on the draft report, Defense stated that “the questions about the ability to achieve both controlled mode security and the QPRs,

as well as the computer capacity needed, have been resolved (underscoring supplied).” The Air Force claims to have identified a hardware

Page 36 GAO/lMTEc89-18 Operations Center Acquisition Delayed


architecture, using IBM model 3090 computers, that not only will meet all block B requirements, including quantitative performance and controlled mode security, but will also possess the growth capabilities needed to meet block C requirements. However, Defense stated that the Air Force “is in the process (underscoring supplied) of developing a block C Requirements Specification, to include block B’s final design and performance capability.” It is unclear to us how a determination can be made that using an IBM 3090 architecture will be the most efficient and cost effective approach to meet block C requirements when these requirements have not been developed. Therefore, after receiving Defense’s comments, we asked Electronic Systems Division to explain their rationale for concluding that the IBM 3090 based architecture will meet all block B requirements and have the growth capabilities needed to meet block C requirements.

We were told that this rationale is based on Ford’s modeling of estimated block C requirements using an improved version of the block B performance prediction model. However, we note that while Ford’s model for predicting system performance has been improved, it has not been validated and problems with it are still being resolved. Further, the Elec- tronic Systems Division has not performed sufficient analyses to conclude that this is the most cost effective and efficient means of achieving SPADOC requirements,

Air Force Management of Defense disagreed with our assessment that the Air Force did not pru-

the SPADOC Program dently manage the SPADOCT effort given the technical difficulties and risks involved. While we reported that the Air Force repeatedly deferred problem resolution to later phases of the acquisition effort, Defense characterized the Air Force’s efforts as “positive actions to manage a very complex major acquisition... .” Three examples are illustrative of the difference of opinion,

First, we reported that while the Air Force identified over 350 action items at the block A critical design review, the Air Force decided to allow the contractor to continue developing the design without first resolving the problems. We noted that following the critical design review, Mitre characterized Ford’s design as being in a state of flux because the design baseline had not been stabilized, software engineering controls were not developed, and the software coding and testing effort was overly ambitious.



Defense stated that the Air Force did not allow the action items to go unresolved; instead, it required Ford to address and correct the 350 action items, an effort that Defense states was completed and approved by the Air Force in July 1984. We did not find this to be the case. In October 1984, Mitre again observed that significant portions of the design had still not stabilized, the test program was still being defined, and the software development effort was unlikely to succeed. In fact, the block A software design was not stable when the system was accepted in April 1988 and Ford’s test program and software coding effort became a major source of problems that continued to delay program development. While the Air Force states it has managed the program prudently and resolved the action items, we found that Air Force management actions taken to date have not resolved identified problems.

Second, we reported that Mitre and Logicon continuously raised concerns about problems Ford was having meeting block A’s security requirements. As early as August 1983, Mitre pointed out that Ford’s design did not meet security requirements, and in February and March 1984, Mitre stated that Ford’s approach to protecting security-relevant software from unauthorized modification did not comply with the contract.

Defense agreed that Mitre raised several issues concerning Ford’s security design; however, Defense stated that we failed to recognize actions taken by the Air Force in response to these concerns. Defense stated that the Air Force made the contractor respond to each issue raised by Mitre and correct its design accordingly. However, we did not find evidence that the Air Force’s actions resolved the problems. In fact, in Sep- tember 1984, the Air Force recognized that the security requirements could not be achieved quickly and therefore, decided to defer achieving them to block B.

However, action to defer achieving the requirement did not solve the problem either. Throughout block B development, Ford maintained that it could not simultaneously achieve the security and quantitative performance requirements, and continuously sought relief. Because Ford’s block B design presented at critical design review could not meet all contract requirements, the Air Force rejected the design. As a result, the Air Force and Ford have spent the last year trying to identify a design to simultaneously achieve the security and quantitative performance requirements. While Defense maintains that the Air Force did not allow the contractor to continue development without resolving problems, 4

Page 38 GAO/WTEG89-19 Operations Center Acquisition Delayed


years have passed since the security issue was raised, but it has yet to be resolved.

Third, we reported that in April 1988 the Air Force acknowledged its inability to meet the block A QPRS without a major redesign of the block A system. Thus, the Air Force deferred achieving block A quantitative performance requirements to block B and accepted block A. According to the Vice Commander, Electronics Systems Division, this was done “to get some minimal and marginal capability into Air Force Space Com- mand’s hands.”

However, in the same month, the Air Force rejected Ford’s block B design. According to Defense, Ford’s effort, which included a major redesign needed to correct block A’s deficiencies, also could not achieve the mission QPRS. As a result, the Air Force now owned block A, which did not meet its needs, and had block B on hold because it too could not satisfy those needs. The Air Force now plans to upgrade the SPADOC system’s computers to faster, more powerful IBM model 3090s in another attempt to achieve controlled mode security and the quantitative performance requirements. This computer upgrade is intended to replace the computers that the Air Force bought when it accepted block A. How- ever, as we noted earlier, the Air Force has not performed the analysis necessary to assure itself that this second set of computers will satisfy SPADoc 4 requirements.

The Department of Defense believes that the Air Force took positive actions in attempts to overcome what it calls “the root causes” of the SPADOC 4 program’s problems. According to Defense, the SPADOC 4 contractor too rigorously implemented the security requirements, inef- ficiently designed the software, poorly integrated the software, and ineffectively managed its subcontractors. Notwithstanding Defense’s claims, it does not alter the fact that Air Force management allowed the situation to go on for many years before initiating action to resolve problems, As we pointed out in our report, the Air Force had numerous opportunities to suspend development until problems were addressed and resolved, but it did not. It was not until April 1988, when the Air Force rejected Ford’s block B design proposal, that the Air Force began to take positive action to halt what it called “the root causes” of SPADOC 4’s program problems.

Finally, Defense disagreed with our recommendations to withhold funding and halt SPADOC 4 development. Defense stated that halting block B

Page 39 GAO/IMTEGS918 Operations Center Acquisition Delayed


would only serve to exacerbate the cost and schedule problems previously experienced and would result in an unnecessary delay in the critical operational capabilities. Defense’s comments provide no evidence to support its claim that halting development would exacerbate cost and schedule problems. In fact, continuing to develop a system with an unstable design, such as a constantly changing hardware architecture, generally results in the need for later changes, as well as cost growth and schedule delays. Our recommendations are designed to minimize future cost growth and schedule delays. Halting development will give the Air Force an opportunity to reassess its critical requirements and resolve the complex technical issues facing the SPADOC program. First, U.S. Space Command must determine whether controlled mode security is absolutely necessary to achieve its mission or whether other alternatives are viable. Second, if controlled mode security is found to be necessary, the Air Force must determine whether this requirement is achievable without undue cost, schedule, and performance impact. Third, before approving an upgrade to the system’s computers, the Air Force must determine that the model being used to predict system performance is capable of accurately predicting the performance of the new computers.

Page 40 GAO/IMTJ3CS9-18 Operations Center Acquisition DeIayed

Page 41 GAO/IMTEG89-18 Opemtions Center Acquisition Delayed

Appendix I

Current ITW/AA System

Missile

1 IJ il

Bomber

-F

Space Defense

-7 lc ommunicationsl W L ~;;~~:pO,

System

Cheyenne Mountain

L+ Military and Civilian Leaders

Page 4 2 GAO/IMTEG89-18 Operations Center Acquisition Delayed

Appendix II

Modernized ITW/AA System

i=l Missile

Cheyenne Mountain

Granite Sentry

Military and Civilian Leaders

+ SCIS Survivable Communications Integration System

e

Page 43 GAO,/BWMXS18 Operations Center Acquisition Delayed

Appendix III

Comments From the Department of Defense.

ASSISTANT SECRETARY OF DEFENSE

Mr. Ralph V. Carlone Assistant Comptroller General Information Management and

Technology Division U.S. General Accounting OfEice Washington, DC 20548

Dear Mr. Carlone:

This is the Department of Defense (DOD) response to the General Accounting Office (GAO) Draft Report, "SPACE DEFENSE: Management and Technical Problems Delay Operations Center Acquisition," dated January 6, 1989 (GAO Code 510275, OSD Case 7872). The DOD partially agrees with the report.

Although many of the facts identified in the draft report are correct, the Department of Defense does not fully concur with several of the findings and conclusions. Therefore, several points of clarification are provided. The Cheyenne Mountain Complex Space Defense Operations Center (SPADOC) 4 is the final phase of an Air Force program to develop and modernize a single center for space defensive agencies, and unfortunately, as originally planned, the program is now over cost and behind schedule. The SPADOC 4 requirements established by the Air Force were the results of four years of effort and reflect what the US Space Command definitely needs in order to perform its space defense mission. The Air Force has taken positive actions to manage a very complex major acquisition which is critical to the defense of this nation.

With the increased management attention the SPADOC 4 program has received, and the significant contractor progress which has recently been achieved, the DOD believes that the Air Force has taken the necessary action to ensure that the later phases of the SPADOC 4 program avoid the pitfalls that have hampered the effort to date. The Air Force has already complied with Congressional direction to combine the Cheyenne Mountain programs into a single program element and is preparing for a Defense Acquisition Board review, now anticipated in June 1989. It continues to be the DOD position that the SPADOC 4 program is still the most prudent approach for extending the space deEense and surveillance missions into the 2lst Century.

Page 44 GAO/lMTEC-S9-18 Operations Center Acquisition Delayed

Appendix JII Comments From the Department of Defense

2

The Department appreciates the opportunity to comment on the report in draft form. The detailed DOD comments on the GAO findings and recommendations are enclosed. Additional technical corrections were separately provided to members of your staff.

Sincerely,

Gordon A. Smith

Enclosure


Appendix III Comments From the Department of Defense

Now on p. 3, pp. 9-10, p. 14, and pp. 18-19.

GAO DRAFT REPORT - DATED JANUARY 6, 1989 (GAO CODE 510275) OSD CASE 7872

"SPACE DEFENSE: MANAGEMENT AND TECHNICAL PROBLEMS DELAY OPERATIONS CENTER ACQUISITION"

DEPARTMENT OF DEFENSE COMMENTS

* * * * *

FINDINGS

9 FINDING A: Potential Development Risks Of The Space Defense Operations Center Not Formally Assessed. The GAO reported that the Space Defense Operations Center (SPADOC) 4 is the final phase in the Air Force program to develop and modernize a single center for all command, control, and communications, and data processing functions for space defense agencies. The GAO explained that the SPADOC 4 is being developed in three evolutionary blocks and was initially scheduled for completion by June 1988, at an estimated cost of $289.6 million. The GAO found, however, that the SPADOC 4 did not become operational in June 1988, as originally planned, and is now significantly over cost and behind schedule. The GAO pointed out that, from the outset, the Air Force was aware that achieving some block A requirements uould be risky and, therefore, the program needed close management oversight. The GAO found, however, that potential development risks of the SPADOC 4 were not formally assessed during the requirement setting and concept development phase. Instead, the GAO found that the Air Force developed detailed requirements for block A, based on the U.S. Space Command space defense mission, with major requirements that the system operate in controlled mode and that it satisfy the quantitative performance requirements. The GAO pointed out that the security level required for block A had not been achieved by any system in 1983, and both Mitre and the Air Force recognized that it would be risky. The GAO concluded that the Air Force did not adequately assess the risks associated with the SPADOC 4 program. (p. 3, pp. 12-13, p. 17, pp. 25-27/GAO Draft Report)

DOD RESPONSE: Partially concur. While some of the GAO basic facts are correct, the DOD nonconcurs with the GAO assumptions and conclusions regarding the SPADOC 4. It is true that SPADOC 4 is the final phase of an Air Force program to develop and modernize a single center for space defense agencies, and unfortunately, as originally planned, the program is now over cost and behind schedule. However, the Air Force did formally assess the potential development risks of the SPADOC 4 requirements and did not attempt to develop a system that was beyond the state-of-the-art.

Page 46 GAO/IMTEESS-18 Operations Center Acquisition Delayed

- Appendix LU Comments From the Department of Defense

When the original SPADOC effort began in 1979, the Air Force spent two years developing requirements for the SPADOC 4 system. These requirements were based upon the necessary actions required by US Space Command (USSPACECOM) to perform its space defense mission. Included in these requirements was the need for a system to perform certain critical tasks within specified time periods in order for the SPADOC to provide adequate threat warning to various satellite owner/ operators. These time critical requirements became known as quantitative performance requirements, or QPR's. Additionally, because of the different security levels involved, as well as the time constraints, the SPADOC 4 system also had to be capable of operating at multiple security levels (controlled mode security).

During 1982 and 1983, the Air Force spent over $12M in a formal, competitive Concept Definition phase for the SPADOC 4. This included the efforts of two contractors (Ford Aerospace Corporation and Martin-Marietta) as well as technical oversight by Mitre Corporation, to take the Air Force mission requirements and develop the system requirements, formally assess the risks of these requirements and present contract proposals for developing the SPADOC 4. Contrary to the GAO report, the Best and Final Offers of both contractors stated that a system meeting both controlled mode security and quantitative performance requirements (QPR's) was achievable. Additionally, Mitre Corporation found both proposals to be technically acceptable and the requirements were approved by the Air Force before a SPADOC 4 contract was awarded. Over the course of the block A and block B devel- opments, these requirements were reassessed and reveriEied at least four more times.

At the time of contract award, the Honeywell Multics system had already been accredited and operating for seven years with controlled mode security requirements very similar to those specified for the SPADOC 4. Today, the Honeywell Secure Communications Processor and the SACDIN program have also been accredited as meeting security requirements significantly more stringent than those speciEied for the SPADOC 4. Still, during Concept Development, the Air Force intentionally selected only the minimum requirements for the SPADOC 4 because it would have been unacceptably costly, time consuming, and unnecessarily technically difficult to require the full set of security features.

In summary, the SPADOC 4 requirements established by the Air Force were the results of four years of effort and reflect what the USSPACECOM definitely needs in order to perform its space defense mission, These requirements were formally assessed by three separate contractors, as well as the Air Force, and were found to be technically feasible and in concert with previously developed systems. Still, the Air

2

Page 47 GAO/IMTFLXS-18 Operations Center Acquisition Delayed

-. ~~~~ Appendix Ill Comments From the Department of Defense

Force took positive action in limiting the scope of the security requirements in order to minimize potential development risks.

. FINDING B: Early Problems In The SPADOC 4 Development. The GAO found that, although block A has been in development since April 1983, it cannot perform 14 of 23 required mission functions quickly enough to satisfy contractually specified quantitative performance requirements. In addition, the GAO found that the system does not ensure that confidential or secret information will not be sent to unauthorized satellite owners and operators. The GAO noted that, from the beginning, the Air Force was aware that achieving some block A requirements would be risky and needed close management oversight. According to the GAO, Mitre repeatedly told the Air Force that it was concerned about the adequacy of the SPADOC security design, inadequacies in a model to predict performance, and the overall software design quality. The GAO reported, for example, that in August 1983, Mitre pointed out that the Ford system did not meet security requirements, and then reemphasized its concerns in early 1984. The GAO also reported that, as early as October 1983, Mitre told the Air Force that the performance model did not accurately reflect the Ford system. In addition, the GAO found that, in March 1984, at the conclusion of the block A critical design review, over 350 design problems had been identified and changes to the block A design were still being made. The GAO found that, in spite of these deficiencies, the Air Force decided to allow the contractor to continue developing that design because (in the Air Force view) it was less risky to continue building based on an incomplete design than to delay development. The GAO concluded that this decision began a continuing pattern of approving key project milestones even though requirements had not been satisfied thus deferring problem resolution. (pp. 3-4, p. 19, pp. 28-32/GAO Draft Report)

DOD RESPONSE; Partially Concur. While problems did surface early during the SPADOC 4 development, the DOD nonconcurs with the GAO conclusions that the Air Force allowed the contractor to continue approving key project milestones even though requirements had not been satisfied. Additionally, the GAO report failed to recognize Air Force actions taken in response to Mitre concerns.

Block A development did begin in April 1983 and as the system exists today, it cannot perform 14 of the 23 originally required mission QPR's. However in April 1988,

as noted in the GAO report, the Air Force decfded that these QPR's could

not be met without a major redesign of the system. The Air Force, therefore, deferred these requirements to block B, which included this redesign. Additionally, it is not true that the block A system does not afford proper security

3

Now on pp.3.4, p. 14,and pp. 1%21.


Appendix IU Comments From the Department of Defense

Now on pp. 3-4, p, 14, p. 18, pp 21-25, and p. 33

protection. Block A will operate at system high security mode, meaning that all messages will be both manually reviewed and automatically tagged as SECRET. All recipients of block A messages are authorized to receive SECRET information. Therefore, classified information will not be sent to unauthorized users.

After development began in April 1983, Mitre did raise several issues concerning the contractor's security design and inadequacies of its model. However, the GAO failed to recognize the actions taken by the Air Force in response to these concerns. With respect to the quality of design, the Air Force made the contractor respond to each issue raised by Mitre and correct its design accordingly. Therefore, it was to be expected that, as the GAO stated, "In many cases, the design presented at the design review differed significantly from earlier versions presented to the Air Force." Additionally, by the time the block A model inadequacies were verified by the Air Force, block A development had progressed far enough to begin making actual performance measurements. The block A model was then deleted as a system performance prediction tool, and only used as a tool to identify functional strings and whether or not these functions would run end-to-end in a non-stressed environment.

Finally, the Air Force did not allow the 350 Critical Design Review (CDR) action items to go unresolved while allowing the contractor to continue developing the system. Instead, all 350 action items were resolved and closed through a series of detailed technical interchanges, and the Air Force, with Mitre's input, approved block A CDR on July 24, 1984.

l FINDING C: Block A Development Phase Was Plagued With Technical Uncertainties And Schedule Delays. The GAO found that, throughout the development phase of block A, beginning in 1984, Mitre and Logicon raised concerns regarding the approach of Ford. According to the GAO, these concerns involved the quality and timeliness of software development, the implementation of security requirements, the adequacy of the Ford performance prediction model, and block A system performance. The GAO found that the Air Force allowed Ford to continue developing block A without solving known problems, despite expresssed concerns and identified problems. The GAO pointed out that, at any point in the process, the Air Force could have suspended development until the problems were addressed and resolved, but it did not do so. The GAO concluded that, as a result, the Air Force did not prudently manage the SPADOC effort. (P. 4, p. 19, p. 25, pp. 31-38, p. 53/GAO Draft Report)

DOD RESPONSE: Partially concur. The DOD concurs block A experienced technical uncertainties and schedule delays and

4

Page 49 GAO/lMTEGl39-18 Operations Center Acquisition Delayed


that these technical uncertainties were indeed in the areas of concern addressed by Mitre and Logicon. However, the DOD does not concur that the Air Force allowed the contractor to continue developing block A without resolving problems, nor that it should have suspended formal development, and therefore did not prudently manage the SPADOC effort.

AS stated in the DOD response to Finding B, the Air Force directed the contractor to incorporate technical issues raised by Mitre and did not approve CDR until all 350 action items were closed. After the CDR, as Mitre and Loqicon continued to express concerns over Ford's performance, the Air Force increased its management oversight of Ford, to include additional reviews and level of detail not required by DOD acquisition regulations. For example:

- In 1984, the Air Force directed Ford to publish a Database Technical Report and Software Integration Plan to better manage the database and software development process, and initiated monthly Software Status Reviews.

- In January 1985, the Air Force initiated management of Ford's year long software integration process through establishment of weekly milestone reports.

- In September 1986, Mitre (representing the Air Force) and Ford began tests to identify ways to improve those computer processing functions that caused the system to perform so slowly.

- From November 1986 through March 1988, the Air Force participated in a performance initiatives program conducted by Ford. Through this program, technical updates to the block A system were applied and performance improvements achieved.

Although formally suspending block A development had been addressed, at no time did the Air Force, including Electronic System's Division and Air Force Space Command (AFSPACECOM), nor the USSPACECOM, feel that it would be in the best interest of the Government to do so. Such action would have only resulted in a loss of trained contractor staff, thus increasing both costs and risk, while further delaying milestones. Both the Air Force and the USSPACECOM believe that the SPADOC 4 effort still appears to be the prudent approach for developing the capabilities needed for the USSPACECOM to perform its space defense mission (See also the DOD response to Finding I).

l FINDING D: Operational Testinq Further Hiqhliqhted Block A System Performance Shortfalls. The GAO reported that, although significant development problems had not been resolved, the Air Force determined that the block A system development tests and subsequent deficiency corrections had

5

Page 50 GAO/IMTlX-S9-18 Operations Center Acquisition Delayed


Now on pp. 3-4 and pp. 25-27.

reduced the block A errors to a "manageable level." The GAO reported that, as a result, the Air Force decided to proceed with Air Force Operational Test and Evaluation Center (AFOTEC) testing in June 1987. According to the GAO, the AFOTEC testing was (1) conducted to determine the readiness of block A to support operations and (2) structured to address critical operational issues. The GAO reported that

. the AFOTEC testing confirmed what the Air Force had been told repeatedly since 1984, that block A could not meet performance requirements. According to the GAO, the AFOTEC found that block A failed to achieve operational capability and concluded that long term efforts would be needed to achieve that capability. The GAO also reported that, beginning in January 1988, the Air Force Space Command began testing block A using the same test procedures used by the AFOTEC, to verify that Ford had corrected the deficiencies previously identified. Although a final test report has not yet been issued, the GAO found that indications are that block A performance problems have continued. (p. 4, pp. 38- 41/GAO Draft Report)

WD RESPONSE: Concur. It is correct that operational testing further highlighted block A system performance shortfalls. In this context, however, the following also needs to be recognized.

In June 1987, a significant number of the AFOTEC trained test team were due for rotation. In order to take advantage of the team's experience, the AFOTEC testing was initiated in the SPADOC Off-Site Test Facility (SOSTF) since their findings would provide early feedback on critical operational and suitability issues. All the discrepancies noted in the AFOTEC report have been closed and today's block A bears little resemblance to the system the AFOTEC tested in June 1987.

l FINDING E: The Air Force Accepted A Marginal Block A System. The GAO reported that in the Ford demonstration test, block A met 7 of 23 mission quantitative performance requirements (QPRS). According to the GAO, in March 1988, the Air Force Space Command stated that, given the block A test results, it was clear that Ford could not meet the QPRs anytime soon, The GAO reported that, in April 1988, the Air Force decided to defer meeting the block A performance requirements and achieving controlled mode security to block B, and accepted the block A system to “get some minimal and marginal capability into Air Force Space Command's hands." In addition, the GAO found that when block A was accepted, 295 other deficiencies and enhancements had been identified. According to the GAO, Ford has agreed to correct 90 deficiencies the Air Force identified as significant and that needed correction before the system could become operational. The GAO noted that no plans have been made to address the other 205 enhancements because they are not considered to be

6

x

Page 61 GAO/IMTEGBB-18 OperationsCenterAcquisitionDdayed


Now on p. 4, p, 14, and p. 27

r contract requirements. According to the GAO, testing to verify that the 90 deficiencies have been corrected began in September 1988, and the Air Force anticipated declaring block A operational in January 1989. (pp. 3-5, pp. 19-28, pp. 42- 43/GAO Draft Report)

DOD RESPONSE: Partially concur. While it is true that block A could not meet a number of performance requirements in April 1988, block A is not now a marginal system. With the correction of a few remaining Initial Operational Capability (IOC) critical service reports, the block A system offers deeinite improvements over current SPADOC 3 capabilities. Functionality, astrodynamic accuracy, and throughput capacity have shown specific improvement. Only system responsiveness remains in the "marginal" category. Block A will provide immediate benefits in the areas where SPADOC 3 is deficient today: nuclear event processing, data management, message preparation, directed energy assessments, and multiple event processing.

It is correct that block A, when accepted, Only met 7 of the 23 mission QPR's. However, as stated in the DOD response to Finding B, the Air Force decided that 14 of these QPR's could not be met within block A and, in April 1988, deferred them to block B. Controlled mode security had already been deferred to block B in 1984, not April 1988. The remaining two QPR's have been Eixed as part of the 90 open deficiencies that the contractor agreed to fix before the system goes operational (now scheduled for March 1989 due to interface problems encountered during live communications testing).

It is incorrect that no plans were made to address the 205 other enhancements. In fact, it was determined that the acceptance of block A was the most prudent way to allow the Air Force to address them. With the acceptance of block A, the contractor could now legally develop these enhancements under the block A maintenance contract. Enhancements with significant operational impact either have, or are currently, being incorporated in the block A system through the Material Improvement Program Review Board. Those enhancements with less operational impact will be analyzed, prioritized, and implemented as required by the block A Software Configuration Control Sub-Board as normal post-IOC version releases.

l FINDING F: Problems Identified In Block A Are Continuinq In Block B. The GAO reported that, when the Air Force began developing block B in June 1986, it knew that many of the block A problems could seriously impair block B development. As examples, the GAO reported that (1) block B was built on unstable block A software, (2) strong indications existed that the performance prediction model was still deficient, (3) Ford and the Air Force remained at a standoff about whether the controlled mode security requirement and the QPR

7

-



t

Now on p. 4, pp. 14-15, and pp. 28-32.

-

requirements could simultaneously be achieved, and (4) Ford and the Air Force could not agree on whether, or how, to increase the block B computing capability to achieve the QPRs. The GAO discussed concerns raised and efforts made to overcome these problems, both before and after the June 1986 decision to begin block B development. The GAO found thatr despite these problems, the Air Force allowed the block B design and development to continue for more than two years. The GAO reported that, in April 1988, after nearly two years of development and with 70 percent of the block B software written, the Air Force rejected the Ford block B design proposal because the design was not expected to meet the QPRs. The GAO recognized that, at the time, the Air Force emphasized to Ford that it is required to deliver a system design that will meet all operational requirements. The GAO reported that, in August 1988, Ford agreed to redesign block B in another attempt to meet all the SPADOC operational requirements, and another design review was scheduled in December 1988, to evaluate the revised proposal. The GAO points out, however, that the questions of whether the system will meet performance specifications and, if so, whether the Government will have to pay additional costs, have yet to be decided. The GAO concluded that, as with block A, continuing on this track only serves to increase program costs and risks, with no discernable beneEit. 44-51/ GAO Draft Report)

(P. 5, pp. 20-21, pp-

DOD RESPONSE: Partially concur. The block B situation as described in the above finding is correct for the time at which it was written. Since then, however, several corrective actions have been taken. In addition, the DOD nonconcurs with the GAO conclusion that block A problems are continuing into block B. Instead, the situation is a reflection of Air Force efforts to prevent such an occurrence, and the significant progress/problem resolutions that have occurred since the GAO report was drafted bear this out.

When block B development began in June 1986, its initial design was based upon what proved to be a changing block A baseline [unstable software). Any changes made to block A had to be incorporated into block B and both Mitre and Logicon raised issues along these lines. Therefore, when the contractor presented its block B design at the CDR, the Air Force disapproved it, due in part to the problems still existing in block A. As mentioned in the GAO finding, the contractor has agreed to redesign block B, now based on a stable block A baseline, and will present this design at a delta CDR, now scheduled for April 1989. Additionally, of the 70 to 80 percent of the block B code already developed, Mitre claims that only 5 to 10 percent will need to be recoded. It should also be noted that the 80 percent developed figure is the contractor's assessment and does not reflect what the Air Force has approved, tested or accepted,

a



Nowon p.3and p.15

With respect to the GAO statements concerning the inadequacy of the block B model, the Air Force has expended substantial effort in evaluating and analyzing the block B performance model. During the second half of October 1988, the Air Force provided full-time, on-site support to Ford to assist with the modeling effort. Considerable effort was spent analyzing the underlying assumptions of the model, and identifying those assumptions which caused the greatest risk to the modeling results. In addition, an experiment was designed and conducted to determine whether or not the model results were sensitive to a variety of system parameters. Any problems that were identified during this effort have either been fixed, or are continuing to be tracked. The contractor is aware of what these problems are, and agrees that they must be resolved prior to the block B CDR.

Finally, the questions about the ability to achieve both controlled mode security and the QPR's, as well as the computer capacity needed, have been resolved. Since April 1988, the Air Force has undertaken a very aggressive effort to process engineering change proposals that will upgrade the hardware architecture, improve software design and add system functionality so that a successful CDR can be achieved. For example, Air Force resources have focused on identifying a hardware architecture that not only will meet all block B requirements, including quantitative performance and controlled mode, but will also possess the growth characteristics needed to meet block C requirements. This move to an IBM 3090 base architecture has been highly endorsed by the USSPACECOM, the AFSPACECOM and Mitre.

l FINDING G: Block C Is Planned, But Not Yet Funded. The GAO reported that block C is intended to complete the transition from the existing SPADOC 3 system, and provide the automated capability needed to manage the U.S. Space Command space defense mission. According to the GAO, block C is to maintain and update orbital data and make orbital position predictions on at least 10,000 potential objects in the data base. The GAO noted that when block C is completed, the SPADOC 3 will be retired and its functions performed by the SPADOC 4. The GAO reported that, as currently planned, block C is to be operational by 1994. The GAO points out, however, that as of November 1988, the Air Force had not obtained congressional funding, nor made specific plans to award a contract to design and develop block C. The GAO noted that in September 1988, the Congress directed that all North American Aerospace Defense Command (NORAD) modernization programs, including the SPADOC, be placed under Defense Acquisition Board oversight, (p. 3, pp. 21-22/GAO Draft Report)

DOD RESPONSE: Concur. Block C funding, in accordance with the revised block B schedule, is within the out years of the

9

Page 64 GAO/IMTEGS9-18 Operations Center Acquisition Delayed

Appendix Ill Comments From the Department nf Defense

Now on p, 3, pp. 16-17, and p. 33.

DOD FY 1990-94 Five Year Defense Plan. The Air Force has not yet obtained congressional funding, because the cycle to forward the request for funds to the Congress has not yet been reached. Additionally, the Air Force is complying with all congressional direction to date, with a Defense Acquisition Board (DAB) review and subsequent report to the Congress anticipated by June 1989.

With respect to the specific plans for awarding a block C contract, the Air Force has decided not to award block C until the completion of block B. However, the Air Force is in the process of developing a block C Requirements Specification, to include block B's final design and performance capability. Additionally, the Air Force is purposefully keeping its acquisition strategy for block C open to incorporate any future lessons learned from block B.

l FINDING H: SPADOC Acquisition Costs Have Risen And Milestones Have Been Delayed. The GAO reported that the estimated cost to acquire the SPADOC 4 has risen from $289.6 million in May 1982, to $437 million, as of March 1988. The GAO reported that, in addition, the scheduled completion of the three blocks of SPADOC 4 has been delayed significantly, with block C now projected for completion in FY 1994, rather than June 1988, as originally planned. The GAO noted that although the SPADOC original costs did not meet the expenditure thresholds that would have required high level oversight throughout development, the OSD and the Air Force did not reconsider this original decision, even though cost estimates rose above these thresholds early in the life of the SPADOC contract. The GAO pointed out that, in conjunction with the direction to consolidate all NORAD modernization programs, including the SPADOC, under Defense Acquisition Board oversight, the Congress also required that, during FY 1989, the Board conduct a management review of the consolidated program and report the results to the Congress. Based on the lack of progress the Air Force has made on the SPADOC to date and the severity of the problems that remain, the GAO questioned whether the system, as currently designed and developed, can meet its required operational capability, and whether the Air Force cost and schedule estimates for attempting to do so are realistic. (p. 3, pp. 22-24, p. 52/GAO DraEt Report)

DOD RESPONSE: Partially concur. The SPADOC 4 costs have risen and block C is now planned for completion in FY 1994. It should be recognized, however, that through proper management attention, the Air Force has taken action to address the SPADOC problems and has accepted the schedule delays to better ensure that the system will meet requirements. The Air Force is complying with all congressional direction to date, with a DAB review and subsequent report to the Congress anticipated by June 1989. Additionally, as stated in the DOD response to Findings E and

10



Now on pp, 3-4 and pp 33-34.

F, substantial progress has been made to both blocks A and B since the GAO drafted this report. Therefore, the DOD is confident that the SPADOC 4 will meet its required capability and that the current cost and schedule estimates, which will be presented to the DAB review, are realistic.

l FINDING I: Future Prospects of The SPADOC Program. The GAO attributed the problems that have affected the SPADOC 4 program to two primary causes, as follows:

- the high complexity and technical risks associated with the program; and

- the failure of the Air Force to prudently manage the SPAWC effort.

The GAO stated that the Air Force is now faced with a dilemma--it now owns a system that is only marginally useful and it now must decide how to use the system cost effectively. The GAO further observed that the Air Force must ensure that later phases of the SPADOC program avoid the pitfalls that have hampered the effort to date. The GAO concluded, however, that the Air Force does not appear to be doing so. The GAO pointed out that Ford has been allowed to continue developing the block B system, even though the Air Force has not approved the hardware configuration, software approach, or system design. The GAO concluded that continuing on this track only serves to increase program costs and risks, with no discernable benefit. S2-54/GAO Draft Report)

(PP. 3-4, PP.

DOD RESPONSE: Nonconcur. The DOD does not concur with the GAO assessment that SPADOC 4 problems were due to high complexity and technical risks and the failure of the Air Force to prudently manage the SPADOC effort.

As stated in the DOD responses to the previous findings, the capability to achieve controlled mode security had been demonstrated by the Honeywell Multics system for seven years prior to contract award. Additionally, as Mitre and Logicon continued to raise concerns over the contractor's design, increasingly more Air Force oversight was applied to the contractor.

The DOD believes that the root causes of SPADOC 4 technical uncertainties and schedule delays were overly rigorous implementation of security features, inefficient software design, poor integration, and lack of subcontractor management by the SPAWC 4 contractor. These areas are being closely monitored by the Air Force, and significant improvements have been seen. Today's block A system will offer definite improvements over existing SPADOC 3 capabilities, and the switch to IBM 3090's provides an architecture that will handle both controlled mode security

11



and the QPR's as well as the growth for block C requirements.

With the increased management attention the SPADOC 4 program has received, and the significant contractor progress which has recently been achieved, the DOD believes that the Air Force has taken the necessary action to ensure that the later phases of the SPADOC 4 program avoid the pitfalls that have hampered the effort to date and that the SPADOC 4 program is still the prudent approach for extending the space defense and surveillance missions into the 21st Century.

12

Page 67 GAO/EVTEGS9-I8 Operations Center Acquisition Delayed

Appendix LII ConunentsFrom the DepartmenCofDefense

Now OII pp 4-5 and p, 34

Now on pp. 4-5 and p. 34.

* * * * * *

RECOMMENDATIONS

l RECOMMENDATION 1: The GAO recommended that the Secretary of Defense halt block B development until the Air Force has complied with congressional requirements to submit NORAD modernization programs, including the SPADOC, to Defense Acquisition Board review. (p. 5, p. 54/GAO Draft Report)

DOD RESPONSE: Nonconcur. Per congressional direction, the Air Force has already combined the Cheyenne Mountain Programs into a single program element and a DAB review is anticipated for June 1989. As discussed in the DOD responses to findings E, F and G, the DOD is confident that the Air Forces has taken the actions necessary to ensure the success of the SPADOC program. Halting block B now would only serve to exacerbate the cost and schedule problems previously experienced and would result in unnecsessary delay in the critical operational capabilities.

l RECOMMENDATION 2: The GAO recommended that, during the Defense Acquisition Board review, the Secretary of the Air Force should specifically submit to the Board his recommendations on the ultimate disposition of the SPADOC block A system. The GAO further recommended that if the Secretary recommends continuing to use block A as an interim improvement over the current, primarily manual system, these plans should include (1) an evaluation of the capabilities and deficiencies of the block A system as accepted, (2) an assessment of the incremental costs and benefits of changes and modifications required to make the system useful operationally, and (3) recommendations on how block A should be used, based on careful analysis of costs incurred and benefits derived. (pp. 5-6, pp. 54-55/GAO Draft Report)

DOD RESPONSE: Concur. The Air Force is preparing for a June 1989 DAB review and report to the Congress. The Secretary of the Air Farce Will address all issues required by the DAB and congressional direction.

l RECOMMENDATION 3: The GAO recommended that, during the Defense Acquisition Board review, the Secretary of the Air Force should specifically submit to the Board plans for the future SPADOC system. The GAO further recommended that these plans should include (1) a thorough analysis of the requirements for the SPADOC and the feasibility of satisfying those requirements, in particular controlled mode security, (2) identification and analysis of alternative technical and

13

Page 68 GAO/KMTEC89-18 Operations Center Acquisition Delayed


Now on pp. 4-5 and p. 34

Now on p, 5 and p 35

contractual approaches to meeting the requirements, and (3) well founded estimates of costs and benefits of the alternative approaches. (pp. 5-6, pp. 54-55/GAO Draft Report)

DOD RESPONSE: Concur. See the DOD response to Recommendation 2.

l RECOMMENDATION 4: The GAO recommended that the Congress withhold funding for the SPADOC 4 acquisition until the Defense Acquisition Board has submitted, and the Secretary of Defense has approved, program plans meeting the objectives described in the previous GAO recommendations. (p. 6, ps 56/GAO Draft Report)

DOD RESPONSE: Nonconcur. As stated in the DOD response to Recommendation 1, the Air Force is already complying with congressianal direction. Withholding funding would have the same impact as halting block B development now and would seriously degrade operational capability.

14


m!ndix IV

Major Contributors to This Report a I_ -

Information Samuel W. Bowlin, Director, Defense and Aeronautics Mission Systems (202) 275-4649

Techrklogy Division, Management and

Washington, DC. Michael Blair, Assistant Director Gregory J. McDonald, Assistant Director

Leonard J. Latham, Technical Advisor

Denver Regional Office

Frederick Day, Evaluator-In-Charge Barry Tidwell, Evaluator Deborah Minnick, Evaluator

Boston Regional Office Frederick Cross, Senior Evaluator

(510276) Page 60 GAO/IMTECW-18 Operations Center Acquisition Delayed

-.

,, :.: . .; .: .’ ,

: , -.

‘, I .‘. : I “. ‘..,

. . :.

_.

Documents

IMTEC-89-18 Space Defense: Management and Technical … · 2020. 9. 1. · Aerospace Defense Command’s ability to provide space surveillance and satellite attack warning ... monitor