Increasing the utilization of commercial items in ... · Commercial Off-The-Shelf Items in Acquisition Programs: Problems, Issues and Best Practices. 6. AUTHOR(S) Willard D. Meyer

Calhoun: The NPS Institutional Archive

Theses and Dissertations Thesis Collection

2001-12

Increasing the utilization of commercial items in

Acquisition Programs problems, issues and best practices

Meyer, Willard D.

Monterey, California. Naval Postgraduate School

http://hdl.handle.net/10945/5912

NAVAL POSTGRADUATE SCHOOL Monterey, California

THESIS

INCREASING THE UTILIZATION OF COMMERCIAL ITEMS IN ACQUISITION PROGRAMS: PROBLEMS,

ISSUES AND BEST PRACTICES

by

Willard D. Meyer

December 2001

Thesis Advisor: Michael W. Boudreau Associate Advisor: Richard B. Doyle

Approved for public release; distribution is unlimited.

THIS PAGE INTENTIONALLY LEFT BLANK

REPORT DOCUMENTATION PAGE Form Approved OMB No. 0704-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 1. AGENCY USE ONLY (Leave blank)

2. REPORT DATE December 2001

3. REPORT TYPE AND DATES COVERED Master’s Thesis

4. TITLE AND SUBTITLE: Increasing the Utilization of Commercial or Commercial Off-The-Shelf Items in Acquisition Programs: Problems, Issues and Best Practices. 6. AUTHOR(S) Willard D. Meyer

5. FUNDING NUMBERS

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) Naval Postgraduate School Monterey, CA 93943-5000

8. PERFORMING ORGANIZATION REPORT NUMBER

9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) N/A

10. SPONSORING / MONITORING AGENCY REPORT NUMBER

11. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government. 12a. DISTRIBUTION / AVAILABILITY STATEMENT Approved for public release; distribution is unlimited.

12b. DISTRIBUTION CODE

ABSTRACT (maximum 200 words) Department of Defense policies mandate an increase in the utilization of commercial items. A great deal of study, writing and analysis has gone into the utilization of commercial item software products but not a great deal of study or analysis has gone into the utilization of commercial item equipment, which this thesis will primarily address. It is generally accepted that the utilization of commercial equipment will substantially reduce cost and lead times necessary to get major acquisition program systems fielded. However, there may be significant downsides and drawbacks to the utilization of commercial items that should be recognized, analyzed and for which corrective actions should be planned in order to mitigate commercial item implementation risks. This thesis identifies many of the Program Manager and industry “issues” associated with implementing and integrating commercial items into a DoD program. It also proposes methods and process solutions that will alleviate many of the implementation problems, thereby mitigating the implementation risks.

15. NUMBER OF PAGES

14. SUBJECT TERMS Commercial Items, Commercial Off-the-Shelf, COTS, Nondevelopmental Items, NDI, Best Practices, Acquisition Problems, Acquisition Issues, Problems, Issues, DoD Acquisition Requirements, DoD Acquisition Goals, Acquisition Requirements, Acquisition Goals.

16. PRICE CODE

17. SECURITY CLASSIFICATION OF REPORT

Unclassified

18. SECURITY CLASSIFICATION OF THIS PAGE

Unclassified

19. SECURITY CLASSIFICATION OF ABSTRACT

Unclassified

20. LIMITATION OF ABSTRACT

UL NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) Prescribed by ANSI Std. 239-18

i


ii

..~

Approved for public release; distribution is unlimited

INCREASING THE UTILIZATION OF COMMERCIAL ITEMS INACQUISITION PROGRAMS: PROBLEMS, ISSUES AND BEST PRACTICES

Willard D. MeyerNH-O3, Department of the Anny CivilianB.S.E.E., San Jose State University, 1969

Submitted in partial fulfillment of therequirements for the degree of

MASTER OF SCIENCE IN MANAGEMENT

from the

NAVAL POSTGRADUATE SCHOOLDecember 2001

Author:

Approved by:

nneth Euske, Deana uate School of Business and Public Policy

I

iii1


iv

ABSTRACT

Commercial and commercial off-the-shelf (COTS) describe products that have

been developed primarily for the use of the commercial industries or for the general

public. The theory behind using commercial items in DoD applications is that products

developed for commercial usage will be competitively priced, and because of the large

user base, will be very low in unit price as compared to a product designed and developed

specifically for DOD’s relatively low-volume usage. Also, in areas such as computers

and software, commercial industry’s speed of technology development and acquisition

surpass that which the Government has been able to achieve.

As of January 2001, Under Secretary of Defense (Acquisition, Technology &

Logistics) (USD(AT&L)), mandated an increase in the utilization of commercial items as

follows: (1) double the dollar value of FAR Part 12 contract actions awarded in 1999 by

the end of fiscal year 2005, and (2) strive to increase the number of FAR Part 12 contract

actions awarded to 50 percent of all Government contracts awarded by the end of FY

2005. A great deal of study, writing and analysis has gone into the utilization of

commercial item software products but less is known about the utilization of commercial

item equipment, which this thesis will primarily address. It is generally thought and accepted that the utilization of commercial equipment

will substantially reduce cost and lead times necessary to get systems fielded. However,

there may be some significant disadvantages to utilization of commercial items that

should be recognized, analyzed and for which corrective actions should be planned to

mitigate the implementation risks.

The objective of this thesis is to identify the critical PM and/or industry “issues”

associated with integrating commercial items into a DoD acquisition program and to

propose methods and process solutions that will alleviate many of the implementation

problems and thereby mitigating the implementation risks.

v


vi

TABLE OF CONTENTS

I. INTRODUCTION........................................................................................................1 A. PURPOSE.........................................................................................................1 B. SCOPE OF RESEARCH ................................................................................1 C. RESEARCH METHODOLOGY ...................................................................1 D. RESEARCH QUESTIONS.............................................................................2 E. ORGANIZATION OF STUDY ......................................................................3

II. BACKGROUND ..........................................................................................................5 A. COMMERCIAL ITEMS DEFINED..............................................................5 B. BROADENED DEFINITION OF COMMERCIAL ITEMS ......................6 C. COMMERCIAL ITEMS CLARIFIED .........................................................7 D. DOD REGULATION 5000.1: COMMERCIAL ITEM

REQUIREMENT.............................................................................................9 E. MARKET RESEARCH ..................................................................................9 F. COMMERCIAL ITEM USAGE CONCEPT .............................................10 G. DOD COMMERCIAL ITEM REQUIREMENTS.....................................11 H. USING COMMERCIAL ITEMS.................................................................12

1. The marketplace, not the PM, drives development of the commercial product. ..........................................................................15

2. A change of the commercial product (which is almost a certainty) will drastically affect program supportability...............15

3. Integrating a commercial item into a program is fundamentally different from developing a custom product, and may require similar PM attention.............................................15

4. Greater teamwork among stakeholders during the selection of commercial items from the marketplace is essential to keep the program on track. ..............................................................................15

5. There must be a willingness to adapt the “requirements” to the capabilities available in the marketplace rather than adapting the commercial capabilities to the DoD “requirements.”...............16

6. Commercial item usage means that many of the normal acquisition activities (such as system development and sustainment) will be repeated throughout the lifecycle of the program, because a commercial item is never really complete (i.e., it is always changing).................................................................16

7. The product must be tested or verified to ensure that required capabilities are in fact met as promised by the commercial item manufacturer......................................................................................16

I. POTENTIAL BENEFITS OF USING COMMERCIAL ITEMS.............17

III. PM & INDUSTRY COMMERCIAL ITEM USAGE EXPERIENCE & PROBLEMS ...............................................................................................................19 A. INTRODUCTION..........................................................................................19

vii

B. BACKGROUND INFORMATION .............................................................19 1. Acquisition Strategy...........................................................................19 2. Market Investigations........................................................................20 3. Concept Formulation.........................................................................21 4. Management of Integration ..............................................................21 5. Logistics Support ...............................................................................22

a. Reliability.................................................................................22 b. Support Requirements.............................................................22 c. Technical Data for Competitive Reprocurement ...................23

6. Test and Evaluation ...........................................................................23 7. Culture ................................................................................................25 8. Army Definitions................................................................................26

C. COMMERCIAL ITEM USAGE ISSUES ...................................................27 1. U.S. Army Audit Agency, NE 90-206, Acquisition of

Nondevelopmental Items: U.S. Army Natick Research, Developmental and Engineering Center, Natick, Massachusetts. 19 March 1990 .........................................................27 a. Acquisition Strategy ................................................................27 b. Market Investigations .............................................................28 c. Concept Formulation..............................................................29

2. U.S. Army Audit Agency, NE 91-204, Acquisition of Nondevelopmental Items. 17 June 1991...........................................29 a. Acquisition Strategy ................................................................29 b. Market Investigations .............................................................30 c. Concept Formulation..............................................................30 d. Management of Integration....................................................30 e. Logistics Support.....................................................................30

3. U.S. Army Audit Agency, NE 90-209, Acquisition of Nondevelopmental Items: Fort Monmouth, New Jersey. 11 September 1990 ..................................................................................32 a. Acquisition Strategy ................................................................32 b. Market Investigations .............................................................33 c. Concept Formulation..............................................................33 d. Test and Evaluation ................................................................33 e. Logistics Support.....................................................................34

4. Bonheim, Mike LTC, PM – Physical Security Equipment (PSE), Commercial Items Acquisition Data Call, 22 February 2001......................................................................................................35 a. Needs Determination...............................................................35 b. Reliability.................................................................................35 c. Market Investigations .............................................................35 d. Logistics Support.....................................................................36 e. Test and Evaluation ................................................................36

viii

5. GAO/NSIAD-95-161, UNMANNED AERIAL VEHICLES: Maneuver System Schedule Includes Unnecessary Risk, 15 September 1995 ..................................................................................36

Test and Evaluation ............................................................................37 6. GAO/NSIAD-96-2, Unmanned Aerial Vehicles: Hunter System

Is Not Appropriate for Navy Fleet Use, 1 December 1995.............37 Needs Determination...........................................................................37

7. The DoD IG Report 96-111, Allegations Involving the Procurement of the Hunter Unmanned Aerial Vehicle-Short-Range System, 7 May 1996................................................................37 a. Needs Determination...............................................................38 b. Reliability.................................................................................38 c. Test and Evaluation ................................................................39

8. DoD IG Report 95-193, DoD Hotline Allegations Regarding the Naval Special Warfare Patrol Coastal Ship and the Rigid Inflatable Boat Acquisition Programs, 15 May 1995......................39 a. Needs Determination...............................................................40 b. Design......................................................................................40 c. Risk Analysis ...........................................................................41 d. Test and Evaluation ...............................................................41

D. COMMERCIAL ITEMS PROBLEMS, ISSUES AND CONSEQUENCES SUMMARY ..................................................................41

IV. ANALYSIS – COMMERCIAL ITEM USAGE EXPERIENCE & PROBLEMS ...............................................................................................................45 A. INTRODUCTION..........................................................................................45 B. ROOT CAUSE PROBLEMS AND ISSUES ...............................................45

1. Inadequate Education or Training within the Acquisition Community .........................................................................................45

2. Inadequate Use of Lessons Learned.................................................46 3. Incorrect Culture-Driven Decision Making ....................................46

C. ANALYSIS OF PROCESS PROBLEMS AND ISSUES ...........................46 1. Needs Determination .........................................................................46 2. Market Investigation .........................................................................48 3. Concept Formulation.........................................................................49 4. Risk Analysis ......................................................................................49 5. Acquisition Strategy...........................................................................49 6. Design..................................................................................................50 7. Management of Integration ..............................................................51 8. Test and Evaluation ...........................................................................51 9. Logistics Support ...............................................................................52


D. COMMERCIAL ITEM - CAUSE - SUMMARY .......................................55

V. CONCLUSIONS AND RECOMMENDATIONS...................................................59 A. INTRODUCTION..........................................................................................59

ix

B. METHODS AND PROCESSES “BEST PRACTICE” CONCLUSIONS ............................................................................................59

1. Lessons Learned.................................................................................59 2. Culture-Driven Mistakes...................................................................59 3. Training and Education ....................................................................60 4. Needs Determination .........................................................................61 5. Acquisition Strategy...........................................................................61 6. Market Investigation .........................................................................62 7. Concept Formulation.........................................................................62 8. Risk Analysis ......................................................................................63 9. Design..................................................................................................63 10. Management of Integration ..............................................................64 11. Test and Evaluation ...........................................................................64 12. Logistics Support ...............................................................................65


C. COMMERCIAL ITEM SUMMARY ..........................................................66 D. COMMERCIAL ITEM BEST PRACTICE SUMMARY .........................69 E. CONCLUSION ..............................................................................................72 F. AREAS FOR FURTHER RESEARCH.......................................................73

LIST OF REFERENCES......................................................................................................75

BIBLIOGRAPHY..................................................................................................................79

LIST OF ACRONYMS AND ABBREVIATIONS .............................................................81

INITIAL DISTRIBUTION LIST .........................................................................................83

x

LIST OF FIGURES Figure 1. Defense Acquisition Management Framework (From: DoDI 5000.2)...........13 Figure 2. Traditional Model vs. Recommended Model ..................................................14

xi


xii

LIST OF TABLES

Table 1 Commercial Item Consequence Matrix ...........................................................41 Table 2 Commercial Item Cause Matrix.......................................................................55 Table 3 Commercial Item Best Practice Matrix............................................................70

xiii


xiv

ACKNOWLEDGMENTS

I wish to acknowledge the direction, support and guidance given to me by my

sponsor, LTC Michael Bonheim, PM-PSE. Without his topic idea and the data he

provided, this thesis would not have been written.

Next, I wish to thank my thesis advisors, Professor Michael Boudreau and Dr.

Richard Doyle, who have given generously of their time, experience and knowledge to

improve upon the form, structure and knowledge contained in this thesis.

A great deal of the data contained in this thesis came from the DoD Inspector

General (IG), General Accounting Office (GAO), and the Army Audit Agency. Most of

the reports were obtained directly from the agencies. I wish to thank those people that

provided swift responses by mailing hard copies or emailing electronic copies of data to

me, as follows:

DoD IG Carol L. Hopper

DoD IG John E. Meling

Army Audit Agency Pamela B. Evans

During my thesis research, I spent untold hours in the NPS Dudley Knox Library,

and received research help from virtually everyone that works at the reference desk, my

thanks to them all. Special thanks go to Ann Jacobson who taught me how to access and

use Lexus-Nexus.

Lastly, to the one who put up with me and supported me throughout this effort,

who read and reread every page, my gratitude and thanks go to my best friend, my wife,

Judith Meyer. Without her unwavering support and editorial help my thesis advisors

would never have succeeded in guiding me through this process in such a smooth and

timely manner.

xv


xvi

I. INTRODUCTION A. PURPOSE

It is generally accepted that the utilization of commercial items will substantially

reduce cost and lead times necessary to get systems fielded. However, there may be

significant downsides and drawbacks to utilization of commercial items that should be

recognized, analyzed and for which corrective actions should be planned to mitigate the

implementation risks.

The purpose and objective of this thesis is to identify many of the Program

Managers’ (PM) or industries’ critical “issues” associated with implementing and

integrating commercial items into a DoD program. The researcher will also propose

process solutions that may alleviate many of the problems and thereby help mitigate

commercial item implementation risks.

B. SCOPE OF RESEARCH The scope will include: (1) a review of DoD regulations as they relate to

commercial items, (2) an in-depth review of the available commercial items literature, (3)

a compilation and evaluation of adverse experiences by PMs and industries using

commercial items from DoD audit reports and (4) an analysis of the audit results of the

PMs and industry implementing current DoD’s commercial item policies. The thesis will

conclude with a recommendation of methods and processes that work effectively when

utilizing commercial items in DoD programs.

C. RESEARCH METHODOLOGY The data for this study was obtained from several sources. First, the researcher

conducted an extensive review of available literature. This literature review consisted of a

search of books, journal articles, microfiche, the worldwide web and other library

information resources pertaining to commercial items. Finally, the researcher conducted a

thorough review of the Internet for current DoD policies, regulations and audit reports

relating to commercial item implementation.

1

Secondly, telephone interviews were conducted with major DoD acquisition

agencies involved in the commercial item acquisition process. These agencies included

Under Secretary of Defense for Acquisition Logistics and Technology, USD(AL&T),

Deputy Under Secretary of Defense for Acquisition Reform, DUSD(AR), Assistant

Secretary of the Army for Acquisition Technology and Logistics, ASA(AT&L), and

Headquarters Army Material Command, HQAMC, where it was determined that there

was no current, consolidated, PM commercial item usage data available. Telephone

interviews were also conducted with a small number of PM and industry personnel in

order to identify current problems and issues being experienced with commercial item

utilization.

Thirdly, a consolidated data call was obtained from Communication Electronics

Command (CECOM), which contained issue papers from three Program Managers (PM):

PM-Physical Security Equipment, (PM-PSE); PM-Defense Communications and Army

Switched Systems, (PM-DCASS); and PM-Global Positioning System, (PM-GPS); and

from the Competition Management Division Legal Office. [SMC]

Finally, additional data was obtained from General Accounting Office (GAO),

Army Audit Agency (AAA), and Department of Defense Inspector General (DoD IG)

audit reports.

D. RESEARCH QUESTIONS

To achieve the objectives of this study, the primary research question was as follows:

What are the most important problems and solutions associated with the use of

commercial products within defense acquisition?

From the basic research question, the following subsidiary questions were developed:

1. What is the definition of commercial items and their relevance to DoD?

2. What are DoD’s requirements and objectives regarding commercial item

usage?

3. What are the critical problems or issues confronted by PMs utilizing

commercial items?

4. How can PM problems or issues utilizing commercial items be resolved?

2

E. ORGANIZATION OF STUDY

The researcher will first review the background of commercial items and the DoD

commercial item requirements in some detail in Chapter II. In Chapter III, the Program

Managers’ and industries’ experiences utilizing commercial products within their DoD

programs will be identified and categorized for further study. In Chapter IV, the PM and

industry critical problems and issues are analyzed. In Chapter V, potential process

solutions will identify the best practices and conclude with areas for further research.

3


4

II. BACKGROUND

A. COMMERCIAL ITEMS DEFINED The Federal Acquisition Regulation (FAR) defines the terms “commercial item”

and “nondevelopmental item” as follows:

1. Commercial Item

a. Any item, other then real property, that is of a type customarily used for nongovernmental purposes and that has been sold, leased, or licensed to the general public, or has been offered for sale, lease or license to the general public.

b. Any item that evolved from an item described in paragraph

a, above through advances in technology or performance that is not yet available in the commercial market in time to meet the delivery requirements of the solicitation; but will be available in the commercial marketplace in time to satisfy the delivery requirements under a Government solicitation.

c. Any item that, but for modifications of a type customarily

available in the commercial market or minor modifications made to meet DoD requirements, would satisfy the criteria in paragraph a or b, above, etc. [FAR Part 2]

2. Nondevelopmental Item

Any previously developed item used exclusively for governmental purposes by a Federal agency, a State or local government, or a foreign government with which the U.S. has a mutual defense cooperation agreement. [FAR Part 2]

Because the specific emphasis within DoD is to increase the utilization of

commercial items, and nondevelopmental items (NDI) are placed second in the priority

of implementation, this thesis will only address commercial item usage. NDI literature

will be reviewed inasmuch as problems experienced utilizing NDI, will for the most part,

apply to commercial item usage as well. Also, there is a great deal of indiscriminate and

imprecise terminology usage. Thus, when the term “NDI” is used, the literature may 5

actually be referring to a commercial item. The specific study of nondevelopmental items

will be excluded from this study, but many of the problems and issues found when

utilizing commercial items also apply when using nondevelopmental items. Thus, the

same types of precautionary and risk-reducing tactics as recommended in this thesis for

commercial items may be used when acquiring nondevelopmental items.

Government Off-The-Shelf (GOTS) is also a commonly used term for

nondevelopmental items (NDI) that are government-unique items in use by federal, state

or other governmental agency or by a foreign government with which the United States

has a mutual defense cooperation agreement. This area will also be excluded and this

term will not be utilized in this thesis.

Commercial Off-The-Shelf (COTS) is a familiar and well-used term within

industry and DoD. However, there is no definition of COTS in the FAR Part 2. As used,

COTS is traditionally defined as its name implies, i.e., a commercial item that is

purchased off the commercial shelf without any modification whatsoever. This term will

not be used in this thesis except where quoted, in favor of the broader term Commercial

Item that includes COTS as a subset.

B. BROADENED DEFINITION OF COMMERCIAL ITEMS The Federal Acquisition Streamlining Act (FASA) of 1994 and the Federal

Acquisition Reform Act (FARA) of 1996 broadened the definition of a commercial item.

As they redefined it, systems, subsystems, and components other than real property are

considered commercial items or modified commercial items if they are of a type that can

be categorized into one of the following areas:

1. Has been offered for sale, sold, leased, or licensed to the general public.

2. Evolved through technology or performance advances and will be

available in the commercial marketplace in time to meet proposed Government delivery

requirements.

3. Required modification of a type customarily available in the commercial

marketplace or required minor modification of a type not customarily available in the

commercial marketplace to meet DoD requirements.

6

Besides broadening the definition of commercial items, the 1994 FASA and 1996

FARA Acts enabled DoD to revise and modify acquisition regulations. DoD was in a

position to:

1. Simplify contracting procedures for acquisitions not costing more than $5

million, when contracting officers reasonably believe that only commercial items will be

offered;

2. Exempt the contract requirement to provide cost or pricing data for

commercial items;

3. Give contracting officers flexibility in determining price reasonableness

for contracts, subcontracts, and modifications;

4. Provide contracting officers a list of Federal Acquisition Regulation

(FAR) provisions that are not applicable for commercially available items; and

5. Remove more than 30 contractor certification requirements that are not

specifically imposed by statue.

[FARA, FASA]

C. COMMERCIAL ITEMS CLARIFIED On January 5, 2001, Dr. J. S. Gansler, then Under Secretary of Defense for

Acquisition, Technology and Logistics, issued a policy memorandum to clarify and to

help overcome some of the barriers being experienced within the Department of Defense

in utilizing commercial items. An Integrated Process Team (IPT) had been formed at his

direction and was headed by both the Deputy Under Secretary of Defense for Acquisition

Reform (DUSD (AR)) and the Director of Defense Procurement. The IPT was chartered

to review DoD commercial item determinations and evaluate whether additional

guidance, tools, or training were necessary. Dr. Gansler’s memorandum says that the IPT

found “inconsistent commercial item determination and weak market research among the

obstacles that exist to broadening the use of commercial items within the DoD.” [DoD.

USD(AT&L), 5 January 2001]

Dr. Gansler’s memorandum also provided clarifying definitions of FAR Part 12

for greater consistency within DoD. Three of the most important of these are as follows:

Commercial Off-the-Shelf (COTS): A product does not have to be commercial-off-the-shelf (COTS) to meet the “commercial item”

7

definition. COTS items are a subset of commercial items. The commercial item definition is much broader than products that are presently available off-the-shelf. It includes items that have only been “offered” for sale, lease, or license to the general public, as well as those that have evolved from a commercial item and are offered for sale, even if not yet available in the commercial marketplace. However, evolved items must be available in the commercial marketplace in time to satisfy solicitation delivery requirements. In addition, all other elements of the commercial item definition at FAR 2.101 must also be met. Modified Commercial Items: When items available in the commercial market cannot meet the Department’s need, DoD must determine whether market items can be or have been modified so that FAR Part 12 can be used. Two types of modifications are available: (1) modifications of a type available in the commercial marketplace; and, (2) minor modifications of a type not customarily available in the commercial marketplace made to Federal Government requirements. For modifications of a type available in the commercial marketplace, the size or extent of modifications is unimportant. For minor modifications, the item must retain a predominance of nongovernmental functions or physical characteristics. “Of a Type”: The phrase “of a type” is not intended to allow the use of FAR Part 12 to acquire sole-source, military unique items that are not closely related to items already in the marketplace. Instead, “of a type” broadens the commercial item definition so that qualifying items do not have to be identical to those in the commercial marketplace. The best value offer in a competitive Part 12 solicitation can be an item that has previously satisfied the Government’s need but has not been sold, leased, licensed, nor offered for sale, lease or license to the general public (a nondevelopmental item as defined in 10 USC 403 (13). In this scenario, the phrase “of a type” allows the best value offer to qualify for a Part 12 contract as long as it is sufficiently like similar items that meet the government’s requirement and are sold, leased, licensed, or offered for sale, lease or license to the general public. In such instances, “of a type” broadens the statutory commercial item definition to allow Part 12 acquisition of a government-unique item that can compete with commercial items that meet the government’s requirement. This avoids the undesirable result of shutting out otherwise price-competitive preexisting suppliers of government-unique items from Part 12 solicitations. [DoD. USD(AT&L), 5 January 2001]

Since COTS has been defined as a subset of “commercial items” and Dr.

Gansler’s memorandum specifically addresses the broader scope of commercial items,

8

this researcher will use the term “commercial item(s)” throughout this thesis and the term

commercial off-the-shelf (COTS) will not be used.

A memorandum titled Commercial Acquisitions, dated 26 March 2001, was

issued by the Acting Assistant Secretary of the Army for Acquisition, Logistics &

Technology, incorporating a plan titled, “Implementation Plan for Increasing the Use of

FAR Part 12”. The plan announced a policy that all contractually provided services (with

the exception of services under FAR Part 36) were presumed to be commercial and that

FAR Part 12 policies and procedures would be used to buy those services. The plan

further states, “For those services where the results of market research indicate that the

service is not commercial, the local Competition Advocate must approve the commercial

determination.” (ASAAL&T) This plan puts even more emphasis on the utilization of

commercial items within the Army, which may open the way for all DoD.

D. DOD REGULATION 5000.1: COMMERCIAL ITEM REQUIREMENT Government regulation echoes the statutory preference for commercial items and indicates their importance to DoD. The best evidence of this is DoDR 5000.1, as noted below.

In response to user requirements, priority consideration shall always be given to the most cost-effective solution over the system’s life cycle. In general, decision-makers, users, and program managers shall first consider the procurement of commercially available products, services, and technologies, or the development of dual-use technologies, to satisfy user requirements, and shall work together to modify requirements, whenever feasible, to facilitate such procurements. Market research and analysis shall be conducted to determine the availability, suitability, operational supportability, interoperability, and ease of integration of existing commercial technologies and products and of non-developmental items prior to the commencement of a development effort. [DoD Directive 5000.1, paragraph 4.2.3]

E. MARKET RESEARCH

Market research is the process of collecting and analyzing data about the products

and technologies available in the marketplace. The market research IPT then determines

which product or technology satisfies the agencies procurement needs. Market research

has been a statutory requirement for almost 20 years, since the passage of the

Competition in Contracting Act of 1984. This act requires the use of market research and

9

procurement planning to promote the use of competitive procedures in federal

contracting.

In November 1990, Congress reemphasized market research for the DoD in the

National Defense Authorization Act for Fiscal Year 1991 (P.L. 101-510). The goal of

this act was to encourage DoD to save money and reduce acquisition cycle time by

buying products that were commercially available or had already been developed.

The Federal Acquisition Streamlining Act (FASA) of 1994 stipulated that all

federal executive agencies must conduct market research before developing new

specifications for procurement or soliciting bids for contracts exceeding $100,000.

FASA (Section 8104, paragraph 2377 of Public Law 103-355) requires heads of agencies

to use the results of market research to determine whether there are commercial items

available that meet the agency’s requirements, if those requirements were modified to a

reasonable extent. For example, if a requirement specifies that an aircraft must meet a

maximum speed of 100 mph but there is a commercial item available with maximum

speed of 90 mph, the requirements community must seriously consider reducing that

requirement so that the commercial item can be used. If suitable commercial items are

not available, the agency will then use nondevelopmental items.

The FAR implemented those market research provisions and essentially

established market research as a tool for identifying sources to ensure competition as well

as commercial products to meet an agency’s needs. FAR Part 10 requires that agencies

conduct market research appropriate to the circumstances before developing new

acquisition requirements documents. If market research establishes that a commercial

item cannot fill the Government’s need, agencies are required by FAR 10.002(c) to

reevaluate the requirement for possible restatement to enable use of commercial or

nondevelopmental items, as defined in FAR 2.101. The findings of market research must

be documented per FAR 10.002 (e).

F. COMMERCIAL ITEM USAGE CONCEPT

The concept behind greatly expanding the usage of commercial items in DoD

applications is best explained in a study produced by the USD(AT&L), as follows:

10

Expanding the use of commercial items in Department of Defense (DoD) systems offers the DoD opportunities for reduced cycle time, faster insertion of new technology, lower life cycle costs, greater reliability and availability, and support from a more robust industrial base. [DoD. ODUSD(AR), 14 July 2000]

Because of the competitive pressures experienced by commercial manufactures to

remain in business, they are forced to develop and produce the best possible quality

products at the most competitive cost. DoD, therefore, will reap the benefits and get the

best value without expending the time and funds necessary to develop, test and field a

similar product. Also, in areas such as computers and software, the speed of technological

development and acquisition by commercial industry surpasses that which the

Government has been able to achieve. Therefore, commercial items will save the DoD

both time and money, as well as keeping them technologically competitive.

Clearly, the mandate of the USD(AT&L) is for DoD to dramatically increase the

utilization of commercial items to improve the acquisition process by speeding it up,

reducing costs, and keeping pace with commercial technology development. As the

USD(AT&L) memorandum says:

To the maximum extent possible, commercial acquisitions should be conducted using FAR Part 12. The use of FAR Part 12 is designed to provide the Department of Defense (DoD) with greater access to commercial markets with increased competition, better prices, and new market entrants and/or technologies. [DoD. USD(AT&L), 5 January 2001]

G. DOD COMMERCIAL ITEM REQUIREMENTS

USD(AT&L’s) memorandum of January 5, 2001 provided further specific

commercial item acquisition dollar-value goals for DoD to meet. The two most important

of these are quoted as follows:

1. Each Service and Defense Agency should double the dollar value of FAR Part 12 contract actions awarded in 1999 by the end of fiscal year (FY) 2005. This would bring the DoD total FAR Part 12 contract actions from $12.6 billion to $25.2 billion.*

11

2. Each Service and Defense Agency should strive to increase the number of FAR Part 12 contract actions awarded to 50 percent of all Government contract awarded by the end of FY 2005.* *For purposes of these goals, a contract action is defined as any new contract award and/or new delivery order placed against a contract awarded with a value greater than $25,000. [DoD. USD(AT&L), 5 January 2001]

In support of these objectives, Dr. Gansler’s memorandum tasked the DUSD(AR)

to track the Department of Defense progress in meeting these goals. He also requested

that the DUSD(AR) and the Director of Defense Procurement charter an Integrated

Process Team (IPT) to: “determine the feasibility of establishing a pilot program so that

the Services and Agencies may collect market research and Commercial Item

Determinations in a central database, or developing tools to assist in ensuring commercial

item determinations are reasonably consistent.” [DoD. USD(ATL), 5 January 2001]

It is clear that it is DoD policy to strongly encourage the utilization of commercial

items in the DoD acquisition process. In support of using commercial items and in order

to shorten the time it takes to complete an acquisition, the process of trading user “gold

plated” requirements for commercial items is an option that must be strongly considered.

However, any such tradeoffs must be critically examined in order that real user needs,

necessary to meet emerging threats, are not sacrificed for the commercial item goal. H. USING COMMERCIAL ITEMS

The use of commercial items in DoD is not new; many programs have used

commercial items to some extent. For example, a vehicle program might use commercial

tires and batteries, with everything else developed especially for the Government

program. Or a complete program, such as a truck, might be a slightly modified

commercial item. The extent of commercial item usage varies from program to program.

With the new emphasis on using commercial items within DoD, a new paradigm of

system acquisition thinking is being created. The 2001 system acquisition management

process graphic representation from DoDI 5000.2 is shown in Figure 1.

12

Figure 1. Defense Acquisition Management Framework

(From: DoDI 5000.2)

The model process is separated into three segments: pre-systems acquisition,

systems acquisition, and sustainment. This process is then broken down contextually, by

OSD, into the Traditional Model, as shown in Figure 2. The Traditional Model consists

of three segments as follows: 1) System Context, which includes the requirements

definition and concept exploration phases of systems acquisition, 2) Architecture &

Design, which includes the design & development phase of systems acquisition, and 3)

Implementation, which includes the building, fielding and supporting phases of the

acquisition system. This Traditional Model is then transposed into the Recommended

Model, which overlaps the three areas above and overlays the process of Simultaneous

Definition and Tradeoffs as the new paradigm for increased utilization of commercial

items within DoD programs. As the OSD says, “increased reliance on commercial items

implies a different paradigm of systems acquisition.” [DoD. ODUSD(AR), pg 3, 14 July

2000]

13

The greatest difference between the two models shown in Figure 2 is that greater

interaction is required between the system context, architecture and design segments

when using commercial items, as shown by the overlapping of contextual areas. This

new model indicates that there is a greater need for use of Integrated Product/Process

Development (IPPD) and IPTs as the overlapping areas indicate. As the OSD states:

“Managing this interaction requires unprecedented cooperation among the program

office, the stakeholders, the contractor, and in many cases the vendor in order to effect

the tradeoffs necessary to keep the program on track.” [DoD. ODUSD(AR), pg 3, 14 July

2000]

Traditional Model Recommended Model

Figure 2. Traditional Model vs. Recommended Model

(From: ODUSD(AR), pg 3, 7/14/2000)

PMs must not only manage a traditional system but now must also integrate the

selection of commercial items in varying degrees into their program, along with all of the

problems associated with commercial item usage. Many acquisition programs have had

problems or have failed due to a lack of consideration for the uniqueness of commercial

item usage in their programs, as well as inattention to the differences noted below.

Some of the most important new paradigm considerations and issues a PM must

deal with are as follows:

14

1. The marketplace, not the PM, drives development of the commercial product.

The commercial marketplace is driven by customer demand, customer satisfaction

and competition. Because DoD’s demand for commercial products is such a small portion

of total commercial demand, DoD cannot dictate development to commercial producers.

Using software as an example, it is noted that, “DoD accounts for less than 1% of the

U.S. market, and less than 0.6% of the international market.” [Anderson, pg. 4, 1998]

2. A change of the commercial product (which is almost a certainty) will drastically affect program supportability.

Change is the norm in the commercial item marketplace, and must be carefully

planned for by PMs when using commercial items in their programs. Planning for

program supportability over the expected life cycle of the product is critical. A method

for handling obsolescence should be devised, such as preplanning product improvement

or upgrading commercial items with new technology insertion along with retirement of

the obsolete items. Another method cited by ODUSD(AR) was “by adopting an open

system architecture with modular designs, maintaining close relationships with

commercial item vendors, and monitoring the marketplace, one particularly successful

program not only avoided technological obsolescence, but also developed a “sparing”

model that reduced the cost of spare components by 40%.” [DoD. ODUSD(AR), pg. 17,

14 July 2000]

3. Integrating a commercial item into a program is fundamentally different from developing a custom product, and may require similar PM attention.

In the development of a custom product, the PM dictates the behavior of

components and interfaces among components. However, as ODUSD(AR) indicates,

“Program managers who use commercial items have little insight into how the

commercial items are put together, how they behave, and why.” [DoD. ODUSD(AR), pg.

5, 14 July 2000]

4. Greater teamwork among stakeholders during the selection of commercial items from the marketplace is essential to keep the program on track.

There is a dynamic interaction necessary between stakeholders when using

commercial items in order to adequately perform commercial item market research,

15

determination and, if necessary, modification of user requirements, establishment of

commercial item logistic support requirements and determination of system reliability.

As ODUSD(AR) states, “Managing this interaction requires unprecedented cooperation

among the program office, the stakeholders, the contractor, and in many cases the vendor

in order to effect the tradeoffs necessary to keep the program on track.” [DoD.

ODUSD(AR), pg. 3, 14 July 2000]

5. There must be a willingness to adapt the “requirements” to the capabilities available in the marketplace rather than adapting the commercial capabilities to the DoD “requirements.”

Paring down requirements to those absolutely essential to the program, as

opposed to those “nice to have,” will allow for requirements flexibility to enable

choosing an acceptable commercial item.

ODUSD(AR) notes that, “Requirements specification must be flexible and

negotiable. A traditional development model that specifies all system requirements prior

to considering the capabilities available in the marketplace is ill suited to the

development of systems incorporating commercial items.” [DoD. ODUSD(AR), pg. 8, 14

July 2000]

6. Commercial item usage means that many of the normal acquisition activities (such as system development and sustainment) will be repeated throughout the lifecycle of the program, because a commercial item is never really complete (i.e., it is always changing).

Frequent change in the commercial item’s configuration will mean constant

monitoring of the impact to the program, as indicated by ODUSD(AR): “However, new,

changed, and obsolete commercial items necessitate repeated cycles of requirement

definition, commercial item evaluation, and system engineering. Some form of

replanning and reengineering will be ongoing throughout the life of the system.” [DoD.


7. The product must be tested or verified to ensure that required capabilities are in fact met as promised by the commercial item manufacturer.

A commercial item’s capability may not always be as stated, and should always

be verified. As noted by ODUSD(AR), “Another program that was using multiple

16

commercial items found that even basic, advertised capabilities of commercial items had

to be tested before the program could begin its planned integration testing.” [DoD.


I. POTENTIAL BENEFITS OF USING COMMERCIAL ITEMS

There are unique obstacles when using commercial items. However, when

addressed correctly, the benefits can far outweigh the risks. Programs have documented

great savings of time and money when the unique challenges encountered by the use of

commercial items have been adequately addressed. However, the use of commercial

items does not reduce or eliminate the risks associated with the traditional acquisition

processes. It is critical that the overall system be guided through all the traditional

acquisition processes of engineering, development, integration, testing, delivery and

sustainment, and that each program be well managed. As the ODUSD(AR) states, “While

there are significant benefits, these benefits can be attained only by understanding and

addressing the significant new challenges that are driven by the fundamental differences

between building items and buying them.” [DoD. ODUSD(AR), pg. 20, 14 July 2000]

17


18

III. PM & INDUSTRY COMMERCIAL ITEM USAGE EXPERIENCE & PROBLEMS

A. INTRODUCTION

This chapter will document many of the PM and industry “issues” associated with

implementing and integrating commercial items into a DoD program. These issues come

from several sources, such as the Army Audit Agency, General Accounting Office

(GAO), DoD Inspector General (DoD IG), and data from a CECOM PM.

Issues will be presented by audit agency report or issue paper topic and then

segregated, when applicable, into the acquisition topic areas as follows: needs

determination; acquisition strategy; market investigation; concept formulation; risk

analysis; management of integration; test and evaluation; logistics support, including

reliability; support requirements; and technical data for competitive reprocurement.

B. BACKGROUND INFORMATION

In many of the referenced Army documents that follow, the term

“nondevelopmental item” is used synonymously with the term “commercial item” as

previously defined in this thesis. Where the term, “nondevelopmental item” is used in the

documentation, it will be replaced with “commercial item” throughout this portion of the

research, except when “nondevelopmental item” is used in a quotation.

Since understanding the issues presented herein requires some understanding of

the acquisition process areas discussed, basic definitions are provided below for each of

the following concepts: requirements determination; acquisition strategy; market

investigation; concept formulation; management of integration; test and evaluation; and

logistics support, which includes: reliability, support requirements, and technical data for

competitive reprocurement.

1. Acquisition Strategy

All major material acquisition programs require an acquisition strategy. In the

Army, for example, the material developer prepares the acquisition strategy during the

concept exploration phase, prior to being approved at the milestone I/A decision point.

19

The acquisition strategy documents the approach for satisfying a requirement that

includes plans and justifications for streamlining the acquisition process - that is, plans to

shorten or eliminate any of the acquisition phases. The goal of the material developer is

to tailor the strategy to create the shortest path through the acquisition process that will

successfully satisfy the material requirement.

If the acquisition strategy is not accurately identified early in the process,

incorrect phases of the acquisition process will be applied to one of the following: the

system, the part, the component, or to the integration of all these into a system. Any

streamlining that occurs which eliminates acquisition phases or steps in the commercial

item acquisition process must be fully addressed in the acquisition strategy. The

treatment of basic low-risk, off-the-shelf commercial items as medium or high-risk

commercial items will add unnecessary delay and cost to the project. Likewise, if high or

medium-risk projects are categorized and treated incorrectly, they will not be processed

correctly. Potentially, additional cost and time will be incurred because these projects

have not been tested and evaluated properly, or had other program risks that have not

been properly addressed. Acquisition strategies should also be updated and resubmitted

promptly when conditions change significantly. This will allow decision-makers to

properly assess the impact of such changes. Users of the items should be involved early

in the acquisition process in order to fully understand the nature, potential limitations,

and advantages of commercial and nondevelopmental items. Only in this way can the

user and acquisition community maintain the flexibility that is needed in establishing

requirements to make effective use of commercial items and nondevelopmental items in

their programs.

2. Market Investigations

Current DoD policy encourages the use of commercial products. However,

procurement agencies have the burden of determining whether commercial items are

available. This process is called market research.

Market research consists of two related techniques: market surveillance and

market investigation. Market surveillance is used to maintain a current knowledge of

market availability within a particular area of technical expertise. Market investigations 20

evaluate commercial items in the marketplace to determine whether they can meet

specific military requirements, or whether a new acquisition development project should

be initiated.

Market research is defined in more detail in chapter II, paragraph E, of this thesis.

Market investigations should satisfy three criteria as follows: (1) identify all potential

commercial item or nondevelopmental item sources (which includes other military

services, other governmental agencies and allied countries), (2) be coordinated with

independent evaluation activities, and (3) obtain enough actual test and performance data,

if applicable, to assess critical issues relating to the operational suitability of the

commercial or nondevelopmental item.

3. Concept Formulation

The materiel developers (U.S. Army Materiel Command) and the combat

developers (U.S. Army Training and Doctrine Command) jointly prepare the concept

formulation package in the Army acquisition process. The concept package includes a

tradeoff determination and a best technical approach, which are prepared by the material

developers. It also includes an Analysis of Alternatives, which is completed by the

combat developers. The concept formulation process provides the initial up-front and

early data for defining and approving the material requirements documents and

supporting program decisions, including analysis of acquisition strategies. A joint process

is used to ensure that the material requirements are realistic, affordable and analytically

supported. These analyses, along with the market investigation, form the basis for

development of a sound acquisition strategy and formal requirements document.

4. Management of Integration

Integration projects - that is, projects that may use commercial items but have to

integrate components or systems to work together - can be high-risk, complex and costly

acquisition efforts and therefore must be managed appropriately. If integration projects

are not identified properly, there will be high risk of a program experiencing technical

problems and not meeting user requirements. Integration projects usually present the

greatest challenges and are among the highest risk programs that utilize commercial

21

items. The adverse results could impact costs and schedules as well as not meeting user

requirements, resulting in potential mission failure.

5. Logistics Support

Logistics support planning is a process that should commence early in a new

acquisition program. The goal is to influence material system requirements and design in

order to ensure that the system has optimum reliability and life cycle costs (LCC) - in

other words, maximum reliability and minimum life cycle costs. Poor logistics support

planning could result in inadequate logistics support, which in turn could adversely affect

the system’s readiness and cost.

a. Reliability

Reliability is defined as the “probability that, when operating under stated

conditions, the system or component will perform its intended function adequately for a

specified period of time.” [Air Force Space and Missile Command, pg. 1, 14 August

1998]

In programs involving R&D, a major issue that developers must consider

is system reliability. Reliability, availability, and maintainability (RAM) are key LCC

drivers. RAM must be addressed early in the acquisition process by establishing

reliability requirements that determine whether the system is economically supportable

over its entire life cycle and is able to achieve required readiness levels.

If the reliability of a commercial item is not verified as being able to meet

the user’s requirements, then the system’s LCC and readiness will be severely impacted.

Lower reliability greatly impacts the support costs, system availability and thus, mission

accomplishment. RAM requirements must be established early in order to enable the

adequate testing of commercial items or nondevelopmental items. Quantitative reliability,

availability and maintainability requirements should be established for all commercial

items, with the exception of passive commercial items with no identifiable failures.

b. Support Requirements

Support requirements have to do with the way the system is maintained. It

can be supported by either a contractor supply and maintenance system or a military

22

organic (in-house) supply and maintenance system. Contractor or organic support

requirements must be analyzed via cost-benefit analysis methodology in order to justify

the selected support decision. Support requirements specifically address how commercial

items will be maintained and supported when fielded due to the fact that Government

access to commercial item technical data will most likely be limited. Consequently,

availability of supplied parts may cease before the intended military lifecycles of the

items expire. Support requirements analysis is crucial, since military systems are kept in

the operational inventory far longer then the lifespan of most commercial items.

c. Technical Data for Competitive Reprocurement

Technical data comes in many forms, including technical maintenance and

operating manuals, engineering drawings and specifications, repair parts lists, and special

tool lists, to name a few. All acquisition programs require technical data to some degree

for logistics support of the system. The data that is acquired for a new program should be

tailored to the specific and well planned needs of the program. Generally, the need for

technical data to support reprocurement decisions should be made via economic analysis,

which will indicate the best overall strategy for obtaining spare parts or replacement

systems. As previously mentioned, Government access to commercial item or

nondevelopmental item technical data will most likely be limited, and availability of parts

supplied may cease before the intended military lifecycles of the items expire. An

analysis of the program’s future requirements for technical data is essential, since

military systems are kept in the operational inventory far longer then the lifespan of most

commercial items. Thus, there must be some provisions made for the supply of spare

parts after the commercial items have become obsolete in the commercial marketplace.

6. Test and Evaluation

The purposes of test and evaluation in the development of a defense system and

its acquisition process are the following: to determine the feasibility of conceptual

approaches in order to minimize design risk, to identify design alternatives, compare and

analyze tradeoffs, and to estimate operational effectiveness and suitability. As a system

progresses through design and development, the emphasis in testing moves gradually

from development test and evaluation (DT&E) - which is chiefly concerned with the

23

attainment of engineering design goals - to operational test and evaluation (OT&E),

which focuses on questions of operational effectiveness, suitability, and supportability.

DT&E is normally planned and conducted by the developing agency during the

concept and technology development phase to assist in the selection of the preferred

alternative system concepts, technologies, and designs. During the system development

and demonstration phase, DT&E demonstrates that engineering is reasonably complete,

that all significant design problems are in hand, and that the design meets its required

specifications in all areas, such as performance, reliability, and maintainability.

The purpose of OT&E is to verify that only operationally effective and

operationally suitable systems are approved for production, thereby meeting both mission

needs and the minimum operational performance requirements of the user. For major

systems, OT&E is normally planned and conducted by a major OT&E field agency

located within the DoD component. This operational test agency must be separate and

independent from the developing/procuring agency.

Operational testing is the primary means of evaluating system performance in a

combat representative environment. It is structured to determine the operational

effectiveness and suitability of a system, as well as to determine if the minimum

acceptable operational performance requirements have been satisfied.

Operational effectiveness refers to the ability of a system to accomplish its

mission in the planned operational environment. Suitability is the degree to which a

system can be placed satisfactorily in field use, taking into account such factors as the

system’s reliability and maintainability. [DSMC, January 1990]

In the commercial arena, commercial items may already have test data available;

however, unless the data has been verified by an independent evaluator it should not be

accepted.

In all usage of commercial items in major programs, OT must be performed in

order to verify or establish the system’s performance in the representative operational

wartime environment.

24

7. Culture

It has been found that culture plays a key role in many negative decisions that are

made in the acquisition community. As used here, culture can be defined as the

background, mores, traditions and customs, or way of life of a group.

In the acquisition arena, there are many key players and contributors to the PM’s

mores, traditions, and way of life. Some of the key players are the military chain-of-

command, the Congressional and military budgeters, the program’s contractor, the

Government contracting agency, the program decision authority, the user community,

and the test community, all with needs that may diverge and actually threaten program

success.

For example, there are Congressional budget “rewards” that force a PM to do

everything within his power to see that his program doesn’t fail a test. If he fails a test, he

may not have a program, because his program funding will be cut. Therefore, he doesn’t

have a keen interest in failing as many times as is necessary to adequately “learn” from a

developmental test. Maturing the technology through the test-fix-test process would

ensure that his program will not have later technological setbacks that would in turn drive

schedule and cost overruns, as well as increasing life-cycle support costs of the program.

Nor does the PM welcome the tester, when he says, “I’m here to help you.” The culture

and reward system by which the PM lives dictates a different response. As testimony to

this, a GAO study found that the processes and culture within DoD have contributed to a

feeling of great aversion by PMs, both towards the test community as well as towards the

testing of the PMs’ weapon systems. The GAO states the following:

Testing plays a less constructive role in DoD because a failure in a key test can jeopardize program support. Specifically, test results often become directly linked to funding and other key decisions for programs. Such a role creates a more adversarial relationship between testers and program managers. [GAO, pg. 6, 31 July 2000]

25

The negative cultural pressures on a PM are further exemplified by an article in

the Marine Corps Times, titled, “We Need to Lie -- To Save the Osprey, Squadron Boss

Ordered Marines to Fake Records.” [Brinkley, pg.8, 29 January 2001] This article

alludes to the negative cultural pressures as perceived by the Marine PM, Lt. COL. Odin

F. Liberman that forced him to lie to protect his program. The article was based upon an

anonymous letter and audiotape received by the administrative offices of Navy Secretary

Richard Danzig. The article quotes the barely audible audiotape of a voice identified as

Liberman’s that states,

The reason we need to lie is … [to] manipulate the data until Milestone III when the aircraft passes its final evaluation and enters full rate production, this program is in jeopardy … Everybody says readiness is in bad … It’s something that everybody is focused upon.

[Brinkley, pgs 8 & 10, 29 January 2001]

8. Army Definitions

During the time of the Army audits cited below, the definition of the various

classifications of what was known then as nondevelopmental items (which would be

classified now as commercial items), is quoted as follows:

• Basic. These are off-the-shelf items used in the same environment for which they were designed. The items should not require development or modification to be acceptable. Available information (such as test data, technical publications and reliability data) should be obtained from the item source and evaluated to avoid costly duplication of effort.

• Adaptation. These are off-the-shelf items adapted for use in a different environment that they were designed for. The items require some modification for military use. Test and evaluation usually is required to ensure the modified items meet user requirements.

• Integration. These are nondevelopmental items used as subsystems or components of larger systems. The efforts require a prove-out phase involving systems engineering and test and evaluation to make sure the total system meets user requirements and is producible.

[USAAA, pg. 8, 19 March 1990] 26

C. COMMERCIAL ITEM USAGE ISSUES

Issues will be presented by audit agency report or issue paper topic and then

segregated into such areas as needs determination, acquisition strategy, market

investigation, concept formulation, design, risk analysis, management of integration, test

and evaluation, and logistics support. Logistics support will be further divided into the

following areas: reliability, support requirements, and technical data for competitive

reprocurement.

1. U.S. Army Audit Agency, NE 90-206, Acquisition of Nondevelopmental Items: U.S. Army Natick Research, Developmental and Engineering Center, Natick, Massachusetts. 19 March 1990

This audit reports that the Army often based acquisition strategies on incomplete

information due to the fact that the Army did not properly conduct market investigations

and concept formulation analysis.

Some of the Army Audit Agency specific findings were as follows: “One system

… involved system integration, but went from program initiation to production without

sufficient testing.” [Pg 2] This resulted in an Army worldwide modular print system

being purchased without sufficient required testing, which would have ensured that the

system met Army requirements. “Planned maintenance concept for an automatic reserve

parachute ripcord release was inconsistent with requirements documents, and … did not

tell decision makers the true cost or extent of required overhauls.” [pg. 2]

a. Acquisition Strategy

The Army Audit Agency reviewed 15 of 22 commercial item projects, and

found that 14 of the 15 projects did not identify the Army identification category of

commercial item (i.e., basic off-the-shelf, adaptation or integration item). For 10 of these

projects, the strategy was inconsistent with the type of category of commercial item being

acquired. For example, basic off-the-shelf items were managed as adaptation projects. In

turn, adaptation and integration items were managed as other types of projects. For basic,

low-risk, off-the-shelf projects, this added unnecessary delay. In addition, complex, high-

risk integration projects that should have been rigorously tested and evaluated were

instead rushed into production as if they had been off-the-shelf projects. For example, as

27

the Army Audit Agency states “One system -- the Modular Print System -- involved

system integration, but went from program initiation to production without sufficient

testing.” [pg. 2] This project was actually a complex system integration effort involving

risks.

These problems resulted from the erroneous supposition that by

eliminating the necessary required analysis developers would be able to streamline the

acquisition process. As the Army Audit Agency report says, “Natick Center personnel

did not appear to fully understand the various categories of commercial items and did not

believe the category was important.” [pg. 11]

b. Market Investigations

The Army Audit Agency found that market investigations either were not

conducted at all, or were inadequate for most of the commercial item projects. Army

Audit Agency reviewed 15 of 22 commercial item projects, and found that five market

investigations were not documented. Furthermore, for seven of the projects, the market

investigations were neither complete, nor had been coordinated with the independent

evaluators as required. The Natick RD&E Center’s policy at the time was to leave market

investigations incomplete or undocumented in order to streamline the acquisition process,

thereby saving time. However, without properly conducted market investigations,

managers had little assurance that all the possible alternatives or material sources had

been identified, investigated, and that the best alternative had been selected. The Army

Audit Agency said, “Natick Center often did not properly conduct market investigations

and other required analyses to develop sound acquisition strategies for nondevelopmental

items.” [pg. 2]

For example, both snow and ice-traversing equipment and a field pack

project experienced costly problems by not having a market investigation completed. In

the case of the snow and ice-traversing equipment, if a proper market investigation had

been performed, the problems with the selected off-the-shelf items could have been

recognized and evaluated prior to acquiring and testing them. In the case of the field

pack, a market investigation would have identified the deficiencies with the commercial

28

pack, which required modification to such an extent that it bore no resemblance to the

original commercial pack.

c. Concept Formulation

The research center had not prepared concept formulation packages for

any of the 15 projects reviewed, resulting in many questionable decisions. For example,

the Army developed a saltwater purification device at a cost of $250,000 and a purchase

cost of $1,320 each, when there was already a Navy-developed system available for a

purchase cost of $980 each. A proper concept formulation package would have identified

the Navy system as a possible alternative. A tradeoff analysis between cost and differing

requirements could have resulted in selection of the Navy system as an alternative, thus

saving the Army $250,000 in development costs as well as the extra $340 purchase cost

per system. This selection would also have saved the Army the two years of

developmental time needed to obtain the system.

Another example, with no concept formulation package, is a five-soldier

crew tent project, which resulted in five commercial tents being purchased and evaluated

three times at a cost of $600,000. A proper concept formulation package, including

tradeoff and other analyses, would have identified the most promising alternative for

decision makers.

2. U.S. Army Audit Agency, NE 91-204, Acquisition of Nondevelopmental Items. 17 June 1991

The audit, among other findings, found that five separate Army commodity

command groups managing 39 commercial item projects didn’t properly support

acquisition strategies.


The analysis to support the selection of commercial items, which is

necessary at the beginning of a project, was not performed. As the Army Audit Agency

found, “DA (Department of the Army) sometimes directed developers to use

nondevelopmental items, and managers therefore believed there was no real need to

perform the comprehensive, upfront analysis required to support their strategies.” [pg. 19]

29


Market investigations were not documented for 11 of the 39 projects. Of

the remaining 28 projects, a total of 21 had incomplete documentation. As the Army

Audit Agency noted, “The investigations didn’t adequately support the streamlining

actions managers took—particularly the elimination of a development or prototype phase

before production … As a result, the development effort required to adapt or integrate

these items had to occur during production.” [pg. 25]


Of 29 projects reviewed for a complete concept formulation package, none

supported this concept formulation characteristic. The majority, 22 projects, didn’t have

any of the concept package completed. For the 7 projects, only the portion of the concept

formulation package that dealt with cost and operational effectiveness was complete.

d. Management of Integration

As the Army Audit Agency stated in this report regarding integration

efforts, “Such efforts not only require research and development, but may need to

undergo both a demonstration and full-scale development phase unless risk is determined

to be sufficiently low.” [pg. 28] Eight of the Army potentially high-risk integration

projects, totaling $1.7 billion, didn’t have a research and development (R&D) phase prior

to production. Although utilizing commercial item off-the-shelf components, the systems

were high-risk integration projects for which the Army had documented only minimal

information to show that the projects would meet user requirements after integration was

accomplished.

e. Logistics Support

The Army Audit Agency reviewed 39 commercial item projects for which

logistics support costs had been projected at $24 billion. These projects had critical issues

related to logistics support, which hadn’t been addressed early enough in the acquisition

process. Three of these issues are as follows: reliability, support requirements, and

technical data for competitive reprocurement.

30

(1) Reliability. The Army Audit Agency reviewed 39

commercial item programs, many with major deficiencies. They found that 11 of these

projects had no quantitative reliability requirements established. In 13 of the projects,

reliability had not been adequately addressed during market investigations. In several

cases, contractors’ reliability estimates had been received without the performance data,

which is necessary in order to evaluate the contractor reliability claims. In 15 of the

reviewed projects, the Army Audit Agency found that material developers hadn’t

adequately conducted reliability testing prior to award of the production contract. It was

found that systems were fielded that had subsequent reliability problems, which could

affect system readiness and significantly increase costs. As the Army Audit Agency said,

“The failure to adequately address reliability early in the acquisition process for new

programs significantly increases the risks associated with using nondevelopmental items

to satisfy Army requirements.” [pg. 40]

(2) Support Requirements. Many of the 39 projects

reviewed had no reliability testing, which would have reduced the risk of critical

decisions about logistics support. In all cases, the most expeditious means available were

used to establish the logistic support plans, which were not necessarily the best or most

cost-effective decisions for the Army.

(3) Technical Data for Competitive Reprocurement. For 14

of the 39 projects reviewed, data adequate for reprocurement was not acquired.

Consequently, the Army will have to use the sole source vendor for any reprocurement of

spare parts or systems throughout their complete lifespan. Some managers assessed the

availability of data suitable for competitive reprocurement in their market analysis. But

many other market investigations either were not documented, or the availability of

technical data was not addressed during the market investigations. Some managers used

lifetime contractor support, not having considered technical reprocurement data as

necessary. Others decided that data necessary for reprocurement was too expensive. In

most cases, managers didn’t use economic analysis or any other proper analytical

technique prior to their decision. It was determined by the Army Audit Agency that

several projects would have benefited from obtaining technical data suitable for

31

competitive reprocurement. Those programs either had to acquire replenishment spares

at significantly higher prices (Lightweight Digital Facsimile), or had to find a

replacement unit because spares and contractor support could not be obtained (Basic

Generation Unit used in the Battlefield Electronics Communication System). [pgs. 44-45]

3. U.S. Army Audit Agency, NE 90-209, Acquisition of Nondevelopmental Items: Fort Monmouth, New Jersey. 11 September 1990

This audit found that the Army advocate for commercial items had not been

involved enough to make sure that program managers considered the commercial item

approach. Furthermore, it was discovered that the advocates were not adequately

monitoring commercial item projects to document “lessons learned,” or to make sure that

the two main objectives of commercial items were achieved: the reduction of program

cost and acquisition time. Also, market investigations and concept formulation analyses

either were not conducted at all or were improperly conducted, and project

documentation was either lax or nonexistent. Complex integration projects were awarded

without adequate testing to assure that they were operationally suitable and that they met

the Army requirements. Support concepts were not adequately investigated or justified to

demonstrate the cost-effectiveness of the selected support concept during daily use,

especially under wartime conditions.


Key phases and critical steps of the acquisition process were eliminated

without explanation for all of the 15 commercial item projects reviewed. Thus,

acquisition strategies that were followed were not adequately supported by analysis. As a

result, managers often followed inappropriate strategies for the category of commercial

item acquired. For example, complex commercial items requiring integration were

frequently acquired with a strategy more appropriate for low-risk, off-the-shelf items.

Seven complex commercial item projects went into production without a separate R&D

phase to obtain and evaluate prototypes. When any phase of the acquisition process is

eliminated or combined, Army regulation requires that the material developer justify their

decision in the acquisition strategy. Most of the strategies reviewed simply stated that the

commercial item approach chosen eliminated the need for research and development.

32


Market investigations were not documented for five of the 15 commercial

item projects reviewed. There was documentation to show that market investigations had

been started for 10 of the 15 projects; however, they were incomplete and unacceptable

for decision-making purposes. In some cases, PMs cited “urgent requirement” as the

reason for inadequate market investigation. However, even follow-on contract options

were awarded without justification for not having a market investigation.


The concept formulation process was not completed for any of the 15

projects reviewed. The acquisition strategies did not address this key step, nor did they

explain why the required concept formulation process was not performed. The Army

Audit Agency found that PMs felt that since they had been directed to use commercial

items by higher management, there was neither a need nor a requirement for the concept

formulation phase.

d. Test and Evaluation

Production contracts for 12 of the 15 projects were begun without

independent test and evaluation to determine whether the items met Army requirements.

These contracts were initiated in spite of the fact that all categories of commercial items

are required to undergo operational test and evaluation (OT&E) before production, unless

an independent market investigation report concludes that such testing can be reduced or

eliminated.

During a review of 15 commercial item projects, the Army Audit Agency

also found that 11 items did not have OT performed. As the Army Audit Agency says,

“In total, about $1.8 billion was spent acquiring nondevelopmental items without

adequately showing the suitability of the items in a realistic operating environment.”

[pg. 31]

33

e. Logistics Support

An Army Audit Agency review of ten commercial item projects, with

estimated LCC support costs of $24 billion, had no analysis performed that justified the

logistics support concept selected.

In the AN/PRC-127 Radio, for example, the determination was made to

use a throwaway repair concept to support the radio’s receiver-transmitter after the

manufacturer’s warranty expired. A maintenance cost-effectiveness analysis performed

prior to contract award concluded that repair costs were about the same as replacement

costs, hence the throwaway decision. However, when the contract was awarded, the

actual unit price was much higher than maintaining the receiver-transmitter would have

been. Nonetheless, the support concept was not reevaluated.

(1) Reliability. The Army Audit Agency reviewed seven

commercial item projects for their reliability. They found that the material developers did

not ensure that qualitative reliability requirements were documented during the concept

exploration phase, prior to awarding the production contracts. Three of the commercial

item programs’ requirements were not established at all. In the other four programs,

managers used the contractor’s predicted values for the quantitative requirements, which

were basically technical estimates and not based on analysis of the item’s operating

mission.

(2) Supportability. An Army Audit Agency review of ten

commercial item projects with estimated life cycle support costs of $24 billion had no

analysis performed that justified the logistics support concept selected. Logistics support

plans were usually prepared, but there was no analysis to justify those plans or to

demonstrate cost-effectiveness of the support concepts. In most cases, the end-item

contractor was to provide maintenance and supply support for the life expectancy of most

items. However, there was no analysis performed that showed that contractor support

was more cost effective than using the Army’s organic support system. For example, as

the Army Audit Agency indicated, “Program plans did not ensure that spare parts for the

AN/PRC-126 Small Unit Radio would be available at fair and reasonable prices.” [pg. 5]

34

4. Bonheim, Mike LTC, PM – Physical Security Equipment (PSE), Commercial Items Acquisition Data Call, 22 February 2001

PM-PSE acts as the PSE focal point for the Army and other Services. The DoD

Physical Security Equipment Action Group has created a Commercial Off-The-Shelf

working group to identify commercial physical security and force protection equipment

that can immediately be used to solve terrorism deficiencies. Under the Chairman Joint

Chiefs of Staff Instruction, CJCSI 5261.01A, commanders are provided funds to react as

follows: “The fund provides a means for CINCs to react to unanticipated requirements

from changes in terrorist threat level or force protection doctrine/ standards.” [CJCS, pg.

1]

Therefore, as the PSE focal point, PM-PSE has been involved in commercial item

security equipment projects, identifying several problems as follows:

a. Needs Determination

A variety of Government users are involved in the purchase of security

equipment. For the most part these users have no experience or training in the acquisition

process. This results in difficulty in obtaining a user statement of work that accurately

identifies what they need or want the product to do.

b. Reliability

There may be a multitude of buyers from different commands for the

security equipment, each having their own certification requirements. Consequently, a

vendor will not be able to certify the product for all scenarios.

Also, the vendors’ performance claims have proven inaccurate. As PM-

PSE states, “In cases involving two portable barriers both barriers failed the initial crash

test and had to be reconfigured by the manufacturer before a successful test could be

accomplished.” [Bonhiem, pg. 2]

c. Market Investigations

There has been no centralized database established for equipment that has

been purchased as well as the results of their operational use or testing.

35

d. Logistics Support

There are no durability or lifespan indicators for these physical security

commercial items. Repair procedures are nonexistent. PM-PSE states, “In many cases,

detailed repair documentation is not available. If the item fails it must be scrapped or

returned to the factory for repair - a much more difficult task in deployed units than the

civilian world.” [Bonhiem, pg 2] Determining the quantity and type of spare parts is an

issue. As PM-PSE indicates, “Determining the amount of repair and spare parts and

where there are stored and subsequently issued needs to be determined.” [Bonhiem, pg.

2]

e. Test and Evaluation

There may be a multitude of buyers from different commands for the

security equipment, each with their own testing requirements. Consequently, a vendor

will not be able to test or certify the product for all scenarios.

5. GAO/NSIAD-95-161, UNMANNED AERIAL VEHICLES: Maneuver System Schedule Includes Unnecessary Risk, 15 September 1995

The GAO states that because of premature entry into production, two previous

UAV programs suffered from costly redesigns in order to achieve acceptable system

performance. The two previous nondevelopmental UAV systems, the Pioneer and the

Hunter, were both started into production without being subjected to any operational

testing.

Premature production of the Pioneer resulted in doubling the costs for nine

systems that did not meet the operational requirements of the user. As of the writing of

this GAO report, the Hunter system was also experiencing problems and an uncertain

future after an expenditure of $627 million.

Because of the previous dubious acquisition track record, the GAO issued this

interim report to bring attention to inadequate testing aspects of the program that, it was

believed, would unnecessarily increase DoD’s program risk.

36

Test and Evaluation

This report states that the Joint Tactical Unmanned Aerial Vehicles project

office planned to begin production of the Maneuver System before making sure that the

system could meet operational requirements via operational testing prior to, or during,

low rate initial production (LRIP). Test articles are normally obtained from LRIP;

however, in this case there was no testing of the LRIP vehicles planned by the project

office.

6. GAO/NSIAD-96-2, Unmanned Aerial Vehicles: Hunter System Is Not Appropriate for Navy Fleet Use, 1 December 1995

The GAO reviewed a variant of the $4.2 billion joint UAV program planned for

use on Navy amphibious assault ships to provide reconnaissance, target acquisition, and

other military missions.

Needs Determination

The GAO finding was that, “The Joint Tactical UAV Projects Office is

proceeding with the acquisition of the Hunter shipboard variant even though all Navy

fleet commanders have stated that they do not want the system on Navy ships.” [Pg 1]

The fleet commanders opposed the Hunter UAV because of the adverse impact it would

have had on all other flight operations from their ships, and because its performance

capability was inadequate for their needs.

Despite fleet concerns, the Chief of Naval Operations planned to proceed

with procurement of the shipboard variant.

7. The DoD IG Report 96-111, Allegations Involving the Procurement of the Hunter Unmanned Aerial Vehicle-Short-Range System, 7 May 1996

From 1990 to 1996 the GAO issued six negative reports concerning the Hunter

UAV Short Range program system deficiencies, some of which are documented above.

The first report in September 1990 indicated that the Navy’s testing strategy for the UAV

would not be in a realistic operational environment, and that full-rate production would

begin prior to the system meeting the Navy’s requirements. The September 1992 report

indicates that adequate user tests were not performed in a realistic operational

environment to assure that critical system performance capabilities satisfactorily existed.

37

The December 1993 report indicates that the system didn’t meet milestone criteria for

entry into LRIP, and that testing had not been done to ensure that operational

requirements were met. The June 1995 report states that the system was logistically

unsupportable, and that tests identified serious performance problems. The September

1995 report states that the acquisition demonstrates the adverse consequences of initiating

production without first obtaining adequate assurance of satisfactory system performance.

This last report of May 1996 states that: “The audit partially or fully substantiated the

allegations that the LRIP system did not conform to contract requirements, operator

safety was at risk, reliability was inadequate and the system was never subjected to

operational testing.” [pg. 1]

In January 1996, the Under Secretary of Defense (AT&L) terminated any future

Hunter system acquisitions.


The Hunter system requirements were not met. As the DoD IG report

states, “The Government bought the first of the seven LRIP systems knowing major

performance deficiencies existed that made the system clearly fall short of contractually

defined requirements. The system has numerous hardware shortages that history shows

that the contractor will never satisfy.” [pg. 4] It can be seen that the PMO staunchly

supported the program as demonstrated by the fact that the Government contracting

officer issued 34 waivers to prevent program delays due to specification

nonconformance. The Hunter Project Office also approved 257 engineering design

changes and 125 specification changes to resolve inconsistencies.

b. Reliability

The DoD IG found that the Hunter Unmanned Vehicle LRIP did not have

adequate reliability. As the DoD IG report stated, “The Hunter system did not meet five

of nine contractual requirements for reliability, availability, maintainability, and built-in-

test.” [pg. 7] Soldiers and military instructors using the system also said that it was

unreliable. The Project Office continued to support the program, indicating that “test

results may not reflect the system’s long-term averages for the time to repair.” [pg. 8]

38

c. Test and Evaluation

The DoD IG found that the Hunter Unmanned Vehicle LRIP was not

subjected to realistic operational testing. The Project Office subjected the prototypes to

limited user tests, but never performed any operational testing on the LRIP systems. The

DoD IG stated,

The technical tests were done in a controlled, sterile environment and any advertised results were not representative of the system’s true capabilities and real limitations. The system was never subjected to test scenarios that replicated the true conditions the hardware would encounter. Credible operational tests have never been accomplished. [pg. 9]

8. DoD IG Report 95-193, DoD Hotline Allegations Regarding the Naval

Special Warfare Patrol Coastal Ship and the Rigid Inflatable Boat Acquisition Programs, 15 May 1995

At the start of the coastal ship portion of this program, the Chief of Naval

Operations (CNO) was considering three alternative approaches: a commercial item

approach using existing patrol craft technology, modification of an existing 110-foot

patrol craft, and new development. The CNO rejected new development due to cost and

time constraints. The modification approach was also rejected, since the 110-foot craft

did not meet all the operational performance requirements. Finally, the CNO approved a

Non-Acquisition Program Definition Document, which “defines and gives direction to

Advanced and Engineering Development programs that explore technologies or integrate

systems not directly related to a procurement.” [pg. 7] The Non-Acquisition Program

Definition Document determined that, “The boat’s performance and configuration

requirements were to be determined based on existing craft in the commercial

marketplace.” [pg. 7]

Thus, it was determined without further review that a commercial item program

would be selected. A firm-fixed price contract was awarded for 13 patrol craft using the

commercial item approach. Over a year later, the Supervisor of Shipbuilding imposed

General Ships Specifications (GSS) requirements on the patrol craft contractor. This

caused the program to evolve from commercial item to a developmental acquisition

39

program. The GSS requirements change contributed to greater than 40 percent cost

growth within the program.

Similarly, the Rigid Inflatable Boat (RIB) program evaluated four boats as

commercial item candidates. None of the boats met operational requirements. After the

evaluation, the PM for Support Ships, Boats, and Crafts directed the Naval Surface

Warfare Center (NSWC), “to develop a technical data package for a 33 foot (10-meter)

RIB.” [pg. 8] This directive effectively changed the program from a commercial item

program to a developmental acquisition program.

The DoD IG found that the Navy deviated from the commercial item acquisition

approach in both cases, without conducting the necessary risk analyses to support the

decision. The Navy had modified the ship requirements beyond the point at which a

commercial item could satisfy them, and then awarded a firm-fixed price contract for the

boat without adequate risk analyses to support the decision.


In the coastal ship program, the Navy combined requirements, which led

to the need for a larger platform. There was no detailed requirements review with the

program sponsor or operators. The increased requirements necessitated changes beyond

which a commercial item acquisition approach could prove satisfactory.

In the RIB program, the requirements within the Operational

Requirements Document (ORD) were more restrictive than those within the Non-

Acquisition Program Definition Document. This should have caused the Naval Special

Warfare Rigid Inflatable Boat program office to determine areas in which requirements

could have been compromised without loss of mission effectiveness.

b. Design

Specifications for the RIB were based on commercial item subsystems and

components. However, their integration into a functioning boat exceeded the design and

performance capabilities of existing vessels.

The RIB design process was accelerated by the RIB PM from 28 weeks to

13 weeks, but no time was allowed to assess or minimize the risk that this created in the

40

acquisition process. It was later determined by the Naval Sea Systems Command

(NSSC) that the design should not have been accelerated.

c. Risk Analysis

The NSCC did not perform an adequate risk analysis that would have

revealed the high technical risk caused by the failure of commercial vessels to meet both

the ORD and the greatly compressed acquisition schedule.

d. Test and Evaluation

There was no first article contract for the LRIP of a limited number of

boats prior to full-rate production for testing purposes. Limiting the initial production

contract to the first two boats for test purposes would have minimized the production risk

for the remaining boats. Because there was no first article test contract for the LRIP of a

limited number of boats prior to full rate production, there was no operational testing,

either. Eliminating testing increased the production risk that boats would not meet user

operational requirements, and would then have to be modified later at greater expense

and time.

D. COMMERCIAL ITEMS PROBLEMS, ISSUES AND CONSEQUENCES SUMMARY

The table below summarizes the problems/issues from this chapter as a “risk”,

and the “potential consequence” of the unresolved risk.

Table 1 Commercial Item Consequence Matrix

RISK POTENTIAL CONSEQUENCE

Needs Determination - User requirements improperly

identified or misunderstood

- Requirements allowed to grow; i.e., requirements are gold plated.

- Need unsatisfied; wrong product produced and purchased; resources wasted; possible

program elimination.

- “Best value” solution will not be produced; impacts to cost, schedule, technical

performance; DoD resources wasted.

41

Table 1 (continued) Commercial Item Consequence Matrix RISK POTENTIAL CONSEQUENCE

Acquisition Strategy

- Incorrect streamlining acquisition strategy used.

- Items procured without key acquisition phases complete; impacts to program cost, schedule and technical performance; DoD

resources wasted.

Market Investigations - Inadequate or no market investigation performed.

- Wrong item procured, “best value” item not procured, impact to cost, schedule and

technical performance, DoD resources wasted.

Concept Formulation

- Inadequate or no concept formulation package prepared.

- Best technical solution not selected; impact to cost, and schedule, DoD resources wasted.

Risk Analysis - A risk analysis was not performed

- Impact to cost, schedule and technical performance; DoD resources wasted; the

program may be eliminated.

Manage Integration - Inaccurate assessment of program

integration risk.

- R&D, or T&E not performed; impact to cost, schedule and technical performance; DoD

resources wasted; the program may be eliminated.

Test & Evaluation

- Testing was inadequate or was not performed.

- Best technical solution not developed; performance, reliability & maintainability

goals not met; impacts to total LCC; operational effectiveness & suitability not

proven; impacts to program cost, schedule, and technical performance; potential mission

failure; DoD resources wasted.

42

Table 1 (continued) C ommercial Item Consequence Matrix

RISK POTENTIAL CONSEQUENCE

Reliability - The program reliability was

inadequate.

- No quantitative reliability requirements established.

- Contractor’s reliability estimates not verified.

- Inadequate reliability testing prior to contract award.

- RAM is adversely impacted; program’s total LCC is adversely impacted; potential mission

failure; DoD resources are wasted.

Support Requirements

- Incorrect program support concept is selected.

- Inadequate logistics support is planned.

- Inability to support the system during a wartime environment; mission failure;

program’s total LCC is adversely impacted; DoD resources are wasted.

- Sub optimal logistics support, increased system LCC, DoD resources wasted.

Tech Data - Competitive Reprocurement

- Data adequate for reprocurement is not obtained.

- A sole source vendor must be used; reprocurement costs are increased; an inability to obtain parts; potential mission failure; DoD

resources are wasted.

Design - The design was underestimated.

- The design process was incorrectly accelerated.

- The best design will not be completed; impacts to program cost, schedule and

technical performance.

- Impact to program cost, schedule and technical performance.

43


44

IV. ANALYSIS – COMMERCIAL ITEM USAGE EXPERIENCE & PROBLEMS

A. INTRODUCTION

In this analysis chapter, all of the possible causes, as well as the underlying

known causes, of the problems and issues in the previous chapter will be presented and

analyzed. It is important to determine the potential critical causes of the problems and

issues in order to arrive at a corrective action, or “best practice,” and thus to provide a

template which PMs can use to preclude similar occurrences in the future.

The analysis will be presented as an array, in the same manner as it was in the

previous chapter. A statement of each issue from the previous chapter will be followed by

examples to demonstrate some of the known causes of that particular issue. At the end of

this chapter is a table summarizing all the known and potential causes, which have been

identified from the reports that were the basis of this research, as cited in Chapter III.

B. ROOT CAUSE PROBLEMS AND ISSUES

For the issues cited below in paragraph C, there are several recurring causes that

underlie nearly every issue. These causes can be called “root causes,” because they

appear to be at the terminus, root, or bottom of the explanation for many of the issues.

Several root causes have been identified, as follows:

1. Inadequate Education or Training within the Acquisition Community

This can always be used as an excuse for errors in judgment. Because their

leadership has told acquisition personnel that they must use commercial items, they

consider this edict as justification for the elimination of critical steps in the acquisition

process when acquiring commercial items. Without the knowledge (that education or

training would give them) of the intricacies or difficulties of each critical part of the

acquisition process, acquisition personnel may not understand that the process cannot be

abbreviated or eliminated without adequate investigation and justification. Consequently,

they suffer the risk of making a disastrously poor decision.

45

Because commercial item usage is fairly new, there are many gray areas with

respect to its correct usage in the acquisition process. Incorrect usage happens frequently

enough to make training and education an important issue.

2. Inadequate Use of Lessons Learned

In several cases previously cited, errors noted in audit reports were repeated. This

indicates a serious failure to learn from previous mistakes. The commercial item advocate

for the Army was supposed to accumulate lessons learned, but failed to do so. Within the

various Army Audit Agency reports, similar errors occurred time after time, indicating

that lessons learned are not being applied to new Army acquisition situations.

LTC Michael Bonheim indicated in his paper that in the area of market

investigations, there is no centralized database for the accumulation of lessons learned.

3. Incorrect Culture-Driven Decision Making

At the root of many poor decisions within the acquisition corps seems to be the

“culture” that drives the decision. That is, the decision is a foregone conclusion, given the

structure and forces that impinge upon the PM during his decision-making process, as

previously noted in Chapter III. This area needs to be researched in greater depth.

C. ANALYSIS OF PROCESS PROBLEMS AND ISSUES

Noted below are examples that highlight the major causes of the issues identified

in commercial item usage.

1. Needs Determination

Issue: Needs or user requirements have been improperly identified,

misunderstood, or ignored.

The Navy coastal ship program started with a requirement to use a non-

acquisition approach to “base the acquisition on existing craft in the commercial

marketplace.” [DoD IG, pg. 7, 15 May 95] Because General Ships Specifications

(military specifications) were imposed on the contractor one year later, the subsequent

acquisition decision changed from the original goal of procuring a commercial vessel to

that of procuring a vessel specifically designed to comply with military specifications.

This decision was made without either risk analysis or cost analysis to support the

46

decision. Certainly, the typical acquisition process was not followed in this case. The

causes of this procedural omission could have been any of the following: inadequate

acquisition education or training, cultural issues that stressed speed, or cutting corners

and costs at the expense of the accurate performance of the acquisition process. Another

potential cause of this procedural omission could have been the powerful influence that

culture has upon the way we perform and operate. The PM’s culture is one that is cost,

schedule, and funding driven, each of these being success oriented - sometimes

constraining or preventing performance of the very activities that would keep the PM out

of difficulty.

In the Navy UAV case, the Project Office blatantly ignored the fleet commanders’

dissatisfaction with the Hunter UAV. The causes here could have been many, such as not

having a clearly written ORD that would have identified the need to minimize

interference with existing onboard flight operations, inadequate use of IPT’s in the

acquisition process that should have included the warfighter as a team member,

inadequate training or education of the Projects Office personnel, or a culture-driven

decision by the PM to ignore the warfighter in the interest of swift project completion.

In the PM-PSE case, a variety of users had differing requirements, but no

experience or training in defining these requirements. The varied causes identified here

are as follows: inadequate teaming between user and PM, inadequate training and/or

education of the user in the acquisition process, lack of standardization of the security

equipment, lack of a centralized force protection database, and failure of the users to be

flexible enough to consider modifying their requirements in order to more easily utilize

commercial force protection items.

Issue: Requirements were allowed to change or grow, i.e., requirements were

“gold plated.”

In the coastal ship program, the Navy’s requirements creep was due in large part

to combining requirements, which led in turn to the need for a larger platform, causing

the length of the vessel to grow from 110’ to 170’ in length. This was probably caused by

not having a detailed requirements review with the program sponsor or operators, as was

47

mentioned in the audit. In other words, there was no IPT, and therefore, no coordination

among the stakeholders, as would have occurred had there been an IPT. An IPT would

have incorporated the teaming tenets of cooperation, coordination, consensus, and

decision streamlining. Additional causal contributors were lack of a clearly written ORD,

lack of adequate acquisition planning, and inadequate market research in the beginning

phase of the program.

2. Market Investigation

Issue: There were inadequate or no market investigations performed.

At the Natick RD&E Center, there was a policy to leave market investigations

either incomplete or undocumented, in order to “streamline” or speed up the process.

This was done without apparent concern for, or knowledge of, the negative impact that

inadequate market investigations would have on the efficiency and effectiveness of the

material selection process. This exemplifies either culturally influenced decision-making

in order to incorrectly “streamline” the process to save time and money, or inadequate

training and/or education in preparation for the acquisition process, which would have

enabled those involved to make the correct decisions from the result of their market

investigations. Another alternative is that organizations might have known the right thing

to do, but were so resource constrained that they couldn’t get the documentation done.

The lack of a database for market investigations was documented by PM-PSE,

LTC Bonheim, and was also an issue in Dr. Gansler’s USD(AT&L) memorandum of

January 5 2001, which is quoted as follows:

To help overcome these barriers to accessing commercial items, I am taking the following actions: … Requesting that the IPT determine the feasibility of establishing a pilot program so that the Services and Agencies may collect market research and Commercial Item Determinations in a central database, or developing tools to assist in ensuring commercial item determinations are reasonably consistent. I request that the recommendation regarding this action be presented to DUSD (AR) within 90 days of the date of this memorandum. [DoD. USD(AT&L), 5 January 2001]

This suggests an awareness within the acquisition community that there

are “issues” related to ease of access to both commercial item information as well

48

as usage data within the acquisition community, which, if readily available, might

make the process more efficient. This commercial item market database could

also capture the valuable lessons learned, which seem to be lost to successive

generations of acquisition members.


Issue: There were inadequate or no concept formulation packages prepared.

Examples demonstrating this issue are to be seen in the 19 March 1990 Army

Audit Agency report. This report is about Natick RD&E Center’s lack of preparation of

the concept formulation packages for two projects, which resulted in increasing costs and

schedules for both projects. One example is the saltwater purification project, for which a

tradeoff analysis would have identified the Naval system as a potential alternative, thus

providing great time and cost savings for the Army project. The second example is the

case of the five-soldier crew tent project, where a concept formulation package with its

tradeoff analysis would have eliminated the need to test five tents, three times each.

4. Risk Analysis

Issue: A risk analysis was not performed.

The decision to eliminate risk analysis in the Navy’s coastal ship program can be

attributed to lack of training, lack of education, or lack of attention to lessons learned in

the acquisition process; alternatively, it might have been culture-driven in the mistaken

premise that time would be saved if this step in the process were eliminated. Although

there is a Systems Acquisition Management curriculum (816) taught at the Naval

Postgraduate School, Naval officers are not enrolled. This is due to the fact that a military

specialty code for Systems Acquisition Management does not exist for Naval officers.

Consequently, the lack of specific training or education in the acquisition field is

understandable as a potential cause for the elimination of risk analysis in their acquisition

process.


Issue: The acquisition process was incorrectly streamlined, resulting in the use

of an incorrect acquisition strategy.

49

In all of the U.S. Army Audit Agency reports, incorrect acquisition strategy was a

significant finding.

In the Natick Center report of 19 March 1990, the strategies were inconsistent

with the categories of commercial items being used. This is an indication of either

inadequate training or lack of education in the acquisition process, crucial for the

identification of each acquisition category (i.e., commercial item, modified commercial

item, nondevelopmental item, or developmental item); or the premeditated incorrect

streamlining of the process to save money and time for culture-driven reasons.

In the 11 September 1990 U.S. Army Audit Agency report, key phases and

critical steps of the acquisition process were eliminated without explanation or

justification. This is an indication of either a lack of training and/or education regarding

the importance and necessity of each part of the acquisition process, or culture-driven

poor decisions in order to save the program time and/or money.

In the 17 June 1991 report of the U.S. Army Audit Agency, the incorrect

acquisition strategy was used because, as the five agencies audited in this report told the

Army Audit Agency, they were directed to use commercial items, and therefore believed

this relieved them of performing the necessary analysis to determine and support strategy

decisions. This is an indication of lack of training and/or education concerning the

importance and necessity for each part of the acquisition process. It is also an example of

how important “culture” is in decision making. The fact that agencies are told to do

something does not relieve them of the responsibility to “think,” and thus make well-

chosen decisions instead of poor ones.

6. Design

Issue: The design was underestimated.

The design issue, exemplified by the Coastal Ship project, can be attributed to

inadequate training or education on the part of the Navy design estimation process when

utilizing commercial items; or it could have been caused by a culture-driven need to

reduce the time and/or cost of the acquisition.

Issue: The design process was incorrectly accelerated. 50

As an example of this issue, the PM accelerated the Rigid Inflatable Boat (RIB)

design process from 28 weeks to 13 weeks without any analysis given to the risk or

consequences of doing so. The clear goal of the PM was to reduce the time required of

the project’s design task. This issue could be attributed to inadequate training or

education on the part of the Navy concerning the use of risk analysis in the acquisition

process; or it could have been caused by the culture-driven need of the PM to reduce the

schedule and/or cost of the program.


Issue: Integration projects associated with high risk for a program were not

managed properly.

An example of this issue was published in the U.S. Army Audit Agency’s 17 June

1991 report, where it was noted that there were four complex commercial item

integration projects for a total cost of $1.7 billion, none of which had a testing phase prior

to awarding their production contract. The commercial items were all off-the-shelf

components. However, the integrated systems were not off-the-shelf and were highly

complex as well. For these four projects, there was little or no information to prove that

the integrated systems would successfully work together to meet the Army’s

requirements. The failure to manage the integration of commercial items within a

program can be attributed to either lack of training and/or education, or a culture-driven

desire to reduce the time and/or cost of the program.


Issue: There was inadequate testing or no testing performed.

An example of this issue is the U.S. Army Audit Agency 11 September 1990

report on the $643 million Regency Net project, which involved a mix of commercial as

well as previously developed military items, all of which had to be integrated via

complex software development. This project included both a basic contract as well as

four production options. It was awarded at a cost of $300 million before any prototypes

were tested. Another project from the Army Audit Agency 11 September 1990 report on

Mobile Subscriber Equipment again involved integration of commercial items with

51

previously developed military items, as well as the development of complex software. It

was not tested until $1.3 billion of the basic production contract had been obligated,

along with two production options. The rationale given to the Army Audit Agency was as

follows: “Acquisition strategies for the projects stated that extensive testing was not

needed because a nondevelopmental item approach was used or that a contractor

demonstrated the item before the production contracts were awarded.” [U.S. Army Audit

Agency, pg 21, 11 September 1990] However, the contractor demonstrations did not

provide sufficient evidence that the items had met the Army operational requirements.

The failure to properly test and evaluate commercial items within a program can be

attributed to lack of training or education, or can be caused by a culture-driven need on

the part of the PM to reduce the time and/or cost of the program by eliminating or

reducing testing.


a. Reliability

Issue: The program reliability was inadequate.

An example of this issue is the PM-PSE paper, which indicated that

vendor claims proved inaccurate. This can be traced to either inadequate early testing of

the products or inadequate training and/or education concerning the need for adequate

evaluation and testing of a commercial item in order to confirm the vendor’s reliability

claims.

Another example of inadequate program reliability is the DoD IG report of

7 May 1996, which showed that the Hunter UAV program continued to receive PM

support even though it did not have adequate reliability. This can be attributed to the

PM’s culture-driven syndrome to protect his program at all costs, in this case by avoiding

early testing that might have demonstrated reliability flaws, resulting in a program

funding cut and, ultimately, delays or even termination of the program.

The primary cause of inadequate reliability can be traced to inadequate

testing early in the program. A major reason for inadequate testing is often the cultural

aversion on the part of the PM to both testing and testers, as previously mentioned in

52

Chapter III, section B-7. Another potential cause, as noted above, is inadequate training

and/or education regarding the importance of attaining adequate system reliability.

Issue: No quantitative reliability requirements established.

This issue is exemplified by the U.S. Army Audit Agency reports of 11

September 1990 and 17 June 1991, which both state that many projects had no

quantitative reliability requirements established. In the absence of reliability

requirements, there is no basis upon which to perform confirmatory tests. However, in

some cases, with or without a reliability requirement, contractor reliability estimates were

verified, but not early enough to allow time for corrective action. In other cases,

reliability testing was not conducted at all.

The cause for not establishing reliability requirements could be one or

more of the following: PMs having been culture-driven to save the program time and/or

money; lack of training/education which enabled the mistaken belief that management’s

edict to use commercial items voids the need for establishing reliability of commercial

items; or the lack of training and/or education of users and PMs regarding the importance

of establishing reliability requirements, especially for commercial items.

Issue: Contractor’s reliability estimates not verified.

This issue is displayed in the U.S. Army Audit Agency reports of 11

September 1990 and 17 June 1991. In both reports, vendors’ reliability estimates were

not verified in the operating environment. In several cases, reliability estimates had been

received, but without the performance data necessary in order to evaluate the contractor

claims. In four programs, the contractor’s predicted values for the quantitative

requirements were used, but not verified. These contractor-predicted values were

basically technical estimates; they were not based on any real data or analyses and

certainly did not reflect the operating environment and conditions where military systems

must perform.

The reasons for not verifying a contractor’s reliability estimates are

attributable to one or all of the following: lack of training and/or education concerning

the importance of commercial item reliability verification through testing; culture-driven 53

decision-making to save the program time and money by eliminating or reducing testing;

or lack of training/education enabling the mistaken belief that management’s edict to use

commercial items voids the need for verifying the reliability of commercial items.

Issue: Inadequate reliability testing prior to contract award.

This issue is exemplified by the U.S. Army Audit Agency 17 June 1991

report, as well as both the DoD IG reports of 15 May 1995 and 7 May 1996. They all

stated that many projects had no or inadequate reliability testing prior to contract award.

In the case of the Hunter UAV, inadequate reliability, among other issues, killed the

program.

Inadequate reliability testing can be attributed to a lack of training and/or

education on the importance of commercial item reliability testing early in the program; a

culture-driven decision to save the program time/money; or the lack of training and/or

education, enabling the mistaken belief that management’s edict to use commercial items

voids the need for verifying the reliability of commercial items through testing.


Issue: Incorrect/Inadequate program support concept is selected.

The U.S. Army Audit Agency report of 11 September 1990 indicated that

for the sake of expediency, the end-item contractor was to have provided both

maintenance and supply support for the expected life-cycle for seven of the ten projects

reviewed. The reason for lack of adequate logistics support in this case was most likely

culture-driven to save the program time and/or money. Lack of training and/or education,

related to the importance of establishing and justifying support concepts, also could have

played a role.


The U.S. Army Audit Agency report of 17 June 1991 indicated that 14

projects reviewed did not acquire technical data suitable for reprocurement of spare parts

or systems. The causes for not obtaining technical data suitable for competitive

reprocurement of parts are attributed to: lack of training and/or education regarding the

54

importance and difficulty of obtaining commercial item technical data; or culture-driven

decisions to save the program money during the acquisition phase.

D. COMMERCIAL ITEM - CAUSE - SUMMARY

In the table below, the problems/issues are listed as risks, along with the potential

consequence of the unresolved risk, and their possible causes, as identified in this

chapter.

Table 2 Commercial Item Cause Matrix

RISK POTENTIAL CONSEQUENCE POSSIBLE CAUSE

Needs Determination - User requirements

are improperly identified,

misunderstood or ignored.

- Requirements allowed to grow, i.e., requirements are gold

plated.

- Need unsatisfied; wrong product produced and

purchased; DoD resources wasted; possible program

elimination.

- “Best value” solution will not be produced; impacts to

cost, schedule, technical performance; DoD resources

wasted.

- No acquisition planning/risk analysis performed; inadequate

training/education; unclear ORD; IPTs not used; culture-driven decisions; no product

standardization; no centralized database; inflexible reqmts.

-No acquisition planning/IPT; needs allowed to grow too easily; inadequate ORD;

insufficient market research.

Acquisition Strategy

- Incorrect streamlining

acquisition strategy used.

- Items procured without key acquisition phases complete;

impacts to program cost, schedule and technical

performance; DoD resources wasted.

- Lessons learned not utilized; no evaluation of streamlining decisions; inadequate training/ education; incorrect culture-

driven decisions.

Market Investigations

- Inadequate or no market investigation

performed.

- Wrong item procured, “best value” item not procured;

impact to cost, schedule and technical performance; DoD

resources wasted.

- No early, robust, market investigation performed;

inadequate training/education; no centralized database/lessons

learned; inadequate funding.

55

Table 2 (continued) Commercial Item Cause Matrix


Concept Formulation - Inadequate or no concept

formulation package prepared.

- Best technical solution not selected; impact to cost,

schedule and technical performance; DoD resources wasted.

- Inadequate training/ education/ or lessons

learned; incorrect culture-driven decisions.

Risk Analysis - A risk analysis was not

performed.

- Impact to cost, schedule and technical performance; DoD resources wasted; the

program may be eliminated.


inadequate training / education / or use of lessons

learned; widespread misunderstanding of the

complexity of integration.

Manage Integration - Inaccurate assessment of program integration risk.

- R&D, or T&E not performed; impact to cost,

schedule and technical performance; DoD

resources wasted; the program may be eliminated.

- No early, robust, market investigation performed,




Test & Evaluation

- Testing was inadequate or was not performed.

- Best technical solution not developed; performance,

reliability & maintainability goals not met; impact total

LCC; operational suitability & effectiveness not proven;

impacts to program cost, schedule, and technical performance; potential

mission failure / program elimination; DoD resources

wasted.

- Culture-driven poor decisions; inadequate

training / education / or use of lessons learned; lack of

funding.

Design - The design was underestimated.


- The best design will not be completed; impacts to

program cost, schedule and technical performance.

- Impact to program cost, schedule and technical

performance.

- Inadequate training / education / or use of lessons learned; culture-driven poor

decision.

- Culture-driven; inadequate training/education/use of

lessons learned.

56

Table 2 (continued) Commercial Item Cause Matrix


Reliability - The program reliability

was inadequate.

- No quantitative reliability requirements established.


- Inadequate reliability testing prior to contract

award.

- RAM is adversely impacted; program’s total

LCC is adversely impacted; potential mission failure;

DoD resources are wasted.

.

- Poor culture-driven decision; inadequate

training / education / use of lessons learned; lack of

funding.

Support Requirements - Incorrect program support

concept is selected.


- Inability to support the system during a wartime

environment; mission failure; program’s total

LCC is adversely impacted; DoD resources are wasted.

- Sub optimal logistics support, increased system

LCC, DoD resources wasted.

- Poor culture-driven decisions; inadequate

training / education / use of lessons learned;

misunderstanding of the schedule needed for

logistics even for CIs/NDIs.

- Insufficient time allotted for logistics support

planning / preparation; EMD phase eliminated.


- Data adequate for reprocurement is not

obtained.

- A sole source vendor must be used; reprocurement

costs increased; an inability to obtain parts; potential

mission failure; DoD resources are wasted.


training / education / use of lessons learned.

57


58

V. CONCLUSIONS AND RECOMMENDATIONS

A. INTRODUCTION

The objective of this thesis has been to study the problems and issues experienced

by PMs and industries utilizing commercial items in their programs. To explore this

subject, the researcher has reviewed and analyzed the history of problems and issues

experienced by acquisition programs using commercial items in their programs. This

chapter presents the conclusions of this thesis, offering recommended best practices for

utilizing commercial items on a program, as well as suggesting areas for further research.

B. METHODS AND PROCESSES “BEST PRACTICE” CONCLUSIONS

It appears that many mistakes or decisions have been made due to the fact that the

decision-makers thought they understood the process leading them to make the correct

streamlining decision. That is, they thought that their expertise would enable them to

bypass certain requirements, or to ‘streamline’ the process for the purpose of saving time

and money, but instead costed their program both time and money due to the resultant

errors.

1. Lessons Learned

It is imperative to learn from our mistakes. There are many lessons to be learned

from the various governmental audits and reports, and from the use of commercial items

in each of the services. These lessons learned should be accumulated in a centralized

database available to the military service PM community as a web site, and should also

be taught in various acquisition courses. This database website should incorporate market

investigation, research material, lessons learned, and the “best practices” to use in

particular situations and circumstances. It should be not only an information database

website, but also a website for learning about the unique aspects of using commercial

items in the acquisition process

2. Culture-Driven Mistakes

As noted previously in Chapter III, section seven, PMs have occasionally been

driven by culture to make negative and flawed decisions that have adversely impacted

59

their programs, and occasionally their careers. In order to change these negative results,

it is necessary to change the basic culture and reward system that impinges upon the

PMs’ world.

The PM-test community relationship is a vivid example of the adverse impacts of

the wrong culture and reward system at work. Rather than Congress killing a program

because it has failed testing, the discovery of technical problems during early testing

instead of during production should be rewarded, before the cost to fix such technical

problems becomes increasingly expensive and time consuming.

To foster a true and lasting attitudinal change of the PMs’ aversion to T&E within

the acquisition realm, it will be necessary to change the current culture and reward

system. The acquisition of knowledge through testing, and the subsequent increase of a

product’s technological maturity, must be viewed by everyone in the acquisition

community as a good and desirable result, and rewarded accordingly. For example, DT

programs that first mature their technologies through adequate testing, thus allowing their

programs to enter the production phase with subsequent low-risk technologies, should be

rewarded by full funding and continuation of their programs. On the other hand,

programs that attempt to enter production with high to medium-risk technologies should

be penalized.

In many cases the PM/PMO is well aware of and sensitive to the issues faced but

the institutional biases are overwhelming and cause them to make poor decisions.

3. Training and Education

Training and education is extremely important within the acquisition community.

The military makes the higher education and leadership training of their personnel a

career-long endeavor. However, within the DoD civilian community that is not

necessarily the case, and this must change. The acquisition of major weapon systems is

the most complex environment within DoD, and all of its personnel must be adequately

trained and educated continuously in order to master its intricacies, maintain current

knowledge and execute the acquisition process properly.

60

There appear to be large gaps within the education process, especially concerning

the use of commercial items and nondevelopmental items within DoD acquisition

programs. Policy drives the acquisition process and is far ahead of the education of its

disciples, especially for commercial item usage. There is inadequate training for user

personnel to learn how to write performance-based requirements. There is no risk

management training oriented toward commercial item/nondevelopmental item usage.

There has been inadequate market investigation training. The acquisition training and

education process should be critically examined for such voids and the deficiencies

corrected.

Training and education alone will not correct the DoD commercial item usage

problems. It is imperative that the “culture” within DoD that drives PMs to make

negative, self-defeating decisions be addressed and changed. The culture and reward

system must promote and reward good decisions. A research article by a professor at the

Naval Postgraduate School about the effect of training on changing the culture within

DoD inventory management noted, “To be effective, it must be coupled with changes in

other organizational design factors such as reward systems and management control

systems. Training is only one of many inputs to organizational behavior. Trying to alter a

culture by changing only one of these factors is usually a wasted effort.” [Kang, pg. 21]

4. Needs Determination

Users initiate an acquisition as they conduct the requirements generation process.

User-led multifunctional teams need to be formed as early as possible to assist the user in

fully grasping the nature, potential limitations, and advantages of available commercial

items. Only through an adequate understanding of the advantages of commercial items

can the users and acquisition community embrace the flexibility necessary in defining

system requirements that will encourage greater consideration of commercial item usage.


It is critical that both the buying organizations and the acquisition community

understand the importance of devising farsighted acquisition strategies, ones that

adequately address the program uncertainties that can result from the use of commercial

items. Acquisition strategies should be devised to target those commercial item 61

uncertainties with the potential to impact product performance, quality, and logistical

support.

6. Market Investigation

When market investigations are not conducted properly, it is possible that some

potential alternatives will not be identified and investigated. Consequently, the best-value

alternative might not be selected. If the best-value is not selected, the result will lead to

greater program cost and/or time, the selection of substandard technology, or the least

supportable material solution.

A good market investigation also would identify deficiencies in the commercial

item that would require modification, and would determine the extent of that

modification. Accurate determination of the necessary modifications is important; having

to modify the selection more then anticipated would add both cost and time to the

program. Extensive modification of commercial items within a program would take the

product out of the category of a commercial item, thus increasing the program’s risk.

Market investigation, along with concept formulation, is a key determiner of the

acquisition strategy. If the market investigation is not performed properly, the selected

acquisition strategy will probably not be the most effective and efficient one to use.

Key personnel within the buying organization must be adequately trained and

experienced in the market investigation process. In the Natick Laboratory case of the

snow and ice-traversing equipment, a proper market investigation would have headed off

the problems with the selected off-the-shelf items prior to acquiring and testing them. In

the case of the field pack, a market investigation would have identified the deficiencies

with the commercial pack, which subsequently required modification to such an extent

that it bore no resemblance to the original commercial pack.


Without an adequate concept formulation package, which includes both a tradeoff

determination and the best technical approach, it is unlikely that the best alternative will

be selected. This will cost the program both time and money, and possibly forego the best

technology as well. Since concept formulation is one of the determiners of the acquisition

62

strategy, incorrect concept formulation is likely to lead to a substandard acquisition

strategy. As the Army Audit Agency states: “The concept formulation process should be

conducted and documented for nondevelopmental items.” [U.S. Army Audit Agency, pg.

23, 11 Sept 1990]

Even in cases where management directs the use of commercial items, there are

always tradeoffs and options that need to be identified, analyzed and evaluated to

establish the best strategy for implementation.

8. Risk Analysis

Program risk will be different when using commercial items than in a traditional

developmental acquisition program. Risk is fairly well defined for commercial items in

the traditional areas of technical performance, cost and scheduling. For example, the

commercial item market investigation will determine cost and technical performance, and

delivery dates are subject to negotiation and discussion. In the areas of integration,

operational suitability and supportability, however, risk analysis should be strongly

emphasized, not eliminated.

The risk that occurs during the integration of commercial parts must also be

recognized, and program strategy to reduce or eliminate that risk should be established.

Commercial items that are to be used in a military environment significantly

different from that for which they are designed carry high risk. These items must be

qualified in the operational environment in which they will be used.

There is inherent supportability risk within the commercial realm due to the rapid

change within the commercial marketplace, as previously discussed. This risk needs to be

recognized and planned for in order to produce adequate total life-cycle support for the

commercial item.

9. Design

The area of design cannot arbitrarily be rushed in order to speed up the program.

Likewise, the use of commercial items does not necessarily mean that the design process

has been reduced or streamlined. In fact, the incorporation of commercial items into a

design may be more difficult, because all of the design parameters of the commercial

63

item may not be known. This is an area that demands careful analysis before attempting

to compress design schedules.


There is widespread misunderstanding of the complexity of commercial item and

nondevelopmental item integration into a program. Integration projects cannot be treated

as normal low-risk commercial item acquisitions. Such integration efforts may not only

require research and development, but also may need both a demonstration and a full-

scale development phase before production, unless the integration risk is determined to

be sufficiently low.


Test and Evaluation (T&E) must be included in the evaluation of both commercial

items and nondevelopmental items to ensure that operational suitability, reliability,

availability and maintainability requirements are met. Only for commercial off-the-shelf

items that are very low risk should developmental T&E be allowed to be eliminated and

operational testing should always confirm the suitability of a commercial item in the

military environment. As the Army Audit Agency indicates, “More effective up-front

planning of independent test and evaluation is needed to ensure that enough data is

obtained to fully evaluate the capabilities of new nondevelopmental items.” [U.S. Army

Audit Agency, pg 28, 11 Sept. 1990]

Testing should be performed on commercial items before and during production,

unless a definitive decision verified by contractor data or previous tests adequately

demonstrates the item’s acceptability. Sufficient data must be available to assess the

critical testing issues of manpower and personnel integration, reliability, safety and

supportability. Any decision to reduce or eliminate developmental testing should be

based on independent evaluation reports that assess either the previous testing or other

data to indicate how much testing will be required. A GAO study found that a DoD

weapon system frequently “experiences ‘late cycle churn’ or the scramble to fix a

significant problem discovered late in development.” [GAO, pg 5, 31 July 2000] This

‘late cycle churn’ is attributable to the inadequate exposure of technical problems early in

the program via a robust test and evaluation process. 64

Operational testing of a commercial item in the military environment it will be

used in order to verify operational suitability, effectiveness and performance should never

be waived unless it will be used in the exact same environment for which it was designed.


A significant problem concerning logistics support for commercial items has

become apparent. Products are constantly being changed within the commercial item

industry to keep contractors competitive. This constant change drastically affects the

manufacturer support for older products. The manufacturer may no longer support older

obsolete products, thus making replacement parts difficult, if not impossible, to obtain.

As an Armed Forces Journal International magazine article puts it, “But today, the true

cost of COTS is beginning to sink in. Sure, DoD’s R&D costs come down, but life-cycle

costs under COTS need to substantially increase. Staying in step with the commercial

world’s technology churn will require continuous mini-development efforts throughout a

program’s life.” [Baker, pg4]

The process of streamlining for commercial items or nondevelopmental items can

result in the elimination of the Engineering & Manufacturing Development (EMD) or

System Development and Demonstration (SDD) phase, which would normally include

detailed logistics planning and acquisition of the necessary logistics support

infrastructure. This may result in not allowing enough time for the logistics planning

process in order to adequately develop and acquire the support materiel package,

maintenance structure, organizational and support training packages, and writing of

technical manuals.

a. Reliability

If a commercial item’s reliability, as stated by the vendor, is not verified

as meeting the user’s requirements, then the system’s LCC may be severely impacted.

Lower reliability greatly impacts the support costs, system availability, and thus the

mission accomplishment. RAM requirements must be established early in order to insure

adequate testing and verification of the reliability of commercial or nondevelopmental

items.

65

Since a commercial or nondevelopmental item has already been designed

and developed, and its reliability already established, the reliability verification should be

an operational assessment of the product within the military wartime environment in

which it will be used.


Because of the volatility of commercial items with respect to design

changes, as previously mentioned, the availability of commercial item parts, components,

or systems may cease before the intended military life-cycle of the commercial item

expires. Therefore, support requirements must focus on the long-term maintenance and

support of commercial items when fielded, because Government access to commercial

item technical data will most likely be limited.


Plans for access to data suitable for competitive reprocurement of

commercial items are critical. Obsolescence within the commercial item marketplace is

usually more rapid and unpredictable than it is for products developed specifically for the

Government. The commercial item marketplace is more volatile; products can be dropped

abruptly, or changed and businesses can go bankrupt or be bought out by other

companies. Without adequate planning for reprocurement, it is assured that there will be

trouble within the program - and sooner then anyone might think.

C. COMMERCIAL ITEM SUMMARY

The inclusion of commercial items in the acquisition process is recognized as an

opportunity to save both time and money. But it is not the Holy Grail. It can do all that

everyone expects it to do, and may be an excellent solution in many cases, but its use

should be as the result of careful analysis, reasoning and multi-functional study. As

Carney and Oberndorf indicate in their research paper on the use of COTS to purchase

software, “The critical point is that using COTS components in any given circumstance

might help, but is not guaranteed to, and such use may even cause greater problems.”

[pg. 1] They propose what they call the “ten commandments,” which must be considered

when using COTS. All of these “commandments” apply directly to the use of commercial

items other than software. 66

For example, their first commandment, “One more time: Do not believe in Silver

Bullets” [pg. 3] applies to this study, also. This has been reiterated in several contexts

involving software, and certainly applies to the use of commercial item equipment in

programs, as well. Anyone who believes that selection of a commercial item and

inserting it into a program will be the quick fix believes in fairy tales, and does not really

understand the process.

The second commandment, “Use the term precisely (and demand like behavior

from others)” [pg. 3] most certainly applies here. We have noted in this research that

terms have been used differently in the past in the Army Audit Agency reports, and that

Dr. Gansler specifically defined terms in order to bring uniformity to the terminology.

Their third commandment, “Understand the impact of COTS products on the

requirements and selection process” [pg. 4] equally applies to this research. It is

important that commercial products are known and understood by those who establish

requirements. The commercial marketplace must be clearly understood in order to have a

flexible range of “requirements” sufficient to allow commercial items to qualify.

The fourth commandment, “Understand the impact of COTS products on the

integration process,” [pg. 5] certainly applies, as demonstrated by the various integration

problems that have occurred within the various services and subsequently reported in the

previously cited audits. There has been a tendency to assume that a commercial item can

be used as-is, without any serious thought given to the difficulty and risk involved in the

integration process. It has been assumed that the use of a commercial item alleviates all

risk of integration. In fact, just the opposite may occur: commercial items may be even

more difficult to integrate.

Their fifth commandment, “Understand their impact on the testing process,”

[pg. 6] also applies, as has been demonstrated by the problems experienced in

commercial item testing. The validation of commercial items is different from the testing

of development projects. This difference in the testing process must be clearly understood

in order for validation of the commercial item to be correctly planned. Testing must then

67

be initiated early enough in the process in order to alleviate the great risk that occurs

when it is performed too late.

Their sixth commandment, “Realize that a COTS approach makes a system

dependent on the COTS vendors,” [pg. 6] was exemplified in the Army Audit Agency

report of 11 September 1990. This report found that in most cases, for the sake of

expediency, PMs opted for the end-item contractor to provide the maintenance and

supply support for the product. Vendor support must be examined critically for several

reasons. First, it’s important to insure that the program doesn’t become too dependent on

the vendor. Second, it must be determined that the vendor has sufficient technical data for

adequate ongoing support. Third, it must be determined that the vendor has sufficient

technical data for reprocurement of the product. Reprocurement is high-risk, made so by

the probability that the commercial item vendor may change the product, or the product

support, go out of business, be bought out by another company, etc. Contingency plans

should therefore be developed for these possibilities.

Their seventh commandment, “Realize that maintenance is not free,” [pg. 7] is

exemplified by the Army Audit Agency 11 September 1990 report about the blind faith

exhibited by the PMO concerning the cost effective vendor support of their products.

This audit report has documented one of many instances where no analysis was

performed to prove that contractor support was more cost-effective than the use of the

military in-house organic support system.

The eight commandment, “You are not absolved of the need to engineer the

system well,” [pg. 8] has been displayed in every one of the issues studied in the previous

chapter. Just because a product may be an off-the-shelf commercial item, there is no

guarantee that it will integrate into the program’s system successfully. Nothing will come

together and work by itself. The integration process must not be accidental, but rather,

must be planned and executed purposefully. As has been seen in the previous problems in

Chapter III, commercial items add another whole dimension to the difficulty of program

management. Elimination of systems engineering in the commercial item acquisition

process is not an option with the use of commercial items. As Carney notes, “This system

will need to be designed, brought together, tested, and managed just the same as any 68

other system you have built or acquired in the past.” [pg 8] Logistics supportability

planning and execution should be added to Carney’s list.

Their ninth commandment, “Just ‘doing COTS’ is not an automatic cost-saver,”

[pg 8] has been alluded to throughout this thesis. There has been a lot of anecdotal

evidence, as well as a few actual cases to substantiate the claim that use of commercial

items saves both time and money. However, it is not a “given” that using a commercial

item will always save time or money. The use of commercial items must be carefully

planned, well engineered, and analyzed in depth.

The tenth commandment, “Just ‘doing COTS’ must be part of a large-scale

paradigm shift,” [pg. 9] was indicated in Chapter II, paragraph H, sections 1-7, of this

thesis. Instead of the Traditional Model, as shown in Figure 2 of this thesis as being a

waterfall acquisition process, it is now imperative that processes overlap and that greater

use of IPTs be made in order to manage the uniqueness of commercial item usage, as

portrayed in the Recommended Process – Figure 2.

D. COMMERCIAL ITEM BEST PRACTICE SUMMARY

The following table summarizes the problem/issue as a “risk,” the possible cause

of the unresolved risk, and the best practice to follow to minimize the risk.

69

Table 3 Commercial Item Best Practice Matrix

RISK POSSIBLE CAUSE BEST PRACTICE

Needs Determination - User requirements are

improperly identified, misunderstood or ignored.

- Requirements allowed to grow, i.e., requirements are

gold plated.

- Inadequate acquisition planning/risk analysis performed; inadequate

training/education; unclear ORD; IPTs not used;

culture-driven decisions; no product standardization; no

centralized database; inflexible reqmts.

-No acquisition planning/IPT; needs

allowed to grow too easily; inadequate ORD;

insufficient market research.

- Use IPTs; clearly written ORD; comprehensive acquisition planning;

thorough market research; adequate acquisition

training and education; use lessons learned.

- Use IPTs; clearly written ORD; comprehensive acquisition planning;

thorough market research; freeze requirements or use P3I process to add needs.

Acquisition Strategy - Incorrect streamlining

acquisition strategy used.

- Lessons learned not utilized; no evaluation of streamlining decisions;

inadequate training/ education; incorrect culture-

driven decisions.

- Utilize lessons learned, closely evaluate stream-lining decisions; provide

adequate acquisition training and education; need comprehensive review of & change of culture paradigm.

Market Investigations - Inadequate or no market investigation performed.


inadequate training/education; no

centralized database/lessons learned.

- Perform robust market investigation early in the

acquisition process; adequately trained

personnel; use centralized database + lessons learned.

Concept Formulation - Inadequate or no concept

formulation package prepared.

- Inadequate training/ education/ or lessons

learned; incorrect culture-driven decisions.

-Use lessons learned; adequately trained and

educated personnel; need comprehensive review of & change of culture paradigm.

70

Table 3 (continued) Commercial Item Best Practice Matrix

RISK POSSIBLE CAUSE BEST PRACTICE

Risk Analysis - A risk analysis was not

performed.






systems acquisition process; adequate acquisition

training and education; centralized database / use

lessons learned.

Manage Integration - Inaccurate assessment of program integration risk.






systems acquisition process; adequately trained and

educated personnel; centralized database / use

lessons learned.

Test & Evaluation - Testing was inadequate or

was not performed.

- Culture-driven poor decisions; inadequate

training / education / or use of lessons learned; lack of funding; lack of resources.

- Change the culture’s paradigm; adequately train

and educate personnel; create centralized database /

use lessons learned.

Design

- The design was underestimated.


- Inadequate training / education / or use of lessons learned; culture-driven poor

decision.

- Culture-driven; inadequate training/education/use of

lessons learned.

- Change the culture’s paradigm; adequately train



Reliability - The program reliability

was inadequate. - No quantitative reliability requirements established.


- Inadequate reliability testing prior to contract

award.


training / education / use of lessons learned, lack of

funding.

- Change the culture’s

paradigm; adequately train and educate personnel;

create centralized database / use lessons learned.

71

Table 3 (continued) Commercial Item Best Practice Matrix RISK POSSIBLE CAUSE BEST PRACTICE

Support Requirements - Incorrect program support

concept is selected.


- Poor culture–driven decisions; inadequate

training / education / use of lessons learned;

misunderstanding of the schedule needed for

logistics even for CI/NDI items.

- Insufficient time allotted for logistics support

planning / preparation; EMD phase eliminated.

- Change culture’s paradigm; adequately train



- Stretch phase to accommodate logistics

support planning such as TMs, training, provisioning.


- Data adequate for reprocurement is not

obtained.


training / education / use of lessons learned.

- Change culture’s paradigm; adequately train


use lessons learned

E. CONCLUSION

In order to reduce the time and cost of fielding new systems and inserting new

technologies rapidly, the well thought out use of commercial items is imperative.

However, the difficult new paradigm of using commercial items must be recognized, and

acquisition personnel educated and trained for its efficient and effective use. The

previous table of best practices is provided in order to highlight the major areas of risk

that have been found to exist within the acquisition community. It also summarizes the

best practices that will serve to minimize those risks.

72

F. AREAS FOR FURTHER RESEARCH

This thesis identified areas that merit additional research, but did not address them

because they were beyond the scope of this study. Those areas are as follows:

1. An in-depth analysis of the cost-benefit relationship of using commercial

items in acquisition programs.

2. An examination of any existing barriers to using commercial items more

efficiently in DoD acquisition programs.

3. A comparative analysis of the problems associated with using commercial

items versus the problems found in traditional acquisition programs to determine the

significant differences, if any.

4. An analysis of major defense acquisition programs’ potential of meeting

the OSD commercial item goals that have been set for DoD.

5. Identification of the extent the current PM community acquisition culture

affects the PM’s correct usage of commercial items, the extent to which it contributes to

the problems, and recommended corrective actions.

6. Identification, via survey or other means, of the extent to which the

“culture” in which the PM is immersed drives incorrect or inadequate decisions within

the acquisition process and potential corrective actions.

73


74

LIST OF REFERENCES Air Force Space and Missile Command. CPATs – Reliability Engineering, 14 August 1998. Military Specifications and Standards Reform Program (MSSRP), Critical Process Assessment Tool (CPAT), SMC/AXM. Retrieved 16 October 2001 from the World Wide Web: http://web2.deskbook.osd.mil/data/012GZDOC.DOC. Anderson, Robert H. and Hundley, Richard O. The Implications of COTS Vulnerabilities for the DoD and Critical U.S. Infrastructures: What Can/Should the DoD Do?. RAND. 1998. ASAAL&T. Acting Assistant Secretary of the Army for Acquisition, Logistics and Technology. Memorandum Subject Commercial Acquisitions. 26 March 2001. Baker, Don. Commercial Off-The-Shelf. Armed Forces Journal International. November, 1997. Retrieved 21 July 2001 from the World Wide Web: http://www.afji.com/mags/1997/Nov/Feature1.html. Bonheim, Mike LTC. PM – Physical Security Equipment. Commercial Items Acquisition Data Call S: COB 22 February 2001. Brinkley, C. Mark. We Need to Lie. Marine Corps Times, pgs 8-10. 29 January 2001. Carney, D. and Oberndorf, P. OSD - Commandments of COTS: Still in Search of the Promised Land. Software Engineering Institute, Carnegie Mellon University. Retrieved 4 September 2001 from the World Wide Web: http://www.deskbook.osd.mil/irp/253.asp - MandRef. CJCS. Chairman of the Joint Chiefs of Staff Instruction 5261.01A. 1 August 1998. Retrieved 24 September 2001 from the World Wide Web: http://www.datashopper.dk/~boo/comterfund.html. Defense Systems Management College. Systems Engineering Management Guide. DSMC, Ft. Belvoir, VA. January 1990. pg 13-1 DoD. DoDD 5000.1. The Defense Acquisition System, paragraph 4.2.3, (Incorporating Change 1, 4 January 2001) 23 October 2000. Retrieved 20 August 2001 from the World Wide Web: http://web2.deskbook.osd.mil/reflib/MDOD/061DD/061DDDOC.HTM#T2. DoD. DoDI 5000.2. Operation of the Defense Acquisition System; (Including Change 1); 4 January 2001. Retrieved 23 August 2001 from the World Wide Web: http://web1.deskbook.osd.mil/htmlfiles/rlframe/REFLIB_Frame.asp?TOC=/htmlfiles/TOC/012ditoc.htm&Doc=/reflib/mdod/012di/012didoc.htm&BMK=T19.

75

http://web2.deskbook.osd.mil/data/012GZDOC.DOC

http://www.afji.com/mags/1997/Nov/Feature1.html

http://www.deskbook.osd.mil/irp/253.asp

http://www.datashopper.dk/~boo/comterfund.html

http://web2.deskbook.osd.mil/reflib/MDOD/061DD/061DDDOC.HTM

http://web1.deskbook.osd.mil/htmlfiles/rlframe/REFLIB_Frame.asp?TOC=/htmlfiles/TOC/012ditoc.htm&Doc=/reflib/mdod/012di/012didoc.htm&BMK=T19

http://web1.deskbook.osd.mil/htmlfiles/rlframe/REFLIB_Frame.asp?TOC=/htmlfiles/TOC/012ditoc.htm&Doc=/reflib/mdod/012di/012didoc.htm&BMK=T19

DoD IG. Department of Defense Inspector General. Allegations Involving the Procurement of the Hunter Unmanned Aerial Vehicle: Short-Range System. Report No. 96-111. 7 May 1996. Retrieved 26 September 2001 from the World Wide Web: http://www.dodig.osd.mil/audit/reports/96report.htm. DoD IG. Department of Defense Inspector General. DoD Hotline Allegation Regarding the Naval Special Warfare Patrol Coastal Ship and Rigid Inflatable Boat Acquisition Programs. Report 95-193. 15 May 1995. DoD IG. Department of Defense Inspector General. Effectiveness of DoD Use of Nondevelopmental Items in Major Defense Acquisition Programs. Report No. 92-107. 22 June 1992. Retrieved 26 September 2001 from the World Wide Web: http://www.dodig.osd.mil/audit/reports/92report.htm. DoD IG. Department of Defense Inspector General. Lessons Learned from Acquisitions of Modified Commercial Items and Nondevelopmental Items. Report No. 97-219. 23 September 1997. DoD. Deputy Under Secretary of Defense (Acquisition Reform). Draft Commercial Item Handbook. 10 January 2001. Retrieved 15 August 2001 from the World Wide Web: http://www.acq.osd.mil/cgi-bin/web_evaluate?dataset= ar&dbs=ar&multi=1&query=Commercial+Item+ Policy&searchButton=Search. DoD. Office of the Deputy Under Secretary of Defense for Acquisition Reform. Commercial Item Acquisition: Considerations and Lessons Learned. 14 July 2000. DoD. Under Secretary of Defense for Acquisition and Technology. Buying Commercial & NonDevelopmental Items: A Handbook. April 1996. DoD. Under Secretary of Defense for Acquisition, Technology and Logistics. Dr. J. S. Gansler. Policy Memorandum: Commercial Acquisition. 5 January 2001. Federal Acquisition Regulation Parts 2 and 12, (25 June 2001). Retrieved 15 August 2001 from the World Wide Web: http://www.arnet.gov/far/loadmain.html. Federal Acquisition Reform Act. Title 10. Section 2304 of Public Law 104-121. 1996. Federal Acquisition Streamlining Act. Title 10. Section 8104, paragraph 2377 of Public Law 103-355. 1994. Flack, Marla. AMSEL-LG-CM. Impact of DA Policy on Commercial Items. Information Paper. 18 May 2001. General Accounting Office. GAO/NSIAD-00-199. Best Practices: A More Constructive Test Approach Is Key. 31 July 2000. Retrieved 30 July 2001 from the World Wide Web: http://www.gao.gov/.

76

http://www.dodig.osd.mil/audit/reports/96report.htm

http://www.dodig.osd.mil/audit/reports/92report.htm

http://www.acq.osd.mil/cgi-bin/web_evaluate?dataset= ar&dbs=ar&multi=1&query=Commercial+Item+ Policy&searchButton=Search

http://www.acq.osd.mil/cgi-bin/web_evaluate?dataset= ar&dbs=ar&multi=1&query=Commercial+Item+ Policy&searchButton=Search

http://www.arnet.gov/far/loadmain.html

http://www.gao.gov/

General Accounting Office. GAO/NSIAD-95-161. UNMANNED AERIAL VEHICLES: Maneuver System Schedule Includes Unnecessary Risk. September 1995. Retrieved 25 September 2001 from the World Wide Web: http://www.gao.gov/. General Accounting Office. GAO/NSIAD-96-2, Unmanned Aerial Vehicles: Hunter System Is Not Appropriate for Navy Fleet Use. 1 December 1995. Retrieved 25 September 2001 from the World Wide Web: http://www.gao.gov/. Kang, Keebom. DoD Inventory Management Culture Changes and Training in Commercial Practices. Research paper NPS-SM-98-002. March 1998. U.S. Army Audit Agency. Acquisition of Nondevelopmental Items U.S. Army Natick Research, Development and Engineering Center Natick, Massachusetts. (Report of Audit NE 90-206). 19 March 1990. U.S. Army Audit Agency. Acquisition of Nondevelopmental Items. (Report of Audit NE 91-204). 17 June 1991. U.S. Army Audit Agency. Acquisition of Nondevelopmental Items, Fort Monmouth, New Jersey. (Report of Audit NE 90-209). 11 September 1990.

77

http://www.gao.gov/

http://www.gao.gov/


78

BIBLIOGRAPHY DoD IG. Department of Defense Inspector General. Contractor’s Performance on the Family of Medium Tactical Vehicles Program. Report 96-020. November 1995. DoD IG. Department of Defense Inspector General. Purchasing Commercial Products. Report 97-145. May 1997. DoDR 5000.2. Department of Defense Regulation. Mandatory Procedures for Major Defense Acquisition Programs (MDAPS) and Major Automated Information System (MAIS) Acquisition Programs. DoD Deskbook, 10 June 2001. DoN. Department of the Navy. NAVSEA -- Acquisition Planning Guide. December 1999. GAO. General Accounting Office. ACQUISITION REFORM - The Government’s Market Research Efforts. GAO/NSIAD-97-3. October 1996. GAO. General Accounting Office. ACQUISITION REFORM - Military-Commercial Pilot Program Offers Benefits but Faces Challenges, Government Accounting Office. GAO/NSIAD-96-53. June 1996. GAO. General Accounting Office. ARMY ACQUISITION - Medium Trucks Passed Key Operational and Technical Tests, Government Accounting Office. GAO/NSIAD-96-4. January 1996. Oberndorf, Tricia, Brownsword, Lisa and Sledge, Carol A. An Activity Framework for COTS-Based Systems. Technical Report CMU/SEI-2000-TR-010. Carnegie Mellon University. October 2000. Oberndorf, Tricia. COTS-Based Systems (CBS) Initiative. Carnegie Mellon University. Retrieved 23 August 2001 from the World Wide Web: http://www.sei.cmu.edu/cbs/cbs_description.html. USAAA. U.S. Army Audit Agency. Acquisition of Nondevelopmental Items – Program Executive Officer, Combat Support and U.S. Army Tank-Automotive Command, Warren, Michigan, Report No. EC-90-211, 21 August 1990.

79

http://www.sei.cmu.edu/cbs/cbs_description.html


80

LIST OF ACRONYMS AND ABBREVIATIONS ASA(AT&L) Assistant Secretary of the Army for Acquisition Technology and

Logistics

CI Commercial Item

CINC Commander in Chief of a Unified Command

CNO Chief of Naval Operations

COTS Commercial Off-The-Shelf

DoD Department of Defense

DoDD Department of Defense Directive

DoDI Department of Defense Instruction

DoDR Department of Defense Regulation

DT Developmental Test

DT&E Developmental Test & Evaluation

DUSD(AR) Deputy Under Secretary of Defense for Acquisition Reform

EMD Engineering & Manufacturing Development

FAR Federal Acquisition Regulation

FASA Federal Acquisition Streamlining Act

GOTS Government Off-the-Shelf

GSA General Services Administration

GSS General Ships Specifications

HQAMC Headquarters of the Army Material Command

IFB Invitation for Bid

IPPD Integrated Product and Process Development

IPT Integrated Process Team

LCC Life Cycle Cost

LRIP Low-Rate Initial Production

MIL-SPEC Military Specification

MOA/MOU Memorandum of Agreement/Understanding

NDI Nondevelopmental Item

NSSC Naval Sea Systems Command

81

NSWC Naval Surface Warfare Center

ORD Operational Requirements Document

OT Operational Test

OT&E Operational Test & Evaluation

PEO Program Executive Officer

PM Program Manager

P3I Preplanned Product Improvement

R&D Research and Development

RDT&E Research, Development, Test & Evaluation

RFP Request for Proposal

RFQ Request for Quotation

RIB Rigid Inflatable Boat

R&M Reliability and Maintainability

SDD System Development and Demonstration

T&E Test and Evaluation

USAAA U.S. Army Audit Agency

USC United States Code

USD(AT&L) Under Secretary of Defense for Acquisition, Technology and

Logistics

UL Underwriters Laboratory

82

INITIAL DISTRIBUTION LIST

1. Defense Technical Information Center 8725 John J. Kingman Road, Ste. 0944 Fort Belvoir, Virginia

2. Dudley Knox Library 411 Dyer Rd.

Naval Postgraduate School Monterey California 3. Mike Boudreau Naval Postgraduate School Monterey California 4. Dr. Richard B. Doyle

Naval Postgraduate School Monterey California 5. Dr. David V. Lamm

Naval Postgraduate School Monterey California 6. LTC Mike Bonheim Attn: AMSEL-DSA-PSE 5900 Putman Rd.

Fort Belvoir, Virginia 22060-5420

83

Documents

Increasing the utilization of commercial items in ... · Commercial Off-The-Shelf Items in Acquisition Programs: Problems, Issues and Best Practices. 6. AUTHOR(S) Willard D. Meyer