Operational Testing: From Basics to System-of-Systems Capabilities

Col Michael W. Kometer, Ph.D.

School of Advanced Air and Space Studies, Maxwell AFB, Alabama

Kent Burkhart

605th Test and Evaluation Squadron, Hurlburt Field, Florida

Susana V. McKee and Dwight W. Polk, Ph.D.

505th Command and Control Wing, Hurlburt Field, Florida

This article examines the need to develop and implement a System of Systems (SoS) operational

test methodology and supporting infrastructure in the U.S. Air Force (USAF). It discusses

impacts on the acquisition and decision authority communities at large, with emphasis on

operational testing in particular. Beginning with the fundamental requirements for testing, the

article presents the realities of the effect of the Information Age on those requirements. The

current progress in developing the acquisition business model and the technical capabilities to

facilitate SoS test are reviewed. The authors then provide a plausible way ahead that will

leverage training and testing capabilities through integrated venues.

Key words: Acquisition/fielding errors; capabilities-based evaluation; Joint Battle

Management Command and Control (JBMC2) roadmap; Joint Theatre Air-Ground

System capability; network-centric warfare; Red and Green Flag training; scientific

method; Single Integrated Air Picture (SIAP); system-of-systems requirements; test and

training integration.

During two-thirds of the 100 years theU.S. Army, and subsequently theU.S. Air Force, has been acquiringaircraft and other systems related toair and space warfare, the Depart-

ment of Defense (DoD) has conducted very littlededicated operational testing to support acquisition orproduction decisions. With the exception of a 16-yearspan (1941–1957, Air Proving Ground, Eglin Field,Florida), the vast majority of government-conductedtesting, even into the Vietnam era, was what is knowntoday as developmental testing, with well-documentedconsequences, such as the first deployment of the F-111A to Southeast Asia. In March 1968, six F-111Aswere sent to Thailand for combat duty. After the lossof three aircraft in less than two months because ofmalfunctioning horizontal stabilizers, the remaining F-111As were returned stateside (Benson 1992).

Tactical Air Command established test centers atEglin Air Force Base (AFB; Tactical Air WarfareCenter, 1963) and Nellis AFB (Tactical FighterWeapons Center, 1966) to help rectify the problem.

The Air Force Test and Evaluation Center (‘‘Opera-tional’’ was a later addition to the title) was activated in1974 to provide operational testing independent of thedevelopment, procurement, and user Commands forthe largest acquisition programs. Quite naturally, thefocus in the ensuing years of operational testing was onthe individual system being acquired or fielded withinthe context of a limited operational environment. Inother words, operational testing was effectively anextension of developmental testing. The larger ques-tions of combat capability and mission contributionwere left to the Modeling and Simulation (M&S)community. Two notable exceptions were the DoD-sponsored Joint Test and Evaluation (JT&E) program,established in 1972, and Tactics Development andEvaluations (TD&Es) dating to the early years of theTactical Fighter Weapons Center.

Recent emphasis on capability-based testing takesthe evolutionary process a step further. In particular,the test community has been tasked to conduct net-centric (or info-centric) and System-of-Systems (SoS)testing. Net-centric testing requires the investigator to

ITEA Journal 2011; 32: 39–51

32(1) N March 2011 39

Report Documentation Page Form ApprovedOMB No. 0704-0188

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering andmaintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information,including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, ArlingtonVA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number.

1. REPORT DATE MAR 2011 2. REPORT TYPE

3. DATES COVERED 00-00-2011 to 00-00-2011

4. TITLE AND SUBTITLE Operational Testing: From Basics to System-of-Systems Capabilities

5a. CONTRACT NUMBER

5b. GRANT NUMBER

5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

5e. TASK NUMBER

5f. WORK UNIT NUMBER

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) School of Advanced Air and Space Studies,600 Chennault Circle,Maxwell AFB,AL,36112-6424

8. PERFORMING ORGANIZATIONREPORT NUMBER

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S)

11. SPONSOR/MONITOR’S REPORT NUMBER(S)

12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited

13. SUPPLEMENTARY NOTES

14. ABSTRACT

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF ABSTRACT Same as

Report (SAR)

18. NUMBEROF PAGES

13

19a. NAME OFRESPONSIBLE PERSON

a. REPORT unclassified

b. ABSTRACT unclassified

c. THIS PAGE unclassified

Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18

evaluate systems and their interoperability as part of aFind-Fix-Track-Target-Engage-Assess (F2T2EA) in-formation network. This requires testers to evaluatecapability beyond the limits of the particular system-under-test and its interoperability/integration withnearest-neighbor systems, as is currently practiced.SoS testing extends the scope of evaluation beyond thatof merely placing a system in the context of a largermulti-system structure; it is the most plausibleapproach to testing that reaches the level of capabil-ity-based evaluation.

This article will examine the basic premise for testingand then show how the Information Age is affectingthat purpose. It will then examine efforts to develop anew business model to facilitate testing to SoSrequirements versus system-level requirements. It willlook at some of the efforts to develop test infrastructureto perform distributed testing with integrated Live,Virtual, and Constructive (LVC) inputs. It will thendiscuss efforts to integrate testing and training and,finally, develop a concept for bringing all thesedevelopments together to accomplish SoS testing.

The primary purpose of testing systems andprocesses has remained unchanged over the years.Testing is evaluation conducted to mitigate the risksassociated with new materiel and non-materiel solu-tions to warfighting capability needs. Operationaltesters need to determine as closely as possible thecapability’s ‘‘state of nature’’ effectiveness and suitabil-ity to avoid making the errors of either recommendingfielding of a non-value-added capability or recom-mending not fielding one that could be value-added.The Information Age changes the focus of operationaltesting by redefining the penalties and benefitsassociated with the decision processes (or the lossfunction)—a system should no longer be measuredagainst system-based performance, but against itscontribution to overall warfighting capability asmeasured by SoS-based requirements. The Office ofthe Secretary of Defense (OSD) has given UnitedStates Joint Forces Command (USJFCOM) the job ofleading the combatant commands (COCOMs) indefining these requirements by identifying JointMission Threads (JMTs) for important Joint missionsthat cross functional lines. OSD has also begunfunding distributed test capabilities that can tietogether the Services’ test ranges and integrate live,virtual, and constructive (LVC) [simulation] methodsto create true Joint environments. Still, assembling theresources for this Joint testing is beyond the constraintsof the current fiscal environment, so it will takeinnovative test and training integration to address SoSrequirements. It will also take work at the grass rootslevel by others in the Testing and Evaluation (T&E)

arena (outside USJFCOM) to ensure they are ready tofeed into emerging processes and infrastructure.Testers should become adept at using graduated levelsof LVC as systems mature through spiral or incre-mental development to ‘‘graduation’’ in integrated testand training events. The 505th Command and ControlWing (505 CCW) is developing a way to join thiseffort by creating a (Joint) Theater Air-Ground System(TAGS) capability that can integrate a distributed SoStest capability with training venues such as Red andGreen Flag.

The theory behind testingMilitary Operational Test and Evaluation (OT&E),

at its most basic level, is simply the application of thescientific method to decision requirements for hard-ware; software; concepts; and tactics, techniques, andprocedures. In the larger context, operational testingcan be explained in the context of game theory. It is anattempt to ascertain the true state of the capabilitybeing tested with respect to fulfillment of certainrequirements. Testers sample that capability in asimulated combat environment or in a real operationalenvironment, and based on the observed results, drawconclusions about the true underlying capability of thesystem, its ‘‘state of nature.’’ The recommendations todecision makers follow from these conclusions. Deci-sion-makers use this information to determine theiractions based on risk considerations. Thus, the test is arisk-reduction tool for the decision-maker—it givesthe best estimate of the state of nature. Appendix Aprovides a more complete mathematical explanationbased on statistical decision theory (Ferguson 1967).

For traditional system-level operational testing, thestates of nature could be considered dichotomous: thesystem meets requirements and performs satisfactorily(it is effective and suitable), or it does not. But we canalso broaden this to a determination that the systemcontributes favorably to warfighting capability or has no(or even negative) impact on warfighting capability. Theactual observed outcome of the test would be multi-dimensional, representing the level of attainment ofobjectives identified by the test team. A decision rulewould be devised (typically subjectively) to map thesepotential observed outcomes into an action (recommen-dation) vector A, say A 5 [a1, a2, a3, a4], where

N a1 is a recommendation to field the system as is;N a2 is a recommendation to field the system after

identified deficiencies are corrected;N a3 is a recommendation not to field the system

but to continue development; andN a4 is a recommendation not to field the system

and to cease development.

Kometer, Burkhart, McKee, & Polk

40 ITEA Journal

Theoretically, a loss function representing theconsequences of the ordered pair (the state of natureand the action taken based on the decision rule andobserved outcomes) could be calculated. The risk foreach state of nature is the statistical expected value ofthe loss function.

Based on the identified risks for each decision rule,decision makers and the test team could develop astrategy for testing, recommendations based on testresults, and subsequent decisions on acquisition andfielding.

Operational testers and decision makers do notdefine explicitly the loss function or documentalternative decision rules—it is highly unlikely theywill ever have sufficient data for this. However,intuitively they should be aware of the impacts offielding immature or deficient systems and of with-holding badly needed capability. Testers, eitherimplicitly or explicitly, must account for Type I Errors(failing a mature or well-functioning system) and TypeII Errors (accepting an immature or deficient system)and the subsequent impact of their recommendationson decision makers and ultimately the U.S. Air Force’s(USAF’s) warfighting capability.

Evaluation that extends the operational tester’spurview to questions of a system’s functioning withinan information network or an SoS architecture requiresa fresh look at the loss function associated with systemcapabilities and impact on warfighting capability aswell as the overall statistical risk associated with testconduct and recommendations. As we shall see below,as systems become more interdependent for informa-tion exchange, the loss function should be based moreon a system’s impact to overall warfighting capabilitythan on a comparison with system-level requirements.

What is different in the Information Age?The military services were designed to organize,

train, and equip forces to fight in their respectivebattlespace environments. This organization has itspurposes—it ensures the particular needs within theseenvironments are accounted for when developing thecapabilities that will allow the military to perform itsfunction. If all the capabilities were independent anddid not come into contact with each other, there wouldbe no need to test the systems in an SoS context. Butthey are not independent, and they do come intocontact with each other. Support functions like closeair support ensure the different services and functionalcomponents have to perform interdependently.

The Information Age has greatly increased theopportunities for integration and interdependence thatdrive this need to perform as an SoS. The ongoingtechnological revolution demands an appropriate

response from those who wish to remain competitive.Command and control in the business world hasdemonstrated this revolution. For a century and a halfthe trend in American business was toward centrallycontrolling massive corporations. From single-unit,owner-managed enterprises with independent mer-chant distributors in the early 19th century, theAmerican firm developed into a colossal, centrallymanaged behemoth in the late 20th century (Chandler1977). But the Information Age pushed the trendtoward decentralization and integration among thelower levels, instead of control from the higher levels.Strategy formerly aimed at controlling the actions ofbusinesses is now instead aimed at constructingrelationships among them, coordinating the use ofresources so operations can be flexible yet focused.With today’s information technology, workers canretrieve all of the information they need at the righttime and place to make decisions on the spot, wherethey are most crucial (Castells 2000). Companies nowlook for others who have the core expertise to performparts of their operations for them. They ‘‘Interlink’’ the‘‘value chains’’ of suppliers, firms, and customers totransform the marketplace (Porter and Millar 1985). Itis now more of a system than a pool of competitors.

The DoD is making a similar transformation. Formore than a decade, Network Centric Warfare (NCW)prophets have urged this transformation. They proposethat the military must prepare to fight NCW, ‘‘anemerging mode of conflict (and crime) at societallevels, short of traditional military warfare, in whichthe protagonists use network forms of organization andrelated doctrines, strategies, and technologies attunedto the information age’’ (Arquilla and Ronfeldt 2001).Technology has enabled these new modes becausecommunication is faster, cheaper, and of higherquality. But NCW is not only about technology. It isabout the linkages among people—networks, unlikeformal hierarchies, are plastic organizations with tiesthat are constantly being formed, strengthened, or cut(Williams 2001). Most important, these analysts claimthat ‘‘it takes networks to fight networks’’ (Arquilla andRonfeldt 2001).

The U.S. military must capitalize on the currentinformation revolution to transform its organization,doctrine, and strategy. It must retain its Command andControl (C2) capability, while becoming flatter—attaining faster response by eliminating some hierar-chical levels in favor of pushing information out to allplayers at the lower levels. Doctrine should be builtaround battle swarming, a process of bringing combatpower to bear at nearly any time and place based onreal-time information (Arquilla and Ronfeldt 1997).The term ‘‘NCW’’ refers to a concept that ‘‘translates

SoS Operational Testing

32(1) N March 2011 41

information superiority into combat power by effec-tively linking knowledgeable entities in the battlespace’’(Alberts, Garstka, and Stein 1999). Its proponentsargue that C2 should not be envisioned as a sequentialprocess as it has been in the past—gathering data,analyzing, making a decision, and then implementingit. Instead, sensors, actors, and decision makers shouldbe networked, so that they have a shared awareness ofthe battlespace. Commanders at the lowest levels willhave enough information to take initiative and speedup the response to changing battlefield conditions(Alberts, Garstka, and Stein 1999).

For the acquisition community, the effect of thistransformation is that the systems it acquires for themilitary Services are increasingly required to interoperatewith systems of other services. As Admiral Cebrowskyput it, ‘‘In reality, what has happened is that a new air-ground system has come into existence where you nolonger talk in terms of one being supported and the othersupporting. That would be like asking if the lungs are insupport of the heart or if the heart is in support of thelungs. It’s a single system’’ (Cebrowsky 2002). The visionof Air Force leadership through the 1990s was thatairpower would be able to execute a ‘‘kill chain’’ as rapidlyas possible due to the smooth integration of a ‘‘system ofsystems’’ (U.S. Air Force 2000). The acquisitioncommunity has taken strides to facilitate machine-to-machine transfer of data among these systems, usingtools like Web services with XML data.

This is the SoS that operational testers must learn toevaluate. In truth, the SoS could take many forms,accomplishing many different operational threads.Close air support, defense against ballistic and cruisemissiles, dynamic targeting of mobile ground targets,and construction of a single integrated air picture areall missions that require an SoS to work in anintegrated fashion. However, the acquisition commu-nity is not structured to consider the needs of the Jointenvironment in the requirements process (DODDOT&E, 2004). Recognizing that this emergingnetwork-centric paradigm required a different systemsengineering approach, the DoD promulgated guidancein Defense Planning Guidance (DPG) 2003 and 2004and Strategic Planning Guidance (SPG) 2006. Thesedocuments moved the DoD towards a Net-CentricGlobal Information Grid (GIG) and Network CentricEnterprise Services (NCES) (DOD AT&L, 2004).

If the decision makers were truly to attempt todefine the loss function, they would have to considerthe impact of the system on the performance of theSoS, not just the comparison of the system to its ownisolated requirements. A new gateway may haverequirements to forward certain message formats, butits real function is to synchronize the situational

awareness of the commanders and troops and allow amore rapid (and effective) transition from informationto action. This is the ‘‘impact on warfightingcapability’’ discussed earlier. The loss function shouldbe defined based on this broader impact, not onwhether it meets the narrower system-based require-ments. The loss is positive (or at least non-negative) ifthe system is fielded but does not increase warfightingcapability, or if it is not fielded but could haveincreased warfighting capability—regardless of wheth-er or not it forwards the required message formats.

Testing only the narrower system-based require-ments actually increases the probability of making aType-II error. Testers could induce loss by recom-mending fielding of a system that will not have apositive impact on warfighting in today’s environment.Warfighters in command and control positions have somuch information that simply adding more informa-tion does not make them more effective—the infor-mation must be added in a way that allows them to dotheir job more effectively.

This brings up a reality that cannot be overlooked:evaluation of the SoS is decidedly incomplete withoutconsideration of the human interactions involved. Allthis information must be organized and presented in away that enables the warfighters to do their jobseffectively and efficiently. The more warfighters areable to cross functional and service lines, the moreavenues they have to be innovative in accomplishingthe mission. However, having more avenues forinnovation also means increased difficulty enforcingglobal procedures. If the people who make commandand control possible are unsure of or drift from globalprocedures meant to avoid fratricide and otherunintended consequences, accidents could occur (Ko-meter 2007). Tests of NCW capability require

1. assessment of the capability of the SoS as an aidto the people to bring combat power to bear atthe right time and place,

2. determination of C2 responsibilities from thelowest tactical level to the strategic level, and

3. development of tasks, techniques, and procedures(TTP) to implement the entire network and SoS.

A business model for SoS requirementsThe problem is that testing is requirements driven,

and right now requirements are mostly system based(U.S. Air Force 2004a). Currently, DODI 5000.2requires all systems to undergo interoperability evalu-ations throughout their life cycles (DOD 2003). Forinformation technology systems with interoperabilityrequirements, the Joint Interoperability Test Com-mand (JITC) must provide certification of critical

Kometer, Burkhart, McKee, & Polk

42 ITEA Journal

interfaces throughout the life cycle, regardless ofacquisition category (Wiegand 2007).

In fact, Network-Readiness (NR) Key PerformanceParameters (KPP) assess NR (which is defined inDODD 4630.5) as follows: ‘‘Net-readiness is thecontinuous capability to interface and interoperate toachieve operationally secure exchanges of informationin conformance with enterprise constraints. The NR-KPP assesses net readiness, information assurancecontrols, and both the technical exchange of informa-tion and end-to-end operational effectiveness of thatexchange’’ (DOD 2007).

But most current testing only validates that thesystems will be able to pass data to the appropriateinteroperable systems. It does not address whether theSoS will function correctly—or, in particular, that thepeople involved will understand how their role changeswhen a mission crosses normal functional lines. To goto this level, the systems must be tested within thelarger SoS. Indeed, JITC is heavily involved in thepush toward end-to-end Joint environment testing toanswer the NR-KPP (TRMC 2006, Clarke 2007).

The conceptual issue with tests of this sort stemsfrom the fact that there is no organizational respon-sibility and, therefore, no resources are available forSoS testing involving multiple command and control,intelligence, surveillance, and reconnaissance (C2ISR)nodes and weapons systems. Systems are currentlyfunded by program, not by capability. Individualprogram offices fund testing of their system require-ments only. Although these requirements now includeNR-KPPs, these KPPs deal only with whether thesystem can receive and provide information in formatsrequired to be interoperable with other nearest-neighbor network systems. They do not specify howthe larger SoS should do its job. Yet in the InformationAge, a program does not constitute a capability.Capabilities cut across multiple programs, requiringthem to interoperate and exchange information.

Tests of capability-based requirements call for a newbusiness model. The Joint Battle Management Com-mand and Control (JBMC2) roadmap is a capabilities-based construct, which lays out the elements of thismodel. The roadmap relies on a Joint Staff–developedconcept of how the Joint force will operate in thefuture across the range of military operations. This‘‘Joint Operations Concept’’ leads to Joint OperatingConcepts, Joint Functional Concepts, Joint EnablingConcepts, and Integrating Concepts. While its futuremay be in doubt, the executor of the JBMC2Roadmap, USJFCOM, currently leads the COCOMsin the development of JMTs—comprehensive descrip-tions of how the Joint force will execute one of sevenwarfighting capabilities. The Joint Staff and Joint

Requirements Oversight Council (JROC) (or func-tional capabilities boards, on behalf of the JROC),reviews and validates requirements for development ofthe JMTs (DOD AT&L 2004). This business modeldoes not change the milestones or purpose of test.Testing of SoS is to be accomplished within thecontext of existing Developmental Test and Evaluation(DT&E) and Operational Test and Evaluation(OT&E) in the Test and Evaluation Master Plans(TEMPs). But the testing must be done in a Jointenvironment, using validated requirements for therelevant Joint mission (DOD DOT&E 2004).

DoD is leading an effort to implement such a modelfor the Single Integrated Air Picture (SIAP)—a ‘‘shiftin the Department of Defense’s traditional focus on anindividual combat system’s performance to the ensem-ble performance of a SoS’’ (JSSEO 2006). The JointTheater Air and Missile Defense 2010 OperationalConcept produced a conceptual template for JointIntegrated Air and Missile Defense that depended on aSIAP (JSSEO 2006). So the Vice Chairman of theJoint Chiefs of Staff (VCJCS), the Under Secretary ofDefense for Acquisition, Technology, and Logistics(USD AT&L), and the DoD Chief InformationOfficer (CIO) chartered the SIAP system engineeringtask force in 2000. They needed a disciplined, Jointprocess to resolve interoperability problems to imple-ment a truly Joint data network that could create theSIAP. As the test strategy puts it, ‘‘Technically, theSIAP is a state of mutual consistency within a weakly-connected, heterogeneous, decentralized, and distrib-uted system’’ (JSSEO 2006). Unfortunately, wherehuman organizations and money are concerned,‘‘weakly-connected’’ and ‘‘heterogeneous’’ do not in-duce action.

In 2003, the JROC designated JFCOM and a SIAPAcquisition Executive to direct the program andestablish funding lines in the Services (JSSEO 2006).The Services will be responsible for ensuring theirindividual systems conform to an Integrated Architec-ture Behavior Model (IABM)—an open architecturecomputer model of prescribed system behavior withbit-level precision that is the product of Joint systemengineering (DOD 2003).

Distributed test capabilities for SoS testGetting the requirements right is just one part of the

equation. Another problem with testing Joint, NCWcapability is the need to construct the SoS for a test.Command and control of the forces requires asophisticated network including Internet, landline,satellite, and line-of-sight protocols. This is the SoSunder test. But T&E of this SoS requires another SoSto monitor and collect data during the test. On top of

SoS Operational Testing

32(1) N March 2011 43

this, the test environment may require augmentation oflive assets with virtual and constructive Modeling andSimulation (M&S) methods in a Hardware-In-The-Loop (HITL) configuration.

‘‘Testing in a Joint Environment Roadmap for2006–2011’’ (DOD DOT&E 2004) presented guid-ance for developing these capabilities. It addressed thefact that the Services each had disparate test capabil-ities that were in some cases redundant and in manycases insufficient. It foresaw the need to create auniversal ‘‘persistent, robust distributed systems engi-neering and test network that can link the specificremote sets of HITL, M&S, and other resources withthe live system in development to accomplish thesystems engineering or testing required for a spectrumof transformational initiatives, as well as to supporttraining exercises and experimentation’’ (DOEDOT&E 2004).

The roadmap set the stage for the Joint MissionEnvironment Test Capability (JMETC). The JMETCprogram office falls under the Test Resource Manage-ment Center (TRMC), which reports to USD AT&L.It collaborates with the Central Test and EvaluationInvestment Program (CTEIP) to fund the programsthat will provide a corporate approach to integratingdistributed LVC capabilities, solving the problemsinherent with the Service-specific capabilities, multiplenetworks, and various different standards that exist. Itwas therefore meant to reduce duplication of effort,provide readily available security agreements, andfacilitate Joint testing and integrated test and training.JMETC establishes persistent connectivity via aVirtual Private Network (VPN) on the SECRETDefense Research and Engineering Network(SDREN), adopts the Test and Training EnablingArchitecture (TENA) middleware and standard inter-face definitions, collaborates with CTEIP to adoptdistributed test support tools, and provides datamanagement solutions and a reuse repository (Fergu-son 2007). The distributed test support tools are beingdeveloped as part of a project called the Joint C4ISRInteroperability Test and Evaluation Capability (Inter-TEC) and include communications control, testcontrol, instrumentation and analysis tools, syntheticbattlespace environment, and simulation/emulationgateways (JITC 2008). As the CTEIP 2006 reportputs it, ‘‘the envisioned end-state is a seamlessly linked,but geographically separated, network of test facilitiesand ranges in which the most modern and technolog-ically advanced defense systems can be tested to the fullextent of their capabilities’’ (TRMC 2006).

These capabilities are still in the maturing phase.The development of Joint Close Air Support (JCAS)mission threads was the impetus for several events

during 2007. The 46th Test Squadron (TS), EglinAFB, conducted a baseline assessment of Link 16 andSituation Awareness Data Link (SADL) to answerquestions about the capability of Joint Terminal AirControllers (JTACs) to send digital 9-line messagesdirectly to the cockpit (46th Test Squadron 2007).

That same year, the Simulation and Analysis Facility(SIMAF) at Wright-Patterson AFB sponsored the AirForce Integrated Collaborative Environment event‘‘Integral Fire 07.’’ This event developed a distributedtest environment to satisfy three different test custom-ers: JFCOM’s Joint Systems Integration Center, theDoD Joint Test and Evaluation Methodology JT&Eprogram, and the Warplan-Warfighter Forwarderinitiative, sponsored by the USAF Command andControl and Intelligence, Surveillance and Reconnais-sance Battlelab. This was the inaugural use of theJMETC to tie together three separate enclaves—15total locations—with an aggregation router. The sitesused the TENA gateways to exchange simulation orinstrumentation information via TENA protocols(TENA 2008).

More recently (2009–2010), JMETC has partici-pated in JEFX events to provide the network backboneto share and exchange near-real-time tactical informa-tion among participants and monitoring test agencies.Two examples are the use of the Guided WeaponsEvaluation Facility (GWEF) to generate simulatedNet-Enabled Weapons (NEW) for management andcontrol by Air Operations Center personnel (JEFX 09)and F-15 and JSTARS Operational Facility (OPFAC)simulation of manned aircraft interacting with livesystems to ‘‘round out’’ testing (JEFX 09, JEFX 10).

In September 2007, USJFCOM J85 conducted anAdvanced Concept Technology Demonstration(ACTD) called BOLD QUEST that provided anotheropportunity to develop distributed test capability.Expanding on the JCAS JMT, JFCOM decided tolook at interoperability of coalition fighters with threeJoint terminal air controller suites: Tactical AirControl Party Close Air Support System (TACP-CASS); Battlefield Air Operations (BAO) Kit; andTarget Location, Designation, and Hand-off Kit.BOLD QUEST had been designed to assess CoalitionCombat Identification (CID) but was deemed theright venue and timing for USJFCOM’s Joint FiresInteroperability and Integration Team (JFIIT) toconduct the JCAS assessment as well (JFIIT 2007).The 46 TS and 640th Electronic Systems Squadron(Hanscom AFB, Massachusetts) deployed three mobiledata link facilities (called ‘‘Winnies’’) to Angels Peak,Antelope Peak, and the National Training Center(NTC). These Winnies supported both data collectionand the tactical infrastructure capability for the event.

Kometer, Burkhart, McKee, & Polk

44 ITEA Journal

In this way the 46 TS demonstrated the capability forremote collections with local, centralized, analysis. The46 TS established network connectivity to Winnies atAngels Peak, Antelope Peak, NTC, and a JFIIT-provided range-instrumentation data-stream at NellisAFB. Additionally, each mountain top Winnie estab-lished connections to the Joint Range Extension (JRE)at the Joint Interface Control Cell (JICC) CombinedAir and Space Operations Center (CAOC) and JFIITGateway Manager (GM). These configurations pro-vided dedicated data to 46 TS, JFIIT, and the JICO(Cebrowski 2002). Remote data collection and analysisare, of course, important components of distributedtesting. JMETC has provided additional networkingcapability during BOLD QUEST 08/09/10 to employ46 TS OPFACs in testing the CID server’s capabilityto determine receipt of fighter Link-16 J12.6 messagesasking for five closest friendly positions near intendedtarget area before employing the CID server in live-flyenvironment. In addition, it supported testing andverification of CID server J3.5 responses to fighterqueries.

The challenge of executing SoS testsSoS testing is not achieved typically because the cost

is prohibitive and dedicated access to key assets isfrequently limited or nonexistent. Some test organiza-tions made progress toward objectives in the develop-ment of warfighting capabilities or processes, but not inproportion to the cost. Typically, with the exception ofBOLD QUEST, these events were underwritten withthe intent of developing the capacity to perform thistype of testing in the future. Until SoS test environ-ments are consistently—or even persistently—avail-able, it will not be feasible for the test customer to testthis way in the absence of sponsorship (46th TestSquadron 2007). Even after the test SoS is available,testing in a Joint environment typically requiressignificant live assets, more coordinated planning,and a robust scenario. ACTDs like BOLD QUESTand other specific projects may have the budget toaccomplish this, but most will not.

Getting all of the assets together at the same timeand appropriate place is also difficult. A robustcommand, control, communications, computers, intel-ligence, surveillance, and reconnaissance (C4ISR), air,space, and cyber network and appropriate-sized strikeforce could include hundreds (or thousands) of entitiesintegrated with real-world communications and otheroperational infrastructure. Operational assets such asairborne C2 platforms, control and reporting centers,and tactical air control parties are in high demand fordeployments and exercises. Scheduling them andgetting them to operate simultaneously in the same

dedicated test battlespace is extremely challenging formany reasons (DOD DOT&E 2004).

One solution being investigated is integrated testand training. Even though the two venues differ intheir objectives, test and training share commonresources and analytical methodologies to some extent(DOD DOT&E 2004). Exercises and experimentsalready attract most of the assets required for end-to-end tests. Green Flags present an air-ground warscenario, while Red Flags are centered on preparing forthe air war in general air warfare. Atlantic Strikeprepares ground troops for coordination with airsupport in the war on terrorism. The added fidelityand infrastructure of assets usually identified withtesting would be welcomed in many cases. For thisreason, OSD has directed the integration of test andtraining in a memo to the services (Krieg, Chu, andMcQuery 2006).

But the two disciplines do not mix easily. Anytesting added to the exercise venues likely wouldrequire transparency to training audiences and have nonegative impact on training objectives. Testers wouldhave to participate throughout the planning process todevelop appropriate scenarios and identify requiredoperator actions to satisfy their data requirements. Theprimacy of training objectives may preclude capturingall the data necessary to complete test objectives. Inaddition, testers might be restricted from repeatingevents where the conditions were not right for testpurposes. For these reasons, the debate about whethertest and training integration will work routinely inpractice rages on.

Some test and training integration efforts havedemonstrated the potential for success. JFIIT conductsaccreditation of C2 procedures at the NationalTraining Facility and at Avon Park during AtlanticStrike exercises for ground troops headed for theCentral Command theatre (USJFCOM 2010). Alongwith this training role, the unit frequently accomplish-es test activity using the same resources. For example,at the Atlantic Strike exercise in November 2007,JFIIT teamed with 605 Test and Evaluation Squadron(TES) and 46 TS to develop the architecture for anddemonstrate the performance of the new Air SupportOperations Center (ASOC) gateway. This was not aformal test, but it reduced risk for the upcomingoperational test of TACP-CASS 1.4.2 and ASOCgateway by solidifying the concept of employment(CONEMP) and TTPs. JT&E organizations like JointDatalink Information Combat Execution (JDICE) andJoint Command and Control of Network EnabledWeapons (JC2NEW) have successfully used trainingvenues to accomplish their objectives. JDICE con-ducted a quick reaction test of Joint Integration of

SoS Operational Testing

32(1) N March 2011 45

Nationally Derived Information at Valiant Shield 07and several real-world events. JDICE also validatedLink 16 TTPs and architectures to enhance the killchain and to filter and de-conflict the targeting picturefor ground troops in Red Flag 04, Valiant Shield 06,and Red Flag 06.1 The effort validated architecturesand TTPs for the national intelligence community toprovide information rapidly to tactical and operationallevel warfighters. It should be noted, however, thatJT&Es focus on tactics, processes, and procedures—not new systems.

On the whole, the test world is still struggling todevelop the methodology for integrated testing andtraining in a way that brings the promised efficiencies.The transition to capability-based requirements andtesting is ponderous. New systems reaching operationaltesting are still, for the most part, subject to the classicrequirements process. This is true to an even greaterdegree for capabilities in the sustainment phase. ForAir Force systems in sustainment (and new ones forwhich Air Force Operational Test and EvaluationCenter [AFOTEC] non-involves) Major Command(MAJCOM) test organizations often assume respon-sibility for operational testing (U.S. Air Force 2004a).

505 CCW concepts for SoS testAs the Air Combat Command (ACC) focal point

for C2 operational testing, 505 CCW is somewhattrapped in the systems-based requirements process.The 605 TES tests only those capabilities that are insustainment or for which AFOTEC waives involve-ment. The squadron tests programs based primarily onACC or other requesting agency requirements, andusually not JROC-approved JMT requirements, asmentioned earlier. To get to true NCW, testers willhave to adopt an SoS methodology for new systems.Upgrades to fielded systems must also be tested fortheir contribution to warfighter capabilities, forconsistency with testing to this standard before andduring initial operational test and evaluation(IOT&E). MAJCOM testers have not been involvedin the SoS requirements development process to thispoint. Testers do not (indeed should not) developrequirements except to the extent that they caninfluence those who do to make the requirements‘‘testable,’’ and program offices are not inclined to fundtesting beyond that which determines the extent ofsystem-level compliance with requirements.

TD&Es are an alternative avenue for addressingmission-level testing. When warfighters need addi-tional capability, new materiel solutions provide oneway to fill the shortfall. Non-materiel solutions areanother possibility. In Air Combat Command, wingssubmit tactics improvement proposals (TIPs), which

are then prioritized at an annual tactics review board inJanuary and subsequent overall combat air forces(CAF) test prioritization process. Predictably, theseTIPs often call for development of TTPs for dealingwith cross-functional problems such as close airsupport, nontraditional ISR, or sensor fusion. C2TD&Es typically are not accomplished because of alack of advocacy at sufficiently high levels andinadequate funding. In spite of being the CAF’s leadfor C2 operational testing, including TD&Es, the 605TES is largely funded by charging program offices (orother requesting agencies) for level-of-effort supportand travel expenses on specific test projects. ACC hasnever formalized TD&E funding even though someTD&Es are among the highest prioritized CAF testprojects each year (Kometer 2007). Piggy-backingTD&Es on training exercises may be the only way toaccomplish them on a recurring, systematic basis, atleast for those involving multiple and varying plat-forms/systems.

To gain access to major training exercises for testpurposes, the test construct must be transparent (ornearly so), and preferably beneficial to the trainingcommunity. The largest training exercises, the Red andGreen Flags and Weapon School Mission Employ-ment Phase, are flown out of Nellis AFB, where the422d Test and Evaluation Squadron and the 57thWing are located. These exercises already use CAOC-Nellis (CAOC-N) to provide a limited operational andtactical C2 experience to enhance the excellent tacticallevel training traditionally associated with these events.

The 505 CCW, in collaboration with 46 TS andSIMAF, is developing a test and training capability tobegin bridging the gap to net-centric and SOS testing.Starting in 2007, the 605 TES earned OSD ResourceEnhancement Program money to fill operationaltesting shortfalls for Theater Battle Management CoreSystems, datalink, TACP-CASS, and network-centriccollaborative targeting. In implementing their shortfallsolution, the team was able to include a gateway toaccess the distributed architecture discussed above. Thewing has successfully obtained CTEIP funding toimplement infrastructure to enable the conduct ofintegrated LVC end-to-end testing in conjunctionwith existing training events to support Joint warfight-ing customers by leveraging the Western RangeComplex, LVC experts, and distributed architectures.OSD is providing additional support toward develop-ing capability with the goal of creating a simulatedTAGS anchored around CAOC-N that can beintegrated with Air Force, Joint, and coalition trainingevents. It will be fully meshed with JMETC andInterTEC to ensure it can be an integral part of thedistributed testing architecture discussed earlier (Wie-

Kometer, Burkhart, McKee, & Polk

46 ITEA Journal

gand 2007). The 505 CCW is reorganizing to focusgreater attention on integrated testing of air, space, andcyber capabilities in this emerging environment. Thenew organizational structure will facilitate bettermutual support among the Wing’s geographicallyseparated testing, simulation, and training compo-nents. It will also provide greater opportunity forleveraging the connectivity provided by its DistributedMission Operations Center at Kirtland AFB to otherDoD developmental, simulation, and testing compo-nents needed for robust SoS testing. Additionally, theaddition of the JDICE mission and expertise to theWing will tremendously enhance the capability to testand train in the emerging net-centric Joint rangeenvironment.

Although the initial drivers for developing theTAGS environment were TD&Es, the project wasundertaken because of the foresight needed to meetSoS testing requirements at every level. Talks with theaircraft and weapons testers in the 53d Wing helpedstimulate this developmental effort since both ACCtest organizations realized they would have to interfacefor net-enabled weapons testing. As C2 and weaponssystems become more interdependent they becomeelements within an integrated SoS architecture andmust be tested as such. Other C2 capabilities alreadydemand SoS test methods.

The proposed concept for transitioning to SoStesting is based in part on evolutionary acquisition (thesequential release of increments and versions ofcapability), the currently favored paradigm for acqui-sition of many C2 systems. This paradigm comes fromthe software development industry and has often beenassociated interchangeably with ‘‘spiral development.’’Software developers realized long ago that firstattempts to deliver a coded product often would revealsignificant problems. Upon seeing early deliveries,users would find glitches and possibly even realize thatoriginally stated requirements did not adequatelydescribe their needs. But software-intensive systemsare often easier and less expensive to change thanhardware acquired from production lines and, thus, aremore amenable to concurrent and sequential develop-ment, correction, and enhancement. Thus, softwaredevelopers evolved the best practice of developingcapability in successive iterations called ‘‘spirals,’’ eachof which provided greater/improved capability whilemore fully satisfying customer needs. When, duringthe spiral process, the capability was finally acceptable,it could be delivered. The current acquisition model forsoftware-intensive C4ISR systems is built around thisapproach. DODI 5000.2 (DOD 2003), AFI 99-103(U.S. Air Force 2004a), and AFI 63-101 (U.S. AirForce 2004b) tell us that capability is to be delivered in

‘‘increments,’’ each of which is to undergo this type ofspiral development.

Marrying the spiral development model with theconstraint on limited open-air test time leads us toconsider a graduated approach to testing, using LVCinfrastructure and capabilities as much as possible. Theincreasingly robust and progressively more operationallyrelevant electronic warfare test process, defined inAFMAN 99-112 (U.S. Air Force 1995), could be astarting point for this approach. This process recom-mends the use of six types of test support/environ-ments—M&S, system integration laboratories, HITL,measurement facilities, installed system test facilities,and open air ranges—each of which tests a system in adifferent state of maturity. Program managers areencouraged to select the appropriate level of testing atthe appropriate time in the life cycle of a system to reducerisks for the acquisition program and for future tests.

C4ISR systems could use a similar progression, usingvarying levels of LVC testing at the appropriate time inthe process. The test strategy should include integrationinto the larger C4ISR SoS within which the individualsystem will function. In early development, this could beaccomplished by using a simulated environment such asthat provided by the JMETC architecture, essentially adistributed HITL network. As the system matures, realsystems could be added to this HITL architecturestructure to ensure that the system can performsatisfactorily in an operationally realistic environment(while retaining the operationally representative stimuliprovided by the simulated environment). During testingwith progressively more live-entity involvement, testerscould examine the system’s performance, effectiveness,and suitability within the SoS and the impact on theTTP employed by the operators within the SoS. Finally,open-air/field exercises could be used to verify systemcapabilities and TTP developed through earlier spirals,in a ‘‘graduation event’’ for OT&E.

For example, TACP-CASS version 1.2.5 test resultsshowed the need for this type of approach. The systempassed Developmental Testing (DT) and proceeded tooperational testing in 2005. One system requirementwas that it successfully interface with the Army BattleCommand System to receive friendly ground forcesituational awareness messages. However, in theoperational environment, it bogged down when themessage load approached that of a division. It also hadproblems reconciling messages from two Army units inclose proximity to each other. The problems were sosignificant that TACP-CASS was not fielded—solutions were postponed to later versions (DiFronzo2006). These problems were not observed during DTbecause of the constrained DT test environment. Theanomalies appeared once the system encountered

SoS Operational Testing

32(1) N March 2011 47

operationally representative information traffic. Test-ing that requires a dedicated large Army force and liveaircraft is too expensive to be accomplished throughoutdevelopment of a system. The proposed alternativewould subject the developing system to a sequence ofprogressively more robust and demanding simulatedenvironments, culminating in multiple systems in adivision-sized configuration. Had this been done, thedeficiencies would have surfaced as the scale andrelative positions of real-world forces evolved. Thesedeficiencies, in turn, could have been corrected prior tooperational testing.

Similarly, during 2009 testing of TACP-CASS1.4.2, the 605 TES established an operationallyrealistic Joint air request High-Frequency (HF) radionetwork to evaluate the capability of contractor off-the-shelf (COTS) equipment designed to provide analternative to the fielded satellite communications(SATCOM) system. The radios performed satisfacto-rily in a limited network designed during DT but failedmiserably in operational field testing. The cause of thefailure is still being investigated, and the system wasfielded without the radios. While TACP units canaccomplish their mission via SATCOM, the opportu-nity to relieve the demand on SATCOM support waslost. A persistent network, readily available to the testcommunity, could have provided the proper environ-ment to detect the deficiency in early TACP-CASS1.4.2 testing rather than in final operational testing.

ConclusionsRight now, it is not clear who will be responsible

ultimately for SoS testing. It is quite likely that for thenear future, Service operational test agencies andMAJCOM operational testers will continue to testsystems to system-level requirements. However, as thesesystems are increasingly seen as families of systems thataffect the JMTs, they eventually need to be tested inJoint SoS environments. That makes it imperative thatthe initial testing lead to and be guided by the eventualrequirements for SoS testing, including commonmeasures and objectives where possible, and by commonLVC-enhanced environments wherever feasible.

As the capability to accomplish SoS testing maturesand becomes persistent and universally available, smallertest organizations, like those of the 505 CCW and otherMAJCOM testers, will be able to access the services ofJMETC and M&S providers more easily. Operationaltest organizations will work with DT organizations toensure M&S adds operational reality and robustenvironments into DT (possibly via operational assess-ments) that will reduce the risk for eventual OT&Eduring an integrated test and training exercise. The datacollection infrastructure employed simultaneously with

training exercises will facilitate data collection andmanagement for both test and exercise agents.

However, the test community has not progressed tothat point yet. Distributed test capabilities are notuniversally available yet, much less embraced by all inthe greater acquisition and user communities. Indeed, itwill take both greater incentives and support from OSDand increased initiative from the grass roots level to leadthe two communities in that direction. Test units, usercommands, and program offices will have to agree tolobby for the addition of testing in the Joint environ-ment, using LVC simulation early on while leading torobust SoS tests. Test units will have to become creativeand develop approaches for testing and trainingintegration acceptable to all responsible organizationsand at the same time champion funding the investmentsrequired for testing at the network and SoS levels.

The transformation of DoD testing will require achange in culture. Testing in conjunction with trainingwill require testers to collect data over multiple events toaccomplish their objectives; they will no longer be ableto rely on planning a single dedicated opportunity tosatisfy all objectives. This effort may even lead to moreopportunities to acquire reliability, availability, andmaintainability data as systems undergo testing overseveral events instead of just one. The outcome could beserendipitous to developers and testers trying to respondto DOT&E’s direction (DOT&E Memo, 30 June2010) for greater rigor in addressing suitability issues.Of course, this could wreak havoc with success-orientedprogram schedules. Testers will have to work withdecision makers to determine what level of uncertainty isacceptable to demonstrate the impact of the newcapability on the warfighting SoS. Seamless verificationwill include seamless transition from M&S-basedevaluation to testing in a robust Joint LVC environ-ment. Unless the acquisition and test communities arewilling to embrace creative means like those discussed,and they receive the necessary support from service andOSD leadership, decision makers will not be able togauge effectively the risks involved with their decisionsfor fielding new capability. Capability-based acquisitionis mandated by OSD, it is gaining momentum, and it isthe right thing to do for our nation’s security. C

COL. MICHAEL W. KOMETER is currently professor of

security studies at the School of Advanced Air and Space

Studies (SAASS), Maxwell AFB, Alabama. Commis-

sioned at the Air Force Academy (Colorado Springs,

Colorado) in 1988, he is an electronic warfare officer with

combat time in the AC-130H and experience as a planner

and strategist in multiple conflicts. He has performed

Kometer, Burkhart, McKee, & Polk

48 ITEA Journal

developmental and operational test, including command-ing the 605th Test and Evaluation Squadron from 2006to 2008. Col. Kometer earned master’s degrees fromGeorgia Tech (Atlanta, Georgia), Air Command and StaffCollege, and SAASS and a doctor of philosophy fromMassachusetts Institute of Technology (Cambridge, Mas-sachusetts). E-mail: michael.kometer@maxwell.af.mil

LYLE KENT BURKHART is an operational test contractor(Sentel) for the 605th Test and Evaluation Squadron,Hurlburt Field, Florida, conducting operational T&E ofC2 systems. He has over 17 years of T&E experience and20 years active duty in the USAF, 8 years in C2. Mr.Burkhart has led initiatives in support of test and traininginterdependencies and continues to support these efforts asa system tester. He received his bachelor’s degree in generalstudies at Louisiana Tech University, Ruston, Louisiana,and is completing his master’s degree in systems engineer-ing with Regis University, Denver, Colorado. E-mail:Lyle.Burkhart.tr@.af.mil

SUSANA V. MCKEE has 24 years of T&E experience. Sheis the 505th Command and Control Wing’s chief ofcapabilities development and integration at HurlburtField, Florida. Previously, Ms. McKee was director of testfor the 605th Test and Evaluation Squadron C4ISRtesting efforts. At Eglin AFB, she served as chief of analysisfor the 53rd Wing’s Air-to Ground Weapon SystemEvaluation Program and was an analyst for operationalflight program upgrades. At the 46th Test Squadron, sheconducted safe escape and ballistics analyses. Ms. McKee

earned undergraduate and graduate degrees from Univer-sity of West Florida, earning mathematics and politicalscience degrees. E-mail: Susana.McKee@Hurlburt.af.mil

DR. DWIGHT WILSON POLK is technical advisor andchief of the Test Office, 505th Command and ControlWing, Hurlburt Field, Florida. Dr. Polk is the wing’stechnical authority on combat air forces command andcontrol T&E. He also provides technical integration forthe wing with the USAF Warfare Center and Air CombatCommand. Dr. Polk earned bachelor of science and masterof science degrees in mathematics, and a doctor ofphilosophy degree in engineering management fromClemson University, Clemson, South Carolina. He hasbeen associated with T&E in some form during 34 of his40 years of combined active duty and civil serviceexperience. E-mail: Dwight.Polk@Hurlburt.af.mil

Endnotes1Suh, Theresa L., Maj., USAF. ‘‘Combining Test and Training.’’

Briefing received via e-mail from Michael Tamburo, Major, USAF to

(M.W.K.) 7 January 2008.

ReferencesAlberts, David S., John J. Garstka, and Frederick P.

Stein. 1999. NetworkCentric warfare: Developing and

leveraging information superiority. Washington, D.C.:DOD C4ISR Cooperative Research Program.

Arquilla, John, and David Ronfeldt. 1997. ‘‘Lookingahead: Preparing for information-age conflict.’’ In InAthena’s camp: Preparing for conflict in the informationage, ed. John Arquilla and David Ronfeldt. SantaMonica, CA: Rand.

Arquilla, John, and David Ronfeldt. 2001. ‘‘Theadvent of netwar (revisited).’’ In Networks and netwars,ed. John Arquilla and David Ronfeldt, 1–25. Wash-ington, D.C.: Rand.

Benson, Lawrence R. 1992. ‘‘History of Air ForceOperational Test and Evaluation (OT&E) mission,organization, and policy.’’ December 1992. Kirtland,AFB, NM: Air Force Operational Test and EvaluationCenter.

Castells, Manuel. 2000. The rise of the networksociety. Malden, MA: Blackwell.

Cebrowsky, Arthur. 2002. Vice Adm. (retired), U.S.Navy. Quoted in Army Times, 25 November 2002.

Chandler, Alfred T. Jr. 1977. The visible hand: Themanagerial revolution in American business. Cambridge,MA: Harvard University Press.

Clarke, Rich. 2007. ‘‘JITC NR-KPP Testing andCertification Presentation to ITEA.’’ Briefing, April2007.

DiFronzo, Vincent P. 2006. ‘‘Memorandum forUSAF WC/CG, ACC/A3, and ACC/A8. Subject:Do Not Field Memo for the TACP-CASS (v) 1.2.5.’’February 23, 2006.

DOD. 2003. Operation of the defense acquisitionsystem. Department of Defense Instruction Number5000.2, 12 May 2003. Washington, D.C.: U.S.Department of Defense, http://www.dtic.mil/whs/directives/corres/pdf/500002p.pdf (accessed December14, 2010).

DOD. 2007. Interoperability and supportability ofInformation Technology (IT) and National SecuritySystems (NSS). Assistant Secretary of Defense (Net-works and Information Integration). Department ofDefense Directive (DODD) 4630.05, 23 April 2007.Washington, D.C.: U.S. Department of Defense.http://www.dtic.mil/whs/directives/corres/pdf/463005p.pdf. (accessed December 14, 2010).

DOD AT&L. 2004. Under Secretary of DefenseAcquisition, Technology, and Logistics (USD AT&L)and United States Joint Forces Command (USJF-COM). ‘‘Joint Battle Management Command andControl (JBMC2) roadmap, Version 1.0.’’ 22 April2004. Washington, D.C.: U.S. Department of Defense.

DOD DOT&E. 2004. Director of Operational Testand Evaluation (DOT&E). ‘‘Testing in a JointEnvironment Roadmap: Strategic Planning Guidance,Fiscal Years 2006–2011.’’ Final Report. 12 November

SoS Operational Testing

32(1) N March 2011 49

2004. Washington, D.C.: U.S. Department of De-fense.

DoD DOT&E. 2010. Director of Operational Testand Evaluation (DOT&E). ‘‘Memorandum for Prin-cipal Deputy Under Secretary of Defense (Acquisition,Technology, and Logistics). Subject: State of Reliabil-ity.’’ June 30, 2010.

Ferguson, C., L. Zimmermann, and F. Barone.2007. ‘‘Joint Mission Environment Test Capability(JMETC) briefing for the Joint Strike Fighter PMO,’’17 April 2007. Washington, D.C.: Test ResourceManagement Center (TRMC).

Ferguson, Thomas S. 1967. Mathematicalstatistics: A decision theoretic approach. New York:Academic Press.

46th Test Squadron. 2007. 46th Test SquadronData Link Test Facility. ‘‘System of Systems ActivityReport for FY 07.’’ Draft. Unpublished. 7 December2007.

JFIIT. 2007. Joint Fires Interoperability and Inte-gration Team (JFIIT). ‘‘Coalition Combat Identifica-tion (CCID) Advanced Concept Technology Dem-onstration (ACTD) Bold Quest Close Air Support(CAS) assessment report.’’ TR-07-06. October 2007.Elgin AFB, FL: JFIIT.

JITC. 2008. Joint C4ISR Interoperability Test &Evaluation (T&E) Capability (InterTEC) Web site.http://jitc.fu.disa.mil/intertec/index.html (accessedJanuary 18, 2008).

JSSEO. 2006. Joint Single Integrated Air PictureSystem Engineering Organization (JSSEO). ‘‘SingleIntegrated Air Picture (SIAP) Test and EvaluationStrategy (TES).’’ Technical Report 2006-018. 3 May2006. Arlington, VA: JSSEO.

Kometer, Michael W., Lt. Col., USAF. 2007.Command in air war: Centralized vs. decentralizedcontrol of combat airpower. Maxwell Air Force Base,Alabama: Air University Press.

Krieg, Kenneth J., David S. Chu, and Charles E.McQueary. 2006. Memorandum for Service Secretar-ies. Subject: Test and Training InterdependencyInitiative. 7 September 2006. Washington, D.C.:DoD.

Porter, M., and V. Millar. 1985. How informationgives you competitive advantage: The informationrevolution is transforming the nature of competition.Harvard Business Review 63 (4), 149–160.

TENA. 2008. Test and Training Enabling Archi-tecture (TENA) Fact Sheet. https://www.tena-sda.org/download/attachments/6750/TENAFactSheet-IF07-2007-10-16.pdf?version51 (accessed 18 January2008).

TRMC. 2006. Test Resource Management Center(TRMC), Central Test and Evaluation Investment

Program (CTEIP). 2006 Annual Report: ImprovingTest and Evaluation Capabilities. http://www.acq.osd.mil/trmc/jipp/cteip/ (accessed December 14, 2010).

U.S. Air Force. 1995. AFMAN 99-112. Electronicwarfare test and evaluation process—direction andmethodology for EW testing. Air Force Manual. 27March 1995. Washington, D.C.: Office of theSecretary of the Air Force.

U.S. Air Force. 2000. Chief of Staff. Globalvigilance, reach, and power: America’s Air Force vision2020. Washington, D.C.: HQ USAF.

U.S. Air Force. 2004a. Capabilities based test andevaluation. Air Force Instruction (AFI) 99-103. 6August 2004. Washington, D.C.: U.S. Department ofDefense, http://www.af.mil/shared/media/epubs/AFI99-103.pdf (accessed December 14, 2010).

U.S. Air Force. 2004b. Operation of the capabilities-based acquisition system. Interim Guidance 63-101.Washington, D.C.: Office of the Assistant Secretary ofthe Air Force.

USJFCOM. 2010. United States Joint ForcesCommand (USJFCOM). Joint Fires Interoperabilityand Integration Team (JFIIT) Web page. https://jfiit.eglin.af.mil/public/index.php (accessed December 14,2010).

Wiegand, Ronald C., Col., USAF. 2007.‘‘USAFWC Capability for Net-centric Test andTraining.’’ Briefing to Test Resource ManagementCenter. Washington, D.C., 19 December 2007.

Williams Phil. 2001. ‘‘Transnational Criminal Net-works.’’ In Networks and Netwars, ed. John Arquilla andDavid Ronfeldt, 61–97. Washington, D.C.: Rand.

AcknowledgmentsThe authors wish to thank Mr. Keith Poch

(JMETC User Support Technical Team), Mr. Joe R.Siebenberg (605 TES), Mr. Robert M. Summitt(JFCOM JFIIT/J3), and Maj. Michael ‘‘Ponch’’Tamburo (JDICE) for their collaboration in research-ing current efforts to develop SoS testing capabilities.Thanks also to Mr. Paul E. Witt, Air Force SpecialOperations Command/History Office (AFSOC/HO)for his help in researching the history of operationaltesting and to M.Sgt. Michelle A. Copenhaver (505CCW/TA) for her editorial and proofreading support

Appendix – Decision theory and testingA good text on the underlying mathematical theory

is that of Thomas S. Ferguson, Mathematical Statistics,A Decision Theoretic Approach (Ferguson 1967).

The following are basic definitions:

N H 5 {h: h is a is a state of nature}N A 5 {a: a is an available action}

Kometer, Burkhart, McKee, & Polk

50 ITEA Journal

N X is a random variable representing an outcomeof the statistical experiment and 1 is a realizationof X.

N S 5 {s:s is a possible outcome of the statisticalexperiment}

N d(X) is a nonrandomized decision rule mapping Sinto A.

N L(h,d(X)) is the loss function for the productspace HxA.

N R(h,d) 5 E [L(h,d(X))] is the risk function,which is assumed to exist and be finite for all h eH, and the expected value is taken over X.

N R(h,d) 5 E [L(h,d(X))] 5 # L(h, d(x))dPh(x)

For traditional system-level operational testing, thestates of nature could be considered dichotomous: thesystem meets requirements and performs satisfactorily(it is effective and suitable), or it does not. Analternative might be a determination that the systemwould contribute favorably to warfighting capability(with possible qualifications) versus one in which itsimpact on warfighting capability would not befavorable. The actual observed outcome of the test isan entry in a multi-dimensional matrix S representingthe level of attainment of objectives identified by thetest team. The random variable X is a function thatmaps S into the real line. A decision rule is devised(typically subjectively) to map X into an action(recommendation) vector A, say A 5 [a1, a2, a3, a4],where

N a1 is a recommendation to field the system as is,N a2 is a recommendation to field the system after

identified deficiencies are corrected,N a3 is a recommendation not to field the system

but to continue development, andN a4 is a recommendation not to field the system

and to cease development.

The loss function represents the consequences of theordered pair: the state of nature and the action takenbased on the decision rule and observed value for X. Therisk is the statistical expected value of the loss functionin the form of a Lebesgue integral. For the purposes ofthis paper, the reader can consider Ph(x) to be thecumulative distribution function for the random variableX when the state of nature is h. A simplified examplefollows, where

N h1 5 the state that the system essentially meetsrequirements;

N h2 5 the state that the system is significantlydeficient;

N x1 5 1 (the system is observed to meet essentialrequirements during testing);

N x2 5 0 (the system is observed to have significantdeficiencies during testing);

N a1 5 action to field the system;N a2 5 action to withhold fielding the system;

d1 xð Þ : x1?a1 d2 xð Þ : x1?a1

x2?a2 x2?a1

d3 xð Þ : x1?a2 d4 xð Þ : x1?a2

x2?a1 x2?a2

N p1(x1) 5 P[X 5 x1 | h 5 h1];N p1(x2) 5 P[X 5 x2 | h 5 h1];N p2(x1) 5 P[X 5 x1 | h 5 h2]; andN p2(x2) 5 P[X 5 x2 | h 5 h2].

Assume that

p1 x1ð Þ~2=3 and p1 x2ð Þ~1=3 when h~h1;

p2 x1ð Þ~1=4 and p2 x2ð Þ~3=4 when h~h2

L h1, a1ð Þ~{600, L h1, a2ð Þ~720, L h2, a1ð Þ~900,

L h2, a2ð Þ~{300

(Note that L(h1, a2) is the opportunity loss corre-sponding to a Type I Error in a statistical test ofhypothesis, and L(h2, a1) is the loss associated with aType II Error in a statistical test of hypothesis.) Thetester is assumed to make recommendations basedsolely on the decision rule d, not on exogenousconsiderations. Choose d(x) 5 d1(x).

Then,

R h1, d1ð Þ~E L h1, d1 Xð Þð Þ½ �

~L h1,a1ð Þp1 x1ð ÞzL h1,a2ð Þp1 x2ð Þ

~{160

R h2, d1ð Þ~E L h2, d1 Xð Þð Þ½ �

~L h2,a1ð Þp2 x1ð ÞzL h2,a2ð Þp2 x2ð Þ

~0

Determination of R(h,d) for d2(x), d3(x), and d4(x) isstraightforward. Based on a complete evaluation ofR(h,d), the investigator can develop an appropriatestrategy for the decision problem.

SoS Operational Testing

32(1) N March 2011 51