Harnessing Innovative Procedures Under an Administration IN …ufdcimages.uflib.ufl.edu/AA/00/06/26/26/00028/04-2017.pdf · 2018. 5. 15. · Defense Acquisition Research Journal A

Defense Acquisition Research Journal A Publication of the Defense Acquisition University

Harnessing Innovative Procedures Under

an Administration IN TRANSITION

2017 Edward Hirsch Acquisition and Writing Award

Presented on behalf of DAU by

April 2017 Vol 24 No 2 | ISSUE 81

Using Analytical Hierarchy and Analytical Network Processes to Create Cyber Security Metrics George C Wilamowski Jason R Dever and Steven M F Stuban

The Threat Detection System That Cried Wolf Reconciling Developers with Operators Shelley M Cazares

Army Aviation Quantifying the Peacetime and Wartime Maintenance Man-Hour Gaps LTC William Bland USA (Ret) CW5 Donald L Washabaugh Jr USA (Ret) and Mel Adams

Complex Acquisition Requirements Analysis Using a Systems Engineering Approach Col Richard M Stuckey USAF (Ret) Shahram Sarkani and Thomas A Mazzuchi

An Investigation of Nonparametric Data Mining Techniques for Acquisition Cost Estimating Capt Gregory E Brown USAF and Edward D White

Online-only Article Critical Success Factors for Crowdsourcing with Virtual Environments to Unlock Innovation Glenn E Romanczuk Christopher Willy and John E Bischoff

The Defense Acquisition Professional Reading List Getting Defense Acquisition Right Written and Introduced by the Honorable Frank Kendall

Article List ARJ Extra

Research Advisory BoardDr Mary C Redshaw

Dwight D Eisenhower School for National Security and Resource Strategy

Editorial BoardDr Larrie D Ferreiro

Chairman and Executive Editor

Mr Richard AltieriDwight D Eisenhower School for NationalSecurity and Resource Strategy

Dr Michelle BaileyDefense Acquisition University

Dr Don Birchler Center for Naval Analyses Corporation

Mr Kevin Buck The MITRE Corporation

Mr John Cannaday Defense Acquisition University

Dr John M Colombi Air Force Institute of Technology

Dr Richard DonnellyThe George Washington University

Dr William T EliasonDwight D Eisenhower School for NationalSecurity and Resource Strategy

Dr J Ronald Fox Harvard Business School

Mr David Gallop Defense Acquisition University

Dr Jacques Gansler University of Maryland

RADM James Greene USN (Ret)Naval Postgraduate School

Dr Mike KotzianDefense Acquisition University

Dr Craig LushDefense Acquisition University

Dr Troy J MuellerThe MITRE Corporation

Dr Andre Murphy Defense Acquisition University

Dr Christopher G PerninRAND Corporation

Dr Richard ShipeDwight D Eisenhower School for NationalSecurity and Resource Strategy

Dr Keith SniderNaval Postgraduate School

Dr John SnoderlyDefense Acquisition University

Ms Dana Stewart Defense Acquisition University

Dr David M TateInstitute for Defense Analyses

Dr Trevor TaylorCranfield University (UK)

Mr Jerry VandewieleDefense Acquisition University

Mr James A MacStravicPerforming the duties of Under Secretary of Defense for Acquisition Technology and Logistics

Mr James P WoolseyPresident Defense Acquisition University

ISSN 2156-8391 (print) ISSN 2156-8405 (online)DOI httpsdoiorg1022594dau042017-812402

The Defense Acquisition Research Journal formerly the Defense Acquisition Review Journal is published quarterly by the Defense Acquisition University (DAU) Press and is an official publication of the Department of Defense Postage is paid at the US Postal facility Fort Belvoir VA and at additional US Postal facilities Postmaster send address changes to Editor Defense Acquisition Research Journal DAU Press 9820 Belvoir Road Suite 3 Fort Belvoir VA 22060-5565 The journal-level DOI is httpsdoiorg1022594dauARJissn2156-8391 Some photos appearing in this publication may be digitally enhanced

Articles represent the views of the authors and do not necessarily reflect the opinion of DAU or the Department of Defense

Director Visual Arts amp Press Randy Weekes

Managing Editor Deputy Director

Visual Arts amp PressNorene L Taylor

Assistant Editor Emily Beliles

Production ManagerVisual Arts amp Press Frances Battle

Lead Graphic Designer Diane FleischerTia GrayMichael Krukowski

Graphic Designer Digital Publications Nina Austin

Technical Editor Collie J Johnson

Associate Editor Michael Shoemaker

Copy EditorCirculation Manager Debbie Gonzalez

Multimedia Assistant Noelia Gamboa

Editing Design and Layout The C3 Group ampSchatz Publishing Group

CONTENTS | Featured Research

A Publication of the Defense Acquisition University April 2017 Vol 24 No 2 ISSUE 81

EARCH PAPER COMPETITIO

N2016 ACS1st

N UNIVERSITY ALUM

p 186 Using Analytical Hierarchy and Analytical Network Processes to Create Cyber Security Metrics George C Wilamowski Jason R Dever and Steven M F Stuban

This article discusses cyber security controls anda use case that involves decision theory methods to produce a model and independent first-order results using a form-fit-function approach as a generalized application benchmarking framework The frameshywork combines subjective judgments that are based on a survey of 502 cyber security respondents with quantitative data and identifies key performancedrivers in the selection of specific criteria for three communities of interest local area network wide area network and remote users

p 222 The Threat Detection System That Cried Wolf Reconciling Developers with Operators Shelley M Cazares

Threat detection systems that perform well intesting can ldquocry wolfrdquo during operation generating many false alarms The author posits that program managers can still use these systems as part of atiered system that overall exhibits better perforshymance than each individual system alone

Featured Research

p 246 Army Aviation Quantifying the Peacetime and Wartime Maintenance Man-Hour Gaps LTC William Bland USA (Ret) CW5 Donald L Washabaugh Jr USA (Ret) and Mel Adams

T he M a i nt en a nc e M a n-Hou r ( M M H ) G a pCa lcu lator conf irms a nd qua ntif ies a la rge persistent gap in Army aviation maintenancerequired to support each Combat Aviation Brigade

p 266 Complex Acquisition Requireshyments Analysis Using a Systems Engineering Approach Col Richard M Stuckey USAF (Ret) Shahram Sarkani and Thomas A Mazzuchi

Programs lack an optimized solution set of requireshyments attributes This research provides a set ofvalidated requirements attributes for ultimateprogram execution success

p 302An Investigation of Nonpara-metric Data Mining Techniques for Acquisition Cost EstimatingCapt Gregory E Brown USAF and Edward D White

Given the recent enhancements in acquisition data collection a meta-analysis reveals that nonpara-metric data mining techniques may improve the accuracy of future DoD cost estimates

Critical Success Factors for Crowdsourcing with Virtual Environments to Unlock Innovation Glenn E Romanczuk Christopher Willy and John E Bischoff

Delphi methods were used to discover critical success factors in five areas virtual environments MBSE crowdsourcing human systems integrashytion and the overall process Results derived from this study present a framework for using virtualenvironments to crowdsource systems design usingwarfighters and the greater engineering staff

httpwwwdaumillibraryarj

Featured Research

p viii From the Chairman and Executive Editor

p xii Research Agenda 2017ndash2018

p xvii DAU Alumni Association

p 368 Professional Reading List

Getting Defense Acquisition Right Written and Introduced by the Honorable Frank Kendall

p 370 New Research in Defense Acquisition

A selection of new research curated by the DAU Research Center and the Knowledge Repository

p 376 Defense ARJ Guidelines for Contributors

The Defense Acquisition Research Journal (ARJ) is a scholarly peer-reviewed journal published by theDefense Acquisition University All submissions receive a blind review to ensure impartial evaluation

p 381 Call for Authors

We are currently soliciting articles and subject matter experts for the 2017ndash2018 Defense ARJ print years

p 384 Defense Acquisition University Website

Your online access to acquisition research consulting information and course offerings

FROM THE CHAIRMAN AND

EXECUTIVE EDITOR

Dr Larrie D Ferreiro

A Publication of the Defense Acquisition University httpwwwdaumil

The theme for this edition of Defense A c q u i s i t i o n R e s e a r c h J o u r n a l i s ldquoHarnessing Innovative Procedures under an Administration in Transitionrdquo Fiscal Year 2017 will see many changes not only in a new administration but also under the National Defense Authorization Act (NDAA) Under this NDAA by February 2018 the Under Secretary of Defense for Acquisition Technology and Logistics (USD[ATampL]) office will be disestabshy

lished and its duties divided between two separate offices The first office the Under Secretary of Defense for Research and Engineering (USD[RampE]) will carry out the mission of defense technological innovation The second office the Under Secretary of Defense for Acquisition and Sustainment (USD[AampS]) will ensure that susshytainment issues are integrated during the acquisition process The articles in this issue show some of the innovative ways that acquishysition can be tailored to these new paradigms

The first article is ldquoUsing Analytical Hierarchy and Analytical Network Processes to Create Cyber Security Metricsrdquo by George C Wilamowski Jason R Dever and Steven M F Stuban It was the recipient (from among strong competition) of the DAU Alumni Association (DAUAA) 2017 Edward Hirsch Acquisition and Writing Award given annually for research papers that best meet the criteria of significance impact and readability The authors discuss cyber

April 2017

security controls and a use case involving decision theory to develop a benchmarking framework that identifies key performance drivers in local area network wide area network and remote user communities Next the updated and corrected article by Shelley M Cazares ldquoThe Threat Detection System That Cried Wolf Reconciling Developers with Operatorsrdquo points out that some threat detection systems that perform well in testing can generate many false alarms (ldquocry wolfrdquo) in operation One way to mitigate this problem may be to use these systems as part of a tiered system that overall exhibits better pershyformance than each individual system alone The next article ldquoArmy Aviation Quantifying the Peacetime and Wartime Maintenance Man-Hour Gapsrdquo by William Bland Donald L Washabaugh Jr and Mel Adams describes the development of a Maintenance Man-Hour Gap Calculator tool that confirmed and quantified a large persistent gap in Army aviation maintenance Following this is ldquoComplex Acquisition Requirements Analysis Using a Systems Engineering Approachrdquo by Richard M Stuckey Shahram Sarkani and Thomas A Mazzuchi The authors examine prioritized requireshyment attributes to account for program complexities and provide a guide to establishing effective requirements needed for informed trade-off decisions The results indicate that the key attribute for unconstrained systems is achievable Then Gregory E Brown and Edward D White in their article ldquoAn Investigation of Nonparametric Data Mining Techniques for Acquisition Cost Estimatingrdquo use a meta-analysis to argue that nonparametric data mining techniques may improve the accuracy of future DoD cost estimates

The online-only article ldquoCritical Success Factors for Crowdsourcing with Virtual Environments to Unlock Innovationrdquo by Glenn E Romanczuk Christopher Willy and John E Bischoff explains how to use virtual environments to crowdsource systems design using warfighters and the engineering staff to decrease the cycle time required to produce advanced innovative systems tailored to meet warfighter needs

This issue inaugurates a new addition to the Defense Acquisition Research Journal ldquoNew Research in Defense Acquisitionrdquo Here we bring to the attention of the defense acquisition community a selection of current research that may prove of further interest These selections are curated by the DAU Research Center and the Knowledge Repository and in these pages we provide the summaries and links that will allow interested readers to access the full works

The featured book in this issuersquos Defense Acquisition Professional Reading List is Getting Defense Acquisition Right by former Under Secretary of Defense for Acquisition Technology and Logistics Frank Kendall

Finally the entire production and publishing staff of the Defense ARJ now bids a fond farewell to Diane Fleischer who has been our Graphic SpecialistLead Designer for this journal since our January 2012 Issue 61 Vol 19 No 1 She has also been with the DAU Press for more than 5 years and has been instrumental in the Defense ARJ team winning two APEX awards for One-of-a-Kind Publicationsmdash Government in both 2015 and 2016 Diane is retiring and she and her family are relocating to Greenville South Carolina Diane we all wish you ldquofair winds and following seasrdquo

Biography

Ms Diane Fleischer has been employed as a Visual Information Specialist in graphic design at the Defense Acquisition University (DAU) since November 2011 Prior to her arrival at DAU as a contractor with the Schatz Publishing Group she worked in a wide variety of commercial graphic positions both print and web-based Dianersquos graphic arts experience spans more than 38 years and she holds a BA in Fine Arts from Asbury University in Wilmore Kentucky

This Research Agenda is intended to make researchers aware of the topics that are or should be of particular concern to the broader defense acquisition community within the federal government academia and defense industrial sectors The center compiles the agenda annually using inputs from subject matter experts across those sectors Topics are periodically vetted and updated by the DAU Centerrsquos Research Advisory Board to ensure they address current areas of strategic interest

The purpose of conducting research in these areas is to provide solid empirically based findings to create a broad body of knowl-edge that can inform the development of policies procedures and processes in defense acquisition and to help shape the thought lead-ership for the acquisition community Most of these research topics were selected to support the DoDrsquos Better Buying Power Initiative (see httpbbpdaumil) Some questions may cross topics and thus appear in multiple research areas

Potential researchers are encouraged to contact the DAU Director of Research (researchdaumil) to suggest additional research questions and topics They are also encouraged to contact the listed Points of Contact (POC) who may be able to provide general guidance as to current areas of interest potential sources of infor-mation etc

DAU CENTER FOR DEFENSE ACQUISITION

RESEARCH AGENDA 2017ndash2018

Competition POCs bull John Cannaday DAU johncannadaydaumil

bull Salvatore Cianci DAU salvatoreciancidaumil

bull Frank Kenlon (global market outreach) DAU frankkenlondaumil

Measuring the Effects of Competition bull What means are there (or can be developed) to measure

the effect on defense acquisition costs of maintaining the defense industrial base in various sectors

bull What means are there (or can be developed) of mea-suring the effect of utilizing defense industria l infrastructure for commercial manufacture and in particular in growth industries In other words can we measure the effect of using defense manufacturing to expand the buyer base

bull What means are there (or can be developed) to deter-mine the degree of openness that exists in competitive awards

bull What are the different effects of the two best value source selection processes (trade-off vs lowest price technically acceptable) on program cost schedule and performance

Strategic Competitionbull Is there evidence that competition between system

portfolios is an effective means of controlling price and costs

bull Does lack of competition automatically mean higher prices For example is there evidence that sole source can result in lower overall administrative costs at both the government and industry levels to the effect of lowering total costs

bull What are the long-term historical trends for compe-tition guidance and practice in defense acquisition policies and practices

April 2017

bull To what extent are contracts being awarded non-competitively by congressional mandate for policy interest reasons What is the effect on contract price and performance

bull What means are there (or can be developed) to deter-mine the degree to which competitive program costs are negatively affected by laws and regulations such as the Berry Amendment Buy American Act etc

bull The DoD should have enormous buying power and the ability to influence supplier prices Is this the case Examine the potential change in cost performance due to greater centralization of buying organizations or strategies

Effects of Industrial Base bull What are the effects on program cost schedule and

performance of having more or fewer competitors What measures are there to determine these effects

bull What means are there (or can be developed) to measure the breadth and depth of the industrial base in various sectors that go beyond simple head-count of providers

bull Has change in the defense industrial base resulted in actual change in output How is that measured

Competitive Contracting bull Commercial industry often cultivates long-term exclu-

sive (noncompetitive) supply chain relationships Does this model have any application to defense acquisition Under what conditionscircumstances

bull What is the effect on program cost schedule and performance of awards based on varying levels of competition (a) ldquoEffectiverdquo competition (two or more offers) (b) ldquoIneffectiverdquo competition (only one offer received in response to competitive solicitation) (c) split awards vs winner take all and (d) sole source

Improve DoD Outreach for Technology and Products from Global Markets

bull How have militaries in the past benefited from global technology development

bull Howwhy have militaries missed the largest techno-logical advances

bull What are the key areas that require the DoDrsquos focus and attention in the coming years to maintain or enhance the technological advantage of its weapon systems and equipment

bull What types of efforts should the DoD consider pursu-ing to increase the breadth and depth of technology push efforts in DoD acquisition programs

bull How effectively are the DoDrsquos global science and tech-nology investments transitioned into DoD acquisition programs

bull Are the DoDrsquos applied research and development (ie acquisition program) investments effectively pursuing and using sources of global technology to affordably meet current and future DoD acquisition program requirements If not what steps could the DoD take to improve its performance in these two areas

bull What are the strengths and weaknesses of the DoDrsquos global defense technology investment approach as compared to the approaches used by other nations

bull What are the strengths and weaknesses of the DoDrsquos global defense technology investment approach as compared to the approaches used by the private sectormdashboth domestic and foreign entities (compa-nies universities private-public partnerships think tanks etc)

bull How does the DoD currently assess the relative benefits and risks associated with global versus US sourcing of key technologies used in DoD acquisition programs How could the DoD improve its policies and procedures in this area to enhance the benefits of global technology sourcing while minimizing potential risks

April 2017

bull How could current DoDUS Technology Security and Foreign Disclosure (TSFD) decision-making policies and processes be improved to help the DoD better bal-ance the benefits and risks associated with potential global sourcing of key technologies used in current and future DoD acquisition programs

bull How do DoD primes and key subcontractors currently assess the relative benefits and risks associated with global versus US sourcing of key technologies used in DoD acquisition programs How could they improve their contractor policies and procedures in this area to enhance the benefits of global technology sourcing while minimizing potential risks

bull How could current US Export Control System deci-sion-making policies and processes be improved to help the DoD better balance the benefits and risks associated with potential global sourcing of key tech-nologies used in current and future DoD acquisition programs

Comparative Studies bull Compare the industrial policies of military acquisition

in different nations and the policy impacts on acquisi-tion outcomes

bull Compare the cost and contract performance of highly regulated public utilities with nonregulated ldquonatu-ral monopoliesrdquo eg military satellites warship building etc

bull Compare contractingcompetition practices between the DoD and complex custom-built commercial prod-ucts (eg offshore oil platforms)

bull Compare program cost performance in various market sectors highly competitive (multiple offerors) limited (two or three offerors) monopoly

bull Compare the cost and contract performance of mil-itary acquisition programs in nations having single ldquopurplerdquo acquisition organizations with those having Service-level acquisition agencies

DAU ALUMNI ASSOCIATION Join the Success Network

The DAU Alumni Association opens the door to a worldwide network of Defense Acquisition University graduates faculty staff members and defense industry representativesmdashall ready to share their expertise with you and benefit from yours Be part of a two-way exchange of information with other acquisition professionals

bull Stay connected to DAU and link to other professional organizations bull Keep up to date on evolving defense acquisition policies and developments

through DAUAA newsletters and the DAUAA LinkedIn Group bull Attend the DAU Annual Acquisition Training Symposium and bi-monthly hot

topic training forumsmdashboth supported by the DAUAA and earn Continuous Learning Points toward DoD continuing education requirements

Membership is open to all DAU graduates faculty staff and defense industrymembers Itrsquos easy to join right from the DAUAA Website at wwwdauaaorg or scan the following QR code

For more information call 703-960-6802 or 800-755-8805 or e-mail dauaa2aolcom

ISSUE 81 APRIL 2017 VOL 24 NO 2

Wersquore on the Web at httpwwwdaumillibraryarj 185185

Image designed by Diane Fleischer

PAPER COMPETITION

2016 ACS 1st

NUNIVERSITY ALU

I ASSO

CIATIO

Using Analytical Hierarchy and Analytical

Network Processes to Create CYBER SECURITY METRICS

George C Wilamowski Jason R Dever and Steven M F Stuban

Authentication authorization and accounting are key access control measures that decision makers should consider when crafting a defense against cyber attacks Two decision theory methodologies were compared Analytical hierarchy and analytical network processes were applied to cyber security-related decisions to derive a measure of effectiveness for risk eval uation A networkaccess mobile security use case was employed to develop a generalized application benchmarking framework Three communities of interest which include local area network wide area network and remote users were referenced while demonstrating how to prioritize alternatives within weighted rankings Subjective judgments carry tremendous weight in the minds of cyber security decision makers An approach that combines these judgments with quantitative data is the key to creating effective defen sive strategies

DOI httpsdoiorg1022594dau16-7602402 Keywords Analytical Hierarchy Process (AHP) Analytical Network Process (ANP) Measure of Effectiveness (MOE) Benchmarking Multi Criteria Decision Making (MCDM)

188 Defense ARJ April 2017 Vol 24 No 2 186ndash221

Cyber Security Metrics httpwwwdaumil

Authentication authorization and accounting (AAA) are the last lines of defense among access controls in a defense strategy for safeguarding the privacy of information via security controls and risk exposure (EY 2014) These controls contribute to the effectiveness of a data networkrsquos system security The risk exposure is predicated by the number of preventative meashysures the Trusted Information Provider or ldquoTIPrdquomdashan agnostic term for the

organization that is responsible for privacy and security of an orgashynizationmdashis willing to apply against cyber attacks (National

Institute of Standards and Technology [NIST] 2014) Recently persistent cyber attacks against the data

of a given organization have caused multiple data breaches within commercial industries and the

US Government Multiple commercial data networks were breached or compromised in

2014 For example 76 million households and 7 million small businesses and other commercial businesses had their data comshypromised at JPMorgan Chase amp Co Home

Depot had 56 million customer accounts compromised TJ Ma xx had 456

million customer accounts comproshymised and Target had 40 million customer accounts compromised (Weise 2014) A recent example of a commercial cyber attack was the attack against Anthem Inc

from January to February 2015 when a sophisticated external attack compromised the data of approximately 80 million customers and employees (McGuire 2015)

C on s e q u e n t l y v a r i o u s effor ts have been made

to combat these increasshyingly common attacks For example on February 13 2015 at a Summit

on Cybersecurity and Consumer Protection

at Stanford University in

April 2017

Palo Alto California the President of the United States signed an executive order that would enable private firms to share information and access classhysified information on cyber attacks (Obama 2015 Superville amp Mendoza 2015) The increasing number of cyber attacks that is currently experienced by many private firms is exacerbated by poorly implemented AAA security controls and risk exposure minimization These firms do not have a method for measuring the effectiveness of their AAA policies and protocols (EY 2014) Thus a systematic process for measuring the effectiveness of defenshysive strategies in critical cyber systems is urgently needed

Literature Review A literature review has revealed a wide range of Multi-Criteria Decision

Making (MCDM) models for evaluating a set of alternatives against a set of criteria using mathematical methods These mathematical methods include linear programming integer programming design of experiments influence diagrams and Bayesian networks which are used in formulating the MCDM decision tools (Kossiakoff Sweet Seymour amp Biemer 2011) The decision tools include Multi-Attribute Utility Theory (MAUT) (Bedford amp Cooke 1999 Keeney 1976 1982) criteria for deriving scores for alternatives decishysion trees (Bahnsen Aouada amp Ottersten 2015 Kurematsu amp Fujita 2013 Pachghare amp Kulkarni 2011) decisions based on graphical networks and Cost-Benefit Analysis (CBA) (Maisey 2014 Wei Frinke Carter amp Ritter 2001) simulations for calculating a systemrsquos alternatives per unit cost and the House of Quality Quality Function Deployment (QFD) (Chan amp Wu 2002 Zheng amp Pulli 2005) which is a planning matrix that relates what a customer wants to how a firm (that produces the products) is going to satisfy those needs (Kossiakoff et al 2011)

The discussion on the usability of decision theory against cyber threats is limited which indicates the existence of a gap This study will employ analytical hierarchies and analytical network processes to create AAA cyber security metrics within these well-known MCDM models (Rabbani amp Rabbani 1996 Saaty 1977 2001 2006 2009 2010 2012 Saaty amp Alexander 1989 Saaty amp Forman 1992 Saaty Kearns amp Vargas 1991 Saaty amp Peniwati 2012) for cyber security decision-making Table 1 represents a networkaccess mobile security use case that employs mathematically based techniques of criteria and alternative pairwise comparisons

TABLE 1 CYBER SECURITY DECISION MAKING USE CASE

Primary Actor Cyber Security Manager

Scope Maximize Network AccessMobilityrsquos Measure of Effectiveness

Level Cyber Security Control Decisions

Stakeholder Security RespondentsmdashOrganizationrsquos Security Decision and Interests Influencers

C-suitemdashResource Allocation by Senior Executives

Precondition Existing Authentication Authorization and Accounting (AAA) Limited to Security Controls Being Evaluated

Main Success Scenario

1 AAA Goal Setting 2 Decision Theory Model 3 AAA Security InterfacesRelationships Design 4 AB Survey Questionnaire with 9-Point Likert scale 5 Survey Analysis 6 Surveyrsquos AB Judgement Dominance 7 Scorecard Pairwise Data Input Into Decision Theory

Software 8 DecisionmdashPriorities and Weighted Rankings

Extensions 1a Goals into Clusters Criteria Subcriteria and Alternatives

3a Selection of AAA Attribute Interfaces 3b Definition of Attribute Interfaces 4a 9-Point Likert Scale Equal Importance (1) to Extreme

Importance (9) 5a Surveyrsquos Margin of Error 5b Empirical Analysis 5c Normality Testing 5d General Linear Model (GLM) Testing 5e Anderson-Darling Testing 5f Cronbach Alpha Survey Testing for Internal

Consistency 6a Dominate Geometric Mean Selection 6b Dominate Geometric Mean used for Scorecard Build

Out 7a Data Inconsistencies Check between 010 and 020 7b Cluster Priority Ranking

April 2017

Research The objective of this research was to demonstrate a method for assessing

measures of effectiveness by means of two decision theory methodologies the selected MCDM methods were an Analytical Hierarchy Process (AHP) and an Analytical Network Process (ANP) Both models employ numerical scales within a prioritization method that is based on eigenvectors These methods were applied to cyber security-related decisions to derive a meashysure of effectiveness for risk evaluation A networkaccess mobile security use case as shown in Table 1 was employed to develop a generalized applicashytion benchmarking framework to evaluate cyber security control decisions The security controls are based on the criteria of AAA (NIST 2014)

The Defense Acquisition System initiates a Capabilities Based Assessment (CBA) to be performed upon which an Initial Capabilities Document (ICD) is built (AcqNotes 2016a) Part of creating an ICD is to define a functional area (or areasrsquo) Measure of Effectiveness (MOE) (Department of Defense [DoD] 2004 p 30) MOEs are a direct output from a Functional Area Assessment (AcqNotes 2016a) The MOE for Cyber Security Controls would be an area that needs to be assessed for acquisition The term MOE was initially used by Morse and Kimball (1946) in their studies for the US Navy on the effecshytiveness of weapons systems (Operations Evaluation Group [OEG] Report 58) There has been a plethora of attempts to define MOE as shown in Table 2 In this study we adhere to the following definition of MOEs

MOEs are measures of mission success stated under specific environmental and operating conditions from the usersrsquo viewpoint They relate to the overall operational success criteria (eg mission performance safety availability and security)hellip (MITRE 2014 Saaty Kearns amp Vargas 1991 pp 14ndash21)

[by] a qualitative or quantitative metric of a systemrsquos overall performance that indicates the degree to which it achieves its objectives under specified conditions (Kossiakoff et al 2011 p 157)

TABLE 2 PANORAMA OF MOE DEFINITIONS

Definition Source The ldquooperationalrdquo measures of success that are closely related to the achievement of the mission or operational objective being evaluated in the intended operational environment under a specified set of conditions ie how well the solution achieves the intended purpose Adapted from DoDI 500002 Defense Acquisition University and International Council on Systems Engineering

(Roedler amp Jones 2005)

ldquohellip standards against which the capability of a (Sproles 2001 solution to meet the needs of a problem may be p 254) judged The standards are specific properties that any potential solution must exhibit to some extent MOEs are independent of any solution and do not specify performance or criteriardquo

ldquoA measure of effectiveness is any mutually (Dockery 1986 agreeable parameter of the problem which induces p 174) a rank ordering on the perceived set of goalsrdquo

ldquoA measure of the ability of a system to meet its specified needs (or requirements) from a particular viewpoint(s) This measure may be quantitative or qualitative and it allows comparable systems to be ranked These effectiveness measures are defined in the problem-space Implicit in the meeting of problem requirements is that threshold values must be exceededrdquo

(Smith amp Clark 2004 p 3)

hellip how effective a task was in doing the right (Masterson 2004) thing

A criterion used to assess changes in system (Joint Chiefs of behavior capability or operational environment Staff 2011 p xxv) that is tied to measuring the attainment of an end state achievement of an objective or creation of an effect

hellip an MOE may be based on quantitative measures (National Research to reflect a trend and show progress toward a Council 2013 measurable threshold p 166)

hellip are measures designed to correspond to (AcqNotes 2016b) accomplishment of mission objectives and achievement of desired results They quantify the results to be obtained by a system and may be expressed as probabilities that the system will perform as required

April 2017

TABLE 2 PANORAMA OF MOE DEFINITIONS CONTINUED

Definition Source The data used to measure the military effect (Measures of (mission accomplishment) that comes from Effectiveness 2015) using the system in its expected environment That environment includes the system under test and all interrelated systems that is the planned or expected environment in terms of weapons sensors command and control and platforms as appropriate needed to accomplish an end-to-end mission in combat

A quantitative measure that represents the (Wasson 2015 outcome and level of performance to be achieved p 101) by a system product or service and its level of attainment following a mission

The goal of the benchmarking framework that is proposed in this study is to provide a systematic process for evaluating the effectiveness of an organishyzationrsquos security posture The proposed framework process and procedures are categorized into the following four functional areas (a) hierarchical structure (b) judgment dominance and alternatives (c) measures and (d) analysis (Chelst amp Canbolat 2011 Saaty amp Alexander 1989) as shown in Figure 1 We develop a scorecard system that is based on a ubiquitous surshyvey of 502 cyber security Subject Matter Experts (SMEs) The form fit and function of the two MCDM models were compared during the development of the scorecard system for each model using the process and procedures shown in Figure 1

FIGURE 1 APPLICATION BENCHMARKING FRAMEWORK

Function 1

Function 2

Function 3

Function 4

FitshyForshyPurpose

Function

Hierarchical Structure

Judgment Dominance Alternatives

Measures

Analysis

Form Methodology The benchmarking framework shown in Figure 1 is accomplished by

considering multiple facets of a problem the problem is divided into smaller components that can yield qualitative and quantitative priorities from cyber security SME judgments Each level within the framework affects the levels above and below it The AHP and ANP facilitate SME knowledge using heushyristic judgments throughout the framework (Saaty 1991) The first action (Function 1) requires mapping out a consistent goal criteria parameters and alternatives for each of the models shown in Figures 2 and 3

FIGURE 2 AAA IN AHP FORM

Criteria

Subcriteria

Alternatives

Maximize Network(s) AccessMobility Measure of Effectiveness for

Trusted Information Providers AAA

Authentication (A1)

Authorization (A2)

Diameter RADIUS Activity QampA User Name Password (Aging)

LAN WAN

Accounting (A3)

Human Log Enforcement

Automated Log Enforcement

RemoteshyUser

Note AAA = Authentication Authorization and Accounting AHP = Analytical Hierarchy Process LAN = Local Area Network QampA = Question and Answer WAN = Wide Area Network

April 2017

FIGURE 3 AAA IN ANP FORM

Maximize Network(s) Access Controls Measure of Effectiveness for

bull Authentication bull RADIUS bull Diameter

Identify (1)

bull LAN bull WAN bull Remote User

bull Authorization bull Activity QampA bull User Name amp

Password Aging

Alternatives (4)

ANALYTICAL NETWORK PROCESS

Access (2)

Elements

bull Accounting bull Human Log

Enforcement bull Automated Log Mgt

Activity (3)

Outer Dependencies

Note AAA = Authentication Authorization and Accounting ANP = Analytical Network Process LAN = Local Area Network Mgt = Management QampA = Question and Answer WAN = Wide Area Network

In this study the AHP and ANP models were designed with the goal of maximizing the network access and mobility MOEs for the TIPrsquos AAA The second action of Function 1 is to divide the goal objectives into clustered groups criteria subcriteria and alternatives The subcriteria are formed from the criteria cluster (Saaty 2012) which enables further decomposition of the AAA grouping within each of the models The third action of Function 1 is the decomposition of the criteria groups which enables a decision maker to add change or modify the depth and breadth of the specificity when making a decision that is based on comparisons within each grouping The final cluster contains the alternatives which provide the final weights from the hierarchical components These weights generate a total ranking priority that constitutes the MOE baseline for the AAA based on the attrishybutes of the criteria

The criteria of AAA implement an infrastructure of access control systems (Hu Ferraiolo amp Kuhn 2006) in which a server verifies the authentication and authorization of entities that request network access and manages their billing accounts Each of the criteria has defined structures for applishycation-specific information Table 3 defines the attributes of the AHP and ANP model criteria subcriteria and alternatives it does not include all of the subcriteria for AAA

TABLE 3 AHPANP MODEL ATTRIBUTES

Attributes Description Source Accounting Track of a users activity (Accounting nd)

while accessing a networks resources including the amount of time spent in the network the services accessed while there and the amount of data transferred during the session Accounting data are used for trend analysis capacity planning billing auditing and cost allocation

Activity QampA Questions that are used when resetting your password or logging in from a computer that you have not previously authorized

(Scarfone amp Souppaya 2009)

Authentication The act of verifying a claimed identity in the form of a preexisting label from a mutually known name space as the originator of a message (message authentication) or as the end-point of a channel (entity authentication)

(Aboba amp Wood 2003 p 2)

Authorization The act of determining if a particular right such as access to some resource can be granted to the presenter of a particular credential

Automatic Log Management

Automated Logs provide (Kent amp Souppaya firsthand information regarding 2006) your network activities Automated Log management ensures that network activity data hidden in the logs are converted to meaningful actionable security information

April 2017

TABLE 3 AHPANP MODEL ATTRIBUTES CONTINUED

Attributes Description Source Diameter Diameter is a newer AAA (Fajardo Arkko

protocol for applications such Loughney amp Zorn as network access and IP 2012) mobility It is the replacement for the protocol radius It is intended to work in both local and roaming AAA situations

Human Accounting Enforcement

Human responsibilities for log (Kent amp Souppaya management for personnel 2006)throughout the organization including establishing log management duties at both the individual system level and the log management infrastructure level

LANmdashLocal A short distance data (LANmdashLocal Area Area Network communications network Network 2008 p 559)

(typically within a building or campus) used to link computers and peripheral devices (such as printers CD-ROMs modems) under some form of standard control

RADIUS RADIUS is an older protocol for (Rigney Willens carrying information related to Rubens amp Simpson authentication authorization 2000) and configuration between a Network Access Server that authenticates its links to a shared Authentication Server

Remote User In computer networking (Mitchell 2016) remote access technology allows logging into a system as an authorized user without being physically present at its keyboard Remote access is commonly used on corporate computer networks but can also be utilized on home networks

User Name Users must change their (Scarfone amp Souppaya amp Password passwords according to a 2009) Aging schedule

WANmdashWide A public voice or data network (WANmdashWide Area Area Network that extends beyond the Network 2008)

metropolitan area

The relationship between authentication and its two subcriteriamdashRADIUS (Rigney Willens Rubens amp Simpson 2000) and Diameter (Fajardo Arkko Loughney amp Zorn 2012)mdashenables the management of network access (Figures 2 and 3) Authorization enables access using Password Activity Question amp Answer which is also known as cognitive passwords (Zviran amp Haga 1990) or User Name amp Password Aging (Zeilenga 2001) (Figures 2 and 3) Accounting (Aboba Arkko amp Harrington 2000) can take two forms which include the Automatic Log Management system or Human Accounting Enforcement (Figures 2 and 3) Our framework enables each TIP to evaluate a given criterion (such as authentication) and its associated subcriteria (such as RADIUS versus Diameter) and determine whether additional resources should be expended to improve the effectiveness of the AAA After the qualitative AHP and ANP forms were completed these data were quantitatively formulated using AHPrsquos hierarchical square matrix and ANPrsquos feedback super matrix

A square matrix is required for the AHP model to obtain numerical values that are based on group judgments record these values and derive priorishyties Comparisons of n pairs of elements based on their relative weights are described in Criteria A1 hellip An and by weights w1 hellip wn (Saaty 1991 p 15)

A reciprocal matrix was constructed based on the following property aji = 1aj where aii = 1 (Saaty 1991 p 15) Multiplying the reciprocal matrix by the transposition of vector wT = (w1hellip wn) yields vector nw thus Aw = nw (Saaty 1977 p 236)

To test the degree of matrix inconsistency a consistency index was genshyerated by adding the columns of the judgment matrix and multiplying the resulting vector by the vector of priorities This test yielded an eigenvalue that is denoted by λ max (Saaty 1983) which is the largest eigenvalue of a reciprocal matrix of order n To measure the deviation from consistency Saaty developed the following consistency index (Saaty amp Vargas 1991)

CI = (λ max ndash n) (n -1)

As stated by Saaty (1983) ldquothis index has been randomly generated for recipshyrocal matrices of different orders The averages of the resulting consistency indices (RI) are given byrdquo (Saaty amp Vargas 1991 p 147)

n 1 2 3 4 5 6 7 8 RI 0 0 058 09 112 124 132 141

April 2017

The consistency ratio (CR) is defined as CR = CIRI and a CR of 20 percent or less satisfies the consistency criterion (Saaty 1983)

The ANP model is a general form of the AHP model which employs complex relationships among the decision levels The AHP model formulates a goal at the top of the hierarchy and then deconstructs it to the bottom to achieve its results (Saaty 1983) Conversely the ANP model does not adhere to a strict decomposition within its hierarchy instead it has feedback relationships among its levels This feedback within the ANP framework is the primary difference between the two models The criteria can describe dependence using an undirected arc between the levels of analysis as shown in Figure 3 or using a looped arc within the same level The ANP framework uses interdependent relationships that are captured in a super matrix (Saaty amp Peniwati 2012)

Fit-for-Purpose Approach We developed a fit-for-purpose approach that includes a procedure

for effectively validating the benchmarking of a cyber security MOE We created an AAA scorecard system by analyzing empirical evidence that introduced MCDM methodologies within the cyber security discipline with the goal of improving an organizationrsquos total security posture

The first action of Function 2 is the creation of a survey design This design which is shown in Table 3 is the basis of the survey questionnaire The targeted sample population was composed of SMEs that regularly manage Information Technology (IT) security issues The group was self-identified in the survey and selected based on their depth of experishyence and prerequisite knowledge to answer questions regarding this topic (Office of Management and Budget [OMB] 2006) We used the Internet surshyvey-gathering site SurveyMonkey Inc (Palo Alto California httpwww surveymonkeycom) for data collection The second activity of Function 2 was questionnaire development a sample question is shown in Figure 4

Defense ARJ April 2017 Vol 24 No 2 186ndash221

FIGURE 4 SURVEY SAMPLE QUESTION AND SCALE

With respect to User NamePasswordshyAging what do you find to be more important

Based on your previous choice evaluate the following statements

Remote User

Importance of Selection

Equal Importance

Moderate Importance

Strong Importance

Very Strong Importance

Extreme Importance

The questions were developed using the within-subjects design concept This concept compels a respondent to view the same question twice but in a different manner A within-subjects design reduces the errors that are associated with individual differences by asking the same question in a difshyferent way (Epstein 2013) This process enables a direct comparison of the responses and reduces the number of required respondents (Epstein 2013)

The scaling procedure in this study was based on G A Millerrsquos (1956) work and the continued use of Saatyrsquos hierarchal scaling within the AHP and ANP methodologies (Saaty 1977 1991 2001 2009 2010 2012 Saaty amp Alexander 1989 Saaty amp Forman 1992 Saaty amp Peniwati 2012 Saaty amp Vargas 1985 1991) The scales within each question were based on the Likert scale this scale has ldquoequal importancerdquo as the lowest parameter which is indicated with a numerical value of one and ldquoextreme importancerdquo as the highest parameter which is indicated with a numerical value of nine (Figure 4)

Demographics is the third action of Function 2 Professionals who were SMEs in the field of cyber security were sampled and had an equal probashybility of being chosen for the survey Using probabilities each SME had an equal probability of being chosen for the survey The random sample enabled an unbiased representation of the group (Creative Research Systems 2012 SurveyMonkey 2015) A sample size of 502 respondents was surveyed in this study Of the 502 respondents 278 of the participants completed all of the survey responses The required margin of error which is also known as the confidence interval was plusmn6 This statistic is based on the concept of how well the sample populationrsquos answers can be considered to represent the ldquotrue valuerdquo of the required population (eg 100000+) (Creative Research

April 2017

Systems 2012 SurveyMonkey 2015) The confidence level accurately measures the sample size and shows that the population falls within a set margin of error A 95 percent confidence level was required in this survey

Survey Age of respondents was used as the primary measurement source for experience with a sample size of 502 respondents to correlate against job position (Table 4) company type (Table 5) and company size (Table 6)

TABLE 4 AGE VS JOB POSITION

Age-Row 1 2 3 4 5 Grand Labels Total

18-24 1 1 4 5 11

25-34 7 2 27 6 28 70

35-44 22 1 63 21 32 139

45-54 19 4 70 41 42 176

55-64 11 1 29 15 26 82

65 gt 1 2 3 6

Grand 60 9 194 85 136 484 Total

SKIPPED 18

Legend 1 2 3 4 5

(Job NetEng Sys- IA IT Mgt Other Position) Admin

Note IA = Information Assurance IT = Information Technology NetEng = Network Engineering SysAdmin = System Administration

TABLE 5 AGE VS COMPANY TYPE

18-24 2 7 2 11

25-34 14 7 35 10 4 70

35-44 13 26 69 19 11 138

45-54 13 42 73 35 13

55-64 7 12 37 22 4

65 gt 5 1 6

Grand 47 87 216 98 35 Total

SKIPPED 19

Legend 1 2 3 4 5

(Job Mil Govt Com- FFRDC Other Position) Uniform mercial

Note FFRDC = Federally Funded Research and Development Center Govrsquot = Government Mil = Military

TABLE 6 AGE VS COMPANY SIZE

Age-Row 1 2 3 4 Grand Labels Total

18-24 2 1 1 7 11

25-34 8 19 7 36 70

35-44 16 33 17 72 138

45-54 19 37 21 99 176

55-64 11 14 10 46 81

65 gt 2 4 6

Grand 58 104 56 264 482 Total

SKIPPED 20

Legend 1 2 3 4

(Company 1-49 50-999 1K-5999 6K gt Size)

The respondents were usually mature and worked in the commercial sector (45 percent) in organizations that had 6000+ employees (55 percent) and within the Information Assurance discipline (40 percent) A high number of

April 2017

respondents described their job descriptions as other (28 percent) The other category in Table 4 reflects an extensive range of job titles and job descripshytions in the realm of cyber security which were not categorized in Table 4

Descriptive statistical analysis is the fourth action of Function 2 This action summarizes the outcomes of the characteristics in concise quantitashytive terms to enable statistical inference (Daniel 1990) as listed in Table 7

TABLE 7 CRITERIA DESCRIPTIVE STATISTICS

A 26 Diameter Protocol

B 74 Automated Log Management

A 42 Human Accounting

Enforcement

B 58 Diameter Protocol

Answered 344 Answered 348 1 11

1 22 1 16

2 2 2 17 2 8 2 7

3 9 3 21 3 19 3 13

4 7 4 24 4 10 4 24

5 22 5 66 5 41 5 53

6 15 6 34 6 17 6 25

7 14 7 40 7 25 7 36

8 3 8 12 8 4 8 9

9 6 9 19 9 7 9 12

Mean 5011 Mean 5082 Mean 4803 Mean 5065

Mode 5000 Mode 5000 Mode 5000 Mode 5000

Standard Deviation

2213 Standard Deviation

Variance 4898 Variance 4792 Variance 4611 Variance 4661

Skewedshyness

-0278 Skewedshyness

-0176 Skewedshyness

-0161 Skewedshyness

Kurtosis -0489 Kurtosis -0582 Kurtosis -0629 Kurtosis -0446

n 89000 n 255000 n 147000 n 201000

Std Err 0235 Std Err 0137 Std Err 0177 Std Err 0152

Minimum 1000 Minimum 1000 Minimum 1000 Minimum 1000

1st Quartile 4000 1st Quartile 4000 1st Quartile 3000 1st Quartile 4000

Median 5000 Median 5000 Median 5000 Median 5000

3rd Quarshytile

7000 3rd Quarshytile

Maximum 9000 Maximum 9000 Maximum 9000 Maximum 9000

Range 8000 Range 8000 Range 8000 Range 8000

Which do you like best Which do you like best

Statistical inference which is derived from the descriptive analysis relates the population demographics data normalization and data reliability of the survey based on the internal consistency Inferential statistics enables a sample set to represent the total population due to the impracticality of surveying each member of the total population The sample set enables a visual interpretation of the statistical inference and is used to calculate the standard deviation mean and other categorical distributions and test the data normality The MiniTabreg software was used to perform these analyses as shown in Figure 5 using the Anderson-Darling testing methodology

FIGURE 5 RESULTS OF THE ANDERSON DARLING TEST

70 60 50 40 30 20

Probability of Plot Q9 Normal

Mean StDev N AD PshyValue

0 3 6 9 12

4839 2138

373 6619

lt0005

The data were tested for normality to determine which statistical tests should be performed (ie parametric or nonparametric tests) We discovshyered that the completed responses were not normally distributed (Figure 5) After testing several questions we determined that nonparametric testing was the most appropriate statistical testing method using an Analysis of Variance (ANOVA)

An ANOVA is sensitive to parametric data versus nonparametric data however this analysis can be performed on data that are not normally distributed if the residuals of the linear regression model are normally distributed (Carver 2014) For example the residuals were plotted on a Q-Q plot to determine whether the regression indicated a significant relationship between a specific demographic variable and the response to Question 9

April 2017

from the survey questionnaire The resulting plot (Figure 6) shows norshymally distributed residuals which is consistent with the assumption that a General Linear Model (GLM) is adequate for the ANOVA test for categorical demographic predictors (ie respondent age employer type employer size and job position)

FIGURE 6 RESIDUAL Q Q PLOT AND ITS GLM ANALYSIS FOR Q9 Factor Information Factor Type Levels Values AGE Fixed 6 1 2 3 4 5 6 SIZE Fixed 4 1 2 3 4 Type Fixed 5 1 2 3 4 5 Position Fixed 5 1 2 3 4 5 Analysis of Variance Source DF Adj SS Adj MS F-Value P-Value

AGE 5 3235 6470 143 0212 SIZE 3 402 1340 030 0828 Type 4 2840 7101 157 0182 Position 4 2364 5911 131 0267

Error 353 159656 4523 Lack-of-Fit 136 63301 4654 105 0376 Pure Error 217 96355 4440

Total 369 169022

Y = Xβ + ε (Equation 1)

Q9 = 5377 - 1294 AGE_1 - 0115 AGE_2 - 0341 AGE_3 - 0060 AGE_4 + 0147 AGE_5 + 166 AGE_6 + 0022 SIZE_1 + 0027 SIZE_2 + 0117 SIZE_3 - 0167 SIZE_4 - 0261 Type_1 + 0385 Type_2 - 0237 Type_3 - 0293 Type_4 + 0406 Type_5 + 0085 Position_1 + 0730 Position_2 - 0378 Position_3 + 0038 Position_4 - 0476 Position_5

Note ε error vectors are working in the background

diamsβ Coefficients Term Coef SE Coef T-Value P-Value VIF Constant 5377 0318 1692 0000 AGE

1 -1294 0614 -211 0036 107 2 -0115 0366 -031 0754 132 3 -0341 0313 -109 0277 176 4 -0060 0297 -020 0839 182 5 0147 0343 043 0669 138

SIZE 1 0022 0272 008 0935 302 2 0027 0228 012 0906 267 3 0117 0275 043 0670 289

Type 1 -0261 0332 -079 0433 149 2 0385 0246 156 0119 128 3 -0237 0191 -124 0216 118 4 -0293 0265 -111 0269 140

Position 1 0085 0316 027 0787 303 2 0730 0716 102 0309 897 3 -0378 0243 -155 0121 306 4 0038 0288 013 0896 303

Parameters

FIGURE 6 RESIDUAL Q Q PLOT AND ITS GLM ANALYSIS FOR Q9 CONTINUED

Q9 What do you like best

Password Activity-Based QampA or Diameter Protocol

Normal Probability Plot (response is Q9)

Residual

70 60 50 40 30 20

01 shy75 shy50 shy25 00 25 50

The P-values in Figure 6 show that the responses to Question 9 have minishymal sensitivity to the age size company type and position Additionally the error ( ε ) of the lack-of-fit has a P-value of 0376 which indicates that there is insufficient evidence to conclude that the model does not fit The GLM model formula (Equation 1) in Minitabreg identified Y as a vector of survey question responses β as a vector of parameters (age job position company type and company size) X as the design matrix of the constants and ε as a vector of the independent normal random variables (MiniTabreg 2015) The equation is as follows

Y = Xβ + ε (1)

Once the data were tested for normality (Figure 6 shows the normally disshytributed residuals and equation traceability) an additional analysis was conducted to determine the internal consistency of the Likert scale survey questions This analysis was performed using Cronbachrsquos alpha (Equation 2) In Equation 2 N is the number of items c-bar is the average inter-item covariance and v-bar is the average variance (Institute for Digital Research and Education [IDRE] 2016) The equation is as follows

April 2017

N c (2)

α = v + (N ndash 1) c

Cronbachrsquos alpha determines the reliability of a survey questionnaire based on the internal consistency of a Likert scale question as shown in Figure 4 (Lehman et al 2011) Cronbachrsquos alpha scores that are greater than 070 are considered to indicate good performance The score for the respondent data from the survey was 098

The determination of dominance is the fifth action of Function 2 which converts individual judgments into group decisions for a pairwise comshyparison between two survey questions (Figure 4) The geometric mean was employed for dominance selection as shown in Equation (3) (Ishizaka amp Nemery 2013) If the geometric mean identifies a tie between answers A (49632) and B (49365) then expert judgment is used to determine the most significant selection The proposed estimates suggested that there was no significant difference beyond the hundredth decimal position The equation is as follows

1NN (3)geometric mean = (prodx)i

The sixth and final action of Function 2 is a pairwise comparison of the selection of alternatives and the creation of the AHP and ANP scorecards The number of pairwise comparisons is based on the criteria for the intershyactions shown in Figures 2 and 3mdashthe pairwise comparisons form the AHP and ANP scorecards The scorecards shown in Figure 7 (AHP) and Figure 8 (ANP) include the pairwise comparisons for each MCDM and depict the dominant AB survey answers based on the geometric mean shaded in red

April 2017

After the scorecard data were populated as shown in Figures 7 and 8 the data were transferred into Super Decisions which is a software package that was employed to complete the final function of the proposed analysis

Function To ensure the validity of the datarsquos functionality in forming the AHP

and ANP models we used the Super Decisions (SD) software to verify the proposed methodology The first action of Function 3 is Measures This action begins by recreating the AHP and ANP models as shown in Figures 2 and 3 and replicating them in SD The second action of Function 3 is to incorporate the composite scorecards into the AHP and ANP model designs The composite data in the scorecards were input into SD to verify that the pairwise comparisons of the AHP and ANP models in the scorecards (Figures 7 and 8) had been mirrored and validated by SDrsquos questionnaire section During the second action and after the scorecard pairwise criteria comparison section had been completed immediate feedback was provided to check the data for inconsistencies and provide a cluster priority ranking for each pair as shown in Figure 9

FIGURE 9 AHP SCORECARD INCONSISTENCY CHECK Comparisons wrt 12_Diameternode in 4Alternatives cluster 1_LAN is moderately more important than 2_WAN 1 1_LAN gt=95 9 8 7 6 5 4 3 2 2 3 4 5 6 7 8 9 gt=95 No comp 2_WAN 2 1_LAN gt=95 9 8 7 6 5 4 3 2 2 3 4 5 6 7 8 9 gt=95 No comp 3_Remote User 3 2_WAN gt=95 9 8 7 6 5 4 3 2 2 3 4 5 6 7 8 9 gt=95 No comp 3_Remote User

Inconsistency 013040

1_LAN 028083

2_WAN 013501

3_Remote 058416

All of the AHP and ANP models satisfied the required inconsistency check with values between 010 and 020 (Saaty 1983) This action concluded the measurement aspect of Function 3 Function 4mdashAnalysismdashis the final portion of the application approach to the benchmarking framework for the MOE AAA This function ranks priorities for the AHP and ANP models The first action of Function 4 is to review the priorities and weighted rankings of each model as shown in Figure 10

FIGURE 10 AHPANP SECURITY METRICS

AHP ANP RADIUS 020000 Authentication RADIUS 018231

Diameter 080000 Diameter 081769

LAN 012950

WAN 033985

Remote User 053065

Password Activity QampA

020000 Authorization Password Activity QampA

020000

User Name amp Password Aging

080000 User Name amp Password Aging

080000

LAN 012807

WAN 022686

Remote User 064507

Human Acct Enforcement

020001 Accounting Human Acct Enforcement

020000

Auto Log Mgt 079999 Auto Log Mgt 080000

LAN 032109

WAN 013722

Remote User 054169

LAN 015873 Alternative Ranking

LAN 002650

WAN 024555 WAN 005710

Remote User 060172 Remote User 092100

These priorities and weighted rankings are the AAA security control meashysures that cyber security leaders need to make well-informed choices as they create and deploy defensive strategies

April 2017

Summary of Analysis Using a GLM the survey data showed normally distributed residuals

which is consistent with the assumption that a GLM is adequate for the ANOVA test for categorical demographic predictors (ie the respondent age employer type employer size and job position)

Additionally using Cronbachrsquos alpha analysis a score of 098 ensured that the reliability of the survey questionnaire was acceptable based on the internal consistency of the Likert scale for each question

The subjective results of the survey contradicted the AHP and ANP MCDM model results shown in Figure 10

The survey indicated that 67 percent (with a plusmn6 margin of error) of the respondents preferred RADIUS to Diameter conversely both the AHP model and the ANP model selected Diameter over RADIUS Within the ANP model the LAN (2008) WAN (2008) and remote user communities proshyvided ranking priorities for the subcriteria and a final community ranking at the end based on the model interactions (Figures 3 and 10) The ranking of interdependencies outer-dependencies and feedback loops is considered within the ANP model whereas the AHP model is a top-down approach and its community ranking is last (Figures 2 and 10)

The preferences between User Name amp Password Aging and Password Activity QampA were as follows of the 502 total respondents 312 respondents indicated a preference for User Name amp Password Aging over Password Activity QampA by 59 percent (with a plusmn6 margin of error) The AHP and ANP metrics produced the same selection (Figures 2 3 and 10)

Of the 502 total respondents 292 respondents indicated a preference for Automated Log Management over Human Accounting Enforcement by 64 percent (with a plusmn6 margin of error) The AHP and ANP metrics also selected Automated Log Management at 80 percent (Figures 2 3 and 10)

The alternative rankings of the final communities (LAN WAN and remote user) from both the AHP and ANP indicated that the remote user commushynity was the most important community of interest

The degree of priority for the two models differed in their ranking weights among the first second and third rankings The differences in the degree of priority between the two models were likely caused by the higher degree of feedback interactions within the ANP model than within the AHP model (Figures 2 3 and 10)

The analysis showed that all of the scorecard pairwise comparisons based upon the dominant geometric mean of the survey AB answers fell within the inconsistency parameters of the AHP and ANP models (ie between 010 and 020) The rankings indicated that the answer ldquoremote userrdquo was ranked as the number one area for the AAA MOEs in both models with priority weighted rankings of 060172 for AHP and 092100 for ANP as shown in Figure 10 and as indicated by a double-sided arrow symbol This analysis concluded that the alternative criteria should reflect at least the top ranking answer for either model based on the empirical evidence presented in the study

April 2017

Study Limitations The study used existing age as an indicator of experience versus responshy

dents security and years of expertise

Areas for Future Research Additional research is recommended regarding the benchmarking

framework application approach for Cyber Security Metrics MOE The authorrsquos dissertation (Wilamowski 2017) includes survey data including empirical analysis and detailed descriptive statistics The scope of the study can be expanded to include litigation from cyber attacks to the main criteria of the AHPANP MCDM models Adding the cyber attack litigation to the models will enable consideration of the financial aspect of the total security controls regarding cost benefit opportunity and risk

Conclusions The research focused on the decision theory that features MCDM AHP

and ANP methodologies We determined that a generalized application benchmark framework can be employed to derive MOEs based on targeted survey respondentsrsquo preferences for security controls The AHP is a suitable option if a situation requires rapid and effective decisions due to an impendshying threat The ANP is preferable if the time constraints are less important and more far-reaching factors should be considered while crafting a defenshysive strategy these factors can include benefits opportunities costs and risks (Saaty 2009) The insights developed in this study will provide cyber security decision makers a method for quantifying the judgments of their technical employees regarding effective cyber security policy The results will be the ability to provide security and reduce risk by shifting to newer and improved requirements

The framework presented herein provides a systematic approach to developing a weighted security ranking in the form of priority rating recshyommendations for criteria in producing a model and independent first-order results An application approach of a form-fit-function is employed as a generalized application benchmarking framework that can be replicated for use in various fields

References Aboba B Arkko J amp Harrington D (2000) Introduction to accounting management

(RFC 2975) Retrieved from httpstoolsietforghtmlrfc2975 Aboba B amp Wood J (2003) Authentication Authorization and Accounting (AAA)

transport profile (RFC 3539) Internet Engineering Task Force Retrieved from httpswwwpotaroonetietfhtmlrfc3500-3599html

Accounting (nd) In Webopedia Retrieved from httpwwwwebopediacom TERMAAAAhtml

AcqNotes (2016a) JCIDS process Capabilities Based Assessment (CBA) Retrieved from httpwwwacqnotescomacqnoteacquisitionscapabilities-basedshyassessment-cba

AcqNotes (2016b) Systems engineering Measures of Effectiveness (MOE) Retrieved from httpwwwacqnotescomacqnotecareerfieldsse-measures-ofshyeffectiveness

Bahnsen A C Aouada D amp Ottersten B (2015) Example-dependent cost-sensitive decision trees Expert Systems with Applications 42(19) 6609ndash6619

Bedford T amp Cooke R (1999) New generic model for applying MAUT European shyJournal of Operational Research 118(3) 589ndash604 doi 101016S0377

2217(98)00328-2 Carver R (2014) Practical data analysis with JMP (2nd ed) Cary NC SAS Institute Chan L K amp Wu M L (2002) Quality function deployment A literature review

European Journal of Operational Research 143(3) 463ndash497 Chelst K amp Canbolat Y B (2011) Value-added decision making for managers Boca

Raton FL CRC Press Cockburn A (2001) Writing effective use cases Addison-Wesley Ann Arbor

Michigan Creative Research Systems (2012) Sample size calculator Retrieved from http

wwwsurveysystemcomsscalchtm Daniel W W (1990) Applied nonparametric statistics (2nd ed) Pacific Grove CA

Duxbury Department of Defense (2004) Procedures for interoperability and supportability of

Information Technology (IT) and National Security Systems (NSS) (DoDI 4630) Washington DC Assistant Secretary of Defense for Networks amp Information IntegrationDepartment of Defense Chief Information Officer

Dockery J T (1986 May) Why not fuzzy measures of effectiveness Signal 40 171ndash176

Epstein L (2013) A closer look at two survey design styles Within-subjects amp between-subjects Survey Science Retrieved from httpswwwsurveymonkey comblogenblog20130327within-groups-vs-between-groups

EY (2014) Letrsquos talk cybersecurity EY Retrieved from httpwwweycomglen servicesadvisoryey-global-information-security-survey-2014-how-ey-can-help

Fajardo V (Ed) Arkko J Loughney J amp Zorn G (Ed) (2012) Diameter base protocol (RFC 6733) Internet Engineering Task Force Retrieved from https wwwpotaroonetietfhtmlrfc6700-6799html

Hu VC Ferraiolo D F amp Kuhn DR (2006) Assessment of access control systems (NIST Interagency Report No 7316) Retrieved from httpcsrcnistgov publicationsnistir7316NISTIR-7316pdf

April 2017

IDRE (2016) What does Cronbachs alpha mean Retrieved from httpwwwats uclaedustatspssfaqalphahtml

Ishizaka A amp Nemery P (2013) Multi-criteria decision analysis Methods and software Somerset NJ John Wiley amp Sons

Joint Chiefs of Staff (2011) Joint operations (Joint Publication 3-0) Washington DC Author

Keeney R L (1976) A group preference axiomatization with cardinal utility Management Science 23(2) 140ndash145

Keeney R L (1982) Decision analysis An overview Operations Research 30(5) 803ndash838

Kent K amp Souppaya M (2006) Guide to computer security log management (NIST Special Publication 800-92) Gaithersburg MD National Institute of Standards and Technology

Kossiakoff A Sweet W N Seymour S J amp Biemer S M (2011) Systems engineering principles and practice Hoboken NJ John Wiley amp Sons

Kurematsu M amp Fujita H (2013) A framework for integrating a decision tree learning algorithm and cluster analysis Proceedings of the 2013 IEEE 12th International Conference on Intelligent Software Methodologies Tools and Techniques (SoMeT 2013) September 22-24 Piscataway NJ doi 101109SoMeT20136645670

LAN ndash Local Area Network (2008) In Newtons Telecom Dictionary (24th ed) New York NY Flatiron Publications

Lehman T Yang X Ghani N Gu F Guok C Monga I amp Tierney B (2011) Multilayer networks An architecture framework IEEE Communications Magazine 49(5) 122ndash130 doi101109MCOM20115762808

Maisey M (2014) Moving to analysis-led cyber-security Network Security 2014(5) 5ndash12

Masterson M J (2004) Using assessment to achieve predictive battlespace awareness Air amp Space Power Journal [Chronicles Online Journal] Retrieved from httpwwwairpowermaxwellafmilairchroniclesccmastersonhtml

McGuire B (2015 February 4) Insurer Anthem reveals hack of 80 million customer employee accounts abcNEWS Retrieved from httpabcnewsgocom Businessinsurer-anthem-reveals-hack-80-million-customer-accounts storyid=28737506

Measures of Effectiveness (2015) In [Online] Glossary of defense acquisition acronyms and terms (16th ed) Defense Acquisition University Retrieved from httpsdapdaumilglossarypages2236aspx

Miller G A (1956) The magical number seven plus or minus two Some limits on our capacity for processing information Psychological Review 63(2) 81ndash97 Retrieved from httpdxdoiorg1010370033-295X1012343

MiniTabreg (2015) Methods and formulas Minitabreg v17 [Computer software] State College PA Author

Mitchell B (2016) What is remote access to computer networks Lifewire Retreived from httpcompnetworkingaboutcomodinternetaccessbestusesfwhat-isshynetwork-remote-accesshtm

MITRE (2014) MITRE systems engineering guide Bedford MA MITRE Corporate Communications and Public Affairs

Morse P M amp Kimball G E (1946) Methods of operations research (OEG Report No 54) (1st ed) Washington DC National Defence Research Committee

National Research Council (2013) Making the soldier decisive on future battlefields Committee on Making the Soldier Decisive on Future Battlefields Board on Army Science and Technology Division on Engineering and Physical Sciences Washington DC The National Academies Press

National Institute of Standards and Technology (2014) Assessing security and privacy controls in federal information systems and organizations (NIST Special Publication 800-53A [Rev 4]) Joint Task Force Transformation Initiative Retrieved from httpnvlpubsnistgovnistpubsSpecialPublicationsNIST SP800-53Ar4pdf

Obama B (2015) Executive ordermdashpromoting private sector cybersecurity information sharing The White House Office of the Press Secretary Retrieved from httpswwwwhitehousegovthe-press-office20150213executive-ordershypromoting-private-sector-cybersecurity-information-shari

OMB (2006) Standards and guidelines for statistical surveys Retrieved from https wwwfederalregistergovdocuments2006092206-8044standards-andshyguidelines-for-statistical-surveys

Pachghare V K amp Kulkarni P (2011) Pattern based network security using decision trees and support vector machine Proceedings of 2011 3rd International Conference on Electronics Computer Technology (ICECT 2011) April 8ndash10 Piscataway NJ

Rabbani S J amp Rabbani S R (1996) Decisions in transportation with the analytic hierarchy process Campina Grande Brazil Federal University of Paraiba

Rigney C Willens S Rubens A amp Simpson W (2000) Remote Authentication Dial In User Service (RADIUS) (RFC 2865) Internet Engineering Task Force Retrieved from httpswwwpotaroonetietfhtmlrfc2800-2899html

shyRoedler G J amp Jones C (2005) Technical measurement (Report No INCOSE TEP-2003-020-01) San Diego CA International Council on Systems Engineering

Saaty T L (1977) A scaling method for priorities in hierarchical structures Journal of Mathematical Psychology 15(3) 234ndash281 doi 1010160022-2496(77)90033-5

Saaty T L (1983) Priority setting in complex problems IEEE Transactions on Engineering Management EM-30(3) 140ndash155 doi101109TEM19836448606

Saaty T L (1991) Response to Holders comments on the analytic hierarchy process Journal of the Operational Research Society 42(10) 909ndash914 doi 1023072583425

Saaty T L (2001) Decision making with dependence and feedback The analytic network process (2nd ed) Pittsburgh PA RWS Publications

Saaty T L (2006) Fundamentals of decision making and priority theory with the analytic hierarchy process Vol VI of the AHP Series (2nd ed) Pittsburgh PA RWS Publications

Saaty T L (2009) Theory and applications of the Analytic Network Process Decision making with benefits opportunities costs and risks Pittsburg PA RWS Publications

Saaty T L (2010) Mathematical principles of decision making (Principia mathematica Decernendi) Pittsburg PA RWS Publications

Saaty T L (2012) Decision making for leaders The analytic hierarchy process for decisions in a complex world (3rd ed) Pittsburg PA RWS Publications

Saaty T L amp Alexander J M (1989) Conflict resolution The analytic hierarchy approach New York NY Praeger Publishers

April 2017

Saaty T L amp Forman E H (1992) The Hierarchon A dictionary of hierarchies Pittsburg PA RWS Publications

Saaty T L Kearns K P amp Vargas L G (1991) The logic of priorities Applications in business energy health and transportation Pittsburgh PA RWS Publications

Saaty T L amp Peniwati K (2012) Group decision making Drawing out and reconciling differences (Vol 3) Pittsburgh PA RWS Publications

Saaty T L amp Vargas L G (1985) Analytical planning The organization of systems (Vol 4) Pittsburgh PA RWS Publications

Saaty T L amp Vargas L G (1991) Prediction projection and forecasting Applications of the analytic hierarchy process in economics finance politics games and sports New York Springer Verlag Science + Business Media

Scarfone K amp Souppaya M (2009) Guide to enterprise password management (NIST Draft Special Publication 800-118) Gaithersburg MD National Institute of Standards and Technology

Smith N amp Clark T (2004) An exploration of C2 effectivenessmdashA holistic approach Paper presented at 2004 Command and Control Research and Technology Symposium June 15-17 San Diego CA

Sproles N (2001) Establishing measures of effectiveness for command and control A systems engineering perspective (Report No DSTOGD-0278) Fairbairn Australia Defence Science and Technology Organisation of Australia

Superville D amp Mendoza M (2015 February 13) Obama calls on Silicon Valley to help thwart cyber attacks Associated Press Retrieved from httpsphysorg news2015-02-obama-focus-cybersecurity-heart-siliconhtml

SurveyMonkey (2015) Sample size calculator Retrieved from httpswww surveymonkeycomblogensample-size-calculator

WANmdashWide Area Network (2008) In Newtons Telecom Dictionary (24th ed) New York NY Flatiron Publications

Wasson C S (2015) System engineering analysis design and development Concepts principles and practices (Wiley Series in Systems Engineering Management) Hoboken NJ John Wiley amp Sons

Wei H Frinke D Carter O amp Ritter C (2001) Cost-benefit analysis for network intrusion detection systems Paper presented at CSI 28th Annual Computer Security Conference October 29-31 Washington DC

Weise E (2014 October 3) JP Morgan reveals data breach affected 76 million households USA Today Retrieved from httpwwwusatodaycomstory tech20141002jp-morgan-security-breach16590689

Wilamowski G C (2017) Using analytical network processes to create authorization authentication and accounting cyber security metrics (Doctoral dissertation) Retrieved from ProQuest Dissertations amp Theses Global (Order No 10249415)

Zeilenga K (2001) LDAP password modify extended operation Internet Engineering Task Force Retrieved from httpswwwietforgrfcrfc3062txt

Zheng X amp Pulli P (2005) Extending quality function deployment to enterprise mobile services design and development Journal of Control Engineering and Applied Informatics 7(2) 42ndash49

Zviran M amp Haga W J (1990) User authentication by cognitive passwords An empirical assessment Proceedings of the Fifth Jerusalem Conference on Information Technology (Catalog No 90TH0326-9) October 22-25 Jerusalem Israel

Author Biographies

Mr George C Wilamowski is currently a sysshytems engineer with The MITRE Corporation supporting cyber security efforts at the Marine Corps Cyber Operations Group He is a retired Marine Captain with 24 yearsrsquo service Mr Wilamowski holds an MS in Software Engineering from National University and an MS in Systems Engineering from The George Washing ton University He is currently a PhD candidate in Systems Engineering at The George Washington University His research interests focus on cyber security program management decisions

(E-mail address Wilamowskimitreorg)

Dr Jason R Dever works as a systems engineer supporting the National Reconnaissance Office He has supported numerous positions across the systems engineering life cycle including requireshyments design development deployment and operations and maintenance Dr Dever received his bachelorrsquos degree in Electrical Engineering from Virginia Polytechnic Institute and State University a masterrsquos degree in Engineering Management from The George Washington University and a PhD in Systems Engineering from The George Washington University His teaching interests are project management sysshytems engineering and quality control

(E-mail address Jdevergwmailedu)

April 2017

Dr Steven M F Stuban is the director of the Nationa l Geospatia l-Intelligence Agency rsquos Installation Operations Office He holds a bachshyelorrsquos degree in Engineering from the US Military Academy a masterrsquos degree in Engineering Management from the University of Missouri ndash Rolla and both a masterrsquos and doctorate in S y s t em s E n g i ne er i n g f r om T he G e or ge Washington University Dr Stuban is an adjunct professor with The George Washington University and serves on a standing doctoral committee

(E-mail address stubangwuedu)

CORRECTION The following article written by Dr Shelley M Cazares was originally published in the January 2017 edition of the Defense ARJ Issue 80 Vol 24 No 1 The article is being reprinted due to errors introduced by members of the DAU Press during the production phase of the publication

The Threat Detection System THAT CRIED WOLF Reconciling Developers with Operators

Shelley M Cazares

The Department of Defense and Department of Homeland Security use many threat detection systems such as air cargo screeners and counter-im provised-explosive-device systems Threat detection systems that perform well during testing are not always well received by the system operators however Some systems may frequently ldquocry wolfrdquo generating false alarms when true threats are not present As a result operators lose faith in the systemsmdashignoring them or even turning them off and taking the chance that a true threat will not appear This article reviews statistical concepts to reconcile the performance metrics that summarize a developerrsquos view of a system during testing with the metrics that describe an operatorrsquos view of the system during real-world missions Program managers can still make use of systems that ldquocry wolfrdquo by arranging them into a tiered system that overall exhibits better performance than each individual system alone

DOI httpsdoiorg1022594dau16-7492401 Keywords probability of detection probability of false alarm positive predictive value negative predictive value prevalence

The Threat Detection System That Cried Wolf httpwwwdaumil

The Department of Defense (DoD) and Department of Homeland Security (DHS) operate many threat detection systems Examples include counter-mine and counter-improvised-explosive-device (IED) systems and airplane cargo screening systems (Daniels 2006 L3 Communications Cyterra 2012 L3 Communications Security amp Detection Systems 2011 2013 2014 Niitek nd Transportation Security Administration 2013 US Army nd Wilson Gader Lee Frigui amp Ho 2007) All of these systems share a common purpose to detect threats among clutter

Threat detection systems are often assessed based on their Probability of Detection (Pd) and Probability of False Alarm (Pfa) Pd describes the fraction of true threats for which the system correctly declares an alarm Conversely

describes the fraction of true clutter (true non-threats) for which the Pfa system incorrectly declares an alarmmdasha false alarm A perfect system will exhibit a Pd of 1 and a Pfa of 0 Pd and Pfa are summarized in Table 1 and disshycussed in Urkowitz (1967)

TABLE 1 DEFINITIONS OF COMMON METRICS USED TO ASSESS PERFORMANCE OF THREAT DETECTION SYSTEMS

Metric Definition Perspective The fraction of all items containing Probability of a true threat for which the system Developer Detection (P )d correctly declared an alarm

The fraction of all items not containing Probability of a true threat for which the system Developer False Alarm (Pfa) incorrectly declared an alarm

Positive Predictive Value (PPV)

The fraction of all items causing an alarm that did end up containing a true threat

Operator

Negative Predictive Value (NPV)

The fraction of all items not causing an alarm that did end up not containing a true threat

Operator

The fraction of items that contained a Prevalence true threat (regardless of whether the mdash (Prev) system declared an alarm)

False Alarm Rate The number of false alarms per unit mdash (FAR) time area or distance

Threat detection systems with good Pd and Pfa performance metrics are not always well received by the systemrsquos operators however Some systems may frequently ldquocry wolfrdquo generating false alarms when true threats are not present As a result operators may lose faith in the systems delaying their response to alarms (Getty Swets Pickett amp Gonthier 1995) or ignoring

April 2017

them altogether (Bliss Gilson amp Deaton 1995) potentially leading to disasshytrous consequences This issue has arisen in military national security and civilian scenarios

The New York Times described a 1987 military incident involving the threat detection system installed on a $300 million high-tech warship to track radar signals in the waters and airspace off Bahrain Unfortunately ldquosomeshybody had turned off the audible alarm because its frequent beeps bothered himrdquo (Cushman 1987 p 1) The radar operator was looking away when the system flashed a sign alerting the presence of an incoming Iraqi jet The attack killed 37 sailors

That same year The New York Times reported a similar civilian incident in the United States An Amtrak train collided near Baltimore Maryland killing 15 people and injuring 176 Investigators found that an alarm whistle

in the locomotive cab had been ldquosubstantially disabled by wrapping it with taperdquo and ldquotrain crew members sometimes muff le the warning whistle because the sound is annoyingrdquo (Stuart 1987 p 1)

Such incidents continued to occur two decades later In 2006 The Los Angeles Times described an incident in which a radar air traffic control system at Los Angeles International Airport (LAX) issued a false alarm prompting the human controllers to ldquoturn off the equipmentrsquos aural alertrdquo (Oldham 2006 p 2) Two days later a turboprop plane taking off from the airport narrowly missed a regional jet the ldquoclosest call on the ground at LAXrdquo in 2 years (Oldham 2006 p 2) This incident had homeland security implications since DHS and the Department of Transportation are co-sector-specific agencies for the Transportation Systems Sector which governs air traffic control (DHS 2016)

The disabling of threat detection systems due to false alarms is troubling This behavior often arises from an inappropriate choice of metrics used to assess the systemrsquos performance during testing While Pd and Pfa encapsushylate the developerrsquos perspective of the systemrsquos performance these metrics do not encapsulate the operatorrsquos perspective The operatorrsquos view can be better summarized with other metrics namely Positive Predictive Value

(PPV) and Negative Predictive Value (NPV) PPV describes the fraction of all alarms that

correctly turn out to be true threatsmdasha measure of how

often the system does not ldquocry wolfrdquo Similarly NPV describes the fraction of all lack of alarms that correctly turn out to be

true clutter From the opershyatorrsquos perspective a perfect system will have PPV and

NPV values equal to 1 PPV and NPV are summarized in Table 1 and discussed in

Altman and Bland (1994b)

Interestingly enough the ver y same threat detection system that satisfies the developerrsquos

desire to detect as much truth as possible can also disappoint the operator by generating

false alarms or ldquocrying wolfrdquo too often (Scheaffer amp McClave 1995) A system

April 2017

can exhibit excellent Pd and Pfa values while also exhibiting a poor PPV value Unfortunately low PPV values naturally occur when the Prevalence (Prev) of true threat among true clutter is extremely low (Parasuraman 1997 Scheaffer amp McClave 1995) as is often the case in defense and homeland security scenarios As summarized in Table 1 Prev is a measure of how widespread or common the true threat is A Prev of 1 indicates a true threat is always present while a Prev of 0 indicates a true threat is never present As will be shown a low Prev can lead to a discrepancy in how developers and operators view the performance of threat detection systems in the DoD and DHS

In this article the author reconciles the performance metrics used to quanshytify the developerrsquos versus operatorrsquos views of threat detection systems Although these concepts are already well known within the statistics and human factors communities they are not often immediately understood in the DoD and DHS science and technology (SampT) acquisition communities This review is intended for program managers (PM) of threat detection systems in the DoD and DHS This article demonstrates how to calculate Pd Pfa PPV and NPV using a notional air cargo screening system as an example Then it illustrates how a PM can still make use of a system that frequently ldquocries wolfrdquo by incorporating it into a tiered system that overall exhibits better performance than each individual system alone Finally the author cautions that Pfa and NPV can be calculated only for threat classification systems rather than genuine threat detection systems False Alarm Rate is often calculated in place of Pfa

Testing a Threat Detection System A notional air cargo screening system illustrates the discussion of pershy

formance metrics for threat detection systems As illustrated by Figure 1 the purpose of this notional system is to detect explosive threats packed inside items that are about to be loaded into the cargo hold of an airplane To detershymine how well this system meets capability requirements its performance must be quantified A large number of items is input into the system and each itemrsquos ground truth (whether the item contained a true threat) is compared to the systemrsquos output (whether the system declared an alarm) The items are representative of the items that the system would likely encounter in an opershyational setting At the end of the test the True Positive (TP) False Positive (FP) False Negative (FN) and True Negative (TN) items are counted Figure 2 tallies these counts in a 2 times 2 confusion matrix

bull A TP is an item that contained a true threat and for which the system correctly declared an alarm

bull An FP is an item that did not contain a true threat but for which the system incorrectly declared an alarmmdasha false alarm (a Type I error)

bull An FN is an item that contained a true threat but for which the system incorrectly did not declare an alarm (a Type II error)

bull A TN is an item that did not contain a true threat and for which the system correctly did not declare an alarm

FIGURE 1 NOTIONAL AIR CARGO SCREENING SYSTEM

NOTIONAL Air Cargo Screening

System

Note A set of predefined discrete items (small brown boxes) are presented to the system one at a time Some items contain a true threat (orange star) among clutter while other items contain clutter only (no orange star) For each item the system declares either one or zero alarms All items for which the system declares an alarm (black exclamation point) are further examined manually by trained personnel (red figure) In contrast all items for which the system does not declare an alarm (green checkmark) are left unexamined and loaded directly onto the airplane

As shown in Figure 2 a total of 10100 items passed through the notional air cargo screening system One hundred items contained a true threat while 10000 items did not The system declared an alarm for 590 items and did not declare an alarm for 9510 items Comparing the itemsrsquo ground truth to the systemrsquos alarms (or lack thereof) there were 90 TPs 10 FNs 500 FPs and 9500 TNs

April 2017

FIGURE 2 2 X 2 CONFUSION MATRIX OF NOTIONAL AIR CARGO SCREENING SYSTEM

Ground Truth

Items (10100)

No Threat (10000)

Threat (100)

NOTIONAL System

Alarm (590)

No Alarm (9510)

TP (90) FN (10)

FP (500) TN (9500)

Probability of Detection P

d = 90 (90 + 10) = 090

(near 1 is better)

Probability of False Alarm P

fa = 500 (500 + 9500) = 005

(near 0 is better)

Positive Predictive Value PPV = 90 (90 + 500) = 015 (near 1 is better)

Negative Predictive Value NPV = 9500 (9500 + 10) asymp 1 (near 1 is better)

The Operatorrsquos View

The Developerrsquos View

Note The matrix tabulates the number of TP FN FP and TN items processed by the system Pd and Pfa summarize the developerrsquos view of the systemrsquos performance while PPV and NPV summarize the operatorrsquos view In this notional example the low PPV of 015 indicates a poor operator experience (the system often generates false alarms and ldquocries wolfrdquo since only 15 of alarms turn out to be true threats) even though the good Pd

and Pfa are well received by developers

The Developerrsquos View Pd and Pfa A PM must consider how much of the truth the threat detection system

is able to identify This can be done by considering the following questions Of those items that contain a true threat for what fraction does the system correctly declare an alarm And of those items that do not contain a true threat for what fraction does the system incorrectly declare an alarmmdasha false alarm These questions often guide developers during the research and development phase of a threat detection system

Pd and Pfa can be easily calculated from the 2 times 2 confusion matrix to answer these questions From a developerrsquos perspective this notional air cargo screening system exhibits good1 performance

TP 90Pd= = = 090 (compared to 1 for a perfect system) (1) TP + FN 90 + 10

FP 500 = = 005 (compared to 0 for a perfect system) (2) Pfa= FP + TN 500 + 9500

Equation 1 shows that of all items that contained a true threat (TP + FN = 90 + 10 = 100) a large subset (TP = 90) correctly caused an alarm These counts resulted in Pd = 090 close to the value of 1 that would be exhibited by a perfect system2 Based on this Pd value the PM can conclude that 90 of items that contained a true threat correctly caused an alarm which may (or may not) be considered acceptable within the capability requirements for the system Furthermore Equation 2 shows that of all items that did not contain a true threat (FP + TN = 500 + 9500 = 10000) only a small subset (FP = 500) caused a false alarm These counts led to Pfa = 005 close to the value of 0 that would be exhibited by a perfect system3 In other words only 5 of items that did not contain a true threat caused a false alarm

The Operatorrsquos View PPV and NPV The PM must also anticipate the operatorrsquos view of the threat detection

system One way to do this is to answer the following questions Of those items that caused an alarm what fraction turned out to contain a true threat (ie what fraction of alarms turned out not to be false) And of those items that did not cause an alarm what fraction turned out not to contain a true threat On the surface these questions seem similar to those posed previously for Pd and Pfa Upon closer examination however they are quite different While Pd and Pfa summarize how much of the truth causes an alarm PPV and NPV summarize how many alarms turn out to be true

PPV and NPV can also be easily calculated from the 2 times 2 confusion matrix From an operatorrsquos perspective the notional air cargo screening system exhibits a conflicting performance

TN 9500 NPV = = asymp 1 (compared to 1 for a perfect system) (3) TN + FN 9500 + 10

TP 90PPV = = = 015 (compared to 1 for a perfect system) (4) TP + FP 90 + 500

Equation 3 shows that of all items that did not cause an alarm (TN + FN = 9500 + 10 = 9510) a very large subset (TN = 9500) correctly turned out to not contain a true threat These counts resulted in NPV asymp 1 approxishymately equal to the 1 value that would be exhibited by a perfect system4 In the absence of an alarm the operator could rest assured that a threat was highly unlikely However Equation 4 shows that of all items that did indeed cause an alarm (TP + FP = 90 + 500 = 590) only a small subset (TP = 90) turned out to contain a true threat (ie were not false alarms) These counts unfortunately led to PPV = 015 much lower than the 1 value that would be

April 2017

exhibited by a perfect system5 When an alarm was declared the operator could not trust that a threat was present since the system generated false alarms so often

Reconciling Developers with Operators Pd and Pfa Versus PPV and NPV

The discrepancy between PPV and NPV versus Pd and Pfa reflects the discrepancy between the operatorrsquos and developerrsquos views of the threat detection system Developers are often primarily interested in how much of the truth correctly cause alarmsmdashconcepts quantified by Pd and Pfa In conshytrast operators are often primarily concerned with how many alarms turn out to be truemdashconcepts quantified by PPV and NPV As shown in Figure 2 the very same system that exhibits good values for Pd Pfa and NPV can also exhibit poor values for PPV

Poor PPV values should not be unexpected for threat detection systems in the DoD and DHS Such performance is often merely a reflection of the low Prev of true threats among true clutter that is not uncommon in defense and homeland security scenarios6 Prev describes the fraction of all items that contain a true threat including those that did and did not cause an alarm In the case of the notional air cargo screening system Prev is very low

TP + FN 90 + 10 Prev = = = 001 (5) TP + FN + FP + TN 90 + 10 + 500 + 9500

Equation 5 shows that of all items (TP + FN + FP + TN = 90 + 10 + 500 + 9500 = 10100) only a very small subset (TP + FN = 90 + 10 = 100) contained a true threat leading to Prev = 001 When true threats are rare most alarms turn out to be false even for an otherwise strong threat detection system leading to a low value for PPV (Altman amp Bland 1994b) In fact to achieve a high value of PPV when Prev is extremely low a threat detection system must exhibit so few FPs (false alarms) as to make Pfa approximately zero

Recognizing this phenomenon PMs should not necessarily dismiss a threat detection system simply because it exhibits a poor PPV provided that it also exhibits an excellent Pd and Pfa Instead PMs can estimate Prev to help determine how to guide such a system through development Prev does not depend on the threat detection system and can in fact be calculated in the absence of the system Knowledge of ground truth (which items contain a true threat) is all that is needed to calculate Prev (Scheaffer amp McClave 1995)

Of course ground truth is not known a priori in an operational setting However it may be possible for PMs to use historical data or intelligence tips to roughly estimate whether Prev is likely to be particularly low in operation The threat detection system can be thought of as one system in a system of systems where other relevant systems are based on record keeping (to provide historical estimates of Prev) or intelligence (to provide tips to help estimate Prev) These estimates of Prev can vary over time and location A Prev that is estimated to be very low can cue the PM to anticipate discrepancies in Pd and Pfa versus PPV forecasting the inevitable discrepshyancy between the developerrsquos versus operatorrsquos views early in the systemrsquos development while there are still time and opportunity to make adjustshyments At that point the PM can identify a concept of operations (CONOPS) in which the system can still provide value to the operator for an assigned mission A tiered system may provide one such opportunity

April 2017

A Tiered System for Threat Detection Tiered systems consist of multiple systems used in series The first

system cues the use of the second system and so on Tiered systems provide PMs the opportunity to leverage multiple threat detection systems that individually do not satisfy both developers and operators simultaneously Figure 3 shows two 2 times 2 confusion matrices that represent a notional tiered system that makes use of two individual threat detection systems The first system (top) is relatively simple (and inexpensive) while the second system (bottom) is more complex (and expensive) Other tiered systems can consist of three or more individual systems

FIGURE 3 NOTIONAL TIERED SYSTEM FOR AIR CARGO SCREENING

Items (590)

= 90 (90 + 10) = 090

= 500 (500 + 9500) = 005

PPV1 = 90 (90 + 500) = 015 NPV

1 = 9500 (9500 + 10) asymp 1

= 88 (88 + 2) = 098

= 20 (20 + 480) = 004

PPV2 = 88 (88 + 20) = 081 NPV

2 = 480 (480 + 2) asymp 1

PPVoverall = 88 (88 + 20) = 081

Pd overall = 88 (88 + (10 + 2)) = 088

Pfa overall= 20 (20 + (9500 + 480)) asymp 0

NPVoverall = (9500 + 480) ((9500 + 480) + (10 + 2)) asymp 1

Items (10100)

Ground Truth No Threat

(10000)

Threat (100)

NOTIONAL System 1

Alarm (590)

No Alarm (9510)

TP1 (90) FN1 (10)

FP1 (500) TN1 (9500)

Threat (90)

NOTIONAL System 2

Alarm (108)

No Alarm (482)

TP2 (88) FN2 (2)

FP2 (20) TN2 (480)

Note The top 2 times 2 confusion matrix represents the same notional system described in Figures 1 and 2 While this system exhibits good Pd Pfa and NPV values its PPV value is poor Nevertheless this system can be used to cue a second system to further analyze the questionable items The bottom matrix represents the second notional system This system exhibits a good Pd Pfa and NPV along with a much better PPV The second systemrsquos better PPV reflects the higher Prev of true threat encountered by the second system due to the fact that the first system had already successfully screened out most items that did not contain a true threat Overall the tiered system exhibits a more nearly optimal balance of Pd Pfa NPV and PPV than either of the two systems alone

The first system is the notional air cargo screening system discussed previshyously Although this system exhibits good performance from the developerrsquos perspective (high Pd and low Pfa) it exhibits conflicting performance from the operatorrsquos perspective (high NPV but low PPV) Rather than using this system to classify items as either ldquoAlarm (Threat)rdquo or ldquoNo Alarm (No Threat)rdquo the operator can use this system to screen items as either ldquoCue Second System (Maybe Threat)rdquo or ldquoDo Not Cue Second System (No Threat)rdquo Of the 10100 items that passed through the first system 590 were classified as ldquoCue Second System (Maybe Threat)rdquo while 9510 were classified as ldquoNo Alarm (No Threat)rdquo The first systemrsquos extremely high

NPV (approximately equal to 1) means that the operator can rest assured that the lack of a cue correctly indicates the very low likelihood of a true threat Therefore any item that fails to elicit a cue can be loaded onto the airplane bypassing the second system and avoiding its unnecessary complexishyties and expense7 In contrast the first systemrsquos low PPV indicates that the operator cannot trust that a cue indicates a true threat Any item that elicits a cue from the first system may or may not contain a true threat and must therefore pass through the secshyond system for further analysis

Only 590 items elicited a cue from the first system and passed through the second system Ninety items contained a true threat while 500 items did not The second system declared an alarm for 108 items and did not declare an alarm for 482 items Comparing the itemsrsquo ground truth to the second systemrsquos alarms (or lack thereof) there were 88 TPs 2 FNs 20 FPs and 480 TNs On its own the second system exhibits a higher Pd and lower Pfa than the first system due to its increased complexity (and expense) In addition its PPV value is much higher The second systemrsquos higher PPV may be due to its higher complexity or may simply be due to the fact that the second system encounters a higher Prev of true threat among true clutter than the first system By the very nature in which the tiered system was assembled the first systemrsquos very high NPV indicates its strong ability to screen out most items that do not contain a true threat leaving only those questionable items for the second system to process Since the second system encounters

April 2017

only those items that are questionable it encounters a much higher Prev and therefore has the opportunity to exhibit higher PPV values The second system simply has less relative opportunity to generate false alarms

The utility of the tiered system must be considered in light of its cost

The utility of the tiered system must be considered in light of its cost In some cases the PM may decide that the first system is not needed since the second more complex system can exhibit the desired Pd Pfa PPV and NPV values on its own In that case the PM may choose to abandon the first sysshytem and pursue a single-tier approach based solely on the second system In other cases the added complexity of the second system may require a large increase in resources for its operation and maintenance In these cases the PM may opt for the tiered approach in which use of the first system reduces the number of items that must be processed by the second system reducing the additional resources needed to operate and maintain the second system to a level that may balance out the increase in resources needed to operate and maintain a tiered approach

To consider the utility of the tiered system its performance as a whole must be assessed in addition to the performance of each of the two individual systems that compose it As with any individual system Pd Pfa PPV and NPV can be calculated for the tiered system overall These calculations must be based on all items encountered by the tiered system as a whole taking care not to double count those TP1 and FP1 items from the first tier that pass to the second

TP2 88Pd= = = 088 (compared to 1 for a perfect system) (6) TP2 + (FN1 + FN2) 88 + (10 + 2)

FP2 20Pfa= = asymp 0 (compared to 0 for a perfect system) (7) FP2 + (TN1 + TN2) 20 + (9500 + 480)

(TN1 + TN2) (9500 + 480) NPV = = asymp 1 (compared to 1 for a perfect (8) (TN1 + TN2) + (FN1 + FN2) (9500 + 480) + (10 + 2)

system)

TP2 88PPV = = = 081 (compared to 1 for a perfect system) (9) TP2 + FP2 88 + 20

Overall the tiered system exhibits good8 performance from the developerrsquos perspective Equation 6 shows that of all items that contained a true threat (TP2 + (FN1 + FN2) = 88 + (10 + 2) = 100) a large subset (TP2 = 88) correctly caused an alarm resulting in an overall value of Pd = 088 The PM can conclude that 88 of items containing a true threat correctly led to a final alarm from the tiered system as a whole Although this overall Pd is slightly lower than the Pd of each of the two individual systems the overall value is still close to the value of 1 for a perfect system9 and may (or may not) be considered acceptable within the capability requirements for the envisioned CONOPS Similarly Equation 7 shows that of all items that did not contain a true threat (FP2 + (TN1 + TN2) = 20 + (9500 + 480) = 10000) only a very small subset (FP2 = 20) incorrectly caused an alarm leading to an overall value of Pfa asymp 0 Approximately 0 of items not containing a true threat caused a false alarm

The tiered system also exhibits good10 overall performance from the opershyatorrsquos perspective Equation 8 shows that of all items that did not cause an alarm ((TN1 + TN2) + (FN1 + FN2) = (9500 + 480) + (10 + 2) = 9992) a very large subset ((TN1 + TN2) = (9500 + 480) = 9980) correctly turned out not to contain a true threat resulting in an overall value of NPV asymp 1 The operator could rest assured that a threat was highly unlikely in the absence of a final alarm More interesting though is the overall PPV value Equation 9 shows that of all items that did indeed cause a final alarm ((TP2 + FP2) = (88 + 20) = 108) a large subset (TP2 = 88) correctly turned out to contain a true threatmdash these alarms were not false These counts resulted in an overall value of PPV = 081 much closer to the 1 value of a perfect system and much higher than the PPV of the first system alone11 When a final alarm was declared the operator could trust that a true threat was indeed present since overall the tiered system did not ldquocry wolfrdquo very often

Of course the PM must compare the overall performance of the tiered sysshytem to capability requirements in order to assess its appropriateness for the envisioned mission (DoD 2015 DHS 2008) The overall values of Pd = 088 Pfa asymp 0 NPV asymp 1 and PPV = 081 may or may not be adequate once these values are compared to such requirements Statistical tests can determine whether the overall values of the tiered system are significantly less than required (Fleiss Levin amp Paik 2013) Requirements should be set for all four metrics based on the envisioned mission Setting metrics for only Pd and Pfa effectively ignores the operatorrsquos view while setting metrics for only PPV and NPV effectively ignores the developerrsquos view12 One may argue that only the operatorrsquos view (PPV and NPV) must be quantified as capability requirements However there is value in also retaining the developerrsquos view

April 2017

(Pd and Pfa) since Pd and Pfa can be useful when comparing and contrasting the utility of rival systems with similar PPV and NPV values in a particular mission Setting the appropriate requirements for a particular mission is a complex process and is beyond the scope of this article

Threat Detection Versus Threat Classification

Unfortunately all four performance metrics cannot be calculated for some threat detection systems In particular it may be impossible to calshyculate Pfa and NPV This is due to the fact that the term ldquothreat detection systemrdquo can be a misnomer because it is often used to refer to threat detecshytion and threat classification systems Threat classification systems are those that are presented with a set of predefined discrete items The systemrsquos task is to classify each item as either ldquoAlarm (Threat)rdquo or ldquoNo Alarm (No Threat)rdquo The notional air cargo screen ing system discussed in this article is actually an example of a threat classification system despite the fact that the author has colloquially referred to it as a threat detection system throughout the first half of this article In contrast genuine threat detection systems are those that are not presented with a set of predefined discrete items The systemrsquos task is first to detect the discrete items from a continuous stream of data and then to classify each detected item as either ldquoAlarm (Threat)rdquo or ldquoNo Alarm (No Threat)rdquo An example of a genuine threat detection system is the notional counter-IED system illustrated in Figure 4

FIGURE 4 NOTIONAL COUNTER IED SYSTEM

Direction of Travel

Convoy

NOTIONAL CountershyIED System

Note Several items are buried in a road often traveled by a US convoy Some items are IEDs (orange stars) while others are simply rocks trash or other discarded items The system continuously collects data while traveling over the road ahead of the convoy and declares one alarm (red exclamation point) for each location at which it detects a buried IED All locations for which the system declares an alarm are further examined with robotic systems (purple arm) operated remotely by trained personnel In contrast all parts of the road for which the system does not declare an alarm are left unexamined and are directly traveled over by the convoy

April 2017

This issue is more than semantics Proper labeling of a systemrsquos task helps to ensure that the appropriate performance metrics are used to assess the system In particular while Pfa and NPV can be used to describe threat classification systems they cannot be used to describe genuine threat detecshytion systems For example Equation 2 showed that Pfa depends on FP and TN counts While an FP is a true clutter item that incorrectly caused an alarm a TN is a true clutter item that correctly did not cause an alarm FPs and TNs can be counted for threat classification systems and used to calcushylate Pfa as described earlier for the notional air cargo screening system

This issue is more than semantics Proper labeling of a systemrsquos task helps to ensure that the appropriate performance metrics are used to assess the system

This story changes for genuine threat detection systems however While FPs can be counted for genuine threat detection systems TNs cannot Therefore while Pd and PPV can be calculated for genuine threat detection systems Pfa and NPV cannot since they are based on the TN count For the notional counter-IED system an FP is a location on the road for which a true IED is not buried but for which the system incorrectly declares an alarm Unfortunately a converse definition for TNs does not make sense How should one count the number of locations on the road for which a true IED is not buried and for which the system correctly does not declare an alarm That is how often should the system get credit for declaring nothing when nothing was truly there To answer these TN-related questions it may be possible to divide the road into sections and count the number of sections for which a true IED is not buried and for which the system correctly does not declare an alarm However such a method simply converts the counter-IED detection problem into a counter-IED classification problem in which disshycrete items (sections of road) are predefined and the systemrsquos task is merely to classify each item (each section of road) as either ldquoAlarm (IED)rdquo or ldquoNo Alarm (No IED)rdquo This method imposes an artificial definition on the item (section of road) under classification How long should each section of road be Ten meters long One meter long One centimeter long Such definitions can be artificial which simply highlights the fact that the concept of a TN does not exist for genuine threat detection systems

Therefore PMs often rely on an additional performance metric for genuine threat detection systemsmdashthe False Alarm Rate (FAR) FAR can often be confused with both Pfa and PPV In fact documents within the defense and homeland security communities can erroneously use two or even all three of these terms interchangeably In this article however FAR refers to the number of FPs processed per unit time interval or unit geographical area or distance (depending on which metricmdashtime area or distancemdashis more salient to the envisioned CONOPS)

FAR = FP total time

FAR = FP total area

FAR = FP total distance

For example Equation 10c shows that one could count the number of FPs processed per meter as the notional counter-IED system travels down the road In that case FAR would have units of m-1 In contrast Pd Pfa PPV and NPV are dimensionless quantities FAR can be a useful performance metric in situations for which Pfa cannot be calculated (such as for genuine threat detection systems) or for which it is prohibitively expensive to conduct a test to fill out the full 2 times 2 confusion matrix needed to calculate Pfa

Conclusions Several metrics can be used to assess the performance of a threat detecshy

tion system Pd and Pfa summarize the developerrsquos view of the system quantifying how much of the truth causes alarms In contrast PPV and NPV summarize the operatorrsquos perspective quantifying how many alarms turn out to be true The same system can exhibit good values for Pd and Pfa during testing but poor PPV values during operational use PMs can still make use of the system as part of a tiered system that overall exhibits better performance than each individual system alone

April 2017

References Altman D G amp Bland J M (1994a) Diagnostic tests 1 Sensitivity and specificity

British Medical Journal 308(6943) 1552 doi101136bmj30869431552 Altman D G amp Bland J M (1994b) Diagnostic tests 2 Predictive values British

Medical Journal 309(6947) 102 doi101136bmj3096947102 Bliss J P Gilson R D amp Deaton J E (1995) Human probability matching behavior

in response to alarms of varying reliability Ergonomics 38(11) 2300ndash2312 doi10108000140139508925269

Cushman J H (1987 June 21) Making arms fighting men can use The New York Times Retrieved from httpwwwnytimescom19870621businessmakingshyarms-fighting-men-can-usehtml

Daniels D J (2006) A review of GPR for landmine detection Sensing and Imaging An International Journal 7(3) 90ndash123 Retrieved from httplinkspringercom article1010072Fs11220-006-0024-5

Department of Defense (2015 January 7) Operation of the defense acquisition system (Department of Defense Instruction [DoDI] 500002) Washington DC Office of the Under Secretary of Defense for Acquisition Technology and Logistics Retrieved from httpbbpdaumildocs500002ppdf

Department of Homeland Security (2008 November 7) Acquisition instruction guidebook (DHS Publication No 102-01-001 Interim Version 19) Retrieved from httpwwwit-aacorgimagesAcquisition_Instruction_102-01-001_-_Interim_ v19_dtd_11-07-08pdf

Department of Homeland Security (2016 March 30) Transportation systems sector Retrieved from httpswwwdhsgovtransportation-systems-sector

Fleiss J L Levin B amp Paik M C (2013) Statistical methods for rates and proportions (3rd ed) Hoboken NJ John Wiley

Getty D J Swets J A Pickett R M amp Gonthier D (1995) System operator response to warnings of danger A laboratory investigation of the effects of the predictive value of a warning on human response time Journal of Experimental Psychology Applied 1(1) 19ndash33 doi1010371076-898X1119

L3 Communications Cyterra (2012) ANPSS-14 mine detection Orlando FL Author Retrieved from httpcyterracomproductsanpss14htm

L3 Communications Security amp Detection Systems (2011) Fact sheet Examiner 3DX explosives detection system Woburn MA Author Retrieved from httpwww sdsl-3comcomformsEnglish-pdfdownloadhtmDownloadFile=PDF-13

L3 Communications Security amp Detection Systems (2013) Fact sheet Air cargo screening solutions Regulator-qualified detection systems Woburn MA Author Retrieved from httpwwwsdsl-3comcomformsEnglish-pdfdownload htmDownloadFile=PDF-50

L3 Communications Security amp Detection Systems (2014) Fact sheet Explosives detection systems Regulator-approved checked baggage solutions Woburn MA Author Retrieved from httpwwwsdsl-3comcomformsEnglish-pdfdownload htmDownloadFile=PDF-17

Niitek (nd) Counter IED | Husky Mounted Detection System (HMDS) Sterling VA Author Retrieved from httpwwwniitekcom~mediaFilesNNiitek documentshmdspdf

Oldham J (2006 October 3) Outages highlight internal FAA rift The Los Angeles Times Retrieved from httparticleslatimescom2006oct03localme-faa3

Parasuraman R (1997) Humans and automation Use misuse disuse abuse Human Factors 39(2) 230ndash253 doi101518001872097778543886

Powers D M W (2011) Evaluation From precision recall and F-measure to ROC informedness markedness amp correlation Journal of Machine Learning Technologies 2(1) 37ndash63

Scheaffer R L amp McClave J T (1995) Conditional probability and independence Narrowing the table In Probability and statistics for engineers (4th ed pp 85ndash92) Belmont CA Duxbury Press

Stuart R (1987 January 8) US cites Amtrak for not conducting drug tests The New York Times Retrieved from httpwwwnytimescom19870108usus-citesshyamtrak-for-not-conducting-drug-testshtml

Transportation Security Administration (2013) TSA air cargo screening technology list (ACSTL) (Version 84 as of 01312013) Washington DC Author Retrieved from httpwwwcargosecuritynlwp-contentuploads201304nonssi_ acstl_8_4_jan312013_compliantpdf

Wilson J N Gader P Lee W H Frigui H and Ho K C (2007) A large-scale systematic evaluation of algorithms using ground-penetrating radar for landmine detection and discrimination IEEE Transactions on Geoscience and Remote Sensing 45(8) 2560ndash2572 doi101109TGRS2007900993

Urkowitz H (1967) Energy detection of unknown deterministic signals Proceedings of the IEEE 55(4) 523ndash531 doi101109PROC19675573

US Army (nd) PdM counter explosive hazard Countermine systems Picatinny Arsenal NJ Project Manager Close Combat Systems SFAE-AMO-CCS Retrieved from httpwwwpicaarmymilpmccspmcountermineCounterMineSys htmlnogo02

Endnotes 1 PMs must determine what constitutes a ldquogoodrdquo performance For some

systems operating in some scenarios Pd = 090 is considered ldquogoodrdquo since only 10 FNs out of 100 true threats is considered an acceptable risk In other cases Pd

= 090 is not acceptable Appropriately setting a systemrsquos capability requirements calls for a frank assessment of the likelihood and consequences of FNs versus FPs and is beyond the scope of this article

2 Statistical tests can determine whether the systemrsquos value is significantly different from the perfect value or the capability requirement (Fleiss Levin amp Paik 2013)

3 Ibid

4 Ibid

5 Ibid

6 Conversely when Prev is high threat detection systems often exhibit poor values for NPV even while exhibiting excellent values for Pd Pfa and PPV Such cases are not discussed in this article since fewer scenarios in the DoD and DHS involve a high prevalence of threat among clutter

7 PMs must decide whether the 10 FNs from the first system are acceptable

April 2017

with respect to the tiered systemrsquos capability requirements since the first systemrsquos FNs will not have the opportunity to pass through the second system and be found Setting capability requirements is beyond the scope of this article

8 PMs must determine what constitutes a ldquogoodrdquo performance when setting the capability requirements for the tiered system

9 Statistical tests can show which differences are statistically significant (Fleiss et al 2013) while subject matter expertise can determine which differences are operationally significant

10 Once again PMs must determine what constitutes a ldquogoodrdquo performance when setting the capability requirements for the tiered system

11 Once again statistical tests can show which differences are statistically significant (Fleiss et al 2013) while subject matter expertise can determine which differences are operationally significant

12 All four of these metrics are correlated since all four metrics depend on the systemrsquos threshold for alarm For example tuning a system to lower its alarm threshold will increase its Pd at the cost of also increasing its Pfa Thus Pd cannot be considered in the absence of Pfa and vice versa To examine this correlation Pd and Pfa are often plotted against each other while the systemrsquos alarm threshold is systematically varied creating a Receiver-Operating Characteristic curve (Urkowitz 1967) Similarly lowering the systemrsquos alarm threshold will also affect its PPV To explore the correlation between Pd and PPV these metrics can also be plotted against each other while the systemrsquos alarm threshold is systematically varied in order to form a Precision-Recall curve (Powers 2011) (Note that PPV and Pd are often referred to as Precision and Recall respectively in the information retrieval community [Powers 2011] Also Pd and Pfa are often referred to as Sensitivity and One Minus Specificity respectively in the medical community [Altman amp Bland 1994a]) Furthermore although Pd and Pfa do not depend upon Prev PPV and NPV do Therefore PMs must take Prev into account when setting and testing system requirements based on PPV and NPV Such considerations can be done in a cost-effective way by designing the test to have an artificial prevalence of 05 and then calculating PPV and NPV from the Pd and Pfa values calculated during the test and the more realistic Prev value estimated for operational settings (Altman amp Bland 1994b)

Biography

Dr Shelley M Cazares is a research staff memshyber at the Institute for Defense Analyses (IDA) Her research involves machine learning and physshyiology to reduce collateral damage in the military theater Before IDA she was a principal research scientist at Boston Scientific Corporation where she designed algorithms to diagnose and treat cardiac dysfunction with implantable medical devices She earned her BS from MIT in EECS and PhD from Oxford in Engineering Science

(E-mail address scazaresidaorg)

Within Army aviation a recurring problem is too many maintenance man-hour (MMH) requirements and too few MMH available This gap is driven by several reasons among them an inadequate number of soldier maintainers inefficient use of assigned soldier maintainers and political pressures to reduce the number of soldiers deployed to combat zones For years contractors have augmented the Army aviation maintenance force Army aviation leadership is working to find the right balance between when it uses soldiers versus contractors to service its fleet of aircraft No stan-dardized process is now in place for quantifying the MMH gap This article

ARMY AVIATION Quantifying the Peacetime and Wartime

MAINTENANCE MAN-HOUR GAPS

CW5 Donald L Washabaugh Jr USA (Ret) and Mel Adams LTC William Bland USA (Ret)

describes the development of an MMH Gap Calculator a tool to quantify the gap in Army aviation It also describes how the authors validated the tool assesses the current and future aviation MMH gap and provides a number of conclusions and recommendations The MMH gap is real and requires contractor support

DOI httpsdoiorg1022594dau16-7512402 Keywords aviation maintenance manpower contractor gap

Quantifying Maintenance Man-Hour Gaps httpwwwdaumil

The Army aviation community has always counted on well-trained US Army helicopter mechanics to maintain Army aircraft Unfortunately a problem exists with too many maintenance man-hour (MMH) requirements and too few MMH available (Nelms 2014 p 1) This disconnect between the amount of maintenance capability available and the amount of mainteshynance capability required to keep the aircraft flying results in an MMH gap which can lead to decreased readiness levels and increased mission risk

The greatest resource available to the aviation commander is the time assigned soldier maintainers are actually turning wrenches on their aircraft

In order to mitigate this MMH gap commanders have hired contractors to augment soldier maintainers and increase the amount of maintenance performed on aircraft for many years (Evans 1997 p 15) This MMH gap can be driven by many reasons among them an inadequate number of soldier maintainers assigned to aviation units inefficient use of assigned soldier maintainers and political pressures to reduce the size of the soldier footprint during deployments Regardless of the reason for the MMH gap the Armyrsquos primary challenge is not managing the cost of the fleet or flying hour program but achieving the associated maintenance challenge and managing the MMH gap to ensure mission success

The purposes of this exploratory article are to (a) confirm a current MMH gap exists (b) determine the likely future MMH gap (c) confirm any requirement for contractor support needed by the acquisition program management and force structure communities and (d) prototype a tool that could simplify and standardize calculation of the MMH gap and proshyvide a decision support tool that could support MMH gap-related trade-off analyses at any level of organization

Background The number of soldier maintainers assigned to a unit is driven by its

Modified Table of Organization and Equipment (MTOE) These MTOEs are designed for wartime maintenance requirements but the peacetime environment is differentmdashand in many cases more taxing on the mainteshynance force There is a base maintenance requirement even if the aircraft are not flown however many peacetime soldier training tasks and off-duty

April 2017

distractions significantly reduce the amount of time soldier maintainers are actually available to work on aircraft (Kokenes 1987 p 9) Another MTOE-related issue contributing to the MMH gap is that increasing airshycraft complexity stresses existing maintenance capabilities and MTOEs are not always updated to address these changes in MMH requirements in a timely manner Modern rotary wing aircraft are many times more comshyplex than their predecessors of only a few years ago and more difficult to maintain (Keirsey 1992 p 2) In 1991 Army aircraft required upwards of 10 man-hours of maintenance time for every flight hour (McClellan 1991 p 31) while today the average is over 16 man-hours for every flight hour

The greatest resource available to the aviation commander is the time assigned soldier maintainers are actually turning wrenches on their aircraft These productive available man-hours are used to conduct both scheduled and unscheduled maintenance (Washabaugh 2016 p 1) Unfortunately too many distractors compete for time spent working on aircraft among them details additional duties and training The goal for soldier direct proshyductive time in peacetime is 45 hours a day (Brooke 1998 p 4) but studies have shown that aviation mechanics are typically available for productive ldquowrench turningrdquo work only about 31 percent of an 8-hour peacetime day which equates to under 3 hours per day (Kokenes 1987 p 12) Finding the time to allow soldiers to do this maintenance in conjunction with other duties is a great challenge to aviation leaders at every level (McClellan 1991 p 31) and it takes command emphasis to make it happen Figure 1 summarizes the key factors that diminish the number of wrench turning hours available to soldier maintainers and contribute to the MMH gap

FIGURE 1 MMH GAP CAUSES

MMH Gap Causes

bull Assigned Manpower Shortages bull Duty Absences

mdash Individual Professional Development Training mdash Guard DutySpecial Assignments mdash LeaveHospitalizationAppointments

bull NonshyMaintenance Tasks mdash Mandatory Unit Training mdash FormationsTool Inventories mdash Travel to and from AirfieldMeals

MMH Gap = Required MMHs Available MMHs

Required MMHs

Available MMHs

Assigned Manpower Shortages

NonshyMaintenance Tasks

Duty Absences

Recently ldquoBoots on the Groundrdquo (BOG) restrictionsmdashdesigned to reduce domestic political riskmdashhave constrained the number of soldiers we can deploy for combat operations (Robson 2014 p 2) The decision is usually to maximize warfighters and minimize maintainers to get the most ldquoBang for the Buckrdquo Despite the reduction in soldier maintainers a Combat Aviation Brigade (CAB) is still expected to maintain and fly its roughly 100 aircraft (Gibbons-Neff 2016 p 1) driving a need to deploy contract maintainers to perform necessary aircraft maintenance functions (Judson 2016 p 1) And these requirements are increasing over time as BOG constraints get tighter For example a total of 390 contract maintainers deployed to maintain aircraft for the 101st and 82nd CABs in 2014 and 2015 while 427 contract maintainers deployed to maintain aircraft for the 4th CAB in 2016 (Gibbons-Neff 2016 p 1)

The Department of Defense (DoD) has encouraged use of Performance Based Logistics (PBL) (DoD 2016) Thus any use of contract support has been and will be supplemental rather than a true outsourcing Second unlike the Navy and US Air Force the Army has not established a firm performance requirement to meet with a PBL vehicle perhaps because the fleet(s) are owned and managed by the CABs The aviation school at Fort Rucker Alabama is one exception to this with the five airfields and fleets

there managed by a contractor under a hybrid PBL contract vehicle Third the type of support provided by contractors across the

world ranges from direct on-airfield maintenance to off-site port operations downed aircraft

recovery depot repairs installation of modifications repainting of aircraft etc Recent experience with a hybrid PBL contract with multiple customers and sources of funding shows that manshyaging the support of several contractors is very difficult From 1995ndash2005 spare

parts availability was a key determinant of maintenance turnaround times But now

with over a decade of unlimited budgets for logistics the issue of spare parts receded

at least temporarily Currently mainshytenance turnaround times are driven

primarily by (a) available labor (b) depot repairs and (c) modifications installed concurrently with reset or

April 2017

phase maintenance This article and the MMH Gap Calculator address only the total requirement for labor hours not the cost or constraints in executing maintenance to a given schedule

The Army is conducting a holistic review of Army aviation and this review will include an assessment of the level of contractor maintenance for Army aviation (McBride 2016 p 1) Itrsquos important to understand the level and mix of mission functions and purpose of contract maintainers in order to find the right balance between when soldiers or contract maintainers are used (Judson 2016 p 2) A critical part of this assessment is understanding the actual size of the existing MMH gap Unfortunately there is no definitive approach for doing so and every Army aviation unit estimates the difference between the required and available MMHs using its own unique heuristic or ldquorule of thumbrdquo calcushylations making it difficult to make an Army-wide assessment

Being able to quantify the MMH gap will help inform the development of new or supplementary MTOEs that provide adequate soldier maintainers Being able to examine the impact on the MMH gap of changing various nonmaintenance requirements will help commanders define more effective manpower management policies Being able to determine an appropriate contract maintainer package to replace nondeployed soldier maintainers will help ensure mission success To address these issues the US Army Program Executive Office (PEO) Aviation challenged us to develop a decishysion support tool for calculating the size of the MMH gap that could also support performing trade-off analyses like those mentioned earlier

Approach and Methodology Several attempts have been made to examine the MMH gap problem in

the past three of which are described in the discussion that follows

McClellan conducted a manpower utilization analysis of his aviation unit to identify the amount of time his soldier maintainers spent performing nonmaintenance tasks His results showed that his unit had the equivashylent of 99 maintainers working daily when 196 maintainers were actually assignedmdashabout a 51 percent availability factor (McClellan 1991 p 32)

Swift conducted an analysis of his maintenance personnel to determine if his MTOE provided adequate soldier maintainers He compared his unitrsquos required MMH against the assigned MMH provided by his MTOE which resulted in an annual MMH shortfall of 22000 hours or 11 contactor man-year equivalents (CME) His analysis did not include the various distractors

described earlier in this article so the actual MMH gap is probably higher (Swift 2005 p 2) Even though his analysis was focused on vehicle mainshytenance some of the same issues plague aviation maintenance

Mead hypothesized that although more sophisticated aviation systems have been added to the fleet the workforce to maintain those systems has not increased commensurately He conducted an analysis of available MMH versus required MMH for the Armyrsquos UH-60 fleet and found MMH gaps for a number of specific aviation maintenance military occupational specialties during both peacetime and wartime (Mead 2014 pp 14ndash23)

The methodology we used for developing our MMH Gap Calculator was to compare the MMH required of the CAB per month against the MMH available to the CAB per month and identify any shortfall The approaches described previously followed this same relatively straightforward matheshymatical formula but the novelty of our approach is that none of these other approaches brought all the pieces together to customize calculation of the MMH gap for specific situations or develop a decision support tool that examined the impact of manpower management decisions on the size of the MMH gap

Our approach is consistent with A rmy R e g u l a t i o n 7 5 0 -1 A r m y M a t e r i e l Maintenance Policy which sets forth guidshyance on determining tactical maintenance augmentation requirements for military mechanics and leverages best practices from Army aviation unit ldquorule of thumbrdquo MMH gap calculations We coordinated with senior PEO Aviation US Army Aviation and Missile Life Cycle Management Command (AMCOM) and CAB subject matter experts (SMEs) and extracted applicable data eleshyments from the official MTOEs for light medium and heavy CAB configurations Additionally we incorporated approved Manpower Requirements Criteria (MARC) data and other official references (Table 1)

April 2017

and established the facts and assumptions shown in Table 2 to ensure our MMH Gap Calculator complied with regulatory requirements and was consistent with established practices

TABLE 1 KEY AVIATION MAINTENANCE DOCUMENTS

Department of the Army (2015) Army aviation (Field Manual [FM] 3-04) Washington DC Office of the Secretary of the Army

Department of the Army (2007) Attack reconnaissance helicopter operations (FM 3-04126) Washington DC Office of the Secretary of the Army

Department of the Army (2007) Aviation brigades (FM 3-04111) Washington DC Office of the Secretary of the Army

Department of the Army (2007) Utility and cargo helicopter operations (FM 3-04113) Washington DC Office of the Secretary of the Army

Department of the Army (2014) Functional userrsquos manual for the Army Maintenance Management System-Aviation (Department of the Army Pamphlet [DA PAM] 738-751) Washington DC Office of the Secretary of the Army

Department of the Army (2013) Army materiel maintenance policy (Army Regulation [AR] 750-1) Washington DC Office of the Secretary of the Army

Department of the Army (2014) Flight regulations (AR 95-1) Washington DC Office of the Secretary of the Army

Department of the Army (2006) Manpower management (AR 570-4) Washington DC Office of the Secretary of the Army

Department of the Army (2013) Aircrew training manual AH-64D (Training Circular [TC] 3-0442) Washington DC Office of the Secretary of the Army

Department of the Army (2013) Aircrew training manual CH-47DF (TC 3-0434) Washington DC Office of the Secretary of the Army

Department of the Army (2013) Aircrew training manual OH-58D (TC 3-0444) Washington DC Office of the Secretary of the Army

Department of the Army (2012) Aircrew training manual UH-60 (TC 3-0433) Washington DC Office of the Secretary of the Army

Department of the Army (2010) Army aviation maintenance (TC 3-047) Washington DC Office of the Secretary of the Army

Force Management System Website (Table of Distribution and Allowances [TDA] Modified Table of Organization and Allowances [MTOE] Manpower Requirements Criteria [MARC] Data) In FMSWeb [Secure database] Retrieved from httpsfmswebarmymil

TABLE 2 KEY FACTS AND ASSUMPTIONS FOR THE MMH GAP MODEL

Factor Reference FactAssumption Number of Aircraft MTOE Varies by unit type assumes

100 fill rate

Number of Flight MTOE Varies by unit type assumes 0 Crews turnover

Number of Maintainers MTOE Varies by unit type assumes all 15-series E6 and below possess minimum school house maintenance skills and perform maintenance tasks

MMH per FH MARC Varies by aircraft type

Military PMAF AR 570-4 122 hours per month

Contract PMAF PEO Aviation 160 hours per month

ARI Plus Up AMCOM FSD 45 maintainers per CAB

Crew OPTEMPO Varies by scenario

MTOE Personnel Fill Varies by scenario

Available Varies by scenario

DLR Varies by scenario

Note AMCOM FSD = US Army Aviation and Missile Life Cycle Management Command Field Support Directorate AR = Army Regulation ARI = Aviation Restructuring Initiative CAB = Combat Aviation Brigade DLR = Direct Labor Rate FH = Flying Hours MARC = Manpower Requirements Criteria MMH = Maintenance Man-Hour MTOE = Modified Table of Organization and Equipment OPTEMPO = Operating Tempo PEO = Program Executive Office PMAF = Peacetime Mission Available Factor

We calculate required MMH by determining the number of flight hours (FH) that must be flown to meet the Flying Hour Program and the associshyated MMH required to support each FH per the MARC data Since several sources (Keirsey 1992 p 14 Toney 2008 p 7 US Army Audit Agency 2000 p 11) and our SMEs believe the current MARC process may undershystimate the actual MMH requirements our calculations will produce a conservative ldquobest caserdquo estimate of the required MMH

We calculate available MMH by leveraging the basic MTOE-based conshystruct established in the approaches described previously and added several levers to account for the various effects that reduce available MMH The three levers we implemented include percent MTOE Fill (the percentage of MTOE authorized maintainers assigned to the unit) percent Availability (the percentage of assigned maintainers who are actually present for duty) and Direct Labor Rate or DLR (the percentage of time spent each day on

April 2017

maintenance tasks) An example MMH Gap Calculation is presented in Figure 2 to facilitate understanding of our required MMH and available MMH calculations

FIGURE 2 SAMPLE MONTHLY CAB MMH GAP CALCULATION

Required MMHs Numbertype of aircraft authorized x Percent Aircraft Fill x Aircraft OPTEMPO x Maintenance Hours required per Flight Hour

Ex) 113 acft x 100 x 1856 FHacft x 15 MMHFH = 31462 MMHs

Available MMHs Numbertype of maintainers authorized x Percent Personnel Fill x Maintainer Availability x Direct Labor Rate (DLR) x Number of Maintenance Hours per maintainer

Ex) 839 pers x 80 x 50 x 60 x 122 MMHpers = 24566 MMHs

MMH Gap = Required MMHs Available MMHs = 6896 MMHs

Defined on per monthly basis

When the available MMH is less than the required MMH we calculate the gap in terms of man-hours per month and identify the number of military civilian or contract maintainers required to fill the shortage We calculate the MMH gap at the CAB level but can aggregate results at brigade comshybat team division corps or Army levels and for any CAB configuration Operating Tempo (OPTEMPO) deployment scenario or CAB maintenance management strategy

Validating the MMH Gap Calculator Based on discussions with senior PEO Aviation AMCOM and CAB

SMEs we established four scenarios (a) Army Doctrine (b) Peacetime (c) Wartime without BOG Constraint and (d) Wartime with BOG Constraint We adjusted the three levers described previously to reflect historical pershysonnel MTOE fill rates maintainer availability and DLR for a heavy CAB under each scenario and derived the following results

bull Army Doctrine Using inputs of 90 percent Personnel MTOE Fill 60 percent Availability and 60 percent DLR no MMH gap exists Theoretically a CAB does not need contractor support and can maintain its fleet of aircraft with only organic mainshytenance assets

bull Peacetime Adjusting the inputs to historical peacetime CAB data (80 percent Personnel MTOE Fill 50 percent Availability and 60 percent DLR) indicates that a typical heavy CAB would require 43 CMEs to meet MMH requirements

bull Wartime without BOG Constraint Adjusting the inputs to typical Wartime CAB data without BOG Constraints (95 Personnel MTOE Fill 80 percent Availability and 65 percent DLR) indicates that a heavy CAB would require 84 CMEs to meet MMH requirements

bull Wartime with BOG Constraint Adjusting the inputs to typical Wartime CAB data with BOG Constraints (50 percent Personnel MTOE Fill 80 percent Availability and 75 percent DLR) indicates that a heavy CAB would require 222 CMEs to meet MMH requirements

The lever settings and results of these scenarios are shown in Table 3 Having served in multiple CABs in both peacetime and wartime as mainshytenance officers at battalion brigade division and Army levels the SMEs considered the results shown in Table 3 to be consistent with current conshytractor augmentations and concluded that the MMH Gap Calculator is a valid solution to the problem stated earlier

April 2017

TABLE 3 MMH GAP MODEL VALIDATION RESULTS FOR FOUR SCENARIOS

Current Army Peacetime Wartime Wartime MTOE and Doctrine (Heavy CAB) wo BOG w BOG

Organization (Heavy CAB) (Heavy CAB) (Heavy CAB) Personnel MTOE Fill Rate

90 80 95 50

Personnel Available 60 50 80 80 Rate

Personnel DLR 60 60 65 75

Monthly 0 6896 23077 61327 MMH Gap

CMEs to fill MMH Gap 0 43 84 222

FIGURE 3 CURRENT PEACETIME amp WARTIME AVIATION MMH GAPS BY MANPOWER FILL

800000

700000

600000

500000

400000

300000

200000

100000

100 90 80 70

Percent Manpower Fill

e of 1

Wartime 362330

489565

616800

113215

744034

151323

Peacetime 36999

To estimate lower and upper extremes of the current MMH gap we ran peacetime and wartime scenarios for the current Active Army aviation force consisting of a mix of 13 CABs in heavy medium and light configurations (currently five heavy CABs seven medium CABs and one light CAB) The results of these runs at various MTOE fill rates are shown in Figure 3

The estimate of the peacetime MMH gap for the current 13-CAB configurashytion is based on (a) 50 percent Availability (b) 60 percent DLR and (c) four levels of Personnel MTOE Fill from 70 percent to 100 percent As shown in Figure 3 the peacetime MMH gap ranges from 36999 to 151323 MMH per month across the current 13-CAB configuration depending on the Personnel MTOE fill rate The number of CMEs needed to address this gap ranges from 215 to 880 CMEs respectively

The estimate of the wartime MMH gap for the current 13-CAB configuration is based on (a) 80 percent Availability (b) 65 pershy

cent DLR and (c) four levels of Personnel MTOE Fill from 70 percent to 100 percent Figure 3 shows the wartime MMH gap

ranges from 362330 to 744034 MMH per month across the current 13-CAB configuration depending on the Personnel MTOE fill rate

The number of CMEs needed to address this gap ranges from 1839 to 3777 CMEs respectively

These CME requirements do not account for any additional program management support requirements In addition it is important to

note that the MMH gaps presented in Figure 3 are not intended to promote any specific planning

objective or strategy Rather these figures present realistic estimates of the MMH gap pursuant to historshy

ically derived settings OPTEMPO rates and doctrinal regulatory guidance on maintainer availability factors

and maintenance requirements In subsequent reviews SMEs val shyidated the MMH gap estimates based on multiple deployments managing

hundreds of thousands of flight hours during 25 to 35 years of service

Quantifying the Future Aviation MMH Gap To estimate the lower and upper extremes of the future MMH gap we

ran peacetime and wartime scenarios for the post-Aviation Restructuring Initiative (ARI) Active Army aviation force consisting of 10 heavy CABs These scenarios included an additional 45 maintainers per CAB as proshyposed by the ARI The results of these runs are shown in Figure 4

April 2017

FIGURE 4 FUTURE PEACETIME amp WARTIME AVIATION MMH GAPS (POST-ARI)

500000

450000

400000

350000

300000

250000

200000

150000

100000

100 90 80 70

e of 1

Wartime 124520

232550

340570

448600

Peacetime 0

The estimate of the peacetime MMH gap for the post-ARI 10-CAB conshyfiguration is based on (a) 50 percent Availability (b) 60 percent DLR and (c) four levels of Personnel MTOE Fill from 70 percent to 100 percent As shown in Figure 4 the peacetime MMH gap ranges from 0 to 88140 MMH per month across the post-ARI 10 CAB configuration The number of CMEs needed to address this gap ranges from 0 to 510 CMEs respectively

The estimate of the wartime MMH gap for the post-ARI 10-CAB configushyration is based on (a) 80 percent Availability (b) 65 percent DLR and (c) four levels of Personnel MTOE Fill from 70 percent to 100 percent Figure 4 shows the wartime MMH gap ranges from 124520 to 448600 MMH per month across the post-ARI 10-CAB configuration The number of CMEs needed to address this gap ranges from 630 to 2280 CMEs respectively As before these CME requirements do not account for any additional program management support requirements

Conclusions First the only scenario where no MMH gap occurs is under exact preshy

scribed doctrinal conditions In todayrsquos Army this scenario is unlikely Throughout the study we found no other settings to support individual and collective aviation readiness requirements without long-term CME support

during either Peacetime or Wartime OPTEMPOs With the proposed ARI plus-up of 45 additional maintainers per CAB the MMH gap is only parshytially addressed A large MMH gap persists during wartime even with a 100 percent MTOE fill rate and no BOG constraint and during peacetime if the MTOE fill rate drops below 100 percent

Second the four main drivers behind the MMH gap are OPTEMPO Personnel MTOE fill rate Availability rate and DLR rate The CAB may be able to control the last two drivers by changing management strategies or prioritizing maintenance over nonmaintenance tasks Unfortunately the CAB is unable to control the first two drivers

The only scenario where no MMH gap occurs is under exact prescribed doctrinal conditions In todayrsquos Army this scenario is unlikely

Finally the only real short-term solution is continued CME or Department of Army Civilian maintainer support to fill the ever-present gap These large MMH gaps in any configuration increase risk to unit readiness airshycraft availability and the CABrsquos ability to provide mission-capable aircraft Quick and easy doctrinal solutions to fill any MMH gap do not exist The Army can improve soldier technical skills lower the OPTEMPO increase maintenance staffing or use contract maintenance support to address this gap Adding more soldier training time may increase future DLRs but will lower current available MMH and exacerbate the problem in the short term Reducing peacetime OPTEMPO may lower the number of required MMHs but could result in pilots unable to meet required training hours to maintain qualification levels Increasing staffing levels is difficult in a downsizing force Thus making use of contractor support to augment organic CAB maintenance assets appears to be a very reasonable approach

Recommendations First the most feasible option to fill the persistent now documented

MMH gap is to continue using contract maintainers With centrally managed contract support efficiencies are gained through unity of effort providing one standard for airworthiness quality and safety unique to Army aviation The challenge with using contractors is to identify the

April 2017

appropriate number of support contractors and program management costs Results of this MMH Gap Calculator can help each CAB and the Army achieve the appropriate mix of soldier maintainers and contractor support

Second to standardize the calculation of annual MMH gaps and support requirements the Army should adopt a standardized approach like our MMH Gap Calculator and continuously improve planning and manageshyment of both soldier and contractor aviation maintenance at the CAB and division level

Third and finally the MMH Gap Calculator should be used to perform various trade-off analyses Aviation leaders can leverage the tool to project the impacts of proposed MMH mitigation strategies so they can modify policies and procedures to maximize their available MMH The Training and Doctrine Command can leverage the tool to help meet Design for Maintenance goals improve maintenance management training and inform MTOE development The Army can leverage the tool to determine the size of the contractor package needed to support a deployed unit under BOG constraints

Our MMH Gap Calculator should also be adapted to other units and main-tenance-intensive systems and operations including ground units and nontactical units While costs are not incorporated in the current version of the MMH Gap Calculator we are working to include costs to support budget exercises to examine the MMH gap-cost tradeoff

Acknowledgments The authors would like to thank Bill Miller and Cliff Mead for leveraging

their real-world experiences and insights during the initial development and validation of the model The authors would also like to thank Mark Glynn and Dusty Varcak for their untiring efforts in support of every phase of this project

References Note Data sources are referenced in Table 1

Brooke J L (1998) Contracting an alarming trend in aviation maintenance (Report No 19980522 012) Retrieved from Defense Technical Information Center Website httpwwwdticmildtictrfulltextu2a344904pdf

Department of Defense (2016) PBL guidebook A guide to developing performance-based arrangements Retrieved from httpbbpdaumildocsPBL_Guidebook_ Release_March_2016_finalpdf

Evans S S (1997) Aviation contract maintenance and its effects on AH-64 unit readiness (Masterrsquos thesis) (Report No 19971114 075) Retrieved from Defense Technical Information Center Website httpwwwdticmildtictrfulltextu2 a331510pdf

Gibbons-Neff T (2016 March 15) How Obamarsquos Afghanistan plan is forcing the Army to replace soldiers with contractors Washington Post Retrieved from https wwwwashingtonpostcomnewscheckpointwp20160601how-obamasshyafghanistan-plan-is-forcing-the-army-to-replace-soldiers-with-contractors

Judson J (2016 May 2) Use of US Army contract aircraft maintainers out of whack DefenseNews Retrieved from httpwwwdefensenewscomstorydefense show-dailyaaaa20160502use-army-contract-aircraft-maintainers-outshywhack83831692

Keirsey J D (1992) Army aviation maintenancemdashWhat is needed (Report No AD-A248 035) Retrieved from Defense Technical Information Center Website httpwwwdticmildtictrfulltextu2a248035pdf

Kokenes G P (1987) Army aircraft maintenance problems (Report No AD-A183shy396) Retrieved from Defense Technical Information Center Website httpwww dticmilcgi-binGetTRDocLocation=U2ampdoc=GetTRDocpdfampAD=ADA183396

McBride C (2016 August) Army crafts holistic review sustainment startegy for aviation InsideDefense Retrieved from httpngesinsidedefensecominsideshyarmyarmy-crafts-holistic-review-sustainment-strategy-aviation

McClellan T L (1991 December) Where have all the man-hours gone Army Aviation 40(12) Retrieved from httpwwwarmyaviationmagazinecomimagesarchive backissues199191_12pdf

Mead C K (2014) Aviation maintenance manpower assessment Unpublished briefing to US Army Aviation amp Missile Command Redstone Arsenal AL

Nelms D (2014 June) Retaking the role Rotor and Wing Magazine 48(6) Retrieved from httpwwwaviationtodaycomrwtrainingmaintenanceRetaking-the shyRole_82268html

Robson S (2014 September 7) In place of lsquoBoots on the Groundrsquo US seeks contractors for Iraq Stars and Stripes Retrieved from httpwwwstripescom in-place-of-boots-on-the-ground-us-seeks-contractors-for-iraq-1301798

Swift J B (2005 September) Field maintenance shortfalls in brigade support battalions Army Logistician 37(5) Retrieved from httpwwwaluarmymil alogissuesSepOct05shortfallshtml

Toney G W (2008) MARC data collectionmdashA flawed process (Report No AD-A479shy733) Retrieved from Defense Technical Information Center Website httpwww dticmilget-tr-docpdfAD=ADA479733

April 2017

US Army Audit Agency (2000) Manpower requirements criteriamdashMaintenance and support personnel (Report No A-2000-0147-FFF) Alexandria VA Author

Washabaugh D L (2016 February) The greatest assetndashsoldier mechanic productive available time Army Aviation 65(2) Retrieved from httpwww armyaviationmagazinecomindexphparchivenot-so-current969-the-greatest shyasset-soldier-mechanic-productive-available-time

Author Biographies

LTC William Bland USA (Ret) currently specializes in developing simulation models and decision support systems for defense clients at Booz Allen Hamilton LTC Bland spent 26 years in the Army primarily as an operations research analyst His past experience includes a tenure teaching Systems Engineering at the United States Military Academy LTC Bland holds a PhD from the University of Virginia

(E-mail address bland_williambahcom)

CW5 Donald L Washabaugh Jr USA (Ret) is currently employed by LMI as the Aviation Logistics and Airworthiness Sustainment liaishyson for TRADOC Capabilities Manager-Aviation Brigades (TCM-AB) working with the Global Combat Support System ndash Army (GCSS-A) Increment 2 Aviation at Redstone Arsenal Alabama He served 31 years in the Army with multiple tours in Iraq and Afghanistan as a mainshytenance officer at battalion brigade division and Army levels Chief Warrant Officer Washabaugh holds a Bachelor of Science from Embry Riddle Aeronautical University

(E-mail address donaldlwashabaughctrmailmil )

April 2017

Author Biographies

LTC William Bland USA (Ret) currently specializes in developing simulation models anddecision support systems for defense clients atBooz Allen Hamilton LTC Bland spent 26 yearsin the Army primarily as an operations researchanalyst His past experience includes a tenureteaching Systems Engineering at the United StatesMilitary Academy LTC Bland holds a PhD fromthe University of Virginia

CW5 Donald L Washabaugh Jr USA (Ret)is currently employed by LMI as the AviationLogistics and Airworthiness Sustainment liai-son for TRADOC Capabilities Manager-AviationBrigades (TCM-AB) working with the GlobalCombat Support System ndash Army (GCSS-A)Increment 2 Aviation at Redstone ArsenalAlabama He served 31 years in the Army withmultiple tours in Iraq and Afghanistan as a main-tenance officer at battalion brigade division andArmy levels Chief Warrant Officer Washabaughholds a Bachelor of Science from Embry RiddleAeronautical University

Dr Mel Adams a Vietnam-era veteran is curshyrently a Lead Associate for Booz Allen Hamilton Prior to joining Booz Allen Hamilton he retired from the University of Alabama in Huntsville in 2007 Dr Adams earned his doctorate in Strategic Management at the University of Tennessee-Knoxville He is a published author in several fields including modeling and simulation Dr Adams was the National Institute of Standards and Technology (NIST) ModForum 2000 National Practitioner of the Year for successes with comshymercial and aerospace defense clients

(E-mail address adams_melbahcom)

COMPLEX ACQUISITION REQUIREMENTS ANALYSIS Using a Systems Engineering Approach

Col Richard M Stuckey USAF (Ret) Shahram Sarkani and Thomas A Mazzuchi

The technology revolution over the last several decades has compounded system complexity with the integration of multispectral sensors and intershyactive command and control systems making requirements development more challenging for the acquisition community The imperative to start programs right with effective requirements is becoming more critical Research indicates the Department of Defense lacks consistent knowledge as to which attributes would best enable more informed trade-offs This research examines prioritized requirement attributes to account for program complexities using the expert judgement of a diverse and experienced panel of acquisition professionals from the Air Force Army Navy industry and additional government organizations This article provides a guide for todayrsquos acquisition leaders to establish effective and prioritized requirements for complex and unconstrained systems needed for informed trade-off decisions The results found the key attribute for unconstrained systems is ldquoachievablerdquo and verified a list of seven critical attributes for complex systems

DOI httpsdoiorg1022594dau16-7552402 Keywords Bradley-Terry methodology complex systems requirements attributes system of systems unconstrained systems

Complex Acquisition Requirements Analysis httpwwwdaumil

Recent Government Accountability Office (GAO) reports outline conshycerns with requirements development One study found programs with unachievable requirements cause program managers to trade away pershyformance and found that informed trade-offs between cost and capability establish better defined requirements (GAO 2015a 2015b) In another key report the GAO noted that the Department of Defense could benefit from ranking or prioritizing requirements based on significance (GAO 2011)

Establishing a key list of prioritized attributes that supports requirements development enables the assessment of program requirements and increases focus on priority attributes that aid in requirements and design trade-off decisions The focus of this research is to define and prioritize requirements attributes that support requirements development across a spectrum of system types for decision makers Some industry and government programs are becoming more connected and complex while others are geographically dispersed yet integrated thus creating the need for more concentrated approaches to capture prioritized requirements attributes

The span of control of the program manager can range from low programmatic authority to highly dependent systems control For example the program manager for a national emergency command and control center typically has low authority to influence cost schedule and performance at the local state and tribal level yet must enable a broader national unconstrained systems capability On the opposite end of the spectrum are complex dependent systems The F-35 Joint Strike Fighterrsquos program manager has highly dependent control of that program and the program is complex as DoD is building variants for the US Air Force Navy and Marine Corps as well as multishyple foreign countries

Complex and unconstrained sysshytems are becoming more prevalent There needs to be increased focus on complex and unconstrained systems requirements attributes development and prioritization to develop a full range of dynamic requirements for decision makers In our research we use the terms

April 2017

systems complex systems and unconstrained systems and their associated attributes All of these categories are explored developed and expanded with prioritized attributes The terms systems and complex systems are used in the acquisition community today We uniquely developed a new category called unconstrained systems and distinctively define complex systems as

Unconstrained System

A collection of component systems simple or complex that is managed operated developed funded maintained and sustained independently of its overarching principal system that creates a new capability

Complex System

A collection of large multifaceted and interrelated comshyponent systems that is dependent on the entirety of the principal system for management operations development funding maintenance and sustainment Complex systems are nondeterministic adaptive holistic and have nonlinear interfaces between attributes

We derived a common set of definitions for requirements systems unconshystrained systems and complex systems using an exhaustive list from government industry and standards organizations Using these definitions we then developed and expanded requirements attributes to provide a select group of attributes for the acquisition community Lastly experts in the field prioritized the requirements attributes by their respective importance

We used the Bradley-Terry (Bradley amp Terry 1952) methodology as amplishyfied in Cooke (1991) to elicit and codify the expert judgment to validate the requirements attributes This methodology using a series of repeatable surveys with industry government and academic experts applies expert judgment to validate and order requirements attributes and to confirm the attributes lists are comprehensive This approach provides an importshyant suite of valid and prioritized requirements attributes for systems unconstrained systems and complex systems for acquisition and systems engineering decision makersrsquo consideration when developing requirements and informed trade-offs

Terms Defined and Attributes Derived We performed a literature review from a broad base of reference mateshy

rial reports and journal articles from academia industry and government Currently a wide variety of approaches defines requirements and the various forms of systems For this analysis we settle on a single definition to comshyplete our research Using our definitions we further derive the requirements attributes for systems unconstrained systems and complex systems (American National Standards InstituteElectronic Industries Alliance [ANSIEIA] 1999 Ames et al 2011 Butterfield Shivananda amp Schwartz 2009 Chairman Joint Chiefs of Staff [CJCS] 2012 Corsello 2008 Customs and Border Protection [CBP] 2011 Department of Defense [DoD] 2008 2013 Department of Energy [DOE] 2002 Department of Homeland Security [DHS] 2010 [Pt 1] 2011 Department of Transportation [DOT] 2007 2009 Institute for Electrical and Electronics Engineers [IEEE] 1998a 1998b Internationa l Council on Systems Eng ineering [INCOSE] 2011 I nt er nat iona l Orga n i zat ion for St a nda rd i zat ion I nt er nat iona l Electrotechnical Commission [ISOIEC] 2008 International Organization for StandardizationInternational Electrotechnical CommissionInstitute for Electrical and Electronics Engineers [ISOIECIEEE] 2011 ISOIEC IEEE 2015 Joint Chiefs of Staff [JCS] 2011 JCS 2015 Keating Padilla amp Adams 2008 M Korent (e-mail communication via Tom Wissink January 13 2015 Advancing Complex Systems Manager Lockheed Martin) Madni amp Sievers 2013 Maier 1998 National Aeronautics and Space Administration [NASA] 1995 2012 2013 Ncube 2011 US Coast Guard [USCG] 2013)

In our study we use the IEEErsquos requirements definition that provides a broad universal and vetted foundation that can be applied to industry government and academia and also aligns with DoD definitions

Requirements Literature research from government and standards organizations

reveals varying definitions for system requirements In our study we use the IEEErsquos requirements definition that provides a broad universal and vetted foundation that can be applied to industry government and academia and also aligns with DoD definitions (IEEE 1998a JCS 2015)

April 2017

Requirement

1 A condition or capability needed by a user to solve a problem or achieve an objective

2 A condition or capability that must be met or possessed by a system or system component to satisfy a contract stanshydard specification or other formally imposed document

3 A document representation of a condition or capability as in definition 1) or 2)

Systems The definitions of systems are documented by multiple government

organizations at the national and state levels and standards organizashytions Our literature review discovered at least 20 existing approaches to defining a system For this research we use a more detailed definition as presented by IEEE (1998a) based on our research it aligns with DoD and federal approaches

Systems

An interdependent group of people objectives and proshycedures constituted to achieve defined objectives or some operational role by performing specified functions A complete system includes all of the associated equipment facilities material computer programs firmware technical documentation services and personnel required for operashytions and support to the degree necessary for self-sufficient use in its intended environment

Various authors and organizations have defined attributes to develop requirements for systems (Davis 1993 Georgiadis Mazzuchi amp Sarkani 2012 INCOSE 2011 Rettaliata Mazzuchi amp Sarkani 2014) Davis was one of the earliest authors to frame attributes in this manner though his primary approach concentrated on software requirements Subsequent to this researchers have adapted and applied attributes more broadly for use with all systems including software hardware and integration In addishytion Rettaliata et al (2014) provided a wide-ranging review of attributes for materiel and nonmateriel systems

The attributes provided in Davis (1993) consist of eight attributes for content and five attributes for format As a result of our research with government and industry we add a ninth and critical content attribute of lsquoachievablersquo and expand the existing 13 definitions for clarity INCOSE and IEEE denote the lsquoachievablersquo attribute which ensures systems are attainable to be built and operated as specified (INCOSE 2011 ISOIECIEEE 2011) The 14 requirements attributes with our enhanced definitions are listed in Table 1 (Davis 1993 INCOSE 2011 ISOIECIEEE 2011 Rettaliata et al 2014)

April 2017

TABLE 1 SYSTEM REQUIREMENTS ATTRIBUTES

Attribute Type Definition Correct Content Correct if and only if every requirement stated

therein represents something required of the system to be built

Unambiguous Content Unambiguous if and only if every requirement stated therein has only one interpretation and includes only one requirement (unique)

Complete Content Complete if it possesses these qualities 1 Everything it is supposed to do is included 2 Definitions of the responses of software to

all situations are included 3 All pages are numbered 4 No sections are marked ldquoTo be determinedrdquo 5 Is necessary

Verifiable Content Verifiable if and only if every requirement stated therein is verifiable

Consistent Content Consistent if and only if (1) no requirement stated therein is in conflict with other preceding documents and (2) no subset of requirements stated therein conflict

Understand- Content Understandable by customer if there exists a able by complete unambiguous mapping between the Customer formal and informal representations

Achievable Content Achievablemdashthe designer should have the expertise to assess the achievability of the requirements including subcontractors manufacturing and customersusers within the constraints of the cost and schedule life cycle

Design Content Design independent if it does not imply a Independent specific architecture or algorithm

Concise Content Concise if given two requirements for the same system each exhibiting identical level of all previously mentioned attributesmdashshorter is better

Modifiable Format Modifiable if its structure and style are such that any necessary changes to the requirement can be made easily completely and consistently

Traced Format Traced if the origin of each of its requirements is clear

Traceable Format Traceable if it is written in a manner that facilitates the referencing of each individual requirement stated therein

TABLE 1 SYSTEM REQUIREMENTS ATTRIBUTES CONTINUED

Attribute Type Definition Annotated Format Annotated if there is guidance to the

development organization such as relative necessity (ranked) and relative stability

Organized Format Organized if the requirements contained therein are easy to locate

While there are many approaches to gather requirements attributes for our research we use these 14 attributes to encompass and focus on software hardware interoperability and achievability These attributes align with government and DoD requirements directives instructions and guidebooks as well as the recent GAO report by DoD Service Chiefs which stresses their concerns on achievability of requirements (GAO 2015b) We focus our research on the nine content attributes While the five format attributes are necessary the nine content attributes are shown to be more central to ensuring quality requirements (Rettaliata et al 2014)

Unconstrained Systems The acquisition and systems engineering communities have attempted

to define lsquosystem of systemsrsquo for decades Most definitions can be traced back to Mark W Maierrsquos (1998) research which provided an early definition

April 2017

and set of requirements attributes As programs became larger with more complexities and interdependencies the definitions of system of systems expanded and evolved

In some programs the program managerrsquos governance authority can be low or independent creating lsquounconstrained systemsrsquomdasha term that while similar to the term system of systems provides an increased focus on the challenges of program managers with low governance authority between a principal system and component systems Unconstrained systems center on the relationship between the principal system and the component system the management and oversight of the stakeholder involvement and governance level of the program manager between users of the principal system and the component systems This increased focus and perspective enables greater requirements development fidelity for unconstrained systems

An example is shown in Figure 1 where a program manager of a national command and communications program can have limited governance authority to influence independent requirements on unconstrained systems with state and local stakeholders Unconstrained systems do not explicitly depend on a principal system When operating collectively the component systems create a unique capability In comparison to the broader definition for system of systems unconstrained systems require a more concentrated approach and detailed understanding of the independence of systems under a program managerrsquos purview We uniquely derive and define unconstrained systems as

Unconstrained Systems

The requirements attributes for unconstrained systems are identical to the attributes for systems as listed in Table 1 However a collection of unconstrained systems that is performing against a set of requirements in conjunction with each other has a different capability and focus than a singular system set of dependent systems or a complex system This perspective though it shares a common set of attributes with a singular or simple system can develop a separate and different set of requirements unique to an unconstrained system

incipa

ined S

amp Co

te amp

l amp Tr

ibal F

pitals

uts amp

April 2017

Complex Systems The systems engineering communities from industry and government

have long endeavored to define complex systems Some authors describe attributes that complex systems demonstrate versus a singular definition Table 2 provides a literature review of complex systems attributes

TABLE 2 LITERATURE REVIEW OF COMPLEX SYSTEM ATTRIBUTES

Attribute Definition Adaptive Components adapt to changes in others as well as to

changes in personnel funding and application shift from being static to dynamic systems (Chittister amp Haimes2010 Glass et al 2011 Svetinovic 2013)

Aspirational To influence design control and manipulate complex systems to solve problems to predict prevent or cause and to define decision robustness of decision and enabling resilience (Glass et al 2011 Svetinovic 2013)

Boundary Liquidity

Complex systems do not have a well-defined boundary The boundary and boundary criteria for complex systems are dynamic and must evolve with new understanding (Glass et al 2011 Katina amp Keating 2014)

Contextual A complex situation can exhibit contextual issues Dominance that can stem from differing managerial world views

and other nontechnical aspects stemming from the elicitation process (Katina amp Keating 2014)

Emergent Complex systems may exist in an unstable environment and be subject to emergent behavioral structural and interpretation patterns that cannot be known in advance and lie beyond the ability of requirements to effectively capture and maintain (Katina amp Keating 2014)

Environmental Exogenous components that affect or are affected by the engineering system that which acts grows and evolves with internal and external components (Bartolomei Hastings de Nuefville amp Rhodes 2012 Glass et al 2011 Hawryszkiewycz 2009)

Functional Range of fulfilling goals and purposes of the engineering system ease of adding new functionality or ease of upgrading existing functionality the goals and purposes of the engineering systems ability to organize connections (Bartolomei et al 2012 Hawryszkiewycz 2009 Jain Chandrasekaran Elias amp Cloutier 2008 Konrad amp Gall 2008)

TABLE 2 LITERATURE REVIEW OF COMPLEX SYSTEM ATTRIBUTES CONTINUED

Attribute Definition Holistic Consider the whole of the system consider the role of

the observer and consider the broad influence of the system on the environment (Haber amp Verhaegen 2012 Katina amp Keating 2014 Svetinovic 2013)

Multifinality Two seemingly identical initial complex systems can have different pathways toward different end states (Katina amp Keating 2014)

Predictive Proactively analyze requirements arising due to the implementation of the system underdevelopment and the systemrsquos interaction with the environment and other systems (Svetinovic 2013)

Technical Physical nonhuman components of the system to include hardware infrastructure software and information complexity of integration technologies required to achieve system capabilities and functions (Bartolomei et al 2012 Chittister amp Haimes 2010 Haber amp Verhaegen 2013 Jain et al 2008)

Interdependenshycies

A number of systems are dependent on one another to produce the required results (Katina amp Keating 2014)

Process Processes and steps to perform tasks within the system methodology framework to support and improve the analysis of systems hierarchy of system requirements (Bartolomei et al 2012 Haber amp Verhaegen 2012 Konrad amp Gall 2008 Liang Avgeriou He amp Xu 2010)

Social Social network consisting of the human components and the relationships held among them social network essential in supporting innovation in dynamic processes centers on groups that can assume roles with defined responsibilities (Bartolomei et al 2012 Hawryszkiewycz 2009 Liang et al 2010)

Complex systems are large and multidimensional with interrelated dependent systems They are challenged with dynamic national-level or international intricacies as social political environmental and technical issues evolve (Bartolomei et al 2012 Glass et al 2011) Complex sysshytems with a human centric and nondeterministic focus are typically large national- and international-level systems or products Noncomplex systems or lsquosystemsrsquo do not have these higher order complexities and relationships Based on our research with federal DoD and industry approaches we uniquely define a complex system as

April 2017

Complex System

It can be argued that complex and unconstrained systems have similar properties however for our research we consider them distinct Complex systems differ from unconstrained systems depending on whether the comshyponent systems within the principal system are dependent or independent of the principal system These differences are shown in Figure 1 Our examshyple is the lsquospace shuttlersquo in which the components of the orbiter external tank and solid rocket boosters are one dependent space shuttle complex system For complex systems the entirety of the principal system depends on component systems Thus the governance and stakeholders of the comshyponent systems depend on the principal system

Complex systems differ from unconstrained systems depending on whether the component systems within the principal system are dependent or independent of the principal system

Complex systems have an additional level of integration with internal and external focuses as shown in Figure 2 Dependent systems within the inner complex systems boundary condition derive a set of requirements attributes that are typically more clear and precise For our research we use the attributes from systems as shown in Table 2 to define internal requirements Using the lsquospace shuttlersquo example the internal requirements would focus on the dependent components of the orbiter external tank and solid rocket boosters

FIGURE 2 COMPLEX SYSTEMS INTERNAL AND EXTERNAL PERSPECTIVES

Complex System Boundary

Adaptive

Technical

Interdependence

Political

Holistic

Environmental Social

Dependent System

Dependent System Dependent

System

(internal)

(external)

Complex systems have a strong external focus As complex systems intershyface with their external sphere of influence another set of requirements attributes is generated as the outer complex boundary conditions become more qualitative than quantitative When examining complex systems extershynally the boundaries are typically indistinct and nondeterministic Using the lsquospace shuttlersquo example the external focus could be Congress the space station the interface with internationally developed space station modules and international partners training management relations and standards

Using our definition of complex systems we distinctly derive and define seven complex system attributes as shown in Table 3 The seven attributes (holistic social political adaptable technical interdependent and envishyronmental) provide a key set of attributes that aligns with federal and DoD approaches to consider when developing complex external requirements Together complex systems with an external focus (Table 3) and an internal focus (Table 2) provide a comprehensive and complementary context to develop a complete set of requirements for complex systems

April 2017

TABLE 3 COMPLEX SYSTEMS EXTERNAL REQUIREMENTS ATTRIBUTES

Attribute Definition Holistic Holistic considers the following

bull Security and surety scalability and openness and legacy systems

bull Timing of schedules and budgets bull Reliability availability and maintainability bull Business and competition strategies bull Role of the observer the nature of systems requirements

and the influence of the system environment (Katina amp Keating 2014)

Social Social considers the following bull Local state national tribal international stakeholders bull Demographics and culture of consumers culture of

developing organization (Nescolarde-Selva amp Uso-Demenech 2012 2013)

bull Subcontractors production manufacturing logistics maintenance stakeholders

bull Human resources for program and systems integration (Jain 2008)

bull Social network consisting of the human components and the relationships held among them (Bartolomei et al 2011)

bull Customer and social expectations and customer interfaces (Konrad amp Gall 2008)

bull Uncertainty of stakeholders (Liang et al 2010) bull Use of Web 20 tools and technologies (eg wikis

folksonomie and ontologies) (Liang et al 2010) bull Knowledge workersrsquo ability to quickly change work

connections (Hawryszkiewycz 2009)

Political Political considers the following bull Local state national tribal international political

circumstances and interests bull Congressional circumstances and interests to include

public law and funding bull Company partner and subcontractor political

circumstances and interests bull Intellectual property rights proprietary information and

patents

Adaptable Adaptability considers the following bull Shifts from static to being adaptive in nature (Svetinovic

2013) bull Systemrsquos behavior changes over time in response to

external stimulus (Ames et al 2011) bull Components adapt to changes in other components as

well as changes in personnel funding and application (Glass et al 2011)

TABLE 3 COMPLEX SYSTEMS EXTERNAL REQUIREMENTS ATTRIBUTES CONTINUED

Attribute Definition Technical Technical considers the following

bull Technical readiness and maturity levels bull Risk and safety bull Modeling and simulation bull Spectrum and frequency bull Technical innovations (Glass et al 2011) bull Physical nonhuman components of the system to include

hardware software and information (Bartolomei et al 2011 Nescolarde-Selva amp Uso-Demenech 2012 2013)

Interde- Interdependencies consider the following pendent bull System and system componentsrsquo schedules for developing

components and legacy components bull Product and production life cycles bull Management of organizational relationships bull Funding integration from system component sources bull The degree of complication of a system or system

component determined by such factors as the number of intricacy of interfaces number and intricacy of conditional branches the degree of nesting and types of data structure (Jain et al 2008)

bull The integration of data transfers across multiple zones of systems and network integration (Hooper 2009)

bull Ability to organize connections and integration between system units and ability to support changed connections (Hawryszkiewycz 2009)

bull Connections between internal and external people projects and functions (Glass et al 2011)

Environshy Environmental considers the following mental bull Physical environment (eg wildlife clean water protection)

bull Running a distributed environment by distributed teams and stakeholders (Liang et al 2010)

bull Supporting integration of platforms for modeling simulation analysis education training and collaboration (Glass et al 2011)

Methodology We use a group of experts with over 25 years of experience to validate

our derived requirements attributes by using the expert judgment methodshyology as originally defined in Bradley and Terry (1952) and later refined in Cooke (1991) We designed a repeatable survey that mitigated expert bias

April 2017

using the pairwise comparison technique This approach combines and elicits expertsrsquo judgment and beliefs regarding the strength of requirements attributes

Expert Judgment Expert judgment has been used for decades to support and solve complex

technical problems Commonly expert judgment is used when substantial scientific uncertainty has an impact on a decision process (Cooke amp Goossens 2008) Thus expert judgment allows researchers and communities of intershyest to reach rational consensus when there is scientific knowledge or process uncertainty (Cooke amp Goossens 2004) In addition it is used to assess outshycomes of a given problem by a group of experts within a field of research who have the requisite breadth of knowledge depth of multiple experiences and perspective Based on such data this research uses multiple experts from a broad range of backgrounds with in-depth experience in their respective fields to provide a diverse set of views and judgments

Commonly expert judgment is used when substantial scientific uncertainty has an impact on a decision process

Expert judgment has been adopted for numerous competencies to address contemporary issues such as nuclear applications chemical and gas indusshytry water pollution seismic risk environmental risk snow avalanches corrosion in gas pipelines aerospace banking information security risks aircraft wiring risk assessments and maintenance optimization (Clemen amp Winkler 1999 Cooke amp Goossens 2004 Cooke amp Goossens 2008 Goossens amp Cooke nd Lin amp Chih-Hsing 2008 Lin amp Lu 2012 Mazzuchi Linzey amp Bruning 2008 Ryan Mazzuchi Ryan Lopez de la Cruz amp Cooke 2012 van Noortwijk Dekker Cooke amp Mazzuchi 1992 Winkler 1986) Various methods are employed when applying this expert judgment Our methodshyology develops a survey for our group of experts to complete in private and allows them to comment openly on any of their concerns

Bradley-Terry Methodology We selected the Bradley-Terry expert judgment methodology (Bradley

amp Terry 1952) because it uses a proven method for pairwise comparisons to capture data via a survey from experts and uses it to rank the selected

requirements attributes by their respective importance In addition to allowshying pairwise comparisons of factors by multiple experts which provides a relative ranking of factors this methodology provides a statistical means for assessing the adequacy of individual expert responses the agreement of experts as a group and the appropriateness of the Bradley-Terry model

The appropriateness of expertsrsquo responses is determined by their number of circular triads Circular triads C(e) as shown in Equation (1) are when an expert (e) ranks one object A in a circular fashion such as A1 gt A2 and A2 gt A3 and A3 gt A1 (Bradley amp Terry 1952 Mazzuchi et al 2008)

t(t2 - 1) 1 1C(e) = minus sum t [a(ie)minus (tminus1)]2 (1) i = 124 2 2

The defined variables for the set of equations are

e = expert t = number of objects n = number of experts A(1) hellip A(t) = objects to be compared a(ie) = number of times expert e prefers A(i)R(ie) = the rank of A(i) from expert eV(i) = true values of the objects V(ie) = internal value of expert e for object i

The random variable C(e) defined in Equation (1) represents the number of circular triads produced when an expert provides an answer in a random fashion The random variable has a distribution approximated by a chi-squared distribution as shown in Equation (2) and can be applied to each expert to test the hypothesis that the expert answered randomly versus the alternative hypothesis that a certain preference was followed Experts for whom this hypothesis cannot be rejected at the 5 percent significance level are eliminated from the study

t(t - 1) (t - 2) 8 1 t 1Cˇ(e) = (t - 4)2 + (t minus 4) [( )( )] 4 3 minus c(e) + 2 ] (2)

The coefficient of agreement U a measure of consistency of rankings from expert to expert (Bradley amp Terry 1952 Cooke 1991 Mazzuchi et al 2008) is defined in Equation (3)

sum t (a(ij))2 sum t i = 1 j = 1 j ne i 2 (3) U = e t minus 1

( )( )2 2

April 2017

When the experts agree 100 percent U obtains its maximum of 1 The coeffishycient of agreement distribution U defines the statistic under the hypothesis that all agreements by experts are due to chance (Cooke 1991 Mazzuchi et al 2008) U has an approximate chi-squared distribution

1 n t n minus 3 i = 1 2sum t sum t a(ij) minusj = 1 j ne i 2 ( )( )( )( 2 2 n minus 2)Uˇ = (4)

n minus 2

The sum of the ranks R(i) is given by

R(i) = sum e R(ie) (5)

The Bradley-Terry methodology uses a true scale value Vi to determine rankings and they are solved iteratively (Cooke 1991 Mazzuchi et al 2008) Additionally Bradley-Terry and Cooke (1991) define the factor F for the goodness of fit for a model as shown in Equation (6) To determine if the model is appropriate (Cooke 1991 Mazzuchi et al 2008) it uses a null hypothesis This approach approximates a chi-squared distribution using (t-1)(t-2)2 for degrees of freedom

t t tF = 2sum i = 1 sum j = 1 j ne i a(i j) ln(R(i j)) minus sum i = 1 a(i) ln(Vi ) + t tsum i = 1 sum j = i + 1 e ln(Vi + Vj ) (6)

Analysis Survey participants were selected for their backgrounds in acquisition

academia operations and logistics For purposes of this study each expert (except one) met the minimum threshold of 25 years of combined experience

and training in their respective fields to qualify as an expert Twenty-five years was the target selected for experts to have the experience perspective and knowledge to be accepted as an expert by the acquisition community at large and to validate the requirements attributes

Survey Design The survey contained four sections with 109 data fields It was designed

to elicit impartial and repeatable expert judgment using the Bradley-Terry methodology to capture pairwise comparisons of requirements attributes In addition to providing definitions of terms and requirements attributes

a sequence randomizer was implemented providing ranshydom pairwise comparisons for each survey to ensure unbiased and impartial results The survey and all required documentation were submitted and subseshyquently approved by the Institutional Review Board in the Office of Human Research at The George Washington University

Participant Demographic Data A total of 28 surveys was received and used to

perform statistical analysis from senior pershysonnel in government and industry Of the

experts responding the average experishyence level was 339 years Government

participants and industry particishypants each comprise 50 percent

of the respondents Table 4 shows a breakout of experishy

ence skill sets from survey participants with an average of

108 years of systems engineering and requirements experience Participants show a

diverse grouping of backgrounds Within the government participantsrsquo group they represent the Army Navy and Air Force

and multiple headquarters organizations within the DoD multiple orgashynizations within the DHS NASA and Federally Funded Research and Development Centers Within the industry participantsrsquo group they repshyresent aerospace energy information technology security and defense sectors and have experience in production congressional staff and small entrepreneurial product companies We do not note any inconsistences within the demographic data Thus the demographic data verify a senior experienced and well-educated set of surveyed experts

April 2017

TABLE 4 EXPERTSrsquo EXPERIENCE (YEARS)

Average Minimum Maximum Overall 342 13 48

Subcategories Program Management 98 3 30

Systems Engineering Requirements 108 1 36

Operations 77 2 26

Logistics 61 1 15

Academic 67 1 27

Test and Evaluation 195 10 36

Science amp Technology 83 4 15

Aerospace Marketing 40 4 4

Software Development 100 10 10

Congressional Staff 50 5 5

Contracting 130 13 13

System Concepts 80 8 8

Policy 40 4 4

Resource Allocation 30 3 3

Quality Assurance 30 3 3

Interpretation and Results Requirements attribute data were collected for systems unconstrained

systems and complex systems When evaluating p-values we consider data from individual experts to be independent between sections The p-value is used to either keep or remove that expert from further analysis for the systems unconstrained systems and complex systems sections As defined in Equation (2) we posit a null hypothesis at the 5 percent significance level for each expert After removing individual experts due to failing the null hypothesis for random answers using Equation (2) we apply the statistic as shown in Equation (4) to determine if group expert agreement is due to chance at the 5 percent level of significance A goodness-of-fit test as defined in Equation (6) is performed on each overall combined set of expert data to confirm that the Bradley-Terry model is representative of the data set A null hypothesis is successfully used at the 5 percent level of significance After completing this analysis we capture and analyze data for the overall set of combined experts We perform additional analysis by dividing the experts into two subsets with backgrounds in government and industry

While it can be reasoned that all attributes are important to developing sound solid requirements we contend requirements attribute prioritization helps to focus the attention and awareness on requirements development and informed design trade-off decisions The data show the ranking of attributes for each category The GAO reports outline the recommendation for ranking of requirements for decision makers to use in trade-offs (GAO 2011 2015) The data in all categories show natural breaks in requirements attribute rankings which requirements and acquisition professionals can use to prioritize their concentration on requirements development

Systems requirements attribute analysis The combined expert data and the subsets of government and industry experts with the associated 90 percent confidence intervals are shown in Figures 3 and 4 They show the values of the nine attributes which provides their ranking

FIGURE 3 SYSTEM REQUIREMENTS ATTRIBUTE RANKINGS OF ALL EXPERTS WITH 90 CONFIDENCE INTERVALS

All Experts (n = 25)

Systems Requirements Attributes

Understandable by Customer

Achievable Design Independent

Concise Consistent Verifiable Complete UnambiguousCorrect

April 2017

FIGURE 4 SYSTEM REQUIREMENTS ATTRIBUTE RANKINGS OF GOVERNMENT AND INDUSTRY EXPERTS WITH 90

CONFIDENCE INTERVALS

Government Experts (n = 12) Industry Experts (n = 13)

Overall the systems requirements attribute values show the top-tier attributes are achievable and correct while the bottom-tier attributes are design-independent and concise This analysis is consistent between the government and industry subsets of experts as shown in Figure 4

The 90 percent confidence intervals of all experts and subgroups overshylap which provide correlation to the data and reinforce the validity of the attribute groupings This value is consistent with industry experts and government experts From Figure 4 the middle-tier attributes from governshyment experts are more equally assessed between values of 00912 and 01617 Industry experts along with the combination of all experts show a noticeable breakout of attributes at the 01500 value which proves the top grouping of systems requirements attributes to be achievable correct and verifiable

Unconstrained requirements attribute analysis The overall expert data along with subgroups for government and industry experts with the associated 90 percent confidence intervals for unconstrained systems are

shown in Figures 5 and 6 This section has the strongest model goodness-of-fit data with a null successfully used at less than a 1 percent level of significance as defined in Equation (6)

FIGURE 5 UNCONSTRAINED SYSTEMS REQUIREMENTS ATTRIBUTE RANKINGS FOR ALL EXPERTS WITH 90 CONFIDENCE INTERVALS

Unconstrained Systems Requirements Attributes

April 2017

FIGURE 6 UNCONSTRAINED SYSTEMS REQUIREMENTS ATTRIBUTE RANKINGS OF GOVERNMENT AND INDUSTRY EXPERTS WITH 90

As indicated in Figure 5 the overall top-tier requirements attributes are achievable and correct These data correlate with the government and indusshytry expert subgroups in Figure 6 The 90 percent confidence intervals of all experts and subgroups overlap which validate and provide consistency of attribute groupings between all experts and subgroups The bottom-tier attributes are design-independent and concise and are consistent across all analysis categories The middle tier unambiguous complete verifiable consistent and understandable by the customer is closely grouped together across all subcategories Overall the top tier of attributes by all experts remains as achievable with a value of 02460 and correct with a value of 01862 There is a clear break in attribute values at the 01500 level

Complex requirements attribute analysis The combined values for comshyplex systems by all experts and subgroups are shown in Figures 7 and 8 with a 90 percent confidence interval and provide the values of the seven attributes

FIGURE 7 COMPLEX SYSTEMS REQUIREMENTS ATTRIBUTE RANKINGS FOR ALL EXPERTS WITH 90 CONFIDENCE INTERVALS

00000 Holistic Social Political Adaptable Technical Interdependent Environmental

Complex Systems Requirements Attributes

April 2017

FIGURE 8 COMPLEX SYSTEMS REQUIREMENTS ATTRIBUTES FOR GOVERNMENT AND INDUSTRY EXPERTS WITH 90

Interdependent Environmental Technical Adaptable PoliticalSocialHolistic

The 90 percent confidence intervals of all experts and subgroups overlap confirming the consistency of the data and strengthening the validity of all rankings between expert groups Data analysis as shown in Figure 7 shows a group of four top requirements attributes for complex systems technical interdependent holistic and adaptable These top four attributes track with the subsets of government and industry experts as shown in Figure 8 In addition these top groupings of attributes are all within the 90 percent confidence interval of one another however the attribute values within these groupings differ

Data conclusions The data from Figures 3ndash8 show consistent agreement between government industry and all experts Figure 9 shows the comshybined values with a 90 percent confidence interval for all 28 experts across systems unconstrained systems and complex systems Between systems and unconstrained systems the expertsrsquo rankings are similar though the values differ The achievable attribute for systems and unconstrained sysshytems has the highest value in the top tier of attribute groups

FIGURE 9 COMPARISON OF REQUIREMENTS ATTRIBUTES ACROSS SYSTEMS UNCONSTRAINED SYSTEMS AND COMPLEX SYSTEMS

WITH 90 CONFIDENCE INTERVALS

0 4 500

0 4000

0 3 500

0 3000

0 2500

0 2000

0 1500

0 1000

Systems Unconstrained Systems Complex Systems

Understandable by Cu

stomer

Design Independen

vable t

ConciseHolist

icSocial

Political

Adaptable

Technical

Interdependent

Environmental

Consistent

Verifiable

Complete

Unambiguous

Correct

Systems and Unconstrained Systems Requirements Attributes

Complex External Requirements Attributes

Our literature research revealed this specific attributemdashachievablemdashto be a critical attribute for systems and unconstrained systems Moreover experts further validate this result in the survey open response sections Experts state ldquoAchievability is the top priorityrdquo and ldquoYou ultimately have to achieve the system so that you have something to verifyrdquo Additionally experts had the opportunity to comment on the completeness of our requirements attributes in the survey No additional suggestions were submitted which further confirms the completeness and focus of the attribute groupings

While many factors influence requirements and programs these data show the ability of management and engineering to plan execute and make proshygrams achievable within their cost and schedule life cycle is a top priority regardless of whether the systems are simple or unconstrained For comshyplex systems experts clearly value technical interdependent holistic and adaptable as their top priorities These four attributes are critical to create achievable successful programs across very large programs with multiple

April 2017

interfaces Finally across all systems types the requirements attributes provide a validated and comprehensive approach to develop prioritized effective and accurate requirements

Conclusions Limitations and Future Work With acquisition programs becoming more geographically dispersed

yet tightly integrated the challenge to capture complex and unconstrained systems requirements early in the system life cycle is crucial for program success This study examined previous requirements attributes research and expanded approaches for the acquisition communityrsquos consideration when developing a key set of requirements attributes Our research capshytured a broad range of definitions for key requirements development terms refined the definitions for clarity and subsequently derived vital requireshyments attributes for systems unconstrained systems and complex systems Using a diverse set of experts it provided a validated and prioritized set of requirements attributes

These validated and ranked attributes provide an important foundation and significant step forward for the acquisition communityrsquos use of a prishyoritized set of attributes for decision makers This research provides valid requirements attributes for unconstrained and complex systems as new focused approaches for developing sound requirements that can be used in making requirements and design trade-off decisions It provides a compelshyling rationale and an improved approach for the acquisition community to channel and tailor their focus and diligence and thereby generate accurate prioritized and effective requirements

Our research was successful in validating attributes for the acquisition community however there are additional areas to continue this research The Unibalance-11 software which is used to determine the statistical information for pairwise comparison data does not accommodate weightshying factors of requirements attributes or experts Therefore this analysis only considers the attributes and experts equally Future research could expand this approach to allow for various weighting of key inputs such as attributes and experts to provide greater fidelity This expansion would determine the cause and effect of weighting on attribute rankings A key finding in this research is the importance of the achievable attribute We recommend additional research to further define and characterize this vital attribute We acknowledge that complex systems their definitions and linkshyages to other factors are embryonic concepts in the systems engineering program management and operational communities As a result we recshyommend further exploration of developing complex systems requirements

References Ames A L Glass R J Brown T J Linebarger J M Beyeler W E Finley P D amp

Moore T W (2011) Complex Adaptive Systems of Systems (CASoS) engineering framework (Version 10) Albuquerque NM Sandia National Laboratories

ANSIEIA (1999) Processes for engineering a system (Report No ANSIEIA-632 shy1998) Arlington VA Author

Bartolomei J E Hastings D E de Nuefville R amp Rhodes D H (2012) Engineering systems multiple-domain matrix An organizing framework for modeling large-scale complex systems Systems Engineering 15(1) 41ndash61

Bradley R A amp Terry M E (1952) Rank analysis of incomplete block designs I The method of paired comparisons Biometrika 39(3-4) 324ndash345

Butterfield M L Shivananda A amp Schwarz D (2009) The Boeing system of systems engineering (SOSE) process and its use in developing legacy-based net-centric systems of systems Proceedings of National Defense Industrial Association (NDIA) 12th Annual Systems Engineering Conference (pp 1ndash20) San Diego CA

CBP (2011) Office of Technology Innovation and Acquisition requirements handbook Washington DC Author

Chittister C amp Haimes Y Y (2010) Harmonizing High Performance Computing (HPC) with large-scale complex systems in computational science and engineering Systems Engineering 13(1) 47ndash57

CJCS (2012) Joint capabilities integration and development system (CJCSI 3170) Washington DC Author

Clemen R T amp Winkler R L (1999) Combining probability distributions from experts in risk analysis Risk Analysis 19(2) 187ndash203

Cooke R M (1991) Experts in uncertainty Opinion and subjective probability in science New York NY Oxford University Press

Cooke R M amp Goossens L H J (2004 September) Expert judgment elicitation for risk assessments of critical infrastructures Journal of Risk 7(6) 643ndash656

Cooke R M amp Goossens L H J (2008) TU Delft expert judgment data base Reliability Engineering and System Safety 93(5) 657ndash674

Corsello M A (2008) System-of-systems architectural considerations for complex environments and evolving requirements IEEE Systems Journal 2(3) 312ndash320

Davis A M (1993) Software requirements Objects functions and states Upper Saddle River NJ Prentice-Hall PTR

DHS (2010) DHS Systems Engineering Life Cycle (SELC) Washington DC Author DHS (2011) Acquisition management instructionguidebook (DHS Instruction Manual

102-01-001) Washington DC DHS Under Secretary for Management DoD (2008) Systems engineering guide for systems of systems Washington DC

Office of the Under Secretary of Defense (Acquisition Technology and Logistics) Systems and Software Engineering

DoD (2013) Defense acquisition guidebook Washington DC Office of the Under Secretary of Defense (Acquisition Technology and Logistics)

DOE (2002) Systems engineering methodology (Version 3) Washington DC Author DOT (2007) Systems engineering for intelligent transportation systems (Version 20)

Washington DC Federal Highway Administration DOT (2009) Systems engineering guidebook for intelligent transportation systems

(Version 30) Washington DC Federal Highway Administration

April 2017

GAO (2011) DoD weapon systems Missed trade-off opportunities during requirements reviews (Report No GAO-11-502) Washington DC Author

GAO (2015a) Defense acquisitions Joint action needed by DoD and Congress to improve outcomes (Report No GAO-16-187T) Testimony Before the Committee on Armed Services US House of Representatives (testimony of Paul L Francis) Washington DC Author

GAO (2015b) Defense acquisition process Military service chiefsrsquo concerns reflect need to better define requirements before programs start (Report No GAO-15 469) Washington DC Author

Georgiadis D R Mazzuchi T A amp Sarkani S (2012) Using multi criteria decision making in analysis of alternatives for selection of enabling technology Systems Engineering Wiley Online Library doi 101002sys21233

Glass R J Ames A L Brown T J Maffitt S L Beyeler W E Finley P D hellip Zagonel A A (2011) Complex Adaptive Systems of Systems (CASoS) engineering Mapping aspirations to problem solutions Albuquerque NM Sandia National Laboratories

Goossens L H J amp Cooke R M (nd) Expert judgementmdashCalibration and combination (Unpublished manuscript) Delft University of Technology Delft The Netherlands

Haber A amp Verhaegen M (2013) Moving horizon estimation for large-scale interconnected systems IEEE Transactions on Automatic Control 58(11) 2834ndash 2847

Hawryszkiewycz I (2009) Workspace requirements for complex adaptive systems Proceedings of the IEEE 2009 International Symposium on Collaborative Technology and Systems (pp 342ndash347) May 18-22 Baltimore MD doi 101109 CTS20095067499

Hooper E (2009) Intelligent strategies for secure complex systems integration and design effective risk management and privacy Proceedings of the 3rd Annual IEEE International Systems Conference (pp 1ndash5) March 23ndash26 Vancouver Canada

IEEE (1998a) Guide for developing system requirements specifications New York NY Author

IEEE (1998b) IEEE recommended practice for software requirements specifications New York NY Author

INCOSE (2011) Systems engineering handbook A guide for system life cycle processes and activities San Diego CA Author

ISOIEC (2008) Systems and software engineeringmdashSoftware life cycle processes (Report No ISOIEC 12207) Geneva Switzerland ISOIEC Joint Technical Committee

ISOIECIEEE (2011) Systems and software engineeringmdashLife cycle processesmdash Requirements engineering (Report No ISOIECIEEE 29148) New York NY Author

ISOIECIEEE (2015) Systems and software engineeringmdashSystem life cycle processes (Report No ISOIECIEEE 15288) New York NY Author

Jain R Chandrasekaran A Elias G amp Cloutier R (2008) Exploring the impact of systems architecture and systems requirements on systems integration complexity IEEE Systems Journal 2(2) 209ndash223

JCS (2011) Joint operations (Joint Publication [JP] 30) Washington DC Author JCS (2015) Department of Defense dictionary of military and associated terms (JP

1-02) Washington DC Author Katina P F amp Keating C B (2014) System requirements engineering in complex

situations Requirements Engineering 19(1) 45ndash62 Keating C B Padilla J A amp Adams K (2008) System of systems engineering

requirements Challenges and guidelines Engineering Management Journal 20(4) 24ndash31

Konrad S amp Gall M (2008) Requirements engineering in the development of large-scale systems Proceedings of the 16th IEEE International Requirements Engineering Conference (pp 217ndash221) September 8ndash12 Barcelona-Catalunya Spain

Liang P Avgeriou P He K amp Xu L (2010) From collective knowledge to intelligence Pre-requirements analysis of large and complex systems Proceedings of the 2010 International Conference on Software Engineering (pp 26-30) May 2-8 Capetown South Africa

Lin S W amp Chih-Hsing C (2008) Can Cookersquos model sift out better experts and produce well-calibrated aggregated probabilities Proceedings of 2008 IEEE International Conference on Industrial Engineering and Engineering Management (pp 425ndash429)

Lin S W amp Lu M T (2012) Characterizing disagreement and inconsistency in experts judgment in the analytic hierarchy process Management Decision 50(7) 1252ndash1265

Madni A M amp Sievers M (2013) System of systems integration Key considerations and challenges Systems Engineering 17(3) 330ndash346

Maier M W (1998) Architecting principles for systems-of systems Systems Engineering 1(4) 267ndash284

Mazzuchi T A Linzey W G amp Bruning A (2008) A paired comparison experiment for gathering expert judgment for an aircraft wiring risk assessment Reliability Engineering amp System Safety 93(5) 722ndash731

Meyer M A amp Booker J M (1991) Eliciting and analyzing expert judgment A practical guide London Academic Press Limited

NASA (1995) NASA systems engineering handbook Washington DC Author NASA (2012) NASA space flight program and project management requirements

NASA Procedural Requirements Washington DC Author NASA (2013) NASA systems engineering processes and requirements NASA

Procedural Requirements Washington DC Author Ncube C (2011) On the engineering of systems of systems Key challenges for the

requirements engineering community Proceedings of International Workshop on Requirements Engineering for Systems Services and Systems-of-Systems (RESS) held in conjunction with the International Requirements Engineering Conference (RE11) August 29ndashSeptember 2 Trento Italy

Nescolarde-Selva J A amp Uso-Donenech J L (2012) An introduction to alysidal algebra (III) Kybernetes 41(10) 1638ndash1649

Nescolarde-Selva J A amp Uso-Domenech J L (2013) An introduction to alysidal algebra (V) Phenomenological components Kybernetes 42(8) 1248ndash1264

Rettaliata J M Mazzuchi T A amp Sarkani S (2014) Identifying requirement attributes for materiel and non-materiel solution sets utilizing discrete choice models Washington DC The George Washington University

April 2017

Ryan J J Mazzuchi T A Ryan D J Lopez de la Cruz J amp Cooke R (2012) Quantifying information security risks using expert judgment elicitation Computer amp Operations Research 39(4) 774ndash784

Svetinovic D (2013) Strategic requirements engineering for complex sustainable systems Systems Engineering 16(2) 165ndash174

van Noortwijk J M Dekker R Cooke R M amp Mazzuchi T A (1992 September) Expert judgment in maintenance optimization IEEE Transactions on Reliability 41(3) 427ndash432

USCG (2013) Capability management Washington DC Author Winkler R L (1986) Expert resolution Management Science 32(3) 298ndash303

Author Biographies

Col Richard M Stuckey USAF (Ret) is a senior scientist with ManTech supporting US Customs and Border Protection Col Stuckey holds a BS in Aerospace Engineering from the University of Michigan an MS in Systems Management from the University of Southern California and an MS in Mechanical Engineering from Louisiana Tech University He is currently pursuing a Doctor of Philosophy degree in Systems Engineering at The George Washington University

(E-mail address richstuckeygwuedu)

Dr Shahram Sarkani is professor of Engineer shying Management and Systems Engineering (EMSE) and director of EMSE Off-Campus Programs at The George Washington University He designs and administers graduate programs that enroll over 1000 students across the United States and abroad Dr Sarkani holds a BS and MS in Civil Engineering from Louisiana State University and a PhD in Civil Engineering from Rice University He is also credentialed as a Professional Engineer

April 2017

Author Biographies

Col Richard M Stuckey USAF (Ret) is asenior scientist with ManTech supporting USCustoms and Border Protection Col Stuckey holdsa BS in Aerospace Engineering from the Universityof Michigan an MS in Systems Management fromthe University of Southern California and an MSin Mechanical Engineering from Louisiana TechUniversity He is currently pursuing a Doctor ofPhilosophy degree in Systems Engineering at TheGeorge Washington University

Dr Shahram Sarkani is professor of Engineer-ing Management and Systems Engineering(EMSE) and director of EMSE Off-CampusPrograms at The George Washington UniversityHe designs and administers graduate programsthat enroll over 1000 students across the UnitedStates and abroad Dr Sarkani holds a BS andMS in Civil Engineering from Louisiana StateUniversity and a PhD in Civil Engineering fromRice University He is also credentialed as aProfessional Engineer

Dr Thomas A Mazzuchi is professor of E n g i ne er i n g M a n a gem ent a n d S y s t em s Engineering at The George Washington University His research interests include reliability life testing design and inference maintenance inspection policy analysis and expert judgment in risk analysis Dr Mazzuchi holds a BA in Mathematics from Gettysburg College and an MS and DSC in Operations Research from The George Washington University

(E-mail address mazzugwuedu)

An Investigation of Nonparametric DATA MINING TECHNIQUES for Acquisition Cost Estimating

Capt Gregory E Brown USAF and Edward D White

The Department of Defense (DoD) cost estimating methodology traditionally focuses on parametric estimating using ordinary least squares (OLS) regres sion Given the recent advances in acquisition data collection however senior leaders have expressed an interest in incorporating ldquodata miningrdquo and ldquomore innovative analysesrdquo within cost estimating Thus the goal of this research is to investigate nonparametric data mining techniques and their application to DoD cost estimating Using a meta-analysis of 14 cost estimating studies containing 32 datasets that predominantly relate to commercial software development the predictive accuracy of OLS regression is measured against three nonparametric data mining techniques The meta-analysis results indicate that on average the nonparametric techniques outperform OLS regression for cost estimating Follow-on data mining research that incor porates DoD-specific acquisition cost data is recommended to extend this articlersquos findings

DOI httpsdoiorg1022594dau16 7562402 Keywords cost estimation data mining nonparametric Cost Assessment Data Enterprise (CADE)

Nonparametric Data Mining Techniques httpwwwdaumil

We find companies in industries as diverse as pharmaceutical research retail and insurance have embraced data mining to improve their decision support As motivation companies who self-identify into the top third of their industry for data-driven decision makingmdashusing lsquobig datarsquo techniques such as data mining and analyticsmdashare 6 percent more profitable and 5 percent more efficient than their industry peers on average (McAfee amp Brynjolfsson 2012) It is therefore not surprising that 80 percent of surveyed chief executive officers identify data mining as strategically important to their business operations (PricewaterhouseCoopers 2015)

We find that the Department of Defense (DoD) already recognizes the potenshytial of data mining for improving decision supportmdash43 percent of senior DoD leaders in cost estimating identify data mining as a most useful tool for analysis ahead of other skillsets (Lamb 2016) Given senior leadershiprsquos interest in data mining the DoD cost estimator might endeavor to gain a foothold on the subject In particular the cost estimator may desire to learn about nonparametric data mining a class of more flexible regression

shying coursework from the Defense Acquisition

University (DAU) does not currently address nonparametric data mining

techniques Coursework instead focuses on parametric estimatshy

ing using ordinary least squares (OLS) regression while omitting nonparametric techniques (DAU

2009) Subsequently t he cos t es t i m ashyt or m ay t u r n t o

past research studshyies however t h is may

prove burdensome if the studies occurred outside the DoD and are not easshy

ily found or grouped together For this reason we strive to provide a consolidation of cost-estimating research

that implements nonparametric data mining Using a technique known as meta-analysis we investigate whether nonparametric techniques can outperform OLS regression for cost-estimating applications

techniques applicable to larger data sets

Initially the estimator may first turn to DoD-provided resources before discovering that cost estimat

April 2017

Our investigation is segmented into five sections We begin with a general definition of data mining and explain how nonparametric data mining difshyfers from the parametric method currently utilized by DoD cost estimators Next we provide an overview of the nonparametric data mining techniques of nearest neighbor regression trees and artificial neural networks These techniques are chosen as they are represented most frequently in cost-esshytimating research Following the nonparametric data mining overview we provide a meta-analysis of cost estimating studies which directly compares the performance of parametric and nonparametric data mining techniques After the meta-analysis we address the potential pitfalls to consider when utilizing nonparametric data mining techniques in acquisition cost estishymates Finally we summarize and conclude our research

Definition of Data Mining So exactly what is data mining At its core data mining is a multishy

disciplinary field at the intersection of statistics pattern recognition machine learning and database technology (Hand 1998) When used to solve problems data mining is a decision support methodology that idenshytifies unknown and unexpected patterns of information (Friedman 1997) Alternatively the Government Accountability Office (GAO) defines data mining as the ldquoapplication of database technologies and techniquesmdashsuch as statistical analysis and modelingmdashto uncover hidden patterns and subshytle relationships in data and to infer rules that allow for the prediction of future resultsrdquo (GAO 2005 p 4) We offer an even simpler explanationmdashdata mining is a collection of techniques and tools for data analysis

Data mining techniques are classified into six primary categories as seen in Figure 1 (Fayyad Piatetsky-Shapiro amp Smyth 1996) For cost estimating we focus on regression which uses existing values to estimate unknown values Regression may be further divided into parametric and nonparametshyric techniques The parametric technique most familiar to cost estimators is OLS regression which makes many assumptions about the distribution function and normality of error terms In comparison the nearest neighbor regression tree and artificial neural network techniques are nonparametshyric Nonparametric techniques make as few assumptions as possible as the function shape is unknown Simply put nonparametric techniques do not require us to know (or assume) the shape of the relationship between a cost driver and cost As a result nonparametric techniques are regarded as more flexible

FIGURE 1 CLASSIFICATION OF DATA MINING TASKS

Anomaly Detection

Data Mining

Association Rule Learning Classification Clustering Regression Summarization

Parametric Nonparametric

Nonparametric data mining techniques do have a major drawbackmdash to be effective these more f lexible techniques require larger data sets Nonparametric techniques utilize more parameters than OLS regression and as a result more observations are necessary to accurately estimate the function (James Witten Hastie amp Tibshirani 2013) Regrettably the gathering of lsquomore observationsrsquo has historically been a challenge in DoD cost estimatingmdashin the past the GAO reported that the DoD lacked the data both in volume and quality needed to conduct effective cost estimates (GAO 2006 GAO 2010) However this data shortfall is set to change The office of Cost Assessment and Program Evaluation recently introduced the Cost Assessment Data Enterprise (CADE) an online repository intended to improve the sharing of cost schedule software and technical data (Dopkeen 2013) CADE will allow the cost estimator to avoid the ldquolengthy process of collecting formatting and normalizing data each time they estishymate a program and move forward to more innovative analysesrdquo (Watern 2016 p 25) As CADE matures and its available data sets grow larger we assert that nonparametric data mining techniques will become increasingly applicable to DoD cost estimating

Overview of Nonparametric Data Mining Techniques

New variations of data mining techniques are introduced frequently through free open-source software and it would be infeasible to explain them all within the confines of this article For example the software Rmdash currently the fastest growing statistics software suitemdashprovides over 8000 unique packages for data analysis (Smith 2015) For this reason we focus solely on describing the three nonparametric regression techniques that comprise our meta-analysis nearest neighbor regression trees and artifishycial neural networks The overview for each data mining technique follows

April 2017

a similar pattern We begin by first introducing the most generic form of the technique and applicable equations Next we provide an example of the technique applied to a notional aircraft with unknown total program cost The cost of the notional aircraft is to be estimated using aircraft data garshynered from a 1987 RAND study consolidated in Appendix A (Hess amp Romanoff 1987 pp 11 80) We deliberately select an outdated database to emphasize that our examples are notional and not necessarily optimal Lastly we introduce more advanced variants of the technique and document their usage within cost-estimating literature

Analogous estimating via nearest neighbor also known as case-based reasoning emulates the way in which a human subject matter expert would identify an analogy

Nearest Neighbor Analogous estimating via nearest neighbor also known as case-based

reasoning emulates the way in which a human subject matter expert would identify an analogy (Dejaeger Verbeke Martens amp Baesens 2012) Using known performance or system attributes the nearest neighbor technique calculates the most similar historical observation to the one being estishymated Similarity is determined using a distance metric with Euclidian distance being most common (James et al 2013) Given two observations p and q and system attributes 1hellipn the Euclidean distance formula is

Distance = radic sumn (pi - qi)2 = radic(p1 - q1)2 + (p2 - q2)2 + hellip + (p - q )2 (1) pq i= 1 n n

To provide an example of the distance calculation we present a subset of the RAND data in Table 1 We seek to estimate the acquisition cost for a notional fighter aircraft labeled F-notional by identifying one of three historical observations as the nearest analogy We select the observation minimizing the distance metric for our two chosen system attributes Weight and Speed To ensure that both system attributes initially have the same weighting within the distance formula attribute values are standardized to have a mean of 0 and a standard deviation of 1 as shown in italics

TABLE 1 SUBSET OF RAND AIRCRAFT DATA FOR EUCLIDIAN DISTANCE CALCULATION

Weight Cost (Thousands of Pounds) Speed (Knots) (Billions)

F-notional 2000 000 1150 -018 unknown

F-4 1722 -087 1222 110 1399

F-105 1930 -022 1112 -086 1221

A-5 2350 109 1147 -024 1414

Using formula (1) the resulting distance metric between the F-notional and F-4 is

DistanceF-notionalF-4 = radic([000 - (-087)]2 + [-018 - (110)]2 = 154 (2)

The calculations are repeated for the F-105 and A-5 resulting in distance calculations of 071 and 110 respectively As shown in Figure 2 the F-105 has the shortest distance to F-notional and is identified as the nearest neighbor Thus the cost estimator would identify the unknown acquisition cost for the notional aircraft to be $1221 billion analogous to the F-105

April 2017

FIGURE 2 EUCLIDIAN DISTANCE PLOT FOR F NOTIONAL

Fshy4 ($1399)

Ashy5 ($1414)

Fshy105 ($1221)

Fshynotional Spee

Weight

shy2 0 2

Moving beyond our notional example we find that more advanced analogy techniques are commonly applied in cost-estimating literature When using nearest neighbor the cost of multiple observations may be averaged when k gt 1 with k signifying the number of analogous observations referenced However no k value is optimal for all data sets and situations Finnie Wittig and Desharnais (1997) and Shepperd and Schofield (1997) apply k = 3 while Dejaeger et al (2012) find k = 2 to be more predictive than k = 1 3 or 5 in predicting software development cost

Another advanced nearest neighbor technique involves the weighting of the system attributes so that individual attributes have more or less influence on the distance metric Shepperd and Schofield (1997) explore the attribute weighting technique to improve the accuracy of software cost estimates Finally we highlight clustering a separate but related technique for estishymating by analogy Using Euclidian distance clustering seeks to partition

a data set into analogous subgroups whereby observations within a subshygroup or lsquoclusterrsquo are most similar to each other (James et al 2013) The partition is accomplished by selecting the clusters minimizing the within cluster variation In cost-estimating research the clustering technique is successfully utilized by Kaluzny et al (2011) to estimate shipbuilding cost

Regression Tree The regression tree technique is an adaptation of the decision tree for

continuous predictions such as cost Using a method known as recursive binary splitting the regression tree splits observations into rectangular regions with the predicted cost for each region equal to the mean cost for the contained observations The splitting decision considers all possishyble values for each of the system attributes and then chooses the system attribute and attribute lsquocutpointrsquo which minimizes prediction error The splitting process continues iteratively until a stopping criterionmdashsuch as maximum number of observations with a regionmdashis reached (James et al 2013) Mathematically the recursive binary splitting decision is defined using a left node (L) and right node (R) and given as

min Σ (ei - eL)2 + Σ (ei - eR)2 (3)iεL iεR

where ei = the i th observations Cost

To provide an example of the regression tree we reference the RAND datashyset provided in Appendix A Using the rpart package contained within the R software we produce the tree structure shown in Figure 3 For simplicity we limit the treersquos growthmdashthe tree is limited to three decision nodes splitshyting the historical observations into four regions Adopting the example of the notional aircraft with a weight of 20 pounds (thousands) and a top speed of 1150 knots we interpret the regression tree by beginning at the top and following the decision nodes downward We discover that the notional airshycraft is classified into Region 3 As a result the cost estimator would identify the unknown acquisition cost for the notional aircraft to be $1305 billion equivalent to the mean cost of the observations within Region 3

April 2017

FIGURE 3 REGRESSION TREE USING RAND AIRCRAFT DATA

Aircraft Cost

Weight lt 3159

Weight lt 1221

Speed lt 992

Weight ge 3159

Weight ge 1221

Speed ge 992

$398 $928 $1305 $2228

0 0 20 40 60 80 100 120

R4 = $2228

Weight (Thousands of Pounds)

As an advantage regression trees are simple for the decision maker to interpret and many argue that they are more intuitive than OLS regresshysion (Provost amp Fawcett 2013) However regression trees are generally

outperformed by OLS regression except for data that are highly nonlinear or defined by complex relationships (James et al 2013) In an effort to improve the performance of regression trees we find that cost-estimating researchers apply one of three advanced regression tree techniques bagging boosting or piecewise linear regression

lsquoBaggingrsquo involves application of the bootstrap method whereby many regression trees are built on the data set but each time using a different subset of the total data set The predicted responses across all trees are then averaged to obtain the final response Within cost-estimating research the bagging technique is used by Braga Oliveria Ribeiro and Meira (2007) to improve software cost-estimating accuracy A related concept is lsquoboostingrsquo for which multiple trees are also developed on the data Rather than resamshypling the original data set boosting works by developing each subsequent tree using only residuals from the prior tree model For this reason boosting is less likely to overfit the data when compared to bagging (James et al 2013) Boosting is adopted by Shin (2015) to estimate building construction costs

lsquoBaggingrsquo involves application of the bootstrap method whereby many regression trees are built on the data set but each time using a different subset of the total data set

In contrast to bagging and boosting the lsquoM5rsquo techniquemdasha type of piecewise linear regressionmdashdoes not utilize bootstrapping or repeated iterations to improve model performance Instead the M5 fits a unique linear regression model to each terminal node within the regression tree resulting in a hybrid treelinear regression approach A smoothing process is applied to adjust for discontinuations between the linear models at each node Within cost research the M5 technique is implemented by Kaluzny et al (2011) to estishymate shipbuilding cost and by Dejaeger et al (2012) to estimate software development cost

Artificial Neural Network The artificial neural network technique is a nonlinear model inspired

by the mechanisms of the human brain (Hastie Tibshirani amp Friedman 2008) The most common artificial neural network model is the feed-forshyward multilayered perceptron based upon an input layer a hidden layer and an output layer The hidden layer typically utilizes a nonlinear logistic

April 2017

sigmoid transformed using the hyperbolic tangent function (lsquotanhrsquo funcshytion) while the output layer is a linear function Thus an artificial neural network is simply a layering of nonlinear and linear functions (Bishop 2006) Mathematically the artificial neural network output is given as

u u (4) omicro = ƒ (ΣWj Vj ) = ƒ [ΣWj gj (Σwjk Ik)]

u = inputs normalized between -1 and 1 Ik

= connection weights between input and output layers wjk

Wj = connection weights between hidden and output layer

Vju = output of the hidden neuron Nj Nj = input element at the output neuron N

gj (hju) = tanh(β frasl 2)

hj micro is a weighted sum implicitly defined in Equation (4)

For the neural network example we again consider the RAND data set in Appendix A Using the JMPreg Pro software we specify the neural network model seen in Figure 4 consisting of two inputs (Weight and Speed) three hidden nodes and one output (Cost) To protect against overfitting one-third of the observations are held back for validation testing and the squared penalty applied The resulting hidden nodes functions are defined as

h1 = TanH[(41281-00677 times Weight + 00005 times Speed)2] (5)

h2 = TanH[(-28327+00363 times Weight + 00015 times Speed)2] (6)

The output function is given as

O = 148727 + 241235 times h1 + 712283 times h2 -166950 times h3 (8)

FIGURE 4 ARTIFICIAL NEURAL NETWORK USING RAND AIRCRAFT DATA

Weight

To calculate the cost of the notional aircraft with a weight of 20 pounds (thousands) and a top speed of 1150 knots the cost estimator would first compute the values for hidden nodes h1 h2 and h3 determined to be 09322 -01886 and 06457 respectively Next the hidden node values are applied to the output function Equation (8) resulting in a value of 13147 Thus the cost estimator would identify the unknown acquisition cost for the notional aircraft to be $1315 billion

In reviewing cost-estimating literature we note that it appears the mulshytilayer perceptron with a logistic sigmoid function is the most commonly applied neural network technique Chiu and Huang (2007) Cirilovic Vajdic Mladenovic and Queiroz (2014) Dejaneger et al (2012) Finnie et al (1997) Huang Chiu and Chen (2008) Kim An and Kang (2004) Park and Baek (2008) Shehab Farooq Sandhu Nguyen and Nasr (2010) and Zhang Fuh and Chan (1996) all utilize the logistic sigmoid function However we disshycover that other neural network techniques are used To estimate software development cost Heiat (2002) utilizes a Gaussian function rather than a logistic sigmoid within the hidden layer Kumar Ravi Carr and Kiran

April 2017

(2008) and Dejaeger et al (2012) test both the logistic sigmoid and Gaussian functions finding that the logistic sigmoid is more accurate in predicting software development costs

Meta-analysis of Nonparametric Data Mining Performance

Having defined three nonparametric data mining techniques common to cost estimating we investigate which technique appears to be the most predictive for cost estimates We adopt a method known as meta-analysis which is common to research in the social science and medical fields In conshytrast to the traditional literature review meta-analysis adopts a quantitative approach to objectively review past study results Meta-analysis avoids author biases such as selective inclusion of studies subjective weighting of study importance or misleading interpretation of study results (Wolf 1986)

Data To the best of our ability we search for all cost-estimating research

studies comparing the predictive accuracy of two or more data mining techshyniques We do not discover any comparative data mining studies utilizing only DoD cost data and thus we expand our search to include studies involvshying industry cost data As shown in Appendix B 14 unique research studies are identified of which the majority focus on software cost estimating

We observe that some research studies provide accuracy results for mulshytiple data sets in this case each data set is treated as a separate research result for a total of 32 observations When multiple variations of a given nonparametric technique are reported within a research study we record the accuracy results from the best performing variation After aggregating our data we annotate that Canadian Financial IBM DP Services and other software data sets are reused across research studies but with significantly different accuracy results We therefore elect to treat each reuse of a data set as a unique research observation

As a summary 25 of 32 (78 percent) data sets relate to software development We consider this a research limitation and address it later Of the remaining data sets five focus on construction one focuses on manufacturing and one focuses on shipbuilding The largest data set contains 1160 observations and the smallest contains 19 observations The mean data set contains 1445 observations while the median data set contains 655 observations

Methodology It is commonly the goal of meta-analysis to compute a lsquopooledrsquo average

of a common statistical measure across studies or data sets (Rosenthal 1984 Wolf 1986) We discover this is not achievable in our analysis for two reasons First the studies we review are inconsistent in their usage of an accuracy measure As an example it would be inappropriate to pool a Mean Absolute Percent Error (MAPE) value with an R2 (coefficient of detershymination) value Second not all studies compare OLS regression against all three nonparametric data mining techniques Pooling the results of a research study reporting the accuracy metric for only two of the data mining techniques would potentially bias the pooled results Thus an alternative approach is needed

We adopt a simple win-lose methodology where the data mining techniques are competed lsquo1-on-1rsquo for each data set For data sets reporting errormdashsuch as MAPE or Mean Absolute Error Rate (MAER)mdashas an accuracy measure we assume that the data mining technique with the smallest error value is optimal and thus the winner For data sets reporting R2 we assume that the data mining technique with the greatest R2 value is optimal and thus the winner In all instances we rely upon the reported accuracy of the validashytion data set not the training data set In a later section we emphasize the necessity of using a validation data set to assess model accuracy

Results As summarized in Table 2 and shown in detail in Appendix C nonshy

parametric techniques provide more accurate cost estimates than OLS regression on average for the studies included in our meta-analysis Given a lsquo1-on-1rsquo comparison nearest neighbor wins against OLS regression for 20 of 21 comparisons (95 percent) regression trees win against OLS regression for nine of 11 comparisons (82 percent) and artificial neural networks win against OLS regression for 19 of 20 comparisons (95 percent)

TABLE 2 SUMMARY OF META ANALYSIS WIN LOSS RESULTS

Nearest N

Wins-Losses

April 2017

We also report the performance of the nonparametric techniques in relashytion to each other It appears that the nearest neighbor technique is the most dominant nonparametric technique However for reasons explained in our limitations we assert that these results are not conclusive For the practitioner applying these techniques multiple data mining techniques should be considered as no individual technique is guaranteed to be the best tool for a given cost estimate The decision of which technique is most appropriate should be based on each techniquersquos predictive performance as well as consideration of potential pitfalls to be discussed later

Limitations and Follow-on Research We find two major limitations to the meta-analysis result As the first

major limitation 78 percent of our observed data sets originate from softshyware development If the software development data sets are withheld we do not have enough data remaining to ascertain the best performing nonshyparametric technique for nonsoftware applications

As a second major limitation we observe several factors that may contribshyute to OLS regressionrsquos poor meta-analysis performance First the authors cited in our meta-analysis employ an automated process known as stepwise regression to build their OLS regression models Stepwise regression has been shown to underperform in the presence of correlated variables and allows for the entry of noise variables (Derksen amp Keselman 1992) Second the authors did not consider interactions between predictor variables which indicates that moderator effects could not be modeled Third with the exception of Dejaeger et al (2012) Finnie et al (1997) and Heiat (2002) the authors did not allow for mathematical transformations of OLS regression variables meaning the regression models were incapable of modeling nonshylinear relationships This is a notable oversight as Dejaenger et al (2012) find that OLS regression with a logarithmic transformation of both the input and output variables can outperform nonparametric techniques

Given the limitations of our meta-analysis we suggest that follow-on research would be beneficial to the acquisition community Foremost research is needed that explores the accuracy of nonparametric techniques for estimating the cost of nonsoftware DoD-specific applications such as aircraft ground vehicles and space systems To be most effective the research should compare nonparametric data mining performance against the accuracy of a previously established OLS regression cost model which considers both interactions and transformations

Potential Data Mining Pitfalls Given the comparative success of nonparametric data mining techshy

niques within our meta-analysis is it feasible that these techniques be adopted by the program office-level cost estimator We assert that nonparashymetric data mining is within the grasp of the experienced cost estimator but several potential pitfalls must be considered These pitfalls may also serve as a discriminator in selecting the optimal data mining technique for a given cost estimate

Interpretability to Decision Makers When selecting the optimal data mining technique for analysis there

is generally a trade-off between interpretability and flexibility (James et al 2013 p 394) As an example the simple linear regression model has low flexibility in that it can only model a linear relationship between a single program attribute and cost On the other hand the simple linear regression

April 2017

offers high interpretability as decision makers are able to easily intershypret the significance of a single linear relationship (eg as aircraft weight increases cost increases as a linear function of weight)

As more f lexible data mining techniques are applied such as bagging boosting or artificial neural networks it becomes increasingly difficult to explain the results to the decision maker Cost estimators applying such data mining techniques risk having their model become a lsquoblack boxrsquo where the calculations are neither seen nor understood by the decision maker Although the model outputs may be accurate the decision maker may have less confidence in a technique that cannot be understood

Risk of Overfitting More flexible nonlinear techniques have another undesirable effectmdash

they can more easily lead to overfitting Overfitting means that a model is overly influenced by the error or noise within a data set The model may be capturing the patterns caused by random chance rather than the fundashymental relationship between the performance attribute and cost (James et al 2013) When this occurs the model may perform well for the training data set but perform poorly when used to estimate a new program Thus when employing a data mining technique to build a cost-estimating model it is advisable to separate the historical data set into training and validation sets otherwise known as holdout sets The training set is used to lsquotrainrsquo the model while the validation data set is withheld to assess the predictive accuracy of the model developed Alternatively when the data set size is limited it is recommended that the estimator utilize the cross-validation method to validate model performance (Provost amp Fawcett 2013)

Extrapolation Two of the nonparametric techniques considered nearest neighbor and

regression trees are incapable of estimating beyond the historical observashytion range For these techniques estimated cost is limited to the minimum or maximum cost of the historical observations Therefore the application of these techniques may be inappropriate for estimating new programs whose performance or program characteristics exceed the range for which we have historical data In contrast it is possible to extrapolate beyond the bounds of historical data using OLS regression As a cautionary note while it is possible to extrapolate using OLS regression the cost estimator should be aware that statisticians consider extrapolation a dangerous practice (Newbold Carlson amp Thorne 2007) The estimator should generally avoid extrapolating as it is unknown whether the cost estimating relationship retains the same slope outside of the known range (DAU 2009)

Spurious Correlation Lastly we introduce a pitfall that is common across all data mining

techniques As our ability to quickly gather data improves the cost estishymator will naturally desire to test a greater number of predictor variables within a cost estimating model As a result the incidence of lsquospuriousrsquo or coincidental correlations will increase Given a 95 percent confidence level if the cost estimator considers 100 predictor variables for a cost model it is expected that approximately five variables will appear statistically sigshynificant purely by chance Thus we are reminded that correlation does not imply causation In accordance with training material from the Air Force Cost Analysis Agency (AFCAA) the most credible cost models remain those that are verified and validated by engineering theory (AFCAA 2008)

Summary As motivation for this article Lamb (2016) reports that 43 percent of

senior leaders in cost estimating believe that data mining is a most useful tool for analysis Despite senior leadership endorsement we find minimal acquisition research utilizing nonparametric data mining for cost estimates A consolidation of relevant non-DoD research is needed to encourage the implementation of data mining techniques in acquisition cost estimating

A consolidation of relevant non-DoD research is needed to encourage the implementation of data mining techniques in acquisition cost estimating

We turn to academic research utilizing industry data finding relevant cost estimating studies that use software manufacturing and construction data sets to compare data mining performance Through a meta-analysis it is revealed that nonparametric data mining techniques consistently outpershyform OLS regression for industry cost-estimating applications The meta-analysis results indicate that nonparametric techniques should at a minimum be at least considered for the DoD acquisition cost estimates

However we recognize that our meta-analysis suffers from limitations Follow-on data mining research utilizing DoD-specific cost data is strongly recommended The follow-on research should compare nonparametric data mining techniques against an OLS regression model which considers both

April 2017

interactions and transformations Furthermore we are honest in recognizshying that the application of nonparametric data mining is not without serious pitfalls including decreased interpretability to decision makers and the risk of overfitting data

Despite these limitations and pitfalls we predict that nonparametric data mining will become increasingly relevant to cost estimating over time The DoD acquisition community has recently introduced CADE a new data collection initiative Whereas the cost estimator historically faced the problem of having too little datamdashwhich was time-intensive to collect and inconsistently formattedmdashit is entirely possible that in the future we may have more data than we can effectively analyze Thus as future data sets grow larger and more complex we assert that the flexibility offered by nonparametric data mining techniques will be critical to reaching senior leadershiprsquos vision for more innovative analyses

References AFCAA (2008) Air Force cost analysis handbook Washington DC Author Bishop C M (2006) Pattern recognition and machine learning New York Springer Braga P L Oliveira A L Ribeiro G H amp Meira S R (2007) Bagging predictors

for estimation of software project effort Proceedings of the 2007 International Joint Conference on Neural Networks August 12-17 Orlando FL doi101109 ijcnn20074371196

Chiu N amp Huang S (2007) The adjusted analogy-based software effort estimation based on similarity distances Journal of Systems and Software 80(4) 628ndash640 doi101016jjss200606006

Cirilovic J Vajdic N Mladenovic G amp Queiroz C (2014) Developing cost estimation models for road rehabilitation and reconstruction Case study of projects in Europe and Central Asia Journal of Construction Engineering and Management 140(3) 1ndash8 doi101061(asce)co1943-78620000817

Defense Acquisition University (2009) BCF106 Fundamentals of cost analysis [DAU Training Course] Retrieved from httpwwwdaumilmobileCourseDetails aspxid=482

Dejaeger K Verbeke W Martens D amp Baesens B (2012) Data mining techniques for software effort estimation A comparative study IEEE Transactions on Software Engineering 38(2) 375ndash397 doi101109tse201155

Derksen S amp Keselman H J (1992) Backward forward and stepwise automated subset selection algorithms Frequency of obtaining authentic and noise variables British Journal of Mathematical and Statistical Psychology 45(2) 265ndash282 doi101111j2044-83171992tb00992x

Dopkeen B R (2013) CADE vision for NDIAs program management systems committee Presentation to National Defense Industrial Association Arlington VA Retrieved from httpdcarccapeosdmilFilesCSDRSRCSDR_Focus_ Group_Briefing20131204pdf

Fayyad U Piatetsky-Shapiro G amp Smyth P (1996 Fall) From data mining to knowledge discovery in databases AI Magazine 17(3) 37ndash54

Finnie G Wittig G amp Desharnais J (1997) A comparison of software effort estimation techniques Using function points with neural networks case-based reasoning and regression models Journal of Systems and Software 39(3) 281ndash289 doi101016s0164-1212(97)00055-1

Friedman J (1997) Data mining and statistics Whats the connection Proceedings of the 29th Symposium on the Interface Computing Science and Statistics May 14-17 Houston TX

GAO (2005) Data mining Federal efforts cover a wide range of uses (Report No GAO-05-866) Washington DC US Government Printing Office

GAO (2006) DoD needs more reliable data to better estimate the cost and schedule of the Shchuchrsquoye facility (Report No GAO-06-692) Washington DC US Government Printing Office

GAO (2010) DoD needs better information and guidance to more effectively manage and reduce operating and support costs of major weapon systems (Report No GAO-10-717) Washington DC US Government Printing Office

Hand D (1998) Data mining Statistics and more The American Statistician 52(2) 112ndash118 doi1023072685468

April 2017

Hastie T Tibshirani R amp Friedman J H (2008) The elements of statistical learning Data mining inference and prediction New York Springer

Heiat A (2002) Comparison of artificial neural network and regression models for estimating software development effort Information and Software Technology 44(15) 911ndash922 doi101016s0950-5849(02)00128-3

Hess R amp Romanoff H (1987) Aircraft airframe cost estimating relationships All mission types Retrieved from httpwwwrandorgpubsnotesN2283z1html

Huang S Chiu N amp Chen L (2008) Integration of the grey relational analysis with genetic algorithm for software effort estimation European Journal of Operational Research 188(3) 898ndash909 doi101016jejor200707002

James G Witten D Hastie T amp Tibshirani R (2013) An introduction to statistical learning With applications in R New York NY Springer

Kaluzny B L Barbici S Berg G Chiomento R Derpanis D Jonsson U Shaw A Smit M amp Ramaroson F (2011) An application of data mining algorithms for shipbuilding cost estimation Journal of Cost Analysis and Parametrics 4(1) 2ndash30 doi1010801941658x2011585336

Kim G An S amp Kang K (2004) Comparison of construction cost estimating models based on regression analysis neural networks and case-based reasoning Journal of Building and Environment 39(10) 1235ndash1242 doi101016j buildenv200402013

Kumar K V Ravi V Carr M amp Kiran N R (2008) Software development cost estimation using wavelet neural networks Journal of Systems and Software 81(11) 1853ndash1867 doi101016jjss200712793

Lamb T W (2016) Cost analysis reform Where do we go from here A Delphi study of views of leading experts (Masters thesis) Air Force Institute of Technology Wright-Patterson Air Force Base OH

McAfee A amp Brynjolfsson E (2012) Big datamdashthe management revolution Harvard Business Review 90(10) 61ndash67

Newbold P Carlson W L amp Thorne B (2007) Statistics for business and economics Upper Saddle River NJ Pearson Prentice Hall

Park H amp Baek S (2008) An empirical validation of a neural network model for software effort estimation Expert Systems with Applications 35(3) 929ndash937 doi101016jeswa200708001

PricewaterhouseCoopers LLC (2015) 18th annual global CEO survey Retrieved from httpdownloadpwccomgxceo-surveyassetspdfpwc-18th-annual-globalshyceo-survey-jan-2015pdf

Provost F amp Fawcett T (2013) Data science for business What you need to know about data mining and data-analytic thinking Sebastopol CA OReilly Media

Rosenthal R (1984) Meta-analytic procedures for social research Beverly Hills CA Sage Publications

Shehab T Farooq M Sandhu S Nguyen T amp Nasr E (2010) Cost estimating models for utility rehabilitation projects Neural networks versus regression Journal of Pipeline Systems Engineering and Practice 1(3) 104ndash110 doi101061 (asce)ps1949-12040000058

Shepperd M amp Schofield C (1997) Estimating software project effort using analogies IEEE Transactions on Software Engineering 23(11) 736ndash743 doi10110932637387

Shin Y (2015) Application of boosting regression trees to preliminary cost estimation in building construction projects Computational Intelligence and Neuroscience 2015(1) 1ndash9 doi1011552015149702

Smith D (2015) R is the fastest-growing language on StackOverflow Retrieved from httpblogrevolutionanalyticscom201512r-is-the-fastest-growing-languageshyon-stackoverflowhtml

Watern K (2016) Cost Assessment Data Enterprise (CADE) Air Force Comptroller Magazine 49(1) 25

Wolf F M (1986) Meta-analysis Quantitative methods for research synthesis Beverly Hills CA Sage Publications

Zhang Y Fuh J amp Chan W (1996) Feature-based cost estimation for packaging products using neural networks Computers in Industry 32(1) 95ndash113 doi101016 s0166-3615(96)00059-0

April 2017

Appendix A RAND Aircraft Data Set

Model Program Cost Airframe Weight Maximum Speed Billions Thousands (Knots)

(Base Year 1977) (Pounds) A-3 1015 2393 546

A-4 373 507 565

A-5 1414 2350 1147

A-6 888 1715 562

A-7 33 1162 595

A-10 629 1484 389

B-52 3203 11267 551

B-58 3243 3269 1147

BRB-66 1293 3050 548

C-130 1175 4345 326

C-133 1835 9631 304

KC-135 1555 7025 527

C-141 1891 10432 491

F3D 303 1014 470

F3H 757 1390 622

F4D 71 874 628

F-4 1399 1722 1222

F-86 248 679 590

F-89 542 1812 546

F-100 421 1212 752

F-101 893 1342 872

F-102 1105 1230 680

F-104 504 796 1150

F-105 1221 1930 1112

F-106 1188 1462 1153

F-111 2693 3315 1262

S-3 1233 1854 429

T-38 437 538 699

T-39 257 703 468

Note Adapted from ldquoAircraft Airframe Cost Estimating Relationships All Mission Typesrdquo by R Hess and H Romanoff 1987 p11 80 Retrieved from httpwwwrandorgpubs notesN2283z1html

Validate

Nearest N

Accuracy Measure

April 2017

Validate

Nearest N

Accuracy Measure

April 2017

coeffi

Appendix C Meta-Analysis Win-Loss Results

Nearest N

1 Lose Win Win Lose Lose Win Win Lose Win Lose Lose Win

2 Lose Win Win Lose Win Lose Win Lose Win Lose Win Lose

3 Lose Win

4 Lose Win

5 Lose Win Lose Win Lose Win Lose Win Win Lose Win Lose

7 Lose Win Lose Win Lose Win Win Lose Win Lose Win Lose

10 Lose Win Lose Win Lose Win Win Lose Lose Win Lose Win

12 Win Lose Lose Win Lose Win Lose Win Lose Win Lose Win

13 Lose Win Lose Win Lose Win Win Lose Win Lose Lose Win

14 Lose Win Lose Win Lose Win

15 Lose Win

17 Win Lose Win Lose Win Lose

18 Lose Win

April 2017

Appendix C continued

Nearest N

20 Lose Win

21 Lose Win

22 Lose Win

23 Lose Win

24 Lose Win

25 Lose Win

26 Lose Win

27 Lose Win

28 Lose Win

29 Lose Win

30 Lose Win

31 Win Lose

32 Lose Win

Wins 1 20 2 9 1 19 8 6 10 5 9 5

Losses

20 1 9 2 19 1 6 8 5 10 5 9

Author Biographies

Capt Gregory E Brown USAF is the cost chief for Special Operations Forces and Personnel Recovery Division Air Force Life Cycle Management Center Wright-Patterson Air Force Base Ohio He received a BA in Economics and a BS in Business-Finance from Colorado State University and an MS in Cost Analysis from the Air Force Institute of Technology Capt Brown is currently enrolled in graduate courseshywork in Applied Statistics through Pennsylvania State University

(E-mail address GregoryBrown34usafmil)

Dr Edward D White is a professor of statistics in the Department of Mathematics and Statistics at the Air Force Institute of Technology Wright-Patterson Air Force Base Ohio He received his MAS from Ohio State University and his PhD in Statistics from Texas AampM University Dr Whitersquos primary research interests include statistical modeling simulation and data analytics

(E-mail address EdwardWhiteafitedu)

April 2017

CRITICAL SUCCESS FACTORS for Crowdsourcing with Virtual Environments TO UNLOCK INNOVATION

Glenn E Romanczuk Christopher Willy and John E Bischoff

Senior defense acquisition leadership is increasingly advocating new approaches that can enhance defense acquisition Their constant refrain is increased innovation collaboration and experimentation The then Under Secretary of Defense for Acquisition Technology and Logistics Frank Kendall in his 2014 Better Buying Power 30 White Paper called to ldquoIncentivize inno vation hellip Increase the use of prototyping and experimentationrdquo This article explores a confluence of technologies holding the key to faster development time linked to real warfighter evaluations Innovations in Model Based Systems Engineering (MBSE) crowdsourcing and virtual environments can enhance collaboration This study focused on finding critical success factors using the Delphi method allowing virtual environments and MBSE to produce needed feedback and enhance the process The Department of Defense can use the emerging findings to ensure that systems developed reflect stakeholdersrsquo requirements Innovative use of virtual environments and crowdsourcing can decrease cycle time required to produce advanced innovative systems tailored to meet warfighter needs

DOI httpsdoiorg1022594dau16 7582402 (Online only) Keywords Delphi method collaboration innovation expert judgment

Crowdsourcing with Virtual Environments httpwwwdaumil

A host of technologies and concepts holds the key for reducing develshyopment time linked to real warfighter evaluation and need Innovations in MBSE networking and virtual environment technology can enable collaboshyration among the designers developers and end users and can increasingly be utilized for warfighter crowdsourcing (Smith amp Vogt 2014) The innoshyvative process can link ideas generated by warfighters using game-based virtual environments in combination with the ideas ranking and filtering of the greater engineering staff The DoD following industryrsquos lead in crowd-sourcing can utilize the critical success factors and methods developed in this research to reduce the time needed to develop and field critical defense systems Innovative use of virtual environments and crowdsourcing can increase the usefulness of weapon systems to meet the real needs of the true stakeholdersmdashthe warfighters

The DoD as a whole has begun looking for efficiency by employing innoshyvation crowdsourcing MBSE and virtual environments (Zimmerman 2015) Industry has led the way with innovative use of crowdsourcing for design and idea generation Many of these methods utilize the public at large However this study will focus on crowdsourcing that uses warfightshyers and the larger DoD engineering staff along with MBSE methodologies This study focuses on finding the critical success factors or key elements and developing a process (framework) to allow virtual environments and MBSE to continually produce feedback from key stakeholders throughout the design cycle not just at the beginning and end of the process The proshyposed process has been developed based on feedback from a panel of experts using the Delphi method The Delphi method created by RAND in the 1950s allows for exploration of solutions based on expert opinion (Dalkey 1967) This study utilized a panel of 20 experts in modeling and simulation (MampS) The panel was a cross section of Senior Executive Service senior Army Navy and DoD engineering staff and academics with experience across the range of virtual environments MampS MBSE and human systems integrashytion (HSI) The panel developed critical success factors in each of the five areas explored MBSE HSI virtual environments crowdsourcing and the overall process HSI is an important part of the study because virtual envishyronments can enable earlier detailed evaluation of warfighter integration in the system design

Many researchers have conducted studies that looked for methods to make military systems design and acquisition more fruitful A multitude of studshyies conducted by the US Government Accountability Office (GAO) has also investigated the failures of the DoD to move defense systems from the early stages of conceptualization to finished designs useful to warfighters The

April 2017

GAO offered this observation ldquoSystems engineering expertise is essential throughout the acquisition cycle but especially early when the feasibility of requirements are [sic] being determinedrdquo (GAO 2015 p 8) The DoD process is linked to the systems engineering process through the mandated use of the DoD 5000-series documents (Ferrara 1996) However for many reasons major defense systems design and development cycles continue to fail major programs are canceled systems take too long to finish or costs are significantly expanded (Gould 2015) The list of DoD acquisition projects either canceled or requiring significantly more money or time to complete is long Numerous attempts to redefine the process have fallen short The DoD has however learned valuable lessons as a result of past failures such as the Future Combat System Comanche Next Generation Cruiser CG(X) and the Crusader (Rodriguez 2014) A partial list of those lessons includes the need for enhanced requirements generation detailed collaboration with stakeholders and better systems engineering utilizing enhanced tradespace tools

Innovative use of virtual environments and crowdsourcing can increase the usefulness of weapon systems to meet the real needs of the true stakeholdersmdashthe warfighters

The DoD is now looking to follow the innovation process emerging in indusshytry to kick-start the innovation cycle and utilize emerging technologies to minimize the time from initial concept to fielded system (Hagel 2014) This is a challenging goal that may require significant review and restructuring of many aspects of the current process In his article ldquoDigital Pentagonrdquo Modigliani (2013) recommended a variety of changes including changes to enhance collaboration and innovation Process changes and initiatives have been a constant in DoD acquisition for the last 25 years As weapons have become more complex software-intensive and interconnected DoD has struggled to find the correct mix of process and innovation The DoD acquisition policy encourages and mandates the utilization of systems engineering methods to design and develop complex defense systems It is hoped that the emergence of MBSE concepts may provide a solid foundation and useful techniques that can be applied to harness and focus the fruits of the rapidly expanding innovation pipeline

The goal and desire to include more MampS into defense system design and development has continually increased as computer power and software tools have become more powerful Over the past 25 years many new efforts have been launched to focus the utilization of advanced MampS The advances in MampS have led to success in small pockets and in selected design efforts but have not diffused fully across the entire enterprise Several different process initiatives have been attempted over the last 30 years The acquisishytion enterprise is responsible for the process which takes ideas for defense systems initiates programs to design develop and test a system and then manages the program until the defense system is in the warfightersrsquo hands A few examples of noteworthy process initiatives are Simulation Based Acquisition (SBA) Simulation and Modeling for Acquisition Requirements and Training (SMART) Integrated Product and Process Development (IPPD) and now Model Based Systems Engineering (MBSE) and Digital Engineering Design (DED) (Bianca 2000 Murray 2014 Sanders 1997 Zimmerman 2015) These process initiatives (SBA SMART and IPPD) helped create some great successes in DoD weapon systems however the record of defense acquisition and the amount of time required to develop more advanced and increasingly complex interoperable weapon systems has been mixed at best The emerging MBSE and DED efforts are too new to fully evaluate their contribution

As weapons have become more complex software-intensive and interconnected DoD has struggled to find the correct mix of process and innovation

The Armyrsquos development of the Javelin (AAWS-M) missile system is an interesting case study of a successful program that demonstrated the abilshyity to overcome significant cost technical and schedule risks Building on design and trade studies conducted by the Defense Advanced Research Projects Agency (DARPA) during the 1970s and utilizing a competitive prototype approach the Army selected an emerging (imaging infrared seeker) technology from the three technology choices proposed The innoshyvative Integrated Flight Simulation originally developed by the Raytheon Lockheed joint venture also played a key role in Javelinrsquos success The final selection was heavily weighted toward ldquofire-and-forgetrdquo technology that although costly and immature at the time provided a significant benefit

April 2017

to the warfighter (David 1995 Lyons Long amp Chait 2006) This is a rare example of warfighter input and unique MampS efforts leading to a successful program In contrast to Javelinrsquos successful use of innovative modeling and simulation is the Armyrsquos development of Military Operations on Urbanized Terrain (MOUT) weapons In design for 20 years and still under developshyment is a new urban shoulder-launched munition for MOUT application now called the Individual Assault Munition (IAM) The MOUT weapon acquisition failure was in part due to challenging requirements However the complex competing technical system requirements might benefit from the use of detailed virtual prototypes and innovative game-based war-

The record of defense acquisition and the amount of time required to develop more advanced and increasingly complex interoperable weapon systems has been mixed at best

fighter and engineer collaboration IAM follows development of the Armyrsquos Multipurpose Individual Munition (MPIM) a program started by the Army around 1994 and canceled in 2001 Army Colonel Richard Hornstein indicates that currently after many program changes and requirements updates system development of IAM will now begin again in the 2018 timeframe However continuous science and technology efforts at both US Army Armament Research Development and Engineering Center (ARDEC) and US Army Aviation and Missile Research Development and Engineering Center (AMRDEC) have been maintained for this type of system Many of our allies and other countries in the world are actively developing MOUT weapons (Gourley 2015 Janersquos 2014) It is hoped that by using the framework and success factors described in this article DoD will accelerate bringing needed capabilities to the warfighter using innovative ideas and constant soldier sailor and airman input With the changing threat environment in the world the US military can no longer allow capability gaps to be unfilled for 20 years or just wait to purchase similar systems from our allies The development of MOUT weapons is an applicashytion area that is ripe for the methods discussed in this article This study and enhanced utilization of virtual environments cannot correct all of the problems in defense acquisition However it is hoped that enhanced utilishyzation of virtual environments and crowdsourcing as a part of the larger

effort into Engineered Resilient Systems (ERS) and expanded tradespace tools can provide acquisition professionals innovative ways to accelerate successful systems development

BACKGROUND Literature Review

This article builds upon detailed research by Murray (2014) Smith and Vogt (2014) London (2012) Korfiatis Cloutier and Zigh (2015) Corns and Kande (2011) and Madni (2015) that covered elements of crowdsourcing virtual environments gaming early systems engineering and MBSE The research study described in this article was intended to expand the work discussed in this section and determine the critical success factors for using MBSE and virtual environments to harvest crowdsourcing data from war-fighters and stakeholders and then provide that data to the overall Digital System Model (DSM) The works reviewed in this section address virtual environments and prototyping MBSE and crowdsourcing The majority of these are focused on the conceptualization phase of product design However these tools can be used for early product design and integrated into the detailed development phase up to Milestone C the production and deployment decision

Many commercial firms and some government agencies have studied the use of virtual environments and gaming to create ldquoserious gamesrdquo that have a purpose beyond entertainment (National Research Council [NRC] 2010) Commercial firms and DARPA have produced studies and programs to utilize an open innovation paradigm General Electric for one is comshymitted to ldquocrowdsourcing innovationmdashboth internally and externally hellip [b]y sourcing and supporting innovative ideas wherever they might come fromhelliprdquo (General Electric 2017 p 1)

Researchers from many academic institutions are also working with open innovation concepts and leveraging input from large groups for concept creation and research into specific topics Dr Stephen Mitroff of The George Washington University created a popular game while at Duke University that was artfully crafted not only to be entertaining but also to provide researchers access to a large pool of research subjects Figure 1 shows a sample game screen The game allows players to detect dangerous items from images created to look like a modern airport X-ray scan The research utilized the game results to test hypotheses related to how the human brain detects multiple items after finding similar items In addition the game allowed testing on how humans detect very rare and dangerous items The

April 2017

game platform allowed for a large cross section of the population to interact and assist in the research all while having fun One of the keys to the useshyfulness of this game as a research platform is the ability to ldquophone homerdquo or telemeter the details of the player-game interactions (Drucker 2014 Sheridan 2015) This research showed the promise of generating design and evaluation data from a diverse crowd of participants using game-based methods

FIGURE 1 AIRPORT SCANNER SCREENSHOT

Note (Drucker 2014) Used by permission Kedlin Company

Process Several examples of process-related research that illustrates beginshy

ning inquiry into the use of virtual environments and MBSE to enhance systems development are reviewed in this section Marine Corps Major Kate Murray (2014) explored the data that can be gained by the use of a conceptual Early Synthetic Prototype (ESP) environment The envisioned environment used game-based tools to explore requirements early in the design process The focus of her study was ldquoWhat feedback can be gleaned and is it useful to decision makersrdquo (Murray 2014 p 4) This innovative thesis ties together major concepts needed to create an exploration of design within a game-based framework The study concludes that ESP should be utilized for Pre-Milestone A efforts The Pre-Milestone A efforts are domishynated by concept development and materiel solutions analysis Murray also discussed many of the barriers to fully enabling the conceptual vision that she described Such an ambitious project would require the warfighters to be able to craft their own scenarios and add novel capabilities An interesting viewpoint discussed in this research is that the environment must be able to interest the warfighters enough to have them volunteer their game-playing time to assist in the design efforts The practical translation of this is that the environment created must look and feel like similar games played by the warfighters both in graphic detail and in terms of game challenges to ldquokeep hellip players engagedrdquo (Murray 2014 p 25)

Corns and Kande (2011) describe a virtual engineering tool from the University of Iowa VE-Suite This tool utilizes a novel architecture includshying a virtual environment Three main engines interact an Xplorer a Conductor and a Computational engine In this effort Systems Modeling Language (SysML) and Unified Modeling Language (UML) diagrams are integrated into the overall process A sample environment is depicted simshyulating a fermentor and displaying a virtual prototype of the fermentation process controlled by a user interface (Corns amp Kande 2011) The extent and timing of the creation of detailed MBSE artifacts and the amount of integration achievable or even desirable among specific types of modeling languagesmdasheg SysML and UMLmdashare important areas of study

In his 2012 thesis Brian London described an approach to concept creation and evaluation The framework described utilizes MBSE principles to assist in concept creation and review The benefits of the approach are explored through examples of a notional Unmanned Aerial Vehicle design project Various SysML diagrams are developed and discussed This approach advoshycates utilization of use-case diagrams to support the Concept of Operations (CONOPS) review (London 2012)

April 2017

Carlini (2010) in the Director Defense Research and Engineering Rapid Toolbox Study called for accelerated concept engineering with an expanded use of both virtual and physical prototypes and support for more innovative interdisciplinary red teams In this article the terms ldquovirtual environmentrdquo and ldquovirtual prototyperdquo can be used interchangeably Korfiatis Cloutier and Zigh (2015) authored a series of articles between 2011 and 2015 related to CONOPS development and early systems engineering design methods The Integrated Concept Engineering Framework evolved out of numerous research projects and articles looking at the combination of gaming and MBSE methods related to the task of CONOPS creation This innovative work shows promise for the early system design and ideation stages of the acquisition cycle There is recognition in this work that the warfighter will need an easy and intuitive way to add content to the game and modify the parameters that control objects in the game environment (Korfiatis et al 2015)

Madni (2015) explored the use of storytelling and a nontechnical narrative along with MBSE elements to enable more stakeholder interaction in the design process He studied the conjunction of stakeholder inputs nontradishytional methods and the innovative interaction between the game engine the virtual world and the creation of systems engineering artifacts The virtual worlds created in this research also allowed the players to share common views of their evolving environment (Madni 2015 Madni Nance Richey Hubbard amp Hanneman 2014) This section has shown that researchers are exploring virtual environments with game-based elements sometimes mixed with MBSE to enhance the defense acquisition process

Crowdsourcing Wired magazine editors Jeff Howe and Mark Robinson coined the

term ldquocrowdsourcingrdquo in 2005 In his Wired article titled ldquoThe Rise of Crowdsourcingrdquo Howe (2006) described several types of crowdsourcing The working definition for this effort is hellip the practice of obtaining needed services ideas design or content by soliciting contributions from a large group of people and especially from the system stakeholders and users rather than only from traditional employees designers or management (Crowdsourcing nd)

The best fit for crowdsourcing conceptually for this current research projshyect is the description of research and development (RampD) firms utilizing the InnoCentive Website to gain insights from beyond their in-house RampD team A vital feature in all of the approaches is the use of the Internet and modern computational environments to find needed solutions or content using the

diversity and capability of ldquothe crowdrdquo at significant cost or time savings The DoD following this lead is attempting to explore the capabilities and solutions provided by the utilization of crowdsourcing concepts The DoD has numerous restrictions that can hinder a full utilization but an artfully crafted application and a focus on components or larger strategic concepts can help to overcome these barriers (Howe 2006)

In a Harvard Business Review article ldquoUsing the Crowd as an Innovation Partnerrdquo Boudreau and Lahkani (2013) discussed the approaches to crowd-sourcing that have been utilized in very diverse areas They wrote ldquoOver the past decade wersquove studied dozens of company interactions with crowds on innovation projects in areas as diverse as genomics engineering operations

research predictive analytics enterprise software development video games mobile apps and marketingrdquo (Boudreau amp Lahkani 2013 p 60)

Boudreau and Lahkani discussed four types of crowdsourcing contests collaborative communities complementors and crowd labor A key enabler of the collaborative communitiesrsquo concept is the utilization of intrinsic motivational factors such as the desire to contribute learn or achieve As evidenced in their article many organizations are clearly taking note of and are beginning to leverage the power of diverse geographically separated ad hoc groups to provide innovative concepts engineering support and a variety of inputs that traditional employees normally would have provided (Boudreau amp Lahkani 2013)

In 2015 the US Navy launched ldquoHatchrdquo The Navy calls this portal a ldquocrowdsourced ideation platformrdquo (Department of the Navy 2015) Hatch is part of a broader concept called the Navy Innovation Network (Forrester 2015 Roberts 2015) With this effort the Navy hopes to build a continuous

April 2017

process of innovation and minimize the barriers for information flow to help overcome future challenges Novel wargaming and innovation pathways are to become the norm not the exception The final tools that will fall under this portal are still being developed However it appears that the Navy has taken a significant step foward to establish structural changes that will simplify the ideation and innovation pipeline and ensure that the Navy uses all of the strengths of the total workforce ldquoCrowdsourcing in all of its forms is emerging as a powerful toolhellip Organizational leaders should take every opportunity to examine and use the various methods for crowdsourcing at every phase of their thinkingrdquo (Secretary of the Navy 2015 p 7)

The US Air Force has also been exploring various crowdsourcing concepts They have introduced the Air Force Collaboratory Website and held a numshyber of challenges and projects centered around three different technology areas Recently the US Air Force opened a challenge prize on its new Website httpwwwairforceprizecom with the goal of crowdsourcing a design concept for novel turbine engines that meet established design requirements and can pass the validation tests designed by the Air Force (US Air Force nd US Air Force 2015)

Model Based Systems Engineering MBSE tools have emerged and are supported by many commercial firms

The path outlined by the International Council on Systems Engineering (INCOSE) in their Systems Engineering Vision 2020 document (INCOSE 2007) shows that INCOSE expects the MBSE environment to evolve into a robust interconnected development environment that can serve all sysshytems engineering design and development functions It remains to be seen if MBSE can transcend the past transformation initiatives of SMART SBA and others on the DoD side The intent of the MBSE section of questions is to identify the key or critical success factors needed for MBSE to integrate into or encompass within a crowdsourcing process in order to provide the benefits that proponents of MBSE promise (Bianca 2000 Sanders 1997)

The Air Force Institute of Technology discussed MBSE and platform-based engineering as it discussed collaborative design in relation to rapidexpeshydited systems engineering (Freeman 2011) The process outlined is very similar to the INCOSE view of the future with MBSE included in the design process Freeman covered the creation of a virtual collaborative environshyment that utilizes ldquotools methods processes and environments that allow engineers warfighters and other stakeholders to share and discuss choices This spans human-system interaction collaboration technology visualshyization virtual environments and decision supportrdquo (Freeman 2011 p 8)

As the DoD looks to use MBSE concepts new versions of the DoD Instruction 500002 and new definitions have emerged These concepts and definitions can assist in developing and providing the policy language to fully utilize an MBSE-based process The Office of the Deputy Secretary of Defense Systems Engineering is working to advance several new approaches related to MBSE New definitions have been proposed for Digital Threads and DED using a DSM The challenges of training the workforce and finding the corshyrect proof-of-principle programs are being addressed (Zimmerman 2015) These emerging concepts can help enable evolutionary change in the way DoD systems are developed and designed

The director of the AMRDEC is looking to MBSE as the ldquoultimate cool wayrdquo to capture the excitement and interest of emerging researchers and scientists to collaborate and think holistically to capture ldquoa single evolving computer modelrdquo (Haduch 2015 p 28) This approach is seen as a unique method to capture the passion of a new generation of government engineers (Haduch 2015)

Other agencies of the federal government are also working on proshygrams based on MBSE David Miller National Aeronautics and Space Administration (NASA) chief technologist indicates that NASA is trying to use the techniques to modernize and focus future engineering efforts across the system life cycle and to enable young engineers to value MBSE as a primary method to accomplish system design (Miller 2015)

The level of interaction required and utilization of MBSE artifacts methods and tools to create control and interact with future virtual environments and simulations is a fundamental challenge

SELECTED VIRTUAL ENVIRONMENT ACTIVITIES

Army Within the Army several efforts are underway to work on various

aspects of virtual environmentssynthetic environments that are importshyant to the Army and to this research Currently efforts are being funded by the DoD at Army Capability Integration Center (ARCIC) Institute for Creative Technologies (ICT) at University of Southern California Naval Postgraduate School (NPS) and at the AMRDEC The ESP efforts managed by Army Lieutenant Colonel Vogt continue to look at building a persistent game-based virtual environment that can involve warfighters voluntarily in design and ideation (Tadjdeh 2014) Several prototype efforts are underway

April 2017

at ICT and NPS to help evolve a system that can provide feedback from the warfighters playing game-based virtual environments that answer real design and strategy questions Key questions being looked at include what metrics to utilize how to distribute the games and whether the needed data can be saved and transmitted to the design team Initial prototype environments have been built and tested The ongoing work also looks at technologies that could enable more insight into the HSI issues by attemptshying to gather warfighter intent from sensors or camera data relayed to the ICT team (Spicer et al 2015)

The ldquoAlways ON-ON Demandrdquo efforts being managed by Dr Nancy Bucher (AMRDEC) and Dr Christina Bouwens are a larger effort looking to tie together multiple simulations and produce an ldquoON-Demandrdquo enterprise repository The persistent nature of the testbed and the utilization of virshytual environment tools including the Navy-developed Simulation Display System (SIMDIStrade) tool which utilizes the OpenSceneGraph capability offers exploration of many needed elements required to utilize virtual envishyronments in the acquisition process (Bucher amp Bouwens 2013 US Naval Research Laboratory nd)

Navy Massive Multiplayer Online War Game Leveraging the Internet

(MMOWGLI) is an online strategy and innovation game employed by the US Navy to tap the power of the ldquocrowdrdquo It was jointly developed by the NPS and the Institute for the Future Navy researchers developed the messhysage-based game in 2011 to explore issues critical to the US Navy of the future The game is played based on specific topics and scenarios Some of the games are open to the public and some are more restrictive The way to score points and ldquowinrdquo the game is to offer ideas that other players comment upon build new ideas upon or modify Part of the premise of the approach is based on this statement ldquoThe combined intelligence of our people is an unharnessed pool of potential waiting to be tappedrdquo (Moore 2014 p 3) Utilizing nontraditional sources of information and leveraging the rapidly expanding network and visualization environment are key elements that can transform the current traditional pace of design and acquisition In the future it might be possible to tie this tool to more highly detailed virshytual environments and models that could expand the impact of the overall scenarios explored and the ideas generated

RESEARCH QUESTIONS The literature review demonstrates that active research is ongoing into

crowdsourcing MBSE and virtual environments However there is not a fully developed process model and an understanding of the key elements that will provide the DoD a method to fully apply these innovations to successful system design and development The primary research questions that this study examined to meet this need are

bull What are the critical success factors that enable game-based virtual environments to crowdsource design and requirements information from warfighters (stakeholders)

bull What process and process elements should be created to inject war fighter-developed ideas metrics and feedback from game-based virtual environment data and use cases

bull What is the role of MBSE in this process

METHODOLOGY AND DATA COLLECTION The Delphi technique was selected for this study to identify the critical

success factors for the utilization of virtual environments to enable crowd-sourced information in the system design and acquisition process Delphi is an appropriate research technique to elicit expert judgment where comshyplexity uncertainty and only limited information available on a topic area prevail (Gallop 2015 Skutsch amp Hall 1973) A panel of MampS experts was selected based on a snowball sampling technique Finding experts across DoD and academia was an important step in this research Expertise in MampS as well as virtual environment use in design or acquisition was the primary expertise sought Panel members that met the primary requirement areas but also had expertise in MBSE crowdsourcing or HSI were asked to participate The sampling started with experts identified from the literature search as well as Army experts with appropriate experience known by the researcher Table 1 shows a simplified description of the panel members as well as their years of experience and degree attainment Numerous addishytional academic Air Force and Navy experts were contacted however the acceptance rate was very low

April 2017

TABLE 1 EXPERT PANEL EXPERTISE

DESCRIPTION EDUCATION EXPERIENCE

Academic ResearchermdashAlabama PhD 20-30 years

NavymdashAcademic ResearchermdashCalifornia PhD 20-30 years

Army OfficermdashRequirementsGame Based EnviromentsmdashVirginia

Masters 15-20 years

Army SESmdashMampSmdashRetiredmdashMaryland PhD 30 + years

Navy MampS ExpertmdashVirgina Masters 10-15 years

MampS ExpertmdashArmy SESmdashRetired Masters 30 + years

MampS ExpertmdashArmymdashVirtual Environments Masters 10-15 years

MampS ExpertmdashArmymdashVampV PhD 20-30 years

MampS ExpertmdashArmymdashVirtual Environments PhD 15-20 years

MampS ExpertmdashArmymdashSimulation Masters 20-30 years

MampS ExpertmdashVirtual EnvironmentsGaming BS 15-20 years

MampS ExpertmdashArmymdashSerious Gamesmdash Colorado

PhD 10-15 years

Academic ResearchermdashVirtual EnvironmentsmdashConopsmdashNew Jersey

PhD lt10 years

MampS ExpertmdashArmymdashVisualization Masters 20-30 years

MampS ExpertmdashArmyMDAmdashSystem of Systems Simulation (SoS)

BS 20-30 years

Academic ResearchermdashFlorida PhD 20-30 years

MampS ExpertmdashArmy Virtual Environmentsmdash Michigan

PhD 15-20 years

MampS ExpertmdashArmymdashSimulation PhD 10-15 years

Army MampSmdashSimulationSoS Masters 20-30 years

ArmymdashSimulationmdashSESmdashMaryland PhD 30 + years

Note CONOPS = Concept of Operations MampS = Modeling and Simulation MDA = Missile Defense Agency SES = Senior Executive Services SoS = System of Systems VampV = Verification and Validation

An exploratory ldquointerview-stylerdquo survey was conducted using SurveyMonkey to collect demographic data and answers to a set of 38 questions This surshyvey took the place of the more traditional semistructured interview due to numerous scheduling conflicts In addition each member of the expert panel was asked to provide three possible critical success factors in the primary research areas Follow-up phone conversations were utilized to

seek additional input from members of the panel A large number of possishyble critical success factors emerged for each focus area Figure 2 shows the demographics of the expert panel (n=20) More than half (55 percent) of the panel have Doctoral degrees and an additional 35 percent hold Masterrsquos degrees Figure 2 also shows the self-ranked expertise of the panel All have interacted with the defense acquisition community The panel has the most experience in MampS followed by expertise in virtual environments MBSE HSI and crowdsourcing Figure 3 depicts a word cloud this figure was created from the content provided by the experts in the interview survey The large text items show the factors that were mentioned most often in the interview survey The initial list of 181 possible critical success factors was collected from the survey with redundant content grouped or restated for each major topic area when developing the Delphi Round 1 survey The expert panel was asked to rank the factors using a 5-element Likert scale from Strongly Oppose to Strongly Agree The experts were also asked to rank their or their groupsrsquo status in that research area ranging from ldquoinnoshyvatorsrdquo to ldquolaggardsrdquo for later statistical analysis

FIGURE 2 EXPERT PANEL DEMOGRAPHICS AND EXPERTISE

Degrees M amp S VE

HSI Crowdsource MBSE

Bachelors 10

Medium 5

Low 10

Low 60

Low 50

High 20

Medium 35

Medium 30

Medium 40

Masters 35

PhD 55 High

95 High 75

Medium 25

April 2017

FIGURE 3 WORDCLOUD FROM INTERVIEW SURVEY

Fifteen experts participated in the Round 1 Delphi study The data generated were coded and statistical data were also computed Figure 4 shows the top 10 factors in each of four areas developed in Round 1mdashvirtual environments crowdsourcing MBSE and HSI The mean Interquartile Range (IQR) and percent agreement are shown for 10 factors developed in Round 1

The Round 2 survey included bar graphs with the statistics summarizing Round 1 The Round 2 survey contained the top 10 critical success factors in the five areasmdashwith the exception of the overall process model which contained a few additional possible critical success factors due to survey software error The Round 2 survey shows an expanded Likert scale with seven levels ranging from Strongly Disagree to Strongly Agree The addishytional choices were intended to minimize ties and to help show where the experts strongly ranked the factors

Fifteen experts responded to the Round 2 survey rating the critical success factors determined from Round 1 The Round 2 survey critical success factors continued to receive a large percentage of experts choosing survey values ranging from ldquoSomewhat Agreerdquo to ldquoStrongly Agreerdquo which conshyfirmed the Round 1 top selections But Round 2 data also suffered from an increase in ldquoNeither Agree nor Disagreerdquo responses for success factors past the middle of the survey

FIGURE 4 CRITICAL SUCESS FACTOR RESULTS ROUND 1

VIRTUAL ENVIRONMENTS CRITICAL SUCCESS FACTOR MEAN IQR AGREE

Real Time Operation 467 1 93

Utility to Stakeholders 447 1 93

Fidelity of ModelingAccuracy of Representation 440 1 87

UsabilityEase of Use 440 1 93

Data Recording 427 1 87

Verification Validation and Accreditation 420 1 87

Realistic Physics 420 1 80

Virtual Environment Link to Problem Space 420 1 80

FlexibilityCustomizationModularity 407 1 80

Return On InvestmentCost Savings 407 1 87

CROWDSOURCING CRITICAL SUCCESS FACTOR MEAN IQR AGREE

AccessibilityAvailability 453 1 93

Leadership SupportCommitment 453 1 80

Ability to Measure Design Improvement 447 1 93

Results Analysis by Class of Stakeholder 433 1 93

Data Pedigree 420 1 87

Timely Feedback 420 1 93

Configuration Control 413 1 87

Engaging 413 1 80

Mission Space Characterization 413 1 87

PortalWeb siteCollaboration Area 407 1 87

MBSE CRITICAL SUCCESS FACTOR MEAN IQR AGREE

Conceptual Model of the Systems 460 1 87

Tied to Mission Tasks 443 1 93

Leadership Commitment 440 1 80

ReliabilityRepeatability 433 1 93

Senior Engineer Commitment 433 1 80

FidelityRepresentation of True Systems 427 1 93

Tied To Measures of Performance 427 1 87

Validation 427 1 93

Well Defined Metrics 427 1 80

Adequate Funding of Tools 420 2 73

April 2017

FIGURE 4 CRITICAL SUCESS FACTOR RESULTS ROUND 1 CONTINUED

HSI CRITICAL SUCCESS FACTOR MEAN IQR AGREE

Ability to Capture Human Performance Behavior 464 1 100

Adequate Funding 457 1 100

Ability to Analyze Mental Tasks 436 1 100

Integration with Systems Engineering Process 433 1 87

Intuitive Interfaces 429 125 79

Consistency with Operational Requirements 427 1 93

Data Capture into Metrics 421 1 86

Fidelity 414 1 86

Note IQR = Interquartile Range

The Round 3 survey included the summary statistics from Round 2 and charts showing the expertsrsquo agreement from Round 2 The Round 3 quesshytions presented the top 10 critical success factors in each area and asked the experts to rank these factors The objective of the Round 3 survey was to determine if the experts had achieved a level of consensus regarding the ranking of the top 10 factors from the previous round

PROCESS AND EMERGING CRITICAL SUCCESS FACTOR THEMES

In the early concept phase of the acquisition process more game-like elements can be utilized and the choices of technologies can be very wide The graphical details can be minimized in favor of the overall application area However as this process is applied later in the design cycle more detailed virtual prototypes can be utilized and there can be a greater focus on detailed and subtle design differences that are of concern to the war-fighter The next sections present the overall process model and the critical success factors developed

Process (Framework) ldquoFor any crowdsourcing endeavor to be successful there has to be a

good feedback looprdquo said Maura Sullivan chief of Strategy and Innovation US Navy (Versprille 2015 p 12) Figure 5 illustrates a top-level view of the framework generated by this research Comments and discussion

from the interview phase have been combined with the literature review data and information to create this process Key elements from the Delphi study and the critical success factors have been utilized to shape this proshycess The fidelity of the models utilized would need to be controlled by the visualizationmodelingprototyping centers These centers would provide key services to the warfighters and engineers to artfully create new game elements representing future systems and concepts and to pull information from the enterprise repositories to add customizable game elements

FIGURE 5 CROWDSOURCE INNOVATION FRAMEWORK

MBSESampT Projects

amp Ideas Warfighter

Ideation

Use Case in SysMLUML

Graphical Scenario Development

VisualizationModeling Prototype Centers

Enterprise RepositoryDigital System Models

Collaborative Crowdsource Innovation

Environment

VoteRankComment Feedback

VotingRankingFilter Feedback MBSE

Artifacts

DeployCapture amp Telemeter Metrics

MBSE UMLSysML Artifacts

MBSE Artifacts Autogenerated

Develop Game Models amp Physics

Innovation Portal

Game Engines

RankingPolling Engines

Engage Modeling Team to Add

Game Features

Play GameCompete

Engineers amp Scientists Warfighters

Environments

Models

Decision Engines

MBSE Artifacts

Lethality

Note MBSE = Model Based Systems Engineering SampT = Science and Technology SysMLUML = Systems Modeling LanguageUnified Modeling Language

The expert panel was asked ldquoIs Model Based Systems Engineering necesshysary in this approachrdquo The breakdown of responses revealed that 63 percent responded ldquoStrongly Agreerdquo another 185 percent selected ldquoSomewhat Agreerdquo and the remaining 185 percent answered ldquoNeutralrdquo These results show strong agreement with using MBSE methodologies and concepts as an essential backbone using MBSE as the ldquogluerdquo to manage the use cases and subsequently providing the feedback loop to the DSM

In the virtual environment results from Round 1 real time operation and realistic physics were agreed upon by the panel as critical success factors The appropriate selection of simulation tools would be required to supshyport these factors Scenegraphs and open-source game engines have been evolving and maturing over the past 10 years Many of these tools were commercial products that had proprietary architectures or were expensive

April 2017

However as the trend toward more open-source tools continues game engines have followed the trend Past research conducted by Romanczuk (2012) linked scenegraph tools such as Prospect Panda3D and Delta3D to high-fidelity human injury modeling and lethality application programming interfaces Currently the DoD has tools like VBS2 and VBS3 available but newer commercial-level engines are also becoming free for use by DoD and the public at large Premier game engines such as Source Unity and Unreal are now open-source engines (Heggen 2015) The trend continues as WebGL and other novel architectures allow rapid development of high-end complex games and simulations

In the MBSE results from Round 1 the panel indicated that both ties to mission tasks and to measures of performance were critical The selection of metrics and the mechanisms to tie these factors into the process are very important Game-based metrics are appropriate but these should be tied to elemental capabilities Army researchers have explored an area called Degraded States for use in armor lethality (Comstock 1991) The early work in this area has not found wide application in the Army However the eleshymental capability methodology which is used for personnel analysis should be explored for this application Data can be presented to the warfighter that aid gameplay by using basic physics In later life-cycle stages by capturing and recording detailed data points engineering-level simulations can be run after the fact rather than in real time with more detailed high-fidelity simulations by the engineering staff This allows a detailed design based on feedback telemetered from the warfighter The combination of telemetry from the gameplay and follow-up ranking by warfighters and engineering staff can allow in-depth high-fidelity information flow into the emerging systems model Figure 6 shows the authorsrsquo views of the interactions and fidelity changes over the system life cycle

FIGURE 6 LIFE CYCLE

Open Innovation Collaboration Strategic Trade Study Analysis of Alternatives Low Fidelity

Competitive Medium Fidelity Evolving Representations

Br oad

Early Concept

Warfighters

EngSci

Warfighters

Prototype Evaluation

C ompar a tiv e

S ampT High Fidelity

Design Features EngSci

Warfighters

F ocused

Note EMD = Engineering and Manufacturing Development EngSci = Engineers Scientists SampT = Science and Technology

Collaboration and Filtering A discussion on collaboration and filtering arose during the interviews

The feedback process from a crowd using virtual environments needs voting and filtering The voting techniques used in social media or on Reddit are reasonable and well-studied Utilizing techniques familiar to the young warfighters will help simplify the overall process The ranking and filtering needs to be done by both engineers and warfighters so the decisions can take both viewpoints into consideration Table 2 shows the top 10 critical success factors from Round 2 for the overall process The Table includes the mean IQR and the percent agreement for each of the top 10 factors A collaboration area ranking and filtering by scientists and engineers and collaboration between the warfighters and the engineering staff are critical success factorsmdashwith a large amount of agreement from the expert panel

TABLE 2 TOP 10 CRITICAL SUCCESS FACTORS OVERALL PROCESS ROUND 2

CRITICAL SUCCESS FACTOR MEAN IQR AGREE

Filtering by ScientistsEngineers 556 1 81

PortalWebsiteCollaboration Area 556 1 81

Leadership Support 6 25 75

Feedback of Game Data into Process 556 275 75

Recognition 513 175 75

Data Security 55 275 75

Collaboration between EngScientist and Warfighters

606 25 75

Engagement (Warfighters) 594 3 69

Engagement (Scientists amp Engineers) 575 3 69

Fidelity Fidelity was ranked high in virtual environments MBSE and HSI

Fidelity and accuracy of the modeling and representations to the true system are critical success factors For the virtual environment early work would be done with low facet count models featuring texture maps for realism However as the system moves through the life cycle higher fidelity models and models that feed into detailed design simulations will be required There must also be verification validation and accreditation of these models as they enter the modeling repository or the DSM

April 2017

Leadership Commitment Leadership commitment was ranked near the top in the MBSE crowd-

sourcing and HSI areas Clearly in these emerging areas the enterprise needs strong leadership and training to enable MBSE and crowdsourcing initiatives The newness of MBSE and crowdsourcing may be related to the expertsrsquo high ranking of the need for leadership and senior engineer commitshyment Leadership support is also a critical success factor in Table 2mdashwith 75 percent agreement from the panel Leadership commitment and support although somewhat obvious as a success factor may have been lacking in previous initiatives Leadership commitment needs to be reflected in both policy and funding commitments from both DoD and Service leadership to encourage and spur these innovative approaches

Critical Success Factors Figure 7 details the critical success factors generated from the Delphi

study which visualizes the top 10 factors in each by using a mind-mapshyping diagram The main areas of study in this article are shown as major branches with the critical success factors generated appearing on the limbs of the diagram The previous sections have discussed some of the emerging themes and how some of the recurring critical success factors in each area can be utilized in the framework developed The Round 3 ranking of the critical success factors was analyzed by computing the Kendallrsquos W coefshyficient of concordance Kendallrsquos W is a nonparametric statistics tool that measures the agreement of a group of raters The expertsrsquo rankings of the success factors showed moderate but statistically significant agreement or consensus

FIGURE 7 CRITICAL SUCCESS FACTOR IN FIVE KEY AREAS

Fi l t e

r i n g b

y S c i e

n t i s t

s g i n

e e r s

Po r t a

l We b

s i t e C

o l l a b

o r a t

L e a d

e r s h

i p S u

p p o r

F e e d

b a c k

I n t o

P r o c

T i m e l y

F e e d

b a c k

R e c o

g n i t i

a t a S

e c u r

ion Be

t we e

n E n g

S c i e

n t i s t

a r fi g

h t e r

E n g a

e n t (

fi g h t

e r s )

S c i e n

t i s t s

amp E n

s s i b i l

v a i l a

b i l i t y

t C o m

m i t m

Abilit

e s i g n

I m p r

o v e m

i s by

C l a s

s o f S

t a k e

h o l d

D a t a

P i g r

C o n fi

o n t r o

Mi s s i

o n S p

a c e C

h c t e

r i z a t

l We b

s i t e

C o l l a

b t i o

n A r e

A b i l i t

y t o C

a p t u

f o r m

a n c e

B e h a

v i o r

A d e q

u a t e

A b i l i t

y t o A

n a l y z

e M e n

t a l T

a s k s

I n t e g

r a t i o

n w i t h

S y s t e

n g i n e

e r i n g

P r o c

L e a d

e r s h

i p S u

p p o r

t C o m

m i t m

I n t u i t

i v e I n

t e r f a

C o n s

i s t e n

i t h O

p e r a

t i o n a

D a t a

C a p t

u r e I

n t o M

e t r i c

F i d e l i

t u a l

M o d e

l o f t

h e S y

s t em

T a s k

L e a d

e r s h

i p C o

i t me n

R e l i a

b i l i t y

e a t a

b i l i t y

S e n i o

r E n g

T i e d t

o M e a

s u r e

o f P e

r f o r m

a n c e

F i d e l i

e p r e

s e n t

a t i o n

o f T r

u e S y

s t e m

We l l

D e fi

n e d M

e t r i c

A d e q

u a t e

F u n d

i n g o f

Tool s

U t i l i t

y t o S

t a k e

h o l d e

R e a l

T i m e O

p e r a

t i o n

F i d e l i

t y o f

e l i n g

A c c u

r a c y

o f Re

D a t a

R e c o

r d i n g

V e r i fi

c a t i o

l i d a t

i o n a n

d A c c r

e d i t a

t i o n

R V irt

F l e x i b

i l ity

u l a r i t

n I n v

e s t m

e n t C

o s t S

a v i n g

ll Proc

Crowdso

urcing

izatio

at iion

Dng di

i ytto

April 2017

LIMITATIONS TO THE RESEARCH The ideas presented here and the critical success factors have been

developed by a team of experts who have on average 20 to 30 years of expeshyrience in the primary area of inquiry and advanced degrees However the panel was more heavily weighted by Army experts than individuals from the rest of the DoD Neither time nor resources allowed for study of other important groups of experts including warfighters industry experts and program managers The Delphi method was selected for this study to genshyerate the critical success factors based on the perceived ease of use of the method and the controlled feedback gathered The critical success factors developed are ranked judgment but based on years of expertise This study considered five important areas and identified critical success factors in those areas This research study is based on the viewpoint of experts in MampS Nonetheless other types of expert viewpoints might possibly genshyerate additional factors Several factor areas could not be covered by MampS experts including security and information technology

The surveys were constructed with 5- and 7- element Likert scales that allowed the experts to choose ldquoNeutralrdquo or ldquoNeither Agree nor Disagreerdquo Not utilizing a forced-choice scale or a nonordinal data type in later Delphi rounds can limit data aggregation and statistical analysis approaches

RECOMMENDATIONS AND CONCLUSIONS

In conclusion innovation tied to virtual environments and linked to MBSE artifacts can help the DoD meet the significant challenges it faces in creating new complex interconnected designs much faster than in the past decade This study has explored key questions and has developed critical success factors in five areas A general framework has also been developed The DoD must look for equally innovative ways to meet numerous informashytion technology (IT) security and workforce challenges to enable the DoD to implement the process successfully in the acquisition enterprise The DoD should also explore interdisciplinary teams by hiring and funding teams of programmers and content creators to be co-located with systems engineers and subject matter experts Artfully crafted game-based scenarios that help explore design and usability issues can be crafted and provided to warfighters as a part of the process and help focus on needed system information The challenge remains for the methods to harvest filter and convert the information gathered to

MBSE artifacts that result from this process An overall process can be enacted that takes ideas design alternatives and data harvestedmdashand then provides a path to feed back this data at many stages in the acquisition cycle The extent to which MBSE tools such as SysML UML and emerging new standards are adopted or utilized in the process may depend upon the emerging training of acquisition professionals in MBSE and the leadership commitment to this approach

Artfully crafted game-based scenarios that help explore design and usability issues can be crafted and provided to warfighters as a part of the process and help focus on needed system information

This article has answered the three research questions posed in earlier discussion Utilizing the expert panel critical success factors have been developed using the Delphi method An emerging process model has been described Finally the experts in this Delphi study have affirmed an essenshytial role of MBSE in this process

FUTURE RESEARCH The DoD is actively conducting research into the remaining challenges

to bring many of the concepts discussed in this article into the acquisition process The critical success factors developed here can be utilized to focus some of the efforts

Key challenges in DoD remain as the current IT environment attempts to study larger virtual environments and prototypes The question of how to utilize the Secret Defense Engineering Research Network High Performance Supercomputing and Secret Internet Protocol Router Network while simultaneously making the process continually available to warfighters will need to be answered The ability of deployed warfighters to engage in future system design efforts is also a risk item that needs to be investigated Research is essential to identify the limitations and inertia associated with the DoD IT environment in relation to virtual environments and crowdsourcing An expanded future research study that uses additional

April 2017

inputs including a warfighter expert panel and an industry expert panel would provide useful data to compare and contrast with the results of this study

An exploration of how to combine the process described in this research with tradespace methodologies and ERS approaches could be explored MBSE methods to link and provide feedback should also be studied

The DoD should support studies that select systems in the early stages of development in each Service to apply the proposed framework and process The studies should use real gaps and requirements and real warfighters In support of ARCIC several studies are proposed at the ICT and the NPS that explore various aspects of the challenges involved in testing tools needed to advance key concepts discussed in this article The Navy Air Force and Army have active programs under various names to determine how MampS can support future systems development as systems and designs become more complex distributed and interconnected (Spicer et al 2015)

The extent to which MBSE tools such as SysML UML and emerging new standards are adopted or utilized in the process may depend upon the emerging training of acquisition professionals in MBSE and the leadership commitment to this approach

When fully developed MBSE and DSM methods can leverage the emerging connected DoD enterprise and bring about a continuous-feedback design environment Applying the concepts developed in this article to assessments conducted by developing concepts Analysis of Alternatives and trade studies conducted during early development through Milestone C can lead to more robust resilient systems continuously reviewed and evaluated by the stakeholders who truly matter the warfighters

References Bianca D P (2000) Simulation and modeling for acquisition requirements and

training (SMART) (Report No ADA376362) Retrieved from httpoaidticmil oaioaiverb=getRecordampmetadataPrefix=htmlampidentifier=ADA376362

Boudreau K J amp Lakhani K R (2013) Using the crowd as an innovation partner Harvard Business Review 91(4) 60ndash69

Bucher N amp Bouwens C (2013) Always onndashon demand Supporting the development test and training of operational networks amp net-centric systems Presentation to National Defense Industrial Association 16th Annual Systems Engineering Conference October 28-31 Crystal City VA Retrieved from http wwwdticmilndia2013systemW16126_Bucherpdf

Carlini J (2010) Rapid capability fielding toolbox study (Report No ADA528118) Retrieved from httpwwwdticmildtictrfulltextu2a528118pdf

Comstock G R (1991) The degraded states weapons research simulation An investigation of the degraded states vulnerability methodology in a combat simulation (Report No AMSAA-TR-495) Aberdeen Proving Ground MD US Army Materiel Systems Analysis Activity

Corns S amp Kande A (2011) Applying virtual engineering to model-based systems engineering Systems Research Forum 5(2) 163ndash180

Crowdsourcing (nd) In Merriam-Websterrsquos online dictionary Retrieved from http wwwmerriam-webstercomdictionarycrowdsourcing

Dalkey N C (1967) Delphi (Report No P-3704) Santa Monica CA The RAND Corporation

David J W (1995) A comparative analysis of the acquisition strategies of Army Tactical Missile System (ATACMS) and Javelin Medium Anti-armor Weapon System (Masterrsquos thesis) Naval Postgraduate School Monterey CA

Department of the Navy (2015 May 20) The Department of the Navy launches the ldquoHatchrdquo Navy News Service Retrieved from httpwwwnavymilsubmitdisplay aspstory_id=87209

Drucker C (2014) Why airport scanners catch the water bottle but miss the dynamite [Duke Research Blog] Retrieved from httpssitesdukeedu dukeresearch20141124why-airport-scanners-catch-the-water-bottle-butshymiss-the-dynamite

Ferrara J (1996) DoDs 5000 documents Evolution and change in defense acquisition policy (Report No ADA487769) Retrieved from httpoaidticmil oaioaiverb=getRecordampmetadataPrefix=htmlampidentifier=ADA487769

Forrester A (2015) Ray Mabus Navyrsquos lsquoHatchrsquo platform opens collaboration on innovation Retrieved from httpwwwexecutivegovcom201505ray-mabusshynavys-hatch-platform-opens-collaboration-on-innovation

Freeman G R (2011) Rapidexpedited systems engineering (Report No ADA589017) Wright-Patterson AFB OH Air Force Institute of Technology Center for Systems Engineering

Gallop D (2015) Delphi dice and dominos Defense ATampL 44(6) 32ndash35 Retrieved from httpdaudodlivemilfiles201510Galloppdf

GAO (2015) Defense acquisitions Joint action needed by DOD and Congress to improve outcomes (Report No GAO-16-187T) Retrieved from httpwwwgao govassets680673358pdf

April 2017

General Electric (2017) GE open innovation Retrieved from httpwwwgecom about-usopeninnovation

Gould J (2015 March 19) McHugh Army acquisitions tale of failure DefenseNews Retrieved from httpwwwdefensenewscomstorydefenseland army20150319mchugh-army-acquisitions-failure-underperformingshycanceled-25036605

Gourley S (2015) US Army looks to full spectrum shoulder-fired weapon Retrieved from httpswwwmilitary1comarmy-trainingarticle572557-us-army-looks-toshyfull-spectrum-shoulder-fired-weapon

Haduch T (2015) Model based systems engineering The use of modeling enhances our analytical capabilities Retrieved from httpwwwarmymile2c downloads401529pdf

Hagel C (2014) Defense innovation days Keynote presentation to Southeastern New England Defense Industry Alliance Retrieved from httpwwwdefensegov NewsSpeechesSpeech-ViewArticle605602

Heggen E S (2015) In the age of free AAA game engines are we still relevant Retrieved from httpjmonkeyengineorg301602in-the-age-of-free-aaa-gameshyengines-are-we-still-relevant

Howe J (2006) The rise of crowdsourcing Wired 14(6) 1ndash4 Retrieved from http wwwwiredcom200606crowds

shyINCOSE (2007) Systems engineering vision 2020 (Report No INCOSE TP-2004-004-02) Retrieved from httpwwwincoseorgProductsPubspdf SEVision2020_20071003_v2_03pdf

Janersquos International Defence Review (2015) Lighten up Shoulder-launched weapons come of age Retrieved from httpwwwjanes360comimagesassets 44249442 shoulder-launched weapon _systems_come_of_agepdf

Kendall F (2014) Better buying power 30 [White Paper] Retrieved from Office of the Under Secretary of Defense (Acquisition Technology amp Logistics) Website httpwwwdefenseinnovationmarketplacemilresources BetterBuyingPower3(19September2014)pdf

Korfiatis P Cloutier R amp Zigh T (2015) Model-based concept of operations development using gaming simulation Preliminary findings Simulation amp Gaming Thousand Oaks CA Sage Publications httpsdoiorg1046878115571290

London B (2012) A model-based systems engineering framework for concept development (Masterrsquos thesis) Massachusetts Institute of Technology Cambridge MA Retrieved from httphdlhandlenet1721170822

Lyons J W Long D amp Chait R (2006) Critical technology events in the development of the Stinger and Javelin Missile Systems Project hindsight revisited Washington DC Center for Technology and National Security Policy

Madni A M (2015) Expanding stakeholder participation in upfront system engineering through storytelling in virtual worlds Systems Engineering 18(1) 16ndash27 httpsdoiorg101002sys21284

Madni A M Nance M Richey M Hubbard W amp Hanneman L (2014) Toward an experiential design language Augmenting model-based systems engineering with technical storytelling in virtual worlds Procedia Computer Science 28(2014) 848ndash856

Miller D (2015) Update on OCT activities Presentation to NASA Advisory Council Technology Innovation and Engineering Committee Retrieved from https wwwnasagovsitesdefaultfilesatomsfilesdmiller_octpdf

Modigliani P (2013 NovemberndashDecember) Digital Pentagon Defense ATampL 42(6) 40ndash43 Retrieved from httpdaudodlivemilfiles201311Modiglianipdf

Moore D (2014) NAWCAD 2030 strategic MMOWGLI data summary Presentation to Naval Air Systems Command Retrieved from httpsportalmmowglinps edudocuments10156108601COMMS+1_nscMMOWGLIOverview_post pdf4a937c44-68b8-4581-afd2-8965c02705cc

Murray K L (2014) Early synthetic prototyping Exploring designs and concepts within games (Masterrsquos thesis) Naval Postgraduate School Monterey CA Retrieved from httpcalhounnpseduhandle1094544627

NRC (2010) The rise of games and high-performance computing for modeling and simulation Committee on Modeling Simulation and Games Washington DC National Academies Press httpsdoiorg101722612816

Roberts J (2015) Building the Naval Innovation Network Retrieved from httpwww secnavnavymilinnovationPages201508NINaspx

Rodriguez S (2014) Top 10 failed defense programs of the RMA era War on the Rocks Retrieved from httpwarontherockscom201412top-10-failed-defenseshyprograms-of-the-rma-era

Romanczuk G E (2012) Visualization and analysis of arena data wound ballistics data and vulnerabilitylethality (VL) data (Report No TR-RDMR-SS-11-35) Redstone Arsenal AL US Army Armament Research Development and Engineering Center

Sanders P (1997) Simulation-based acquisition Program Manager 26(140) 72ndash76 Secretary of the Navy (2015) Characteristics of an innovative Department of the Navy

Retrieved from httpwwwsecnavnavymilinnovationDocuments201507 Module_4pdf

Sheridan V (2015) From former NASA researchers to LGBT activists ndash meet some faces new to GW The GW Hatchet Retrieved from httpwwwgwhatchet com20150831from-former-nasa-researchers-to-lgbt-activists-meet-someshyfaces-new-to-gw

Skutsch M amp Hall D (1973) Delphi Potential uses in educational panning Project Simu-School Chicago Component Retrieved from httpseric edgovid=ED084659

Smith R E amp Vogt B D (2014 July) A proposed 2025 ground systems ldquoSystems Engineeringrdquo process Defense Acquisition Research Journal 21(3) 752ndash774 Retrieved from httpwwwdaumilpublicationsDefenseARJARJARJ70ARJshy70_Smithpdf

Spicer R Evangelista E Yahata R New R Campbell J Richmond T Vogt B amp McGroarty C (2015) Innovation and rapid evolutionary design by virtual doing Understanding early synthetic prototyping (ESP) Retrieved from httpictusc edupubsInnovation20and20Rapid20Evolutionary20Design20by20 Virtual20Doing-Understanding20Early20Syntheticpdf

Tadjdeh Y (2014) New video game could speed up acquisition timelines National Defense Retrieved from httpwwwnationaldefensemagazineorgbloglists postspostaspxID=1687

US Air Force (nd) The Air Force collaboratory Retrieved from https collaboratoryairforcecom

US Air Force (2015) Air Force prize Retrieved from httpsairforceprizecomabout US Naval Research Laboratory (nd) SIMDIStrade presentation Retrieved from https

simdisnrlnavymilSimdisPresentationaspx

April 2017

Versprille A (2015) Crowdsourcing to solve tough Navy problems National Defense Retrieved from httpwwwnationaldefensemagazineorgarchive2015June PagesCrowdsourcingtoSolveToughNavyProblemsaspx

Zimmerman P (2015) MBSE in the Department of Defense Seminar presentation to Goddard Space Flight Center Retrieved from httpssesgsfcnasagovses_ data_2015150512_Zimmermanpdf

Author Biographies

Mr Glenn E Romanczuk is a PhD candishydate at The George Washington University He is a member of the Defense Acquisition Corps matrixed to the Operational Test Agency (OTA) evaluating the Ballistic Missile Defense System He holds a BA in Political Science from DePauw University a BSE from the University of Alabama in Huntsville (UAH) and an MSE from UAH in Engineering Management His research includes systems engineering lethality visualization and virtual environments

(E-mail address gromanczukgwmailgwuedu)

Dr Christopher Willy is currently a senior systems engineer and program manager with J F Taylor Inc Prior to joining J F Taylor in 1999 he completed a career in the US Navy Since 2009 he has taught courses as a professoshyrial lecturer for the Engineering Management and Systems Engineering Department at The George Washington University (GWU) Dr Willy holds a DSc degree in Systems Engineering from GWU His research interests are in stochastic processes and systems engineering

(E-mail address cwillygwmailgwuedu)

April 2017

Dr John E Bischoff is a professorial lecturer of Engineering Management at The George Washington University (GWU) He has held execshyutive positions in several firms including AOL Time Warner and IBM Watson Research Labs Dr Bischoff holds a BBA from Pace University an MBA in Finance from Long Island University an MS in Telecommunications Management from the Polytechnic University and a Doctor of Science in Engineering Management from GWU

(E-mail address jebemailgwuedu)

T h e D e f e n s e A c q u i s i t i o n Professional Reading List is intended to enrich the knowledge and under-standing of the civilian military contractor and industrial workforce who participate in the entire defense acquisition enterprise These book recommendations a re desig ned to complement the education and training vital to developing essential competencies and skills of the acqui-sition workforce Each issue of the Defense Acquisition Research Journal will include one or more reviews of suggested books with more available on our Website httpwwwdaumillibrary

We encourage our readers to submit book reviews they believe should be required reading for the defense acquisition professional The books themselves should be in print or generally available to a wide audi-ence address subjects and themes that have broad applicability to defense acquisition profession-a ls and provide context for the reader not prescriptive practices Book reviews should be 450 words or fewer describe the book and its major ideas and explain its rele-vancy to defense acquisition Please send your reviews to the managing editor Defense Acquisition Research Journal at DefenseARJdaumil

Featured Book Getting Defense Acquisition Right

Author The Honorable Frank Kendall Former Under Secretary of Defense for Acquisition Technology and Logistics Publisher Defense Acquisition University Press Fort Belvoir VA Copyright Date 2017 Hardcover 216 pages ISBN TBD Introduction by The Honorable Frank Kendall

April 2017

Review For the last several years it has been my great honor and privilege to

work with an exceptional group of public servants civilian and military who give all that they have every day to equip and support the brave men and women who put themselves in harms way to protect our country and to stand up for our values Many of these same public servants again civilian and military have put themselves in harms way also

During this period I wrote an article for each edition of the Defense ATampL Magazine on some aspect of the work we do My goal was to communicate to the total defense acquisition workforce in a manner more clearly directly and personally than official documents my intentions on acquisition policy or my thoughts and guidance on the events we were experiencing About 6 months ago it occurred to me that there might be some utility in organizing this body of work into a single product As this idea took shape I developed what I hoped would be a logical organization for the articles and started to write some of the connecting prose that would tie them together and offer some context In doing this I realized that there were some other written communications I had used that would add to the completeness of the picshyture I was trying to paint so these items were added as well I am sending that product to you today It will continue to be available through DAU in digital or paper copies

Frankly Im too close to this body of work to be able to assess its merit but I hope it will provide both the acquisition workforce and outside stakeholdshyers in and external to the Department with a good compendium of one acquisition professionals views on the right way to proceed on the endless journey to improve the efficiency and the effectiveness of the vast defense acquisition enterprise We have come a long way on that journey together but there is always room for additional improvement

I have dedicated this book to you the people who work tirelessly and proshyfessionally to make our military the most capable in the world every single day You do a great job and it has been a true honor to be a member of this team again for the past 7 years

Getting Defense Acquisition Right is hosted on the Defense Acquisition Portal and the Acquisition Professional Reading Program websites at

httpsshortcutdaumilcopgettingacquisitionright

httpdaudodlivemildefense-acquisition-professional-reading-program

New Research in DEFENSE ACQUISITION

Academics and practitioners from around the globe have long con-sidered defense acquisition as a subject for serious scholarly research and have published their findings not only in books but also as Doctoral dissertations Masterrsquos theses and in peer-reviewed journals Each issue of the Defense Acquisition Research Journal brings to the attention of the defense acquisition community a selection of current research that may prove of further interest

These selections are curated by the Defense Acquisition University (DAU) Research Center and the Knowledge Repository We present here only the authortitle abstract (where available) and a link to the resource Both civil-ian government and military Defense Acquisition Workforce (DAW) readers will be able to access these resources on the DAU DAW Website httpsidentitydaumilEmpowerIDWebIdPFormsLoginKRsite Nongovernment DAW readers should be able to use their local knowledge management cen-ters and libraries to download borrow or obtain copies We regret that DAU cannot furnish downloads or copies

We encourage our readers to submit suggestions for current research to be included in these notices Please send the authortitle abstract (where avail-able) a link to the resource and a short write-up explaining its relevance to defense acquisition to Managing Editor Defense Acquisition Research Journal DefenseARJdaumil

Developing Competencies Required for Directing Major Defense Acquisition

Programs Implications for Leadership Mary C Redshaw

Abstract The purpose of this qualitative multiple-case research

study was to explore the perceptions of government proshygram managers regarding (a) the competencies program

managers must develop to direct major defense acquisition proshygrams (b) professional opportunities supporting development of

those competencies (c) obstacles to developing the required competencies and (d) factors other than the program managers competencies that may influence acquisition program outcomes The general problem this study addressed was perceived gaps in program management competencies in the defense acquisition workforce the specific problem was lack of information regarding required competencies and skills gaps in the Defense Acquisition Workforce that would allow DoD leaders to allocate resources for training and development in an informed manner The primary sources of data were semistructured in-depth interviews with 12 major defense acquisition program managers attending the Executive Program Managers Course (PMT-402) at the Defense Systems Management College School of Program Managers at Fort Belvoir Virginia either during or immediately prior to assignments to lead major defense acquisition programs The framework for conducting the study and organizing the results evolved from a primary

research question and four supporting subquestions Analysis of the qual-itative interview data and supporting information led to five findings and associated analytical categories for further analysis and interpretation Resulting conclusions regarding the competencies required to lead program teams and the effective integration of professional development opportu-nities supported recommendations for improving career management and professional development programs for members of the Defense Acquisition Workforce

APA Citation Redshaw M C (2011) Developing competencies required for directing major defense

acquisition programs Implications for leadership (Order No 1015350964) Available from ProQuest Dissertations amp Theses Global Retrieved from https searchproquestcomdocview1015350964accountid=40390

Exploring Cybersecurity Requirements in the Defense Acquisition Process

Kui Zeng

Abstract The federal government is devoted to an open safe free and

dependable cyberspace that empowers innovation enriches business develops the economy enhances security fosters education upholds

democracy and defends freedom Despite many advantagesmdashfederal and Department of Defense cybersecurity policies and standards the best military power equipped with the most innovative technologies in the world and the best military and civilian workforces ready to perform any missionmdashdefense cyberspace is vulnerable to a variety of threats This study explores cybersecurity requirements in the defense acquisition process The literature review exposes cybersecurity challenges that the govern-ment faces in the federal acquisition process and the researcher examines cybersecurity requirements in defense acquisition documents Within the current defense acquisition process the study revealed that cybersecurity is not at a level of importance equal to that of cost technical and perfor-mance Further the study discloses the defense acquisition guidance does not reflect the change in cybersecurity requirements and the defense acqui-sition processes are deficient ineffective and inadequate to describe and consider cybersecurity requirements thereby weakening the governmentrsquos overall efforts to implement a cybersecurity framework into the defense acquisition process Finally the study recommends defense organizations

elevate the importance of cybersecurity during the acquisition process to help the governmentrsquos overall efforts to develop build and operate in an open secure interoperable and reliable cyberspace

APA Citation Zeng K (2016) Exploring cybersecurity requirements in the defense

acquisition process (Order No 1822511621) Available from ProQuest Dissertations amp Theses Global Retrieved from httpsearchproquestcom docview1822511621accountid=40390

Improving Defense Acquisition Outcomes Using an Integrated Systems Engineering Decision Management (ISEDM) Approach

Matthew V Cilli

Abstract The US Department of Defense (DoD) has recently revised

the defense acquisition system to address suspected root causes of unwanted acquisition outcomes This dissertation

applied two systems thinking methodologies in a uniquely inte-grated fashion to provide an in-depth review and interpretation of the

revised defense acquisition system as set forth in Department of Defense Instruction 500002 dated January 7 2015 One of the major changes in the revised acquisition system is an increased emphasis on systems engineer-ing trade-offs made between capability requirements and life-cycle costs early in the acquisition process to ensure realistic program baselines are established such that associated life-cycle costs of a contemplated system are affordable within future budgets Given that systems engineering trade-off analyses will play a pivotal role in future defense acquisition efforts this research employed a two-phased exploratory sequential and embedded mixed-methods approach to take an in-depth look at the state of literature surrounding systems engineering trade-off analyses The research also aimed to identify potential pitfalls associated with the typical execution of a systems engineering trade-off analysis quantify the risk that potential pitfalls pose to acquisition decision quality suggest remedies to mitigate the risk of each pitfall and measure the potential usefulness of contemplated innovations that may help improve the quality of future systems engineering trade-off analyses In the first phase of this mixed-methods study qualita-tive data were captured through field observations and direct interviews with US defense acquisition professionals executing systems engineering

April 2017

trade analyses In the second phase a larger sample of systems engineering professionals and military operations research professionals involved in defense acquisition were surveyed to help interpret qualitative findings of the first phase The survey instrument was designed using Survey Monkey was deployed through a link posted on several groups within LinkedIn and was sent directly via e-mail to those with known experience in this research area The survey was open for a 2-month period and collected responses from 181 participants The findings and recommendations of this research were communicated in a thorough description of the Integrated Systems Engineering Decision Management (ISEDM) process developed as part of this dissertation

APA Citation Cilli M V (2015) Improving defense acquisition outcomes using an Integrated

Systems Engineering Decision Management (ISEDM) approach (Order No 1776469856) Available from ProQuest Dissertations amp Theses Global Retrieved from httpsearchproquestcomdocview1776469856accountid=40390

Arming Canada Defence Procurementfor the 21st Century

Elgin Ross Fetterly

Abstract The central objective of this thesis is to examine how the Canadian

government can make decisions that will provide the government with a defence procurement process better suited to the current

defence environmentmdashwhich places timeliness of response to changing operational requirements at a premium Although extensive research has described the scope and depth of shortcomings in the defence procurement process recommendations for change have not been translated into effective and comprehensive solutions Unproductive attempts in recent decades to reform the defence procurement process have resulted from an overwhelm-ing institutional focus on an outdated Cold War procurement paradigm and continuing institutional limitations in procurement flexibility adapt-ability and responsiveness This thesis argues that reform of the defence procurement process in Canada needs to be policy-driven The failure of the government to adequately reform defence procurement ref lects the inability to obtain congruence of goals and objectives among participants in that process The previous strategy of Western threat containment has

changed to direct engagement of military forces in a range of expedition-ary operations The nature of overseas operations in which the Canadian Forces are now participating necessitates the commitment of significant resources to long-term overseas deployments with a considerable portion of those resources being damaged or destroyed in these operations at a rate greater than their planned replacement This thesis is about how the Canadian government can change the defence procurement process in order to provide the Canadian Forces with the equipment they need in a timely and sustained basis that will meet the objectives of government policy Defence departments have attempted to adopt procurement practices that have proven successful in the private sector without sufficient recognition that the structure of the procurement organisation in defence also needed to change significantly in order to optimize the impact of industry best practices This thesis argues that a Crown Corporation is best suited to supporting timely and effective procurement of capital equipment Adoption of this private sector-oriented organisational structure together with adoption of industry best practices is viewed as both the foundation and catalyst for transformational reform of the defence procurement process

APA Citation Fetterly E R (2011) Arming Canada Defence procurement for the 21st

century (Order No 1449686979) Available from ProQuest Dissertations amp Theses Global Retrieved from httpsearchproquestcom docview1449686979accountid=40390

April 2017

Defense ARJ Guidelines FOR CONTRIBUTORSThe Defense Acquisition Research Journal (ARJ) is a scholarly peer-reviewed journal published by the Defense Acquisition University (DAU) All submissions receive a blind review to ensure impartial evaluation

Defense ARJ April 2017 Vol 24 No 2 376-380

IN GENERAL We welcome submissions from anyone involved in the defense acquishy

sition process Defense acquisition is defined as the conceptualization initiation design development testing contracting production deployshyment logistics support modification and disposal of weapons and other systems supplies or services needed for a nationrsquos defense and security or intended for use to support military missions

Research involves the creation of new knowledge This generally requires using material from primary sources including program documents policy papers memoranda surveys interviews etc Articles are characterized by a systematic inquiry into a subject to discoverrevise facts or theories with the possibility of influencing the development of acquisition policy andor process

We encourage prospective writers to coauthor adding depth to manuscripts It is recommended that a mentor be selected who has been previously pubshylished or has expertise in the manuscriptrsquos subject Authors should be familiar with the style and format of previous Defense ARJs and adhere to the use of endnotes versus footnotes (refrain from using the electronic embedshyding of footnotes) formatting of reference lists and the use of designated style guides It is also the responsibility of the corresponding author to furnish any required government agencyemployer clearances with each submission

MANUSCRIPTS Manuscripts should reflect research of empirically supported experishy

ence in one or more of the areas of acquisition discussed above Empirical research findings are based on acquired knowledge and experience versus results founded on theory and belief Critical characteristics of empirical research articles

bull clearly state the question

bull define the methodology

bull describe the research instrument

bull describe the limitations of the research

bull ensure results are quantitative and qualitative

bull determine if the study can be replicated and

bull discuss suggestions for future research (if applicable)

Research articles may be published either in print and online or as a Web-only version Articles that are 4500 words or less (excluding abstracts references and endnotes) will be considered for print as well as Web pubshylication Articles between 4500 and 10000 words will be considered for Web-only publication with an abstract (150 words or less) included in the print version of the Defense ARJ In no case should article submissions exceed 10000 words

Book Reviews Defense ARJ readers are encouraged to submit reviews of books they

believe should be required reading for the defense acquisition professional The reviews should be 450 words or fewer describing the book and its major ideas and explaining why it is relevant to defense acquisition In general book reviews should reflect specific in-depth knowledge and understanding that is uniquely applicable to the acquisition and life cycle of large complex defense systems and services

Audience and Writing Style The readers of the Defense ARJ are primarily practitioners within the

defense acquisition community Authors should therefore strive to demonstrate clearly and concisely how their work affects this community At the same time do not take an overly scholarly approach in either content or language

Format Please submit your manuscript with references in APA format (authorshy

date-page number form of citation) as outlined in the Publication Manual of the American Psychological Association (6th Edition) For all other style questions please refer to the Chicago Manual of Style (16th Edition) Also include Digital Object Identifier (DOI) numbers to references if applicable

Contributors are encouraged to seek the advice of a reference librarian in completing citation of government documents because standard formulas of citations may provide incomplete information in reference to governshyment works Helpful guidance is also available in The Complete Guide to Citing Government Documents (Revised Edition) A Manual for Writers and Librarians (Garner amp Smith 1993) Bethesda MD Congressional Information Service

Pages should be double-spaced in Microsoft Word format Times New Roman 12-point font size and organized in the following order title page (titles 12 words or less) abstract (150 words or less to conform with forshymatting and layout requirements of the publication) two-line summary list of keywords (five words or less) reference list (only include works cited in the paper) authorrsquos note or acknowledgments (if applicable) and figures or tables (if any) Manuscripts submitted as PDFs will not be accepted

Figures or tables should not be inserted or embedded into the text but segregated (one to a page) at the end of the document It is also importshyant to annotate where figures and tables should appear in the paper In addition each figure or table must be submitted as a separate file in the original software format in which it was created For additional information

April 2017

on the preparation of figures or tables refer to the Scientific Illustration Committee 1988 Illustrating Science Standards for Publication Bethesda MD Council of Biology Editors Inc

The author (or corresponding author in cases of multiple authors) should attach a signed cover letter to the manuscript that provides all of the authorsrsquo names mailing and e-mail addresses as well as telephone and fax numbers The letter should verify that the submission is an original product of the author(s) that all the named authors materially contributed to the research and writing of the paper that the submission has not been previously pubshylished in another journal (monographs and conference proceedings serve as exceptions to this policy and are eligible for consideration for publication in the Defense ARJ ) and that it is not under consideration by another journal for publication Details about the manuscript should also be included in the cover letter for example title word length a description of the computer application programs and file names used on enclosed DVDCDs e-mail attachments or other electronic media

COPYRIGHT The Defense ARJ is a publication of the United States Government and

as such is not copyrighted Because the Defense ARJ is posted as a complete document on the DAU homepage we will not accept copyrighted manushyscripts that require special posting requirements or restrictions If we do publish your copyrighted article we will print only the usual caveats The work of federal employees undertaken as part of their official duties is not subject to copyright except in rare cases

Web-only publications will be held to the same high standards and scrushytiny as articles that appear in the printed version of the journal and will be posted to the DAU Website at wwwdaumil

In citing the work of others please be precise when following the author-date-page number format It is the contributorrsquos responsibility to obtain permission from a copyright holder if the proposed use exceeds the fair use provisions of the law (see US Government Printing Office 1994 Circular 92 Copyright Law of the United States of America p 15 Washington DC) Contributors will be required to submit a copy of the writerrsquos permission to the managing editor before publication

We reserve the right to decline any article that fails to meet the following copyright requirements

bull The author cannot obtain permission to use previously copyshyrighted material (eg graphs or illustrations) in the article

bull The author will not allow DAU to post the article in our Defense ARJ issue on our Internet homepage

bull The author requires that usual copyright notices be posted with the article

bull To publish the article requires copyright payment by the DAU Press

SUBMISSION All manuscript submissions should include the following

bull Cover letter

bull Author checklist

bull Biographical sketch for each author (70 words or less)

bull Headshot for each author should be saved to a CD-R disk or e-mailed at 300 dpi (dots per inch) or as a high-print quality JPEG or Tiff file saved at no less than 5x7 with a plain backshyground in business dress for men (shirt tie and jacket) and business appropriate attire for women All active duty military should submit headshots in Class A uniforms Please note low-resolution images from Web Microsoft PowerPoint or Word will not be accepted due to low image quality

bull One copy of the typed manuscript including

deg Title (12 words or less)

deg Abstract of article (150 words or less)

deg Two-line summary

deg Keywords (5 words or less)

deg Document double-spaced in Microsoft Word format Times New Roman 12-point font size (4500 words or less for the printed edition and 10000 words or less for the online-only content excluding abstract figures tables and references)

These items should be sent electronically as appropriately labeled files to the Defense ARJ Managing Editor at DefenseARJdaumil

CALL FOR AUTHORS We are currently soliciting articles and subject matter experts for the 2017 Defense Acquisition Research Jourshynal (ARJ) print year Please see our guidelines for conshytributors for submission deadlines

Even if your agency does not require you to publish consider these career-enhancing possibilities

bull Share your acquisition research results with the Acquisition Technology and Logistics (ATampL) community

bull Change the way Department of Defense (DoD) does business bull Help others avoid pitfalls with lessons learned or best practices from your project or

program bull Teach others with a step-by-step tutorial on a process or approach bull Share new information that your program has uncovered or discovered through the

implementation of new initiatives bull Condense your graduate project into something beneficial to acquisition professionals

ENJOY THESE BENEFITS bull Earn 25 continuous learning points for We welcome submissions from anyone inshy

publishing in a refereed journal volved with or interested in the defense acshybull Earn a promotion or an award quisition processmdashthe conceptualization bull Become part of a focus group sharing initiation design testing contracting proshy

similar interests duction deployment logistics support modshybull Become a nationally recognized expert ification and disposal of weapons and other

in your field or specialty systems supplies or services (including conshybull Be asked to speak at a conference struction) needed by the DoD or intended for

or symposium use to support military missions

If you are interested contact the Defense ARJ managing editor (DefenseARJdaumil) and provide contact information and a brief description of your article Please visit the Defense ARJ Guidelines for Contributors at httpwwwdaumillibraryarj

The Defense ARJ is published in quarterly theme editions All submis-sions are due by the first day of the month See print schedule below

Author Deadline Issue

July January

November April

January July

April October

In most cases the author will be notified that the submission has been received within 48 hours of its arrival Following an initial review submis-sions will be referred to peer reviewers and for subsequent consideration by the Executive Editor Defense ARJ

Defense ARJ PRINT SCHEDULE

Contributors may direct their questions to the Managing Editor Defense ARJ at the address shown below or by calling 703-805-3801 (fax 703-805-2917) or via the Internet at norenetaylordaumil

The DAU Homepage can be accessed at httpwwwdaumil

DEPARTMENT OF DEFENSE

DEFENSE ACQUISITION UNIVERSITY

ATTN DAU PRESS (Defense ARJ)

9820 BELVOIR RD STE 3

FORT BELVOIR VA 22060-5565

January

Defense Acquisition University

WEBSITEhttpwwwdaumil

Your Online Access to Acquisition Research Consulting Information and Course Offerings

Now you can search the New DAU Website and our online publications

Defense ARJ

New Online Subscription

Defense ATampL

Cancellation

Change E-mail Address

Last Name

First Name

DayWork Phone

E-mail Address

Signature (Required)

ver 01032017

PLEASE FAX TO 703-805-2917

The Privacy Act and Freedom of Information Act In accordance with the Privacy Act and Freedom of Information Act we will only contact you regarding your Defense ARJ and Defense ATampL subscriptions If you provide us with your business e-mail address you may become part of a mailing list we are required to provide to other agencies who request the lists as public information If you prefer not to be part of these lists please use your personal e-mail address

FREE ONLINES U B S C R I P T I O N

S U B S C R I P T I O N

Thank you for your interest in Defense Acquisition Research Journal and Defense ATampL magazine To receive your complimentary online subscription please write legibly if hand written and answer all questions belowmdashincomplete forms cannot be processed

When registering please do not include your rank grade service or other personal identifiers

S U R V E Y

Please rate this publication based on the following scores

5 mdashExceptional 4 mdash Great 3 mdash Good 2 mdash Fair 1 mdash Poor

Please circle the appropriate response

1 How would you rate the overall publication 5 4 3 2 1

2 How would you rate the design of the publication 5 4 3 2 1

True Falsea) This publication is easy to readb) This publication is useful to my careerc) This publication contributes to my job effectivenessd) I read most of this publicatione) I recommend this publication to others in the acquisition field

If hand written please write legibly

3 What topics would you like to see get more coverage in future Defense ARJs

4 What topics would you like to see get less coverage in future Defense ARJs

5 Provide any constructive criticism to help us to improve this publication

6 Please provide e-mail address for follow up (optional)

Wersquore on the Web at httpwwwdaumillibraryarj

Current Connected Innovative

Contents

From the Chairman and Executive Editor

DAU Center for Defense Acquisition | Research Agenda 2017-2018

DAU Alumni Association

Article 1 Using Analytical Hierarchy and Analytical Network Processes to Create CYBER SECURITY METRICS

Article 2 The Threat Detection System13THAT CRIED WOLF13Reconciling Developers with Operators

Article 3 ARMY AVIATION13Quantifying the Peacetime and Wartime13MAINTENANCE MAN-HOUR GAPS

Article 4 COMPLEX ACQUISITION13REQUIREMENTS ANALYSIS13Using a Systems Engineering Approach

Article 5 An Investigation of Nonparametric13DATA MINING TECHNIQUES13for Acquisition Cost Estimating

Article 6 CRITICAL SUCCESS FACTORS13for Crowdsourcing13with Virtual Environments13TO UNLOCK INNOVATION

Professional Reading List

New Research in13DEFENSE ACQUISITION

Defense ARJ Guidelines13FOR CONTRIBUTORS

CALL FOR AUTHORS

Defense ARJ13PRINT SCHEDULE

Using Analytical Hierarchy and Analytical Network Processes to Create Cyber Security Metrics George C Wilamowski Jason R Dever and Steven M F Stuban

The Threat Detection System That Cried Wolf Reconciling Developers with Operators Shelley M Cazares

Army Aviation Quantifying the Peacetime and Wartime Maintenance Man-Hour Gaps LTC William Bland USA (Ret) CW5 Donald L Washabaugh Jr USA (Ret) and Mel Adams

Complex Acquisition Requirements Analysis Using a Systems Engineering Approach Col Richard M Stuckey USAF (Ret) Shahram Sarkani and Thomas A Mazzuchi

An Investigation of Nonparametric Data Mining Techniques for Acquisition Cost Estimating Capt Gregory E Brown USAF and Edward D White

Online-only Article Critical Success Factors for Crowdsourcing with Virtual Environments to Unlock Innovation Glenn E Romanczuk Christopher Willy and John E Bischoff

The Defense Acquisition Professional Reading List Getting Defense Acquisition Right Written and Introduced by the Honorable Frank Kendall

Article List ARJ Extra

N2016 ACS1st

N UNIVERSITY ALUM

Featured Research

EXECUTIVE EDITOR

April 2017

Biography

April 2017

PAPER COMPETITION

2016 ACS 1st

NUNIVERSITY ALU

I ASSO

CIATIO

April 2017

Function 1

Function 2

Function 3

Function 4

FitshyForshyPurpose

Function

Measures

Analysis

Criteria

Subcriteria

Alternatives

Authentication (A1)

Authorization (A2)

LAN WAN

Accounting (A3)

RemoteshyUser

April 2017

Identify (1)

Password Aging

Alternatives (4)

Access (2)

Elements

Activity (3)

Outer Dependencies

April 2017

metropolitan area

n 1 2 3 4 5 6 7 8 RI 0 0 058 09 112 124 132 141

April 2017

Remote User

Equal Importance

Moderate Importance

Strong Importance

Extreme Importance

April 2017

18-24 1 1 4 5 11

25-34 7 2 27 6 28 70

35-44 22 1 63 21 32 139

45-54 19 4 70 41 42 176

55-64 11 1 29 15 26 82

65 gt 1 2 3 6

Grand 60 9 194 85 136 484 Total

SKIPPED 18

Legend 1 2 3 4 5

18-24 2 7 2 11

25-34 14 7 35 10 4 70

35-44 13 26 69 19 11 138

45-54 13 42 73 35 13

55-64 7 12 37 22 4

65 gt 5 1 6

Grand 47 87 216 98 35 Total

SKIPPED 19

Legend 1 2 3 4 5

18-24 2 1 1 7 11

25-34 8 19 7 36 70

35-44 16 33 17 72 138

45-54 19 37 21 99 176

55-64 11 14 10 46 81

65 gt 2 4 6

Grand 58 104 56 264 482 Total

SKIPPED 20

Legend 1 2 3 4

April 2017

Enforcement

1 22 1 16

2 2 2 17 2 8 2 7

3 9 3 21 3 19 3 13

4 7 4 24 4 10 4 24

5 22 5 66 5 41 5 53

6 15 6 34 6 17 6 25

7 14 7 40 7 25 7 36

8 3 8 12 8 4 8 9

9 6 9 19 9 7 9 12

Standard Deviation

Skewedshyness

-0278 Skewedshyness

-0176 Skewedshyness

-0161 Skewedshyness

n 89000 n 255000 n 147000 n 201000

3rd Quarshytile

70 60 50 40 30 20

0 3 6 9 12

4839 2138

373 6619

lt0005

April 2017

Total 369 169022

1 -1294 0614 -211 0036 107 2 -0115 0366 -031 0754 132 3 -0341 0313 -109 0277 176 4 -0060 0297 -020 0839 182 5 0147 0343 043 0669 138

SIZE 1 0022 0272 008 0935 302 2 0027 0228 012 0906 267 3 0117 0275 043 0670 289

Type 1 -0261 0332 -079 0433 149 2 0385 0246 156 0119 128 3 -0237 0191 -124 0216 118 4 -0293 0265 -111 0269 140

Position 1 0085 0316 027 0787 303 2 0730 0716 102 0309 897 3 -0378 0243 -155 0121 306 4 0038 0288 013 0896 303

Parameters

Residual

70 60 50 40 30 20

01 shy75 shy50 shy25 00 25 50

Y = Xβ + ε (1)

April 2017

N c (2)

April 2017

1_LAN 028083

2_WAN 013501

3_Remote 058416

LAN 012950

WAN 033985

Remote User 053065

020000

080000

LAN 012807

WAN 022686

Remote User 064507

020000

LAN 032109

WAN 013722

Remote User 054169

LAN 002650

WAN 024555 WAN 005710

April 2017

Author Biographies

April 2017

Shelley M Cazares

Operator

April 2017

System

April 2017

Ground Truth

Items (10100)

No Threat (10000)

Threat (100)

NOTIONAL System

Alarm (590)

No Alarm (9510)

TP (90) FN (10)

FP (500) TN (9500)

d = 90 (90 + 10) = 090

(near 1 is better)

fa = 500 (500 + 9500) = 005

(near 0 is better)

April 2017

Items (590)

= 90 (90 + 10) = 090

= 500 (500 + 9500) = 005

PPV1 = 90 (90 + 500) = 015 NPV

1 = 9500 (9500 + 10) asymp 1

= 88 (88 + 2) = 098

= 20 (20 + 480) = 004

PPV2 = 88 (88 + 20) = 081 NPV

2 = 480 (480 + 2) asymp 1

PPVoverall = 88 (88 + 20) = 081

Pd overall = 88 (88 + (10 + 2)) = 088

Items (10100)

(10000)

Threat (100)

NOTIONAL System 1

Alarm (590)

No Alarm (9510)

TP1 (90) FN1 (10)

FP1 (500) TN1 (9500)

Threat (90)

NOTIONAL System 2

Alarm (108)

No Alarm (482)

TP2 (88) FN2 (2)

FP2 (20) TN2 (480)

April 2017

system)

April 2017

Direction of Travel

Convoy

April 2017

FAR = FP total time

FAR = FP total area

April 2017

3 Ibid

4 Ibid

5 Ibid

April 2017

Biography

April 2017

MMH Gap Causes

Required MMHs

Available MMHs

Duty Absences

April 2017

100 fill rate

April 2017

90 80 95 50

Monthly 0 6896 23077 61327 MMH Gap

800000

700000

600000

500000

400000

300000

200000

100000

100 90 80 70

e of 1

Wartime 362330

489565

616800

113215

744034

151323

Peacetime 36999

April 2017

500000

450000

400000

350000

300000

250000

200000

150000

100000

100 90 80 70

e of 1

Wartime 124520

232550

340570

448600

Peacetime 0

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Complex System

April 2017

Requirement

Systems

April 2017

incipa

ined S

amp Co

te amp

l amp Tr

ibal F

pitals

uts amp

April 2017

Boundary Liquidity

April 2017

Complex System

Adaptive

Technical

Interdependence

Political

Holistic

Dependent System

System

(internal)

(external)

April 2017

patents

April 2017

( )( )2 2

April 2017

n minus 2

April 2017

Operations 77 2 26

Logistics 61 1 15

Academic 67 1 27

Policy 40 4 4

April 2017

0 4 500

0 4000

0 3 500

0 3000

0 2500

0 2000

0 1500

0 1000

stomer

Design Independen

vable t

ConciseHolist

icSocial

Political

Adaptable

Technical

Interdependent

Environmental

Consistent

Verifiable

Complete

Unambiguous

Correct

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Anomaly Detection

Data Mining

April 2017

F-4 1722 -087 1222 110 1399

F-105 1930 -022 1112 -086 1221

A-5 2350 109 1147 -024 1414

April 2017

Fshy4 ($1399)

Ashy5 ($1414)

Fshy105 ($1221)

Fshynotional Spee

Weight

shy2 0 2

April 2017

Aircraft Cost

Weight lt 3159

Weight lt 1221

Speed lt 992

Weight ge 3159

Weight ge 1221

Speed ge 992

$398 $928 $1305 $2228

0 0 20 40 60 80 100 120

R4 = $2228

April 2017

Weight

April 2017

Nearest N

Wins-Losses

April 2017

A-4 373 507 565

A-5 1414 2350 1147

A-6 888 1715 562

A-7 33 1162 595

A-10 629 1484 389

B-52 3203 11267 551

B-58 3243 3269 1147

BRB-66 1293 3050 548

C-130 1175 4345 326

C-133 1835 9631 304

KC-135 1555 7025 527

C-141 1891 10432 491

F3D 303 1014 470

F3H 757 1390 622

F4D 71 874 628

F-4 1399 1722 1222

F-86 248 679 590

F-89 542 1812 546

F-100 421 1212 752

F-101 893 1342 872

F-102 1105 1230 680

F-104 504 796 1150

F-105 1221 1930 1112

F-106 1188 1462 1153

F-111 2693 3315 1262

S-3 1233 1854 429

T-38 437 538 699

T-39 257 703 468

Validate

Nearest N

Accuracy Measure

April 2017

Validate

Nearest N

Accuracy Measure

April 2017

coeffi

Nearest N

3 Lose Win

4 Lose Win

15 Lose Win

18 Lose Win

April 2017

Nearest N

20 Lose Win

21 Lose Win

22 Lose Win

23 Lose Win

24 Lose Win

25 Lose Win

26 Lose Win

27 Lose Win

28 Lose Win

29 Lose Win

30 Lose Win

31 Win Lose

32 Lose Win

Wins 1 20 2 9 1 19 8 6 10 5 9 5

Losses

20 1 9 2 19 1 6 8 5 10 5 9

Author Biographies

April 2017

Masters 15-20 years

PhD 10-15 years

PhD lt10 years

BS 20-30 years

PhD 15-20 years

Degrees M amp S VE

Bachelors 10

Medium 5

Low 10

Low 60

Low 50

High 20

Medium 35

Medium 30

Medium 40

Masters 35

PhD 55 High

95 High 75

Medium 25

April 2017

Engaging 413 1 80

Validation 427 1 93

April 2017

Fidelity 414 1 86

MBSESampT Projects

Ideation

Environment

Artifacts

Innovation Portal

Game Engines

Game Features

Play GameCompete

Environments

Models

Decision Engines

MBSE Artifacts

Lethality

April 2017

FIGURE 6 LIFE CYCLE

Br oad

Early Concept

Warfighters

EngSci

Warfighters

C ompar a tiv e

Warfighters

F ocused

606 25 75

April 2017

Fi l t e

r i n g b

y S c i e

n t i s t

s g i n

e e r s

Po r t a

l We b

s i t e C

o l l a b

o r a t

L e a d

e r s h

i p S u

p p o r

F e e d

b a c k

I n t o

P r o c

T i m e l y

F e e d

b a c k

R e c o

g n i t i

a t a S

e c u r

ion Be

t we e

n E n g

S c i e

n t i s t

a r fi g

h t e r

E n g a

e n t (

fi g h t

e r s )

S c i e n

t i s t s

amp E n

s s i b i l

v a i l a

b i l i t y

t C o m

m i t m

Abilit

e s i g n

I m p r

o v e m

i s by

C l a s

s o f S

t a k e

h o l d

D a t a

P i g r

C o n fi

o n t r o

Mi s s i

o n S p

a c e C

h c t e

r i z a t

l We b

s i t e

C o l l a

b t i o

n A r e

A b i l i t

y t o C

a p t u

f o r m

a n c e

B e h a

v i o r

A d e q

u a t e

A b i l i t

y t o A

n a l y z

e M e n

t a l T

a s k s

I n t e g

r a t i o

n w i t h

S y s t e

n g i n e

e r i n g

P r o c

L e a d

e r s h

i p S u

p p o r

t C o m

m i t m

I n t u i t

i v e I n

t e r f a

C o n s

i s t e n

i t h O

p e r a

t i o n a

D a t a

C a p t

u r e I

n t o M

e t r i c

F i d e l i

t u a l

M o d e

l o f t

h e S y

s t em

T a s k

L e a d

e r s h

i p C o

i t me n

R e l i a

b i l i t y

e a t a

b i l i t y

S e n i o

r E n g

T i e d t

o M e a

s u r e

o f P e

r f o r m

a n c e

F i d e l i

e p r e

s e n t

a t i o n

o f T r

u e S y

s t e m

We l l

D e fi

n e d M

e t r i c

A d e q

u a t e

F u n d

i n g o f

Tool s

U t i l i t

y t o S

t a k e

h o l d e

R e a l

T i m e O

p e r a

t i o n

F i d e l i

t y o f

e l i n g

A c c u

r a c y

o f Re

D a t a

R e c o

r d i n g

V e r i fi

c a t i o

l i d a t

i o n a n

d A c c r

e d i t a

t i o n

R V irt

F l e x i b

i l ity

u l a r i t

n I n v

e s t m

e n t C

o s t S

a v i n g

ll Proc

Crowdso

urcing

izatio

at iion

Dng di

i ytto

April 2017

Author Biographies

April 2017

Kui Zeng

Matthew V Cilli

April 2017

Elgin Ross Fetterly

April 2017

bull Cover letter

July January

November April

January July

April October

January

Defense ARJ

Defense ATampL

Cancellation

Last Name

First Name

DayWork Phone

E-mail Address

ver 01032017

S U R V E Y

Contents

CALL FOR AUTHORS

N2016 ACS1st

N UNIVERSITY ALUM

Featured Research

EXECUTIVE EDITOR

April 2017

Biography

April 2017

PAPER COMPETITION

2016 ACS 1st

NUNIVERSITY ALU

I ASSO

CIATIO

April 2017

Function 1

Function 2

Function 3

Function 4

FitshyForshyPurpose

Function

Measures

Analysis

Criteria

Subcriteria

Alternatives

Authentication (A1)

Authorization (A2)

LAN WAN

Accounting (A3)

RemoteshyUser

April 2017

Identify (1)

Password Aging

Alternatives (4)

Access (2)

Elements

Activity (3)

Outer Dependencies

April 2017

metropolitan area

n 1 2 3 4 5 6 7 8 RI 0 0 058 09 112 124 132 141

April 2017

Remote User

Equal Importance

Moderate Importance

Strong Importance

Extreme Importance

April 2017

18-24 1 1 4 5 11

25-34 7 2 27 6 28 70

35-44 22 1 63 21 32 139

45-54 19 4 70 41 42 176

55-64 11 1 29 15 26 82

65 gt 1 2 3 6

Grand 60 9 194 85 136 484 Total

SKIPPED 18

Legend 1 2 3 4 5

18-24 2 7 2 11

25-34 14 7 35 10 4 70

35-44 13 26 69 19 11 138

45-54 13 42 73 35 13

55-64 7 12 37 22 4

65 gt 5 1 6

Grand 47 87 216 98 35 Total

SKIPPED 19

Legend 1 2 3 4 5

18-24 2 1 1 7 11

25-34 8 19 7 36 70

35-44 16 33 17 72 138

45-54 19 37 21 99 176

55-64 11 14 10 46 81

65 gt 2 4 6

Grand 58 104 56 264 482 Total

SKIPPED 20

Legend 1 2 3 4

April 2017

Enforcement

1 22 1 16

2 2 2 17 2 8 2 7

3 9 3 21 3 19 3 13

4 7 4 24 4 10 4 24

5 22 5 66 5 41 5 53

6 15 6 34 6 17 6 25

7 14 7 40 7 25 7 36

8 3 8 12 8 4 8 9

9 6 9 19 9 7 9 12

Standard Deviation

Skewedshyness

-0278 Skewedshyness

-0176 Skewedshyness

-0161 Skewedshyness

n 89000 n 255000 n 147000 n 201000

3rd Quarshytile

70 60 50 40 30 20

0 3 6 9 12

4839 2138

373 6619

lt0005

April 2017

Total 369 169022

1 -1294 0614 -211 0036 107 2 -0115 0366 -031 0754 132 3 -0341 0313 -109 0277 176 4 -0060 0297 -020 0839 182 5 0147 0343 043 0669 138

SIZE 1 0022 0272 008 0935 302 2 0027 0228 012 0906 267 3 0117 0275 043 0670 289

Type 1 -0261 0332 -079 0433 149 2 0385 0246 156 0119 128 3 -0237 0191 -124 0216 118 4 -0293 0265 -111 0269 140

Position 1 0085 0316 027 0787 303 2 0730 0716 102 0309 897 3 -0378 0243 -155 0121 306 4 0038 0288 013 0896 303

Parameters

Residual

70 60 50 40 30 20

01 shy75 shy50 shy25 00 25 50

Y = Xβ + ε (1)

April 2017

N c (2)

April 2017

1_LAN 028083

2_WAN 013501

3_Remote 058416

LAN 012950

WAN 033985

Remote User 053065

020000

080000

LAN 012807

WAN 022686

Remote User 064507

020000

LAN 032109

WAN 013722

Remote User 054169

LAN 002650

WAN 024555 WAN 005710

April 2017

Author Biographies

April 2017

Shelley M Cazares

Operator

April 2017

System

April 2017

Ground Truth

Items (10100)

No Threat (10000)

Threat (100)

NOTIONAL System

Alarm (590)

No Alarm (9510)

TP (90) FN (10)

FP (500) TN (9500)

d = 90 (90 + 10) = 090

(near 1 is better)

fa = 500 (500 + 9500) = 005

(near 0 is better)

April 2017

Items (590)

= 90 (90 + 10) = 090

= 500 (500 + 9500) = 005

PPV1 = 90 (90 + 500) = 015 NPV

1 = 9500 (9500 + 10) asymp 1

= 88 (88 + 2) = 098

= 20 (20 + 480) = 004

PPV2 = 88 (88 + 20) = 081 NPV

2 = 480 (480 + 2) asymp 1

PPVoverall = 88 (88 + 20) = 081

Pd overall = 88 (88 + (10 + 2)) = 088

Items (10100)

(10000)

Threat (100)

NOTIONAL System 1

Alarm (590)

No Alarm (9510)

TP1 (90) FN1 (10)

FP1 (500) TN1 (9500)

Threat (90)

NOTIONAL System 2

Alarm (108)

No Alarm (482)

TP2 (88) FN2 (2)

FP2 (20) TN2 (480)

April 2017

system)

April 2017

Direction of Travel

Convoy

April 2017

FAR = FP total time

FAR = FP total area

April 2017

3 Ibid

4 Ibid

5 Ibid

April 2017

Biography

April 2017

MMH Gap Causes

Required MMHs

Available MMHs

Duty Absences

April 2017

100 fill rate

April 2017

90 80 95 50

Monthly 0 6896 23077 61327 MMH Gap

800000

700000

600000

500000

400000

300000

200000

100000

100 90 80 70

e of 1

Wartime 362330

489565

616800

113215

744034

151323

Peacetime 36999

April 2017

500000

450000

400000

350000

300000

250000

200000

150000

100000

100 90 80 70

e of 1

Wartime 124520

232550

340570

448600

Peacetime 0

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Complex System

April 2017

Requirement

Systems

April 2017

incipa

ined S

amp Co

te amp

l amp Tr

ibal F

pitals

uts amp

April 2017

Boundary Liquidity

April 2017

Complex System

Adaptive

Technical

Interdependence

Political

Holistic

Dependent System

System

(internal)

(external)

April 2017

patents

April 2017

( )( )2 2

April 2017

n minus 2

April 2017

Operations 77 2 26

Logistics 61 1 15

Academic 67 1 27

Policy 40 4 4

April 2017

0 4 500

0 4000

0 3 500

0 3000

0 2500

0 2000

0 1500

0 1000

stomer

Design Independen

vable t

ConciseHolist

icSocial

Political

Adaptable

Technical

Interdependent

Environmental

Consistent

Verifiable

Complete

Unambiguous

Correct

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Anomaly Detection

Data Mining

April 2017

F-4 1722 -087 1222 110 1399

F-105 1930 -022 1112 -086 1221

A-5 2350 109 1147 -024 1414

April 2017

Fshy4 ($1399)

Ashy5 ($1414)

Fshy105 ($1221)

Fshynotional Spee

Weight

shy2 0 2

April 2017

Aircraft Cost

Weight lt 3159

Weight lt 1221

Speed lt 992

Weight ge 3159

Weight ge 1221

Speed ge 992

$398 $928 $1305 $2228

0 0 20 40 60 80 100 120

R4 = $2228

April 2017

Weight

April 2017

Nearest N

Wins-Losses

April 2017

A-4 373 507 565

A-5 1414 2350 1147

A-6 888 1715 562

A-7 33 1162 595

A-10 629 1484 389

B-52 3203 11267 551

B-58 3243 3269 1147

BRB-66 1293 3050 548

C-130 1175 4345 326

C-133 1835 9631 304

KC-135 1555 7025 527

C-141 1891 10432 491

F3D 303 1014 470

F3H 757 1390 622

F4D 71 874 628

F-4 1399 1722 1222

F-86 248 679 590

F-89 542 1812 546

F-100 421 1212 752

F-101 893 1342 872

F-102 1105 1230 680

F-104 504 796 1150

F-105 1221 1930 1112

F-106 1188 1462 1153

F-111 2693 3315 1262

S-3 1233 1854 429

T-38 437 538 699

T-39 257 703 468

Validate

Nearest N

Accuracy Measure

April 2017

Validate

Nearest N

Accuracy Measure

April 2017

coeffi

Nearest N

3 Lose Win

4 Lose Win

15 Lose Win

18 Lose Win

April 2017

Nearest N

20 Lose Win

21 Lose Win

22 Lose Win

23 Lose Win

24 Lose Win

25 Lose Win

26 Lose Win

27 Lose Win

28 Lose Win

29 Lose Win

30 Lose Win

31 Win Lose

32 Lose Win

Wins 1 20 2 9 1 19 8 6 10 5 9 5

Losses

20 1 9 2 19 1 6 8 5 10 5 9

Author Biographies

April 2017

Masters 15-20 years

PhD 10-15 years

PhD lt10 years

BS 20-30 years

PhD 15-20 years

Degrees M amp S VE

Bachelors 10

Medium 5

Low 10

Low 60

Low 50

High 20

Medium 35

Medium 30

Medium 40

Masters 35

PhD 55 High

95 High 75

Medium 25

April 2017

Engaging 413 1 80

Validation 427 1 93

April 2017

Fidelity 414 1 86

MBSESampT Projects

Ideation

Environment

Artifacts

Innovation Portal

Game Engines

Game Features

Play GameCompete

Environments

Models

Decision Engines

MBSE Artifacts

Lethality

April 2017

FIGURE 6 LIFE CYCLE

Br oad

Early Concept

Warfighters

EngSci

Warfighters

C ompar a tiv e

Warfighters

F ocused

606 25 75

April 2017

Fi l t e

r i n g b

y S c i e

n t i s t

s g i n

e e r s

Po r t a

l We b

s i t e C

o l l a b

o r a t

L e a d

e r s h

i p S u

p p o r

F e e d

b a c k

I n t o

P r o c

T i m e l y

F e e d

b a c k

R e c o

g n i t i

a t a S

e c u r

ion Be

t we e

n E n g

S c i e

n t i s t

a r fi g

h t e r

E n g a

e n t (

fi g h t

e r s )

S c i e n

t i s t s

amp E n

s s i b i l

v a i l a

b i l i t y

t C o m

m i t m

Abilit

e s i g n

I m p r

o v e m

i s by

C l a s

s o f S

t a k e

h o l d

D a t a

P i g r

C o n fi

o n t r o

Mi s s i

o n S p

a c e C

h c t e

r i z a t

l We b

s i t e

C o l l a

b t i o

n A r e

A b i l i t

y t o C

a p t u

f o r m

a n c e

B e h a

v i o r

A d e q

u a t e

A b i l i t

y t o A

n a l y z

e M e n

t a l T

a s k s

I n t e g

r a t i o

n w i t h

S y s t e

n g i n e

e r i n g

P r o c

L e a d

e r s h

i p S u

p p o r

t C o m

m i t m

I n t u i t

i v e I n

t e r f a

C o n s

i s t e n

i t h O

p e r a

t i o n a

D a t a

C a p t

u r e I

n t o M

e t r i c

F i d e l i

t u a l

M o d e

l o f t

h e S y

s t em

T a s k

L e a d

e r s h

i p C o

i t me n

R e l i a

b i l i t y

e a t a

b i l i t y

S e n i o

r E n g

T i e d t

o M e a

s u r e

o f P e

r f o r m

a n c e

F i d e l i

e p r e

s e n t

a t i o n

o f T r

u e S y

s t e m

We l l

D e fi

n e d M

e t r i c

A d e q

u a t e

F u n d

i n g o f

Tool s

U t i l i t

y t o S

t a k e

h o l d e

R e a l

T i m e O

p e r a

t i o n

F i d e l i

t y o f

e l i n g

A c c u

r a c y

o f Re

D a t a

R e c o

r d i n g

V e r i fi

c a t i o

l i d a t

i o n a n

d A c c r

e d i t a

t i o n

R V irt

F l e x i b

i l ity

u l a r i t

n I n v

e s t m

e n t C

o s t S

a v i n g

ll Proc

Crowdso

urcing

izatio

at iion

Dng di

i ytto

April 2017

Author Biographies

April 2017

Kui Zeng

Matthew V Cilli

April 2017

Elgin Ross Fetterly

April 2017

bull Cover letter

July January

November April

January July

April October

January

Defense ARJ

Defense ATampL

Cancellation

Last Name

First Name

DayWork Phone

E-mail Address

ver 01032017

S U R V E Y

Contents

CALL FOR AUTHORS

N2016 ACS1st

N UNIVERSITY ALUM

Featured Research

EXECUTIVE EDITOR

April 2017

Biography

April 2017

PAPER COMPETITION

2016 ACS 1st

NUNIVERSITY ALU

I ASSO

CIATIO

April 2017

Function 1

Function 2

Function 3

Function 4

FitshyForshyPurpose

Function

Measures

Analysis

Criteria

Subcriteria

Alternatives

Authentication (A1)

Authorization (A2)

LAN WAN

Accounting (A3)

RemoteshyUser

April 2017

Identify (1)

Password Aging

Alternatives (4)

Access (2)

Elements

Activity (3)

Outer Dependencies

April 2017

metropolitan area

n 1 2 3 4 5 6 7 8 RI 0 0 058 09 112 124 132 141

April 2017

Remote User

Equal Importance

Moderate Importance

Strong Importance

Extreme Importance

April 2017

18-24 1 1 4 5 11

25-34 7 2 27 6 28 70

35-44 22 1 63 21 32 139

45-54 19 4 70 41 42 176

55-64 11 1 29 15 26 82

65 gt 1 2 3 6

Grand 60 9 194 85 136 484 Total

SKIPPED 18

Legend 1 2 3 4 5

18-24 2 7 2 11

25-34 14 7 35 10 4 70

35-44 13 26 69 19 11 138

45-54 13 42 73 35 13

55-64 7 12 37 22 4

65 gt 5 1 6

Grand 47 87 216 98 35 Total

SKIPPED 19

Legend 1 2 3 4 5

18-24 2 1 1 7 11

25-34 8 19 7 36 70

35-44 16 33 17 72 138

45-54 19 37 21 99 176

55-64 11 14 10 46 81

65 gt 2 4 6

Grand 58 104 56 264 482 Total

SKIPPED 20

Legend 1 2 3 4

April 2017

Enforcement

1 22 1 16

2 2 2 17 2 8 2 7

3 9 3 21 3 19 3 13

4 7 4 24 4 10 4 24

5 22 5 66 5 41 5 53

6 15 6 34 6 17 6 25

7 14 7 40 7 25 7 36

8 3 8 12 8 4 8 9

9 6 9 19 9 7 9 12

Standard Deviation

Skewedshyness

-0278 Skewedshyness

-0176 Skewedshyness

-0161 Skewedshyness

n 89000 n 255000 n 147000 n 201000

3rd Quarshytile

70 60 50 40 30 20

0 3 6 9 12

4839 2138

373 6619

lt0005

April 2017

Total 369 169022

1 -1294 0614 -211 0036 107 2 -0115 0366 -031 0754 132 3 -0341 0313 -109 0277 176 4 -0060 0297 -020 0839 182 5 0147 0343 043 0669 138

SIZE 1 0022 0272 008 0935 302 2 0027 0228 012 0906 267 3 0117 0275 043 0670 289

Type 1 -0261 0332 -079 0433 149 2 0385 0246 156 0119 128 3 -0237 0191 -124 0216 118 4 -0293 0265 -111 0269 140

Position 1 0085 0316 027 0787 303 2 0730 0716 102 0309 897 3 -0378 0243 -155 0121 306 4 0038 0288 013 0896 303

Parameters

Residual

70 60 50 40 30 20

01 shy75 shy50 shy25 00 25 50

Y = Xβ + ε (1)

April 2017

N c (2)

April 2017

1_LAN 028083

2_WAN 013501

3_Remote 058416

LAN 012950

WAN 033985

Remote User 053065

020000

080000

LAN 012807

WAN 022686

Remote User 064507

020000

LAN 032109

WAN 013722

Remote User 054169

LAN 002650

WAN 024555 WAN 005710

April 2017

Author Biographies

April 2017

Shelley M Cazares

Operator

April 2017

System

April 2017

Ground Truth

Items (10100)

No Threat (10000)

Threat (100)

NOTIONAL System

Alarm (590)

No Alarm (9510)

TP (90) FN (10)

FP (500) TN (9500)

d = 90 (90 + 10) = 090

(near 1 is better)

fa = 500 (500 + 9500) = 005

(near 0 is better)

April 2017

Items (590)

= 90 (90 + 10) = 090

= 500 (500 + 9500) = 005

PPV1 = 90 (90 + 500) = 015 NPV

1 = 9500 (9500 + 10) asymp 1

= 88 (88 + 2) = 098

= 20 (20 + 480) = 004

PPV2 = 88 (88 + 20) = 081 NPV

2 = 480 (480 + 2) asymp 1

PPVoverall = 88 (88 + 20) = 081

Pd overall = 88 (88 + (10 + 2)) = 088

Items (10100)

(10000)

Threat (100)

NOTIONAL System 1

Alarm (590)

No Alarm (9510)

TP1 (90) FN1 (10)

FP1 (500) TN1 (9500)

Threat (90)

NOTIONAL System 2

Alarm (108)

No Alarm (482)

TP2 (88) FN2 (2)

FP2 (20) TN2 (480)

April 2017

system)

April 2017

Direction of Travel

Convoy

April 2017

FAR = FP total time

FAR = FP total area

April 2017

3 Ibid

4 Ibid

5 Ibid

April 2017

Biography

April 2017

MMH Gap Causes

Required MMHs

Available MMHs

Duty Absences

April 2017

100 fill rate

April 2017

90 80 95 50

Monthly 0 6896 23077 61327 MMH Gap

800000

700000

600000

500000

400000

300000

200000

100000

100 90 80 70

e of 1

Wartime 362330

489565

616800

113215

744034

151323

Peacetime 36999

April 2017

500000

450000

400000

350000

300000

250000

200000

150000

100000

100 90 80 70

e of 1

Wartime 124520

232550

340570

448600

Peacetime 0

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Complex System

April 2017

Requirement

Systems

April 2017

incipa

ined S

amp Co

te amp

l amp Tr

ibal F

pitals

uts amp

April 2017

Boundary Liquidity

April 2017

Complex System

Adaptive

Technical

Interdependence

Political

Holistic

Dependent System

System

(internal)

(external)

April 2017

patents

April 2017

( )( )2 2

April 2017

n minus 2

April 2017

Operations 77 2 26

Logistics 61 1 15

Academic 67 1 27

Policy 40 4 4

April 2017

0 4 500

0 4000

0 3 500

0 3000

0 2500

0 2000

0 1500

0 1000

stomer

Design Independen

vable t

ConciseHolist

icSocial

Political

Adaptable

Technical

Interdependent

Environmental

Consistent

Verifiable

Complete

Unambiguous

Correct

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Anomaly Detection

Data Mining

April 2017

F-4 1722 -087 1222 110 1399

F-105 1930 -022 1112 -086 1221

A-5 2350 109 1147 -024 1414

April 2017

Fshy4 ($1399)

Ashy5 ($1414)

Fshy105 ($1221)

Fshynotional Spee

Weight

shy2 0 2

April 2017

Aircraft Cost

Weight lt 3159

Weight lt 1221

Speed lt 992

Weight ge 3159

Weight ge 1221

Speed ge 992

$398 $928 $1305 $2228

0 0 20 40 60 80 100 120

R4 = $2228

April 2017

Weight

April 2017

Nearest N

Wins-Losses

April 2017

A-4 373 507 565

A-5 1414 2350 1147

A-6 888 1715 562

A-7 33 1162 595

A-10 629 1484 389

B-52 3203 11267 551

B-58 3243 3269 1147

BRB-66 1293 3050 548

C-130 1175 4345 326

C-133 1835 9631 304

KC-135 1555 7025 527

C-141 1891 10432 491

F3D 303 1014 470

F3H 757 1390 622

F4D 71 874 628

F-4 1399 1722 1222

F-86 248 679 590

F-89 542 1812 546

F-100 421 1212 752

F-101 893 1342 872

F-102 1105 1230 680

F-104 504 796 1150

F-105 1221 1930 1112

F-106 1188 1462 1153

F-111 2693 3315 1262

S-3 1233 1854 429

T-38 437 538 699

T-39 257 703 468

Validate

Nearest N

Accuracy Measure

April 2017

Validate

Nearest N

Accuracy Measure

April 2017

coeffi

Nearest N

3 Lose Win

4 Lose Win

15 Lose Win

18 Lose Win

April 2017

Nearest N

20 Lose Win

21 Lose Win

22 Lose Win

23 Lose Win

24 Lose Win

25 Lose Win

26 Lose Win

27 Lose Win

28 Lose Win

29 Lose Win

30 Lose Win

31 Win Lose

32 Lose Win

Wins 1 20 2 9 1 19 8 6 10 5 9 5

Losses

20 1 9 2 19 1 6 8 5 10 5 9

Author Biographies

April 2017

Masters 15-20 years

PhD 10-15 years

PhD lt10 years

BS 20-30 years

PhD 15-20 years

Degrees M amp S VE

Bachelors 10

Medium 5

Low 10

Low 60

Low 50

High 20

Medium 35

Medium 30

Medium 40

Masters 35

PhD 55 High

95 High 75

Medium 25

April 2017

Engaging 413 1 80

Validation 427 1 93

April 2017

Fidelity 414 1 86

MBSESampT Projects

Ideation

Environment

Artifacts

Innovation Portal

Game Engines

Game Features

Play GameCompete

Environments

Models

Decision Engines

MBSE Artifacts

Lethality

April 2017

FIGURE 6 LIFE CYCLE

Br oad

Early Concept

Warfighters

EngSci

Warfighters

C ompar a tiv e

Warfighters

F ocused

606 25 75

April 2017

Fi l t e

r i n g b

y S c i e

n t i s t

s g i n

e e r s

Po r t a

l We b

s i t e C

o l l a b

o r a t

L e a d

e r s h

i p S u

p p o r

F e e d

b a c k

I n t o

P r o c

T i m e l y

F e e d

b a c k

R e c o

g n i t i

a t a S

e c u r

ion Be

t we e

n E n g

S c i e

n t i s t

a r fi g

h t e r

E n g a

e n t (

fi g h t

e r s )

S c i e n

t i s t s

amp E n

s s i b i l

v a i l a

b i l i t y

t C o m

m i t m

Abilit

e s i g n

I m p r

o v e m

i s by

C l a s

s o f S

t a k e

h o l d

D a t a

P i g r

C o n fi

o n t r o

Mi s s i

o n S p

a c e C

h c t e

r i z a t

l We b

s i t e

C o l l a

b t i o

n A r e

A b i l i t

y t o C

a p t u

f o r m

a n c e

B e h a

v i o r

A d e q

u a t e

A b i l i t

y t o A

n a l y z

e M e n

t a l T

a s k s

I n t e g

r a t i o

n w i t h

S y s t e

n g i n e

e r i n g

P r o c

L e a d

e r s h

i p S u

p p o r

t C o m

m i t m

I n t u i t

i v e I n

t e r f a

C o n s

i s t e n

i t h O

p e r a

t i o n a

D a t a

C a p t

u r e I

n t o M

e t r i c

F i d e l i

t u a l

M o d e

l o f t

h e S y

s t em

T a s k

L e a d

e r s h

i p C o

i t me n

R e l i a

b i l i t y

e a t a

b i l i t y

S e n i o

r E n g

T i e d t

o M e a

s u r e

o f P e

r f o r m

a n c e

F i d e l i

e p r e

s e n t

a t i o n

o f T r

u e S y

s t e m

We l l

D e fi

n e d M

e t r i c

A d e q

u a t e

F u n d

i n g o f

Tool s

U t i l i t

y t o S

t a k e

h o l d e

R e a l

T i m e O

p e r a

t i o n

F i d e l i

t y o f

e l i n g

A c c u

r a c y

o f Re

D a t a

R e c o

r d i n g

V e r i fi

c a t i o

l i d a t

i o n a n

d A c c r

e d i t a

t i o n

R V irt

F l e x i b

i l ity

u l a r i t

n I n v

e s t m

e n t C

o s t S

a v i n g

ll Proc

Crowdso

urcing

izatio

at iion

Dng di

i ytto

April 2017

Author Biographies

April 2017

Kui Zeng

Matthew V Cilli

April 2017

Elgin Ross Fetterly

April 2017

bull Cover letter

July January

November April

January July

April October

January

Defense ARJ

Defense ATampL

Cancellation

Last Name

First Name

DayWork Phone

E-mail Address

ver 01032017

S U R V E Y

Contents

CALL FOR AUTHORS

N2016 ACS1st

N UNIVERSITY ALUM

Featured Research

EXECUTIVE EDITOR

April 2017

Biography

April 2017

PAPER COMPETITION

2016 ACS 1st

NUNIVERSITY ALU

I ASSO

CIATIO

April 2017

Function 1

Function 2

Function 3

Function 4

FitshyForshyPurpose

Function

Measures

Analysis

Criteria

Subcriteria

Alternatives

Authentication (A1)

Authorization (A2)

LAN WAN

Accounting (A3)

RemoteshyUser

April 2017

Identify (1)

Password Aging

Alternatives (4)

Access (2)

Elements

Activity (3)

Outer Dependencies

April 2017

metropolitan area

n 1 2 3 4 5 6 7 8 RI 0 0 058 09 112 124 132 141

April 2017

Remote User

Equal Importance

Moderate Importance

Strong Importance

Extreme Importance

April 2017

18-24 1 1 4 5 11

25-34 7 2 27 6 28 70

35-44 22 1 63 21 32 139

45-54 19 4 70 41 42 176

55-64 11 1 29 15 26 82

65 gt 1 2 3 6

Grand 60 9 194 85 136 484 Total

SKIPPED 18

Legend 1 2 3 4 5

18-24 2 7 2 11

25-34 14 7 35 10 4 70

35-44 13 26 69 19 11 138

45-54 13 42 73 35 13

55-64 7 12 37 22 4

65 gt 5 1 6

Grand 47 87 216 98 35 Total

SKIPPED 19

Legend 1 2 3 4 5

18-24 2 1 1 7 11

25-34 8 19 7 36 70

35-44 16 33 17 72 138

45-54 19 37 21 99 176

55-64 11 14 10 46 81

65 gt 2 4 6

Grand 58 104 56 264 482 Total

SKIPPED 20

Legend 1 2 3 4

April 2017

Enforcement

1 22 1 16

2 2 2 17 2 8 2 7

3 9 3 21 3 19 3 13

4 7 4 24 4 10 4 24

5 22 5 66 5 41 5 53

6 15 6 34 6 17 6 25

7 14 7 40 7 25 7 36

8 3 8 12 8 4 8 9

9 6 9 19 9 7 9 12

Standard Deviation

Skewedshyness

-0278 Skewedshyness

-0176 Skewedshyness

-0161 Skewedshyness

n 89000 n 255000 n 147000 n 201000

3rd Quarshytile

70 60 50 40 30 20

0 3 6 9 12

4839 2138

373 6619

lt0005

April 2017

Total 369 169022

1 -1294 0614 -211 0036 107 2 -0115 0366 -031 0754 132 3 -0341 0313 -109 0277 176 4 -0060 0297 -020 0839 182 5 0147 0343 043 0669 138

SIZE 1 0022 0272 008 0935 302 2 0027 0228 012 0906 267 3 0117 0275 043 0670 289

Type 1 -0261 0332 -079 0433 149 2 0385 0246 156 0119 128 3 -0237 0191 -124 0216 118 4 -0293 0265 -111 0269 140

Position 1 0085 0316 027 0787 303 2 0730 0716 102 0309 897 3 -0378 0243 -155 0121 306 4 0038 0288 013 0896 303

Parameters

Residual

70 60 50 40 30 20

01 shy75 shy50 shy25 00 25 50

Y = Xβ + ε (1)

April 2017

N c (2)

April 2017

1_LAN 028083

2_WAN 013501

3_Remote 058416

LAN 012950

WAN 033985

Remote User 053065

020000

080000

LAN 012807

WAN 022686

Remote User 064507

020000

LAN 032109

WAN 013722

Remote User 054169

LAN 002650

WAN 024555 WAN 005710

April 2017

Author Biographies

April 2017

Shelley M Cazares

Operator

April 2017

System

April 2017

Ground Truth

Items (10100)

No Threat (10000)

Threat (100)

NOTIONAL System

Alarm (590)

No Alarm (9510)

TP (90) FN (10)

FP (500) TN (9500)

d = 90 (90 + 10) = 090

(near 1 is better)

fa = 500 (500 + 9500) = 005

(near 0 is better)

April 2017

Items (590)

= 90 (90 + 10) = 090

= 500 (500 + 9500) = 005

PPV1 = 90 (90 + 500) = 015 NPV

1 = 9500 (9500 + 10) asymp 1

= 88 (88 + 2) = 098

= 20 (20 + 480) = 004

PPV2 = 88 (88 + 20) = 081 NPV

2 = 480 (480 + 2) asymp 1

PPVoverall = 88 (88 + 20) = 081

Pd overall = 88 (88 + (10 + 2)) = 088

Items (10100)

(10000)

Threat (100)

NOTIONAL System 1

Alarm (590)

No Alarm (9510)

TP1 (90) FN1 (10)

FP1 (500) TN1 (9500)

Threat (90)

NOTIONAL System 2

Alarm (108)

No Alarm (482)

TP2 (88) FN2 (2)

FP2 (20) TN2 (480)

April 2017

system)

April 2017

Direction of Travel

Convoy

April 2017

FAR = FP total time

FAR = FP total area

April 2017

3 Ibid

4 Ibid

5 Ibid

April 2017

Biography

April 2017

MMH Gap Causes

Required MMHs

Available MMHs

Duty Absences

April 2017

100 fill rate

April 2017

90 80 95 50

Monthly 0 6896 23077 61327 MMH Gap

800000

700000

600000

500000

400000

300000

200000

100000

100 90 80 70

e of 1

Wartime 362330

489565

616800

113215

744034

151323

Peacetime 36999

April 2017

500000

450000

400000

350000

300000

250000

200000

150000

100000

100 90 80 70

e of 1

Wartime 124520

232550

340570

448600

Peacetime 0

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Complex System

April 2017

Requirement

Systems

April 2017

incipa

ined S

amp Co

te amp

l amp Tr

ibal F

pitals

uts amp

April 2017

Boundary Liquidity

April 2017

Complex System

Adaptive

Technical

Interdependence

Political

Holistic

Dependent System

System

(internal)

(external)

April 2017

patents

April 2017

( )( )2 2

April 2017

n minus 2

April 2017

Operations 77 2 26

Logistics 61 1 15

Academic 67 1 27

Policy 40 4 4

April 2017

0 4 500

0 4000

0 3 500

0 3000

0 2500

0 2000

0 1500

0 1000

stomer

Design Independen

vable t

ConciseHolist

icSocial

Political

Adaptable

Technical

Interdependent

Environmental

Consistent

Verifiable

Complete

Unambiguous

Correct

April 2017

Author Biographies

April 2017

Author Biographies

April 2017

Anomaly Detection

Data Mining

April 2017

F-4 1722 -087 1222 110 1399

F-105 1930 -022 1112 -086 1221

A-5 2350 109 1147 -024 1414

April 2017

Fshy4 ($1399)

Ashy5 ($1414)

Fshy105 ($1221)

Fshynotional Spee

Weight

shy2 0 2

April 2017

Aircraft Cost

Weight lt 3159

Weight lt 1221

Speed lt 992

Weight ge 3159

Weight ge 1221

Speed ge 992

$398 $928 $1305 $2228

0 0 20 40 60 80 100 120

R4 = $2228

April 2017

Weight

April 2017

Nearest N

Wins-Losses

April 2017

A-4 373 507 565

A-5 1414 2350 1147

A-6 888 1715 562

A-7 33 1162 595

A-10 629 1484 389

B-52 3203 11267 551

B-58 3243 3269 1147

BRB-66 1293 3050 548

C-130 1175 4345 326

C-133 1835 9631 304

KC-135 1555 7025 527

C-141 1891 10432 491

F3D 303 1014 470

F3H 757 1390 622

F4D 71 874 628

F-4 1399 1722 1222

F-86 248 679 590

F-89 542 1812 546

F-100 421 1212 752

F-101 893 1342 872

F-102 1105 1230 680

F-104 504 796 1150

F-105 1221 1930 1112

F-106 1188 1462 1153

F-111 2693 3315 1262

S-3 1233 1854 429

T-38 437 538 699

T-39 257 703 468

Validate

Nearest N

Accuracy Measure

April 2017

Validate

Nearest N

Accuracy Measure

April 2017

coeffi

Nearest N

3 Lose Win

4 Lose Win

15 Lose Win

18 Lose Win

April 2017

Nearest N

20 Lose Win

21 Lose Win

22 Lose Win

23 Lose Win

24 Lose Win

25 Lose Win

26 Lose Win

27 Lose Win

28 Lose Win

29 Lose Win

30 Lose Win

31 Win Lose

32 Lose Win

Wins 1 20 2 9 1 19 8 6 10 5 9 5

Losses

20 1 9 2 19 1 6 8 5 10 5 9

Author Biographies

April 2017

Masters 15-20 years

PhD 10-15 years

PhD lt10 years

BS 20-30 years

PhD 15-20 years

Degrees M amp S VE

Bachelors 10

Medium 5

Low 10

Low 60

Low 50

High 20

Medium 35

Medium 30

Medium 40

Masters 35

PhD 55 High

95 High 75

Medium 25

April 2017

Engaging 413 1 80

Validation 427 1 93

April 2017

Fidelity 414 1 86

MBSESampT Projects

Ideation

Environment

Artifacts

Innovation Portal

Game Engines

Game Features

Play GameCompete

Environments

Models

Decision Engines

MBSE Artifacts

Lethality

April 2017

FIGURE 6 LIFE CYCLE

Br oad

Early Concept

Warfighters

EngSci

Warfighters

C ompar a tiv e

Warfighters

F ocused

606 25 75

April 2017

Fi l t e

r i n g b

y S c i e

n t i s t

s g i n

e e r s

Po r t a

l We b

s i t e C

o l l a b

o r a t

L e a d

e r s h

i p S u

p p o r

F e e d

b a c k

I n t o

P r o c

T i m e l y

F e e d

b a c k

R e c o

g n i t i

a t a S

e c u r

ion Be

t we e

n E n g

S c i e

n t i s t

a r fi g

h t e r

E n g a

e n t (

fi g h t

e r s )

S c i e n

t i s t s

amp E n

s s i b i l

v a i l a

b i l i t y

t C o m

m i t m

Abilit

e s i g n

I m p r

o v e m

i s by

C l a s

s o f S

t a k e

h o l d

D a t a

P i g r

C o n fi

o n t r o

Mi s s i

o n S p

a c e C

h c t e

r i z a t

l We b

s i t e

C o l l a

b t i o

n A r e

A b i l i t

y t o C

a p t u

f o r m

a n c e

B e h a

v i o r

A d e q

u a t e

A b i l i t

y t o A

n a l y z

e M e n

t a l T

a s k s

I n t e g

r a t i o

n w i t h

S y s t e

n g i n e

e r i n g

P r o c

L e a d

e r s h

i p S u

p p o r

t C o m

m i t m

I n t u i t

i v e I n

t e r f a

C o n s

i s t e n

i t h O

p e r a

t i o n a

D a t a

C a p t

u r e I

n t o M

e t r i c

F i d e l i

t u a l

M o d e

l o f t

h e S y

s t em

T a s k

L e a d

e r s h

i p C o

i t me n

R e l i a

b i l i t y

e a t a

b i l i t y

S e n i o

r E n g

T i e d t

o M e a

s u r e

o f P e

r f o r m

a n c e

F i d e l i

e p r e

s e n t

a t i o n

o f T r

u e S y

s t e m

We l l

D e fi

n e d M

e t r i c

A d e q

u a t e

F u n d

i n g o f

Tool s

U t i l i t

y t o S

t a k e

h o l d e

R e a l

T i m e O

p e r a

t i o n

F i d e l i

t y o f

e l i n g

A c c u

r a c y

o f Re

D a t a

R e c o

r d i n g

V e r i fi

c a t i o

l i d a t

i o n a n

d A c c r

e d i t a

t i o n

R V irt

F l e x i b

i l ity

u l a r i t

n I n v

e s t m

e n t C

o s t S

a v i n g

ll Proc

Crowdso

urcing

izatio

at iion

Dng di

i ytto

April 2017

Author Biographies

April 2017

Kui Zeng

Matthew V Cilli

April 2017

Elgin Ross Fetterly

April 2017

bull Cover letter

July January

November April

January July

April October

January

Defense ARJ

Defense ATampL

Cancellation

Last Name

First Name

DayWork Phone

E-mail Address

ver 01032017

S U R V E Y

Contents

CALL FOR AUTHORS