Validation Methodologies for Netcentric Warfare Simulations
Wesley N. Colley, Ph.D.Center for Modeling, Simulation and Analysis
University of Alabama in Huntsville
CMSA SPARTA
Jared Biggs Chris Noller
Randy Harrell Jeff Roberts
Jim Walsh
Sponsor: Naval Air Warfare Center, Weapons Division
Validation for Netcentric Warfare Simulations
Presentation outline
• Motivation
• Validation Context
• Quantitative Validation
• InterTEC Joint Fires Exercise
Validation for Netcentric Warfare Simulations
Problem Statement / Project Focus
• Netcentric Warfare is new– Classic “face” validation by battle-experienced
SMEs not generally possible
• Networks are classically difficult to characterize– Nonlinear/brittle behaviors mean much more
careful quantitative analysis is necessary
• Focus: Develop quantitative validation techniques for NCW simulations.
Validation for Netcentric Warfare Simulations
TestComponent
PostProcessing
MetricsGathering
Real WorldTest Data
DesignMOP/MOE
MetricsAnalysis
V&VAnalysis
met
rics
responses
outputs
sta
tistic
s
experiment design
MathematicalV&V
stim
uli
validmetricvalues
Project Overview
Validation for Netcentric Warfare Simulations
Scope of Validation• Our Goal: Validate the NCW-specific aspects of systems (in
constructive simulation)
• Two possibilities:– Inherent NCW system (Link-16)
– NCW components of non-NCW system (Link-16 on F35)
NetcentricC2 Environment NCW System (e.g., Link-16)
Weapons System (e.g., F35)
Weapons System NCWInterface (F35 Link-16 Hardware)
Validation for Netcentric Warfare Simulations
NCW Effectiveness and Networks • NCW Goal: Full-spectrum information
dominancetimeliness
accuracyrelevance
InformationSuperiorityProduct
Easiest axis: timeliness“Simply” improve network
(Albert’s NCW, 2000)Network metric is thusTimeliness of messages
Validation for Netcentric Warfare Simulations
ServiceRegistry
Identity
InfrastructureInfrastructure
B2B Enterprise C2 Logicw/Distributed Services
L&
A
Pub/Sub
•Logon & Authenticate•Publish
•Sensor (SPY)•Distributed Sensor CPU•Fires (SM)•Alerts•PLI (Position)
•Subscribe•Threat•Environment•Blue Force•Combat ID (IFF)•ROE•Target ID•Alerts
Logon & Authenticate (L&A)
Subsc
ribe
Publis
h
•Logon & Authenticate•Publish
•PLI (Position)•Subscribe
•Threat•Environment•Blue Force ID•Combat ID (IFF)•ROE•Alerts
•Logon & Authenticate•Publish
•Sensor (MP-RTIP)•Forward Pass (SM)•PLI (Position)
•Subscribe•Hand-Off (SM)•Target ID (HRR)
Establishing ServicesEstablishing Services
What Does the Network Have to Do?
L&A
Pub/Sub
TrackService
AlertService
Publish Track
RC
V &
Pro
cess
Tra
ck/T
arge
t
Gener
ate A
lert/T
arge
t ID
RC
V A
lert/Target ID
RCV Alert/Target ID
Weapon Data Link
Weapon DeliveryWeapon Data Link
Weapon Provider
(Sensor to Shooter) (Sensor to Weapon)
Forward Pass R
cv
Enga
gem
ent w
/For
war
d Pa
ss
Delivering ServicesDelivering Services
DETECT
.
Validation for Netcentric Warfare Simulations
Network Composeability Table
Communications linkages
Dial-a-Comm Link Select the software programmable or networked radios connection and waveform
JTRS, VRC-99, EHF MDR, TC, Teleport, SHF/CA FCs, TCDL,MUOS
Compose the kind of connectivity and raw bandwidth you need to support your mission
Operational networkingDial-a-Network
Operational network formation to meet the force and mission needs (e.g. GIG-BE, TCS, JTF Warnet, EHF MDR)
GIG-BE, TCS, JTF Warnet, EHF MDR,
MUOS, TSAT
Compose the community of interests you need to support your mission
Information flow management
Dial-a-Precedence
Establish bandwidth allocation and priority for applications and key individuals
ADNS, BMAC QOS, TCS
Compose lanes with different speed limits and priority for your mission related data flows
Distributed information processing and storage
Dial-a-Computer
Establish roles of the computers that will support the mission, their interfaces to your FORCEnet, and the information managers who run them
NCES, XTCF, DJC2, OA, RAPIDS, CORBA, GRS, IT-21, NMCI, CAS, COWAN/CENTRIX
Compose your information management environment
Full spectrum military and civil affairs program applications
Dial-an-Application
Establish the operational cells and specific applications that will pubish information into the FORCEnet
GCCS-M, GCSS-M, OA, TMIP, IBV, RAMIS, NTCSS, TBMCS, ADOCS, JBMCS
Compose the sensing, planning, decision support, and weapons applications that will publish into the FORCEnet to support support your mission
Virtual collaborationDial-a-Meeting
Establish the collaborative environment that the participants in the mission use to coordinate actions and activities
GeoViz, IWS, DCTS, NetMeeting, VoIP, IP VTC
Compose the virtual rooms, participants, schedule, and battle rhythm
Functional, temporal, and geospacial visualization
Dial-a-GUI
Establish the standards for the form of presentation and FORCEnet subscription rules to be used to support your mission
GeoViz, WebCOP, K-WEB,
Compose the Alerting, Status Board, and COP views to be shared by the forceScads of systems with scads of uses
Validation for Netcentric Warfare Simulations
Recommendation• Use Missions × Means breakdown for validation
– Provides hierarchical decomposition of• Missions: the desired tasks or capabilities
• Means: the hardware and/or software assets available
– Successfully used by Petty, Colley for validation of JTLS in Terminal Fury ’05 (published at SIW Fall 2005)
• Select from broad NCW categories– Use M×M to drill down for specific systems
– Methodologies should be similar within broad categories
Validation for Netcentric Warfare Simulations
Missions × Means Example Ground Force Comm.
SecureConnection
Single Soldier
Receive
Voice Data Voice
GIG
JTR
S JTR
S L
AN
JTR
S r
adio
voic
eda
ta
Send
outg
oing
inco
min
g
voic
eda
talo
gon
band
wid
thal
loca
tion
Data
auth
en-
tica
te
= Evidence Item
Validation for Netcentric Warfare Simulations
Evidence Item• Element in M×M matrix where means
supports mission
• Each will have quantitative measures of performance (MOP)– Latency (e.g., time to authenticate connection)– Bandwidth consumed– Total packets sent/received– Compliance with protocol
Validation for Netcentric Warfare Simulations
Validation• Compare MOPs to known standards
– TCP/IP– JTRS specification– FORCEnet documentation, etc.
FORCEnet Architecture & Standards Volume I (Cont)
5.0 FORCEnet Functional Architecture5.1 DoD Architecture Framework Document
5.2 Mission Capability Packages
5.3 Communications and Networks
5.4 Intelligence, Surveillance, and Reconnaissance (ISR)
5.5 Distributed Services, Common Operational and Tactical
6.0 Implementation Reference Mission6.1 SEA STRIKE6.2 SEA SHIELD6.3 SEA BASE6.4 FORCEnet
AppendicesAppendix A AV-1
Appendix B ICD – FORCEnet Capabilities
Appendix C FORCEnet Service Category Definitions
Appendix D Allied and Coalition FORCEnet Architecture
Appendix E Human Systems Integration
Appendix F FORCEnet System Descriptions
Appendix G SP-21 Pillar POR Standards Compliance
Appendix H List of Acronyms
Appendix I References
Statement A: Approved for public release; distribution is unlimited (4/20/04)
F O R C E n e t A rc h ite c tu re & S ta n d a rd s V o lu m e I
T a b le o f C o n te n ts :1 .0 F O R C E n e t D e fin itio n
1 .1 F O R C E n e t O b je c tiv e -T ra ns form a tio n1 .2 A rc h ite c tu re O v e rv ie w
2 .0 B a c k g ro u n d2 .1 S e a P o w e r 21 P illa rs2 .2 F O R C E n e t P re c e p ts2 .3 F O R C E n e t A rc h ite c tu ra l S c op e2 .4 F O R C E n e t D e v e lo pm e n t P ro c e ss2 .5 F O R C E n e t A rc h ite c tu re Ap p ro a c h
3 .0 F O R C E n e t R e q u ire m e n ts3 .1 O p e ra tio n a l V ie w (O V -1 )3 .2 In itia l C a pa b ilit ie s D o c um e n t (IC D )3 .3 F le e t C a p a b ility N e e d s
4 .0 F O R C E n e t D riv e rs4 .1 D o D D riv e rs4 .2 N a v a l4 .3 C o m m erc ia l T ec h n o lo gy U p d a te4 .4 A llie d /C o a lit io n In te ro p e ra b ility
A r c h it e c t u r e & S ta n d a r d s
V o lu m e I
O p e r a t io n a l & S y s t e m s V ie w
O f f ic e o f t h e C h ie f E n g in e e r
S P A W A R 0 5
D is t r ib u t io n D : D is t r ib u t io n a u th o r i z e d to t h e D e p a rt m e n t o f D e f e n s e a n d U .S . D o D c o n t r a c t o rs o n l y ( O p e r a tio n a l U s e ) 0 3 N o v e m b e r 2 0 0 3 . O th e r re q u e s t s s h a l l b e r e fe r r e d t o t h e S p a c e a n d N a v a l W a r f a r e S y s t e m s C o m m a n d O f f i c e ( A r c h i t e c t u r e a n d S t a n d a rd s S P A W A R C o d e 0 5 2 ) o r P u b l i c A f f a i r s a n d C o r p o r a t e C o m m u n ic a t io n s O f f i c e ( S P A W A R 0 0 P ) .
“ D e s t ru c t io n N o t ic e ” - F o r u n c la s s i fi e d , l im i te d d is t r ib u tio n d o c u m e n t s , d e s t r o y b y a n y m e t h o d t h a t w i l l p r e v e n t d is c lo s u r e o f c o n t e n t s o r r e c o n s t r u c t io n o f th e d o c u m e n t . ”
F O R C E n e t
V e rs io n 1 .4 3 0 A p r il 2 0 0 4
S ta tem e nt A : A p p ro ve d fo r p u b lic re le a se ; d is tr ib u tio n is u n lim ite d (4 /2 0 /0 4 )
Validation for Netcentric Warfare Simulations
Quantitative Analysis• Experimental Design
– Flex system in statistically interesting ways
– Push simulations hard to identify problem areas
• Non-linear behaviors
– Networks are inherently susceptible to non-linear response
– Must create mathematical means of handling non-linearity
Network in non-linear regime
Mes
sage
Lat
ency
Runtime
Validation for Netcentric Warfare Simulations
Validation Testbed—NETE• Netcentric End-to-End Simulation• Based in Extend discrete event simulation
environment• Modeled NCW Elements:
– GIG BE, TSATs, Link-16– TADIL-J messages– AEGIS cruisers, FBX-T (Sea of Japan)
• Thread– DPRK launches missiles– Tracks formed, passed by GIG to STRATCOM
Validation for Netcentric Warfare Simulations
NETE Features• Realistic message processing• Link-16 message slotting, jitter• Latency computation• Threat tracks based on Lincoln Lab models
Validation for Netcentric Warfare Simulations
Validation for Netcentric Warfare Simulations
• < NETE Demo >
• Metric under test = TADIL-J latency from Link-16 hub in Sea of Japan back to STRATCOM
Validation for Netcentric Warfare Simulations
• 10 Gbits/sec– Very stable at low latency
throughout scenario
• 1.0 Gbits/sec– Latency grows as number
of tracks increases– Highly unstable run to
run
10.0 Gbits/sec
1.0 Gbits/sec
Bandwidth Sensitivity
Wide variationrun to run
Validation for Netcentric Warfare Simulations
0.8 Gbits/sec
0.1 Gbits/sec
• 0.8 Gbits/sec– Latency now grows
throughout run, most of the time
• 0.1 Gbits/sec– Nearly zero througput– Latency grows linearly as
run proceeds
Bandwidth Sensitivity
Validation for Netcentric Warfare Simulations
Quantitative Analysis• Onset of latency …
– rapid, non-linear, brittle, etc.
• Repeatability in this regime is limited– Statistics difficult to quantify
– Validity assessment very difficult with only MOPs, MOEs or a few live exercises as a guide
• Tack:– Quantify statistical behavior of simulations
– Validate where possible
– Assess likely validity breakdowns
Validation for Netcentric Warfare Simulations
Validity Roll-Up—Indexing• Recent NASA V&V work provides a roll-up framework• Index validity on 0 – 5 scale• Assign target validity index to each component• Measured validity vs. target validity generates validity gap
– How to change statistical measures into 0 – 5 scale– How to assign target validity?
012345
Measured
Target
Gap of 2
012345
Gap of 1
012345
Gap of 3
012345
No Gap
Component 1 Component 2 Component 3 Component 4
Validation for Netcentric Warfare Simulations
Validity Roll-Up• Assign risk associated with each component• Roll-up validity as sum of risk-weighted gaps
– May use max rather than sum (weakest link premise)
components
gap)validity ((risk) ii
Component:Validity scoreTarget ValidityRisk Factor
Component:Validity scoreTarget ValidityRisk Factor
Component:Validity scoreTarget ValidityRisk Factor
Component:Validity scoreTarget ValidityRisk Factor
Module Validity Gap:
ii gap)validity ((risk)
components
gap)validity ((risk) ii
maxcomponents
or
Validation for Netcentric Warfare Simulations
InterTEC Joint Fires Exercise
REDRED
BLUEBLUE
RED = small enemy nationBLUE = small friendly nation
RED forces invade Boron-rich area in BLUE lands
Validation for Netcentric Warfare Simulations
Primary Battlespace Objects• Threat
– Integrated Air Defenses with CAP– Concentrations of Ground Forces Stationary and Mobile– Truck Convoy (HVT)
• Joint Force– C2
• CVN• AOC• GCCS-A• AEGIS• E-2C• E-3 AWACS
– Sensor Platforms• EC-135• JSTARS
– Strike Elements• F-16, F-22, F-35• F-18• MLRS• AEGIS
– ECM• EA6-B Prowler
Validation for Netcentric Warfare Simulations
Mission Thread Overview
• Carrier Air Wing and Joint Air Forces Conduct Coordinated Strike to Destroy HVT Truck Convoy
• Targeting Thread:– EW Provides Initial Indicators and Focused Search for JSTARS– JSTARS Provides Continuous Track for Targeting– E-3 and E-2 Coordinate Strike and Targeting– Also Control Strike Aircraft– EA-6 Provides SEAD Support– FA-18, F-16 Conduct Strike– AEGIS and MLRS also Included in Joint Strike– E-2 Provides SAM Location with EA-6/EC-135– Objective is Closely Spaced (in time) Weapon Arrival on Target
• Threat Uses IAMD to Disrupt Strike
Validation for Netcentric Warfare Simulations
Joint Fires Scenario in Extend (so far)
Link-16 models pulled from NETEGround models pulled from ESP (FCS simulation)
Enemyconvoy
Validation for Netcentric Warfare Simulations
Possible Network Topology
Surveillance Network Link-16
Surveillance Network Link-16 JRE
Strike Control Network Link-16
EPLRS
Validation for Netcentric Warfare Simulations
• < Joint Fires Scenario Demo >
Validation for Netcentric Warfare Simulations
• Currently underway at Point Mugu
• Awaiting post-exercise artifacts– Clearance recently granted
• Plan– Examine exercise in detail
• Model closely in Extend• Construct particular validation methodologies
– Compare to live artifacts• Assess validity• Identify weaknesses/gaps in methodologies
– Identify methodology weaknesses/gaps
Status of Exercise
Validation for Netcentric Warfare Simulations
“High-Level” Validation• Is the level of fidelity appropriate?
• Is the constructive setup appropriate?– If network is analyzed and computed ahead of time
• Is line-of-sight fidelity compromised?
• Do latencies change as assets move?
• Do connections change (a la cell tower hand-offs)?
• Red Force– Any networking capabilities modeled here?
– Jamming, other counter-measures on blue comm assets?
– Blue force anti-comm efforts against red?
• Other information axes– Accuracy, relevance (should realistic error rates be played?)
Validation for Netcentric Warfare Simulations
• Robust, quantitative validation needed for NCW simulations
• Missions × Means decomposition helps structure methodology
• Quantitative analysis just beginning– Still must address statistics of non-linear onset, etc.
• Validation roll-up methods provide an overall validity picture– “Recomposition” from M × M decomposition
• Support of Live Joint Fires Exercise just underway
Wrap-Up
Validation for Netcentric Warfare Simulations
Contact:Wesley N. Colley, Ph.D.
Senior Research Scientist, CMSA