Live-Virtual-Constructive Test and Evaluation of Intelligent Sensor Networks

(410) 271-2855 david.scheidt@jhuapl.edu

Intelligent sensor networks differ from traditional sensor networks in several important ways: i.  Intelligent sensor networks use advanced processing, including machine learning, to develop an

understanding of the scene they are observing and the actors within the scene. ii.  Intelligent sensor networks are adaptive, capable of using their scene understanding to

a.  change their focus, following and monitoring information deemed more important, including real-time adaptation of sensor location, pose, and orientation;

b. change the network’s information-processing methods, adapting the frequency, node responsibility, and methods used to extract useful knowledge from data; and

c.  adapt the network’s information distribution strategy, modifying the dissemination structure, frequency, and routing strategy to assure that information is timely, relevant, and does not overload users.

In complex, constantly changing scenes there exists an optimal organizational structure and strategy for exchanging and processing information used to understand the

operating environment.

This plot compares the utility of alterative information exchange topology. Each color represents a dominant area in which unique topological structure used to ingest, process, and disseminate knowledge performed better than all others. The horizontal axis represents variance in the rate of unpredictable change in the operating environment. The vertical axis represents the complexity of the environment being observed. [2]

When testing an intelligent sensor network, it is inadequate to simply measure whether the sensor network did or did not detect or track the target. When measuring an intelligent sensor network, we need to understand, in a battlefield sense, who knew what when, what they did with the information, and how that decision impacted the engagement.

Accurate testing of intelligent sensor networks requires a test infrastructure that accurately models the use of the information provided by the sensor network. To do this, the test infrastructure must incorporate C2 and decision processes of sensor network users and adversaries alike.

At APL, software-in-the-loop testing of intelligent systems uses faster-than-real-time intelligent batch testing of the system-under-test’s intelligent software. APL’s batch testing tool uses a variety of automated analysis tools to select additional tests based on past results. Depending on what we seek to learn from the testing, we can select from half a dozen vignette production algorithms.

In this partial data set, we explore the use of alternate methods to define tests: Monte Carlo methods produce random samples that can be used for statistical analysis; T-wise testing maximizes coverage of the state space; genetic algorithms are effective at identifying boundary conditions and allow us to characterize the conditions in which critical properties do,

or do not, hold; and criticality testing produces test sets that allow us to prove key capabilities will always hold. [3]

References   [1] D. Scheidt, M. Pekala, “The Impact of Entropic Drag on Command and Control,” Proc. of 12th International Command and Control Research and Technology Symposium (ICCRTS), Newport, RI, June 19–21, 2007.

[2] D. Scheidt, K. Schultz, “On Optimizing Command and Control, International Command and Control Research Technology Symposium (ICCRTS),” Quebec City, ON, June 21–23, 2011 (Gary F. Wheatley Best Paper Award Winner).

[3] B. Bauer, D. Scheidt, P. Rosendall, N. Rolander, R. Holder, E. Schmidt, “Evaluating Test Methods for a Complex Connected System,” AAIA Infotech@Aerospace Conference, St. Louis, MO, March 30, 2011.

[4] B. D’Amico, R. Lutz, D. Scheidt, “Testing of Autonomy in Complex, Interactive Environments,” International Test and Evaluation Association Journal, December 2014.

Monte Carlo, criticality, and T-wise testing all examine different portions of the state space. T-wise was the most effective at identifying areas of low fitness (orange); criticality did not identify any areas of low fitness because all low-fitness examples occurred in noncritical conditions.

Cost-effective hardware-in-the-loop testing of intelligent systems requires a live-virtual-constructive test infrastructure that models the information gathering, flow, and use for

friendly and adversarial forces.

TACE’s unique live-virtual-constructive infrastructure models complex cognitive tasks and live-virtual-constructive interactions on the ground over a Test and Training Enabling Architecture (TENA)  bus. Abstract interactions are sent to TACE hardware co-located with the system under test (SUT) over a real-time network where high-fidelity synthetic images are produced from the abstract models and injected into the sensor’s data stream. [4]

Testing of Autonomy in Complex Environments (TACE) is a distributed live-virtual-constructive test infrastructure capable of testing complex interactive engagements in

which both friendly and adversarial forces observe, communicate, and react realistically in complex, interactive engagements.

Johns Hopkins University Applied Physics Laboratory 11100 Johns Hopkins Road, Laurel, MD 20723

This plot shows the amount of uncertainty, measured in Shannon bits, as a function of the amount of sensor observations used to estimate a randomly moving target. The local minima identify the amount of data that should be processed to provide the least uncertain target estimate. Attempting to fuse additional observations causes a net loss of information at the time the information is used. This is because the decay in information produced by target movement since the time of initial observation is greater than the net information gain produced by fusing additional data. [1]

In complex, constantly changing scenes there exists an optimal amount of information that should be used to understand the operating environment.

David Scheidt

TACE has been tested at the Aberdeen Proving Ground using real unmanned air vehicles conducting intelligence, surveillance, and reconnaissance missions tracking a mixture of synthetic and real ground forces. TACE is being transitioned to the Navy’s Atlantic Test Range and will be an operational part of range test equipment in 2017.

Criticality-based testing produces an ordered list of tests starting with the most “critical.” By testing in criticality order, we can define severity and temporal boundaries on system response. In this example, tests with two sets of mission goals identified that no critical error could occur within the system within six seconds.

TENA

Constructive

Virtual

Sensor Network

Live Live

Live

This work was funded by TRMC under contract W900KK-13-C-0036-P00006 and ONR under contract N00013-01-0155.