PROCEEDINGSof theHUMANFACTORSANDERGONOMICSSOCIETY45thANNUALMEETING-2001

EFFECT OF DISPLAY DESIGN AND SITUATION COMPLEXITYON OPERATOR PERFORMANCE

Robert A. WillisDepartment of Systems and InformationEngineering

University of Virginia

The U.S. Navy is currently developing its next-generation cruise missile, the Tactical Tomahawk,which improves upon current versions by its ability to be retargetedin flight against emergenttime-critical targets. In this study, we developed an advanced operator interfaceprototype formonitoring, controlling and retargeting the Tactical Tomahawk missile and empirically tested theeffect of mission complexity on the ability of shipboard operators to maintain situationalawareness in various operationalscenarios. The first phase of research involved a domainanalysis of three primary domains: the weapon system; time-critical decisionmaking; andprinciples of interface design. The second phase was a concurrent Cognitive Work/TaskAnalysis(CW/TA) and scenario development effort. The third phase was interface component designfollowed by complete prototype implementation. In the final phase, we trainedand tested twentygraduate students on the dynamic and interactive prototype, based on hypotheses pertaining toboth monitoring and retargeting tasks. Statistical results support two primary conclusions. First,operators can maintain adequatesituational awareness when monitoring eight missiles and twelvetargets simultaneously. Second, results support the use of the missile timebar feature in theinterface to compare events. Subjective results indicate the requirement for a robust decisionsupport tool to facilitate rapid retargeting decisions. These and other results form the basis forrecommendations to the Naval SurfaceWarfare Center, Dahlgren Division (NSWCDD) abouthow to most effectively allocate personnel resources in the design of a command and controlwatchstation for the Tactical Tomahawk cruise missile system.

INTRODUCTION operational capability in 2003. TTWCS requirements(Operational Requirements Document, 1997) include the

The Tomahawk cruise missile system has given our following:military forces a formidable tool in their arsenal for use

c3 Enable rapid tactical mission planning on boardagainst many types of enemy targets. Previously, targetsdeep in enemy territory would have to be attacked by the launching vessel.manned aircraft or special operations personnel. With c3 Enable the missile to 'loiter' near the defaultthe addition of the Tomahawk missile system in the target area to decrease the time required to1980' s, these targets can be attacked with precision from attack preplanned alternate (flex) targets as wellsafe distance (up to a thousand miles) thereby reducing as newly identified emergent targets.the threat to friendly servicemen. The capabilities andadvantages of this system were greatly exploited in [] Enable in-flight reprogramming of the missileOperation Desert Storm (1991), during the strikes (to change route and/or target).against Afghanistan and Sudan in 1998, and in The challenges introduced with the TTWCS centerOperation Allied Force in Serbia (1999). primarily around the operator's ability to maintain

The U.S. Navy' s current cruise missile is launched situational awareness when multiple missiles are beingfrom either a ship or a submarine, and flies a controlled, and to rapidly make retargeting decisions byprogrammed route to a predetermined target. Target and evaluating specified criteria. Some of the criteria, suchroute planning is a complex process conducted in Hawaii as the opportunity cost of striking a flex target over aand Virginia, with some limited capability on aircraft default target, are somewhat subjective. Others, such ascarriers (Cruise Missile Support Activity, 1997). Target missile fuel remaining, coverage area, and target priorityand route data is downloaded to the missile on the vessel can be quantitatively compared, suggesting a decisionprior to launch, and the ATWCS operator cannot modify process that is to some degree automated. Our goal wasor abort a mission once the missile is launched, to develop a set of operator displays for this domain,Because of the huge success of early and current help define the desired level of automation, determineTomahawk versions, the Navy is developing the Tactical the number of missiles one operator can manage, andTomahawk Weapon Control System (TTWCS) for initial test the effectiveness of those displays on operator

346

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001

performance in multiple operating scenarios, for this project is that a targeting center issues the attackorder which specifies the "what, where, when", while the

APPROACH TTWCS operator and/or system decides "how" (withwhat missile(s)). Furthermore, there is assumed to be

The research was conducted in four phases (Figure 1). two-way communication to facilitate feedback to theThe first phase was a Domain Analysis of three primary targeting center regarding any preplanned targets thatdomains: the TTWCS system properties; the larger were given up as an opportunity cost for servicing ancontext of air campaign command and control; and unplanned emergent target. Finally, the system mustgeneric interface design principles. Second, potential address the likelihood of the targeting cell "asking" if ascenarios were developed to identify functional vessel could accomplish a certain mission. The displaysrequirements. Next, interface components were must therefore facilitate rapid analysis of "what-if'developed and iteratively synthesized into increasingly scenarios to determine the acceptance/rejection of arobust prototypes. Finally, Subject Testing and Analysis possible time-critical mission.was conducted using the interactive, dynamic prototype.

iiM!!_ii_iiii{iiii!iii_iiii_i_iiii_!i_iii_!iii!i!_i!i!!!i!_i!iii_!ii_i_iiiii_i_i_!iii_iiiiiiii_ii_iiiiiiiiiii_i_iiiii_i_i_!!iiiii!iiiiii_J_i!!_i_!ii!_!_!ii!i_i_i_!!}_!!i_ii!_iii_{ii_iiiii_iiiii_ii_ii!iii_

Figure I Phases of Research

Domain Analysis

Weapon system and environment. The schematic in Figure 2 TacticalTomahawkMissionsFigure 2 depicts the three basic types of Tactical (revised from LoPresto and Rice, 2000)Tomahawk missions (the figure is not a proposedoperator display). The class of target to which the Products. Our initial products from the Domainmissile is assigned defines the missions. Target classes Analysis phase included a list of seven object types that(and thus mission types) include default, flex, and are inherent in the weapon system's environment. Theseemergent (or d-, f-, and e-targets). A missile is launched are Targets, Missiles, Airspace Constraints/Threats,along a path initially toward its d-target. While enroute, Vessels, Missions, Campaigns, and BattleGroups.the operator can command the missile to abort its default Properties for the "Target" and "Missile" objects aremission and execute one of several preprogrammed flex depicted in Figure 3.missions. The operator can also respond to the

Scenario Developmentappearance of an e-target by rapidly programming and

transmitting a completely new route and aimpoint. Continuing the decomposition of the system intoTime-critical targeting. As a retargetable system, products, a set of seven scenarios was next developed to

TTWCS brings the cruise missile into the realm of on- facilitate the identification of operator informationcall weapons available to an air campaign or ground requirements and interface functional requirements. Acommander for prosecution of time-critical targets sample scenario was:(Scott, 2000). Time-critical targets are those enemy "Three inflight missiles are along separatetargets identified in the field that are likely to move, and routes prior to the branch points for a commontherefore need to be serviced as quickly as possible, e.g., flex target. The operator receives the commanda mobile surface-to-air missile launcher that has been to service the flex target as well as a newpicked up via satellite imagery data. The Department of emergent target that is within range of twoDefense has established a goal of ten minutes for the missiles. Priorities of all targets are known, andservicing of time-critical targets (Scott, 2000). the expected move time of the e-target is given.

Consequence in interface design. TTWCS display Select the best course of action."development must consider from whom an attack orderis received, and in what format. A central assumption

347

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001

Other scenarios included interaction with no-fly described scenario requires an interpretation of thezones, targets requiring multiple missiles, multiple missiles with respect to time, and requires the operator toemergent targets, and targets closer to a launching vessel rapidly evaluate the problem, reduce and comparethananinflightmissile, options,andmakea decision. ThemissileTimebar

component prototype shown in Figure 5 maps object......_..............................................................................................................................................._......................_: ?iiiiiiiiiiii_i_iiii_iii_iiiiiii_iii_i_iiii_i_iiiii!_iiiii_iii_iiiiiiii_iiiiii_ii_ii_iiiiiiii_iii_!_iii:_iiiiii......: _i!i_ii___i_!#!i_!_%!_!i properties into a graphical feature for a missile assigned

'_i_'_,_,_,_,_,_,_,_,_,i!'iii'ii!'_i_ii'_ii!'_iii'_ii?_ii _, to an emergent target. This missile-time componentwouldbemultipliedforeachcandidatemissile

'_ ......_ _,_ i _i_i_i_i___ii_i ..................................................................................................................................................................................................................................................................................................................................................................................................,..................................i

i_iiiii _ _ii_i!_ iiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii______:___:_:_

iiiiiiiiiiiiiiii_i_iiii_iiiiii i _iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

___S_i_i_ _": iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiiii_ii_iiiiiiiiiiiiiiiiiiiiiiii_iiiiii iiiiiiiiiiiiii_iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii::

::::::: :::::::::::::::::::::::::::::

Figure3 Propertiesfortheiii_!_!_iii_i!i_J;_ii_i_iiiiiiiiiiiiiiiiii_:iiii_iiMissile and Target Objects Figure 5 Timebar Component Prototype

Other interface components include missile routesInterface Design and Prototype Development (actual route representations or merely assignment

Following the Domain Analysis, component and indicators between missiles and targets), coverage zonessystem prototypes were iteratively synthesized as (a circle shown around each missile that represents thedepicted in Figure 4. areathat that the missile can reach in a certain amountof

time), airspace control measures (no-fly zones), andthreats.

System prototype. The synthesis of componentswith major tasks such as that described in the retargetingtask flowchart product lead to a system prototype similarto the one shown in Figure 6. Static, non-interactivePowerPoint slides were used for cognitive walkthroughpurposes (Kellmeyer & Osga, 2000). The iterativeprototyping process progressed through scripted"movies", and into static yet interactive slides todemonstrate basic system display functions.

A design document was submitted to severalprofessional multimedia producers, and CreativePerspectives (Charlottesville, VA) was selected to code adynamic and interactive "Version 3" prototype suitablefor subject testing in the near-term, as well as a "Version4" prototype incorporating subject testing results.

Figure 4 Prototype Development Macromedia Director 8 was selected as the prototypingsoftware.

Components. The most basic types of components The prototype architecture enables the programmingare the symbols used to represent the individual missiles of new scenarios (number of missiles & targets,and targets. To the greatest extent possible, the symbol waypoints, messages, etc.), and continuously updates alldevelopment is in accordance with Department of the geometric expressions inherent in the system, withDefense common interface symbology (Mil Std-2525B, the exception of missile altitude and dive angle. The1999). prototype automatically reduces the solution space to

Requirements for subsequent components were courses of action ("candidate missiles") duringmade clear in the analysis of other scenarios and theirfunctional requirements. For example, the previously

348

PROCEEDINGSof theHUMANFACTORSANDERGONOMICSSOCIETY45thANNUALMEETING-2001

retargeting scenarios (this is Level 3 automation as Design of Experimentdefined by Endsley and Kiris, 1995).

The pop-up window shown in Figure 6 is a decision Three experimental variables were controlled using asupport tool that facilitates course of action comparison, split-plot experimental design. Each test run consistedThis Level 2 automated system presents processed of a monitor scenario followed by a retarget scenario.information and allows the user to investigate an For monitoring tasks, we varied the number of "missilesautomated recommendation using the triangular tool to plus targets" objects in the system (10 vs. 20 objects), asrapidly explore sensitivities to different criteria well as the display type, which either requires constantweighting schemes. As the user changes the weighted viewing of the coverage zone (CZ) features withcriteria, either directly or by moving the dot around the selectable time factors (radii), or doesn't permit thetriangle, the recommended action is displayed for each viewing of coverage zones at all. For retargeting tasks,of the criteria and for the composite. This decision display type was similarly varied, as well as the overallsupport tool was designed after subject testing revealed a number of candidate missiles for an emergent targetneed for improved situation assessment (Endsley & occurrence (2 vs. 4 candidates).Kiris, 1995) during retargeting tasks.

Participants

Time and resources prevented us from using Navypersonnel as subjects. However, this task is for aweapon that is in development so no true "experienced"operators exist. Twenty engineering graduate studentsserved as subjects, and attended 2 ½ hours of trainingand certification testing. Immediately prior to theexperiment, subjects attended an individual review toensure their understanding of the system and userinterface.

Scenario Design and Description

Each test scenario included an eight-minutemonitoring scene followed by a five-minute retargetingscene. The message section of the interface was used to

Figure6 Emergent TargetingEvent (Prototype Version4) issue both inherent system messages as well asadministrative testing questions to the subjects. Inherent

Hypotheses messages included changes in missile health status,increasing priority of flex targets, appearance of

The testing scenarios were storyboarded based on emergent targets, and orders to service flex or emergenthypotheses pertaining to two major types of targets. All administrative testing questions (shown inoperator/system tasks: monitoring and retargeting. It Table 1) supported at least one hypothesis. Questions 1-was expected that, during monitoring tasks: 7 were administered during the monitoring scenario and

questions 8-11 were administered during the retargetingca Operators' overall situational awareness (asmeasured by time and accuracy) will decrease as scenario.the number of total icons (missiles plus targets) Questions were measured by response time andincreases, accuracy. Questions 7, 10 and 11 were also rated by two

o Operators using the "coverage zone" feature will observers on a subjective situational awareness scale (1-show poorer performance and accuracy due to 7), as these questions required the subject to verballythe added screen clutter than those operators not compare courses of action.usingthefeature,exceptduringtasksthat RESULTS

specifically require estimating coverage. Table 1 shows the variables affected by responseFurther, when performing retargeting tasks, time. The "number of objects" variable (10 vs. 20

ca An operator's overall situational awareness will missiles plus targets) was significant to response timedecrease as the number of candidate missiles (RT) for questions 4-6, 8, 10 and 11. This findingincreases, suggestsa limitonthenumberof objectsoneoperator

can handle. A post-test survey found that 20 objects (8

349

PROCEEDINGSof theHUMANFACTORSANDERGONOMICSSOCIETY45thANNUALMEETING-2001

missiles, 12 targets) was the most subjects felt they of action (COAs) during retargeting tasks. Three orcould handle. An interesting finding was that the more COAs required an inordinate amount of mental"number of objects" variable was insignificant during processing time, and short term memory limitationstasks that required the operator to compare time became significant. Thus, we subsequently developedattributes of multiple missiles (questions 1-3 and 9). In (but have not tested) the COA decision support tool (theother words, an increase in the number of missiles did pop-up window in Figure 6).not significantly affect response time (or accuracy). Thisfindingsupports the use of the timebar component for CONCLUSIONS

timecomparisons. Thisprojectaimedto identifythecommandand

Table 1. Reactiontime results control decision factors that will affect future navalofficers when equipped with retargetable long-range_]_Q_:[I___J_:'_____________________[8[_[fi______missiles. By following a systematic cognitive

........................!..............................................................................................................................................................111__ i __ ....1 engineering design and testing process, we haveI Which missile will reach ifS ...... N0ne ....................Nsdefaulttargetlast if allmissiles designed and tested two of three decision support toolsarecommandedto godirectlyto (timebars, coverage zones and a multi-criteria weightingtheirdefaulttarget? tool) integratedintoanoverallworkspaceto support

2 Which missile(s) will impact None NS operators. Further, we have begun to classify thebefore12:30? scenariosandlevelof automationforthis systemandto

3 Point to the timebar associated None NS quantify the number of missiles one operator canwiththe southern-mostmissile effectively handle. These results are likely transferable

4 Howmany targetsrequireor Objects .0023 at some level to uninhabited vehicles being developedprefermorethanonemissile? for reconnaissance and intelligencein both the air and

5 Which soft warhead missilesare Objects .008 underwater, and help define how to support operators inhittingtargetswithapriorityof managing dynamic spatial information.6 or more?

6 Howmanymissilesareyou Objects .0006 ACKNOWLEDGEMENTSmonitoring right now?

7 Describe/showthe general areas Coverage .0377 This work was sponsored by the Naval Surfacein which you could NOT service Zonesan e-targetthatis expectedto Warfare Center, DahlgrenDivision (NSWCDD),move12minutesfromnow. StephanieGuerlain, Principal Investigator.

8 Point to the candidate missiles Candidates .0001that, if retargeted, will sacrifice REFERENCESboth a default and a flex target.

9 Which missile will reach the e- None NS Cruise Missile Support Activity, 1997.RetrievedNovember

target first? 27, 2000 from:www.fas.org/irp/agency/dod/uspacom/cmsa/1 With respect to targetpriority, Candidates .0001 Endsley, M., and Kiris, E. (1995).The out-of-the-loop0 timeto getthere,andfuel performanceproblemandlevelofcontrolin automation.

remaining,verballycompare the Human Factors, 37(2), 381-394.candidates.Istherea Kellmeyer,D.andOsga,G.A.(2000).Usabilitytesting&dominatingsolution? analysisofadvancedmultimodalwatchstationfunctions.

1 If the system were designed to Candidates .0001 Proceedings of the Human Factors and Ergonomics Society1 automatically select the best 44thAnnual Meeting, SanDiego, California.

missilefor ane-target,whydo LoPresto,M. andRice,K. (2000). TomahawkStrikeNetworkyou think itwouldselectMissile [Powerpointpresentation].JohnsHopkinsUniversity(010,031,32,18)inthiscase? AppliedPhysicsLab.

Operational Requirements Document (ORD) For Tomahawk

Only question 7 was significantly affected by the Weapon SystemBaseline IV (U) (Revision 2), January,1998.presence of the coverage zones. Subjects who had Scott, W. (2000, October 2). Experimental center nails time-access to the coverage zone feature were significantly critical targets. Aviation Week and Space Technology, p.70-faster and were rated as havingbetter situation 72.awareness (accuracy) in their descriptive response(p<.O001).

Not surprisingly, the "number of candidates"variable was significant (p<.O001) in comparing courses

350