The DEFACTO System:Training Tool for Incident Commanders ∗

Nathan Schurr, Janusz Marecki,J.P. Lewis and Milind Tambe

University of Southern California, PHE 5143737 Watt Way, Los Angeles, CA 90089-0781

{schurr, marecki, tambe}@usc.edu, zilla@computer.orgphone: 213.740.9560 fax: 213.740.7877

Paul ScerriCarnegie Mellon University,

5000 Forbes Avenue, Pittsburgh, PA 15213pscerri@cs.cmu.edu

phone: 412.268.2145 fax: 412.268.6436

Abstract

Techniques for augmenting the automation of routine coordi-nation are rapidly reaching a level of effectiveness where theycan simulate realistic coordination on the ground for largenumbers of emergency response entities (e.g. fire engines,police cars) for the sake of training. Furthermore, it seemsinevitable that future disaster response systems will utilizesuch technology. We have constructed a new system, DE-FACTO (Demonstrating Effective Flexible Agent Coordina-tion of Teams through Omnipresence), that integrates state-of-the-art agent reasoning capabilities and 3D visualizationinto a unique high fidelity system for training incident com-manders. The DEFACTO system achieves this goal via threemain components: (i) Omnipresent Viewer - intuitive inter-face, (ii) Proxy Framework - for team coordination, and (iii)Flexible Interaction - between the incident commander andthe team. We have performed detailed preliminary experi-ments with DEFACTO in the fire-fighting domain. In addi-tion, DEFACTO has been repeatedly demonstrated to key po-lice and fire department personnel in Los Angeles area, withvery positive feedback.

IntroductionIn the wake of large-scale national and international terroristincidents, it is critical to provide first responders and res-cue personnel with tools and techniques that will enablethem to evaluate response readiness and tactics, measureinter-agency coordination and improve training and decisionmaking capability. We focus in particular on building toolsfor training and tactics evaluation for incident commanders,who are in charge of managing teams of fire fighters at criti-cal incidents. Such tools would provide intelligent softwareagents that simulate first responder tactics, decisions, andbehaviors in simulated urban areas and allow the incidentcommander (human) to interact. These agents form teams,where each agent simulates a fire engine, which plans andacts autonomously in a simulated environment. Through in-teractions with these software agents, an incident comman-

∗This research was supported by the United States Departmentof Homeland Security through the Center for Risk and EconomicAnalysis of Terrorism Events (CREATE). However, any opinions,findings, and conclusions or recommendations in this document arethose of the author and do not necessarily reflect views of the U.S.Department of Homeland Security.

der can evaluate tactics and realize the consequences of keydecisions, while responding to such disasters.

To this end, we have constructed a new system, DE-FACTO (Demonstrating Effective Flexible Agent Coordina-tion of Teams through Omnipresence). DEFACTO incorpo-rates state of the art artificial intelligence, 3D visualizationand human-interaction reasoning into a unique high fidelitysystem for training incident commanders. By providing theincident commander interaction with the coordinating agentteam in a complex environment, the commander can gainexperience and draw valuable lessons that will be applicablein the real world. The DEFACTO system achieves this viathree main components: (i) Omnipresent Viewer - intuitiveinterface, (ii) Proxy Framework - for team coordination, and(iii) Flexible Interaction - between the incident commanderand the team.

Omnipresent Viewer:As sensor capabilities and networksimprove, command and control centers will have access tovery large amounts of up-to-date information about the en-vironment. A key open question is how that information canbe used to allow most effective incident command decision-making. We are exploring one promising option that lever-ages advances in 3D visualization and virtual reality: aviewer for virtual omnipresence.

Proxy Framework:Techniques for automation of routinecoordination are rapidly reaching a level of effectiveness andmaturity where they can be applied to incident commandertraining simulations. To examine the impact of such tech-nology, we have included state-of-the-art autonomous coor-dination into the heart of the DEFACTO system. This allowsother aspects of the system to be developed with a realisticrepresentation of how the coordination will proceed. Con-versely, it can assist developers of coordination technologyto identify the key problems for real-world deployment oftheir technology.

Flexible Interaction: Previous work has shown thatin a multiagent context, flexible interactions between au-tonomous agents and humans are critical to the success ofthe system. While intuitive human interfaces and intelli-gent team coordination are important, critical failures willoccur unless human-agent interaction is appropriately man-aged. Hence, we are developing new, flexible techniques formanaging this interaction. These techniques build on theo-ries of teamwork to manage the distributed response (Scerri,

IAAI-05 / 1555

Pynadath, & Tambe 2002).Training incident commanders provides a dynamic sce-

nario in which decisions must be made correctly and quicklybecause human safety is at risk. However, it also holds po-tential for agents to take on responsibility in order to free thehumans involved from being unnecessarily overburdened ina time of crisis. When using DEFACTO, incident comman-ders have the opportunity to see the disaster and the coordi-nation/resource constraints unfold so that they can be betterprepared when commanding over an actual disaster. Apply-ing DEFACTO to disaster response holds a great potentialbenefit to the training of incident commanders in the firedepartment. DEFACTO has been repeatedly demonstratedto key police and fire department personnel in Los Angelesarea with very positive feedback.

In addition, we have performed detailed preliminary ex-periments with DEFACTO being used as a training tool inthe firefighting domain. We present results from these ex-periments that reveal results pointing to the value of usingsuch tools. First, contrary to previous results (Scerriet al.2003), human involvement is not always beneficial to anagent team; despite their best efforts, humans may some-times end up hurting an agent team’s performance. Second,increasing the number of agents in an agent-human teammay degrade the team performance, even though increasingthe number of agents in a pure agent team under identicalcircumstances improves team performance. These resultsillustrate that DEFACTO can create a challenging environ-ment for incident commanders, and that there is a need forthe flexible interaction strategies between teams and inci-dent commanders incorporated into DEFACTO (which do-main experts verify is a requirement in this domain (LAFD2001)).

With DEFACTO, our objective is to both enable thehuman to have a clear idea of the team’s state and im-prove agent-human team performance. We want DEFACTOagent-human teams to better prepare firefighters for currenthuman-only teams and better prepare them for the future, inwhich agent-human teams will be inevitable and invaluable.We believe that leveraging the above aspects DEFACTO willresult in better disaster response methods and better incidentcommanders.

System OverviewFirst, DEFACTO incorporates a visualizer that allows for thehuman to have anomnipresentinteraction with remote agentteams. We refer to this as the Omni-Viewer (see Figure 1),and it combines two modes of operation. The NavigationMode allows for a navigable, high quality 3D visualizationof the world, whereas the Allocation Mode provides a tra-ditional 2D view and a list of possible task allocations thatthe human may perform. Human experts can quickly absorbon-going agent and world activity, taking advantage of boththe brain’s favored visual object processing skills (relativeto textual search, (Paivio 1974)), and the fact that 3D repre-sentations can be innately recognizable, without the layer ofinterpretation required of map-like displays or raw computerlogs. The Navigation mode enables the human to understandthe local perspectives of each agent in conjunction with the

Disaster Scenario

Proxy

Proxy

Proxy

Proxy

Team

Omni-Viewer

DEFACTO

Navigation Allocation

Incident Commander

Figure 1: DEFACTO system architecture.

global, system-wide perspective that is obtained in the Allo-cation mode.

Second, DEFACTO utilizes a proxy framework that han-dles both the coordination and communication for thehuman-agent team (see Figure 1). We use coordination al-gorithms inspired by theories of teamwork to manage thedistributed response (Scerri, Pynadath, & Tambe 2002). Thegeneral coordination algorithms are encapsulated inproxies,with each team member having its own proxy and represent-ing it in the team.

Third, we also use the above mentioned proxy frame-work to implement adjustable autonomy (AA) between eachmember of the human-agent team. The novel aspect of DE-FACTO’s flexible AA is that we generalize the notion ofstrategiesfrom single-agent single-human context to agent-team single-human. In our work, agents may flexibly chooseamong team strategies for adjustable autonomy instead ofonly individual strategies; thus, depending on the situation,the agent team has the flexibility to limit human interaction,and may in extreme cases exclude humans from the loop.

Domain

The DEFACTO system is currently focused on illustratingthe potential of future disaster-response to disasters that mayarise as a result of large-scale terrorist attacks. Constructedas part of the effort at the first center for research excellenceon homeland security (the CREATE center), DEFACTO ismotivated by a scenario of great concern to first responderswithin Los Angeles and other metropolitan areas. In ourconsultations with the Los Angeles fire department and per-sonnel from the CREATE center, this scenario is of greatconcern. In particular, a shoulder-fired missile could poten-tially be used to attack a low-flying civilian jet-liner that ispreparing to land at Los Angeles International Airport caus-ing the jet-liner to crash into an urban area and resulting in alarge-scale disaster on the ground. This scenario could leadto multiple fires in multiple locations with potentially manycritically injured civilians. While there are many longer-term implications of such an attack, we focus on assistingfirst responders, namely fire fighters.

IAAI-05 / 1556

(a) (b)

Figure 2: Current Training Methods: (a) projected photo and(b) incident commanders at a table

We had the opportunity to study the current methods thatthe Los Angeles Fire Department (LAFD) implements totrain incident commanders. The LAFD uses a projectionscreen to simulate the disaster (Figure 2-(a)). In addition,the participating incident commander is seated at a desk,directing an assistant to take notes (Figure 2-(b)). Otherfirefighters remain in the back of the room and communi-cate to the incident commander via radios. Firefighters aretaken temporarily out of duty in order to help act out thesepre-determined scenarios in order to test the incident com-mander’s abilities. DEFACTO aims to automate this processand enable higher fidelity and larger scale training scenari-ons which require less personnel.

Underlying TechnologiesIn this section, we will describe the technologies used inthree major components of DEFACTO: the Omni-Viewer,proxy-based team coordination, and proxy-based adjustableautonomy. The Omni-Viewer is an advanced human inter-face for interacting with an agent-assisted response effort.The Omni-Viewer provides for both global and local viewsof an unfolding situation, allowing a human decision-makerto obtain precisely the information required for a particulardecision. A team of completely distributed proxies, whereeach proxy encapsulates advanced coordination reasoningbased on the theory of teamwork, controls and coordinatesagents in a simulated environment. The use of the proxy-based team brings realistic coordination complexity to theprototype and allows more realistic assessment of the inter-actions between humans and agent-assisted response. Thesesame proxies also enable us to implement the adjustable au-tonomy necessary to balance the decisions of the agents andhuman.

We have applied DEFACTO to a disaster rescue domain,where the incident commander of the disaster acts as thehu-man userof DEFACTO. We focus on two urban areas: asquare block that is densely covered with buildings (we useone from Kobe, Japan) and the University of Southern Cali-fornia (USC) campus, which is more sparsely covered withbuildings. In our scenario, several buildings are initially onfire, and these fires spread to adjacent buildings if they arenot quickly contained. The goal is to have a human inter-

act with the team of fire engines in order to save the mostbuildings. Our overall system architecture applied to disas-ter response can be seen in Figure 1.

Omni-Viewer

Our goal of allowing fluid human interaction with agents re-quires a visualization system that provides the human witha global view of agent activity as well as shows the localview of a particular agent when needed. Hence, we havedeveloped an omnipresent viewer, or Omni-Viewer, whichwill allow the human user diverse interaction with remoteagent teams. While a global view is obtainable from a two-dimensional map, a local perspective is best obtained froma 3D viewer, since the 3D view incorporates the perspectiveand occlusion effects generated by a particular viewpoint.The literature on 2D- versus 3D-viewers is ambiguous. Forexample, spatial learning of environments from virtual nav-igation has been found to be impaired relative to studyingsimple maps of the same environments (Richardson, Mon-tello, & Hegarty 1999). On the other hand, the problem maybe that many virtual environments are relatively bland andfeatureless. Ruddle points out that navigating virtual en-vironments can be successful if rich, distinguishable land-marks are present (Ruddle, Payne, & Jones 1997).

To address our discrepant goals, the Omni-Viewer incor-porates both a conventional map-like 2D view, AllocationMode (Figure 3-d), and a detailed 3D viewer, NavigationMode (Figure 3-b). The Allocation mode shows the globaloverview as events are progressing and provides a list oftasks that the agents have transfered to the human. TheNavigation mode shows the same dynamic world view, butallows for more freedom to move to desired locations andviews. In particular, the user can drop to the virtual groundlevel, thereby obtaining the world view (local perspective) ofa particular agent. At this level, the user can “walk” freelyaround the scene, observing the local logistics involved asvarious entities are performing their duties. This can behelpful in evaluating the physical ground circumstances andaltering the team’s behavior accordingly. It also allows theuser to feel immersed in the scene where various factors(psychological, etc.) may come into effect.

In order to prevent communication bandwidth issues, weassume that a high resolution 3D model has already beencreated and the only data that is transfered during the dis-aster are important changes to the world. Generating thissuitable 3D model environment for the Navigation modecan require months or even years of manual modeling ef-fort, as is commonly seen in the development of commercialvideo-games. However, to avoid this level of effort we makeuse of the work of You et. al. (Suya You & Fox 2003) inrapid, minimally assisted construction of polygonal modelsfrom LiDAR (Light Detection and Ranging) data. Given theraw LiDAR point data, we can automatically segment build-ings from ground and create the high resolution model thatthe Navigation mode utilizes. The construction of the USCcampus and surrounding area required only two days usingthis approach. LiDAR is an effective way for any new geo-graphic area to be easily inserted into the Omni-Viewer.

IAAI-05 / 1557

(a) (b) Navigation Viewer (c)

(d) Allocation Viewer (e) (f)

Figure 3: Omni-Viewer during a scenario: (a) Multiple fires start across the campus, (b) The Incident Commander uses theNavigation mode to quickly grasp the situation, (c) Navigation mode shows a closer look at one of the fires, (d) Allocationmode is used to assign a fire engine to the fire, (e) The fire engine has arrived at the fire, (f) The fire has been extinguished.

Proxy: Team CoordinationA key hypothesis in this work is that intelligent distributedagents will be a key element of a future disaster response.Taking advantage of emerging robust, high bandwidth com-munication infrastructure, we believe that a critical role ofthese intelligent agents will be to manage coordination be-tween all members of the response team. Specifically, we areusing coordination algorithms inspired by theories of team-work to manage the distributed response (Scerri, Pynadath,& Tambe 2002). The general coordination algorithms areencapsulated inproxies, with each team member having itsown proxy and representing it in the team. The current ver-sion of the proxies is calledMachinetta(Scerriet al. 2003)and extends the successful Teamcore proxies (Pynadath &Tambe 2003). Machinetta is implemented in Java and isfreely available on the web. Notice that the concept of areusable proxy differs from many other “multiagent toolk-its” in that it provides the coordinationalgorithms, e.g., al-gorithms for allocating tasks, as opposed to theinfrastruc-ture, e.g., APIs for reliable communication.

Communication: communication with other proxiesCoordination: reasoning about team plans and communi-

cationState: the working memory of the proxyAdjustable Autonomy: reasoning about whether to act au-

tonomously or pass control to the team memberRAP Interface: communication with the team member

Figure 4: Proxy Architecture

The Machinetta software consists of five main modules,three of which are domain independent and two of which aretailored for specific domains. The three domain independentmodules are for coordination reasoning, maintaining localbeliefs (state) and adjustable autonomy. The domain spe-cific modules are for communication between proxies andcommunication between a proxy and a team member. Themodules interact with each other only via the local state witha blackboard design and are designed to be “plug and play.”Thus new adjustable autonomy algorithms can be used withexisting coordination algorithms. The coordination reason-ing is responsible for reasoning about interactions with otherproxies, thereby implementing the coordination algorithms.The adjustable autonomy algorithms reason about the inter-action with the team member, providing the possibility forthe team member to make any coordination decision insteadof the proxy.

IAAI-05 / 1558

Teams of proxies implementteam oriented plans(TOPs)which describe joint activities to be performed in terms ofthe individualroles to be performed and any constraints be-tween those roles. Generally, TOPs are instantiated dynam-ically from TOP templates at runtime when preconditionsassociated with the templates are filled. Typically, a largeteam will be simultaneously executing many TOPs. For ex-ample, a disaster response team might be executing multiplefight fire TOPs. Such fight fire TOPs might specify a break-down of fighting a fire into activities such as checking forcivilians, ensuring power and gas is turned off, and sprayingwater. Constraints between these roles will specify interac-tions such as required execution ordering and whether onerole can be performed if another is not currently being per-formed. Notice that TOPs do not specify the coordination orcommunication required to execute a plan; the proxy deter-mines the coordination that should be performed.

Current versions of Machinetta include a token-based roleallocation algorithm. The decision for the agent becomeswhether to assign values represented by tokens it currentlyhas to its variable or to pass the tokens on. First, theteam member can choose the minimum capability the agentshould have in order to assign the value. This minimum ca-pability is referred to as thethreshold. The threshold is cal-culated once (Algorithm 1, line 6), and attached to the tokenas it moves around the team.

Second, the agent must check whether the value can beassigned while respecting its local resource constraints (Al-gorithm 1, line 9). If the value cannot be assigned within theresource constraints of the team member, it must choose avalue(s) to reject and pass on to other teammates in the formof a token(s) (Algorithm 1, line 12). The agent keeps valuesthat maximize the use of its capabilities (performed in theMAX CAP function, Algorithm 1, line 10).

Algorithm 1 TOKENMONITOR(Cap,Resources)1: V ← ∅2: while truedo3: msg ← getMsg()4: token← msg5: if token.threshold = NULL then6: token.threshold← COMPUTETHRESHOLD(token)7: if token.threshold ≤ Cap(token.value) then8: V ← V ∪ token.value9: if

Pv∈V Resources(v) ≥ agent.resources then

10: out← V− MAX CAP(V alues)11: for all v ∈ out do12: PASSON(newtoken(v))13: V alues← V alues− out14: else15: PASSON(token) /* threshold > Cap(token.value) */

Proxy: Adjustable AutonomyIn this paper, we focus on a key aspect of the proxy-basedcoordination: Adjustable Autonomy. Adjustable autonomyrefers to an agent’s ability to dynamically change its ownautonomy, possibly to transfer control over a decision to ahuman. Previous work on adjustable autonomy could be

categorized as either involving a single person interactingwith a single agent (the agent itself may interact with oth-ers) or a single person directly interacting with a team. Inthe single-agent single-human category, the concept of flex-ible transfer-of-control strategy has shown promise (Scerri,Pynadath, & Tambe 2002). A transfer-of-control strategy isa preplanned sequence of actions to transfer control over adecision among multiple entities. For example, anAH1H2strategy implies that an agent (A) attempts a decision and ifthe agent fails in the decision then the control over the deci-sion is passed to a humanH1, and then ifH1 cannot reacha decision, then the control is passed toH2. Since previ-ous work focused on single-agent single-human interaction,strategies were individual agent strategies where only a sin-gle agent acted at a time.

An optimal transfer-of-control strategy optimally bal-ances the risks of not getting a high quality decision againstthe risk of costs incurred due to a delay in getting that deci-sion. Flexibility in such strategies implies that an agent dy-namically chooses the one that is optimal, based on the situ-ation, among multiple such strategies (H1A, AH1, AH1A,etc.) rather than always rigidly choosing one strategy. Thenotion of flexible strategies, however, has not been applied inthe context of humans interacting with agent-teams. Thus,a key question is whether such flexible transfer of controlstrategies are relevant in agent-teams, particularly in a large-scale application such as ours.

DEFACTO aims to answer this question by implementingtransfer-of-control strategies in the context of agent teams.One key advance in DEFACTO is that the strategies are notlimited to individual agent strategies, but also enables team-level strategies. For example, rather than transferring controlfrom a human to a single agent, a team-level strategy couldtransfer control from a human to an agent-team. Concretely,each proxy is provided with all strategy options; the key isto select the right strategy given the situation. An exampleof a team level strategy would combineAT Strategy andHStrategy in order to makeAT H Strategy. The default teamstrategy,AT , keeps control over a decision with the agentteam for the entire duration of the decision. TheH strategyalways immediately transfers control to the human.AT Hstrategy is the conjunction of team levelAT strategy withHstrategy. This strategy aims to significantly reduce the bur-den on the user by allowing the decision to first pass throughall agents before finally going to the user, if the agent teamfails to reach a decision.

Experiments and EvaluationOur DEFACTO system was evaluated through experimentscomparing the effectiveness of Adjustable Autonomy (AA)strategies over multiple users. In order to provide DE-FACTO with a dynamic rescue domain, we connected it tothe previously developed RoboCup Rescue simulation envi-ronment (Kitanoet al. 1999). To interface with DEFACTO,each fire engine is controlled by a proxy in order to han-dle the coordination and execution of AA strategies. Con-sequently, the proxies can try to allocate fire engines to firesin a distributed manner, but can also transfer control to themore expert user (incident commander). The user can then

IAAI-05 / 1559

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

A H AH ATH

(a) Subject 1

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

A H AH ATH

(b) Subject 2

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

A H AH ATH

(c) Subject 3

Figure 5: Performance.

use the Omni-Viewer in Allocation mode to allocate enginesto the fires that he has control over. In order to focus onthe AA strategies (transferring the control of task allocation)and not have the users ability to navigate interfere with re-sults, the Navigation mode was not used during this first setof experiments.

The results of our experiments are shown in Figure 5,which shows the results of subjects 1, 2, and 3. Each sub-ject was confronted with the task of aiding fire engines insaving a city hit by a disaster. For each subject, we testedthree strategies, specifically,H, AH andAT H; their perfor-mance was compared with the completely autonomousATstrategy.AH is an individual agent strategy, tested for com-parison withAT H, where agents act individually, and passthose tasks to a human user that they cannot immediatelyperform. Each experiment was conducted with the same ini-tial locations of fires and building damage. For each strategywe tested, varied the number of fire engines between 4, 6 and10. Each chart in Figure 5 shows the varying number of fireengines on the x-axis, and the team performance in terms ofnumbers of building saved on the y-axis. For instance, strat-egyAT saves 50 building with 4 agents. Each data point onthe graph is an average of three runs. Each run itself took15 minutes, and each user was required to participate in 27experiments, which together with 2 hours of getting orientedwith the system, equates to about 9 hours of experiments pervolunteer.

Figure 5 enables us to conclude the following:

• Human involvement with agent teams does not neces-sarily lead to improvement in team performance.Con-trary to expectations and prior results, human involvementdoes not uniformly improve team performance, as seenby human-involving strategies performing worse than theAT strategy in some cases. For instance, for subject 3AH strategy provides higher team performance thanATfor 4 agents, yet at 10 agents human influence is clearlynot beneficial.

• Providing more agents at a human’s command does notnecessarily improve the agent team performance.As seenfor subject 2 and subject 3, increasing agents from 4 to 6

givenAH andAT H strategies is seen to degrade perfor-mance. In contrast, for theAT strategy, the performanceof the fully autonomous agent team continues to improvewith additions of agents, thus indicating that the reductionin AH andAT H performance is due to human involve-ment. As the number of agents increase to 10, the agentteam does recover.

• No strategy dominates through all the experiments givenvarying numbers of agents.For instance, at 4 agents,human-involving strategies dominate theAT strategy.However, at 10 agents, theAT strategy outperforms allpossible strategies for subjects 1 and 3.

• Complex team-level strategies are helpful in practice:AT H leads to improvement overH with 4 agents forall subjects, although surprising domination ofAH overAT H in some cases indicates thatAH may also need auseful strategy to have available in a team setting.

Note that the phenomena described range over multipleusers, multiple runs, and multiple strategies. The most im-portant conclusion from these figures is thatflexibility is nec-essary to allow for the optimal AA strategy to be applied.The key question, then, is then how to select the appropriatestrategy for a team involving a human whose expected de-cision quality isEQH . In fact, by estimating theEQH ofa subject by checking the “H” strategy for small number ofagents (say 4), and comparing toAT strategy, we may beginto select the appropriate strategy for teams involving moreagents. In general, higherEQH lets us still choose strate-gies involving humans for a more numerous team. For largeteams however, the number of agentsAGH effectively con-trolled by the human does not grow linearly, and thusATstrategy becomes dominant.

Unfortunately, the strategies including the humans andagents (AH andAT H) for 6 agents show a noticeable de-crease in performance for subjects 2 and 3 (see Figure 5). Itwould be useful to understand which factors contributed tothis phenomena.

Our crucial predictions were that while numbers of agentsincrease,AGH steadily increases andEQH remains con-stant. Thus, the dip at 6 agents is essentially affected by

IAAI-05 / 1560

2

2.5

3

3.5

4

3 4 5 6 7 8 9 10 11Number of Agents

Age

nts/

Fire

AH ATH

(a) Subject 1

2

2.5

3

3.5

4

3 4 5 6 7 8 9 10 11Number of Agents

Age

nts/

Fire

AH ATH

(b) Subject 2

2

2.5

3

3.5

4

3 4 5 6 7 8 9 10 11Number of Agents

Age

nts/

Fire

AH ATH

(c) Subject 3

Figure 7: Amount of agents assigned per fire.

Strategy H AH AT H

# of agents 4 6 10 4 6 10 4 6 10

Subject 1 91 92 154 118 128 132 104 83 64

Subject 2 138 129 180 146 144 72 109 120 38

Subject 3 117 132 152 133 136 97 116 58 57

Table 1: Total amount of allocations given.

0

2

4

6

8

10

12

3 4 5 6 7 8 9 10 11Number of Agents

Effe

ctiv

e A

gent

s

Subject 1 Subject 2 Subject 3 A

(a)

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

Subject 1 Subject 2 Subject 3

(b)

Figure 6: Agents effectively used (AGH ) by the subjects (a)and Performance of strategyH. (b)

eitherAGH or EQH . We first testedAGH in our domain.The amount of effective agents,AGH , is calculated by di-viding how many total allocations each subject made by howmany theAT strategy made per agent, assumingAT strategyeffectively uses all agents. Figure 6-(a) shows the numberof agents on the x-axis and the number of agents effectiveused,AGH , on the y-axis; theAT strategy, which is usingall available agents, is also shown as a reference. However,the amount of effective agents is actually about the same in4 and 6 agents. This would not account for the sharp dropwe see in the performance. We then shifted our attentionto theEQH of each subject. One reduction inEQH couldbe because subjects simply did not send as many allocationstotally over the course of the experiments. This, however isnot the case as can be seen in Table 1 where for 6 agents, asthe total amount of allocations given is comparable to thatof 4 agents. To investigate further, we checked if the qual-ity of human allocation had degraded. For our domain, themore fire engines that fight the same fire, the more likely it

is to be extinguished and in less time. For this reason, theamount of agents that were tasked to each fire is a good in-dicator of the quality of allocations that the subject makes6-(b). Figure 7 shows the number agents on the x-axis andthe average amount of fire engines allocated to each fire onthe y-axis.AH andAT H for 6 agents result in significantlyless average fire engines per task (fire) and therefore loweraverageEQH .

Related WorkWe have discussed related work throughout this paper. How-ever, we now provide comparisons with key previous agentsoftware prototypes and research. Hill et al’s work is a simi-lar immersive training tool (Swartoutet al. 2001). However,their work focused more on multi-modal dialogue and em-phasize single agent interaction along predefined story lines,whereas our work focuses on adjustable autonomy and co-ordinating large numbers of agents in a dynamic, complexfire-fighting domain.

Among the current tools aimed at simulating rescue en-vironments, it is important to mention products like JCATS(Technology 2005) and EPICS (Laboratory 2005). JCATSrepresents a self-contained, high-resolution joint simulationin use for entity-level training in open, urban and subter-ranean environments. Developed by Lawrence LivermoreNational Laboratory, JCATS gives users the capability to de-tail the replication of small group and individual activitiesduring a simulated operation. At this point however, JCATScannot simulate agents. Finally, EPICS is a computer-based,scenario-driven, high-resolution simulation. It is used byemergency response agencies to train for emergency situ-ations that require multi-echelon and/or inter-agency com-munication and coordination. Developed by the U.S. ArmyTraining and Doctrine Command Analysis Center, EPICSis also used for exercising communications and commandand control procedures at multiple levels. Similar to JCATShowever, EPICS does not currently allow agents to partici-pate in the simulation.

Given our application domains, Scerri et al’s work onrobot-agent-person (RAP) teams for disaster rescue (Scerriet al. 2003) is closely related to DEFACTO. Our work takes

IAAI-05 / 1561

a significant step forward in comparison. First, the omni-viewer enables navigational capabilities improving humansituational awareness not present in previous work. Second,we provide team-level strategies, which we experimentallyverify, absent in that work. Third, we provide extensive ex-perimentation, and illustrate that some of the conclusionsreached in (Scerriet al. 2003) were indeed preliminary,e.g., they conclude that human involvement is always ben-eficial to agent team performance, while our more exten-sive results indicate that sometimes agent teams are betteroff excluding humans from the loop. Human interactions inagent teams has also been investigated in (Burstein, Mulve-hill, & Deutsch 1999; Suya You & Fox 2003), and there issignificant research on human interactions with robot-teams(Fong, Thorpe, & Baur 2002; Crandall, Nielsen, & Goodrich2003). However, they do not use flexible AA strategiesand/or team-level AA strategies. Furthermore, our exper-imental results may assist these researchers in recognizingthe potential for harm that humans may cause to agent orrobot team performance. Significant attention has been paidin the context of adjustable autonomy and mixed-initiativein single-agent single-human interactions (Horvitz 1999;Allen 1995). However, this paper focuses on new phenom-ena that arise in human interactions with agent teams.

SummaryThis paper presents an operational prototype training toolfor incident commanders: DEFACTO. DEFACTO incor-porates state of the art artificial intelligence, 3D visualiza-tion and human-interaction reasoning into a unique highfidelity system for training incident commanders. TheDEFACTO system achieves this via three main compo-nents: (i)Omnipresent Viewer - intuitive interface, (ii)ProxyFramework - for team coordination, and (iii) Flexible In-teraction - between the incident commander and the team.We present results preliminary experiments with DEFACTOin the firefighting domain. These results illustrate that anagent team must be equipped with flexible strategies for ad-justable autonomy so that the appropriate strategy can beselected. Positive feedback from DEFACTO’s ultimate con-sumers, LAFD, illustrates its promise and potential for real-world application.

AcknowledgmentsThanks to Nikhil Kasinadhuni for his work on the DE-FACTO Navigation Viewer. Also, thanks to Ronald Roe-mer, David Perez, and Roland Sprewell for their time andinvaluable input to this project.

ReferencesAllen, J. F. 1995. The TRAINS project: A case study inbuilding a conversational planning agent.Journal of Ex-perimental and Theoretical AI (JETAI)7:7–48.

Burstein, M. H.; Mulvehill, A. M.; and Deutsch, S. 1999.An approach to mixed-initiative management of heteroge-neous software agent teams. InHawaii International Con-ference on System Sciences, 8055. IEEE Computer Society.

Crandall, J. W.; Nielsen, C. W.; and Goodrich, M. A. 2003.Towards predicting robot team performance. InIEEE In-ternational Conference on Systems, Man and Cybernetics(SMC).Fong, T.; Thorpe, C.; and Baur, C. 2002. Multi-robot re-mote driving with collaborative control.IEEE Transactionson Industrial Electronics.Horvitz, E. 1999. Principles of mixed-initiative user in-terfaces. InProceedings of ACM SIGCHI Conference onHuman Factors in Computing Systems (CHI’99), 159–166.Kitano, H.; Tadokoro, S.; Noda, I.; Matsubara, H.; Taka-hashi, T.; Shinjoh, A.; and Shimada, S. 1999. Robocup res-cue: Search and rescue in large-scale disasters as a domainfor autonomous agents research. InIEEE InternationalConference on Systems, Man and Cybernetics (SMC), vol-ume VI, 739–743.Laboratory, L. L. N. 2005. Jcats - joint conflict and tacticalsimulation. Inhttp://www.jfcom.mil/about/factjcats.htm.LAFD. 2001. High-rise incident command system. InLosAngeles Fire Department.Paivio, A. 1974. Pictures and words in visual search.Mem-ory & Cognition2(3):515–521.Pynadath, D. V., and Tambe, M. 2003. Automated team-work among heterogeneous software agents and humans.Journal of Autonomous Agents and Multi-Agent Systems(JAAMAS)7:71–100.Richardson, A.; Montello, D.; and Hegarty, M. 1999. Spa-tial knowledge acquisition from maps and from navigationin real and virtual environments.Memory and Cognition27(4):741–750.Ruddle, R.; Payne, S.; and Jones, D. 1997. Navigat-ing buildings in desktop virtual environments: Experimen-tal investigations using extended navigational experience.J. Experimental Psychology - Applied3(2):143–159.Scerri, P.; Pynadath, D. V.; Johnson, L.; Rosenbloom, P.;Schurr, N.; Si, M.; and Tambe, M. 2003. A prototypeinfrastructure for distributed robot-agent-person teams. InInternational Joint Conference on Autonomous Agents andMulti-Agent Systems (AAMAS).Scerri, P.; Pynadath, D.; and Tambe, M. 2002. Towards ad-justable autonomy for the real world.Journal of ArtificialIntelligence Research17:171–228.Suya You, Jinhui Hu, U. N., and Fox, P. 2003. Urban sitemodeling from lidar. InProc. 2nd Int’l Workshop Com-puter Graphics and Geometric Modeling (CGGM), 579–588.Swartout, W., R. H.; Gratch, J.; Johnson, W. L.; Kyri-akakis, C.; LaBore, C.; Lindheim, R.; Marsella, S.; Mi-raglia, D.; Moore, B.; Morie, J.; Rickel, J.; Thiebaux, M.;Tuch, L.; Whitney, R.; and Douglas, J. 2001. Towardthe holodeck: integrating graphics, sound, character andstory. InProceedings of the Fifth International Conferenceon Autonomous Agents, 409–416. ACM Press.Technology, A. S. 2005. Epics - emergencypreparedness incident commander simulation. Inhttp://epics.astcorp.com.

IAAI-05 / 1562