Lessons Learned from Disaster Management ∗

Nathan Schurr, Pratik Patil, Fred Pighin, Milind Tambe,University of Southern California, Los Angeles, CA 90089, {schurr, pratiksp, pighin, tambe}@usc.edu

ABSTRACTThe DEFACTO system is a multiagent based tool for training inci-dent commanders of large scale disasters. In this paper, we high-light some of the lessons that we have learned from our interactionwith the Los Angeles Fire Department (LAFD) and how they haveaffected the way that we continued the design of our disaster man-agement training system. These lessons were gleaned from LAFDfeedback and initial training exercises and they include: system de-sign, visualization, improving trainee situational awareness, adjust-ing training level of difficulty and situation scale. We have takenthese lessons and used them to improve the DEFACTO system’straining capabilities. We have conducted initial training exercisesto illustrate the utility of the system in terms of providing usefulfeedback to the trainee.

1. INTRODUCTIONThe recent hurricanes that have hit the gulf coast of the US have

served to reaffirm the need for emergency response agencies to bebetter prepared for large scale disasters. Both natural and man-made (terrorism) disasters are growing in scale, however the re-sponse to these incidents continues to be managed by a single per-son, namely the incident commander. The incident commandermust monitor and direct the entire event while maintaining com-plete responsibility. Because of this, incident commanders muststart to be trained to handle these large scale events and assist inthe coordination of the team.

In order to fulfill this need and leverage the advantages of multia-gents, we have continued to develop the DEFACTO system (Demon-strating Effective Flexible Agent Coordination of Teams via Om-nipresence). DEFACTO is a multiagent based tool for training in-cident commanders for large scale disasters (man-made or natural).

Our system combines a high fidelity simulator, a redesigned hu-

∗This research was supported by the United States Departmentof Homeland Security through the Center for Risk and EconomicAnalysis of Terrorism Events (CREATE). However, any opinions,findings, and conclusions or recommendations in this document arethose of the author and do not necessarily reflect views of the U.S.Department of Homeland Security.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Copyright 200X ACM X-XXXXX-XX-X/XX/XX ... $5.00.

man interface, and a multiagent team driving all of the behaviors.Training incident commanders provides a dynamic scenario in whichdecisions must be made correctly and quickly because human safetyis at risk. When using DEFACTO, incident commanders have theopportunity to see the disaster in simulation and the coordinationand resource constraints unfold so that they can be better preparedwhen commanding over an actual disaster. Applying DEFACTO todisaster response aims to benefit the training of incident comman-ders in the fire department.

With DEFACTO, our objective is to both enable the human tohave a clear idea of the team’s state and improve agent-human teamperformance. We want DEFACTO agent-human teams to betterprepare firefighters for current human-only teams. We believe thatby leveraging multiagents, DEFACTO will result in better disasterresponse methods and better incident commanders.

Previously, we have discussed building our initial prototype sys-tem, DEFACTO [8]. Recently, the Los Angeles Fire Department(LAFD) have begun to evaluate the DEFACTO system. In this pa-per, we highlight some of the lessons that we have learned from ourinteraction with the LAFD and how they have affected the way thatwe continued to design of our training system. These lessons weregleaned from LAFD feedback and initial training exercises.

The lessons learned from the feedack from the LAFD include:system design, visualization, improving trainee situational aware-ness, adjusting training level of difficulty and situation scale. Wehave taken these lessons and used them to improve the DEFACTOsystem’s training capabilities.

We have also perfromed initial training exercise experiments toillustrate the utility of the system in terms of providing useful feed-back to the trainee. We ended up finding that allowing more fireengines to be at the disposal of the incident commander sometimesnot only didn’t improve, but rather worsened team performance.There were even some instances in which the agent team wouldhave performed better had the team never listened to human adviceat all. We also provide analysis of such behaviors, thereby illustrat-ing the utility of DEFACTO resulting from the feedback given totrainees.

2. MOTIVATIONIn this section, we will first start with an explanation of the cur-

rent methods for training that the LAFD currently use. Then weexplain some of the advantages that our multiagent approach hasover these methods.

The incident commander’s main duties during a fire shoulder allresponsibility for the safety of the firefighters. In order to do this,the incident commander must have constant contact with the fire-fighters and have a complete picture of the entire situation. Theincident commander must make certain that dangerous choices are

LAFD Exercise: Simulations by People Playing Roles

� LAFD officials simulate fire progression and the resource availability

� Battalion Chief allocates available resources to tasks

(a) Current Incident Commander Training Exercise(b) Fire Captain Roemer using the DEFACTO trainingsystem

Figure 1: Old vs. New training methods

avoided and the firefighters are informed and directed as needed.We were allowed to observe a Command Post Exercise that sim-

ulated the place where the incident commander is stationed duringa fire (see Figure 1(a)). The Incident commander has an assistantby his side who keeps track on a large sheet of paper where all ofthe resources (personnel and equipment) are located. A sketch ofthe fire is also made on this sheet, and the fire and fire engines’location is also managed.

The Command Post is currently simulated by projecting a sin-gle static image of a fire in an apartment. In the back of the room,several firefighters are taken off duty in order to play the role of fire-fighters on the scene. They each communicate on separate channelsover walkie talkies in order to coordinate by sharing informationand accepting orders. The fire spreading is simulated solely byhaving one of the off-duty firefighters in the back speaking over thewalkie talkie and describing the fire spreading.

The LAFD’s curent approach, however, has several limitations.First, it requires a number of officers to be taken off duty, whichdecreases the number of resources available to the city for a disas-ter during training. Second, the disaster conditions created are notaccurate in the way that they appear or progress. Since the imagethat the incident commander is seeing is static, there is no infor-mation about state or conditions of the fire that can be ascertainedfrom watching it, which is contrary to the actual scene of a disas-ter response. Furthermore, the fire’s behavior is determined by thereports of the acting fire fighters over the walkie talkie, which attimes might not be a plausible progression of fire in reality. Third,this method of training restricts it to a smaller scale of fire becauseof the limited personnel and rigid fire representation.

Our system aims to enhance the training of the incident com-manders (see Figure 1(b)). Our approach allows for training to notbe so personnel heavy, because fire fighter actors will be replacedby agents. By doing this we can start to train incident comman-ders with a larger team. Through our simualtion, we can also startto simulate larger events in order to push the greater number ofavailable resources to their limit. Also, by simulating the fire pro-gression, we can place the Incident commander in a more realisticsituation and force them to react to realistic challenges that arise.

3. SYSTEM ARCHITECTUREIn this section, we will describe the technologies used in three

major components of DEFACTO: the Omni-Viewer, proxy-based

Disaster Scenario

Proxy

Proxy

Proxy

Proxy

Team

Omni-Viewer

DEFACTO

Incident Commander

Figure 2: System Architecture

team coordination, and proxy-based adjustable autonomy. The Omni-Viewer is an advanced human interface for interacting with an agent-assisted response effort. The Omni-Viewer has been introducedbefore [8], however has since been redesigned after incorporatinglessons learned from the LAFD. The Omni-Viewer now providesfor both global and local views of an unfolding situation, allow-ing a human decision-maker to obtain precisely the information re-quired for a particular decision. A team of completely distributedproxies, where each proxy encapsulates advanced coordination rea-soning based on the theory of teamwork, controls and coordinatesagents in a simulated environment. The use of the proxy-basedteam brings realistic coordination complexity to the training systemand allows a more realistic assessment of the interactions betweenhumans and agent-assisted response. These same proxies also en-able us to implement the adjustable autonomy necessary to balancethe decisions of the agents and human.

DEFACTO operates in a disaster response simulation environ-ment. The simulation environment itself is provided by the RoboCupRescue Simulator [3]. To interface with DEFACTO, each fire en-gine is controlled by a proxy in order to handle the coordinationand execution of adjustable autonomy strategies. Consequently, theproxies can try to allocate fire engines to fires in a distributed man-ner, but can also transfer control to the more expert user (incidentcommander). The user can then use the Omni-Viewer to allocateengines to the fires that he has control over. In our scenario, sev-eral buildings are initially on fire, and these fires spread to adjacentbuildings if they are not quickly contained. The goal is to have a

Figure 3: Proxy Architecture

human interact with the team of fire engines in order to save thegreatest number of buildings. Our overall system architecture ap-plied to disaster response can be seen in Figure 2.

3.1 Omni-ViewerOur goal of allowing fluid human interaction with agents re-

quires a visualization system that provides the human with a globalview of agent activity as well as shows the local view of a particu-lar agent when needed. Hence, we have developed an omnipresentviewer, or Omni-Viewer, which will allow the human user diverseinteraction with remote agent teams. While a global view is ob-tainable from a two-dimensional map, a local perspective is bestobtained from a 3D viewer, since the 3D view incorporates the per-spective and occlusion effects generated by a particular viewpoint.

To address our discrepant goals, the Omni-Viewer allows forboth a conventional map-like top down 2D view and a detailed 3Dviewer. The viewer shows the global overview as events are pro-gressing and provides a list of tasks that the agents have transferredto the human, but also provides the freedom to move to desired lo-cations and views. In particular, the user can drop to the virtualground level, thereby obtaining the perspective (local view) of aparticular agent. At this level, the user can fly freely around thescene, observing the local logistics involved as various entities areperforming their duties. This can be helpful in evaluating the phys-ical ground circumstances and altering the team’s behavior accord-ingly. It also allows the user to feel immersed in the scene wherevarious factors (psychological, etc.) may come into effect.

3.2 Proxy: Team CoordinationA key hypothesis in this work is that intelligent distributed agents

will be a key element of a disaster response. Taking advantageof emerging robust, high bandwidth communication infrastructure,we believe that a critical role of these intelligent agents will be tomanage coordination between all members of the response team.Specifically, we are using coordination algorithms inspired by the-ories of teamwork to manage the distributed response [6]. The gen-eral coordination algorithms are encapsulated inproxies, with eachteam member having its own proxy which represents it in the team.The current version of the proxies is calledMachinetta[7] and ex-tends the earlier Teamcore proxies [5]. Machinetta is implementedin Java and is freely available on the web. Notice that the conceptof a reusable proxy differs from many other “multiagent toolkits”in that it provides the coordinationalgorithms, e.g., algorithms forallocating tasks, as opposed to theinfrastructure, e.g., APIs for re-liable communication.

Communication: communication with other proxiesCoordination: reasoning about team plans and communicationState: the working memory of the proxyAdjustable Autonomy: reasoning about whether to act autonomously

or pass control to the team member

RAP Interface: communication with the team member

The Machinetta software consists of five main modules, threeof which are domain independent and two of which are tailoredfor specific domains. The three domain independent modules arefor coordination reasoning, maintaining local beliefs (state) and ad-justable autonomy. The domain specific modules are for commu-nication between proxies and communication between a proxy anda team member. The modules interact with each other only via theproxy’s local belief state with a blackboard design and are designedto be “plug and play.” Thus new adjustable autonomy algorithmscan be used with existing coordination algorithms. The coordina-tion reasoning is responsible for reasoning about interactions withother proxies, thereby implementing the coordination algorithms.

Teams of proxies implementteam oriented plans(TOPs) whichdescribe joint activities to be performed in terms of the individualroles and any constraints between those roles. Generally, TOPsare instantiated dynamically from TOP templates at runtime whenpreconditions associated with the templates are filled. Typically, alarge team will be simultaneously executing many TOPs. For ex-ample, a disaster response team might be executing multiple fightfire TOPs. Such fight fire TOPs might specify a breakdown offighting a fire into activities such as checking for civilians, ensur-ing power and gas is turned off, and spraying water. Constraintsbetween these roles will specify interactions such as required exe-cution ordering and whether one role can be performed if anotheris not currently being performed. Notice that TOPs do not specifythe coordination or communication required to execute a plan; theproxy determines the coordination that should be performed.

Current versions of Machinetta include a token-based role allo-cation algorithm. The decision for the agent becomes whether toassign values from the tokens it currently has to its variable or topass the tokens on. First, the team member can choose the mini-mum capability the agent should have in order to assign the value.This minimum capability is referred to as thethreshold. The thresh-old is calculated once (Algorithm 1, line 6), and attached to thetoken as it moves around the team.

Second, the agent must check whether the value can be assignedwhile respecting its local resource constraints (Algorithm 1, line9). If the value cannot be assigned within the resource constraintsof the team member, it must choose a value(s) to reject and pass onto other teammates in the form of a token(s) (Algorithm 1, line 12).The agent keeps values that maximize the use of its capabilities(performed in the MAX CAP function, Algorithm 1, line 10).

Algorithm 1 TOKENMONITOR(Cap, Resources)

1: V ← ∅2: while truedo3: msg ← getMsg()4: token← msg5: if token.threshold = NULL then6: token.threshold← COMPUTETHRESHOLD(token)7: if token.threshold ≤ Cap(token.value) then8: V ← V ∪ token.value9: if

Pv∈V Resources(v) ≥ agent.resources then

10: out← V− MAX CAP(V alues)11: for all v ∈ out do12: PASSON(newtoken(v))13: V alues← V alues− out14: else15: PASSON(token) /* threshold > Cap(token.value) */

3.3 Proxy: Adjustable Autonomy

One key aspect of the proxy-based coordination is AdjustableAutonomy. Adjustable autonomy refers to an agent’s ability to dy-namically change its own autonomy, possibly to transfer controlover a decision to a human. Previous work on adjustable autonomycould be categorized as either involving a single person interactingwith a single agent (the agent itself may interact with others) or asingle person directly interacting with a team. In the single-agentsingle-human category, the concept of flexible transfer-of-controlstrategy has shown promise [6]. A transfer-of-control strategy is apreplanned sequence of actions to transfer control over a decisionamong multiple entities. For example, anAH1H2 strategy impliesthat an agent (A) attempts a decision and if the agent fails in the de-cision then the control over the decision is passed to a humanH1,and then ifH1 cannot reach a decision, then the control is passedto H2. Since previous work focused on single-agent single-humaninteraction, strategies were individual agent strategies where only asingle agent acted at a time.

An optimal transfer-of-control strategy optimally balances therisks of not getting a high quality decision against the risk of costsincurred due to a delay in getting that decision. Flexibility in suchstrategies implies that an agent dynamically chooses the one thatis optimal, based on the situation, among multiple such strategies(H1A, AH1, AH1A, etc.) rather than always rigidly choosing onestrategy. The notion of flexible strategies, however, has not been ap-plied in the context of humans interacting with agent-teams. Thus,a key question is whether such flexible transfer of control strategiesare relevant in agent-teams, particularly in a large-scale applicationsuch as ours.

DEFACTO aims to answer this question by implementing transfer-of-control strategies in the context of agent teams. One key ad-vance in DEFACTO is that the strategies are not limited to indi-vidual agent strategies, but also enables team-level strategies. Forexample, rather than transferring control from a human to a singleagent, a team-level strategy could transfer control from a human toan agent-team. Concretely, each proxy is provided with all strategyoptions; the key is to select the right strategy given the situation. Anexample of a team level strategy would combineAT Strategy andH Strategy in order to makeAT H Strategy. The default team strat-egy,AT , keeps control over a decision with the agent team for theentire duration of the decision. TheH strategy always immediatelytransfers control to the human.AT H strategy is the conjunction ofteam levelAT strategy withH strategy. This strategy aims to sig-nificantly reduce the burden on the user by allowing the decision tofirst pass through all agents before finally going to the user, if theagent team fails to reach a decision.

4. LESSONS LEARNED FROM INITIAL DE-PLOYMENT FEEDBACK

Through our communication with strategic training division ofthe LAFD (see Figure 1(b)), we have learned a lot of lessons thathave influenced the continuing development of our system.

4.1 PerspectiveJust as in multiagent systems, the Incident commander must over-

come the challenge of managing a team that each possess only apartial local view. This is highlighted in fighting a fire by incidentcommanders keeping in mind that there are five views to every fire(4 sides and the top). Only by taking into account what is hap-pening on all five sides of the fire, can the fire company make aneffective decision on how many people to send where. Because ofthis, a local view (see Figure 4(a)) can augment the global view (see

Figure 4(b)) becomes helpful in determining the local perspectivesof team members. For example, by taking the perspective of a firecompany in the back of the building, the incident commander canbe aware that they might not see the smoke from the second floor,which is only visible from the front of the building. The incidentcommander can then make a decision to communicate that to thefire company or make an allocation accordingly.

The 3D perspective of the Omni-Viewer was initially thought tobe an example of a futuristic vision of the actual view given to theincident commander. But after allowing the fire fighters to lookat the display, they remarked, that they have such views availableto them already, especially in large scale fires (the very fires we aretrying to simulate). At the scene of these fires are often a news heli-copter is at the scene and the incident commander can patch into thefeed and display it at his command post. Consequently our trainingsimulation can already start to prepare the Incident commander toincorporate a diverse arrary of information sources.

4.2 Fire BehaviorWe also learned how important smoke and fire behavior is to the

firefighters in order affect their decisions. Upon our first showingof initial prototypes to the incident commanders, they looked at oursimulation, with flames swirling up out of the roof (see Figure 5(a)).We artificially increased fire intensity in order to show off the firebehavior and this hampered their ability to evaluate the situationand allocations. They all agreed that every firefighter should bepulled out because that building is lost and might fall at any minute!In our efforts to put a challenging fire in front of them to fight, wehad caused them to walk away from the training. Once we startto add training abilities, such as to watch the fire spread in 3D, wehave to also start to be more aware of how to accurately show afire that incident commander would face. We have consequentlyaltered the smoke and fire behavior (see Figure 5(b)). The smokeappears less “dramatic” to the a lay person than a towering inferno,but provides a more effective training environment.

4.3 Gradual TrainingInitially, we were primarily concerned with changes to the sys-

tem that allowed for a more accurate simulation of what the inci-dent commander would actually see. Alternatively, we have alsoadded features, not because of their accuracy, but also to aid intraining by isolating certain tasks. Very often in reality and in oursimulations, dense urban areas obscure the ability to see where allof the resources (i.e. fire engines) are and prevent a quick view ofthe situation (see Figure 6(a)). To this aim, we have added a newmode using the 3D, but having the buildings each have no height,which we refer to as Flat World (see Figure 6(b)). By using thisflat view, the trainee is allowed to concentrate on the allocating re-sources correctly, without the extra task of developing an accurateworld view with obscuring high rise buildings.

4.4 User IntentA very important lesson that we learned from the LAFD, was

that the incident commander cannot be given all information for theteam and thus the human does not know all about the status of theteam members and vice versa. Consequently, this lack of completeawareness of the agent team’s intentions can lead to some harmfulallocations by the human (incident commander). In order for infor-mation to be selectively available to the incident commander, wehave allowed the incident commander to query for the status of aparticular agent. Figure 7 shows an arrow above the Fire Engineat the center of the screen that has been selected. On the left, thestatistics are displayed. The incident comander is able to select a

(a) Local Perspective (b) Global Perspective

Figure 4: Local vs. Global Perspectives in the Omni-Viewer

(a) Old Fire (b) New Smoke

Figure 5: Improvement in fire visualization

(a) Normal (b) Flat World

Figure 6: Improvement in locating resources (fire engines and ambulances)

Figure 7: Selecting for closer look at a Fire Engine.

particular fire engine and find out the equipment status, personnelstatus, and the current tasks are being performed by the fire fightersaboard that engine. This detailed information can be accessed ifdesired by the incident commander, but is not thrown to the screenby all agents, in order to not overwhelm the incident commander.

4.5 ScaleIn addition, we have also learned of new challenges that we are

currently attempting to tackle by enhancing the system. One of thebiggest challenges in order to start simulating a large urban fire isthe sheer scale of the resources that must be managed. Accordingto the fire captains, in order to respond to a single high rise build-ing with a few floors on fire, roughly 200 resources (fire engines,paramedics etc.) would need to be managed at the scene. Coor-dinating such a large number of agents on a team is a challenge.Also, as the incident scales to hundreds of resources, the incidentcommander ends up giving more autonomy to the team or else facebeing overwhelmed. We argue, that adjustable autonomy will startto play a bigger and more essential roll in allowing for the incidentcommander to monitor the situation.

5. LESSONS LEARNED FROM TRAININGEXERCISES

5.1 Training ExercisesIn order to study the potential of DEFACTO, we performed some

training exercises with volunteers. These initial experiments showedus that humans can both help and hurt the team performance. Thekey point is that DEFACTO allows such experiments with trainingexercises and more importantly allows for analysis and feedbackregarding the exercises. Thus trainees can gain useful insight as towhy their decisions led to problematic/beneficial situations.

In particular, some of our initial experimental results were pub-lished earlier in [8], but now we are able to provide analysis andfeedback. The results of our training exercise experiments are shownin Figure 8, which shows the results of subjects 1, 2, and 3. Eachsubject was confronted with the task of aiding fire engines in sav-ing a city hit by a disaster. For each subject, we tested three strate-gies, specifically,H, AH (individual agent, then human) andAT H(agent team, then human); their performance was compared withthe completely autonomousAT strategy.AH is an individual agentstrategy, tested for comparison withAT H, where agents act indi-vidually, and pass those tasks to a human user that they cannot im-mediately perform. Each experiment was conducted with the same

AH ALL

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

NUMBER OF AGENTS SENT TO BUILDING

PR

OB

AB

ILIT

Y B

UIL

DIN

G S

AV

ED

AH ALL

Figure 10: AH for all subjects.

initial locations of fires and building damage. For each strategy wetested, varied the number of fire engines between 4, 6 and 10. Eachchart in Figure 8 shows the varying number of fire engines on thex-axis, and the team performance in terms of numbers of buildingsaved on the y-axis. For instance, strategyAT saves 50 buildingwith 4 agents. Each data point on the graph is an average of threeruns. Each run itself took 15 minutes, and each user was requiredto participate in 27 experiments, which together with 2 hours ofgetting oriented with the system, equates to about 9 hours of exper-iments per volunteer.

Figure 8 enables us to conclude the following:

• Human involvement with agent teams does not necessarilylead to improvement in team performance.Contrary to ex-pectations and prior results, human involvement does notuniformly improve team performance, as seen by human-involving strategies performing worse than theAT strategyin some cases. For instance, for subject 3AH strategy pro-vides higher team performance thanAT for 4 agents, yet at10 agents human influence is clearly not beneficial.

• Providing more agents at a human’s command does not nec-essarily improve the agent team performance.As seen forsubject 2 and subject 3, increasing agents from 4 to 6 givenAH andAT H strategies is seen to degrade performance. Incontrast, for theAT strategy, the performance of the fully au-tonomous agent team continues to improve with additions ofagents, thus indicating that the reduction inAH andAT Hperformance is due to human involvement. As the number ofagents increase to 10, the agent team does recover.

• Complex team-level strategies are helpful in practice: AT Hleads to improvement overH with 4 agents for all subjects,although surprising domination ofAH over AT H in somecases indicates thatAH may also need a useful strategy tohave available in a team setting.

Note that the phenomena described range over multiple users,multiple runs, and multiple strategies. Unfortunately, the strate-gies including the humans and agents (AH andAT H) for 6 agentsshow a noticeable decrease in performance for subjects 2 and 3(see Figure 8). It would be useful to understand which factors con-tributed to this phenomena from a trainee’s perspective.

5.2 Analysis

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

A H AH ATH

(a) Subject 1

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

A H AH ATH

(b) Subject 2

0

50

100

150

200

250

300

3 4 5 6 7 8 9 10 11Number of Agents

Bui

ldin

gs S

aved

A H AH ATH

(c) Subject 3

Figure 8: Performance.

2

2.5

3

3.5

4

3 4 5 6 7 8 9 10 11Number of Agents

Age

nts/

Fire

AH ATH

(a) Subject 1

2

2.5

3

3.5

4

3 4 5 6 7 8 9 10 11Number of Agents

Age

nts/

Fire

AH ATH

(b) Subject 2

2

2.5

3

3.5

4

3 4 5 6 7 8 9 10 11Number of Agents

Age

nts/

Fire

AH ATH

(c) Subject 3

Figure 9: Amount of agents assigned per fire.

ATH ALL

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

NUMBER OF AGENTS SENT TO BUILDING

PR

OB

AB

ILIT

Y B

UIL

DIN

G S

AV

ED

ATH ALL

Figure 11: ATH for all subjects.

We decided to a more in depth analysis of what exactly was caus-ing the degrading performance when 6 agents were at the disposalof the incident commander. Figure 9 shows the number agents onthe x-axis and the average amount of fire engines allocated to eachfire on the y-axis.AH andAT H for 6 agents result in significantlyless average fire engines per task (fire) and therefore lower average.Another interesting thing that we found was that this lower aver-age was not due to the fact that the incident commander was over-whelmed and making less decisions (allocations). Figures 12(a),12(b), and 12(c) all show how the number of buildings attacked donot go down in the case of 6 agents, where poor performance isseen.

Figures 10 and 11 show the number of agents assigned to abuilding on the x-axis and the probability that the given buildingwould be saved on the y-axis. The correlation between these val-ues demonstrate the correlation between number of agents assignedto the quality of the decision.

We can conclude from this analysis that the degradation in per-formance occurred at 6 agents because fire engine teams were splitup, leading to fewer fire-engines being allocated per building onaverage. Indeed, leaving fewer than 3 fire engines per fire leadsto a significant reduction in fire extinguishing capability. We canprovide such feedback of overall performance, showing the perfor-mance reduction at six fire engines, and our analysis to a trainee.The key point here is that DEFACTO is capable of allowing forsuch exercises, and their analyses, and providing feedback to po-tential trainees, so they improve their decision making, Thus, inthis current set of exercises, trainees can understand that with sixfire engines, they had managed to split up existing resources inap-propriately.

0

50

100

150

200

250

300

2 4 6 8 10 12

Number of Agents

Bu

ildin

gs

Att

acke

d

AH ATH

(a) Subject 1

0

50

100

150

200

250

2 4 6 8 10 12

Number of Agents

Bu

ildin

gs

Att

acke

d

AH ATH

(b) Subject 2

0

50

100

150

200

250

300

350

2 4 6 8 10 12

Number of Agents

Bu

ildin

gs

Att

acke

d

AH ATH

(c) Subject 3

Figure 12: Number of buildings attacked.

6. RELATED WORK AND SUMMARYIn terms of related work, it is important to mention products

like JCATS [9] and EPICS [4]. JCATS represents a self-contained,high-resolution joint simulation in use for entity-level training inopen, urban and subterranean environments. Developed by LawrenceLivermore National Laboratory, JCATS gives users the capabilityto detail the replication of small group and individual activities dur-ing a simulated operation. At this point however, JCATS cannotsimulate agents. Finally, EPICS is a computer-based, scenario-driven, high-resolution simulation. It is used by emergency re-sponse agencies to train for emergency situations that require multi-echelon and/or inter-agency communication and coordination. De-veloped by the U.S. Army Training and Doctrine Command Analy-sis Center, EPICS is also used for exercising communications andcommand and control procedures at multiple levels. Similar toJCATS however, EPICS does not currently allow agents to par-ticipate in the simulation. More recently multiagents have beensuccesfully applied to training navy tactics [10] and teams of Un-inhabited Air Vehicles [1, 2]. Our work is similar to these in spirit,however our focus and lessons learned are based on the train ofincident commanders in disaster rescue environments.

In summary, in order to train incident commanders for large scaledisasters, we have been working on the DEFACTO training system.This multiagent system tool has begun to be used by fire captainsfrom the Los Angeles Fire Department. We have learned somevaluable lessons from their feedback and the analysis of some ini-tial training exercise experiments. These lessons were gleaned fromLAFD feedback and initial training exercises. The lessons learnedfrom the feedack from the LAFD include: system design, visual-ization, improving trainee situational awareness, adjusting traininglevel of difficulty and situation scale. We have taken these lessonsand used them to improve the DEFACTO system’s training abili-ties. We have conducted initial training exercises to illustrate theutility of the system in terms of providing useful feedback to thetrainee. Through DEFACTO, we hope to improve training tools forand consequently improve the preparedness of incident comman-ders.

7. ACKNOWLEDGMENTSThanks to CREATE center for their support. Also, thanks to Fire

Captains of the LAFD: Ronald Roemer, David Perez, and RolandSprewell for their time and invaluable input to this project.

8. REFERENCES[1] J. W. Baxter and G. S. Horn. Controlling teams of

uninhabited air vehicles. InProceedings of the fourth

international joint conference on Autonomous agents andmultiagent systems (AAMAS), 2005.

[2] S. Karim and C. Heinze. Experiences with the design andimplementation of an agent-based autonomous uavcontroller. InProceedings of the fourth international jointconference on Autonomous agents and multiagent systems(AAMAS), 2005.

[3] H. Kitano, S. Tadokoro, I. Noda, H. Matsubara, T. Takahashi,A. Shinjoh, and S. Shimada. Robocup rescue: Search andrescue in large-scale disasters as a domain for autonomousagents research. InIEEE SMC, volume VI, pages 739–743,Tokyo, October 1999.

[4] L. L. N. Laboratory. Jcats - joint conflict and tacticalsimulation. Inhttp://www.jfcom.mil/about/factjcats.htm,2005.

[5] D. V. Pynadath and M. Tambe. Automated teamwork amongheterogeneous software agents and humans.Journal ofAutonomous Agents and Multi-Agent Systems (JAAMAS),7:71–100, 2003.

[6] P. Scerri, D. Pynadath, and M. Tambe. Towards adjustableautonomy for the real world.Journal of ArtificialIntelligence Research, 17:171–228, 2002.

[7] P. Scerri, D. V. Pynadath, L. Johnson, P. Rosenbloom,N. Schurr, M. Si, and M. Tambe. A prototype infrastructurefor distributed robot-agent-person teams. InAAMAS, 2003.

[8] N. Schurr, J. Marecki, P. Scerri, J. P. Lewis, and M. Tambe.The defacto system: Training tool for incident commanders.In The Seventeenth Innovative Applications of ArtificialIntelligence Conference (IAAI), 2005.

[9] A. S. Technology. Epics - emergency preparedness incidentcommander simulation. Inhttp://epics.astcorp.com, 2005.

[10] W. A. van Doesburg, A. Heuvelink, and E. L. van den Broek.Tacop: A cognitive agent for a naval training simulationenvironment. InProceedings of the fourth international jointconference on Autonomous agents and multiagent systems(AAMAS), 2005.