Cost Adaptation for Robust Decentralized Swarm Behaviour

Peter Henderson1, Matthew Vertescher2, David Meger1, and Mark Coates2

Abstract— Decentralized receding horizon control (D-RHC) provides a mechanism for coordination in multi-agent settings without a centralized command center.However, combining a set of different goals, costs, andconstraints to form an efficient optimization objectivefor D-RHC can be difficult. To allay this problem, weuse a meta-learning process – cost adaptation – whichgenerates the optimization objective for D-RHC to solvebased on a set of human-generated priors (cost andconstraint functions) and an auxiliary heuristic. We use thisadaptive D-RHC method for control of mesh-networkedswarm agents. This formulation allows a wide rangeof tasks to be encoded and can account for networkdelays, heterogeneous capabilities, and increasingly largeswarms through the adaptation mechanism. We leveragethe Unity3D game engine to build a simulator capable ofintroducing artificial networking failures and delays in theswarm. Using the simulator we validate our method onan example coordinated exploration task. We demonstratethat cost adaptation allows for more efficient and safertask completion under varying environment conditions andincreasingly large swarm sizes. We release our simulatorand code to the community for future work.

I. INTRODUCTION

In recent years, unmanned aerial vehicles (UAVs)have proven useful for a number of tasks – such assearching, exploration, and area mapping [1], [2], [3].However, the maximum flight time of these systems isstill limited. To accomplish such tasks efficiently, with-out need for a command-and-control center, decentral-ized swarm systems – consisting of many agents whichcoordinate among themselves – have been used. Onemethod for coordination of a swarm is via decentralizedreceding horizon control (D-RHC) [4]. This involves for-mulating and optimizing a cost-based objective functionusing predictive information about neighbouring swarmmembers and the agent’s goals.

However, in a complex swarm, it is difficult to se-lect the most efficient and safe combination of costsa priori. Rather, agents receive feedback about theirlocal environment and task performance during execu-tion. Varying network delays, heterogeneous mixturesof robot capabilities, and other environmental factorsmay render a pre-defined D-RHC optimization objectiveineffective. As such, we introduce the notion of costadaptation in the context of a swarm D-RHC problem

1Department of Computer Science, McGill University, Montreal,Quebec, Canada peter.henderson@mail.mcgill.ca

2Department of Electrical, Computer, and Software Engineering,McGill University, Montreal, Quebec, Canada

formulation. Using a heuristic-based guided search, welearn to generate the swarm’s D-RHC objective bymodifying and combining a set of human-generatedprior cost and constraint functions. We leverage theswarm’s interaction with the environment to perform thismeta-learning optimization, in our case through adaptivesimulated annealing [5]. This allows the adaptive agentsto perform the present task optimized for safety andefficiency. We provide a simple method for combininggoal-driven objectives and adapting to a myriad ofenvironmental perturbations. This method has the addedbenefit of improving interpretability over a heuristic-optimized action policy (as with reinforcement learningin [6]). The learned D-RHC objective (which uses aninterpretable combination of human priors) can giveinsight into the decision-making process and can bereviewed/edited.

To implement and evaluate our adaptive D-RHCmethod, we focus on a cooperative exploration task witha UAV swarm, as this has been a widely cited problemwith a variety of applications and goals [2], [3]. We firstdevelop a set of costs and constraints based on naturalflocking behaviour [7] and an agent’s goals, along with adecision-making framework for D-RHC. For communi-cation, we formulate a mesh network which can sendmessages within the swarm based on neighbourhoodtopology. To determine an agent’s current goals, we alsoimplement a bidding system to actively divide and assignsub-goals within the swarm in a decentralized manner.This allows for large numbers of agents to extend theswarm and actively partition swarms if this results inmore efficient global coordination. All of these goals(as in any cost-based formulation) can be difficult tobalance, thus we augment the D-RHC decision-makingframework with cost adaptation to learn the optimalD-RHC objective under varying conditions based onthe aforementioned priors inspired from related D-RHCwork.

For evaluating adaptation to varying conditions, weconstruct a 3D simulation capable of mimicking packetloss and communication delays between agents. Throughour simulator, we show the feasibility of our approachby accomplishing a cooperative exploration task. Wedemonstrate that the cost adaptation in our systemallows for safe and efficient exploration even underlarge communication delays. The swarm can actively re-adapt to keep a safer distance between agents or even

arX

iv:1

709.

0711

4v2

[cs

.AI]

30

Sep

2018

cluster together under negligible communication delays.We release our simulator and code to the community1.

II. RELATED WORK

Several coordinated swarm systems use centralizeddecision-making for coordination of complex taskswhere fine-grained control is needed [8], [9]. However,as ours does, many others focus on decentralized coor-dination of UAVs for swarm task completion. In [10],the authors present the “first decentralized multicopterflock that performs stable autonomous outdoor flight”. Adifferential update rule for the velocity is derived, basedon natural flocking rules and information belief aboutneighbouring swarm members. Other works favour con-tinual cost optimization to find the next optimal velocityor position based on the belief of the neighbours states,rather than differential updates. This is accomplishedby setting a time horizon and computing the optimalpredicted action at the horizon. The action is taken, in-formation is then gathered, and the optimization problemsolved again repeatedly. These are called decentralizedreceding horizon control (D-RHC) methods [4], [11].Other works also present similar methods for cost-basedswarm formulation for specific domains, including [1].In our work, we aim to move toward decentralizedcontrol by exchanging minimal coordination informationusing a mesh-network. This allows for extended swarmsand dynamic swarm partitioning and re-joining in localeswhere it may not be feasible to establish a centralcommand-and-control center. As such, we extend the D-RHC scheme posed in [4].

Recent work has also used meta-learning to generatea loss function in deep learning settings for classificationtasks [12]. We add to this expanding research area andlearn the D-RHC objective from a combination of humanpriors (pre-generated cost and constraint functions).

III. BACKGROUND AND PRELIMINARIES

In Receding Horizon Control (RHC) control systems,a constrained optimization problem is used to determinethe next action for a projected time frame ending witha “receding horizon” (i.e. the next timestep where anaction should be computed) [13]. RHC systems are usedacross many domains including decentralized swarmrobotics [14], [11]. A common theme throughout theRHC literature is its usage in mission critical systemswhere fast and reliable decision making is needed. Withan optimized and appropriate set of cost functions, ithas been shown that a system using RHC can “performnear its physical limits” [13] and, as such, is ideal forour application. In particular, we focus on D-RHC asa control scheme. D-RHC models can be generally for-mulated as solving a constrained optimization problem

1https://github.com/Breakend/SocraticSwarm

at each timestep. The D-RHC controller first predictsstate information of its neighbouring nodes and anyother variable data inputs at the next time horizon. Tomake these predictions it uses any model informationit may have about the optimization functions of theother controllers or simply current state information likevelocity. The system then solves the constrained costminimization problem to determine the optimal solutionat the next time horizon for a variable it can control,such as the velocity of the agent.

The “boids” model presented by Reynolds [7] isbased on natural flocking patterns in birds. Three corerules make up natural flocking. According to this work,each agent in a flock aims only to: “steer to avoidcrowding local flockmates” (separation), “steer towardsthe average heading of local flockmates” (alignment),and “steer to move toward the average position of localflockmates” (cohesion). By following these fundamen-tal laws, simulated agents can mimic natural flockingbehaviour often seen in birds. These have been usedas inspiration for cost formulation in some cost-basedmethods [10], [4], as we follow here.

IV. METHOD

For the central decision-making problem, we frameour work in the context of a coordinated explorationtask where the state space is easily divisible by anarea grid. In this formulation, the mesh-networkeddecision-making process can be viewed in four parts:(1) knowledge distribution and gathering; (2) global goalassignment through bidding within agents connectedthrough the mesh; (3) local action choice through cost-optimization; (4) meta-learning through cost adaptation.In the knowledge acquisition phase, agents broadcasttheir beliefs about their state to their neighbours. Inthe global goal assignment step, agents try to determinetheir next sub-goal via a decentralized bidding process.In the local action choice step, the information gainedthrough propagation and bidding is used to optimizethe local action choice used by low-level controllers viaD-RHC. In the meta-learning step, the provided set ofcosts are used to formulate a new D-RHC objective tosuit the properties of the environment that the agentsare in. We decompose the global goal assignment fromthe D-RHC process to simplify the problem based onprior work [15]. This also allows for more efficient taskcompletion – if the highest bidding areas are at separateends of the space, the swarm can decompose itself intoseveral swarms and rejoin later. A general outline of thiscan be seen in Algorithm 1.

A. Knowledge Propagation and Mesh Networking

We assume a fluid mesh network for our swarm:agents can drop in and out of the network without

Algorithm 1: Agent Decision Process

1 Divide task on-board each agent according to sameprocess into a set possible goals Gi associatedwith a cost Ci.

2 while Task Not Complete do3 while not WonBid do4 bid = CalculateNextBid()5 SendToNeighbours(bid)6 ReceiveAndCorrectNeighbourBids()7 WonBid = CompleteBidding()8 if WonBid then9 AddCost(bid.SubGoalCost)

10 end11 end12 while not Completed(bid.SubGoal) do13 belief=predictBeliefAt(ti+1)14 nextPos = optimize(costs, belief)15 applyVelocityToward(nextPos)16 broadcast(currentPosition)17 updates = ListenForPositionUpdates()18 updateBeliefState(updates)19 end20 end

affecting the rest of the swarm (e.g. if that is the mostoptimal action to take or if the agent is destroyed). Eachagent has the ability to send information to other swarmmembers about itself, the environment, or bidding re-sults. Before an agent makes a decision, it must processall new messages and directly update its knowledge base.Each agent keeps a record of its current belief about aneighbour’s position, goal, and the information about theworld.

At the start of the decision loop, agents broadcasttheir position to their immediate neighbours in the mesh.Incoming updates are processed and placed into theknowledge base. Similar to D-RHC, agents will try topredict their neighbours positions at the next horizonusing any position, velocity, and acceleration informa-tion in addition to the time elapsed since the messagewas sent (via timestamp on the information packet). Forbidding, messages are propagated throughout the meshand agents are made aware if new swarm members join.

B. Bidding

For global coordination, a task must be divided intosub-goals. In an exploration task, the sub-goals are for-mulated as searching an area subsection, or tile. To coor-dinate the distribution of these sub-goals, we formulate aconsensus bidding mechanism wherein agents: calculatetheir next desired sub-goal to complete, broadcast a bid,and enter an auction process until they claim a sub-goal.

All receiving agents keep a belief state of the state ofall sub-goals (e.g. in bidding, claimed, completed). Bidsare propagated through the mesh, unlike current agentpositions which are only sent to immediate neighbours.Agents are not guaranteed to know the entire biddingprocess due to packet delays or drops and must maketheir decisions in a decentralized manner based on theircurrent belief of the bidding process.

If an agent does not have a sub-goal assigned to it or acandidate sub-goal (where a bid has been submitted) itwill generate a new bid. A bid is calculated as gj =wdistDj + wnearλij based on [15] and the largest bidfrom all unclaimed sub-goals is chosen. In the contextof exploration, here Dj is the distance to the centre ofthe tile and λij is the cohesion factor from other agentsto the tile. This bid formulation encourages a choice ofsub-goals that are close the current position, with somespread-out factor from other agents. This helps to reducecontention for tiles, which increases search efficiency.

If an agent has the highest bid within some auctiontimeout, it claims the sub-goal. If there is a higher bid,it marks its own belief of the sub-goal as claimed byanother agent and re-bids. Once an agent completesa sub-goal, it sends this update to its neighbours forpropagation in the mesh as needed. With larger com-munication delays, race conditions can occur duringbidding. To address this, we use a simple method: if anagent receives information contrary to its own beliefs,it attempts to correct its current belief state or sendout corrective messages to the swarm. Some of thesescenarios are described in Figure 1. We find that if wechange bidding from propagation of messages throughthe mesh to the restricted sending of bids to immediateneighbours, the swarm is still able to complete tasksefficiently. This is because belief states are inherentlyupdated through correction during the bidding processas we discuss earlier. These corrections cascade throughthe network, effectively accomplishing selective propa-gation. This effect is further seen when new membersjoin a swarm or multiple swarms join together.

C. Local Action Choice through D-RHC

As aforementioned, D-RHC optimizes for the bestpossible position to be at the next time horizon giventhe cost function. While we rely on cost adaptationto provide the optimal cost function, we generate aset of initial priors for the guided search to modifyand combine. We formulate our costs with inspirationof previous work, aiming for finite, normalized, andinterpretable cost functions. We highlight, however, thatgiven any set of reasonable costs and constraints thecost adaptation method should converge to a functioningobjective.

We define three costs (priors) which the adaptation

Agent 1 Agent 2

Bid on ti

TimeoutClaimed ti

(a)

Agent 1 Agent 2Bid on ti

Higher Bid on ti

Gives up

TimeoutClaimed ti

(b)

Agent 1 Agent 2Bid on ti

Timeout

Claimed

Higher Bid on ti

Drops claim

TimeoutClaimed ti

(c)

Agent 1 Agent 2Bid on ti

Timeout

Claimed

Searched

Higher Bid on ti

Searched ti

Drops claim

(d)

Fig. 1. Illustrations of several bidding scenarios. (a) shows the common no contention bid. (b) shows two agents bidding over the same tilewhere a winner is achieved without race conditions. (c) illustrates an agent bidding on an already claimed tile. The agent with the lower biddrops its claim. (d) shows an agent claiming a searched tile. The agent that searched the tile corrects its neighbours’ knowledge.

method can used to generate the D-RHC objective:cohesion, safety, and goal. These are based on theformulations seen in [10], [4]. While we initiallyconsidered the rote boids formulation (i.e. alignment,separation, and cohesion), we empirically found that wecould combine separation and cohesion into one termand remove the alignment penalty for simplicity andreduction of the adaptation search space. We furthermodify the costs such that they are normalized between0 and 1. This ensures that the cost adaptation methoddoes not need to re-scale significantly over time basedon exploding costs.

Cohesion Our combined cohesion cost is:

f(dj , rc) =

{(2dj/rc)− 1 if dj < 0.75rc

Cpenalty otherwise(1)

cη = min

(1,

∑Nj=1 α

jf(dj , rc)

N

)(2)

Where dj = dist(pj , pi) and d1 ≤ d2 · · · ≤ dN(i.e. the distance are sorted in order). Here Ndesignates the local neighbour space (i.e. the numberof immediately connected neighbours in the mesh). rcis the communication range; α ∈ [0, 1] is a fading factor.

Safety Cost While the safety cost can adapt to manycontexts, here we simply formulate it as keeping a safealtitude off of the ground. Note, that we explicitly modelcollision with other agents as a constraint, while altitudeis a cost. This is by design, as in the case of landing,the altitude safety cost can be decreased.

cz = min

(1,

{((zi/zmin)− 1)2 if zmin > zi

((zi − zmin)/zmax)2 otherwise

)(3)

Here, zi is the altitude to be tested and zmin, zmax arethe bounds for the altitude.

Goal Cost Lastly, we must formulate a goal cost associ-ated with the completion of a sub-goal. Once a sub-goalhas been claimed, this goal cost is added to the D-RHCoptimization problem, driving task completion in localactions. This is formulated as:

cg = min

(1,

2

πarctan

(dist(pi, pgoal)

Cdist

))(4)

Where pi is the agent’s position to be tested and pgoalis the position of the centre of the goal tile which theagent intends to search. Cdist is a centroid point whichdetermines the slope of the cost according to arctan.

Full Optimization Problem Thus our modifiable swarmaction decision process can be viewed as a weighted sumof costs resulting in the optimization problem:

min (wηcη + wzcz + wgcg)s.t. pt+1

i < pti + (vti + atiδt)δt∀j∈neighbours dist(pt+1

i , pt+1j ) > ∆min

(5)

Here, wη, wy, wg are the cost weights. vti is the currentvelocity, ati is the current acceleration, δt is the timeuntil the next horizon, pt+1

i is the next position withcost to be minimized, pti is the current position, pt+1

j isa neighbour’s predicted position at the next horizon, and∆min is some minimum distance kept between an agentand any neighbour.

Figure 2 shows the functions modelled by these costs.Note, that we formulate collision as a constraint, bound-ing the next optimal position such that it is not withinsome sphere of radius ∆min surrounding an agent. Forhigher communication delays, this ∆min can be adaptedto prevent agents from risking collision with an uncertain

0 rc/2 rc0

1Cpenalty

(a) cη0 zmin zmax

0

1

(b) cz0 1

0

1

(c) cgFig. 2. Visualizations of each of the three cost functions. cη , cz and cg describe the cohesion, safety and goal cost respectively.

agents position. Furthermore, we bound the search spaceto only points that are reachable by the agent by thenext horizon based on the agent’s current velocity andacceleration.

To minimize our D-RHC objective, we attemptedseveral different techniques. We investigated ParticleSwarm Optimization with constraints and DifferentialEvolution with constraints2. Both PSO and DE per-formed well, but DE with constraints proved to be theleast computationally intensive approach with acceptablesolutions. As such, we simply use this method as a blackbox for constrained optimization of D-RHC problem.

V. COST ADAPTATION

Finally, we introduce a meta-learning process – costadaptation – which uses the human generated prior costs(defined previously) to generate a new D-RHC objec-tive. Cost adaptation optimizes an auxiliary heuristicin successive trials based on swarm performance underthe current system dynamics (such as conditions withheavy packet loss) through a guided search mechanism.In the previously defined set of human priors (thecosts and constraints), the variables in the meta-learningsearch space are: wη, wy, wg,∆min, Cpenalty ∈ Θ. Eachvariable in the cost adaptation problem is bounded suchthat θ = [Θmin,Θmax], where all are bounded between 0and 1, except for ∆min, which has a large bound of 100.We evaluate each trial using a set of costs and constraintsaccording to a heuristic, Ec. We generate this heuristicbased on our prior intuition, just as a reward is generatedin many reinforcement learning domains [16]. Generally,we try to normalize each part of the heuristic functionbetween 0 and 1, re-weighting portions by importance.We give penalties for collisions between agents andbonuses for completing tasks quickly and completely.

Ec = ctime + 2cnsrh + 4cclsn (6)

ctime =ttrial

tmax(7)

cclsn =

{0.25 + 0.75 nclsn

nagents, nclsn > 0

0 , otherwise(8)

2Both PSO and DE from C# framework found at:http://www.hvass-labs.org/projects/swarmops/cs/

Here, pcomplete is the fraction of the task completed(for the coordinated exploration task this is the per-centage of tiles searched); nclsn is the number of agentcollisions with the total number of agents being: nagents;ttrial is the time it took to complete a trial and tmax isthe maximum allowed trial time.

While we initially formulated the cost adaptationproblem with simulated annealing [17], we found this tobe a sample inefficient search method. Since the desire isto find a suitable D-RHC objective in as few simulationsas possible, we turned to Adaptive Simulated Annealing(ASA) [5], [18]. We found poor convergence propertieswith the rote version of ASA, as such, we modify it forour own purposes, seen in Algorithm 2. We choose atemperature decay of .95 and a re-annealing schedule of25 according experimental search performance.

Algorithm 2: Adapting Costs and ConstraintsInput : w ∈W for all weights assigned to costs

and constraints. wmin, wmax,∀w ∈WOutput: w ∈ W

1 Set a temperature TC = T0 = .95−MAX TRIALS

2 Initialize w0 = w = w;∀w ∈W, w ∈ W , w ∈ W3 Set Emin =∞4 for i = 0, 1, 2, . . . ,MAX TRIALS do5 Sample wi+1 ∈ Wi+1 with the random variable

Yi ∼ U [(wmin − w)TC

T0, (wmax − wi)TC

T0]

according to:

wi+1 = wi + yi;wi+1 ∈Wi+1, wi ∈Wi

Run trial with W and evaluate heuristic Ec.

6 if Ec < Emin or eEcprev−Ec

TC > v ∼ U [0, 1]then

7 Emin = EcSet w = wi;∀wi ∈ Wi, w ∈ W

8 end9 Set TC = T0(.95)i

10 if i mod 25 = 0 then11 Re-anneal, by setting TC = T012 end13 end

TrialRunner

WorldState

CommManager

Agent

Communication

KnowledgeBase

BiddingKnowledge

PossibleGoals

Goal Bids

RectZoneTile

SensorsAltimeter

GPS

DecisionManager

Cost ManagerCohesion Cost

Safety Cost

Goal Cost

FlightController

Fig. 3. Simplified class diagram of the simulation structure. The colours green, blue and orange represent the Learning, World State andAgent modules respectively. The Learning module consists of the trial runner, responsible for conducting a single trial. The World State modulefacilitates inter-agent communication and maintains the true state of the task. It also serves as the communication manager. The Agent moduleconsists of several sub modules designed to support each of the required functionality from movement to decision making. Agents communicatewith each other through the world module.

VI. EXPERIMENTS

To validate the benefits of cost adaptation, weformulate several sets of experiments where we use adefault setting (optimized for the base problem with novariation) and compare it against cost adaptation undervarious environment perturbations. In these experiments,we use an example coordinated exploration problem.Here, the task is to search a given a 600m × 400marea. This area can be divided into 384 (25m × 25m)tiles. To search a tile properly, an agent must be adistance of 5 meters or less from the centre of the tileand 40 meters above the ground. This is to simulatetaking an aerial photograph. The goal is to search theentire space in a decentralized manner in the leastamount of time possible without collisions. The timeoutfor tiles after the first bid (auction time, tauction) was0.5 seconds. Agents update their next desired locationevery tupdate = 0.05s and send agent update messageevery tbroadcast = 0.25s. The communication range wasset to 200m based on several known 802.11s capablechipsets [19]. The maximum velocity of an agent wasset to 40ms−1 in any direction. With maximum verticalacceleration at 6ms−2 and horizontal acceleration at3ms−2. These settings allow for a balance betweenrealistic dynamics and speed of simulation.

Simulator Several promising 3D multi-agent roboticssimulators exist, including Stage [20] and Argos [21].However, we found that existing simulators provideda large amount of overhead in fast implementation ofmesh-networked agents or were poorly maintained. In-stead, we built our simulation on the actively maintainedUnity3D game engine3. Unity supports basic physicssimulations, networking, and all other necessary compo-nents for our swarm simulation including the ability toadd visual cues which indicate algorithm performance.

3https://unity3d.com

The overall architecture of the simulator encompassesthree main modules: Learning, World State, and Agent.The Learning module facilitates the execution of mul-tiple trials for weight optimization and cost adaptation.The World State module is in charge of running a singletrial, simulated communication, and keeping track of theoverall world state. Finally, the Agent module consistsof several sub modules designed to simulate a swarmagent and decision making algorithms. A simplified classdiagram of the framework can be seen in Figure 3.

Each agent has a simulated GPS sensor for positionand altitude as well as an IMU for heading, velocity,acceleration. To simulate the real world inaccuracy ofsensors, random noise of up to two meters is added tothe agents’ position. We also add an simulated altime-ter based on ray-casting to the “sea-level”. A simpleflight controller is implemented which calculates thenecessary pitch, roll, yaw velocities to move towardthe next desired position. The maximum velocity andacceleration are scaled in the positive and negative ydirections representing the agility of the quad copterto gain or drop altitude quickly. Agents send eitherposition updates or bidding related messages through theWorldStateManager, which represents the mesh networkprotocol’s lower communication layers. Figure 4 showsan example of five agents in simulation4.

VII. RESULTS

A. Adaptation to Network Delays

Table I compares using increasing communicationdelays with five agents. The average completion time(µt) and variance (σ2

t ) across 100 simulated trials –along with the percent of area searched in the restrictedtime frame (300 seconds) and number of collisionsbetween agents in that time frame. Communication delaywas variable with the average delay as shown in the

4Video demo: https://youtu.be/2TUSXMo493I

Fig. 4. Five agents searching 384, 25× 25m2 tiles. Agent start in the bottom left corner and search clockwise around the grid as illustratedby the spread of the green tiles. Black lines show an agents position to where it believes each of the other agents to be. Agents tend to searchtiles at a comfortable distance to swarm. Red, blue, yellow, green are the states of the tiles (not claimed, in auction, claimed, searched).

TABLE ICOMMUNICATION DELAY TRIALS

Tdelay (s) µt (s) σ2t (s) µ% Searched µcollisions

Without AdaptationCentralized Flight Plan 107.08 0.47 100 0

0 175.15 48.26 100 00.2 177.30 42.79 100 00.4 174.68 53.80 100 00.8 171.60 49.73 100 01.6 198.86 370.13 100 0.123.2 221.56 449.73 98 2.25

With Adaptation0 164.44 53.86 100 0

0.2 162.22 45.12 100 00.4 168.15 56.72 100 00.8 171.34 33.14 100 01.6 195.93 67.26 100 03.2 191.07 77.45 100 0

table. The control scenario consists of each agent havinga pre-calculated flight plan which avoids all other agents.As the communication delay increases past the auctiontime of 0.5 seconds, collisions and slower search timesappear when the cost and nearness constraint is notproperly tuned for uncertainty in neighbouring position.

In the second half of Table I, costs were adaptedwith ASA for 50 iterations, then 100 trials were usedfor evaluation. As can be seen, performance drasticallyimproves as the D-RHC objective is adapted to completetasks in the safest and most efficient manner possible.While we note that the performance of the control in thisscenario is obviously faster, it is not robust to changinggoals and conditions (such as destruction of membersof the swarm). In our approach, the swarm recovers andpicks up other workers’ tasks if they are not completed.

B. Adaptation to Heterogenous Capabilities

To further expand on our analysis of the system’sability to adapt to new conditions, we investigate usingheterogeneous mixtures of agents. For these purposes,we spawn 10 agents and limit trials to 100 seconds suchthat goal is to complete as many tiles as possible in thistime with heterogeneous dynamics. We use the same384 tile grid and run 100 trials. We vary the maximumvelocity of the agents by adding a fixed Gaussian noiseof the form Z ∼ N (0, σmaxVel) to the maximum velocity

TABLE IIADAPTATION TO VELOCITY GAUSSIAN NOISE

σmaxV elocity µ% Searched σ2% Searched µcollisions

Without Adaptation10 51.4 % 3.9 % 020 30.7 % 7.4% 030 40.1 % 5.7% 0

With Adaptation10 63.2 % 3.4 % 020 53.2 % 7.9% 030 70.1 % 13.5% 0

TABLE IIIADAPTATION TO ACCELERATION GAUSSIAN NOISE

σacceleration µ% Searched σ2% Searched µcollisions

Without Adaptation.5 51.4 % 1.8% 01.0 61.1% 3.3% 02.0 67.2% 2.4% 02.5 51% 12.15% 0

With Adaptation.5 63.3% 0.2% 01.0 70.1% 0.8& 02.0 73.6% 3.6% 02.5 77.9% 6.7% 0

of each agent at the start of the experiment such thatevery agent has different flight dynamics.

We use the original parameters used for the resultsseen Table II and perform 50 iterations of ASA to re-adapt our D-RHC cost formulation before our 100 trialiterations. We similarly add a fixed Gaussian noise of theform Z ∼ N (0, σacc) to the acceleration of each agent inanother set of experiments. As demonstrated in Tables IIand III, we find that in nearly every case, adaptationsignificantly improves performance. At higher variancesin heterogeneity, we found that the non-adaptive swarmsometimes tried to to stick with a slower agent and endedup hurting efficiency. In these cases, the adaptive swarmfound converged to a D-RHC objective which left behindlagging members by lowering the cohesion cost.

C. Scalability

Table IV shows the scalability of cost adaptation tovarying number of agents. The communication delaywas fixed at zero and trials were simulated 100 times

TABLE IVTRIALS ON SCALABILITY TO NUMBER OF AGENTS.

Agents µt (s) σ2t (s) µcollisions

5 175.15 48.26 010 97.73 23.84 015 95.00 25.00 020 70.66 4.80 025 51.60 3.30 030 49.37 2.32 0

each after adaptation. As more agents are added, wesee search times decrease as expected, without anycollisions. However, at around 30 agents, we find thatfor our particular grid test case, performance gains beginto level off. This is expected, as the grid is rectangularof size 25, a line of 25 agents can reach maximumefficiency by sweeping across the grid (as they do).

VIII. DISCUSSION

We demonstrate a meta-learning method of costadaptation which leverages current swarm performanceto generate an optimization problem for D-RHC. Wedemonstrate that cost adaptation successfully conformsto new environment conditions. We build a simulator ontop of Unity3D, evaluate our system on a coordinatedexploration task, and release all code to the public.

The benefits and applications of our method to anyrobotic system are clear. The adaptive nature of ourformulation allows for heterogeneous mixtures of agentswith different capabilities and goals to participate in theswarm. This has the potential to be used in robot convoy-ing [22], cooperative exploration [23], or heterogeneouscoordination for marine monitoring [9]. Our methodol-ogy follows recent trends in meta-learning [12] and canbe built upon for similar approaches in control settingswith varying degrees of prior knowledge incorporated(e.g. a neural network approximator could model the D-RHC objective with no human priors). This work is eas-ily expandable and a base for future implementations ofadaptable systems. The heuristic-based cost adaptationmethods here can leverage reinforcement learning in thefuture for more efficient online learning. The robustnessof the methods we present here are building blocks forfuture advances in swarm behaviour and control systems.

REFERENCES

[1] M. Schwager, B. J. Julian, M. Angermann, and D. Rus, “Eyesin the sky: Decentralized control for the deployment of roboticcamera networks,” Proceedings of the IEEE, vol. 99, no. 9, pp.1541–1561, 2011.

[2] J. Banfi, A. Q. Li, I. Rekleitis, F. Amigoni, and N. Basilico,“Strategies for coordinated multirobot exploration with recurrentconnectivity constraints,” Autonomous Robots, pp. 1–20, 2017.

[3] I. Rekleitis, V. Lee-Shue, A. P. New, and H. Choset, “Limitedcommunication, multi-robot team based coverage,” in Roboticsand Automation, 2004. Proceedings. ICRA’04. 2004 IEEE Inter-national Conference on, vol. 4. IEEE, 2004, pp. 3462–3468.

[4] T. Keviczky, F. Borrelli, K. Fregene, D. Godbole, and G. J. Balas,“Decentralized receding horizon control and coordination ofautonomous vehicle formations,” IEEE Transactions on ControlSystems Technology, vol. 16, no. 1, pp. 19–33, 2008.

[5] L. Ingber., “Adaptive Simulated Annealing (ASA): LessonsLearned,” Control and Cybernetics, vol. 25, pp. 33–54, 1996.

[6] J. Hwangbo, I. Sa, R. Siegwart, and M. Hutter, “Control ofa quadrotor with reinforcement learning,” IEEE Robotics andAutomation Letters, vol. 2, no. 4, pp. 2096–2103, 2017.

[7] C. W. Reynolds, “Flocks, Herds and Schools: A DistributedBehavioral Model,” SIGGRAPH Comput. Graph., vol. 21,no. 4, pp. 25–34, Aug. 1987. [Online]. Available: http://doi.acm.org/10.1145/37402.37406

[8] J. A. Preiss, W. Honig, G. S. Sukhatme, and N. Ayanian,“Crazyswarm: A large nano-quadcopter swarm,” in Robotics andAutomation (ICRA), 2017 IEEE International Conference on.IEEE, 2017, pp. 3299–3304.

[9] F. Shkurti, A. Xu, M. Meghjani, J. C. G. Higuera, Y. Gird-har, P. Giguere, B. B. Dey, J. Li, A. Kalmbach, C. Prahacs,et al., “Multi-domain monitoring of marine environments usinga heterogeneous robot team,” in Intelligent Robots and Systems(IROS), 2012 IEEE/RSJ International Conference on. IEEE,2012, pp. 1747–1753.

[10] G. Vasarhelyi, C. Viragh, G. Somorjai, N. Tarcai, T. Szorenyi,T. Nepusz, and T. Vicsek, “Outdoor Flocking and FormationFlight with Autonomous Aerial Robots,” in IEEE/RSJ Interna-tional Conference on Intelligent Robots and Systems (IROS).IEEE, 2014, pp. 3866–3873.

[11] T. Keviczky, F. Borrelli, and G. J. Balas, “Decentralized Reced-ing Horizon Control for Large Scale Dynamically DecoupledSystems,” Automatica, vol. 42, no. 12, pp. 2105–2115, 2006.

[12] Y. Fan, F. Tian, T. Qin, X.-Y. Li, and T.-Y. Liu, “Learning toteach,” in International Conference on Learning Representations,2018.

[13] J. Mattingley, Y. Wang, and S. Boyd, “Receding Horizon Con-trol,” Control Systems, IEEE, vol. 31, no. 3, pp. 52–65, 2011.

[14] T. Keviczky, K. Fregene, F. Borrelli, G. J. Balas, and D. Godbole,“Coordinated Autonomous Vehicle Formations: Decentralization,Control Synthesis and Optimization,” in American Control Con-ference, 2006, 2006, pp. 6–pp.

[15] W. Sheng, Q. Yang, J. Tan, and N. Xi, “Distributed Multi-RobotCoordination in Area Exploration,” Robotics and AutonomousSystems, vol. 54, no. 12, pp. 945–955, 2006.

[16] R. S. Sutton and A. G. Barto, Reinforcement learning: Anintroduction. MIT press Cambridge, 1998, vol. 1, no. 1.

[17] P. J. Van Laarhoven and E. H. Aarts, Simulated Annealing.Springer, 1987.

[18] L. Ingber, A. Petraglia, M. R. Petraglia, and M. A. S. Machado,“Adaptive Simulated Annealing,” in Stochastic Global opti-mization and Its Applications with Fuzzy Adaptive SimulatedAnnealing. Springer, 2012, pp. 33–62.

[19] L. Li, X. Hu, and B. Zhang, “A routing algorithm for wifi-basedwireless sensor network and the application in automatic meterreading,” Mathematical Problems in Engineering, 2013.

[20] R. Vaughan, “Massively Multi-Robot Simulation in Stage,”Swarm Intelligence, vol. 2, no. 2-4, pp. 189–208, 2008.

[21] C. Pinciroli, V. Trianni, R. O’Grady, G. Pini, A. Brutschy,M. Brambilla, N. Mathews, E. Ferrante, G. Di Caro, andF. Ducatelle, “ARGoS: A Modular, Parallel, Multi-Engine Simu-lator for Multi-Robot Systems,” Swarm Intelligence, vol. 6, no. 4,pp. 271–295, 2012.

[22] F. Shkurti, W.-D. Chang, P. Henderson, M. J. Islam, J. CamiloGamboa Higuera, J. Li, T. Manderson, A. Xu, G. Dudek, andJ. Sattar, “Underwater multi-robot convoying using visual track-ing by detection,” in Proc. of The IEEE International Conferenceon Intelligent Robots and Systems (IROS), 2017.

[23] C. Trevai, Y. Fukazawa, J. Ota, H. Yuasa, T. Arai, andH. Asama, “Cooperative exploration of mobile robots usingreaction-diffusion equation on a graph,” in Robotics and Automa-tion, 2003. Proceedings. ICRA’03. IEEE International Confer-ence on, vol. 2. IEEE, 2003, pp. 2269–2274.