Distributed Model Shaping for Scaling to Decentralized POMDPs with hundreds of agents

D-TREMOR - AAMAS2011 1

Distributed Model Shaping for Scaling to Decentralized POMDPs

with hundreds of agents

Prasanna Velagapudi

Pradeep Varakantham

Paul Scerri

Katia Sycara


Motivation

• 100s to 1000s of robots, agents, people

• Complex, collaborative tasks• Dynamic, uncertain

environment• Offline planning

Search & Rescue Military C2

Disaster Response

ConvoyPlanning


Motivation

• Exploit three characteristics of these domains1. Explicit Interactions

• Specific combinations of states and actions where effects depend on more than one agent

2. Sparsity of Interactions• Many potential interactions could occur between agents• Only a few will occur in any given solution

3. Distributed Computation• Each agent has access to local computation• A centralized algorithm has access to 1 unit of computation• A distributed algorithm has access to N units of computation


Review: Dec-POMDP

: Joint Transition

: Joint Reward

: Joint Observation

1

2


Distributed POMDP with Coordination Locales

[Varakantham, et al 2009]

CL =Nature of time constraint (e.g. affects only same-time, affects any future-

time)

Relevant region of joint state-action space

Time constraint


CL =

:

:




Decentralized auction

EVA POMDP solver

Policy sub-sampling and Coordination Locale (CL)

messages

Prioritized/randomized reward and transition shaping

D-TREMOR (extending TREMOR [Varakantham, et al 2009])

Task Allocation

Local Planning

Interaction Exchange

Model Shaping


D-TREMOR: Task Allocation

• Assign “tasks” using decentralized auction– Greedy, nearest allocation

• Create local, independent sub-problem:


D-TREMOR: Local Planning

• Solve using off-the-shelf algorithm (EVA)

• Result: locally-optimal policies


D-TREMOR: Interaction Exchange

Find PrCLi and ValCLi:

• Send CL messages to teammates:

No collision

Collision

+1

-6

ValCLi= -7

[Kearns 2002]

Entered corridor in 95 of 100 runs:

PrCLi= 0.95


D-TREMOR: Model Shaping

• Shape local model rewards/transitions based on interactions

11

Probability of interaction

Interactionmodel functions

Independentmodel functions


D-TREMOR: Local Planning (again)

• Re-solve shaped local models to get new policies

• Result: new locally-optimal policies new interactions

12


D-TREMOR: Adv. Model Shaping

• In practice, we run into three common issues faced by concurrent optimization algorithms:– Slow convergence– Oscillation– Local optima

• We can alter our model-shaping to mitigate these by reasoning about the types of interactions we have



• Slow convergence Prioritization– Assign priorities to agents, only model-shape collision

interactions for higher priority agents

– Can quickly resolve purely negative interactions• Negative interaction: when every agent is guaranteed to have a

lower-valued local policy if an interaction occurs



• Oscillation Probabilistic shaping– Often caused by time dynamics between agents

• Agent 1 shapes based on Agent 2’s old policy• Agent 2 shapes based on Agent 1’s old policy

– Each agent only applies model-shaping with probability δ [Zhang 2005]

– Breaks out of cycles between agent policies



• Local Optima Optimistic initialization– Agents cannot detect mixed interactions (e.g. debris)

• Rescue agent policies can only improve if debris is cleared• Cleaner agent policies can only worsen if they clear debris

I’m not going near the

debrisIf no one is going through debris, I

won’t clear it

I’m not clearing the

debris





– Let each agent solve an initial model that uses an optimistic assumption of interaction condition


Experimental Setup

• D-TREMOR policies– Max-joint-value– Last iteration

• Comparison policies– Independent– Optimistic– Do-nothing– Random

• Scaling:– 10 to 100 agents– Random maps

• Density– 100 agents– Concentric ring maps

• 3 problems/condition• 20 planning iterations• 7 time step horizon• 1 CPU per agent

D-TREMOR produces reasonable policies for 100-agent planning problems in under 6 hrs.

(with some caveats)


Experimental Datasets

Scaling Dataset Density Dataset


Experimental Results: Scaling

Naïve Policies

D-TREMOR Policies


Experimental Results: Density

D-TREMOR rescues the most victims D-TREMOR does not

resolve every collision+10 ea. -5 ea.


Experimental Results: Time

Increase in time related to # of CLs, not # of agents

# of

CLs

Acti

ve


Conclusions

• D-TREMOR: Decentralized planning for sparse Dec-POMDPs with many agents

• Demonstrated complete distributability, fast heuristic interaction detection, and local message exchange to achieve high scalability

• Empirical results in simulated search and rescue domain


Future Work

• Generalized framework for distributed planning under uncertainty through iterative message exchange

• Optimality/convergence bounds

• Reduce necessary communication• Better search over task allocations• Scaling to larger team sizes


Questions?



Motivation

• Scaling planning to large teams is hard– Need to plan (with uncertainty) for each agent in team– Agents must consider the actions of a growing number of

teammates– Full, joint problem has NEXP complexity [Bernstein 2002]

• Optimality is going to be infeasible• Find and exploit structure in the problem• Make good plans in reasonable amount of time


Motivation

• Exploit three characteristics of these domains1. Explicit Interactions

• Specific combinations of states and actions where effects depend on more than one agent

2. Sparsity of Interactions• Many potential interactions could occur between agents• Only a few will occur in any given solution

3. Distributed Computation• Each agent has access to local computation• A centralized algorithm has access to 1 unit of computation• A distributed algorithm has access to N units of computation



Do-nothing does the best?

Ignoring interactions = poor performance



Why is this increasing?


Related WorkSc

alab

ility

Opti

mal

ity Generality EDI-CRTD-Dec-POMDP

TREMOR

Dynamic Networks

JESP

Prioritized Planning

D-TREMOR

DPC

OC-Dec-MDP

SPIDER

Structured Dec-(PO)MDP planners– JESP

[Nair 2003]

– TD-Dec-POMDP[Witwicki 2010]

– EDI-CR[Mostafa 2009]

– SPIDER[Marecki 2009]

• Restrict generality slightly to get scalability

• High optimalityOptimal Decoupling


Related WorkSc

alab

ility

Opti

mal


TREMOR

Dynamic Networks

JESP


D-TREMOR

DPC

OC-Dec-MDP

SPIDER

Heuristic Dec-(PO)MDP planners– TREMOR

[Varakantham 2009]

– OC-Dec-MDP[Beynier 2005]

• Sacrifice optimality for scalability

• High generality

Optimal Decoupling


Related WorkSc

alab

ility

Opti

mal


TREMOR

Dynamic Networks

JESP


D-TREMOR

DPC

OC-Dec-MDP

SPIDER

Optimal Decoupling

Structured multiagent path planners– DPC

[Bhattacharya 2010]

– Optimal Decoupling[Van den Berg 2009]

• Sacrifice generality further to get scalability

• High optimality


Related WorkSc

alab

ility

Opti

mal


TREMOR

Dynamic Networks

JESP


D-TREMOR

DPC

OC-Dec-MDP

SPIDER

Optimal Decoupling

Heuristic multiagent path planners– Dynamic Networks

[Clark 2003]

– Prioritized Planning[Van den Berg 2005]

• Sacrifice optimality to get scalability


Scal

abili

ty

Opti

mal

ity Generality

Related Work

EDI-CRTD-Dec-POMDP

TREMOR

Dynamic Networks

JESP

Our approach:

• Fix high scalability and generality

• Explore what level of optimality is possible


D-TREMOR

DPC

OC-Dec-MDP

SPIDER

Optimal Decoupling


A Simple Rescue Domain

Rescue Agent

Cleaner Agent

Narrow Corridor

Victim

Unsafe Cell

Clearable Debris


A Simple (Large) Rescue Domain


Distributed POMDP with Coordination Locales (DPCL)

• Often, interactions between agents are sparse

Only fits one agent Passable if

cleaned



Distributed, Iterative Planning

• Inspiration:– TREMOR

[Varankantham 2009]

– JESP[Nair 2003]

• Reduce the full joint problem into a set of smaller, independent sub-problems

• Solve independent sub-problems with local algorithm

• Modify sub-problems to push locally optimal solutions towards high-quality joint solution


Distributed Team REshaping of MOdels for Rapid execution (D-

TREMOR)

• Reduce the full joint problem into a set of smaller, independent sub-problems (one for each agent)

• Solve independent sub-problems with existing state-of-the-art algorithms

• Modify sub-problems such that local optimum solution approaches high-quality joint solution

Task Allocation

Local Planning


Model Shaping


Decentralized auction

EVA POMDP solver

Policy sub-sampling and Coordination Locale (CL)

messages

Prioritized/randomized reward and transition shaping

D-TREMOR (extending [Varakantham, et al 2009])

Task Allocation

Local Planning


Model Shaping


D-TREMOR: Task Allocation

• Assign “tasks” using decentralized auction– Greedy, nearest allocation

• Create local, independent sub-problem:


D-TREMOR: Local Planning

• Solve using off-the-shelf algorithm (EVA)

• Result: locally-optimal policies



Finding PrCLi

• Evaluate local policy

• Compute frequency of associated si, ai

[Kearns 2002]:

Entered corridor in 95 of 100 runs:

PrCLi= 0.95



Finding ValCLi

• Sample local policy value with/without interactions– Test interactions independently

• Compute change in value if interaction occurred

No collision

Collision

+1

-6

ValCLi= -7

[Kearns 2002]:



• Send CL messages to teammates:

• Sparsity Relatively small # of messages


D-TREMOR: Model Shaping

• Shape local model rewards/transitions based on remote interactions

47

Probability of interaction

Interactionmodel functions

Independentmodel functions


D-TREMOR: Local Planning (again)

• Re-solve shaped local models to get new policies

• Result: new locally-optimal policies new interactions

48



• In practice, we run into three common issues faced by concurrent optimization algorithms:– Slow convergence– Oscillation– Local optima

• We can alter our model-shaping to mitigate these by reasoning about the types of interactions we have



• Slow convergence Prioritization– Majority of interactions are collisions

– Assign priorities to agents, only model-shape collision interactions for higher priority agents

– From DPP: prioritization can quickly resolve collision interactions

– Similar properties for any purely negative interaction• Negative interaction: when every agent is guaranteed to have a

lower-valued local policy if an interaction occurs



• Oscillation Probabilistic shaping– Often caused by time dynamics between agents

• Agent 1 shapes based on Agent 2’s old policy• Agent 2 shapes based on Agent 1’s old policy

– Each agent only applies model-shaping with probability δ [Zhang 2005]

– Breaks out of cycles between agent policies





PrCL = low, ValCL = lowIf (ValCL = low):

optimal policy do nothing

PrCL = low, ValCL = low





– Let each agent solve an initial model that uses an optimistic assumption of interaction condition


Experimental Setup

• D-TREMOR policies– Max-joint-value– Last iteration

• Comparison policies– Independent– Optimistic– Do-nothing– Random

• Scaling:– 10 to 100 agents– Random maps

• Density– 100 agents– Concentric ring maps

• 3 problems/condition• 20 planning iterations• 7 time step horizon• 1 CPU per agent

D-TREMOR produces reasonable policies for 100-agent planning problems in under 6 hrs.

(with some caveats)


Experimental Datasets

Scaling Dataset Density Dataset


Experimental Results: Scaling

Naïve Policies

D-TREMOR Policies



Do-nothing does the best?

Ignoring interactions = poor performance



D-TREMOR rescues the most victims D-TREMOR does not

resolve every collision+10 ea. -5 ea.



Why is this increasing?



Increase in time related to # of CLs, not # of agents


Conclusions

• D-TREMOR: Decentralized planning for sparse Dec-POMDPs with many agents

• Demonstrated complete distributability, fast heuristic interaction detection, and local message exchange to achieve high scalability

• Empirical results in simulated search and rescue domain


Conclusions

D-TREMOR produces reasonable policies for 100-agent planning problems in under 6 hrs.– Partially-observable, uncertain world– Multiple types of interactions & agents

• Improves over independent planning• Resolved interactions in large problems• Still some convergence/efficiency issues


DPCL vs. other models

• EDI/EDI-CR– Adds complex transition functions

• TD-Dec-MDP– Allows simultaneous interaction (within epoch)

• Factored MDP/POMDP– Adds interactions that span epochs


D-TREMOR


D-TREMOR


D-TREMOR: Reward functions

• Probability that a debris will not allow a robot to enter the cell: – P_Debris = 0.9;

• Probability of action failure– P_ActionFailure = 0.2;

• Probability that success is observed if the action succeeded.– P_ObsSuccessOnSuccess = 0.8;

• Probability that success is observed if the action failed– P_ObsSuccessOnFailure = 0.2;

• Probability that a robot will return to the same cell after collision– P_ReboundAfterCollision = 0.5;

• Reward of saving a victim– R_Victim = 10.0;

• Reward of cleaning debris– R_Cleaning = 0.25;

• Reward of moving– R_Move = -0.5;

• Reward of observing– R_Observe = -0.25;

• Reward for a collision– R_Collision = -5.0;

• Reward for landing in an unsafe cell– R_Unsafe = -1;


Review: POMDP

+100

-10

: Set of States

: Set of Actions

: Set of Observations

: Transition function

: Reward function

: Observation function



[Varakantham, et al 2009]• Extension of Dec-POMDP which modifies ,• Coordination locales (CLs) represent interactions:

Explicit time

Explicit time constraint

Implicitly construct interaction functions

CL =


Proposed Approach: DIMSDistributed Iterative Model Shaping

Task Allocation

Local Planning


Model Shaping

• Assign tasks to agents• Reduce search space considered by agent• Define local sub-problem for each robot



Task Allocation

Local Planning


Model Shaping

• Assign tasks to agents• Reduce search space considered by agent• Define local sub-problem for each robot

Full SI-Dec-POMDP

Local (Independent) POMDP



Task Allocation

Local Planning


Model Shaping

• Solve local sub-problems using off-the-shelf centralized solver

• Result: Locally-optimal policy



Task Allocation

Local Planning


Model Shaping

• Given local policy: estimate local probability and value of interactions

• Communicate local probability and value of relevant interactions to team members

• Sparsity Relatively small # of messages



Task Allocation

Local Planning


Model Shaping

• Modify local sub-problems to account for presence of interactions



Task Allocation

Local Planning


Model Shaping

• Reallocate tasks or re-plan using modified local sub-problem


Any decentralized allocation mechanism (e.g. auctions)

Stock graph, MDP, POMDP solver

Lightweight local evaluation and low-bandwidth messaging

Methods to alter local problem to incorporate non-local effects


Task Allocation

Local Planning


Model Shaping


Example: Interactions

Rescue robot

Cleaner robot

Debris

Victim


Example: Sparsity

Documents

Distributed Model Shaping for Scaling to Decentralized POMDPs with hundreds of agents