Multi-swarm Problem Multi-swarm Problem Solving in NetworksSolving in Networks

Tony WhiteTony Whiteemail: email: tony@sce.carleton.catony@sce.carleton.ca

OverviewOverview• IntroductionIntroduction

– What is it and why is it interesting?What is it and why is it interesting?

– Problem solving and stigmergy.Problem solving and stigmergy.

• Swarm problem solvingSwarm problem solving– Swarm agent architectureSwarm agent architecture

– Connection-oriented routingConnection-oriented routing

– Simple diagnosisSimple diagnosis

• Extensions and FutureExtensions and Future

What is Swarm What is Swarm Intelligence?Intelligence?• ““Swarm Intelligence is a property of Swarm Intelligence is a property of

systems of non-intelligent robots systems of non-intelligent robots exhibiting collectively intelligent exhibiting collectively intelligent behavior.” [Beni, 89]behavior.” [Beni, 89]

• Characteristics of a swarm:Characteristics of a swarm:– distributed, no central control or data source;distributed, no central control or data source;

– no (explicit) model of the environment;no (explicit) model of the environment;

– perception of environment, i.e. sensing;perception of environment, i.e. sensing;

– ability to change environment.ability to change environment.

What is Swarm What is Swarm Intelligence (cont.)?Intelligence (cont.)?• Swarm systems are examples of Swarm systems are examples of

behavior-based systems exhibiting:behavior-based systems exhibiting:

– multiple lower level competencies;multiple lower level competencies;

– situated in environment;situated in environment;

– limited time to act;limited time to act;

– autonomous with no explicit control autonomous with no explicit control provided;provided;

– problem solving is emergent behavior;problem solving is emergent behavior;

– strong emphasis on reaction and adaptation; strong emphasis on reaction and adaptation;

MotivationsMotivations

• Robust nature of animal problem-solvingRobust nature of animal problem-solving– simple creatures exhibit complex behavior;simple creatures exhibit complex behavior;

– behavior modified by dynamic environment. behavior modified by dynamic environment.

• Emergent behavior observed in:Emergent behavior observed in:– bacteriabacteria

– antsants

– beesbees

– ......

Emergent Problem SolvingEmergent Problem Solving

• For For Lasius NigerLasius Niger ants, [Franks, 89] ants, [Franks, 89] observed:observed:– regulation of 1 degree celcius range;regulation of 1 degree celcius range;

– forming bridges;forming bridges;

– raiding specific areas for food;raiding specific areas for food;

– building and protecting nest;building and protecting nest;

– sorting brood and food items;sorting brood and food items;

– cooperating in carrying large items;cooperating in carrying large items;

– emigration of a colony;emigration of a colony;

– finding shortest route from nest to food source;finding shortest route from nest to food source;

– preferentially exploiting the richest food source available.preferentially exploiting the richest food source available.

StigmergyStigmergy

• Indirect communication via interaction Indirect communication via interaction with environment [Grassé, 59]with environment [Grassé, 59]– Sematonic [Wilson, 75] stigmergySematonic [Wilson, 75] stigmergy

• action of agent directly related to problem action of agent directly related to problem solving and affects behavior of other agents.solving and affects behavior of other agents.

– Sign-based stigmergySign-based stigmergy• action of agent affects environment not action of agent affects environment not

directly related to problem solving activity. directly related to problem solving activity.

Ant ColonyAnt Colony

• Ants are behaviorally Ants are behaviorally unsophisticated; collectively perform unsophisticated; collectively perform complex tasks. complex tasks.

• Ants have highly developed Ants have highly developed sophisticated sign-based stigmergysophisticated sign-based stigmergy– communicate using pheromones;communicate using pheromones;

– trails are laid that can be followed by trails are laid that can be followed by other ants.other ants.

• Species lay pheromone trails travelling from Species lay pheromone trails travelling from nest, to nest or possibly in both directions.nest, to nest or possibly in both directions.

• pheromones evaporate.pheromones evaporate.

• pheromones accumulate with multiple ants pheromones accumulate with multiple ants using path.using path.

Pheromone TrailsPheromone Trails

Food sourceNest

Pheromone Trails Pheromone Trails continuedcontinued

D

E

H C

A

B

d=0.5

d=0.5

d=1.0

d=1.0

E

H

E

D

H C

A

B

30 ants

D

C

A

B

30 ants

15

ants

15

ants

15

ants

15

ants

30

ants

10

ants

20

ants

20

ants

10

ants

30

ants

T = 0 T = 1

Swarm operationSwarm operation

• Swarm agents:Swarm agents:– arrive at a node,arrive at a node,

– sense environment,sense environment,

– undertake local activity,undertake local activity,

– modify environment,modify environment,

– use sensory input to make migration use sensory input to make migration decisiondecision

Swarm Agent ArchitectureSwarm Agent Architecture

A=(E,R,C,MDF,m)

• Agents have a uniform architecture Agents have a uniform architecture consisting of five components:consisting of five components:

emitters (emitters (EE),), receptors (receptors (RR),), chemistry (chemistry (CC),), a migration decision function (a migration decision function (MDFMDF),), memory (memory (mm))

Multi-swarm ArchitectureMulti-swarm Architecture

EmittersEmitters

• Generators of chemical messages Generators of chemical messages ((EE))

Chemical modification

of environment

ReceptorsReceptors

• Sensors of chemical messages (Sensors of chemical messages (RR) ) from local environmentfrom local environmentDetector for local

environment

ChemicalsChemicals

• Chemicals are digitally-encoded using Chemicals are digitally-encoded using a {1, 0, #} alphabet.a {1, 0, #} alphabet.

• The # symbol unifies with 1 or 0.The # symbol unifies with 1 or 0.

• Chemicals have two attributes:Chemicals have two attributes:

– encodingencoding

– concentrationconcentration

• Chemicals participate in reactions.Chemicals participate in reactions.

ChemistryChemistry

• The set of chemical reactions (The set of chemical reactions (CC) ) that can operate on sensed and that can operate on sensed and locally stored chemicals.locally stored chemicals.

Chemical interactions

ZWYX

ZXYX

ZYX

YYX

nothingX

''

Examples of chemical Examples of chemical reactionsreactionsCatalytic breakdown of 011Catalytic breakdown of 011

0#10#1011

ZWTX

ZXTX

ZYT

YYT

TWYX

TXYX

TYX

Endothermic Endothermic reactionsreactions

Exothermic Exothermic reactionsreactions

Migration Decision Migration Decision FunctionFunction• The The MDFMDF is used to determine the next node is used to determine the next node

to visit in the networkto visit in the network

ppijijkk (t) = (t) = pp[T[Tijkpijkp(t) ](t) ]--kpkp[C(i,j)][C(i,j)]-- / / NNkk(i,j,t)(i,j,t)

NNkk(i,j,t) = (i,j,t) = j in A(i)j in A(i) pp[T[Tijkpijkp(t) ](t) ]- - kpkp[C(i,j)][C(i,j)]--

kpkp, , are control parameters are control parameters

NNkk(i,j,t) is a normalization term(i,j,t) is a normalization term

A(i) is the set of available egress linksA(i) is the set of available egress links

Multi-Swarm scenarioMulti-Swarm scenario

• Distributed Network Management Distributed Network Management employing delegation [Yemini, 91]employing delegation [Yemini, 91]

• Three interacting swarms:Three interacting swarms:– connection finding,connection finding,

– connection monitoring,connection monitoring,

– connection fault diagnosisconnection fault diagnosisA

B

C

D

E

Routing ProblemRouting Problem

• IdeaIdea– Ants dropping different pheromones Ants dropping different pheromones

used to compute “shortest” path from used to compute “shortest” path from source to destination(s);source to destination(s);

– more flexible adaptation to failures and more flexible adaptation to failures and network congestion;network congestion;

– use only local knowledge for routing use only local knowledge for routing and avoid costly communication of and avoid costly communication of state to all network nodes. state to all network nodes.

Why Routing?Why Routing?

• Conventional routing often relies on:Conventional routing often relies on:– global state available at all nodes;global state available at all nodes;– centralized control;centralized control;– fixed “shortest path” (Dijkstra) algorithms;fixed “shortest path” (Dijkstra) algorithms;– limited ability to deal with congestion or limited ability to deal with congestion or

failure.failure.

• Ideally, would like to have network Ideally, would like to have network adapt routing patterns to take adapt routing patterns to take advantage of free resources and move advantage of free resources and move existing traffic if possible. existing traffic if possible.

Routing ResearchRouting Research

• Three approaches so far investigated:Three approaches so far investigated:

– [White et al, 96+][White et al, 96+]

– [Schoonderwoerd et al, 97] (*)[Schoonderwoerd et al, 97] (*)

– [Di Caro and Dorigo, 97][Di Caro and Dorigo, 97]

• DifferencesDifferences

– Link cost metric constant in *Link cost metric constant in *

– Point to point traffic only in *Point to point traffic only in *

– AS parameter settings constant in *AS parameter settings constant in *

Routing agentsRouting agents

• Agent types:Agent types:– explorerexplorer

• used for route determinationused for route determination

– allocatorallocator• allocates resources in network when route allocates resources in network when route

emergedemerged

– deallocatordeallocator• deallocates resources in network at end of deallocates resources in network at end of

callcall

Point-2-Point ConnectionsPoint-2-Point Connections

• For explorer agents:For explorer agents:– At each node, they choose path with At each node, they choose path with

probability proportional to f(cprobability proportional to f(cee, p, pee););

– explorers visit edges once only (achieved explorers visit edges once only (achieved through use of tabu list);through use of tabu list);

– when destination reached, ants return along when destination reached, ants return along the path explored laying down pheromone the path explored laying down pheromone trail;trail;

– when explorers return a decision is made when explorers return a decision is made regarding path emergenceregarding path emergence

Path EmergencePath Emergence

• At source node (“nest”):At source node (“nest”):– store paths for previous m explorer agents;store paths for previous m explorer agents;

– when p% follow same path allocator agent is when p% follow same path allocator agent is sent to allocate bandwidth in network;sent to allocate bandwidth in network;

– explorer agents continue to look for new explorer agents continue to look for new (possibly better) paths.(possibly better) paths.

• Applies for one or many pt-2-pt Applies for one or many pt-2-pt connections: connections: – ants use different, non-reacting pheromones.ants use different, non-reacting pheromones.

Explorer agent algorithmExplorer agent algorithm1. Initialize1. Initialize

set t:= 0set t:= 0

For every edge (i,j) set an initial For every edge (i,j) set an initial value Tvalue Tijij(t) for trail intensity. (t) for trail intensity. Place m ants on the source Place m ants on the source node. [Generate new explorers node. [Generate new explorers at freq. eat freq. ef] f] ]]

2. Set s:= 1 { tabu list index)2. Set s:= 1 { tabu list index)for k:= 1 to m dofor k:= 1 to m do Place starting town of the kthPlace starting town of the kth

ant in tabuant in tabukk(s).(s).

3. Repeat until dest’n reached:3. Repeat until dest’n reached:

Set s := s + 1Set s := s + 1for k:=1 to m dofor k:=1 to m do Choose the node j to move toChoose the node j to move to

with probability pwith probability pijijkk (t) (t)

Move the kth ant to node j.Move the kth ant to node j.Update explorer route cost:Update explorer route cost:rrkk = r = rkk + c(i,j) + c(i,j)if (rif (rkk > r > rmaxmax)) kill explorerkill explorerkk

Insert town j in tabuInsert town j in tabukk(s).(s).At destination go to 4.At destination go to 4.

4. While s > 14. While s > 1 traverse edge (i,j)traverse edge (i,j) T(i,j) = T(i,j) + pT(i,j) = T(i,j) + pee s := s - 1s := s - 1

5. At source node do:5. At source node do:if (pathif (pathee = pathBuffer * d) = pathBuffer * d) create and send allocatorcreate and send allocator

if t > Tmax if t > Tmax create and send allocatorcreate and send allocator

Evaporation occurs concurrently with explorationEvaporation occurs concurrently with exploration

Point-2-Multipoint Point-2-Multipoint ConnectionsConnections• For j destinations, consider as j pt-2-For j destinations, consider as j pt-2-

pt connections with:pt connections with:– same pheromone, i.e. all explorers same pheromone, i.e. all explorers

communicate;communicate;

– j allocator agents only allocate j allocator agents only allocate bandwidth once;bandwidth once;

– allocator send decision made when % allocator send decision made when % of all j explorer ants agree on spanning of all j explorer ants agree on spanning tree.tree.

Routing FunctionRouting Function

Transition probability Transition probability (mdf)(mdf)::

ppijijkk (t) = [T (t) = [Tijkijk(t) ](t) ]--[C(i,j)][C(i,j)]-- / / NNkk(i,j,t)(i,j,t)

NNk k (i,j,t) = (i,j,t) = j in (S-Tabu(k))j in (S-Tabu(k)) [T [Tijkijk(t) ](t) ]--[C(i,j)][C(i,j)]--

, , are control parameters that determine the sensitivity are control parameters that determine the sensitivity of the algorithm to link cost and pheromone.of the algorithm to link cost and pheromone.

C(i,j) a C(i,j) a functionfunction that depends upon the that depends upon the type of traffictype of traffic, , the the lengthlength and and utilizationutilization of the link. of the link.

Allocator agentsAllocator agents

• Allocator agents can fail:Allocator agents can fail:

– bandwidth already allocated by time bandwidth already allocated by time allocator is sent;allocator is sent;

– allocator agent backtracks to source rolling allocator agent backtracks to source rolling back resource allocation and decreases back resource allocation and decreases pheromone levels;pheromone levels;

– decision to re-send allocator made at a decision to re-send allocator made at a later time (a backoff period is observed);later time (a backoff period is observed);

– explorer ants continue to search for routes.explorer ants continue to search for routes.

Connection monitoring Connection monitoring agentsagents• Connection’s quality of service Connection’s quality of service

monitored.monitored.

• Changes in QoS cause Connection Changes in QoS cause Connection monitoring agents to be sent into monitoring agents to be sent into network, laying down q-chemical.network, laying down q-chemical.

2

A

B

C

D

•E

Connection diagnosis Connection diagnosis agentsagents• Examples of Examples of netletsnetlets

• SenseSense q-chemicalq-chemical concentrations concentrations

2A

B

C

D

E)()(

)(/)()(

tQtN

tNtQtp

ijj

ijij

Mdf

Experimental Parameters Experimental Parameters (fixed)(fixed)• Number of ants to create = 20Number of ants to create = 20

• Frequency of creation = every 10 cyclesFrequency of creation = every 10 cycles

• Amount of pheromone dropped = 10Amount of pheromone dropped = 10

• pheromone evaporation rate = 0.9pheromone evaporation rate = 0.9

• Sample window size = 50Sample window size = 50

• Emergence criterion = 0.9Emergence criterion = 0.9

Routing ResultsRouting Results

• Shortest paths emerged quicklyShortest paths emerged quickly

• Mixed pt-2-pt, pt-2-mpt routes emergedMixed pt-2-pt, pt-2-mpt routes emerged

• Routing responds to changes in Routing responds to changes in environment:environment:– node failure;node failure;

– link failure;link failure;

– link cost changeslink cost changes

Parameter SensitivityParameter Sensitivity

• Bad solutions and stagnationBad solutions and stagnation– For high values of For high values of the algorithm enters the algorithm enters

stagnation behavior very quickly without stagnation behavior very quickly without finding very good solutions.finding very good solutions.

• Bad solutions and no stagnationBad solutions and no stagnation– too low, insufficient importance too low, insufficient importance

associated with trail.associated with trail.• Good solutionsGood solutions

– , , in the central area (1,1), (1,2), in the central area (1,1), (1,2), (1,5), (0.5, 5) (1,5), (0.5, 5)

Routing Results Routing Results

• Link cost functions: C(i,j)Link cost functions: C(i,j)

Five functions studied

experimentally:• constant• linear• linear threshold• quadratic• server (1/1-u)

At high occupancy (> 50%) server appeared to give the bestresults. At low occupancy (<25%) function relatively unimportant.

constantconstant

linearlinear

thresholdthreshold

quadraticquadratic

serverserver

DiagnosisDiagnosis

• Diagnosis agent is ‘hill climbing’ in Diagnosis agent is ‘hill climbing’ in space of q-chemical.space of q-chemical.

• Very quickly finds high q-chemical Very quickly finds high q-chemical concentrations and initiates concentrations and initiates diagnostic activity.diagnostic activity.

• An example of distributed diagnosis An example of distributed diagnosis through constructive chemical through constructive chemical interference.interference.

Self AdaptationSelf Adaptation

• pheromone and cost sensitivities pheromone and cost sensitivities shouldshould vary during search:vary during search:

– avoid premature convergence;avoid premature convergence;

– speed up search considerably.speed up search considerably.

• Explorers encode sensitivity values:Explorers encode sensitivity values:

– fitness of encoding is cost of route;fitness of encoding is cost of route;

– new agents are created with and use new agents are created with and use genetically-manipulated values for route genetically-manipulated values for route finding.finding.

Extensions to current m-Extensions to current m-Swarm systemSwarm system• Multi-class routing Multi-class routing

– Higher priority traffic causes pheromone Higher priority traffic causes pheromone levels of lower priority traffic to decay;levels of lower priority traffic to decay;

– automated re-routing of traffic performed.automated re-routing of traffic performed.

• Swarms learn to avoid regions of Swarms learn to avoid regions of network providing low quality network providing low quality connections.connections.

Other (potential) Other (potential) applications of m-Swarm applications of m-Swarm systemssystems• Behavior-based Network Management;Behavior-based Network Management;

– explorer agents allocate routes;explorer agents allocate routes;

– parent agents monitor “health” of explorers;parent agents monitor “health” of explorers;

– congestion agents identify global network congestion agents identify global network congestion;congestion;

– congestion agents signal re-planning of congestion agents signal re-planning of network;network;

– congestion agent pheromones react with congestion agent pheromones react with routing agent pheromones in order to cause re-routing agent pheromones in order to cause re-routing after network re-planned.routing after network re-planned.

Futuristic?Futuristic?

• Active networks are being researched at:Active networks are being researched at:– M.I.T.M.I.T.

– U. Penn.U. Penn.

– CMUCMU

– Georgia Tech.Georgia Tech.

• Management by delegation [Yemini, 91] Management by delegation [Yemini, 91] considered an essential design criterion for considered an essential design criterion for next generation network management next generation network management systems.systems.