Scalable Management for Networks and Services

Rolf Stadler

Laboratory for Communication NetworksKTH Royal Institute of Technology

Stockholm

HP Laboratories, Palo Alto, March 31, 2003

node

Managementstation

Managementstation

Manager-Agent based management

node

A A A

M

results

download &execute

P

Management Program

Manager

Agent

•Centralized Control•Management protocols: SNMP, CMIP•Program runs on Management Station

•Decentralized Control•Program runs on network nodes

P

The Shift of a Management Paradigm

AP

A A

Router

Execution Environment

Management station

Management Program

navigation Code Server

Architecture for Pattern-based Management

Weaver—A Testbed for pattern-based Management

WANA

WANB

WANC

WAND

Management Station

Router A Router B Router C Router D

FastEthernet Switch

Simple Navigation Patterns

1

21

2

1

2

2

2

3

3

34

operation typical application navigation pattern

type 1: node-to-node 1 node control/monitor(get/set of variables)

type 2: visit all nodesalong a path/flow

1 flow/path controle.g.: traceroute, bottleneckdetection, signalling, VPNoperation

type 3: distribute agentto all nodes in subnet(parallel control)

subnet controle.g.: topology detection

type 4: visit all nodes insubnet (sequent ialcontrol)

12

3

4

5

subnet control, messagebroadcaste.g.: congestion locationdetection

Echo Pattern (expansion)

droot=1

Echo Pattern (expansion)

droot=2

Echo Pattern (expansion)

droot=3

Echo Pattern (expansion)

droot=4

Echo Pattern (expansion)

Echo Pattern

droot=5

droot=4

Echo Pattern (contraction)

droot=3

Echo Pattern (contraction)

droot=2

Echo Pattern (contraction)

droot=1

Echo Pattern (contraction)

Echo Pattern (contraction)

Echo Pattern (contraction)

The Echo Pattern

• Two phases of traversal– expansion phase: explorers flood network with requests

for local operations– contraction phase: echoes return and aggregate results

• Properties– Generates balanced traffic load– Traffic load depends on network topology,

not on speed of traversal– Time complexity increases linearly with network

diameter.

Examples of Echo-based Management

• Get information on topology

– compute the current number of leaf nodes, the connectivity distribution

– discover current topology within 10 hops of node x

• Get information on network state

– identify 10 most congested links

– compute distribution of link utilization, queue lengths

– identify sub topologies with highly loaded links

– find a resource R closest to node x

Pattern-based Management—An Engineering Approach to Decentralized Management

• A management program consists of– A navigation pattern (distr. graph traversal algorithm)– An operation on nodes– An aggregation function

• Relevance of this approach– Provides a basis to analyze management operation for

performance, scalability, robustness– Supports concept of re-usable patterns, hides

complexity

Composing Management Programs

Segall

Echo Patterns

Navigation Patterns

Chang

Skip

Wait

Scope

Multi

Echo Aggregators

Res. Disc.

Aggregators

Leaf Count

Load. Hist.

Conn. Hist.

C LI

HT

TP

X ML

SN MP Local Operations

Node Access

Management Program

Properties of Patterns

Echo Aggregators

Res. Disc.

Aggregators

Leaf Count

Load. Hist.

Conn. Hist.

C LI

HT

TP

X ML

SN MP

Node Access

Management Program

Segall

Simple Echo Robust Echo Others

Echo Patterns

Navigation Patterns

Chang

Skip

Wait

Scope

Multi

• A pattern can be used for many management operations. • A pattern can be chosen according to performance objectives.• A pattern hides the complexity of a distributed operation.• Network failures can be handled within patterns.• Code mobility can be controlled.

visitedi : boolean init false;Gi : set of integers init neighbors();parenti : integer init -1;

Echo(inmsg: bytes, from: integer) { Gi := Gi - from; if visitedi = false { parenti := from; visitedi := true;

OnInitiate(inmsg, outmsg); if Gi != empty dispatch(parenti, outmsg, i); } else OnAggregate(inmsg); if Gi = empty {

OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}

The Interface between Pattern and Aggregator

OnBegin

OnTerminate

OnInitiate OnComplete

OnAggregate

…av_loadi := av_load;…

…av_load := load();n:=1;…

…av_load := (av_load*n + av_loadj)/(n+1);n:=n+1;…

SIMPSON: A SIMple Pattern Simulator fOr Large Networks

0

200000

400000

600000

800000

1e+06

1.2e+06

0 1 2 3 4 5 6T

raff

ic (b

yte

s)

Time (secs)

Traffic vs Time for 221 node grid network

"trace1.txt"

Analyzing Management Operations

A

CD

E

F

B

A

B C

DEF

E D F E

A

B D FEC

Network Graph G=(V,E) Execution Graphs G’=(V’,E’)

Centralized Management Distributed Management

Star Pattern Echo Pattern

Traffic Complexity of Management Operations

Amount of traffic placed on the network during execution.

C t ra f f ic hopcount v' childk v' Iq Ir+

v' V' 0 k ch i ldcou n t v'

=

Ctrafficecho

Iq Ir+ Edegree G 2 V–

2------------------------------------------- 1+ +

=

Ctrafficstar Iq Ir+ hopcount v'ro ot v'

v' V'

=

Time Complexity of Management Operations

Time needed from invocation until completion of a operation.

C time C time v'root =

Ctime v' tc tr+ if childcount v' 0=

tc tr M v' + + otherwise

=

M v' max ktq 2+ hopcount v' childk v' tl Ctime childk v' + 1 k childcount v'

=

Ctimes tar

O V =

Ct imeecho

O d =

Performing Echo-based Operations on the Entire Internet

• Purpose is illustrating the scalability of echo-based operations.

• What we needed:

– Complexity analysis of pattern

– Estimation of Internet topological properties

• diameter

• connectivity distribution

• number of nodes

Estimated Performance of Echo-based Operation on the Internet

Assumptions:Process-level transmission time: 5msNetwork delay per hop: 4msMessage size: 1KBLocal operation: 500ms per executionDiameter of Internet: 34 hops

Echo Pattern Star Pattern

Aggregated Traffic 2.25 x1011 bytes 1.31 x 1012 bytes

Max Traffic on a Link 4'096 bytes 1.8 x 109 bytes

Completion Time 17.48 seconds 5.09 days

Active Node ManagerSource

Repository

BinariesRepository

Preprocessor

TransportAccessPoint

ExecutionEnvironment

ManagementOperationResults

DeviceManager

C++Compiler

Source, State

Source, State

SNMP sets

Management commands

Weaver Active Node

Source code,Active node management

Router

NodeState

LocalProgramStates

Source, State

Events

SNMP gets/traps

Source Code

results

Active Node Engine

Management Station

Suboperations in Weaver

Node A Node B

start

Execution (T1)

Serialization (T2)

Dispatch (T3)

Receiving (T4)

Loading (T5) or Instantiation (T6)

De-serialization (T7)

Execution (T1)

Serialization (T2)

Dispatch (T3)

Receiving (T4)Resolving (T8)

end

Tim

e

De-serialization (T7)Execution (T1)

TC1

TC2

Measuring Execution Times on Weaver

    Duration in ms Performed by Module

Execution (T1) 1.57 (σ = 0.48) Execution Environment

Serialization (T2) 3.46 (σ = 0.71) Execution Environment

Dispatch (T3) 1.67 (σ = 0.49) Transport Access Point

Receiving (T4) 0.62 (σ = 0.30) Transport Access Point

Loading (T5) 23.42 (σ = 0.70) Execution Environment

Instantiation (T6) 0.77 (σ = 0.015) Execution Environment

De-serialization (T7) 2.04 (σ = 0.49) Execution Environment

Resolving (T8) 0.15 (σ = 0.001) Execution Environment

Communications Delay (TC) 4.04 (σ = 0.10) ---

Estimating Execution Times of Echo-based Operations on Weaver

Skip EchoSkipEcho(inmsg: bytes from: integer) {

if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg, i); Gi = up_neighbors() - from; if Gi != empty dispatch(parenti, outmsg, i); } else { Gi = Gi - from; OnAggregate(inmsg); } if completei != true and Gi = empty { OnComplete(outmsg); completedi := true; if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}alarm(type: {failure, recovery}, affected: integer){ if visitedi = true { if type = failure { Gi := Gi - affected if completei != true and Gi = empty { completei := true; OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); } } }}

Wait EchoSkipEcho(inmsg: bytes from: integer) {

if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg, i); Gi = up_neighbors() - from; if Gi != empty dispatch(parenti, outmsg, i); } else { Gi = Gi - from; OnAggregate(inmsg); } if completei != true and Gi = empty { OnComplete(outmsg); completedi := true; if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}alarm(type: {failure, recovery}, affected: integer){ if visitedi= true { if type == failure { Gi = Gi - affected Bi = Bi + affected if completei != true and Gi = empty { completei := true; OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i) else OnTerminate(inmsg); } } else { if affected is in Bi { Bi = Bi - affected Gi = Gi + affected } } }}

Designing Robust PatternsPlain EchoEcho(inmsg: bytes, from: integer) { Gi := Gi - from; if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg); if Gi != empty dispatch(parenti, outmsg, i); } else OnAggregate(inmsg); if Gi = empty { OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}

100

120

140

160

180

200

220

52.5 53 53.5 54 54.5 55 55.5

Cove

rage (

nodes)

Time (mins)

Coverage Vs Time for skipecho

MTTF=3.683 hrsMTTF=7.367 hrsMTTF=11.05 hrs

MTTF=14.733 hrsMTTF=29.467 hrsMTTF=73.67 hrs

Network Coverage vs. Execution Timefor Skip Echo

MTTR=1 minMTTR=11 minMTTR 0

MTTR inf

MTTF = 3.6 hrsMTTF = 7.3 hrsMTTF = 11.0 hrsMTTF = 14.7 hrsMTTF = 29.4 hrsMTTF = 73.6 hrs

Current and Planned Work

• Self-organizing, adaptable Networks and Systems:Patterns for routing and dynamic construction of network control structures. (Constantin Adam)

• WQL: A table-based Network Query Language on Weaver.

(Koon-Seng Lim)

• Policy-based Management: Patterns for distribution and dynamic re-computation of policies.(Alberto Gonzalez)

Literature on this Work

• K.S. Lim, R. Stadler: “Weaver—Realizing a scalable management paradigm on commodity routers,” Eighth IFIP/IEEE International Symposium on Integrated Network Management (IM 2003), Colorado Springs, Colorado, USA, March 24-28, 2003.

• K.S. Lim and R. Stadler: "Developing pattern-based management programs," IFIP/IEEE International Conference on Management of Multimedia Networks and Services (MMNS 2001), Chicago, IL, October 29 - November 1, 2001.

• K.S. Lim and R. Stadler: "A navigation pattern for scalable Internet management,"IFIP/IEEE International Symposium on Integrated Network Management (IM 2001), Seattle,Washington, 14-18 May, 2001.

• R. Kawamura and R. Stadler: "A middleware architecture for active distributed management of IP networks, "IEEE/IFIP Network Operations and Management Symposium (NOMS 2000), Honolulu, Hawaii, April 10-14, 2000.