Scalable Management for Networks and Services

Scalable Management for Networks and Services

Rolf Stadler

Laboratory for Communication NetworksKTH Royal Institute of Technology

Stockholm

HP Laboratories, Palo Alto, March 31, 2003

node

Managementstation

Managementstation

Manager-Agent based management

node

A A A

M

results

download &execute

P

Management Program

Manager

Agent

•Centralized Control•Management protocols: SNMP, CMIP•Program runs on Management Station

•Decentralized Control•Program runs on network nodes

P

The Shift of a Management Paradigm

AP

A A

Router

Execution Environment

Management station

Management Program

navigation Code Server

Architecture for Pattern-based Management

Weaver—A Testbed for pattern-based Management

WANA

WANB

WANC

WAND

Management Station

Router A Router B Router C Router D

FastEthernet Switch

Simple Navigation Patterns

1

21

2

1

2

2

2

3

3

34

operation typical application navigation pattern

type 1: node-to-node 1 node control/monitor(get/set of variables)

type 2: visit all nodesalong a path/flow

1 flow/path controle.g.: traceroute, bottleneckdetection, signalling, VPNoperation

type 3: distribute agentto all nodes in subnet(parallel control)

subnet controle.g.: topology detection

type 4: visit all nodes insubnet (sequent ialcontrol)

12

3

4

5

subnet control, messagebroadcaste.g.: congestion locationdetection

Echo Pattern (expansion)

droot=1

Echo Pattern (expansion)

droot=2

Echo Pattern (expansion)

droot=3

Echo Pattern (expansion)

droot=4

Echo Pattern (expansion)

Echo Pattern

droot=5

droot=4

Echo Pattern (contraction)

droot=3

Echo Pattern (contraction)

droot=2

Echo Pattern (contraction)

droot=1

Echo Pattern (contraction)

Echo Pattern (contraction)

Echo Pattern (contraction)

The Echo Pattern

• Two phases of traversal– expansion phase: explorers flood network with requests

for local operations– contraction phase: echoes return and aggregate results

• Properties– Generates balanced traffic load– Traffic load depends on network topology,

not on speed of traversal– Time complexity increases linearly with network

diameter.

Examples of Echo-based Management

• Get information on topology

– compute the current number of leaf nodes, the connectivity distribution

– discover current topology within 10 hops of node x

• Get information on network state

– identify 10 most congested links

– compute distribution of link utilization, queue lengths

– identify sub topologies with highly loaded links

– find a resource R closest to node x

Pattern-based Management—An Engineering Approach to Decentralized Management

• A management program consists of– A navigation pattern (distr. graph traversal algorithm)– An operation on nodes– An aggregation function

• Relevance of this approach– Provides a basis to analyze management operation for

performance, scalability, robustness– Supports concept of re-usable patterns, hides

complexity

Composing Management Programs

Segall

Echo Patterns

Navigation Patterns

Chang

Skip

Wait

Scope

Multi

Echo Aggregators

Res. Disc.

Aggregators

Leaf Count

Load. Hist.

Conn. Hist.

C LI

HT

TP

X ML

SN MP Local Operations

Node Access

Management Program

Properties of Patterns

Echo Aggregators

Res. Disc.

Aggregators

Leaf Count

Load. Hist.

Conn. Hist.

C LI

HT

TP

X ML

SN MP

Node Access

Management Program

Segall

Simple Echo Robust Echo Others

Echo Patterns

Navigation Patterns

Chang

Skip

Wait

Scope

Multi

• A pattern can be used for many management operations. • A pattern can be chosen according to performance objectives.• A pattern hides the complexity of a distributed operation.• Network failures can be handled within patterns.• Code mobility can be controlled.

visitedi : boolean init false;Gi : set of integers init neighbors();parenti : integer init -1;

Echo(inmsg: bytes, from: integer) { Gi := Gi - from; if visitedi = false { parenti := from; visitedi := true;

OnInitiate(inmsg, outmsg); if Gi != empty dispatch(parenti, outmsg, i); } else OnAggregate(inmsg); if Gi = empty {

OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}

The Interface between Pattern and Aggregator

OnBegin

OnTerminate

OnInitiate OnComplete

OnAggregate

…av_loadi := av_load;…

…av_load := load();n:=1;…

…av_load := (av_load*n + av_loadj)/(n+1);n:=n+1;…

SIMPSON: A SIMple Pattern Simulator fOr Large Networks

0

200000

400000

600000

800000

1e+06

1.2e+06

0 1 2 3 4 5 6T

raff

ic (b

yte

s)

Time (secs)

Traffic vs Time for 221 node grid network

"trace1.txt"

Analyzing Management Operations

A

CD

E

F

B

A

B C

DEF

E D F E

A

B D FEC

Network Graph G=(V,E) Execution Graphs G’=(V’,E’)

Centralized Management Distributed Management

Star Pattern Echo Pattern

Traffic Complexity of Management Operations

Amount of traffic placed on the network during execution.

C t ra f f ic hopcount v' childk v' Iq Ir+

v' V' 0 k ch i ldcou n t v'

=

Ctrafficecho

Iq Ir+ Edegree G 2 V–

2------------------------------------------- 1+ +

=

Ctrafficstar Iq Ir+ hopcount v'ro ot v'

v' V'

=

Time Complexity of Management Operations

Time needed from invocation until completion of a operation.

C time C time v'root =

Ctime v' tc tr+ if childcount v' 0=

tc tr M v' + + otherwise

=

M v' max ktq 2+ hopcount v' childk v' tl Ctime childk v' + 1 k childcount v'

=

Ctimes tar

O V =

Ct imeecho

O d =

Performing Echo-based Operations on the Entire Internet

• Purpose is illustrating the scalability of echo-based operations.

• What we needed:

– Complexity analysis of pattern

– Estimation of Internet topological properties

• diameter

• connectivity distribution

• number of nodes

Estimated Performance of Echo-based Operation on the Internet

Assumptions:Process-level transmission time: 5msNetwork delay per hop: 4msMessage size: 1KBLocal operation: 500ms per executionDiameter of Internet: 34 hops

Echo Pattern Star Pattern

Aggregated Traffic 2.25 x1011 bytes 1.31 x 1012 bytes

Max Traffic on a Link 4'096 bytes 1.8 x 109 bytes

Completion Time 17.48 seconds 5.09 days

Active Node ManagerSource

Repository

BinariesRepository

Preprocessor

TransportAccessPoint

ExecutionEnvironment

ManagementOperationResults

DeviceManager

C++Compiler

Source, State

Source, State

SNMP sets

Management commands

Weaver Active Node

Source code,Active node management

Router

NodeState

LocalProgramStates

Source, State

Events

SNMP gets/traps

Source Code

results

Active Node Engine

Management Station

Suboperations in Weaver

Node A Node B

start

Execution (T1)

Serialization (T2)

Dispatch (T3)

Receiving (T4)

Loading (T5) or Instantiation (T6)

De-serialization (T7)

Execution (T1)

Serialization (T2)

Dispatch (T3)

Receiving (T4)Resolving (T8)

end

Tim

e

De-serialization (T7)Execution (T1)

TC1

TC2

Measuring Execution Times on Weaver

    Duration in ms Performed by Module

Execution (T1) 1.57 (σ = 0.48) Execution Environment

Serialization (T2) 3.46 (σ = 0.71) Execution Environment

Dispatch (T3) 1.67 (σ = 0.49) Transport Access Point

Receiving (T4) 0.62 (σ = 0.30) Transport Access Point

Loading (T5) 23.42 (σ = 0.70) Execution Environment

Instantiation (T6) 0.77 (σ = 0.015) Execution Environment

De-serialization (T7) 2.04 (σ = 0.49) Execution Environment

Resolving (T8) 0.15 (σ = 0.001) Execution Environment

Communications Delay (TC) 4.04 (σ = 0.10) ---

Estimating Execution Times of Echo-based Operations on Weaver

Skip EchoSkipEcho(inmsg: bytes from: integer) {

if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg, i); Gi = up_neighbors() - from; if Gi != empty dispatch(parenti, outmsg, i); } else { Gi = Gi - from; OnAggregate(inmsg); } if completei != true and Gi = empty { OnComplete(outmsg); completedi := true; if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}alarm(type: {failure, recovery}, affected: integer){ if visitedi = true { if type = failure { Gi := Gi - affected if completei != true and Gi = empty { completei := true; OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); } } }}

Wait EchoSkipEcho(inmsg: bytes from: integer) {

if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg, i); Gi = up_neighbors() - from; if Gi != empty dispatch(parenti, outmsg, i); } else { Gi = Gi - from; OnAggregate(inmsg); } if completei != true and Gi = empty { OnComplete(outmsg); completedi := true; if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}alarm(type: {failure, recovery}, affected: integer){ if visitedi= true { if type == failure { Gi = Gi - affected Bi = Bi + affected if completei != true and Gi = empty { completei := true; OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i) else OnTerminate(inmsg); } } else { if affected is in Bi { Bi = Bi - affected Gi = Gi + affected } } }}

Designing Robust PatternsPlain EchoEcho(inmsg: bytes, from: integer) { Gi := Gi - from; if visitedi = false { parenti := from; visitedi := true; OnInitiate(inmsg, outmsg); if Gi != empty dispatch(parenti, outmsg, i); } else OnAggregate(inmsg); if Gi = empty { OnComplete(outmsg); if parenti >= 0 dispatch(parenti, outmsg, i); else OnTerminate(inmsg); }}

100

120

140

160

180

200

220

52.5 53 53.5 54 54.5 55 55.5

Cove

rage (

nodes)

Time (mins)

Coverage Vs Time for skipecho

MTTF=3.683 hrsMTTF=7.367 hrsMTTF=11.05 hrs

MTTF=14.733 hrsMTTF=29.467 hrsMTTF=73.67 hrs

Network Coverage vs. Execution Timefor Skip Echo

MTTR=1 minMTTR=11 minMTTR 0

MTTR inf

MTTF = 3.6 hrsMTTF = 7.3 hrsMTTF = 11.0 hrsMTTF = 14.7 hrsMTTF = 29.4 hrsMTTF = 73.6 hrs

Current and Planned Work

• Self-organizing, adaptable Networks and Systems:Patterns for routing and dynamic construction of network control structures. (Constantin Adam)

• WQL: A table-based Network Query Language on Weaver.

(Koon-Seng Lim)

• Policy-based Management: Patterns for distribution and dynamic re-computation of policies.(Alberto Gonzalez)

Literature on this Work

• K.S. Lim, R. Stadler: “Weaver—Realizing a scalable management paradigm on commodity routers,” Eighth IFIP/IEEE International Symposium on Integrated Network Management (IM 2003), Colorado Springs, Colorado, USA, March 24-28, 2003.

• K.S. Lim and R. Stadler: "Developing pattern-based management programs," IFIP/IEEE International Conference on Management of Multimedia Networks and Services (MMNS 2001), Chicago, IL, October 29 - November 1, 2001.

• K.S. Lim and R. Stadler: "A navigation pattern for scalable Internet management,"IFIP/IEEE International Symposium on Integrated Network Management (IM 2001), Seattle,Washington, 14-18 May, 2001.

• R. Kawamura and R. Stadler: "A middleware architecture for active distributed management of IP networks, "IEEE/IFIP Network Operations and Management Symposium (NOMS 2000), Honolulu, Hawaii, April 10-14, 2000.