Jon TurnerApplied Research LabComputer Science & EngineeringWashington University

www.arl.wustl.edu

Supercharged Planetlab PlatformGENI Experimenters’ Workshop – 6/2010

2

Shameless PlugThis talk is just a teaserTo really learn how to use the SPPs, come to

our tutorial at GEC-8 (Thursday, 7/22)» more detailed presentations» live demonstrations» supervised hands-on session

If you can’t do that» read the online tutorial at wiki.arl.wustl.edu/spp» get an account at spp_plc.arl.wustl.edu» start playing

3

SPP Nodes

SPP is a high performance overlay hosting platform Designed to be largely compatible with PlanetLab How it’s different from PlanetLab

» multiple processors, including an NP-blade for slice fastpaths» multiple 1 GbE interfaces» support for advance reservation of interface bw, NP resources

Chassis Switch

10x1 GbE

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

4

SPP Deployment in Internet 2

2 2

2

two more nodesHouston and Atlanta

later this year

5

Washington DC Installation

6

Details

I2 Optical Connections

ProtoGENI ProtoGENI ProtoGENI

I2 Internet ServiceI2 Router I2 RouterI2 Router

SPP SPP SPPSPP_PLCSALT KANS WASHspp_plc.arl.wustl.edu

7

Hosting Platform Details

Chassis Switch

10x1 GbE

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

PLOS

VM VM . . .

General PurposeProcessing Engine

filter

. . .

. . .

filter

Line Cardlookupparse header

format

... ...

queues

...

Network Processing Engine

8

Key Control Software ComponentsSystem Resource Manager (SRM)

» runs on Control Processor» retrieves slice definitions from SPP-PLC» manages all system-level resources and reservations

Resource Management Proxy RMP)» runs on GPEs (in root vServer)» provides API for user slices to access resources

Slice Configuration tool (scfg)» command-line interface to RMP

Substrate Control Daemon (SCD)» runs on NPE management processor» supports NPE resource configuration, statistics reporting

9

Application Framework Fastpath/slowpath

» fastpath mapped onto NPE» control daemon in vServer

on GPE Configurable elements

» code option – determines how packets processed by parse, header format

» fastpath interfaces• map to physical interface• provisioned bandwidth

» TCAM filters» Queues

• length, bandwidthControl daemon can configure fastpath through RMP

»or users can configure manually with scfg

Parse Lookup

Filters

Control Interface

HeaderFormat

QueueManagerFast Path

...

...

...

outp

utin

terf

aces

inpu

tin

terf

aces

controldaemon

(in vServer)

GPE

Remote Login Interface

exce

ptio

n pa

cket

s&

in-b

and

cont

rol

exce

ptio

n pa

cket

s&

in-b

and

cont

rol

10

Working with SPPsDefine new slice using SPP-PLC

» just like PlanetLabLogin to SPPs in slice to install application codeReserve resources needed for your experiment

» includes interface bandwidth on external ports and NPE fastpath resources

To run experiment during reserved period» “claim” reserved resources» setup slowpath endpoints» configure fastpath (if applicable)» setup real-time monitoring» run application and start traffic generators

11

Web Site and SPP_PLChttp://wiki.arl.wustl.edu/index.php /Internet_Scale_Overlay_Hostinghttp://wiki.arl.wustl.edu/index.php /The_SPP_Tutorial

spp_plc.arl.wustl.edu

12

Creating a SliceSPP_PLC

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

SRM

13

Preparing a Slice

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

Line Card

SLM

Logging into slice

SFTP connection for downloading code

Requires Network Address Translation

datapath detects new connection, LC

control processor adds filters

14

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

LC

SRM

RMP

Configuring a Slowpath Endpoint

●Request endpoint with desired port number on specific interface thru Resource Manager Proxy (RMP)

●RMP relays request to System Resource Manager (SRM)

●SRM configures LC filters for interface●Arriving packets directed to slice,

which is listening on socket

15

Setting Up a Fast Path

●Request fastpath through RMP●SRM allocates fastpath●Specify logical interfaces and

interface bandwidths●Specify #of filters, queues, binding

of queues to interfaces, queue lengths and bandwidths

●Configure fastpath filters

Chassis Switch

CP

ExternalSwitch

netFPGA

GPE GPE NPE

LC

SRM

RMP

lookupparse headerformat

queues

...

16

Displaying Real-Time Data

Fastpath maintains traffic counters and queue lengths

To display traffic data» configure an external TCP port» run sliced within your vServer, using configured port

• sliced --ip 64.57.23.194 --port 3552 &» on remote machine, run SPPmon.jar

• use provided GUI to setup monitoring displays• SPPmon configures sliced, which polls the NPE running the

fastpath• SCD-N reads and returns counter values to sliced• can also display data written to file within your vServer

NPE

CPGPE

GPE

Line Card

SCD

sliced

SPPmon

17

Command Line Tools

scfg – general slice configuration tool» scfg –-cmd make_resrv (cancel_resrv, get_resrvs,..)» scfg –-cmd claim_resources» scfg –-info get_ifaces (...)» scfg –-cmd setup_sp_endpoint (setup_fp_tunnel,..)» same features (and more) available thru C/C++ API

sliced – monitor traffic and display remotelyip_fpd – fastpath daemon for with IPv4 fastpath

» handles exception packets (e.g. ICMP ping)» can extend to add features (e.g. bw reservation option)

ip_fpc – fastpath configuration tool for IPv4 fastpath

18

Forest Overlay NetworkOverlay for real-time

distributed applications» large online virtual worlds» distributed cyber-physical systems

Large distributed sessions» endpoints issue periodic status reports» and subscribe to dynamically changing sets of reports» requires real-time, non-stop data delivery, even as

communication pattern changesPer-session provisioned channels (comtrees)

» unicast data delivery with route learning» dynamic multicast subscriptions using multicast core

overlayrouter

client

serveraccess

connection

comtree

19

Unicast Addressing and ForwardingEvery comtree has its own topology and routesTwo level unicast addresses

» zip code and endpoint numberNodes with same zip code form subtree

» all nodes in “foreign” zips reached through same branchUnicast routing

» table entry for each foreign zip code and local endpoint

» if no route table entry, broadcastand set route request flag

» first router with a route responds

1

5

4

3

2

3.1

20

Multicast RoutingFlat addresses

» on per comtree basisHosts subscribe to multicast

addresses to receive packetsEach comtree has “core” subtree

» multicast packets go to all core routers» subscriptions propagate packets outside core

Subscription processing» propagate requests towards first core router» stop if intermediate router already subscribed» can subscribe/unsubscribe to many multicasts at once

Core size can be configured to suit application

coresubtree

subscriptionspropagate

towards core

21

Forest Packet FormatEncapsulated in IP/UDP packet

» IP (addr,port#) identify logical interface to Forest router

Types include» user data packets» multicast subscribe/unsubscribe» route reply

Flags include» unicast route request

IP/UDP Header(addr,port#) identify

logical interfacever length type flags

comtreesrc adrdst adr

header err check

payload(≤1400 bytes)

payload err check

22

Basic Demo SetupWASH

KANS

SALT 10.1.7.*

.2

.2

10.1.1.*

• .1

10.1.3.*.2

.1.1

64.57.23.*

2.100

3.1001.100.218

planetlabnodes

1.1 1.2 1.3

.186

2.1 2.2 2.3

.204

3.1 3.2 3.3

laptop

23

Simple Unicast Demo

Uses one host at each routerSingle comtree with root at KANSHost 1.1 sends to 2.1 and 3.1

» packet to 3.1 flagged with route-request and “flooded”» router 2.100 responds with route-reply

Host 2.1 sends to 1.1 and 3.1» already has routes to both so no routing updates needed

Host 3.1 sends to 1.1 and 2.1» packet to 1.1 flagged with route-request and “flooded”» router 2.100 responds with route-reply

1.*

2.*

3.*

24

new route from 1.100 (at salt) to

3.0 (at wash)

new route from 3.100 (at salt) to

1.0 (at wash)

2.100 receives flagged packet

from 1.100 to 3.1

sending route-reply

forwarding to 3.1and sendingroute-reply

forwarding to 3.1and sendingroute-reply

use UserData to display data in

stats files on GPEs

run script to launch Forest

routers and hosts

25

Setting up & Running the DemoPrepare Forest router configuration files

» config info for Forest links, comtrees, routes, statisticsPrepare/save SPPmon config – specify chartsReserve SPP resources

» bandwidth on four external interfaces (one for sliced)Start session

» claim reserved resources» setup communication endpoint for router logical interfaces

and for sliced to report statistics» start sliced on SPP and SPPmon on laptop

Start Forest routers & hosts, then observe traffic» done remotely from laptop, using shell script

26

Reservation ScriptOn spp, execute reservation 03151000 03152200

#! /bin/bashcat >res_file.xml <<foobar<?xml version="1.0" encoding="utf-8" standalone="yes"?><spp> <rsvRecord> <rDate start="2010${1}00" end="2010${2}00" /> <plRSpec> <ifParams> <ifRec bw="10000" ip="64.57.23.186" /> ... </ifParams> </plRSpec> </rsvRecord></spp>foobarscfg --cmd make_resrv --xfile res_file.xml

reservation start and end times

(mmddhhmm) GMT

reserve interface

bandwidth (4 of these)

invoke scfg on reservation file

copy reservation to a file

27

Setup ScriptOn spp, execute setup

#! /bin/bash# claim reserved resourcesscfg --cmd claim_resources# configure interfaces, binding port numbersscfg –cmd setup_sp_endpoint –bw 10000 –ipaddr 10.1.1.1 --proto 17 –port 30123scfg –cmd setup_sp_endpoint –bw 10000 –ipaddr 10.1.3.1 --proto 17 –port 30123scfg –cmd setup_sp_endpoint –bw 10000 –ipaddr 64.57.23.186 --proto 17 –port 30123scfg –cmd setup_sp_endpoint –bw 2000 –ipaddr 64.57.23.182 --proto 6 –port 3551# run monitoring daemoncat </dev/null >statssliced –ip 64.57.23.182 &

interfaces to other

SPPs

“public” interface

interface for traffic

monitoring

28

Run Demo On laptop, run script fdemo1

#! /bin/shtlim=50 # time limit for hosts and routers (sec)dir=fdemo1 # directory in which code is executed...# public ip addresses used by forest routersr1ip=64.57.23.218 ...# names and addresses of planetlab nodes used as forest hostsh11host=planetlab6.flux.utah.eduh11ip=155.98.35.7 ...ssh ${sppSlice}@${salt} fRscript ${dir} 1.100 ${tlim} &...sleep 2ssh ${plabSlice}@${h11host} fHscript ${dir} ${h11ip} ${r1ip} 1.1 ${rflg} ${minp} ${tlim} &...

29

Basic Multicast Demo

Uses four hosts per router» one source, three receivers

Comtree centered at each Forest router» each source sends to a multicast group on each comtree

Phase 1 – uses comtree 1» each receiver subscribes to multicast 1, then 2 and 3;

then unsubscribes – 3 second delay between changes» receivers all offset from each other

Phases 2 and 3 are similar» use comtrees 2 and 3, so different topology

1.*

2.*

3.*

30

run script to launch Forest

routers and hosts

all hosts subscribe in turn to three multicasts

on comtree 1

salt to kans traffic

salt to wash traffic