Design of a High Performance Active Router · front panel SE SE SE SE O I O I O I O I O I O I O I O I line cards main board Power Supply Fans WUGS20. Washington WASHINGTON UNIVERSITY

Jonathan Turner - 3/18/99 WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

1

Design of a HighPerformance Active Router

Jonathan Turner, Guru Parulkar (on leave)Dan Decasper, John Dehart, Sumi Choi, Tilman Wolf,

Bernhard Plattner (ETHZ), Ralph Keller (ETHZ)

WashingtonWASHINGTON UNIVERSITY IN ST LOUIS

2Jonathan Turner - 3/18/99

Project Objectives

l Demonstrate potential of high performance active networking.l Design and implement a prototype active networking platform.

» high performance - active processing at gigabit rates» scalable - prototype capacity of 20 Gb/s - scalable design to over 1 Tb/s

l Develop general methods for high performance.» scalable and extensible hardware platform» software architecture supporting parallel processing» kernel plug-ins for efficient software processing» use of compiled code from distributed code servers» authentication to ensure code integrity» reconfigurable hardware components

l Demonstrate active applications» protocol deployment» topology-aware reliable multicast» video distribution over reliable multicast



Key Technology Components

l Scalable Switch Core» initial prototype 20 Gb/s, 160 Gb/s configuration under construction» built around custom ATM chip set developed at Washington University

l Embedded Processor at each Port» Pentium Processor with 32 MB memory» NetBSD operating system with active network extensions» high performance two port network interface chip

l Kernel Plugins for Efficient Packet Processing» configurable software components in OS kernel» allows application-specific processing of compiled code in kernel

l Distributed Code Caches for On-Demand Plugin Retrieval» non-resident plugins retrieved from hierarchy of trusted code servers» authentication mechanisms used to verify integrity of retrieved code



Washington University Prototype Switch

l 20 Gb/s core capacity on single PC boardl efficient multicast support in hardwarel variety of line cards (up to 2.4 Gb/s)l Copies being distributed as open

research platform to 30 universitiesunder NSF sponsorship

Switch Element

IPP

IPPIPPIPPIPPIPPIPP

OPP

OPP

OPPOPPOPPOPPOPPOPPIPP

LC

LC

LC

LCLCLCLCLC

LC

LCLC

LCLCLCLCLC

front panel

SE

SE

SE

SE

OI

OI

OI

OI

OI

OI

OI

OI

line

card

s

mainboard

Power SupplyFans

WUGS20



Active Node Processing Element

l Active Processing Element (PE) ateach switch port for scalable activepacket processing.

l Can stack multiple PEs per port forhigher performance.

l APIC enables efficient input oroutput side processing.

IPP

OPPAPIC

Pentium166 MHz

cachePCI

Bridge

32+ MB

ROE

X

SystemFPGAPELine Card

IF0

IF1

OF0

OF1

Ext. Mem. Int.

VC

XT

CST

R

PCI 64/32



Principal Data Flows Through PE

l Active packets move through configured kernel plugins.» active function dispatcher passes packets to instances of plugin objects» instantiates objects or triggers download of plugin class, as needed» streamlined processing of SAPF packets using pre-established state

l Standard processing for “plain” IP packets.» classification and routing, header processing, output queueing

PacketClassificationand Routing

Dri

ver

SAPF Packet

IPv4/6Header

Processing

Packet +Flow Id

Pac

ket

Sche

dule

r

ActiveFunction

Dispatcher

Resource Controller

Selector/Dispatcher

Dri

ver

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

Active Packets

Plain Packets

Kernel Plugins

. . .

. . .

. . .

. . .

. . .

. . .



Overall System Architecturel Switch Controller (SC) manages

system.» global coordination & control» routing protocols» build routing tables and other

information needed by PEs» first level code server» reprogrammable for active

processing

SC

Scalable Switch Core

PP

PELC

PP

PELC

PP

PEPP

PEPP

PEPP

PEPP

PELC LC LC LC LC

API

C

CPU

MEM

l Processing Elements (PE) do per packet processing.» standard processing of IP packets» various types of active processing

–– Active IP packets using Active Network Encapsulation Protocol (ANEP)Active IP packets using Active Network Encapsulation Protocol (ANEP)–– Simple Active Packet Format (SAPF) using pre-established flow stateSimple Active Packet Format (SAPF) using pre-established flow state

(conceptually similar to tag switching)(conceptually similar to tag switching)–– ANTS for capsule execution in user-space Java VMANTS for capsule execution in user-space Java VM

» control mechanisms for demand-loading of plugins



Software Architecture


Dri

ver

SAPF Packet

IPv4/6Header

Processing

Pac

ket

Sche

dule

r

ActiveFunction

Dispatcher

Resource Controller

Selector/Dispatcher

Dri

ver

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

. . .

KernelPlugins

. . .

. . .

. . .

. . .

. . .

. . .

PluginCtl. Unit

TCP/UDP

anetd

JavaVM

ANTS

NOSI

Active Plugin Loader

PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

PluginDB

KeyDB

PolicyRules

RSVP/SSPRouting

CodeServer

CodeServer

ActiveCode DB

PE Kernel Space

PE User Space,Switch Controller,& Remote Servers



Active Packet Processing ComponentsActive Function Dispatcherl parses packets of new active flowsl determines what plugins need to

be loaded and instantiatedResource Controllerl track CPU and memory usage by

active flowsl enforce usage limitsl DRR-style algorithmPlugin Control Unitl maintain resident active pluginsl support plugin

loading/unloadingSelector/Dispatcherl implements the Simple Active

Packet Formatl enables more efficient packet

classification for active flows


Dri

ver

SAPF Packet

IPv4/6Header

Processing

Pac

ket

Sche

dule

r

ActiveFunction

Dispatcher

Resource Controller

Selector/Dispatcher

Dri

ver

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

. . .

KernelPlugins

. . .

. . .

. . .

. . .

. . .

. . .

PluginCtl. Unit

TCP/UDP

anetd

JavaVM

ANTS

NOSI


PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

PluginDB

KeyDB

PolicyRules

RSVP/SSPRouting

CodeServer

CodeServer

ActiveCode DB

PE Kernel Space


ANTSl user space execution environment

developed by MITNodeOS APIl standard AN interface on which for user

space EEsl provides abstractions for computation,

storage and communication.



Plugin Management

Plugin Requestorl communicates with code servers

to retrieve new pluginsActive Plugin Loaderl provides flow of control among

the other plugin managementcomponents

Plugin Database Controllerl administers local plugin

database (cache)Policy Controllerl controls admission of new

plugins based on policy rulesl maintains policy databaseSecurity Gatewayl checks integrity and origin of

new pluginsl maintains database (cache) of

public keys for developers, codeservers.


Dri

ver

SAPF Packet

IPv4/6Header

Processing

Pac

ket

Sche

dule

r

ActiveFunction

Dispatcher

Resource Controller

Selector/Dispatcher

Dri

ver

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

. . .

KernelPlugins

. . .

. . .

. . .

. . .

. . .

. . .

PluginCtl. Unit

TCP/UDP

anetd

JavaVM

ANTS

NOSI


PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

PluginDB

KeyDB

PolicyRules

RSVP/SSPRouting

CodeServer

CodeServer

ActiveCode DB

PE Kernel Space


AN Node Managerl user interface for node

managementl supports privileged access to

other software components



Basic Demo & Performance Measurement

l User-level program (start) sends active packet referencingping plugin to remote machine.

l Ping plugin swaps source/destination addresses andforwards packet.

l After specified number of round trips, packet returned touser level program to generate measurement.

ATMNIC

ATMNIC

Start

PingPlugin

PingPlugin

150 Mb/s ATM Link



Ping (first packet)

l Plugin retrieved, installed,instantiated.

l Packet processed by plugin.


Dri

ver

SAPF Packet

IPv4/6Header

Processing

Pac

ket

Sche

dule

r

ActiveFunction

Dispatcher

Resource Controller

Selector/Dispatcher

Dri

ver

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

. . .

KernelPlugins

. . .

. . .

. . .

. . .

. . .

. . .

PluginCtl. Unit

TCP/UDP

anetd

JavaVM

ANTS

NOSI


PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

PluginDB

KeyDB

PolicyRules

RSVP/SSPRouting

CodeServer

CodeServer

ActiveCode DB

PE Kernel Space




Ping (second packet)

l AFD dispatches packet directlyto plugin.


Dri

ver

SAPF Packet

IPv4/6Header

Processing

Pac

ket

Sche

dule

r

ActiveFunction

Dispatcher

Resource Controller

Selector/Dispatcher

Dri

ver

IP Packets

Dri

ver

Dri

ver

Pac

ket

Sche

dule

r

. . .

KernelPlugins

. . .

. . .

. . .

. . .

. . .

. . .

PluginCtl. Unit

TCP/UDP

anetd

JavaVM

ANTS

NOSI


PluginRequestor

Plugin DBController

PolicyController

SecurityGateway

ANNManager

PluginDB

KeyDB

PolicyRules

RSVP/SSPRouting

CodeServer

CodeServer

ActiveCode DB

PE Kernel Space




Ping Performance

l Experimental Setup» 233 MHz Pentium Pro» Efficient Networks ATM interface (150 Mb/s)

l Results (after first packet)» round trip delay 169 µsec» one way delay 84.5 µsec» shortcut forwarding 50 µsec (hardware 40, software 10)» AFD & plugin 34.5 µsec

l Comparisons» U Penn PLAN (PII 300 MHz) 200 µsecs» MIT, ANTS (UltraSparc 167 MHz) 588 µsecs» Arizona, Scout (200 MHz) 518 µsecs



ANN Project Schedule (page 1)

Code ServerDesign

Was

hU

ET

HZ

Q3/98 Q4/98 Q1/99 Q2/99 Q4/99Q3/99

SelectorDispatcher

Design ANPESoftware

Dan

Assemble/Test SPC

Protocol Deployment

Tilmanunassigned/others JohnSumi

Design ANPEHardware

Demo

Publications

allRalph

ActiveMulticast

Test ANPESoftware

Security Gateway

Policy Controller

ResourceController

(CPU)

Port NetBSD to SPC

Port ANEPSoftware to SPC

HFSC

Active FunctionDispatcher

Modificationsto IP stack/PCU/

Sockets


PluginRequester

RefinePlugin

Management

ResourceController(Memory)

Plugin DatabaseController

Dissertation

Load-Sharing

Simple DAN PrePrototypeup and running

ANN v1.0 upand running

Single ANPE softwarecompleted

RefineDAN kernel

support

Javaapplications

ANTS/Javasupport

WDR/Ed/John

SPC HWDesign Spec

AN protocoldocument

Last Modified 10:27 AM, Tuesday, March 16, 1999

Java support

APIC gettingFabbed (11/18/98)

Assemble/Test SPC Dev. Sys.

APIC Card/Driver Test

APIC NicTestbed

ANN ManagerProgram



ANN Project Schedule (page 2)W

ash

UE

TH

Z

Q1/00 Q2/00 Q3/00 Q4/00 Q2/01Q1/01

Publications

IPv6Mobility

Load-Sharingalgorithm

Integrate I/Osubsystem

Databases forCode Server Reliable Real

Time VideoMulticast

Demo FinalDemo

ANN hard and softwarecompleted

More applications

Killer Apps up andrunning

DanTilmanunassigned/others JohnSumi allRalph

Last Modified 10:29 AM, Tuesday, March 16, 1999



Configurable “Active Hardware”

l Port Extender can be stacked with PE at each portl On-board FPGAs can provide support for active processing.

» fast packet classification and “plain” IP forwarding» output queue scheduling

l FPGAs can potentially be “programmed” for active processing» “hardware plugins” for computationally intensive active applications

IPP

OPPAPIC

Pentium166 MHz

cachePCI

Bridge

32+ MB

ROE

X

SystemFPGAPELine Card

FPGA

Port Extender

FPGA

Memory

Memory



Towards Commodity Active Processing

l Large-scale success of active networks will require to single chips capable ofactive processing at 1-10 Gb/s.

l Technology trends favor multiple processors on a chip.» single chip processor performance scales sublinearly with chip area» multiprocessor performance scales linearly

l Design trade-offs different than with commercial single processor chips.

Lookup &Class.Engine

ActiveProcessor

Cache

ActiveProcessor

Cache

ActiveProcessor

Cache

PacketScheduler

External Memory

. . .

Memory Interface

Commercial Processors (desktop)» few concurrent tasks» virtual memory, floating point important» coarse-grained processor scheduling is

adequate

Active Processors» many concurrent tasks» VM, FP much less important» finer grained scheduling needed



Active Processing for Wireless Communication

l Constraints on computational power ofwireless devices.» power, cost, weight

l Active router can act as computationalagent for wireless devices.» wireless communication triggers activation

of device-specific active program» may be downloaded from remote server» enables rapid bug fixes, upgrades and

user-specific capabilitiesl Generalizes to wireless hierarchy.

» portable hand-held devices at base ofhierarchy

» vehicle-mounted routers provide relay tosatellites and wired network infrastructure

WiredNetwork

ActiveRouter

Wireless Devices



Conclusions

l High performance Active Networking need not be anoxymoron.

» scalable systems with gigabit links and terabit throughputs are possiblewith current/near-term technology

» on-going technology improvements will make AN economically viablel Our approach sacrifices some flexibility relative to capsules.

» sacrifice seems appropriate for performance gains» since same platform can also support capsules, no real loss of flexibility

l More work needed on active applications» need some compelling instances of active applications to focus design

efforts» benchmarks needed for performance evaluation» need better understanding of “API” for active code developers

l More information:http://www.arl.wustl.edu/arl/projects/ann/ann.html

Documents

Design of a High Performance Active Router · front panel SE SE SE SE O I O I O I O I O I O I O I O I line cards main board Power Supply Fans WUGS20. Washington WASHINGTON UNIVERSITY