Protocol Boosters A Kernel-Level Implementation A Hardware-Level Implementation And then some ….. W. S. Marcus Senior Research Scientist Telcordia Technologies

Protocol Boosters A Kernel-Level ImplementationA Hardware-Level ImplementationAnd then some …..

W. S. Marcus

Senior Research Scientist

Telcordia Technologies

[email protected]

WSMSlide 2

My Objectives

• Introduce Protocol Booster design methodology

• Describe a kernel-level implementation

• Describe a hardware-level implementation

• Highlight applications and results

WSMSlide 3

Protocol Boosters:What are they?

• Conceived as software/hardware modules that:– Transparently, selectively, and robustly enhance existing

communications protocols.

– Are used to incrementally construct new communications protocols.

– Allow protocol design to track improvements in networking technologies.

– Reduces inefficiencies associated with designing for the worst-case.

– Permit on-the-fly adaptation/customization of a protocol.

WSMSlide 4

Protocol Boosters:What are they? (continued)

SingleElementProtocolBooster

NewProtocol

Host X Host Y

BASE PROTOCOL

Booster Message Exchange

Multi-ElementProtocolBooster

Element A

Multi- ElementProtocolBooster

Element A’

WSMSlide 5

Protocol Boosters:Example: FZC

UDP/IP’Protocol

Host X Host Y

UDP/IP

FZCParityPacket

Generator

FZCPacket

Re-Generation

MSGn

FZC Parity Packet Stream

MSGn+1 MSGn-1MSGn+2 MSGn-2

FZCyFZCy+1

lossy wireless network

WSMSlide 6

Protocol Boosters:Example: FZC

UDP/IP’Protocol

Host X Host Y

UDP/IP

FZCParityPacket

Generator

MSGn

FZC Parity Packet Stream

MSGn+1 MSGn-1MSGn+2 MSGn-2

FZCyFZCy+1

lossy wireless network

WSMSlide 7

Protocol Boosters:Example: FZC (continued)

• Desirable for– Applications with latency constraints

– Applications for multicast distribution

– Applications that respond to packet loss with retransmissions

– Networks with no, or slow, return channels

• Not desirable for congestion loss (use other boosters)

• Employs a novel FEC scheme that – Based on Reed-Solomon Code

– Allows fast software implementation (Mb/s)

– Receiver need not know the number of parities sent

– Receiver need not wait for, nor calculate, any parity packets beyond the number of missing data packets.

WSMSlide 8

Protocol Boosters: Protocol Boosters and other adaptation technologies

• Link Layer Services

• Protocol Conversion/Termination

• Protocol Boosters

• Active Networking

• Basic differences– transparency

– robustness

– selectivity

– dynamism

– ease of deployment

– flexibility

– generality

WSMSlide 9

Protocol Boosters: Some Guiding Principles:

• Protocol Boosters– Can reside anywhere in the network.

– Can operate within the confines of a single network element.

– Can be distributed over several network elements.

– Can add, delay, or delete end-to-end messages.

– Never modifies syntax or semantics of end-to-end protocol exchanges.

– Transparent to protocol being boosted and elimination of any part of the booster does not, in itself, prevent end-to-end communications.

– Are instantiated or revoked on-the-fly based on policies (e.g., observed network behavior, time of day, etc.)

– Can be nested and concatenated.

– Are an Active Networking programming model.

WSMSlide 10

Active Networks: General Principles:

• Architecture for supporting rapid, reconfigurable, and dynamic services on a per packet basis!!!

• Each packet can deliver new functionality into the interior of the network.

• Secure packet processing

• Safe packet processing

• Techniques for code mobility and service deployment

WSMSlide 11

Protocol Boosters: Booster Modules

• Monitoring – History, Trace

• Debugging– Add, Delay, Delete, Duplicate

• Error Control Family– ARQ-R, ARQ-A, ACK-COMP

– FZC, ED, RO, DUP-DTEC, MSS

– FEC, DAT-COMP, SEC <-- Violate guiding principles

– PMOD

WSMSlide 12

Protocol Boosters: Implementations

• Adhere to guiding principles.

• Focus on performance, e.g., below the EE/NodeOs line.

• Kernel-level implementation for boosting IP– Procedures followed by a protocol

– ARQ, FZC, flow control, signaling etc.

– Packet oriented processing

• Hardware-level implementation for boosting ATM at the OC-3c line rate.– Core functions used by a protocol

– FEC, CRC checking and calculation, encryption, authentication, packet filtering, compression etc.

– Bit oriented processing

WSMSlide 13

Protocol BoostersKernel-level implementation II: Linux 2.0.32

• Individual protocol booster modules are implemented as loadable kernel modules, lkms.

• Six interfaces per module– instance manager (includes /proc file system hooks)

– input, output, and forward interfaces

– booster channel interface (out-of-band channel)

– ioctl interface

• A policy manager (lkm) uses a variant of the Linux IP firewall mechanism to construct flow traps directed at sequences of booster modules.

• A booster manager manages the loading/unloading of the individual boosters via the kerneld facilities, presents statistics via the /proc filesystem, manages the flow of packets through the booster modules.

• Booster deployment via a client/server paradigm

(presently under human intervention).

WSMSlide 14

Protocol BoostersKernel-level implementation II: Linux 2.0.32 (continued)

IP

USER B USER C USER D USER E

TCP UDP

BOOST

PolicyManager

BoosterA

BoosterB

BoosterC

ICMP IGMP

SERVER

BoosterManager

user

kernel

Device

WSMSlide 15

Protocol BoostersKernel-level implementation II: Linux 2.0.32 (continued)

• Booster Construction

Trapboosterinstance

boosterinstance

boosterinstance

eventqueue 0

timerqueue

Muxbooster sequence

WSMSlide 16

Protocol Boosters :Programmable Protocol Processing Pipeline (P4)

Switching array

Bypass buffer

PE PE PE

IIF OIF

……

WSMSlide 17

Protocol Boosters :Programmable Protocol Processing Pipeline (P4)

• SRAM (Altera 10K-series) based programmable devices

• Switching array (reconfiguration time = 1us)

• Processing implemented in hardware

• Dynamic reconfiguration during run time (device download time = 100ms)

PE PE PE PEPE PE

WSMSlide 18

Protocol Boosters:Experimental Setup

• Uses ttcp [with UDP and TCP options]

• F*C adds redundant packets

• Loss module removes packets

• FZC operates at access link line rates

• Unicast and multicast tested

• FEC operates at OC3c rates

F*CXXP Loss ttcpF*C

Transmitter Receiver

EthernetXXPttcp

WSMSlide 19

Protocol Boosters:Kernel-level Implementation: FZC with Random Errors

84%

86%

88%

90%

92%

94%

96%

98%

100%

No loss 2% loss 4% loss 6% loss, 8% loss 10% loss

Loss

Perc

enta

ge o

f pac

kets

whi

ch a

rriv

ed s

ucce

ssfu

lly

no FEC

FEC 20,2

FEC 20,4

FEC 20,6

WSMSlide 20

Protocol Boosters:Kernel-level Implementation: FZC with Bursty Errors

80%

82%

84%

86%

88%

90%

92%

94%

96%

98%

100%

NoFEC

FEC20,2

FEC20,4

FEC20,6

FEC20,8

FEC20,10

FEC20,12

FEC20,14

FEC20,16

FEC20,18

FEC20,20

FEC Ratio

Perc

entg

e of

pac

kets

whi

ch a

rriv

ed s

ucce

ssfu

lly

16% Bursty Loss

16% Random Loss

WSMSlide 21

Protocol Boosters:P4 Implementation: FEC results

Measured TCP throughput

0.1

1

10

100

1.0E-12

1.0E-10

1.0E-09

1.0E-08

1.0E-07

1.0E-06

1.0E-05

1.0E-04

BER

Mb

/s

without P4 with P4, no FEC no FEC, choked

with FEC Adaptive

WSMSlide 22

Protocol Boosters:Kernel-Level: Demonstration QoS Testbed

DynamicNetwork

High Delay Network

Noisy Wireless Link

Host C

Host B

Host A

Congested Network Host D

Local retransmissions across noisy wireless access networkAdd (or retransmit) parity packets on multicast treesPerform ACK manipulation over high delay networksLocal forwarding of packets to host in dynamic networkApplied selectively (e.g., mobile node signalling)

WSMSlide 23


Roaming

Wired-1 Net

Backbone Net

Wireless-1 NetWireless-2 Net

192.4.12.206

192.168.51.129

192.4.12.151

192.168.52.129

192.4.12.150

192.168.50.129

Wired-2 Net

192.168.51.130

192.168.51.193

192.168.51.194

192.168.52.194 192.168.50.194

• 5 PCs running Linux 2.0.32 enhanced with:– DVMRP (xerox mrouted v3.9.beta3), Mobile IP (Stanford MosquitoNet v1.0.4), DHCP

(ISC dhcpd v1.0)

– Protocol Boosters (v0.3), Multicast Proxy (Bellcore v0.1)

2 Mb/s (915 MHz ISM band) WaveLAN (2 virtual nets)

10 Mb/s wired Ethernet

lotus

rose

daisy

twolf tdevil

WSMSlide 24


10 Mb/s wired Ethernet

• 6 boosters used!– FZe adds parity packets; FZd regenerates data packets

– DLY delays packets (sets RTT); LOS drops packets (sets p)

– MSS reduces TCP’s Max Segment by 16 bytes (for parity)

– POR limited reordering (to prevent triple-DUP ACK)

• TCP throughput is O(RTT/px) where x=1/2 for small p– FEC reduces p to increase effective throughput

MSS FZe DLY LOS FZd POR

TCP TCP

Source Router Router

WSMSlide 25


0

1000

2000

3000

4000

5000

6000

7000

0 5 10 15Packet Loss Probability (p)

Thro

ughput (K

bps)

RTT=2ms (no FEC)RTT=2ms (FEC)RTT=500ms (no FEC)RTT=500ms (FEC)

WSMSlide 26


• FEC booster in multicast distribution tree– Transparently adds h parity packets (not change data packets)

– Downstream receivers recover any h missing data packets

• Same parity recovers different packets at each receiver– Preemptively add parity packets (real-time applications)

– Reactively retransmit parity (max number of missing packets)

Data 1Parity Data 2Data 3



FEC Booster

WSMSlide 27

Protocol Boosters:Hardware-level: PMODs

General:• Decide which booster is necessary• Activate booster at right place and right time• Coordinate dependencies among boosters• Signaling

P4 Specific:• Manage limited hardware resources• Predict the need for booster and configure in advance

WSMSlide 28

Protocol Boosters:Hardware-level: PMODs

P4:• BER-monitor based on AAL-5 CRC-32• Checks AAL-5 packet

• bad: X=1, good: X=0• Calculates moving average:

• Y[n]=Y[n-1]-X[n-256]+X[n]Controller:

• Collects data from BER monitor (reads Y)• Relates BER-monitor data with actual BER

WSMSlide 29

Protocol Boosters:Conclusions

� Devised an useful model for incremental protocol construction

� Demonstrated high-performance implementations of the model

� Demonstrated the value of Protocol Boosters for UDP and TCP applications

� Anticipate porting ‘best of the breed’ to SwitchWare/PLAN

� Technology Transfer to Army Research Lab Work (Airborne Communications Node - ACN)

� TCP performance improvements promising, but open issues

�How adapt to different versions of TCP (e.g., when ACK)

�Better understand FZC booster and TCP interaction

�Policies

Documents

Protocol Boosters A Kernel-Level Implementation A Hardware-Level Implementation And then some ….. W. S. Marcus Senior Research Scientist Telcordia Technologies