A Multicast Transport Driver for Globus XIO

Karl Jeaclekarl.jeacle@cl.cam.ac.uk

ETNGRID 2005

Rationale

Would like to achieve high-speed bulk data delivery to multiple sites

Multicasting would make sense Existing multicast research has focused on

sending to a large number of receivers But Grid is an applied example where sending to

a moderate number of receivers would be extremely beneficial

ETNGRID 2005

Multicast availability

Deployment is a problem!Protocols have been defined and implementedValid concerns about scalability; much FUD “chicken & egg” means limited coverage

Clouds of native multicastBut can’t reach all destinations via multicastSo applications abandon in favour of unicast

What if we could multicast when possible……but fall back to unicast when necessary?

ETNGRID 2005

Multicast TCP?

TCP “single reliable stream between two hosts”

Multicast TCP “multiple reliable streams from one to n hosts”

May seem a little odd, but there is precedent…TCP-XMO – Liang & CheritonM-TCP – Mysore & VargheseM/TCP – Visoottiviseth et alPRMP – Barcellos et alSCE – Talpade & Ammar

ETNGRID 2005

ACK implosion

ETNGRID 2005

Building Multicast TCP

Want to test multicast/unicast TCP approachBut new protocol == kernel changeWidespread test deployment difficult

Build new TCP-like engineEncapsulate packets in UDPRun in userspacePerformance is sacrificed……but widespread testing now possible

ETNGRID 2005

TCP/IP/UDP/IP

SendingApplication

TCP

IP

UDP

IP

ReceivingApplication

TCP

IP

UDP

IP

If natively implemented

test deployment

ETNGRID 2005

TCP engine

Where does initial TCP come from? Could use BSD or Linux

Extracting from kernel could be problematic

More compact alternative lwIP = Lightweight IPSmall but fully RFC-compliant TCP/IP stack

lwIP + multicast extensions = “TCP-XM”

ETNGRID 2005

TCP-XM overview

Primarily aimed at push applications Sender initiated – advance knowledge of receivers Opens sessions to n destination hosts simultaneously Unicast is used when multicast not available Options headers used to exchange multicast info API changes

Sender incorporates multiple destination and group addresses Receiver requires no changes

TCP friendly, by definition

ETNGRID 2005

TCP

SYN

SYNACK

ACK

FIN

FIN

ACK

ACK

ACK

DATASender Receiver

ETNGRID 2005

TCP-XM

Sender

Receiver 1

Receiver 2

Receiver 3

ETNGRID 2005

TCP-XM connection

ConnectionUser connects to multiple unicast destinationsMultiple TCP PCBs created Independent 3-way handshakes take placeSSM or random ASM group address allocated

(if not specified in advance by user/application)

Group address sent as TCP optionAbility to multicast depends on TCP option

ETNGRID 2005

TCP-XM transmission

Data transferData replicated/enqueued on all send queuesPCB variables dictate transmission modeData packets are multicast (if possible)Retransmissions are unicastAuto fall back/forward to unicast/multicast

CloseConnections closed as per unicast TCP

ETNGRID 2005

Fall back / fall forward

TCP-XM principle “Multicast if possible, unicast when necessary”

Initial transmission mode is group unicastEnsures successful initial data transfer

Fall forward to multicast on positive feedbackTypically after ~75K unicast data

Fall back to unicast on repeated mcast failure

ETNGRID 2005

TCP-XM reception

ReceiverNo API-level changesNormal TCP listenAuto-IGMP join on TCP-XM connectAccepts data on both unicast/multicast ports tcp_input() accepts:

packets addressed to existing unicast destination……but now also those addressed to multicast group

Tracks how last n segs received (u/m)

ETNGRID 2005

Grid multicast?

How can multicast be used in Grid environment?

TCP-XM is new multicast-capable protocol

Globus is de-facto Grid middleware

Would like TCP-XM support in Globus…

ETNGRID 2005

Globus XIO

eXtensible Input Output libraryAllows “i/o plugins” to Globus

APISingle POSIX-like API / set of semanticsSimple open/close/read/write API

Driver abstractionHides protocol details / Allows for extensibilityStack of 1 transport & n transform driversDrivers can be selected at runtime

ETNGRID 2005

XIO architecture

ETNGRID 2005

XIO implementation

ETNGRID 2005

XIO/XM driver specifics

Two important XIO data structures

1. Handle Returned to user when XIO framework ready Used for all open/close/read/write calls lwIP netconn connection structure used

2. Attribute Used to set XIO driver-specific parameters… … and TCP-XM protocol-specific options List of destination addresses

ETNGRID 2005

XIO code example// init stackglobus_xio_stack_init(&stack, NULL);

// load drivers onto stackglobus_xio_driver_load("tcpxm", &txdriver);globus_xio_stack_push_driver(stack, txdriver);

// init attributesglobus_xio_attr_init(&attr);globus_xio_attr_cntl(attr, txdriver, GLOBUS_XIO_TCPXM_SET_REMOTE_HOSTS,

hosts, numhosts);

// create handleglobus_xio_handle_create(&handle, stack);

// send dataglobus_xio_open(&handle, NULL, target);globus_xio_write(handle, "hello\n", 6, 1, &nbytes, NULL);globus_xio_close(handle, NULL);

ETNGRID 2005

One-to-many issues

Stack assumes one-to-one connectionsXIO user interface requires modificationNeeds support for one-to-many protocolsMinimal user API changesFramework changes more significant

GSI is one-to-oneAuthentication with peer on connection setupBut cannot authenticate with n peersNeed some form of “GSI-M”

ETNGRID 2005

LAN speed

ETNGRID 2005

LAN efficiency

ETNGRID 2005

WAN speed

ETNGRID 2005

WAN efficiency

ETNGRID 2005

Driver availability

Multicast transport driver for Globus XIORequires Globus 3.2 or later

Source code onlineSample clientSample serverDriver installation instructions

http://www.cl.cam.ac.uk/~kj234/xio/

ETNGRID 2005

mcp & mcpd

Multicast file transfer application using TCP-XM

‘mcpd &’ on servers ‘mcp file host1 host2… hostN’ on client

http://www.cl.cam.ac.uk/~kj234/mcp/Full source code onlineFreeBSD, Linux, Solaris

ETNGRID 2005

All done!

Thanks for listening!

Questions?