Implementing TCP Sockets over RDMA · Implementing TCP Sockets over RDMA MacArthur and Russell...

ImplementingTCP Socketsover RDMA

MacArthurand Russell

Background

RSockets

UNH EXS

PerformanceEvaluation

Conclusions

References

Implementing TCP Sockets over RDMA

Patrick MacArthur <pio3@cs.unh.edu>Robert D. Russell <rdr@cs.unh.edu>

Department of Computer ScienceUniversity of New HampshireDurham, NH 03824-3591, USA

2nd Annual 2014 InfiniBand User Group WorkshopApril 3, 2014 2:30pm

1 / 31

Background

RSockets

UNH EXS

Conclusions

References

Acknowledgements

The authors would like to thank the University of NewHampshire InterOperability Laboratory for the use of theirRDMA cluster for the development, maintenance, and testingof UNH EXS. We would also like to thank the UNH-IOL andIxia for the use of an Anue network emulator for performancetesting.This material is based upon work supported by the NationalScience Foundation under Grant No. OCI-1127228 and underthe National Science Foundation Graduate Research FellowshipProgram under award number DGE-0913620.

2 / 31

Background

RSockets

UNH EXS

Conclusions

References

Outline

1 Background

2 RSockets

3 UNH EXS

4 Performance Evaluation

5 Conclusions

6 References

3 / 31

Background

RSockets

UNH EXS

Conclusions

References

Differences Between RDMA and TCP Sockets

“Kernel bypass”: data transfers with no OS involvement

“Zero-copy”: Direct virtual memory to virtual memorytransfers

Message-oriented

Asynchronous programming interface

TCP Sockets

Kernel involvement in all data transfers

Buffered in kernel-space on both sides of connection

Byte-stream oriented protocol

Synchronous programming interface

4 / 31

Background

RSockets

UNH EXS

Conclusions

References

TCP Sockets Data Transfer

sender receiver

User recvbuffer

User sendbuffer

IntermediateReceiveBuffer

(Kernel)IntermediateSendBuffer

(Kernel)

5 / 31

Background

RSockets

UNH EXS

Conclusions

References

Message vs. Byte Stream Semantics

Message Transfer (RDMA, UDP, SOCK SEQPACKET)

sender receiver

He l l o

Wo r l d He l l o

Byte Stream Transfer (TCP/IP)

sender receiver

He l l o

Wo r l d He l l o

Wo r l d

6 / 31

Background

RSockets

UNH EXS

Conclusions

References

Issue: O NONBLOCK is not asynchronous

Try-and-fail

recv: if no data in buffer, fail immediately with EAGAINsend: if buffer is full, fail immediately with EAGAIN

POSIX poll/select notify when an operation can start, notwhen operations complete

recv: poll/select returns when data in buffersend: poll/select returns when empty space in buffer

This is incompatible with RDMA semantics

RDMA: send or recv queued, call returns immediately,proceeds in background

7 / 31

Background

RSockets

UNH EXS

Conclusions

References

Issue: Implementing “Zero-copy”

Memory to be used for RDMA must (currently) beregistered

Existing sockets programs do not register memory to beused in I/O operations

May use any malloc’d/stack variableMay be freed at any timeSockets programmers assume memory can be reused assoon as send() returns

Not respecting adapter’s natural alignment can causesevere performance degradation, especially on FDRadapters

8 / 31

Background

RSockets

UNH EXS

Conclusions

References

Prior Implementations of Sockets over RDMA

Sockets Direct Protocol (SDP) (defined by InfiniBandspecification [InfiniBand 2011])

BCopy (buffering on both sides)ZCopy (zero-copy, send() blocks) [Goldenberg 2005]AZ-SDP (asynchronous, zero-copy, segfault handler)[Balaji 2006]

uStream (asynchronous but not zero-copy) [Lin 2009]

9 / 31

Background

RSockets

UNH EXS

Conclusions

References

Current Implementations of Sockets over RDMA

SMC-R (100% compatibility with TCP/IP and sockets)

rsockets (high-performance sockets replacement)[Hefty 2012]

UNH EXS (extended sockets)[ISC 2005, Russell 2009, MacArthur 2014]

10 / 31

Background

RSockets

UNH EXS

Conclusions

References

Outline

1 Background

2 RSockets

3 UNH EXS

5 Conclusions

6 References

11 / 31

Background

RSockets

UNH EXS

Conclusions

References

RSockets

Goal: compatibility with sockets, high performance

Built on RDMA, so kernel bypass for data transfer path

Buffer copies on both sides of connection

Supports SOCK STREAM (TCP-like) and SOCK DGRAM(UDP-like) modes

API is currently synchronous only

12 / 31

Background

RSockets

UNH EXS

Conclusions

References

RSocket Data Transfer with rsend/rrecv

sender receiver

User recvbuffer

User sendbuffer

IntermediateSendBuffer

RDMA WRITE WITH IMM

all in user space

13 / 31

Background

RSockets

UNH EXS

Conclusions

References

“Zero-copy” with rsockets

Can perform zero-copy using riomap and riowrite

riomap maps a virtual memory region to an offset

riowrite directly transfers data to iomap’d buffer identifiedby offset

Example

/* *************** at receiver ***************** */

off_t target_offset = riomap(fd, target_buf , len , PROT_WRITE , -1);

rsend(fd, &target_offset , sizeof(target_offset), 0);

rrecv(fd, empty , sizeof(empty), MSG_WAITALL );

/* *************** at sender ***************** */

off_t target_offset;

rrecv(fd, &target_offset , sizeof(target_offset), MSG_WAITALL );

/* write big buffer to server */

riowrite(fd, local_buf , length , target_offset , 0);

/* notify recipient of completion */

rsend(fd, &empty , sizeof(empty), 0);

14 / 31

Background

RSockets

UNH EXS

Conclusions

References

Outline

1 Background

2 RSockets

3 UNH EXS

5 Conclusions

6 References

15 / 31

Background

RSockets

UNH EXS

Conclusions

References

UNH EXS (Extended Sockets)

Based on ES-API (Extended Sockets API) published bythe Open Group [ISC 2005]

Extensions to sockets API to provide asynchronous,zero-copy transfers

Memory registration (exs mregister(), exs mderegister())Event queues for completion of asynchronous events(exs qcreate(), exs qdequeue(), exs qdelete())Asynchronous operations (exs send(), exs recv(),exs accept(), exs connect())

UNH EXS supports SOCK SEQPACKET (reliablemessage-oriented) and SOCK STREAM (reliablestream-oriented) modes

16 / 31

Background

RSockets

UNH EXS

Conclusions

References

UNH EXS Programming

Example

Example asynchronous send operation

exs_mhandle_t mh = exs_mregister(buf , bufsize , EXS_ACCESS_READ );

exs_qhandle_t qh = exs_qcreate (10);

if (exs_send(fd, buf , bufsize , 0, mh , 0, qh) < 0) {

perror("Could not start send operation");

/* bail out */

/* do work in parallel with data transfer */

exs_event_t ev;

if (exs_qdequeue(qh , &ev, 1, NULL) < 0) {

perror("Could not get send completion event");

/* bail out */

fprintf(stderr , "Send of %d/%d bytes complete with errno=%d\n",

bufsize , ev.exs_evt_union.exs_evt_xfer.exs_evt_length ,

ev.exs_evt_errno );

17 / 31

Background

RSockets

UNH EXS

Conclusions

References

UNH EXS Protocol

Direct Transfer (SOCK STREAM and SOCK SEQPACKET)

ADVERT

DIRECT

sender receiver

User recvbuffer

User sendbuffer

exs_send

exs_recv

Indirect Transfer (SOCK STREAM only)

INDIRECT

sender receiver

User recvbuffer

User sendbuffer

exs_send

exs_recv

18 / 31

Background

RSockets

UNH EXS

Conclusions

References

Outline

1 Background

2 RSockets

3 UNH EXS

5 Conclusions

6 References

19 / 31

Background

RSockets

UNH EXS

Conclusions

References

Performance Evaluation

Comparison of TCP, rsockets, and EXS using rstream andriostream tools

rsockets rrecv()/rsend(): rstreamrsockets riomap()/riowrite(): riostreamTCP: rstreamEXS: rstream-like utility modified to take advantage ofasynchronous operations

No optimization done for rsocket and TCP cases

Systems used: Intel Xeon 2.40 GHz E5-2609 CPUs, 64 GBRAM, PCI-e Gen 3

HCAs: Mellanox ConnectX-3 FDR InfiniBand HCAsconnected via Mellanox SX6036 FDR InfiniBand switch

20 / 31

Background

RSockets

UNH EXS

Conclusions

References

Throughput Comparison

512 B 8 KiB 128 KiB 2 MiB 32 MiB 512 MiB

ughput

Message size

streamexsrstream

riostreamrstream-tcp

21 / 31

Background

RSockets

UNH EXS

Conclusions

References

CPU Usage Comparison

Message size

Client CPU Utilization

streamexsrstream

Message size

CPU Percent Server

streamexsrstream

22 / 31

Background

RSockets

UNH EXS

Conclusions

References

Latency Comparison

32 B 128 B 512 B 2 KiB 8 KiB 32 KiB 128 KiB 512 KiB

conds)

Message size

streamexsrstream

23 / 31

Background

RSockets

UNH EXS

Conclusions

References

UNH EXS SOCK SEQPACKET vs.SOCK STREAM

ughput

Message size

pumpexs-seqpacketpumpexs-stream

Message size

Client CPU Utilization

100000

conds)

Message size

Server CPU Utilization

24 / 31

Background

RSockets

UNH EXS

Conclusions

References

Outline

1 Background

2 RSockets

3 UNH EXS

5 Conclusions

6 References

25 / 31

Background

RSockets

UNH EXS

Conclusions

References

Conclusions

Socket buffering is implicit in sockets API

Efficient zero-copy requires extensions to API

ES-API provides asynchronous operationrsockets provides riomap()/riowrite()

Message-oriented is more efficient to implement overRDMA than byte stream-oriented

26 / 31

Background

RSockets

UNH EXS

Conclusions

References

Outline

1 Background

2 RSockets

3 UNH EXS

5 Conclusions

6 References

27 / 31

Background

RSockets

UNH EXS

Conclusions

References

References I

Infiniband Trade Association,“Supplement to Infiniband Architecture SpecificationVolume 1, Release 1.2.1: Annex A4: Sockets DirectProtocol (SDP),”Oct. 2011.

D. Goldenberg, M. Kagan, R. Ravid, and M. S. Tsirkin,“Zero copy sockets direct protocol overInfiniband—preliminary implementation and performanceanalysis,”in High Performance Interconnects, 2005. Proceedings.13th Symposium on. IEEE, 2005, pp. 128–137.

28 / 31

Background

RSockets

UNH EXS

Conclusions

References

References II

P. Balaji, S. Bhagvat, H.-W. Jin, and D. K. Panda,“Asynchronous zero-copy communication for synchronoussockets in the sockets direct protocol (SDP) overInfiniBand,”in Parallel and Distributed Processing Symposium, 2006.IPDPS 2006. 20th International. IEEE, 2006, pp. 8–pp.

Y. Lin, J. Han, J. Gao, and X. He,“uStream: a user-level stream protocol over InfiniBand,”in Parallel and Distributed Systems (ICPADS), 2009 15thInternational Conference on. IEEE, 2009, pp. 65–71.

29 / 31

Background

RSockets

UNH EXS

Conclusions

References

References III

S. Hefty,Rsockets.Available: https://www.openfabrics.org/

ofa-documents/doc_download/495-rsockets.html

Interconnect Software Consortium in association with theOpen Group,“Extended Sockets API (ES-API) Issue 1.0,”Jan. 2005.

——,“A General-purpose API for iWARP and InfiniBand,”in the First Workshop on Data Center Converged andVirtual Ethernet Switching (DC-CAVES), Sep. 2009.

30 / 31

Background

RSockets

UNH EXS

Conclusions

References

References IV

P. MacArthur and R. Russell,“An Efficient Method for Stream Semantics over RDMA,”in Proceedings of the 28th IEEE International Parallel andDistributed Processing Symposium (IPDPS 2014),May. 2014, to appear.

31 / 31

Implementing TCP Sockets over RDMA · Implementing TCP Sockets over RDMA MacArthur and Russell...

Documents

OFED Storage Protocols...Reliable Datagram Service RDS iSCSI RDMA Protocol (Initiator) iSER SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP IPoIB IP over InfiniBand

Elementary TCP Sockets Unix Network Programming Ch # 4

Elementary TCP Sockets

Tcp Sockets

REXX/SOCKETS VM REXX TCP/IP Socket -

TCP, UDP, and Sockets: rigorous and experimentally

Offload Sockets Framework and Sockets Direct ... - InfiniBandinfiniband.sourceforge.net/NW/SDP/OSF_SDP_HLD.pdf · utilizes various InfiniBand features (such as remote DMA (RDMA),

TEE Sockets API Specification v1.0 - GlobalPlatform · TEE Sockets API Specification Annex A: TCP/IP Specification of TEE Sockets API Specification [Sockets TCP/IP] GPD_SPE_102 :

TCP/IP Sockets in C: Practical Guide for Programmers

Elementary TCP Sockets - WPIrek/Undergrad_Nets/B07/TCP_Sockets.pdfUNIX Network Programming Vol. 1, Second Ed. Stevens ... System Calls for Elementary TCP Sockets #include

Using TCP Through Sockets - Stanford University

TCP/IP Sockets in C: Practical Guide for Programmerscs.ecs.baylor.edu/~donahoo/practical/CSockets/PracticalSocketC.pdf · TCP/IP Sockets in C: Practical Guide for Programmers Michael

Chapter04 Elementary TCP Sockets

TCP, UDP, and Sockets: rigorous and experimentally-validated … · 2005. 3. 18. · TCP, UDP, and Sockets: rigorous and experimentally-validated behavioural speciﬁcation Volume

TCP/IP Sockets in Java - doc.lagout.org Sockets in Java... · TCP/IP sockets in Java : ... visit our Web site at or ... students can then move on

TCP/IP Sockets in Java - WordPress.com · 2012-04-16 · 1.6 Exercises 8 2 Basic Sockets 9 2.1 Socket Addresses 9 2.2 TCP Sockets 15 2.2.1 TCP Client 16 2.2.2 TCP Server 21 2.2.3

Fermi Cluster for Real-Time Hyperspectral Scene Generation · SDP Sockets Direct Protocol SRP SCSI RDMA Protocol (Initiator) iSER iSCSI RDMA Protocol (Initiator) RDS Reliable Datagram

Sockets Tcp Ip Cet Arci

NFS/RDMA over 40Gbps iWARP · 2019. 12. 21. · Reliable Datagram Service . RDS iSCSI RDMA Protocol (Initiator) iSER . SCSI RDMA Protocol (Initiator) SRP Sockets Direct Protocol SDP

UDP and TCP Sockets in Java