Oct 2012

Socket Direct Protocol over PCI Express Interconnect: Design, Implementation and Evaluation

By Ahmed Bukhamsin

1

School of Computing Science / Ahmed Bukhamsin

Outline

• PCI Express (PCIe) Overview• PCIe Interconnect• Proposed System• Implementation• Evaluation• Conclusion

2

School of Computing Science / Ahmed Bukhamsin

PCI Express Overview 1

• PCI = Peripheral Component Interconnect

3

School of Computing Science / Ahmed Bukhamsin

PCI Express Overview 2

4

School of Computing Science / Ahmed Bukhamsin

PCI Express Overview 3

• PCIe Evolution• High throughput• Low latency• Low power consumption• PCIe over cable

* Aggregate bandwidth implies simultaneous traffic in both directions.

5

School of Computing Science / Ahmed Bukhamsin

PCIe Interconnect 1

• Issues with connecting two root complexes• Bus ownership• Endpoints resource allocation

• Solution: Non-transparent bridge (NTB)

6

School of Computing Science / Ahmed Bukhamsin

PCIe Interconnect 2

• NTB isolates the connected PCIe domains• NTB provides a communication mechanism between the connected PCIe

domains

7

School of Computing Science / Ahmed Bukhamsin

PCIe Interconnect 3

• NTB communication mechanism • Acquires memory in both systems• Provides address translation to allow data exchange• Provides doorbell and scratchpad registers accessible from both sides

8

School of Computing Science / Ahmed Bukhamsin

Proposed System 1

• We propose to develop the Socket Direct Protocol (SDP)

• SDP makes it possible to run TCP based socket applications without modifications

9

School of Computing Science / Ahmed Bukhamsin

Proposed System 2

10

Applications

SDP Socket Preloaded Library

Protocol Switch

SDP

TCP

IP

Communication API

NTB Driver

User Space

Kernel Space

NIC Driver

PCIe NTB CardEthernet NIC

DMA Driver

School of Computing Science / Ahmed Bukhamsin

Proposed System 2

• SDP Socket Preloaded Library• Rule based protocol switch

11

Applications

SDP Socket Preloaded Library

Protocol Switch

SDP

TCP

IP

Communication API

NTB Driver

User Space

Kernel Space

NIC Driver

PCIe NTB CardEthernet NIC

DMA Driver

School of Computing Science / Ahmed Bukhamsin

Proposed System 2

• SDP Socket Preloaded Library• Rule based protocol switch

• SDP Kernel Module• The main component of the solution• Register itself as new address family• Implements all TCP functions

12

Applications

SDP Socket Preloaded Library

Protocol Switch

SDP

TCP

IP

Communication API

NTB Driver

User Space

Kernel Space

NIC Driver

PCIe NTB CardEthernet NIC

DMA Driver

School of Computing Science / Ahmed Bukhamsin

Proposed System 2

• SDP Socket Preloaded Library• Rule based protocol switch

• SDP Kernel Module• The main component of the solution• Register itself as new address family• Implements all TCP functions

• Communication API• Group of functions exported by the

driver for SDP to access the driver

13

Applications

SDP Socket Preloaded Library

Protocol Switch

TCP

IP

Communication API

NTB Driver

User Space

Kernel Space

NIC Driver

PCIe NTB CardEthernet NIC

DMA Driver

SDP

School of Computing Science / Ahmed Bukhamsin

Proposed System 2

14

Applications

SDP Socket Preloaded Library

Protocol Switch

TCP

IP

Communication API

NTB Driver

User Space

Kernel Space

NIC Driver

PCIe NTB CardEthernet NIC

DMA Driver

SDP

• SDP Socket Preloaded Library•Rule based protocol switch

• SDP Kernel Module• The main component of the solution•Register itself as new address family• Implements all TCP functions

• Communication API•Group of functions exported by the

driver for SDP to access the driver

• DMA and NTB Drivers•Cards manufacturer drivers

School of Computing Science / Ahmed Bukhamsin

Implementation 1

• SDP Connection

Server Client

- Listen()

- accept()

Scratchpad registers

- Connect()

Interrupt

Create client Buffer

- Read registersServer BufferCreate server Buffer

IP and Port

Interrupt

Connection establishedInterrupt

Connection established

Server port

Ready

Conniction ID

client buf addr

Client BufferServer buff addr

-Read Reg.

15

School of Computing Science / Ahmed Bukhamsin

Implementation 2

• Send and Receive

Data Source Data SinkTime

recv 1

Data Ready Signal

DMA Data Copy

Wa

itin

g

sendmsg 1

...

sendmsg 2sendmsg 3

sendmsg n

copy 1

recv 2

copy 2

recv 3

copy 3

...

recv n

copy n

SDP Buf SDP BufApplication Application

... ... ...

...

16

School of Computing Science / Ahmed Bukhamsin

Evaluation 1

• Benchmarks• Throughput• Latency• CPU Utilization

• Application Testing• Apache Web Server

17

School of Computing Science / Ahmed Bukhamsin

Evaluation 2

• Throughput

18

School of Computing Science / Ahmed Bukhamsin

Evaluation 3

• Throughput

19

School of Computing Science / Ahmed Bukhamsin

Evaluation 3

• Throughput

20

School of Computing Science / Ahmed Bukhamsin

Evaluation 3

• Throughput

21

School of Computing Science / Ahmed Bukhamsin

Evaluation 3

• Throughput

22

School of Computing Science / Ahmed Bukhamsin

Evaluation 4

• Latency

23

School of Computing Science / Ahmed Bukhamsin

Evaluation 5

• Application

24

School of Computing Science / Ahmed Bukhamsin

Evaluation 5

• Application

25

School of Computing Science / Ahmed Bukhamsin

Evaluation 5

• Application

26

School of Computing Science / Ahmed Bukhamsin

Evaluation 5

• Application

27

School of Computing Science / Ahmed Bukhamsin

Conclusion

• We design and implement SDP• We evaluate it and compare it to SDP over Infiniband• Our SDP offers comparable performance to Infiniband for concurrent

transfers of large message sizes• Application testing shows more u 10% improvement in performance

over Infiniband• PCIe can serve as an interconnect for communication-intensive

applications

28