View
223
Download
0
Tags:
Embed Size (px)
Citation preview
netkit ftpd/ftp migration
Part 3Yufei
10/01/2010
• Review
• Result– reasons
– methods
• mr and cm_id• bugs
Migrate ftpd/ftp to RDMA environment
Server start listening, and wait for new connection request
Client open new connection, then login to the server
Child use the established connection to transfer
COMMANDs and REPLIES information with Client
fork()USER, PASS,PORT, PASV,RETR, STOR
communication channel
data transfer channel
• Use librdmacm to establish the RDMA data transfer channel instead of socket TCP
Bad result
File Size RDMA TCP
1GB 1.87 secs (5.5e+05 Kbytes/sec)
1.8 secs (5.7e+05 Kbytes/sec)
100GB 444 secs (2.3e+05 Kbytes/sec)
311 secs (3.3e+05 Kbytes/sec)
reason
• http://hpdtl.cewit.stonybrook.edu/bugs/show_bug.cgi?id=1
Example – rput file1. client tells the server, use RDMA_WRITE to transfer data, so
the server has to prepare for the mr to recv data
2. the server response the client, preparation finished, and send the data sink address to the client, the longest data the server can receive
3. the client start data transfer, use RDMA_WRITE, include the information header
4. the client wait for the cq of RDMA_WRITE (RDMA_WRITE_COMPLETE)
5. the client send information to the server that the RDMA_WRITE finished
6. the server write data to the file system
7. the server tell the client continue
method• Independent Data Loading Module
– Reader and Writer
• Independent Data Transfer Module– Sender and Receiver
• Difficultly– data structure design(encapsulation)
– task decomposition
– threads communication and synchronization• Signal and condition variable
– error handling
method cont’d - Batch mr post• rdma.c
• Ibv_post_send– Ibv_send_wr
• Struct– Ibv_send_wr *next;
• In iperf and previous version of netkit ftp, I always use one task each time. If use this work list, application could save much time related to information exchange.
The relationship between memory region (mr) and communication id (cm_id) from programmer’s view
Separated? or Combined?
cm_id: communication identifymr: memory region
pd: protection domainqp: queue pair
cm_id
• int rdma_create_id ( \
struct rdma_event_channel *channel, \ struct rdma_cm_id **id, \
void *context, \
enum rdma_port_space ps);
cm_id
• int ibv_alloc_pd ( \
struct ibv_context *context);
Example:
ibv_alloc_pd(cm_id->verbs);
pd
cm_id
• int rdma_create_qp ( \
struct rdma_cm_id *id, \
struct ibv_pd *pd, \
struct ibv_qp_init_attr *qp_init_attr);
Example:
rmda_create_qp(cm_id, pd, …);
qp = cm_id->qp
pd
qp
cm_id
• struct ibv_mr *ibv_reg_mr ( \
struct ibv_pd *pd, \
void *addr, size_t length, int access);
pd
qp
mr
cm_id
pd
qp
mr
cm_id
pd
qp
mr
The relationship between memory region (mr) and communication id (cm_id) from programmer’s view
Separated: de-re-register each timeCombined: attach to the
established link
Q & A