Docker infiniband

Docker + InfiniBand@syoyo

Docker Meetup in Tokyo #1(Feb 12, 2014)

Thursday, February 13, 14

Why InfiniBand?

• Faster networking

• 3.2 GB/s on QDR

• Low laterncy

• ~3 us

• de-facto inter-connect standard in HPC environment


Crehan Research


My contribution (1/2)


My contribution (2/2)

• OFED userland port for OpenIndiana(Solaris)

• https://github.com/syoyo/oi-build/tree/ofed-1.5.4.1/components/open-fabrics


Existing virtualization technique

• SR-IOV

• HW dependent.

• Need driver/firmware support

• PCI-passthrough

• Only one guest can access

• ZeroVM

• NaCl-based application isolation


LXC

• Lightweight virtualization

• Direct-access to PCI-ex devices


Existing research

• Performance Evaluation of Container-based Virtualization for High Performance Computing Environments

• Xavier, M.G. et. al. PDP2013

• http://marceloneves.org/papers/pdp2013-containers.pdf


http://marceloneves.org/papers/pdp2013-containers.pdf




• Simple, easy but powerful!


User A User B User C

Memory

CPU

OS

(*) http://commons.wikimedia.org/wiki/Smiley

(*)


http://commons.wikimedia.org/wiki/Smiley

http://commons.wikimedia.org/wiki/Smiley

Container A

Memory

CPU

Container B Container C

OS

Memory

CPU

Memory

CPU


User A User B User C

/opt/gcc-4.4

/opt/gcc-4.5

/opt/gcc-4.6


Container A

gcc4.4

Container B Container C

OS

gcc4.5 gcc4.6


Container(CentOS6.5)

CentOS 6.5

IB switch

ConnectX-2

CentOS 6.5

ConnectX-2

192.168.11.16

192.168.11.8


• Privilege : on

• bind /dev/infiniband

$ sudo docker run -privileged \ -v /dev/infiniband:/dev/infiniband \ -i -t centos6-ib /bin/bash



# ib_read_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 1000 3008.83 3008.80 0.048141----------------------------------------------------

# ib_write_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 5000 3066.07 3066.05 0.049057---------------------------------------------------------------------------------------

# ib_read_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 1.66 20.13 2.65 ---------------------------------------------------------------------------------------

# ib_write_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 0.80 7.12 0.86 ---------------------------------------------------------------------------------------

LXC Container


$ ib_read_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 1000 3015.43 3011.54 0.048185---------------------------------------------------------------------------------------

$ ib_write_bw 192.168.11.16…----------------------------------- #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 65536 5000 3065.79 3065.63 0.049050--------------------------------------------------------------------------------

$ ib_read_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 1.68 39.48 2.67 ---------------------------------------------------------------------------------------

$ ib_write_lat 192.168.11.16…----------------------------------- #bytes #iterations t_min[usec] t_max[usec] t_typical[usec] 2 1000 0.80 4.70 0.86 ---------------------------------------------------------------------------------------

Native


Details

• http://qiita.com/syoyo/items/bea48de8d7c6d8c73435

• in Japanese


Summary

• Docker + InfiniBand works!

• No virtualization overhead!


Future work(1/3)

• Per-container(per-user) access control for IB device

• e.g. IB port0 -> User A, IB port1 -> User B

• Security concern

• Other PCI-ex devices

• GPU

• PCI-SSD

• etc.


Future work(2/3)

• Virtualized, highly-efficient cloud platform for rendering task.


Render A Render B Render C Render D

Render E

InfiniBand

etcd

Future work(3/3)


• Thank you!


Self Improvement

Docker infiniband