EtherPIPE: an Ethernet character device for network scripting

Yohei Kuga (Keio University)Takeshi Matsuya (Keio University)

Hiroaki Hazeyama (NAIST)Kenjiro Cho (IIJ)

Osamu Nakamura (Keio University)

HotSDN’13, August 16, 2013, Hong Kong, China 1

Motivation• Shell scripting is powerful to process files• We want to use it for network processing

2

$ cat /dev/port0 > ./capture

Our Goal

Use Basic UNIXcommand Capture

packetsReceive packets

Motivation• Shell scripting is powerful to process files• We want to use it for network processing

3

$ cat /dev/port0 >/dev/port1

Our Goal

Use Basic UNIXcommand

Receive packets Send packets

Motivation• Shell scripting is powerful to process files• We want to use it for network processing

• It’s useful for SDN prototyping and debugging4

$ ./FW.sh </dev/port0 >/dev/port0

Our Goal

Send packetsReceive packetsBuild a networkapplication

EtherPIPE

• A character device for packet processing– It accesses the device using open(2)– And, handles packets using read(2) and write(2)Then UNIX commands can read/write packets as a file

5

Network Scripting

• Lightweight network programming by shell scripts– Combining UNIX commands, redirections, pipes with

EtherPIPE devices

6

EtherPIPE design: Device namespace

7

• ‘0’ and ‘1’: Shell Interface– ASCII format. For shell scripting

• ‘r0’ and ‘r1’: Raw Interface– Binary format. For high-bandwidth and complex applications

EtherPIPE design: Shell Interface (ASCII)

8

• one packet per line• Device driver converts raw data to ASCII• Each byte is separated by “space” character

• To make string-based UNIX commands to process packets– But the data size is 3 times larger

12 words 12 words 4 words 2 words

Shell Interface: Applications• Sending a packet

$ echo “FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00 22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20“ >/dev/ethpipe/0

• Port mirroring$ cat /dev/ethpipe/0 | tee /dev/ethpipe/0 > /dev/ethpipe/1

• MAC address filtering$ grep ‘^FFFFFFFFFFFF’ /dev/ethpipe/0 > /dev/ethpipe/1

• VLAN untagging$ sed -e 's/8100 00 01 //' < /dev/ethpipe/0 > /dev/ethpipe/1

9

EtherPIPE design: Raw interface (Binary)

• For simple capturing and replaying, and for optimized applications

• Raw format includes fields for offloading– Hardware timestamp

• nanosecond RX timestamp for precise capturing

– Five-tuple hash for flow processing• Hash # <src and dst IP addr,

proto num, src and dst port num>

10

Prototype implementations

1. Device driver (Shell IF) for generic NICs– For proof-of-concept

2. FPGA card + device driver– Dev kit: LatticeECP3 versa kit ($99, 1GE x2 and PCIe)– Supports 1000BASE-T full duplex– Shell IF device driver (working)– Raw IF device driver (under development)

• FPGA logic and drivers source code are available– https://github.com/sora/ethpipe

11

Prototype implementations

1. Device driver (Shell IF) for generic NICs– For proof-of-concept

2. FPGA card + device driver– Dev kit: LatticeECP3 versa kit ($99, 1GE x2 and PCIe)– Supports 1000BASE-T full duplex– Shell IF device driver (working)– Raw IF device driver (under development)

• FPGA logic and drivers source code are available– https://github.com/sora/ethpipe

12

RX: 1GE line-rateTX: near 1GE line-rate

13

How to measure the transmit PPS

• Measured the TX PPS of Shell IF by `wc`, `cat` and `time`

• Performs the near 1GE line-rate PPS1,481,481 pps vs.1,488,095 pps for 64B pkts

$ head ./4m.pktFFFFFFFFFFFF 0022CF63967B 8899 23 .. FFFFFFFFFFFF 0022CF63967B 8899 23 ..FFFFFFFFFFFF 0022CF63967B 8899 23 ..

$ wc ./4m.pkt 4000000 196000000 676000000 4m.pkt

$ time cat ./4m.pkt >/dev/ethpipe/0real 0m2.700sUser 0m0.004ssys 0m2.661s

$ bcscale=34000000/2.7001481481.481

Enough for prototyping

Summary• We proposed “Network scripting”, new design of network

IO for packet processing, and showed a prototype implementation– Allows shell scripting to handle network devices w/ ‘<, >, |’ – Simple data format for UNIX commands and scripting languages

• Future work– Fix Shell and Raw IF formats

• E.g,: More offloading fields (currently: RX timestamp and 5-tuple hash)– Finish Raw IF driver and its userland libraries

14

15

FPGA adapter design

16Receiving Sending

Performance transmit PPS of FPGA prototype: data

$ head 4m.pktFFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00 22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20FFFFFFFFFFFF 0022CF63967B 8899 23 9C 15 E3 FE 32 40 00 22 CF 63 96 7B 00 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20…

17

Host operations (tappipe.c)

• To bind EtherPIPE device with OS network stack by TAP device– To use existing network commands (arp, dhclient, tcpdump,

etc)– Build custom layer 2~3 functions by shell script (ping,

traceroute, nmap, etc)

$ ./tappipe pipe0 </dev/ethpipe/0 >/dev/ethpipe/0 &$ ip addr pipe0$ tcpdump -i pipe0$ dhclient –r pipe0 && dhclient pipe0

$ echo ${PING_REQUEST} > /dev/ethpipe/018

Performance result ofcommand-line based packet processing

19

• Measuring the potential of Shell IF throughput by dummy driver– Read(): Dummy driver prepare the 64 Byte packet in kernel– Write(): the driver simply copies the data into a buffer in the kernel

• Result– ‘Capture’ (memory copy from kernel to userland) performs over 10GE– Some command is very slow such as `cut`

Our focus• A typical network device file is designed for

communications between hosts– A network device file has to provide all functions / APIs

from Layer 2 to Layer 4 – An implementation would be as a special device– Device specific settings and methods are required to

send/receive packets

• We focus on a network device file for packet processing– Only provide IO of physical ethernet ports and doesn't care

other functions (MTU, reassemble ,FIB, etc)– An implementation is developed as a common device file– Basic system calls are enough to send/receive packets 20