Tackling Parallelization Challenges of Kernel Network

Tackling Parallelization Challenges of Kernel Network Stack for Container Overlay Networks

Jiaxin Lei*, Kun Suo+, Hui Lu*, Jia Rao+

* SUNY at Binghamton+ University of Texas at Arlington

Containers Are Widely Adopted by Industry

•OS level virtualization

•Lightweight

•Higher consolidation density

•Lower operational cost



•Lightweight





•Lightweight



Overlay Networks Are the Technique For Containers Connectivity•Typical overlay network solutions:

Docker Overlay, Flannel, Calico, Weave

•They are generally built upon the tunneling approach like using VxLAN protocol.

Overlay Networks Are the Technique For Containers Connectivity

Inner IP Header

Inner Ethernet Header Payload

Original L2 Frame

•Typical overlay network solutions: Docker Overlay, Flannel, Calico, Weave



Inner IP Header


Original L2 Frame

VxLAN Header

Outer UDP Header

Outer IP Header

Outer Ethernet Header FCS

VxLAN Encapsulated Packet




Inner IP Header


Original L2 Frame

VxLAN Header

Outer UDP Header

Outer IP Header



VxLAN Flags Reserved VNI Reserved

VxLAN Network Identifier




Inner IP Header


Original L2 Frame

VxLAN Header

Outer UDP Header

Outer IP Header








Inner IP Header


Original L2 Frame

VxLAN Header

Outer UDP Header

Outer IP Header







Network Packet Processing Path•Prolonged network packet processing path

•Additional virtual devices overhead

Network Packet Processing Path

Receiving Side

•Prolonged network packet processing path


Native


NIC

Receiving Side Kernel Space

Container Applications



Native

User Space


NIC

Receiving Side


IRQ



Kernel Space User Space

Native


NIC

Receiving Side

Network Stack


IRQ SoftIRQ




Native


NIC

Receiving Side

Network Stack


IRQ SoftIRQ




Native


NIC

Receiving Side

Network Stack


IRQ SoftIRQ




Native


NIC

Receiving Side

Network Stack


IRQ SoftIRQ




Native


NIC

Receiving Side

Network Stack


IRQ SoftIRQ




Native



NIC

Receiving Side



Overlay




NIC

Receiving SideIRQ SoftIRQ

Network Stack



Overlay




NIC

Receiving SideIRQ

VxLANSoftIRQ

Network StackSoftIRQ




Overlay



NIC

Receiving SideIRQ

VxLANSoftIRQ


vBridge Veth

Packet decapsulation




Overlay



NIC

Receiving SideIRQ

VxLANSoftIRQ


vBridge Veth Network StackSoftIRQ





Overlay



NIC

Receiving SideIRQ

VxLANSoftIRQ







Overlay



NIC

Receiving SideIRQ

VxLANSoftIRQ







Overlay



NIC

Receiving SideIRQ

VxLANSoftIRQ







Overlay


NIC

Receiving SideIRQ

VxLANSoftIRQ



Existing Optimizations for Packet Processing




NIC

Receiving SideIRQ

VxLANSoftIRQ





IRQ coalescing GRO



NIC

Receiving Side User Space

IRQVxLAN

SoftIRQNetwork Stack

SoftIRQ




IRQ coalescing GRO RPSMulti-queue

Kernel Space

Experimental SettingsHardware Software

CPU Memory NIC Throughput CPU Usage2.2 GHz 64GB 40Gbps iPerf3 mpstat10 cores Multi-queue

Experimental SettingsHardware Software

CPU Memory NIC Throughput CPU Usage2.2 GHz 64GB 40Gbps iPerf3 mpstat10 cores Multi-queue

S1 - Native S2 - Linux Overlay S3 - Docker Overlay

Experimental Settings

Host1 Host2

eth0

iPerf3 iPerf3

eth0

Hardware SoftwareCPU Memory NIC Throughput CPU Usage

2.2 GHz 64GB 40Gbps iPerf3 mpstat10 cores Multi-queue



Host1 Host2 Host1 Host2

eth0

VxLANiPerf3 iPerf3

iPerf3 iPerf3

eth0 eth0 eth0

VxLAN





S3 - Docker Overlay

Host1 Host2 Host1 Host2 Host1 Host2

eth0

VxLAN

vBridge

Veth0

iPerf3

iPerf3 Container

iPerf3 Container

iPerf3

iPerf3 iPerf3

eth0 eth0 eth0 eth0 eth0

VxLAN VxLAN VxLAN

vBridge

Veth0

S2 - Linux Overlay



S1 - Native

Single Flow Performance

TCP and UDP Throughputs under Three Different Cases

Thro

ughp

ut (G

b/s)

0

5

10

15

20

25

S1 S2 S3

6.46.5

23 TCP

• TCP Throughput of Docker Overlay case drops 72% compared with native case.



Thro

ughp

ut (G

b/s)

0

5

10

15

20

25

S1 S2 S3

6.46.5

23 TCP

• TCP Throughput of Docker Overlay case drops 72% compared with native case.

• UDP Throughput drops 58%.



Thro

ughp

ut (G

b/s)

0

5

10

15

20

25

S1 S2 S3

3.94.7

9.36.46.5

23 TCPUDP


CPU Usage under Three Different Cases for TCP

Tota

l CPU

Usa

ge

0%

2%

4%

6%

8%

10%

S1 S2 S3

User

System

SoftIRQ

* 5% indicates one cpu core is fully saturated.


• Packet processing overhead fully saturates one cpu core in two overlay cases.


Tota

l CPU

Usa

ge

0%

2%

4%

6%

8%

10%

S1 S2 S3

User

System

SoftIRQ

~5%




Tota

l CPU

Usa

ge

0%

2%

4%

6%

8%

10%

S1 S2 S3

User

System

SoftIRQ

~5% • Packet processing overhead fully saturates one cpu core in two overlay cases.

• Current solutions can’t scale single flow performance.


Multiple Flows Performance

0

10

20

30

40

1 2 3 4 5 6 ... 10 20 40 80

Native

Pair Number of Iperf Connection

Thro

ughp

ut (G

b/s)


0

10

20

30

40

1 2 3 4 5 6 ... 10 20 40 80

Native


Thro

ughp

ut (G

b/s)

• Native case quickly reaches ~37 Gbps under TCP with only 2 pairs.

~37Gbps


0

10

20

30

40

1 2 3 4 5 6 ... 10 20 40 80

NativeLinux OverlayDocker Overlay


Thro

ughp

ut (G

b/s)


• In two overlay cases, TCP throughput grows slowly.


0

10

20

30

40

1 2 3 4 5 6 ... 10 20 40 80



Thro

ughp

ut (G

b/s)


• In two overlay cases, TCP throughput grows slowly.


• Under the same throughput (e.g., 40 Gbps), overlay networks consume much more CPU resources (e.g., around 2.5 times) than the native case.

0%10%20%30%40%50%60%70%

1 2 3 4 5 6 ... 10 20 40 80 1 2 3 4 5 6 ... 10 20 40 80 1 2 3 4 5 6 ... 10 20 40 80

UserSystemSoftIRQ

Native Linux Overlay Docker Overlay Pair Number of Iperf Connection

Tota

l CPU

Usa

ge~2.5 times


• Bad scalability is largely due to the inefficient interplay of many tasks.

0%10%20%30%40%50%60%70%

1 2 3 4 5 6 ... 10 20 40 80 1 2 3 4 5 6 ... 10 20 40 80 1 2 3 4 5 6 ... 10 20 40 80

UserSystemSoftIRQ

Native Linux Overlay Docker Overlay Pair Number of Iperf Connection

Tota

l CPU

Usa

ge~2.5 times

Small Packet Performance

0

60,000

120,000

180,000

240,000

64B128B

256B512B

1KB2KB

4KB8KB


Pack

et N

umbe

r /s

Packet Size of Iperf Connection

Small Packet Performance

0

60,000

120,000

180,000

240,000

64B128B

256B512B

1KB2KB

4KB8KB


Pack

et N

umbe

r /s

Packet Size of Iperf Connection

• Docker overlay achieves as low as 50% packet processing rate of that in the native case.

Interrupt Number with Varying Packet Sizes

0

10,000

20,000

30,000

40,000

64B128B256B512B1KB2KB4KB8KB



IRQ SoftIRQ Packet Size of Iperf Connection

IRQ

/s (S

oftIR

Q/s

)


IRQ

/s (S

oftIR

Q/s

)

0

100,000

200,000

300,000

400,000




Interrupt number for TCP Interrupt number for UDP

• IRQ number increases dramatically in the Docker overlay UDP case — 10x of that in the TCP case.

Interrupt Number with Varying Packet Sizes

0

10,000

20,000

30,000

40,000





IRQ

/s (S

oftIR

Q/s

)


IRQ

/s (S

oftIR

Q/s

)

0

100,000

200,000

300,000

400,000




Interrupt number for TCP Interrupt number for UDP

• IRQ number increases dramatically in the Docker overlay UDP case — 10x of that in the TCP case.

• 3x softIRQ numbers are observed in Docker Overlay case compared with the IRQ numbers.

Insights and Conclusions

Insights and Conclusions• Kernel does not provide per-packet level parallelization.

Insights and Conclusions• Kernel does not provide per-packet level parallelization. • Kernel does not efficiently handle various packet processing tasks.

Insights and Conclusions• Kernel does not provide per-packet level parallelization. • Kernel does not efficiently handle various packet processing tasks. • Bottlenecks become more severe for small packets.


Thinking about future works:


Thinking about future works:• Is it feasible to provide packet-level parallelization for a single network

flow?



flow?

• How can the kernel perform a better isolation among multiple flows especially for efficiently utilizing shared hardware resources?



flow?

• How can the kernel perform a better isolation among multiple flows especially for efficiently utilizing shared hardware resources?

• Can the packets be further coalesced with optimized network path for reduced interrupts and context switches?

Thank you!

Documents

Tackling Parallelization Challenges of Kernel Network