Pushing Suricata Towards 90 Gbit s and More€¦ · Pushing Suricata Towards 80 Gbit/s and More | 2019-10-31 | Tobias Appel 20 . Conclusion • 100Gbps in a Single Server is still

Pushing Suricata Towards 80Gbps and More

Leibniz Supercomputing Centre | October 30th 2019 | Tobias Appel

Motivation

•  Our old Suricata installation needed a hardware upgrade due to increase in bandwidth

•  Nearly no hardware recommendations for Suricata operating at high-speed available

•  New installation should be „future-proof“ using cutting-edge technology (XDP & Flow Shunting)

•  CPU recommendation is easy ! buy as many cores as you can afford (same for RAM)

•  But what about the NIC?

2 Pushing Suricata Towards 80 Gbit/s and More | 2019-10-31 | Tobias Appel

Acknowledgement

•  Thanks to the Suricata community and the Suricata developers •  Peter Manev •  Eric Leblond

•  Without them, this project wouldn‘t have been possible

•  Also thanks to the hardware vendors for providing us with their latest NICs free of charge during the evaluation period •  Napatech •  Accolade


High Performance Computing SuperMUC-NG, LRZ Linux Cluster

Virtual Reality and Visualisation V2C (CAVE, Powerwall)

IT Service Backbone for the Advancement of Research Science LRZ as an IT Center of Excellence

Storage

Network

Cloud Computing

Cluster

HPC

Training

Consultancy

Email

High Speed Networking Munich Scientific Network

Big Data Bavarian State Library Digital Archive

Pushing Suricata Towards 80 Gbit/s and More | 2019-10-31 | Tobias Appel 4

5

LRZ “SuperMUC-NG” Top500 (June 2019): #9

Pushing Suricata Towards 80 Gbit/s and More | 2019-10-31 | Tobias Appel

The Munich Scientific Network


•  Backbone for Munich Universities and Scientific Institutions

>136.000 Students >30.000 Scientific Staff 14 Core-Router 72 On-Site-Router >1.900 Switches >3800 Access Points 77 Leased Dark Fibre >200.000 Devices 59 Locations and >600 Buildings

•  Transmitted Data 3.400 / 1.200 TByte / Month (in-/outgoing)

Septun Mark I and II

•  Extreme Performance Tuning Guide from 2016 aims at sustained 20Gbps (Standard Hardware with Intel and AF_Packet)

•  https://github.com/pevma/SEPTun

•  SEPTun Mark II Guide from 2017 – focuses on the introduction of eXpress Data Path (XDP)

•  https://github.com/pevma/SEPTun-Mark-II

•  Conclusion: No problem to handle 20Gbps per Server (with some tuning)

•  But what about 100Gbps?


XDP & Flow Shunting

•  With the power of XDP we can inspect and discard unwanted packets

•  No need to waste CPU resources on Netflix streams or encrypted Traffic

•  XDP allows us to drop packets at various stages:

•  On the NIC directly (requires compatible hardware, best option) •  At driver level (requires XDP compatible driver) •  At kernel level (fallback, slowest option)


Network Card Considerations

•  We choose well-known vendors with Suricata and (hopefully) XDP/Flow shunting support

•  Due to NUMA limitations, we plan a dual CPU / dual NIC setup per server


Hardware Setup

•  Identical Servers •  Dell R740 Chassis •  Dual CPU Intel Xeon Platinum 8168 (24 Cores + HT) •  256 GB RAM •  2x 480GB SAS SSD in Raid 1 •  Debian 10 (Kernel 4.19) •  Latest Suricata with all rules enabled

•  Using passive Taps:


Ixia Vision Edge 100 Configuration

•  8x 100Gbps configured as Network Port (Traffic from Wire Tap) •  Multiple 40/100Gbps Tool Ports ! Mirror Traffic to each Server


Generating Traffic in the Lab is not as easy expected

•  PCAP Replay not suitable for us •  If done locally, it will not use the physical NIC •  Replay using the network requires powerful hardware at 100 Gbit/s (bottlenecks are

HDD / RAM / CPU / NIC) •  Multi-Threaded trafgen (part of netsniff-ng) got us not even 10 Gbit/s

•  trex performs well •  But requires compatible OS and hardware (best use CentOS / RHEL) •  No problem to generate 10 Gbit/s even on older hardware

•  Use multiple servers and Ixia Packetbroker to combine generated traffic with real traffic


NIC Overview


Intel XL710 Napatech NT200A02

Mellanox ConnectX-5

Netronome Agilio CX

Accolade ANIC-80Ku

Ports 1x 10/40G 2x 40/100G 2x 40/100G 2x 40G 2x 40G

PCIe 3.0 3.0 4.0 3.0 3.0

XDP Support driver See notes driver hardware See notes

Price Cheap Expensive (FPGA)

Cheap Cheap Expensive (FPGA)

Notes Plug‘n‘Play, just works

Used to require pf_ring, not anymore. Driver still in beta, bypass in hardware

Complicated way to install driver, be ready to spent much time reading docs

Painless driver installation

Officially not compatible with Suricata yet, bypass in hardware

Preliminary Results

•  Bandwidth Average: 22.7 Gbit/s | Peak: 38.6 Gbit/s •  Packets /s Average: 2,717,952 pps | Peak: 4,397,240 pps •  Bypass hard to calculate, depends on incoming traffic ~ 35-40%


Intel XL710 Napatech NT200A02

Mellanox ConnectX-5

Netronome Agilio CX

Accolade ANIC-80Ku

CPU Average 32.5% n/a 27.4% 24.7% n/a

CPU Peak 40.1% n/a 43.1% 30.8% n/a

Side-Effects from high bandwidth •  At those speeds Suricata becomes the „lesser“ problem •  Log Shipping and writing to disk is a challenge

•  Bug: writing large number of json events on high speed traffic results in packet drops •  https://redmine.openinfosecfoundation.org/issues/2726


How to break your Splunk license in under one hour

•  More than 50.000 events / second - Too much for Splunk Forwarder?

•  Analyzing all events is an even bigger challenge


Tuning Rules becomes even more important

•  The Top 20 Signatures make for more than 10 Million alerts (in one hour)

•  Top hitting Signatures: •  GPL ICMP_INFO PING BSDtype •  GPL ICMP_INFO PING *NIX •  SURICATA TLS on unusual port •  ET INFO WinHttp AutoProxy Request wpad.dat Possible BadTunnel •  ET CHAT IRC PRIVMSG command •  ET CHAT IRC JOIN command •  ET CHAT IRC NICK command •  ET CHAT IRC USER command


Lessons learned

•  Be comfortable to experiment with latest (non-stable) software versions •  Nearly everything is bleeding-edge, update often

•  Issues with compiling Kernel, drivers or Suricata happened all the time •  Latest version of one part, causes a chain-reaction of things you need to update – if you

are even able to •  E.g. Debian 10 Hyperscan Bug •  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903675 •  Mellanox Driver checks OS, but is rarely updated to support newer versions of Debian


Preliminary Results - Conclusion

•  Current hardware can handle 20+ Gbps without any problems – even without bypass or shunting of „elephant flows“

•  Commodity hardware can achieve 40Gbps, with dedicated NIC and shunting, even more

•  If you use the NIC only for Suricata, buy a cheaper NIC and invest the money into more cores (SEPTun recommendation) ! still true up to 40Gbps. At 100Gbps it will a be different story


Conclusion

•  100Gbps in a Single Server is still difficult – bypass / filter required

•  NUMA Node placement is important

•  PCIe Bottleneck at 100Gbps – again, bypass / filter required

•  2x 40Gbps on Dual CPU should be possible on commodity hardware


Outlook

•  Waiting for updates •  Eagerly waiting to hear Peter‘s and Eric‘s talk on Friday morning •  Stable Kernel and driver releases •  Napatech will release stable firmware at the end of November •  Accolade is still working on integration with Suricata 5.x

•  Final tests (Stability + Performance)

•  Planned to deploy into production Q1/2020


Documents

Pushing Suricata Towards 90 Gbit s and More€¦ · Pushing Suricata Towards 80 Gbit/s and More | 2019-10-31 | Tobias Appel 20 . Conclusion • 100Gbps in a Single Server is still