© 2013 - ntop.org

Monitoring Network Traffic using ntopng

Luca Deri <deri@ntop.org>

© 2013 - ntop.org

Outlook

• What are the main activities of ntop.org ?• ntop’s view on network monitoring.• From ntop to ntopng.• ntopng architecture and design.• Using ntopng.• Advanced monitoring with ntopng.• Future roadmap items.

2

© 2013 - ntop.org

Introduction

3

© 2013 - ntop.org

About ntop.org [1/2]

• Private company devoted to development of open source network traffic monitoring applications.

• ntop (circa 1998) is the first app we released and it is a web-based network monitoring application.

• Today our products range from traffic monitoring, high-speed packet processing, deep-packet inspection, and IDS/IPS acceleration.

4

© 2013 - ntop.org

About ntop.org [2/2]

• Our software is powering many commercial products...

5

© 2013 - ntop.org

Problem Statement [1/2]• In 90s standard-based (e.g. RMON) and proprietary solutions (e.g. HP OpenView) were dominating the market.

• The explosion of the Internet(working) reduced the number of vendors but increased the number of monitoring devices and protocols.

• ntop tried to fill the lack of simple traffic monitoring application releasing the original ntop application aimed at simplifying the analysis of network traffic.

6

© 2013 - ntop.org

Problem Statement [2/2]• Today the market is partially driven by hardware manufacturers that manufacture costly, limited (usually no analysis beyond packet header) and often not extensible solutions.

• Consequences:◦Monitoring evolution is capped by hardware vendors.◦Commercial probes monitor what the vendor wants (e.g. Cisco TelePresence) and not what the user needs.◦Often people believe in speed=hardware, so they buy hardware probes that are unlike to provide timely answers to continuously evolving monitoring needs.

7

© 2013 - ntop.org

ntop Goals

• Provide better, yet price effective, traffic monitoring solution by enabling users to have increased traffic visibility.

• Go beyond standard metrics and increase traffic visibility by analyzing key protocols in detail.

• Provide users comprehensive and accurate traffic reports able to offer at a fraction of price what many commercial products do together.

• Promote open-source software, while protecting selected IPRs.

8

© 2013 - ntop.org

ntop’s Approach to Traffic Monitoring

• Ability to capture, process and (optionally) transmit traffic at line rate, any packet size.

• Leverage on modern multi-core/NUMA architectures in order to promote scalability.

• Use commodity hardware for producing affordable, long-living (no vendor lock), scalable (use new hardware by the time it is becoming available).

• Use open-source to spread the software, and let the community test it on unchartered places.

9

© 2012 - ntop.org

Network Health Monitoring is Complex

• Traditional network monitoring systems can limit their supervision activities to “home” networks.

• Mobile users, intra-VM data exchange, or cloud services are often invisible to network monitoring systems.

• Traditional firewalls are becoming blind as:◦Generic (e.g. HTTP) protocols can be used to tunnel non-

hypertext services (e.g. music streaming).◦HTTPS is not checked so often it flows unrestricted.

• Cloud services access can’t be monitored with simple “periodic pings” as provides IPs are often unresponsive and change according to our location.

10

© 2013 - ntop.org

Limitations of (Many) Monitoring Systems

• Visibility limited to packet header (payload agnostic).

• Packet encapsulations (e.g. GRE, PPP, GTP) are not always handled, so that we don’t know what happens inside tunnels.

• Unable to monitor intra-virtual machines (VMs) traffic (no cloud-friendly).

• Windows PCs are not first-class citizens.• Inability to map user traffic to actions performed or user’s interests.

11

© 2013 - ntop.org

Some History

• In 1998, the originalntop has been created.

• It was a C-based appembedding a web serverable to capture trafficand analyze it.

• Contrary to many tools available at that time, ntop used a web GUI to report traffic activities.

• It is available for Unix and Windows under GPL.

12

© 2013 - ntop.org

ntop Architecture

13

HTTP/HTTPS RRDCisco NetFlowInMon sFlow

© 2013 - ntop.org

Why was ntop obsolete?

• Its original LAN-oriented design prevented ntop from handling more than a few hundred Mbit.

• The GUI was an old (no fancy HTML 5) monolithic piece written in C so changing/extending a page required a programmer.

• ntop could not be used as web-less monitoring engine to be integrated with other apps.

• Many components were designed in 1998, and it was time to start over (spaghetti code).

14

© 2013 - ntop.org

ntopng Design Goals

• Clean separation between the monitoring engine and the reporting facilities.

• Robust, crash-free engine (ntop was not really so).• Platform scriptability for enabling extensions or changes at runtime without restart.

• Realtime: most monitoring tools aggregate data (5 mins usually) and present it when it’s too late.

• Many new features including HTML 5-based dynamic GUI, categorization, DPI.

15

© 2013 - ntop.org

ntopng Architecture

16

• Three different and self-contained components, communicating with clean API calls.

Users

HTTPLua-based Web Reports

nDPI-based C++ Monitoring Engine

PF_RING Kernel Module and Drivers

InternetTraffic

Lua API Calls

PF_RING C API Calls(Linux) Kernel

Data Cache

© 2013 - ntop.org17

PF_RING (Circa 2003)

ReadIndex

WriteIndex

Incoming Packets

Outgoing Packets Userspace

Kernel

Socket(ring)

NetworkAdapter

mmap()

Socket(ring)

PF_RING

Application A Application Z

© 2013 - ntop.org

Vanilla vs DNA (Direct NIC Access)

18

Device Driver

Application

DMA

PF_RING

Userland

Kernel

CircularBuffer

NAPIPolling

PF_RINGPolling

PF_RING DNAVanilla PF_RING

Device Driver

Application

DMA

AcceleratedCards

Userland

Kernel

NICMemory

Map FPGA

ApplicationPolling

© 2013 - ntop.org

PF_RING DNA: Features

• DNA is a Linux kernel-bypassing technology developed by ntop/Silicom for 1/10 Gbit network adapters.

• It can receive/transmit traffic at line rate any packet size (14.88 Mpps, 60+4 bytes) using x86 servers.

• Feature-rich set of companion tools distributed in source format (e.g. line-rate traffic generation and pcap traffic replay) so that people can customize and use existing (pcap-based) apps as low-cost alternatives to hardware-based tools (e.g. traffic generator).

19

© 2013 - ntop.org

ntopng Monitoring Engine

• Coded in C++ and based the concept of flow (se of packets with the same 6-tuple).

• Flows are inspected with a home-grown DPI-library named nDPI aiming to discovering the “real” application protocol (no ports are used).

• Information is clustered per: (Capture) Network Device FlowHostHigh-level Aggregations

20

© 2013 - ntop.org

Information Lifecycle

• All information (e.g. hosts and flows) is stored in memory.

• Using command line options, users can specify how many hosts/flows can be kept in memory.

• Idle flows and are periodically purged to free memory.

• Hosts are serialized and stored in JSON format in Redis for 1 hour, so that in case new traffic is detected ntopng can restore them from cache.

21

© 2013 - ntop.org

Packet Processing Journey

1.Packet capture: PF_RING (Linux) or libpcap.2.Packet decoding: no IP traffic is accounted.3.IPv4/v6 Traffic only:1.Map the packet to a 6-tuple flow and increment stats.2.Identify source/destination hosts and increment stats.3.Use nDPI to identify the flow application protocol1.UDP flows are identified in no more than 2 packets.2.TCP Flows can be identified in up to 15 packets in total, otherwise the flow is marked as “Unknown”.

4.Move to the next packet.

22

© 2013 - ntop.org

The need for DPI in Monitoring [1/2]

• Limit traffic analysis at packet header level it no longer enough (nor cool).

• Network administrators want to know the real protocol without relying on the port being used.

• Selected protocols can be “precisely dissected” (e.g. HTTP) in order to extract information, but on the rest of the traffic it is necessary to tell network administrators what is the protocol flowing in their network.

23

© 2013 - ntop.org

The need for DPI in Monitoring [2/2]• DPI (Deep Packet Inspection) is a technique for inspecting the packet payload for the purpose of extracting metadata (e.g. protocol).

• There are many DPI toolkits available but they are not what we looked for as:◦ They are proprietary (you need to sign an NDA to use them), and

costly for both purchase and maintenance.◦ Adding a new protocol requires vendor support (i.e. it has a high

cost and might need time until the vendor supports it) = you’re locked-in.

• On a nutshell DPI is a requirement but the market does not offer an alternative for open-source.

24

© 2013 - ntop.org

Say hello to nDPI• ntop has decided to develop its own GPL DPItoolkit in order to build an open DPI layer for ntop and third party applications.

• Supported protocols (~170) include:◦P2P (Skype, BitTorrent)◦Messaging (Viber, Whatsapp, MSN, The Facebook)◦Multimedia (YouTube, Last.gm, iTunes)◦Conferencing (Webex, CitrixOnLine)◦Streaming (Zattoo, Icecast, Shoutcast, Netflix)◦Business (VNC, RDP, Citrix, *SQL)

25

© 2013 - ntop.org

nDPI on ntopng: Sample Report

26

© 2013 - ntop.org

Is nDPI Computationally Expensive?

• nDPI requires◦Full packet payload capture (free with PF_RING DNA).◦ Inspection of the packet payload.

• The above activities are not cheap, but they are performed only at the beginning of the connection: the longer is the connection the cheaper.

• nDPI increases the application load of less than 5-10% so it is almost unnoticeable.

27

© 2013 - ntop.org

ntopng and Redis

• Redis is an open source key-value in-memory database.

• ntop uses it to cache data such as:◦Configuration and user preferences information.◦DNS name resolution (numeric to symbolic).◦Volatile monitoring data (e.g. hosts JSON representation).

• Some information is persistent (e.g. preferences) and some is volatile: ntopng can tell redis how long a given value must be kept in cache.

28

© 2013 - ntop.org

Welcome to the MicroCloud [1/2]

29

Cloud Node(redis)

Cloud Node(redis)

Cloud Node(redis)

Cloud Get/PutCloud Get/Put

ntopngnProbentopng

nProbe

nProbentopng

Cloud Get/Put

Cloud Get/Put

MicroCloud

Monitoring ApplicationCloud Subscribe

Cloud Getn

nnnn

n

© 2013 - ntop.org

Welcome to the MicroCloud [2/2]

• The microcloud is a semi-persistent cache, used to aggregate heterogeneous data onto a single place for promoting data correlation and thus better traffic visibility.

• ntopng is a first-class microcloud citizen, but additional applications (e.g. WinLogon, Captive Portals) can fill the cloud or access (read-only) data present on it.

30

© 2013 - ntop.org

Using the MicroCloud• Typical usage scenarios include:◦Store the DNS cache and HTTP host, so that we can associate the correct symbolic hostname to a flow (often a issue in particular with CDNs).◦Keep coherency across tunnels (e.g. GTP) so that probes can associate traffic directions properly.◦Store user-data (e.g. Radius IP/User association) so that the probe, can report collectors both the username who produced a given flow and the symbolic flow hostnames.◦Trigger data capture on demand (via publish/subscribe).

31

© 2013 - ntop.org

Lua-based ntopng Scriptability [1/3]

• A design principle of ntopng has been the clean separation of the GUI from engine (in ntop it was all mixed).

• This means that ntopng can (also) be used (via HTTP) to feed data into third party apps such as Nagios or OpenNMS.

• All data export from the engine happens via Lua.• Lua methods invoke the ntopng C++ API in order to interact with the monitoring engine.

32

© 2013 - ntop.org

Lua-based ntopng Scriptability [2/3]

• /scripts/callback/scripts are executedperiodically to performspecific actions.

• /scripts/lua/ scriptsare executed only by theweb GUI.

• Example:http://ntopng:3000/lua/flow_stats.lua

33

© 2013 - ntop.org

Lua-based ntopng Scriptability [3/3]

• ntopng defines (in C++) two Lua classes:◦interfaceHook to objects that describe flows and hosts.Access to live monitoring data.◦ntopGeneral functions used to interact with ntopng configuration.

• Lua objects are usually in “read-only” mode◦C++ sets their data, Lua reads data (e.g. host.name).◦Some Lua methods (e.g. interface.restoreHost()) can however modify the information stored in the engine.

34

© 2013 - ntop.org

Multithreading in ntopng

• ntopng can monitor simultaneously various network interfaces, each using a thread.

• Packet processing per-interface is lockless.• Lua scripts can access monitoring data while packets are being processed: locks occur in very few places and are not per-packet.

• Multiple Lua scripts and HTTP requests can be served (on different threads) while packets are being processed.

35

© 2013 - ntop.org

Using ntopng

36

© 2013 - ntop.org

Logging into ntopng

37

© 2013 - ntop.org

Dashboard

38

© 2013 - ntop.org

Available Menu Items

39

© 2013 - ntop.org

Dynamic Web Interface

40

© 2013 - ntop.org

Flows Monitoring [1/2]

41

© 2013 - ntop.org

Flows Monitoring [2/2]

42

© 2013 - ntop.org

Host Monitoring [1/3]

43

© 2013 - ntop.org

Host Monitoring [2/3]

44

© 2013 - ntop.org

Host Monitoring [3/3]

45

© 2013 - ntop.org

Activity Map

• 1 second resolution host and aggregation activity• Compressed bitmap

• Saved persistently on disk (Local Hosts only)

46

> ls -l client14.dropbox.com4 -rw-rw-rw- 1 nobody nogroup 24 Oct 11 02:31 client14.dropbox.com

© 2013 - ntop.org

Traffic Aggregations [1/2]

• nDPI extracts specific attributes from traffic that ntopng aggregates (if configured):◦DNS/Whois responses◦HTTP host names◦Operating System (from HTTP headers)

• Aggregations can be enabled (they are off by default) and are handled just as flows and hosts.

47

© 2013 - ntop.org

Traffic Aggregations [2/2]

48

© 2013 - ntop.org

Hosts and Aggregations Interaction

49

© 2013 - ntop.org

Geolocation

50

Map Centered UsingHTML 5 Geolocation

Maxmind GeoIP

© 2013 - ntop.org

Live Host Activities

51

© 2013 - ntop.org

Historical Activities

• All relevant counters are saved on disk in RRD.• Interface counters are saved with 1 second resolution. Hosts counters every 5 minutes.

52

RRD values correlated with top talkers

Ajax-based charts (no RRD graphs)

© 2013 - ntop.org

Using ntopng as a Live Data Source

• ntopng is a server able to serve data to third party applications via HTTP.

• Data is exported via JSON.• This mechanism can be extended via Lua scripts.

53

© 2013 - ntop.org

Top Hosts as a Clickable Map

54

© 2013 - ntop.org

Using ntopng with NetFlow/sFlow

• ntopng can handle flows (Net/sFlow) via nProbe.

• Data Collector (ntopng)◦ ntopng -i tcp://127.0.0.1:5556

• Probe (nProbe)◦ nprobe --zmq "tcp://*:5556" -i eth1 -n none (probe mode)◦ nprobe --zmq "tcp://*:5556" -i none -n none --collector-port

2055 (sFlow/NetFlow collector mode)

55

© 2013 - ntop.org

Embedding ntopng [1/2]

• Historically we have started our first embed attempt in 2003 with the Cyclades TS100.

• The nBox was used to analyze traffic then sent to ntop for representation.

• After 10 years we tried again with ntopng.

56

© 2013 - ntop.org

Embedding ntopng [2/2]

• The ntopng code compiles smoothly for cheap (36 Euro) boxes such as the BeagleBone Black.

• You can now createyour personal/cheaptraffic analyzerwithout having touse a PC.

• Post 1.1 release wewill optimize supportfor these devices.

57

© 2013 - ntop.org

Final Remarks

• Over the past 15 years ntop created a software framework for efficiently monitoring traffic.

• “We have a story to tell you, not just hacks”.• Commodity hardware, with adequate software, can now match the performance and flexibility that markets require. With the freedom of open source.

• Available under GNU GPLv3 fromhttp://www.ntop.org/.

58