Monitoring and Debugging

MUM EU 2019 Vienna | Patrik Schaub | © FMS Internetservice GmbH

(with) RouterOS

Agenda

ƒ Company introduction

ƒ Network operationthe big picture

ƒ Management approachesƒ Network debuggingƒ RouterOS debugging

FMS Internetservice GmbH

Value Added Distribution

FMS Internetservice GmbH

ƒ Value Added Distributor

ƒ Distribution

ƒ Training

ƒ Consulting

ƒ Support

ƒ Founded 1997

ƒ 11 employees

ƒ Southern Germany

FMS Internetservice GmbH

ƒ Inhouse training facility

ƒ All certification levels

ƒ First German speaking Training partnerTR11 & TR23

ƒ First MTCSA certified German distributor

See Training Schedule

Distributor Table

10G Radio Links Wireless LoRaWAN IoT Solution

5 year warranty & next day replacement 3km transmition & 10 years battery life

Network Operation – Big Picture

Challenges and Elements

The Challenge of Operation

ƒ Growing number of devicesƒ More critical servicesƒ Higher bandwidth (more packets)ƒ Heavy interconnection of sites

ƒ Networksƒ Become largerƒ Become more complexƒ Require higher availabilityƒ Require effective security

Operational Tasks

ƒ Management

ƒ Inventory

ƒ Maintenance

ƒ Debugging

ƒ Monitoring

RouterOS

Network Inventory Management

ƒ Dudeƒ Script based databaseƒ TR069ƒ CAPsMAN

Access to management

ƒ Dudeƒ Management VLANƒ RoMONƒ CAPsMAN

Management technologies

ƒ Webboxƒ Winboxƒ Terminalƒ APIƒ TR069ƒ SNMPƒ Appƒ CAPsMAN

General Tools

ƒ Time / SNTPƒ Watchdogƒ Scripting & APIƒ Netwatchƒ SSH keys

Maintenance

ƒ RouterOS & bootloader updatesƒ Backup/Restore & Import-Export

RouterOS

Debugging (Router)

ƒ Healthƒ Historyƒ local loggingƒ /system ressourcesƒ /system routerboardƒ /tools profileƒ Supout

Debugging (Traffic and Network)

ƒ Neighboursƒ Bandwidth test (old and new)ƒ Traffic generatorƒ Torchƒ Ping, Flood Ping, Ping Speedƒ Tracerouteƒ IP Scanƒ Packet Sniffer (and TZSP

streams)ƒ Port Mirroring (Switch chip)

Logging & 3rd Party Integration

ƒ IP Accountingƒ Traffic Flow (Netflow)ƒ SNMPƒ Graphingƒ Syslogƒ TR069

Management Topologies

Secure and Convenient Management Access

Management Approaches

ƒ Considerationsƒ Securityƒ Convenienceƒ Efficiency

ƒ Common Approachesƒ Separate management and user trafficƒ Management VLANƒ Tunneling payload (e.g. PPPoE)ƒ Tunneling of management (VPN)

Management Approaches

ƒ Central MikroTik toolsƒ The Dudeƒ CAPsMANƒ Usermanager

ƒ Detailed examplesƒ RoMONƒ API (Application programming interface)

RoMON

Simplify Discovery and Access

RoMON

ƒ Router Management Overlay Networkƒ Proprietary MikroTik protocol

ƒ Device discoveryƒ Device access

ƒ Layer-2 & layer-3 networksƒ Without layer-3 routingƒ Winbox support

RoMON + MAC Winbox vs. Neighbours + MAC Winbox

RoMON

ƒ Creates overlay networkƒ Only with MikroTik devicesƒ Not limited to layer-2 broadcast domainƒ Winbox: discovery and MAC connectionƒ Winbox: RoMON agent connection

ƒ On ethernet like interfaces (Ethernet,WLAN, EoIP, VLAN …)

Neighbour discovery (MNDP)

ƒ Using existing networkƒ Compatible with CDP and LLDPƒ Limited to layer-2 broadcast domainƒ Winbox: discovery and MAC

connection

Local Device Discovery across Routers

192.0.2.0/24 203.0.113.0/24 198.51.100.0/24

Winbox RoMON Agent RoMON enabled Router

RoMON enabled devices RoMON enabled devices RoMON enabled devices

Discovery with MNDPConnect by IP or MAC Winbox

Discovery with RoMON, Connect by RoMON Winbox

Connect to RoMON

RoMON Setup

ƒ Enable RoMON

ƒ Optional but recommended

ƒ Set ID manually

ƒ Use secret(s)

ƒ Optional

ƒ Customize interface

configuration

RoMON Tools

ƒ Discovery

ƒ Ping

ƒ CLI: ssh

ƒ Winbox

Standard Tools in RoMON Network

Ping / MAC Ping RoMON Ping

Winbox Discovery and RoMON Connection

Devices within

the layer-2

network

discovered

Use router as

RoMON agent

1

2

Winbox Discovery and RoMON Connection

Connected to

RoMON agent

Two hops to

reachRoMON

discovery

through agent

3

4

Local Device Discovery across Routers

192.0.2.0/24 203.0.113.0/24 198.51.100.0/24

Winbox RoMON Agent RoMON enabled Router

Discovery with RoMON, Connect by RoMON Winbox

Connect to RoMON

R1 R2

A11 A12 A21 A22 A31 A32

Path to A32 as seen from agent R1

1

2

21

Remote RoMON Agent

ƒ RoMON agent connection by IPƒ Across layer-3 networkƒ E.g. internet

ƒ Remote discovery and management

ƒ Branch officesƒ Customer networks

Remote Network Discovery

INET

203.0.113.0/24

198.51.100.0/24

RoMONenableddevices

RoMONenableddevices

RoMON AgentCustomer 1

RoMON AgentCustomer 2

OperatorWinbox

eth5Disable RoMON on WAN port

Security Considerations

ƒ Disable RoMON on WANƒ Don’t enable Winbox on WAN

ƒ Management VPNƒ VPN to reach RoMON agentƒ RoMON to reach remote devicesƒ VLAN to limit RoMON locally

MikroTik API

Custom-tailored Management Access

FMS Management Platform

Initial situation:ƒ Distributed Hotspot Systemƒ Hundreds of sitesƒ New gateways will be deployed

ƒ 100+ third party devices per siteƒ Fixed local IPv4 addressesƒ Conflicting local subnets

ƒ Two small NOCs, Road Warrior

Initial requirements:ƒ Easy operation

ƒ Auto configuration of gatewaysƒ site to site VPN, INET access, basics

ƒ Third party devicesƒ Direct access to WEB interfacesƒ Central inventoryƒ Central monitoring with dependencies

FMS Management Platform

FMS Management Platform

INET INET

Road Warrior Datacenter Customer Site 3

NOC 1 Customer Site 1

NOC 2 Customer Site 2

FMS Management Plattform

Captive Portal

RouterOSVPN

Concentrator

MikroTikRouter

Third PartyAccess Points

MikroTikRouter

1 2 3

Adding new Sites

INET INET

NOC 1 Customer Site 1

FMS Management Platform

Captive Portal

RouterOSVPN

Concentrator

MikroTikRouter

ƒ Site routerconfigurationƒ VPN server

configuration

ƒ Captive Portalconfigurationƒ Monitoring of site

router

Adding new Devices

INET INET

NOC 1 Customer Site 1

FMS Management Platform

Captive Portal

RouterOSVPN

Concentrator

MikroTikRouter

Tasks:ƒ Central access to

third party devicesƒ Monitoring of third

party devices

Coping with IP Conflicts / Management

INET INET

NOC 1 Customer Site 1

Customer Site 2

FMS ManagementPlattform

Captive Portal

RouterOSVPN

Concentrator

MikroTikRouter

192.168.40.10/24

192.168.40.10/24

ƒ Tunnel end point knownƒ Port forwarding on site

router by APIƒ EndPointIP:DevicePort

Coping with IP Conflicts / Monitoring

INET INET

NOC 1 Customer Site 1

Customer Site 2

FMS ManagementPlattform

Captive Portal

RouterOSVPN

Concentrator

MikroTikRouter

192.168.40.10/24

192.168.40.10/24

ƒ API to execute ping onMikroTik site router

Local Retailer

INET INET

Stock & Payment Management Datacenter Customer Site 3

NOC Customer Site 1

Security Services Customer Site 2

FMS Management Plattform

CaptivePortal

RouterOSVPN

Concentrator

MikroTikRouter

Third PartyAccess Points

DVR /Surveillance

ERP /Cash register

IoT – Management “only” Networks

ƒ Dedicated networks for managementand monitoring

ƒ Often small but many sitesƒ Only purpose is managementƒ Lack of trained network staffƒ Efficiency and simplicity most

important

Access for vendors(e.g. CNC machines)

Smart metering for

transformer stations

Get in Touch

Are you looking for a centralised andindividual management platform?

+49 761 2926500 | sales@fmsweb.de | Web form

Network Debugging

The Needle in a (huge) Haystack

Packet Sniffer

Last Resort for Networking Problems

Network Debugging

ƒ Planning / checking firewall settingsƒ Networking problemsƒ Faulty client / server applications

ƒ Things go wrong?ƒ Real insight is necessary

ƒ Packet sniffingƒ De facto standard: Wiresharkƒ RouterOS packet sniffer

MikroTik Packet Sniffer

ƒ General settingsƒ Filterƒ Start/Stop

ƒ Results in CLI / Winboxƒ Results in file, analyse in Wireshark

ƒ Streaming to Wireshark

Remote Packet Sniffing

INET 198.51.100.0/24

Customer 1

Operator

Packet SnifferLocally analyse packets from

a remote sniffer in real time

Sniffer Stream

ƒ Enable “Stream”ƒ Set Wireshark host IPƒ Enable “Filter Stream”

ƒ TZSP stream is sent

ƒ Filter stream in Wiresharkƒ UDP port 37008

ƒ Start sniffer in Winbox

1

2

Live Output

1

Traffic Flow

Statistical Network Information

Traffic Flow

ƒ Compatible with Netflowƒ Statistical network

informationƒ Byte and packet counterƒ Source and destination IP

addressesƒ Source and destination ports

ƒ Top talkersƒ Top protocolsƒ Utilisation

Netflow Collector and Anlysis

ƒ ntop (former) free standardƒ Successor ntop-ngƒ Requires commercial nProbe

to collect Netflow

ƒ Alternative free and opensource collectors available

ƒ E.g as in FMS ManagementPlattform Former ntop GUI

Netflow in FMS Management Plattform

xxxxxxxx

xxxxxxxx

xxxxxxxx

xxxxxxxxxxxxxxxxxxxxxxxx

Debugging RouterOS Installations

The other Needle in another (huge) Haystack

RouterOS Debugging

ƒ Source for networkdebugging = packets andpacket statistics

ƒ Source for device debugging= local status information

ƒ SNMPƒ Local logging

Log Output

Central Syslog

ƒ External, central syslog server

ƒ Will survive reboots / crashesƒ No tampering from deviceƒ Better searchƒ Correlation across devices

ƒ Example: Investigate VRRP changeƒ Involved: Master, slave, crosslink

switch

VRRP Setup

VRRP1 VRRP2

RSTP

FMS Management Platform

ƒ Syslog, Netflow, SNMP traps …ƒ MongoDB, Elasticsearch …

ƒ Central storageƒ Powerful searchƒ Dashboardsƒ Alerts

ƒ Enhanced MikroTik supportƒ E.g. MikroTik MIB, Log syntax Remote Syslog Configuration

?

WIFI Connects from Syslog across complete Network

10.10.0.29

10.10.0.22

system,error,critical login failure for user admin from 10.10.0.55 via web

Enhanced Log Message Processing

ƒ Make syslog serverunderstand message

ƒ Database fields

ƒ Searchƒ Sortingƒ Analyse

ƒ Login FailureDashboard

1

Failed Logins including Username and Login Type

10.10.0.29

10.10.0.22

Get in Touch

Are you looking for centralisedand MikroTik aware logging?

+49 761 2926500 | sales@fmsweb.de | Web form

FMS Internetservice GmbH

Services and Contact

CentralLoggingTraining

CentralManagement

RouterOSHosting

ConsultingSupport

ServiceContracts Distribution

+49 761 2926500 | sales@fmsweb.de | Web form

www.fmsweb.de | www.mikrotik-shop.de

Thank You