Collec:ng telemetry data with pmacct - AfPIF · Presentaon history § 1.1: – MENOG 13 mee:ng,...

AfPIF2017,Abidjan–Aug2017

Collec:ngtelemetrydatawithpmacct

PaoloLucentepmacct

Presenta:onhistory

§  1.1:–  MENOG13mee:ng,KuwaitCity,Sep2013

§  1.2:–  SEE3mee:ng,Sofia,Apr2014

§  1.3:–  AfPIF2017mee:ng,Abidjan,Aug2017

Agenda

§  Introduc)on§  pmacctarchitecture&benefits§  example,dataaggrega:on:trafficmatrices§  example,loggingmicro-flowsorevents§  tee:brieflyonhorizontalscalability

whoami

PaoloLucenteGitHub:paololucenteLinkedIn:plucenteDiggingdataoutofnetworksworldwideforfunandprofitformorethan10years

libpcap

pmacctisopen-source,free,GPL’edso[ware

MySQL PgSQL SQLite

MongoDB BerkeleyDB

flat-files

RabbitMQ Kafka

memory tables

NetFlow IPFIX

h\p://www.pmacct.net/

Streaming Telemetry

BMP GeoIP

pmacct:afewsimpleuse-cases

flat-files

NetFlow IPFIX

IPFIX libpcap

pmacct:aslightlymorecomplexuse-case

flat-files

NetFlow IPFIX

aggregation method #1

aggregation method #2

nfacctd

Theuse-caseformessagebrokers

pmacct-to-elas:csearch0.3.0

Creditsto:PierCarloChiodi,h\ps://github.com/pierky/pmacct-to-elas:csearch

Usecasesbyindustry

Keypmacctnon-technicalfacts

§  10+yearsoldproject§  Can’tspellthenamea[ertheseconddrink§  Free,open-source,independent§  Underac:vedevelopment§  Innova:onbeingintroduced§ Welldeployedaround,alsoinlargeSPs/IXPs§  ClosetotheSP/IXPcommunityneeds

Agenda

§  Introduc:on§  pmacctarchitecture&benefits§  example,dataaggrega:on:trafficmatrices§  example,loggingmicro-flowsorevents§  tee:brieflyonhorizontalscalability

Sometechnicalfacts

§  Pluggablearchitecture:•  Caneasilyaddsupportfornewdatasourcesandbackends

§  Correla:onofdatasources:•  Na:velysupporteddatasources(ie.flowtelemetry,BGP,BMP,IGP,StreamingTelemetry)

•  Enrichwithexternaldatasourcesviatagsandlabels

§  Enableanaly:csagainsteachdatasource:•  Streamreal-:me•  Dumpatregular:meintervals(possiblestatecompression)

Sometechnicalfacts(cont.d)

BGP thread

NetFlow thread

Abstraction layer

Core Process

MySQLplugin

Printplugin(toflat-files)

Observed network Backends

Plugins

Sometechnicalfacts(cont.d)

§  Buildmul:pleviewsoutoftheverysamecollectednetworktraffic,ie.:•  Unaggregatedtoflat-filesforsecurityandforensics;ortomessagebrokers(RabbitMQ,Kaia)forBigData

•  Aggregatedas[<ingressrouter>,<ingressinterface>,<BGPnext-hop>,<peerdes:na:onASN>]andsenttoaSQLDBtobuildaninternaltrafficmatrixforcapacityplanningpurposes

§  Pervasivedata-reduc:ontechniques,ie.:•  Dataaggrega:on•  Filtering•  Sampling

Touchingground:aconfigsnippetforbothaggregatedandunaggregatedviews

nfacctd_ip: 10.0.0.1 nfacctd_port: 2100

plugins: print[forensics], mysql[int_traffic_matrix]

aggregate[forensics]: etype, src_host, dst_host, \

peer_src_ip, peer_dst_ip, in_iface, out_iface, \

timestamp_start, timestamp_end, src_port, \

dst_port, proto, tos, src_mask, dst_mask, src_as, \

dst_as, tcpflags, export_proto_seqno

aggregate[int_traffic_matrix]: in_iface, peer_src_ip, \

peer_dst_ip, peer_dst_as

! ! <rest of config skipped>

Basic daemon config

Instantiating plugins

Defining plugin aggregation

methods

Touchingground:dataaggrega:on&custom-definedprimi:ves

§  Configfilesnippet: ! … aggregate[int_traffic_matrix]: peer_src_ip, \ mplsTopLabelIPv4Address

! aggregate_primitives: /path/to/primitives.lst

§  Customprimi:vedefini:on: ! …

name=mplsTopLabelIPv4Address field_type=47 len=4 semantics=ip

Node-to-node internal TM (egress NetFlow)

mplsTopLabelIPv4Address not supported natively, let’s

define it!

Primitive name, will be used for

everything

NetFlow field type, IPFIX Information Element NetFlow/IPFIX

field length

Data presentation: [ u_int, hex, ip, mac, str ]

BGPintegra:on

§  pmacctintroducedaQuagga-basedBGPdaemon:•  Implementedasaparallelthreadwithinthecollector•  Doesn’tsendUPDATEswhatsoever•  BehavesasapassiveBGPneighbor•  Maintainsper-peerBGPRIBs•  Supports32-bitASNs;IPv4,IPv6andVPNfamilies•  SupportsADD-PATH:dra[-ieq-idr-add-paths

§  WhyBGPatthecollector?•  Telemetryreportsonforwarding-plane,andabitmore•  Extendedvisibilityintocontrol-planeinforma:on

BMPintegra:on

§  pmacctintroducedaBMPdaemonwri\enfromscratch:•  BMPis:BGPMonitoringProtocol•  Implementedasaparallelthreadwithinthecollector•  AllgoodiesalreadydescribedfortheBGPdaemon:

§ Contribu:ngtoIETFdra[-ieq-grow-bmp-local-rib

•  VisibilityinAdj-RIB-In•  VisibilityinAdj-RIB-Out:

§ Contribu:ngtoIETFdra[-ieq-grow-bmp-adj-rib-out

IGP(IS-IS)integra:on

§  AQuagga-basedIS-ISdaemonwasintroduced:•  Implementedasaparallelthreadwithinthecollector•  IS-ISneighborshipoveraGREtunnel•  Currentlylimitedtosingleneighborhip,singlelevel,singletopology

•  UsefultolookupnonBGP-routednetworks§  Itwillgeteventuallyreplaced(BGP-LS)

Storingdatapersistently§  Dataneedtobeaggregatedbothinspa:alandtemporaldimensionsbeforebeingwri\endown:•  Op:malusageofsystemresources•  Avoidsexpensiveconsolida:onofmicro-flows

§  Buildproject-drivendataset(s):•  Noshameinmul:plepartlyoverlappingdata-sets•  Op:mizecompu:ng

Storingdatapersistently(cont.d)§  “noSQL”databases(BigDataJ):•  Abletohandlelarge:me-seriesdata-sets•  MeaningfulsubsetofSQLquerylanguage•  Innova:vestorageandindexingengines•  Scalable:clustering,spa:alandtemporalpar::oning•  UI-ready:ie.ELKandTICKstacks

§  Open-sourceRDBMS:•  Abletohandlelargedata-sets•  FlexibleandstandardizedSQLquerylanguage•  Solidstorageandindexingengines•  Scalable:clustering,spa:alandtemporalpar::oning

Storingdatapersisently:MongoDB

§  OnceitwaspmacctopeningtonoSQLdatabases:•  MessupwiththeCAPIover:me•  2017realitycheck:lackofinterest,discon:nuing

§  noSQLlandscapedifficulttomovethrough,ie.fragmentedandlacksofstandardiza:on

§ MongoDBseemedinteres:ngfor:•  Na:vegroupingopera:on(moreperformingandlesscomplexthanmap/reduce)

•  Horizontalscalingconcept(sharding)

Brokeringdataaround:RabbitMQ,Kaiamessageexchanges

§  noSQLlandscapedifficulttomovethrough,ie.fragmentedandlacksofstandardiza:on,amirepea:ngmyself?J

§  Datacanbepickedupatthemessageexchangeinthepreferredprogramming/scrip:nglanguage

§  Datacanbetheneasilyinsertedinthepreferredbackend,ie.notna:velysupportedbypmacct

Agenda

§  Introduc:on§  pmacctarchitecture&benefits§  example,dataaggrega)on:trafficmatrices§  example,loggingmicro-flowsorevents§  tee:brieflyonhorizontalscalability

Whyspeakingoftrafficmatrices?

–  Aretrafficmatricesusefultoanetworkoperatorinthefirstplace?Yes…§  Capacityplanning(buildcapacitywhereneeded)§  TrafficEngineering(steertrafficwherecapacityisavailable)

§  Be\erunderstandtrafficpa\erns(whattoexpect,withoutacrystalball)

§  Supportpeeringdecisions(trafficinsight,trafficengineeringattheborder,supportwhatifscenarios)

Reviewoftrafficmatrices:internal

§  POPtoPOP,ARtoAR,CRtoCR

Customers Customers

AS1 AS2 AS3 AS4 AS5

Server Farm 1 Server Farm 2

Reviewoftrafficmatrices:external

§  Router(ARorCR)toexternalASorexternalAStoexternalAS(forIPtransitproviders)

Customers Customers

AS1 AS2 AS3 AS4 AS5

Server Farm 1 Server Farm 2

Let’sfocusonanexternaltrafficmatrixtosupportpeeringdecisions

§  Analysisofexis:ngpeersandinterconnects:•  Supportpolicyandrou:ngchanges•  Fine-grainedaccoun:ngoftrafficvolumesandra:os•  Determinebackbonecostsassociatedtopeering•  Determinerevenueleaks

§  Planningofnewpeersandinterconnects:•  Whotopeernext•  Wheretoplacenextinterconnect•  Modelingandforecas:ng

Ourtrafficmatrixvisualized

A={src_as,in_iface,peer_src_ip,peer_dst_ip,as_path,tstamp,bytes}

{CZ,X(->CY),PEC,PEA,AZ_AY,<:me>,<bytes>}

{BX,Y(->BY),PEB,PEC,CY_CX,<:me>,<bytes>}

{AX,Z(->AZ),PEA,PEB,BY_BZ,<:me>,<bytes>}

Ge|ngBGPtothecollector

§  Neededfortechnicalreasons:•  FlowexportersuseNetFlowv5,ie.noBGPnext-hop•  FlowexportersareunawareofBGP•  Libpcapisusedtocollecttrafficdata

§  Neededfortopologyortrafficrelatedreasons:•  Transi:ngtrafficto3rdpar:es•  Dominatedbyoutboundtraffic

§  LetthecollectorBGPpeerwithallPEdevices:facingpeers,transitandcustomers.

§  Determinememoryfootprint(belowinMB/peer,usingBGPbest-pathsessions)

19.97 18.59 18.12 17.89 17.76 17.57 17.48 17.39

0 200 400 600 800 1000 1200 1400

MB/peer>=0.12.4

MB/peer<0.12.4

NumberofBGPpeers

500KIPv4routes,50KIPv6routes,64-bitexecutable

~9GBtotalmemory@500peers

Ge|ngBGPtothecollector(cont.d)

§  SetthecollectorasiBGPpeeratthePEdevices:•  ConfigureitasaRRclienttogetfulltable•  CollectoractsasiBGPpeeracross(sub-)ASes

§  BGPnext-hophastorepresenttheremoteedgeofthenetworkmodel:•  TypicalscenarioforMPLSnetworks•  Canbefolloweduptocoverspecificscenarioslike:

§ BGPconfedera:ons§ defaultgatewaydefinedduetopar:alordefault-onlyrou:ngtables

Ge|ngBGPtothecollector(cont.d)

Ge|ngtelemetrytothecollector

–  Exportingress-onlymeasurementsatallPEdevices:facingpeers,transitandcustomers.§  Trafficisroutedtodes:na:on,soplentyofinforma:ononwhereit’sgoingto•  True,someeBGPmul:-pathscenariosmaygetchallenging

§  It’scrucialinsteadtogetasmuchaspossibleaboutwheretrafficiscomingfrom,ie.:•  inputinterfaceatingressrouter•  sourceMACaddress

–  Performdatareduc:onatthePE(ie.sampling)

Telemetrydata/BGPcorrela:on

Touchingground:aconfigsnippetfortrafficmatrices

plugins: mysql[int_tm], mysql[ext_tm] !

aggregate[int_tm]: in_iface, peer_src_ip, \

peer_dst_ip, peer_dst_as

aggregate[ext_tm]: src_as, in_iface, peer_src_ip, \

peer_dst_ip, peer_dst_as, as_path, dst_as

sql_table[int_tm]: int_tm-%Y%m%d_%H%M

sql_table[ext_tm]: ext_tm-%Y%m%d_%H%M

sql_cache_entries[ext_tm]: 99991

sql_refresh_time: 300

sql_history: 5m

! <rest of config skipped>

Dynamic SQL table names, ie.: XXX_tm-20130803_1400

Build time bins of 5 mins

Insert data every 5 mins

AS-to-AS external TM

Port-to-port internal TM

Touchingground:howdatawouldlooklike:internaltrafficmatrixexample(1/2)

mysql> SHOW TABLES FROM pmacct; +----------------------+

| Tables_in_pmacct |

+----------------------+

| … |

| int_tm-20130803_1400 |

| int_tm-20130803_1405 |

| int_tm-20130803_1410 |

| … |

| ext_tm-20130803_1400 |

| ext_tm-20130803_1405 |

| ext_tm-20130803_1410 |

| … |

+----------------------+

Set of internal traffic matrices

Set of external traffic matrices

NOTE: sub-aggregation is expensive: we could also have had our traffic matrices over multiple temporal aggregations in parallel, ie. 5 mins (as above) but also hourly and daily.

Touchingground:howdatawouldlooklike:internaltrafficmatrixexample(2/2)

mysql> SELECT * FROM int_tm-20130803_1400 LIMIT 10; +----------+-------------+-------------+-------------+------------------+-------+ | iface_in | peer_ip_src | peer_ip_dst | peer_dst_as | stamp_inserted | bytes | +----------+-------------+-------------+-------------+------------------+-------+ | 212 | 10.0.0.107 | 10.0.0.3 | 65000 | 03-08-2013 14:00 | 859 | | 212 | 10.0.0.107 | 10.0.0.253 | 65001 | 03-08-2013 14:00 | 5358 | | 212 | 10.0.0.107 | 10.0.0.234 | 65002 | 03-08-2013 14:00 | 6181 | | 212 | 10.0.0.107 | 10.0.0.251 | 65003 | 03-08-2013 14:00 | 27002 | | 205 | 10.0.0.107 | 10.0.0.233 | 65004 | 03-08-2013 14:00 | 1200 | | 258 | 10.0.0.107 | 10.0.0.240 | 65005 | 03-08-2013 14:00 | 560 | | 212 | 10.0.0.107 | 10.0.0.252 | 65006 | 03-08-2013 14:00 | 62682 | | 212 | 10.0.0.107 | 10.0.0.234 | 65007 | 03-08-2013 14:00 | 3843 | | 212 | 10.0.0.107 | 10.0.0.17 | 65008 | 03-08-2013 14:00 | 21074 | | 205 | 10.0.0.107 | 10.0.0.254 | 65009 | 03-08-2013 14:00 | 2023 | +----------+-----------+---------------+-----------------+--------------+-------+ 10 rows in set (0.03 sec)

Getting ingress NetFlow from 10.0.0.7 Time reference

Here is our matrix Here is our matrix Here is our matrix Here is our matrix

Amount of traffic sent in the time window 14:00 – 14:05

Case-study:peeringatAS286[year2009]

Nego:a:on

Engineering

Produc:on

Op:miza:on

§  Peeringasacycle§  NetFlow+BGPtrafficmatrix

steerspeeringop:miza:on:•  Iden:fynewand“old”peers•  Trafficanalysis:backbone

costs,95thpercen:les,ra:os•  Analysisofinterconnec:on

densityandtrafficdispersion•  Forecas:ngandtrending•  Ad-hocqueriesfromDesign

&Engineeringandindeed…theIPTProductManager

§  250+Gbpsrou:ng-domain§  100+high-endroutersaround

theglobe:•  ExportsampledNetFlow•  Adver:sefullrou:ngtable•  MixofJuniperandCisco

§  Collectorenvironment:•  Runson2Solaris/SPARCzones•  pmacct:dual-core,4GBRAM•  MySQL:quad-core,24GBRAM,

500GBdisk§  Datareten:on:6months

AS286rou:ngdomain

pmacct

Internalusers

§  AS286backboneroutersarefirstconfiguredfromtemplates:•  NetFlow+BGPcollectorIPaddressdefinedoverthere•  Enablerforauto-discoveryofnewdevices

§  Edgeinterfacesareprovisionedfollowingservicedeliverymanuals:•  RelevantmanualsandTSDsincludeNetFlowac:va:on•  PeriodicchecksNetFlowisac:vewhereitshould

§  Maps,ie.sourcepeer-AStoingressinterfaces,arere-builtperiodically

Agenda

pmacct,logging&flat-files:briefhistory(1/2)

§  OriginallypmacctwasaboutmemorytablesandRDBMS(noflat-files)

§  Itwasalsoaboutdataaggrega:on(nologgingmicro-flows)

§  Othertoolsweredoingthisgreatlyalready–be\erinvestinnewideas

§  In2011pmacctopenedtoflat-files(seenextslide)

pmacct,logging&flat-files:briefhistory(2/2)

§  InrecentyearsthelandscapechangedandNetFlowandIPFIXprotocolswerebeinggeneralizedbesidesflows

§  No:ceably,theyenteredintheEventLoggingspace,ie.:•  CiscoNEL,ie.(CG)NATevents•  CiscoNSEL,ie.securityevents

§  Thiswasgreat:metoreviewthestrategyaroundflat-filesandlogging(ofeventsand,asaconsequence,ofmicro-flows)

Logginguse-cases

§ Micro-flows:•  R&D,labac:vi:es•  Security,DDoSdetec:on,forensics•  Relatedtodataaggrega:on:

§ Analysis,temporary:elaborateaggrega:onmethods§ Back-up,permanent:keeprawdataincaseuseful

§  Events:•  NATevents:compliancywithlocalauthori:es•  FWevents:securityreports

Loggingfeatures

§  Splitdatainfilesanddirectoriesbasingon:meofthedayand(some)primi:vevalues,ie.:•  5mins,hourly,dailyfiles(fullycustomizable)•  IPaddressoftelemetryexporter

§  Canappendtoanexis:ngfile:ie.hourlyfilesbutrefreshedevery5mins

§  Filescanbethenarchivedrightawayviatriggers,print_trigger_exec)

§  Pointertolatestfilein:me-series

Loggingdataformats

§  Textformatssupported:tab-spaced,CSVandJSON•  CSV:quickestserializa:on,notself-descrip:ve•  JSON:verbose,(hence)slowertoserialize,self-descrip:ve

§  Binaryformatsupported:ApacheAvro•  Spaceefficient,schema-based,indirectaccess•  GPBbeingavoided•  MayberevisitCap’nProtoinfuture

Touchingground:aconfigsnippetforloggingmicro-flows

plugins: print[forensics] !

aggregate[forensics]: src_host, dst_host, \

peer_src_ip, peer_dst_ip, in_iface, out_iface, \

timestamp_start, timestamp_end, src_port, \

dst_port, proto, tos, src_mask, dst_mask, src_as, \

dst_as, tcpflags

print_output_file[forensics]: /path/to/forensics-%Y%m%d_%H%M.txt

print_output[forensics]: csv

print_refresh_time[forensics]: 300

print_history[forensics]: 5m

print_output_file_append[forensics]: true

print_latest_file[forensics]: /path/to/forensics-latest

Micro-flow (de)aggregation

Dynamic file names, ie.: forensics-20130803_1400

[ formatted, csv, json ]

Insert data every 5 mins, append to file if exists

Pointer to latest file to become optional and explicitly configured

Insert data every 5 mins, append to file if exists

Touchingground:howdatawouldlooklike:loggingmicro-flows(1/2)

shell> ls –la …

-rw------- 1 pmacct pmacct <size> Aug 02 13:50 forensics-20130802-1345.txt

lrwxrwxrwx 1 pmacct pmacct 10 Aug 02 14:05 forensics-latest -> /path/to/forensics-20130802-1400.txt

Pointer to latest finalized file

Configurable ownership

Touchingground:howdatawouldlooklike:loggingmicro-flows(2/2)

shell> cat forensics-latest

SRC_AS,DST_AS,PEER_SRC_IP,PEER_DST_IP,IN_IFACE,OUT_IFACE,SRC_IP,DST_IP,SRC_MASK,DST_MASK,SRC_PORT,DST_PORT,TCP_FLAGS,PROTOCOL,TOS,TIMESTAMP_START,TIMESTAMP_END,PACKETS,BYTES 65001,65002,10.0.0.1,10.0.0.100,101,8,192.168.158.133,192.168.126.141,24,24,61912,22,24,tcp,16,2013-08-04 17:40:12.167216,2013-08-04 17:41:36.140279,21,1407

[ .. ]

Touchingground:aconfigsnippetforloggingNATevents

plugins: print[cgn] !

aggregate[cgn]: src_host, post_nat_src_host, src_port, \

post_nat_src_port, proto, nat_event, timestamp_start

print_output_file[cgn]: /path/to/cgn-%Y%m%d_%H%M.txt

print_output[cgn]: json

print_refresh_time[cgn]: 300

print_history[cgn]: 5m

print_cache_entries[cgn]: 9999991

print_output_file_append[cgn]: true

Well-defined NAT event

Bigger cache size to cope with increased number of records

NOTE 2: a bigger cache is beneficial to limit scattering of writes to the backend. If the configured cache is unable to contain all records, a purge of data to the backend is triggered and cache content is

flushed so to make room to new data.

NOTE 1: see config snippet for micro-flows (a few slides back) for additional comments on configuration directives listed above.

Let’s increase readability J

Touchingground:howdatawouldlooklike:loggingNATevents(1/2)

shell> ls –la …

-rw------- 1 pmacct pmacct <size> Aug 02 13:50 cgn-20130802-1345.txt

lrwxrwxrwx 1 pmacct pmacct 10 Aug 02 14:05 cgn-latest -> /path/to/cgn-20130802-1400.txt

Touchingground:howdatawouldlooklike:loggingNATevents(2/2)

shell> cat cgn-latest

{"timestamp_start": "2013-02-21 16:56:33.518000000", "ip_proto": "tcp", "post_nat_ip_src": ”1.2.179.16", "ip_src": ”192.168.37.51", "port_src": 61020, "nat_event": 1, "post_nat_port_src": 31392} [ .. ]

Asingle(setof)collector(s)forbothmicro-flowsandeventslogging?

§  Yes,possible:•  AllNetFlow,regardless,pointedtothesameplace•  Makessenseonsmall-mediumdeployments•  Onlargeronespoten:allypressuringthe(samesetof)collector(s)with,say,anongoingDDoSandaCGNbladereboo:ngisnotagoodidea.Goforspli|ng.

§  pmacctabletotagandchannelizedata(ie.senddataselec:velytoplugins)basingonanumberofclauses

Touchingground:aconfigsnippetforbothmicro-flowsandeventlogging(1/2)plugins: print[forensics], print[cgn] ! pre_tag_filter[forensics]: 10 aggregate[forensics]: <micro-flows aggregation method> print_output_file[forensics]: /path/to/forensics-%Y%m%d_%H%M.txt ! pre_tag_filter[cgn]: 20 aggregate[cgn]: <NAT events aggregation method> print_output_file[cgn]: /path/to/cgn-%Y%m%d_%H%M.txt print_cache_entries[cgn]: 9999991 ! print_output: csv print_refresh_time: 300 print_output_file_append: true ! pre_tag_map: /path/to/pretag.map ! <rest of config skipped>

NOTE: This configuration merely merges the micro-flows and event logging configurations seen before. Check them out (a few slides back) for additional comments on configuration directives listed above.

Enabler for channelizing data to the correct plugin

instance: map to assign tags to records

Allowing only flows through (see next slide)

Allowing only events through (see next slide)

Touchingground:aconfigsnippetforbothmicro-flowsandeventlogging(2/2)shell> cat pretag.map

set_tag=10 ip=0.0.0.0/0 sample_type=flow

set_tag=20 ip=0.0.0.0/0 sample_type=event

Tags are assigned in pretag.map and recalled in the configuration by pre_tag_filter

Apply to all NetFlow/IPFIX exporters

Apply heuristics to classify records among flows and

events

Apply heuristics to classify records among flows and

events

Agenda

Brieflyonscalability§  Asinglecollectormightnotfititall:•  Memory:can’tstoreallBGPfullrou:ngtables•  CPU:can’tcopewiththepaceoftelemetryexport

§  Divide-et-imperaapproachisvalid:•  AssignPEs(bothtelemetryandBGP)tocollectors•  Ifna:velysupportedDB:

§ AssigncollectorstoDBnodes§ ClustertheDB

•  Ifnot-na:velysupportedDB:§ Assigncollectorstomessagebrokers§ Clusterthemessaginginfrastructure

Brieflyonscalability(cont.d)§  Intui:vely,thematrixcanbecomebig:•  Canbereducedbyexcludingen::esnegligibletothespecificscenario:§ Keepsmallerroutersoutoftheequa:on§  Filteroutspecific(classof)customers§  FocusondownstreamifCDN,upstreamifISP§  Sampleorputthresholdsontrafficrelevance

Needforhorizontalscalabilityinatelemetrycollector

§  High-frequencysampling(ie.security)§  Copewithincreasingdatarates:•  10Gto100Gbut,dependingontheapplica:on,samplingratesmightstaythesame

•  Eventslogging:ie.NetFlowchallengesSysloginthespaceofloggingCarrierGradeNAT(CGN)andfirewallevents

§  Scalewithoutsuper-compu:ngpowers

pmacct&horizontalscaling

§  Supportsa‘tee’plugin•  Receiverscanbeadded/changed/removedonthefly•  Load-balencedtee’ing(hashedorround-robin)•  Selec:vetee’ing

§ Mul:plepmacctcollectorscanruninparallel•  Couplingtelemetryandrou:ngdatafromsamePE

Touchingground:aconfigsnippetfortransparenthashingbalancer(1/2)

plugins: tee[balancer] !

plugin_buffer_size[blabla]: 100000

plugin_pipe_zmq[blabla]: true

tee_receivers[balancer]: /path/to/tee_receivers.lst

tee_transparent: true

Instantiating a tee plugin

File containing receivers definitions

Transparent balancing enabled. Disabling it acts as

a proxy

Queueing and buffering to handle sustained packet

Touchingground:aconfigsnippetfortransparenthashingbalancer(2/2)

shell> cat tee_receivers.lst

id=1 \

ip=192.168.5.1:2100,192.168.5.2:2100,192.168.5.3:2100 \

balance-alg=hash-agent

Touchingground:aconfigsnippetfortransparentselec:vebalancer(1/2)

plugins: tee[balancer] !

plugin_buffer_size[blabla]: 100000

plugin_pipe_zmq[blabla]: true

tee_receivers[balancer]: /path/to/tee_receivers.lst

tee_transparent: true

pre_tag_map: /path/to/pretag.map

NOTE: see config snippet for transparent hashing balancer (a few slides back) for additional comments on configuration directives listed above.

Enabler for selective balancing: map to assign tags

to NetFlow/IPFIX exporters

Touchingground:aconfigsnippetfortransparentselec:vebalancer(2/2)

shell> cat tee_receivers.lst

id=2 ip=192.168.4.1:2100 tag=100id=3 ip=192.168.4.2:2100 tag=200

shell> cat pretag.map

set_tag=100 ip=10.0.0.0/25

set_tag=200 ip=10.0.0.128/25

Tags are assigned in pretag.map and recalled in

tee_receivers.lst

Tags are assigned in pretag.map and recalled in

tee_receivers.lst

Furtherinforma:onaboutpmacct

§  h\ps://github.com/pmacct/pmacct•  OfficialGitHubrepository,wherestarandwatchusJ

§  h\p://www.pmacct.net/lucente_pmacct_uknof14.pdf•  MoreaboutcouplingtelemetryandBGP

§  h\p://ripe61.ripe.net/presenta:ons/156-ripe61-bcp-planning-and-te.pdf•  Moreabouttrafficmatrices,capacityplanning&TE

§  h\ps://github.com/pmacct/pmacct/wiki/•  Wiki:docs,implementa:onnotes,ecosystem,etc.

Thanks!Ques:ons?

PaoloLucente<paolo@pmacct.net>

h\p://www.pmacct.net/|h\ps://github.com/pmacct/pmacct

Backupslides

Post-processingRDBMSandrepor:ng(1/2)§  TrafficdeliveredtoaBGPpeer,perloca:on: mysql> SELECT peer_as_dst, peer_ip_dst, SUM(bytes), stamp_inserted

FROM acct_bgp

WHERE peer_as_dst = <peer | customer | IP transit> AND

stamp_inserted = < today | last hour | last 5 mins >

GROUP BY peer_as_dst, peer_ip_dst;§  AggregateASPATHstothesecondhop:mysql> SELECT SUBSTRING_INDEX(as_path, ‘.’, 2) AS as_path, bytes

FROM acct_bgp

WHERE local_pref = < IP transit pref> AND

stamp_inserted = < today | yesterday | last week >

GROUP BY SUBSTRING_INDEX(as_path, ‘.’, 2)

ORDER BY SUM(bytes);

§  Focuspeakhour(say,8pm)data:mysql> SELECT … FROM … WHERE stamp_inserted LIKE ‘2010-02-% 20:00:00’

§  Trafficbreakdown,ie.topNgroupingBGPpeersofthesamekind(ie.peers,customers,transit):

mysql> SELECT … FROM … WHERE …

local_pref = <<peer | customer | IP transit> pref>

§  Downloadtrafficmatrix(orasubsetofit)to3rdpartybackboneplanning/trafficengineeringapplica:on(ie.Cariden,Wandl,etc.):

mysql> SELECT peer_ip_src, peer_ip_dst, bytes, stamp_inserted

FROM acct_bgp

WHERE [ peer_ip_src = <location A> AND

peer_ip_dst = <location Z> AND … ]

stamp_inserted = < today | last hour | last 5 mins >

GROUP BY peer_ip_src, peer_ip_dst;

Post-processingRDBMSandrepor:ng(2/2)

Collec:ng telemetry data with pmacct - AfPIF · Presentaon history § 1.1: – MENOG 13 mee:ng,...

Documents

AFPIF: The Neutral Case Study

AfPIF 2013 Casablanca, Morocco Prenesh Padayachee Chief Technology Officer Internet Solutions

The African Peering and Interconnection Forum 2018 @iWeek fileAssociation (ISPA). The AfPIF conference will run from 21-23 August in conjunction with South Africa’s leading Internet

AFIX - AfPIF · The African IXP Association AfPIF 2015 / August 25th-27th, 2015 / Maputo, ... Annual survey of IXPs in Africa Launched in 2015 Helps us better understand our region

MEETING SUMMARY - AfPIF · Google, Netflix, Simbanet, iColo.io, Kooba, AFRINIC, Akamai, FranceIX, Netnod, Teraco, TISPA, Spicenet, and CTV. This year’s event theme was “Back to

“pmacct, a new player in the network management arena” · able to choose how to report them and supporting multiple collection methods. • A straight way to feed network data

The Peering and Interconnection The Africa Peering ForumThe second day at the Africa Peering and Interconnection Forum (AfPIF) is dedicated to plenary presentations and discussions

AfPIF-2015 “Lessons Learnt” · 2019-05-27 · Peering Introductions •Your AS Number •Where you Peer •Your Peering Policy •Contact Information •Email the above to mwangi@isoc.org

Andrew Owens - AfPIF · AFPIF: The Neutral Case Study Andrew Owens Africa’sInternet Timeline FIRST TELEPHONE FIRST EMAIL START OF

AfPIF 2016 "Back to Basics": Welcome and Introductions

Peering Negotiations at AfPIF

THE $1,000 INTERNET EXCHANGE WARNING: Contains up-and-to-the-right charts Remco van Mook AFPIF, September 2015

ARDiscovery#//€˜community‘discovery‘task‘force‘at ar‘summit/# # 3.#####Par:cipate#in#the#Barcelona Meetup#group#mee:ng#on#October#13## hap://€˜Reality‘Barcelona#

pmacct - NLNOG...aggregation method #1 aggregation method #2 nfacctd Usage scenarios Key pmacct non-technical facts 10+ years old project v[ oo Z vu ( Z }v ]vl Free, open-source, independent

CONWAY...Danny Rassmussen - 1008 Front Street Joanna Nabholz – H+N Architects - Architect Rik Sowell - Architect - 1315 North Street Ma@ Malle@ - St. Joseph Church and School Mee:ng

pmacct and Streaming Telemetry - pmacct project: IP ... · libpcap pmacct is open-source, free, GPL’edsoftware sFlow BGP maps IGP MySQL PgSQL SQLite MongoDB BerkeleyDB flat-files

National Information Center Afpif Sudan Exchange Point ... 2016 - 02 - Sudan IXP - Introduction.pdf · National Information Center August 2016 Afpif 1 Sudan Exchange Point (SIXP)

“IP accounting reloaded, the - pmacct · “IP accounting reloaded, the pmacct project: NetFlow, sFlow, SQL, RRD, stream classification and much more …” Paolo Lucente

The African Peering and Interconnection Forum 2018 @iWeek · Association (ISPA). The AfPIF conference will run from 21-23 August in conjunction with South Africa’s leading Internet

Protecting your peering edge. - AfPIF · Protecting your peering edge. Graham Beneke AfPIF 2015. #include std-disclaimer. IXP Peer 3 Peer 1 Peer 2 ISP. Expect to receive trafﬁc