• Home

  • Documents

  • using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup...
40
Cloud monitoring using CHR and Big-Data

using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

Embed Size (px)

Citation preview

Page 1: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

Cloud monitoring using

CHR and Big-Data

Page 2: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

SummaryAbout Us

General background on CHR

CHR Amazon EC2 installing

CHR Use cases

Cloud monitoring elements

Reporting , Alerting and Trigger

Page 3: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

<whoami

2

Shlomi GutmanCTO of Voicenter (Israel)

VP of Cloud Products at QXIP (Amsterdam)

Page 4: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

3

Voicenter is A leading telecommunication technology company providing top-tier business telephony since 2007

We are delivering a ‘One-stop-shop’ solution for business all around the world

TelecomServices

PBX Call CenterSolution

Page 5: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

4

QXIP - Voice Capture Engineering & Development

QXIP {QuickSIP} is an R&D Company specializing in Open-Source and Commercial Voice Technology Development.

Page 6: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

5

What’s CHR?

Cloud Hosted Router (CHR) is a RouterOS version intended for running as a virtual machine.

It supports the x86 64-bit architecture and can be used on most of the popular hypervisors such as VMWare, Hyper-V, VirtualBox and others.

CHR has full RouterOS features enabled by default but has a different licensing model than other RouterOSversions.

Page 7: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

6

CHR Licensing License

Perpetual is a lifetime license -buy once, use forever .

It is possible to transfer a perpetual license to another CHR instance.

License Speed limit Price

Free 1Mbit FREE

P1 1Gbit $45

P10 10Gbit $95

P-Unlimited Unlimited $250

If the CHR instance will not be able to access the account server to renew the license ,it will behave as if the trial period has ran out and will not allow an upgrade of RouterOS to a newer version.

Page 8: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

CHR hosting environment

Page 9: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

8

Installing CHR on AWS – Step 1

Page 10: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

9

Select CHR Image (AMI)– Step 2

Page 11: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

10

Select Instance Tab – Step 3

Page 12: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

11

Setup your network – Step 4

Page 13: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

12

Configure – Step 5

Page 14: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

13

Installing CHR on AWS – Step 6

Page 15: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

15

Finally… Winbox ... IP… Connect ...

Page 16: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

16

Change Password !!!

Page 17: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

17

CHR - Use case Types

• Virtual Instance

Custom hardware

Management - Dude ,RADIUS

Labs setup

Page 18: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

18

Virtualization – CHR vs x86Why use the CHR instead of the traditional x86 VM?

• Optimized for Virtualization 64 bit support Fastpath support Driver support

• Paravirtualized NIC –Using the CHR allows us to use the a paravirtualized NIC which is capable of speeds beyond 10 Gbps. The E1000 NIC used in the x86 VM is only capable of 1Gbps.

• Future proof – The CHR will continue to be developed

Page 19: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

19

CHR - Use case Types

• Cloud Connectivity VPN cloud - Road Warrior

Direct Connect alternative

Secure distributed cloud environment

Page 20: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

20

CHR - Use case Types

• Cloud monitoring Cyber Defense

Billing Logic on Steroids

Centralized Log Analyze

Page 21: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch
Page 22: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

21

Cyber crime top 20 countries attracts

Page 23: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

22

IOT – the missing S

Page 24: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

23

General background on cyber attracts

Page 25: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

24

Who is behind cyber crime ?

Page 26: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

25

Page 27: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

26

Page 28: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

27

How to ship your data(Syslog) …..

Page 29: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

28

How to ship your data (NetFlow)

/ip traffic-flowset cache-entries=4M enabled=yes

interfaces=BRIDGE

/ip traffic-flow targetadd dst-address=66.66.66.66

port=1234 version=5

Page 30: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

29

Shipping Big Data Log• paStash is a tool to manage spaghetti I/O

with input, processors and output. • modules for all seasons and protocols.

https://github.com/sipcapture/paStash

Page 31: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

30

PaStash Config

Input plugins•File

•Syslog

•ZeroMQ

•Redis

•HTTP

•Websocket

•TCP / TLS

•Google app engine

•AMQP

•SQS

•NetFlow

•Freeswitch ESL

•Asterisk AMI

Outputs•ZeroMQ

•ElasticSearch

•Statsd

•Gelf

•File

•HTTP Post

•Websocket

•Redis

•Logio

•TCP / TLS

•AMQP

•SQS

•HEP

Filter plugins•Regex

•Grok

•Mutate Replace

•Grep

•Reverse DNS

•Compute field

•Compute hash

•Compute date field

•Split

•Rename

•Multiline

•Json fields

•Geoip

•Eval

•Bunyan

•HTTP Status Classifier

input {udp {host => 0.0.0.0port => 514type => syslog

}}

filter {regex {regex => /^(\S)/+/fields => [toto]

}}

output {elasticsearch {host => localhostport => 9200

}}

https://github.com/sipcapture/paStash/blob/master/docs/inputs/file.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/syslog.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/zeromq.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/redis.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/http.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/ws.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/tcp_tls.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/gae.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/amqp.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/sqs.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/netflow.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/esl.md
https://github.com/sipcapture/paStash/blob/master/docs/inputs/ami.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/zeromq.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/elasticsearch.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/statsd.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/gelf.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/file.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/http_post.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/ws.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/redis.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/logio.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/tcp_tls.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/amqp.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/sqs.md
https://github.com/sipcapture/paStash/blob/master/docs/outputs/hep.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/regex.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/grok.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/mutate_replace.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/grep.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/reverse_dns.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/compute_field.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/compute_hash.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/compute_date_field.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/split.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/rename.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/multiline.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/json_fields.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/geoip.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/eval.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/bunyan.md
https://github.com/sipcapture/paStash/blob/master/docs/filters/http_status_classifier.md
Page 32: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

31

Parsing Mikrotik Netflow

Page 33: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

32

Parsing Mikrotik Log

Page 34: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

33

Mikrotik Netflow Dashboards

Page 35: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

33

Mikrotik Logs Dashboards

Page 36: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

34

ElasticsearchElasticsearch is a search engine based on Lucene. It provides a distributed,multitenant-capable

full-text search engine with an HTTP web interface and schema-free JSON documents.

Page 37: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

35

Siren alerting & reporting application

SENTINL extends Siren with Alerting and Reporting functionality to monitor, validate and inform users and administrators on data series changes using standard queries or join queries, programmable validators, transformers and messages to send out using a variety of configurable actions including sending action to the Mikrotik API as well as sending Emails, Slack Messages, API Webhooks, PDF Snapshots of Charts, creating new Documents and much more.

Page 38: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

33

Siren Alerting & Reporting App

Siren Enterprise provides many unique features and enables integrators to realize unique Business Intelligence creatures. With such power, automating workflows and being able to get notified with data detections quickly becomes a key requirement.

Page 39: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

[email protected] us today for a month free trial

Pushing your data out of the box

Page 40: using CHR and Big-Data - MUM - MikroTik User Meeting · Select Instance Tab –Step 3. 11 Setup your network –Step 4. 12 Configure –Step 5. 13 ... fields => [toto]}} output {elasticsearch

THANK YOU!


Elasticsearch Aggregations

Elasticsearch Aggregations

Software
Neo4j Integration with ElasticSearch - ElasticSearch Meetup

Neo4j Integration with ElasticSearch - ElasticSearch Meetup

Data & Analytics
Elasticsearch, Logstash, and Other Data · Elasticsearch, Logstash, and Other Data Preamble and Introduction Overview • Elasticsearch – a search engine • Logstash – filters

Elasticsearch, Logstash, and Other Data · Elasticsearch, Logstash, and Other Data Preamble and Introduction Overview • Elasticsearch – a search engine • Logstash – filters

Documents
RA-SBT-CHR, RB-SBT-CHR - Lutron Electronics · A Step-by-Step Guide for Installion, Programming, and Operation Note: Please leave this manual with homeowner. RA-SBT-CHR, RB-SBT-CHR

RA-SBT-CHR, RB-SBT-CHR - Lutron Electronics · A Step-by-Step Guide for Installion, Programming, and Operation Note: Please leave this manual with homeowner. RA-SBT-CHR, RB-SBT-CHR

Documents
Elasticsearch 5 in Amazon Elasticsearch Service

Elasticsearch 5 in Amazon Elasticsearch Service

Technology
Elasticsearch for dummies · Jakub Kluvánek jakub.kluvanek@biddingfox.com @kluvi, @biddingfox_devs Elasticsearch for dummies

Elasticsearch for dummies · Jakub Kluvánek [email protected] @kluvi, @biddingfox_devs Elasticsearch for dummies

Documents
Elasticsearch, Logstash, Kibana Technical Walk-Throughfiles.meetup.com/16806932/ES-Technical-Walk-Through_ADL_201508.pdf · 3 Elasticsearch Terminology •A node is a single Elasticsearch

Elasticsearch, Logstash, Kibana Technical Walk-Throughfiles.meetup.com/16806932/ES-Technical-Walk-Through_ADL_201508.pdf · 3 Elasticsearch Terminology •A node is a single Elasticsearch

Documents
Genetic Linkage 1 rs12426597 rs6447271 Chr. 4 Chr. 12

Genetic Linkage 1 rs12426597 rs6447271 Chr. 4 Chr. 12

Documents
Introduction to Elasticsearch

Introduction to Elasticsearch

Software
Solr vs ElasticSearch

Solr vs ElasticSearch

Technology
Elasticsearch SQL Webinar · 2020-06-07 · Elasticsearch SQL RDBMS Terminological Differences Relational Databases (RDBMS) Elasticsearch SQL Column Field Row Document Table

Elasticsearch SQL Webinar · 2020-06-07 · Elasticsearch SQL RDBMS Terminological Differences Relational Databases (RDBMS) Elasticsearch SQL Column Field Row Document Table

Documents
Book Exposition ( Ma + chr ) / Book exhibition ( Ma + Chr )

Book Exposition ( Ma + chr ) / Book exhibition ( Ma + Chr )

Documents
Elasticsearch - DevNexus 2015

Elasticsearch - DevNexus 2015

Technology
Elasticsearch Workshop

Elasticsearch Workshop

Data & Analytics
ElasticSearch 5.x - New Tricks - 2017-02-08 - Elasticsearch Meetup

ElasticSearch 5.x - New Tricks - 2017-02-08 - Elasticsearch Meetup

Data & Analytics
Elasticsearch @ Keboola

Elasticsearch @ Keboola

Software
TGCP Optimizer - Step by Step Rev.PA2 - 3GCellPro fileIn case loading (from Files->Read ... 2.2 How to import Huawei RNC TRACE CHR&MNCDT? ... SOFT_HANDOVER_EXECUTION and MISSING_NEIGBOUR

TGCP Optimizer - Step by Step Rev.PA2 - 3GCellPro fileIn case loading (from Files->Read ... 2.2 How to import Huawei RNC TRACE CHR&MNCDT? ... SOFT_HANDOVER_EXECUTION and MISSING_NEIGBOUR

Documents
Elasticsearch: Accelerating the Django Admin · Elasticsearch Service elastic cloud Elasticsearch Reference + + + + + + + Elasticsearch Reference: 6.4 (current) Getting Started Set

Elasticsearch: Accelerating the Django Admin · Elasticsearch Service elastic cloud Elasticsearch Reference + + + + + + + Elasticsearch Reference: 6.4 (current) Getting Started Set

Documents
Elasticsearch 101

Elasticsearch 101

Software
Dell EMC Search · above Elasticsearch. Elasticsearch cluster ports NGINX TCP/ HTTPS 9300– 9400 Ports for communicating with Elasticsearch (Index data nodes). Elasticsearch cluster

Dell EMC Search · above Elasticsearch. Elasticsearch cluster ports NGINX TCP/ HTTPS 9300– 9400 Ports for communicating with Elasticsearch (Index data nodes). Elasticsearch cluster

Documents
ElasticSearch Basics

ElasticSearch Basics

Technology
Full Chr-17 & Chr-18

Full Chr-17 & Chr-18

Documents
ELASTICSEARCH | LOGSTASH | KIBANA”Elasticsearch, along with Logstash and Kibana, provides a powerful open source platform for indexing, searching and ... datadog_metrics elasticsearch

ELASTICSEARCH | LOGSTASH | KIBANA”Elasticsearch, along with Logstash and Kibana, provides a powerful open source platform for indexing, searching and ... datadog_metrics elasticsearch

Documents
elasticsearch - Berlin Buzzwords 20112011.berlinbuzzwords.de/.../files/elasticsearch-bbuzz2011.pdf · elasticsearch The Road to a Distributed, (Near) Real Time, Search Engine Shay

elasticsearch - Berlin Buzzwords 20112011.berlinbuzzwords.de/.../files/elasticsearch-bbuzz2011.pdf · elasticsearch The Road to a Distributed, (Near) Real Time, Search Engine Shay

Documents
Elasticsearch and Spark

Elasticsearch and Spark

Data & Analytics
ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG

ELASTIC SEARCH, LOGSTASH, KIBANA & · PDF fileInstall LogSTASH Install java much like elasticsearch installation step Run the following command to import the Elasticsearch public GPG

Documents
Http Www.hotelschool.cornell.edu Chr PDF Showpdf Chr Research Tools Carrolltool.pdf My Path Info=Chr Research Tools Carrolltool

Http Www.hotelschool.cornell.edu Chr PDF Showpdf Chr Research Tools Carrolltool.pdf My Path Info=Chr Research Tools Carrolltool

Documents
Elasticsearch Py

Elasticsearch Py

Documents
Elasticsearch Documentation

Elasticsearch Documentation

Documents
Meetup ElasticSearch : « Booster votre Magento avec Elasticsearch »

Meetup ElasticSearch : « Booster votre Magento avec Elasticsearch »

Engineering