Elastic Stack

Technology Innovation Group

2017.06.20(Tue)Masamitsu Maehara

What happens when you visualize servers exposed to the world?

Self Introductionl Masamitsu Maehara

l Future Architect, Inc.

l Technology Innovation Group

l Messing around with AWS

l yurufuwa Engineer

@micci184

The Purposel Get to know the wonders of Elastic Stack

l Get to know the fun part of visualizing logs

Elastic Stack??

Elastic Stack

Elastic CLoud

LogStash Beats

Elasticsearch

Kibana

+

Security

Alert

Monitor

Graph

l Logstash/Beats：Import Logs

l Elasticsearch：Store/Index/Analyze

l Kibana：User Interface

Expose to the world?

HoneyPot

??

HoneyPotl High Interactive HoneyPot

l Use real OS and applications

l Easy access to information

l High Risk

l Low Interactive Honeypot

l Audit by emulating OS and applications

l Limited function

l Easily noticed by attackers

l Safer than high interactive HoneyPot

Dionaeal Low Interactive HoneyPot

l Gathers malware

l SMB/HTTP/HTTPS/FTP/TFTP/MSSQL/SIP

l Will create front-end view(just to make it look real)

l Low Interactive HoneyPot

l Specialized for SSH

l Better than Kippo

Cowrie

Configuration

l Built on AWS

l Install Beats on HoneyPot

l Gather data into Elastic Stack

Dionaea

Region@Virginia

HaneyPot VPC

Public Subnet

Cowrie ElasticStack

Wordpress

Client・・・

Monitoring

Attack

Logging

Install Dionaea### Ubuntu 14.04$ sudo apt-get update

$ sudo apt-get dist-upgrade

$ sudo apt-get install software-properties-common

$ sudo add-apt-repository ppa:honeynet/nightly

$ sudo apt-get update

$ sudo apt-get install dionaea

### Start Dionaea

$ sudo service dionaea start

Install Cowrie### Ubuntu 16.04$ sudo apt-get install git python-virtualenv libmpfr-dev libssl-dev libmpc-dev libffi-dev build-essential libpython-dev python2.7-minimal authbind

### adduser Cowrie

$ sudo adduser --disabled-password cowrie

$ sudo su - cowrie

### Setup Virtual Enviroment

$ virtualenv cowrie-env

$ source cowrie-env/bin/activate

### Install configuration file

$ export PYTHONPATH=/home/cowrie/cowrie

### Start Cowrie

$ bin/cowrie start

Activating virtualenv “cowrie-env”

Starting cowrie: [twistd -l log/cowrie.log --umask 0077 --pidfile var/run/cowrie.pid cowrie ]...

$ bin/cowrie status

cowrie is running (PID: 5979).

Beatsl Data Shipper

l The Beats FamBam

l Filebeat：Sends log files

l Metricbeat：Sends metric data（CPU/Mem..etc）l Packetbeat：Sends packet capture data

l Winlogbeat：Sends Windows event logs

l Heartbeat：Audits system stats

Data Flowl Store Apache/MySQL for WordPress directly into Elasticsearch

l Store logs from HoneyPot to Elasticsearch via Logstash

Dionaea Filebeat

Ubuntu

+ Log/dionaea.log/binalies/*

Amazon Linux

Logstash+ Input+ Filter+ Output

Cowrie Filebeat

Ubuntu

+ Log/cowrie.log

Elasticsearch

KibanaMetricBeat

Filebeat

Amazon Linux

+ Apache

Apache

WordPress

Packetbeat+ MySQL

Filebeat Modulesl Install Filebeat

$ curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.0.0-alpha2-x86_64.rpm

$ sudo rpm -vi filebeat-6.0.0-alpha2-x86_64.rpm

### Configuring

$ vim /etc/filebeat/filebeat.yml

#------------------------------- Apache2 Module ------------------------------

- module: apache2

# Access logs

access:

enabled: true

var.paths: ["/var/log/httpd/access_log"]

error:

enabled: true

var.paths: ["/var/log/httpd/error_log"]

#-------------------------- Elasticsearch output -------------------------------

output.elasticsearch:

hosts: [“xxx.xxx.xxx.xxx:9200"]

Ingest Pluginsl Install Ingest Geoip & Ingest user agent

l Ingest Geoip：Maps IP addresses to maps

l Ingest user agent：Deals with user agent as it thinks best

l Install Ingenst Plugins on the Elastic Stack server

### Ingest Geoip

$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip

### Ingest user agent

$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent

!!Attention #01l Be mindful of proxy environments

l Will receive a timeout error when installing Ingest Plugins

l Make sure to define proxy setting on the startup script before installing

$ sudo /usr/share/elasticsearch/bin//elasticsearch-plugin install ingest-user-agent

-> Downloading ingest-user-agent from elastic

Exception in thread "main" java.net.ConnectException: Connection timed out

### Setup Proxy

$ export ES_JAVA_OPTS="-Dhttp.proxyHost=xxx -Dhttp.proxyPort=xxx -Dhttps.proxyHost=xxx -Dhttps.proxyPort=xxx"

### Install ingest-user-agent

$ /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent

-> Downloading ingest-user-agent from elastic

[=================================================] 100% ### Ingest Geoip

!!Attention #02l Be mindful of proxy environments

l Handy setting that imports Dashboards when starting Filebeat＃Configure dashboard settings on filebeat.yml

l However in environments with proxy settings it will not work! :(

l In that case, install it manually

$ sudo vim /etc/filebeat/filebeat.yml

#============================== Dashboards =====================================

- #setup.dashboards.enabled: false

+ #setup.dashboards.enabled: enable

$ sudo /usr/share/filebeat/scripts/import_dashboards -file /tmp/beats-dashboards-x.x.zip -eshttp://xxx:9200

Visualization♥

Beautiful♥

Until recently…

l Send logs to be visualized from Filebeat to Logstash

l Normalize received logs with Logstash and store it on Elasticsearch

l Create a dashboard on Kibana to make it cool

Sooooo,,,Who needs Logstash?

Wait!

Logstash & Dionaea

Malwarel Are these malwares on Dionaea?

l They’re all over /opt/dionaea/var/dionaea/binaries…

$ ll /opt/dionaea/var/dionaea/binaries

-rw------- 1 dionaea dionaea 53 Jun 6 02:59 d41d8cd98f00b204e9800998ecf8427e.gz

-rw------- 1 dionaea dionaea 162168 Jun 7 22:56 dc8c32d7f26352c8484bc490b6467843.gz

-rw------- 1 dionaea dionaea 153820 Jun 7 02:34 dd0400bed68d272b08d1d0272bc18462.gz

-rw------- 1 dionaea dionaea 129803 Jun 5 01:01 de1e602b2452a95ba57ef53347e50094.gz

-rw------- 1 dionaea dionaea 22778 Jun 6 17:38 e0ddd8bf8e3b97ad25855721dc75daae.gz

-rw------- 1 dionaea dionaea 155154 Jun 7 04:33 e53ed987e82ad7bf076c23d91401cac7.gz

-rw------- 1 dionaea dionaea 1189 Jun 8 15:32 ead49a9b7b0c8ad6894be45674cebf77.gz

-rw------- 1 dionaea dionaea 22777 Jun 6 17:39 eb18a7d302bbc8c0b3ed2cd1612e8d59.gz

…

-rw------- 1 dionaea dionaea 21966 Jun 5 16:52 ee0efafc69a13cd57d714ffdc603d8fc.gz

-rw------- 1 dionaea dionaea 154329 Jun 4 16:48 f09ee5028fd1b1eaaf22df1538de159b.gz

-rw------- 1 dionaea dionaea 156637 Jun 9 08:51 f5f1fd0d093d81a4a769c20aca1d6232.gz

-rw------- 1 dionaea dionaea 29643 Jun 8 15:34 fc9b0b8b711e44ce0d4f91b0cedb1c76.gz

ClamScanl What do you do when you suspect a malware? You scan it.

l Malware FOUND$ clamscan /opt/dionaea/var/dionaea/binaries/

/opt/dionaea/var/dionaea/binaries/f09ee5028fd1b1eaaf22df1538de159b.gz: Win.Worm.Kido-200 FOUND

/opt/dionaea/var/dionaea/binaries/621c0b356c49edc5ce4cf3ee88c30f82.gz: OK

/opt/dionaea/var/dionaea/binaries/90e02a26204ade7771acf7e8521bdf09.gz: Win.Worm.Kido-297 FOUND

/opt/dionaea/var/dionaea/binaries/02830b424d88664cc3576941dd9841f9.gz: Win.Worm.Kido-307 FOUND

/opt/dionaea/var/dionaea/binaries/a7bc14c1bd7271a45391f1e1541afe43.gz: Win.Worm.Downadup-110 FOUND

/opt/dionaea/var/dionaea/binaries/87136c488903474630369e232704fa4d.gz: Win.Worm.Kido-113 FOUND

/opt/dionaea/var/dionaea/binaries/1195dfde6305980ed050a9751b157f42.gz: Win.Worm.Kido-293 FOUND

/opt/dionaea/var/dionaea/binaries/1b4cd56e54d3f9030a153590fb3fa9e5.gz: Win.Worm.Kido-316 FOUND

/opt/dionaea/var/dionaea/binaries/fc9b0b8b711e44ce0d4f91b0cedb1c76.gz: OK

/opt/dionaea/var/dionaea/binaries/cae8a8524eeb0e7de1fb3704bd14b7ba.gz: Win.Trojan.Ramnit-1847 FOUND

/opt/dionaea/var/dionaea/binaries/7bb455ea4a77b24478fba4de145115eb.gz: Win.Worm.Kido-197 FOUND

/opt/dionaea/var/dionaea/binaries/eb18a7d302bbc8c0b3ed2cd1612e8d59.gz: OK

/opt/dionaea/var/dionaea/binaries/smb-az4poq4s.tmp.gz: OK

/opt/dionaea/var/dionaea/binaries/16acf30169d089b8a967f40d9a38d8f7.gz: Win.Trojan.Agent-129152 FOUND

What if we want to surveillance these malwares in realtime?

Data Flowl Regularly runs ClamScan to output logs

l Filebeat sends the logs to the Elastic Stack server

l Logstash normalizes the logs and stores them into Elasticsearch

l Visualize with Kibana

Dionaea Filebeat

Ubuntu

+ Log/binalies/*/log/scan.log

clamscan.sh

Amazon Linux

Logstash+ Input+ Filter+ Output

Elasticsearch

KibanaMetricBeat

By the way, do you use

Logstash?

Logstash vs fluentdl Compared on Google Trend

l By country

l Blue：Logstash

l Red：fluentd

Grok filter

Grok Filter

l Below were the ClamScan results

l We need to somehow normalize it to get certain Key-Value data

l Data we want

l OK/FOUND (Key : check)

l Malware Name (Key : malware)$ clamscan /opt/dionaea/var/dionaea/binaries/

/opt/dionaea/var/dionaea/binaries/f09ee5028fd1b1eaaf22df1538de159b.gz: Win.Worm.Kido-200 FOUND

/opt/dionaea/var/dionaea/binaries/621c0b356c49edc5ce4cf3ee88c30f82.gz: OK

/opt/dionaea/var/dionaea/binaries/90e02a26204ade7771acf7e8521bdf09.gz: Win.Worm.Kido-297 FOUND

/opt/dionaea/var/dionaea/binaries/02830b424d88664cc3576941dd9841f9.gz: Win.Worm.Kido-307 FOUND

Such a pain in the neck…

Grok Constructorhttp://grokconstructor.appspot.com/do/match

Grok Constructor

l Let’s you test on web browsers

l You can also check stdout on Logstash

l Convenient if you don’t want to rewrite Logstash.conf

Paste log here

Grok Filter

Click GO! after pasting

Grok Constructor

l Results look like this

lOK/FOUND is contained in “check”

l But, where is the malware name?

l Work on that Grok Filter again

use contents detected in data

Grok Filter

Click GO! after pasting

Grok Constructor

l Results look like this

l Malware name is matched in malware!

Logstash.conf

l Final product looks like this

input {beats {

port => 5044}

}filter {

grok {match => [ "message", "/[^/]+/[^/]+/[^/]+/[^/]+/(?<field>[^/]+)/%{GREEDYDATA:data}%{WORD:check}"]remove_field => [ "host", "message" ]

}grok {

match => [ "data", "(?:[¥w._/%-]+)%{WORD}(?:[:]*)%{GREEDYDATA:malware}"]remove_field => [ "data" ]

}}output {

elasticsearch {hosts => "http://xxx.xxx.xxx.xxx:9200/"

}}

Looking good

Malware♥

Summaryl Let Elastic Stack do everything from input to output

l Easy visualization with Beats

l Expose your server to expand your log variation

l Are you excited to share your server with the public?

l Don’t let Logstash beat you!

Thanks