Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye

Using the SDACK Architecture on Security Event Inspection

Darren Chen

Evans YeSr. Software Engineer @ Trend Micro

Sr. Software Engineer @ Trend Micro

2016 DockerCon | Copyright© 2016 Trend Micro Inc.

About Darren• Darren Chen (Yu-Lun Chen)• Sr. Software Engineer @ Trend Micro• Enthusiast in big data and cloud computing

technologies• Docker experience – 1.5 years


About Evans• Evans Ye (Yu-Hsin Yeh) • Sr. Software Engineer @ Trend Micro• Apache Bigtop PMC member• Develop big data apps & infra• Docker experience – 2.5 years


How to make a software product ?


How to make a Dockerize

software product ?



BeforeMotivationWhat is SDACK

Agenda

DuringWhy DockerizeSecurityMonitor

AfterLessons LearnedConclusionsQ&A


Motivation

Target Scenario


Problems• Too many log to investigate• Lack of actionable, prioritized

recommendations


AD WindowsEvent

DNS Proxy Web server

…..

ThreatAnalytic System


But we faced Two problems…….


How to deal with

Customers’ Private data ?


Cloud On Premises

How to deal with Big Volume logs ?

2,000,000,000 per day


We need to build

an On-Premises product

which can deal with Big Data


How to deal with Big Data?



Toolbox for building wide variety of big data product

SDACK Architecture


What is SDACK

SDACK

Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka

fast and general engine for large-scale data processing

deployment and resource management

toolkit and runtime for building highly concurrent,distributed, and resilient message-driven applications

distributed, highly available database designedto handle large amounts of data across datacenters

high-throughput, low-latency distributed pub-submessaging system for real-time data feeds


Data Storage

Data Analysis

Data Preprocessing

Data PipelinePackage


Threat Analytic System Architecture


Log

APIServer

WebServer 2016 DockerCon | Copyright© 2016 Trend Micro Inc.

APIServer

Web Server


Medium-sized Enterprises


Large Enterprises


Fortune 500


With Docker• Easy to scale• Test once, run anywhere• Widely supported by many platforms



Why Dockerize

Dockerize – Benefit


Deploy Develop

Test Scale


Deploy Develop

Test Scale

Dockerize – Benefit 1


APIWeb

Challenge• Setup• Operate• Update


APIServer

Web Server

Dockerize Software Technologies

Docker Compose for Operation


Web Server

APIServer Docker Compose

kafka: build: . ports: - “9092:9092”spark: image: spark port: - “8080:8080” ……

APIServer

Web Server

Docker Hub for Updating


APIServer

Docker Hub

APIServer

Web Server



Deploy Develop

Test Scale

Benefit for Development• Docker provides two benefits in our Spark jobs

development – Reproducibility– Flexibility



Reproducibilityin

Spark Streaming Job Development


Dev Cluster


Data Streams


Local


Data Streams

SnapshotData Set

(Date : Jan. 04 ~ Jan. 08)

Freq. : 1 minBatch size : 1000


Local


Data Streams

SnapshotData Set

(Date : Jan. 04 ~ Jan. 08)


Freq. : 0.5 minBatch size : 5000


1

2

3

Quick Development IterationLocal

LocalData StreamsSnapshotData Set


Local

Deploy

Test

Destroy

ModifyJob

Job


Flexibilityin

Hybrid Architecture

Data Research in Dev Cluster

2016 DockerCon | Copyright© 2016 Trend Micro Inc.2016 DockerCon | Copyright© 2016 Trend Micro Inc.

APIWeb

Dev ClusterData scientists submit spark jobs

Job



APIWeb

Dev Cluster

Job

Result

Data scientists submit spark jobs



APIWeb

Dev ClusterData scientists submit spark jobs



APIWeb

Dev Cluster

Job

Other memberssubmit spark jobs



APIWeb

Dev Cluster

Job

Wrong Result

Other memberssubmit spark jobs

Hybrid Architecture


Dev ClusterSubmit Spark Job

APIWeb

APIWeb

Job

Result

Local

What’s More


Dev ClusterWeb Service Development

APIWeb

APIWeb

Local



Deploy Develop

Test Scale

APIServer

Web Server

• Test case 1• sub-test 1a• sub-test 1b

• Test case 2• sub-test 2a• sub-test 2b

• Test case n• sub-test na• sub-test nb


APIServer

Web Server

APIServer

Web Server

…

Clean & Consistent Environment



Deploy Develop

Test Scale

Distributed Software Components


Akka• High performance concurrency framework• Clustering mechanism available• Leverage on Akka, we build up our Akka

cluster system


Our Akka Cluster System


Client

Master

LDAPServer

1

2 3

4

Query account information

Send the job

Query LDAP ServerReturn the result LDAPService

Our Akka Cluster System


Master

LDAP HostName DB

DataProcessEndpoint

JobJobJob

Dockerize for Each Micro-service


LDAP

DB

DataProcess

Endpoint

HostName

Master

Dockerize for Scale Out


DataProcess

HostName

DB LDAP Endpoint

DataProcess

DataProcess


Security

Docker Vulnerabilities since 1st release


The only high severity vulnerability was fixed within 2 days.

Misconfiguration


Open it without ACL ?

Open Docker Registry


AU BE CA CN DE FI FR GB HK HR IE IR IT JP KR NL PL RU SE SG TW US ZA0

10

20

30

40

50

60

70

80

90

Open Docker Registry w/o Access Control


Some tools can make your Dockerize product more secure

Docker Bench for Security• Check

– Host configuration– Docker daemon configuration– Docker daemon configuration files– Container images and build files– Container runtime


CoreOS Clair• Static analysis of vulnerabilities

– Debian security bug tracker– Ubuntu CVE tracker– Red Hat security data


Docker Cloud



Monitor

Web Server

APIServer

Monitor stack


Grafana

CPU, Memory, Network Metrics

Monitor stack


Grafana

Metrics

APPMetrics

Issue on cAdvisor• cAdvisor can not send network usage correctly

to InfuxDB– when the container use host network on a

multiple network cards machine• Use Telegraf to fix this problem



BeforeMotivationWhat is SDACK

Agenda

DuringWhy DockerizeSecurityMonitor

AfterLessons LearnedConclusionsQ&A


Lessons Learned

Lessons Learned• Mount the stuff you may change it frequently

to your Docker containers– For example, on PoC, mount your configuration

files into Docker containers directly


On PoC


Change Settings

Re-build Images Deploy

APIServer

Web Server

Mount configuration files


Host machine

Conf

Kafka container

Conf Conf

Spark container

Conf Conf Conf

Conf Conf Conf

Kafka Configurations

Conf Conf Conf

Spark Configurations


Conclusions

Summary


Dockerize

• Deploy• Develop• Test• Scale

Security

• Misconfiguration• Docker Bench• CoreOS Clair• Docker Cloud

Monitor

• Visibility• cAdvisor• InfluxDB• Grafana

APIServer

Web Server

for Security


We Need To build an On-Premises product


In the beginning …


We Need To build

an On-Premises product


Have NowBuild

Ship

Run

Conclusions


Go aheadDockerize your product


Thank you!


Q & A


Thank you!

Technology

Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye