Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye

Using the SDACK Architecture on Security Event Inspection

Darren Chen

Evans YeSr. Software Engineer @ Trend Micro

Sr. Software Engineer @ Trend Micro

About Darren• Darren Chen (Yu-Lun Chen)• Sr. Software Engineer @ Trend Micro• Enthusiast in big data and cloud computing

technologies• Docker experience – 1.5 years

About Evans• Evans Ye (Yu-Hsin Yeh) • Sr. Software Engineer @ Trend Micro• Apache Bigtop PMC member• Develop big data apps & infra• Docker experience – 2.5 years

How to make a software product ?

How to make a Dockerize

software product ?

BeforeMotivationWhat is SDACK

Agenda

DuringWhy DockerizeSecurityMonitor

AfterLessons LearnedConclusionsQ&A

Motivation

Target Scenario

Problems• Too many log to investigate• Lack of actionable, prioritized

recommendations

AD WindowsEvent

DNS Proxy Web server

ThreatAnalytic System

But we faced Two problems…….

How to deal with

Customers’ Private data ?

Cloud On Premises

How to deal with Big Volume logs ?

2,000,000,000 per day

We need to build

an On-Premises product

which can deal with Big Data

How to deal with Big Data?

Toolbox for building wide variety of big data product

SDACK Architecture

What is SDACK

Source: http://www.slideshare.net/akirillov/data-processing-platforms-architectures-with-spark-mesos-akka-cassandra-and-kafka

fast and general engine for large-scale data processing

deployment and resource management

toolkit and runtime for building highly concurrent,distributed, and resilient message-driven applications

distributed, highly available database designedto handle large amounts of data across datacenters

high-throughput, low-latency distributed pub-submessaging system for real-time data feeds

Data Storage

Data Analysis

Data Preprocessing

Data PipelinePackage

Threat Analytic System Architecture

APIServer

Web Server

Medium-sized Enterprises

Large Enterprises

Fortune 500

With Docker• Easy to scale• Test once, run anywhere• Widely supported by many platforms

Why Dockerize

Dockerize – Benefit

Deploy Develop

Test Scale

Deploy Develop

Test Scale

Dockerize – Benefit 1

APIWeb

Challenge• Setup• Operate• Update

APIServer

Web Server

Dockerize Software Technologies

Docker Compose for Operation

Web Server

APIServer Docker Compose

kafka: build: . ports: - “9092:9092”spark: image: spark port: - “8080:8080” ……

APIServer

Web Server

Docker Hub for Updating

APIServer

Docker Hub

APIServer

Web Server

Deploy Develop

Test Scale

Benefit for Development• Docker provides two benefits in our Spark jobs

development – Reproducibility– Flexibility

Reproducibilityin

Spark Streaming Job Development

Dev Cluster

Data Streams

SnapshotData Set

(Date : Jan. 04 ~ Jan. 08)

Freq. : 1 minBatch size : 1000

Data Streams

SnapshotData Set

(Date : Jan. 04 ~ Jan. 08)

Freq. : 0.5 minBatch size : 5000

Quick Development IterationLocal

LocalData StreamsSnapshotData Set

Deploy

Destroy

ModifyJob

Flexibilityin

Hybrid Architecture

Data Research in Dev Cluster

APIWeb

Dev ClusterData scientists submit spark jobs

APIWeb

Dev Cluster

Result

Data scientists submit spark jobs

APIWeb

Dev ClusterData scientists submit spark jobs

APIWeb

Dev Cluster

Other memberssubmit spark jobs

APIWeb

Dev Cluster

Wrong Result

Other memberssubmit spark jobs

Hybrid Architecture

Dev ClusterSubmit Spark Job

APIWeb

Result

What’s More

Dev ClusterWeb Service Development

APIWeb

Deploy Develop

Test Scale

APIServer

Web Server

• Test case 1• sub-test 1a• sub-test 1b

• Test case 2• sub-test 2a• sub-test 2b

• Test case n• sub-test na• sub-test nb

APIServer

Web Server

APIServer

Web Server

Clean & Consistent Environment

Deploy Develop

Test Scale

Distributed Software Components

Akka• High performance concurrency framework• Clustering mechanism available• Leverage on Akka, we build up our Akka

cluster system

Our Akka Cluster System

Client

Master

LDAPServer

Query account information

Send the job

Query LDAP ServerReturn the result LDAPService

Our Akka Cluster System

Master

LDAP HostName DB

DataProcessEndpoint

JobJobJob

Dockerize for Each Micro-service

DataProcess

Endpoint

HostName

Master

Dockerize for Scale Out

DataProcess

HostName

DB LDAP Endpoint

DataProcess

Security

Docker Vulnerabilities since 1st release

The only high severity vulnerability was fixed within 2 days.

Misconfiguration

Open it without ACL ?

Open Docker Registry

AU BE CA CN DE FI FR GB HK HR IE IR IT JP KR NL PL RU SE SG TW US ZA0

Open Docker Registry w/o Access Control

Some tools can make your Dockerize product more secure

Docker Bench for Security• Check

– Host configuration– Docker daemon configuration– Docker daemon configuration files– Container images and build files– Container runtime

CoreOS Clair• Static analysis of vulnerabilities

– Debian security bug tracker– Ubuntu CVE tracker– Red Hat security data

Docker Cloud

Monitor

Web Server

APIServer

Monitor stack

Grafana

CPU, Memory, Network Metrics

Monitor stack

Grafana

Metrics

APPMetrics

Issue on cAdvisor• cAdvisor can not send network usage correctly

to InfuxDB– when the container use host network on a

multiple network cards machine• Use Telegraf to fix this problem

BeforeMotivationWhat is SDACK

Agenda

DuringWhy DockerizeSecurityMonitor

AfterLessons LearnedConclusionsQ&A

Lessons Learned

Lessons Learned• Mount the stuff you may change it frequently

to your Docker containers– For example, on PoC, mount your configuration

files into Docker containers directly

On PoC

Change Settings

Re-build Images Deploy

APIServer

Web Server

Mount configuration files

Host machine

Kafka container

Conf Conf

Spark container

Conf Conf Conf

Kafka Configurations

Conf Conf Conf

Spark Configurations

Conclusions

Summary

Dockerize

• Deploy• Develop• Test• Scale

Security

• Misconfiguration• Docker Bench• CoreOS Clair• Docker Cloud

Monitor

• Visibility• cAdvisor• InfluxDB• Grafana

APIServer

Web Server

for Security

We Need To build an On-Premises product

In the beginning …

We Need To build

an On-Premises product

Have NowBuild

Conclusions

Go aheadDockerize your product

Thank you!

Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and Evans Ye

Technology

Lun Provision

Feature selection and transduction for prediction of molecular bioactivity for drug design Reporter: Yu Lun Kuo (D95922037) E-mail: sscc6991@gmail.comsscc6991@gmail.com

NANO LETTERS Diameter-Dependent Electron …nano.eecs.berkeley.edu/publications/NanoLett_2009_InAs...Diameter-Dependent Electron Mobility of InAs Nanowires , Yu-Lun Chueh, Yu-Chih

SCIENTIFIC ABSTRACT MOSKALENKO, YU. YE. - MOSKALEV, …...SCIENTIFIC ABSTRACT MOSKALENKO, YU. YE. - MOSKALEV, A. N. Subject: SCIENTIFIC ABSTRACT MOSKALENKO, YU. YE. - MOSKALEV, A

Deep Video Frame Interpolation using Cyclic Frame Generationyulunliu/CyclicGen_/liu.pdf · 2019-02-12 · Deep Video Frame Interpolation using Cyclic Frame Generation Yu-Lun Liu,1,2,3

Computer Architecture Chapter 4 Assessing and Understanding Performance Yu-Lun Kuo 郭育倫 Department of Computer Science and Information Engineering Tunghai

Sketch of Adiponectin Receptors Hongkui Yu Hongqiang Ye Jie Nie Xiangyu Sun These authors contributed equally to this work

C. Mitchell | F. Ye | N. Wiseman Shang Han Lun on Cold

Autoinhibitory Mechanisms in Receptor Tyrosine Kinases Reporter: Yu Lun Kuo E-mail: sscc6991@gmail.comsscc6991@gmail.com Date: Nov. 29, 2007 Front Biosci

Colloidal Lithography Chapter 1 Ye Yu and Gang …cdn.intechopen.com/.../InTech-Colloidal_lithography.pdfChapter 1 Colloidal Lithography Ye Yu and Gang Zhang Additional information

Cloud meets Big Data - IDGweb.idg.no/app/web/online/Event/Energyworld/2012/Presentasjoner/E… · 16 tb lun 16 tb lun 16 tb lun 16 tb lun 16 tb lun 16 tb lun 16 tb lun 16 tb lun 16

Manhattan Scene Understanding via XSlit Imaging...Manhattan Scene Understanding Via XSlit Imaging Jinwei Ye Yu Ji Jingyi Yu University of Delaware, Newark, DE 19716, USA {jye,yuji,yu}@cis.udel.eduAbstract

1/17 Identification of thermophilic species by the amino acid compositions deduced from their genomes Reporter: Yu Lun Kuo E-mail: sscc6991@gmail.comsscc6991@gmail.com

Implementation In Tree Stat 6601 November 24, 2004 Bin Hu, Philip Wong, Yu Ye

SCIENTIFIC ABSTRACT PULYER, YU. M. - PUMPER, YE. YA

Ye Nwe Moe - Interview with Dr Lun Swe

CONTENTS · Mr YU Lup Fat Joseph BOARD COMMITTEES AUDIT COMMITTEE Mr CHENG Yuk Wo (Chairman) Mr CHOW Cheuk Yu Alfred BBS, JP Ms CHEN Jing Mr LEE Ka Lun Mr YU Lup Fat Joseph CONNECTED

Launching a Security Testbed for Wireless Networks with ... · National Chiao Tung University bortingchen.ece98g@nctu.edu.tw Yu-Lun Huang National Chiao Tung University ylhuang@cn.nctu.edu.tw

Taipei 101 Mall George Tian Pei-Yu Tsai Tzen Chan Yu-Lun Huang

SCIENTIFIC ABSTRACT MELNIKOV, YE. F. - MELNIKOV, … · Title: SCIENTIFIC ABSTRACT MELNIKOV, YE. F. - MELNIKOV, YU. S. Subject: SCIENTIFIC ABSTRACT MELNIKOV, YE. F. - MELNIKOV, YU