Upload
garron
View
101
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Threat Connect : a visualized cyber-threats entity reporting system backed with H adoop ecosystem. Scott Miao, Trend Micro s [email protected] @ takeshi.miao. Who am I. RD, SPN, Trend Micro 3+ years for Hadoop eco system Expertise in HDFS/MR/ HBase @ takeshi.miao. Agenda. - PowerPoint PPT Presentation
Citation preview
2013 Trend Micro 25th Anniversary
Threat Connect : a visualized cyber-threats entity reporting system backed with Hadoop ecosystem
Scott Miao, Trend [email protected]
@takeshi.miao
2013 Trend Micro 25th Anniversary
Who am I
• RD, SPN, Trend Micro• 3+ years for Hadoop eco system• Expertise in HDFS/MR/HBase• @takeshi.miao
2013 Trend Micro 25th Anniversary
Agenda• Threat intelligence problem• Challenges and Solutions• Summary
2013 Trend Micro 25th Anniversary
THREAT INTELLIGENCE PROBLEM
“I want to quickly get an overview of the incident, including its scope, timeline, and impact.”
2013 Trend Micro 25th Anniversary
2013 Trend Micro 25th Anniversary
2013 Trend Micro 25th Anniversary 7
2013 Trend Micro 25th Anniversary
2013 Trend Micro 25th Anniversary
Threat Connect
• A Web Service for Threat Information Report– RESTful Interface to access– Integrated with TM Deep Discovery products
• Relevant and Actionable Intelligence
2013 Trend Micro 25th Anniversary
2013 Trend Micro 25th Anniversary
IP, domain, URL, filename, process, file hash, Virus detection, registry key, etc.
Product 1 Product 2 Product 3 …
Threat Conne
ct
Sand-box File
Detection
Threat
Web
Web Reputatio
nFamil
y Write-up
TE
Virus DB
APT KB
Most relevant threat report with actionable
intelligenceon a single portal
Process and correlates different data sources
2013 Trend Micro 25th Anniversary
CHALLENGES AND SOLUTIONS
2013 Trend Micro 25th Anniversary
StoringReal Time AccessPick Your right tool
Big DataMoving
Process & CorrelateGraph Problem
2013 Trend Micro 25th Anniversary
MOVING
2013 Trend Micro 25th Anniversary
Hadoop
Event Logs
FBSFBS
FBS
Feed Back log ServiceDear users/services
Accumulate small files
2013 Trend Micro 25th Anniversary
STORING
2013 Trend Micro 25th Anniversary
Cost
Easy Process
Archive
HDFS
2013 Trend Micro 25th Anniversary
PROCESS & CORRELATE
2013 Trend Micro 25th Anniversary
Pig/MR
• UDFs• MRs for special
cases
Store
• HDFS• Hbase• Solr• RDB
Time
• Batch• Performance
2013 Trend Micro 25th Anniversary
REAL TIME ACCESS
2013 Trend Micro 25th Anniversary
Real Time
Access
Free form
search
Random Access
Solr Cloud
HBase
• EX. Sandbox Reports
EX. Threat Detection DBs
2013 Trend Micro 25th Anniversary
GRAPH MODEL
2013 Trend Micro 25th Anniversary
Massive scalable ?
Active community ?
Analyzable ?
2013 Trend Micro 25th Anniversary
• We use HBase as a Graph Storage– Google BigTable and PageRank– HBaseCon2012
2013 Trend Micro 25th Anniversary
HGraph
Schema Design
Blueprints API
Graph Analysis MRs
https://github.com/tinkerpop/blueprints/wiki
2013 Trend Micro 25th Anniversary
PICK RIGHT TOOL
2013 Trend Micro 25th Anniversary
Pick right tool for right usecases
• Silver bullet ?• No one project fits all• One problem may has several choices
http://www.neevtech.com/blog/2013/03/18/hadoop-ecosystem-at-a-glance/
2013 Trend Micro 25th Anniversary
SUMMARY
2013 Trend Micro 25th Anniversary
Small files
• Namenode fsimage would explore the memory
• Too many map tasks to run for a job
FBSFBSFBS
2013 Trend Micro 25th Anniversary
Store your data anyway
• Store all the raw data on the HDFS– Break invisible isolation from different data
sources• Archive your data with deduced easy to use
FileFormat– Trenvi, RC file, ORC file
2013 Trend Micro 25th Anniversary
Know MR more
• Even you are the pig developer– Deal with MR issues– Write better pig-latin– Sometimes you can only use MR
2013 Trend Micro 25th Anniversary
Know your data & usecases
• Realtime ? Batch ?• Access Pattern ?• Therefore, you can pick right tool
2013 Trend Micro 25th Anniversary
THANK YOU GUYS