Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
1 © Hortonworks Inc. 2011–2018. All rights reserved
27-29 November, Vilnius
Apache Metron in the Real WorldDave Russell - Hortonworks
www.roaringelephant.org
2 © Hortonworks Inc. 2011–2018. All rights reserved
Who am I?
3 © Hortonworks Inc. 2011–2018. All rights reserved
4 © Hortonworks Inc. 2011–2018. All rights reserved
Why Apache Metron?
5 © Hortonworks Inc. 2011–2018. All rights reserved
Months until breach noticed
Avg. months log retention
9 6
VS
3Months missing
6 © Hortonworks Inc. 2011–2018. All rights reserved
28 Months
Police One/Berkut
Yahoo/FSB FB/Cambridge Analytica
35 Months 48 Months
Time until breach actually noticed
7 © Hortonworks Inc. 2011–2018. All rights reserved
“Sometime in the next few years we're going to have our first
category-one cyber-incident; one that will need a national response.”
Ian LevyTechnical Director
National Cyber Security Centre
8 © Hortonworks Inc. 2011–2018. All rights reserved
Andhra Pradesh Police, IndiaAristotle University of Thessaloniki, GreeceAutomobile Dacia, RomaniaCambrian College, CanadaChinese public security bureauCJ CGVDalian Maritime UniversityDeutsche BahnDharmais Hospital, IndonesiaFaculty Hospital, Nitra, SlovakiaFedExGarena Blade and SoulGuilin University Of Aerospace TechnologyGuilin University Of Electronic TechnologyHarapan Kita Hospital[disambiguation needed], IndonesiaHezhou University
SandvikSão Paulo Court of JusticeSaudi Telecom CompanySberbankShandong UniversityState Governments of India Government of GujaratGovernment of KeralaGovernment of MaharashtraGovernment of West BengalSuzhou Vehicle AdministrationSun Yat-sen University, ChinaTelefónicaTelenor Hungary, HungaryTelkom (South Africa)Timrå Municipality, SwedenUniversitas Jember, IndonesiaUniversity of Milano-Bicocca, ItalyUniversity of Montreal, CanadaVivo, Brazil
HitachiHondaInstituto Nacional de Salud, ColombiaLakeridge HealthLAKSLATAM Airlines GroupMegaFonMinistry of Internal Affairs of the Russian FederationMinistry of Foreign Affairs (Romania)National Health Service (England)NHS ScotlandNissan Motor Manufacturing UKO2, GermanyPetrobrásPetroChinaPortugal TelecomPulse FMQ-ParkRenaultRussian Railways
9 © Hortonworks Inc. 2011–2018. All rights reserved
2018 so far...
340M Records
150M Records
92M RecordsAnd many, many, many more..https://en.wikipedia.org/wiki/List_of_data_breaches
10 © Hortonworks Inc. 2011–2018. All rights reserved
What Does Apache Metron Look Like?
11 © Hortonworks Inc. 2011–2018. All rights reserved
Security telemetry source: authentication logsSecurity telemetry source: authentication logs
12 © Hortonworks Inc. 2011–2018. All rights reserved
13 © Hortonworks Inc. 2011–2018. All rights reserved
14 © Hortonworks Inc. 2011–2018. All rights reserved
15 © Hortonworks Inc. 2011–2018. All rights reserved
What is Apache Metron?
16 © Hortonworks Inc. 2011–2018. All rights reserved
Built on top on proven open source big data technology
17 © Hortonworks Inc. 2011–2018. All rights reserved
An architecture for real-time cybersecurity analytics
18 © Hortonworks Inc. 2011–2018. All rights reserved
Telemetry Data Source
19 © Hortonworks Inc. 2011–2018. All rights reserved
Telemetry Data Collectors
20 © Hortonworks Inc. 2011–2018. All rights reserved
Cyber Security Stream Processing Pipeline
21 © Hortonworks Inc. 2011–2018. All rights reserved
Profiling by time
t = 1 t = 2 t = 3 t = n
Wide range of algorithms including:à HyperLogLogPlusà Bloom filtersà T-digestsà Statistical Baseliningà Hashing functionsà Outlier detectionà GeoHashing over timeà Locality Sensitive HashingApprox. Data
SketchApprox. Data
SketchApprox. Data
SketchApprox. Data
Sketch
Combined Baseline
Statistic
22 © Hortonworks Inc. 2011–2018. All rights reserved
Cyber Security Stream Processing Pipeline
23 © Hortonworks Inc. 2011–2018. All rights reserved
Apache Metron Modules
24 © Hortonworks Inc. 2011–2018. All rights reserved
Who is Using Apache Metron (Part 1)
25 © Hortonworks Inc. 2011–2018. All rights reserved
26 © Hortonworks Inc. 2011–2018. All rights reserved
27 © Hortonworks Inc. 2011–2018. All rights reserved
28 © Hortonworks Inc. 2011–2018. All rights reserved
The Wider Apache Metron Ecosystem
29 © Hortonworks Inc. 2011–2018. All rights reserved
30 © Hortonworks Inc. 2011–2018. All rights reserved
Who is Using Apache Metron (Part 2)
31 © Hortonworks Inc. 2011–2018. All rights reserved
32 © Hortonworks Inc. 2011–2018. All rights reserved
33 © Hortonworks Inc. 2011–2018. All rights reserved
34 © Hortonworks Inc. 2011–2018. All rights reserved
35 © Hortonworks Inc. 2011–2018. All rights reserved
Deploying Apache Metron
36 © Hortonworks Inc. 2011–2018. All rights reserved
AD/AssetDB/HR/Threat
HDF
HDFS
NiFi - Ingest
HDP
Phase 0 – Current State
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets
3
1
2
4
5
37 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
SolrInvestigator UI
HDP
Phase 1 - Ingest and Archive
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
10
6
87
9
11
12
13
Banana
38 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
Solr
Enrichment Data
Investigator UI
HDP
Phase 2 – Enrich and Threat Intel
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
14
Banana / Kibana / ZoomData
39 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Zeppelin
HDFS
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
Solr
Enrichment Data
Metron Profiler
Triage
Alert
Investigator UI
HDP
Phase 3 – NiFi Data Ingestion + Analytics / UEBA Profiling
ADP Event Broker (Kafka)
ADP Smart Connectors
ADP Logger
ArcSight ESM
Security Assets AD/AssetDB/HR/Threat
Spark
Historical Analysis
Source Data (via NiFi)
15
16
Banana
40 © Hortonworks Inc. 2011–2018. All rights reserved
HDF
Source Data (via NiFi)
Zeppelin
HDFS
Spark
Historical Analysis
Model as a Service
NiFi - Ingest
Kafka MQ
Storm Parse / Enrich / GeoIP / Index
Automated Response
Solr
Enrichment Data
Netflow / PCAP /
Snort (Kafka direct)
Metron Profiler
Triage
Alert
Investigator UI
HDP
Phase 4 – ArcSight Logger Migration + New Data Sources
ADP Event Broker (Kafka)
ADP Smart
Connectors
ADP Logger
ArcSight ESM
Security Assets
Banana
AD/AssetDB/HR/Threat
17
18
19
20
41 © Hortonworks Inc. 2011–2018. All rights reserved
Considerations for Sizing Apache Metron
42 © Hortonworks Inc. 2011–2018. All rights reserved
• Events per second (average and peak)• Retention time for Hot / Warm / Cold
zones• Enrichments• Node sizing• I/O Considerations• PCAP?
Sizing an HCP deployment
43 © Hortonworks Inc. 2011–2018. All rights reserved
3 Months
Hot
Warm
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~3 months
44 © Hortonworks Inc. 2011–2018. All rights reserved
12 Months
Hot
Warm
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
45 © Hortonworks Inc. 2011–2018. All rights reserved
Hot
Warm Cold
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
Cold HDFS layer +12 months
24 Months
46 © Hortonworks Inc. 2011–2018. All rights reserved
Cold
Beyond 24 months
Hot
Warm ColdColdColdCold
Fast indexed layer (Solr / ES) ~3 months
Warm HDFS layer ~12 months
Cold HDFS layer +12 months
47 © Hortonworks Inc. 2011–2018. All rights reserved
Questions?
48 © Hortonworks Inc. 2011–2018. All rights reserved
49 © Hortonworks Inc. 2011–2018. All rights reserved
Appendix