Solving Cyber at Scalewith Hadoop, Storm and Metron

Simon Elliston Ball

• Product Manager

• Data Scientist

• Elephant herder

• @sireb

Threat Sources

IoT: Mirai

Reports of 1.2 Tbps

500,000 devices at peak

DDoS attacks on Dyn DNS services

Insiders

Ransomware and spears

Who are we up against?

MEECES

Money

Ego

Entertainment

Cause

Entrance (social acceptance)

Status

Big Business

• $tn market

• Access is bought and sold: 5 bitcoin for 100m accounts

• Sharing networks

• Criminals as a Service

• DDoS attacks: cost attackers $5 per hour, defenders ~$40k

Sources: BT and KPMG Report, Taking the Offensive

Challenges for the Modern SOC

Drowning in Data

Staff shortage

Long tail problem

What we have now

Silos

Packet

Store

SIEM

Log

StoreForensics Tools

Endpoint Agents

Cases

Threat Intel

UEBA

Anti Virus

Email filter

Rules: Asset or

Liability

Shiny

new tools

Solutions: machine learning! magic!

Triage Automation

Detecting the unknown unknowns

Explaining yourself

The value of real time

Data in Motion: why wait until it’s at rest?

Correct context: the world moved on

Better data = analyst efficiency

Fully enriched data

Real context

Consistency

= faster triage and better coverage

Single View of Business & Security

Risks

HR

Finance

Web Logs

Security Appliances

Email

Syslogs

Geolocation

Network Data

IoT

Telemetry Data

Operations

CRM

Longer term data

• Attacks last months

• So should your queryable data

Executable solutions

• Orchestration

• Machine-time response

How to do it

Network Level Taps

Data Sources and Aggregation

Open standards for data models = more productive data scientists + shareable models

Business level data sources link security to real business risk.

Massively scalable platforms

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Data Se

rvices an

d In

tegratio

n Laye

r

Search andDashboarding

Portal

Security Data Vault

CommunityAnalytical

Models

Provisioning,Management

and Monitoring

ModulesReal-time ProcessingCyber Security Engine

TelemetryParsers Enrichment

ThreatIntel

AlertTriage

Indexersand

Writers

Cyber SecurityStream Processing Pipeline

Apache Metron: a framework for Big Data Driven cyber security

Tele

metry In

gest B

uffe

r

TelemetryData Collectors

Real-timeEnrich / ThreatIntel Streams

PerformanceNetwork

IngestProbes

/ OtherMachine Generated Logs(AD, App / Web Server,

firewall, VPN, etc.)

Security Endpoint Devices (Fireye, Palo Alto,

BlueCoat, etc.)

Network Data(PCAP, Netflow, Bro, etc.)

IDS(Suricata, Snort, etc.)

Threat Intelligence Feeds(Soltra, OpenTaxi,third-party feeds)

TelemetryData Sources

Community Development

• http://metron.apache.org

• https://github.com/apache/incubator-metron/

Thank you!

• Apache Metron: http://metron.apache.org

• Twitter: @sireb