All Things Open

Agenda• Intro, who I am.• Cybersecurity • ONI now Apache Spot (incubating)• Apache Spot (incubating)• Demo• Call to Action.• Q&A

Cybersecurity• We have gaps… The analysis of billions of events , orchestrate our

data sources (logs in different forms), and sometimes the documentation of our security products is not the best.

The hacker community collaborates everyday, it’s time we do the same.

Services Products Training

Free$100Learn to Crack

WifiHack a Corporate

Email AccountAngler

Exploit Kits

$500

AssetInventory

ConfigurationGuidanceAnalysis

VulnerabilityAnalysis

ThreatAnalysis

IntrusionDetection

VulnerabilityAlert

ConfigurationGuidance

AssetDefinition

ThreatAlert

IncidentReport

IncidentManagement

Operational Enterprise Networks

Centralized ReportingEnterprise ITChange Management

Development & SustainmentSecurity ManagementProcesses

Assessment of System

Development,Integration, &Sustainment

Activitiesand

Certification &Accreditation

System &Software

AssuranceGuidance/

Requirements

Operations Security Management Processes

Enterprise IT Asset Management

CCE/CCSS/OVAL/ARF/XCCDF/CPE

CVE/CWE/CVSS/ARF/CCE/CCSS/ARF/CWSS/OVAL/CPE/XCCDF

CVE/CWE/CVSS/ARF/CCE/CCSS/OVAL/CWSS/XCCDF/CPE/CAPEC/MAEC

CVE/CWE/CVSS/ARF/.CCE/OVAL/CCSS/XCCDF/CPE/CAPEC/CWSS/MAEC/CEE

CPE/OVAL/ARF

CWE/CAPEC/CWSS/MAEC/OVAL/OCIL/XCCDF/CCE/CPE/ARF/SAFES/SACM

CWE/CAPEC/SBVR/CWSS/MAEC

OVAL/XCCDF/CCE/CCSS/CPE/ARF

CPE/OVAL XCCDF/OVAL/CCE/CCSS

CVE/CWE/OVAL/CVSS/CWSS

CAIF/IDMEF/IODEF/CVE/CWE/OVAL/CPE/MAEC/CCSS/CWSS/CEE/ARF

CVE/CWE/CVSS/CPE/CWSS/CAPEC/MAEC

SCAP

SwAAPCVE/CWE/CVSS/CCE/CCSS/OVAL/XCCDF/CPE/CAPEC/MAEC/CWSS/CEE/ARF

CVE/CWE/CVSS/CCE/CCSS/OVAL/XCCDF/CPE/CAPEC/MAEC/CWSS/CEE/ARF

TrustManagement

IdentityManagement

EMAP

ERAP ECAP

ESIP ITAPTAAP

ONI -> Apache Spot (incubating)• Apache spot (incubating) is an advanced analytic solution that will help us to

close the gaps that we are mention on the previous slides.

• Ingesting billions of records in HDFS and execute machine learning algorithms, to detect potential threats in our environment.

Apache Spot (incubating) Core

ON

I Dat

a So

urce

s

DNSInfrastructure Logs

ProxyInfrastructureLogs

Routers with Netflow Protocol Enabled onInterfaces

New Data Source

New DataSource

ONIVisualization

Server / iPython Server

ON

I GU

I

TLS Https 443

Security and Context Use

Cases Develop in Conjunction with

Intel Security

Assumes Cloudera Hadoop Environment

Data Integration

Data Store Machine Learning

CollectorsOnlineNoSQL(HBase)

Filesystem(HDFS)

Spot MLAlgorithms

Spark

Master Node (s) ClouderaManager/Navigator

Machine learningAlgoritms Output,ONI Recommendedthe Intel MPILibraries. Scala

Native AdministrationCloudera Manager

Cluster AuthenticationLDAP/KerberosAuthentication

Machine learningGenerates CSV Files

with the Results

Operational Analytics AddingContext Using Reputation Services

for Public IP Address (GTI)

Defining the Interface to Share

the Suspicious Connections with I-

Sec Products and Other Brands.

Product Architectural Overview

Apache Spot delivers…1. Scalable Data and Analytics Platform 2. Open Data Models3. Analytic Collaboration Across the Community4. Growing Application Ecosystem

… to address cybersecurity use cases.• Network Traffic Analytics• Threat Hunting• Incident Detection and Resolution• Cybersecurity Data Management• Custom Use Case

Platf

orm

Apache Spot, bringing all of the components together.

Data

Man

agem

ent

Apache Spot Sample Data Sets

Apache Spot Open Data Models (ODM)

Data Platform (CDH)

Ingestion (Kafka, Flume, Streamsets)

Anal

ytics Apache Spot OSS Analytics

Analytic Services (Jupyter, Apache Spark)

Apps Apache Spot ODM Marketplace

Infra Intel Hardware, On-Prem, AWS, Azure M

anag

emen

t, Se

curit

y, Go

vern

ance

(Dire

ctor

, Man

ager

, Sen

try,

Nav

igat

or)

Public or private clouds

Scalable storage and distributed processing

Provisioning, management, and security

Batch and stream data ingestion

Logical and physical models

Data Science workbench

Network traffic analytics, Add’l OSS analytics

ODM Compliant ecosystem, both open source and ISV

Community sourced, anonymized data sets for model development

Demo

Call to Action.Contribute for the Apache Spot (incubating) project.1. Develop connectors to ingest more data2. Develop new algorithms that help us to increase the detection rate of the tool3. Contribute to add Context to this results, adding threat intelligence feeds

connector to databases that will help us to present meaningful information to the end user.

4. Develop the User Interface, propose changes, technologies, operational summaries, reports, etc.

Call to Action.5. Integrate Apache Spot (incubating) with other security tools, that have the

capabilities to enforce / change security postures. (Firewall consoles, IPS consoles, Proxies, Endpoint Security Solutions, E-mail proxies)

6. Contac us • Web page: http://spot.apache.org/• slack: slack.apache-spot.io/ • twitter @ApacheSpot

7. Contribute to the Apache Spot (incubating) project.

With Apache Spot, you are joining a community.

Collaborate with industry leaders using a common framework.

Apache Spot(Incubating)

Join the community that is fighting cyber threats today.

spot.incubator.apache.org