Security and Audit for Big Data

© 2013 IBM Corporation

IBM Security Systems

1


Security and Audit for Big Data

Tina ChenGuardium Enablement

[email protected]



2

Please note

� IBM’s statements regarding its plans, directions, and intent are subject to change or

withdrawal without notice at IBM’s sole discretion.

� Information regarding potential future products is intended to outline our general product

direction and it should not be relied on in making a purchasing decision.

� The information mentioned regarding potential future products is not a commitment,

promise, or legal obligation to deliver any material, code or functionality. Information about

potential future products may not be incorporated into any contract. The development,

release, and timing of any future features or functionality described for our products remains

at our sole discretion.



3

Bring your own IT

Social business

Cloud and virtualization

1 billion mobile workers

1 trillion connected

objects

Innovative technology changes everything



4

Compromises take weeks and months to discover & remediate

http://www.verizonbusiness.com/resources/reports/rp_data-breach-investigations-report-2012_en_xg.pdf?CMP=DMC-SMB_Z_ZZ_ZZ_Z_TV_N_Z038

Time span of events by percent of breaches



5

Four key drivers for data security

CLOUD ADOPTIONEnterprise Security #1 Inhibitor1

APTs DATA BREACHES98% Stolen RecordsFrom Large Orgs2

From Vormetric1. Global State of Information Security® Survey by PwC, CIO magazine, and CSO magazine – October 2012 2. Verizon Data Breach Investigation Report – March 2012

BIG DATABig Data is a Big Target

GLOBALCOMPLIANCEAggressive NewRegulations

� Compliance– PCI-DSS, HIPAA/HITECH, SOX, GLBA, Basel III and others

– National Data Encryption Laws: UK Data Protection Act, EU Data Protection Directive, South Korean PIPA and others

– Require encryption, separation of duties, privileged user controls

� Data Breaches – Valuable data is being targeted by sophisticated attackers - data

breaches increasingly result

– IP Protection, US Federal and State Data Protection laws, Data across Borders

– Encryption + access controls limit risk, meet safe harbor requirements

� Cloud Adoption – Security the #1 concern– Cloud efficiency and flexibility highly desired

– Encryption + access control limits exposure to cloud adminsand other new security risks

� Big Data = Big Risks– Large data sets inevitably include sensitive data

– All data stores and report locations require protection

“To mitigate business risk, you must proactively protect what matters — customer data, financial data and intellectual property — from both outside attackers and privileged insiders.”



6

Structured

Unstructured

Streaming

Big Data Platform

Hadoop Cluster

Clients

The importance of monitoring Can you answer these questions?

- Who is running map reduce jobs and what are those jobs accessing?

- Is there a new job in the system that hasn’t been vetted?

- Is someone possibly trying to hack into the file system?



7



8

Complement existing security with secure databases



9

InfoSphere Guardium protects sensitive data in Hadoop environments and helps ensure compliance

o Protect your sensitive data with real time activity monitoring

o Gain insights into data activity throughout the stack: Hive, MapReduce,

HBase and HDFS

o Detect unauthorized applications or users

o Real time alerts reduce time to discovery for possible breach or infraction of

compliance

o Automate compliance and management tasks

o Infrastructure in place to provide additional real-time controls over time

Introducing HadoopActivity Monitoring

Application

Storage

MapReduce

Oozie

HDFS

HBase

HiveApplication

Storage

MapReduce

HDFS

HBase

Hive

Monitor and Audit



10

How it’s done

InfoSphere Guardium monitors key Hadoop events:•Session and user information

•HDFS operations – Commands (cat, tail, chmod, chown, expunge, etc), files, permissions

•MapReduce jobs - Job, operations, permissions

•Exceptions, such as authorization and access control failures

•Hive/HBase queries , - Alter, count, create, drop, get, put, list,..

Heavy lifting occurs on Guardium collector! Very low

overhead on monitored nodes. Architecture supports separation of duties

Hadoop ClusterClients

InfoSphereGuardiumCollector Appliance

S-TAPs

InfoSphere Guardium Reports

Sensitive data alert!



11

How it is done



1212

Capture and Parsing Overview

HadoopClient

GuardiumCollector

Analysis engine

Hadoop fs –mkdir /user/data/sundari

Hadoop fs –mkdir ….

Sessions

Commands

Objects

Read Only Hardened Repository

(no direct access)

Hadoop commands

mkdirs

Joe /user/data/sundari

Parse commands

then log

Joe

Namenode

S-TAP

Hadoop fs –mkdir …

Hadoop fs –mkdir/user/data/sundari



13

A recommended approach

1. Identify users and classes of users – “privileged” users, data scientists…Who is allowed to access sensitive data

� Validate with activity monitoring

2. Identify the applications, jobs, ad-hoc analysis

� Validate with activity monitoring

3. When possible identify and mask sensitive data before it enters the cluster and identify specific directory location in cluster for that data. Put tighter monitoring controls around that data.

4. Look at exceptions – permission exceptions, other operational errors



14

Use cases

Let’s do the following:

• Log and/or alert access to sensitive files by “unauthorized or unknown” user

• Reporting on new jobs entering the system (identify new MapReduce jobs in the

system)

• Exception reporting for permission errors on sensitive data

Plan

Monitor

Automate

And for each scenario, how to:



15

Planning for sensitive data access and monitoring

o Do you have PCI or other sensitive data? Is sensitive data already identified in “source” systems?

o How do you carry that sensitive identification over to Hadoop?

o What are the internal and external compliance requirements for monitoring sensitive data access?

o What is the plan for handling violations? Who need to be alerted and when?

PlanPlan



16

Where is sensitive data?

Configuration

files

Sensitive data or

unknown

PlanKeep sensitive data localized, encrypted, and

under monitoring control

Non-Sensitive

data or known



17

Monitoring sensitive data

Your policy rules go here, such as sensitive data alerting…

Real-time security policies

Default Hadoop Policy

Flexible, granular rules….

MonitorMonitor



18

Determine who is accessing sensitive data

When?From where?

Who?

MonitorMonitor

What?



19

Alerts reduce time to discovery

Incorporate Data Events

into QRadar unified view and real time analytics

AutomateAutomate

Unauthorized access to sensitive data!



20

Planning for application access

What MapReduce jobs are being used? What kind of ad hoc analysis is allowed on the system?

Have they been vetted for access to sensitive data?

What is a normal pattern of activity?

What process should the team use to communicate new deployments?

No human communication process is infallible…

Task Map(break task into

small parts)

Reduce(many results to a

single result set)

TaskTask Map(break task into

small parts)

Map(break task into

small parts)


single result set)


single result set)

Plan



21

What applications are using the data?

Now, reduce the noise by filtering out authorized jobs….

MonitorMonitor

MapReduce reports ….



22


Focus your resources on the unknown –

unauthorized MapReduce jobs

MonitorMonitor



23


Audit process workflow and administrative automation AutomateAutomate

Should this job be approved?



24


Audit process workflow and administrative automation

Business Owner approves or rejects new applications/jobs

AutomateAutomate



25




Information Security confirms Business Owner recommendation

AutomateAutomate



26




Information Security confirms Business Owner recommendation

Guardium Admin adds authorized jobs to “authorized job list”

AutomateAutomate



27


Populate new vetted applications automatically AutomateAutomate



28





29





30

Alerting off the Exception reports

IO exception

MonitorMonitor AutomateAutomate

Table already exists



31

• PCI, SOX and HIPAA accelerators included with DAM (guidance, reports, and more)

Continuous database activity monitoring



32

PCI – Data Access Report



33

Streamline and simplify compliance processes for Hadoop

Proven track record in data security

Complete separation of duties

Sensitive data monitoring to pass compliance audits

Privileged user monitoring

Real-time alerting for abnormal/suspicious activity

Full forensics: Any activity – views, changes, updates…

Heterogeneous support – IBM, HortonWorks, Cloudera,

Greenplum…

Same platform for all databases in your enterprise

Proven track record in data security

Complete separation of duties

Sensitive data monitoring to pass compliance audits

Privileged user monitoring

Real-time alerting for abnormal/suspicious activity

Full forensics: Any activity – views, changes, updates…

Heterogeneous support – IBM, HortonWorks, Cloudera,

Greenplum…

Same platform for all databases in your enterprise

InfoSphere Guardium Top BenefitsInfoSphereInfoSphere GuardiumGuardium Top BenefitsTop Benefits



34



35

IBM Securing all types of data…

� Data privacy for non-production environments:

�Optim Data Privacy (DP)

�Optim Test Data Management (TDM)

� Securing static data on repository :

�Guardium Data Encryption

Stored(Databases, File Servers, Big Data, Data

Warehouses, Application Servers, Cloud/Virtual ..)

Over Network(SQL, HTTP, SSH, FTP, email,. …)

Data in MotionData in Motion

Repository Vulnerability(Database Configuration, Patch Level,

OS Security, …)

ConfigurationConfiguration

� Data privacy for unstructured data (documents)– Guardium Data Redaction

� Data privacy for production environments:– Guardium Data Activity Monitoring

� Ensuring database is configured and patched properly

– Guardium Vulnerability Assessment

– QRadar/QVM

Data at Rest



36

Data Masking for Data PrivacyMask confidential data to avoid data breach & meet privacy compliance

Database

InfoSphere Optim

JASON MICHAELSJASON MICHAELS ROBERT SMITHROBERT SMITH

Mask

Before Masking After Masking

• Protect sensitive information from misuse and fraud

• Prevent data breaches and associated fines

• Achieve better information governance

• Protect confidential data while preserving analytics

• Mask data anytime, anywhere

• Mask data in Hadoopusing MapReduce

• Implement proven built-in masking algorithms

• Support compliance with privacy regulations

Requirements

Benefits

Mask in-databaseMask in-database Mask in-Hadoopusing MapReduce

Mask in-Hadoopusing MapReduce

Extract, mask & loadExtract, mask & load

IMS

VSAM

More…

Mask filesMask files

Hadoop



37

Optim Data Masking implementation in Hadoop

JASON MICHAELSJASON MICHAELS ROBERT SMITHROBERT SMITH

OPTIM

Before Masking After Masking

� Optim Masking can also be executed in Hadoop for delimited files.

� Java application/interface for masking.

• MapReduce base classes and helpers.

• Distributed cache.

• Shared libraries.

• Use of masking in Reducers.

� Declarative specification of:

• Metadata of data files

• Masking rules



38Products Services

Intelligence: A comprehensive portfolio of products and services

New in 2012



39

All domains feed Security Intelligence

Endpoint Management vulnerabilities enrich QRadar’s

vulnerability database

AppScan Enterprise

AppScan vulnerability results feed QRadar SIEM for improved

asset risk assessment

Tivoli Endpoint Manager

Guardium Identity and Access Management

IBM Security NetworkIntrusion Prevention System

Flow data into QRadar turns NIPS

devices into activity sensors

Identity context for all security domains w/ QRadar as the dashboard

Database assets, rule logic and database activity information

Correlate new threats based on X-Force IP reputation feeds

Hundreds of 3rd party information sources



40

Key Business Drivers for InfoSphere GuardiumContinuously Monitor All Access too..

Prevent data breaches

Assure data governance

Reduce cost of compliance



41

Extend real-time Data Activity Monitoring to also protect sensitive data in data warehouses, Big Data Environments and file shares

Integration with

LDAP, IAM,

SIEM, TSM,

Remedy, …

Big Data Environments

DATA

InfoSphereBigInsights

CouchDB

GreenPlum



42

Information and community

� InfoSphere Guardium YouTube Channel – includes overviews and technical demos

�developerWorks forum (very active)

�Guardium DAM User Group on Linked-In (very active)

�Community on developerWorks (includes content and links to a myriad of sources, articles, etc)

New! InfoSphere Guardium Virtual User Group. Open, technical discussions with other users.

Send a note to [email protected] if interested.

42



43

ibm.com/guardium

© Copyright IBM Corporation 2012. All rights reserved. The information contained in these materials is provided for informational purposes

only, and is provided AS IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use

of, or otherwise related to, these materials. Nothing contained in these materials is intended to, nor shall have the effect of, creating any

warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement

governing the use of IBM software. References in these materials to IBM products, programs, or services do not imply that they will be available in

all countries in which IBM operates. Product release dates and/or capabilities referenced in these materials may change at any time at IBM’s sole

discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any

way. IBM, the IBM logo, and other IBM products and services are trademarks of the International Business Machines Corporation, in the United

States, other countries or both. Other company, product, or service names may be trademarks or service marks of others.

Statement of Good Security Practices: IT system security involves protecting systems and information through prevention, detection and response

to improper access from within and outside your enterprise. Improper access can result in information being altered, destroyed or misappropriated

or can result in damage to or misuse of your systems, including to attack others. No IT system or product should be considered completely secure

and no single product or security measure can be completely effective in preventing improper access. IBM systems and products are designed to

be part of a comprehensive security approach, which will necessarily involve additional operational procedures, and may require other systems,

products or services to be most effective. IBM DOES NOT WARRANT THAT SYSTEMS AND PRODUCTS ARE IMMUNE FROM THE

MALICIOUS OR ILLEGAL CONDUCT OF ANY PARTY.



44

Disclaimer

Please Note:

IBM’s statements regarding its plans, directions, and intent are subject to change or

withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product

direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise,

or legal obligation to deliver any material, code or functionality. Information about potential

future products may not be incorporated into any contract. The development, release, and

timing of any future features or functionality described for our products remains at our sole

discretion.

Technology

Security and Audit for Big Data