Putting Big D ata to Work

Putting Big Data to Work

AURIMS/ANZUIAG Conference 2014

University of Adelaide 2

Who Am I

• Mathew Benwell

• Information Security Specialist at the University of Adelaide

• Worked in Information Security for 8 years


First, a Disclaimer

• I work in a highly technical field, there will be a technology slant to this talk!

• However, the concepts in this talk translate to non technical fields

• My experiences are with a specific product called Splunk


About This Presentation

• What is Big Data?

• Big Data at the University of Adelaide

• Technology Use Cases


What is Big Data

• Big Data 3 V’s

VarietyVelocity

Volume


How is Big Data Useful?

• Analyse very large data sets quickly

• Add context using variety

• Can help spot unusual events



• Analysis

– Arithmetic operations

– Trending

– Anomalous data



• Visualisations


A Simple Big Data Analytics Process

What do you want to know?

• Be precise!

What dataset holds the answer?

• If required data is not logged, start logging now!

Collect data • Multiple sources

Analyse and get the answer!

• Get answers all the time, continuously


Big Data and Audit

• Why wait for the good old 90 day review??

• Why not have our Big Data system tell use when an interesting event occurs?

• Why not take it a step further and add context

• Advise system owner at the time it occurred


Big Data and Audit

• During an Audit we ask lots of questions

The Question: – Who maintains access to privileged information?

– More specifically, we aim to identify those with unauthorised access to privileged information

Data that could support an answer:– System logs of changes to user groups

– List of groups which maintain privileged access

– Change system records


Big Data and AuditQuestion: Is Domain Admins group restricted to authorised IT personnel?

Required Data: Current Members + Active Directory event log that fires when someone is added to the Domain Admins group

Active Directory

BIG DATA SYSTEM

John Doe added to Domain Admins

Alert

Could be any question:• Monitoring changes to bank transaction file• Monitoring anomolous pay runs• Overrides in requisition request• Mismatched invoices


Big Data and Compliance

• Assist with Compliance to standards

• Payment Card Industry – Digital Security Standard (PCI-DSS)

• ISO 27001


Big Data and Compliance

• PCI-DSS

• Many technical controls

• Identify credit card data– Known pattern

– On the network

– Emails


Big Data and Risk

• We could use Big Data to identify financial risks

• Help prioritise risk treatment

• Identify unusual events– Transaction without a purchase order

– Higher than normal transaction

– High volume or scheduled, low value transactions


Big Data and Risk

• Profiling financial transactions

• Say we see a regular payment that occurs routinely

• Imagine the transaction one day starts occurring more frequently, or the transaction value changes significantly?

• This would be worth investigation







What is Splunk

• First the most asked question!

Where did the name come from?

• Derived from the word ‘Spelunk’ ‘to explore caves, especially as a hobby’

Our customers told us that finding their IT problems was like "digging through caves with headlamps and helmets, crawling through the muck"


What is Splunk

• Software that can be used to store, analyse and report on Big Data!

• Simple licence model, based on the total volume of data consumed daily

• Highly scalable. Performance is only limited by hardware resources


What Data Can Splunk Consume

• Machine data, any data generated by a computer

– System logs

– Text files

– Databases

– Output from systems


Getting Data into Splunk

• Getting data into Splunk• Syslog

• Splunk Forwarder• Tail/dump any local file• Windows registry• WMI• Script• Active Directory

• DB Connect – Oracle, MSSQL, MySql, PostGres

• API – Push data using Splunk API


Splunk at the University of Adelaide• Community driven collaboration


Splunk at the University of Adelaide• Initially purchased for the Security team to help

deal with the ‘Phishing’ problem

• Uses are expanding significantly

• Quick Statistics– 3 Primary Servers

– Total 19TB storage capacity

– 89 billion events, 30 event sources


Splunk at the University of Adelaide• Google for your data


Splunk at the University of Adelaide• More than Google for your data


Splunk at the University of Adelaide• Analysis







Use Case – Vulnerability Data

• System vulnerability data (Nessus, Nexpose, Qualys, etc)


Use Case – Vulnerability Data

• Add context, this data becomes far more Useful!– Is the system accessible from the Internet (Firewall

policies)

– Is the system actively being attacked (Intrusion Detection System data)

– Is the system actually vulnerable

• Additional information leads to a more educated assessment of impact and likelihood of occurrence


Use Case – Internet Charges

• AARNet users pay subscription costs

• Most Australian Universities control using quota systems

• Beginning 2014, the University of Adelaide removed the quota system



• Potential Financial Risk– High volume of Internet usage

– Internet usage is not cheap when you account for ~25k students!

– We have a budget to stick to

• What are we doing to control the cost?

• Big Data!!





• Constantly analysing Internet traffic

• Comparing our traffic with a list of unmetered content

• Applying technical controls to limit impact of known high cost, non University related activities




Putting Big Data to Work

• In Summary:– Big Data systems are very powerful

– Big Data principles can be applied to many needs, just ask the question

– Big Data can help find needles in many haystacks

• I hope you enjoyed my presentation!

• Thank You

Documents

Putting Big D ata to Work