Splunk Conf 2014 - Getting the message

Damien DallimoreDev Evangelist , CSO Office @ Splunk

Getting the Message

Nimish DoshiPrincipal Systems Engineer @ Splunk

DisclaimerDuring the course of this presentation, we may make forward looking statements regarding future events or the

expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important

factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other

commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

Agenda

Damien’s Section

What is messagingJMS + DemoAMQP + DemoKafka + DemoCustom message handlingArchitecting for scale

Nimish’s Section

Using ZeroMQUsing JMS for underutilized computersQuestion time

Damien’s Section

From Middle Earth

Make Splunk Apps & Add-ons

Messaging background

apps.splunk.com

github.com/damiendallimore

What is messaging ?Messaging infrastructures facilitate the sending/receiving of messages between distributed systems

Message can be encoded in one of many available protocols

A common paradigm involves producers and consumers exchanging via topics or queues

Topics (publish subscribe)

Queues (point to point)

Why are messaging architectures used ?

Integrating Legacy Systems

Integrating Heterogeneous Systems

Distributed Applications

Cluster Communication

High Performance Streaming

There’s a lot of information in the pipes

The data opportunity

Easily tap into a massive source of valuable inflight data flowing around the veins

Don’t need to access the application directly ,pull data off the messaging bus

I can not think of a single industry vertical that does not use messaging

Getting this data into Splunk

Many different messaging platforms and protocols

JMS (Java Message Service)

AMQP (Advanced Message Queueing Protocol)

Nimish will cover some more uses cases also

Not a messaging protocol , but a programming interface to many different underlying message providers

WebsphereMQ , Tibco EMS , ActiveMQ , HornetQ , SonicMQ etc…

Very prevalent in the enterprise software landscape

RabbitMQ

Supports AMQP 0.9.1, 0.9, 0.8

Common in financial services and environments that need high performance and low latency

Cluster centric design = strong durability and fault tolerance

Scales elastically

Producers and Consumers communicate via topics in a Kafka node cluster

Very popular with open source big data / streaming analytics solutions

Custom message handling

These Modular Inputs can be used in a multitude of scenarios

Message bodies can be anything : JSON, XML, CSV, Unstructured text, Binary

Need to give the end user the ability to customize message processing

So you can plugin your own custom handlers

Need to write code , but it is really easy , and there are examples on GitHub

I’m a big data pre processing fan

Cut the code

Compile, bundle into jar file, copy to Splunk

Declaratively apply it

Let’s see if it works

Achieving desired scale

AMQP Queue

AMQP Mod Input

Single Splunk Instance

With 1 Modular Input instance , only so much performance / throughput can be achieved

You’ll hit limits with JVM heap , CPU , OS STDIN/STDOUT Buffer , Splunk indexing pipeline

So go Horizontal

AMQP Queue

Universal Forwarders

Splunk Indexer Cluster

AMQP Broker

AMQP Mod Input AMQP Mod Input

Nimish’s Section

About Me

• Principal Systems Engineer at Splunk in the NorthEast• Session Speaker at all past Splunk .conf user conferences• Catch me on the Splunk Blogs

Problem with Getting Business Data from JMS

The goal is to index the business message contents into SplunkMessage Uncertainty Principal:If you de-queue the message to look at it, you have affected the TXNIf you use various browse APIs for content, you may miss it– Message may have already been consumed by TXN

Suggestion: Use a parallel queue to log the message– Suggestion: Try ZeroMQ

Why use ZeroMQ

Light WeightMultiple Client language support (Python, C++, Java, etc)Multiple design patterns (Pub/Sub, Pipeline, Request/Reply, etc)Open Source with community support

Application Queue and ZeroMQ Example

Auto Load Balance

Example Python Sender

context = zmq.Context()socket = context.socket(zmq.PUSH)socket.connect('tcp://127.0.0.1:5000')sleeptime=0.5

while True: num=random.randint(50,100) now = str(datetime.datetime.now()) sleep(sleeptime) payload = now + " Temperature=" + str(num) socket.send(payload)

Python Receiver (Scripted Input)

context = zmq.Context()socket = context.socket(zmq.PULL)# Change address and port to match your environmentsocket.bind("tcp://127.0.0.1:5000")

while True: msg = socket.recv() print "%s" % msgexcept: print "exception"

Python Subscriber (Scripted Input)

context = zmq.Context()socket = context.socket(zmq.SUB)

socket.connect ("tcp://localhost:5556")

# Subscribe to directionfilter = "east"socket.setsockopt(zmq.SUBSCRIBE, filter)

while True: string = socket.recv() print string

Parallel Pipeline Example

Getting Events out of SplunkSplunk SDK

Use Cases:– In Depth processing of Splunk events in a queued manner– Use as pivot point to drop off events into a Complex Event Processor– Batch Processing of Splunk events outside of Splunk

Divide and Conquer Approach as seen in last slide

Java Example using SDK to load ZeroMQString query=search;Job job = service.getJobs().create(query, queryArgs);while (!job.isDone()) {

Thread.sleep(100);job.refresh();

}// Get Query Results and store in String str… (Code Omitted)// Assuming single line events StringTokenizer st = new StringTokenizer(str, "\n");while(st.hasMoreTokens()) {

String temp= st.nextToken();sock.send(temp.getBytes(), 0);byte response[] = sock.recv(0);

Idle Computers at a Corporation

Idea: Use Ideas from SETI @ Home

Idle Computers Put to Work Using JMS

Applications for Distributing Work

Application Server would free up computing resourcesWork could be pushed to underutilized computersExamples:– Massive Mortgage Calculation Scenarios– Linear Optimization Problems– Matrix Multiplication– Compute all possible paths for combinatorics

Architecture

Optional

Algorithm

Application servers push requests to queues, which may include data in the request object called a Unit of WorkJMS client implements doWork() interface to work with dataMessage Driven Bean receives finished work and implements doStore() interfaceWhat does this have to do with Splunk?– Time Series results can be stored in Splunk for further or historical analytics

Matrix Example High Level Architecture

Search Language Against Matrix ResultList Column Values of Each Stored Multiplied Matrix using Multikv

Screenshot here

Search Language Against Matrix ResultVisualize the Average for Columns 2 to 5

Screenshot here

Search Language Against Matrix ResultPerform arbitrary math on aggregate columns

Screenshot here

Reference

ZeroMQ– http://apps.splunk.com/app/1000/– Blog: http://blogs.splunk.com/2012/06/08/zeromq-as-a-splunk-input/

Using JMS for Underutilized Computers– Github Reference: https://github.com/nimishdoshi/JMSClientApp/– Blog: http://blogs.splunk.com/2014/04/11/splunk-as-a-recipient-on-the-jms-grid/– Article:http:

//www.oracle.com/technetwork/articles/entarch/jms-distributed-work-082249.html

Questions ?

THANK YOUddallimore@splunk.comndoshi@splunk.com

Splunk Conf 2014 - Getting the message

Software

Splunk App for StreamMany Solutions, One Goal. Some history •Splunk acquires Cloudmeter, December 2013 •Renamed Splunk App for Stream •Released with Splunk 6.0 (August, 2014)

FireEye Splunk InterFireEye + Splunk: Intermediate Guidemediate Guide

Splunk conf2014 - Using Selenium and Splunk for Transaction Monitoring Insight

Splunk For IT Operational Intelligence · • Splunk Apps –Splunk App for VMware –Splunk Apps for Citrix & Hyper V –Splunk App for Microsoft Exchange –Splunk App for Active

Monitoring Docker Containers with Splunk - Splunk .conf · Splunk and Docker – At A Glance Visibility in your Container Environments Delivering Splunk as Containers Monitoring for

Fortscale Splunk Integrationinfo.fortscale.com/hubfs/UEBA Content/Fortscale Splunk Integration.pdf · Fortscale Splunk Integration ... Fortscale digests access and authentication

Tenable and Splunk Integration · Splunk Splunkreceivesvulnerabilitydatacollectedby SecurityCenter Nessus NessusHost Scans,Nes-susPlugins Splunk SplunkreceivesvulnerabilitydatacollectedbyNes-

Splunk Validated Architectures · Splunk Engineers Responsible for managing the Splunk lifecycle. Managed Service Providers Entities that deploy and run Splunk as a service for customers

Splunk Review - University of Birmingham · Ref: Splunk Review, Dec 2014 Mohammad Rameez Shafsad itinnovation@contacts.bham.ac.uk Indexing 2.2.1 Searching using splunk Splunk uses

Security Investigation and Rapid Response Using Splunk and ... · Enter Splunk Splunk Enterprise and Splunk Cloud provide an analytics-driven approach to security on the AWS Cloud

Referent / Redner Benjamin Tiggemann · 2015-07-08 · Splunk for Exchange Splunk for Enterprise Security* Splunk for Vmware Splunk for PCI Compliance* Splunk for Windows Splunk for

Our experiences throughout the 3 year journey · Splunk – Hardware is now available for Splunk expansion • Splunk begins to ﬁll monitoring gaps, acts as “glue” • Splunk

Netﬁlter Iptables for Splunk Documentation - Read … · Netﬁlter Iptables for Splunk Documentation, Release 0 Splunk Answers Splunk has a strong community of active users and

Agency Chargeback Models to Enable Enterprise Splunk ......Workshops: Get Splunk Hands -on Experience Attend a Splunk Workshop Upcoming Schedule December 1: Introduction to Splunk

© 2019 SPLUNK INC. and ELK Let's Chat About Splunk

TimeACerTime*–** ComparingTimeRangesinSplunk* · AboutMe! SplunkSeniorInstructorsince2009! FrequentcontributortoSplunkAnswers! LoveSplunksearchlanguagepuzzles*

stoQ’ing your Splunk - SANS · PDF filestoQ’ingyour Splunk Ryan Kovar, Splunk Marcus LaFerrera, PUNCH SANS DFIR 2016

Splunk Conf 2014 - Splunking the Java Virtual Machine

SplunkingYour Mobile Apps - .conf19 | Splunk · Introducing Splunk for Mobile Intelligence (Splunk MINT) Deploying Splunk MINT – Intro to SDKs – Getting Started Using Splunk MINT

ReversingLabs Explainable Threat Intelligence Enriches ... Data Sheets/RL-Splunk...sent to Splunk. • In Splunk, the TitaniumScale report is correlated with other available Splunk