Embed Size (px)
Citation preview
Copyright © 2013 Splunk Inc.
Raanan Dagan Sr. Sales Engineer -‐ Hadoop Domain Export, Splunk #splunkconf
Technical Deep Dive: Data IntegraIon Between Splunk and RelaIonal Databases
Legal NoIces During the course of this presentaIon, we may make forward-‐looking statements regarding future events or the expected performance of the company. We cauIon you that such statements reflect our current expectaIons and esImates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-‐looking statements, please review our filings with the SEC. The forward-‐looking statements made in this presentaIon are being made as of the Ime and date of its live presentaIon. If reviewed aTer its live presentaIon, this presentaIon may not contain current or accurate informaIon. We do not assume any obligaIon to update any forward-‐looking statements we may make. In addiIon, any informaIon about our roadmap outlines our general product direcIon and is subject to change at any Ime without noIce. It is for informaIonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaIon either to develop the features or funcIonality described or to include any such feature or funcIonality in a future release.
Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respecCve
owners.
©2013 Splunk Inc. All rights reserved.
2
Agenda
! Background and Overview ! DB Connect Demo ! Technical Overview ! Customer Examples and Summary
3
Background and Overview
4
What About Structured Data?
5
Customer profile
Product a1ributes
Employee details
Pricing and Rate plans
Asset info
Machine Data – Delivers Real-‐Ime Insights
6
Media server logs
(machine data)
Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for [email protected] 10.164.232.181 from 12.130.60.5 recorded OK.!2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3" 503 0 0 825 1680!Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for [email protected] 10.164.232.181 from 12.130.60.5 recorded OK.!
Phone Number IP Address Track ID
Structured Data – Contains Business Context
7
Media server logs
(machine data)
Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for [email protected] 10.164.232.181 from 12.130.60.5 recorded OK.!2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3" 503 0 0 825 1680!Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for [email protected] 10.164.232.181 from 12.130.60.5 recorded OK.!
Track ID ArIst Title Format ID Run Ime
01011207201000005652000000000053 Maroon 5 Moves like Jagger MP3 4:30
Phone # Subscriber ID
2172618992 53546
Subscriber ID
First Name Last Name Age State Customer Score
53546 Jim Morrison 25 CA 93
Customer, product databases
Phone number IP address Track ID
Enrich Machine Data with Structured Data
8
Structured databases
CSV lookup
DB Connect >10,000 downloads
Introducing Splunk DB Connect
! Enrich search results with addiIonal business context
! Easily import data into Splunk for deeper analysis
! Integrate mulIple DBs concurrently ! Simple set-‐up, non-‐evasive and secure
Reliable, scalable, real-‐Jme integraJon between Splunk and tradiJonal relaJonal databases
MicrosoT SQL server
JDBC
Database lookup
Database query
ConnecIon pooling
Other databases
Oracle database
Java Bridge Server
9
Splunk DB Connect Demo
10
Splunk DB Connect Technical Overview
Splunk DB Connect: Main Features
12
! Database connecIon management ! SQL database lookups ! Splunk search language extensions – Database query – Database info – Database output
! SQL database input ! Access control
Installing Splunk DB Connect
13
! Simple app setup, no configuraIon files to touch ! AutomaIcally checks for the required Java version
Database ConnecIon Management Configure new database connecIon sefngs in minutes
from the Splunk user interface
14
MicrosoT SQL server
JDBC
Database lookup
Database query
ConnecIon pooling
Other databases
Oracle database
Java Bridge Server
Works With Many Databases
15
! Supports mainstream databases – Oracle database – MicrosoT SQL server – MySQL – PostgreSQl – Sybase – DB2 – Generic JDBC support
! Database connecIon pooling limits load on database
Database Lookups
16
Enrich machine data by adding structured data from tradiIonal relaIonal databases
Three Steps to Enriching Machine Data
17
1. Connect
2. Configure
3. Enrich
Splunk Search Language Extensions
18
Execute database queries directly from the Splunk user interface with new Dbquery, Dbinfo, and DBoutput Splunk search commands
Explore Database Structure
19
! Wrapping dbinfo and dbquery
Browse and navigate database schemas and tables from the Splunk DB Connect user interface
Import and Index Database Data Combine machine data with structured data from relaIonal databases
20
New dbmon-‐tail and dbmon-‐dump input types can be used to import rows from the database
Access Control Database ConnecIon
21
Access Control 1. Splunk administrator can set
users / roles with permissions for the connecIon (block, read, write)
2. Database connecIon can be set to a ‘Read-‐Only’
3. DBA can set permission on the database side
Technical Summary
• Quick to set-‐up, scales to mulIple concurrent databases • Enrich machine data with database data in three easy steps • Execute SQL queries to visualize database data directly in the
Splunk user interface • Import and index database data for historical analysis and
correlaIon with machine data
22
Success Stories
Enabling ExcepIonal Customer Service
24
Users to customers mapping
SQL SQL
User acJvity
= Customer details, external/internal details
Database
+
Machine Data
= User acIvity data from SaaS applicaIon, websites
SaaS
Real-‐Jme visibility of customer experience
Website
Driving ProacIve Network Management
25
Network Switch Data
Splunk DB Connect imports millions of records per day: ! Visualize graph of outliers ! Detect high uIlizaIon paqerns
Import and index data
• CPU uIlizaIon • Inbound packets • Network staIsIcs • Data for 10,000+
switches
Problem management dashboards
Summary
• Machine data contains a categorical record of acIvity and behavior
• Enrich with structured data to provide business context – for beqer IT, security and business insights
• Splunk DB Connect delivers reliable, scalable, real-‐Ime integraIon between Splunk and tradiIonal relaIonal databases
26
Next Steps
27
1
2
3
Download the .conf2013 Mobile App If not iPhone, iPad or Android, use the Web App
Take the survey & WIN A PASS FOR .CONF2014… Or one of these bags!
Go to the Splunk DB Connect demo staJon on level 3
QuesJons Raanan Dagan [email protected]
THANK YOU
