SPLUNK CONF 2013 DEPLOYMENT SERVER IN THE REAL WORLD€¦ · financial, Amazon.com (14 dog years),...

Craig Nelson Sr. System Admin, F5 Networks #splunkconf

SPLUNK CONF 2013 DEPLOYMENT SERVER IN THE REAL WORLD

#splunkconf

2 © F5 Networks, Inc.

INTRODUCTION

3 © F5 Networks, Inc.

About Me

•  Craig Nelson, “No, I’m not the actor”

•  25+ years in IT including healthcare, start-ups, ISP’s, large financial, Amazon.com (14 dog years), and now F5 Networks

•  Operations and infrastructure focused, primarily web and security

•  Architect and support F5’s customer facing applications

•  Architect and deploy internal solutions for operations (like Splunk)

•  SME for load balancing and web app security within IT

4 © F5 Networks, Inc.

About F5 Networks •  F5 manufactures and supports solutions in the Application Delivery

Controller (ADC) space

•  ADC is not just load balancing

•  Security: network & application firewalls (AFM/ASM)

•  Performance: load balancing and web acceleration (LTM/AAM)

•  DNS: load balancing, caching, forwarding (GTM)

•  Access technologies: SSL VPN (APM)

•  Service provider: traffic flow & control (PEM/CGNAT)

•  www.f5.com/products

5 © F5 Networks, Inc.

Agenda

•  Deployment Goals

•  Guiding Principles

•  Deployment Approach

•  Infrastructure

•  Deployment and Licensing Server

•  Heavy Forwarders

•  Indexers

•  Search Heads

6 © F5 Networks, Inc.

Project Outline

7 © F5 Networks, Inc.

Deployment Goals & Guiding Principles

•  Enterprise security installation

•  Operational IQ

•  Simple and supportable deployment

•  Prevent configuration drift

•  Stop rogue changes

•  Support easy rebuild and lower environment duplication

•  Use KISS principle at all levels

8 © F5 Networks, Inc.

Deployment Approach

•  Proof of concept stack built in a separate environment to prove out the approach and learn how the deployment server works prior to onsite engagement

•  Onsite Splunk professional services came for 3 weeks

•  Pick the sources in priority order and tune/filter/type them

•  Customize ES after the sources are flowing correctly

9 © F5 Networks, Inc.

Infrastructure

10 © F5 Networks, Inc.

Infrastructure

•  2 physical indexers

•  2 search heads, 1 physical for ES and 1 virtual for misc

•  1 license server (70GB license)

•  Main physical heavy forwarder

•  Heavy forwarder in each DMZ

•  Every “heavy” server gets the same base configuration •  Listeners •  Deployment server (chained deployment server are not fun)

•  Deployment server does the rest

11 © F5 Networks, Inc.

Infrastructure

12 © F5 Networks, Inc.

Common Configuration

•  We already had scripts that we used to push standard applications from SVN out to target hosts (not a complete CM)

•  Every “heavy” server shared a base configuration, laid down by our configuration scripts •  /etc/init.d/splunk startup script!•  $SPLUNKHOME/etc/auth/splunkweb/mysplunkweb SSL key and

certificate •  $SPLUNKHOME/etc/system/local/authentification.conf!•  $SPLUNKHOME/etc/system/local/authorize.conf!•  $SPLUNKHOME/etc/system/local/limits.conf!•  $SPLUNKHOME/etc/system/local/serverclass.conf!

13 © F5 Networks, Inc.

Common Configuration (cont.)

deployment-apps were placed outside $SPLUNKHOME and configured via serverclass.conf [global] !repositoryLocation = /data/splunk/deployment-apps targetRepositoryLocation = $SPLUNK_HOME/etc/apps !

Thus every single “heavy” server was a deployment server and got the base deployment applications from our synchronization script. All servers are also deployment clients, pulling their customizations from an appropriate deployment server – DMZ’s clients pulled from their local DMZ server. This approach allowed me to avoid the dreaded chained deployment servers, which seemed overly complex and un-needed given my existing scripts could make all the servers identical.

14 © F5 Networks, Inc.

License and Primary Deployment Server

•  Loaded with the base configuration, web GUI enabled

•  Custom configuration includes the licenses

•  Loaded with SOS and SIDEVIEW from deployment-apps

15 © F5 Networks, Inc.

Indexers

•  Loaded with the base configurations

•  Get custom apps •  Index configurations •  Inputs and listeners

•  Plus all apps and TA’s appropriate to indexers

16 © F5 Networks, Inc.

Search Heads

•  Loaded with the base configurations

•  Get custom apps •  Define our custom props and transforms

•  Plus all apps and TA’s appropriate to search heads

17 © F5 Networks, Inc.

Syslog Aggregator / Heavy Forwarders

•  Loaded with the base configurations

•  Get custom apps from the deployment server •  Define our custom props and transforms •  Local files inputs (these are our syslog aggregation points) •  TCP and UDP inputs identical to the indexers •  Outputs everything to the Index servers

•  Guaranteed delivery of the TCP inputs was deemed “not required” to reduce cost and complexity •  These are not indexers and non-redundant (for now) if they’re down then

the logs are lost

18 © F5 Networks, Inc.

Clients

•  Load the basic universal forwarder

•  Load the appropriate deployment client app files as a “bootstrap” •  Different deployment app depending on the client location internal network

versus various DMZ networks

•  Don’t set the deployment client via the installation wizard or by changing files in the etc/system/local since these can’t be changed later by the deployment server •  Everything that gives a client a “personality” is controlled by the

deployment client apps

19 © F5 Networks, Inc.

Summary & Further Reading

20 © F5 Networks, Inc.

Summary

I live and architect solutions by the saying:

“Just because you can technically do a thing, doesn’t make it a good idea.”

The Splunk deployment server facility allows for significant flexibility and with that can come painful complexity. Here I attempted to show some options for limiting the complexity by making all the server’s base build identical but still use the deployment server to manage some of the complexity around the different server’s usage.

21 © F5 Networks, Inc.

Further Reading

http://docs.splunk.com/

http://docs.splunk.com/Special:SplunkSearch/docs?q=deployment

http://docs.splunk.com/Documentation/Splunk/5.0.4/Deploy/Scaleyourdeployment

http://docs.splunk.com/Documentation/Splunk/5.0.4/Deploy/Aboutdeploymentserver

22 © F5 Networks, Inc.

Configuration File Segments

23 © F5 Networks, Inc.

serverclass.conf #################################################################!# All non-dmz clients get the deployment client app to remain consistent with the!# deployment server.!#################################################################![serverClass:all_nondmz_deployclients]!filterType = blacklist!#SEA DMZ PROD 10.10.232.0/22!blacklist.0 = 10.10.23[2-5].*!blacklist.1 = 10.10.24[0-3,8,9].*!blacklist.2 = 10.10.25[0,1,4].*!#the dmz syslog aggs still need to check back home!whitelist.0=dmz1syslog*!whitelist.1=dmz2syslog*![serverClass:all_nondmz_deployclients:app:f5_all_deploymentclient]!!#################################################################!# All dmz1 clients point to the local DMZ syslog aggs as their deployment client.!#################################################################![serverClass:all_dmz1_deployclients]!#STGDMZ PROD 10.10.232.0/22!whitelist.0 = 10.10.23[2-5].*!#DMZ1 PROD 10.10.240.0/22!whitelist.1 = 10.10.24[0-3].*!#the dmz syslog aggs still need to check back home!blacklist.0=dmz1syslog*!blacklist.1=dmz2syslog*![serverClass:all_dmz1_deployclients:app:f5_dmz1_deploymentclient]![serverClass:all_dmz1_deployclients:app:f5_dmz1_outputs]!!#################################################################!# All dmz2 clients point to the local DMZ syslog aggs as their deployment client.!#################################################################![serverClass:all_dmz2_deployclients]!#PUBDMZ2 PROD 10.10.248.0/22!whitelist.0 = 10.10.24[8,9].*!whitelist.1 = 10.10.25[0,1,4].*!

#PUBDMZ2 Management!#the dmz syslog aggs still need to check back home!blacklist.0 = dmz1syslog*!blacklist.1 = dmz2syslog*![serverClass:all_dmz2_deployclients:app:f5_ldmz2_deploymentclient]![serverClass:all_dmz2_deployclients:app:f5_dmz2_outputs]!!#################################################################!#class defines all non-dmz forwarders and has them send directly to the 2 indexers!#################################################################![serverClass:all_forwarders]!filterType = blacklist!#the dmz syslog aggs still need to check back home!whitelist.0 = dmz1syslog*!whitelist.1 = dmz2syslog*!blacklist.0 = splunkidx01!blacklist.1 = splunkidx02!blacklist.4 = 10.10.23[2-5].*!#PUBDMZ1 PROD 10.10.240.0/22!blacklist.5 = 10.10.24[0-3,8,9].*!#PUBDMZ2 Management!blacklist.6 = 10.10.25[0,1,4].*!![serverClass:all_forwarders:app:f5_all_forwarder_outputs]!!

24 © F5 Networks, Inc.

serverclass.conf (cont) #################################################################![serverClass:deployment_servers]!whitelist.0 = splunkdply01![serverClass:deployment_servers:app:sos]![serverClass:deployment_servers:app:sideview_utils]!!#################################################################![serverClass:notbatch_search]!whitelist.0 = splunkweb02!whitelist.1 = splunkweb03!#disable summary indexing on web02 and web03![serverClass:notbatch_search:app:Splunk_for_Exchange_disablesummary]!!#################################################################![serverClass:all_indexers]!whitelist.0 = splunkidx01!whitelist.1 = splunkidx02![serverClass:all_indexers:app:f5_all_indexer_base]![serverClass:all_indexers:app:f5_all_indexes]![serverClass:all_indexers:app:f5_full_instance_inputs]![serverClass:all_indexers:app:f5_indexer_filtering_props]![serverClass:all_indexers:app:SA-ForIndexers]![serverClass:all_indexers:app:Splunk_TA_windows]!#Exchange apps - all the TA's are needed on the indexers![serverClass:all_indexers:app:Splunk_for_Exchange]!#Active Directory app![serverClass:all_indexers:app:Splunk_for_ActiveDirectory]![serverClass:all_indexers:app:SA-ldapsearch]![serverClass:all_indexers:app:sideview_utils]![serverClass:all_indexers:app:TA-fire_brigade]![serverClass:all_indexers:app:sos]![serverClass:all_indexers:app:maps]!#force everyone to search app by default![serverClass:all_indexers:app:user-prefs]!#################################################################![serverClass:all_syslog_agg]!whitelist.0 = syslog*!whitelist.1 = dmz1syslog*!whitelist.2 = dmz2syslog*!

restartSplunkd = False![serverClass:all_syslog_agg:app:f5_syslog_inputs]![serverClass:all_syslog_agg:app:f5_all_search_base]![serverClass:all_syslog_agg:app:f5_syslog_props]![serverClass:all_syslog_agg:app:f5_full_instance_inputs]![serverClass:all_syslog_agg:app:f5_indexer_filtering_props]![serverClass:all_syslog_agg:app:TA-sos]!!#################################################################![serverClass:all_parsers]!whitelist.0 = splunkidx01!whitelist.1 = splunkidx02!# These apps is added so that the parse-time rules contained therein are!# present when the data arrives at the indexers. Heavy (event-aware)!# forwarders will also require these apps.![serverClass:all_parsers:app:f5_syslog_props]!# End parse-time required apps.![serverClass:all_parsers:app:TA-sos]!!#################################################################![serverClass:syslog]!whitelist.0 = syslog01!restartSplunkd = false![serverClass:syslog:app:lea-loggrabber-splunk]!#run the mail server reputation checker on the syslog server![serverClass:sea_syslog:app:TA-SMTP-Reputation]!!

25 © F5 Networks, Inc.

serverclass.conf (cont) #################################################################![serverClass:all_search]!whitelist.0 = splunkweb01!whitelist.1 = splunkweb02!whitelist.2 = splunkweb03![serverClass:all_search:app:f5_all_search_base]![serverClass:all_search:app:f5_syslog_props]!# Homegrown![serverClass:all_search:app:f5_TA-f5]!# From Splunkbase![serverClass:all_search:app:splunk_deployment_monitor]![serverClass:all_search:app:SplunkforF5Access]![serverClass:all_search:app:SplunkforF5Networks]![serverClass:all_search:app:SplunkforF5Security]![serverClass:all_search:app:sos]!#Exchange apps - all the TA's are needed on the search heads![serverClass:all_search:app:Splunk_for_Exchange]!#utlity apps required for Exchange![serverClass:all_search:app:maps]![serverClass:all_search:app:SA-ldapsearch]![serverClass:all_search:app:sideview_utils]!#Active Directory apps![serverClass:all_search:app:Splunk_for_ActiveDirectory]![serverClass:all_search:app:Splunk_TA_windows]!#Splunk Windows![serverClass:all_search:app:windows]!#FireBrigade app from Sanford (bucket views)![serverClass:all_search:app:fire_brigade]!#need the user-prefs to get people to search by default.![serverClass:all_search:app:user-prefs]!!#################################################################![serverClass:all_windows]!whitelist.0=*!# Deploy this app only to Windows boxes.!machineTypesFilter=windows-x64, windows-intel![serverClass:all_windows:app:Splunk_TA_windows]!restartSplunkd = true!!

#################################################################![serverClass:all_windows_dc]!whitelist.0=*domain-controller*!# Deploy this app only to Windows boxes.!machineTypesFilter=windows-x64, windows-intel![serverClass:all_windows_dc:app:Splunk_TA_windows]![serverClass:all_windows_dc:app:TA-DomainController-NT6]!restartSplunkd = true!!!

26 © F5 Networks, Inc.

deploymentclient.conf and outputs.conf deployment-apps/f5_all_deploymentclient/local/deploymentclient.conf![deployment-client] !disabled = false ![target-broker:deploymentServer] !targetUri = 10.11.11.101:8089 !phoneHomeIntervalInSecs=60!!deployment-apps/f5_llx_dmz_deploymentclient/local/deploymentclient.conf![deployment-client] !disabled = false ![target-broker:deploymentServer] !targetUri = 10.10.250.14:8089 !phoneHomeIntervalInSecs=60!!deployment-apps/f5_llx_dmz_outputs/local/outputs.conf![tcpout]!defaultGroup=llx_dmz_splunk!!# FORWARD ALL INTERNAL DATA!forwardedindex.0.whitelist = .*!forwardedindex.1.blacklist =!forwardedindex.2.whitelist =!![tcpout:llx_dmz_splunk]!server=10.10.250.14:9997!compressed=false!!deployment-apps/f5_all_forwarder_outputs/local/outputs.conf![tcpout]!defaultGroup=prod_splunk!!# FORWARD ALL INTERNAL DATA!forwardedindex.0.whitelist = .*!forwardedindex.1.blacklist =!forwardedindex.2.whitelist =!!

[tcpout:prod_splunk]!server=10.11.11.17:9997,10.11.11.18:9997!compressed = false!

27 © F5 Networks, Inc.

Next Steps

Download the .conf2013 Mobile App If not iPhone, iPad or Android, use the Web App

Take the survey & WIN A PASS FOR .CONF2014… Or one of these bags!

Go to “Architecting and Sizing Your Splunk Deployments” Room: Brera 2&3, Level 3 Today, 3-4pm

1

2

3

1

2

3

28 © F5 Networks, Inc.

devcentral.f5.com facebook.com/f5networksinc linkedin.com/companies/f5-networks twitter.com/f5networks youtube.com/f5networksinc