Copyright © 2014 Splunk Inc.

Vladimir Had Security Engineer, First Data

Change control using GIT

Splunk Disclaimer

2

During the course of this presentation, we may make forward looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and

estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in the this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other

commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

3

Around the world, every second of every day, First Data makes payment transactions secure, fast and easy for merchants, financial institutions and their customers.

Whether the choice of payment is by debit or credit card, gift card, check or mobile phone, online or at the checkout counter, First Data takes every opportunity to go beyond the transaction by leveraging

its vast product portfolio and expertise to drive customer revenue

and profitability.

More information about the company is available at: FirstData.com (http://firstdata.com)

Twitter (http://twitter.com/firstdata)

LinkedIn (http://www.linkedin.com/company/first-data-corporation)

Facebook (http://www.facebook.com/firstdata)

YouTube (http://www.youtube.com/firstdata)

About Me

4

Security Engineer in First Data

Working in Information Security for 10 years, various jobs from technical roles to management level

Holding CISSP, CISA, CISM, OSCP, OSWP

Splunker for 2 years

Agenda

5

Splunk in First Data

Challenges

Change control

Change control using GIT

Common use cases

Splunk Live!

Next Steps

Summary

Splunk at First Data Global log collection tool for SIEM – APAC, EMEA, LAC, NA

6

Operations Management Metrics and Reporting

Analytics and Visualization Application Management

LOB Owners/ Executives

System Administrator

Operations Teams

Security Analysts

IT Executives

Application Developers

Auditors Website/Business

Analysts Customer Support

Security and Compliance

Splunk at First Data Users and Production Usage

7

Splunk at First Data Administrators and Security Engineers

8

Challenges

1. We need to know what each admin/party is doing

2. Share this in effective way

3. Distribute changes to regions

4. Apply changes to minimize impact/outage based on local time

9

Challenges

1. We need to know what each admin/party is doing

2. Share this in effective way

3. Distribute changes to regions

4. Apply changes to minimize impact/outage based on local time

There are additional steps:

5. Review changes

6. Fix bugs

7. Apply changes to minimize impact/outage based on local time

10

Change control

11

Keep comments in configuration files when each change is introduced

Change control Keep copies of the modified file

12

Change control using GIT Distributed version control system

13

Change Control Using GIT

14

Reference architecture

Master GIT server Remote repository

Private/Local GIT repository Private/Local GIT repository

Deployment server

Admin/Engineer

Add-ons Apps Knowledge objects

• GIT Bare repository

• All changes are distributed as deployment-apps – Rule: Minimal changes in local

configs on Splunk instances $SPLUNK_HOME/etc/system/local/

• Access to repo via SSH

• Authentication by OS – E.g. local user accounts, via LDAP

• Access control (rw/ro) based on file system permissions

Change Control Using GIT GIT Server

15

Master GIT server Remote repository # Appendix 1

Add-ons Apps Knowledge objects

• GIT works in snapshot mode – each

Admin needs to create local snapshot (exact clone) of master GIT repository

• Admin works locally on local GIT repository, makes changes on deployment apps to local repo

• No network connectivity required for 99% of operations

• Admin to synchronize work with master GIT repository – to git-push changes to shared place

Change Control Using GIT

16

Admin / Engineer

Private/Local GIT repository Master GIT server Remote repository

# Appendix 2

• Local GIT repository using read-only

user to perform clone

• Update local GIT repository by GIT pull when required

• Repository location pointing to local GIT repository serverclass.conf

• Reload/Restart deployment server to apply the changes

• serverclass.conf also served as an app

• Symlink from local GIT repo to etc/system/local/<conf>

Change Control Using GIT

17

Deployment Server

Master GIT server Remote repository

Deployment server

Local GIT repository

# Appendix 3 & 4

Change Control Using GIT

18

Distributed version

Master GIT server Remote repository

Add-ons Apps

Local GIT repository Local GIT repository

Local GIT repository

Local GIT repository

Local GIT repository

Knowledge objects

Common use cases

19

Basic workflow in GIT

Basic workflow for deployment server

History review

Identify differences between last and previous change

Recover deleted file

Distributed version control system

Splunk Live!

Use cases

21

Basic workflow 1/3

Use cases

22

Basic workflow 2/3

Use cases

23

Basic workflow 3/3

Use cases

24

Deployment server

Use cases

25

History review 1/3

Use cases

26

History review 2/3

Use cases

27

History review 3/3

Use cases

28

File recovery 1/2

Use cases

29

File recovery 2/2

Next steps

30

Use long-running branches – master, development, test

Different branches for introducing changes such as feature, fix, updates, upgrades – Define naming convention - region/type/name of the change (e.g. emea/upgrade/TA-windows)

Use pull-requests to review/push change into long-running branches

Granular access control to branches – No direct access to production branch

Summary

31

Using GIT for version control and change control in Splunk

GIT repository tracks all add-ons, apps and knowledge objects

Admins/Engineers push the changes to master GIT repository

Deployment server pulls the changes from master GIT Repository and provides deployment apps from local GIT repository

Deployment server is reloaded when required

All history is kept, changes can be reviewed and reverted to any previous working state

References

32

• Splunk – https://docs.splunk.com – $SPLUNK_HOME/etc/system/README/ – Creating Splunk Knowledge Objects – Developing Apps with Splunk

• GIT – GIT manual pages – http://git-scm.com – Introduction to GIT: Scott Chacon

THANK YOU

Appendix 1

34

# on GIT Server

mkdir -p /opt/git # make directory where our git repository will be located

cd /opt/git # change to our directory

git init --bare conf2014.repo.git # create bare GIT repository

chmod -R 2775 conf2014.repo.git/ # our access control is based on filesystem access

groupadd git-rw-group # create one group that will have read write access to GIT repository

chgrp -R git-rw-group conf2014.repo.git # change group for all files/directories recursively

useradd vlad # create user that will be member of read-write group

usermod -a -G git-rw-group vlad # assign user the permission rw to GIT repository

useradd git-ro-user # create read-only user that will be used by deployment server to sync the repositories

GIT Server Setup 1/2

Appendix 1

35

# on Workstation

cd ~ # change to home directory

mkdir local && cd local # we will store local GIT repository in local directory called local

git clone ssh://vlad@gitserver/opt/git/conf2014.repo.git # perform git clone, see that vlad is doing it so it will have read write access to GIT

cd conf2014.repo # change directory to cloned GIT repository

git config --global user.name "Vlad“ # configure author, nickname, color and some nice aliases

git config --global user.email vladimir.had@firstdata.sk

git config --global color.ui true

git config --global alias.lol "log --graph --decorate --pretty=oneline --abbrev-commit"

git config --global alias.lola "log --graph --decorate --pretty=oneline --abbrev-commit --all"

echo 'GIT repository for conf2014 Demo' >> readme # prepare for the first commit

git add readme # add file called readme to index

git commit -m 'initialization of GIT repository‘ # commit readme file

git push origin master # push changes to remote GIT repository and create branch ‘master’

GIT Server Setup 2/2

Appendix 2

36

# on Workstation

git pull # update working tree

vim outputs.conf # make modifications

git status # find what is actual status of our local GIT repository

git diff # find changes

git add outputs.conf # store changes

git commit –m ‘CR12345: adding additional indexer’ # add changes to index

git push # push changes to remote repository

Basic workflow: Read-write user 1/2

Appendix 2

37

# on Workstation

cd ~ && cd local/conf2014.repo # go to local GIT repository

git pull # update working tree

mkdir –p deployment-apps/conf2014_myapp/local # create directory structure

mkdir –p deployment-apps/conf2014_myapp/metadata # create directory structure

cat << EOF > deployment-apps/conf2014_myapp/local/inputs.conf

[monitor:///var/log/messages]

disabled = 0

sourcetype = syslog

index = main

EOF

git add . # add all changes to index

git commit –m ‘conf2014: adding new app’ # commit changes

git push # push changes to remote repository

Basic workflow: Create app 2/2

Appendix 3

38

# on Deployment server

su – splunk # ensure all steps are done under user called splunk

cd $SPLUNK_HOME # go to default splunk directory

mkdir etc/git # create empty holder of git repositories

git clone ssh://git-ro-user@gitserver/opt/git/conf2014.repo.git

# clone master GIT repository

cd conf2014.repo # change directory to local GIT repository

git log -3 # find last 3 changes in GIT repository

Basic workflow: Splunk user 1/2 - First pull

Appendix 3

39

# on Deployment server

su – splunk # splunk is running under user called splunk

cd $SPLUNK_HOME/etc/git/conf2014.repo # change directory to local GIT repository

git pull # fetch all changes from remote GIT repository and merge them automatically

/opt/splunk/bin/splunk reload deploy-server # reload deployment server

Basic workflow: Splunk user 2/2 – Update repository

Appendix 4

40

# on Workstation

cd ~ && cd local/conf2014.repo # change to local repository

git pull # fetch and merge latest changes from remote

mkdir deployment-apps # create folder for deployment apps

mkdir –p deployment-apps/conf2014_serverclass/local # create app

cat <<EOF > deployment-apps/conf2014_serverclass/local/serverclass.conf

[global]

repositoryLocation = $SPLUNK_HOME/etc/git/conf2014.repo/deployment-apps

EOF

Echo ‘#Autogenerated file’ > deployment-apps/conf2014_serverclass/local/app.conf

git add . # add all changes to index

git commit –m ‘conf2014: serverclass app’ # commit changes

git push # push app to remote GIT repository

Serverclass.conf into GIT 1/2

Appendix 4

41

# on Deployment server

su – splunk # splunk is running under user called splunk

cd $SPLUNK_HOME/etc/git/conf2014.repo # change directory to local GIT repository

git pull # fetch all changes from remote GIT repository and merge them automatically

cd $SPLUNK_HOME/etc/system/local # change directory to local system directory

ln –sf /opt/splunk/etc/git/conf2014.repo/deployment-apps/conf2014_serverclass/local/serverclass.conf .

# create symbolic link from serverclass.conf in GIT repository to etc/system/local

/opt/splunk/bin/splunk reload deploy-server # reload deploy-server

Serverclass.conf into GIT 2/2