Splunk Deep Dive… · 2020. 8. 4. · Elastic Certified Engineer Splunk Enterprise Features • Index • Source data from websites, applications, servers, databases, and more •

Elastic Certified Engineer

Splunk Deep Dive

Course Overview


Course Overview

• Non-Certification• Who it is for:

• Big Data Enthusiasts

• Developers

• Administrators

• Pre-requisites:

• Basic Linux experience (LPI Essentials)

• Basic networking knowledge


Splunk Deep Dive

Splunk Enterprise Overview


Splunk Enterprise Features

• Index

• Source data from websites, applications, servers, databases, and more

• Index your IT data into Splunk

• Search

• Primary way you will navigate your data in Splunk

• Search can also be used as a report to power dashboard panels

• Alerts

• Get notified when search results meet specific conditions

• Alert actions can send an email, post to an RSS feed, or execute a script


Splunk Enterprise Features (Continued)

• Dashboard

• Combine panels into a wholistic view of your data

• Panels can contain search boxes, fields, charts, and more

• Pivot

• Map attributes to a table, chart, or data visualization

• Can be saved as reports and added to dashboards

• Reports

• Save searches as pivot reports and then add reports to dashboards as panels

• Run them ad-hoc or on a schedule


Splunk Enterprise Components

• Forwarder

• Collects and forwards data to an indexer

• Low resource usage allowing them to reside on many machines with little impact

• Indexer

• Indexes data received from a forwarder

• Searches indexed data when requested by a search head

• Search Head

• Interacts with users by directing search requests to indexer

• Merges search results when directing multiple indexers


Splunk Deep Dive

Splunk Enterprise Installation Recommendations


Installation Requirements

• Platforms

• VMs are not recommended and will decrease performance

• Network File Systems (NFS) are not recommended

• Containers are supported with Docker Enterprise or Docker Community Edition

• 64-bit OS

• Linux with kernel versions 2.6, 3.x, and 4.x

• Windows Server 2012, 2012 R2, and 2016


Installation Requirements (Continued)

• Recommended Hardware

• Two 6-core processors at or above 2 GHz

• 12 GB of RAM

• RAID 0 or RAID 10 storage

• Should be capable of 800 IOPS

• Solid state drives are recommended

• NVMe drives will provide the most performance

• 1 Gb NIC with network latency under 100 ms between Splunk nodes

• Supported File Systems

• Linux: ext3, ext4, btrfs, XFS, NFS

• Windows: NTFS, FAT32


Capacity Planning

• Considerations

• How much data do you expect to index daily?

• How much data do you need to retain and for how long?

• How many users do you expect to search through the data at any one time?

• Do you plan to use certain specific searches more than once?

• Do you want or need to use a Splunk app to present or manipulate your data?

• Storage

• Experiment with indexing data samples and checking the size of the Splunk DB

• Rawdata file is about 10% of the size of the incoming data

• Index files can range from 10% to 110% of the size of the rawdata file depending on the amount of unique terms


Capacity Planning (Continued)

< 2 GB/day 2-300 GB/day 300-600 GB/day 600 GB-1 TB/day 1-2 TB/day 2-3 TB/day

Less than 4 users

1 combined instance

1 combined instance

1 Search Head,2 Indexers


Search Head,7 Indexers


Up to 8 users

1 combined instance






Up to 16 users




2 Search Heads,4 Indexers



Up to 24 users







Up to 48 users







In this lesson, we will be installing a standalone Splunk Enterprise instance on our very own Linux Academycloud server via the Cloud Playground.

From Linux Academy's site, click on Cloud Servers > Playground

Create a new cloud server with the following specifications:

Distribution: CentOS 7Zone: Your preferred zone. Mine is going to be North America.Size: LargeTag: Splunk Deep Dive

After your server starts up, log in and reset the temporary password to one of your choosing.

You can use the following link to create or log in to a Splunk account and download the Splunk Enterprise7.2.4.2 Linux x86_64 RPM: Splunk 7.2.4.2

Once you create an account and log in, the download will begin automatically.

Note: Uploading the Splunk Enterprise installer can take some time depending on your internet upload speed.

If you are using a Windows machine, you can use PuTTY's PSCP to SCP from the Windows command line:

pscp -scp splunk-7.2.4.2-fb30470262e3-linux-2.6-x86_64.rpm cloud_user@public_ip_of_cloud_server:/tmp

Standalone Splunk Enterprise Installation

Create Your Cloud Server

Download Splunk Enterprise

Copy the Splunk Enterprise RPM to your cloud server

Windows Users

Mac and Linux Users

https://www.splunk.com/page/download_track?file=7.2.4.2/linux/splunk-7.2.4.2-fb30470262e3-linux-2.6-x86_64.rpm&platform=Linux&architecture=x86_64&version=7.2.4.2&product=splunk&typed=release&name=linux_installer

For Mac or Linux users, you can just use scp from the command line:

scp splunk-7.2.4.2-fb30470262e3-linux-2.6-x86_64.rpm cloud_user@public_ip_of_cloud_server:/tmp

Log in to your cloud server as the user cloud_user with your preferred method. I will be using my nativeTerminal application.

Become the root user:

sudo su -

Install the Splunk RPM:

rpm -i /tmp/splunk-7.2.4.2-fb30470262e3-linux-2.6-x86_64.rpm

Create the file /opt/splunk/etc/system/local/user-seed.conf with the following contents,substituting in the password of your choosing:

[user_info]USERNAME = adminPASSWORD = your_password

Save and close the file.

Enable boot-start and accept the license as this is the first time we will be executing the Splunk binary:

/opt/splunk/bin/splunk enable boot-start --accept-license

Install Splunk Enterprise

Manually create the administrator credentials

Configure Splunk to start on boot

Start Splunk Enterprise

Start Splunk using the splunk binary:

/opt/splunk/bin/splunk start

Or, start Splunk using systemctl :

systemctl start Splunkd

Check the splunkd.log log file:

less /opt/splunk/var/log/splunk/splunkd.log

In your web browser, go to http://public_ip_of_cloud_server:8000

Log in with the administrator credentials you configured earlier.

Log in to the Splunk Web Console


Splunk Deep Dive

Splunk Enterprise Security Overview


Install Splunk Securely

• Verify installer integrity

• MD5

• SHA512

• GnuPG Public Key

• Create Secure Administrator Credentials

• Use a strong password

• Hash the password in the user-seed.conf file

• Enable the Federal Information Processing Standard (FIPS)

• Enable if it is a regulatory requirement for your environment

• Automatically enabled on a Linux machine that runs a kernel in FIPS mode


Install Splunk Securely (Continued)

• Encrypt Web Console Traffic

• Use either the default or your own certificates

• Enables HTTPS for the Splunk Web Console

• Configure Splunk password policies

• Create your own password requirements for Splunk users

• Limit Data Access with Role-Based User Access Control

Let's secure our standalone Splunk Enterprise instance that we installed earlier on our Linux Academy cloudserver via the Cloud Playground.

Become the root user

sudo su -

Copy the default web.conf from /opt/splunk/etc/system/default/ into/opt/splunk/etc/system/local/ :

cp /opt/splunk/etc/system/default/web.conf /opt/splunk/etc/system/local/.

Change the permissions to enable write permissions on /opt/splunk/etc/system/local/web.conf :

chmod 600 /opt/splunk/etc/system/local/web.conf

Edit the /opt/splunk/etc/system/local/web.conf file by changing the following lines:

# port number tag is missing or 0 the server will NOT start an http listener# this is the port used for both SSL and non-SSL (we only have 1 port now).httpport = 8000

# this determines whether to start SplunkWeb in http or https.enableSplunkWebSSL = false

To:

Secure Splunk Enterprise

Configure HTTPS and the use of port 443 for the SplunkWeb Console

# port number tag is missing or 0 the server will NOT start an http listener# this is the port used for both SSL and non-SSL (we only have 1 port now).httpport = 443

# this determines whether to start SplunkWeb in http or https.enableSplunkWebSSL = true

Save and close the file.

Restart Splunk:

/opt/splunk/bin/splunk restart

Or:

systemctl restart Splunkd

Browse to the web console over HTTPS using https://public_ip_of_cloud_server .

Browse to the web console over HTTPS using https://public_ip_of_cloud_server and log in usingyour administrator credentials.

Go to Settings > Access Control, then click + Add new next to Users.

Name: yourfirstnameFull Name: yourfirstname yourlastnameEmail address: youremailaddressSet password: weakpasswordConfirm password: weakpasswordTime zone: DefaultDefault app: launcherAssign to roles: userCreate a role for this user: falseRequire password change on first login: false

Click Save.

Log out of admin and log in as your new user to test it out.

Create a new user with basic access

Browse to the web console over HTTPS using https://public_ip_of_cloud_server and log in usingyour administrator credentials.

Go to Settings > Access Control > Password Policy Management

Minimum characters: 8Numeral: 1Lowercase: 1Uppercase: 1Special character: 1Expiration: EnableDays until password expires: 90Expiration alert in days: 15History: EnablePassword history count: 24Constant login time: 0Login fail message: SimpleForce existing users to change weak passwords: TrueLockout: EnableFailed login attempts: 5Lockout threshold in minutes: 5Lockout duration in minutes: 30

Click Save.

Log out as the administrator user and log back in as the user you created earlier and change the password tosomething that meets our new password requirements when prompted (example: $trongPassword123 ).

Configure stricter Splunk password policies


Splunk Deep Dive

What’s Next?


What’s Next?

• Hadoop Quick Start• Elastic Stack Essentials• DevOps Essentials• Big Data Essentials• Configuration Management:

• Chef

• Puppet

• Ansible

• Salt

Documents

Splunk Deep Dive… · 2020. 8. 4. · Elastic Certified Engineer Splunk Enterprise Features • Index • Source data from websites, applications, servers, databases, and more •