© 2019 SPLUNK INC.© 2019 SPLUNK INC.

Kubernetes Observability with Splunk Connect for Kubernetes (SCK)

October 2019

© 2019 SPLUNK INC.

During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC.

The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.

Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved.

Forward-Looking Statements

THIS SLIDE IS REQUIRED, DO NOT DELETE

© 2019 SPLUNK INC.

Senior Software Engineer | Splunk

Donald Tregonning

Software Engineer | SplunkChaitanya Phalak

Use this if there will be two speakers for your session.

Shubham JainSenior Software Engineer | Splunk

© 2019 SPLUNK INC.

DemoSneak peek into today’s presentation

© 2019 SPLUNK INC.

1. Environment Overview – Splunk & K8s

2. Observability Challenges & Problems solved

3. Splunk Connect for K8s (Intro, Design and Architecture)

4. SCK Installation and Configuration

5. Kubernetes Metrics and Splunk Metrics Workspace

6. Advanced Splunk Tooling with SCK

7. SCK Highlights and Roadmap

8. Q&A

Agenda

© 2019 SPLUNK INC.

Splunk & SCK Environment

© 2019 SPLUNK INC.

Challenges with Observability

▶ 3 pillars of observability

▶ Loosely tied together

▶ Resides in different systems

Observability

Logs

MetricsTraces

© 2019 SPLUNK INC.

Splunk Connect for Kubernetes brings together all three pillars into

one solution…

© 2019 SPLUNK INC.

✔ Tackled Heapster deprecation and removal from K8s Project

✔ Options to build out metrics

• Prometheus

• Simple Summary API Client

• Custom Metrics solution (Winner)

✔ Reliability

✔ Metric Store Compatibility

✔ Elasticity

✔ Version Support

✔ Scalable

Problems tackled with SCK

© 2019 SPLUNK INC.

Different Types of Data in Splunk

▶ Metric data▶ Event data

• Log data

• Objects metadata

© 2019 SPLUNK INC.

Splunk Connect for Kubernetes InternalsAn Introduction, Design Patterns and Product Architecturehttps://github.com/splunk/splunk-connect-for-kubernetes

© 2019 SPLUNK INC.

SCK Component

Component types• Logs

• Metrics

• ObjectsCollector agent: Fluentd

• Fluentd plugins for component functionality

Deployment types• K8s Deployment

• K8s Daemonset

© 2019 SPLUNK INC.

Logging Component Deployment - daemonset

• configmap

• secret

• serviceAccountFluentd plugins

• in_tail

• systemd

• monitor_agent

• concat

• jq_transformer

• splunk_hecLogs are sent to Splunk using Splunk’s fluentd HEC plugin (splunk_hec)

© 2019 SPLUNK INC.

Metrics Component Deployments

• daemonset

• Supporting kubernetes objects (configmaps, secrets and serviceAccounts)

Fluentd plugins • kubernetes_metrics

• kubernetes_metrics_aggregator

• splunk_hec

Metrics are sent to Splunk using Splunk’s fluentd HEC plugin (splunk_hec)

© 2019 SPLUNK INC.

Objects Component Deployment

• configmap

• secret

• serviceAccountFluentd plugins

• kubernetes_objects

• splunk_hecObjects plugin • Cluster’s events metadata for

configured kubernetes objects (pods, nodes, configmaps)

Collected events are sent to Splunk using Splunk’s fluentd HEC plugin(splunk_hec)

© 2019 SPLUNK INC.

Installation and ConfigurationHow to install and configure Splunk Connect for K8s using Helm

© 2019 SPLUNK INC.

Splunk Enterprise 7.0 or later version

Splunk HTTP Event Collector

Two Splunk indexes ready

Splunk Prerequisites What you need to get started

© 2019 SPLUNK INC.

Helm to deploy SCK Admin access to Kubernetes cluster to run

helm

Ability to install SCK using yaml manifests

Kubernetes Prerequisites What you need to get started

© 2019 SPLUNK INC.

DemoSplunk Connect for Kubernetes

© 2019 SPLUNK INC.

K8s Metrics Endpoints and Splunk Metrics Workspace.

Where we collect things and where they end up

© 2019 SPLUNK INC.

Metrics Collected in SCK:

https://github.com/splunk/fluent-plugin-kubernetes-metrics/blob/develop/metrics-information.md

Metrics Collection Endpoints:

1.) @kubelet_url = "https://#{ env_host}:#{env_port}/stats/summary"

2.) @kubelet_url_stats = "https://#{ env_host}:#{env_port}/stats/"

3.) @cadvisor_url = "https://#{ env_host}:#{env_port}/metrics/cadvisor"

4.) @kubernetes_url_final = "https://#{env_host}:#{env_port}/api/"

Where do we collect K8s

Metrics from?

© 2019 SPLUNK INC.

Splunk Metrics Workspace

© 2019 SPLUNK INC.

Advanced Splunk Tooling with SCKIntroducing Splunk App for Infrastructure with SCK

© 2019 SPLUNK INC.

• Unifies and correlates logs and metrics for comprehensive infrastructure monitoring and troubleshooting

• Guided, out-of-the-box data onboarding to monitor Kubernetes performance and health

• Identify trends and find root causes fast

Splunk App for InfrastructureEasily deploy Splunk Connect for Kubernetes and monitor your k8 environment

© 2019 SPLUNK INC.

DemoSplunk App for Infrastructure

© 2019 SPLUNK INC.

Splunk Connect Kubernetes Highlights and What’s NextNext Release – Now!

© 2019 SPLUNK INC.

1. Works well with all Kubernetes flavors (self-managed or hosted: EKS, GKE, AKS, OpenShift and IBM IKS…and everything else)

2. Reliable event delivery leveraging Splunk’s HTTP Event Collector.

3. Scalable and secured way of ingesting k8s data into Splunk

4. Ships with features like: • Index routing

• Data filtering

• Support for Custom Metadata

• Support for audit logs

5. Actively developed and maintained by Splunk and its community!!

6. Support for upcoming Splunk Data Ingestion product. Ingest API

7. Simple, performs exceptionally well

Product summary

SCK Highlights

© 2019 SPLUNK INC.

What’s next for the SCK!

Support for PKS

Flexible security

configurations (PSP)

More metrics alongside improved

performanceBetter out of

the box tracing capability

© 2019 SPLUNK INC.

Github repo links for different component of SCK • https://github.com/splunk/splunk-connect-for-kubernetes

• https://github.com/splunk/fluent-plugin-kubernetes-objects

• https://github.com/splunk/fluent-plugin-splunk-hec

• https://github.com/splunk/fluent-plugin-kubernetes-metrics

• https://github.com/splunk/fluent-plugin-k8s-metrics-agg

▶ Contributions - https://github.com/splunk/splunk-connect-for-kubernetes/issues/new

Important Project Links & Contributions

© 2019 SPLUNK INC.

You!Thank

© 2019 SPLUNK INC.

Don’t forget to rate our session

© 2019 SPLUNK INC.

Q&ACome visit us at the GDI booth on the main floor!!