Inputs:File,Network,* Script,andmore!* Splunkd:Pipelines& … · 2017-10-13 · 24 Line%Breaker% _raw* Sep1206:11:58abathlmba13.no.cox.netstoreagent[49597]:StarSngupdatescan

Copyright © 2014 Splunk Inc.

Inputs: File, Network, Script, and more! Splunkd: Pipelines & Processors & Queues, Oh my! Amrit Bath, Jag Kerai

Agenda

!   Splunkd building blocks: –  Pipelines, processors, queues –  Not here: Indexing/clustering, searching.

!   Where data goes & how !   Where data comes from !   Debugging/opSmizing

(What Would Octavio Do?)

About Us

3

!   Jag Kerai (2008) – Engineer & Manager –  Forwarding/receiving, Splunkd framework,

HA/Clustering, Deployment server… –  Previously: Xsigo Systems, Brocade

CommunicaSons, SAP Lab – Web apps in 1995!

–  Is a smart dude

!   Amrit Bath (2005) – Engineer & Hungover –  CLI, Deploy Serv, Tailing Processor, REST API,

Universal Forwarder, Indexed ExtracSons… –  Previously: Unemployment, College –  Tries to enjoy working on cars

Splunk Architecture

4

Index Cluster

Splunk Web Interface Splunk CLI Interface Other Interfaces

Search

Monitor Files Network input Detect File Changes

Parse / Extract / Manipulate

Distributed Search

REST API

Users / ACL

Scheduling/AlerSng ReporSng Knowledge

Splunk > Engine

Distributed Search

Deployment Server

Script / Stdout

Forward

It’s All a Pipeline

5

A B D E

C

Really

6

A B

C

D E

Forwarder

Indexers

Appserver

100Mb?

7200 RPM/ 10k/15k?

Internally, too!

7

A B

C

D E Index

Queue Queue

Monitor input

Processors

And Across MulSple Instances!

8

Index

Queue Queue

Monitor input

Processors

TCP Output

TCP input

Queue 100 Mb

(This is the biggun’)

UF IDX

Data Structures & RouSng

Pipeline Data

10

Host www2

Index prod_servers

_raw 10.3.1.92 -‐ -‐ [21/Jul/2011:20:34:44 -‐0700] "GET /results/bonnie-‐solns_vm_nick.html HTTP/1.1" 200 2938

UTF-‐8 Line Broken

_conf www2, access_log, /var/log/hqpd/access_log

Pipelines

11

!   Pipeline: thread !   Data flows through linearly, hits mulSple pipelines before indexing !   Naturally allow parallelism, modularity !   Config files: $SPLUNK_HOME/etc/modules/ Merged: var/run/splunk/composite.xml

Input UTF-‐8 Converter Line Breaker Header

ExtracSon Output

Parsing Pipeline

Pipeline Data

Processor

12

!   Processor: Performs small but logical unit of work !   Contained within a Pipeline !   Executed by Pipeline thread !   Example: LineBreaker, Aggregator, TcpInput, Index

Pipeline

Processor 2 Output Input Processor 1

Pipeline Data

Pipelines/Processors (Debugging)

13

Parsing Queue

Agg Queue

Typing Queue

Index Queue

TCP/UDP Pipeline

Tailing

FIFO Pipeline

FSChange

Exec Pipeline

uu8

Header

Parsing Pipeline

Linebreaker Aggregator

Merging Pipeline

Regex Replacement

Annotator

Typing Pipeline

TCP Out

Syslog Out

Indexer

Index Pipeline

UF

UF + IDX_E


14

Parsing Queue

Agg Queue

Typing Queue

Index Queue

TCP/UDP pipeline

Tailing

FIFO pipeline

FSChange

Exec pipeline

uu8

Header

Parsing Pipeline

Linebreaker Aggregator

Merging Pipeline

Regex Replacement

Annotator

Typing Pipeline

TCP Out

Syslog Out

Indexer

Index Pipeline


15

Parsing Queue

Agg Queue

Typing Queue

Index Queue

uu8

Header

Aggregator

Regex Replacement

Annotator

TCP Out

Syslog Out

Indexer

Parsing Pipeline

Merging Pipeline

Typing Pipeline

Index Pipeline

Linebreaker

TCP/UDP pipeline

Tailing

FIFO pipeline

FSChange

Exec pipeline

Queue

16

!   Queue size bounded by memory !   Variable size pipeline data

pData pData pData pData

Queue

Thread Thread

Process

Process Remove Insert

Queue

17

Pipeline Thread

Insert Blocked

1.   Remove 2.   Process 3.   Insert (Blocked)

pData pData pData pData

Index Queue (Full) Tcp Output Typing Pipeline

Process Remove Insert (Blocks) Network

Down

pData pData pData

Pipeline Thread

Insert Blocked

1.   Remove 2.   Process 3.   Insert (Blocked)

Write to Network (Fails)

Queue (Full) Queue (Full) Queue (Full)

Processors

Persistent Queue

18

!   If memory used up, use file system !   Writer does not block if memory is used up

!   Think virtual memory

pData pData

RAM

Regular Queues

pData pData

RAM File System

Persistent Queues

!  Writer blocks if Q is full

Persistent Queue

19

Splunk Host

Internal Queues Full

pData pData Tcpout Q Input Q Persistent Q Full

Network

Much Bigger Queue

Network

Processors: Input & Parsing

Input Pipelines

21

Network File Monitor FIFO FS Change Script Registry Monitor

Host www2

Index prod_servers

_raw Jul 30 00:21:19 amritDesktop sshd[30416]: Accepted publickey for amrit from 10.3.1.52 port 59426 ssh2 \n

Jul 30 00:21:26 amritDesktop sshd[30418]: Received disconnect from 10.3.1.52: 11: disconnected by user \n

Pipeline Data

Parsing Queue

…

Monitor Input (aka Tailing Processor)

22

!   Two synchronous components: –  File Update NoSficaSon (FUN!) –  Tailer: reads files

!   Files are read: 1) One at a Sme 2) In 64KB chunks 3) UnSl EOF

–  Can read large files & archives in parallel !   Send <=64 KB chunks to output queue

…

Tailer Batch Archive

Parsing Queue

FUN!

Parsing: UTF-‐8 Processor

23

UTF-‐8 Processor

Host www2

Charset UTF-‐8

_raw ポンティアック・トランザム

Host www2

Charset Shiz-‐JIS

_raw ƒ|ƒ“ƒeƒBƒAƒbƒN�Eƒgƒ‰ƒ“ƒUƒ€

Pipeline Data

Pipeline Data

24

Line Breaker

_raw

Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: StarSng update scan Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: UpdateController: Message tracing { "interval_since_last_invocaSon" = 23000; "power_source" = ac; "power_state" = wake; "start_date" = "2014-‐08-‐21 20:10:39 +0000"; } Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: Asserted BackgroundTask power asserSon (returned 0)

Pipeline Data

_raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: StarSng update scan Pipeline Data

Parsing: Line Breaker

_raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: UpdateController: Message tracing { Pipeline Data

_raw "interval_since_last_invocaSon" = 23000; Pipeline Data

_raw "power_source" = ac; Pipeline Data

. . .

25

Pipeline Data

**OR** Parsing: CSV/JSON Line Breaker (6.0+)

CSV/JSON Line Breaker

_raw How Splunkd works,”Learn about how Splunk ingests and parses data”,Amrit/Jag How search works,Learn how Splunk’s search language works,Dr. Z

_raw How Splunkd works,”Learn about how Splunk ingests and parses data”,Amrit/Jag

Pipeline Data

sourcetype csv

See Also: sourcetype=_json INDEXED_EXTRACTIONS=(csv|json|…)

sourcetype csv

_meta Subject::”How Splunkd works” DescripSon::”Learn about how Splunk ingests and\nparses data” Presenter::Amrit/Jag

_raw How search works,Learn how Splunk’s search language works,Dr. Z

sourcetype csv

_meta Subject::”How search works” DescripSon::”Learn how Splunk’s search language works” Presenter::”Dr. Z”

Pipeline Data

From File Containing: Subject,DescripSon,Presenter ↵ How Splunkd works,”Learn about how Splunk ingests and ↵ parses data”,Amrit/Jag ↵ How search works,Learn how Splunk’s search language works,Dr. Z

Parsing: Header Processor

26

Header Processor

Host database_1

_raw Unknown database error

Pipeline Data

Pipeline Data

Host www2

_raw Unknown database error

Pipeline Data

Host www2

_raw ***SPLUNK*** host=database_1

27

Merging: Line Merge Pipeline Data _raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: StarSng update scan Pipeline Data

Line Merge

_raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: UpdateController: Message tracing { Pipeline Data

_raw "interval_since_last_invocaSon" = 23000; Pipeline Data

_raw "power_source" = ac; Pipeline Data

. .

Pipeline Data _raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: StarSng update scan

Pipeline Data

_raw

Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: UpdateController: Message tracing { "interval_since_last_invocaSon" = 23000; "power_source" = ac; "power_state" = wake; "start_date" = "2014-‐08-‐21 20:10:39 +0000"; }

Pipeline Data

_raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: Asserted BackgroundTask power asserSon (returned 0) Pipeline Data

Typing: Regex Replacement

28

Regex Replacement

Host logbox

Sourcetype syslog

_raw Sep 12 06:11:58 abath-‐mba13.no.cox.net storeagent[49597] <CriScal>: StarSng update scan

Host abath-‐mba13.no.cox.net

Sourcetype syslog


Pipeline Data

Pipeline Data

Typing: Annotator

29

Annotator



Pipeline Data


Punct __::_-‐_[]_<>:___


Pipeline Data

Pipeline Data

Indexer Pipeline: TCP/Syslog Out, Indexer

30

TCP Output


Index main


Pipeline Data

Syslog Output

Indexer

To remote server…

To remote server…

To disk.. Or cluster!

Scripted Input (aka Exec Processor)

31

mysql_fetch.sh web_ping.bat

tracert.exe

data

args

data

args

args data

Exec Processor

Parsing Q

TCP/UDP Input

32

Pipeline Data

TCP/UDP Pipeline

Parsing Q

Splunk

Host

TCP

UDP

Read

Port

Wait

Parsing Pipeline

Remove

Insert to Queue

Processors

Insert

Network

TCP Output Qs

33

pData

pData

Load Balance Group

pData

Load Balance Group

Typing Pipeline

Insert (Blocks)

Processors

Splunk

Splunk

Clone or Route

TCPOut Q

TCPOut Q

Index Q

Splunk

Splunk

Network Down

Network

Nerdier Stuff

34

Resource Management

35

2007-‐01-‐23 21:15:46,674 DEBUG [com.splunk.doom] com.sun.ebank.ejb.customer…. \n java.lang.NumberFormatExcepSon: For input string: "fish!" \n at java.lang.NumberFormatExcepSon.forInputString(NumberFormatExcepSon.java:48) \n at java.lang.Integer.parseInt(Integer.java:447) \n …

Pipeline Data Raw Storage

offset @0 + 100

_raw 2007-‐01-‐23 21:15:46,674 DEBUG [com.splu....

Pipeline Data 1

offset @100 + 50

_raw java.lang.NumberFormatExcepSon: For in…

Pipeline Data 2

offset @150 + 80

_raw at java.lang.NumberFormatExcepSon.f…

Pipeline Data 3

offset @230 + 40

_raw at java.lang.Integer.parseInt(Integer.jav…

Pipeline Data 4

Debugging! Metrics! S.O.S App!

36

metrics.log: Queues via S.O.S

37

metrics.log: Queues via Search

38

Search: index=_internal group=queue | eval pc=(current_size_kb*100)/max_size_kb | Smechart perc90(pc) by name

metrics.log: Processor Time via S.O.S

39

metrics.log: Processor Time via Search

40

Search: index = _internal group = pipeline processor != sendout | Smechart perc90(cpu_seconds) by processor

metrics.log: Indexing Rate via S.O.S

41

metrics.log: Indexing Rate via Search

42

Search: index=_internal source=*metrics.log* group=thruput | Smechart per_second(kb)

metrics.log: Scenarios

43

!   Indexing Instance: Index Queue at 100% –  Forwarding disabled: Indexing rate? Slow disk? Full disk? –  Forwarding enabled:

Indexing rate? Slow disk on remote indexer? Full remote disk? TCPOut rate? Low network bandwidth? High network latency? Local indexing rate? Slow local disk? Full local disk?

!   Universal Forwarder: Parsing Queue at 100% –  Indexing rate? Slow disk on remote indexer? Full remote disk?

TCPOut rate? Low bandwidth? High latency? (No local indexing here)

!   Start from end, work backwards…

metrics.log: Universal Forwarder

44

!   No indexing/searching capability !   Can forward metrics to indexer…

–  May not get there! ê  Configure S.O.S: hqp://splunk-‐base.splunk.com/answers/48874/how-‐can-‐i-‐monitor-‐the-‐resource-‐usage-‐of-‐my-‐forwarder-‐using-‐the-‐sos-‐app#50315

ê  Fix forwarding: hqp://splunk-‐base.splunk.com/answers/38091/best-‐pracSces-‐to-‐deploy-‐the-‐splunk-‐on-‐splunk-‐app-‐in-‐a-‐distributed-‐search-‐environment

!   Fallback to raw file (grep!) $ grep group=queue metrics.log | grep –color 'max_size.*current_size_kb[^,]*,’

Metrics -‐ group=queue, name=typingqueue, blocked=true, max_size_kb=500, current_size_kb=499, current_size=1821, largest_size=1821, smallest_size=0

Metrics Log

45

!   Search: index=_internal source=*metrics.log*

!   Groups –  pipeline –  queue –  per_source_thruput –  per_sourcetype_thruput –  per_index_thruput –  per_host_thruput –  …

Recap

!   Splunk instance consists of linear pipelines !   Splunk topology emulates pipelines !   Downstream slowdown results in upstream blockage !   metrics.log across the topology reveals the whole picture

–  Queue sizes –  Indexing thruput –  Forwarding thruput –  CPU usage per PipelineData Processor

!   This is how you should debug – the same way we do!

QuesSons?

47

Documents

Inputs:*File,*Network,* Script,*and*more!* Splunkd:Pipelines& … · 2017-10-13 · 24 Line%Breaker% _raw* Sep*12*06:11:58*abathlmba13.no.cox.netstoreagent[49597]*:*StarSng*update*scan*

Inputs:File,Network,* Script,andmore!* Splunkd:Pipelines& … · 2017-10-13 · 24 Line%Breaker% _raw* Sep1206:11:58abathlmba13.no.cox.netstoreagent[49597]:StarSngupdatescan