Upload
hathu
View
223
Download
3
Embed Size (px)
Citation preview
‹#›
Unifying logs and metrics data with Elastic BeatsMonica Sarbu Team lead, Elastic Beats
Who am I
2
https://www.flickr.com/photos/ofernandezberrios/7176474422
• Team lead at Elastic Beats
• Software engineer
• Joined Elastic 1 year ago
@monicasarbu
http://github.com/monicasarbu
Beats are lightweight shippers that collect and
ship all kinds of operational data to Elasticsearch
3
Beats are lightweight shippers that collect and
ship all kinds of operational data to Elasticsearch
4
5
Lightweight shippers
• Lightweight application
• Written in Golang
• Install as agent on your servers
• No runtime dependencies
• Single purpose
Beats are lightweight shippers that collect and
ship all kinds of operational data to Elasticsearch
6
7
All kinds of operational data
• Filebeat • collects logs
• Winlogbeat • collects Windows event logs
• Packetbeat • collects insides from the
network packets
not released
• Topbeat • collects system statistics like
CPU usage, disk usage, memory usage per process, etc
• Metricbeat • collects metrics by
interrogating periodically external services
Beats are lightweight shippers that collect and
ship all kinds of operational data to Elasticsearch
8
‹#›
In Elasticsearch .. you are storing the raw value … You have the
ability to ask and answer questions that you didn’t think about when
the data was stored!
Felix Barnsteiner
The Elastic Stack
10
‹#›
Captures insights from network packets
12
Packetbeat
Sniffing the network traffic
13
• Copy traffic at OS or hardware level
• ZERO latency overhead
• Not in the request/response path, cannot break your application
Client
Server
sniff sniff
14
Sniffing use cases
• Security
• Intrusion Detection Systems
• Troubleshooting network issues
• Troubleshooting applications
• Performance analysis
Monitor the network traffic with OSS tools
15
1 2 3 4ssh to each of your
serverstart a trace using tcpdump on each
of your server
download trace from each server to a common location
merge all traces
5analyze it with
Wireshark
The Problem
16
1 2 3 you have lots of
serverschallenging to see
the traffic exchanged
between your servers
Packetbeat makes it easy
Packetbeat overviewIt does all of this in real time directly on the target servers
17
1 2 3 4capture network
trafficdecodes network
trafficcorrelates request & response into
transactions
extract measurements
5send data to Elasticsearch
Packetbeat: Available decoders
18
HTTP
MySQL
PostgreSQL MongoDB (community)
Memcache
ICMP (community) + Add your own
Thrift-RPC DNS (community)
Redis
AMQP (community)
NFS (community)
Packetbeat: Configuration
19
# Network interfaces where to sniff the datainterfaces: device: any
# Specify the type of your network dataprotocols: dns: ports: [53] http: ports: [80, 8080, 8081, 5000, 8002] mysql: ports: [3306] …
20
21
22
23
24
25
26
27
28
29
30
‹#› 31
Packetbeat flows• Look into data for which we don’t
understand the application layer protocol
• TLS
• Protocols we don’t yet support
• Get data about IP / TCP / UDP layers
• number of packets
• retransmissions
• inter-arrival time
flows: # network flow timeout timeout: 30s
# reporting period period: 10s
32
33
34
Collects log lines
35
Filebeat
36
Filebeat overview
• Simple log forwarder that sends the log lines to Elasticsearch
• Successor of Logstash Forwarder
• It remembers how far it read, so it never loses log line
• Reads the log files line by line
• It doesn’t parse the log lines!
Filebeat: Parse logs with Logstash
37
• Filebeat sends out unparsed log lines
• Use filters like Grok, mutate, geoip to parse the log lines
• Combine the filters with conditionals or create custom filters in ruby
• Forward data to other systems using the Logstash output plugins
Filebeat
Elasticsearch
Logstash
Other systems
Filebeat: Parse logs with Ingest Node
38
• Ingest node plugin is available starting with Elasticsearch 5.0.0-alpha1
• Filebeat sends out unparsed log lines directly to Elasticsearch
• Use Ingest Node processors to parse the log lines
• Easier to setup
Filebeat
Elasticsearch
Filebeat: ConfigurationConfigure prospectors to forward the log lines
39
filebeat: # List of prospectors to fetch data. prospectors:
# Type of files: log or stdin - input_type: log
# Files that should be crawled and fetched. paths: - “/var/log/apache2/*”
# File encoding: plain, utf-8, big5, gb18030, … encoding: plain
40
41
‹#›
Multiline
42
multiline: # Sticks together all lines # that don’t start with a [ pattern: ^\[ negate: true match: after
Filebeat extra power
• Sticks together related log lines in a single event
• For all those long exceptions
• Can also be done by Logstash, but it’s sometimes easier to configure the patterns closer to the source
43
‹#›
‹#›
45
json: keys_under_root: false message_key: “message” overwrite_keys: false add_error_key: false
Filebeat extra power JSON logs
• application logs in JSON format
• you don’t have to choose what data to include in the log line
• don’t need to use grok filters from Logstash to parse the application logs
46
47
‹#›
Basic filtering
48
# Only send lines starting with# ERR or WARNinclude_lines: [“^ERR”, “^WARN”]
# Exclude lines containing # a keywordexclude_lines: [“Request received”]
# Exclude files all togetherexclude_files: [“.gz$”]
Filebeat extra power
• Because removing stuff at the source is more efficient
• Flexible Whitelist + Blacklist regexp log line filtering
• Efficient log files filtering (excluded files are never opened)
• Works on multiline too
Collects Windows Event logs
49
Winlogbeat
50
Winlogbeat overview
• Sends out unparsed Windows event logs
• Remembers how far it read, so it never loses any Windows event logs
• Use Ingest Node or Logstash to parse the Windows event logs
Winlogbeat: ConfigurationSpecify the event logs that you want to monitor
51
winlogbeat: #list of event logs to monitor event_logs: - name: Application - name: Security - name: System
52
Collects system statistics
53
Topbeat
54
Topbeat overview
• Like the Unix top command but instead of printing the system statistics on the screen it sends them periodically to Elasticsearch
• Works also on Windows
Topbeat: Exported data
55
• system load • total CPU usage • CPU usage per core • Swap, memory usage
System wide
• state • name • command line • pid • CPU usage • memory usage
Per process
• available disks • used, free space • mounted points
Disk usage
Topbeat configurationSpecify the system statistics that you want to monitor
56
topbeat: # how often to send system statistics period: 10
# specify the processes to monitor procs: [".*"]
# Statistics to collect (all enabled by default) stats: system: true process: true filesystem: true
57
‹#›
‹#›
‹#›
‹#›
‹#›
‹#›
‹#›
‹#›
Collects periodically metrics from external systems.
66
Metricbeat in progress
Metricbeat: how it works
67
1 2 3Periodically polls monitoring APIs
of various services
Groups performance
data into documents
Ships them to Elasticsearch
Metricbeat: A module for each metric type
68
Metricbeat
apache module
mysql module
redis module
system module +
69
Metricbeat: It is also a library!
• Use the Metricbeat infrastructure, to create a standalone Beat
• You can create a Beat with a single module that exports your custom data
• Can use the built in Metricbeat modules
Metricbeat
df module
github.com/ruflin/df2beat
Metricbeat module vs standalone Beat
70
• Contributed via PR to the elastic/beats Github repository
• Officially supported
• Supports common systems
• Docker based integration tests
Metricbeat module
• In a separate Github repository
• Supported by the community
• Supports specialized systems
• Optional Docker based integration tests
Standalone Beat
Provide a platform to make it easier to build custom Beats
on top of it
71
Beats platform
72
Beat 1
libbeat
Beat 2 Beat 3 +
libbeat
73
• Written in Go
• Provide common functionality for reading configuration files, for handling CLI arguments, for logging
• Makes sure reliably send the data out
• Provide things like encryption, authentication with certificates
• Has support for different outputs: Elasticsearch, Logstash, Redis, Kafka
libbeat
Outputs
‹#›
Community Beats
Community Beats
75
libbeat
Community Beats
Elastic Beats
Collect, Parse & Ship
• Standalone projects
• Written in Go
• Use libbeat
• Concentrate only on collecting the data
• Solve a specific use case
Official vs Community Beats
76
• In the elastic/beats Github repository
• Officially supported
• Synced releases with the whole stack
Official Beats
• In another Github repository
• Supported by the community
• Releases at any time
Community Beats
77
1 Apachebeat
2 Dockerbeat
3 Elasticbeat
4 Execbeat
5 Factbeat
6 Hsbeat
20COMMUNITY
BEATS Sending all sorts of
data to Elasticsearch
7 Httpbeat
8 Nagioscheckbeat
9 Nginxbeat
10 Phpfpmbeat
11 Pingbeat
13 Unifiedbeat
12 Redisbeat
14 Uwsgibeat
15 Flowbeat
16 Lmsensorsbeat
17 Twitterbeat
18 Upbeat
19 Wmibeat
20 Packagebeat
‹#› 78
input: # Loop every 5 seconds period: 5 # Use raw sockets for ping # Requires root! privileged: true # Whether to perform IPv4/v6 pings useipv4: true useipv6: false
# List targets under the tag # you want assigned to targets: # tag: google google: - google.com.au - google.com
You know, for pings• Sends ICMP (v4 or v6) pings
periodically to a list of hosts
• Can send also UDP pings (no root required)
• Resolves DNS
• Records RTT
Pingbeat
Pingbeat output
79
{ "@timestamp": "2016-02-08T11:02:22.675Z", "beat": { "hostname": "Tudors-MBP", "name": "Tudors-MBP" }, "count": 1, "rtt": 25.336089, "tag": "google", "target_addr": "216.58.213.227", "target_name": "google.com.au", "type": "pingbeat"}
‹#›
80
Execbeat
execbeat: execs: # Each - Commands to execute. - # Cron expression # Default is every 1 minute. cron: "@every 10s"
# The command to execute command: echo args: "Hello World" document_type: jolokia
fields: host: test2
• Accepts cron expressions
• Sends stdout and stderr to Elastic search
• Use Logstash and Grok to further parse the output
Run any command
Execbeat output
81
{ "@timestamp": "2016-02-08T11:59:36.007Z", "beat": { "hostname": "Tudors-MBP", "name": "Tudors-MBP" }, "exec": { "command": "echo", "stdout": "Hello World\n" }, "fields": { "host": "test2" }, "type": "jolokia"}
‹#› 82
Dockerbeat
Docker Monitoring• Uses the Docker API
• Exports per container stats about:
• CPU
• Memory
• Disk
• Network
• IO access
• Log
input: # In seconds, defines how often to # read server statistics period: 5
# Define the docker socket path # By default, this will get the # unix:///var/run/docker.sock socket:
Dockerbeat output
83
{ "@timestamp": "2016-02-08T12:44:56.136Z", "containerID": "17021c571d69fe4e93ee395b129c0f073d8aed6d618c9d0d805f68e0b66b2c3f", "containerName": "kibana", "memory": { "failcnt": 0, "limit": 1044586496, "maxUsage": 68485120, "usage": 9732096, "usage_p": 0.009316697121077851 }, "type": "memory"}
‹#› 84
Nagioscheckbeat
Run Nagios checks• Can execute any Nagios plugin
• Execution period configurable per check
• Sends alerts (Warning/Critical) to Elasticsearch
• Sends performance data to Elasticsearch
input: checks: - name: "disks" cmd: "plugins/check_disk" args: "-w 80 -c 90 -x /dev" period: "1h" - name: "load" cmd: "plugins/check_load" args: "-w 5 -c 10" period: "1m"
Nagioscheckbeat output
85
{ "@timestamp": "2015-12-30T18:56:33.933Z", "args": "-w 5 -c 10", "cmd": "/usr/lib64/nagios/plugins/check_load", "count": 1, "message": "OK - load average: 0.16, 0.05, 0.06", "status": "OK", "took_ms": 14, "type": "nagioscheck"}
Provide a platform to make it easier to build custom Beats
on top of it
86
Beat generatorGenerate the boilerplate code for you
87
$ pip install cookiecutter
$ cookiecutter https://github.com/elastic/beat-generator.git
project_name [Examplebeat]: Mybeatgithub_name [your-github-name]: monicasarbubeat [examplebeat]: mybeatbeat_path [github.com/your-github-name]: github.com/monicasarbufull_name [Firstname Lastname]: Monica Sarbu
88
Beats Packer
• Cross-compiles to all our supported platforms
• Produces RPMs, DEBs,
• Same tools that we use to build the official Elastic Beats
• Can be executed from Travis CI
Multiple data types, one view in Kibana
89
• metrics
• flows
• logs
• system stats
• transactions
• transactions
• metrics
• metrics
• logs • logs
• system stats
• flows• flows
• metrics
• logs
Monitor MySQL with Elastic Stack
90
Metricbeat
mysql …
Filebeat
log …
Packetbeat
mysql …
Elasticsearch
Kibanastats queries
slow queries
Monitor web server with Elastic Stack
91
Metricbeat
mysql apache
Filebeat
log …
Packetbeat
mysql http
Elasticsearch
Kibanamysql & apache stats queries & HTTP transactions
slow queries apache logs
‹#›
Thank you
‹#›
93
Want to hear more about Logstash?
Don’t miss Ingest Logs with Style by Pere Urbon-Bayes
Thursday 12:00pm - 1:00pm in MOA 05
‹#›
Q&A
Find us on:
• github.com/elastic/beats • discuss.elastic.co • @elastic #elasticbeats • #beats on freenode
Or Here. In Real Life!
‹#›
Please attribute Elastic with a link to elastic.co
Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/
Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries.
Third party marks and brands are the property of their respective holders.
95