Embed Size (px)
Citation preview
These aren't the logs you're looking for
Alicja Kwasniewska, Intel Eric Lemoine, MirantisMichal Jastrzebski, Intel
Alicja Kwasniewska Eric Lemoine Michal Jastrzebski
Agenda
KollaLogging
in dockers
Hacking Rsyslog
Central logging service
Elk or Ehk? Demo
KollaDocker containers Ansible playbooks
Kolla
● Production-ready Docker containers and deployment tools foroperating OpenStack clouds
● Both out of the box configuration and possibility for complete customization
● Images built in the kollaglue namespace for every tagged release, but you can also build images on your own
Docker containers
Deployment tools
Out of the boxconfiguration
Kollaglue
Customization
Logging in
docker containers
Docker logs● What? Anything that is written out to the container’s standard output/error● Logging capabilities available in the Docker are exposed as drivers
Default
● How to get path to this file? docker inspect -f "{{.LogPath}}" containerName● The docker logs command retrieves logs present at the time of execution.
NOTE: available only for containers with json-file and journald logging drivers
json syslog journald gelf fluentd
Hacking
rsyslog
Previous solution based on rsyslog
● Hacky● Shared /dev/log● Not meant for containers● New versions of docker breaks● Syslog doesn’t work well with tracebacks● Removed in both Liberty and Mitaka
Central logging serviceadvantages
“One Ring to rule them all, One Ring to find them...”
Which node? Which
container? Where should
I look first?
No need to search many places !
→ data stored in one location and accessed from single interface
One more second and I will find the right place in one of the
10000+ lines...
“Developer looking at production logs after a regression with downtime” oil canvas, circa 1580; source: http://classicprogrammerpaintings.tumblr.com
No need to search for one line in million !
→ interface provides filtering options and visual representation of data using graphs and charts
Kolla Central logging serviceNew feature in Mitaka
How it works?
Heka Feed data Elasticsearch View Kibana/var/lib/kolla/...
Search
Interact
HekaFeed data
/var/lib/kolla/...
Heka Feed data
/var/lib/kolla/...
HekaUnified data processing
Introduction to HekaStream processing software
● Open source● Developed by Mozilla● Used in production (by Mozilla)● Written in Go● http://hekad.readthedocs.org
Why Heka?
→“Unified Data Processing”
One tool for all the data:
● Ops data/metrics● Business data● Logging output● Error
reports/tracebacks
Ops data/ metrics
Business data
Logging output
Error reports/ tracebacks
Heka PipelinePlugins at each stage of the pipeline
Heka Highlights
● Lightweight● Flexible● Configurable● Easily and safely extendable
Lightweight Flexible
Configurable Easily and safely
extendable
ELK or EHK?
Logstash vs HekaWhy did we choose Heka over Logstash?
● Avoid running a JVM on each node● Heka is faster and more lightweight*● Heka is very flexible (plugins defined w/ code)
* https://github.com/elemoine/heka-logstash-comparison
→ EHK
ElasticsearchOpen-source text search and analytics engine
Introduction to Elasticsearch
● Open source product from Elastic group - https://github.com/elastic/elasticsearch
● Highly-scalable full-text search and analytics engine● Used by applications with complex search requirements● Written in Java
Elasticsearch Highlights
HighlyScalable
HighlyAvailable
RESTfulAPI
Full-TextSearch
Document-oriented
KibanaOpen-source analytics and visualization platform
Introduction to Kibana
● Open source product from Elastic group - https://github.com/elastic/kibana
● Written mainly in JavaScript, comes with Node.js server app ● Server component has a built-in proxy to Elasticsearch (designed to
work with Elasticsearch)
Kibana concepts
● Allows to search, view, and interact with data stored in Elasticsearch indices● Possibility to analyse and visualize data in a variety of charts, tables, and
maps
Search
Interact
View
Visualize
Analyse
Kibana - authentication
Elastic Shield - provides security for entire stack, including Kibana plugin for session support and login screen, but requires a license…
Nginx - use it as proxy with simple auth mechanism for Kibana
HAProxy - add HTTP Basic authentication with ACL that uses the http_auth criterion and http-request auth rule
ShieldFrom Elastic
GroupNginx HAProxy
HTTP Basic authentication with ACL
Deploy
EHK with Kolla
3 easy steps
Build images with one
command
Adapt configuration
files and enable central logging
Deploy Kolla with one
command
For details please watch webinar presented by Steven Dake - link provided at the last slide
Demo →
Thank you for your attentionAlicja Kwasniewska [email protected]
Eric Lemoine [email protected] Michal Jastrzebski [email protected]
Irc: #openstack-kolla Webinar: https://vimeopro.com/midokura/345kl392
