Embed Size (px)
Citation preview
LMA Toolchain
Guide to the Elasticsearch-Kibana Server
Plugin for Fuel (version 0.7.0)
1
Revision history Document purpose Key terms, acronyms and abbreviations The ElasticsearchKibana Server Plugin
Requirements Installation Guide
Install the plugin on the Fuel Master Node Plugin configuration Plugin Install Verification
User Guide Exploring the Data with Kibana
Dashboard management Filters and queries
Troubleshooting Appendix
References
2
Revision history
Version Revision date Editor Comment
0.1 03.09.2015 Simon Pasquier ([email protected])
Initial revision.
0.2 03.30.2015 Irina Povolotskaya ([email protected])
Added Document purpose and Key terms, acronyms and abbreviations sections.
1.0 04.16.2015 Simon Pasquier ([email protected])
Major version.
2.0 22.05.2015 Patrick Petit Revised document to make it consistent with InfluxDBGrafana plugin user guide
3
Document purpose This document provides instructions for installing, configuring and using ElasticsearchKibana plugin for Fuel.
Key terms, acronyms and abbreviations
Term/acronym/abbreviation Definition
LMA Collector Logging, Monitoring and Alerting Collector. A service running on each OpenStack node that collects logs and notifications.
Elasticsearch An opensource search engine built on top of Apache Lucene™, a fulltext searchengine library. Elasticsearch is written in Java and uses Lucene internally for all of its indexing and searching, but it aims to make fulltext search easy by hiding the complexities of Lucene behind a simple, coherent, RESTful API.
Kibana An open source (Apache Licensed), browser based analytics and search dashboard for Elasticsearch. Kibana is a snap to setup and start using.
The Elasticsearch-Kibana Server Plugin The LMA (Logging, Monitoring & Alerting) Toolchain is a serviceoriented monitoring solution designed for Mirantis OpenStack. It is comprised of a collection of opensource tools to help you monitor and diagnose service availability problems in your OpenStack environment at the infrastructure level. Those tools are packaged and delivered as Fuel plugins that you can install from within the graphic user interface of Fuel. The LMAToolchain plugins version 0.7 are compatible with Mirantis OpenStack version 6.1. The LMAToolchain is composed of operations support functions based on bestofbreed tools that we strived to keep to a minimum:
4
● The LMA Collector (or just collector) that is used to collect, aggregate, transform and route three types of operational data:
○ Logs and notifications which are indexed and stored as discrete events in ElasticSearch
○ Monitoring metrics which are data points organised in groups of consecutive and chronologically order lists a.k.a timeseries. Those timeseries are stored in InfluxDB
○ Metrics annotations which are produced by an anomaly detection module of the collector.
● The Events Analytics that are used to search and correlate serviceaffecting events produced by OpenStack such as notifications and services logs. This is an indispensable tool for troubleshooting errors and work with root cause analysis.
● The Metric Analytics that are used to search and display timeseries of resource usage, performance and service availability state indicators. Service availability states are annotated to provide insights as why a particular OpenStack service has changed a state. This is an indispensable tool for finding the answer to the critical question of what has changed, when and whenever possible why.
The LMA Toolchain includes Elasticsearch combined with Kibana for the Events Analytics functions. The ElasticsearchKibana plugin is installed with two preconfigured dashboards (one for logs and one for notifications). However, it is important to understand that using the plugin to install Elasticsearch and Kibana is more a convenience to get you started quickly than a requirement of the LMA Toolchain. As such, the configuration of the LMA Collector provides support to either:
● specify the hostname (or IP address) of an Elasticsearch server (or cluster of servers) that is already installed in the datacenter or,
● specify a node name with a baseos role where to install Elasticsearch and Kibana in the OpenStack environment.
ElasticSearch is a powerful opensource tools based on Lucene search engine and which when combined with Kibana makes operational data like the OpenStack log messages and notifications easy to explore and analyse. To help you get started, the plugin is preinstalled with two dashboards you can use to search and analyse the OpenStack logs and notifications. The software installed with the plugin in version 0.7.0 is as follow:
Components Version
Elasticsearch v0.9.1 for CentOS and Ubuntu (64bit)
Kibana v3.1.2 for CentOS and Ubuntu
5
Nginx Version coming by default with the Linux distribution
Requirements
Requirement Version/Comment
Fuel 6.1 or higher
Hardware It is highly recommended to use dedicated disk(s) for data storage. Otherwise, Elasticsearch will store its data on the root filesystem.
Resources consumption The amount of resources (RAM, CPU, disk) required for Elasticsearch depends on the number of clients, but a typical setup would have at least 8GB of RAM and fast disks (ideally, SSD drives).
Installation Guide As mentioned earlier, you can install Elasticsearch and Kibana using the Fuel plugin or separately in which case no ElasticsearchKibana plugin setup is required. You only need to follow the instructions to setup the LMA Collector plugin.
Install the plugin on the Fuel Master Node To install the ElasticsearchKibana plugin on the Fuel Master node follow these steps:
1. Download the plugin from the Fuel Plugins Catalog 1
2. Copy the plugin rpm file to the Fuel Master node:
[root@home ~]# scp elasticsearch_kibana-0.7-0.7.0-0.noarch.rpm root@<Fuel Master node
IP address>:/tmp
3. Log into the Fuel Master node. Install the plugin using the Fuel CLI:
[root@fuel ~]# fuel plugins --install /tmp/elasticsearch_kibana-0.7-0.7.0-0.noarch.rpm
1 Alternatively, you may want to build the rpm file of the plugin yourself if, for example, you want to modify some configuration elements. But note that this is at your own risk. Detailed instructions to build the ElasticsearchKibana plugin are provided in the README.md file of the StackForge project. See https://github.com/stackforge/fuelpluginelastisearchkibana for further details.
6
4. Verify that the plugin is installed correctly:
[root@fuel ~]# fuel plugins id | name | version | package_version
---|----------------------|---------|----------------
1 | elasticsearch_kibana | 0.7.0 | 2.0.0
To install the LMA Collector plugin, please follow instructions from the LMA Collector Plugin Guide.
Plugin configuration
To configure the ElasticsearchKibana plugin, follow these steps:
1. Create a new environment using the Fuel UI wizard. 2. Click on the Settings tab and scroll down until you reach the ElasticsearchKibana
plugin section 3. Select the ElasticsearchKibana Plugin checkbox and fill in the configuration
parameters as shown in the figure below:
a. The only required field that you must fill in is the Node name field. All the other
fields are either optional or are associated with a default value. You should enter in the Node name field the name of the node Elasticsearch and Kibana will be installed during the deployment. For example elasticsearch. See node setup instructions below.
b. Optionally, you can specify up to three disks where the Elasticsearch databases will be created. The plugin will automatically create a logical volume on top of those disks. You can also specify an allocation size in the Size field if you want to partition the disk. If no size is specified, then the plugin will create a
7
logical volume taking all the available disk space. If no disk is specified at all, then the Elasticsearch databases will be created on the root filesystem.
4. Select the Logging, Monitoring and Alerting (LMA) Collector Plugin checkbox and fill in the “Events Analytics” section as shown in the figure below. Please check the Collector Plugin User Guide for further instructions about the LMA Collector.
a. You can optionally select the OpenStack Notifications checkbox if you want to include the OpenStack services notifications.
5. Click Save Settings at the bottom of the page to save the configuration parameters 6. Then switch to the Nodes tab to add a node with the baseos role. 7. Before deploying any changes edit the name of the baseos node by clicking on
"Untitled (xx:yy)" and modify it to "elasticsearch".
8. Scroll down the “Networks” page and verify your networks. 9. Then finally deploy your changes.
Plugin Install Verification
Depending on the number of nodes and setup complexity, deploying a new Mirantis OpenStack environment typically may take between 30 minutes to several hours. Once your environment has been successfully deployed, you should receive in the Fuel UI a notification which looks something like this.
8
A current limitation of the current Fuel plugin framework is that it is not possible to display notifications containing dynamic information such as the URL of the Grafana server. This limitation will be fixed in a next release of Mirantis OpenStack but meanwhile you will have to retrieve that information using the fuel command line. [root@fuel ~]# fuel nodes id | status | name | cluster | ip | […] | roles ---|--------|------------------|---------|-----------| |------
1 | ready | elasticsearch | 1 | 10.20.0.3 | | base-os 6 | ready | Untitled (55:b3) | 1 | 10.20.0.7 | | ceph-osd, compute
3 | ready | Untitled (73:19) | 1 | 10.20.0.5 | | controller 4 | ready | Untitled (1c:5c) | 1 | 10.20.0.6 | | controller 5 | ready | Untitled (75:35) | 1 | 10.20.0.8 | | ceph-osd, compute 2 | ready | Untitled (ec:a2) | 1 | 10.20.0.4 | | controller The above command line output tells you that the ElasticsearchKibana server has been deployed on node1 with the IP address 10.20.0.3. To verify that Elasticsearch is running properly, use the following steps. [root@fuel ~]# curl http://10.20.0.3:9200/ The expected output should look like something like this:
{
"status" : 200,
"name" : "node-1",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "1.4.4",
"build_hash" : "c88f77ffc81301dfa9dfd81ca2232f09588bd512",
"build_timestamp" : "2015-02-19T13:05:36Z",
"build_snapshot" : false,
"lucene_version" : "4.10.3"
},
"tagline" : "You Know, for Search"
}
9
User Guide
Exploring the Data with Kibana The Kibana user interface runs on port 80. Pointing your browser to the URL http://10.20.0.3:80/ you should see a screen like this:
The Kibana dashboard is divided into 4 main sections:
1. A timepicker control section that lets you choose the time period you want to select and refresh frequency
2. A query and filter section where all the filters are displayed 3. A log analytics row which contains 4 panels:
a. The number of log messages for the chosen time period b. The top 10 hosts c. The top 10 log sources d. The number of log messages grouped by severity
4. A table of log messages sorted in chronological order
Dashboard management
10
There are 2 dashboards available:
● The Logs Dashboard that is the Kibana Home Dashboard for viewing the log messages.
● The Notifications Dashboard for viewing the OpenStack notifications if you enabled this option in the LMA Collector settings.
You can switch from one dashboard to another by clicking on the topright Load icon in the toolbar and select the requested dashboard.
You can modify the dashboard as you like and save it using the topright Save icon in the toolbar. You can also save a copy of a modified dashboard by changing its name.
Filters and queries Filters and queries have similar syntax but they are used for different purposes:
● Filters are used to restrict the data that is displayed in the dashboard ● Queries are used for freetext search. You can also combine multiple queries and
compare their results. To further filter the log messages based, for example, on the deployment_id, you need to expand a log entry and then select the deployment_id field by clicking on the magnifying glass icon as shown below.
11
This will apply a new filter in the dashboard as shown below. This would work for any field that has been indexed in the log entry.
12
Filter and query conditions can also contain wildcards. The wildcard can be requested according to the following template: programname: <name>*. For example, to display only Nova logs in the dashboard, you should enter programname:nova* in the query textbox:
13
You can also specify multiple queries to compare different data sets. To add a new query, click on the + sign at the rightend of the query textbox and enter a new search query.
14
Troubleshooting If you get no data in the dasboards, use the instructions below to troubleshoot the problem:
1. Check if the LMA collector service is up and running: # On CentOS
/etc/init.d/lma_collector status
# On Ubuntu
status lma_collector
2. If the LMA Collector is down, start it again: # On CentOS
/etc/init.d/lma_collector start
# On Ubuntu
15
start lma_collector 3. Look for errors in the LMA collector log file (located at /var/log/lma_collector.log) on
the different nodes. 4. Check if the nodes are able to connect to the Elasticsearch server on port 9200.
Appendix
References ● the official Elasticsearch documentation ● the official Hardware recommendations for Elasticsearch ● the official Kibana documentation
16
