Centralized Logfile Search (a.k.a. Tracing)

Vito Baggioliniwith Gergo Horanyi, Felix Ehm, Stephen Page

Vito Baggiolini 2

Outline

• Introduction• Architecture and current logfile sources (FECs)• Kibana, the search GUI• Demo• Next steps

16/05/2014

Vito Baggiolini 3

Introduction

• Purpose: Facilitate diagnostics of the operational control system– No need to know where exactly logfiles are stored– Make logfile history easy to search and correlate– Search logfiles across all levels (Java, CMW, FESA, Drivers, OS, …)

• Scope:– For operations, not for development (maybe later)– Mainly to search history, not for live logging (-> use cmw-admin or tail -f)– To search many systems at once, less suitable to for individual systems (better to

go the log file directly)• Implementation:

– Before: $plunk, very intuitive and user friendly, commercial, closed-source, – Now: ElasticSearch + Kibana, free, open-source, less intuitive + user friendly

16/05/2014

Vito Baggiolini 4

Current Architecture and log sources

SyslogOS

SyslogFESA2

cmw-logFESA3

cmw-logCMW

NFSfgcd-log Sources

Java ProcessLogstash1 Logstash2Parse to

key-value

ElasticSearch

Store and search

Kibana User Interface

16/05/2014

Vito Baggiolini 5

Query Bar

Filters

Fields

Results vs Time

Detailed Messages

To select Time Window

16/05/2014

Kibana

Vito Baggiolini 616/05/2014

Query Bar

Filters

To select Time Window

To share searches

Vito Baggiolini 716/05/2014

Fields

Results vs Time

Detailed Messages

Click on field-> top values

Filter out repeated

Vito Baggiolini 8

Kibana Demo

• http://logs.cern.ch• Select time window, zoom in, zoom out• Search

– Enter query for keyword– Enter query for a field

• Click on fields to see top values and filter• Filter out repeated messages• Modify existing filters• Display results as tables• Share results with others16/05/2014

Vito Baggiolini 9

Demo…

16/05/2014

Vito Baggiolini 10

How to make the best of Kibana• Do a simple Search (e.g. for *) and filter• Click on field names (not on boxes) to see top values)

– Top values only over last 500!– Focus on or exclude a given event

16/05/2014

Vito Baggiolini 11

How to make the best of Kibana• Do a simple Search (e.g. for system:FESA3) and then use filters• Click on field names (not on boxes) to see top values)

– Top values only over last 500!– Focus on or exclude a given event

• Play with Filters – E.g. edit existing filters

• Share temporary views with others• Don’t yet invest too much in Dashboards => fix displays(?)• Use Lucene Query Language for advanced queries and filters

16/05/2014

Vito Baggiolini 12

Next steps• Make system operational (now in BETA)

– Verify scalability– Add redundancy where feasible– Add sanity checks to verify proper functioning

• Make sure log sources and are parsed correctly to key/value– Finalize standardization of fields pushed into Elastic Search – https://wikis.cern.ch/pages/viewpage.action?pageId=73041150

• Add Java logging input (gradually, servers first)

16/05/2014