Introducing the Jahia Log Analyzer

#jahiaXDC 1

Digital CSI : The Jahia Log Analyzer

Serge Huber Jahia CTO & Co-Founder

[email protected]

mailto:[email protected]

Title of Presentation I Date of Presentation 2

Introduction

Digital CSI : The Jahia Log Analyzer I #jahiaXDC 3

Have you ever…

Needed to analyze (very quickly) large amount of log file data ? Been trying to make some sense of large amount of thread dumps ? Been trying to understand exception frequency and seriousness ? Needed to understand performance issues without impacting a live production system ? Wanted to generate reports in multiple formats such as CSV, JSON or push into ElasticSearch ? Wanted to customize/extend the log analysis tool in Java ? Want a solution that won’t require sending your logs to an external company in the cloud ?


How it works


How it works

Log Files

CSV

JSON

ElasticSearch

…

Jahia Log Analyzer

Line Analyzers

Exceptions

Garbage Collection

Log line filter

…

Progressive

Stack Trace Analysis

GeoIP


Features


Features

Input processing - Can process log files or thread dump files - No impact on production servers ! - “Live” or post-mortem analysis - Single node log processing - Cluster nodes log processing - Pattern-based log line matching - Progressive analysis (BETA) - Can easily handle large files (over 800MB), they are not fully loaded in

memory


Features

Line Analyzers • Generic

• Exceptions • Log lines with level filter (>= WARN) • Thread dumps (BETA) • Garbage collection activity

• Jahia-specific • Jahia DX server-side page processing time with GeoIP resolution • Jahia DX Request Load Average • Jackrabbit Bundle Cache logs


Features

Output Writers: - CSV - JSON - ElasticSearch (embedded or remote) Stack trace analysis: (beta) - Pattern-based stack element matching - Tagging based on recognized patterns Progressive Analysis (beta) User interfaces: - Swing UI - Command line UI Kibana predefined dashboard and visualizations


User Interfaces


Swing UI


Command line interface

java -DremoteServers=localhost:9300 -jar loganalyzer/target/loganalyzer-0.5-SNAPSHOT.jar cluster-logs [INFO] LogParserConfiguration - Using default parser-config.xml file for configuration... [INFO] JahiaLogAnalyzer - Analyzing logs in cluster-logs... [INFO] ElasticSearchService - Connecting to remote ElasticSearch at addresses: [localhost:9300]... avr. 19, 2016 3:23:11 PM org.elasticsearch.plugins.PluginsService <init> INFO: [Katherine Reynolds] modules [], plugins [], sites [] [INFO] JahiaLogAnalyzer - File cluster-logs/.DS_Store already parsed previously, skipping [INFO] JahiaLogAnalyzer - File cluster-logs/jvm-app1/.DS_Store already parsed previously, skipping [INFO] JahiaLogAnalyzer - Parsing file cluster-logs/jvm-app1/catalina.out-20160301_app1... [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 132517=tcher.include(ApplicationDispatcher.java:523) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 614108=lect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 631819=ava:206) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 691268=a:206) [WARN] ExceptionLineAnalyzer - Couldn't parse first line of exception, ignoring. Line 754226=ilter.TemplateNodeFilter.prepare(TemplateNodeFilter.java:142)


Live demo


Thread Dump Line Analyzer

Processes Thread Dumps and generates - A detail entry for each thread in the dump containing

- Timestamp - Thread ID - Thread State - Stack trace, length, dashcode and stack analysis tags (BETA) - Locks information

- A global entry for each thread dump containing - Timestamp - Number of threads in dump - Potential “Stuck” threads (in same state since last dump) (BETA) - New/dead threads since last dump


Stack Trace Analysis

{"id":"jackrabbit-bundle-load-from-database","level":"NORMAL","threadState":"RUNNING","ignore":false,"linesToMatch":["\\Qatorg.apache.jackrabbit.core.util.db.ConnectionHelper.execute(ConnectionHelper.java:\\E\\d+\\Q)\\E","\\Qatorg.apache.jackrabbit.core.util.db.ConnectionHelper.reallyExec(ConnectionHelper.java:\\E\\d+\\Q)\\E","\\Qatorg.apache.jackrabbit.core.util.db.ConnectionHelper$3.call(ConnectionHelper.java:\\E\\d+\\Q)\\E","\\Qatorg.apache.jackrabbit.core.util.db.ConnectionHelper$3.call(ConnectionHelper.java:\\E\\d+\\Q)\\E","\\Qatorg.apache.jackrabbit.core.util.db.ConnectionHelper$RetryManager.doTry(ConnectionHelper.java:\\E\\d+\\Q)\\E","\\Qatorg.apache.jackrabbit.core.util.db.ConnectionHelper.exec(ConnectionHelper.java:\\E\\d+\\Q)\\E","\\Qatorg.apache.jackrabbit.core.persistence.pool.BundleDbPersistenceManager.loadBundle(BundleDbPersistenceManager.java:\\E\\d+\\Q)\\E"]}


Stack Trace Analysis, continued


Kibana integration

• All data is timestamped

• Common fields • Predefined

dashboards • Predefined

visualizations • Cluster aware


Roadmap


Project Status

Ready for usage Lots of improvements possible (modules, new line analyzers) Some features still under development: - Thread dump analyzer - Stack trace analysis - Progressive analysis Configuration is not yet as flexible as desired, needs improvements (configurable patterns for example)


Roadmap

Develop proper plugin system (OSGi?) Make it possible to quickly define new filters using regexps (no code) Improve configuration system Make it possible to use scripts (Groovy, Javascript) to develop line analyzers Migrate to Apache Karaf ?


Q & A


Thank you !


Resources

We need your help !

Don’t hesitate to get involved !

Get/fork it now here : https://github.com/Jahia/jahia-loganalyzer

Reach me at : [email protected], @sergehuber

https://github.com/Jahia/jahia-loganalyzer

mailto:[email protected]

Software

Introducing the Jahia Log Analyzer