1

Generic logging layer for the distributed computing

by

Gene Van BurenValeri Fine

Jerome Lauret

2

Logging API for HENP application• Most HENP experiment software includes a

logging or tracing API allowing for displaying in a particular format important feedback coming from the core application.

• However, inserting log statements into the code is a low-tech method for tracing the program execution flow and often leads to a flood of messages in which the relevant ones are occluded.

• In a distributed computing environment, accessing the information via a log-file is no longer applicable and the approach fails to provide runtime tracing

3

Practical Grid case example• A “job” is often a complex workflow or Meta-job

– A Meta-Job can be composed of step1 = several sub-jobs running in parallel, the result collected and N jobs in step2 process the result etc ….

– We want to have a consistent picture of the Meta-Job (not only individual ones)

• A given job may be re-directed elsewhere (pre-emption, re-submit)

• An individual job log may warrant the triggering of some action (abort the entire task or Meta-job, recovery service take action, workflow is altered etc …)

• To accommodate for the needed components, we first need to provide the necessary tools to collect the information => logger

4

Consistency• Running a job involves a chain of events where

many components are involved often written in diverse languages and not offering a consistent and easily adaptable interface for logging important events.

• Meta-Job requires a Meta-log– Workflow should be represented– All steps may be inspected in details (individual steps)– An overview of the a global log (Meta-log) MUST be

possible

5

STAR meta-job

ST

AR

GR

ID jo

b sc

hedu

ler

GRID realm

GRID jobs Job log file

Met

a-lo

g of

met

a-jo

b

Job log fileGRID jobs

workflow

6

The job steps …• Job initiation

– – local OS shell script– Job generation/ submission – Java code (for example, STAR

Unified Meta-Scheduler, SUMS)• Job initialization at the remote node – Shell script• Application (user) job

– Starting, application preparation – C++, Fortran, …– Running the application itself – C++, Fortran, …– Job finishing (from the application user point of view) –

C++/Fortran/etc• Job normal completion

– transferring the log files and clean up – Shell script, Services (SRM)

– file registration, …• job abnormal completion at remote site – Perl script, …• job abnormal completion at submitting side – Java code,

…• … etc …

7

Jakarta log4j project

• An approach based on a new generic layer built on top of a logger family derived from the Jakarta log4j project that includes log4cxx, log4c, log4perl packages may provide a consistency across packages and framework

• A community supported project

8

Main components

• Log4cxx has 3 main components:– loggers, – appenders and– layouts.

• These three types of components work together to enable developers to log messages according to message type and level, and to control at runtime how these messages are formatted and where they are reported.

9

STAR Scheduler

Log4cxx.xml

Current job logging

configuration

Production Manager

Job

Abstract STAR loggerAbstract STAR logger

Abstract STAR loggerAbstract STAR logger

List of appendersList of layouts

Hierarchy of the loggers

“Console” Appender

“HTML” Appender

“MySQL Appender

“File” Appender “Custom” Appender

10

Event severity & verbosity• The first and foremost advantage of any logging

API over plain stdout resides – in its ability to disable certain log statements while

allowing others to print unhindered. – This capability assumes that the logging space, that

is, the space of all possible logging statements, is categorized according to some developer-chosen criteria

– Repetitive events can be filtered

• Selection of log event severity or verbosity level

11

Run-time log configuration• Additionally, the power of using log4j design, is the

possibility to enable logging (or features) at runtime without modifying the application binary or the wrapper layers.

• The picture shows a dedicated layer implemented as C++ abstract class library that serves as a proxy between the application framework and the distributed environment.

• The approach is designed so that the debugging statements can remain in shipped code without incurring a heavy performance cost.

• Logging equips the developer with as detailed context as necessary for application failures, from testing, quality assurance to a production mode limited amount of information.

12

STAR logger configuration file<?xml version="1.0" encoding="UTF-8" ?><!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration debug="true" xmlns:log4j="http://jakarta.apache.org/log4j/">

<appender name="D1" class="org.apache.log4j.ConsoleAppender"> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%-3c{2}:%-5p - %m%n"/> </layout> </appender>

<appender name="D2" class="org.apache.log4j.FileAppender"> <param name="File" value="temp.html" /> <layout class="org.apache.log4j.HTMLLayout"> <param name="ConversionPattern" value="%x: %-5p %c{2} - %m%n"/> </layout> </appender>

<appender name="MYSQL" class="org.apache.log4j.MySQLAppender"> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="INSERT INTO StarLogger VALUES (&quot;%-3c{2}:%-5p %d -

%m&quot;);"/> </layout> </appender>

</appender>

<root> <priority value ="DEBUG" /> <appender-ref ref="D1" /> <appender-ref ref="D2" /> <appender-ref ref="D3" /> <appender-ref ref="MYSQL" /> </root>

</log4j:configuration>

13

• The implementation of the STAR logger layer via log4cxx API allows to independently set the level of the verbosity for each STAR production “module”. It is convenient for the debug purposes of some particular piece of the code. Each developer knows his/ her module and can set the message level” for that module.

• Capability to include the special appenders like “MySQL”and “HTML” for certain critical points does allow to get the job status on-line (via remote Db or Web page) before the GRID deliveries the entire usually huge log file to dig through.

• If the needs of the special treatment arisen the custom dynamically loaded appender filter can be deployed also.