SEAL: Core Libraries and Services Project

SEAL: Core Libraries and Services Project

CERN/IT After-C5 Meeting6 June 2003P. Mato / CERN

6 June 2003 SEAL - Core Libraries and Services ProjectP. Mato/CERN2

Contents

History: The blueprint RTAG Project Aims Current Work Areas:

– Foundation Libraries– Math Libraries– Plugin Management– LCG Dictionary– Scripting services

Software Process Project Status and Summary


The Blueprint RTAG The mandate of the RTAG was to define the architectural ‘blueprint’

for LCG applications:– Define the main architectural domains (‘collaborating frameworks’) of

LHC experiments and identify their principal components. – Define the architectural relationships between these ‘frameworks’ and

components, including Grid aspects, identify the main requirements for their inter-communication, and suggest possible first implementations.

– Identify the high-level milestones for each domain and provide a first estimate of the effort needed.

– Derive a set of requirements for the LCG Any piece of software developed by any LCG common project must

conform to a coherent overall architectural vision The end goal is the integration of LCG and non-LCG software to

build coherent applications The blueprint will be established in terms of a set of requirements,

suggested approaches and guidelines, and recommendations


Architecture requirements Long lifetime: technology evolution Languages: C++ today; allow multi-language and evolution Distributed applications TGV and airplane work Modularity of components Component communication via public interfaces Interchangeability of implementations Integration into coherent framework Design for end-user’s convenience more than the developer’s Re-use existing implementations Software quality at least as good as any LHC experiment Meet performance, quality requirements of trigger/DAQ software Platforms Trigger/DAQ environment


Architectural Elements

Interface model– Abstract interfaces, versioning,

guidelines,… Component model

– Communication via public interfaces (no hidden channels)

– Plug-ins (run-time loading)– Life-time management (reference

counting)– Configuration

Design guidelines– Dependencies– Exception handling– Interface to external components

Basic Framework

Foundation Libraries

Simulation Framework

Reconstruction Framework

Visualization Framework

Applications

. . .

Optional Libraries

OtherFrameworks

Software Structure


Architectural Elements (2)

Object Dictionary– The ability to query a class about its internal structure

(Introspection)– Essential for data browsing, rapid prototyping, persistency, etc.

Object Whiteboard– Uniform access to

application-defined objects (equivalent to the Gaudi transient stores)

Component Bus– Integration of components

providing a wide variety of functionality

GUI

Python

mathmathshell

PyGaudi

DatabaseEDG APIGUI

XML

GaudiFramework

PyROOT

RootClasses

PVSS

JPE

JavaClasses


The Blueprint RTAG Recommendations

RTAG establishes a user/provider relationship between LCG software an ROOT– LGC software will not be based on ROOT, it will use ROOT

where adequate Start common project on core tools and services

SEAL Project Start common project on physics interfaces Start RTAG on analysis, including distributed

aspects Tool/technology recommendations

– CLHEP, CINT, Python, Qt, AIDA, … Develop a clear process for adopting third party

software


SEAL Overview

SEAL aims to– Provide the software infrastructure, basic frameworks, libraries

and tools that are common among the LHC experiments– Select, integrate, develop and support foundation and utility

class libraries– Develop a coherent set of basic framework services to facilitate

the integration of LCG and non - LCG software Scope

– Foundation Class Libraries» Basic types (STL, Boost, CLHEP, …), utility libraries, system

isolation libraries, domain specific foundation libraries – Basic Framework Services

» Component model, reflection, plugin management, incident (event) management, distributed computing, grid, scripting


Domain Coverage

EventGeneration

Core Services

Dictionary

Whiteboard

Foundation and Utility Libraries

DetectorSimulation

Engine

Persistency

StoreMgr

Reconstruction

Algorithms

Geometry Event Model

GridServices

I nteractiveServices

Modeler

GUIAnalysis

EvtGen

Calibration

Scheduler

Fitter

PluginMgr

Monitor

NTuple

Scripting

FileCatalog

ROOT GEANT4 DataGrid Python Qt

Monitor

. . .MySQLFLUKA

EventGeneration

Core Services

Dictionary

Whiteboard


DetectorSimulation

Engine

Persistency

StoreMgr

Reconstruction

Algorithms

Geometry Event Model

GridServices

I nteractiveServices

Modeler

GUIAnalysis

EvtGen

Calibration

Scheduler

Fitter

PluginMgr

Monitor

NTuple

Scripting

FileCatalog

ROOT GEANT4 DataGrid Python Qt

Monitor

. . .MySQLFLUKA


Assumptions, constraints, risks

Do not re-invent the wheel – Most of the core software to be delivered by SEAL exists -

more or less - in experiments’ core software» We will re-use as much as possible existing software» Most of the work will be in re- packaging existing pieces of

software If wheel squeaks…

– Develop / adapt / generalize in order to achieve the necessary level of coherency and conformance to the architectural vision already established

Adopt a Seal – In order to use SEAL, projects will need to replace their own

software elements with SEAL functionally equivalent ones. This will certainly imply some period of instability for the experiment applications


Project Work Packages


Math Libraries

Component Model and Plug-in Manager

LCG Object Dictionary

Basic Framework Services

Scripting Services

Grid Services

Education and Documentation

New work package added to the project by incorporating the MathLib project (F. James et al.)

This work package will probably be split between “Foundation and Utility Libraries” and “Basic Framework services” later

Not yet started working in this area


SEAL Versions Road Map

Release Date Status Description (goals)

V 0.1.0 14/02/03 internal Establish dependency between POOL and SEALDictionary generation from header files

V 0.2.0 31/03/03 public Essential functionality sufficient for the other existing LCG projects (POOL)Foundation library, system abstraction, etc.Plugin management

V 0.3.0 16/05/03 internal Improve functionality required by POOLBasic framework base classes

V 1.0.0 30/06/03 public Essential functionality sufficient to be adopted by experimentsCollection of basic framework servicesScripting support

Released 04/04/03Released 14&26/02/03

Released 23/05/03


1. Foundation and Utility Libraries

Inventory of existing utility classes Support for Boost library

– Open source utility library (parts of which are destined to be included in the following C++ standard)

Participation to CLHEP project Adopted CMS classlib

– A set of foundation and utility classes mainly used in Iguana (CMS)

Development of SealUtil and SealKernel packages– The idea is to develop SEAL utility and system isolation library

complementary to Boost, CLHEP and STL Guidelines for selecting external libraries

– Set of criteria to define


2. Math Libraries GSL (Gnu Scientific Library) evaluation

– How it compares with NagC– Automatic testing of GSL (with cppUnit) as part of every SEAL

release Re-implementation of MINUIT in C++

– Prototype available in current SEAL release – Migrad and Minos available – The numerical results of the two prototypes compared to the

Fortran version. Compatible within the errors. Other libraries

– MathLibs/GSLAlgebra. Prototype wrapper around GSL. Not complete but restricted to Minuit needs.

Provide to experiments with math and statistics libraries to be used in analysis, reconstruction, simulation. – GSL support, …


3. Component Model and Plug-in Manager

Plug-in Management– Service in charge of managing, querying, [un]loading plug-

ins Implementation in two levels

– Lower level (no framework constraints)» Ability to create “module” libraries that contain “plug-ins”» Ability to instantiate concrete implementations knowing

the plug-in category and the concrete type» Caching information about what modules contain what

plugins and their categories – Plugin instances management (assuming a given framework

model)» Locate other plug-ins and manage their lifetime» Framework base classes to obey plug-in/framework model


3. Component Model and Plug-in Manager(2)

Howto’s available– Host modules in your project– Create a new plugin factory– Create a new plugin module library

Plug-InCachePlug-InCache

ObjectFactory

ObjectFactoryPlug-in

DatabasePlug-InCache

ModuleModuleModuleModuleModule ObjectFactory

Attached

Unattached

// Module DefinitionDEFINE_SEAL_MODULE();DEFINE_SEAL_PLUGIN(SomeFactory, MyObject);

// Usage by a clientBaseObject* p;p = SomeFactory::get()->create(“MyObject”);


4. Object Dictionary

Adding “introspection/reflection” capabilities to C++ Reflection packages

– The Dictionary/Reflection package provides the capability to introspect and interact with any C++ object at run-time

– The Dictionary/ReflectionBuilder package is the “write” interface of the reflection information. Typically generated code uses this interface to built the reflection information at run-time

Dictionary generation– Main goal: Full support of C++ without any class instrumentation– DictionaryGenerator package provides the command to do it

Usages so far– Object persistency (POOL)– Interactivity and scripting (Python)


Dictionary generation from header files– Uses gccxml (0.4.0) to parse header-files (extension to

gcc-compiler), generates intermediate XML file with dictionary information

– XML file parsed by a python script which generates dictionary building C++ code (for “selected classes”)

– Compiled to shared library and loaded at run-time to create dictionary in memory

4. Object Dictionary (2)

.xml

.h

.h lcgdict.h.h

gccxml .xml filter+

exten

gendictparser

_dict.cpp .somake

selection file

(python script)

#include files


LCG Object Dictionary: Usage

ROOT I/O

LCGDictionaryCINT

DictStreamer

.h

in

out

ROOTCINT

CINT generatedcode Dict generating

code

.adl.xml

ADL/GOD

OtherClients:

(python, GUI, etc.)

LCG

to C

INT

Dic

t gate

way

(2)

(1)

Population

Conversion

Reflection

GCC-XML


5. Basic Framework Services

Developing the basic framework– Component model definition

» component identification, configuration, lifetime strategy, interface model, etc.

– Started to review existing designs (Gaudi, Cobra, Iguana,…) Basic Component Model

– Set of “base classes” and interfaces that developers will use to build their components

Developing Basic Common Services– Started design and implementation for the message reporting

service– Other services like component configuration, “event”

management, object white board, etc. will continue soon after


5. Basic Framework Services (2)

The current ideas– Give to the “users” (framework

developers) a number pre-defined service base classes (from simple to more complex)

– Developers should “model” their services using one of the proposed base classes

– Standard functionality (instantiation, configuration, lifetime management, etc.) should be provided by these bases classes

Service

MyService

Context

PropMgr

BServLetAServLet

Client NameSvc


6. Scripting Services

Scripting needed for Interactivity, Configuration, Rapid prototyping, Integration, etc.

GOAL: Bring Python and C++ (CINT) at the same level of functionality– Interoperability between them. The end-user choice.

Define guidelines for developing Python bindings– Evaluate existing options: SWIG, Boost.Python, SIP, and raw

Python C-API– Study how each technology handles a number of predefined

cases» Examples: Method overloading, inheritance across

boundaries, templated classes and methods, natural mapping of Python constructs, etc.

– Study interoperability between binding technologies


6. Scripting Services (2)

PyROOT: Python bindings for ROOT (former RootPython)– PyROOT is a Python extension module that allows

the user to interact with any ROOT class»Done generically using the ROOT dictionary. No

need to generate any Python wrapper code to include new ROOT classes

PyLCGDict: Python extension module– Started to work on the LCG Dictionary bindings– Basically repeating the work of PyROOT using the

LCG dictionary instead


8. Education/Documentation

Main activities and tasks– Produce documentation– Produce training material (tutorials)– Help incorporating SEAL components into LCG projects and

experiment frameworks Existing documentation

– Very limited for the time being– Produced a number of HowTo pages for the released elements

» Plugin Manager, Dictionary generation, PyROOT, etc. Helping POOL to incorporate released components

– We have a team member working for both POOL and SEAL Will be preparing tutorials for June release

– Time of possible adoption by experiments


Software Process SPI provided infrastructure

– Savannah Web portal, CVS repository, development tools (SCRAM, Doxygen, CppUnit, etc.) and services (external tool repository, etc.)

– Slowly getting the good dynamics and practices Efforts to “standardize” tool usage and conventions

– Avoiding divergences between LCG projects – Adaptation of tools to “LCG proper” working models

Release procedures– Developing SEAL proper procedures– Tagging, building, testing, releasing, documentation generation,

announcing, etc.– Automation of the process is essential (rotating release

manager role) Platforms

– Single release platform for the moment: Linux RedHat 7.3/gcc-3.2


Status and Summary The SEAL project started last November as a result of

the “blueprint” RTAG recommendations SEAL has had three main releases: 0.1.0, 0.2.0, 0.3.0

since the beginning of the year at the scheduled times– The main emphasis has been to support POOL (Dictionary,

Plugin Management, Foundation classes, etc.) The functionality available is not yet enormous but its

development has help us– To get experience on new tools and procedures– Build up the development team

No technical problem encountered so far Next scheduled release

– Basic functionality completed and ready to be adopted by experiments at end-June


http://seal.cern.ch/

Documents

SEAL: Core Libraries and Services Project