LHCb Software Week12-14 November 2003 Gennady Kuznetsov

Production Manager Tools (New Architecture)

Data Challenges 2001

Gennady Kuznetsov

History: Controlling Monte-Carlo simulation.

In 2001, control system was based on a single Java Servlet which in fact was a single file with Java code. The sequence of execution was fixed and all site dependencies had to be programmed inside. Code had to be redistributed for each location with the Servlet runner.

Increased complexity of the tasks and demands for flexibility (requirement to run more then one type of job) required architectural changes.

As a result, the “Data Management” group produced a document with a new flexible architecture.

The Grand Design

Gennady Kuznetsov

Job Configuration, Data Production, Bookkeeping - Design Document, M. Frank (Editor), 04/12/2001

Data Challenges 2003

Gennady Kuznetsov

Agent

Agent

Agent

Agents interpret XML file with Workflow description

Advantages:This very flexible architecture gives us freedom to adjust any parameters in the workflow and also allows us to manipulate Steps known to the “Agent”.Disadvantages:• Agent had to exist on the site or had to be submitted.• For any change in execution, we had to update the agents everywhere.

Components - New System

Module

Module

Module

Module

Module

Module

Module

Step

Step

Step StepStep

Step Step

Step

WorkflowJob

Job Job Job

Job Job

JobJob

Job Job Job

Job Job

Job

Job

Job Job Job

Job Job

Job

ProductionLevels of usage:1. Module – Programmer2. Step – Production

manager3. Workflow –

User/Production manager

We add Module as additional

component of the architecture

Each step generates job as a Python

program.

This module structure allow Production Manager to construct any algorithm as a combination of modules.

Gennady Kuznetsov

Architecture - New System

Presenter Name:

Variable

VariableCollection

ModuleDefinition ModuleInstance

n

1

StepDefinition

n

1

n

StepInstance

WorkflowDefinition1

1

n n

1

Module Editor

Step Editor

Workflow Editor

edit

edit

edit

Module/Step exampleExecution Sequence

Gennady Kuznetsov

DistributionSoftware checkand installation

Example of the Workflow 370000-Sv260r2-Bv17r4

BrunelReconstruction

GaussSimulation

Mon. ServiceMonitoring

Mon. ServiceMonitoring

Data quality “rawh” validation

File ServiceFile transfer

Data quality “oodst” validation

File ServiceFile transfer

Bookkeeping. ServiceBookkeeping

GaussSimulation

Mon. ServiceMonitoring

Data quality “rawh” validation

File ServiceFile transfer

Bookkeeping. ServiceBookkeeping

GaussSimulation

Mon. ServiceMonitoring

Data quality “rawh” validation

File ServiceFile transfer

Bookkeeping. ServiceBookkeeping

GaussSimulation

Mon. ServiceMonitoring

Data quality “rawh” validation

File ServiceFile transfer

Bookkeeping. ServiceBookkeeping

Bookkeeping. ServiceBookkeeping

Another three lines (not enough space)

Module

Module/Step exampleData propagation

Gennady Kuznetsov

DistributionSoftware checkand installation

Example of the Workflow 370000-Sv260r2-Bv17r4

GaussSimulation

Mon. ServiceMonitoring

Data quality “rawh” validation

File ServiceFile transfer

Bookkeeping. ServiceBookkeeping

Module

Module Editor

Python code of single module. Can

be many classes.Module

variables.

Description

Module Name Stored as XML file

Gennady Kuznetsov

Step Editor

Step Name

Description

Definitions of Modules

Instances of Modules

Variables of currently selected instance

Selected instance

Stored as XML file, where all modules are embedded

Gennady Kuznetsov

Step variables.

Workflow Editor

Gennady Kuznetsov

Workflow Name

Description

Step Definitions

Step Instances

Variables of currently

selected Step Instance

Selected Step Instance

Workflow Variables.

Stored as XML file

Job Splitting

Gennady Kuznetsov

Step

Step StepStep

Step Step

Step

Workflow DefinitionJob

Job Job Job

Job Job

JobJob

Job Job Job

Job Job

Job

Job

Job Job Job

Job Job

Job

Production

Python List

The input value for the job splitting is a Python list object. Every single (top level) element of this list applies to the Workflow Definition and propagates through the code and generates single element of the production (one or several jobs).

Current Status

Gennady Kuznetsov

Done:• All three editors (Module, Step, Workflow) ready.• XML schema seems stable.

To do for the Data Challenge 2004:• Define variable types.• Extend Python code generator to cover Workflow.• Create appropriate module library.• Implement “list input” for the Workflow Editor (relates to the job splitting).

Future development:• Add graphical editing panel to Step and Workflow editors.• Finish embedded Python editor with highlighting feature.