Geant4 physics validation on the GRID

George Lestaris supervisors: W. Pokorski & A. Ribon

Geant4 physics validation on the GRID

SFT meeting - 28/04/2014


Introduction to the problem

• Geant4 is a simulation framework

• Physics validation is form of regression testing but:

• simulation data can be compared with respect to real (reference) data (from papers, test beams, etc)

• Monte Carlo simulation does contain by its nature the concept of randomness

• need to run many simulation events to get data that could be statistically close to previous versions

2


Previous work• Solution tailored for Simplified calorimeter

• a Geant4 application that simulates the showers that are produced inside simplified geometries of LHC calorimeters

• DIANE used to run jobs on GRID resources

• not very flexible:

• unable to easily restart single jobs

• scheduling among different GRID sites was round-robin

• not maintained/supported anymore

3


Previous work

• Process required manual intervention in different stages..

• Error-prone actions

• Minor problems

• Slow access to stored data: mis-index MySQL database

• Slow plotting: a new ROOT session per graph

4


Previous work

5

Worker

Step 1Build and store

binaries to CVMFS

Step 2Setup DIANE Master

Step 3Submit GANGA Jobs

Step 4Run analysis of results

CVMFS

MySQL DB

GANGA Iterface

WLCG

LxBatch

WorkerWorkerWorker...

DIANE Master

Web application

AFS


Motivation• Extensible system to

support multiple tests

• Use of a better supported WMS

• Automations to reduce the room for user errors during jobs submission and running

• Efficient storage and plotting

6

Creation of DIANE master

Preparation of AFS directory

Submission of pilot jobs

Results merging, analysis and

storage

...

Old

Submission of production

...

New

Results become available in the web application


g4.Val

7


Architecture• Component-based architecture

• Core component contains essential classes and modules, referenced by all other components

8

Production side: • Production: Submission

of jobs. The job wrapper that runs the tests, collects stores and merges results.

• Client: Communication with the Web API.

Presentation side: • DB: Abstract interface to

the database

• Analysis: Runs lightweight analysis tasks (e.g.: plotting) over the results.

• Web: Hosts the UI and the results submission API.


Architecture

9

Core

API Client

Production

Web

DB

Analysisspeaks API

DIRACserver EOS MySQLRedis

queue


User roles (actors)

Test manager: person that imports a test in the system

• not necessarily the developer of the test but someone who understands how to run it

Production manager: submits the productions (sets of jobs) for tests

End user: that looks into the final results (web UI) by running on-demand analysis (plotting) tasks

10


End user

11

http://g4-val.cern.ch/

Tests list Analysis plugins

http://g4-val.cern.ch/

Geant4 physics validation on the GRID12

Filters

Tabs

Results UI


Results UI components

13

Version selection

Filters can be lists

of values


Results UI components

14

Bookmarkable pages

Different analysis (plotting) plugins


Production manager

• Submission with a recipe

• predefined recipe that submits a production by changing the Geant4 version

• Command line tool to interactively create custom production

• Python API

• if need custom input sandbox, environment, dependancies

15


Recipe submission

$> source /cvmfs/geant4.cern.ch/grid/etc/login.sh# prompt to type the certificate password$> g4val-prod-recipe -t sc -v 10.0.ref03 -r test# some settings.........[g4val.production.submitter.dirac INFO] 2014-04-25 16:38:43 CEST - Submitted job with reference: Id = 64849[g4val.production.submitter.dirac INFO] 2014-04-25 16:38:43 CEST - 8 jobs were successfully submitted!

16

Test code Version Recipe


DIRAC

The DIRAC (Distributed Infrastructure with Remote Agent Control) project is a complete Grid solution for a community of users needing access to distributed computing resources. DIRAC forms a layer between a particular community and various compute resources to allow optimized, transparent and reliable usage.

17


DIRAC (in English)

• Developed for the LHCb experiment initially

• Today a collaboration with many users around the world

• Geant4 uses a small subset of its functionality

• Pilot job approach as well

• Scheduling is done opportunistically, i.e.: the site that runs the most pilot jobs runs the most DIRAC jobs

• Able to restart/reset/reschedule jobs

• Web UI to list, filter, monitor and manage jobs

18


Test manager

Needs to define the test model

• what are the variables that can uniquely identify a test run?

• what are the variables, extracted from the test results, that are interesting to present?

19


Test modelAttributes and parameters: variables that characterise a test instance (execution)

Test run: assignment of values to all the test parameters

• A test run is unique

• Parameters are indexing list of runs

Observables: variables that describe results

• Can be tuples, list of numbers, or pointers to (ROOT) files

• A test observable is also having its attributes and parameters

20


Test model

21

str codestr namestr description

Teststr codestr nameint typemixed default_valuebool allow_null

Attribute

bool use_referenceParameterstr code

str namestr description

Observable*

1

*

* 1

1


Test runningA lot of extensions points available to override system default behaviours for a test:

• Test running: production hooks

• add dependencies (libraries from CernVM-FS)

• environment (variables)

• macro file generation (template)

• macro file is used as an input to a Geant4 application

22


Results storage

• Run splitting and merging:

• big (long) runs can be split to smaller jobs (e.g.: reducing number of events)

• Results storage: production hooks

• raw results harvesting if necessary

• format of registered results (observables)

23


Results presentation

On-demand analysis

• analysis plugins are independent of a test, i.e.: can be reused

• mapping between what observables can be plotted with which analysis plugin

24


Technical details (backup slides)

25


Production: Job splitting and submission

• Large test runs can be split to smaller jobs with reduced number of events

• Each job appends its results to the run results

• List of jobs is submitted with the submitter interface

• Current implementation with DIRAC, but can be any

• Job scheduling is done by the WMS

26


Production: Job wrapper

g4.Val gets involved again when the job arrives to a worker node (WN)

• Job wrapper is stored in CernVM-FS

• Job is a simple JSON file with its input sandbox

27


Production: Job wrapper1. Input sandbox checking

2. Macro file generation given the run parameters

3. Dependency resolution per test for WNs platform

4. Starts the Geant4 process

5. Transfers results to EOS when done

6. Appends job results to the run results if run was split

7. Registers the results to the database with the API client component

28


Database component

Database schema is generated upon test import

• table for storing the runs

• one table for each observable

• attributes are fields and parameters make up the table indexes

29


Web component technologies

30


Analysis component• Web component submits

analysis task to redis queue

• Analysis workers are processes that execute these tasks

• Disposable in case of crashes

• Analysis load balancer starts new workers if the queue has many waiting jobs

31

AJAXServer

Web interface

User Browser

HTTP

Request

Request

Request

...

Workers

Database


Examples (backup slides)

32


Simplified calorimeter• Big runs, split to many jobs

• Raw results as ROOT files (merged with TFileMerger)

• Registered results are value-error tuples for the six observables

• Using many runs in one graph as beam energy is the X-axis parameter

• Plotted with HighCharts JavaScript library (client side)

• or ROOT TGraphError (server side)

33


Simplified calorimeter

34

http://bit.ly/1lZQef0

http://bit.ly/1lZQef0


Geant4 test test30

• Quick runs, no need of splitting

• One pilot job can run tens of test30 jobs

• Raw results are ROOT files

• Registered results are these ROOT files URLs (EOS supports natively XRootD)

• Each TObject is an observable

• Plotted with ROOT TObject superimpose plugin

35


Geant4 test test30

36

http://bit.ly/1lXtSLc

http://bit.ly/1lXtSLc


Example of test’s modelfrom g4val.model import Test, Attribute, Parametertest = Test( code="sc", name="Simplified calorimeter", description="...")# Parameterstest.add_attribute(Parameter( code="physics_list", type=Attribute.STRING, type_size=100, name="Physics list", use_reference=False))test.add_attribute(Parameter( code="calorimeter", type=Attribute.STRING, type_size=100, name="Calorimeter"))test.add_attribute(Parameter( code="beam_particle", type=Attribute.STRING, type_size=5, name="Beam projectile", default_value="pi-"))test.add_attribute(Parameter( code="beam_energy", type=Attribute.FLOAT, name="Beam energy"))# Attributestest.add_attribute(Attribute( code="events_amt", type=Attribute.SHORT, name="Number of events", default_value=5000))

37


Example of macro generation hookdef get_job_macro(job): return """#verbose1#events%(events_amt)d#particle%(beam_particle)s#material%(material)s#energy(MeV)%(beam_energy)s#generator%(generator)s#run#exit""" % job

38

• From test test30

• Test parameters are:

• Number of events

• Beam particle

• Beam energy

• Material

• and Generator

• job is inheriting Python dict


Conclusion

g4.Val is a flexible, extensible system for Geant4 physics validation in our group

• Introduces a Geant4 test formalisation

• Uses DIRAC for job submission and running on GRID and potentially other batch resources

• Efficient storage of results

• On-demand analysis tasks through an highly interactive intuitive web interface

39

Science

Geant4 physics validation on the GRID