Grid Update

Henry Nebrensky

Brunel University

MICE Collaboration Meeting CM23

The Awesome Power of Grid Computing

The Grid provides seamless interconnection between thousands of computers.

It therefore generates new acronyms and jargon at

superhuman speed.

MICE and the Grid (1)

● Computing: 10 UK sites & Sofia supporting MICE● Storage: 9 sites & RAL Tier 1 supporting MICE● Job submission: 2 each at Glasgow and RAL● UK centric!

Even though we don't yet need to ask to ask for more resources, does everyone know whom they should ask and how long the bureaucracy takes?

MICE and the Grid (2)

● g4Beamline tested on Grid over a year ago● G4MICE in regular use

– ~1000 simulation jobs/day● Job submission: recent RB->WMS change● I will make a bundle of config files for the gLite

CLI available, for people to install their own UI – gLite User Guide:

http://glite.web.cern.ch/glite/documentation/default.asp● Ganga GUI (as used by Atlas) also works for

MICE, see D. Forrest MICE Note soon.

MICE and the Grid (3)

● Grid interface to CASTOR storage at RAL Tier1 has been deployed (not sure what MICE quota is)

● Space unused – need to test (both interface and storage). We should ensure that the Tier 1 is aware of our activities; check:

– https://www.gridpp.ac.uk/wiki/RAL_Tier1_CASTOR_planning

and route requests via Paul Kyberd. ● Storage elsewhere has been used (e.g. staging of

simulation output) but as yet we are not yet formally storing data on the Grid.

MICE and the Grid (4)

● We are currently using EGEE/WLCG middleware and resources, as they are receiving significant development effort and are a reasonable match for our needs (shared with various minor experiments such as LHC)

● Outside Europe other software may be expected –

e.g. the OSG stack in the US. Interoperability is

from our perspective a “known unknown”...

MICE and Grid Data Storage

● The Grid can provide provide MICE not only with computing (number-crunching) power, but also with a secure global framework allowing users access to data

● Good news: storing development data on the Grid

keeps it available to the collaboration – not stuck

on an old PC in the corner of the lab ● Bad news: loss of ownership – who picks up the

data curation responsibilities?

Grid File Management

● Each file is given a unique, machine-generated, GUID when stored on the Grid

● The file is physically uploaded to one (or more) SEs (Storage Elements) where it is given a machine-generated SURL (Storage URL)

● Machine-generated names are not (meant to be) human-usable

● A “replica catalogue” tracks the multiple SURLs of a GUID

File Catalogue

● For sanity's sake we would like to associate nice sensible filenames with each file (LFN, Logical File Name)

● A “file catalogue” is basically a database that translates between something that looks like a Unix filesystem and the GUIDs and SURLs needed to actually access the data on the Grid

● MICE has an instance of LFC (LCG File Catalogue) run by the Tier 1 at RAL

● The LFC service can do both the replica and LFN cataloguing

File Catalogue Namespace

● We need to agree on a consistent namespace for the file catalogue

● The aim is NOT to replicate the experiment structure, instead we want an easy-to-understand structure that is compact but without too many entries in each low-level directory (aim for 20-50)

Concepts (1)

1) At present it is hard-to-impossible to delete a directory out of the LFC namespace. Avoid excess complexity – prevents creating dead branches

2) Ideally a directory should contain either only

more subdirectories, or only some files

Concepts (2)

1) Don't assume it will be possible to browse this from a graphical interface with thumbnails – if you have to search through the LFC by hand, it will be painful even with the ideal structure

2) Moving things will cause confusion (though LFC

allows multiple “soft” links)

3) MC simulations close to that which they model

Namespace (half-baked proposal)

● We get given /grid/mice/ by the server● Four upper-level directories:

– Construction/historical data from detector development and QA

– Calibration/needed during analysis (large datasets, c.f. DB)

– TestBeam/test beam data

– MICE/DAQ output and corresponding MC simulation

Namespace proposal (1)

/grid/mice/MICE/StepN/ RefMomentum? / MC or data

● Split into MICE Step. ● Do we need more subdivision e.g. by momentum?● Separate directory for MC results, or in filename?

/TestBeam/place/...● KEK, Fermi, RAL (Tracker Cosmics)?● The subdivisions to be place-specific

Namespace proposal (2)

/grid/mice/Construction/module/data

/Construction/Tracker/TrackerQA/Construction/Tracker/OpticalQA/Construction/Tracker/FibreDamage

● What about the other modules?● Should we bother with raw files, or tarball/zip

each subset up?

Namespace proposal (3)

/grid/mice/Calibration/Tracker/VLPC/Calibration/BeamLine/FieldMaps

● How should this be split up = what else will be in

here – e.g. are the solenoid field maps part of the

spectrometer/grid/mice/Calibration/Spectrometer/FieldMaps

or part of the beamline/grid/mice/Calibration/BeamLine/FieldMaps/SpectrometerSolenoid

or do we put the field maps together/grid/mice/Calibration/FieldMaps/SpectrometerSolenoid/grid/mice/Calibration/FieldMaps/Quads

Namespace proposal (3)/grid/mice

/Users/name

● For people to use as scratch space for their own

purposes● Encourage people to do this through LFC – helps

avoid “dark data”● LFC allows Unix-style access permissions

Namespace

We are starting to deploy this now, so we need a (crude) idea of – what calibration data will be needed– how it will be provided - O(# of files), format, size

I have no ( = zero ) idea of what to expect from most of MICE for this !!!

What info should be represented by the directory structures and what within the filename?

Need to be consistent in use of upper and lower case in LFNs, else will create bogus entries.

Metadata Catalogue (1)

● For many applications – such as analysis – you will want to identify the list of files containing the data that matches some parameters

● This is done by a “metadata catalogue”.For MICE this doesn't yet exist

● A metadata catalogue can in principle return either the GUID or an LFN

Metadata Catalogue (2)

● We need to select a technology to use for this– use the conditions database– gLite AMGA (who else uses it – will it remain

supported?)● Need to implement – i.e. register metadata to files

– What metadata will be needed for

analysis?● Should the catalogue include the file format and

compression scheme (gzip ≠ PKzip)?

Metadata Catalogue (2) for Humans

or, in non-Gridspeak:● we have several databases (conditions DB, EPICS,

e-Logbook) where we should be able to find all sorts of information about a run/timestamp.

● but how do we know which runs to be interested in, for our analysis?

● we need an “index” to the MICE data, and for this we need to define the set of “index terms” that will be used to search for relevant datasets.

Grid Data Access ProtocolsOur current tools are based around the transfer of

whole files to/from a local disk on the processing machine.

The Grid also allows “POSIX I/O” (i.e. random access) directly to files on the local SE, using a secure version of the RFIO protocol. This would require compiling libshift into G4MICE.

This would be useful in cases where we need to access only a small part of the data in a file.

We currently don't see any need for this in MICE.

Last Slide

● “My laptop battery is flat” is no longer an excuse for not getting some simulation/analysis done!