S.A. 1

AMI

04/21/23 S.A. 1

AMI and its place in ATLAS Metadata

 Solveig AlbrandJerome FulachierFabian Lambert

S.A. 2

AMI

Part I

1. Introduction to AMI

2. Data sources

3. Known problems

S.A. 3

AMI

Brief Intoduction to AMI

Connections

ORACLE mySQL

Generic functions

Datasets TagCollectorSQ NICOS

ATLAS Metadata Interface A generic catalogue application. Used by

several ATLAS applications.

AMIWhen you make a catalogue in AMI you immediately get a web browser interface, a python CLI + generic query and data manipulation functions.

Application specific layer

S.A. 4

AMI

Interfaces

•Web interface :

•Browsing + all server commands (add, clone, update) if authorised.

•Personal pages with bookmarks

•pyAMI :

•A SOAP web service + examples of use in the ATLAS release. All server commands accessible.

•Data Puller Tasks (see next slide)

AMI

ami.in2p3.fr pyAMI

dataPuller

S.A. 5

AMI

Data Puller Tasks

• Run in a separate instance of AMI.

• Each task runs in its own thread.

• TaskThreads managed by a “master” thread and database tables.– Can modify frequency of tasks, and mutual

exclusion dynamically.

• Email alerts

• Watch dog mechanism.

S.A. 6

AMI

AMI Features• Uses "MQL" from GLite. Users do not need to know the

relational structure of the tables.• A powerful graphic query builder for ad-hoc queries.• Multi-threaded queries and "natural" partioning.• Supports some schema evolution. • Indirect DB connections. RDBMS & database site transparent

for user.• Efficient connection pool and transaction management.• Exploits prepared statements.• Manages its own user database, and autorisation. (users

roles commands catalogues). VOMS proxy upload possible.

• Possibility to browse third party databases if they have a simple relational structure. (Primary key/ Foreign key)

S.A. 7

AMI

AMI catalogues OFFICIAL datasets (data_*, mc_*, and SOME physics group datasets.No USER datasets as yet.

By default :-

•No searching in "archived" catalogues.

•Datasets known to be bad are hidden.

OVERVIEW PAGE

S.A. 8

AMI

Search Results

S.A. 9

AMI

AMI is a MEDIATOR interface• “a software module that exploits encoded

knowledge about some sets or subsets of data to create information for a higher layer of applications”. (Gio Wiederhold, Mediators in the architecture of future information systems, IEEE Computer Magazine, Vol 25, No3, p38-49 March 1993)

• To put it another way, a mediator is an application which puts some domain expertise between the user and a group of data sources, so that information coming from these different sources can be aggregated.

S.A. 10

AMI

04/21/23 S.A. 10

Sources of AMI data• Real data : From the Tier 0 : DAQ data, and first

reconstruction is registered in AMI < 5 minutes after it is declared “ready” (both datasets and files).

• Monte Carlo and reprocessing.– From the Task Request DB : Tasks, dataset names, MC and

reprocessing configuration tags ("AMI tags")– From the production DB : Finished tasks – files and metadata.

• From physicists with AMI writer role.– M.C. GEN datasets– MC Dataset number info, physics group owner,…– Corrected cross sections and comments. (coordinated by Borut K

and Claire Gwenlan.)– Tier0 configuration tags (AMI tags for real data.)– …

N.B. I have neglected Tag Collector here and in what follows – for clarity

S.A. 11

AMI

Tier 0

AMI

Tier 0Ready

Done

This AMI task reads datasets, files and some metadata from Tier 0 using a semaphore mechanism.

Simple and efficient – AMI only get datasets when they are completed.

"AMITags" are written in AMI

before use

S.A. 12

AMI

AKTR (Task Request)

AMI

AKTR

1. Reads new tasks, once they are no longer pending, if they are not aborted before submission.

2. Reads provenance,3. Reads updates to a bad status. 4. Reads MC and reprocessing AMITags.

N.B. Does NOT read finished/done status here.

Read AMITags

S.A. 13

AMI

ProdDB

AMI

ProdDB

1. Follows RUNNING status of tasks.

2. When FINISHED reads metadata.xml for all jobs of the task.

S.A. 14

AMI

DQ2

AMI

DQ2

Data deletednomenclature

DQ2Info

S.A. 15

AMI

PROBLEMS

• Problems we've been dealing with for such a long time that it's almost not worth mentioning them, as I fear that in some cases nothing will ever get done. "Minor Problems"

• Mechanism problems. Major problems we really SHOULD do something about

• Content problems. I am not competent to say anything about these, so I won't.

S.A. 16

AMI

Minor Problems

• Tier 0 : No update mechanism. (Have not needed one up to now.)

• AKTR :– Provenance very acrobatic. (c.f. AMI Review 2006).

Input Dataset column in AKTR DB very messy.

– No "dataset container" "task number" link

• General : No finite state description as far as I know. We had to guess. It is sometimes confusing for users. Dataset states need some work from the collaboration.

S.A. 17

AMI

Important Problems - 1• Validation of metadata.xml

– Validation of metadata output is not part of release validation, and it should be.

– Is it entirely reliable even when it works? There have been many examples of one job out of many which fails to produce metadata. Why are these jobs still declared "done"? The result is that a wrong number of files/events is reported. (https://savannah.cern.ch/bugs/?70591It's because we run with "--ignoreerror=True" . Pavel suggested how to fix it. Alvin agreed. Will anything get done?)

S.A. 18

AMI

Important problems - 2

• Relation between TRANSFORMS (e.g.https://twiki.cern.ch/twiki/bin/view/Atlas/RecoTrf ) and METADATA.– An improved grammar for metadata.xml (task

aware so more compact) is available, but has never been put into production.

– Manpower - Who is going to replace Alvin Tan?

S.A. 19

AMI

Important problems - 3• Lack of coherence between AMI and DQ2 for dataset

container contents (and status).– At the moment AMI guesses that a finished TID dataset will be

placed in its container by the production system when the task is declared FINISHED, using information from AKTR/ProdDB) This is sometimes the case, but not always. What is the logic? Is it documented?

– Solution Stop guessing and get the info from DQ2. Have not done this up to now, but currently working on improving AMI DQ2 information exchange using ACTIVE MQ server. Need confirmation that this can go into production.

• Need a policy for file status, but that can come later when we have more Active MQ experience.

S.A. 20

AMI

Requirements• Our requirements come from many sources.

– Data Preparation coordination + physics groups.– Reprocessing Coordination– Metadata coordination – MC generator group– Fast Reconstruction/Tier 0– DA tools (Ganga) – ADC (DQ2)– User suggestions.

• Coming soon : – better treatment of Physics Containers, – better support for physics groups,– User datasets ???

• Reported until recently to DB, and "close contact" with Data preparation (AMI review recommendations). Now in ADC dev (as well as? instead of ?)

S.A. 21

AMI

Part II

The ATLAS metadata context

S.A. 22

AMI

Other Metadata Applications in ATLAS

• There are many metadata applications in ATLAS and many granularities of data/metadata.

• Metadata Task Force Report 2006/2007

S.A. 23

AMI

Metadata use (MTF report)• To determine the status of a process such as calibration or

analysis (e.g. stage of processing).• To provide statistics on some process without having to

read in large amounts of data (e.g. run completed gracefully or terminated abnormally).

• To provide early indications of problems and clear indications of their source (e.g. detector status summary).

• To allow the cross referencing of data in different data stores or different formats (conditions, events, TDAQ), (POOL, n-tuple), …

• To locate other types of data.• To encapsulate the state of a process so that it can be

repeated or easily modified (e.g.versions of software and configuration used for a process).

S.A. 24

AMI

Granularity of metadata (MTF Report)

Event ; 1 File ; 102 – 104 Luminosity Block ; 104

Run ; 105-106 Datastream per run;104 – 106 Dataset ;105 – 109 "A means to navigate between these granularities

of scale will be needed to support queries where the user indicates a requirement on a given granularity but where the data needed is stored in a different granularity."

S.A. 25

AMI

Metadata Flow (MTF Report)

• From the detector to analysis

• From the MC production to analysis

• Through different granularities

S.A. 26

AMI

There are many metadata applications in ATLAS

• Developed independently.

• Different positions in the metadata flow.

• Different DB structures and interface choices.

• Links are ad hoc – these has never been any concerted design or effort to develop an overall metadata structure (with the notable exception of dataset nomenclature).

S.A. 27

AMI

It is too complicated to try and classify all of them by granularity.

event file LB Run Stream Dataset

And not very useful!

App

lica

tion

S.A. 28

AMI

User Data Discovery Intefaces• The AIM of AMI is to show physicists the

DATASETS which are available for analysis, and to allow SEARCHING on PHYSICS criteria.

• There are two other important ATLAS mediator applications for Data Discovery at other granularities. – RunQuery (Runs)– ELSSI/TAG DB (Events)

• Granularity Clouds ? (c.f. Grid architecture)

S.A. 29

AMI

Overview of Data Discovery in ATLAS

EventEventRunRun

DatasetDataset

AMI

RunQuery ELSSI

"Navigation Parameters"

runNumber

eventNumber

datasetName

S.A. 30

AMI

Inside the Dataset cloudAMI

AKTR

PANDA

DQ2

PRODDBTier0

S.A. 31

AMI

Links from (and to) AMI

AMI

Twiki

Run Query

Run Summary

Config DBGeometry

PandaTrigger

COMA

ProdDB

ELSII

pAThena

GANGA

SVN/NICOS

S.A. 32

AMI

Links from Run Query (taken from RQ tutorial)

The actual call of AtlRunQuery after parsing

The actual call of AtlRunQuery after parsing

Which criterion are applied

Which criterion are applied

Total number of runs / events selected and execution time

Total number of runs / events selected and execution time

Table with runs selected with show results

(run, links, LB, time, #events always shown)

Table with runs selected with show results

(run, links, LB, time, #events always shown)

Summary (where useful)Summary (where useful)

Links to other run information (DS, BS, AMI, DQ, E-log)

Links to other run information (DS, BS, AMI, DQ, E-log)

S.A. 33

AMI

21/04/23 33

TAG DB Structure & Processes

Conditions Metadata DB (MC and Prod)

Application and Service Knowledge DB

POOL Relational CollectionsTAG

UploadTier-0

COMA Run Fill Progs

Repro Data

Loader

COOL

AMI

DQ2

TAG DBTrigger

DB

Slide from Florbela Viegas, SW week July 2010

S.A. 34

AMI

Some "Blue Sky" Questions• Remove the "ad-hoc" mechanisms for

communication?– Generalise Active MQ communication between

applications?

• Move to an ATLAS wide query grammar for data discovery?– "Ontology" (http://en.wikipedia.org/wiki/Ontology_%28computer_science%29)

• A common vocabulary for sharing information about a domain. It includes definitions of the basic concepts in the domain, and the relations among them. Each actor would then implement its own adaptor to translate to and from the common language

S.A. 35

AMI

Each application implements an adaptor to a common language.

AMI METADATA PORTAL

Query BuilderResult Aggregation

Metadata Dictionary

User QueryResult

ATLAS METADATA ONTOLOGY

Translations done by each application

The Good Run List is

almost a prototype for this!

S.A. 36

AMI

?

• A very interesting long term project for a real computer scientist, but do we really need it?

• Implies a common desire to achieve such an interface.

S.A. 37

AMI

Conclusions.• It is important that each mediator interface is well

anchored in its granularity cloud.

• It is equally important to keep good coordination between them.

• The physicist user is the boss! Make it easy for him/her.

AMI

RunQuery ELSSI

S.A. 38

AMI

Extra slides

S.A. 39

AMI

Key for interface diagrams

AMI

Data Puller Tasks

pyAMIami.in2p3.fr

KeytriggerDB

prodDB

Tier0

AMI

AMI

AMI

AMI links to

AMI reads from

AMI writes to

S.A. 40

AMI

GranularityGranularity MTF selection

Event 1 2872903475 1

File 102 – 104 631715 4548

Luminosity Block 104 No easy way of counting

Run 105-106 5006 573892

stream per run 104-106 126 ?

Dataset 105 – 109 9304 308781

S.A. 41

AMI

Container incoherence example• Task 73194, marked "finished" in AKTR 2010-04-22 11:03:54.0

(actually was probably finished on 2009-07-16 09:44:24) • All containers are empty.

– data09_cos.00121238.physics_RPCwBeam.recon.TAG_COMM.r733/– data09_cos.00121238.physics_RPCwBeam.recon.NTUP_MUONCALIB.r

733/– data09_cos.00121238.physics_RPCwBeam.recon.ESD.r733/– data09_cos.00121238.physics_RPCwBeam.recon.AOD.r733/– data09_cos.00121238.physics_RPCwBeam.recon.log.r733/– data09_cos.00121238.physics_RPCwBeam.recon.HIST.r733/

• But the TID datasets exist in DQ2 (i.e. the data is not deleted)• Example :

data09_cos.00121238.physics_RPCwBeam.recon.TAG_COMM.r733_tid073194 (3827 files)