Upload
ella-keith
View
215
Download
2
Tags:
Embed Size (px)
Citation preview
Metadata
(data about data)
GridPP-15, Paul Millar
2006-01-11 GridPP-15 Metadata 2
Contents
Monitoring metadata service Event-level metadata Work improving AMI Cataloguing ATLAS metadata Conclusions
2006-01-11 GridPP-15 Metadata 3
Monitoring a metadata service
Requirements document has been released Why do people want to
monitor metadata services? How to make that happen? What already exist out there? What still needs to be done?
2006-01-11 GridPP-15 Metadata 4
JMX – servlet monitoring
2006-01-11 GridPP-15 Metadata 5
Monitoring Architecture
2006-01-11 GridPP-15 Metadata 6
MonAMI architecture
2006-01-11 GridPP-15 Metadata 7
MonAMI summary
Follows the UNIX “do one thing well” philosophy. Takes data from from a site and pushes it
somewhere. Plugin architecture: easy to add extra targets Easy to configure (really!) Multiligual: currently speaks Ganglia, but will
(soon) speak Nagios, LEMON, R-GMA?, ... Monitors: Apache, tomcat/JMX, MySQL, ...
2006-01-11 GridPP-15 Metadata 8
ATLAS Event Level Metadata• Event tag infrastructure is being developed to:
– Allow physicists to exclude uninteresting events from data sample
– Extract samples of specific interest into a smaller fileset, for repeated running
– Provide a global view of the data, useful for data mining • Event tag infrastructure is currently under review• Content & usage of event tags under discussion with
physics groups• Tags produced from Rome data for 2.3 million events,
stored at CERN– https://uimon.cern.ch/twiki/bin/view/Atlas/RomeTagFAQ
2006-01-11 GridPP-15 Metadata 9
Integration with DDM• For event tag infrastructure to be used successfully, it
must be able to work with ATLAS Distributed Data Management system (DQ2)
• DQ2 uses datasets as basic units of data manipulation
• Event tag tools currently refer only to files• Currently working on ways to solve this problem
DQ2
Dataset 1
File 1
File 2
File 3
Tag browser
File 1
File 3
2006-01-11 GridPP-15 Metadata 10
Cataloguing Event Tags
• Event tags can now be built and written to a master database.– This may be replicated to T1 and T2s– Different physics groups may build their own
tags• Geographically diverse locations for data.
• How should these be catalogued? – Many open questions (e.g., should AOD and
Tag be distributed together, and how?)– Aim to build on prototype CollectionCatalog, to
design and implement Event Tag Catalogue
2006-01-11 GridPP-15 Metadata 11
AMI VOMS
Completed requirements analysis of the impact of VOMS on AMI.
AMI will use ATLAS-wide agreed VOMS groups and roles.
Mapping of VOMS groups to AMI decided. Problem of getting the proxy certificate into AMI
... using your web browser... Solutions? mod_gridsite or MyProxy.
2006-01-11 GridPP-15 Metadata 12
AMI and SQLite
2006-01-11 GridPP-15 Metadata 13
Cataloguing metadata Looking at ATLAS, there's metadata in:
DDM (several databases) Production system (ProdDB) Event-level (tag) database(s) COOL (used for correlation of DAQ and run Numbers) AMI Catalogues datasets by physics metadata. Detector Description (file catalogues, but should be able to ignore these)
Risk of information being duplicated, unknown or impossible to get at.
Implement easy navigation between different metadata
2006-01-11 GridPP-15 Metadata 14
Summary
The metadata collaboration's work ongoing with various aspects of metadata.
Monitoring requirements document released. Opportunity for working together for developing
additional monitoring tools. AMI will soon support off-line analysis How to implement event-level metadata is going
through discussion stage and into prototyping.
2006-01-11 GridPP-15 Metadata 15
Questions
Comments
Thoughts