The ATLAS Database Project Richard Hawkings, CERN Torre Wenaus, BNL/CERN ATLAS plenary meeting June 24, 2004

The ATLAS Database Project

Richard Hawkings, CERNRichard Hawkings, CERN

Torre Wenaus, BNL/CERNTorre Wenaus, BNL/CERN

ATLAS plenary meetingATLAS plenary meeting

June 24, 2004June 24, 2004

June 24th, 2004June 24th, 2004Richard HawkingsRichard Hawkings 2

Outline

Mandate and scopeMandate and scope

Project definition processProject definition process

Organization and communicationOrganization and communication

Subproject surveySubproject survey

Concluding remarksConcluding remarks

The current draft plan can be found at: The current draft plan can be found at: http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/calibration/AtlasDBProjectPlan.{ps,http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/calibration/AtlasDBProjectPlan.{ps,

pdfpdf

}}

(temporary location until new database web is up and running)(temporary location until new database web is up and running)


Project Mandate and Scope

Lead and coordinate all ATLAS database activitiesLead and coordinate all ATLAS database activities Including those so far under Software, TC, TDAQ, Detector Projects

Software, servers, distributed data management infrastructure

Specifically, databases and data management forSpecifically, databases and data management for Detector production, detector installation, survey data

Detector geometry

Online configuration, run bookkeeping, run conditions

Event data and metadata

Calibration and alignment (online and offline)

Offline processing configuration and bookkeeping

Grid based access to event and non-event data


Project Definition Process

RH, TW appointments (each currently at 50%) effective May 1RH, TW appointments (each currently at 50%) effective May 1stst 2004 2004

Serious project definition work began late AprilSerious project definition work began late April Based on the project outline presented at January 2004 software week Project should strengthen, not weaken or delay, DB activities across ATLAS

Many individual discussions, feeding into plan iterationsMany individual discussions, feeding into plan iterations

Public draft circulated to ATLAS software community in advance of BNL Public draft circulated to ATLAS software community in advance of BNL

software week, and discussed during the weeksoftware week, and discussed during the week No major new input – generally favourable impression

Draft plan was approved by May 28Draft plan was approved by May 28thth CMB/SPMB CMB/SPMB

Continuing to gather input (TDAQ community, subdetectors, EB, …)Continuing to gather input (TDAQ community, subdetectors, EB, …)

The plan will evolve, but the current version will guide project launchThe plan will evolve, but the current version will guide project launch


Organization

Part of Software and Computing Project (Dario Barberis) Part of Software and Computing Project (Dario Barberis)

Richard Hawkings, Torre Wenaus Co-Leaders 5/04-5/06Richard Hawkings, Torre Wenaus Co-Leaders 5/04-5/06 Computing Management Board (CMB) members Ex-officio Trigger/DAQ steering group (TDAQ-SG) members

Database Steering Group chaired by the Project Leaders is Database Steering Group chaired by the Project Leaders is

planning and decision making bodyplanning and decision making body

Twelve subprojects cover the mandated scopeTwelve subprojects cover the mandated scope

Some subprojects embedded in other parts of ATLASSome subprojects embedded in other parts of ATLAS Where tight integration of DB activities must be preserved Subprojects being organized in close consultation with the projects

concerned, to ensure this happens


Work Breakdown

1)1) Project managementProject management - steering, planning, coordination, strategy - steering, planning, coordination, strategy

2)2) Detector productionDetector production - long term storage of subdetector production data - long term storage of subdetector production data

3)3) Detector installationDetector installation - manufacturing and test (MTF), racks, cabling, survey - manufacturing and test (MTF), racks, cabling, survey

4)4) Detector geometryDetector geometry - primary numbers for detector description - primary numbers for detector description

5)5) Online databasesOnline databases - configuration, conditions, bookkeeping, offline transfer - configuration, conditions, bookkeeping, offline transfer

6)6) Calibration and alignment Calibration and alignment - central tools, not subdetector algorithm work- central tools, not subdetector algorithm work

7)7) Conditions database infrastructureConditions database infrastructure - core sw and tools; framework integration - core sw and tools; framework integration

8)8) Event dataEvent data - events and metadata from raw to analysis. Common core sw - events and metadata from raw to analysis. Common core sw

9)9) Distributed data managementDistributed data management - event and conditions data. Grid integration - event and conditions data. Grid integration

10)10) Offline processing configuration and bookkeepingOffline processing configuration and bookkeeping - production metadata - production metadata

11)11) Distributed database servicesDistributed database services - physical databases, distributed infrastructure - physical databases, distributed infrastructure

12)12) Software support servicesSoftware support services - supporting users and deployers of DB software - supporting users and deployers of DB software


Steering Group

Planning and decision making bodyPlanning and decision making body

Integration mechanism to ensure synergy and coherenceIntegration mechanism to ensure synergy and coherence Across subdetectors, across database areas

Representation from subprojects, associated projectsRepresentation from subprojects, associated projects

Decisions taken following consensus in the steering groupDecisions taken following consensus in the steering group

Project Leaders have full authority for planning and execution, including Project Leaders have full authority for planning and execution, including

cases lacking full consensuscases lacking full consensus In cases of serious dissent, CMB and TDAQ-SG (where appropriate) take

final decision

Strategic decisions go to the CMB (and TDAQ-SG) for endorsementStrategic decisions go to the CMB (and TDAQ-SG) for endorsement

Steering Group is a large body, as the broad scope requires…Steering Group is a large body, as the broad scope requires…


Steering Group Composition

Still have to fill some appointments in consultation with appropriate communitiesStill have to fill some appointments in consultation with appropriate communities Technical coordination - Kathy Pommes, Luc Poggioli Online - Antonio Amorim, Mihai Caprini, Igor Soloviev High level trigger - TBD Calibration and Alignment - Richard Hawkings Detector geometry - Joe Boudreau Inner Detector - TBD LAr calorimeter - Hong Ma Tile calorimeter – Karl Gellerstedt Muon spectrometer - Joe Rothberg Conditions database infrastructure - RD Schaffer Event data - David Malon Distributed data management - TBD Offline processing - TBD Distributed database services - Alexandre Vaniachine User feedback - TBD, an informed+noisy+constructive user voice Persistency Framework Project (LCG Apps Area) - Dirk Duellmann Computing Coordinator (ex officio) - Dario Barberis Software Project Leader (ex officio) - David Quarrie

Liaison from DB Project to

Software Project Management Board - David Malon


Communication

MeetingsMeetings All with agendas in advance, and minutes documenting technical progress,

planning and decisions. Phone connections to allow wide participation Steering Group meeting bi-weekly (Friday 15:30 starting June 25th) Weekly meeting covering primarily offline – continuation of existing meeting

Technical planning and execution, within overall guidelines and plan of the SG

A second weekly slot (to be defined) for Online database meeting, roughly bi-weekly TC database meeting (production, installation) every 2-4 weeks Conditions data working group meeting periodically Associated mailing lists for all these communities (online/TC to be setup)

WebWeb Project web as a comprehensive and current source of technical and

planning information and documentation is a very high priority We take this as a project management responsibility We will both write web content and nag others to do the same!


Subproject Survey

A compressed survey of the subprojects…A compressed survey of the subprojects…


1) Project Management

Most of this area already addressed…Most of this area already addressed… Planning and steering

Project meetings

Project web

……but also includes…but also includes… Monitoring of QA, testing and validation

Strategy and technology evolution


2) Detector Production

The many production/construction DBs used worldwide in the The many production/construction DBs used worldwide in the

subdetectors are not the responsibility of this projectsubdetectors are not the responsibility of this project

Ensuring Ensuring allall data of long-term interest to ATLAS is gathered into central data of long-term interest to ATLAS is gathered into central

(CERN IT Oracle) databases (CERN IT Oracle) databases isis in the mandate in the mandate Central system for uniform access and long-term maintainability

Provision of tools, standards, guidelines to subdetectors

Data definition and entry is subdetector responsibility

Central DB exists and some subsystems are entering data, but there is Central DB exists and some subsystems are entering data, but there is

a great deal of central/common work to be donea great deal of central/common work to be done

Personnel and oversight for central/common work is mostly absentPersonnel and oversight for central/common work is mostly absent Some ideas and possibilities – benefit from subdet production finishing?


3) Detector Installation

MTF (manufacturing and test) installation databaseMTF (manufacturing and test) installation database Installed parts with links to production database

Rack database (exists; being populated)Rack database (exists; being populated)

Cabling database (partially exists)Cabling database (partially exists)

Survey database (does not exist)Survey database (does not exist)

Extraction tools to e.g. use cabling data for online config Extraction tools to e.g. use cabling data for online config

(not existing, and needed soon – e.g. comissioning)(not existing, and needed soon – e.g. comissioning)

Here again, personnel and oversight for central/common Here again, personnel and oversight for central/common

work (this project’s mandate) is severely lackingwork (this project’s mandate) is severely lacking


4) Detector Geometry

Primary numbers used by detector description softwarePrimary numbers used by detector description software

NOVA-based system deployed and operating for some timeNOVA-based system deployed and operating for some time

Work is underway to move to a successor with versioning Work is underway to move to a successor with versioning

supportsupport Approach is consistent with EB-mandated push to implement final

‘as-built’ detector geometry before subdetector engineers leave

Involves a ‘fast track’ implementation using standard relational DB

tools to quickly support gathering and loading data

Offline access (via LCG ‘relational POOL’, conditions DB) on longer

timescale when that software is ready


5) Online Databases

Configuration database - 30% of the online system; must Configuration database - 30% of the online system; must

remain integral to onlineremain integral to online

Online run bookkeeping - expect to employ standard Online run bookkeeping - expect to employ standard

offline/online tools offline/online tools

Conditions database interfaces - ditto. Conditions database interfaces - ditto. Online, through the Lisbon group, has provided the standard tool

used also offline, now also contributing to LCG CondDB project

External interfaces and data flow - Information Server (IS), External interfaces and data flow - Information Server (IS),

slow controls (DCS), offline (to AMI, production mgmt)slow controls (DCS), offline (to AMI, production mgmt)


6) Calibration and Alignment

Activities organized via conditions data working groupActivities organized via conditions data working group

Communication forum for developing strategies, preparing Communication forum for developing strategies, preparing

online and offline algorithmsonline and offline algorithms

Conditions database loading and accessConditions database loading and access

Contribute to computing model Contribute to computing model Little manpower for central/common tasks

Both subdetector and coordination effort currently focused on CTB


7) Conditions Database Infrastructure

Conditions database core software developmentConditions database core software development

Supporting tools (browsers, data distribution and synchronization, Supporting tools (browsers, data distribution and synchronization,

subsetting, etc.)subsetting, etc.)

Athena services for conditions dataAthena services for conditions data

ATLAS participation in LCG CondDB common projectATLAS participation in LCG CondDB common project Activity is increasing, with ATLAS the largest experiment participant New: Relational DB support for POOL, versioning system component

Short-term focus is on CTB support and stabilityShort-term focus is on CTB support and stability

Planning (after CTB) to converge from many tools …Planning (after CTB) to converge from many tools … Lisbon CondDB, NOVA, POOL, geometry DB

… … to essentially one, incorporating all experience gainedto essentially one, incorporating all experience gained Common project CondDB with POOL support


8) Event Data

Core software support for event data, from raw data to analysisCore software support for event data, from raw data to analysis Including event collections and physics datasets

Athena integration - both event data specific and common persistency Athena integration - both event data specific and common persistency

services services This activity moved from Software Project to Database Project

Event data access outside Athena, e.g. in ROOT analysis environmentEvent data access outside Athena, e.g. in ROOT analysis environment

ATLAS participation in POOL common projectATLAS participation in POOL common project

Event data storage for CTB and DC2 is generally OK

File-level data management is handled by the next subproject


9) Distributed Data Management

Management of ATLAS data around the worldManagement of ATLAS data around the world Cataloging, replication, synchronization, access control, …

Event, conditions and other data; files and relational DBsEvent, conditions and other data; files and relational DBs

Integration/interfacing with grid tools for data managementIntegration/interfacing with grid tools for data management And working around grid software deficiencies

Present focus on DC2 production needs – Present focus on DC2 production needs – Key tool: Don Quixote – interface to heterogeneous grids

As yet no overall strategy for DDM today and in the futureAs yet no overall strategy for DDM today and in the future Need urgently to address user-level data management tools


10) Offline Processing Configuration and Bookkeeping

Databases cataloging metadata that is input to and output Databases cataloging metadata that is input to and output

from offline processing jobsfrom offline processing jobs Both managed production and (in the future) group and individual

level jobs

Cataloging of provenance information to unambiguously define

job/software configuration

Key tools at present are AMI and the production DBKey tools at present are AMI and the production DB

Again needs plan & strategy, including technology choicesAgain needs plan & strategy, including technology choices

Present focus on DC2 and CTB supportPresent focus on DC2 and CTB support


11) Distributed Database Services

Support for deployed database and data management Support for deployed database and data management

services at CERN and throughout ATLASservices at CERN and throughout ATLAS

Physical servers, distributed (heterogeneous!) database Physical servers, distributed (heterogeneous!) database

infrastructureinfrastructure

Support and/or liaison for admin and operations of Support and/or liaison for admin and operations of

databases away from CERNdatabases away from CERN

Liaison to CERN IT/DB for CERN-based servicesLiaison to CERN IT/DB for CERN-based services Possible common project in distributed database infrastructure

under discussion, initiated by ATLAS (David Malon) Present focus is again on CTB and DC2 support


12) Database Software Support Services

Support for software, distinct from support for physical Support for software, distinct from support for physical

services (preceding subproject)services (preceding subproject)

DocumentationDocumentation Not authoring (developers are responsible), but organization,

usability, monitoring and review, ‘encouragement’ to authors

Tutorials and trainingTutorials and training

User support servicesUser support services E.g. Savannah problem reporting, feature requests


The challenges ahead

ATLAS database project is a big projectATLAS database project is a big project Covers many different areas, diverse communities

Key objectives: Improving communication, facilitating data transfers

Short and medium-term concernsShort and medium-term concerns Manpower for TC-related areas (detector production / installation)

Missing both sub-project leadership effort and workers

Becoming increasingly important as we approach commissioning Can we exploit effort freed up from sub-detectors ?

Data management strategies and needs – DC2 and vision beyond Large scale distributed infrastructure – LCG common project initiative Individual doing analysis/development – end user tools

New contributions / efforts are needed and welcome !New contributions / efforts are needed and welcome !

Documents

The ATLAS Database Project Richard Hawkings, CERN Torre Wenaus, BNL/CERN ATLAS plenary meeting June 24, 2004