Data Management at CERN’s Large Hadron Collider (LHC) Dirk D ü llmann CERN IT/DB, Switzerland

D. Duellmann, CERN Data Management at the LHC 1

Data Management at CERN’s Data Management at CERN’s Large Hadron Collider (LHC)Large Hadron Collider (LHC)

Dirk DDirk DüüllmannllmannCERN IT/DB, SwitzerlandCERN IT/DB, Switzerland

http://cern.ch/dbhttp://cern.ch/db

http://pool.cern.chhttp://pool.cern.ch


Outline

• Short Introduction to CERN & LHC• Data Management Challenges • The LHC Computing Grid (LCG)

• LCG Data Management Components • Object Persistency and the POOL Project• Connecting to the GRID – LCG Replica Location

Service

CERN - The European Organisation for Nuclear ResearchThe European Laboratory for Particle Physics

• Fundamental research in particle physics• Designs, builds & operates large accelerators• Financed by 20 European countries (member

states) + others (US, Canada, Russia, India, ….)~€650M budget - operation + new accelerators 2000 staff + 6000 users (researchers) from all over the

world

• Next Major Research Project - LHC start ~2007 • 4 LHC Experiments, each with

• 2000 physicists, 150 universities, apparatus costing ~€300M, computing ~€250M to setup, ~€60M/year to run

• 10-15 year lifetime


airp

ort

Computer Centre Geneva

27km


The LHC machineThe LHC machine

Two counter- circulating

proton beams

Collision energy 7+7 TeV

27 Km of magnetswith a field of 8.4 Tesla

Super-fluid Heliumcooled to 1.9°K

The world’s largest superconducting structure


online systemmulti-level triggerfilter out backgroundreduce data volume from40TB/s to 500MB/s

level 1 - special hardware

40 MHz (40 TB/sec)level 2 - embedded processorslevel 3 - PCs

75 KHz (75 GB/sec)5 KHz (5 GB/sec)100 Hz(500 MB/sec)data recording &

offline analysis


LHC Data Challenges

• 4 large experiments, 10-15 year lifetime

• Data rates: 500MB/s – 1.5GB/s

• Total data volume: 12-14PB / year• Several hundred PB total !

• Analysed by thousands of users world-wide

• Data reduced from “raw data” to “analysis data” in a small number of well-defined steps

interactivephysicsanalysis

batchphysicsanalysis

batchphysicsanalysis

detector

event summary data

rawdata

eventreprocessing

eventreprocessing

eventsimulation

eventsimulation

analysis objects(extracted by physics topic)

Data Handling and Computation for

Physics Analysisevent filter(selection &

reconstruction)

event filter(selection &

reconstruction)

processeddata

les.

rob

ert

son

@ce

rn.c

h

CERN

Estimated DISK Capacity at CERN

0

1000

2000

3000

4000

5000

6000

7000

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

year

Tera

Byt

es

LHC

Other experiments

Estimated Mass Storage at CERN

LHC

Other experiments

0

20

40

60

80

100

120

14019

98

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

Year

Pet

aByt

es

Estimated CPU Capacity at CERN

0

1,000

2,000

3,000

4,000

5,000

6,000

1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

year

K S

I95

LHC

Other experiments

Moore’s law

Planned capacity evolution at CERN

Mass Storage Disk

CPU

physics group

regional group

les.

rob

ert

son

@ce

rn.c

h

CERNTier2

Lab a

Uni a

Lab c

Uni n

Lab m

Lab b

Uni bUni y

Uni x

Tier3physics

department

Desktop

Germany

Tier 1

USAUK

France

Italy

……….

CERN Tier 1

……….

The LHC Computing

Centre

Multi Tiered Computing Models - Computing GridsMulti Tiered Computing Models - Computing Grids


LHC Data ModelsLHC Data Models

• LHC data models are complex!• Typically hundreds (500-1000) of

structure types (classes in OO)• Many relations between them• Different access patterns

• LHC experiments rely on OO technology• OO applications deal with networks of

objects• Pointers (or references) are

used to describe inter object relations

• Need to support this navigational model in our data store

EventEvent

TrackListTrackList

TrackerTracker Calor.Calor.

TrackTrackTrackTrackTrackTrack

TrackTrackTrackTrack

HitListHitList

HitHitHitHitHitHitHitHitHitHit


What is POOL?What is POOL?

• POOL is the common persistency framework for physics applications at the LHC • Pool Of persistent Objects for LHC

• Hybrid Store – Object Streaming & Relational Database• Eg ROOT I/O for object streaming

- complex data, simple consistency model (write once)• Eg RDBMS for consistent meta data handling

- simple data, transactional consistency

• Initiated in April 2002 • Ramping up over the last year from 1.5 FTE to ~10 FTE

• Common effort between LHC experiments and the CERN Database group • project scope and architecture and development • => Rapid feedback cycles between project and its users

• First larger data productions starting now!


Component ArchitectureComponent Architecture

• POOL (as most other LCG software) is based on a strict component software approach• Components provide technology neutral APIs • Communicate with other components only via abstract

component interfaces

• Goal: Insulate the very large experiment software systems from concrete implementation details and technologies used today

• POOL user code is not dependent on any implementation libraries • No link time dependency on any implementation packages

(e.g. MySQL, Root, Xerces-c..) • Component implementations are loaded at runtime via a plug-

in infrastructure

• POOL framework consists of three major, weakly coupled, domains


POOL ComponentsPOOL Components

POOL API

Storage Service FileCatalog Collections

ROOT I/OStorage Svc

XMLCatalog

MySQLCatalog

EDG Replica Location Service

ExplicitCollection

ImplicitCollection

RDBMSStorage Svc


POOL Generic Storage HierarchyPOOL Generic Storage Hierarchy

• A application may access databases (eg streaming files) from one or more file catalogs

• Each database is structured into containers of one specific technology (eg ROOT trees or RDBMS Tables)

• POOL provides a “Smart Pointers” type pool::Ref<UserClass>• to transparently load objects from

the back end into a client side cache

• define persistent inter object associations across file or technology boundaries

POOL Context

FileCatalog

Database

Container

Object


Data Dictionary & StorageData Dictionary & Storage

Technology dependent

DictionaryGeneration

CIN

T

dic

tio

nar

yC

INT

d

icti

on

ary

I/O I/O

Data I/O

GCC-XMLGCC-XML

LCG dictionary codeLCG dictionary code

Abstract DDL

Abstract DDL

Code GeneratorCode Generator

LC

Gd

icti

on

ary

LC

Gd

icti

on

ary

Gat

eway

Gat

eway

Reflection

Oth

er

Cli

en

tsO

the

r C

lie

nts

C++ Header

C++ Header


POOL File CatalogPOOL File Catalog

• Files are referred to inside POOL via a unique and immutable file identifier which is system generated at file creation time• This allows to provide stable inter-file reference

• FileID are implemented as Global Unique Identifier (GUID) • Allows to create consistent sets of files with internal references

- without requiring a central ID allocation service• Catalog fragments created independently can later be merged

without modification to corresponding data file

Logical NamingLogical Naming

Object LookupObject Lookup

FileIDLFN1LFN1 PFN1, technologyPFN1, technology

LFN2LFN2

LFNnLFNn

PFN2, technologyPFN2, technology

PFNn, technologyPFNn, technology

File Identity and File Identity and metadatametadata


StorageElement

EDG Replica Location Services - Basic Functionality

Replica ManagerReplica Location

Service

Replica Metadata Catalog

StorageElement

Files have replicas stored at many Grid sites on Storage Elements.

Each file has a unique GUID.Locations corresponding to the GUID are kept in the Replica Location Service.

Users may assign aliases to the GUIDs. These are kept in the Replica Metadata Catalog.

The Replica Manager provides atomicity for file operations, assuring consistency of SE and catalog contents.

jam

es.

case

y@

cern

.ch


StorageElement

Interactions with other Grid Middleware Components

Replica ManagerReplica Location

Service

Replica Optimization Service

Replica Metadata Catalog

SEMonitor

Network Monitor

Information Service

Resource Broker

User Interface orWorker Node

StorageElement

Virtual OrganizationMembership Service

Applications and users interface to data through the Replica Manager either directly or through the Resource Broker.

jam

es.

case

y@

cern

.ch


RLS Service Goals

• To offer production quality services for LCG 1 to meet the requirements of forthcoming (and current!) data challenges

• e.g. CMS PCP/DC04, ALICE PDC-3, ATLAS DC2, LHCb CDC’04

• To provide distribution kits, scripts and documentation to assist other sites in offering production services

• To leverage the many years’ experience in running such services at CERN and other institutes

• Monitoring, backup & recovery, tuning, capacity planning, …

• To understand experiments’ requirements in how these services should be established, extended and clarify current limitations

• Not targeting small-medium scale DB apps that need to be run and administered locally (to user)


Conclusions

• Data Management at LHC remains a significant challenge because of data volume, project lifetime, complexity of S/W and H/W setups.

• The LHC Computing Grid (LCG) approach is based on eg the EDG and GLOBUS Middleware projects and uses a strict component approach for physics application software

• The LCG-POOL project has developed a technology neutral persistency framework which is currently being integrated into the experiment production systems

• In conjunction with POOL a data catalog production service is provided to support several upcoming data productions in the 100 of terabyte area


LHC Software Challenges

• Experiment software systems are large and complex• Developed by teams of expert developers• Permanent evolution and improvement for years…

• Analysis is performed by many end user developers• Often participating only for short time • Usually without strong computer science background• Need simple and stable software environment

• Need to manage change over a long project lifetime • Migration to new software, implementation languages• New computing platforms, storage media• New computing paradigms ???

• Data management system needs to be designed such confine the impact of unavoidable change during the project

Documents

Data Management at CERN’s Large Hadron Collider (LHC) Dirk D ü llmann CERN IT/DB, Switzerland