DESY, 10 May 2004

“The ARDA project: Grid analysis prototypes of the LHC

experiments”Massimo LamannaARDA Project Leader

Massimo.Lamanna@cern.ch

http://cern.ch/arda

www.eu-egee.org cern.ch/lcgEGEE is a project funded by the European Union under contract IST-2003-508833

DESY, 10 May 2004 - 2

Contents

• A bit of history…LHC experiments and the LCG projectEGEE project

• ARDA ProjectMandate and organisation

• ARDA activities during 2004Now…Second half of 2004

• Conclusions and Outlook

DESY, 10 May 2004 - 3

LHC Experiments

CMSATLAS

LHCbLHCbALICE

Storage –Raw recording rate 0.1 – 1 GByte/s

Accumulating at 5-8 PetaByte/year

10 PetaByte of disk

Processing –200,000 of today’s fastest PCs

Multi-Tiered View of LHC Computing

2.5-10Gbs

2.5-10Gbs

1-10Gbs

FranceUK Italia

JapanUSDeutschland

DESY, 10 May 2004 - 5

The LHC Computing Grid Project

• Prepare and deploy the computing environment for the LHC experiments

• Common applications, tools, frameworks and environments,• Move from testbed systems to real production services:

Experiments need a dependable system: • Operated and supported 24x7 globally• Computing fabrics run as production physics services• Computing environment must be robust, stable, predictable, and supportable

• Foster collaboration, coherence of the LHC computing centres• LCG is not a grid technology R&D project:

• Enable physics data analysis and distributed collaboration to a new scale

DESY, 10 May 2004 - 6

The LHC Computing Grid Project:Phase 1 and Phase 2

Phase 1 – 2002-05Development and prototypingApproved by CERN Council 20 September 2001

Phase 2 – 2006-08Installation and operation of the full world-wide initial production Grid

• Exploiting Phase 1 experience

Costs (materials + staff) included in the LHC cost to completion estimates

DESY, 10 May 2004 - 7

The LCG Phase 1: Goals

• Prepare the LHC computing environmentProvide the common tools and infrastructure for the physics application softwareEstablish the technology for fabric, network and grid managementOperate a series of data challenges for the experimentsBuild a solid collaboration and a fertile exchange of experience within the community of the centres contributing to the LCG.Validate the technology and models by building progressively more complex Grid prototypesDevelop models for building the Phase 2 GridMaintain reasonable opportunities for the re-use of the results of the project in other fields

• Deploy a 50% model* production GRID including the committed LHC Regional Centres

• Produce a Technical Design Report for the full LHC Computing Grid to be built in Phase 2 of the project

* 50% of the complexity of one of the LHC experiments

DESY, 10 May 2004 - 8

Too early?

• First collisions in Spring 20071 year to procure, install, and test the full LHC computing fabrics

• Infrastructure work like civil engineering already startedThe Computing TDR must be ready in mid-2005At least 1 year of experience in operating a production grid to validate the computing model

• Experiments’ data challenges should run within LCG in 2004

With a reasonable level of “production” service

• How do we evolve the present services (LCG-2) into the final system?

DESY, 10 May 2004 - 9

The EGEE project• Create a European-wide Grid production quality

infrastructure for multiple sciences• Profit from current and planned national and regional

Grid programmes, building onthe results of existing projects such as DataGrid (EDG), LCG and othersEU Research Network and industrial Grid developers

• Support Grid computing needs common to the different communities

integrate the computing infrastructures and agree oncommon access policies

• Exploit International connections (US and AP)Provide interoperability with other major Grid initiatives such as the US NSF Cyberinfrastructure, establishing aworldwide Grid infrastructure

• Leverage national resources in a more effective way• 70 leading institutions in 27 countries

(including Russia and US)

DESY, 10 May 2004 - 10

EGEE Scope

• The project started April 2004First phase will last 2 years with EU funding of ~32M€

Possibility of 2nd phase if successful• EGEE Scope : ALL-inclusive for academic applications• Open to industrial and socio-economic world as well

Industrial participation both as potential end-users and IT technology and service suppliersEGEE organises an Industry Forum to keep Industrial and Commercial parties in close contactServices developed in 2004-5 may be tendered to Industry in the second phase (2006-7)

• The major success criterion of EGEE: how many satisfied users from how many different domains ?

5000 users from at least 5 disciplines2 Pilot Application Domains: Physics & Bioinformatics

DESY, 10 May 2004 - 11

EGEE and LCG

• Strong links already established between EDG and LCG and this approach will continue in the scope of EGEE

• The core infrastructure of the LCG and EGEE grids will be operated as a single service, and will grow out of LCG service

LCG includes US and AsiaEGEE includes other sciences Substantial part of infrastructure common to both

• Parallel production linesLCG-2

• 2004 data challengesPre production prototype

• EGEE MW• ARDA playground Globus 2 based Web Service based

EGEE-2EGEE-1LCG-2LCG-1

EDGVDT. . .

LCG EGEE. . .

DESY, 10 May 2004 - 12

ARDA working group recommendations

• New service decompositionStrong influence of Alien system

• the Grid system developed by the ALICE experiments and used by a wide scientific community (not only HEP)

• Role of experience, existing technology…Web service framework

• Interfacing to existing middleware to enable their use in the experiment frameworks

• Early deployment of (a series of) prototypes to ensure functionality and coherence

EGEE Middleware

ARDA project

DESY, 10 May 2004 - 13

Web Services• Web services:

The term Web services describes a standardized way of integrating Web-based applications using the XML, SOAP, WSDL and UDDI open standards over an Internet protocol backbone. XML is used to tagthe data, SOAP is used to transfer the data, WSDL is used for describing the services available and UDDI is used for listing what services are available. Used primarily as a means for businesses to communicate with each other and with clients, Web services allow organizations to communicate data without intimate knowledge of each other's IT systems behind the firewall.

Unlike traditional client/server models, such as a Web server/Web page system, Web services do not provide the user with a GUI. Web services instead share business logic, data and processes through a programmatic interface across a network. The applications interface, not the users. Developers can then add the Web service to a GUI (such as a Web page or an executable program) to offer specific functionality to users.

Web services allow different applications from different sources to communicate with each other without time-consuming custom coding, and because all communication is in XML, Web services are not tied to any one operating system or programming language. For example, Java can talk with Perl, Windows applications can talk with UNIX applications.

N.B.: Web services do not require the use of browsers or HTML.

From: http://www.webopedia.com

DESY, 10 May 2004 - 14

End-to-end prototypes: why?

• Provide a fast feedback to the EGEE MW development teamAvoid uncoordinated evolution of the middlewareCoherence between users expectations and final product

• Experiments ready to benefit from the new MW as soon as possibleFrequent snapshots of the middleware availableExpose the experiments (and the community in charge of the deployment) to the current evolution of the whole systemExperiments system are very complex and still evolving

• Move forward towards new-generation real systems (analysis!)Prototypes should be exercised with realistic workload and conditions

• No academic exercises or synthetic demonstrations• LHC experiments users absolutely required here!!!

A lot of work (and useful software) is involved in current experiments data challenges: this will be used as a starting point

• Adapt/complete/refactorise the existing: we do not need another system!

DESY, 10 May 2004 - 15

E2E Prototypes implementation

• Every experiment has already at least one systemAnalysis/Production typically distinct entitiesUsing a variety of back-ends (Batch systems, different grid systems)

• ARDA will put its effort on the experiment (sub)system the experiment choosesEGEE MW as foundation layer

• Multigrid interfaces outside our scope• Experiments do know how to deal with this

By default, we expect 4 systems

• There is nothing like an ARDA prototypeAdapt/complete/refactorise the existing (sub)system!Collaborative effort (not a parallel development)Commonality is not ruled out, but it should emerge and become attractive for the experiments. Anyway not imposed “from outside”Users users users!!!First important checkpoint: December 2004

DESY, 10 May 2004 - 16

Experiment End-to-End Prototypes

• The initial prototype will have a reduced scopeComponents selection for the first prototypeExperiments components not in use for the first prototype are not ruled out (and used/selected ones might be replaced later on)Not all use cases/operation modes will be supported

• Attract and involve usersMany users are absolutely required

• The Use Cases are still being definedExample:

• A physicist selects a data sample (from current Data Challenges)• With an example/template as starting point (s)he prepares a job to scan the data• The job is split in sub-jobs, dispatched to the Grid, some error-recovery is

automatically performed, merged back in a single output• The output (histograms, ntuples) is returned together with simple information on

the job-end status

DESY, 10 May 2004 - 17

E2E Prototypes

EGEE Middleware Interface Layer

FileCatalog CE WorkloadFileCatalogFileCatalog

Exp

erim

ent

softw

are

Other systems in use(LCG2, G2003, NorduGrid,

LSF, PBS, …)

• Each experiment chooses the “starting point” (1 system)

Subset of the existing systemEmphasis on analysisEGEE MW as foundation layer

• There is nothing like an ARDA prototype!

Adapt/complete/refactorise the existing one together with the experiments’teams

• The initial prototype will have a reduced scope

Just the most sensible starting point

SE

Exp

erim

ent-s

peci

fic

mid

dlew

are

“Gen

eric

”m

iddl

ewar

e

DESY, 10 May 2004 - 18

ARDA Project current set up

• LCG:Project leader (Massimo Lamanna/CERN)4 LCG staff (100% at CERN) matching the 4 EGEE staff1 more staff from LCG (100% at CERN)About 4 FTEs from other sources (20% at CERN)

• EGEE:4 NA4 staff (100% at CERN)

• Experiments:4 experiments interfaces

• Represent the experiments in project definition, implementation and evaluation• Identify and coordinate the experiment contributions

– analysis groups in the experiments with whom the middleware people can work to specify the services and validate the implementations

– “upper middleware” teams (experiment-specific MW)Users

Exp.System

Strong link with exp. teams

Strong link with exp. teams

Strong link with regional centres

Strong link with exp. teams

DESY, 10 May 2004 - 19

People• Massimo Lamanna• Birger Koblitz

• Dietrich Liko• Frederik Orellana

• Derek Feichtinger• Andreas Peters

• Julia Andreeva• Juha Herrala

• Andrew Maier• Kuba Moscicki

• Andrey Demichev• Viktor Pose

• Wei-Long Ueng• Tao-Sheng Chen

Russia

Taiwan

Experiment interfaces

Piergiorgio Cerello (ALICE)David Adams (ATLAS)Lucia Silvestris (CMS)Ulrik Egede (LHCb)

ALICE

ATLAS

CMS

LHCb

DESY, 10 May 2004 - 20

ARDA @ Regional Centres

• “Deployability” is a key factor of MW success• A few Regional Centres will have the responsibility to provide early

installation for ARDAUnderstand “Deployability” issuesExtend the ARDA test bedThe ARDA test bed will be the next step after the most complex “EGEE Middleware” test bed

• Stress and performance tests could be ideally located outside CERNThis is for experiment-specific components (e.g. a Meta Data catalogue)Leverage on Regional Centre local know how

• Data base technologies• Web services• …

Pilot sites might enlarge the resources available and give fundamental feedback in terms of “deployability” to complement the EGEE SA1 activity (EGEE/LCG operations)

DESY, 10 May 2004 - 21

Coordination and forum activities

• The coordination activities would flow naturally from the fact that ARDA will be open to provide demonstration benches

Since it is neither necessary nor possible that all projects could be hosted inside the ARDA experiments’ prototypes, some coordination is needed to ensure that new technologies can be exposed to the relevant community

• Transparent processARDA should organise a set of regular meetings (one per quarter?) to discuss results, problems, new/alternative solutions and possibly agree on some coherent program of work.The ARDA project leader organises this activity which will be truly distributed and lead by the active partners

• Special relation with LCG GAGLCG forum for Grid requirements and use casesExperiments representatives coincide with the EGEE NA4 experiments representatives

• ARDA will channel this information to the appropriate recipients

ARDA workshop (January 2004 at CERN; open; over 150 participants)ARDA workshop (June 21-23 at CERN; by invitation)

• “The first 30 days of EGEE middleware”ARDA workshop (September 2004?; open)

DESY, 10 May 2004 - 22

Coordination and forum activities

ALICEDistr. Analysis

EGEEmiddleware

ResourceProviders

Community

ATLASDistr. Analysis

CMSDistr. Analysis

LHCbDistr. Analysis

SEAL

PROOF

GAE

POOL

ARDACollaborationCoordinationIntegrationSpecificationPrioritiesPlanning

Experience Use Cases

EGEE NA4Application

identification and support

LCG-GAGGrid Application

Group

DESY, 10 May 2004 - 23

Plans and activity within the experiments

• General patternPlanningExample

• LHCb• CMS• ATLAS• ALICE

DESY, 10 May 2004 - 24

Example of activity

• Existing system as starting pointEvery experiment has different implementations of the standard services Used mainly in production environments

• Few expert users• Coordinated update and read actions

ARDA• Interface with the EGEE middleware• Verify (help to evolve to) such components to analysis environments

– Many users» Robustness

– Concurrent “read” actions» Performance

• One prototype per experimentA Common Application Layer might emerge in future

• ARDA emphasis is to enable each of the experiment to do its job

Very soonVery soon

Already startedAlready started

Milestone Date Description

1.x.1 May 2004 E2E x prototype definition agreed with the experiment

1.x.2 September 2004 E2E x prototype using basic EGEE middleware

1.x.3 November 2004 E2E x prototype improved functionality

1.x December 2004 E2E prototype for experiment x, capable of analysis

2.x December 2005 E2E prototype for experiment x, capable of analysis and production

DESY, 10 May 2004 - 25

LHCb

• The LHCb system within ARDA uses GANGA as principal component.

• The LHCb/GANGA plans to enable physicists to use GANGA to analyse the data being produced during 2004 for their studies naturally matches the ARDA mandate

• At the beginning, the emphasis will be to validate the tool focusing on usability, validation of the splitting and merging functionality for users jobs

• The DIRAC system (LHCb grid system, used mainly in production so far, could be a useful playground to understand the detailed behaviour of some components, like the file catalog)

DESY, 10 May 2004 - 26

GANGAGaudi/Athena aNd Grid Alliance

• Gaudi/Athena: LHCb/ATLAS frameworksThe Athena uses Gaudi as a foundation

• Single “desktop” for a variety of tasks

• Help configuring and submitting analysis jobs

• Keep track of what they have done, hiding completely all technicalities

Resource Broker, LSF, PBS, DIRAC, CondorJob registry stored locally or in the roaming profileAutomate config/submit/monitor procedures

• Provide a palette of possible choices and specialized plug-ins (pre-defined application configurations, batch/grid systems, etc.)

• Friendly user interface (CLI/GUI) is essentialGUI Wizard Interface

• Help users to explore new capabilities• Browse job registry

Scripting/Command Line Interface• Automate frequent tasks• python shell embedded into the Ganga GUI

GAUDI Program

GANGAGUI

JobOptionsAlgorithms

Collective&

ResourceGrid

Services

HistogramsMonitoringResults

Grid Services

GANGA

UI

BkSvc

BookkeepingService

WorkLoadManager

SE Filecatalog

WLM ProSvc Monitor

Internal Model

ProfileService

GAUDIProgram

Instr.

CE

DESY, 10 May 2004 - 27

ARDA contribution to Ganga

• Integration with EGEE middlewareWaiting for the EGEE middleware, we developed an interface to CondorUse of Condor DAGMAN for splitting/merging and error recovery capability

• Design and DevelopmentCommand Line InterfaceFuture evolution of Ganga

• Release managementSoftware process and integration

• Testing, tagging policies etc.Infrastructure

• Installation, packaging etc.

DESY, 10 May 2004 - 28

LHCb Metadata catalog

• Used in production (for large productions)• Web Service layer being developed (main developers in the UK)• Oracle backend• ARDA contributes a “testing” focused on the analysis usage

RobustnessPerformances under high concurrency (read mode)

Measured network rate vs no. of concurrent clients

DESY, 10 May 2004 - 29

Client

TAIWAN

CERN

Oracle DB

Oracle DB

Bookkeeping Server

Bookkeeping Server CPU Load

Network Process time

Web & XML-RPC Service performance tests

CPU LoadNetwork Process time

DB I/O Sensor

Network monitorVirtual Users

CERN/Taiwan tests

• Clone Bookkeeping DB in Taiwan• Install the WS layer• Performance Tests

Database I/O SensorBookkeeping Server performance tests

• Taiwan/CERN Bookkeeping Server DB• XML-RPC Service performance tests• CPU Load, Network send/receive sensor, Process time

Client Host performance tests• CPU Load, Network send/receive sensor, Process time

DESY, 10 May 2004 - 30

CMS

• The CMS system within ARDA is still under discussionThis Wednesday CMS session during CMS software week…

• It is already clear that the complex RefDB system (the heart of the data challenge DC04, recently finished) will be one of the area of collaboration between CMS and the corresponding ARDA team

RefDB is the bookkeeping engine to plan and steer the production across different phases (simulation, reconstruction, to some degree into the analysis phase) . It contained all necessary information except file physical location (RLS) and information related to the transfer management system (TMDB).Measuring performances underway (similar philosophy as for the LHCb Metadata catalog measurements)

DESY, 10 May 2004 - 31

DC04 data flow at T0 (CERN)

RefDB McRunjobT0 worker

nodes

GDB castor poolTapes

ExportBuffers

Transfer agent

RLS TMDB

Reconstruction instructions

Reconstructionjobs

Reconstructeddata

Reconstructeddata

Checks what hasarrived

Updates Updates

Summaries of successful jobs

DESY, 10 May 2004 - 32

ATLAS

• The ATLAS system within ARDA has been agreedATLAS has a complex strategy for distributed analysis, adressingdifferent area with specific projects (Fast response, user-driven analysis, massive production, etc…: see http://www.usatlas.bnl.gov/ADA/)Starting point is the DIAL system

• The AMI metadata catalog is a key componentmySQL as a back endGenuine Web Server implementationRobustness and performance tests from ARDA

• In the start up phase, ARDA provided some help in developing ATLAS production tools

Finishing…

DESY, 10 May 2004 - 33

What is DIAL?

Interactive analysise.g. ROOT, JAS, ...

DIAL

Distributed processing running data-specific application

Dataset Scheduler AAAJob

DESY, 10 May 2004 - 34

ATLAS Metadata Catalog (AMI)

Use

rU

ser

Use

r

SOA

P-Pr

oxy

SOA

P-Pr

oxy

Met

a-D

ata

(MyS

QL

)

Met

a-D

ata

(MyS

QL

)M

eta-

Dat

a(M

ySQ

L)

Planned

• Atlas Metadata- Catalogue, contains File Metadata:

Simulation/Reconstruction-VersionFile-Content:Event typesDoes not contain physical filenames

• SOAP-Proxy (in Java) front-end to hierarchical databases (institute →collaboration)

• Proxy allows database schema evolution

• SOAP allows automatic code generation for client

DESY, 10 May 2004 - 35

AMI studies in ARDA

SOA

P-Pr

oxy

Met

a-D

ata

(MyS

QL

)

Use

rU

ser

Use

r

• Many problems still open:• Large network traffic overhead due to schema independent tables• SOAP proxy supposed to provide DB properties:

Browsable resultsNote that Web Services are “stateless” (not automatic handles to have the concept of session, transaction, etc…): 1 query = 1 (full) responseLarge queries crashed server

• Shall proxy re-implement all database functionality?• Nice collaboration in place with ATLAS-Grenoble

Studied behaviour using many concurrent clients:

DESY, 10 May 2004 - 36

ATLAS ATCOM

• AtCom II: planned successor of AtCom

• Graphical interactive tool to support production management in ATLAS

• Large scale job definition, submission and progress monitoring

• Linked to several bookkeeping databases (AMI and Magda)

• Plug-ins for LSF, EDG and Nordugrid

DESY, 10 May 2004 - 37

ALICE

• The ALICE system within ARDA will be the evolution of the analysis system presented by ALICE at SuperComputing2003 (SC2003)

With the new EGEE middleware (at SC2003, AliEn was used)

• Some activity on the PROOF systemRobustnessError recovery

DESY, 10 May 2004 - 38

ALIEN system/Grid enabled PROOF (SC2003 Demo)

PROOFPROOF

PROOF PROOF SLAVESSLAVES

PROOF MASTERPROOF MASTERSERVERSERVER

PROOF PROOF SLAVESSLAVESPROOF PROOF SLAVESSLAVES

TcpRouter

TcpRouterTcpRouter

TcpRouter

Site A

Site C

Site B

USER SESSIONUSER SESSION

DESY, 10 May 2004 - 39

ALICE-ARDA prototype improvements

• SC2003:The setup was heavily connected with the middleware servicesSomewhat “inflexible” configurationNo chance to use PROOF on federated grids like LCG in AliEnTcpRouter service needs incoming connectivity in each siteLibraries can not be distributed using the standard rootd functionality

• Improvement ideas:Distribute another daemon with ROOT, which replaces the need for aTcpRouter serviceConnect each slave proofd/rootd via this daemon to two central proofd/rootd master multiplexer daemons, which run together with the proof masterUse Grid functionality for daemon start-up and booking policies througha plug-in interface from ROOTPut PROOF/ROOT on top of the grid servicesImprove on dynamic configuration and error recovery

DESY, 10 May 2004 - 40

ALICE-ARDA improved system

• The remote proof slaves looklike a local proof slave onthe master machine

• Booking service is usable also on local clusters

PROOFPROOF

PROOF PROOF SLAVE SLAVE SERVERSSERVERS

Proxy proofd Proxy rootd

Master

Grid ServicesBooking

DESY, 10 May 2004 - 41

Conclusions and Outlook

• ARDA is startingMain tool: experiment prototypes for analysisDetailed project plan being preparedGood feedback from the LHC experimentsGood collaboration with EGEE NA4Good collaboration with Regional Centres

• Look forward to contribute to the success of EGEEHelping EGEE Middleware to deliver a fully functionally solution

• ARDA main focusCollaborate with the LHC experiments to set up the end-to-end prototypesAggressive schedule

• First milestone for the end-to-end prototypes is Dec 2004

DESY, 10 May 2004 - 42

Links

• LCG:http://cern.ch/lcg

• EGEE:www.eu-egee.orgNA4 (Application Identification and Support): http://egee-na4.ct.infn.it/index.phpNA4 HEP:http://egee-na4.ct.infn.it/hep/

• ARDA:http://cern.ch/arda

• GAG:http://project-lcg-gag.web.cern.ch/project-lcg-gag/