5 November 2001 F Harris GridPP Edinburgh 1

WP8 status for validating Testbed1 and middleware

F Harris(LHCb/Oxford)

http://datagrid-wp8.web.cern.ch/DataGrid-WP8/

5 November 2001 F Harris GridPP Edinburgh 2

Overview of presentation

• Overview of WP8 and Testbed1 validation planning

• Updated plan – WP6 and WP8 cooperation-the ‘loose cannons’

• Examples of specific planning for Datagrid Testbed1 validation by an experiment

• The way forwards on integrating the services and

enhancing the architecture

• Conclusions

5 November 2001 F Harris GridPP Edinburgh 3

WP8 and Middleware release status (Testbed1)

• An integration team has been putting together products from the various work packages and doing basic tests. (Bob Jones will give overall status here). Work ongoing……

• Documentation exists (variable quality) for the middleware. HEP Applications have conducted a survey. See …

– http://datagrid-wp8.web.cern.ch/DataGrid-WP8/Testbed1/Doc_Assessment/middleware_documentation_assessm.htm

5 November 2001 F Harris GridPP Edinburgh 4

Experiment Testbed Validation plan(updated on October 29)

8 15 22 29 5 12 19 26 3 10 17

Doc review

WP1 M.Reale

WP2 J.Templon

WP3 I.Augustin

WP4 A.De Salvo

WP5 JJ.Blaising

Integration assessment1 person / application

Application integration + Test plan

1 person / application

TWG + WP8-10 General Meeting

Validation plans presented

WP8 staffVO set up

Basic testing (?)

Message to WP9 & WP10

Software integration

WP8

WP9

WP10

Party!

Start writing 8.2

5 November 2001 F Harris GridPP Edinburgh 5

WP8 generic testing and helping WP6 deploy software

• As per the original plan the WP8 ‘loose cannons’ (LCs) will run a series of generic tests which mimic typical HEP jobs (job submission, file transfer,copying)..

• In addition(this was not foreseen) WP8 will help deployment of software in Testbed1 by taking responsibility for

– Replica catalog (one per VO)– Resource broker (seems we can get away with one to

start)– VO user LDAP9 (one/expt at Nikhef)

5 November 2001 F Harris GridPP Edinburgh 6

What WP8 need from WP6• Installation of basic PM9 software on testbed sites

– We should be notified as soon as a new site is WP6 certified

– Start with sites that are ready to participate and progressively expand to all the 40 sites!

– Take into account the experiment priority list

• Run and operate all services required by WP8 to be able to use the Testbed (including LDAP and other servers)

• Manage the relations with the local site administrator

• Support the use of the TestBed

5 November 2001 F Harris GridPP Edinburgh 7

Validation of middleware using existing MC production system

Submit jobs remotelyvia Web

Executeon farm

Monitorperformanceof farm viaWeb

Update bookkeepingdatabase

Transfer data toCASTOR (and HPSS, RAL Datastore)

Data Quality Check ‘Online’

WP 1 job submission tools WP 4

environment

WP 1 job submission

tools

WP 3 monitoring

tools

WP 2 data replication

WP 5 API for mass storage

Online histogram production using

GRID pipes

WP 2 meta data tools

WP1 tools

5 November 2001 F Harris GridPP Edinburgh 8

Operative Procedure for Atlas (S Resconi,Milan)

ATLAS Testbed0 kit is suited to start No change in the code (script level only) for first STEPS which will be devoted to test WP1 functionalities After first STEPS have been successful, use GDMP (by WP2) for file transfer Integrate in the software WP2 API to access the Globus Replica Catalog Think to implement a NEW ATLAS kit that should include ATHENA framework, ATLFASTOO (C++ fast simulation program), Objectivity output and integrate WP2 API near end of November if everything is OK !

In the meanwhile, enlarge gradually the Validation setup: starting from the 2 sites already involved in the integration passing to 4 sites (Lyon, Cern, RAL, CNAF) involving, if possible, also second round sites

5 November 2001 F Harris GridPP Edinburgh 9

Testbed1 sites for experiments

• First round priority sites

– CERN, CCIN2P3/Lyon, CNAF/Bologna,RAL

• 15-17 EU sites for the second round

– UK 6 sites Bristol ,Edinburgh, Glasgow, Lancaster, Liverpool, Oxford

–

5 November 2001 F Harris GridPP Edinburgh 10

Meetings of experiment software architects and middleware managers

First meeting (Sep 24,2001) 10:00 F.Carminati Introduction10:20 LHCb architecture (P.Mato)10:55 CMS architecture (V.Innocente)11:30 ATLAS architecture (C.Tull)12:05 Lunch14:00 ALICE architecture (P.Buncic/F.Rademakers)14:35 Presentations by middleware packages (10’) Discussion 

• Define interactions between applications and middleware to develop architecture

• Evolution of delivered software: what we will get in PM9 and how will it evolve? Tightening up interfaces for existing services (e.g. duplication of functions…)

• Next in series will be late November

5 November 2001 F Harris GridPP Edinburgh 11

GAUDI/GANGA use of Grid services (P Mato)

GAUDI Program

GANGAGU

I

Collective&

ResourceGrid

Services

HistogramsMonitoringResults

JobOptionsAlgorithms

• Translation between high-level input file specification to JDL. Is it at all possible?

• Resource estimation and allocation (WP1)• Job scheduling (WP1)

• Job Monitoring (WP1, WP3)• Logging (WP1)

• File Data Access (local and remote) (WP2)

• Replica Location and LFN translation (WP2)

5 November 2001 F Harris GridPP Edinburgh 12

Conclusions• Integration still going on – hopefully released for generic tests by

applications during week of Nov 5. Released to experiments mid-late November?

• WP8 are working with WP6 to help get system deployed (e.g. Resource Broker, Resource Catalogues, LDAP servers). Hopefully by Testbed2 all will be done by WP6 organisation

• The way forward for proper integration of Grid services is by having an architecture activity embracing the experiment s/w architects and the work packages. Lets look for activities common to all experiments (I.e. independent of s/w framework). All experiments (Alice,Atlas,CMS,LHCB) agree on this. Next meeting end of November.

• The UK should enhance its efforts in WP8. And there should be close working with WP6 in UK.