Metadata: ARDA Implementation
PrototypeValidate our ideas and expose a concrete example to interested partiesMultiple back endsCurrently: Oracle, PostgreSQL, SQLite, MySQLDual front endsTCP Streaming Chosen for performanceSOAPFormal requirement of EGEECompare SOAP with TCP StreamingAlso implemented as standalone Python libraryData stored on the file system
Massimo Lamanna / CERN
Python Interpreter
Metadata Python API
Client
filesystem
Database
MDServer
Metadata Server
SOAP
TCP Streaming
Oracle
PostgreSQL
SQLite
Client
Client
INFSO-RI-508833
NA4 biomedical gLite tests on the PPSPPS components tested: Weekly testing of WMS, FiReMan, security related components (ACLs for file access), R-GMAOther gLite testingSite installation (v1.0, 1.1, 1.2 and 1.3)Advanced data management on prototype testbedMedical ImagerTrigger:Retrieve DICOM files from imager.Register file in FiremangLite EDS client for data encryptionRegister Metadata in AMGAClient Library:Lookup file through Metadata (AMGA)Use gLite EDS client for data decryption Serve it up to the application
INFSO-RI-508833
All gLite Services are available and tested on GILDAgLite Services
INFSO-RI-508833
2. The capture of requirements, and the assessment of middlewareThe status at PM18
A database of requirements has been constructed by the Project Technical Forum(PTF)This database already contains >400 requirementsThis database is populated by the scientific communities deploying applications on EGEE and by the FP6 projects (eg Diligent, Seegrid,...)
The capture of requirements from FP5 projects
Several FP5 projects were granted access to the database (Grace, Mammogrid)But it has proved to be difficult to collect their requirementsNo formalization of requirementsProjects finishedRequirements were collected mainly from the scientific communities (Crossgrid, Datagrid)
The assessment of middleware w.r.t requirements
There is on-going work with JRA1 within the framework of the PTF and the TCG
INFSO-RI-508833
3. The clarifification of the true motivation of usersAn improved Memorandum of Understanding(MOU) between EGEE and the scientific communities has been designed and implemented
Detailed SA1 questionnaire to evaluate the required resourcesRegular phone conferences with the Operation Advisory Group
An improved application integration process is under design
Creation of lightweight recognised VOs vs supported Vos
Only supported VOs will be asked to write an MOU
Decentralised integration vs centralised EGAAP-like approach
Initial deployment will be at regional level
INFSO-RI-508833
The status of the MoUs with the generic supported applications
INFSO-RI-508833
The status of generic applications deployment on EGEEApplications accepted by the Project Executive Board(PEB)
Earth Science Research (Earth Observation, Hydrology, Climate)Geophysics (Industry)Computational ChemistryAstrophysics (MAGIC and Planck collaborations) Finance (EGRID)
New Applications recommended to the PEB by the EGEE Generic Applications Advisory Panel (EGAAP)
Fusion (ITER)ArchaeologyEC projects (EELA, EUMEDGRID, EUCHINAGRID, BIOINFOGRID)
INFSO-RI-508833
4. The evaluation of user satisfactionFirst user survey at PM1534 users filled a questionnaire available onlineResults available in DNA4.3.2User satisfaction is good for user supportUser satisfaction on grid services is intermediateUsers would like improvements in documentation (with the exception of system manuals which are well appreciated)Action: creation of the User information Group to address issues related to documentation in the project Next user survey at PM21
INFSO-RI-508833
5. Demonstration of the scientific benefits of the EGEE gridThe number of users in the VOs related to NA4 activity has continued to grow
from ~500 at PM9 to ~1000 at PM18 More than 20 applications are deployed on the production infrastructure
WE HAVE DEMONSTRATED
added value in terms of deployment scale
The very large scale production of simulated data for HEP experiments
In particular this has been essential for the commissioning of the LHC experiments
added value in terms of response time
The computation of the seismic epicentre by ESRThe search for new drugs: large scale in silico docking (WISDOM) - access to large scale resources gives much improved turnaround
added value in terms of data distribution
The distribution of LHC data on the grid, and the large users activity (~150k jobs/month in the CMS case)
INFSO-RI-508833
Large Scale Production on EGEE by several HEP experiments Fundamental activity in the preparation of the LHC experiments start up
Physics and detector studiesComputing systems commissioning
Examples:
ATLAS: Ran on 3 grids (2/3 of work on EGEE) Peaks of 10,000 jobs per day on EGEE200 CPU years in 2005 ; 45 TB dataComprehensive analysis: see S.Campana et al., e-Science 2005, Melbourne, Australia
LHCb:Peaks of 5,500 jobs/day on EGEE700 CPU/years in 2005 ; 77 TB of data
A lot of activity within non-LHC experiments (BaBar,CDF, D, ZEUS,H1)
More details in DNA4.3.2ATLAS production evolution 2004-2005
INFSO-RI-508833
First biomedical data challenge: World-wide In Silico Docking On Malaria (WISDOM)
Significant biological parameterstwo different molecular docking applications (Autodock and FlexX)about one million virtual ligands selectedtarget proteins from the parasite responsible for malariaSignificant numbers Total of about 46 million ligands docked in 6 weeks1TB of data produced Up 1000 computers in 15 countries used simultaneously corresponding to about 80 CPU yearsAverage crunching factor ~600WISDOM open dayDecember 16th, 2005, Bonn (Germany)
Discuss Data Challenge resultsPrepare next steps towards a malaria Grid (EGEE-II, Embrace, Bioinfogrid)Information: http://wisdom.eu-egee.fr
Graph2
597
1072
383
715
5122
7580
2004
263
3687
3356
1877
337
218
827
8106
Domain distribution of Flexx run jobs
flexx
flexx
479
62
56
659
55
283
1072
662
129
151
383
1877
189
56
1759
766
874
1014
811
612
2488
2600
1361
382
133
74
460
97
125
282
644
9
102
2690
1347
4940
3
337
666
715
56
263
827
380
2763
218
233
nb of run jobs
CE distribution for Flexx run jobs
autodock
479
62
56
659
55
283
1072
662
129
151
383
1877
189
56
1759
766
874
1014
811
612
5088
1361
382
133
74
460
97
125
282
644
9
102
2690
1347
4940
3
337
666
715
56
263
827
380
2763
218
233
Site distribution for DC run jobs (Flexx)
paysFlexx
paysFlexx
229
14
286
9
17
174
301
603
757
141
230
1082
143
13
699
108
418
708
436
255
912
1136
445
201
483
5
45
107
9
324
165
59
703
1080
585
1716
414
119
470
128
35
250
465
221
854
17
83
842
nb of run jobs
CE distribution for Autodock run jobs
paysautodock
229
14
286
9
17
174
301
603
757
141
230
1082
143
13
699
108
418
708
436
255
2048
445
201
483
5
45
107
9
324
165
59
703
1080
585
1716
414
119
470
128
35
250
465
221
854
17
83
842
Site distribution of DC Biomed run jobs (Autodock)
data
paysnb
bg479
bg62
bg56
Total bg597bg597
com1072com1072
Total com1072com1072
cy383cy383
Total cy383cy383
de715de715
Total de715de715
es283es283
es662es662
es151es151
es874es874
es9es9
es380es380
es2763es2763
Total es5122es5122
fr55fr55
fr1014fr1014
fr811fr811
fr612fr612
fr2488fr2488
fr2600fr2600
Total fr7580fr7580
gr189gr189
gr56gr56
gr1759gr1759
Total gr2004gr2004
il263il263
Total il263il263
it129it129
it1361it1361
it382it382
it133it133
it74it74
it460it460
it97it97
it125it125
it282it282
it644it644
Total it3687it3687
nl2690nl2690
nl666nl666
Total nl3356nl3356
pl1877pl1877
Total pl1877pl1877
ro337ro337
Total ro337ro337
ru218ru218
Total ru218ru218
tw827tw827
Total tw827tw827
uk659uk659
uk766uk766
uk102uk102
uk1347uk1347
uk4940uk4940
uk3uk3
uk56uk56
uk233uk233
Total uk8106uk8106
Total36144
data
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Domain distribution of Flexx run jobs
AUTODOCKpaysnb
bg229
bg14
Total bg243bg243
com301com301
Total com301com301
cy230cy230
Total cy230cy230
de128de128
Total de128de128
es174es174
es603es603
es141es141
es418es418
es59es59
es221es221
es854es854
Total es2470es2470
fr17fr17
fr708fr708
fr436fr436
fr255fr255
fr912fr912
fr1136fr1136
Total fr3464fr3464
gr143gr143
gr13gr13
gr699gr699
Total gr855gr855
il250il250
Total il250il250
it757it757
it445it445
it201it201
it483it483
it5it5
it45it45
it107it107
it9it9
it324it324
it165it165
Total it2541it2541
nl1080nl1080
nl470nl470
Total nl1550nl1550
pl1082pl1082
Total pl1082pl1082
ro119ro119
Total ro119ro119
ru17ru17
ru83ru83
Total ru100ru100
tw465tw465
Total tw465tw465
uk286uk286
uk9uk9
uk108uk108
uk703uk703
uk585uk585
uk1716uk1716
uk414uk414
uk35uk35
uk842uk842
Total uk4698uk4698
Total18496
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Domain distribution of Autodock run jobs
SiteCEdomainrun jobs
-1BG01-IPPce001.grid.bas.bgbg229AUTODOCKBG01-IPP229
-1BG02-IMce001.imbm.bas.bgbg14BG02-IM14
-1BHAM-LCG2epgce1.ph.bham.ac.ukuk286BHAM-LCG2286
-1CAVENDISH-LCG2serv03.hep.phy.cam.ac.ukuk9CAVENDISH-LCG29
-1CEA-DAPNIA-SACLAYnode03.datagrid.cea.frfr17CEA-DAPNIA-SACLAY17
-1CESGA-EGEEce2.egee.cesga.eses174CESGA-EGEE174
-1CGG-LCG2ce1.egee.fr.cgg.comcom301CGG-LCG2301
-1CIEMAT-LCG2lcg02.ciemat.eses603CIEMAT-LCG2603
-1CNAF-T1ce01-lcg.cr.cnaf.infn.itit757CNAF-T1757
-1CNB-LCG2mallarme.cnb.uam.eses141CNB-LCG2141
-1CY01-LCG2ce101.grid.ucy.ac.cycy230CY01-LCG2230
-1CYFRONET-LCG2zeus02.cyf-kr.edu.plpl1082CYFRONET-LCG21082
-1GR-01-AUTHnode001.grid.auth.grgr143GR-01-AUTH143
-1GR-04-FORTH-ICSgrid001.ics.forth.grgr13GR-04-FORTH-ICS13
-1HG-01-GRNETce01.isabella.grnet.grgr699HG-01-GRNET699
-1IC-LCG2gw39.hep.ph.ic.ac.ukuk108IC-LCG2108
-1IFCA-LCG2egeece.ifca.org.eses418IFCA-LCG2418
-1IN2P3-CCcclcgceli01.in2p3.frfr708IN2P3-CC708
-1IN2P3-CPPMmarseillece01.mrs.grid.cnrs.frfr436IN2P3-CPPM436
-1IN2P3-LALgrid10.lal.in2p3.frfr255IN2P3-LAL255
-1IN2P3-LPCclrce01.in2p3.frfr912IN2P3-LPC2048
-1IN2P3-LPCclrce02.in2p3.frfr1136INFN-BARI445
-1INFN-BARIgridba2.ba.infn.itit445INFN-CAGLIARI201
-1INFN-CAGLIARIgrid002.ca.infn.itit201INFN-CATANIA483
-1INFN-CATANIAgrid012.ct.infn.itit483INFN-CNAF5
-1INFN-CNAFgridit-ce-001.cnaf.infn.itit5INFN-FERRARA45
-1INFN-FERRARAgrid0.fe.infn.itit45INFN-NAPOLI107
-1INFN-NAPOLIgriditce01.na.infn.itit107INFN-PADOVA9
-1INFN-PADOVAprod-ce-01.pd.infn.itit9INFN-ROMA1324
-1INFN-ROMA1t2-ce-01.roma1.infn.itit324INFN-ROMA1-VIRGO165
-1INFN-ROMA1-VIRGOvirgo-ce.roma1.infn.itit165INTA-CAB59
-1INTA-CABce00.inta.eses59ManHEP-LCG2703
-1ManHEP-LCG2bohr0001.tier2.hep.man.ac.ukuk703NIKHEF-ELPROD1080
-1NIKHEF-ELPRODtbn20.nikhef.nlnl1080OXFORD-01-LCG2585
-1OXFORD-01-LCG2t2ce02.physics.ox.ac.ukuk585RAL-LCG21716
-1RAL-LCG2lcgce01.gridpp.rl.ac.ukuk1716RALPP-LCG414
-1RALPP-LCGheplnx201.pp.rl.ac.ukuk414ROGRID-ICI119
-1ROGRID-ICItestbed001.grid.ici.roro119SARA-LCG2470
-1SARA-LCG2mu6.matrix.sara.nlnl470SCAI128
-1SCAIscaicl0.scai.fraunhofer.dede128ScotGRID-Edinburgh35
-1ScotGRID-Edinburghce.epcc.ed.ac.ukuk35TAU-LCG2250
-1TAU-LCG2lcfgng.cs.tau.ac.ilil250Taiwan-LCG2465
-1Taiwan-LCG2lcg00125.grid.sinica.edu.twtw465UPV-GRyCAP221
-1UPV-GRyCAPramses.dsic.upv.eses221pic854
-1picce01.pic.eses854ru-Moscow-SINP-LCG217
-1ru-Moscow-SINP-LCG2lcg06.sinp.msu.ruru17ru-PSN-LCG283
-1ru-PSN-LCG2lcgce.psn.ruru83scotgrid-gla842
-1scotgrid-glace1-gla.scotgrid.ac.ukuk842
1BG01-IPPce001.grid.bas.bgbg479FLEXXBG01-IPP479
1BG02-IMce001.imbm.bas.bgbg62BG02-IM62
1BG04-ACADce01.grid.acad.bgbg56BG04-ACAD56
1BHAM-LCG2epgce1.ph.bham.ac.ukuk659BHAM-LCG2659
1CEA-DAPNIA-SACLAYnode03.datagrid.cea.frfr55CEA-DAPNIA-SACLAY55
1CESGA-EGEEce2.egee.cesga.eses283CESGA-EGEE283
1CGG-LCG2ce1.egee.fr.cgg.comcom1072CGG-LCG21072
1CIEMAT-LCG2lcg02.ciemat.eses662CIEMAT-LCG2662
1CNAF-T1ce01-lcg.cr.cnaf.infn.itit129CNAF-T1129
1CNB-LCG2mallarme.cnb.uam.eses151CNB-LCG2151
1CY01-LCG2ce101.grid.ucy.ac.cycy383CY01-LCG2383
1CYFRONET-LCG2zeus02.cyf-kr.edu.plpl1877CYFRONET-LCG21877
1GR-01-AUTHnode001.grid.auth.grgr189GR-01-AUTH189
1GR-04-FORTH-ICSgrid001.ics.forth.grgr56GR-04-FORTH-ICS56
1HG-01-GRNETce01.isabella.grnet.grgr1759HG-01-GRNET1759
1IC-LCG2gw39.hep.ph.ic.ac.ukuk766IC-LCG2766
1IFCA-LCG2egeece.ifca.org.eses874IFCA-LCG2874
1IN2P3-CCcclcgceli01.in2p3.frfr1014IN2P3-CC1014
1IN2P3-CPPMmarseillece01.mrs.grid.cnrs.frfr811IN2P3-CPPM811
1IN2P3-LALgrid10.lal.in2p3.frfr612IN2P3-LAL612
1IN2P3-LPCclrce01.in2p3.frfr2488IN2P3-LPC50885088
1IN2P3-LPCclrce02.in2p3.frfr2600INFN-BARI1361
1INFN-BARIgridba2.ba.infn.itit1361INFN-CAGLIARI382
1INFN-CAGLIARIgrid002.ca.infn.itit382INFN-CATANIA133
1INFN-CATANIAgrid012.ct.infn.itit133INFN-CNAF74
1INFN-CNAFgridit-ce-001.cnaf.infn.itit74INFN-FERRARA460
1INFN-FERRARAgrid0.fe.infn.itit460INFN-NAPOLI97
1INFN-NAPOLIgriditce01.na.infn.itit97INFN-PADOVA125
1INFN-PADOVAprod-ce-01.pd.infn.itit125INFN-ROMA1282
1INFN-ROMA1t2-ce-01.roma1.infn.itit282INFN-ROMA1-VIRGO644
1INFN-ROMA1-VIRGOvirgo-ce.roma1.infn.itit644INTA-CAB9
1INTA-CABce00.inta.eses9ManHEP-LCG2102
1ManHEP-LCG2bohr0001.tier2.hep.man.ac.ukuk102NIKHEF-ELPROD2690
1NIKHEF-ELPRODtbn20.nikhef.nlnl2690OXFORD-01-LCG21347
1OXFORD-01-LCG2t2ce02.physics.ox.ac.ukuk1347RAL-LCG24940
1RAL-LCG2lcgce01.gridpp.rl.ac.ukuk4940RALPP-LCG3
1RALPP-LCGheplnx201.pp.rl.ac.ukuk3ROGRID-ICI337
1ROGRID-ICItestbed001.grid.ici.roro337SARA-LCG2666
1SARA-LCG2mu6.matrix.sara.nlnl666SCAI715
1SCAIscaicl0.scai.fraunhofer.dede715ScotGRID-Edinburgh56
1ScotGRID-Edinburghce.epcc.ed.ac.ukuk56TAU-LCG2263
1TAU-LCG2lcfgng.cs.tau.ac.ilil263Taiwan-LCG2827
1Taiwan-LCG2lcg00125.grid.sinica.edu.twtw827UPV-GRyCAP380
1UPV-GRyCAPramses.dsic.upv.eses380pic2763
1picce01.pic.eses2763ru-PSN-LCG2218
1ru-PSN-LCG2lcgce.psn.ruru218scotgrid-gla233
1scotgrid-glace1-gla.scotgrid.ac.ukuk233
INFSO-RI-508833
Example of a generic applications use of EGEE (ESR) OzonemapsEarthquakes epicentre determination
INFSO-RI-508833
Conclusions and a forward lookAchievements
The number of users in the VOs related to NA4 activity has doubled
The usage of the grid by the pilot applications has significantly evolved into large scale production
EGAAP applications have also achieved production status on EGEE (ESR,MAGIC)
The migration of several existing applications to gLite has been achieved
A detailed MOU procedure has been defined and executed for several selected applications
The first User Survey has been performed Issues being actively addressed
Reduction of the amount of skilled effort required by the VOs for production use of the grid (EGEE is improving middleware, monitoring, user interfaces and user support)
The availability, accessibility and quality of user documentation (an inter-activity group is working on this)
Looking forwards : User Forum, EGEE-II
INFSO-RI-508833
EGEE first User ForumDates: March 1-3 2006Location: CERN, SwitzerlandTarget attendance: 150 participantsGoalsObtain a consistent understanding across the EGEE related projects of their expectations, present status and possible evolutionLearn how to improve EGEE quality of servicePromote cross-application fertilisationPrepare EGEE-IIParticipation open to external projects and EGEE membersFormat: 3-day workshopPresentations by thematic areas selected by invitation and through a call for contributionsEGEE presentations (integration of new applications, access to resources, status of middleware,...)With a lot of time for discussionURL:http://egee-intranet.web.cern.ch/egee-intranet/User-Forum/index.html
Montrer un slide sur le user survey (evaluation of user satisfaction)Mieux expliciter la user documentation qui manque, Expliquer ce qui est en cours pour repondre au besoin
****ACLs for file access -> which componentChanger ce slide pour mettre les tests sur le PPSMake it explicit why egeode MoU is not finished Adding Diligent
LHCb list of texts to eraseImprove resolution, too heavy, too many images, bad slide
Expliquer pourquoi on a besoin Speak about EGEE-II