Upload
nickolas-baker
View
214
Download
0
Embed Size (px)
Citation preview
Henryk Palka NEC’07 Varna, Sept 2007 1
HEP GRID computing
in
Poland
Henryk PalkaInstitute of Nuclear Physics, PAN, Krakow, Poland
Henryk Palka NEC’07 Varna, Sept 2007 2
● LHC data rates and computing model
● LCG : LHC Computing GRID project
● Polish Grid infrastructure
● Sharing of Central Europe Grid resources
● ATLAS MC production and Data Challanges
at ACC Cyfronet
● BalticGrid project
Topics:
Henryk Palka NEC’07 Varna, Sept 2007 3
LHC experiments data rates
Rate
[Hz]
RAW
[MB]
ESDReco[MB]
AOD
[kB]
MonteCarlo
[MB/evt]
MonteCarlo
% of real
ALICE HI 100 12.5 2.5 250 300 100
ALICE pp 100 1 0.04 4 0.4 100
ATLAS 200 1.6 0.5 100 2 20
CMS 150 1.5 0.25 50 2 100
LHCb 2000
0.025
0.025 0.5 20
107 seconds/year pp from 2008 on (?) ~109 events/experiment106 seconds/year heavy ion
For LHC computing: 100M SpecInt2000 or 100K
of ~3GHz cores is needed!
For data storage: 20 Peta Bytes or 100K of
disks/ tapes per year is needed!
Henryk Palka NEC’07 Varna, Sept 2007 4
LHC Computing Modelorganisation of WLCG
Henryk Palka NEC’07 Varna, Sept 2007 6
LHC Computing Grid project - LCGObjectives
- design, prototyping and implementation of the computing environment for LHC experiments (Monte Carlo simulation, reconstruction and data analysis): - infrastructure (PC farms, networking) - middleware (based on EDG, VDT, gLite….) - operations (experiment VOs, operation and support centres)
Schedule - phase 1 (2002 – 2005; ~50 MCHF); R&D and prototyping (up to 30% of the final size)- phase 2 (2006 – 2008 ); preparation of a Technical Design Report, Memoranda of Understanding, deployment (2007)
Full physics run
2005 20072006 2008
First physics
First beams
Cosmics
2005 20072006 2008
First physics
First beams
Cosmics
Henryk Palka NEC’07 Varna, Sept 2007 7
Planned sharing of capacity between CERN and Regional Centres in 2008
CPU
CERN20%
All Tier-1s39%
All Tier-2s41%
Disk
CERN7%
All Tier-1s58%
All Tier-2s35%
Tape
CERN33%
All Tier-1s67%
Requirements from December 2004 Computing model papers, reviewed by LHCC Jan 05
Pre
limin
ary
pla
nnin
g d
ata
Tape
8
LHC ComputingWLCG is based on few computing scientific grids
9
GTS 1,6 Gb/s
GDAŃSK
POZNAŃ
ZIELONA GÓRA
KATOWICE
KRAKÓW
LUBLIN
WARSZAWA
BYDGOSZCZ TORUŃ
CZĘSTOCHOWA
BIAŁYSTOK
OLSZTYN
RZESZÓW
Bielsko-Biała
GÉANT 10+10 Gb/s
KOSZALIN
SZCZECIN
WROCŁAW
ŁÓDŹ
KIELCEPUŁAWYOPOLE
RADOM
BASNET 34 Mb/s
CESNET, SANET
GÉANT/TELIA 2x2,5 Gb/s
DFN 10 Gb/s Gorzów
MAN
10 Gb/s(1 lambda)
2 x 10 Gb/s
1 Gb/s
CBDF 10 Gb/s
PIONIER’S FIBERS
Polish Grid infrastructure Networking – PIONIER project
Tier1 FZKKarlsruhe
Tier2 PCSSPoznań Tier2 ICM
Warszawa
Tier2 ACKCyfronet Kraków
HEP VLAN1 Gb/s
HEP VLAN1 Gb/s
10
Polish Grid infrastructure Tier2: ACC Cyfronet – ICM – PSNC
Three computing centres contribute to the Polish Tier2 (as part of EGEE/ LCG ROC)
• ACC Cyfronet Krakow• ~300 (450) Pentium 32 bit processors• connected to PSNC via 1 Gbs HEP VLAN
• ICM Warsaw• ~180 (340) AMD-64 Opteron processors• connected to PSNC via 1 Gbs HEP VLAN
•PSNC Poznan • ~240 Itanium IA-64 processors• connected to GEANT and DFN – 10 Gbs
In the hierarchy of WLCG the Polish Tier2 is connected to Tier1 at FZK Karlsruhe
Building Tier3 at IFJ PAN Krakow and IPJ/ FP TU Warsaw
Warsaw-ICM
Cracow-CYFRONET
Poznan-PSNC
11
Polish Grid infrastructure Disk storage at ACC Cyfronet
HP EVA 8000: - 8GB cache; - 8 FC shortwave ports; - 240 FATA 500GB 7200rpm HDDs (120TB)2nd HP EVA with 90 TB underway
12
Polish Grid infrastructure Towards Polish National Grid
Poland has been/ is involved in number of EU Grid projects
• 5 FP: EUROGRID, GridLab, CrossGrid, GridStart, GRIP,…• 6 FP: EGEE, EGEE2, K-WF Grid, BalticGrid, Core GRID, ViroLab, Gredia, int.eu.grid, UNIGRIDS, EUChinaGrid,…
In 2007 five major Polish computing centers (Krakow, Gdansk, Poznan, Warsaw and Wroclaw) signed an agreement to form Polish National Grid called PL-Grid.
Yearly Cracow Grid Workshops• about 150 participants (in 2007: W. Boch, F. Gagliardi, W. Gensch, K. Kasselman, D. Kranzmueller, T. Priol, P. Sloot, and others…),
• this year workshop, 7th in row, will take place on 15-18 October 2007
Henryk Palka NEC’07 Varna, Sept 2007 13
Sharing of CEGC resources
Henryk Palka NEC’07 Varna, Sept 2007 14
Sharing of CEGC resources
Henryk Palka NEC’07 Varna, Sept 2007 15
EGEE-CEGC computing resources
usage by LHC experiments and other VOs
ICMWarszawa
Cyfronet
Kraków
WCSS64Wrocław
PCSSPoznań
Henryk Palka NEC’07 Varna, Sept 2007 16
ATLAS MC production at ACC Cyfronet
Atlas production at Cyfronet was running very well and with high efficiency. ATLAS is regularly getting its fairshare recently running at the level of more than 100 CPU constantly.
17
ATLAS Data Challenges Status
ATLAS
ATLAS DC2 - CPU usage
41%
30%
29%
LCG
NorduGrid
Grid3
ATLAS DC2 - LCG - September 71%
2%
0%
1%
2%
14%
3%
1%
3%
9%
8%
3%2%5%1%4%
1%
1%
3%
0%
1%
1%
4%1%
0%
12%
0%
1%
1%
2%
10%
1% 4%
at.uibk
ca.triumf
ca.ualberta
ca.umontreal
ca.utoronto
ch.cern
cz.golias
cz.skurut
de.fzk
es.ifae
es.ific
es.uam
fr.in2p3
it.infn.cnaf
it.infn.lnl
it.infn.mi
it.infn.na
it.infn.na
it.infn.roma
it.infn.to
it.infn.lnf
jp.icepp
nl.nikhef
pl.zeus
ru.msu
tw.sinica
uk.bham
uk.ic
uk.lancs
uk.man
uk.rl
~ 1350 kSI2k.months~ 120,000 jobs~ 10 Million events fully simulated (Geant4)~ 27 TB
All 3 Grids have been proven to be usable for a real productionabout 1% of the events have been generated in Cracow
- DC2 Phase I started in July, finished in October 2004
- 3 Grids were used LCG ( ~70 sites, up to 7600 CPUs) NorduGrid (22 sites, ~3280 CPUs (800), ~14TB) Grid3 (28 sites, ~2000 CPUs)
LCG 41%
Grid3 29%
NorduGrid30%
from L. Robertson at C-RRB 2004
Henryk Palka NEC’07 Varna, Sept 2007 18
Data transfer T0->T1->T2
T0 T1 tests started in May
Mostly FZK Tier1 involved
End of May: proposal to include Tier2s from FZK Cloud
Delayed due to a high rate of errors at FZK(even though nominal transfer rates has been achieved)
Mid June: T1(FZK) (Cloud)T2 functional test started
DQ2 tool at FZK worked well
CYFRONET and 4 other sites out of total 7 tested had ~100% file transfer efficiency
Transfer rates FZKCYF as high as 60 Mbyte/s
FZK dCacheTransfer Rates
19
Where is the digital divide in Europe?
courtesy of D. Foster
20
BalticGrid in One SlideBalticGrid in One Slide
■ Started 1 Nov 2005 (duration 30m)■ Partners:
10 Leading institutions in six countries in the Baltic Region and Switzerland (CERN)
■ Budget: 3.0 M€
■ Coordinator: KTH PDC, Stockholm
■ Compute Resources: 17 resource centres
■ Pilot applications: HEP, material science, biology, linguistics SA - Specific Service Activities
NA - Networking ActivitiesJRA - Joint Research Activities
■ Estonian Educational and Research Network, EENet■ Keemilise ja Bioloogilse Füüsika Instituut, NICPB■ Inst. of Mathematics and Computer Science, IMCS
UL■ Riga Technical University, RTU■ Vilnius University, VU■ Institute of Theoretical Physics and Astronomy, ITPA■ Poznan Supercomputing and Networking Center,
PSNC■ Instytut Fizyki Jadrowej, im. H Niewodniczanskiego,
Polskiej Akademii Nauk, IFJ PAN■ Parallelldatorcentrum, Kungl Tek. Högskolan, KTH
PDC■ CERN
21
Grid OperationsGrid Operations
Activity Leader: Lauri AntonKrakow Coordinator: Marcin Radecki
(Status end of 2006)
22
BaltiGrid recources in IFJ PANBaltiGrid recources in IFJ PAN
The seed of Tier 3:
■ Development of local GRID installations Access GRID from local UI
■ Support for HEP users Installation of experimental applications Development and tests of user algorithms Submit jobs to GRID – distributed analysis
■ Mini cluster (blade technology) Financed from associated national project 32 cores, 2 GB RAM/core, 2 TB disks To be extended in future (local Tier 3)
Henryk Palka NEC’07 Varna, Sept 2007 23
The insurmountable problem of LHC computing seems to be solvable, thanks to rapid progress in IT technologies
Polish Tier-2 LCG infrastructure and organisation are sound and they are being developed further to meet committments for 2008
HEP GRID community plays also essential role in removing ‘digital divides’ and in bringing the GRID technology to other branches of science
Summary and conclusions
Henryk Palka NEC’07 Varna, Sept 2007 24
The material used in this presentations comes from many sources: the LHC collider and LCG projects, the LHC experimental teams… Special thanks are to Michal Turala, the spiritus movens of Polish GRID computing.I also thank my other Krakow colleagues: P. Lason, A. Olszewski M. Radecki and M. Witek.
Acknowledgements
Henryk Palka NEC’07 Varna, Sept 2007 25
Thank you for your attention
26
Backup
Henryk Palka NEC’07 Varna, Sept 2007 27
LHC Computing
Henryk Palka NEC’07 Varna, Sept 2007 28
Data preselection in real time
- many different
physics processes
- several levels
of filtering
- high efficiency
for events of interest
- total reduction factor of about 107
LHC experiments and data rate
Level 1 - Special Hardware
Level 2 - Embedded Processors/Farm
40 MHz 40 MHz (1000 TB/sec) equivalent)
(1000 TB/sec) equivalent)
Level 3 – Farm of commodity CPU
75 KHz 75 KHz (75 GB/sec)fully digitised
(75 GB/sec)fully digitised5 KHz5 KHz (5 GB/sec)
(5 GB/sec)100 Hz100 Hz (100 MB/sec)
(100 MB/sec)
Data Recording &
Data Recording &
Offline Analysis
Offline Analysis
29
ICFA Network Task Force (1998): required network bandwidth (Mbps)
1998 2000 2005
BW Utilized Per Physicist (and Peak BW Used)
0.05 - 0.25 (0.5 - 2)
0.2 – 2 (2-10)
0.8 – 10 (10 – 100)
BW Utilized by a University Group 0.25 - 10 1.5 - 45 34 - 622
BW to a Home Laboratory Or Regional Center
1.5 - 45 34 - 155 622 - 5000
BW to a Central Laboratory Housing Major Experiments
34 - 155 155 - 622
2500 - 10000
BW on a Transoceanic Link 1.5 - 20 34 - 155 622 - 5000
100–1000 X Bandwidth Increase Foreseen for 1998-2005100–1000 X Bandwidth Increase Foreseen for 1998-2005See the ICFA-NTF Requirements Report:See the ICFA-NTF Requirements Report:
http://l3www.cern.ch/~newman/icfareq98.html
Henryk Palka NEC’07 Varna, Sept 2007 30
Progress on IT technology
1
10
100
1,000
10,000
100,000
1,000,000
10,000,000
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
Farm CPU box MIPS/$M
Doubling in 1.2 years
Raid Disk GB/$M
Doubling in 1.4 years
Transatlantic WAN kB/s per$M/yr
Doubling in 8.4 or 0.7 years
from R. Mount
Performance per unit cost in function of time
31
Where is the digital divide in Europe?
32
BalticGrid initiative
Estonian Educational and Research Network, EENet
Keemilise ja Bioloogilse Füüsika Instituut, NICPB
Institute of Mathematics and Computer Science, IMCS UL
Riga Technical University, RTU Vilnius University, VU
Institute of Theoretical Physics and Astronomy, ITPA Poznan Supercomputing and Networking Center, PSNC Instytut Fizyki Jadrowej, im. Henryka Niewodniczanskiego, Polskiej
Akademii Nauk, IFJ PAN Parallelldatorcentrum at Kungl Tekniska Högskolan, KTH PDC CERN
Proposal to the recent EU 6 FP call (research infrastructure) submitted
33
BalticGrid PartnersBalticGrid Partners
■ Estonia Tallinn, Tartu
■ Lithuania Vilnius
■ Latvia Riga
■ Poland Kraków, Poznan
■ Switzerland Geneva
■ Sweden Stockholm
Details on www.balticgrid.org