Upload
giles-norman
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
Grid Computing for High Energy Grid Computing for High Energy Physics in JapanPhysics in Japan
Grid Computing for High Energy Grid Computing for High Energy Physics in JapanPhysics in Japan
Hiroyuki MatsunagaHiroyuki MatsunagaInternational Center for Elementary Particle Physics (ICEPP), International Center for Elementary Particle Physics (ICEPP),
The University of TokyoThe University of Tokyo
International Workshop on e-Science for Physics 2008International Workshop on e-Science for Physics 2008
2
Major High Energy Physics Program in JapanMajor High Energy Physics Program in JapanMajor High Energy Physics Program in JapanMajor High Energy Physics Program in Japan
KEK-B (Tsukuba)– Belle
J-PARC (Tokai)– Japan Proton Accelerator Research Complex– Operation will start within this year – T2K (Tokai to Kamioka)
long baseline neutrino experiment Kamioka
– SuperKamiokande– KamLAND
International collaboration– CERN LHC (ATLAS, ALICE)– Fermilab Tevatron (CDF)– BNL RHIC (PHENIX)
3
Grid Related ActivitiesGrid Related ActivitiesGrid Related ActivitiesGrid Related Activities
ICEPP, University of Tokyo– WLCG Tier2 site for ATLAS
Regional Center for ATLAS-Japan group Hiroshima University
– WLCG Tier2 site for ALICE KEK
– Two EGEE production sitesBELLE experiment, J-PARC, ILC…
– University support– NAREGI
Grid deployment at universities– Nagoya U. (Belle), Tsukuba U. (CDF)…
Network
4
Grid Deployment at University of TokyoGrid Deployment at University of TokyoGrid Deployment at University of TokyoGrid Deployment at University of Tokyo
ICEPP, University of Tokyo– Involved in international HEP experiments since 1974
Operated pilot system since 2002 Current computer system started working last year
– TOKYO-LCG2. gLite3 installed CC-IN2P3 (Lyon, France) is the associated Tier 1 site within A
TLAS computing model– Detector data from CERN go through CC-IN2P3– Exceptionally far distance for T1-T2
RTT ~280msec, ~10 hopsChallenge for efficient data transferData catalog for the files in Tokyo located at Lyon
– ASGC (Taiwan) could be additional associated Tier1Geographically nearest Tier 1 (RTT ~32msec)Operations have been supported by ASGC
– Neighboring timezone
5
Hardware resourcesHardware resourcesHardware resourcesHardware resources
Tier-2 site plus (non-grid) regional center facility
– Support local user analysis by ALTAS Japan group
Blade servers– 650 nodes (2600 cores)
Disk arrays– 140 Boxes (~6TB/box)– 4Gb Fibre-Channel
File servers– Attach 5 disk arrays– 10 GbE NIC
Tape robot (LTO3)– 8000 tapes, 32 drives
Pledged Planned to be pledged
2007 2008 2009 2010
CPU (kSI2k) 1000 1000 1000 3000
Disk (Tbyes) 200 400 400 600
Nominal WAN (Mbits/sec)
2000 2000 2000 2000
Tape robotBlade
serversDisk arraysDisk arrays
6
SINET3SINET3SINET3SINET3
SINET3 (Japanese NREN)– Third generation of SINET, since Apr. 2007– Provided by NII (National Institute of Informatics)
Backbone: up tp 40Gbps Major universities connect with 1-10 Gbps
– 10 Gbps to Tokyo RC International links
– 2 x 10 Gbps to US– 2 x 622 Mbps to Asia
7
International LinkInternational LinkInternational LinkInternational Link 10Gbps between Tokyo and CC-IN2P3
– SINET3 + GEANT + RENATER (French NREN)– public network (shared with other traffic)
1Gbps link to ASGC (to be upgraded to 2.4 Gbps)
TokyoNew York Lyon
SINET3 (10Gbps)GEANT (10Gbps)
RENATER (10Gbps)
Taipei
8
Network test with IperfNetwork test with IperfNetwork test with IperfNetwork test with Iperf
Memory-to-memory test performed with Iperf program Use Linux boxes dedicated for iperf test at both ends
– 1Gbps limited by NIC– Linux kernel 2.6.9 (BIC TCP)– Window size 8Mbytes, 8 parallel streams
For Lyon-Tokyo: long recovery time due to long RTT
Lyon <-> Tokyo (RTT: 280ms) Taipei <-> Tokyo (RTT: 32ms)
9
Data Transfer from Lyon Tier1 centerData Transfer from Lyon Tier1 centerData Transfer from Lyon Tier1 centerData Transfer from Lyon Tier1 center
Data transferred from Lyon to Tokyo– Used Storage Elements in production– ATLAS MC simulation data
Storage Elements– Lyon: dCache (>30 gridFTP servers, Solaris, ZFS)– Tokyo: DPM (6 gridFTP servers, Linux, XFS)
FTS (File Transfer System)– Main tool for bulk data transfer– Execute multiple file transfers (by using gridFTP) concurrent
lySet number of streams for gridFTP
– Used in ATLAS Distributed Data Management system
10
Performance of data transferPerformance of data transferPerformance of data transferPerformance of data transfer
>500 Mbytes/s observed in May, 2008– Filesize: 3.5Gbytes– 20 files in parallel, 10 streams each– ~40Mbytes/s for each file transfer
Low activity at CC-IN2P3 during the period (other than ours)
500 Mbytes/s
0 20 40Mbytes/s
10
1
100Throughput per file transfer
11
Data transfer between ASGC and TokyoData transfer between ASGC and TokyoData transfer between ASGC and TokyoData transfer between ASGC and Tokyo
Transferred 1000 files at a test (1Gbytes filesize) Tried various numbers of concurrent files / streams
– From 4/1 to 25/15 Saturate 1Gbps WAN bandwidth
Tokyo -> ASGC
ASGC -> Tokyo
20/10
4/2 4/4
16/1
20/1025/15
4/18/1 4/2 8/24/416/1
25/10 25/10
25/10
12
CPU Usage in the last year (Sep 2007 – Aug 2008)CPU Usage in the last year (Sep 2007 – Aug 2008)CPU Usage in the last year (Sep 2007 – Aug 2008)CPU Usage in the last year (Sep 2007 – Aug 2008)
3,253,321 CPU time (kSI2k*hours) in last year– Most jobs are ATLAS MC simulation
Job submission is coordinated by CC-IN2P3 (the associated Tier1)Outputs are uploaded to the data storage at CC-IN2P3
– Large contribution to the ATLAS MC production
Top 10 Tier2 site for ATLAS VO
0500000
100000015000002000000250000030000003500000400000045000005000000
AGLT
2
IN2P
3-CC
-T2
MWT2_U
CGRI
F
TOKY
O-LCG2 WT2
BU_A
TLAS
_Tier
2
MWT2_IU
UKI-N
ORTHG
RID-
MAN-H
EP
OU_OCH
EP_S
WT2
Nor
mal
ized
CP
U t
ime
atlas
TOKYO-LCG2 CPU time per month CPU time at Large Tier2 sitesTOKYO-LCG2 CPU time per month
0
100000
200000
300000
400000
500000
600000
7-Sep 7-Oct 7-Nov 7-Dec 8-Jan 8-Feb 8-Mar 8-Apr 8-May 8-Jun 8-Jul 8-Aug
Nor
mal
ized
CP
U t
im
atlas
13
ALICE Tier2 center at Hiroshima UniversityALICE Tier2 center at Hiroshima UniversityALICE Tier2 center at Hiroshima UniversityALICE Tier2 center at Hiroshima University
WLCG/EGEE site – “JP-HIROSHIMA-WLCG”
Possible Tier 2 site for ALICE
14
Status at HiroshimaStatus at HiroshimaStatus at HiroshimaStatus at Hiroshima
Just became EGEE production site– Aug. 2008
Associated Tier1 site will likely be CC-IN2P3– No ALICE Tier1 in Asia-Pacific region
Resources– 568 CPU cores
Dual-Core Xeon(3GHz) X 2cpus X 38boxesQuad-Core Xeon(2.6GHz) X 2cpus X 32boxesQuad-Core Xeon(3GHz) X 2cpus X 20blades
– Storage: ~200 TB next year
Network: 1Gbps– On SINET3
15
KEKKEKKEKKEK
Belle experiment has been running– Need to have access to existing peta-bytes of data
Site operations– KEK does not support any
LHC experiment– Try to gain experience by
operating sites in order to prepare for future Tier1 level Grid center
University support NAREGI
KEK Tsukuba campus
Mt. Tsukuba
KEKB
Linac
Belle exp.
16
Grid Deployment at KEKGrid Deployment at KEKGrid Deployment at KEKGrid Deployment at KEK
Two EGEE sites– JP-KEK-CRC-1
Rather experimental use and R&D– JP-KEK-CRC-2
More stable services NAREGI
– Used beta version for testing and evaluation Supported VOs
– belle (main target at present), ilc, calice, …– Not support LCG VOs
VOMS operation– belle (registered in CIC)– ppj (accelerator science in Japan), naokek– g4med, apdg, atlasj, ail
17
Belle VOBelle VOBelle VOBelle VO
Federation established– 5 countries, 7 institutes, 10 sites
Nagoya Univ., Univ. of Melbourne, ASGC, NCU, CYFRONET, Korea Univ., KEK
VOMS is provided by KEK Activities
– Submit MC production jobs– Functional and performance tests– Interface to existing peta-bytes of data
19
ppj VOppj VOppj VOppj VO
Federated among major universities and KEK– Tohoku U. (ILC, KamLAND)– U. Tsukuba (CDF)– Nagoya U. (Belle, ATLAS)– Kobe U. (ILC, ATLAS)– Hiroshima IT (ATLAS, Computing Science)
Common VO for accelerator science in Japan– NOT depend on specific projects, but resources shared
KEK acts as GOC– Remote installation– Monitoring
Based on Nagios and Wiki– Software update
20
KEK Grid CAKEK Grid CAKEK Grid CAKEK Grid CA
Started since Jan. 2006 Accredited as an IGTF
(International Grid Trust Federation) compliant CA
JFY 2006
Apr 2006 - Mar 2007
JFY 2007
Apr2007 – Mar 2008
Personal cert. 68 119
Host cert. 139 238
Web server cert. 4 0
Numbers of Issued certificates
21
NAREGINAREGINAREGINAREGI
NAREGI: NAtional REsearch Grid Initiative– Host institute: National Institute of Infrmatics (NII) – R&D of the Grid middleware for research and indus
trial applications – Main targets are nanotechnology and biotechnolog
yMore focused on computing gridData grid part integrated later
Ver. 1.0 of middleware released in May, 2008– Software maintenance and user support services w
ill be continued
22
NAREGI at KEKNAREGI at KEKNAREGI at KEKNAREGI at KEK
NAREGI- version installed on the testbed– 1.0.1: Jun. 2006 – Nov. 2006
Manual installation for all the steps
– 1.0.2: Feb 2007– 2.0.0: Oct. 2007
apt-rpm installation
– 2.0.1: Dec. 2007
Site federation test– KEK-NAREGI/NII: Oct. 2007– KEK-National Astronomy Observatory (NAO): Mar. 2008
Evaluation of application environment of NAREGI– job submission/retrieval, remote data stage-in/out
24
Data Storage: GfarmData Storage: GfarmData Storage: GfarmData Storage: Gfarm
Gfarm: distributed file system– DataGrid part in NAREGI– Data are stored in multiple disk servers
Tests performed :– Stage-in and stage-out to the Gfarm storage– GridFTP interface
Between gLite site and NAREGI site
– File access from applicationHave access with FUSE (Filesystem in userspace)
– Without the need of changing application program– IO speed is several times slower than local disk
25
Future Plan on NAREGI at KEKFuture Plan on NAREGI at KEKFuture Plan on NAREGI at KEKFuture Plan on NAREGI at KEK
Migration to the production version Test of interoperability with gLite Improve the middleware in the application dom
ain– Development of the new API to the application
Virtualization of the middlewarefor script languages (to be used at web portal as well)
– MonitoringJobs, sites,…
26
SummarySummarySummarySummary
WLCG– ATLAS Tier2 at Tokyo
Stable operation
– ALICE Tier2 at HiroshimaJust started operation in production
Coordinated effort lead by KEK– Site operations with gLite and NAREGI middlewares
Belle VO: SRB – Will be replaced with iRODs
ppj VO: deployment at universities– Supported and monitored by KEK
– NAREGIR&D, interoperability