View
216
Download
2
Tags:
Embed Size (px)
Citation preview
Int'l HEP Data Grid Work Shop in Daegu
1
Status and Prospective in Japan
KEK High Energy Accelerator Research Organization
Atsushi Manabe
Computing Research Center (CRC)
KEK
Int'l HEP Data Grid Work Shop in Daegu
2
Outline
• Grid projects (budget) in Japan academic region.
• HEP Data Grid activities in Japan.– KEK
• Needs in KEK• Strategy of KEK CRC• Pacifi-Grid• Gfarm project• Grid/HPSS test bed on SupserSINET.
– ICEPP: ATLAS Japan Regional Center
Int'l HEP Data Grid Work Shop in Daegu
3
Grid Projects(budget) in Japan (academic area only)
• MEXT(Ministry of Education, sports …. and technology) projects– SuperSINET; Gigabit backbone for academic researchers in Japan
– MEXT Informatics Project
• A05: Application Grid projects for science ~ US$ 6M /5 y Astronomy (Virtual Observatory), io-Infomatics/Genome (BioGrid) ……
– Networking supercomputers: ITBL ~US$ 100M /5y– National Research Grid Initiative ~US$ 300M/5y
focused in Nano-tech application– (LHC/ATLAS(ICEPP,KEK)
• METI (Ministry of Economy, Trade and Industry) project– AIST GTRC(Grid Technology Research Center) ~ US$ 140M
Int'l HEP Data Grid Work Shop in Daegu
4
Gigabit network
• Super SINET– 10 Gbps MEXT DWDM nationwide network– About 40 universities and labs – US$ 60M/yr– 5 application areas: HEP, Nanotech,
Bioinformatics, Astronomy, Grid Computing
• Tsukuba-WAN– 330Gbps DWDM ring in the Tsukuba area– Tens of national labs– 6 supercomputers in TOP100– Testbed for Grid infrastructure and applications
Int'l HEP Data Grid Work Shop in Daegu
5
Nagoya Hub
Internet
Osaka U
Kyoto U
Nagoya U
NIFS
NIG
KEK
Tohoku U
IMSU-Tokyo
U Tokyo
NII
IP 10 GbpsWDM path
IP router
OXC
Tokyo HubOsaka Hub
ICRKyoto-U
NII Chiba
US, EU
Waseda
TITEC
Tsukuba
Hokkaido
IMS
Kyushu UKorea
NAO
ISAS
Network Topology of SuperSINET
Int'l HEP Data Grid Work Shop in Daegu
6
SuperSINET Backbone
Titech
KEK
ICEPP
7Int'l HEP Data Grid Work Shop in Daegu
Network InfrastructureNII- NY
TRANSPAC
Taiwan-USJapan (NII) -NY : 2.4G x 2 Jan. 2003Japan – US: 622M x 2 (TRANSPAC)Korea – US: 45 MKorea – Japan: 2.4G Jan. 2003Korea – Europe: 2 M (TEIN)China(IHEP) – Japan(KEK): 128 kbps (HEP)China – US: 10 MTaiwan – Japan: 155 MTaiwan – US: 622 M (Dec. 2002)
TEIN
Int'l HEP Data Grid Work Shop in Daegu
8
MEXT Grid Projects• MEXT “Informatics” Project : A05 “Grid” Area
– Lead by Shinji Shimozo(Osaka U.) & Satoshi Matsuoka(Titech)– 11 groups of universities and labs– Research and development on application, middleware, and infrastruct
ure– Support the Virtual Observatory(astronomy) and the BIO
-Grid(bio-Infomatics/Genome) projects
• Tokyo Institute of Technology “Titech Campus Grid” (US$ 2M)
– Lead by Satoshi Matsuoka(Titech)– Total of 800 CPU PC clusters at 13 locations in a 30km range connect
ed by Super TITANET(1-4Gbps) backbone– Pilot project for virtual parallel computer infrastructure for high end co
mputational e-Science
Int'l HEP Data Grid Work Shop in Daegu
9
ITBL• ITBL(IT-based Laboratory)
– To Connect Super-Computers in Government Labs: NAL, RIKEN, NIED, NIMS, JST, JAERI
– Project period: 2001-2005 (3-stage project) with total of US$ 105M funding
– Software development project.– Applications: mechanical simulation, computational biology, material
science, environment, earthquake engineering– Step 1: Supercomputer centers of government lab are
networked via SuperSINET– Step 2: “Virtual Research Environment”: Grid-enabling
laboratory applications– Step 3: Sharing information among researchers from widely
distributed disciplines and institutions
Int'l HEP Data Grid Work Shop in Daegu
10
NAREGI (National Research Grid Initiative)
• NII (The National Institute of Informatics) under MEXT project.
• Making Center of Grid Research & Development– Development of Grid Lower and Middleware
(Grid scheduler,VM, RPC, MPI, GUI tools…)
– Networking, Security and User management(Routing, robust TCP/IP protocol, Traffic management)
– Applications for Nano-science and technology.Joint with Institute for Molecular Science.
– http://www.grid.nii.ac.jp/
Int'l HEP Data Grid Work Shop in Daegu
11
Grid Technology Research Center• Part of AIST under METI (Ministry of Economy, Trade and Industry) • GTRC (Grid Technology Research Center)
– Established in Jan. 2002– Director: Satoshi Sekiguchi
• US$140M for building, equipment, R&D and operations
• AIST super server; PC cluster2004-2Q 11TFLOPS/ AMD Operon x 2116CPUs
• To be central institute for Grid research in Japan– Not just R&D, but serve as central CA, ApGrid’s principal site– Collaboration with institutions and other Grid projects of both domestic and
international scale– Organizing GRID Consortium Japan
– http://unit.aist.go.jp/grid/
Int'l HEP Data Grid Work Shop in Daegu
12
Grid related Activity at KEK
Int'l HEP Data Grid Work Shop in Daegu
13
• KEK sponsors projects which use accelerators and also projects in related fields– KEKB accelerator
• BELLE experiment
– Proton synchrotron • nuclear and elementary particle physics • medical
– Synchrotron Radiation Facility • bio, material science and so on
– Also many theorists in related fields
• Support International HEP collaboration– ATLAS Japan
Major projects in KEK
Int'l HEP Data Grid Work Shop in Daegu
14
KEK Future projects• J-PARC
– will be built in Tokai div. of Japan Atomic Energy Research Institute (JAERI) where ~50km north from KEK start the operation in JFY2005
– 50 GeV high-intensity proton synchrotron – nuclear physics and elementary particle physics
• Super B factory– x30 Luminosity(require x10 CPU power). Under
discussion.
• Linear Collider – discussion is still under way
Int'l HEP Data Grid Work Shop in Daegu
15
Computing Facilities at KEK
• Computing Research Center offers group oriented computing services – Exclusive computing resource for main groups
• Networking, storage and computing resource
• User group owns their own computer systems – PC farms
Int'l HEP Data Grid Work Shop in Daegu
16
Main computing facilities
• KEKCC computer system – for experiments at PS, linear collider R&D, theory, AT
LAS and so on• IBM SP3, 120TB HPSS storage
• KEKB computer system for Belle– Sun WSs and PC farms. 500TB storage.
• KEKSC: Super-computer system for mainly lattice QCD G.– Hitachi SR-8000 100 nodes; almost 1TFLOPTS
Int'l HEP Data Grid Work Shop in Daegu
17
Our needs in Grid Era
• ATLAS Regional Center (Grid is premise)– Being set up at ICEPP (Int’l Center for Particle Physics, U.
Tokyo)
– Still KEK have large portion of ATLAS Japan members and they will make data analysis using computing resource at KEK. We will support them.
• And– BELLE; international collaboration and require more
and more CPU and storage.– J-PARC; joint project of other Inst. regionally
separated.
Int'l HEP Data Grid Work Shop in Daegu
18
BELLE
• BELLE is the major project at KEK• They have many collaborators world wide • They always needs additional CPU power because
acc. luminosity improving and improving.• They seek a solution of their problems related com
puting issue using GRID. • We interviewed the BELLE computing coordinato
r and got the wish list shown in the following slides
Int'l HEP Data Grid Work Shop in Daegu
19
Wish lists from BELLE• What BELLE wants:
– A way to integrate CPU powers and way to distribute data among collaborators’ Univ.s and Inst.s for their data production.
– Automatic way to submit analysis jobs in the distributed environment.
• once analysis codes have placed somewhere, process for all of experimental data and Monte Carlo should be done automatically and simply users gain the result
– Automatic Monte Carlo production – Automatic version control for libraries, constants and others on
all of the computer system in BELLE – Automatic update on reconstructed data or Monte Carlo if
libraries or constants are once updated
Int'l HEP Data Grid Work Shop in Daegu
20
Wish lists from BELLE 2
• It is not easy to achieve their requests technically – simplest solution will be hire more Post-Docs
or find more grad-students, however
• We have decomposed their requests – should be done in application level (BELLE)or
– can be solved by GRID
Int'l HEP Data Grid Work Shop in Daegu
21
What they need related GRID• An efficient way to distribute jobs and way to
manage/share/distribute data for the jobs.– effective data catalogue
• physical to logical mapping • Seamless and unique file tree• query
– file names, contents and others
• works on the Internet but secure
– efficient and easy to distribute data to remote sites– easy to use
• web interface and GUI also
Int'l HEP Data Grid Work Shop in Daegu
22
continued
• mass storage management system– not only tapes but also large amount of disks – robust and fast transfer – easy integration with other component
• We already have them partially on LAN, but we need to expand them to the Internet. We hope GRID will solve them.
Int'l HEP Data Grid Work Shop in Daegu
23
KEK CRC strategy towards GRID
Int'l HEP Data Grid Work Shop in Daegu
24
Done or On going.• Install GT2(Globus toolkit ver.2) and applications (LSF)
to two CRC Systems (KEKCC & KEKB) ) and will open to some limited users. (remote Job submission)
• Build temporal CA with single Virtual Organization(VO).
• Build a inter-institute wide Grid test bed on SuperSINET and make a performance test. (talk later)
• Trial use of SRB with Belle data in a test system.• Start to touch LCG tools by ATLAS Japan people.• watch technical trends carefully.
Int'l HEP Data Grid Work Shop in Daegu
25
SRB
• SRB is available today and looks useful even in HEP ( http://www.npaci.edu/DICE/SRB/ )– developed at San Diego Super Computer Center – seamless solution to HPSS
• good for the KEKCC (central computer) users
– Data catalogue and replication– – SLAC is also testing
Int'l HEP Data Grid Work Shop in Daegu
26
Until early 2004• Deploy GT3, evaluate LCG tools and other something new tec
hnology.• Build own VO at each sites and integrate them.• File sharing by SRB (or ?) and seamless job submission betwee
n KEKCC and KEKB• Pacifi-Grid test environment;
Pan-Pacific collaboration (see later)– KNU (Korea), Univ. of Melbourne (Australia), SLAC and KEK will t
est SRB
• Start to develop higher level software (experiment side) on the top of Globus and SRB.– Message passing with GSI authentication ?
Int'l HEP Data Grid Work Shop in Daegu
27
~2004• KEKB and KEKCC (5yeas rental )system
replacement (2005-4Q) .– GRID will become concrete until this stage or
die– Final specification have to be decided
• Decision of middle ware (Grid or else?)• CA • Security • human resource
– trained engineer on site
Int'l HEP Data Grid Work Shop in Daegu
28
Other topics discussed internally • CA
– nation wide, institution wide or else? KEK certify all Japanese HEP user?– Rules for certification of users.
• PKI– secure enough? Safety private key management (IC card?)– management costs? – seamless integration with local authentication
• Network– How we can overcome latency?– better connectivity to the Internet, routing and throughput
• middleware– Globus, Avaki or something else – SRB, LCG or other data replication solution
• Storage– SAN– StorageTank (IBM)
• security vs usability
Int'l HEP Data Grid Work Shop in Daegu
29
Pacifi-Grid:Proposal on
Open Pacific Grid collaboration
Takashi Sasaki (KEK)
Int'l HEP Data Grid Work Shop in Daegu
30
charter• Give research institutions and IT vendors opportunity
of testing Grid technologies under inter-state environment.– Emphasis on Data Grid– Open to everybody who wants to join– Grid is a newly emerging technology and more experiences
are necessary before deploying at the large scale production systems
• Test on high latency network• robustness tests• interoperability tests among sites and vendors • security
Int'l HEP Data Grid Work Shop in Daegu
31
Collaboration (under discussion)
• Korea• U of Melbourne, Australia• SLAC, USA• JAPAN
– ICEPP (Univ. of Tokyo)
– KEK
• Some venders– Under discussing with Fujitsu, IBM Japan and so on
Int'l HEP Data Grid Work Shop in Daegu
32
Gfarm R&D
Development R&DGrid middleware of Cluster File System
with job execution.
Int'l HEP Data Grid Work Shop in Daegu
33
R&D of Gfarm (Grid data farm)• Gfarm
– Development by CS scientist of AIST, Titech and KEK since 2000
• AIST :Advanced Inst. for Science and Technology
– Architecture• PC farm with large local disk/node as a GRID fabric
• Large data file is divided into fragments and stored in the disks
• Data I/O by parallel file system
• Affinity scheduling of process and storage for data residence
Int'l HEP Data Grid Work Shop in Daegu
34
Data Access Locality
computingnode
storagenode
high speedswitch
jobmanager
jobA
jobB
user job
jobmanager
jobA
jobB
user job
computingnode
storageelement
Exploit the data access locality as much as possible
Int'l HEP Data Grid Work Shop in Daegu
35
Gfarm• Grid Data Farm : Beta 3 release is now available.• Parallel I/O: exploit the data access locality, store and access files by “fr
agments”• Parallel Job: program runs on the
nodes where the file fragmentsreside: “owner computes”
• Programs and file fragments,job history, data checksum,file replication are managed withMetaDB
• Fragments are replicated forbackup and load balancing
• User sees the fragmented files asa single Gfarm URL
• Provide system call hooks foropen(), close(), read(), write() etc
http://datafarm.apgrid.org/
• Authentication:Globus GSI and/orShared Private Key
Int'l HEP Data Grid Work Shop in Daegu
36
FADS/Goofy in ATLAS• Used in sub-detector software developments and physics vali
dations• Detector modules also run in ATLAS mainstream framewor
k (Athena)
• Testbed for ROOT I/O• Bandwidth Challenge in
SC2002• Generated 106 fully simulated
higss 4 events in 2 dayswith ~ 400 CPU
Int'l HEP Data Grid Work Shop in Daegu
37
Gfarm achievement• “File affinity scheduling” provides parallel processi
ng capability for both I/O-bound and CPU-intensive jobs.
• FADS/Goofy was successfully implemented on Gfarm architecture. In the process, ROOT I/O module works with Gfarm parallel file system using the system call hook, without a changes to the ROOT package.
• ROOT I/O files has been successfully replicated at 2.286 Gbps using the SC2002 Bandwidth Challenge testbed with 12 nodes ~ 190Mbps/node.
• Please refer to CHEP03 reports by Y.Morita and O.Tatebe
Int'l HEP Data Grid Work Shop in Daegu
38
Status of ATLAS Japan RC (ICEPP U. of Tokyo)
• R&D is ongoing – Test-buy of PC farm and storage
• 1U P3, P4 servers ( >150nodes and more) , LTO (10TB) with CASTOR and >20TB disk servers.
– Nordu GRID was tested first • Data transfer tests between U. Tokyo and KEK was
done
– ATLAS data challenges – LCG 0 and 1
Int'l HEP Data Grid Work Shop in Daegu
39
Int'l HEP Data Grid Work Shop in Daegu
40
Int'l HEP Data Grid Work Shop in Daegu
41
LCG-0 Deployment Status
Site Scheduled StatusTier 1
0 CERN 15/2/03 Done
1 CNAF 28/2/03 Done
2 RAL 28/2/03 Done
3 FNAL 30/3/03 Done
4 Taipei 15/4/03 Done
5 FZK 30/4/03 Done
6 IN2P3 7/5/03 In prep.
7 BNL 15/5/03 Done
8 Russia (Moscow) 21/5/03 In prep.
9 Tokyo 21/5/03 Done
Tier 2
10 Legnaro (INFN) After CNAF Done
These sites deployed the LCG-0 pilot system and will be the first sites to deploy LCG-1
Int'l HEP Data Grid Work Shop in Daegu
42
LCG Resource Commitments – 1Q04 CPU
(kSI2K)Disk TB
Support FTE
Tape TB
CERN 700 160 10.0 1000
Czech Republic 60 5 2.5 5
France 420 81 10.2 540
Germany 207 40 9.0 62
Holland 124 3 4.0 12
Italy 507 60 16.0 100
Japan 220 45 5.0 100
Poland 86 9 5.0 28
Russia 120 30 10.0 40
Taiwan 220 30 4.0 120
Spain 150 30 4.0 100
Sweden 179 40 2.0 40
Switzerland 26 5 2.0 40
UK 1656 226 17.3 295
USA 801 176 15.5 1741
Total 5600 1169 120.0 4223
Int'l HEP Data Grid Work Shop in Daegu
43
Int'l HEP Data Grid Work Shop in Daegu
44
Status of ATLAS Japan RC 2
• Budget requests have been submitted to the funding agency – still no decision has been made
• and no decision will be made within this FY– All of national universities and some national laboratories
including KEK will be non-nationalized (not privatized, but will be a kind of independent governmental agencies)
Int'l HEP Data Grid Work Shop in Daegu
45
Summary of KEK Grid activity
• KEK had a step forward for the Grid computing • People at ATLAS Japan RC are working on LCG • Realistic environment is necessary to test
middleware and applications – We have started the discussion among Pan-Pacific sites
to collaborate each other
– We want to exchange knowledge and experiences
Int'l HEP Data Grid Work Shop in Daegu
46
End of my presentation
Thank you