GRID in Japan

Takashi Sasaki Computing Research Center

KEK

Contents

• NAREGI• GRID Interoperability • An example of application

– Particle therapy

NAREGI• National flagship project for developing the

middleware– Leader: Ken Miura, NII

• NAREGI version 1.0 will be released in the spring of 2008(April?)– After April 2008, NAREGI will be a part of the peta-

flops computer project – Originally, focused on super computers

• Posters at SC2007– NAREGI architecture– How to use NAREGI– Cyber Science Infrastructure

http://www.naregi.org/papers/data/sc07_panel_01.pdfhttp://www.naregi.org/papers/data/sc07_panel_02.pdfhttp://www.naregi.org/papers/data/sc07_panel_03.pdf

GRID Interoperability in HEP

NEGST workshop Tokyo 2007NEGST workshop Tokyo 2007

Representing the collaboration between Representing the collaboration between

CCCC--IN2P2 and KEKIN2P2 and KEK--CRCCRC

Takashi SasakiTakashi Sasaki

KEK Computing Research CenterKEK Computing Research Center

This presentation is based on DominiqueThis presentation is based on Dominique’’s presentation at s presentation at FJPPL2007 workshopFJPPL2007 workshop

GRIDs

• The Global Grid Forum web site is mentioning ~45 major different grid organizations

• W-LCG is federating resources from 3 different Grids:– EGEE (Europe + Asia + Canada)– OSG (US)– NorduGrid (Nordic countries)

– NAREGI (Japan ?? )

• Having 1 single grid is unrealistic• One has to deal with several Grids and provide

interoperability– From the user point of view– From the sites' point of view

NAREGI (NAREGI (National Research National Research Grid InitiativeGrid Initiative) is developed in ) is developed in JapanJapan

Project presentation• This project is proposing to work toward Grid

Interoperability

• Work will concentrate mainly on– EGEE / NAREGI interoperability

• Crucial for ILC• Will become important for LHC

– NAREGI is a huge effort in Japan, and will certainly become a piece of the W-LCG organization (I hope)

– SRB / iRODS data grid (see later)• Implementation• Development• Interoperability with EGEE / NAREGI

NAREGI: 2003-2007 10 billion

People concerned in the project

• KEK

– S. Kawabata– T. Sasaki– G. Iwai– K. Murakami– Y. Iida

• CC-IN2P3

– D. Boutigny– S. Reynaud– F. Hernandez– J.Y. Nief

NAREGI at KEK• KEK installed and is maintaining NAREGI middleware

NAREGI is based on Grid standards NAREGI is based on Grid standards defined at the OGF (Open Grid Forum)defined at the OGF (Open Grid Forum)

NAREGI betaNAREGI beta--1 released in May 2006 and 1 released in May 2006 and installed at KEKinstalled at KEK•• With some help from the NAREGI support With some help from the NAREGI support team for the last stageteam for the last stage•• Compute part (6 nodes)Compute part (6 nodes)•• Data grid part (3 nodes)Data grid part (3 nodes)

Tested at KEK with P152 (heavy ions) and Tested at KEK with P152 (heavy ions) and Belle simulationBelle simulation

Experience with middleware being built upExperience with middleware being built up

Interoperability between EGEE and NAREGI

• 2 possible approaches– Implement the GIN (Grid Interoperability Now) layer in

NAREGI• Defined by the GIN group from the OGF• Short term solution in order to get the Interoperability Now !• Pragmatic approach

– Work with longer term standards defined within the OGF• Develop a Meta Scheduler compatible with many Grid

implementations• Based on SAGA (Simple API for Grid Applications) "Instead of

interfacing directly to Grid Services, the applications can soaccess basic Grid Capabilities with a simple, consistent andstable API"

• and JSDL (Job Submission Description Language)

GIN• Pushed by KEK and others, NAREGI has done

considerable efforts to implement the GIN layer"Trying to identify islands of interoperation between production grids and grow those islands"

Developing an interoperation island with EGEE

Common framework for Authentication and VO management

Cross job submission between gLite/EGEE and NAREGI

Data transfer between gLite and Gfarm

Grid resource information service around the world

From NAREGI presentation in 02/07

An example of interoperability

GridFTPServer

EGEE

gLite ClientgLite Client

SRM Client

NAREGI

NAREGI ClientNAREGI Client

SRMClient

GfarmAPI

NAREGIMetadata ServerLFC(Metadata Server)

GfarmServer

DPM(SRM Server)

StorageStorage

GIN-data: Data Management and Movement

Interoperability between NAREGI and EGEE at the data level

Work also on: Work also on:

GINGIN--auth GINauth GIN--jobs jobs GINGIN--info info GINGIN--opsops

The SAGA / JSDL approach

• This approach is being developed at CC-IN2P3 (Sylvain Reynaud)

• This interoperability tool is sharing several modules with a software layer which is being developed at CC-IN2P3 in order to easily interface our local batch system to any Grid Computing Element

Next steps on NAREGI / EGEE interoperability

• Continue work on both directions: GIN and SAGA / JSDL

• Try cross job submission on both Grid middleware

• Explore data exchange between NAREGI and EGEE

The Storage Resource Broker (SRB)

• SRB is relatively light data grid system developed at SDSC

• Considerable experience has been gained at KEK, SLAC, RAL and CC-IN2P3

• Heavily used for BaBar data transfer since years (up to 5 TB/day)

• Very interesting solution to store and share biomedical data (images)

• Advantages– Easy and fast development of applications– Extensibility– Reliability– Easiness of administration

From SRB to iRODS

• iRODS (iRule Oriented Data Systems ) is the SRB successor– CC-IN2P3 and KEK are both involved in

iRODS developments and tests

– Should bring many new functionalities

From SRB to iRODSiRule Oriented Data Systems

Definition of rules and Definition of rules and micromicro--servicesservices

Allows to fully Allows to fully customize the customize the system in order system in order to adapt it to the to adapt it to the applicationapplication

SRB-DSI Being developed by IidaBeing developed by Iida--sansan

SRBSRB--DSI is a software layer which allows the Grid world based on DSI is a software layer which allows the Grid world based on GlobusGlobusto interoperate with SRBto interoperate with SRB

GlobusGlobus world:world: data transfer based on data transfer based on GridFTPGridFTP

SRB:SRB: based on its own protocolbased on its own protocolCrucial to interoperate the Crucial to interoperate the LCG and SRB worldsLCG and SRB worlds

The SRB-DSI architecture

rls09 rls10 rsr01

rac01

srbServersrbServer srbServer

MCAT

Scommand

globus-url-copy

GridFTP

globus-gridftp-server SRB-DSIglobus-gridftp-server SRB-DSIglobus-gridftp-server SRB-DSI

edg-gridftp-ls

ExternalDisk Sub-sys

SRB (GSI_AUTH) SRB (GSI_AUTH)

GridFTPSRB Protocol

SRB Protocol(Svr-Interconnection)

HPSS API I/O

HPSS

POSIX I/O (LOCAL) POSIX I/O (LOCAL)

hpssfsd

HPSS API I/OEach DSI accesse to the following SRB resource- gsiftp://rsr01 -> rsr01-ufs- gsiftp://rls09 -> rls09-hpss-hsi- gsiftp://rls10 -> rls10-hpss-vfs

“rls09-hpss-hsi” “rls10-hpss-vfs” “rsr01-ufs”

SRB (GSI_AUTH)

Next step for SRB / LCG interoperability

• In order to have LCG and SRB fully interoperable we need to develop an SRB / SRM interface

• This will be a common area of work for CC-IN2P3 and KEK in the near future– One of us will stay at CC-IN2P3 for 8month (in plan)

• Then we will explore the possibility to make SRB an alternative for LCG storage

• SRB / iRODS is probably a good candidate to store user's files Grid wide An idea to be explored

Software Suite for Particle Therapy Simulation

Takashi SasakiKEK Computing Research Center

and CREST/JST

Takashi.Sasaki@kek.jp

The Project

• “The Development of Software Framework for Simulation in Radiotherapy”– funded by the Core Research for Evolutional

Science and Technology (CREST) program organized by Japan Science and Technology Agency (JST) from 2003 to 2008

• Joint project among medical physicists, astro-physicists and Geant4 developers in Japan

Member Institutes• High Energy Accelerator Research Organization (KEK)• Ritsumeikan University (RITS)• Kobe University• Naruto University of Education • Toyama National College of Maritime Technology• Japan Aerospace Exploration Agency (JAXA)• National Institute of Radiological Science (NIRS)• National Cancer Center, Kashiwa• Gunma University Faculty of Medicine • Hyogo Ion Beam Medical Center (HIBMC)• Kitasato University

Goal of The Project

• Provide the software suit for simulation in radiotherapy, especially, particle therapy– Software framework and tools

• Implementation of geometry of facilities • DICOM interface• Visualization • GRID

– Tuned physics models• Validation of simulation results

– Collaboration with facilities

EGS4JQMD

modeler

framework for medical application

others...

Scoring/Tally Package

Physics Listfor Radiotherapy

Knowledge DB

visualization/interactivity

DICOM interface

The system architecture

Geant4

Dose Calculation Engines

GRID Deployment

PTSsim

Basic design of Beam irradiation system

patient

Wobblermagnets

scatterer Dose monitors flatness monitor

Ridge filterRange shifter

MLC

RangeCompensator

beam

Purpose:Widen the beam size to fit the tumor size with keeping lateral flatness of beam fluxAdjust the depth of Bragg peak in a patient volume with the tumor position

Other technology:Double scattering, Spiral wobbling system for shortening the irradiation systemBeam scanning in three dimensions using small beam spot and variable beam energy

T.Aso IEEE NSS 2007 N60‐1

Implemented irradiation systems using PTSsimFacility Accelerator Beam Energy

(MeV)Lateral Spreading System Range Modulator

HIBMCGantry

Synchrotron Proton150,190,230

Wobbler magnetsand scatter

Ridge filter

NCCGantry

Cyclotron Proton150,190,235

Scatter and double scatter Ridge filter

UCSF Cyclotron Proton67.5

N/A Propeller blades

HIMAC Synchrotron Carbon400 MeV/u

Wobbler magnets and scatter Ridge filter

HIBMC Synchrotron Carbon320 MeV/u

Wobbler magnets and scatter Ridge filter

GSI Synchrotron Carbon~400 MeV/u

Beam scanning Fine Ridge filter

T.Aso IEEE NSS 2007 N60‐1

Example of irradiation systemsScatter

Ridge filter Propeller blade

Ionization Chamber

Wire Chamber

Wobbler Magnets

Collimator

Multi-leaf collimator

Bolus

Water phantom

DICOM data

Double Scatter

/G4M/ChangeSystem NCCGantry

/G4M/System HIBMCGantry

/G4M/ChangeSystem UCSFSetup

T.Aso IEEE NSS 2007 N60

DICOM interface• DICOM handler had been developed independently

– This is not one in geant4 examples– TOSHIBA, SIEMENS, and GE DICOM data had been tested.– DICOM network is partly supported using DCMTK (OFFIS)– Filter modules are plugged in for converting original CT data into

a suitable format for making a geometry.• Extraction of patient geometry• Reformation of voxels• Density conversion from HU

etc.– DICOM-RT

• Not well standardized • Only HIBMC-Mitsubishi is tested

Load DICOM data

Compensation of lack slices

Select window of interest

Outline extraction

Reformation of voxles

Material creation

Geometry construction

http://geant4.kek.jp/gMocren/

Calculated dose distributionCalculated dose distribution

color mapping

contour plot

Particle trajectoriesParticle trajectories

Opacity curve and color map editorOpacity curve and color map editor

free hand or templates with WW&WL editing

Supported system :- Windows 2k/XP or PC Linux OS- Pentium 4 or faster- more than 1 GB (recommend)

• no special hardware is necessary

Supported system :- Windows 2k/XP or PC Linux OS- Pentium 4 or faster- more than 1 GB (recommend)

• no special hardware is necessary

3D (ray casting) 2D (MPR)

Opacity curve and color map editor

gMocren and utility softwares are freely available.gMocren and utility softwares are

freely available.

Functionality Requirements Functionality Requirements ::

• To visualize- the modality image used by the simulation,- the calculated dose distribution and- the particle trajectories- in an agreeable speed

• Transfer function editor• Multi-platform

• To visualize- the modality image used by the simulation,- the calculated dose distribution and- the particle trajectories- in an agreeable speed

• Transfer function editor• Multi-platform

Trajectory information in the simulation is available.

gMocren : DICOM visualizergMocrengMocren : DICOM : DICOM visualizervisualizer

2/Nov/2007 29

IEEE NSS/MIC (San Diego, Nov/2006)

User InterfacesUser InterfacesExample of accelerator head designbbbbbbb

2/Nov/2007 30

IEEE NSS/MIC (Puerto Rico, Oct/2005)

Lead Scatterer

0 0.5 1 1.5 2 2.5 30102030405060708090

散乱体厚 (mm)

陽子

線の

広が

り (m

m)

□ 150MeV 測定値■ 150MeV ｼﾐｭﾚｰｼｮﾝ値△ 190MeV 測定値▲ 190MeV ｼﾐｭﾚｰｼｮﾝ値○ 230MeV 測定値● 230MeV ｼﾐｭﾚｰｼｮﾝ値

Nuclear Interaction Effect

MLC Water phantom

beam

50mm depth 250mm depth● ： Measurements－ ： TPS calculations□ ： G4 (histograms)

by T.Akagi (HIBMC)Bragg Peak

Spread Out Bragg Peak (SOBP)

( with Wobbler and Scatter )

(with Wobbler, Scatter, and Ridge filter)

Depth‐Dose distribution

Proton range

ーNIST PSTAR●Simulation

Material Properties

Wobbler Magnet

Uniform Irradiation Field

Edges of Multi-Leaf Collimator

Beam Delivery system validation

Wobbler demonstration:Spiral Wobbling

Stopping Power/Range , checked with NIST data 

― Simulation■ Measurement

Validation ActivitiesValidation ActivitiesValidation Activities for Protonsfor Protons

2/Nov/2007 31IEEE TNS V52,Issue4, (2005) 896pp

NIRS /IHI experimental beam lineNIRS /IHI experimental beam line

2/Nov/2007 32

for  Carbonsfor  CarbonsValidation ActivitiesValidation ActivitiesValidation Activities

IEEE NSS/MIC (San Diego, Nov/2006)

Job SubmissionJob Submission

MatchMakerMatchMatchMakerMaker

InformationSupermarketInformationInformationSupermarketSupermarket

Task QueueTask QueueTask Queue

Network ServerNetwork ServerNetwork Server

Resource BrokerResource BrokerResource Broker

Site‐ASiteSite‐‐AA

Site‐BSiteSite‐‐BB

Site‐CSiteSite‐‐CC

Grid Web UIGrid Web UIGrid Web UI

HospitalHospitalHospital

Log on to Grid Web UILog on to Grid Web UILog on to Grid Web UI

Obtain the proxy/resource informationInitialize/destroy proxy certificateObtain the proxy/resource informationObtain the proxy/resource informationInitialize/destroy proxy certificateInitialize/destroy proxy certificate

Input the job parametersInput the job Input the job parametersparameters

Submit the jobSubmit the jobSubmit the job

Jobset History View* Jobset is a set of one job and overJobsetJobset History ViewHistory View* * JobsetJobset is a set of one job and overis a set of one job and over

Jobset in detailJobsetJobset in detailin detail

Get and merge the resultsGet and merge the resultsGet and merge the results

HospitalHospitalHospital

GRID/Web GRID/Web インターフェースインターフェース33外部評価委員会2/Nov/2007

WNWNWN WNWNWN

WNWNWN WNWNWN WNWNWN

WNWNWN WNWNWN WNWNWN

WNWNWN WNWNWN WNWNWN

WNWNWN WNWNWN WNWNWN

WNWNWN WNWNWN WNWNWN

WNWNWN

WMSWMSWMS WMSWMSWMS WMSWMSWMS

GlobusGlobus I/FI/F GlobusGlobus I/FI/FGlobusGlobus I/FI/F

Job SubmissionJob Submission

MatchMakerMatchMatchMakerMaker

InformationSupermarketInformationInformationSupermarketSupermarket

Task QueueTask QueueTask Queue

Network ServerNetwork ServerNetwork Server

Resource BrokerResource BrokerResource Broker

Grid Web UIGrid Web UIGrid Web UI

SESESE SESESE SESESE

Grid access via HTTPJob submission, management, monitoringGet and browse results 

Grid access via HTTPGrid access via HTTPJob submission, management, monitoringJob submission, management, monitoringGet and browse results Get and browse results 

Resource BrokerInquiry resource informationJob queuing and logging 

Resource BrokerResource BrokerInquiry resource informationInquiry resource informationJob queuing and logging Job queuing and logging 

File CatalogueIndependent of physical location of filesReplication and transfer automatically

File CatalogueFile CatalogueIndependent of physical location of filesIndependent of physical location of filesReplication and transfer automaticallyReplication and transfer automatically

Virtual OrganizationBased on GSIAcross the institutes 

Virtual OrganizationVirtual OrganizationBased on GSIBased on GSIAcross the institutes Across the institutes 

Site‐ASiteSite‐‐AA Site‐BSiteSite‐‐BB Site‐CSiteSite‐‐CC

GRID: distributed computing over the InternetGRID: distributed computing over the InternetGRID: distributed computing over the Internet

2/Nov/2007 34外部評価委員会

IEEE NSS/MIC (Puerto Rico, Oct/2005)

Application to educationApplication to education

35

• 1.198 MeV gammas entering 3.004 cm of water• 40 events are displayed to “measure” the probability of the through gammas

Examples of Virtual Laboratory

Virtual Lab on WebWeb Textbook

2/Nov/2007

Geant4 Space workshop 2008

• Annual workshop for the geant4 users in space related field– Covering bio medical topics also related

human mission to Mars• Feb. 12-15, 2008 at Univ. of Tokyo

organized by JAXA• Details in the bottom of

http://cern.ch/geant4

http://cern.ch/geant4

GRID in JapanContentsNAREGIGRID Interoperability in HEPGRIDsProject presentationPeople concerned in the projectNAREGI at KEKInteroperability between EGEE and NAREGIGINAn example of interoperabilityThe SAGA / JSDL approachNext steps on NAREGI / EGEE interoperabilityThe Storage Resource Broker (SRB)From SRB to iRODSFrom SRB to iRODSSRB-DSI The SRB-DSI architecture Next step for SRB / LCG interoperabilitySoftware Suite for Particle Therapy Simulation The ProjectMember InstitutesGoal of The ProjectThe system architecture Basic design of Beam irradiation systemImplemented irradiation systems using PTSsimExample of irradiation systemsDICOM interfaceUser InterfacesApplication to educationGeant4 Space workshop 2008