Embed Size (px)
Citation preview
Contact: Junwei Cao ([email protected])SC2005, Seattle, WA, November 12-18, 2005The authors gratefully acknowledge the support of the United States National Science Foundation for the construction and operation of the LIGO Laboratory and the Particle Physics and Astronomy Research Council of the United Kingdom, the Max-Planck-Society and the State of Niedersachsen/Germany for support of the construction and operation of the GEO600 detector. The authors also gratefully acknowledge the support of the research by these agencies and by the Australian Research Council, the Natural Sciences and Engineering Research Council of Canada, the Council of Scientific and Industrial Research of India, the Department of Science and Technology of India, the Spanish Ministerio de Educacion y Ciencia, the John Simon Guggenheim Foundation, the Leverhulme Trust, the David and Lucile Packard Foundation, the Research Corporation, and the Alfred P. Sloan Foundation. This work is directly supported under National Science Foundation award #0326281.
Grid Enabled LIGO Data MonitoringJunwei Cao1, Erik Katsavounidis1, and John Zweizig2
1LIGO Laboratory - Massachusetts Institute of Technology, Cambridge, MA 02139, USA2LIGO Laboratory - California Institute of Technology, Pasadena, CA 91125, USA
LIGO: Laser Interferometer Gravitational-wave Observatory
The LSC Data Grid (LDG)
AEI/Golm
CardiffBirmingham•
AEI/GolmAEI/Golm
CardiffBirmingham•
CardiffCardiffBirmingham•
Users are interfaced with a LIGO friendly language.
[job]id = testmonitor = rmonargs = -opt optinput = opt[data]observatory = @H@Ltype = @RDS_R_L3@RDS_R_L3start = 751658000end = 751676993
ldm.sub
[jcao@ldaspc1 ~]$ cd ldm[jcao@ldaspc1 ldm]$ source setup.sh[jcao@ldaspc1 ldm]$ cd ../rmon[jcao@ldaspc1 rmon]$ ldm_agent[jcao@ldaspc1 rmon]$ ldm_submit ldm.subJob test has been submitted.[jcao@ldaspc1 rmon]$ more ldm_test_condor.outProcessing multi list file: ldm_test_CIT_multilist.txt
Number of lists added: 2 Total data streams: 2……startgps=751658000 stride=16 r-statistic=-0.00251782……
grid-enabled run of rmon DMT offline monitor using LDM
universe = globusglobusscheduler = ldas-grid.ligo.caltech.edu/jobmanager-condorlog = ldm_test_condor.logoutput = ldm_test_condor.outerror = ldm_test_condor.errshould_transfer_files = YESwhen_to_transfer_output = ON_EXITtransfer_input_files = ldm_test_CIT_multilist.txt, ldm_test_CIT_filelist1.txt,ldm_test_CIT_filelist2.txt, /home/jcao/rmon/optarguments = -inlists ldm_test_CIT_multilist.txt -opt optenvironment = LD_LIBRARY_PATH=/dso-test/jcao/dol/libexecutable = /home/jcao/rmon/rmonQueue
automatically generated Condor submission file
Users do not bother with technical details of LSC
data grid services.
Data are located and file lists are generated automatically
Users are interfaced with a LIGO friendly language.
[job]id = testmonitor = rmonargs = -opt optinput = opt[data]observatory = @H@Ltype = @RDS_R_L3@RDS_R_L3start = 751658000end = 751676993
ldm.sub
[jcao@ldaspc1 ~]$ cd ldm[jcao@ldaspc1 ldm]$ source setup.sh[jcao@ldaspc1 ldm]$ cd ../rmon[jcao@ldaspc1 rmon]$ ldm_agent[jcao@ldaspc1 rmon]$ ldm_submit ldm.subJob test has been submitted.[jcao@ldaspc1 rmon]$ more ldm_test_condor.outProcessing multi list file: ldm_test_CIT_multilist.txt
Number of lists added: 2 Total data streams: 2……startgps=751658000 stride=16 r-statistic=-0.00251782……
grid-enabled run of rmon DMT offline monitor using LDM
universe = globusglobusscheduler = ldas-grid.ligo.caltech.edu/jobmanager-condorlog = ldm_test_condor.logoutput = ldm_test_condor.outerror = ldm_test_condor.errshould_transfer_files = YESwhen_to_transfer_output = ON_EXITtransfer_input_files = ldm_test_CIT_multilist.txt, ldm_test_CIT_filelist1.txt,ldm_test_CIT_filelist2.txt, /home/jcao/rmon/optarguments = -inlists ldm_test_CIT_multilist.txt -opt optenvironment = LD_LIBRARY_PATH=/dso-test/jcao/dol/libexecutable = /home/jcao/rmon/rmonQueue
automatically generated Condor submission file
Users do not bother with technical details of LSC
data grid services.
Data are located and file lists are generated automatically
/data/node10/frame/S3/L3/LHO/H-RDS_R_L3-751658016-16.gwf/data/node11/frame/S3/L3/LHO/H-RDS_R_L3-751658032-16.gwf/data/node12/frame/S3/L3/LHO/H-RDS_R_L3-751658048-16.gwf/data/node13/frame/S3/L3/LHO/H-RDS_R_L3-751658064-16.gwf/data/node14/frame/S3/L3/LHO/H-RDS_R_L3-751658080-16.gwf/data/node15/frame/S3/L3/LHO/H-RDS_R_L3-751658096-16.gwf/data/node16/frame/S3/L3/LHO/H-RDS_R_L3-751658112-16.gwf
/data/node10/frame/S3/L3/LLO/L-RDS_R_L3-751658016-16.gwf/data/node11/frame/S3/L3/LLO/L-RDS_R_L3-751658032-16.gwf/data/node12/frame/S3/L3/LLO/L-RDS_R_L3-751658048-16.gwf/data/node13/frame/S3/L3/LLO/L-RDS_R_L3-751658064-16.gwf/data/node14/frame/S3/L3/LLO/L-RDS_R_L3-751658080-16.gwf/data/node15/frame/S3/L3/LLO/L-RDS_R_L3-751658096-16.gwf/data/node16/frame/S3/L3/LLO/L-RDS_R_L3-751658112-16.gwf
rmon
filelist1.txtfilelist2.txt
multilist.txt
stride 16.0channel_1 H1:LSC-AS_Qchannel_2 L1:LSC-AS_Q
opt
filelist1.txt
filelist2.txt
[jcao@ldaspc1 rmon]$ export LD_LIBRARY_PATH=/opt/lscsoft/dol/lib[jcao@ldaspc1 rmon]$ ./rmon -opt opt -inlists multilist.txtProcessing multi list file: multilist.txt
Number of lists added: 2 Total data streams: 2Processing frame list file: /home/jcao/rmon/filelist1.txt
Number of files added: 1188 Total frame files: 1188Processing frame list file: /home/jcao/rmon/filelist2.txt
Number of files added: 1188 Total frame files: 1188channel[1]=H1:LSC-AS_Q channel[2]=L1:LSC-AS_Qstartgps=751658000 stride=16 r-statistic=-0.00251782startgps=751658016 stride=16 r-statistic=-0.0122699startgps=751658032 stride=16 r-statistic=0.0168868……
standalone run of rmon DMT offline monitor
/data/node10/frame/S3/L3/LHO/H-RDS_R_L3-751658016-16.gwf/data/node11/frame/S3/L3/LHO/H-RDS_R_L3-751658032-16.gwf/data/node12/frame/S3/L3/LHO/H-RDS_R_L3-751658048-16.gwf/data/node13/frame/S3/L3/LHO/H-RDS_R_L3-751658064-16.gwf/data/node14/frame/S3/L3/LHO/H-RDS_R_L3-751658080-16.gwf/data/node15/frame/S3/L3/LHO/H-RDS_R_L3-751658096-16.gwf/data/node16/frame/S3/L3/LHO/H-RDS_R_L3-751658112-16.gwf
/data/node10/frame/S3/L3/LLO/L-RDS_R_L3-751658016-16.gwf/data/node11/frame/S3/L3/LLO/L-RDS_R_L3-751658032-16.gwf/data/node12/frame/S3/L3/LLO/L-RDS_R_L3-751658048-16.gwf/data/node13/frame/S3/L3/LLO/L-RDS_R_L3-751658064-16.gwf/data/node14/frame/S3/L3/LLO/L-RDS_R_L3-751658080-16.gwf/data/node15/frame/S3/L3/LLO/L-RDS_R_L3-751658096-16.gwf/data/node16/frame/S3/L3/LLO/L-RDS_R_L3-751658112-16.gwf
rmon
filelist1.txtfilelist2.txt
multilist.txt
stride 16.0channel_1 H1:LSC-AS_Qchannel_2 L1:LSC-AS_Q
opt
filelist1.txt
filelist2.txt
[jcao@ldaspc1 rmon]$ export LD_LIBRARY_PATH=/opt/lscsoft/dol/lib[jcao@ldaspc1 rmon]$ ./rmon -opt opt -inlists multilist.txtProcessing multi list file: multilist.txt
Number of lists added: 2 Total data streams: 2Processing frame list file: /home/jcao/rmon/filelist1.txt
Number of files added: 1188 Total frame files: 1188Processing frame list file: /home/jcao/rmon/filelist2.txt
Number of files added: 1188 Total frame files: 1188channel[1]=H1:LSC-AS_Q channel[2]=L1:LSC-AS_Qstartgps=751658000 stride=16 r-statistic=-0.00251782startgps=751658016 stride=16 r-statistic=-0.0122699startgps=751658032 stride=16 r-statistic=0.0168868……
standalone run of rmon DMT offline monitor
Applications
Infrastructures
End users & applications
Application enabling
Middleware / Services
Operating Systems and …FC3 GCC AutotoolsPython MySQL
Job scheduling / Condor
Condor-G
Data transfer / GridFTP
Worklfow management / Condor DAGman
Grid security / Globus GSI Resource management / Globus GRAM
Resource location service / GlobusCatalog service / Globus Information service / Globus MDS
VDS
LSCcertUtilsLSC Security management
LSCdataFind LSCsegFindLDRLSC Data management
The LIGO Data Monitoring (LDM) Environment Glue OnasysLSC Job management
LDAS DMT LALApps
LSC CA
VOMS
The LSC Data Grid Client/Server Environment Version 3.5 (using VDT 1.3.5)
Matlab
Applications
Infrastructures
End users & applications
Application enabling
Middleware / Services
Operating Systems and …FC3 GCC AutotoolsPython MySQL
Job scheduling / Condor
Condor-G
Data transfer / GridFTP
Worklfow management / Condor DAGman
Grid security / Globus GSI Resource management / Globus GRAM
Resource location service / GlobusCatalog service / Globus Information service / Globus MDS
VDS
LSCcertUtilsLSC Security management
LSCdataFind LSCsegFindLDRLSC Data management
The LIGO Data Monitoring (LDM) Environment Glue OnasysLSC Job management
LDAS DMT LALApps
LSC CA
VOMS
The LSC Data Grid Client/Server Environment Version 3.5 (using VDT 1.3.5)
Matlab
QUEUED
SCHEDULED
LOCATING
LOCATED
RUNNING
FINISHED
ldm_agent
ldm_locate_script
ldm_exec_script
ldm_submit
RELEASED
ldm_rm
ldm_q
REJECTED
LSCdataFind
condor_submit
LD
M
Oth
er t
oo
ls
client
serv
er
LSCdataFindServer
GlobusJob Manager
ldm_agent
Condor[AGENT]RESOURCES = @MIT@CIT@LHO@LLOSITES = /home/jcao/ldm/etc/LDM_SITESEXEC = /home/jcao/ldm/bin/ldm_exec_scriptLOCATE = /home/jcao/ldm/bin/ldm_locate_scriptPID = /home/jcao/ldm/var/ldm.pidLOG = /home/jcao/ldm/var/ldm.logLDG = /home/jcao/ldg-3.0/
[MIT]lscdatafindserver = ldas-gridmon.mit.eduglobusscheduler = ldas-grid.mit.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/dso-test/home/jcao/dol/lib[CIT]lscdatafindserver = ldas-gridmon.ligo.caltech.eduglobusscheduler = ldas-grid.ligo.caltech.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/dso-test/jcao/dol/lib[LHO]lscdatafindserver = ldas-gridmon.ligo-wa.caltech.eduglobusscheduler = ldas-grid.ligo-wa.caltech.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/dso-test/jcao/dol/lib[LLO]lscdatafindserver = ldas-gridmon.ligo-la.caltech.eduglobusscheduler = ldas-grid.ligo-la.caltech.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/data2/jcao/dol/lib
LDM_CONFIG
LDM_SITES
Modules developedor deployed
Modules designedand underdeveloped
condor_master
QUEUED
SCHEDULED
LOCATING
LOCATED
RUNNING
FINISHED
ldm_agent
ldm_locate_script
ldm_exec_script
ldm_submit
RELEASED
ldm_rm
ldm_q
REJECTED
LSCdataFind
condor_submit
LD
M
Oth
er t
oo
lsLD
M
Oth
er t
oo
ls
client
serv
er
client
serv
er
LSCdataFindServer
GlobusJob Manager
ldm_agent
Condor[AGENT]RESOURCES = @MIT@CIT@LHO@LLOSITES = /home/jcao/ldm/etc/LDM_SITESEXEC = /home/jcao/ldm/bin/ldm_exec_scriptLOCATE = /home/jcao/ldm/bin/ldm_locate_scriptPID = /home/jcao/ldm/var/ldm.pidLOG = /home/jcao/ldm/var/ldm.logLDG = /home/jcao/ldg-3.0/
[MIT]lscdatafindserver = ldas-gridmon.mit.eduglobusscheduler = ldas-grid.mit.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/dso-test/home/jcao/dol/lib[CIT]lscdatafindserver = ldas-gridmon.ligo.caltech.eduglobusscheduler = ldas-grid.ligo.caltech.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/dso-test/jcao/dol/lib[LHO]lscdatafindserver = ldas-gridmon.ligo-wa.caltech.eduglobusscheduler = ldas-grid.ligo-wa.caltech.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/dso-test/jcao/dol/lib[LLO]lscdatafindserver = ldas-gridmon.ligo-la.caltech.eduglobusscheduler = ldas-grid.ligo-la.caltech.edu/jobmanager-condorenvironment = LD_LIBRARY_PATH=/data2/jcao/dol/lib
LDM_CONFIG
LDM_SITES
Modules developedor deployed
Modules designedand underdeveloped
condor_master
• The LIGO project aims to make the first direct detection of gravitational waves as predicted by Einstein’s General Theory of Relativity.
• LIGO manages and operates two national observatories for gravitational waves, located at Hanford, WA and Livingston, LA.
• The LIGO Scientific Collaboration (LSC) consists of nearly 400 scientists from about 40 institutions worldwide.
• In the LSC, a grid computing infrastructure is utilized for LIGO data analysis and monitoring with thousands of CPUs and terabytes of data storage capabilities distributed over 10 sites in the USA and Europe.
• The LSC Data Grid (LDG) client/server environment is built on top of the Virtual Data Toolkit (VDT).
The LIGO Data Monitoring (LDM) Environment
• LDM interfaces users with a LIGO friendly language instead of the more CS flavor Condor language.
• Technical details of grid computing are hidden from end users (mostly gravitational wave physicists in the LSC).
• The experience gained in this work will be applied for future deployment of the Open Science Grid (OSG) infrastructure for LIGO applications.
The Data Monitoring Toolkit (DMT)
• DMT defines tools and environments necessary to support continuous data monitoring of LIGO interferometers.
• While DMT is essentially a production-level software package being used in LIGO control rooms, researchers find the enrichment of DMT libraries make it an ideal environment for offline LIGO data analysis.
For More Information …
http://www.ligo.caltech.edu
http://www.ligo.mit.edu/lsc-itr2003
http://www.ligo.org
http://www.lsc-group.phys.uwm.
edu/lscdatagrid
http://www.ligo.mit.edu/ldg
http://www.griphyn.org
http://www.ivdgl.org
http://www.opensciencegrid.org
For the LIGO ScientificCollaboration (LSC)
LIGO Document No. G050573-00-E
