Upload
shanon-jenkins
View
217
Download
0
Embed Size (px)
Citation preview
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 1
JIM and JIM and SAMGrid for SAMGrid for
CDFCDF
Mòrag Burgon-LyonMòrag Burgon-Lyon University of GlasgowUniversity of Glasgow
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 2
ContentsContents
• What is JIM?
• What is SAMGrid?
• How does JIM relate to SAMGrid?
• How does JIM work?
• Using JIM
• What will JIM do once complete?
• Deployment Plan
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 3
What is What is JIMJIM??
Job Management Infrastructure and Information Management
• Job Management Infrastructure is the framework allowing job submissions for execution on a cluster that has enough resources to complete the job.
• Information Management is knowing what resources are available for use.
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 4
What is SAMGrid?What is SAMGrid?
• SAMGrid is a grid infrastructure whose goal is to allow globally distributed computing for current experiments, CDF and DØ.
• Minos is strongly considering SAM
• Communicating with LHC experiments CMS (Fermilab) and Atlas (Brookhaven)
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 5
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
Complements Sequencial Access via Metadata to provide complete grid services
Job ManagementInformation
and Monitoring
Data Handling
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 6
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 7
How does JIM relate to SAMGrid?How does JIM relate to SAMGrid?
• JIM allows a user to submit jobs to SAMGrid and to access the output files on completion.
• JIM chooses which resources will be used to execute a job. This decision is based on the nearest location of the required data files.
• SAM executes the submitted job on the chosen cluster providing the I/O data management for the files.
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 8
How does JIM relate to SAMGridHow does JIM relate to SAMGrid
• Condor MMS was expanded for SAMGrid– Broker can query SAM station to see how much
data is already present.
• Globus toolkit is used for job transfer and monitoring
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 9
How does JIM work?How does JIM work?
JOB
Computing Element
Submission Client
User Interface
QueuingSystem
User Interface
User Interface
Broker
Match Making Service
Information Collector
Execution Site #1
Submission Client
Submission Client
Match Making Service
Match Making Service
Computing Element
Grid Sensors
Execution Site #n
Queuing System
Queuing System
Grid Sensors
Storage Element
Storage Element
Computing Element
Storage Element
Data Handling System
Data Handling System
Storage Element
Storage Element
Storage Element
Storage Element
Information Collector
Information Collector
Grid Sensor
s
Grid Sensor
s
Grid Sensor
s
Grid Sensor
s
Computing Element
Computing Element
Data Handling System
Data Handling System
Data Handling System
Data Handling System
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 10
How does JIM work?How does JIM work?
• Distinguishes grid level (global) scheduling (selection of a cluster to run) from local scheduling (distribution of the job within the cluster).
• Distinguishes structured jobs (where the details are known to Grid middleware) from unstructured jobs (where the whole job is mapped onto a single cluster).
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 11
Using JIM – Submitting a jobUsing JIM – Submitting a job
User creates a jdl file such as the example shown:
sam_dataset = jpmm08-1fileexecutable = retrieve.shinput_sandbox = /home_scotgrid/m/mlyon/test/testjobcpu-per-event = 1sjob_manager = samjob_type = sam_analysissam_universe = prdsam_experiment = cdflog = testjob.logoutput = testjob.outerror = testjob.errarguments = Download Output Yet?group = testinstances = 1
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 12
Using JIM – Submitting a jobUsing JIM – Submitting a job
• The job is submitted by typing:
samg submit testjobfile.jdf• The progress of the job can be viewed by selecting
the submission site from the list:
http://samgrid.fnal.gov:8080• Select the job from the list. Details of the job state
are displayed• Once the job has been completed the output may
be downloaded
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 13
Using JIM – Viewing MapUsing JIM – Viewing Map
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 14
Using JIM – Viewing submission sitesUsing JIM – Viewing submission sites
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 15
Using JIM – Viewing submitted jobsUsing JIM – Viewing submitted jobs
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 16
Using JIM – Downloading outputUsing JIM – Downloading output
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 17
What will JIM do once complete?What will JIM do once complete?
• Current functionality allows job submission and retrieval
• Next steps:– Making the web-download secure– User support and defect fixing through testing phase– Roll out of SAMGrid to all CDF sites with available
resources– Add more brokering criteria
• Directors review of Run II computing has recommended the expansion of SAM to be a lab-wide product
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 18
Deployment PlanDeployment Plan
Two plans running in parallel:• Testing JIM with non-trivial submissions:
– Large Monte Carlo Simulation– Reconstruction– Analysis
• JIM software installations:– Glasgow ScotGrid IBM and CDFG Dell cluster– Oxford Dell cluster– Liverpool Dell cluster– UCL Dell cluster– Wisconsin
In progress
CompletePlanned for October
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 19
Deployment PlanDeployment Plan
• Glasgow University has a complete installation of SAMGrid on both ScotGrid and the CDF cluster. This installation is being tested with Monte Carlo simulation.
• Oxford University will be installed in the first week of October.
• Other UK institutions and Wisconsin to follow
2222ndnd September 2003 September 2003 JIM for CDFJIM for CDF 20
CreditsCredits
• Thanks to Gabriele Garzoglio and the JIM team at Fermilab for providing material for this presentation.