Condor Project Computer Sciences Department University of Wisconsin-Madison condor-admin@cs.wisc.edu Grids and Condor Barcelona,

  • View
    226

  • Download
    6

Embed Size (px)

Text of Condor Project Computer Sciences Department University of Wisconsin-Madison condor-admin@cs.wisc.edu...

  • Slide 1

Condor Project Computer Sciences Department University of Wisconsin-Madison condor-admin@cs.wisc.edu http://www.cs.wisc.edu/condor Grids and Condor Barcelona, 2006 Slide 2 2 http://www.cs.wisc.edu/condor Agenda Extended users tutorial Advanced Uses of Condor Java programs DAGMan Stork MW Grid Computing Case studies, and a discussion of your applications needs Slide 3 3 http://www.cs.wisc.edu/condor Resources There are many resources (machines) in the world, and many are or can be made available! Groups of machines may be labeled as grids Welcome to the power of the grid ! Slide 4 4 http://www.cs.wisc.edu/condor Condor and Grids Condor has always been a tool to harness grid computing Condors mechanisms have evolved as technologies have evolved. Roughly categorized: Flocking Glidein The grid universe Slide 5 5 http://www.cs.wisc.edu/condor Flocking A way for jobs to run within a different, separate Condor pool Condor runs here, and Condor runs there here there Slide 6 6 http://www.cs.wisc.edu/condor Connect Condor Pools with Flocking Flocking is a Condor-specific technology Flocking is enabled with configuration Jobs flock from here to there when they cannot be run here due to lack of available machines Slide 7 7 http://www.cs.wisc.edu/condor Configuration Configuration files contain lots of the administrative information used by Condor Format is like that in submit description files: AttributeName = Value Slide 8 8 http://www.cs.wisc.edu/condor Configuration here For jobs to be able to flock from here to there In the configuration file on the pool where jobs flock from: FLOCK_TO = FLOCK_COLLECTOR_HOSTS = $(FLOCK_TO) FLOCK_NEGOTIATOR_HOSTS = $(FLOCK_TO) HOSTALLOW_NEGOTIATOR_SCHEDD = $(COLLECTOR_HOST), $(FLOCK_NEGOTIATOR_HOSTS) Slide 9 9 http://www.cs.wisc.edu/condor Configuration there In the configuration file on the pool where jobs flock to: FLOCK_FROM =,..., To make security work: HOSTALLOW_WRITE_COLLECTOR = $(HOSTALLOW_WRITE), $(FLOCK_FROM) HOSTALLOW_WRITE_STARTD = $(HOSTALLOW_WRITE), $(FLOCK_FROM) HOSTALLOW_READ_COLLECTOR = $(HOSTALLOW_READ), $(FLOCK_FROM) HOSTALLOW_READ_STARTD = $(HOSTALLOW_READ), $(FLOCK_FROM) Slide 10 10 http://www.cs.wisc.edu/condor Submit Description File Enable file transfer: universe = vanilla executable = myjob.exe input = myjob.input output = myjob.output log = myjob.log should_transfer_files = YES when_to_transfer_output = ON_EXIT queue Slide 11 11 http://www.cs.wisc.edu/condor The Glidein Concept Assume: We need more machines, and we have permission to use a set of machines Glidein temporarily adds a set of machines to the local pool Slide 12 12 http://www.cs.wisc.edu/condor Glidein In addition, Glidein solves the problem: My job needs to run on that particular resource, and my job needs Condor. For example: a job that must run under the standard universe Slide 13 13 http://www.cs.wisc.edu/condor Glidein Condor sends and runs its own executables on the resource The needed resource appears to temporarily join the local Condor pool ! Slide 14 14 http://www.cs.wisc.edu/condor Glidein run condor_glidein to add the remote resource to the local pool local pool remote resource the master and startd daemons become grid universe jobs using gt2 Slide 15 15 http://www.cs.wisc.edu/condor Making Glidein Work Change the configuration to give access permission ( HOSTALLOW_WRITE ) to the remote resource No changes to jobs submit description files! But, do enable file transfer in the submit description file: universe = vanilla executable = myjob.exe input = myjob.input output = myjob.output log = myjob.log should_transfer_files = YES when_to_transfer_output = ON_EXIT queue Slide 16