1 Using Condor An Introduction ICE 2010

1http://www.cs.wisc.edu/condor

Using Condor An Introduction

ICE 2010


The Condor Project (Established ‘85)

Distributed High Throughput Computing research performed by a team of ~35 faculty, full time staff and students.


Definitions› Job

The Condor representation of your work

› Machine The Condor representation of

computers and that can perform the work

› Match Making Matching a job with a machine

“Resource”


Job

Jobs state their requirements and preferences:I need a Linux/x86 platformI need the machine at least 500 MbI prefer a machine with more memory


Machine

Machines state their requirements and preferences:Run jobs only when there is no keyboard

activityI prefer to run Frieda’s jobsI am a machine in the econ departmentNever run jobs belonging to Dr. Smith


The Magic of Matchmaking

› Jobs and machines state their requirements and preferences

› Condor matches jobs with machinesbased on requirements and

preferences


Getting Started:Submitting Jobs to

Condor› Overview:

Choose a “Universe” for your job Make your job “batch-ready” Create a submit description file Run condor_submit to put your job in

the queue


1. Choose the “Universe”› Controls how Condor handles jobs

› Choices include: Vanilla Standard Grid Java Parallel VM


Using the Vanilla Universe

• The Vanilla Universe:– Allows running

almost any “serial” job

– Provides automatic file transfer, etc.

– Like vanilla ice cream• Can be used in just

about any situation


2. Make your job batch-ready

Must be able to run in the background

• No interactive input

• No GUI/window clicks

• No music ;^)


Make your job batch-ready (continued)…

Job can still use STDIN, STDOUT, and STDERR (the keyboard and the screen), but files are used for these instead of the actual devices

Similar to UNIX shell:•$ ./myprogram <input.txt >output.txt


3. Create a Submit Description File

› A plain ASCII text file

› Condor does not care about file extensions

› Tells Condor about your job: Which executable, universe, input, output and

error files to use, command-line arguments, environment variables, any special requirements or preferences (more on this later)

› Can describe many jobs at once (a “cluster”), each with different input, arguments, output, etc.


Simple Submit Description File

# Simple condor_submit input file# (Lines beginning with # are comments)# NOTE: the words on the left side are not# case sensitive, but filenames are!Universe = vanillaExecutable = my_jobOutput = output.txt Queue


4. Run condor_submit› You give condor_submit the name

of the submit file you have created: condor_submit my_job.submit

› condor_submit: Parses the submit file, checks for

errors Creates a “ClassAd” that describes

your job(s) Puts job(s) in the Job Queue


The Job Queue

› condor_submit sends your job’s ClassAd(s) to the schedd

› The schedd (more details later): Manages the local job queue Stores the job in the job queue

• Atomic operation, two-phase commit• “Like money in the bank”

› View the queue with condor_q


Examplecondor_submit and

condor_q% condor_submit my_job.submitSubmitting job(s).1 job(s) submitted to cluster 1.

% condor_q

-- Submitter: perdita.cs.wisc.edu : <128.105.165.34:1027> : ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

1.0 frieda 6/16 06:52 0+00:00:00 I 0 0.0 my_job

1 jobs; 1 idle, 0 running, 0 held

%


Input, output & error files› Controlled by submit file settings

› You can define the job’s standard input, standard output and standard error: Read job’s standard input from “input_file”:

• Input = input_file• Shell equivalent: program <input_file

Write job’s standard ouput to “output_file”:• Output = output_file• Shell equivalent: program >output_file

Write job’s standard error to “error_file”:• Error = error_file• Shell equivalent: program 2>error_file


Email about your job

• Condor sends email about job events to the submitting user

• Specify “notification” in your submit file to control which events:

Notification = completeNotification = neverNotification = errorNotification = always

Default


Feedback on your job

› Create a log of job events

› Add to submit description file:log = sim.log

› Becomes the Life Story of a Job Shows all events in the life of a job Always have a log file


Sample Condor User Log

000 (0001.000.000) 05/25 19:10:03 Job submitted from host: <128.105.146.14:1816>

...

001 (0001.000.000) 05/25 19:12:17 Job executing on host: <128.105.146.14:1026>

...

005 (0001.000.000) 05/25 19:13:06 Job terminated.

(1) Normal termination (return value 0)

...


Example Submit Description File With

Logging# Example condor_submit input file# (Lines beginning with # are comments)# NOTE: the words on the left side are not# case sensitive, but filenames are!Universe = vanillaExecutable = /home/frieda/condor/my_job.condorLog = my_job.log ·Job log (from Condor)Input = my_job.in ·Program’s standard inputOutput = my_job.out ·Program’s standard outputError = my_job.err ·Program’s standard errorArguments = -a1 -a2 ·Command line argumentsInitialDir = /home/frieda/condor/runQueue


Let’s run a job

› First, need a terminal emulator http://www.putty.org

• (or similar)

› Login to chopin.cs.wisc.edu as cguserXX, and the given password

http://www.putty.org/


Logged In?

› condor_q

› condor_status


Create submit file

› nano submit.your_initials• universe = vanilla• executable = /bin/echo• Arguments = hello world• Output = out• Log = log• queue


And submit it…

› condor_submit submit.your_initials

› (wait… remember the HTC bit?)

› Condor_q xx

› cat output


“Clusters” and “Processes”

› If your submit file describes multiple jobs, we call this a “cluster”

› Each cluster has a unique “cluster number”› Each job in a cluster is called a “process”

Process numbers always start at zero› A Condor “Job ID” is the cluster number, a period,

and the process number (i.e. 2.1) A cluster can have a single process

• Job ID = 20.0 ·Cluster 20, process 0 Or, a cluster can have more than one process

• Job ID: 21.0, 21.1, 21.2 ·Cluster 21, process 0, 1, 2


Submit File for a Cluster# Example submit file for a cluster of 2 jobs

# with separate input, output, error and log filesUniverse = vanillaExecutable = my_jobArguments = -x 0log = my_job_0.logInput = my_job_0.inOutput = my_job_0.outError = my_job_0.errQueue ·Job 2.0 (cluster 2, process 0)Arguments = -x 1log = my_job_1.logInput = my_job_1.inOutput = my_job_1.outError = my_job_1.errQueue ·Job 2.1 (cluster 2, process 1)


% condor_submit my_job.submit-file

Submitting job(s).

2 job(s) submitted to cluster 2.

% condor_q

-- Submitter: perdita.cs.wisc.edu : <128.105.165.34:1027> :

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

1.0 frieda 4/15 06:52 0+00:02:11 R 0 0.0 my_job –a1 –a2

2.0 frieda 4/15 06:56 0+00:00:00 I 0 0.0 my_job –x 0

2.1 frieda 4/15 06:56 0+00:00:00 I 0 0.0 my_job –x 1


%

Submitting The Job


Organize your files and directories for big runs

› Create subdirectories for each “run” run_0, run_1, … run_599

› Create input files in each of these run_0/simulation.in run_1/simulation.in … run_599/simulation.in

› The output, error & log files for each job will be created by Condor from your job’s output


Submit Description File for 600 Jobs

# Cluster of 600 jobs with different directoriesUniverse = vanillaExecutable = simLog = simulation.log...Arguments = -x 0

InitialDir = run_0 ·Log, input, output & error files -> run_0

Queue ·Job 3.0 (Cluster 3, Process 0)

Arguments = -x 1

InitialDir = run_1 ·Log, input, output & error files -> run_1

Queue ·Job 3.1 (Cluster 3, Process 1)

·Do this 598 more times…………


Submit File for a Big Cluster of Jobs

› We just submitted 1 cluster with 600 processes

› All the input/output files will be in different directories

› The submit file is pretty unwieldy (over 1200 lines)

› Isn’t there a better way?


Submit File for a Big Cluster of Jobs (the better

way) #1› We can queue all 600 in 1 “Queue” command Queue 600

› Condor provides $(Process) and $(Cluster) $(Process) will be expanded to the

process number for each job in the cluster• 0, 1, … 599

$(Cluster) will be expanded to the cluster number• Will be 4 for all jobs in this cluster


Submit File for a Big Cluster of Jobs (the better

way) #2› The initial directory for each job can

be specified using $(Process) InitialDir = run_$(Process) Condor will expand these to “run_0”,

“run_1”, … “run_599” directories

› Similarly, arguments can be variable Arguments = -x $(Process) Condor will expand these to “-x 0”, “-x 1”, … “-x 599”


Better Submit File for 600 Jobs

# Example condor_submit input file that defines# a cluster of 600 jobs with different directoriesUniverse = vanillaExecutable = my_jobLog = my_job.logInput = my_job.inOutput = my_job.outError = my_job.errArguments = –x $(Process) ·–x 0, -x 1, … -x

599InitialDir = run_$(Process) ·run_0 … run_599Queue 600 ·Jobs

4.0 … 4.599


Now, we submit it…$ condor_submit my_job.submitSubmitting

job(s) ...............................................................................................................................................................................................................................................................

Logging submit event(s) ...............................................................................................................................................................................................................................................................



And, Check the queue$ condor_q

-- Submitter: x.cs.wisc.edu : <128.105.121.53:510> : x.cs.wisc.edu

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD

4.0 frieda 4/20 12:08 0+00:00:05 R 0 9.8 my_job -arg1 –x 0

4.1 frieda 4/20 12:08 0+00:00:03 I 0 9.8 my_job -arg1 –x 1



...





Removing jobs› If you want to remove a job from

the Condor queue, you use condor_rm

› You can only remove jobs that you own

› Privileged user can remove any jobs “root” on UNIX “administrator” on Windows


Removing jobs (continued)

› Remove an entire cluster: condor_rm 4 ·Removes the whole

cluster

› Remove a specific job from a cluster: condor_rm 4.0 ·Removes a single job

› Or, remove all of your jobs with “-a” condor_rm -a ·Removes all jobs /

clusters


Submit cluster of 10 jobs

› nano submit• universe = vanilla• executable = /bin/echo• Arguments = hello world $(PROCESS)• Output = out.$(PROCESS)• Log = log• Queue 10


And submit it…

› condor_submit submit

› (wait…)

› Condor_q xx

› cat log

› cat output.yy


My new jobs run for 20 days…

› What happens when a job is forced off it’s CPU? Preempted by higher priority

user or job Vacated because of user

activity

› How can I add fault tolerance to my jobs?


Condor’s Standard Universe to the rescue!› Support for transparent process

checkpoint and restart› Remote system calls (remote

I/O)Your job can read / write files as if they were local


Remote System Calls inthe Standard Universe

› I/O system calls are trapped and sent back to the submit machineExamples: open a file, write to a file

› No source code changes typically required

› Programming language independent


Process Checkpointing in the

Standard Universe› Condor’s process checkpointing provides a mechanism to automatically save the state of a job

› The process can then be restarted from right where it was checkpointed After preemption, crash, etc.


Checkpointing:Process Starts

checkpoint: the entire state of a program, saved in a file CPU registers, memory image, I/O

time


Checkpointing:Process Checkpointed

time

1 2 3


Checkpointing:Process Killed

time

3

3

Killed!


Checkpointing:Process Resumed

time

3

3

goodput badput goodput


When will Condor checkpoint your job?

› Periodically, if desired For fault tolerance

› When your job is preempted by a higher priority job

› When your job is vacated because the execution machine becomes busy

› When you explicitly run condor_checkpoint, condor_vacate, condor_off or condor_restart command


Making the Standard Universe Work

› The job must be relinked with Condor’s standard universe support library

› To relink, place condor_compile in front of the command used to link the job:

% condor_compile gcc -o myjob myjob.c

- OR -

% condor_compile f77 -o myjob filea.f fileb.f

- OR -

% condor_compile make –f MyMakefile


Limitations of the Standard Universe

› Condor’s checkpointing is not at the kernel level. Standard Universe the job may not:

• Fork()• Use kernel threads• Use some forms of IPC, such as pipes and shared

memory

› Must have access to source code to relink

› Many typical scientific jobs are OK


Submitting Std uni job

› #include <stdio.h>

› int main(int argc, char **argv) {

› int i;for(i = 0 ; i < 10000000; i++) {}

› }


And submit…

› condor_compile gcc –o foo foo.c

-- Change "vanilla" to "standard" -- Change "/bin/echo" to "foo" (or

above)


My jobs have have dependencies…

Can Condor help solve my dependency problems?


Condor Universes:Scheduler and Local

› Scheduler Universe Plug in a meta-scheduler Developed for DAGMan (more later) Similar to Globus’s fork job manager

› Local Very similar to vanilla, but jobs run on

the local host Has more control over jobs than

scheduler universe


DAGMan

› Directed Acyclic Graph Manager

› DAGMan allows you to specify the dependencies between your Condor jobs, so it can manage them automatically for you.

› (e.g., “Don’t run job “B” until job “A” has completed successfully.”)


What is a DAG?

› A DAG is the data structure used by DAGMan to represent these dependencies.

› Each job is a “node” in the DAG.

› Each node can have any number of “parent” or “children” nodes – as long as there are no loops!

Job A

Job B

Job C

Job D


Defining a DAG

› A DAG is defined by a .dag file, listing each of its nodes and their dependencies:# diamond.dagJob A a.subJob B b.subJob C c.subJob D d.subParent A Child B CParent B C Child D

› each node will run the Condor job specified by its accompanying Condor submit file

Job A

Job B Job C

Job D


Submitting a DAG

› To start your DAG, just run condor_submit_dag with your .dag file, and Condor will start a personal DAGMan daemon which to begin running your jobs:

% condor_submit_dag diamond.dag

› condor_submit_dag is run by the schedd DAGMan daemon itself is “watched” by

Condor, so you don’t have to


DAGMan

Running a DAG

› DAGMan acts as a “meta-scheduler”, managing the submission of your jobs to Condor based on the DAG dependencies.

CondorJobQueue

B C

D

A

A

.dagFile


DAGMan

Running a DAG (cont’d)

› DAGMan holds & submits jobs to the Condor queue at the appropriate times.

CondorJobQueue D

B

C

B

A

C


Running a DAG (cont’d)

› In case of a job failure, DAGMan continues until it can no longer make progress, and then creates a “rescue” file with the current state of the DAG.

CondorJobQueue DAGMan

X

D

A

BRescue

File


Recovering a DAG

› Once the failed job is ready to be re-run, the rescue file can be used to restore the prior state of the DAG.

CondorJobQueue

RescueFile

CDAGMan D

A

B C


DAGMan

Recovering a DAG (cont’d)

› Once that job completes, DAGMan will continue the DAG as if the failure never happened.

CondorJobQueue

C

D

A

B

D


DAGMan

Finishing a DAG

› Once the DAG is complete, the DAGMan job itself is finished, and exits.

CondorJobQueue

C

D

A

B


Additional DAGMan Features

› Provides other handy features for job management…

nodes can have PRE & POST scripts failed nodes can be automatically re-

tried a configurable number of times job submission can be “throttled”


What about Licensed Jobs?

› e.g. matlab Site license? matlab compiler Octave


Chirp

› condor_chirp get_file remote local

› condor_chirp put_file local remote


General User Commands› condor_status View Pool Status

› condor_q View Job Queue› condor_submit Submit new Jobs› condor_rm Remove Jobs› condor_prio Intra-User Prios› condor_history Completed Job Info› condor_submit_dag Submit new DAG› condor_checkpoint Force a checkpoint› condor_compile Link Condor library


Statistical Bootstrap› Build up from the worker side out

› The matlab/octave worker:

› worker.m:

#!/s/octave/bin/octave -q

load "subset" subset;

subset = subset(floor(rand(10,1) .* 1000));

printf("%f ", mean(subset));


Run the worker alone

› (won’t work – why?)

› ./worker.m


Submit file

universe = vanilla

executable = worker.m

should_transfer_files = true

when_to_transfer_output = on_exit

transfer_input_files = subset

output = mean.$(PROCESS)

error = foo

log = log

queue 10


Create the initial datadriver.m

#!/s/octave/bin/octave –qdist_size = 100000;d = rand(dist_size, 1) .* 500;subset = d(floor(rand(1000,1) .*

100000));save "subset" subset;


And submit the job…

› condor_submit submit


Add the submission tothe driver script…

#!/s/octave/bin/octave –q

dist_size = 100000;

d = rand(dist_size, 1) .* 500;

subset = d(floor(rand(1000,1) .* 100000));

save "subset" subset;

system("condor_submit submit");

system("condor_wait log");


And run the driver!

› ./driver.m


Parallel convergence checking:

Another DAGman example› Evaluating a function at many

points

› Check for convergence -> retry

› Particle Swarm Optimization


Prepare

Compute Compute Compute Compute

Converge?

Done

Yes!

No


Any Guesses?

› Who has thoughts?

› Best to work from “inside out”

79


The job itself.

#!/bin/sh

###### random.sh

echo $RANDOM

exit 0

80


The submit file

› Any guesses?

81


The submit file# submitRandom

universe = vanilla

executable = random.sh

Should_transfer_files = yes

When_to_transfer_output = on_exit

output = out

log = log

queue

82


Next step: the inner DAG

83

First Last Node

Node0

Node1 Node2 Node3 Node4

Node11


The DAG file

› Any guesses?

84


The inner DAG file

Job Node0 submitRandom




PARENT Node0 CHILD Node1




PARENT Node1, Node2, Node3 CHILD Node11

85


Inner DAG

› Does this work?

› At least one iteration?

86


How to iterate

› DAGman has simple control structures (Makes it reliable)

› SUBDAGs!

› Remember what happens if post fails?

87


The Outer Dag› Another Degenerate Dag

(But Useful!)

88

Post Script (with exit value)

SubDag (with retry)t


This one is easy!

› Can you do it yourself?

89


The outer DAG file

####### Outer.dag #############

SUBDAG EXTERNAL A inner.dag

SCRIPT POST A converge.sh

RETRY A 10

#### converge.sh could look like

#!/bin/sh

echo "Checking convergence" >> converge

exit 1

90


Let’s run that…

› condor_submit_dag outer.dag

› Does it work? How can you tell?

91


DAGman a bit verbose…$ condor_submit_dag outer.dag

-----------------------------------------------------------------------

File for submitting this DAG to Condor : submit.dag.condor.sub

Log of DAGMan debugging messages : submit.dag.dagman.out

Log of Condor library output : submit.dag.lib.out

Log of Condor library error messages : submit.dag.lib.err

Log of the life of condor_dagman itself : submit.dag.dagman.log

-no_submit given, not submitting DAG to Condor. You can do this with:

"condor_submit submit.dag.condor.sub"

-----------------------------------------------------------------------

-----------------------------------------------------------------------

File for submitting this DAG to Condor : outer.dag.condor.sub

Log of DAGMan debugging messages : outer.dag.dagman.out

Log of Condor library output : outer.dag.lib.out

Log of Condor library error messages : outer.dag.lib.err

Log of the life of condor_dagman itself : outer.dag.dagman.log

Submitting job(s).

Logging submit event(s).


-----------------------------------------------------------------------

92


Debugging helps

› Look in the user log file, “log”

› Look in the DAGman debugging log

› “foo”.dagman.out

93


What does converge.sh need

› Note the output files?

› How to make them unique?

› Add DAG variables to inner dag And submitRandom file

94


The submit file (again)

# submitRandom

universe = vanilla


output = out

log = log

queue

95


The submit file

# submitRandom

universe = vanilla


output = out.$(NodeNumber)

log = log

queue

96


The inner DAG file (again)

Job Node0 submit_pre







Job Node11 submit_post




97


The inner DAG file (again)

Job Node0 submit_pre




…

VARS Node1 NodeNumber=“1”



…

98


Then converge.sh sees:

$ ls out.*

out.1 out.10 out.2 out.3 out.4 out.5 out.6 out.7 out.8 out.9

$

› And can act accordingly…

99


Thank you!

Check us out on the Web:http://www.condorproject.org

Email:[email protected]

Documents

1 Using Condor An Introduction ICE 2010