High Throughput Computing with Condor at Purdue

XSEDE ECSS Monthly Symposium

Condor

• What is Condor?

• What is High Throughput Computing?

• Why Condor? Why not Condor?

• Condor at Purdue

• Submitting and managing jobs

• Suitable jobs

Topics

• A product of the University of Wisconsin-Madison

• A job scheduler• A resource manager• A workflow management system• Focused on High Throughput Computing

What is Condor?

What is High Throughput Computing (HTC)?

• Large amounts of processing

• Long period of time

HTC v. HPC

• FLOPS extracted v. FLOPS

• Distributed Ownership v. Central Ownership

• Capturing Idle Cycles v. Losing Idle Cycles

• Throughput v. Response Time

• Distributed Memory v. Tightly-coupled Memory

• 1,000 Jobs v. 1 Job

Why Condor?

• Wasted compute cycles

• Scheduling of related jobs

• Access to more cores

Advantages of Condor

• Many tasks running at once

• Access to more powerful computers

• Using wasted cycles

• Minimal impact on remote computers

• Security

• Little or no code modification

Disadvantages of Condor

• Compete for access• Task may take longer to complete• Processing can be lost• Parallel jobs aren’t available• Large files can impact the remote computer• Heterogeneity of the remote computers• Few compatible compilers

Condor at Purdue

• Installed on large cyberinfrastructure clusters• Installed in distributed desktops• Used as a scavenger of free cycles• Parallel jobs not supported• ~27K Linux cores and 1K Windows cores• Several more kilocores at DiaGrid partner sites

Condor at Purdue

• Jobs are vacated when a PBS job starts– Long running jobs may never complete

• Common home directory across clusters• Scratch directories roughly per-cluster• ~7 TB of checkpoint storage for standard

universe jobs

Job Universes

• Vanilla universe– Doesn't require a recompile– No native checkpoint mechanism

• Standard universe– Streams I/O (can overload the submit node)– Supports checkpointing– No fork(), shared memory, pipes

File transfer

• A vanilla universe feature• Allows jobs to flow to other sites

Compiling for Condor

• A standard universe requirement• The condor_compile command wraps a

limited compiler set.• Links against Condor libraries to add support for

I/O streaming and checkpointing

Checkpointing

• Saves all state information

• Transfers state information to Condor management

• Deletes job from processor

• Restarts interrupted job on another unused processor

Job lifecycle

• Job is submitted• Scheduler process contacts negotiator process• Negotiator matches job to an available slot• If no slots are available, scheduler contacts

remote negotiator• Execute node runs job• If job gets evicted, scheduler process contacts

negotiator process again

Submitting a job

• Create a submit file:# Simple Condor job file

Executable = bin/simpletest

Arguments = 600

Universe = standard

Log = log/$(Cluster).$(Process).log

Error = log/$(Cluster).$(Process).err

Output = log/$(Cluster).$(Process).out

+TGProject = TG-STA060013N

Queue 10

Submitting a job

• With file transfer:# Simple Condor job file

Executable = bin/process_files.sh

Universe = vanilla

ShouldTransferFiles = if_needed

Transfer_input_files = input.dat

Transfer_output_files = output.png

Log = log/$(Cluster).$(Process).log

+TGProject = TG-STA060013N

Queue

Submitting a job

• Job submitted with the condor_submit command:

condor_submit myjobfile.condor

Managing jobs

• Get all jobs in queue: condor_q• Get only user's jobs: condor_q user• Why isn't my job running?

condor_q -better-analyze jobid• Remove a job: condor_rm jobid

Getting the most cores: Requirements = ...

• Condor tries to be helpful by inserting automatic job requirements

• OpSys• Arch• FileSystemDomain• Memory >= ImageSize

• This sometimes over-constrains jobs

Getting the most cores: Requirements = ...

• The Requirements attribute gives you the flexibility to add or remove execute nodes

• Example: job files are in your home directory

Requirements = regexp(“rcac.purdue.edu”,FilesystemDomain)

• Example: job executable is a Windows binary

Requirements = (OpSys==“WINNT61”)

A special note about Memory

• Condor sometimes overestimates the memory usage of a job

• Condor reports totalmemory/cores, but jobs are not memory constrained

• It’s best to put a dummy memory requirement in the submission file

Getting the most out of your cores: Rank = ...

• You can prefer a job land on particular nodes• Example: prefer 64-bit nodes with lots of

memory

Rank = (ARCH==“X86_64”)*1000 + Memory

Workflow management with DAGman

• Directed Acyclic Graph Manager

• Defines parent-child relationships among jobs

• Allows pre- and post-execution hooks

• Submit with condor_submit_dag

Diamond DAG

C

A

B1 B2

Diamond DAG

# Diamond-shaped DAG

Job First p_00060.A.sub

Job Second_1 p_00060.B1.sub

Job Second_2 p_00060.B2.sub

Job Third p_00060.C.sub

PARENT First CHILD Second_1 Second_2

PARENT Second_1 Second_2 CHILD Third

More complex DAGs

Who Benefits from Condor?

• Monte Carlo simulations

• Parameter sweeps

• “Embarrassingly parallel” jobs

Purdue’s Condor Users

• Structural Biology

• Education

• Chemical Engineering

• Bioinformatics

• Climate Visualization

• Distributed Rendering

• High Energy Physics

For more information

• University of Wisconsin website:

• http://research.cs.wisc.edu/condor

• Email:

• bcotton@purdue.edu

• rcac-help@purdue.edu