• Home

  • Documents

  • An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch...
17
An overview of batch processing 2-June-2016

An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Embed Size (px)

Citation preview

Page 1: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Anoverviewofbatchprocessing

2-June-2016

Page 2: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Yourcomputer

Yourprogram

One-on-one

Page 3: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Yourcomputer(mul?plecores)

YourprogramYourprogram Yourprogram YourprogramYourprogram

Mul?pleprogramsonasinglecomputer

Page 4: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Yourcomputer(mul?plecores)

YourprogramYourprogram Yourprogram YourprogramYourprogram

Abatchsystemmanagingmul?pleprogramsonasinglecomputer

YourprogramYourprogram Yourprogram YourprogramYourprogram

onhold

Page 5: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Batchnode

YourprogramYourprogram Yourprogram YourprogramYourprogram

Abatchsystemmanagingmul?pleprogramsonmul?plecomputers

YourprogramYourprogram Yourprogram YourprogramYourprogram

onhold

Yourcomputer

BatchnodeBatchnode

Batchnode BatchnodeBatchnode

Batchmanager

Page 6: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

ThestandardsoFwareformanagingbatchsystemsinscien?ficcompu?ngisHTCondor(orjustCondor)

MainwebpagehKp://research.cs.wisc.edu/htcondor/

Quickstart

hKp://research.cs.wisc.edu/htcondor/quick-start.html

FullmanualhKp://research.cs.wisc.edu/htcondor/manual/v7.6/2_Users_Manual.html

•  WeuseanolderversionofCondorintheNevispar?cle-physics

systems.•  S?cktothe“vanilla”universe;the“standard”universewon’twork

forROOToranyotherpar?cle-physicssoFware(soyoudon’tneedcondor_compile).

Page 7: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Batchnode

YourprogramYourprogram Yourprogram YourprogramYourprogram

Condormanagingmul?pleprogramsonmul?plecomputerswithmul?plequeues

YourprogramYourprogram Yourprogram YourprogramYourprogram

onhold

Submitmachine

BatchnodeBatchnode

Batchnode BatchnodeBatchnode

Condormaster

Condorpool

Page 8: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Batchnode

YourprogramYourprogram Yourprogram YourprogramYourprogram

Condorwillhaltaqueueinfavorofaninterac?veprogram

YourprogramYourprogram Yourprogram YourprogramYourprogram

onhold

Submitmachine

BatchnodeBatchnode

Batchnode BatchnodeBatchnode

Condormaster

Condorpool

Someoneloggedin!

Page 9: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Batchnode

YourprogramYourprogram Yourprogram YourprogramYourprogram

Condormanagingmul?pleprogramsonmul?plecomputerswithmul?pleconfigura?ons

YourprogramYourprogram Yourprogram YourprogramYourprogram

onhold

Submitmachine

BatchnodeBatchnode

Batchnode BatchnodeBatchnode

Condormaster

Condorpool

Page 10: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Batchnode

YourprogramYourprogram Yourprogram YourprogramYourprogram

Condoruses“ClassAds”tomatchyourrequirementswithwhateachnodeoffers

YourprogramYourprogram Yourprogram YourprogramYourprogram

onhold

Submitmachine

BatchnodeBatchnode

Batchnode BatchnodeBatchnode

Condormaster

Condorpool

Yourrequirements(jobClassAd)

Whatanodeoffers(machineClassAd)

Page 11: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

ResourcePlanning•  Condorcan’tdoeverythingforyou.•  Thinkaboutinputfiles(includingprograms)andoutputfilesandhow

they’llbeaccessed.•  Thinkaboutdiskspace.“df -h”and“du -shx *”canhelp.•  Funfact:Thepar?cle-physicsCondorpoolscan’tseeyourhomedirectory!•  Moral:Letcondortransferyourfiles…whenpossible.

Whenyoucan’tletcondortransferyourfiles,herearedisk-sharingmethodsoutsideofcondor:•  NFS–usedatNevis•  CVMFS–FermilabandCERN•  Grid,BlueArc–onlyusedatFermilab•  AFS–obsolete,s?llusedatCERN

Page 12: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

ResourcePlanning

Yourserver

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

/home

Whatwedon’tdo

•  Condorcan’tdoeverythingforyou.•  Thinkaboutinputfiles(includingprograms)andoutputfilesandhow

they’llbeaccessed.•  Thinkaboutdiskspace.“df -h”and“du -shx *”canhelp.•  Funfact:Thepar?cle-physicsCondorpoolscan’tseeyourhomedirectory!•  Moral:Letcondortransferyourfiles…whenpossible.

Page 13: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

ResourcePlanning•  Condorcan’tdoeverythingforyou.•  Thinkaboutinputfiles(includingprograms)andoutputfilesandhow

they’llbeaccessed.•  Thinkaboutdiskspace.“df -h”and“du -shx *”canhelp.•  Funfact:Thepar?cle-physicsCondorpoolscan’tseeyourhomedirectory!•  Moral:Letcondortransferyourfiles…whenpossible.

Yourserver

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

node

/home

Fileserver

/share

/data

Whatwedo

Page 14: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Computer Systems at NevisLinux Cluster

hypa?aadministra?on,NIS

kolyaATLASfranklin

Mail

karthurATLAS

hogwartsstaff

Administra?veservers Workgroup/Loginservers

Clientsand

x-terms

Clientsand

x-terms

Worksta?onsbatchnodesstudentboxes

shangDOE

annexoff-sitebackup&mail

adawebserver

sullivanmailing-listserver

<hKp://www.nevis.columbia.edu/linux/><hKp://www.nevis.columbia.edu/linux/cluster-names.html>

tehanuVERITAS

shelleybackupserver

xenia

tangoSMB

hermesDNS,batch

virtualmachines

houstonNeutrino

Fileservers

xenia2

vetchgedserret

amsterdam

bleeker

westside

riverside

Page 15: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

node05

Bringingthejobtothedata

Submitmachine

node06node04

node02 node03node01

Condormaster

requirements = (machine = node04.nevis.columbia.edu)

bigfile1.root bigfile2.root bigfile3.root

bigfile4.root bigfile5.root bigfile6.root

Somewrapperscript

Page 16: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Final?ps•  Splitupyourtasksoeachcondorjobtakes20-60minutes

•  Ifyourjobmustbepreempted,itwillhavetorunfromthebeginningonthesamemachinethatcancelledthejob

•  Testyourjobwithoneprocessbeforesubmiqngitfor10,000processes!

Page 17: An overview of batch processing - Nevis Laboratories · PDF fileAn overview of batch processing 2-June-2016 Your computer Your program One-on-one ... hKp://research.cs.wisc.edu/htcondor/quick-start.html

Resources

MainwebpagehKp://research.cs.wisc.edu/htcondor/

QuickstarthKp://research.cs.wisc.edu/htcondor/quick-start.html

FullmanualhKp://research.cs.wisc.edu/htcondor/manual/v7.6/2_Users_Manual.html

Nevispar?cle-physicscondorguide

hKps://twiki.nevis.columbia.edu/twiki/bin/view/Nevis/Condor

BasicCondor@NevistutorialhKp://www.nevis.columbia.edu/~seligman/root-class/


HKR / HKA / HKP /HKS - Amazon S3 · SS-GHKR 9/13 Supersedes 6/13 HKR / HKA / HKP /HKS ElEctRic HEAt KitS foR AiR HAndlERS And PAcKAgEd ElEctRic UnitS *Complete warranty details available

HKR / HKA / HKP /HKS - Amazon S3 · SS-GHKR 9/13 Supersedes 6/13 HKR / HKA / HKP /HKS ElEctRic HEAt KitS foR AiR HAndlERS And PAcKAgEd ElEctRic UnitS *Complete warranty details available

Documents
HKR / HKA / HKP/ HKS · SS-AHKR 4/13 A m cpora ela g c, l.P., H, t. Supersedes 10/12 dRAft HKR / HKA / HKP/ HKS ElEctRic HEAt KitS foR AiR HAndlERS And PAcKAgEd ElEctRic UnitS …

HKR / HKA / HKP/ HKS · SS-AHKR 4/13 A m cpora ela g c, l.P., H, t. Supersedes 10/12 dRAft HKR / HKA / HKP/ HKS ElEctRic HEAt KitS foR AiR HAndlERS And PAcKAgEd ElEctRic UnitS …

Documents
Concrete Batch CONCRETE BATCH PLANTSPlants

Concrete Batch CONCRETE BATCH PLANTSPlants

Documents
Batch, Batch, Batch: What Does It Really Mean? Matthias Wloka

Batch, Batch, Batch: What Does It Really Mean? Matthias Wloka

Documents
BATCH INFORMATION COCKPIT - batch history

BATCH INFORMATION COCKPIT - batch history

Documents
 · Twin-Wall Sheets (HKP) Planchas con cámaras huecas (HKP) Lastre alveolari (HKP) Key Leyenda Legenda Sheets Planchas Lastre HKP Corner Elements HKP esquineras Elementi angolari

 · Twin-Wall Sheets (HKP) Planchas con cámaras huecas (HKP) Lastre alveolari (HKP) Key Leyenda Legenda Sheets Planchas Lastre HKP Corner Elements HKP esquineras Elementi angolari

Documents
Deaths l952-1953 Shirley Louise Ames, Ply, res Warren, May ... · Apr 14, lb69, w, hkp, dau Alfred Lamonde and Adeline Guyette, Dr Feiner, buried Green ... Jan 2, 1876, w, hkp, dau

Deaths l952-1953 Shirley Louise Ames, Ply, res Warren, May ... · Apr 14, lb69, w, hkp, dau Alfred Lamonde and Adeline Guyette, Dr Feiner, buried Green ... Jan 2, 1876, w, hkp, dau

Documents
Batch and Fed-batch Fermentation

Batch and Fed-batch Fermentation

Documents
HKR / HKA / HKP /HKS

HKR / HKA / HKP /HKS

Documents
USING LIGHTWEIGHT CHECKPOINT/RECOVERY TO IMPROVE THE AVAILABILITY AND DESIGNABILITY …research.cs.wisc.edu/multifacet/theses/daniel_sorin_ph… ·  · 2002-07-24USING LIGHTWEIGHT

USING LIGHTWEIGHT CHECKPOINT/RECOVERY TO IMPROVE THE AVAILABILITY AND DESIGNABILITY …research.cs.wisc.edu/multifacet/theses/daniel_sorin_ph… ·  · 2002-07-24USING LIGHTWEIGHT

Documents
Rerun: Exploiting Episodes for Lightweight memory Race …research.cs.wisc.edu/multifacet/papers/isca08_rerun.pdf · 2.1. Motivating Example ... (EQ 1) 2.3. Establishing Episodes

Rerun: Exploiting Episodes for Lightweight memory Race …research.cs.wisc.edu/multifacet/papers/isca08_rerun.pdf · 2.1. Motivating Example ... (EQ 1) 2.3. Establishing Episodes

Documents
GPU Computing Pipeline Inefficiencies and Optimization ...research.cs.wisc.edu/multifacet/papers/iiswc15_pc_characterize.pdf · GPU Computing Pipeline Inefficiencies and Optimization

GPU Computing Pipeline Inefficiencies and Optimization ...research.cs.wisc.edu/multifacet/papers/iiswc15_pc_characterize.pdf · GPU Computing Pipeline Inefficiencies and Optimization

Documents
Batch to Batch shade variation

Batch to Batch shade variation

Engineering
Docker and HTCondor - University of …research.cs.wisc.edu/.../presentations/ThainG_Docker.pdf · All docker commands are bound into the “docker” executable “run” command

Docker and HTCondor - University of …research.cs.wisc.edu/.../presentations/ThainG_Docker.pdf · All docker commands are bound into the “docker” executable “run” command

Documents
HKP Senior Magazine, 2016

HKP Senior Magazine, 2016

Documents
HKP Editor Guidelines and Publishing Tutorial.pdfHKP Editor Guidelines and Publishing Tutorial

HKP Editor Guidelines and Publishing Tutorial.pdfHKP Editor Guidelines and Publishing Tutorial

Documents
The Boys in Blue- A HKP Chief Inspector & A P.I. · 2019. 10. 30. · The Boys in Blue: A HKP Chief Inspector. & A P.I. Early Days 1909 - 1927 “Enough’s enough!” Jack growled

The Boys in Blue- A HKP Chief Inspector & A P.I. · 2019. 10. 30. · The Boys in Blue: A HKP Chief Inspector. & A P.I. Early Days 1909 - 1927 “Enough’s enough!” Jack growled

Documents
“Batch, Batch, Batch:” What Does It Really Mean?

“Batch, Batch, Batch:” What Does It Really Mean?

Documents
Batch Man - Batch Processing

Batch Man - Batch Processing

Documents
Batch script file Additional information. Objectives… What is a Batch File? Stopping a Runaway Batch File Creating a Batch File Editing a Batch

Batch script file Additional information. Objectives… What is a Batch File? Stopping a Runaway Batch File Creating a Batch File Editing a Batch

Documents
Batch to Batch Reproducibility - Nearchimicaextranet.nearchimica.it/admin/newspdf/batch-to-batch and creases.pdf · Batch-to-Batch Reproducibility • Use the same standard program

Batch to Batch Reproducibility - Nearchimicaextranet.nearchimica.it/admin/newspdf/batch-to-batch and creases.pdf · Batch-to-Batch Reproducibility • Use the same standard program

Documents
HKR / HKA / HKP / HKS - Daikinshumate.daikincomfort.com/media/pdfs/spec_sheets/SS-DHKR.pdf · HKR / HKA / HKP / HKS ELECTRIC HEAT KITS FOR AIR HANDLERS AND PACKAGED ELECTRIC UNITS

HKR / HKA / HKP / HKS - Daikinshumate.daikincomfort.com/media/pdfs/spec_sheets/SS-DHKR.pdf · HKR / HKA / HKP / HKS ELECTRIC HEAT KITS FOR AIR HANDLERS AND PACKAGED ELECTRIC UNITS

Documents
Batch Fed Batch

Batch Fed Batch

Documents
M18 HKP - kesko-onninen-pim-resources-production.s3 ...kesko-onninen-pim-resources-production.s3-website-eu-west-1.amazonaws.com/...M18 HKP Original instructions Originalbetriebsanleitung

M18 HKP - kesko-onninen-pim-resources-production.s3 ...kesko-onninen-pim-resources-production.s3-website-eu-west-1.amazonaws.com/...M18 HKP Original instructions Originalbetriebsanleitung

Documents
HKP 3C (&B) Painting instructions Camouflage scheme€¦ · HKP 3C (&B) Painting instructions Camouflage scheme HKP 3C, 03115, Z55 - AF1 Boden. ... Close Tamiya: Close Humbrol:

HKP 3C (&B) Painting instructions Camouflage scheme€¦ · HKP 3C (&B) Painting instructions Camouflage scheme HKP 3C, 03115, Z55 - AF1 Boden. ... Close Tamiya: Close Humbrol:

Documents
HTCondor and Python - research.cs.wisc.edu

HTCondor and Python - research.cs.wisc.edu

Documents
IN A ENVIRONNENT WALTER - University of …research.cs.wisc.edu/htcondor/doc/mutka-dissertation.pdf · WORKSTATION ENVIRONNENT ... Global Reservation Table ... opportunities for people

IN A ENVIRONNENT WALTER - University of …research.cs.wisc.edu/htcondor/doc/mutka-dissertation.pdf · WORKSTATION ENVIRONNENT ... Global Reservation Table ... opportunities for people

Documents
Hkp Presentation

Hkp Presentation

Technology
HKP 2015 Wedding Guide

HKP 2015 Wedding Guide

Documents
Batch to Batch and Creases

Batch to Batch and Creases

Documents