Current SituationStable SeriesCurrent: Condor ver 6.8.4. (Feb 5th)Development SeriesCurrent : Condor ver 6.9.2. (April 10th)
Major v6.9 ChangesVirtual Machine Universe See Jaeyoungs talkQuill 2.0See Jeff and Eriks talkScalability ImprovementsGCB ImprovementsPrivilege Separation
Team Scalability!Dan Two-Faces Bradley (half CMS, half Condor), Papa Todd Tannenbaum, and Uncle Greg Thain
The new condor_q GUI?
How we lookedIntuition / Heated hallway discussionWageringExamination of the log filesSome Toolscallgrind and kcachegrindOr Compuware DevPartner on Win32gprofstracetcpdump
What we foundInappropriate Data Structures
Implementation issuesBad hash function, table sizesBuffer copy, copy, copy, copy, copy, copy
I am sooo embarrassed!schedd
What we didMore non-blocking I/O in critical areas to eliminate timeouts/embracesCleansed a bunch o embarrassmentsReuse of a claimCarefully cache candidate jobsUse autoclustersAuto-adapting parameters based on workloadOld way: do some bookeeping every 5 secondsNew way: spend 5% of time doing bookeeping
Creepy fork()Got This: 0.125sec/fork!Wanted this:
Lets fix this one, please.(patch circa 2003)
And last but not least, thanks you toBuffered writes toschedd transaction log(i.e. Francesco Prelz!)
Numerous scalabilityimprovements to Condor-C
Lets see some results to date
SWEEEEET!!Todd, lets see thatgraph again!!!
condor_q performance(sans Quill)Already donebatch sending of ads (eliminate latency, let tcp window warm up)projection of attributes (note : now condor_q l more expensive than "condor_q -format").Still Todo ?i/o in another thread href protocol on the wirecaching of parsed expressions -- classads are very redundantsame improvements into condor_status
Collector PerformanceFixing Dropped updatesincreased incoming buffer sizes; problems caused by synchronization via condor_reconfig -all etc. Also, with Winsock, UDP sendto() is always successful. (!)Added DNS caching for unauthenticated connections in Condor. profiling was important; we had no suspicion this was the problem. Collector was spending 20% of its time in the DNS resolver library!Todo: Ian Alderman discovered non-blocking communication assumptions violated by authentication methods that require round-trips.
Negotiation Performancev6.8 -> automatic significant attributes.v6.9 -> resource request adsSimple explanation: Resource request ad == a count plus all significant attributes.Inserted into a schedd submitter ad.Give me 400 resources like this, and 200 resources like that, etc.Matchmaking algorithms remains the same, just how it learns about jobs changes.Disabled by default.Possibilities, possibilitiesMore robust against unresponsive scheddsNo startd Rank preemption?Others?
Impact of negotiation changesUW CS Pool Negotiation cycle times:2583 seconds baselineDropped to 366 secs w/ autoclusteringAdd matchlist caching, dropped to 223 secsAdd resource request ads, drops again to 129 seconds.CM memory footprint increased by 80k.
Team GCB!Derek Mr. Follow-CVS-rules-or-ELSE Wright, Alan Ask me about Social Security #s DeSmet, and Jaime The GridMan Frey
Improved Scalability: Only use the broker if required!Local Host Optimizations (6.9.1) Bypass GCB if two daemons are talking on the same hostLocal Network Optimizations (6.9.3)Two hosts on the same private net bypass the brokerEvery network is assigned a unique network nameDaemons advertise (a) public accessible IP; (b) real IP; (c) network name.Names match ? use real ip : use public IP.Improved RobustnessBroker dies -> master finds another broker and restarts.When master starts up, it pings a list o brokers and randomly chooses from those that respond.Bug fixesImproved Logging now they are helpful and sane.
Team Privilege Separation!Cousin Greg Quinn, Pete Psilord Keller, and Zach When the grid relaxes, its Zmiller time Miller
Condors Privilege SeparationApply principle of least privilege to CondorNo more root / super-user privilege requiredCurrently completed on execute side (v6.9.3), almost on submit sideUse glexec or Condors own sudoCan still run the old way if you wantRefer to Greg Quinns Talk
Minor v6.9 changesLeases added to COD.Simple best-fit algorithm added to dedicated scheduler.Can reference resource usage and quota information in preemption policy.condor_config_val dump [-v]Chirp improvementsJobs can write messages into the user logCan use proc 0 ClassAd as a scratch padCondor shutdown via expressions External AwarenessPlug: Talk w/ Joe Meehan @ the Research BOF!
Minor v6.9 changes, cont.More types of jobs can survive across a shutdown/crash of submit machineSuch as jobs that stream stdout/err.Users job log changes.Can have a centralized job log file.Get values of any job ad attribute in log.
Hoping for v6.9, but no promisesRich Wolskis prediction workSupport for VOMS attributesUpdate condor binaries on job boundariesSecure install by defaultVia pool password?
So 2-3 more developer releases, then new stable series Condor 7.0 ( or Condor Vista? )
And the next developer series after v6.9 ?
Terms of LicenseAny and all dates in these slides are relative from a date hereby unspecified in the event of a likely situation involving a frequent condition. Viewing, use, reproduction, display, modification and redistribution of these slides, with or without modification, in source and binary forms, is permitted only after a deposit by said user into PayPal accounts registered to Todd Tannenbaum .
Beyond v6.9For the next year, so far we have identified the following intial focus areas:Continue our work w/ Storage Management (MOPS)Refer to Dan Frasers talkContinue our work w/ Virtual MachinesRefer to Jaeyoung Yoons talkScheduling WorkStartd Enhancements
Scheduling in Condor TodayDistributed OwnershipSettings reflect 3 separate viewpoints:Pool manager, Resource Owner, Job SubmitterCMscheddCMscheddscheddscheddschedd
But some sites want to use Condor like this:Just one submission point (schedd)All resources owned by one entityWe can do better for these sites.Policy configurations are complicated.Some useful policies not present because they are hard to do a wide-area distributed system.Today the dedicated scheduler only supports FIFO and a naive Best Fit algorithms.schedd
So what to do?Give the schedd more scheduling options.Examples: why cant the schedd do priority preemption without the matchmakers help? Or move jobs from slow to fast claimed resources ?Pluggable scheduler routines.schedd
StartD Enhancements: New sources of workCondor-G enabled the SchedD to talk to many different scheduling systems to run jobsNow the StartD will be able to talk to different managers to fetch jobs and work.StartD configured to be claimed at boot so that you dont need the overhead of match-making.Dont necessarily need a SchedD -- fetch jobs (work units) from other systems (DB of jobs, etc).
StartD Enhancements: Dynamic slotsCurrently, resource slots are static -- some changes require restarting the StartD.Would like to add dynamic computing slots:Dual-core machines are ubiquitous.1 or 2 gigs of RAM is commodity.Instead of statically partitioning the RAM (1 gig for each slot), itd be nice to advertise 2 CPUs, 2 gigs of RAM, and once 1 CPU is claimed for 1/2 gig, to advertise 1 CPU, 1.5 gigs of RAM for the other slot.
StartD Enhancements: Dynamic slots (contd)Can also be used to simplify complex policies:Currently checkpoint to swap implemented with static slots and pre-configured policy.Would like to just dynamically allocate new slots, and make it easier to have global, slot-wide policy expressions, not just per-slot policies.Could have implications for COD, GlideIn and other uses of the StartDGlideIn under an existing Condor pool might just allocate a new slot on the parent StartD, instead of spawning a whole new StartD under the parent StartDCOD claims could allocate new dynamic slots, too