Todd TannenbaumComputer Sciences DepartmentUniversity of Wisconsin-Madison
[email protected]://www.cs.wisc.edu/condor
Condor RoadMapCondor Week 2007
2
Current Situation› Stable Series
Current: Condor ver 6.8.4. (Feb 5th)› Development Series
Current : Condor ver 6.9.2. (April 10th)
3
Major v6.9 Changes› Virtual Machine Universe
See Jaeyoung’s talk› Quill 2.0
See Jeff and Erik’s talk› Scalability Improvements› GCB Improvements› Privilege Separation
4
Team Scalability!
Dan “Two-Faces” Bradley (half CMS, half Condor), “Papa” Todd Tannenbaum, and “Uncle”
Greg Thain
The new condor_q GUI?
7
How we looked…› Intuition / Heated hallway discussion› Wagering› Examination of the log files› Some Tools
callgrind and kcachegrind• Or Compuware DevPartner on Win32
gprof strace tcpdump
8
What we found…› Inappropriate Data Structures
› Deadly embraces
› Implementation issues Bad hash function, table sizes Buffer copy, copy, copy, copy, copy,
copy
“I am sooo embarrassed!”
scheddschedd
Bill and Monica
9
What we did…› More non-blocking I/O in critical areas to
eliminate timeouts/embraces› Cleansed a bunch o embarrassments› Reuse of a claim
Carefully cache candidate jobs Use “autoclusters”
› Auto-adapting parameters based on workload Old way: “do some bookeeping every 5
seconds” New way: “spend 5% of time doing bookeeping”
10
Creepy fork()Got This: 0.125sec/fork! Wanted this:
11
Lets fix this one, please.(patch circa 2003)
12
And last but not least, thanks you to
Buffered writes toschedd transaction log
(i.e. Francesco Prelz!)
Numerous scalabilityimprovements to Condor-C
13
Let’s see some results to date
15
SWEEEEET!!Todd, lets see that
graph again!!!
21
condor_q performance(sans Quill)
› Already done batch sending of ads (eliminate latency, let
tcp window warm up) projection of attributes (note : now “condor_q
–l” more expensive than "condor_q -format").› Still Todo ?
i/o in another thread href protocol on the wire caching of parsed expressions -- classads are
very redundant same improvements into condor_status
23
Collector Performance› Fixing Dropped updates
increased incoming buffer sizes; problems caused by synchronization via condor_reconfig -all etc.
Also, with Winsock, UDP sendto() is always successful. (!)
› Added DNS caching for unauthenticated connections in Condor. profiling was important; we had no suspicion this was
the problem. Collector was spending 20% of its time in the DNS resolver library!
› Todo: Ian Alderman discovered non-blocking communication assumptions violated by authentication methods that require round-trips.
25
Negotiation Performance› v6.8 -> automatic “significant attributes”.
› v6.9 -> “resource request” ads Simple explanation: Resource request ad == a count
plus all significant attributes. Inserted into a schedd submitter ad. “Give me 400 resources like this, and 200 resources
like that, etc”.› Matchmaking algorithms remains the same,
just how it “learns” about jobs changes.› Disabled by default.› Possibilities, possibilities…
More robust against unresponsive schedds No startd Rank preemption? Others?
26
Impact of negotiation changes
› UW CS Pool – Negotiation cycle times: 2583 seconds baseline Dropped to 366 secs w/ autoclustering Add matchlist caching, dropped to 223 secs Add resource request ads, drops again to
129 seconds. CM memory footprint increased by 80k.
28
Team GCB!
Derek “Mr. Follow-CVS-rules-or-ELSE” Wright, Alan “Ask me about Social Security #s” DeSmet,
and Jaime “The GridMan” Frey
29
› Improved Scalability: Only use the broker if required! Local Host Optimizations (6.9.1)
• Bypass GCB if two daemons are talking on the same host Local Network Optimizations (6.9.3)
• Two hosts on the same private net bypass the broker• Every network is assigned a unique network name• Daemons advertise (a) public accessible IP; (b) real IP; (c)
network name.• Names match ? use real ip : use public IP.
› Improved Robustness Broker dies -> master finds another broker and restarts. When master starts up, it pings a list o brokers and
randomly chooses from those that respond. Bug fixes
› Improved Logging – now they are helpful and sane.
30
Team Privilege Separation!
“Cousin” Greg Quinn, Pete “Psilord” Keller, and Zach “When the grid relaxes, its Zmiller time” Miller
31
Condor’s Privilege Separation› Apply principle of
least privilege to Condor
› No more root / super-user privilege required
› Currently completed on execute side (v6.9.3), “almost” on submit side
› Use glexec or Condor’s own sudo
› Can still run the “old way” if you want
› Refer to Greg Quinn’s Talk
32
Minor v6.9 changes› Leases added to COD.› Simple best-fit algorithm added to dedicated
scheduler.› Can reference resource usage and quota
information in preemption policy.› condor_config_val –dump [-v]› Chirp improvements
Jobs can write messages into the user log Can use proc 0 ClassAd as a “scratch pad”
› Condor shutdown via expressions External Awareness Plug: Talk w/ Joe Meehan @ the Research BOF!
33
Minor v6.9 changes, cont.
› More types of jobs can survive across a shutdown/crash of submit machine Such as jobs that stream stdout/err.
› User’s job log changes. Can have a centralized job log file. Get values of any job ad attribute in log.
34
Hoping for v6.9, but no promises
› Rich Wolski’s prediction work
› Support for VOMS attributes
› Update condor binaries on job boundaries
› Secure install by default Via pool password?
35
So 2-3 more developer releases, then new stable
series Condor 7.0 (… or Condor Vista? … )
And the next developer series after v6.9 ?
Terms of LicenseAny and all dates in these slides are relative from a date hereby unspecified in the event of a likely situation involving a frequent condition. Viewing, use, reproduction, display, modification and redistribution of these slides, with or without modification, in source and binary forms, is permitted only after a deposit by said user into PayPal accounts registered to Todd Tannenbaum ….
37
Beyond v6.9› For the next year, so far we have
identified the following intial focus areas: Continue our work w/ Storage Management
(MOPS)• Refer to Dan Fraser’s talk
Continue our work w/ Virtual Machines• Refer to Jaeyoung Yoon’s talk
Scheduling Work Startd Enhancements
38
Scheduling in Condor Today
CM
scheddCMschedd
scheddschedd
schedd
startdstartdstartdstartdstartdstartdstartdstartdstartdstartd
› Distributed Ownership› Settings reflect 3 separate viewpoints:
Pool manager, Resource Owner, Job Submitter
39
But some sites want to use Condor like this:
schedd
startdstartdstartdstartdstartd
› Just one submission point (schedd)› All resources owned by one entity› We can do better for these sites.
Policy configurations are complicated. Some useful policies not present because they
are hard to do a wide-area distributed system. Today the dedicated “scheduler” only supports
FIFO and a naive Best Fit algorithms.
40
So what to do?
schedd
startdstartdstartdstartdstartd
› Give the schedd more scheduling options. Examples: why can’t the schedd do
priority preemption without the matchmakers help? Or move jobs from slow to fast claimed resources ?
› Pluggable scheduler routines.
41
StartD Enhancements: New sources of work
› Condor-G enabled the SchedD to talk to many different scheduling systems to run jobs…
› Now the StartD will be able to talk to different managers to fetch jobs and work.
› StartD configured to be “claimed at boot” so that you don’t need the overhead of match-making.
› Don’t necessarily need a SchedD -- fetch jobs (work units) from other systems (DB of jobs, etc).
42
StartD Enhancements: Dynamic slots
› Currently, resource slots are static -- some changes require restarting the StartD.
› Would like to add dynamic computing slots: Dual-core machines are ubiquitous. 1 or 2 gigs of RAM is “commodity”. Instead of statically partitioning the RAM (1 gig
for each slot), it’d be nice to advertise “2 CPUs, 2 gigs of RAM”, and once 1 CPU is claimed for 1/2 gig, to advertise “1 CPU, 1.5 gigs of RAM” for the other slot.
43
StartD Enhancements: Dynamic slots (cont’d)
› Can also be used to simplify complex policies: Currently “checkpoint to swap” implemented with static
slots and pre-configured policy. Would like to just dynamically allocate new slots, and
make it easier to have global, slot-wide policy expressions, not just per-slot policies.
› Could have implications for COD, GlideIn and other uses of the StartD… GlideIn under an existing Condor pool might just
allocate a new slot on the “parent” StartD, instead of spawning a whole new StartD under the parent StartD
COD claims could allocate new dynamic slots, too…
44
Thank you!
Recommended
