Upload
quentin-fennessy
View
219
Download
3
Embed Size (px)
Citation preview
Recursive Computing
AMD on AMD…
Quentin Fennessy
Oct 2009 2
Who am I?
•Quentin Fennessy–Worked at AMD (Advanced Micro Devices) for 10 years–Compute Clusters: 10 years with clustered computing–Unix: 20+ years in various industries (telecomm, automation,
semiconductors)–BA in Computer Science from University of Massachusetts–Manager for Core Services of Global Engineering IT
3
Recursive Computing--What?
•Definition: See RECURSIVE
Oct 2009
4
Goal of AMD Compute Clusters
•Develop, test, revise and completemicroprocessor designs
•Do it efficiently–time-wise–$$-wise–people-wise
•Support concurrent design projects–5 or 6 at any given time
Oct 2009
5
High Level Attainable Goals
•Plan to meet your business needs
•Understand the technical possibilities now–work with your vendors–hire and grow a great staff
•Understand the technical possibilities for the future
•Be flexible to accommodate changing business needs and technical possibilities
Oct 2009
6
Compute Clusters at AMD
• Installed at each AMD design center (Austin x 2, Fort Collins, Sunnyvale, Boxborough, Dresden, Bangalore, )
• cluster size ranges from 200 to 10K+ cpus
• 98+% compute servers are AMD Opteron™ and AMD Athlon™ MP processor-based
• AMD Opteron and AMD Athlon MP processor-based desktops are also used as compute resources
• AMD processor-based systems run 64bit and 32bit Linux (Red Hat Enterprise 3 and 4)
Oct 2009
7
History of AMD Clusters
c 1998: AMD K6 processors, Linux, ~400 systems
• c 2000: AMD Athlon™ processors, Linux, ~1K systems• c 2001: More AMD Athlon processors, Linux, ~2K systems• c 2002: More AMD Athlon processors, Linux, ~3K systems
•c 2003: AMD Opteron™ processors, Linux, ~4.5K systems
•c 2004: More AMD Opteron processors, Linux, ~6K systems
•c 2005: Dual Core AMD Opteron processors, Linux, ~7K systems, ~15K+ cpus
•c 2006: ~8K systems, ~23K+ cpus
Oct 2009
8
OS Transitions for AMD Clusters
•HP-UX → Solaris, painful as it was our first transition
•Solaris → HP-UX, painful because we forgot our first
•Solaris+HP-UX → 32 bit Linux, easier
•32bit Linux → 64bit Linux, easy! because of compatibility
•What makes an OS transition hard?– implicit assumption that we will always use OS Foo-X– the imagination and creativity of OS vendors
•What makes an OS transition easy?–never assume anything will be the same next year–avoid buying into OS-specific infrastructure tools
Oct 2009
9
HW Transitions for AMD Clusters
• HP → Sun, easy (Sun does a great job maintaining systems)
• Sun → HP, easy (HP does a great job maintaining systems)
• Sun, HP → AMD Athlon™ processor-based systems (32bit), HARD (Linux device issues, no system integration)
• AMD Athlon™ MP (32bit) → AMD Opteron™ processors, easy, it just worked
• Transition → Sun and HP AMD Opteron™ processor-based systems (easy, fast, very nice systems)
Oct 2009
10
Historic Bottlenecks
•Every system, every cluster has a bottleneck—the slowest part of the system
•Goal—provide a balanced cluster
•Bottleneck Candidates–Fileservers–Network–Application licenses–Cluster manager systems
Oct 2009
11
Data Storage
•2PB+ of network attached storage in 46 Netapp filers
•>50% are Opteron-based Netapp filers
•Typically Quad-GbE attached, with 10GbE testing in 1H07
•Fibre-channel and ATA disks, RAID-DP and RAID4 volumes
•Challenge 1: a few hundred jobs can overwhelm a filer...either with raw I/O or relentless meta-data requests
•Challenge 2: moving data between filers is a division-visible change and makes fileserver upgrades difficult
•Goal: a fileserver that can add cpu and network capacity as easily as we add disk capacity
Oct 2009
12
Networking
•We use commodity networking from Nortel, Cisco (100baseT, GbE)
•Post-2003 compute servers are connected via GbE switches
•Older systems are connected via 100baseT
•We use VLANs for partitioning, routing to connect to the rest of AMD
•Our network provides redundant paths and management components, except for the last mile to each compute server.
Oct 2009
13
Cluster Management via LSF
•Currently—excellent performance for job submission, dispatch and status updates
•Our LSF job scheduler systems (for clusters with 10k cpus) are available for under $25,000 from tier 1 vendors.
•We have a good upgrade path
•Challenge: Match Resource Allocation to Business Needs
Oct 2009
14
Best Practices
•Use revision control tools (RCS, Subversion, CVS, etc)
•Use OS-independent and vendor-independent tools
•Strive for uniformity in h/w and system s/w
•Reserve sample systems for testing and integration
•Plan for the failure of systems
•Use collaborative tools for communication, planning and documentation (we use TWiki, irc, audio and video conferencing)
Oct 2009
15
Our Fastest Systems are…
•AMD Opteron™ processor-based systems of course…
•Some optimizations:
•Fully populate memory DIMM slots for max bandwidth (typ 4 dimms/socket)
•Use ECC/Chipkill (x4) memory to correct up to 4bit errors
•Enable memory interleaving in the BIOS
•Use a 64bit NUMA-aware OS (Red Hat has done well for us)
•Recompile your applications in 64bit mode
Oct 2009
16
System Types in the Cluster
•AMD Opteron™ processor–64bit, Linux, 2p→8p, 2GB→128GB–Most with single ATA disk, some w/SCSI–Most with single power supply–Gigabit Ethernet, single connection
•AMD Athlon™ MP processor–32bit, Linux, 1p→2p, 1GB→4GB–ATA disk–Single power supply–100Mb Ethernet, single connection
• Other Unix Systems– 64bit, 2p-8p, 2GB→28GB
Oct 2009
17
System Types in the Cluster
Cluster Capacity by Throughput
64%
35%
1%0%
20%
40%
60%
80%
100%
AMD Opteron™64bit
AMD Athlon™32bit
Other
Oct 2009
18
Show Me Some Numbers
CPU and System Totals
0
2,000
4,000
6,000
8,000
10,000
ASDC ANDC SVDC BDC IDC Total
Total CPUsOpteron System TotalAthlon System Total
Oct 2009
19
More Numbers
Total Capacity (Megabytes) per cluster
0
10
20
30
40
50
60
70
ASDC ANDC SVDC BDC IDC Total
Mill
ions
Total RAM (MB)Total Swap (MB)
Oct 2009
20
Internal Benchmark Comparison
K9mark for System Types
42
114 115
259
356
532
0
100
200
300
400
500
600
K7 Clas
sic 1P
K7 MP 2
P
K7 Bart
on 1P
Opteron
2P
Opteron
4P
Opteron
8P
Processor type and qty
k9ma
rk sc
ore
K9mark
Oct 2009
21
Large Cluster Throughput, for Texas2(Year to Date)
Utilization 95%
LSF Jobs/Day 40K – 100K
Average Job turnaround 8-9 hours
Average CPU seconds/job 10,728
Oct 2009
22
Large Cluster Throughput, for Texas2
Max Job Throughput/hour4250 (was 2500 last
year)
Jobs/day (peak) 120K+
Jobs/day (average) 50K
Oct 2009
23
Crunchy LSF Details
•Job Scheduler for Texas2 Cluster (3900 systems, 11k cpus)–Hewlett Packard DL585
• 4 Single Core Opteron 854 (2.8Ghz)• 16GB RAM• 64bit Redhat Enterprise Linux 4, Update 1
•System Load for Job Scheduler–Typically 40% busy–10.5MB/sec network traffic–Manages 3900 compute nodes
• Queues jobs• Monitors system load• Monitors running jobs
Oct 2009
24
Job Types
•Architecture – what should it do?
•Functional Verification – will it work?
•Circuit Analysis – transistors, library characterization
•Implementation – put the pieces together
•Physical Verification – timing, capacitance
•Tapeout – send it to the fab
Oct 2009
25
Resource Usage by Job Types
Approximate Resource Usage
0%
20%
40%
60%
80%
100%
Func
tiona
l Ver
ificat
ion
Circu
it An
alysis
Arch
itectu
re
Phys
ical V
erific
ation
Other
Tape
out
Oct 2009
26
Architecture
•Highest level description of the cpu–functional units (FP, Int, cache)–bus connections (number, type)–cache design (size, policy, coherence)
•Architectural Verification – up to multi-GB processes
•Job pattern – 100s or 1000s of jobs run overnight for experiments
•Fundamental early phase of each project
•Re-done during design to validate
Oct 2009
27
Functional Verification
•CPU-intensive, relatively low memory
•Huge quantities of similar jobs
•RTL 1-2GB processes
•Gates 2-8GB processes
Oct 2009
28
Circuit Analysis
•Many small jobs, some large jobs
•Peaky pattern of compute requirements
•Compute needs can multiply quickly when manufacturing processes change
•Challenge: too-short jobs can be scheduled inefficiently
Oct 2009
29
Physical Verification
•Physical Design & Routing
•Extraction of Electrical Characteristics including Timing and Capacitance
•Memory intensive + compute intensive
Oct 2009
30
Tapeout – next stop, the FAB
•Compute intensive, one task may use >400 systems
•Memory intensive, approaching 128GB
•Longest-running jobs–Fortunately clustered AMD Opteron™ processor-based systems
have reduced our longest job run-time to less than one week
•Last engineering step before manufacturing –Time-to-market critical
Oct 2009
31
Challenges
•Growth –Cluster size = X today, 2X in 18 months?
•Manageability–Sysadmin/system ratio – can we stay the same or improve?–Since 1999 the ratio has improved 3X
•Linux–Improve quality–Manage the rapid rate of change
•Scalability–What decisions today will help us grow?
Oct 2009
32
Linux Challenges
•Linux Progression–Redhat 6.x–Redhat 7.x–Suse Linux Enterprise Server 8.x–Redhat Fedora Core 1–Redhat Enterprise Linux 3.x–Redhat Enterprise Linux 4.x
•Additional efforts include:–Revision Control with CVS–System installation with Kickstart–Configuration Management with cfengine, yum
Oct 2009
33
Actual Train Wrecks
• Power Loss for one or multiple buildings– Breakers, City cable cuts, human
error
• Cooling loss• Cooling loss + floods!• NFS I/O overloads• Network failures
– hardware– human error– software
• Job Scheduler Overload– 100K pending jobs– Relentless job status queries
Oct 2009
34
System Installation Progression
•1 Manual installation, no updates
•2 Automated installation, no updates
•3 Automated installation, manual updates
•4 Automated installation, automated updates
•We are currently at level 3, approaching level 4–Kickstart for installation–cfengine for all localization–yum for package management
Oct 2009
35
Tools for Clusters
•We use LSF from Platform Computing on our clusters
•Locally written tools are easier sed than done
•Freely available software keeps everything working
•Perl, CVS, kickstart, cfengine, yum
•ethereal, tcpdump, ping, mtr, …
•ssh, rsync, clsh, syslog-ng, …
•Apache, TWiki
•Mysql
•RT
Oct 2009
36
Out of the Box?
• Can a Compute Cluster be an “Out of the Box” experience?– (will it just work?)
• Not for large clusters
• Why?
• These factors – Applications – Operating Systems– System Hardware– Network Hardware– Network Configuration– Physical Infrastructure (space,
power, cooling)
Oct 2009
37
Recursive Computing? What?
Our clusters are used to to design faster processors and better systems for our customers – processors for your clusters and our own.
•1999: AMD(AMD K6, HP-PA,SPARC) → AMD K7
•2000: AMD(AMD K6, SPARC) → AMD K7
•2001: AMD(AMD K7, AMD K6, SPARC) → AMD K7, K8
•2002: AMD(AMD K7, SPARC) → AMD K8
•2003: AMD(AMD K7, AMD K8) → AMD K8+++
•2004: AMD(AMD K7, AMD K8) → AMD K8+++
•2005: AMD(AMD K7, AMD K8(dual-core)) → AMD K8+++
Oct 2009
38
Trademark Attribution
AMD, the AMD Arrow Logo, AMD Athlon, AMD Opteron and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this presentation are for identification purposes only and may be trademarks of their respective companies.
Oct 2009