Upload
xkinanx
View
1.506
Download
11
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
1
Sun SPARC Enterprise T5140 and T5240Rack ServersTechnical Training Course
<Name><Job Title>
Sun Microsystems, Inc.
Sun Internal and Approved Partners Only 2
UltraSPARC T2: TrueSystem On a Chip
• Up to 8 cores @1.2GHz or 1.4GHz• Up to 64 threads per CPU • Up to 16 FB-DIMMs, 4 memory
controllers> Up to 64GB memory (4GB DIMMs)> 2.5x memory BW of UltraSPARC T1
• 8 x fully pipelined Floating Point units / core, 1 per core
• Dual 10Gbit Ethernet and PCI-E integrated onto chip
• 4MB L2 (8 banks) 16 way associative• Enhanced MAU/Security co-processor
per core> DES, 3DES, AES, RC4, SHA1,
SHA256, MD5, RSA to 2048 key, ECC,CRC32
• Advanced Power saving features• 65nm process technology
x8 @2.5GHz4GBytes/s bi-directional
Full Cross Bar
C0 C1 C2 C3 C4 C5 C6 C7FPU FPU FPU FPU FPU FPU FPU FPU
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$
FB DIMM FB DIMM FB DIMM FB DIMM
FB DIMM FB DIMM FB DIMM FB DIMM
PCI-ENIU(E-net+)
Sys I/FBuffer Switch Core
2x 10GE EthernetPower ~95W<1.5W/thread
MCU MCU MCU MCU
MAU MAUMAUMAUMAUMAUMAUMAU
DMA Sub-System
Sun Internal and Approved Partners Only 3
UltraSPARC T2 ProcessorL2 Data
Bank 0
SPARC
Core 0
SPARC
Core 1
SPARC
Core 5
SPARC
Core 4L2 Data
Bank 1
L2 Data
Bank 4
L2 Data
Bank 5
L2 Data
Bank 7
L2 Data
Bank 6
L2 Data
Bank 3
L2 Data
Bank 2
L2B0
L2B1
L2B2
L2B3
L2B5
L2B4
L2B6
L2B7
SPARC
Core 2
SPARC
Core 3
SPARC
Core 7
L2
TAG2
L2
TAG3
L2
TAG7
L2
TAG6
L2
TAG0
L2
TAG1
L2
TAG5
L2
TAG4MCU0
MCU1
MCU2
MCU3
DMU
PEU
RTX
RDP TDS
CCXSII
SIO
CCU
NC
U
EF
U
SPARC
Core 6
MACFSR
FSR
FSR
PSR ESR
For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4
SPC SPARCv9 CMT CoreCCX Core-Cache CrossbarL2$ Level 2 Cache BankCLC Coherence & Link ControllerMCU Memory Control UnitSIU System Interface UnitNCU Non-cacheable UnitMAC Media Access ControllerDMU Data Management UnitPEU PCI Express Unit
Sun Internal and Approved Partners Only 4
UltraSPARC T2 Plus: Multi-Socket US T2
• Per Socket> Up to 8 cores @1.2GHz or
1.4GHz> Up to 64 Threads per socket> 4MB L2$ 8 banks x 16 Way SA> Up to 8 FPU's, 1 per core> Up to 8 Crypto cores, 1 per core
> DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC,CRC32
> 2 memory Controllers, 16 DIMMS> 25Read+13WriteGB/s Memory BW> One x8 PCIe interface
• Per system> 2 sockets glue-less> 4 sockets using UST2 Plus XBR> 16 or 32 PCIe lanes on 2 or 4
Socket systems• 65nm process technology
x8 @2.5GHz4GBytes/s
bi-directional
Full Cross Bar
Power ~105W<1.5W/thread
CO
FPUMAU
C1
FPUMAU
C2
FPUMAU
C3
FPUMAU
C4
FPUMAU
C5
FPUMAU
C6
FPUMAU
C7
FPUMAU
L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$CU CU CU CU
MCU MCU
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
FB-DIMM
4 x Coherency
Links
System I/F PCI-Express
SSI Bus
Each Coherency Link 6.4GBytes/s per direction, 12.8GB/s total4 x Coherency links give Snoop bandwidth of 51.2GB/s
Sun Internal and Approved Partners Only 5
UltraSPARC T2 Plus ProcessorSPC SPARCv9 CMT CoreCCX Core-Cache CrossbarL2$ Level 2 Cache BankCLC Coherence & Link ControllerMCU Memory Control UnitSIU System Interface UnitNCU Non-cacheable UnitNCX Non-cacheable CrossbarDMU Data Management UnitPEU PCI Express Unit
Sun Internal and Approved Partners Only 6
CMT Processor ComparisonUltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus
Technology 90nm 65nm 65nm# of Cores 8 8 8# of Threads 32 64 64# of Sockets 1 1 1 – 4
1 – 1.4 1.2 – 1.4 1.2 – 1.4Execution pipelines per core 1 2 2Instructions per cycle per pipe 1 1 1FPUs per chip 1 8 8I$ size (per core) 16KB 16KB 16KBD$ size (per core) 8KB 8KB 8KBL2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-WayMemory Type DDR2 FBDIMM-1 FBDIMM-1Memory Bandwidth (GB/s) 21.3
- -IO Bus JBUS x8 PCIe x8 PCIeIO Bandwidth (GB/s) 2.5 2 + 2 2 + 2On-chip Networking - 2 x 10GE -Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5
Power (Watts, Watts/thread) 72, 2.25 95, 1.5
Core Frequency (Ghz)
42READ + 21WRITE 25READ + 13WRITE
Coherence Bandwdth (GB/s) 4 x (6.4READ+6.4WRITE)
105, 1.6 *
Sun Internal and Approved Partners Only 7
UltraSPARC T2 Plus Cores, Crossbar, L2$(8 cores, 64 threads, 4MB L2$)
CoherenceUnit
CoherenceUnit
CoherenceUnit
CoherenceUnit
Memory Controller Memory Controller
PCI-Express
UltraSPARC T2 Plus Cores, Crossbar, L2$(8 cores, 64 threads, 4MB L2$)
CoherenceUnit
CoherenceUnit
CoherenceUnit
CoherenceUnit
Memory Controller Memory Controller
PCI-Express
UltraSPARC T2 Plus 2-Socket System
System IO (Network, Disk, etc.)
NCX NCU, DMU NCX NCU, DMU
Dual ChannelFBDIMM
Dual ChannelFBDIMM
Dual ChannelFBDIMM
Dual ChannelFBDIMM
Coherency Links
Sun Internal and Approved Partners Only 8
UltraSPARC T2/Plus CPU Core Block Diagram
EXU1
IFU
LSUSPU
TLU
MMU/HWTW
FGU
Gasket
xbar/L2
EXU0
• IFU – Instruction Fetch Unit> 16 KB I$, 32B lines, 8-way SA> 64-entry fully-associative ITLB
• EXU0/1 – Integer Execution Units> 4 threads share each unit> Executes one integer
instruction/cycle• LSU – Load/Store Unit
> 8KB D$, 16B lines, 4-way SA> 128-entry fully-associative DTLB
• FGU – Floating/Graphics Unit• SPU – Stream Processing Unit
> Cryptographic acceleration• TLU – Trap Logic Unit
> Updates machine state, handles exceptions and interrupts
• MMU – Memory Management Unit> Hardware tablewalk (HWTW)
– TLB miss handled by hardware > 8KB, 64KB, 4MB, 256MB pages
Sun Internal and Approved Partners Only 9
●
●
Data
• Each Core has its own Floating Point Unit. This removes completely the 2% FP restriction of T1.
• Only a 6 cycle penalty to the FPU> T1 required upwards of 28 cycles
• Fully-pipelined (except divide/sqrt)> Divide/sqrt in parallel with add or multiply operations of other
threads• Full VIS 2.0 implementation in hardware, no emulation• FPU performs integer multiply, divide, population count• Multiplier incorporates enhancements for offloading modular
arithmetic operations to the Crypto Unit
Floating Point Unit
Sun Internal and Approved Partners Only 10
●
●
Data
FB-DIMM Memory Interface
Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 PlusRequire Pins 1000 440 208Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socketMemory Capacity up to 16x DIMM slots
Memory Controllers 4 4 2Memory Latency 96ns 126ns 126nsDIMM speeds 400MHz 667MHz 667MHzDIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM
Up to 64x DIMM slotsT5x20 uses 16 DIMM slots
Up to 32x DIMM slots/socketT5x40 uses 8/16 DIMM slots
Sun Internal and Approved Partners Only 11
●
●
Data
10/100/1000/10GbE Network Interfaces• Uses the Neptune Chip on the Motherboard which provides
> Two RJ-45 ports dedicated for 10/100/1000 > Two RJ-45 ports capable of 10/100/1000
– Fitting a XAUI adapter disables the one of these ports per XAUI • Same 10 GbE XAUI cards as T5x20 server• 16 Rx and 24 Tx DMA channels for virtualization
– Integrated NIU in T2 has 16 Tx channels– Raw UDP Tx performance on Neptune > T2 NIU
– T2 processor network typically outperforms Neptune on most network benchmarks
• Packet classification> Rx data can be HW classified at layer 2 ,3 and 4
– Put into the separate Rx DMA buffers by classification rules
• Same 'nxge' device driver as T2 NIU.• Performance estimates, testing/tuning ongoing
– 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle– 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports)
12Sun Internal and Approved Partners Only
T5140/T5240Dual UltraSPARC T2 Plus
Extreme Rackmount Density• 1 / 2RU chassis, 28” depth• Two sockets
> Up to 8 cores @ 1.2GHz at RR – 1.4GHz to follow
> Up to 128 threads • 161U/2U/ 322U FB-DIMM memory slots
> 64/128GB of memory, 4GB DIMMsHigh Reliability• Up to 4 (1U) or 8/16 (2U) Hot-Plug
SATA/SAS 2.5” HDD (Raid 0,1) > RAID 5 BBWC card post-RR
• Redundant, hot-swappable PSUs and Fans. DC Option for 2U
• Certain Configs are 'Hi-Line Power' onlyExpandable• 3/6 PCI-E expansion slots
> including 2 x 10 GBE I/0 XAUI slots• 4 x 1GbE Ethernet as standard• 4 USB ports
• Up to 128 threads / 128GB memory / 128 LDoms in a 1U / 2U redundant chassis
• Over 5,100 threads per rack !• High RAS and I/O bandwidth• Scale-Up / Scale-Out of application
infrastructure
Sun Internal and Approved Partners Only 13
T5x40 Standard Configurations (PTO)Part Number Description 1RU
SETPAAE1Z
SETPBGE1Z
SETPCJE1Z
SETPCPE1Z
Part Number Description 2RUSEUPBAE1Z
SEUPCJE1Z
SEUPCPE1Z
SEUPCPE2Z
SEUPDPE1Z
2 x 4 Core, 64 Threads, 1.2Ghz, 8GB (8 X 1GB),4 disk capable, 2x 146GB disks, 2x 720W PSU, DVD2 x 6 Core, 96 Threads, 1.2Ghz, 16GB (8 X 2GB), 4 disk capable, 2x 146GB disks, 2x 720W PSU, DVD2 x 8 Core, 128 Threads, 1.2Ghz, 32GB (16 X 2GB),4 disk capable, 2x 146GB disks, 2x 720W PSU, DVD2 x 8 Core, 128 Threads, 1.2Ghz, 64GB (16 X 4GB),4 disk capable, 2x 146GB disks, 2x 720W PSU, DVD
2 x 6 Core, 96 Threads, 1.2Ghz, 8GB (8 X 1GB),8 disk capable, 2x 146GB disks, 2x 1100W PSU, DVD2 x 8 Core, 128 Threads, 1.2Ghz, 32GB (16 X 2GB),8 disk capable, 2x 146GB disks, 2x 1100W PSU, DVD2 x 8 Core, 128 Threads, 1.2Ghz, 64GB (16 X 4GB),8 disk capable, 2x 146GB disks, 2x 1100W PSU, DVD2 x 8 Core, 128 Threads, 1.2Ghz, 64GB (16 X 4GB),16 disk capable, 2x 146GB disks, 2x 1100W PSU, DVDHIGHLINE POWER ONLY (220VAC)2 x 8 Core, 128 Threads, 1.4Ghz, 64GB (16 X 4GB),16 disk capable, 2x 146GB disks, 2x 1100W PSU, DVDHIGHLINE POWER ONLY (220VAC)
Sun Internal and Approved Partners Only 14
Sun Fire T1000/T5120/T5140 Differences
Feature Sun Fire T1000 SPARC Enterprise T5120 SPARC Enterprise T5140Form Factor 1U, 18¾” deep 1U, 28” deep 1U, 28” deep
CPU
Memory
Network 4 x GbE 4 x GbE + 2 x 10GbE
Internal Storage
Removable Media None 1x DVD-RW 1x DVD-RWSerialPCI Express slots
Power Supply
Fans Non-Hot-Swap 4 x Redundant/Hot-Swap 6 x Redundant/Hot-Swap
UltraSPARC T11.0 GHz, 16-32 threads
UltraSPARC T21.2/1.4 GHz 32-64 threads
2x UltraSPARC T2 Plus1.2/1.4 GHz 64-128 threads
DDR2 32GB Max8x slots
FB-DIMM 64GB MAX16x slots
FB-DIMM 64GB MAX16x slots
4 x 1GbE or3 x 1Gbe + 1 x 10GbE or
2 x 1GbE + 2 x 10GbE1x 3.5” SATA, non-hot-swap
OR 2x 2.5” SAS Up to 8 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap
1x RS-232, No USB 1x RS-232, 4x USB 1x RS-232, 4x USB1x (low profile) 3x(low profile) 3x(low profile)
1x 300W ACNon-Redundant
2x 650W AC & DC optionRedundant/Hot-Swap
2x 720W ACRedundant/Hot-Swap
Sun Internal and Approved Partners Only 15
Sun Fire T2000/T5220/T5240 Differences
Feature Sun Fire T2000 SPARC Enterprise T5220 SPARC Enterprise T5240Form Factor 2U, 24” deep 2U, 28” deep 2U, 28” deep
CPU
Memory
Network 4x GbE4x GbE + 2x 10GbE
Internal Storage
Removable Media 1x DVD-ROM 1x DVD-RW 1x DVD-RWSerialPCI Express slotsPCI-X slots None None
Power SupplyFans 3 x Redundant Hot-Swap 3 x Redundant Hot-Swap 5x Redundant/Hot-Swap
UltraSPARC T11.0 - 1.4GHz 32 threads
UltraSPARC T21.2/1.4 GHz 64 threads
2x UltraSPARC T2 Plus1.2/1.4 GHz 64-128 threads
DDR2, 64GB MAX16x Slots
FB-DIMM, 64GB MAX16x Slots
FB-DIMM 128GB MAX32x slots
4 x GbE or3 x GbE + 1 x 10GbE or
2 x GbE + 2 x 10GbE
4x 2.5” SAS, hot-swap 8 x 2.5” SAS, hot-swap Up to 16 x 2.5” SAS, hot-swap
1x RS-232, 4x USB 1x RS-232, 4x USB 1x RS-232, 4x USB3x (low profile) 6x (low profile) 6x (low profile)2x (low profile)
2x 450W ACRedundant/Hot-Swap
2x 750W ACRedundant/Hot-Swap
2x 1100W AC/2 x1200Watt DCRedundant/Hot-Swap
Sun Internal and Approved Partners Only 16
Front Panel
DISK 0DISK 0 DISK 1DISK 1 DISK 2DISK 2 DISK 3DISK 3
DISK 0DISK 0
DISK 1DISK 1 DISK 3DISK 3
DISK 2DISK 2
DISK 5DISK 5
DISK 4DISK 4 DISK 6DISK 6 DISK 7DISK 7
Locator LED/Button
Fault LEDStatus LEDPower Button
Locator LED/Button
Fault LEDStatus LEDPower Button
Serial Number label
Serial Number
Disk Drive MapTop Fault LEDPSU Fault LEDTemp Fault LED
Top Fault LEDPSU Fault LEDTemp Fault LED
Disk Drive MapDVD-ROMDVD-ROM
DVD-RWDVD-RW
Dual USB 2.0 ports
Dual USB 2.0 ports
DISK 0DISK 0DISK 1DISK 1
DISK 3DISK 3DISK 2DISK 2
DISK 5DISK 5DISK 4DISK 4 DISK 10DISK 10 DISK 13DISK 13
DVD-RWDVD-RWDISK 6DISK 6
DISK 7DISK 7
DISK 8DISK 8DISK 9DISK 9
DISK 11DISK 11DISK 12DISK 12 DISK 15DISK 15
DISK 14DISK 14
Sun Internal and Approved Partners Only 17
Rear Panel
PSU 0PSU 0 PSU 1PSU 1
PSU 1PSU 1
PSU 0PSU 0
PSUStatusLED's
SPSerialPort
ChassisStatus LED's
and Locator Button SP
NetworkPort
Quad GigabitEthernet Ports Dual
USB 2.0Ports
POSIX DB-9Serial Port
PCI/XAUI 0PCI/XAUI 0 PCI/XAUI 1PCI/XAUI 1 PCI 2PCI 2
PCI/XAUI 0PCI/XAUI 0 PCI/XAUI 1PCI/XAUI 1 PCI 2PCI 2
PCI 5PCI 5PCI 4PCI 4PCI 3PCI 3
PSUStatusLED's SP
SerialPort
ChassisStatus LED's
and Locator Button SP
NetworkPort
Quad GigabitEthernet Ports Dual
USB 2.0Ports
POSIX DB-9Serial Port
Sun Internal and Approved Partners Only 18
x4x4
x4
33MHz32bit
USB
2.0
2.0
2.0
x8
x8
USB 2.0Hub
Front PanelUSB Quad GbE
ConnectorsSerialMgt
NetworkMgt
POSIXSerial DB-9Rear Panel
CL
SSIx8 x8
x1
PCIeto
PCI
PCIeSwitch
PLX 8548
PCIeSwitch
PLX 8548x8 x8
x8
QUAD GbEPHY
BCM5466R
0 1 2 30
NEPTUNEEthernet
2 x 10/100/10002 x 10/100/1000/10000
USBto IDE
Disk Chassis1RU=4 Disks
PCIto
USB
LSI1068E
MPC885ILOM
ServiceProcessor
FPGA
SSI
DVD
T5140 Block Diagram
1RU=16 DIMMS
XAUI XAUI
PCIex16
Slot2
PCIex8
Slot0
PCIex8
Slot1
Sun Internal and Approved Partners Only 19
x4x4
x4
x4
33MHz32bit
USB
2.0
2.0
2.0
x8
x8x8
USB 2.0Hub
Front PanelUSB Quad GbE
ConnectorsPCIex16
Slot2
PCIex8
Slot5
PCIex8
Slot0
PCI-ex8
Slot3
2RU
SerialMgt
NetworkMgt
POSIXSerial DB-9Rear Panel
CL
SSIx8 x8
x1
PCIeto
PCI
x8 x8
PCIeSwitch
PLX 8548
PCIeSwitch
PLX 8548x8 x8
2RU 2RU
PCIex8
Slot1
PCIex8
Slot4
x8
QUAD GbEPHY
BCM5466R
0 1 2 30
NEPTUNEEthernet
2 x 10/100/10002 x 10/100/1000/10000
USBto IDE
Disk Chassis2RU = 8 or 16 Disks
PCIto
USB
LSI1068E
MPC885ILOM
ServiceProcessor
FPGA
SSI
DVD
T5240 Block Diagram2RU Memory Mezzanine Tray
Option2RU=16 DIMMS without Mezannine, 32 DIMMS with mezannine
XAUI XAUI
Sun Internal and Approved Partners Only 20
Internal View (T5140)
DiskDiskChassisChassis
SAS x4 cable to Disk Chassis 6 Fan Assemblies are required
(chassis can take up to 8)
System Configuration Card (SCC)EEPROM
PCIe Slot 2x16(m)/x8(e)
Chassis Intrusion Reed Switch(magnet on underside of lid)
X4 SAS Connector 0 (cable attached)X4 SAS Connector 1 (not used in 1RU)
Service Processor
UltraSPA
RC
UltraSPA
RC
T2 Plus 0T2 Plus 0
PowerPowerBoardBoard
PSU1PSU1
PSU0PSU0
PCIe Slot 1x8(m)/x8(e)
orXAUI 1
PCIe Slot 0x8(m)/x8(e)
orXAUI 0
Power BUS BarsRED = +12VoltsBLUE = Ground
FB0/FM
1FB
0/FM1
FB1/FM
0FB
1/FM0
FB1/FM
1FB
1/FM1
FB1/FM
2FB
1/FM2
Fault RemindButton
FB0/FM
0FB
0/FM0
FB0/FM
2FB
0/FM2
UltraSPA
RC
UltraSPA
RC
T2 Plus 1T2 Plus 1
Memory Mezannine Connectors (not used on T5140)
FB-DIMMsFB-DIMMs
FB-DIMMsFB-DIMMs
Power Power SuppliesSupplies
NeptuneNetwork ASIC
Sun Internal and Approved Partners Only 21
Internal View (T5240 without Mezzanine)
DiskDiskChassisChassis
5 Fan Assemblies are required(chassis can take up to 6)
System Configuration Card (SCC)EEPROM
Chassis Intrusion Reed Switch(magnet on underside of lid)
X4 SAS Connector 0 (cable attached)X4 SAS Connector 1 (not used in 1RU)
Service Processor
UltraSPA
RC
UltraSPA
RC
T2 Plus 0T2 Plus 0
PowerPowerBoardBoard
Power BUS BarsRED = +12VoltsBLUE = Ground
FB0/FM
1FB
0/FM1
FB1/FM
0FB
1/FM0
FB1/FM
1FB
1/FM1
Fault RemindButton
FB0/FM
0FB
0/FM0
FB0/FM
2FB
0/FM2
UltraSPA
RC
UltraSPA
RC
T2 Plus 1T2 Plus 1
Power Power SuppliesSupplies
PCIe Slot 2x8(m)/x8(e)x16(m)/x8(e)
PSU's PSU's Vertically Vertically StackedStacked
PCIe Slot 1x8(m)/x8(e)x8(m)/x8(e)
orXAUI 1
PCIe Slot 0x8(m)/x8(e)x8(m)/x8(e)
orXAUI 0
Memory Mezannine Connectors
FB-DIMMsFB-DIMMs
FB-DIMMsFB-DIMMs
Sun Internal and Approved Partners Only 22
Internal View (T5240 with Mezzanine)
DiskDiskChassisChassis
5 Fan Assemblies are required(chassis can take up to 6)
System Configuration Card (SCC)EEPROM
Chassis Intrusion Reed Switch(magnet on underside of lid)
Service Processor
PowerPowerBoardBoard
Power BUS BarsRED = +12VoltsBLUE = Ground
FB0/FM
1FB
0/FM1
FB1/FM
0FB
1/FM0
FB1/FM
1FB
1/FM1
Fault RemindButton
FB0/FM
0FB
0/FM0
FB0/FM
2FB
0/FM2
Memory Mezannine Handles
FB-DIMMsFB-DIMMs
FB-DIMMsFB-DIMMs
Power Power SuppliesSupplies
PCIe Slot 2x8(m)/x8(e)x16(m)/x8(e)
PSU's PSU's Vertically Vertically StackedStacked
PCIe Slot 1x8(m)/x8(e)x8(m)/x8(e)
orXAUI 1
PCIe Slot 0x8(m)/x8(e)x8(m)/x8(e)
orXAUI 0
Mezzanine Air Guide
Sun Internal and Approved Partners Only 23
Motherboard Memory Physical LayoutChassis Front
Chassis Rear
112233
UltraSPARCT2 Plus 0
Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1
Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1
UltraSPARCT2 Plus 1
Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1CMP0/Branch1/Channel1/DIMM0
Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1
Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1
Channel0/DIMM 1/Branch 1Channel0/DIMM 1/Branch 1
CMP0/Branch1/Channel1/DIMM1
CMP0/Branch1/Channel0/DIMM1CMP0/Branch1/Channel0/DIMM0CMP0/Branch0/Channel0/DIMM0CMP0/Branch0/Channel0/DIMM1
CMP0/Branch0/Channel1/DIMM1CMP0/Branch0/Channel1/DIMM0
CMP1/Branch1/Channel1/DIMM0CMP1/Branch1/Channel1/DIMM1
CMP1/Branch1/Channel0/DIMM1CMP1/Branch1/Channel0/DIMM0CMP1/Branch0/Channel0/DIMM0CMP1/Branch0/Channel0/DIMM1
CMP1/Branch0/Channel1/DIMM1CMP1/Branch0/Channel1/DIMM0
2211
11
1122
22
11
1111
11
33
33
33
33
FIRSTInstall ALL DIMMs in
Group 18 DIMM Minimum
Config
SECONDInstall ALL DIMMs in
Group 212 DIMM Config
THIRDInstall ALL DIMMs in
Group 316 DIMM Config
All DIMMs MUST be the same Size i.e 1GB, 2GB, 4GB, no mixingAll DIMMs in a Branch must be the same Sun Part NumberUnused DIMM locations must be occupied with DIMM Fillers
At RR, Supported configs will be ...Fully Populate Group 1 and 2 (8 DIMMS)
Fully Populate Groups 1, 2 and 3 (16 DIMMS)
Sun Internal and Approved Partners Only 24
Mezzanine Memory Physical LayoutChassis Front
Chassis Rear
4433Channel0/DIMM 1/Branch 1
Channel0/DIMM 1/Branch 1CMP0/MR0/BR1/CH1/DIMM3
Channel0/DIMM 1/Branch 1
55
FOURTHInstall ALL DIMMs in
Group 424 DIMM Config
FIFTHInstall ALL DIMMs in
Group 532 DIMM Config
All DIMMs MUST be the same Size i.e 1GB, 2GB, 4GB, no mixingAll DIMMs in a Branch must be the same Sun Part NumberUnused DIMM locations must be occupied with DIMM Fillers
CMP0/MR0/BR1/CH1/DIMM2
CMP0/MR0/BR1/CH0/DIMM2CMP0/MR0/BR1/CH0/DIMM3CMP0/MR0/BR0/CH0/DIMM3CMP0/MR0/BR0/CH0/DIMM2
CMP0/MR0/BR0/CH1/DIMM2CMP0/MR0/BR1/CH1/DIMM3
CMP1/MR1/BR1/CH1/DIMM3CMP1/MR1/BR1/CH1/DIMM2
CMP1/MR1/BR1/CH0/DIMM2CMP1/MR1/BR1/CH0/DIMM3CMP1/MR1/BR0/CH0/DIMM3CMP1/MR1/BR0/CH0/DIMM2
CMP1/MR1/BR0/CH1/DIMM2CMP1/MR1/BR0/CH1/DIMM3
44
44444444
4444
55
55555555
5555
At RR, Supported configs will be ...Fully Populate Groups 1, 2, 3 , 4 and 5 (32 DIMMS)
Sun Internal and Approved Partners Only 25
Memory Configuration Rules• Populate at least 4 DIMMS at a time• Populate by complete group only (set 1, 2, 3, 4, 5)
> Groups may be 4 or 8 DIMMs, check service labels on lid • Ensure proper installation of Airflow Guides
> All Configs require DIMM fillers for correct air-flow> 1RU has no Airflow Guide> 2RU has two configurations
– No Mezzanine – Install Lower Air Guide
– With Mezzanine– Install Lower PDB Guide and Mezzanine Guide
• All DIMMS must be the same size(i.e all 1GB, 2GB or 4GB, no mixing)• In a Group all DIMMS must have the same SUN Part No.• No clocking down of memory in any config.• 8GB DIMMs will be qualified/evaluated after RR
Sun Internal and Approved Partners Only 26
Network Ports and Options
XAUI 0
NOTESAll Ethernet ports provided by a single Neptune Chip on the motherboard
NET 0 and NET 1 are disabled by XAUI cards if installed to take advantage of 10GbE networking* If a XAUI card is installed, the matching RJ-45 port is disabledXAUI cards take one of three current Fibre 10GbE transceiversShort Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM)
Copper versions of the XAUI card are expected Post-RR XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive
with PCI cards in the same location
Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1Driver nxge0 nxge1 nxge2 nxge3 PCI Device PCI@300/0/8/0,0 PCI@300/0/8/0,1 PCI@300/0/8/0,2 PCI@300/0/8/0,3 PCI@300/0/8/0,0 PCI@300/0/8/0,1
nxge0* nxge1*
XAUI 1
Sun Internal and Approved Partners Only 27
10GbE XAUI Option Cards• Interface to the Neptune chip to upgrade two of its ports to 10GbE• Cards each consume a PCIe slot if installed• Attach to dedicated XAUI sockets on special risers• 10Gbe XAUI Card
– SESY7XA1Z> Requires Fibre Tranceivers
– 10GbE Short Reach(SR)– SESY7XT1Z
– 10GbE Long Reach (LR)– SESY7XT2Z
– 10GbE Long Reach multimode (LRM)– part No. TBD
• Copper connector XAUI versions to follow post-RR
Sun Internal and Approved Partners Only 28
PCI Riser Cards• Various Riser Cards are used• Risers should not be moved between slots• All slots accommodate low Profile PCI-Express cards• T5140 – 25 Watts per card, 75 Watts Max (3 Slots)• T5240 – 25 Watts per card, 102 Watts Max (6 Slots)
> Riser Card Mechanical and Electrical widths for T5140 and T5240
T5140 Slot 0 Slot 1 Slot 2Only Slot x8/XAUI(m) x8(e) x8/XAUI(m) x8(e) x16(m) x8(e)
T5240 Slot 1 Slot 1 Slot 2Top Slot x8(m) x8(e) x8(m) x8(e) x8(m) x8(e)
Bottom Slot x8/XAUI(m) x8(e) x8/XAUI(m) x8(e) x16(m) x8(e)
Sun Internal and Approved Partners Only 29
Disk Cage• Disk chassis is a FRU
– 4 screws and disconnect cables• Contains DVD-RW/Dual USB module as a
sub-FRU/CRU– Standard in all default configs– remove disk 3/7 to access release latch, pre-RR
• Non-DVD module available as X-Option– Still provides Dual USB ports
• T5140 will be available with only a 4 disk chassis (8 disk may be post RR)> Only uses one of the x4 SAS links
• T5240 will be available with 8 or 16 disk chassis> 16 Disk systems requires High-Line Power (220VAC)
Remove Disk and locate latch beneath DVD
Press up and pull DVD out
Sun Internal and Approved Partners Only 30
Disk Drives• Disk drives are all 2.5” form factor• Chassis support both SAS and SATA (post-RR)• SAS Drives
– 73GB @ 10K RPM– 146GB @ 10K RPM– 73GB @ 15K RPM– 146GB @ 15K RPM (post-RR)
• SATA Drives– 300GB @ 5400 RPM (H2CY08)– 500GB @ 7200 RPM (end CY08)
• New mounting bracket “Marlin”> Allows better cooling in denser chassis
• Same disks as T5x20 systems
Ready to Remove
FaultStatus
Disk LED's
Improved Airflow above
and below disks
Sun Internal and Approved Partners Only 31
Disk Controller Options• T5x40 has embedded LSI1068E on the motherboard
– JBOD up to 8 disks– Controls internal disks only– RAID 0 (striping) and RAID 1(mirroring), using 'radictl'– Hot-plug supported– Hot-swap in non-RAID configs is done with 'cfgadm'
• Optional Extended RAID functionality cards> SPARC Drivers are available, SAS cables are being produced now> SGXPCIESAS-R-INT-Z
– 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available)– Adaptec Card and Firmware with Intel SunRiseLake chipset– Battery Backed Write Cache (BBWC) of 256MB– BBWC battery at least 72 hours (much longer when new/fully charged)– Only for internal disks, replaces cables to route to standard disk chassis
> SGXPCIESAS-R-EXT-Z– External port version, for external disk arrays only (no internal ports)
> Not available for T5440 due to chassis design and only 4 disk capacity
Sun Internal and Approved Partners Only 32
Fans and Cooling• T5140
> Requires 6 Fan modules– Main System Cooling
– FANBD0/FM0– FANBD0/FM1– FANBD0/FM2– FANBD1/FM0– FANBD1/FM1– FANBD1/FM2
– No Modules in FM3• T5240
> Requires 5 Fan Modules– Main system Cooling
– FANBD0/FM0– FANBD0/FM1– FANBD0/FM2– FANBD1/FM0– FANBD1/FM1
– No Modules in FANBD1/FM2
PSU'sPSU'sFBD
1FBD
1FM
3FM
3FBD
1FBD
1FM
2FM
2FBD
1FBD
1FM
1FM
1FBD
1FBD
1FM
0FM
0
FBD0
FBD0
FM0
FM0
FBD0
FBD0
FM1
FM1
FBD0
FBD0
FM2
FM2
FBD0
FBD0
FM3
FM3
MotherboardMotherboard
Disk C
hassisD
isk Chassis
PSU'sPSU's
MotherboardMotherboard
Disk C
hassisD
isk Chassis
FBD0
FBD0
FM1
FM1
FBD0
FBD0
FM0
FM0
FBD0
FBD0
FM2
FM2
FBD1
FBD1
FM1
FM1
FBD1
FBD1
FM0
FM0
FBD1
FBD1
FM2
FM2
FrontBack
Unused Fan slots can be populated, but not recommended, fans will run and wear, so not a great way of keeping spares....
Sun Internal and Approved Partners Only 33
Fans modules and boards• Fans are speed controlled by the Service Processor• Each fan has an Amber FAULT LED
– if lit the fan module is faulty• 1RU modules
– 2 Fans at 15K RPM Max.• 2RU modules
– 2 Fans at 12K RPM Max• Monitored through ILOM/IPMI/Web
Fan Fault Amber LED
Lit if fan faulty
Vibration Absorbing
Silicon Dampers
Sun Internal and Approved Partners Only 34
Power Supplies• PSU's are high efficiency• Output 12V (and 3.3v standby)• Most DC-DC conversion done in MBD• Shared with Doradi/o, Tucani/a• Integrated Fan• T5140
> Side-by-Side> AC 720Watts
• T5240> Vertically Stacked> 1100Watts> DC 1200 Watts option coming post-RR> DC or AC, no sharing> BEWARE...
– Same form factor as SunFire X4200 PSU's ... DO NOT MIX– 1.4Ghz and 16 Disk configs require 220VAC (called “High Line Power”)
Ready to Remove
Fault
AC Present
PSU LED's
Sun Internal and Approved Partners Only 35
Rack Mounting Options
• Two Rack mounting options are available..• Standard Rack Mount (4 post)SESY9RK1Z
> Standard configs contain this kit– ATO : SESY9RK1Z, X-ATO : SESY9RK1Z
• Tool-less Rack Kit (4 post)> Optional kit, purchased seperately> Snaps into most 3rd party Racks> Requires no tools to fit or remove
– ATO : SESY9RK2Z, X-ATO : SESX9RK2Z
Sun Internal and Approved Partners Only 36
Solaris for Sun SPARC Enterprise T5x40• T5x40 servers will require Solaris10 8/07 (update 4) with
patches> A Patch CD is supplied in the box
– Mandatory patches = 127111-08,126434-05,125416-05– Available patches = 124235-02, 125369-13,125476-02,127111-07– Likely to be out of date, so customers should check SunSolve...
• This incorporates several specific UltraSPARC T2/T2 Plus optimizations and for CMT servers in general
• Also includes some specific T5x40 optimisations> UltraSPARC T2 Plus MPO optimisations> nxge driver tweaks for 10GbE devices> MMU DMA relaxed scheduling > LDOMs pre-install upgrade to version 1.0.2 (US T2 Plus support)
Sun Internal and Approved Partners Only 37
Solaris FMA• Now aware of CPU Core offlining• PCIe Fabric diagnostics improved• Crypto Unit offlining• Neptune awareness (though XAUI's not diagnosed)• POST memory CE tolerance improved• FRU identity info improved in fault reports (part/ser #)• Diagnostic Engines (DE)
– Existing DE have been extended for T5x40– New Eversholt DE used for HostBridge, PCIe diagnostics
• Limitations...– Can't isolate Uncorrectable Errors to a single DIMM– PCIe riser cards not in topology– XAUI card and XFP not in the topology– SouthBridge chip is not diagnosed at this time (DVD, USB,...)– IO Expansion box is not diagnosed via FMA
Sun Internal and Approved Partners Only 38
ILOM Features at a Glance• Hardware Control, Inventory, Monitoring• Host Configuration and Administration• SP Configuration and Administration• Logging, Auditing and Alerting• Authentication :
– LDAP, RADIUS, Active Directory, Local Users
• Services :– SSH, HTTP(S), NTP, Syslog, SMTP
• Out-of-Band– Via Serial port, or dedicated Ethernet port
• In-Band– Via Host OS, firmware upload, Explorer, Syslog
Sun Internal and Approved Partners Only 39
ILOM – Industry Standard Manageability• ILOM enables Sun systems to be easily integrated into third
party management infrastructures• Key functionality
> IPMI 2.0> SNMP V1, V2c, V3> SSH 2.0> LDAP, RADIUS & MS Active Directory authentication> Browser UI, SMASH DMTF CLP + ALOM style CLI
• NOTE : No RKVMS functionality like Galaxy ILOM> No Java plugin required
• Browsers tested include, these base versions an later releases
– Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox 1.0 and 1.5 on W2K server
Sun Internal and Approved Partners Only 40
ILOM – Industry Standard Manageability
• Identical to ILOM 2.0 for T5x20• Default user is 'root' with password 'changeme'
> Use for ILOM WEB GUI access and SSH access and IPMI• The ALOM 'admin' user is not defined by default, create this
user with an ALOM CLI• Lost 'root' password and ILOM config is reset using jumpers
> T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2) • Note... 'sunservice' user shares the password of the 'root'
user ONLY FOR SERVICE ACCESS*• Passwords must be 8 to 16 chars (used to be max 8)• Max of 10 users allowed (used to be 16)
Sun Internal and Approved Partners Only 41
ILOM Common Tasks
• Setting up an ALOM 'admin' user from the ILOM CLI> SUNSP00144F4641BD login: root
Password: changemeWaiting for daemons to initialize...Daemons readySun(TM) Integrated Lights Out ManagerVersion 2.0.4.0Copyright 2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.Warning: password is set to factory default.->create /SP/users/admin role=Administrator cli_mode=alomCreating user...Enter new password: ********Enter new password again: ********Created /SP/users/admin2-> exitSUNSP00144F4641BD login: adminPassword: ********Waiting for daemons to initialize...Daemons readySun(TM) Integrated Lights Out ManagerVersion 2.0.4.0Copyright 2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.sc>
Sun Internal and Approved Partners Only 42
ILOM Common Tasks
• Setting up ILOM IP configuration from the ILOM CLI> SUNSP00144F4641BD login: root
Password: changemeWaiting for daemons to initialize...Daemons readySun(TM) Integrated Lights Out ManagerVersion 2.0.4.0Copyright 2007 Sun Microsystems, Inc. All rights reserved.Use is subject to license terms.Warning: password is set to factory default.-> set /SP/network state=enabledSet ’state’ to ’enabled’-> set /SP/network pendingipaddress=10.8.31.163Set ’pendingipaddress’ to ’10.8.31.163’-> set /SP/network pendingipdiscovery=staticSet ’pendingipdiscovery’ to ’static’-> set /SP/network pendingipnetmask=255.255.252.0Set ’pendingipnetmask’ to ’255.255.252.0’-> set /SP/network pendingipgateway=10.8.31.248Set ’pendingipgateway’ to ’10.8.31.248’-> set /SP/network commitpending=trueSet ’commitpending’ to ’true’->
-> set/SP/pendingipdiscovery=dhcpSet ’pendingipdiscovery’ to ’dhcp’-> set /SP/network commitpending=trueSet ’commitpending’ to ’true’->
Static IP configuration
Dynamic IP (DHCP)
configuration
Sun Internal and Approved Partners Only 43
ILOM IPMI functionality• IPMI is a widely used management protocol for x86 servers• ... But its new for SPARC boxes...• Download the required tools from
> The Solaris Supplemental CD (an older binary package)> Latest is available at http://ipmitool.sourceforge.net
– Binary packages (usually older versions) are available for Solaris x86 and SPARC, as well as Linux, Windows, Mac...etc
> There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC now too)
– http://www.sun.com/blueprints/0107/820-1011.pdf
-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr/MB/V_+1V0_VDD | 1.01 Volts | ok/MB/V_+3V3_STBY | 3.36 Volts | ok/MB/V_+3V3_MAIN | 3.36 Volts | ok/MB/V_+5V0_VCC | 4.97 Volts | ok/MB/V_+12V0_MAIN | 12.16 Volts | ok....-bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set /SYS/LOCATE on/SYS/LOCATE | ON-bash-ventus2$
Get the Sensor Data
Repository
Set the System Locator LED
flashing
Sun Internal and Approved Partners Only 44
ILOM Common Tasks
• Logging into the ILOM Web GUI• Point Web Browser at the IP address of
the ILOM> Will automatically redirect to a
secure HTTPS port> Best to disable any Web Proxies for
best performance> Make sure port 80, 443 are open on
any firewall
• Log in as any defined user, default Administration user is 'root'
Sun Internal and Approved Partners Only 45
ILOM/ALOM Common Tasks
• A Tour of the ILOM Web GUI • GUI provides ways to set up IP address configurations for ILOM
• Configure additional ILOM/ALOM users
• Access Sensor Readings
• Control System configurations
• Carry out firmware updates.
• etc....
Sun Internal and Approved Partners Only 46
ALOM 2.0 vs. 1.4 CLI Differences• Default CLI appearance is ILOM
– User is 'root'– Must create an 'admin' Administrator user with the ALOM CLI
• Flashupdate in new ALOM expects a TFTP server not an FTP server • ALOM IP address changes now active only after running ...
– 'setsc netsc_commit true'– Previously changes were active only after SC reset– DHCP is enabled by default
• View firmware versions with – 'showhost version'
• Only Admin and Operator rolesare defined
sc> showhost version Sun System Firmware 7.0.0_build_13 2007/08/10 23:44
Host flash versions: Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54 OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25 Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:51
Sun Internal and Approved Partners Only 47
ALOM backward Compatibilty CLI
• Uses ALOM-CMT 1.2 for a base feature set• Includes ALOM-CMT 1.4 features...
> break -d> Host Watchdog Timer> Bootmode LDOMs awareness> Forward SP events to Hot/Solaris syslog via inband channel
Sun Internal and Approved Partners Only 48
Logical Domains 1.0.2 Features
• UltraSPARC T2 Plus Platform Support> 128 domains supported at RR
– 256 domains work on T5440, may be qualified later, requires Solaris fixes• Feature Set very similar to 1.0.1• I/O domain can reboot while the other domains keep running
> Dependant domains would wait for IO transactions to complete• LDoms SNMP MIB
> Modeled after industry standard DMTF CIM (Common Information Model)> Running on control domain to enable remote SNMP monitoring
• Domain minimization support> Minimal/reduced Solaris installation> Base minimal packages plus optional ones for guest, control, I/O or service
domains
Release in Q1CY08 aligned with UltraSPARC T2 Plus launch
Sun Internal and Approved Partners Only 49
Logical Domains 1.0.2 Requirements
• Solaris 10 08/07 (update 4) and required patches• Systems firmware version
– T5x40 Firmware 7.1.0 = SP FW 2.0.4.20+– T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+– T2000/T1000/T6300/Pelton Firmware 6.5.x
• SUNWldm.v LDOMs Manager Package– Version 1.0.2
Sun Internal and Approved Partners Only 50
CRU/FRU List - T5140• Lots shared with other Jake and Elwood systems
Item Description FRU/CRUDisk Backplane 4 drive Disk Backplane to support 4 Drives only FRUDisk Backplane 8 drive Disk Backplane to support 8 Drives (post -RR) FRUFan Module 1U Dual Fan module, 4 populated in system CRUFan Board 1U Fan connector board to accommodate Fan Modules, 2 per system FRUDVD Assembly DVD Assembly including front USB board FRUPaddle board 1U FRUPDB 1U FRUPSU 1U Power supply (720 Watts) CRU
PCI Riser assembly for left and centre motherboard slots CRUPCI Riser for right motherboard slot CRU
Rack mount kit (requires tools) Rack mount kit CRURack mount kit, no tools required for installation CRU
CMA Cable management arm CRUBus Bars Power Bus Bars between PDB and Motherboard FRUPDB to MB Ribbon Cable FRULight pipe kit Light pipe assemblies for left and right of disk cage FRUMotherboard assembly Motherboard assembly, part no varies depending on Core count FRU
CRUXAUI board optical/copper XAUI Adapter card (two different parts for copper/optical) CRUXFP module XFP GBIC module for use in XAUI cards (three different parts SR, HR, HRMM) CRUBattery Battery backup for NVRAM of Service Processor CRUHDD 73GB/146GB SAS Hard Disk drive, different part numbers for capacities CRUSystem Configurations PROM System Configuration PROM FRU
SAS Cable kit FRU
Conection between PDB, fan boards and Disk BackplanePower distribution board, contains master chassis serial number
X8 &XAUI, 1U PCIe Riserx16 1U PCIe Riser
Rack Mount Kit (tooless)
Ribbon cable for control connections between PDB abd Motherboard
DIMMs FBDIMMs (three different part numbers for 1GB, 2GB, 4GB)
Maramba Cable Kit
Sun Internal and Approved Partners Only 51
CRU/FRU List - T5240• Lots shared with other Jake and Elwood systems
Item Description FRU/CRUDisk Backplane 16 drive Disk Backplane to support 16 Drives only FRUDisk Backplane 8 drive Disk Backplane to support 8 Drives only FRUFan Module 2U Dual Fan module, 3 populated in system CRUFan Board 2U Fan connector board to accommodate Fan Modules, 2 per system FRUDVD Assembly DVD Assembly including front USB board FRUPaddle board 2U Connection between PDB, fan boards and Disk Backplane FRUPDB 2U FRUPSU 2U Power supply (1100 Watts) CRU
PCI Riser assembly for left and centre motherboard slots CRUPCI Riser for right motherboard slot CRU
Rack mount kit (requires tools) Rack mount kit CRURack mount kit, no tools required for installation CRU
CMA Cable management arm CRUBus Bars Power Bus Bars between PDB and Motherboard FRUPDB to MB Ribbon Cable FRULight pipe kit Light pipe assemblies for left and right of disk cage FRUMotherboard assembly Motherboard assembly, part no varies depending on Core count FRU
CRUXAUI board optical/copper XAUI Adapter card (two different parts for copper/optical) CRUXFP module XFP GBIC module for use in XAUI cards (three different parts SR, HR, HRMM) CRUBattery Battery backup for NVRAM of Service Processor CRUHDD 73GB/146GB SAS Hard Disk drive, different part numbers for capacities CRUSystem Configurations PROM System Configuration PROM FRU
SAS Cable kit FRU
Power distribution board, contains master chassis serial number
X8,X8, XAUI, 2U PCIe RiserX8, x16 2U PCIe Riser
Rack Mount Kit (tooless)
Ribbon cable for control connections between PDB abd Motherboard
DIMMs FBDIMMs (three different part numbers for 1GB, 2GB, 4GB)
Maramba Cable Kit
Sun Internal and Approved Partners Only 52
Option Parts List for T5140/T5240Part Number Option Description AvailableSECY9PS41Z AC PSU 1RU (720 Watts) RRSEDY9PS51Z AC PSU 2RU (1100 Watts) RR- DC PSU 2RU (1200Watts) post-RRSESY2A1Z 2GB Memory (2x1GB) RRSESY2B1Z 4GB memory (2x2GB) RRSESY2C1Z 8GB Memory (2x4GB) RRSESY9FF1Z FB-DIMM Filler panel RRSEUY2MM1Z Memory Mezzanine Assembly 2U only RRSEU9AD1Z Memory Mezzanine Air Duct RRSETY9BA1Z 4 Disk Backplane for 1U RRSETY9BB1Z 8 Disk Backplane for 1U post-RRSEUY9BB1Z 8 Disk Backplane for 2U RRSEUY9BC1Z 16 Disk Backplane for 2U (High Line only (220VAC) RRSESY3A11Z 73GB, 10KRpm 2.5” SAS Drive RRSESY3C11Z 146GB, 10KRpm 2.5” SAS Drive RRSESX3A21Z 73GB, 15KRpm, 2.5” SAS Drive RR- SATA Drives (300,500GB) post-RRSESY9MF1Z HDD Filler Panel RRSESY9DV1Z DVD, 8X RW, Slot Load RRSESY9RK1Z Slide Rail Kit, requires screwdriver (1RU and 2RU) RRSESY9RK2Z Slide Rail Kit, Tool-less (1RU and 2RU) RR
Sun Internal and Approved Partners Only 53
Option Parts List for T5140/T5240 (cont..)Part Number Option Description Max no.(1RU/2RU)
SG-PCIE1FC-QF4 RR 3/6SG-PCIE2FC-QF4 RR 3/6SG-PCIE1FC-EM4 RR 3/6SG-PCIE2FC-EM4 RR 3/6SG-PCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 3/6SG-PCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 3/6SGCPCIESAS-R-INT-Z SAS, 8 Port, RAID, PCI-E, External Drive Controller RR 1/1SGCPCIESAS-R-EXT-Z SAS, 8 Port, RAID, PCI-E, External Drive Controller RR 1/1SECX9SA1Z Cable X-option for T5140 RR 1/1SECX9SA2Z Cable X-option for T5240 RR 1/17281A-2 RR 3/67280A-2 RR 3/61236A-Z RR 3/64447A-X RR 3/3X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 3/3X5558A RR -X5560A-Z RR -X3000A XVR-300, PCI-E x8 Graphics cards TBD 1/1X4240A XVR-300, PCI-E x16 Graphics cards, requires 605-4668-04 cable TBD 1/1SESY7XA1Z 10GbE XAUI card RR 2/2TBD 10GbE XAUI Copper 10g Base-T TBD 2/2X6000A Sun Crypto Accelerator 6000 RR 1/ 2
AvialableQlogic, 4Gb, PCI-E Single PortQlogic, 4Gb, PCI-E Dual Port
Emulex, 4Gb, PCI-E Single Port Emulex, 4Gb, PCI-E Dual Port
Northstar Dual port GigE MMF Low profile PCI-ENorthstar Dual port GigE UTP Low profile PCI-E
Infiniband HBA, Dual 4x ports, PCI-EAtlas Quad Port, GigE Copper, PCI-E (up to 3/6)
Tranceiver 10GE SR XFP (XAUI/Atlas)Tranceiver 10GE LR XFP (XAUI/Atlas)
Sun Internal and Approved Partners Only 54
Tuning Out of the Box• Using Solaris 10 08/07 (update 4) with Patches
> MPO optimisations used> DMA relaxed Ordering implemented> Default pages sizes for
– Initdata Segments is 64kB pages– Private Anon, Heap, Process Stack, Text, Shared Anon segments are 4MB pages– ISM and DISM use 256MB pages– New variables introduced to control these sizes if changes are needed
– max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize, max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize, max_shm_lpsize
> L2 Cache hashing implemented– helps randomise accesses to L2 caches and ease thrashing
> Stack biasing implemented– D-Cache is only 4 Way associative... 8 threads– Running multiple copies of a binary can lead process stacks aligning in the cache
and to hot lines in the cache– Stack Bias slews the stack variable for processes.
Sun Internal and Approved Partners Only 55
Tuning Out of the Box (continued..)> Processor Groups introduced
– Helps associate CPUs with some shared resources (int/FP pipelines, Crypto units etc)
– Enables improved multi-level CMT load balancing and affinity policies> 'bcopy'/'uiomove' Optimizations
– Key kernel operations– new implementation about 2x faster than previous
• Optimisations will require patching of S10 8/07> These will be post RR patches as they become available> Most fixes are already committed in update 5> Check patch levels for customers with performance issues> Also check Firmware versions as this can improve performance
too...
Sun Internal and Approved Partners Only 56
Tuning Recommendations• Less involved than with T2000• S10 8/07 is more 'automatic'
– OOB optimised page size selection, thread scheduling, block copy, IP instances for zones, page coloring etc...
• If tuning is necessary remember...> Improved pipeline results in Single threaded performance around 20% better> Floating Point is no longer an issue> Start with T2000 tuning options> Use updated trapstat, cpustat,
• LDOMs may be an easier way to scale poorly threaded applications• Solaris Crypto framework is still being tuned
> Great for big packets, working on improving small packets• A Blueprint is being prepared (Denis Sheahan)
Sun Internal and Approved Partners Only 57
Tuning Recommendations (cont..)• Refer to UltraSPARC T1 applications tuning best practices...
> http://www.sun.com/servers/coolthreads/tnb/applications.jsp• CoolTools are being updated for UltraSPARC T2 and T2 Plus
> http://www.sun.com/servers/coolthreads/overview/cooltools.jsp• Sun Studio 12 is UltraSPARC T1 and T2/T2 Plus aware
> Recompiling without code changes should benefit T1/T2/T2 Plus apps• LDOMs configs can benefit significantly from a transition to T5x20 platform
> Multiple IO/Service domains with higher bandwidth PCI busses> Make sure appropriate LDOMs tuning and sizing is carried out
• Wider Crypto capability, so ensure code is using PKCS#11 framework> Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers),
n2rng(random number generator) drivers are used appropriately
Sun Internal and Approved Partners Only 58
PCIe Performance
• Up to two times the performance of T5x20 (two PCIe Busses)
• All PCI card slots are x8 PCI electrically> Single level hierarchy> Cut through switching> PCI-E relaxed ordering for higher performance
• Performance per bus...> Marketing performance = 4000MB/sec> Theoretical Rate with overheads = 3790MB/sec> Max. DMA Achievable(bidirectional) = 3096MB/sec
– with 128byte packets– And 512 byte read size
Sun Internal and Approved Partners Only 59
Tools and Information
• Cool Tools– http://www.sun.com/servers/coolthreads/overview/cooltools.jsp
• SUNWIN Tokens> Sun SPARC Enterprise T5140/T5240
– Architecture Whitepaper : 526121> T5140/T5240 RAS Whitepaper : TBA
• Video Tours> Check the Sun.com product pages
• Power Calculators> Check the Sun.com product pages
60
<Name><Job Title>Sun Microsystems
60
Sun Internal and Approved Partners Only 61
Breakthrough Price, Breakthrough Value
HP DL580G5 IBM p550
$32,319 $28,234 $77,906
Virtualization $0 $9,074 $5,2402 x 10Gb Ethernet $1,788 $7,198 $9,484Crypto Acceleration $0 $1,500 $9,000Grand Total $34,107 $46,006 $101,630Introductory Offer $17,922
Sun SPARC Enterprise T5240
32GB Memory, OS Support
See Pricing Substantiation Slide. Introductory price based on buying Try and Buy unit before June 30th
Sun Internal and Approved Partners Only 62
Massive Internal Storage Expansion
IBM Power 6 p520 IBM Power 6 p550 IBM x3850 M2 HP Dell
Internal Disk Drives 2.5x 2.5x 4x 2x 2x
Rack Space ½ ½ ½ Same Same
Lower costs, lower power, and lower space as it avoids the need to add disk arrays for local data storage
• T5240 supports up to 16 internal disks• 2-4x than IBM, HP, and Dell
Sun Internal and Approved Partners Only 63
T5240 supports 4X more Memory
T5240 HP DL380G5
Dual socket / 2U
Sun SPARC Enterprise T5240 Beats HP
Dual socket / 2U
Sun Internal and Approved Partners Only 64
Pricing SubstantiationSun SPARC Enterprise T5240 Comparisons:DL580G5, - Base server price for HP DL580G5 with 4 x Intel Xeon x7350 2.93GHz processors, 16 x 2GB FB-DIMMs, Slimline DVD-ROM/CD-RW drive, 2 x 146GB 10k rpm SAS, RAID 1, Redundant PSUs, 2 x 1 Gigabit Ethernet, HP iLO Select Pack, 4 x 1GbE, Microsoft Windows 2003 Enterprise Edition with 25 CALs . Virtualization price for VMware ESX Virtual Infrastructure Standard 4 Processor License with 1 year coverage. 10GbE price for 2 x SR-fiber NC510F PCI-E adaptors. Crypto Acceleration price for Sun Crypto Accelerator 6000. From Sun Store pricing www.sun.com 01/25/08. Pricing from HP Store, www.hp.com, 02/15/08IBM pricing from Sun Market Intelligence, 02/13/08. IBM p550 configured with 2 x 4.2 GHz Power6 processors, 16 x 2 GB FBDIMMS, 2 x 146 GB SAS 2.5” HDD, RAID, 4 x 1GbE ports, 2 x 1700 Watt PSU, DVD-ROM, Rail kit, AIX 6.1. IBM Virtualisation costs based on $1,310 per core, IBM 10GbE costs based on Short Range fiberSun SPARC Enterprise T5240 configured with 2 x 1.2 GHz UltraSPARC T2 Plus 8 Core processors, 16 x 2 GB FBDIMMS, 2 x 146 GB SAS 2.5” HDD, RAID, 4 x 1GbE ports, 2 x 1100 Watt PSU, Slim DVD RW, Rail kit, 2 x 10GbE XAUI Adaptors, Solaris 10
Sun Internal and Approved Partners Only 65
10GbE Networking Substantiation10GbE price for 2 x SR-fiber PCI-E adaptors = $6,740. From HP DL380G5 pricing on HP Store, www.hp.com, 12/17/07. Cost for 2 x 10GbE and SR transceivers for Maramba and Batoka are $1,800. From sun.com, 12/17/07