gx60607r100

________________________________________________________________________

Global Support Enablement Education

Servicing the IBM Flex Chassis (8721)

November 2011

Study guide

GX60607 Release 1.00

This course is owned and published by the IBM MTS

Global Support Skills and Knowledge Enablement Team and created by Alex Badia.

Servicing the IBM Flex Chassis (8721) Preface

November 2011 2 GX60607.pdf

International Business Machines Corporation, 2011 All rights reserved. IBM MTS Global Support Skills and Knowledge Enablement IBM Systems, Department EYGA. Building 203, Post Office Box 12195, Research Triangle Park, North Carolina 27709-2195 IBM reserves the right to change specifications or other product information without notice. This publication could include technical inaccuracies or typographical errors. References herein to IBM products and services do not imply that IBM intends to make them available in other countries. IBM provides this publication as is, without warranty of any kind either expressed or impliedincluding the implied warranties of merchantability or fitness for a particular purpose. Some jurisdictions do not allow disclaimer of expressed or implied warranties. Therefore, this disclaimer may not apply to you. Data on competitive products is obtained from publicly obtained information and is subject to change without notice. Please contact the manufacturer for the most recent information. The following terms are trademarks or registered trademarks of IBM Corporation in the United States, other countries or both: Active Memory, Active PCI, AT, BladeCenter, the e-business logo, EasyServ, Enterprise X-Architecture, EtherJet, HelpCenter, HelpWare, IBM RXE-100 Remote Expansion Enclosure, IBM XA-32, IBM XA-64, IntelliStation, LANClient Control Manager, Memory ProteXion, NetBAY3, Netfinity, Netfinity Manager, Predictive Failure Analysis, RXE Expansion Port, SecureWay, ServeRAID, ServerProven, ServicePac, SMART Reaction, SMP Expansion Module, SMP Expansion Port, UM Services, Universal Manageability, Update Connector, Wake on LAN, XceL4 Server Accelerator Cache, XpandOnDemand scalability. IBM Corporation Subsidiaries: Lotus, Lotus Notes, Domino, and SmartSuite are trademarks of Lotus Development Corporation. Tivoli and Planet Tivoli are trademarks of Tivoli Systems, Inc. LLC, Adobe, and PostScript are trademarks of Adobe Systems, Inc. Intel Celeron, LANDesk, MMX, Pentium II, Pentium III, Pentium 4, SpeedStep, and Xeon are trademarks or registered trademarks of Intel Corporation. Linux is a trademark of Linus Torvalds. Microsoft Windows and Windows NT are trademarks or registered trademarks of Microsoft Corporation. Other company, product, and service names may be trademarks or service marks of others. For more information, visit: http://www.ibm.com/legal/us/en/copytrade.shtml.

Servicing the IBM Flex Chassis (8721) Preface


Preface Servicing the IBM Flex Chassis (8721) This document may not be copied or sold, either in part or in whole, to non-IBM personnel. Please write your name and address below to personalize your copy.

Issued to: Address:

Current release date: November 2011 Current release level: 1.00

The information contained within this publication is current as of the date of the latest revision and is subject to change at any time without notice. Please forward all comments and suggestions regarding the course material format and content to your local IBM System x Service and Support Education country coordinator or contact.

Servicing the IBM Flex Chassis (8721) Table of Contents


Table of Contents Preface ............................................................................................................................ 3

Servicing the IBM Flex Chassis (8721) ....................................................................... 3

Table of Contents ........................................................................................................... 4 Prerequisites ............................................................................................................... 5 Objectives ................................................................................................................... 6

Servicing the IBM Flex Chassis (8721) ......................................................................... 7 Overview ..................................................................................................................... 7

Flex Chassis Front View ........................................................................................ 7 Flex Chassis Inside View ...................................................................................... 8 Flex Chassis Rear View ........................................................................................ 9 Product Description ............................................................................................. 10

IBM Flex Chassis (8721) Features and Specifications ............................................. 19 Problem Determination and Troubleshooting ........................................................... 41

Summary ....................................................................................................................... 42

Servicing the IBM Flex Chassis (8721) Prerequisites and Objectives


Prerequisites A curriculum is a combination of specific roadmaps that are attempted in a logical sequence. Each roadmap is a set of course modules that are also attempted in a logical sequence. A brand curriculum consists of the roadmap types listed below. Attempting the courses in each roadmap in the sequence presented affords the best change of successful completion. Access the complete IBM Flex System curriculum using the following Global Support Enablement University (GSEU) link:

https://w3-connections.ibm.com/wikis/home?lang=en#/wiki/Global%20Support%20Enablement%20University/page/Flex%20MTS%20RSC%20and%20SSR%20Combined%20Curriculum

Prerequisites: Prerequisite courses provide basic concepts in technologies, service tasks, and products. All prerequisites should be completed prior to attempting subsequent roadmaps in the curriculum.

Fundamentals: Fundamental courses describe in general the architectures, technologies, capabilities, and limitations of the specific product brand. Fundamental courses should be attempted prior to problem determination and product specific course modules.

Point of Reference Training: Point of Reference Training (PORT) is the use of the brand external product documentation to learn about the products as do our customers. This training is designed to provide service personnel the customer's point of reference for the products they have installed.

New Technologies: New technologies courses introduce new major concepts not previously seen within the brand. Over time, new technologies courses are integrated into the fundamentals courses.

Access and Interface Training: Access and Interface (A and I) training modules detail the ability to access service data and the use of typical application interfaces, such as system management, service management, and configuration management.

Problem Determination Training: Problem Determination (PD) training details the processes and tasks used to isolate failures and generate a resolution plan for the products covered within the curriculum.

Product and Options modules: Product and Options course modules provide specific information for each product covered within the curriculum. These course modules specifically relate to machine types and option part numbers, and provide the details of a particular product.

Service Task Training: Service task training describes the processes and practices used by service personnel to perform problem determination, problem isolation, and action plans to resolve customer issues. Service task training is typically hands on classroom or virtual exercises and practice sessions using brand products.

Servicing the IBM Flex Chassis (8721) Prerequisites and Objectives


Compute Environment Training: Compute environment training details the typical customer use of the products within the curriculum, as well as external factors that can cause non obvious issues with these products.

Education Assist Elements: Education Assist (EA) elements include installation and removal videos, simulators, trifold summaries, and self-running product or application demonstrations. EA elements are designed to be used as daily work aids, not as stand-alone course modules.

Curriculum Assessments: Assessments for each course module within the curriculum are provided at the completion of the course module. Curriculum assessments are taken when a large majority of the curriculum has been completed.

The prerequisites for this course include the roadmaps and courses listed below:

http://xsnsftp.raleigh.ibm.com/docs/university3/prereq_sum_bycrs.asp?crscode=GX60607

Objectives Upon completion of this course, you will be able to:

1. Provide an overview of the Flex chassis. 2. Describe the components within the Flex chassis and their locations. 3. Highlight some similarities and differences between the IBM BladeCenter and the

IBM Flex System. 4. Understand the chassis lightpath and LEDs. 5. Describe power and cooling of individual components within the chassis. 6. Perform problem determination activities and troubleshoot the Flex chassis.

Servicing the IBM Flex Chassis (8721) Overview


Servicing the IBM Flex Chassis (8721) Overview The Flex chassis is 10U tall. It features fourteen horizontal node bays in the front. At the rear, it can hold up to four switches, two chassis management modules (CMM), six power supplies, and ten fans. This new chassis was designed to support technology advancements through the next several years. Technology includes future processors, memory, I/O, storage, and specialty hardware. In addition, the Flex chassis features faster internal bus and I/O speeds, energy efficient cooling and power system, high efficiency components, and sophisticated control systems. The Flex chassis is compatible with future high efficiency data center infrastructure and features new compute and storage nodes with new scalable switches and I/O adapters. Lastly, there is the Flex System Manager (FSM), which is a compute node that has the capability of managing multiple Flex chassis.

Flex Chassis Front View

Figure 1 Flex Chassis, Front View



Flex Chassis Inside View

Figure 2 Flex Chassis, Front Inside View

Figure 3 Flex Chassis, Rear Inside View



Flex Chassis Rear View

Figure 4 Flex Chassis, Rear View



Product Description At the time of this course, Flex chassis accepts x86 Intel and Power compute nodes. It can be managed by an FSM and comes with a minimum of one CMM, but is capable of containing a maximum of two. I/O modules are in a different form factor then BladeCenter and still come from multiple vendors such as Cisco and Emulex. Comparison can be made between BladeCenter H and Flex chassis.

Table 1 BladeCenter H & Flex Chassis

BladeCenter H Flex

Height (U) 9 10

Blades/compute node bays 14

7 full-wide or 14 half-wide compute nodes

Switch bays 4 LSSM + 4 high speed 4 high speed scalable switches

Blade/compute node pitch (mm) 30 57

Chassis height 395 mm (15.55 in) 440 mm (17.32 in)

Chassis depth 737 mm (29 in) 801 mm (31.5 in)

Chassis width 444 mm (17.5 in) 447 mm (17.6 in)

System weight (maximum) 164 kg (360 lb) 205 kg (450 lb)



The Flex chassis has an outer skeleton. Inside there are compute node bay shelves, midplane, fan distribution cards, personality card, rear LED card, fan logic module, fillers (node, power, switch, CMM, fan), fans, and power supplies. Some of these are labeled in Figure 5 below.

Figure 5 Flex Chassis Components



The following figures, 6 and 7, display front and rear chassis locations for Flex. Bay 1 starts at the bottom left of the chassis and ends with bay 14 in the top right front side corner. At the rear, it displays locations for four switches, two CMMs, six power supplies, and ten fans.

Figure 6 Flex Chassis Bay Locations



Figure 7 Flex Chassis Rear Power Supply, Fan, and Switch Bay Locations



The Flex chassis accepts multiple node configurations: half-wide (standard), full-wide, and specialty. A specialty node can occupy up to four standard slots, which is the equivalent of two full-wide nodes stacked on top one another. Node examples are provided in Figure 8.

Figure 8 Flex Chassis Node Examples



Depending on the node, Flex can hold fourteen, seven, or three per chassis.

Figure 9 Node Form Factors



There are four LEDs on the front panel, all located at the bottom left corner of the chassis. The front customer interface panel card has chassis information LEDs and the fault LED light-up button. When the combination fault LED button is in deferred maintenance mode, the fault LED of a failed component (e.g., compute node, switch, power supply, fan, etc.) does not light. Pushing the button illuminates the fault LED of all failed components, which are visible from the front, or rear of the chassis. Internal fault LEDs (i.e., board LEDs next to DIMMs) does not light until activated by internal lightpath switch (todays blue marked button on the system board).

Figure 10 Flex Chassis Information LEDs

There is a personality card in addition to the front panel card. The personality card provides two EEPROMs that contain chassis manufacturing vital product data (VPD) and the remaining space is for CMM control and usage. This card is also the control point for the front and rear service indicators. The state of the front chassis fault switch is via the personality card. Two temperature sensors on the personality card provide inlet temperature information from the switches to the CMMs. The personality card interfaces via a cable to the front panel card which provides a light up logo (used to indicate power on), a blue location LED, and a combination fault LED / switch.



The personality card logic is power by 3.3V received from the CMMs. If there is no CMM in the system, the personality card forces the fault LED to turn on.

Figure 11 Personality Card



Figure 12 Flex Chassis Block Diagram

Servicing the IBM Flex Chassis (8721) Features and Specifications


IBM Flex Chassis (8721) Features and Specifications The Flex chassis is 800.5 mm deep, 440 mm tall, and 447 mm wide. Connections exist through the midplane. Those connections are achieved with mezzanine cards (daughter cards), onboard node Ethernet connections, CMMs, and connections to fans and power. Horizontal node packaging allows maximum board size to include two sockets and twenty-four DIMM slots on a standard compute node. A 57 mm node pitch provides low airflow impedance for energy efficiency in the design. Flexible node form factors within the chassis support half- and full-width boards, double-high nodes, and interconnection of expansion boards.

Figure 13 Flex Chassis, Front View



Figure 14 Flex Chassis, Rear View

CMM and I/O modules come in a new form factor due to the Flex systems chassis design. The CMM is now 183.5 mm wide, 19.6 mm tall, and 296mm deep while lying flat. The new I/O modules are 401.6 mm wide, 30 mm tall, and 296 mm deep while lying flat. The I/O module is vertically oriented when inserted into the chassis and its connections line up at the midplane.



Figure 15 CMM

Figure 16 I/O Modules

The fan logic module plugs into the fan distribution card. This distribution card is connected at the midplane. It contains connectors for the 40 mm fan pack cable and four 80 mm fan pack power and signal connectors.



Figure 17 Fan Logic Module



Figure 18 Fan Distribution Card



Both 40 mm and 80 mm fan packs are dual sourced. One comes from Nidec and the other from Delta. Also notice that the 80 mm fan pack is keyed for orientation. The orientation is important because of the fan packs square shape.

Figure 19 40 mm Fan Pack



Figure 20 80 mm Fan Pack

The Flex chassis is optimized for 3-phase, 60A, 200VAC. It contains up to six 2500W power supplies and supports both N+N and N+1 power redundancy. There is efficient balancing of N+N power across all three phases. The power supply is sized to efficiently use PDU power. This chassis is compatible with 1-phase 60A, but using 1-phase 60A is an ineffective use of power. These power supplies are certified to 80 Plus Platinum. There is low power loss across the midplane and connectors. Features enhanced power capping and management with expanded power monitoring capabilities. They monitor both AC and DC power. Power monitors are located on all major components in the chassis (e.g., nodes, fan packs, switches, and CMM). There is a single power domain. The midplane consists of only one power plane common to all chassis elements. It is highly reliable with no active components. The power supply is designed to provide fault isolation.



Figure 21 2500W Power Supply



Figure 22 Flex Chassis Power (3-phase, 60A, 200VAC)



Figure 23 Four Flex Chassis Configured in a Rack Power (3-phase, 60A, 200VAC)



Figure 24 Flex Chassis Power (1-phase, 60A, 200VAC)

Flex takes advantage of zoned cooling for nodes (right / left) and switches. Fans may be added as more nodes are added for scale up/out cooling. The chassis features easy path fan upgrades. There is a standardized fan interface that gets coded (no code change if the fan is changed). Adaptive cooling controls minimize system cooling power consumption and airflow requirements. The cooling controls are node based rather than chassis based. Chassis cooling is optimized as a function of node configurations, power monitor circuits, component temperatures, and ambient temperature. This optimization results in lower Cubic Feet per Minute (CFM) and cooling energy spent at the chassis level and maximizes the delta T across the chassis for more efficient room level integration. The Flex chassis has enhanced acoustic optimization with multi-select states. States one to five are customer selectable acoustic states that balance acoustic and performance needs. The chassis has industry leading fan efficiency and improved serviceability features. The high efficiency node dampers tightly seal the node bay when a node is removed. A chassis level CFM usage is available to customers for planning purposes. The ambient health awareness console detects hot air recirculation to the chassis.



Table 2 BladeCenter H & Flex Chassis Cooling

Cooling BladeCenter H Flex

Low speed / high speed fan operating conditions altitude at which values are determined

LS sea level HS 7000ft / high power

blades

LS sea level HS 10000ft / high power nodes

Environmental support (operating temperature and altitude)

10-32 C / 7000ft 10-35 C / 3000ft

5-28 C / 3000ft 5-40 C / 3000ft

Air moving devices (AMD)

2 forward-curved blowers and 12 power supply (PS)

fans

8 x 80 mm axial flow, counter rotating fans and 12 PS fans

AMD speed control method

Linear based on inlet fan fail, PS fail, or CPU over

temp

Node-based cooling controls given configuration, power, ambient inlet

and component temp dependencies

Altitude compensation Yes pressure sensor Intelligent sensor controls



Figure 25 Node Cooling Path



Figure 26 I/O Module / CMM Cooling Path



Figure 27 Power Supply Cooling Path

Servicing the IBM Flex Chassis (8721) System Configurations and Diagrams


System Configurations and Diagrams A base model 8721-A1Y has a CMM, two power supplies, fillers (thirteen node, four power, three switch, one CMM, four fan), one breakout cable, two power distribution units (PDU) jumper cables (two meter), documentation, rack kit, packaging and labels.

Figure 28 Flex Chassis



Figure 29 Midplane, Front View



Figure 30 Midplane, Rear View



Figure 31 Node Bay Shelf, Flex Chassis, Shuttle



Figure 32 CPU Node I/O Options: LOM or Mezzanine Card



Figure 33 Switch Interconnect



There are 3 different power policies: 1. Basic Power Management

a. Allows the chassis to fully use power supply power, i.e., no N+N or N+1 redundancy

2. Power Supply Redundancy a. Assumes no node throttling upon a power supply fault b. Most restrictive redundancy mode

3. Power Supply Redundancy with Throttling a. Allows power supplies to operate beyond their rating for a brief period of

time b. Nodes must throttle within 20ms upon a power supply fault

Figure 34 Flex AC and DC distribution (N+N configuration)

Servicing the IBM Flex Chassis (8721) Problem Determination


Problem Determination and Troubleshooting Flex chassis problem determination methodology is similar to the BladeCenter environment. Obtain the service data log information from either the CMM or FSM. The most efficient way to populate the chassis with nodes is from the bottom up. For hardware, ensure that all connectors are intact and the correct parts are in the proper locations according to installation and configurations guides. Lightpath is used to alert the end user of a condition that requires attention. Not all notifications are fault based. Some instances that are not fault based include, but are not limited to information notifications, power good, and locator LEDs. Before removing the flex cable connecting the personality card and the chassis customer interface panel, attach a 1m string to the front of the cable. This string saves time in the event removal and installation is necessary. Keep in mind that there is firmware for the fans that is verified and updated automatically by the CMM. The fan firmware versions may differ when replacing Flex chassis fans.

Servicing the IBM Flex Chassis (8721) Summary


Summary This course has enabled you to:

1. Provide an overview of the Flex chassis. 2. Describe the components within the Flex chassis and their locations. 3. Highlight some similarities and differences between the IBM BladeCenter and the

IBM Flex System. 4. Understand the chassis lightpath and LEDs. 5. Describe power and cooling of individual components within the chassis. 6. Perform problem determination activities and troubleshoot the Flex chassis.

Documents

gx60607r100