Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

An Oracle White Paper

March, 2011

Oracle Exalogic Elastic Cloud: X2-2 Hardware Overview

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

Disclaimer

The following is intended to outline our general product direction. It is intended for information

purposes only, and may not be incorporated into any contract. It is not a commitment to deliver

any material, code, or functionality, and should not be relied upon in making purchasing

decisions. The development, release, and timing of any features or functionality described for

Oracle‟s products remains at the sole discretion of Oracle.

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

1

Introduction Balancing cost, performance, complexity in engineering the most optimal hardware and

software system is non-trivial. Regardless of approach, the end result may not yield the

desired outcome. As a result, IT has started the inevitable shift to tightly integrated

engineered systems as a cost effective simple way of obtaining an optimized purposefully

built system that considers all of the traditional variables. In addition, engineered

systems offered by systems companies offer improved quality and efficiency over custom

systems built and assembled ad-hoc within IT departments.

Software vendors, by and large, write software to function on a variety of platforms varied

by device drivers, operating systems, chip sets, network and storage devices. Java™

made this considerably simpler by abstracting the hardware layer away from the

developer offering a development platform well suited to building portable enterprise

class applications regardless of hardware and operating system. However, this portability

has come at a cost, specifically performance. While many applications built on Java™

perform extremely well, tuning applications is often an expensive and time consuming

process.

Exalogic Elastic Cloud takes performance of Java™ and Java EE™ based applications to

the next level by offering a platform optimally suited for deployment of Java™ based

applications with a particular focus on WebLogic server and the rest of Oracle Fusion

Middleware.

This paper focuses on the hardware aspects of Exalogic Elastic Cloud, including the

selection process and rationale, as well as technical insights into what systems

modifications have been engineered to offer a system optimally suited for middleware

workloads.

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

2

Hardware: Foundation for Acceleration

Hardware Design Choices

Specifying and selecting the proper hardware is a time consuming and labor intensive process,

assuming that testing of the various configurations is performed. The Exalogic engineering staff

has conducted extensive testing on a wide range of hardware configurations to arrive at the

optimal configuration for middleware type deployments. Rationales for these decisions are

outlined below.

Design Goals

The following design goals guided the system architecture of Exalogic Elastic Cloud X2-2:

Density: A key driver in the decision making process was to architect a system that provided a

high level of compute density which is necessary for both high performance applications and for

the use case of consolidating disparate middleware and application deployments.

“State of the art”- Since Exalogic was to represent the best general purpose middleware

deployment platform, it was necessary to pursue the most advanced technologies available in the

Sun/Oracle hardware portfolio. This is evidenced by the choices in all component areas:

compute, network and storage.

Balance: Architecting a high performance system is not necessarily a difficult proposition.

However, designing a high performance system that is balanced in terms of power consumption,

storage, compute, cost, serviceability etc. is non-trivial. Exalogic puts forth a balanced system

design that considers all of the relevant system design factors suited for middleware type

workloads.

Serviceability: The ability to field service the system is extremely important when it comes to

providing a fault tolerant system. The ability to perform reliable field upgrades1 as well as the

swapping of failed components is important in maintaining uptime in the overall system.

Serviceability is not only evident in the mundane (but important) aspect of how the cable design

has manifested itself in Exalogic, but also the choice for individual hardware components such as

1 For example, the ability to upgrade a quarter rack to a half rack or from a half rack to a full rack configuration is intended to be performed in a reliable predictable fashion.

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

3

redundant InfiniBand switches and storage heads that can be replaced without requiring any

downtime.

High performance networking: At the center of any clustered super computer sits a high

performance network; preferably a scalable fat-tree network such as InfiniBand, enabling high

speed communication between constituent applications and components. As described below,

InfiniBand is a key technology behind the success of Exalogic as it provides a reliable high

performance network interconnect while at the same time offloading compute cycles from host

CPUs further increasing performance of the overall system.

Consolidation: A primary goal was also to provide a platform optimally suited for the

consolidation of heterogeneous applications. This implies the ability to support multiple

application stacks running on various operating systems.

Compute Node

Selecting a compute system optimally suited for high performance Java workloads leads to

selecting a processor that has the ability to efficiently support multiprocessing. Complimentary to

this, is the ability to efficiently be able to support multiple hardware threads utilized by the

executing Java software stack which tends to be highly threaded. The compute node within

Exalogic is comprised of multiple 1U X4170 M22 compute nodes which hosts dual socket 5600

Westmere series rack optimized Intel Xeon processor. Each processor consists of 6 cores for a

total core count of 12 cores per compute node with 24 hyper-threads allowing for highly

concurrent workloads to execute efficiently.

The specific configuration of X4170 M2 includes twelve 8 GB DIMMs (Dual in-line memory

modules) and two 32 GB SATA high performance high MTBF solid state disk drives configured

in RAID 1 with redundant fans and redundant power supplies. The end result is a high

performance, fault tolerant compute node with no moving parts besides the dual redundant fans.

Besides high performance processors, Java benefits from high performing RAM to efficiently

handle frequent memory allocation and garbage collection. For this reason, the X4170M2 is

equipped with exactly 96GB of RAM; the maximum amount of RAM enabling the higher

memory bus speed of 1333 MHz. Although the X4170M2 has a max capacity of 144GB,

increasing memory beyond the 96Gb barrier requires a 40% drop in the memory bus speed to

800Mhz which is sub-optimal for Java based applications. The end result is 8GB of RAM per

2 Sun Fire X4170 Data Sheet: http://www.oracle.com/us/products/servers-storage/servers/x86/sun-fire-x4170-m2-ds-079875.pdf

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

4

core, which is more than enough for the most memory intensive applications. Figure 1 and figure

2 illustrate the front and back view of the physical X4170M2 unit.

FIGURE 1: COMPUTE NODE FRONT VIEW3

FIGURE 2: COMPUTE NODE REAR VIEW

Storage

In order to facilitate use cases that involve high levels of elasticity and high availability, Exalogic

is outfitted with the 7320 ZFS storage appliance4. As individual compute nodes are intended to

only host operating system binaries, the 7320 storage appliance stores all other application

binaries, log files and content necessary for the applications to execute. The compute nodes

3 The X4170M2 Exalogic Elastic Cloud configuration does not include a DVD drive

4 See 7320 Data Sheet: http://www.oracle.com/us/products/servers-storage/zfs-storage-family-ds-173093.pdf

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

5

mount configured partitions of the 7320 to access necessary data over the InfiniBand network.

The notion of mounting the 7320 networked attached storage device eliminates the need to back

up individual compute nodes and in the event of failure of a compute node, the partition can

simply be mounted from another compute node while the failed node can be replaced.

The 7320 supports multiple protocols (NFS v2/v3/v4, CIFS, HTTP, WebDAV,

FTP/SFTP/FTPS) and are all made available for use. However, NFS on Internet Protocol over

InfiniBand (IPoIB) is the recommended configuration as a result of extensive reliability and

performance testing. More information on this topic can be found in the Oracle Exalogic

Enterprise Deployment Guide5.

The 7320 appliance in Exalogic contains 20 2TB drives which can be configured in numerous

ways depending on high availability needs and desire for replication. Typically, two 2TB drives

are set aside as „hot‟ spares in case of drive failure. Any drive failures are automatically handled by

the storage system through re-partitioning with the spare drive being put into service.

Figure 3 shows an illustration of the 7320 ZFS storage appliance. As shown, the two redundant

storage heads (Processor 1 and Processor 2) contain 2TB of high performance read cache, often

referred to as „readzillas‟. These caches store data retrieved through previous reads performed on

the spinning disks contained in the JBOD6 tray. The cache significantly enhances performance of

reads as access to slower spinning drives are not necessary.

Complimentary to the readzilla caches is the logzilla cache co-located in the storage tray. This

cache is a highly performant write cache which allows writes to the appliance to be done with

minimal latency. Data that is written to the logzilla cache is asynchronously written to the

spinning drives.

5 The Exalogic Enterprise Deployment Guide can be found here: http://download.oracle.com/docs/cd/E18476_01/doc.220/e18479/toc.htm

6 Just a Bunch of Disks

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

6

FIGURE 3: 7320 STORAGE APPLIANCE FRONT VIEW

Networking

During the last several years, chips have not become significantly faster in terms of clock speed.

However, chips are able to deliver on much higher workloads through denser core counts. This

trend is likely to continue based on current roadmaps from microprocessor leading vendors7.

The implication of denser core counts is that each processor is able to perform more work.

Therefore each compute node is likely to demand escalating network resources in the form of

remote method invocations, file transfers and database interactions. The network can easily

become a bottleneck from both a throughput and latency perspective. In a traditional IP over

Ethernet network configuration, the compute node requesting network resources is involved in

the processing of the top layers in the TCP stack, requiring additional compute resources for all

network interactions. Providing a high performance network as the foundation for a high

performance system is fundamental in constructing a highly performant system with today‟s high

core count processors. It is for this reason that both Exalogic and Exadata are built from the

ground up using InfiniBand.

7 See Intel‟s microprocessor roadmap: http://download.intel.com/products/roadmap/roadmap.pdf and Oracle‟s SPARC roadmap: http://www.oracle.com/ocom/groups/public/@ocom/documents/webcontent/044518.pdf

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

7

The InfiniBand Trade Association (IBTA) was founded in 1999 is led today by Oracle, Intel,

IBM, Mellanox, QLogic and Voltaire. The IBTA is responsible for the advancement of the

InfiniBand specification. The goal of the IBTA is to put forth a communication link suitable for

use in high performance computing systems such as Oracle Exadata and Oracle Exalogic.

Comparing state of the art 10Gb/s Ethernet networks with InfiniBand Quad Data Rate (QDR)

shows us the advantage of InfiniBand as a high performance network infrastructure.

In addition to raw network performance and reduced latency, the compute node does not need

to be concerned with TCP management resulting in more cycles spent executing the actual

application.

While InfiniBand is a high performance network, all applications are not able to take advantage

of the native features offered by its infrastructure, namely the sockets direct protocol (SDP), a

lower level protocol enabling „writing directly to the wire‟. This requires an InfiniBand aware

software stack such as Oracle Fusion Middleware or Oracle Database.

Although most applications cannot take advantage of the native InfiniBand SDP protocol, the

Open Fabrics Enterprise Distribution (OFED) stack that ships with Exalogic provides an IP

compatible implementation known as Internet Protocol over InfiniBand (IPoIB). IPoIB is an IP

network interface implementation operating over InfiniBand allowing all InfiniBand unaware

applications to communicate unchanged over InfiniBand. Applications utilizing this compatibility

network stack see significant benefit of the InfiniBand network through higher throughput and

lower latency compared to a 10Gb/sec Ethernet network. Applications running on compute

nodes communicate with applications on other compute nodes using this InfiniBand network.

Interconnected Exalogic and Exadata Machines communicate via IPoIB. Exalogic machines can

be connected to an external network, including a standard database hosted on a machine outside

of the Exalogic machine, via the InfiniBand-to-10 Gb Ethernet gateways using Ethernet over

InfiniBand (EoIB). Each Exalogic machine configuration includes at least 2 such gateways, which

also act as InfiniBand switches connecting all compute nodes and the Sun ZFS Storage 7320

appliance within the Exalogic machine.

InfiniBand Switches

Within Exalogic, the InfininBand gateway switches provide a shared, high speed network for

application clusters that are comprised of compute nodes and the shared storage appliance. The

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

8

gateway switch features a standard 1u data center form factor, thirty two 40Gb InfiniBand ports,

eight 10Gb Ethernet ports, and the Sun ILOM8 management interface.

Exalogic‟s converged fabric leverages the properties of the InfiniBand architecture to enable

high-performance in demanding, clustered data center environments. The fabric supports the

creation of logically isolated network partitions, as well as advanced features for traffic isolation

and quality of service (QoS) management. QDR InfiniBand delivers 40 Gigabits of raw

bandwidth. The InfiniBand base rate is 10 Gigabits per second. QDR capable products deliver

four times the base rate.

Deploying a converged fabric minimizes the cost and complexity of building and operating

scalable systems by reducing the number of adapters, cables, and switches to install, manage, and

maintain. LAN traffic carried over Oracle's converged data center fabric uses familiar Ethernet

semantics. Furthermore, the LAN ports on the Sun Network QDR InfiniBand Gateway Switch

appear as endpoints to the LAN, thus avoiding potential interoperability issues with network

management tools and methods.

Within the Sun Network QDR InfiniBand Gateway Switch, the embedded fabric manager is

enabled to support active/standby dual manager configurations, ensuring a seamless migration of

the fabric management services in the event of a module failure. The switch is also provisioned

with redundant power and cooling for high availability.

The front panel of the Sun Network QDR InfiniBand Gateway Switch has 36 QSFP connectors

(See Figure 4). The connectors are each dedicated to InfiniBand or Gateway functions as follows:

The “left-most” thirty-two ports support QDR InfiniBand connectivity.

The two “upper right-most” ports support Ethernet connectivity.

The two “lower right-most” ports are covered with a plug marked “Do Not Remove”. No

connections should be made to these ports.

8 See http://www.sun.com/systemmanagement/ilom.jsp for more information.

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

9

FIGURE 4: INFINIBAND SWITCH

The switch has two physical ports dedicated to Ethernet functionality. These ports can support

up to four 1/10 Gb Ethernet connections each. Oracle provides “splitter” cables which fan out a

single QSFP (Quad Small Form Pluggable) switch port to four optical fiber LC connectors. A

QSFP transceiver is available to connect the “splitter” cable to the switch.

Network Extensibility

All X4170 compute nodes and the 7320 storage appliance are connected via the fully switched

InfiniBand backplane. If desired, additional Exalogic and Exadata can be added to the network

fabric to facilitate high performance computing across eight full racks without requiring

additional networking hardware such as a data center switch. A common use case is the

integration of Exalogic and Exadata to form a high performance compute system combining raw

compute power with Online Transaction Processing (OLTP).

Operating System

Exalogic provides users with a choice of Oracle Linux or Oracle Solaris operating systems.

Solaris is the number one UNIX operating system in the market and Oracle continues to invest

in Oracle Linux. Exalogic is 100% compatible with all standard Oracle Linux 5 and Solaris 11

applications. To this end, no special certification for Exalogic is required. As a result, all Oracle

applications that are certified for the appropriate releases of Oracle Linux and Solaris are

supported on Exalogic.

When Oracle released Exadata and Exalogic, Oracle tested those servers with the old Red Hat

compatible kernel and found that the OS based on the 2.6.18 kernel was not able to drive the

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

10

system given the high performance networking and storage infrastructure. In order for Exalogic

to leverage the hardware, SDP and IPoIB, a modern OFED stack was needed. This was not

available with RHEL and therefore set out to evolve the RHEL kernel. To produce UEK

(Unbreakable Enterprise Kernel), Oracle began with a stable release of the 2.6.32 kernel,

representing a major leap forward from the RHEL5 2.6.18 kernel. This kernel included

enhancements focused on performance and stability. Considerable work went into ensruing that

heavy network and IO loads could be spread across all CPUs in order to optimize RDS and

InfiniBand performance. UEK represents the best Linux performance Oracle can deliver today.

By moving to 2.6.32 as base for the kernel many features in the areas of hardware fault

management, data integrity and diagnostics are advanced, such as

Detecting and logging hardware errors before any affect to OS or application

Automatic isolation of defective CPUs and memory

Avoiding system crashes

Improved application uptime

Taking advantage of these enhancements, require no changes to existing applications. The

optimizations provide up to 12x InfiniBand performance, 60% higher workload, 50% reduced

latency, and 50% improved IPoIB performance9.

Oracle Solaris 11 Express10 raises the bar on the functionality introduced in Oracle Solaris 10,

continuing Oracle's leadership for providing the best choice in mission critical operating systems

for enterprise environments. As a result of a host of new features such as:

Network based package management tools to greatly decrease planned system downtime and

provide for a complete safe system upgrade

Built-in network virtualization and delegated administration for an unprecedented level of

flexibility for application consolidation

Continued leadership for providing the highest levels of security

Oracle Solaris 11 Express is simply put, the most exciting release of the Oracle Solaris platform

to date.

9 See Oracle Exalogic Elastic Cloud: A Brief Introduction for optimization details http://www.oracle.com/us/products/middleware/exalogic-wp-173449.pdf

10 For detailed documentation on Oracle Solaris 11 Express see

http://www.oracle.com/technetwork/server-storage/solaris11/overview/index.html

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

11

Oracle Solaris Zones is built-in OS virtualization which continues to build on its long and

distinguished pedigree. One of the most highly adopted, highly used, mature virtualization

technologies, Oracle Solaris Zones was first introduced as a core part of Oracle Solaris 10. With

Oracle Solaris 11 Express, Oracle Solaris Zones11 has become even more central to both the

application and the end user, and continues to introduce enhancements with new improvements

and features including:

Integration into the new packaging system

Improved observability

Increased control over administration

Tight integration with ZFS

Support for Oracle Solaris 10 Zones

Integration with the new Oracle Solaris 11 network stack architecture

Best in class manageability

Oracle Enterprise Manager provides application-to-disk management through Enterprise

Manager Grid Control and Enterprise Manager OpsCenter. Enterprise Manager allows every

individual hardware component within an Exalogic deployment to be monitored in real time and,

at the customer‟s option, have system status automatically reported to Oracle Support for

proactive system maintenance. Through integration with Oracle Support, Enterprise Manager

can apply tested patch bundles tailored for Exalogic that cover every layer of the system, from

device firmware and operating system to JVM, application server, upper-stack Fusion

Middleware, and Oracle applications.

11 See Oracle Solaris Virtualization:

http://www.oracle.com/technetwork/articles/servers-storage-admin/sol11evirt-186209.pdf

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

12

FIGURE 5: EXALOGIC ELASTIC CLOUD MANAGEMENT INFRATRUCTURE

Enterprise Manager Grid Control (EMGC)

Enterprise Manager Grid Control focuses on managing the software assets deployed on

Exalogic, with particular attention to Fusion middleware and Oracle applications. Exalogic

specific functionality allows representation, management and monitoring of Exalogic

deployments.

Enterprise Manager Operations Center (EMOC)

Enterprise Manager Operations Center is Oracle's single, integrated solution for managing all

aspects of the data center. It allows for the discovery, provisioning, update, management, and

monitoring of the physical and virtual assets in multiple data centers from a single console.

Enterprise Manager Ops Center is an enterprise management solution that sits off the servers

and communicates to the hardware assets through the ILOM (Integrated Lights Out Manager).

Operating System agents can also be deployed for insight into Operating System level metrics.

The remote management capabilities are designed to help increase availability and utilization and

minimize downtime. Although Oracle Enterprise Manager is optional in the Exalogic machine

environment, Oracle recommends the use of EMGC to monitor deployed software assets and

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

13

EMOC to monitor Exalogic hardware components. Oracle Enterprise Manager 11g Ops Center

supports the following key features in the Exalogic machine environment:

Hardware lifecycle management

InfiniBand fabric views and Ethernet network view

Console access to launch the browser user interface for managing the Sun ZFS Storage 7320

appliance and the InfiniBand switches

Serial console access to the service processors of compute nodes, switches, and storage

appliance

Integration with Oracle Services such as My Oracle Support

Problem management for the Sun ZFS 7320 Storage appliance

In addition, Oracle Enterprise Manager Ops Center supports bare metal provisioning, discovery

of hardware assets, patch automation, import of firmware images, creation policies, and firmware

upgrade for the hardware components of an Exalogic machine.

ILOM (Integrated Lights Out Manager):

Oracle Integrated Lights Out Manager (ILOM) provides advanced service processor (SP)

hardware and software that can be uses to manage and monitor Exalogic components, such as

compute nodes, gateway switches, storage appliance, InfiniBand switches and power distribution

units. ILOM's dedicated hardware and software is preinstalled on all Exalogic components.

ILOM enables the ability to actively manage and monitor compute nodes in the Exalogic

machine independently of the operating system state, providing a reliable Lights Out

Management (LOM) system. The ILOM service processor (SP) runs its own embedded operating

system and has a dedicated Ethernet port, which together provides out-of-band management

capability. In addition, ILOM can be accessed from the compute node's operating system. Using

ILOM, compute node can be remotely managed, effectively providing a virtual KVM switch. The

ILOM management interface is integrated with Oracle Enterprise Manager Ops Center

Summary

The need for enterprise IT organizations to provide next-generation cloud features such as elastic

capacity while meeting ever more demanding performance and reliability requirements is driving

demand for a new approach to infrastructure. Whether workloads are Web-based or thick-client,

whether data-intensive or processing-intensive, whether homogeneous or highly heterogeneous,

the key to success is hardware and software engineered together for performance, reliability, and

scale. Building or using custom, special purpose systems for different applications is wasteful and

expensive. Oracle Exalogic Elastic Cloud, the world‟s first and only integrated middleware

Oracle White Paper—Oracle Exalogic Elastic Cloud X-2 Hardware Overview

14

machine, dramatically surpasses alternatives and provides enterprises the best possible

foundation for running applications.

By consolidating applications to Oracle Exalogic Elastic Cloud, enterprises will:

Accelerate the performance of Java applications by as much as 10X

Improve reliability and scalability beyond even the most mission-critical requirements

Reduce deployment effort by up to 95% and reduce costs by as much as 60%

We invite you to begin your datacenter transformation with Exalogic today.

White Paper Title

February 2011

Author: Erik Bergenholtz

Contributing Authors: Michael Palmeter

Oracle Corporation

World Headquarters

500 Oracle Parkway

Redwood Shores, CA 94065

U.S.A.

Worldwide Inquiries:

Phone: +1.650.506.7000

Fax: +1.650.506.7200

oracle.com

Copyright © 2009, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and

the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other

warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or

fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are

formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any

means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective

owners.

0109