View
16
Download
0
Category
Preview:
Citation preview
DESIGN AND ARCHITECTURE OF ORACLE RAC/ASM CLUSTERED SOLUTIONArmando RodriguezAdvisory Systems Engineer, EMC armando.rodriguez@emc.com
2014 EMC Proven Professional Knowledge Sharing 2
Table of Contents
Business Challenge......................................................................................................... 3
Design Philosophy........................................................................................................... 3
Building Block Description ............................................................................................... 3
VNX MCx multi-core technology Benefits ........................................................................ 4
VNX 8000 Building Block ................................................................................................ 5
Storage Grid Design utilizing VNX MCx 8000 Building Blocks ........................................ 5
Seven important design considerations for the Storage Grid .......................................... 7
Manage efficiently by leveraging EMC Storage software tools ........................................ 9
Conclusion .................................................................................................................... 13
References .................................................................................................................... 14
Disclaimer: The views, processes, or methodologies published in this article are those of the
author. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.
2014 EMC Proven Professional Knowledge Sharing 3
Business Challenge
Demand for high performance, high capacity, workload-optimized data warehouse and analytics
solutions have never been higher. The ability to extract information from today’s large scale data
warehouse environment requires a flexible, highly scalable storage solution that provides
demonstrable return on investment required in today’s competitive environment. EMC’s VNX®
MCx Multi-Core platform provides a balance of performance, capacity, capital cost, and
operating cost that is unrivaled in the industry. This article will explore a scale out, building block
approach for a multi-cluster Storage Grid architecture utilizing VNX with an Oracle RAC (Real
Application Clusters) data warehouse solution.
Design Philosophy
The Storage Grid concept design philosophy is rooted in a few basic principles.
Achieve high levels of scale out performance or capacity by linking low cost, high
performance storage arrays together in a single turnkey solution.
Utilize low cost commodity-based components in all layers of the design and include
only the minimal required software to operate the solution.
The solution specifications should have limited variations to allow for repeatable build
processes in order to minimize time and services required to deploy. A building block
approach is the preferred method to approach the design.
Taken as a whole, the design philosophy allows for a low cost scalable storage solution to meet
a variety of Oracle RAC ASM (Automatic Storage Management) use cases. This includes
everything from high performance OLTP transactional systems to multi-petabyte data
warehouse applications. The focus of this article will be on the data warehouse use case.
Building Block Description
The EMC VNX MCx storage line includes multiple models and sizes to meet different customer
requirements. The entry-level VNXe includes both block and file virtualized into a dual controller
system. The general purpose VNX arrays start with a 250-drive model—VNX5400 —through the
multiprocessor VNX8000 system that scales to 1500 drives. The VNX8000 is the array selected
as the base building block for the scale-out Storage Grid. The rationale for this is three-fold.
1. The array scalability allows the solution to minimize the number of arrays per Storage
Grid required, easing operational management of the solution. Reducing the number of
2014 EMC Proven Professional Knowledge Sharing 4
widget one has to manage improves the administration overhead associated with large
scale out storage farms.
2. The large amount of system cache offered by this particular model allows, 256GB
(Gigabytes) per array allows for improved efficiency in managing the IO workload in
particular write bursts into the system.
3. The VNX8000 includes a powerful Intel-based Dual Socket 8 Core 2.7GHz processor
which in conjunction with the VNX multithreaded code, enables maximum scalability.
Each array has a total of 32 Intel processor cores, making it an ideal candidate as a building
block of a high bandwidth-based Oracle RAC data warehouse cluster.
VNX MCx multi-core technology Benefits
VNX MCx Multi-core technology makes the VNX platform an ideal candidate for a scale out
storage data warehouse application. The performance-optimized multithreaded technology
allows for linear I/O performance scalability utilizing all processor core available in the storage
system. Figure 1 depicts the CPU core balance utilization of each of the 16 Intel cores on the
VNX8000 platform under high bandwidth Oracle RAC I/O load, which peaks at 18GB/s in this
example.
Figure 1: CPU Core Utilization under heavy I/O bandwidth (balance I/O against all Intel cores)
2014 EMC Proven Professional Knowledge Sharing 5
VNX 8000 Building Block
VNX MCx 8000 Array
Dual Socket 8 Core 2.7Ghz Intel Processors
2-Rack Design utilizing 2.5” drives
o Balance floor space efficiency and power
o Power envelope of 2 x 30A circuits per rack
900 2.5” Drive slots per Building Block
256 GB System Memory per Array
Dual redundant system architecture
No single point of failure
Up to 25GB/s+ large block Read Bandwidth
Lithium ION Battery-backed Cache
Figure 2: VNX8000 2 Rack Design
The building block outlined in Figure 2 includes the following characteristics. Each array is
composed of 900 2.5” drive slots per building block. The VNX has many drive options available
that include 7200 RPM SATA, 10k RPM SAS, 15k RPM SAS, and Enterprise Flash Drive (EFD)
in the 2.5” form factor. All drive types are available in multiple capacity amounts and can be
chosen to optimize the capacity / performance requirements for the specific workload profile
being targeted by the design. If targeting a high capacity large scale data warehouse
application, the 900GB 10k 2.5” drive is an ideal candidate that balances performance, capacity,
density, and cost. Additional drive options can be utilized depending on unique application I/O
requirements. Factors to consider when selecting drive types include workload skew, I/O size,
read/write mix, capacity requirements, and cost constraints. Your EMC sales performance
experts can help select the right drive types for your unique requirements.
Storage Grid Design utilizing VNX MCx 8000 Building Blocks
Now, let’s compose an example solution. The requirements are for a high bandwidth Oracle
RAC data warehouse cluster using ASM to stripe the entire workload across all storage
resources. Oracle ASM enables a database to spread workload across multiple LUNs. This
software load balancing technique allows a database administrator to tie together multiple
2014 EMC Proven Professional Knowledge Sharing 6
storage resources on a single array or multiple arrays to achieve very high performance. The
bandwidth design targets in the example below are for 100GB/s composed of 95% large block
reads (256kb+). The Storage Grid is composed of 6 VNX 8000 arrays each with 900 2.5” drive
slots (5400 drives total) with three 16-node Oracle RAC clusters. Host storage Fiber Channel
connectivity is designed to allow for bursts up to 6 GB/s per node based on available HBA
bandwidth utilizing 4 x 16Gb connections per host.
2014 EMC Proven Professional Knowledge Sharing 7
Seven important design considerations for the Storage Grid
1. Balance Fiber Channel connectivity to match performance capability of the storage array
and hosts.
To maximize the build, cost, and support efficiency of the solution, do not over-build the fiber
channel connectivity of the solution. Consider the storage bandwidth capabilities of the
nodes and storage platform when making the selection. The VNX8000 is capable of
providing 72 8Gb/s fiber channel connections per array. However, 32-48 8Gb/s fiber channel
connections are more in line with the platform bandwidth capabilities and strike a nice
balance between performance and cost. 600 MB/s per 8Gb connection is a good guideline
for a high bandwidth read-focused design on the VNX 8000. Since host bandwidth
capabilities depend heavily on the number of cores available to Oracle for I/O processing, a
good rule of thumb for the host is to build to 150MB/s peak per processor core of read
bandwidth. Thus, if you have a quad socket (8 core per socket) host, you should build the
Fiber Channel connectivity to 150MB/s * 8 * 4 = 4800 MB/s per node. Four 16Gb/s fiber
channel connections per host strikes a good balance between bandwidth, cost and cabling
complexity for a typical Oracle RAC data warehouse Storage Grid deployment utilizing quad
processor socket hosts.
2. Even the load distribution between all storage array resources (storage processors, disks,
buses, front end connections).
An important design consideration for Oracle RAC and ASM is to ensure even distribution of
all storage array resources to each ASM disk group. Balancing all resources such as
storage processors, Fiber Channel connectivity, LUN paths, backend storage bus layout,
and all disks is critical to achieve maximum scalability. All clusters should touch all disks in
the solution. This way, the ebb and flow of a given cluster/host workload can leverage all
available storage processing resources. Balancing storage I/O processing resources over
time will maximize the number of Oracle loads and extracts one can achieve in a given day.
3. Assess criteria in selecting Pools or classic RAID group storage layout to be utilized with the
building blocks.
Pools versus classic RAID group storage are an important design consideration for the
layout of the storage arrays. The key consideration is the amount of writes the solution is
2014 EMC Proven Professional Knowledge Sharing 8
expected to have during production workloads. A classic RAID should be considered if a lot
of logging activity and loads occur in the data warehouse on a regular basis. Meanwhile,
Pools provide more flexibility and ease of use but at a cost to in write performance. Writes
coming into the system with Pools both have to commit the writes with the associated RAID
protection overhead and also conduct meta data updates. A heavy write environment favors
classic RAID group over Pools to maximize performance. Pools should always be thickly
provisioned (full allocation) for high bandwidth read Oracle RAC data warehouse use cases.
4. Utilize Brocade or Cisco edge switches as Fibre Channel connectivity mechanisms to
maximize the solution’s cost efficiency. Leverage 16GB/s connectivity wherever possible for
data warehouse high bandwidth application use cases.
The fixed configuration Storage Grid design favors simplicity and cost efficiency. The
selection of low cost, high port density edge switches is ideal for Fibre Channel connectivity.
Selecting low cost edge switches will provide all the availability and performance required by
the Storage Grid. A minimum of two edge switches per Storage Grid is recommended, with
four being the ideal Fibre Channel fabric design. No inter switch links (ISL’s) are needed on
the Fibre Channel switches. Each switch operates on its own unique fabric with no port over
subscription to maximize performance and availability of the Storage Grid.
5. Utilize high speed, low latency network interconnect between host nodes. Two typical
examples are Infiniband or low latency, hardware accelerated TCP.
To provide reliable and efficient scalability of the host Oracle RAC cluster, it is best practice
to utilize a low latency network interconnect between the host nodes. The larger the cluster,
the more a low latency interconnect is required. Infiniband is a common interconnect
technology that provides reliable, low latency and high bandwidth capabilities which are
ideal for the node interconnect in a large scale multimode Oracle RAC cluster.
6. Utilize two or three Oracle RAC Clusters behind each Storage Grid to maximize disk
utilization over time.
To maximize storage bandwidth use over time, using multiple clusters enables the natural
ebb and flow of job processing to share a common storage bandwidth pipe. The goal of
having multiple clusters is to smooth out bandwidth utilization during daily, weekly, and
monthly cluster job loads. The primary drawback of multiple clusters is higher seek
2014 EMC Proven Professional Knowledge Sharing 9
distances on the individual disk drives supporting the solution. However, the bandwidth
utilization gains will typically outweigh this drawback by maximizing capital expenditures of
the solution over its life span.
7. Leveraging management tools and fault alert automation to efficiently support each Storage
Grid is critical.
A VNX scale-out model can cause the number of widgets to manage to grow exponentially.
It is important that automation is used to monitor for key performance indicators and
hardware faults on the storage solution. Knowing when response time volatility is being
observed or when component failures occur will provide early warning and narrow the focus
of the issue to allow for speedier resolution. Alert automation enables the user to narrow the
focus of troubleshooting efforts to specific arrays and specific LUNs that may be having
performance issues. Absent robust monitoring tools, a user is left hunting in the dark to
address performance and or availability issues that are affecting the Oracle RAC database
instance. Three key management tools are available for the VNX to address these
challenges:
1. Unisphere® Remote
2. VNX Monitoring and Reporting
3. EMC’s Secure Remote Support Gateway
Manage efficiently by leveraging EMC Storage software tools
Unisphere Remote
Unisphere Remote, a management tool included with a VNX MCx array that is packaged as a
VMware vAPP (pre-packaged application stack), is able to manage up to 1000 individual VNX
arrays via a single GUI. The scalability limits increase with each generation of the product.
Unisphere Remote monitors health and alerts for hardware faults across the entire VNX
footprint on a given network. Each individual VNX can be tagged into unique groups to enable
viewing system health state and alerts for a particular Storage Grid. Limiting views to specific
VNX arrays can be very helpful when troubleshooting a potential storage issue against a given
Storage Grid composed of multiple arrays.
2014 EMC Proven Professional Knowledge Sharing 10
Figure 4: Unisphere Remote Dashboard
Alert automation and viewing in Unisphere Remote
The Alert dashboard eases monitoring for alerts by array or storage array. With a couple of
mouse clicks, all system alerts can be identified on a single or multiple arrays based on severity
by utilizing tags. These tags enable VNX arrays to be grouped to get a single alert view,
providing an efficent method to review for faults against a particular Storage Grid composed of
multiple arrays. Unisphere Remote enable quick identification of any hardware issue that may
be effecting application performance or availability, thus narrowing the focus of troubleshooting
effort to only the component having problems.
2014 EMC Proven Professional Knowledge Sharing 11
Figure 5: Unisphere Remote Alert Dashboard
VNX Monitoring and Reporting
VNX Monitoring and Reporting (M&R), another tool included with the purchase of a VNX MCx
array, provides performance and capacity reporting capabilities for both block and file storage
types on a VNX array. Each VNX M&R instance can support up to 5 VNX arrays. Note: the full
Watch4Net package on which VNX M&R is based has higher support limits and increased
capabilities, albeit at additional cost. VNX M&R has dozens of default reports that can provide
valuable insight to the performance and capacity utilization of VNX arrays. Examples of the
default performance reports include Storage Pool IOPs bottleneck, LUN performance
Bottlenecks, Disk Utilization Heat Map, and many more. In addition to generating reports on
performance and capacity utilization, inventory reports can also be generated that include all
arrays, firmware levels, and associated storage layout resources (LUNs, RAID Groups, Storage
Pools, Host, Ports, and applications). Along with reporting, VNX M&R can also provide
performance and threshold alerts to both SNMP and SMTP targets. Simple monitoring around
key performance indicators such as response time threshold can provide early warning of
potential issues that will impact Oracle RAC performance.
2014 EMC Proven Professional Knowledge Sharing 12
Figure 6: VNX Monitoring and Reporting Dashboard
EMC Secure Remote Support Gateway
A third tool included with the purchase of any VNX MCx array, EMC Secure Remote Support
Gateway (ESRS) provides an asynchronous messaging system to enable secure encrypted file
transfer for device monitoring of EMC hardware products as well as outbound notification to
EMC of hardware and software faults that require support. The ESRS platform also provides the
remote access needed to support diagnostic activities of EMC support staff to speed time to
resolution of hardware/software related faults that occur. Such automation helps minimize the
time to resolution of any hardware/software problem that may occur on EMC storage products.
ESRS is a critical piece of efficient management of a scale out Storage Grid solution that could
be composed of dozens or even hundreds of storage arrays for the most complex customer
environments.
2014 EMC Proven Professional Knowledge Sharing 13
Figure 7: EMC Secure Remote Support Gateway Archiecture
Conclusion
The VNX MCx platform provides the ideal storage solution to build a scale out, high
performance Oracle RAC data warehouse application. Providing the high bandwidth and
storage capacity demanded by Oracle RAC data warehouses can be very challenging. The
exceptional performance scalability provided by VNX Multi-core I/O technology enables the
solution to scale linearly based on the available disk I/O processing power. This article provides
a starting point to develop a custom building block-based solution that meets all the application
requirements and does so with an easy to manage, high availability solution.
2014 EMC Proven Professional Knowledge Sharing 14
References
EMC documentation utilized for ESRS, VNX M&R, and Unisphere Remote. Figure 4, 5, 6, and 7
screen shots taken from the following documents.
Operation Guide - EMC Secure Remote Support Gateway for Linux Release 2.22 Rev 03
White Paper – EMC Unisphere Remote: Next Generation Storage Monitoring, A Detailed
Review November 2012
EMC VNX Monitoring and Reporting Version 1.1 User Guide P/N 300-014-420
EMC believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION
MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO
THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an
applicable software license.
Recommended