Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS · Installation and configuration ... Database tuning ... Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS

Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS

Applied Technology

Abstract

This white paper introduces IP storage technologies and the platform benefits for Oracle running on consolidated IP storage over NFS. It also reviews the features of Oracle 11g Direct NFS and summarizes the business and performance benefits observed after extensive lab testing on EMC® Celerra® network-attached storage (NAS).

October 2010

Copyright © 2009, 2010 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

P/N H6180.1

Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS Applied Technology 2

Table of Contents Executive summary ............................................................................................4 Introduction.........................................................................................................4

Audience ...................................................................................................................................... 4 Technology overview .........................................................................................5

IP storage technology and NFS protocols ................................................................................... 5 Benefits..................................................................................................................................... 5 Challenges................................................................................................................................ 5

Oracle 11g DNFS......................................................................................................................... 6 EMC Celerra unified, multi-protocol storage architectures .......................................................... 7 Celerra IP storage solution features ............................................................................................ 8

Consolidated storage ............................................................................................................... 8 High-speed backup .................................................................................................................. 8 Data replication and restore ..................................................................................................... 9 Database environment cloning for QA, test, and development................................................ 9

Testing Celerra IP storage with Oracle DNFS ..................................................9 Test environment ......................................................................................................................... 9

IP architecture .......................................................................................................................... 9 Database environment ........................................................................................................... 10 IP storage server environment ............................................................................................... 10 IP storage layout for Oracle databases.................................................................................. 10 Network connections .............................................................................................................. 11

Test setup for DNFS .................................................................................................................. 11 Installation and configuration.................................................................................................. 11 Network setup ........................................................................................................................ 12 Database tuning ..................................................................................................................... 13

Test methodology ...................................................................................................................... 14 Performance and resilience tests........................................................................................... 14 Clustered versus non-clustered solutions .............................................................................. 14 Scaled network ports.............................................................................................................. 15 Data collection........................................................................................................................ 15

Test scenarios............................................................................................................................ 15 DSS workload testing ............................................................................................................. 15 OLTP workload testing ........................................................................................................... 19

Conclusion ........................................................................................................23 References ........................................................................................................23


Executive summary Network-attached storage (NAS) technology, also referred to as IP storage technology, is now an established storage infrastructure that leverages the IP/Ethernet backbone of an organization to deliver consolidated, high availability storage solutions for mission-critical applications. The Oracle database is no exception to the trend, and the number of IP storage technologies using the Network File System (NFS) protocol has grown aggressively in recent years.

This white paper describes how EMC® Celerra® IP storage architectures can leverage the benefits of NFS enhancements introduced when Oracle embedded the NFS client directly into the Oracle 11g database application kernel.

Introduction NAS is now common in corporate data centers worldwide. This is largely due to simplified storage provisioning and the widespread adoption of the NFS protocol, which operates over low-cost, legacy, Ethernet/IP network infrastructure.

NAS storage products, such as EMC Celerra, offer a wide range of highly available and flexible NAS/IP storage platforms that meet the needs of today’s businesses. While offering a wide variety of functionality, they also provide a differential in cost that makes NAS storage products viable for small, medium, and large businesses and organizations.

NAS/IP storage not only focuses on centralizing and consolidating storage for multiple systems, but also serves the requirements of a single IP storage array. Regardless of the size of the system, centrally managed provisioning for high availability, backup, and data recovery is a valuable commodity.

Critics argue the suitability of NAS/IP storage solutions for commercial and enterprise applications and question the maturity of the technologies. This is unfounded and already dispelled by large and prestigious organizations around the world that are deploying Oracle with business-critical applications on Celerra and other IP storage platforms. The question posed by Oracle 11g Direct NFS is not the viability of the NFS technologies and protocol, but whether they bring additional benefits.

Oracle acknowledges the importance of IP storage technologies by making the NFS protocol engine, previously hosted in the operating system, a core component of the Oracle 11g database. This is now Oracle 11g Direct NFS.

Aside from opening up the NFS protocol to Windows and other operating system platforms, which were previously precluded because they did not negatively incorporate NFS engines, other questions related to this feature still remain unanswered.

Oracle’s support for NFS as a base protocol for Oracle databases is intriguing to many, especially those well versed in traditional SAN-based storage products. EMC supports Oracle’s approach to a NAS-based database solution, since it validates our own findings that have evolved over several years.

Now that the NFS engine is incorporated as a free, operating system-agnostic feature in the Oracle 11g Database product, it begs to question: How does it work? What are the benefits? How much does it cost? Does it perform and scale better? And, will it work for my organization?

An introduction to NAS and DNFS technologies and the answers to many of these questions are addressed in this white paper and backed by lab-based validation tests and empirical test data.

Audience The audience for this white paper includes non-technical business persons, technical managers, and other decision-makers looking for insight into Oracle IP storage technologies.


Technology overview Storage area networks (SAN) and NAS architectures are the primary industry-standard connection architectures adopted by organizations today.

SAN technologies use storage-specific transport protocols at a data block level to read and write data to storage devices. This approach provides good performance, but requires a high-cost, dedicated Fibre Channel connection infrastructure.

On the other hand, IP storage technology uses existing Ethernet connection fabric and standard protocols, such as TCP/IP, to connect servers and storage devices—sometimes across vast distances. NAS solutions are less expensive than SAN technology and, because Ethernet is present as the backbone of most corporate networks, in-house expertise is predominant with respect to management and maintenance.

NAS architectures free the database server from file and volume management overhead introduced by block storage devices used with SAN technology. IP storage technology reduces the need for host-based file and volume management software by hosting these management applications on the IP storage array. This practice centralizes management and typically lifts the burden from the database server and places it on the storage array. This is highly beneficial in a database environment since significant cost is invested in purchasing and licensing database servers.

Using IP storage technology allows the database server to perform its prime function—the management of database processes and query workload.

IP storage technology and NFS protocols The following section describes the benefits as well as the challenges of using NFS protocols with IP storage technology.

Benefits IP storage consolidation for Oracle database solutions typically benefits greatly from using the NFS protocols, over and above the Internet Small Computer System Interface (iSCSI) or the Common Internet File System (CIFS) protocols. This is primarily due to the following:

• The CIFS protocol is typically used in Windows operating systems, with little use on other operating system platforms. It also lacks the file-locking capabilities needed in enterprise database environments, including Oracle environments.

• The iSCSI protocol is a SCSI block protocol designed for Ethernet, but has noticeable performance drawbacks in database environments.

• The NFS protocol delegates file and volume management to the NAS server, thereby freeing CPU cycles for database operations.

• The NFS protocol is an established industry-standard protocol with adoption on most common operating systems, including UNIX, Linux, MAC, OS/2, AIX, and HP-UX. NFS provides support for many of the most popular enterprise application platforms.

• The NFS protocol also supports distributed and grid computing architectures, which have gained significant traction in the Oracle database environment over the last few years.

• The NFS protocol is used widely across the world by IT departments, and Oracle database administrators (DBA) have skills and experience with the associated technologies.

Challenges NFS is an excellent choice for an IP storage file system protocol for many of the reasons cited in the previous section. However, some unique challenges are faced when implementing the NFS protocol in enterprise database architectures.


IP storage offloads the file and volume management operations to the storage array, but works with the host operating system in caching I/O reads and writes to the database. This is because the NFS client software that manages the NFS operations is embedded in the host operating system of the database server.

In non-database environments, the use of the operating system to mediate, cache, and control the I/O for the NFS protocol is acceptable. After all, the NFS protocol was designed to handle file read/write patterns observed on the most common operating systems (Figure 1).

Unfortunately, the read/write patterns of a database are different in nature from most file operations. The immediacy and asynchronous nature of database I/O (DB I/O) is neither accommodated nor managed well by most operating system NFS client kernels.

Furthermore, an operating system has varying I/O caching and performance characteristics, whether Linux is used as opposed to Solaris, or whether you use Windows or HP-UX operating systems. This leads to varying NFS performance across different operating systems and across different release versions of the same operating system, as evidenced by the performance variance between RHEL 6.2 and RHEL 7.2 and so forth.

As the result of these factors, varying NFS performance can be manifested across implementations using enterprise database architectures.

Figure 1 DNFS elimination of OS memory dependency

Oracle 11g DNFS Oracle 11g DNFS overcomes the variability in NFS I/O performance by hosting the NFS client software inside the database kernel, not the operating system kernel.

With the NFS client embedded in an Oracle 11g Database, it has high impact in the following ways:

• Constant NFS performance is observed across all operating systems. • The DNFS kernel is modified to better cache and manage I/O patterns typically experienced in

database environments, such as larger and more efficient reads and writes. • The DNFS kernel allows Asynchronous Direct I/O, typically most efficient for database I/O. This

significantly improves read/write database performance by allowing I/O to continue while other requests are being submitted and processed.

• Database integrity requires immediate write access to the database when requested. Operating system caching delays write access for efficiency reasons, potentially compromising data integrity during


failure scenarios. DNFS uses database caching techniques with Asynchronous Direct I/O to ensure almost immediate data writes, thus reducing data integrity risks.

• The NFS client manages load balancing and high availability by applying this in the DNFS client, rather than at the operating system level. This greatly simplifies network setup in high availability environments and reduces dependence on network administrators, eliminating the need to set up network subnets and bond ports, such as Link Aggregation Control Protocol (LACP) bonding.

• DNFS allows up to four parallel network paths/ports to be used for I/O between the database server and the IP storage array. For efficiency and performance, these are managed and load-balanced by the DNFS client, not the operating system.

• DNFS overcomes operating system write-locking, which can be inadequate in some operating systems and can cause I/O performance bottlenecks in others.

• Database server CPU and memory usage are reduced by eliminating the overhead of copying data to and from the operating system memory cache to the database System Global Area (SGA) cache.

EMC Celerra unified, multi-protocol storage architectures Celerra Unified Storage Platform products offer a flexible architecture and multi-protocol connectivity. This enables connectivity over IP/Ethernet, iSCSI, and Fibre Channel SAN environments. Multi-protocol functionality is available on integrated Celerra storage arrays—and for very low cost.

EMC Celerra IP solutions address many of the concerns described by Oracle customers, including the following:

• Cost of network infrastructure • Cost of disparate storage • Volume management software overhead • System manageability and administration • Using high-cost database servers for non-database workloads (backup) • Difficulties implementing replication and mirroring • Provisions for high availability and failover • Provisions for security

Some EMC customers have successfully deployed Oracle solutions in so-called “blended configurations,” using specific protocols to host different types of system data and applications, or both. For more information on unified storage platform products and blended solutions for Oracle environments, see EMC Powerlink® or the Reference section of this document.

Figure 2 on page 8 represents the connectivity and configuration options available on Celerra Unified Storage Platforms.


Figure 2 Celerra with Oracle RAC 11g solution architecture option Oracle 11g Direct NFS leverages the power of the Celerra IP storage environment and provides functionality and performance that corresponds to Fibre Channel SAN connectivity in a midrange database environment.

Multi-protocol Celerra architectures offer SAN connectivity, but this document demonstrates that Fibre Channel speeds and throughput can be achieved using pure NAS/IP storage architectures with Oracle 11g Direct NFS.

Celerra IP storage solution features Celerra IP storage platforms offer a number of tangible benefits to our customers. A typical solution incorporates architectural elements described in the following sections.

Consolidated storage Oracle storage consolidation offers centralized information, simplified management, and improved efficiency. Integrated volume management and the NFS file protocol eliminate the need for additional expense and complexity when managing and sharing files and database data.

High-speed backup It is critical to have high-speed database backup and recovery in order to adequately support Oracle environments. Celerra SnapSure™ provides high-speed, non-disruptive, array-based snapshots, which can be used for either quick data restoration, or as the basis for a full database backup. SnapSure snapshots can be checked into the Oracle RMAN repository and are fully integrated with RMAN for restoration and


recovery operations. Snaps provide an efficient way to back up quickly and easily, without impacting database server performance and without taking the database server offline.

Validated solutions include SnapSure recovery checkpoints, blended backup, and high-speed seeding for Celerra Replicator™ or Oracle Data Guard, NDMP, or Oracle Secure backup to tape.

Data replication and restore Protecting your Oracle environment from disasters provides business protection using remote disaster recovery. Celerra Replicator, utilizing SnapSure checkpoints, allows quick replication and seeding of production environments without impacting database server performance. This also allows you to save money on per-CPU licensing costs. Array-based synchronization of production environments further eliminates the database log shipping overhead on the server CPU. Array-based synchronization also protects the database from a variety of additional logical storage errors, and production environments can be restored to a point in time in a fraction of the time.

Validated solutions include Celerra Replicator and Celerra array-based seeding for Oracle Data Guard in multiple configurations and OS environments. More information is available in the Reference section of this white paper.

Database environment cloning for QA, test, and development With Celerra SnapSure writeable checkpoints, customers can replicate one or many copies of the production database environment. Then, if it becomes necessary later, the customer can roll changes into the production database environment. SnapSure writeable checkpoints can be created quickly, use storage efficiently, and are easy to manage. Production database replicas are great for creating multiple reporting environments and development environments, applying patches, or creating backups. Writeable snaps are compatible with virtualization technologies, such as VMware®, to create cost-effective portable copies of production database environments

EMC Celerra solutions offer a mix of hardware and software products to ease the challenges of administering any Oracle database environment.

Testing Celerra IP storage with Oracle DNFS EMC takes a holistic approach to serving our customer needs. The EMC Celerra Solutions group and EMC Global Solutions collaborated to provide lab-based validation of solutions with formal documentation deliverables. These deliverables include documents to help customers, plan, design, and deliver IP solutions in an Oracle database environment on an EMC Celerra.

During Oracle 11g product beta testing, the EMC Solutions groups evaluated Oracle 11g DNFS. This involved systematic testing of the Oracle 11g DNFS database functionality to prove performance, scalability, resilience, and manageability.

The following sections detail the Oracle 11g DNFS validation testing conducted by EMC, working closely with the Oracle DNFS development team.

Test environment

IP architecture Figure 3 on page 10 illustrates the NAS architecture and IP network infrastructure used during the validation of Oracle 11g Direct NFS with Celerra.


Figure 3 Celerra NS20 RAC architecture (Windows and RHEL 64-bit)

Database environment Testing was conducted on two separate four-node Oracle Real Application Cluster (RAC) environments as follows:

• 4 x Dell 2950 RAC Nodes, 16 GB Memory – Windows 2003 and RHEL, 64-bit • 4 x Dell 6950 RAC Nodes, 16 GB Memory – RHEL

IP storage server environment An EMC Celerra NS40 integrated storage array was used as the basis for all DNFS testing.

IP storage layout for Oracle databases The storage layouts used in the testing were configured in line with IP storage best practices and reference architectures jointly developed by EMC Global Solutions and Celerra Solutions. Exhaustive validation testing with Celerra in Oracle environments over many years has contributed to the development of this approach. See the Reference section for more details.

A summary of the storage layout for the Oracle data is provided in Table 1 on page 11.


Table 1 Storage layout summary

Celerra file system name

Windows file system type

Contents Storage access method

RAID level Disk type

dbfs NFS Oracle data files Direct NFS RAID 5 146GB FC log1fs NFS Online redo log

files group1 Direct NFS RAID 1 146GB FC

log2fs NFS Online redo log files group2

Direct NFS RAID 1 146GB FC

flashfs NFS Flashback recovery area

Direct NFS RAID 5 500GB SATA

archfs OCFS2 Archived log files iSCSI RAID 5 500GB SATA

ocrfs1 RAW OCR file copy 1, voting disk copy 1

iSCSI RAID 1 146GB FC

ocrfs2 RAW OCR file copy 2, voting disk copy 2

iSCSI RAID 1 146GB FC

ocrfs3 RAW voting disk copy 3 iSCSI RAID 1 146GB FC

Table 1 describes the layout of the file systems that host the system data files, logs, and other Oracle components. It also denotes the RAID level adopted and the types of storage disks used.

For more information on Oracle NFS best practices on Celerra, see the Reference section of this document.

Network connections An EMC Celerra NS20 was configured as the IP storage server with the following network IP connectivity:

• 4 x 1 GbE connections per Data Mover • Four connections were left as four individual (non-trunked by operating system) • Four network ports configured for full use by DNFS • 2 x 1 GbE connections were used for the Oracle RAC Interconnect

Test setup for DNFS Oracle 11g DNFS is a free product option that ships with the Standard Edition and Enterprise Editions of the Oracle 11g database. This section describes the test setup for Oracle 11g DNFS.

Installation and configuration There are two steps to install and configure DNFS as follows:

1. Create an oranfstab file, as shown in Figure 4 and Figure 5 on page 12. This defines the following:

Database server Source and target NIC IP addresses for the paths Export mounting points Outgoing message routing


server: rtpsol35b local: 10.6.24.124 path: 10.6.24.220 local: 10.6.25.125 path: 10.6.25.167 local: 10.6.26.126 path: 10.6.26.107 local: 10.6.27.127 path: 10.6.27.108 export: /datafs mount: /home/oracle/FTS_Lineitem/data dontroute

Figure 4 Linux: Example of Oranfs tab file

server: rtpsol35b local: 10.6.24.124 path: 10.6.24.220 local: 10.6.25.125 path: 10.6.25.167 local: 10.6.26.126 path: 10.6.26.107 local: 10.6.27.127 path: 10.6.27.108 export: /datafs mount: c:\home\oracle\FTS_Lineitem\data dontroute

Figure 5 Windows: Example of Oranfstab file

2. Replace the ODM library file with the version that implements DNFS, as shown in Figure 6 and Figure 7.

cd $ORACLE_HOME/lib cp libodm11.so libodm11.so_stub ln -s libnfsodm11.so libodm11.so

Figure 6 Linux: Example of OS commands used to swap in the ODM library files for DNFS

cd %ORACLE_HOME%/bin copy oraodm11.dll oraodm11.dll.stub copy /Y oranfsodm11.dll oraodm11.dll dir libnfsodm11.so libodm11.so

Figure 7 Windows: Example of OS commands used to swap in the ODM library files for DNFS

Once configured, no additional maintenance or configuration is required.

Network setup Port bonding and load balancing are managed by the Oracle DNFS client in the database; therefore, there are no additional network setup steps.

If operating system NIC/connection bonding is already configured, the operating system must be configured to release the connections so they work as independent ports. DNFS then manages the bonding, HA, and load balancing for the connections.

The dontroute setting specifies that outgoing messages should not be routed using the operating system, but should be sent using the IP address to which they are bound. If dontroute is not specified, it is mandatory that all paths to the Celerra are configured in separate network subnets.


The network setup can now be managed by an Oracle DBA through the oranfstab file. This frees the SYSDBA database administrator from performing specific bonding tasks previously necessary for operating system LACP-type bonding, such as creating separate subnets.

Database tuning Oracle 11g DNFS requires little additional tuning, other than the tuning considerations necessary in any IP storage environment with Oracle. In an unchanging environment, once tuned, DNFS requires no ongoing maintenance.

Examples of Celerra database tuning for Oracle 11g DNFS are provided in the following sections.

Reserved port configuration Some NFS file servers require NFS clients to connect using reserved ports. If a file server is running with reserved port-checking, it must be disabled for DNFS to operate.

Mounting DNFS If DNFS is used, a new configuration file, oranfstab, must be created to specify the options, attributes, and parameters that enable Oracle database to use DNFS. This process includes the following tasks:

• Add the oranfstab file to the ORACLE_BASE\ORACLE_HOME\dbs directory. • For Oracle RAC, replicate the oranfstab file on all nodes and keep synchronized. Mounting multiple servers When the new oranfstab configuration file is placed in the ORACLE_BASE\ORACLE_HOME\dbs directory, the entries in this file are specific to a single database. The DNFS client searches for the mount point entries as they appear in the oranfstab file. DNFS uses the first matched entry as the mount point.

NOTE: See EMC Solutions for Oracle RAC 11g on Windows Direct NFS with EMC Celerra Unified Storage Platforms Solution Guide for more details on configuring multiple mounting points.

Degree of parallelism (DOP) Business Intelligence/Decision Support System (BI/DSS) and data warehousing workloads with complex query generation can involve outer table joins or full database table scans, or both. This can be optimized on DNFS by configuring the degree of parallelism (DOP) used by the database in the int.ora file. The DOP is set to 8 by default for a standard database installation. Validation testing for DSS workloads with DNFS concluded that a DOP set to 32 was optimum for the TPC-H type of workloads applied to the servers during this testing.


Manual Memory Management For best database performance on Linux with DNFS and KNFS the SGA memory was manually tuned:

Some of the Database tuning parameters and values used are detailed below:

SGA_TARGET = 40 GB db_file_multiblock_read_count = 8 db_writer_processes = 8 sysctl -w sunrpc.tcp_slot_table_entries=128 vm.nr_hugepages=20481 wmem_max = 64K

OS configuration Best performance results on Linux for both KNFS and DNFS were observed when hyperthreading was disabled in the DB server bios.

NOTE: For more information on specific DNFS tuning for Oracle on Celerra IP storage, see related DNFS white papers and solutions implementation documentation in the Reference section.

Test methodology Modern database solutions incorporate multiple database workload types, each with different characteristics. All must be accommodated with consistent and predictable performance. For this reason, two distinct workload types were incorporated into the testing as follows:

• OLTP (similar to TPC-C)—Random read/write query workloads observed in most transactional database systems.

• DSS (similar to TPC-H)—Large sequential read query workloads observed in BI and data warehousing database systems.

To conduct the DNFS testing, two benchmarking tools were utilized.

• The OLTP workload was generated using the Quest Benchmark Factory tool. • The DSS workload was generated using DSS simulation scripts provided by Oracle. (These scripts

instigate full table scan operations on large tables and are run in parallel on a single node in multiple SQLPlus sessions.)

Lab-based validation testing was conducted on various configuration combinations as described in the following sections.

Performance and resilience tests Performance and resilience tests were conducted to compare results obtained from operating system-managed (OS-managed) LACP bonded Kernel NFS (KNFS), as compared to results obtained from database-managed bonding using DNFS.

These tests helped to provide an understanding of the dynamics of port failover and load balancing, as well as the relative performance of the operating system kernel compared to database-hosted NFS clients.

Clustered versus non-clustered solutions Multiple database configurations were used to test clustered versus non-clustered solutions, including the following configurations:

• Oracle Database Single Instance configuration, with Cluster Ready Services (CRS) services stopped • Oracle RAC configurations with one, two, three, and four database nodes started, with CRS services

enabled


Scaled network ports The database servers were configured to utilize up to four 1-GbE network ports. The DNFS and KNFS testing validated performance and resilience using one, two, three, and four network ports in distinct and separate performance tests.

Data collection During distinct runs on the separate configurations, data was collected from the following sources:

• Database logs • Database AWR reports • Celerra storage array logs • Database server operating system utilities (for example, VMStat) • Quest Benchmark Factory reports Painstaking analysis of each run was undertaken to compile a comparative summary of Oracle DNFS versus operating system KNFS performance and resilience.

Quest Benchmark Factory was configured for OLTP (TPC-C like) user load testing. This was configured with a 2,000 warehouse TPC-C database. During execution 24 agents were used.

Test scenarios This section presents an executive summary of the findings from the DNFS validation testing for DSS and OLTP workload types.

DSS workload testing Two types of workload testing scenarios were conducted in this portion of the validation testing as follows: • Query DSS performance on DNFS • DSS port scaling on DNFS The analysis and accompanying charts in subsequent sections represent DSS testing on the following architecture:

• 4 x Node Oracle RAC architecture (Dell 2950, dual-core 3 GHz DB servers) • 4 x NIC port connections to the Celerra NS40 IP storage array • Comparing LACP bonded KNFS to unbonded (database-managed) DNFS • 64-bit Windows Server 2003 operating system:


Query performance on DNFS This section shows how large sequential database read operations typical in data warehousing (similar to TPC-H), DSS, and BI queries perform on Oracle DNFS compared to OS-managed KNFS on Oracle 11g.

DSS Workload - DNFS vs. KNFS Performance

125100

439

330

50

100

150

200

250

300

350

400

450

500

Max MB/s Data Throughput Max % DB CPU Utilization

4 Port, LACP Bonded KNFS 4 Port, non-Trunked DNFS

Figure 8 DSS Query: Data throughput and DB CPU load – Linux Figure 8 illustrates that the maximum MB/s throughput on the Oracle database server for DNFS is more than three times the throughput of OS-managed KNFS on Windows.

The maximum database server CPU usage is close to 100 percent for KNFS; this never exceeds 35 percent on the same load and hardware using DNFS.

DSS WorkloadAvg % DB CPU Utilization

17

61

12

97

71

50102030405060708090

100

4 ports,DNFS

4 ports,KNFS

2 ports,DNFS

2 ports,KNFS

1 ports,DNFS

1 ports,KNFS

% C

PU U

tiliz

atio

n

Figure 9 DSS Query: Average DB server CPU utilization – Linux

On average, the percentage of database server CPU utilization never exceeds 17 percent for DNFS, while the chart in Figure 9 shows averages for KNFS are consistently above 71 percent.

All other tests are comparable in terms of high availability, resilience, port failover, and so forth. Effectively, there are no differences in functionality between KNFS and DNFS, except the reduction in CPU load and the increased MB/s data throughput using DNFS.


Database performance consistency is maintained on DNFS regardless of the operating system platform on which it is used. Further tests conducted on this configuration with RHEL 64-bit showed no significant differences in DNFS performance across all operating systems tested.

Port scaling on DNFS Tests were conducted to identify the scaling capabilities of DNFS compared to OS-managed KNFS. In a series of four tests, the database servers were subjected to simulated concurrent DSS workloads. In each test, the number of ports used to connect to the storage array was incremented from one to four NIC ports. The results below show the maximum combined port throughput for each of the single- and multi-port configurations.

DSS Workload Port Scaling Throughput

0

50

100

150

200

250

300

350

400

450

500

1 Port 2 Ports 3 Ports 4 Ports

MB

/s

OS Bonded KNFS Direct NFS

Figure 10 DSS Query: Data throughput scaling (bonded connections)

KNFS has reduced port throughput compared to DNFS. DNFS shows significantly better maximum throughput (Figure 10). Both DNFS and KNFS exhibit linear scalability for one-, two-, three-, and four-port configurations for DSS workloads.

All these configurations offer HA and load balancing for port connections. Load balancing and HA for failed connections is managed by the operating system for KNFS and by the database for DNFS.

Figure 11 shows port scaling and elevated performance for KNFS when LACP bonding has not been applied. In this configuration, each KNFS connection is independent and unprotected from connection/port failure.


DSS Workload Port Scaling Throughput

0

50

100

150

200

250

300

350

400

450

500

1 Port 2 Ports 3 Ports 4 Ports

MB

/s

Unbonded KNFS Direct NFS

Figure 11 DSS Query: Data throughput scaling (bonded DNFS and unbonded KNFS)

Although KNFS performance is significantly better than LACP-bonded KNFS performance, this configuration is unprotected from connection/port failure. Furthermore, load balancing across port connections is unavailable. This type of unbounded KNFS configuration does not meet the high availability requirements of most organizations today.

Test conclusions The conclusions drawn from DSS workload testing are described in the following sections.

Administration Operating system KNFS manages caching of data read/writes and, through operating system-NIC bonding (for example, LACP bonding), the operating system manages load balancing and HA. DNFS eliminates bonding and allows the database-resident NFS client to manage trunking, load balancing, and port availability.

Performance and power efficiency Results categorically show DNFS boosts the read/write performance of the database, while significantly reducing the database server CPU load.

This is excellent news for the highly demanding BI/DSS application and their users, keen to maintain query performance. Furthermore, efficient use of CPU cycles on high-specification database server hardware can potentially reduce power consumption and increase database server CPU life.

Linear data throughput scalability Test results clearly show DNFS and KNFS data throughput, which is necessary for efficient and timely query performance, scales linearly with the addition of extra NIC ports. However, DNFS outperforms KNFS in all single- and multi-port configurations.


Manageability Oracle DNFS significantly simplifies load balancing and HA by eliminating the need for network administration resources to set up and maintain port bonding/trunking. DNFS in the database also load-balances database loads in a more efficient manner than the operating system-based KNFS NFS client that is not optimized.

DNFS is easy to set up and the “ease-of-use” nature of the option means it needs no maintenance once configured.

Free option Oracle DNFS is a free option of the Oracle 11g database. It requires no additional database option and ships with every product release of Oracle 11g.

Test summary BI/DSS users are sensitive to performance and query response degradation. DNFS proves consistent data throughput and it remains significantly higher than KNFS. Furthermore, linear scalability is maintained by adding extra 1-GbE NIC connections.

When deployed with the EMC Celerra, DNFS proves to be a performant and efficient vehicle for deploying business-critical BI and DSS database solutions in a Celerra IP storage environment.

OLTP workload testing Two types of QLPT (similar to TPC-C) workload testing scenarios were conducted in this portion of the validation testing as follows: • Query OLTP performance on DNFS • OLTP port scaling on DNFS The analysis and accompanying charts in subsequent sections represent OLTP testing on the following architecture:

• 4 x Node Oracle RAC architecture (Dell 2950, Dual-core 3 GHz DB Servers) • 4 x NIC port connections to the Celerra NS40 IP storage array • Comparing LACP bonded KNFS to unbonded (database-managed) DNFS • 64-bit Windows Server 2003 operating system OLTP query performance on DNFS Most production databases run OLTP random or batched query workloads generated by a vast array of database applications, such as Oracle E-Business suite or SAP running, for example, ERP or CRM query workloads.

In these tests, OLTP workloads were simulated. Oracle DNFS was compared to operating system-hosted (OS-hosted) KNFS and query performance and other critical performance indicators were observed while running in an IP environment.


Figure 12 OLTP Query: Averaged transactions per second throughput and DB server CPU utilization – Linux

Several tests with varying OLTP workloads were run on Linux database servers. Figure 12 shows that an average TPS for DNFS was approximately 15 percent better than regular KNFS.

Significant difference in Celerra CPU utilization was also observed during the comparison of DNFS and KNFS. On Linux the DNFS tests showed almost half the CPU utilization on the Celerra server with a OLTP workload.

Figure 13 OLTP Query: Example TPS throughput under user query workload – Windows

OLTP workloads were scaled up to up to almost 19,000 concurrent users on Linux database servers running DNFS (Figure 13). DNFS demonstrated consistent TPS workload performance over one, two, three, and four NIC ports (DNFS paths). Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS Applied Technology 20

Figure 14 OLTP Query: DB Server CPU Utilization under user query workload

CPU utilization for random OLTP workloads showed little, if any, variation for KNFS versus DNFS. This is due to the nature of the small random workloads and the fact that the random OLTP workloads leverage less of the synchronous DNFS capabilities that were so prevalent for large sequential workloads (BI/DSS queries).

OLTP port scaling on DNFS Tests were conducted to identify the scaling capabilities of DNFS compared to OS-managed KNFS with OLTP database query workloads.

In a series of four tests, the database servers were subjected to simulated OLTP workloads using the Quest Benchmark Factory tool. In each test, the number of ports used to connect to the storage array was incremented from one to four ports.

The results that follow show the maximum combined TPS achieved for each of the single- and multi-port configurations. Averages across multiple tests are shown and, in each data set, varying workloads between 3,600 and 19,000 simulated OLTP users were applied.


Figure 15 OLTP Query: TPS scaling with additional NICs – DNFS

Figure 15 shows linear scalability of DNFS ports. It is observed that port scalability and TPS are optimal when using four NIC ports for DNFS.

Port scaling analysis Oracle 11g DNFS demonstrates linear port scaling and broadly similar TPS performance to KNFS with identical query workloads in all port configurations.

In all cases except TPS and Storage CPU Utilization (where DNFS outperformed KNFS), both configurations exhibited broadly similar profiles with regard to memory usage, database CPU load, and other key measurements.

Test conclusions The conclusions drawn from OLTP workload testing are described in the following sections.

Administration Operating system KNFS manages cached data reads and writes, and through operating system NIC bonding (for example, LACP bonding), the operating system manages load balancing and HA. DNFS eliminates bonding and allows the database-resident NFS client to manage trunking, load balancing, and port availability. Furthermore, DNFS load balancing and DB connectivity are tuned specifically for database workloads.

Performance and efficiency Results categorically show DNFS boosts OLTP TPS by approximately 15 percent over KNFS when four network NIC ports are used for the storage connection. DNFS also exhibited significantly less memory usage on the Celerra storage processors. On average Celerra memory utilization was approx 40 percent less than the equivalent KNFS footprint. In all other respects, resource usage (including database CPU utilization) is almost identical for both configurations.

Linear scalability Test results clearly show DNFS scales linearly with the addition of extra NIC ports. Testing indicated concurrent user load can be maintained to around 19,000 OLTP users in a four-node RAC configuration on Linux.


Manageability Oracle DNFS significantly simplifies load balancing and HA by eliminating the need for network administration resources to set up and maintain port bonding/trunking.

DNFS is easy to set up and the “ease-of-use” nature of the option means it needs no maintenance after DNFS is configured. Operating system bonding setup is required for KNFS, but the setup varies and is specific for different operating systems. This is typically a network administration task requiring tasks such as subnet configuration.

Free option Oracle DNFS is a free option of the Oracle 11g database. It requires no additional database options and ships with every product release of Oracle 11g.

Test summary Transactional and batch OLTP systems are prevalent across the globe and the overhead of administration, continued performance, and scalability can be significant. In complex Oracle RAC environments, high availability and scalability of architecture are elements that are essential to business growth.

Management configuration and cost overheads are significantly reduced by DNFS. The tuned nature of the DNFS client makes it a great OS-agnostic, scalable alternative to operating system-specific KNFS.

Celerra IP storage and Oracle 11g DNFS are an exceptional combination for OLTP database systems by providing a highly optimized, performant, centrally managed, and consolidated IP solution.

Conclusion Businesses need robust, scalable solutions that meet the challenges of today’s tough economic climate. EMC Celerra IP storage provides a scalable, performant, and highly available solution for Oracle database and business application infrastructures. The EMC Celerra validated solution and experience from our customer implementations show that IP storage solutions for Oracle are both safe and secure, and also perform data transactions with levels of data throughput equal to traditional SAN storage.

Oracle 11g DNFS provides highly tuned NFS solutions that EMC Celerra NS and NX platforms use to provide tangible benefits to customers. These benefits materialize in cost savings, performance improvements, and scalability enhancements that greatly strengthen the already established benefits of implementing IP solutions for Oracle database environments.

Oracle 11g DNFS is a free, easy-to-use option that significantly improves performance and offers efficiency and manageability savings across the board.

Why wouldn’t you choose Oracle 11g DNFS to enhance Celerra IP storage solutions in your organization’s database environment?

Reference The following references are located on EMC Powerlink:

• Solution Guide: EMC Solutions for Oracle RAC 11g on Windows Direct NFS with EMC Celerra Unified Storage Platforms

• Solution Guide: EMC Solutions for Oracle 11g on Linux using Direct NFS on VMware ESX Server and EMC Celerra Unified Storage Platforms

• White Paper: EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platform - Applied Technology Guide

• Reference Architecture: EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises—Physically Booted Solutions With EMC Celerra NS40 Unified Storage Platform

• Best Practices Planning: EMC Solutions for Oracle Database 10g/11g for Midsize Enterprises EMC Celerra Unified Storage Platforms


• Tech Note: Celerra NS Series NFS Oracle Database 10g Migration and Upgrade

The following reference is located on Oracle.com:

• Oracle Database 11g Direct NFS Client white paper


Documents

Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS · Installation and configuration ... Database tuning ... Optimizing EMC Celerra IP Storage on Oracle 11g Direct NFS