HUAWEI OceanStor V3 Converged Storage Systems … Systems — OLTP Oracle Database Reference Architectures HUAWEI OceanStor V3 converged storage systems are next-generation ... and

HUAWEI OceanStor V3 Converged

Storage Systems — OLTP Oracle

Database Reference Architectures

HUAWEI OceanStor V3 converged storage systems are next-generation storage products intended for enterprise

customers. How to select proper storage models and configurations based on the database scale and performance

requirements for the maximum return on investment (ROI) is a challenge faced by customers and Huawei. The

solutions described in this document verify OLTP Oracle Database of several common scales, providing reference

about OLTP databases for enterprise customers.

The solutions use OceanStor 5300 V3, 5500 V3, 5600 V3, and 5800 V3 converged storage systems to respectively

provide at least 20,000, 40,000, 60,000, and 80,000 transaction IOPS for OLTP Oracle 12c RACs whose scales are 2

TB, 4 TB, 6 TB, and 8 TB respectively.

Based on Oracle 12c clusters and pluggable database architectures, the solutions use RH2288 servers as database

nodes and are oriented to enterprise databases that require high transaction IOPS.

Wang Yaohui

Storage Solutions, IT, Huawei Enterprise BG

2015-03-17 V1.0

Huawei Technologies Co., Ltd.

mailto:[email protected]

Huawei Proprietary and Confidential Copyright © Huawei Technologies Co., Ltd. 2

HUAWEI OceanStor V3 Converged Storage Systems — OLTP Oracle Database Reference

Architectures

Contents

1 Overview ......................................................................................................................................... 4

1.1 Introduction .................................................................................................................................................................. 4

1.2 Solution Architectures .................................................................................................................................................. 4

1.3 Intended Audience ........................................................................................................................................................ 8

1.4 Customer Benefits ........................................................................................................................................................ 9

1.5 Key Components .......................................................................................................................................................... 9

1.6 Workload Model ........................................................................................................................................................... 9

2 Products and Technologies ....................................................................................................... 10

2.1 OceanStor V3 Converged Storage Systems ................................................................................................................ 10

2.1.1 Next-Generation Hardware ...................................................................................................................................... 11

2.1.2 Multi-Controller Architecture .................................................................................................................................. 11

2.1.3 Convergence Design ................................................................................................................................................ 11

2.1.4 Smart Software ........................................................................................................................................................ 11

2.1.5 Unified and Easy Management ................................................................................................................................ 12

2.1.6 RAID 2.0+ Block Virtualization .............................................................................................................................. 12

2.1.7 SmartTier ................................................................................................................................................................. 13

2.2 Oracle Database and Cluster ....................................................................................................................................... 15

2.2.1 Oracle RAC and ASM ............................................................................................................................................. 15

2.2.2 Oracle System Architecture ..................................................................................................................................... 17

2.2.3 Oracle Application Types ......................................................................................................................................... 19

3 Tiny-Size OLTP Database Reference Architecture .............................................................. 21

3.1 Huawei Solution ......................................................................................................................................................... 21

3.1.1 Solution Architecture ............................................................................................................................................... 21

3.1.2 Solution Configuration ............................................................................................................................................ 22

3.2 Verification Procedure ................................................................................................................................................ 24

3.3 Verification Results ..................................................................................................................................................... 27

4 Small-Size OLTP Database Reference Architecture ............................................................ 28

4.1 Huawei Solution ......................................................................................................................................................... 28







Architectures

5 Medium-Size OLTP Database Reference Architecture ....................................................... 33

5.1 Huawei Solution ......................................................................................................................................................... 33





6 Large-Size Transaction Database Reference Architecture .................................................. 38

6.1 Huawei Solution ......................................................................................................................................................... 38





7 Appendix ...................................................................................................................................... 43

7.1 Reference Documents ................................................................................................................................................. 43

7.2 Acronyms and Abbreviations ...................................................................................................................................... 43



Architectures

1 Overview

1.1 Introduction

HUAWEI OceanStor V3 converged storage systems are next-generation storage products

intended for enterprise customers. How to select proper storage models and configurations

based on the database scale and performance requirements for the maximum return on

investment (ROI) is a challenge faced by customers and Huawei. The solutions described in

this document verify OLTP Oracle Database of several common scales, providing reference

about OLTP databases for enterprise customers.

The solutions use OceanStor 5300 V3, 5500 V3, 5600 V3, and 5800 V3 converged storage

systems to respectively provide at least 20,000, 40,000, 60,000, and 80,000 transaction IOPS

for OLTP Oracle 12c clusters whose scales are 2 TB, 4 TB, 6 TB, and 8 TB respectively.

Based on Oracle 12c clusters and pluggable database architectures, the solutions use RH2288

servers as database nodes and are oriented to enterprise databases that require high transaction

IOPS.

1.2 Solution Architectures

The following table describes the verified reference architectures.

Table 1-1 Reference architecture list

Solution Component Amount of Data Transaction IOPS

T (tiny-size

OLTP database)

OceanStor 5300 V3

converged storage

system

2 x RH2288 V2 servers

2 TB

(Lab test data: 2.5

TB of table and

index data)

20,000

(Lab test data:

28,877)

S (small-size

OLTP database)

OceanStor 5500 V3

converged storage

system


4 TB

(Lab test data: 5 TB

of table and index

data)

40,000

(Lab test data:

44,172)

M (medium-size

OLTP database)

OceanStor 5600 V3

converged storage

system


6 TB

(Lab test data: 8 TB

of table and index

data)

60,000

(Lab test data:

63,477)



Architectures

Solution Component Amount of Data Transaction IOPS

L (large-size

OLTP database)

OceanStor 5800 V3

converged storage

system


8 TB

(Lab test data: 10

TB of table and

index data)

80,000

(Lab test data:

86,429)

Figure 1-1 Tiny-size OLTP database reference architecture

[1] The detailed configuration of the RH2288 V2 is as follows: 2 x E5-2660 CPUs, 256 GB memory, 1 x

QLogic 8 Gbit/s dual-port Fibre Channel HBA, and 1 x Intel 10 Gbit/s Ethernet HBA.

[2] The detailed configuration of the OceanStor 5300 V3 converged storage system is as follows: 32 GB

cache, 1 x controller enclosure (18 x 600 GB 10k rpm SAS disks, 7 x 200 GB SLC SSDs, 2 x 8 Gbit/s

four-port Fibre Channel I/O modules), 1 x disk enclosure (25 x 600 GB 10k rpm SAS disks), 1 x disk

domain containing all the 50 disks (22.7 TB capacity), 1 x storage pool (RAID 10 and 6 TB capacity

configured for the SAS tier, and RAID 5-5 and 800 GB capacity configured for the SSD tier), 10 x 500

GB LUNs (eight LUNs used as the data area, and two LUNs used as the log area).

[3] The tested database contains 2.5 TB of table and index data.

[4] The tested transaction IOPS is 28,877, based on the Order Entry 2.0 order processing model.

5300 V3 [2]

43 x 600 GB 10k rpm SAS disks7 x 200 GB SLC SSDs

2 x

12

Gb

it/s

SA

S

4 x 8 Gbit/s FC

1 x disk enclosureOracle 12c RAC

2 x RH2288 V2 [1]

Cluster Private Interconnection

S6700 10GE switch

2 x SNS2224 FC switches

Tiny-Size OLTP Database Solution2 TB[3] data, 20,000[4] transaction IOPS



Architectures

Figure 1-2 Small-size transaction database reference architecture





four-port Fibre Channel I/O modules), 3 x disk enclosures (75 x 600 GB 10k rpm SAS disks), 1 x disk



GB LUNs (16 LUNs used as the data area, and 4 LUNs used as the log area).

[3] The tested database contains 5 TB of table and index data.


5500 V3 [2]


4 x

12

Gb

it/s

SA

S

4 x 8 Gbit/s FC

3 x disk enclosures

Oracle 12c RAC

4 x RH2288 V2 [1]


S6700 10GE switch


Small-Size OLTP Database Solution4 TB[3] data, 40,000[4] transaction IOPS



Architectures

Figure 1-3 Medium-size OLTP database reference architecture




cache, 1 x controller enclosure (2 x 8 Gbit/s four-port Fibre Channel I/O modules, 2 x 12 Gbit/s

four-port SAS I/O modules), 6 x disk enclosures (132 x 600 GB 10k rpm SAS disks, 18 x 200 GB SLC

SSDs), 1 x disk domain containing all the 150 disks (69.1 TB capacity), 1 x storage pool (RAID 10 and

18 TB capacity configured for the SAS tier, and RAID 5-9 and 2400 GB capacity configured for the

SSD tier), 30 x 500 GB LUNs (24 LUNs used as the data area, and 6 LUNs used as the log area).



6 x

12 G

bit/s

SA

S

4 x 8 Gbit/s FC

6 x disk enclosures

Oracle 12c RAC

6 x RH2288 V2 [1]


S6700 10GE switch


Medium-Size OLTP Database Solution6 TB[3] data, 60,000[4] transaction IOPS

5600 V3 [2]




Architectures

Figure 1-4 Large-size transaction database reference architecture



[2] The detailed configuration of the OceanStor 5800 V3 converged storage system is as follows: 128

GB cache, 1 x controller enclosure (1 x 8 Gbit/s four-port Fibre Channel I/O module, 2 x 12 Gbit/s







1.3 Intended Audience

This document is intended for Huawei partners and customers as well as storage and database

administrators and IT engineers who want to deploy OLTP Oracle Database 12c based on

HUAWEI OceanStor V3 converged storage systems.

It is assumed that the readers are familiar with the following products and technologies:

8 x

12 G

bit/s

SA

S

Oracle 12c RAC

8 x RH2288 V2 [1]


S6700 10GE switch


Large-Size OLTP Database Solution8 TB[3] data, 80,000[4] transaction IOPS

8 x disk enclosures

5800 V3 [2]


4 x 8 Gbit/s FC



Architectures

Architecture and working principles of OceanStor V3 converged storage systems

Architecture and working principle of Oracle Database 12c

Linux operating system basics

1.4 Customer Benefits

The solutions described in this document are designed to accelerate the process of designing,

verifying, and delivering an OLTP database solution. Based on a typical OLTP model, this

document provides verified solutions that use OceanStor V3 converged storage systems.

Typical configurations and performance indicators are provided for reference, simplifying

selection of storage models as well as storage planning and configuration. The verified

solutions are expected to help customers obtain the maximum ROI.

1.5 Key Components

The hardware and software covered in this document are as follows:

Storage system: OceanStor V3 converged storage system V300R001C10

Operating system: Red Hat Enterprise Linux 6.5

Multipathing software: UltraPath for Linux 8.01.024

Database software: Oracle Database 12.1.0.2

Cluster software: Oracle RAC 12.1.0.2

Test tool: Huawei SwingBench Test Suite 1.0 for Oracle 12c

1.6 Workload Model

This document uses the mainstream OLTP test model SwingBench Order Entry to test the

recommended planning and configuration strategies. This model defines a type of online order

business and simulates a scenario where a large number of users are querying products,

placing orders, processing orders, and viewing orders online. Those operations are the most

common ones in transaction systems. In this workload model, there are two main performance

indicators: transactions per second (TPS) and average transaction response time. The TPS

indicates the number of transactions processed per second. A higher TPS indicates higher

productivity. The average transaction response time directly affects the speed of user

operations. A shorter response time indicates better user experience.

Order Entry defines nine tables, storing information about products, customers, orders,

warehouses, and login. During the workload tests, 50% of operations are SELECT, 30%

INSERT, 20% UPDATE, and no DELETE operations. From the perspective of I/O layer, the

workload model is the most typical OLTP workload model, where small data blocks are

accessed at random and the ratio between reads and writes is 6:4.



Architectures

2 Products and Technologies

2.1 OceanStor V3 Converged Storage Systems

HUAWEI OceanStor V3 converged storage systems are next-generation unified storage

products designed for enterprise-class applications. Leveraging a storage operating system

oriented to cloud architecture, a powerful next-generation hardware platform, and a full range

of intelligent management software, OceanStor V3 converged storage systems deliver

industry-leading functionality, performance, efficiency, reliability, and ease of use. They

provide data storage for applications such as large-scale database OLTP/OLAP, file sharing,

and cloud computing, and can be used in industries ranging from government, finance,

telecommunications, energy, to media and entertainment (M&E). Meanwhile, OceanStor V3

converged storage systems can provide a wide range of efficient and flexible backup and

disaster recovery solutions to ensure business continuity and data security, delivering

excellent storage services.

For details about HUAWEI OceanStor V3 converged storage systems, click the following

link:

http://e.huawei.com/en/products/cloud-computing-dc/storage/unified-storage/mid-range

Figure 2-1 HUAWEI OceanStor V3 converged storage systems

http://e.huawei.com/en/products/cloud-computing-dc/storage/unified-storage/mid-range



Architectures

2.1.1 Next-Generation Hardware

OceanStor V3 converged storage systems employ next-generation Intel multi-core processors,

PCIe 3.0 buses, 12 Gbit/s SAS 3.0 high-speed disk ports, and a variety of host ports such as

16 Gbit/s Fibre Channel, 10 Gbit/s FCoE, and 56 Gbit/s InfiniBand host ports. The storage

systems provide up to 28 GB/s of system bandwidth to meet the requirements of

bandwidth-intensive application scenarios. They also offer million-level IOPS performance,

outshining products from other vendors.

OceanStor V3 converged storage systems are equipped with exclusive SmartIO cards. A

SmartIO card supports 8 Gbit/s Fibre Channel, 16 Gbit/s Fibre Channel, 10 Gbit/s iSCSI, and

10 Gbit/s FCoE. Users can specify the protocols that a SmartIO card is required to support.

The deduplication/compression cards used by OceanStor V3 converged storage systems

support lossless data deduplication and compression, efficiently reducing data storage costs.

In addition, the storage systems can implement data encryption to secure data.

2.1.2 Multi-Controller Architecture

The multi-controller architecture used by OceanStor V3 converged storage systems supports

online horizontal expansion. An OceanStor V3 converged storage system can be

non-disruptively expanded to a maximum of eight controllers, 1 TB of cache, and 5 TB of

storage space, meeting customers' future capacity needs. The multi-controller architecture

allows load balancing among controllers and eliminates single points of failure, thereby

ensuring high availability and stable service running.

2.1.3 Convergence Design

Convergence of SAN and NAS: SAN and NAS services are converged to provide

elastic storage, simplify service deployment, improve storage resource utilization, and

reduce the total cost of ownership (TCO). Underlying storage resource pools directly

provide both block and file services, thereby shortening storage resource access paths to

ensure that the two services are equally efficient.

Convergence of heterogeneous storage systems: Based on the built-in heterogeneous

virtualization function, OceanStor V3 converged storage systems can efficiently manage

storage systems from other mainstream vendors and unify resource pools for central and

flexible resource allocation.

Convergence of entry-level, mid-range, and high-end storage systems: OceanStor V3

converged storage systems are the only storage systems in the industry that enable

entry-level, mid-range, and high-end storage systems to interwork seamlessly with each

other. Data can freely flow among storage products of different models without the

assistance of third-party systems.

Convergence of SSDs and HDDs: The advantages of traditional and solid-state storage

media are combined, bringing the performance of different types of storage media into

full play and striking an optimal balance between performance and cost.

Convergence of primary and backup storage: The built-in backup function enables

data to be efficiently backed up without additional backup software, simplifying backup

solution management.

2.1.4 Smart Software

Multi-tenancy and service level agreement (SLA): OceanStor V3 converged storage

systems intelligently allocate storage resources in cloud computing environments to meet

the needs of enterprises and organizations. The storage systems also leverage data



Architectures

isolation and a range of data security policies such as data encryption and data

destruction to meet varying data security requirements. OceanStor V3 converged storage

systems provide four service levels and allocate storage resources based on service

priorities. Storage resources are first allocated to high-priority services to ensure system

performance and shorten response time.

Smart-series efficiency improvement suite: OceanStor V3 converged storage systems

use SmartTier (dynamic storage tiering), SmartMotion (intelligent data migration), and

innovative SmartVirtualization (heterogeneous virtualization) to achieve vertical,

horizontal, and cross-system 3D data flowing, improving storage resource utilization by

three times.

Hyper-series data protection software: Data protection software such as remote

replication, snapshot, and LUN copy software meets user needs for local, remote, and

multi-region data protection, maximizing business continuity and data availability.

2.1.5 Unified and Easy Management

Unified management: OceanStor V3 converged storage systems provide powerful

storage management software that supports global topology view, capacity analysis,

performance analysis, fault diagnosis, and end-to-end service visualization to manage a

wide range of devices.

Convenient management: OceanStor V3 converged storage systems can be initially

configured in 5 steps which take about 40 seconds, and expanded in two steps which take

about 15 seconds.

Mobile management: Users can use tablets and mobile phones to manage storage

systems in real time. System status is sent automatically, making constant attendance by

an engineer unnecessary.

2.1.6 RAID 2.0+ Block Virtualization

RAID 2.0+ block virtualization of the OceanStor V3 implements virtualization for underlying

disk management and upper-layer resource management. Inside the system, the storage space

of each disk is divided into fine-grained data blocks, which comprise RAID groups. In doing

so, data is evenly distributed to all disks in the storage pool. In addition, data block–based

resource management largely improves the resource management efficiency.

1. The V3 storage systems support SSDs, SAS disks, and NL-SAS disks. These disks

comprise disk domains. In a disk domain, disks of the same type comprise disk groups

(DGs).

2. In a DG, the storage space of disks is divided into chunks (CKs) of a fixed size. Then the

system consolidates CKs from random disks into CK groups (CKGs) based on RAID

algorithms.



Architectures

3. CKGs are divided into logical storage space called extents, which have a fixed size too.

Extents are the minimum unit for comprising thick LUNs. On thin LUNs, extents are

further divided into smaller grains.

2.1.7 SmartTier

SmartTier is a DST feature independently developed by Huawei based on RAID 2.0+. It

creates different storage tiers based on disk types, analyzes the activity levels of data blocks,

and migrates data among tiers based on analysis results. This feature ensures that data is

stored in proper storage media, improving system performance and reducing the TCO.

OceanStor V3 converged storage systems support three types of disks, namely, SSDs, SAS

disks, and NL-SAS disks. Each type of storage medium has its unique advantages and

disadvantages in performance and cost. As a result, it is hard for customers to strike a balance

between storage costs and storage performance.

SSDs: Feature a short response time, a low storage request processing cost, but a high

storage capacity cost per gigabyte.

NL-SAS disks: Have a high request processing cost but a low capacity cost per gigabyte.

SAS disks: Fall in between the previous two types in terms of performance and cost.

To strike a balance among storage performance, capacity, and cost, SmartTier incorporates the

following advantages:

High performance and multi-application support: various applications and differentiated

application performance demands

Flexibility and ease-of-use: convenient configuration and migration of data to a suitable

tier

High efficiency and low power consumption: energy-saving and efficient use of space

Table 2-1 Properties of the three tiers in SmartTier

Tier Disk Type Application

High-performa

nce tier

SSD Applicable to applications with intensive

random access requests

Performance

tier

SAS Applicable to applications with medium access

requests

Capacity tier NL-SAS Applicable to applications with low access

requests but high capacity requirements

As shown in the following figure, SmartTier performs intelligent data storage at a LUN level and divides data on LUNs based on the default data migration granularity of 4 MB. The data



Architectures

migration granularity is called extent and ranges from 512 KB to 64 MB. SmartTier collects

statistics and analyzes data activity levels based on extents, and matches data of different

activity levels with storage media. Active data will be promoted to higher-performance

storage media (such as SSDs), whereas inactive data will be demoted to more cost-effective

storage media with larger capacities (such as NL-SAS disks).

Figure 2-2 Working principle of SmartTier

A complete SmartTier service process involves three phases:

Phase I: Collecting hotspot statistics

SmartTier allows user-defined I/O monitoring periods. During the scheduled periods, it

collects I/O statistics. Activity levels of data will change throughout a data lifecycle. By

comparing the activity level of one extent with that of another, the storage system

determines which data block is more or less frequently accessed. The activity level of

each extent is obtained based on the performance indicator statistics of data blocks.

Phase II: Analyzing data placement

The collected hotspot statistics are analyzed. This analysis ranks extents within the

storage pool. The ranking progresses from the most frequently accessed extents to the

least frequently accessed extents in the same storage pool. Note that only extents in the

same storage pool are ranked. Then a data migration solution is created. Before data

migration, SmartTier determines the direction of migrating extents according to the latest

data migration solution.



Architectures

Phase III: Migrating data

SmartTier has two migration triggering modes: manual and periodic. The manual

triggering mode has a higher priority than the periodic mode. In manual triggering mode,

data migration can be triggered immediately when necessary. In periodic triggering mode,

data migration is automatically triggered based on a preset migration start time and

duration. The start time and duration of data migration are user-definable.

2.2 Oracle Database and Cluster

Oracle Database is one of the most widely used relational databases. This section briefly

introduces Oracle Database 12c and focuses on Multitenant-related components and features,

including RAC, ASM, Multitenant, data files, database instance architecture, and application

types.

2.2.1 Oracle RAC and ASM

As shown in Figure 2-3, an Oracle 12c RAC contains two types of nodes: Hub nodes and Leaf

nodes. Hub nodes have direct access to shared storage, whereas Leaf nodes access shared

storage through Hub nodes. When a database is deployed on an Oracle RAC, the nodes can be

grouped into multiple server pools. Each database is deployed in a server pool, and every

node in a server pool runs a database instance. Application servers access the virtual IP

addresses (VIPs) of nodes to store data. If a node fails, its VIP network is restored on another

node of Oracle RAC. Application servers reconnect to the Oracle database through a

reconnection mechanism. Setting connection character strings on application servers can

enable multiple modes of accessing Oracle RAC nodes, including load balancing and failover

modes. In these modes, a multi-node Oracle cluster is presented as a single database to

application servers.

The shared storage of the Oracle RAC Hub nodes includes Oracle Cluster Registry (OCR),

voting disks, and database. OCR records information about node statuses, voting disks

synchronize data between nodes, and the database is a set of files.

Figure 2-3 Oracle Flex Cluster 12c

Oracle ASM provides a simple storage management interface for database administrators to

manage servers and storage across different platforms. As a built-in file system and volume manager, Oracle ASM is exclusive to Oracle database files. ASM simplifies file system



Architectures

management, provides asynchronous I/O performance tuning, saves management time for

administrators, and offers a flexible, efficient database environment.

ASM can consolidate LUNs into a disk group and use Allocation Units (AUs) to allocate

storage space from the disk group. ASM supports three types of disk groups.

External: Data is not mirrored between LUNs, and the storage system provides data

protection.

Normal: A normal disk group consists of two failure groups between which data is

mirrored.

High: A high disk group consists of three failure groups among which data is mirrored.

When an OceanStor V3 storage system is used to create ASM disk groups, it is recommended

that controllers A and B evenly own the LUNs in the disk groups before external or normal

disk groups are created.

Oracle Flex ASM is a new Oracle ASM deployment model that increases database instance

availability and reduces Oracle ASM related resource consumption. Oracle Flex ASM

facilitates cluster based database consolidation, as it ensures that Oracle Database 12c

instances running on a particular server will continue to operate, should the Oracle Flex ASM

instance on that server fail.

Figure 2-4 ASM before Oracle Database 12c

Figure 2-5 Oracle 12c Flex ASM



Architectures

Oracle RAC provides the following key characteristics, essential for HA data management:

Reliability — Oracle Database is known for its reliability. Oracle RAC takes this step

further by removing the database server as a single point of failure. If an instance fails,

the remaining instances in the cluster remain open and active. Oracle Clusterware

monitors all Oracle processes and immediate restarts any failed component.

Error detection — Oracle Clusterware automatically monitors all Oracle RAC databases

as well as other Oracle processes (Oracle ASM, instances, Listeners, etc.) and provides

fast detection of problems in the environment. It also automatically recovers from

failures often before users notice that a failure has occurred.

Recoverability — The Oracle Database includes many features that make it easy to

recover from various types of failures. If an instance fails in an Oracle RAC database, it

is recognized by another instance in the cluster and recovery will start automatically. Fast

Application Notification (FAN) and Fast Connection Failover (FCF) and especially the

Oracle RAC 12c Application Continuity feature make it easy to mask any component

failure from the user.

Continuous Operations — Oracle RAC provides continuous service for both planned and

unplanned outages. If a server (or an instance) fails, the database remains open and

applications continue to be able to access data, allowing for business critical workloads

to finish, mostly without a delay in service delivery.

For more information about Oracle 12c RAC and ASM, refer to the following documents:

White Paper: Oracle Real Application Clusters (RAC)

A Technical Overview of New Features for Automatic Storage Management in Oracle

Database 12c

2.2.2 Oracle System Architecture

Figure 2-6 Oracle system architecture

http://www.oracle.com/technetwork/database/options/clustering/rac-wp-12c-1896129.pdf

http://www.oracle.com/technetwork/products/cloud-storage/oracle-12c-asm-overview-1965430.pdf

http://www.oracle.com/technetwork/products/cloud-storage/oracle-12c-asm-overview-1965430.pdf



Architectures

As shown in the preceding figure, System Global Area (SGA) and Program Global Area

(PGA) of Oracle databases consume memory. SGA stores system information and page cache

information, and PGA stores session information. SGA contains the following parts:

Buffer Cache: buffers data blocks.

Redo Log Buffer: buffers log records as a recycle data group.

Share Pool: buffers data dictionaries and shared SQL information.

Oracle files are categorized as follows:

Control file: records the database structure, parameters, and locations of other data fails.

Data file: stores user data and temporary data.

Online log: record changes to data blocks and consists of several log groups. Files in the

log groups are mirrored to each other. After a log group is used up, data in written to the

next log group. After the first log group is used up, data is written to the first log group

again.

Archive log: In archive mode, databases copy fully written log groups to the archive area

for data restoration when anomalies occur.

Among Oracle processes, the Listener process monitors client connections. Clients are

connected in two modes:

In shared mode, the listener redirects client requests to the dispatcher process, which

places the request in the request queue of the large pool. Then the shared server obtains

and processes the request in the request queue and puts the processing results in the

response queue of the large pool. At last, the dispatcher returns the processing results to

the client.

In dedicated mode, a dedicated server process serves each client connection. After

receiving the request from a client, the Oracle server process looks for the data block in

the buffer cache. If the data block is found, data is read, computed, and changed in the

buffer cache. If the data block does not exist in the buffer cache, the Oracle server

process writes the data block from data files to the buffer cache, and then computes and

changes it.

Oracle uses the LRU algorithm to eliminate outdated data in the buffer cache so that the

released storage space can be used by new data blocks. Data that has been changed in the

buffer cache is called "dirty data", which is written by the DB writer (DBW) process to data

files.

To ensure data integrity and reliability, relational databases use "transaction" to indicate an

atomic operation. When processing a transaction, the Oracle server process records changed

data and the change time in the log buffer. When the transaction is delivered, the log writer

(LGWR) synchronizes the log data in the log buffer to online log files. The log buffer is a

memory area where data can be written in a circular manner. When the log buffer is one-third

full, the LGWR synchronizes the log records to the log file, regardless of whether the

transaction is committed. In addition, Oracle databases synchronize logs automatically every

three minutes.

By default, Oracle databases perform a checkpoint every 30 minutes. When the checkpoint is

performed, the Checkpoint (CKPT) process synchronizes the control file and triggers the

DBW to write dirty data to data files.

Online logs of Oracle are divided into several groups, each of which contains one or multiple

log files. When multiple log files exist, the files are mirrored to each other. Oracle databases

write logs to the log groups in sequence. When the last log group is filled, the databases write

to the first log group, restarting the cycle. Before a log group switch, Oracle checks whether



Architectures

the dirty data recorded in the next log group is completely written to data files. If not, Oracle

waits until the DBW process writes all the dirty data to the data files before it starts the log

group switch.

When an Oracle database is in archive mode, the Archive (ARC) process copies filled logs to

the archive area. If data anomalies occur, the archived logs are used for precise data recovery.

Oracle Database 12c also supports a Multitenant that allows multiple PDBs to run in one

CDB. Figure 2-7 shows a CDB with four containers: the root, seed, and two PDBs (hrpdb and

salespdb). Each PDB has its own dedicated application. A different PDB administrator

manages each PDB. User SYS can manage the root and every PDB. At the physical level, this

CDB has a database instance and database files. The Multitenant feature brings better

serviceability to the Oracle Database.

The seed PDB is a system-supplied template that the CDB can use to create new PDBs. The seed PDB is

named PDB$SEED. You cannot add or modify objects in PDB$SEED.

Figure 2-7 Oracle Multitenant architecture

For more information about Oracle Multitenant Architecture, refer to the following document:

White Paper: Oracle Multitenant

2.2.3 Oracle Application Types

Data transactions are categorized as two types: OLTP and OLAP.

http://www.oracle.com/technetwork/database/multitenant-wp-12c-1949736.pdf



Architectures

OLTP: A number of online users perform transactions.

OLAP: A small number of users perform long-term complex statistical queries.

OLTP applications have the following I/O characteristics:

From the perspective of database

The reading, writing, and changing of each transaction involve a small amount of data.

Database data must be up-to-date. Therefore, OLTP applications require a high database

availability.

Many users connect to and use the database concurrently.

The database must be highly responsive and able to complete a transaction within

seconds.

From the perspective of storage

Every I/O is small-sized, ranging from 2 KB to 8 KB.

Disk data is randomly accessed.

At least 30% of data is generated by random writing operations.

Redo logs are written frequently.



Architectures

3 Tiny-Size OLTP Database Reference Architecture

3.1 Huawei Solution

3.1.1 Solution Architecture

Figure 3-1 Tiny-size OLTP database reference architecture





four-port Fibre Channel I/O modules), 1 x disk enclosure (25 x 600 GB 10k rpm SAS disks), 1 x disk



GB LUNs (eight LUNs used as the data area, and two LUNs used as the log area).

5300 V3 [2]


2 x

12

Gb

it/s

SA

S

4 x 8 Gbit/s FC

1 x disk enclosureOracle 12c RAC

2 x RH2288 V2 [1]


S6700 10GE switch


Tiny-Size OLTP Database Solution2 TB[3] data, 20,000[4] transaction IOPS



Architectures

[3] The tested database contains 2.5 TB of table and index data.


3.1.2 Solution Configuration

Table 3-1 Hardware configuration

Solution Hardware Component Quantity

Tiny-Size OLTP

Database

Solution

Server: RH2288

V2

2

Intel® Xeon

® E5-2660 @ 2.20 GHz 2

Memory 256 GB

QLogic 8 Gbit/s dual-port Fibre

Channel HBA

1

Intel 10 Gbit/s dual-port Ethernet

HBA

1

Storage:

OceanStor 5300

V3

1

Cache 32 GB

2 U controller enclosure with 25 slots 1

2 U disk enclosure with 25 slots 1

600 GB 10k rpm SAS disk 43

200 GB SLC SSD 7

8 Gbit/s four-port Fibre Channel I/O

module

2

Fibre Channel

switch: SNS2224

2

Private network

switch: S6700

1

Table 3-2 Software configuration

Solution Hardware Software Quantity

Tiny-Size OLTP

Database

Solution

RH2288 V2 2

Operating system: Red Hat Enterprise

Linux 6.5

Multipathing software: UltraPath for

Linux 8.01.024



Architectures


Database cluster: Oracle Grid 12.1.0.2

Database software: Oracle Database

12.1.0.2

Test tool: Huawei SwingBench Test

Suite 1.0 for Oracle 12c

OceanStor

5300 V3

1

SmartTier license

SmartThin license

SmartMotion license

SmartQoS license

Figure 3-2 Storage configuration

[1] The disk domain configuration is as follows: The hot spare policy of the SAS tier is High (default

policy), and that of the SSD tier is Low.

[2] The storage pool configuration is as follows: RAID 10 and 6 TB capacity are configured for the SAS

tier. RAID 5-5 and 800 GB capacity are configured for the SSD tier. The SmartTier monitoring period is

from 08:00 to 18:00. The data migration plan is carried out from 02:00 to 06:00 each day. Other

parameters keep the default values.

[3] The LUN configuration of the data area is as follows: The SmartTier policy is automatic migration.

LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.

The LUN configuration of the log area is as follows: The read/write policy is reclamation. The priority is

high. LUNs are evenly allocated to controllers A and B. Other parameters keep the default values.

Disk

Disk Domain [1]

LUN [3]

ASM Disk Group [4]+DATA +LOG+GRID

…43 x 600 GB SAS disks

13 TB allocated 8 TB free 1.7 TB hot spare

Storage Pool [2] 1844 GB free

1 x 100 GB LUN

…

8 x 500 GB LUNs 2 x 500 GB LUNs

Oracle Database [5]

2.5 TB table & index data

Order Entry Workload

…

… 7 x 200 GB SLC SSDs

5100 GB allocated



Architectures

[4] The external redundancy policy is configured for all ASM disk groups. Other parameters keep the

default values.

[5] For details about the database configuration, see Table 3-4.

3.2 Verification Procedure

Step 1 Prepare the environment.

Deploy a network based on the solution architecture. Configure the OceanStor 5300 V3

converged storage system and create disk domains, storage pools, LUNs, and LUN groups as

instructed in the OceanStor V3 converged storage system user manual.

Pre-configure host operating systems as instructed in the Oracle 12c cluster deployment guide.

Set host parameters based on the following table. Install UltraPath as instructed in the

UltraPath user manual. Scan for hosts on the OceanStor 5300 V3 converged storage system.

Create host groups and mapping views.

Table 3-3 Host parameter settings

Configuration File Parameter Value

/etc/sysctl.conf vm.nr_hugepages 102400 (2 MB)

kernel.shmmax 214748364800 (bytes)

kernel.shmall 52428800 (4 KB)

/etc/security/limits.conf oracle soft nproc 16384

oracle hard nproc 65536

oracle soft nofile 16384

oracle hard nofile 65536

oracle soft memlock 209715200 (1 KB)

oracle hard memlock 209715200 (1 KB)

/sys/block/sd*/queue/scheduler deadline

/sys/block/sd*/queue/max_sectors_kb 1024

On each host, run the following commands to scan for LUNs, create UDEV policies, and

generate UDEV devices. (The commands only apply to Red Hat Enterprise Linux 6 and

UltraPath V100R008.)

upadm start hotscan

upadm show lun array=0 | grep -v LUNV | awk '/LUN Name/{printf "KERNEL==\"sd?*\",

BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id -g -u /dev/$name\", RESULT==\"3%s\",

NAME=\"huawei/%s\", OWNER=\"oracle\", GROUP=\"dba\", MODE=\"0660\"\n",$6,$9}' >

/etc/udev/rules.d/99-huawei-devices.rules

udevadm control --reload-rules

start_udev

ls -l /dev/huawei/



Architectures

Step 2 Deploy cluster and database software.

Deploy Oracle 12c cluster and database software as instructed in the Oracle 12c cluster and

database configuration guide. Use DBCA to create a container database. Set

db_create_file_dest to +DATA. Create five 128 MB log groups for each instance and enable

the archive mode. Place online logs and archive logs to the +LOG disk group. Set database

parameters based on the following table.

Table 3-4 Database parameter settings

Parameter Value

db_create_file_dest +DATA

db_block_size 8192

db_file_multiblock_read_count 128

sga_target 160 GB

pga_aggregate_target 40 GB

lock_sga TRUE

use_large_pages ONLY

db_files 512

process 1024

parallel_max_servers 256

fast_start_mttr_target 30

db_writer_processes 2

Step 3 Load data.

Use Huawei SwingBench Test Suite 1.0 for Oracle 12c to run data loading scripts that create

pluggable databases pdbt, pdbs, and pdbm and load test data at a scale of 250, 500, and 1000

for the three databases respectively (about 2 TB in total).

sh 0.load.sh pdbt 250 DATA

sh 0.load.sh pdbs 500 DATA

sh 0.load.sh pdbm 1000 DATA

The parameters of a data loading script (format: 0.load.sh pdb scale asmdg) are described as follows:

pdb: pluggable database name

scale: data scale, with about 1 GB per unit

asmdg: target ASM disk group without the plus symbol (+)

Step 4 Perform a warm-up test.

Run the following OLTP script to start a one-hour test. After that, manually start hotspot data

migration based on SmartTier.

sh 1.run.sh -p ora -c 2 -d pdbt,pdbs,pdbm -n 20,40,80 -i 10 -r 60 -w 10 -o xxx -s xxx



Architectures

The parameters of an OLTP script (format: 1.run.sh -p NODE_PREFIX -c NODE_COUNT -d

PDB1,PDB2,PDB3... -n NumUsers1,NumUsers2,NumUsers3... -i INTERVAL -r

RUN_TIME_MINS -w WARMUP_MINS -o OS_PSWD -s SYSTEM_PSWD) are described as

follows:

-p NODE_PREFIX Oracle RAC node prefix, for example, RAC nodes are

'ora1,ora2,ora3', then set NODE_PREFIX to 'ora'

-c NODE_COUNT: Number of Oracle RAC nodes to run the test, if set to 2, then

swingbench will connect PDBs of 'ora1' and 'ora2'

-d PDB1,... PDBs to connect

-n NumUsers1,... Sessions to create for PDBs

-i INTERVAL Performance statistic & swingbench verbose print interval

seconds

-r RUN_TIME_MINS Run minutes, including warmup time

-w WARMUP_MINS Warm up minutes

-o OS_PSWD current OS user password

-s SYSTEM_PSWD password of dba user 'system'

In DeviceManager of OceanStor 5300 V3 converged storage system, manually migrate

hotspot data as follows:

1. Click the Provisioning icon on the right.

2. Click Resource Performance Tuning.

3. Click SmartTier.

4. Set Data Migration Speed to High.

5. Select the desired storage pool.

6. Choose More > Start.

Figure 3-3 Manually executing SmartTier hotspot data identification

Step 5 Start OLTP testing.

Run the following OLTP script to keep increasing the number of users until 20,000 transaction

IOPS is reached (the read latency is shorter than 10 ms.)

sh 1.run.sh -p ora -c 2 -d pdbt,pdbs,pdbm -n 35,70,140 -i 10 -r 20 -w 10 -o xxx -s xxx

---End



Architectures

3.3 Verification Results

Table 3-5 Verification results

Database Amount of Data

Session Quantity

TPS IOPS I/O Latency

Disk Quantity

PDBT 389 GB 70 414

PDBS 730 GB 140 905

PDBM 1463 GB 280 1610

CDB 38 GB

Total 2620 GB 490 2929 28,877 8.83 ms 43 SAS

disks, 7 SSDs

Objective 2 TB 2000 20,000

Figure 3-4 OLTP test process

The y-axis indicates IOPS. The x-axis indicates the execution duration (expressed in minutes). riops

indicates read IOPS. wiops indicates write IOPS.

0

5000

10000

15000

20000

25000

30000

35000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

riops wiops



Architectures

4 Small-Size OLTP Database Reference Architecture

4.1 Huawei Solution


Figure 4-1 Small-size OLTP database reference architecture



5500 V3 [2]


4 x

12

Gb

it/s

SA

S

4 x 8 Gbit/s FC

3 x disk enclosures

Oracle 12c RAC

4 x RH2288 V2 [1]


S6700 10GE switch


Small-Size OLTP Database Solution4 TB[3] data, 40,000[4] transaction IOPS



Architectures



four-port Fibre Channel I/O modules), 3 x disk enclosures (75 x 600 GB 10k rpm SAS disks), 1 x disk



GB LUNs (16 LUNs used as the data area, and 4 LUNs used as the log area).






Small-Size

OLTP Database

Solution

Server: RH2288

V2

4

Intel® Xeon

® E5-2660 @ 2.20 GHz 2

Memory 256 GB


Channel HBA

1


HBA

1

Storage:

OceanStor 5500

V3

1

Cache 48 GB

2 U controller enclosure with 25

disk slots

1



200 GB SLC SSD 12

8 Gbit/s four-port Fibre Channel

I/O module

2

Fibre Channel

switch: SNS2224

2

Private network

switch: S6700

1



Small-Size Data RH2288 V2 4



Architectures


Warehouse

Solution Operating system: Red Hat Enterprise

Linux 6.5


Linux 8.01.024



12.1.0.2



OceanStor

5500 V3

1

SmartTier license

SmartThin license

SmartMotion license

SmartQoS license




Disk

+DATA +LOG+GRID

...88 x 600 GB SAS disks

25.8 TB allocated 17.6 TB free 2.7 TB hot spare

3788 GB free

1 x 100 GB LUN

...


5 TB table & index data


...

... 12 x 200 GB SLC SSDs

10,100 GB allocated

Disk Domain [1]

LUN [3]

ASM Disk Group [4]

Storage Pool [2]

Oracle Database [5]



Architectures

[2] The storage pool configuration is as follows: RAID 10 and 12 TB capacity are configured for the

SAS tier. RAID 5-9 and 1600 GB capacity are configured for the SSD tier. The SmartTier monitoring

period is from 08:00 to 18:00. The data migration plan is carried out from 02:00 to 06:00 each day.

Other parameters keep the default values.






default values.




For details, see section 3.2 "Verification Procedure."



Step 3 Load data.


pluggable databases pdbs, pdbm, and pdbl and load test data at a scale of 500, 1000, and

2000 for the three databases respectively (about 4 TB in total).



sh 0.load.sh pdbl 2000 DATA

For details about the script parameter meanings, see section 3.2 "Verification Procedure."




Run an OLTP script to keep increasing the number of users until 40,000 transaction IOPS is

reached (the read latency is shorter than 10 ms.)

sh 1.run.sh -p ora -c 4 -d pdbs,pdbm,pdbl -n 28,55,110 -i 10 -r 20 -w 10 -o xxx -s xxx

---End




Session Quantity


Disk Quantity

PDBS 730 GB 112 665



Architectures


Session Quantity


Disk Quantity

PDBM 1463 GB 220 1347

PDBL 2943 GB 440 2576

CDB 38 GB

Total 5140 GB 772 4588 44,172 7.84 ms 88 SAS disks,

12 SSDs

Objective 4 TB 4000 40,000




0

10000

20000

30000

40000

50000

60000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 19 20

riops wiops



Architectures

5 Medium-Size OLTP Database Reference Architecture

5.1 Huawei Solution


Figure 5-1 Medium-size OLTP database reference architecture



6 x

12 G

bit/s

SA

S

4 x 8 Gbit/s FC

6 x disk enclosures

Oracle 12c RAC

6 x RH2288 V2 [1]


S6700 10GE switch


Medium-Size OLTP Database Solution6 TB[3] data, 60,000[4] transaction IOPS

5600 V3 [2]




Architectures


cache, 1 x controller enclosure (2 x 8 Gbit/s four-port Fibre Channel I/O modules, 2 x 12 Gbit/s










Medium-Size

Data

Warehouse

Solution

Server: RH2288

V2

6

Intel® Xeon

® E5-2660 @ 2.20 GHz 2

Memory 256 GB


Channel HBA

1


HBA

1

Storage:

OceanStor 5600

V3

1

Cache 64 GB

3 U controller enclosure 1



200 GB SLC SSD 18


I/O module

2

12 Gbit/s four-port SAS I/O module 2

Fibre Channel

switch: SNS2224

2

Private network

switch: S6700

1



Architectures



Medium-Size

Data

Warehouse

Solution

RH2288 V2 6


Linux 6.5


Linux 8.01.024



12.1.0.2



OceanStor

5600 V3

1

SmartTier license

SmartThin license

SmartMotion license

SmartQoS license




+DATA +LOG+GRID


38.6 TB allocated 27.4 TB free 3 TB hot spare

5732 GB free

1 x 100 GB LUN

...




...


15,100 GB allocated

Disk Domain [1]

LUN [3]

ASM Disk Group [4]

Storage Pool [2]

Oracle Database [5]

Disk



Architectures










default values.







Step 3 Load data.


pluggable databases pdbs, pdbm, and pdbxl and load test data at a scale of 500, 1000, and

4000 for the three databases respectively (about 6 TB in total).



sh 0.load.sh pdbxl 4000 DATA







sh 1.run.sh -p ora -c 6 -d pdbs,pdbm,pdbxl -n 30,60,120 -i 10 -r 20 -w 10 -o xxx -s

xxx

---End



Architectures




Session Quantity


Disk Quantity

PDBS 752 GB 180 531

PDBM 1483 GB 360 1097

PDBXL 5759 GB 720 3870

CDB 41 GB

Total 8035 GB 1260 5498 63,477 9.66 ms 132 SAS disks,

18 SSDs

Objective 6 TB 60,000




0

10000

20000

30000

40000

50000

60000

70000

80000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 17 18 19 20

riops wiops



Architectures

6 Large-Size Transaction Database Reference Architecture

6.1 Huawei Solution


Figure 6-1 Large-size transaction database reference architecture

8 x

12 G

bit/s

SA

S

Oracle 12c RAC

8 x RH2288 V2 [1]


S6700 10GE switch


Large-Size OLTP Database Solution8 TB[3] data, 80,000[4] transaction IOPS

8 x disk enclosures

5800 V3 [2]


4 x 8 Gbit/s FC



Architectures



[2] The detailed configuration of the OceanStor 5800 V3 converged storage system is as follows: 128

GB cache, 1 x controller enclosure (1 x 8 Gbit/s four-port Fibre Channel I/O module, 2 x 12 Gbit/s










Large-Size Data

Warehouse

Solution

Server: RH2288

V2

8

Intel® Xeon

® E5-2660 @ 2.20 GHz 2

Memory 256 GB

QLogic 8 Gbit/s FC dual-port HBA 1


HBA

1

Storage:

OceanStor 5800

V3

1

Cache 128 GB

3 U controller enclosure 1



200 GB SLC SSD 25


I/O module

2

12 Gbit/s four-port SAS I/O module 2

Fibre Channel

switch: SNS2224

2

Private network

switch: S6700

1



Architectures



Large-Size Data

Warehouse

Solution

RH2288 V2 8


Linux 6.5


Linux 8.01.024



12.1.0.2



OceanStor

5800 V3

1

SmartTier license

SmartThin license

SmartMotion license

SmartQoS license


+DATA +LOG+GRID


51.5 TB allocated 36.6 TB free 3.7 TB hot spare

7676 GB free

1 x 100 GB LUN

...




...


20,100 GB allocated

Disk Domain [1]

LUN [3]

ASM Disk Group [4]

Storage Pool [2]

Oracle Database [5]

Disk



Architectures












default values.







Step 3 Load data.


pluggable databases pdbm, pdbl, and pdbxland and load test data at a scale of 1000, 2000,

and 4000 for the three databases respectively (about 8 TB in total).


sh 0.load.sh pdbl 2000 DATA

sh 0.load.sh pdbxl 4000 DATA







sh 1.run.sh -p ora -c 8 -d pdbm,pdbl,pdbxl -n 45,90,180 -i 10 -r 20 -w 10 -o xxx -s

xxx

---End



Architectures




Session Quantity


Disk Quantity

PDBM 1481 GB 360 1023

PDBL 2912 GB 720 2016

PDBXL 5753 GB 1440 3960

CDB 39 GB

Total 10,185 GB 1260 6999 86,429 9.54 ms 175 SAS disks,

25 SSDs

Objective 8 TB 80,000




0

20000

40000

60000

80000

100000

120000

0 1 2 3 4 5 6 7 8 9 11 12 13 14 15 16 17 18 19 20

riops wiops



Architectures

7 Appendix

7.1 Reference Documents

For details about the best practices for deploying OLTP Oracle Database based on HUAWEI

OceanStor V3 converged storage systems, refer to the following document:

Planning and Configuring HUAWEI OceanStor V3 Converged Storage Systems to Maximize OLTP Oracle Database's Performance and Availability

7.2 Acronyms and Abbreviations

Table 7-1 Acronyms and abbreviations

Acronym and Abbreviation Full Name

ASM Automatic Storage Management

ARC Archive (Oracle log archive process)

AU Allocation Unit (of an ASM disk group)

BI Business Intelligence

CDB Oracle Container Database

CK Chunk

CKG Chunk Group

CKPT Checkpoint (Oracle checkpoint)

DBW DB Writer (Oracle disk flush process)

DG Disk Group

ETL Extract, Transform, Load (OLAP data processing phase)

FC Fibre Channel

FCoE Fibre Channel over Ethernet

GE Gigabytes Ethernet

HBA Host Bus Adapter



Architectures

Acronym and Abbreviation Full Name

HDD Hard Disk Drive

LUN Logical Unit Number (Huawei storage volume)

OCR Oracle Cluster Registry

OLAP Online Analytical Processing (oriented to complex

analytical query)

OLTP Online Transaction Processing (oriented to multi-user

online transactions)

PDB Oracle Pluggable Database

PGA Program Global Area (a memory region that contains

data and control information for a server process)

SAS Serial Access SCSI

SAN Storage Area Network

SGA System Global Area (a group of shared memory

structures that contain data and control information for

one Oracle database instance)

SSD Solid State Disk

NAS Network Attached Storage

NL-SAS Near-line SAS

RAC Real Application Clusters

VIP Virtual IP



Architectures

Huawei Technologies Co., Ltd.

Address: Huawei Industrial Base

Bantian, Longgang

Shenzhen 518129

People's Republic of China

Website: http://www.huawei.com

Email: [email protected]

Copyright © Huawei Technologies Co., Ltd. 2015. All rights reserved.

No part of this document may be reproduced or transmitted in any form or by any means without prior written

consent of Huawei Technologies Co., Ltd.

Trademarks and Permissions

and other Huawei trademarks are trademarks of Huawei Technologies Co., Ltd.

All other trademarks and trade names mentioned in this document are the property of their respective holders.

Notice

The purchased products, services and features are stipulated by the contract made between Huawei and the

customer. All or part of the products, services and features described in this document may not be within the

purchase scope or the usage scope. Unless otherwise specified in the contract, all statements, information,

and recommendations in this document are provided "AS IS" without warranties, guarantees or

representations of any kind, either express or implied.

The information in this document is subject to change without notice. Every effort has been made in the

preparation of this document to ensure accuracy of the contents, but all statements, information, and

recommendations in this document do not constitute a warranty of any kind, express or implied.

http://www.huawei.com/

mailto:[email protected]