Deploy Large Unstructured Data Storage in a Small Data Center ... - · PDF fileMount the GlusterFS volume gvol0 on client 1: # mkdir /mnt/gv0 # mount -t glusterfs gluster1:/gv0 /mnt/gv0

© 2015 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 25

White Paper

Deploy Large Unstructured Data Storage in a Small Data Center Space: Cisco UCS C240 M4 Rack Server and Red Hat Gluster Storage 100-TB Solution

Executive Summary

Today, companies face scenarios that require an IT architecture that can hold tens to hundreds of terabytes of

unstructured data. Storage-intensive enterprise workloads can encompass the following:

● Archive and backup content, including backup images and near-online (nearline) archives

● Rich media content storage and delivery, such as videos, images, and audio files

● Enterprise drop-box content

● Cloud and business applications, including log files, and RFID and other machine-generated data

● Virtual and cloud infrastructure, such as virtual machine images

● Emerging workloads, such as co-resident applications

This document is intended to help organizations seeking an ultra-dense, high-throughput solution that can store up

to 100-TB of unstructured data in a small amount of rack space. The document provides test results that

demonstrate how Cisco UCS® C240 M4 Rack Servers and Red Hat Gluster Storage, along with Cisco Unified

Computing System™

(Cisco UCS) fabric interconnects, can be optimized to serve in these scenarios. It includes

system setup information, test methodology, and results for a 100-terabyte (TB) solution. Initial tests indicate that

the Cisco® and Red Hat technologies together provide an ultra-dense, scalable, high-throughput solution that can

store unstructured data that is easy to manage.

Solution Overview

The Cisco UCS C240 M4 Rack Server and Red Hat Gluster Storage provide a complete 100-TB solution for high-

volume, unstructured data storage.

Cisco UCS C240 M4 Rack Server

The Cisco UCS C240 M4 large form-factor (LFF) server is the newest 2-socket, 2-rack-unit (2RU) rack server from

Cisco. It is designed for both performance and expandability over a wide range of storage-intensive infrastructure

workloads, from big data to collaboration.

The enterprise-class Cisco UCS C240 M4 small form-factor (SFF) server extends the capabilities of the Cisco UCS

portfolio in a 2RU form factor with the addition of the Intel® Xeon

® processor E5-2600 v3 series, which delivers the

best combination of performance, flexibility, and efficiency gains.

In addition, the Cisco UCS C240 M4 LFF server provides 24 DIMM slots, up to 6 PCI Express (PCIe) 3.0 slots, and

up to 12 front-loading LFF drives plus 2 (optional) internal SFF SATA boot drives for a total of 14 internal drives.


The Cisco UCS C240 M4 server includes a modular LAN-on-motherboard (mLOM) slot for installation of a Cisco

virtual interface card (VIC) or third-party network interface card (NIC) without consuming a PCI slot; it also includes

two 1 Gigabit Ethernet embedded (on the motherboard) LOM ports. These features together provide an

outstanding amount of internal memory and storage expandability plus exceptional performance.

The Cisco UCS C240 M4 can be used as a standalone server or as part of the Cisco Unified Computing System,

which unifies computing, networking, management, virtualization, and storage access into a single integrated

architecture enabling end-to-end server visibility, management, and control in both bare-metal and virtualized

environments.

Cisco UCS 6200 Series Fabric Interconnects

The Cisco UCS 6200 Series Fabric Interconnects are a core part of Cisco UCS, providing both network

connectivity and management capabilities for the system. The Cisco UCS 6200 Series offers line-rate, low-latency,

lossless 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE), and Fibre Channel functions.

The Cisco UCS 6200 Series provides the management and communications backbone for the Cisco UCS Servers

and 5100 Series Blade Server Chassis. All chassis, and therefore all blades and rack servers, attached to the

Cisco UCS 6200 Series Fabric Interconnects become part of a single, highly available management domain. In

addition, by supporting unified fabric, the fabric interconnect provides both the LAN and SAN connectivity for all

blades within its domain.

The Cisco UCS 6200 Series uses a cut-through networking architecture, supporting deterministic, low-latency, line-

rate 10 Gigabit Ethernet on all ports, switching capacity of 2 terabits (Tb), and 320 Gbps of bandwidth per chassis,

independent of packet size and enabled services. The product family supports Cisco low-latency, lossless 10

Gigabit Ethernet unified network fabric capabilities, which increase the reliability, efficiency, and scalability of

Ethernet networks. The fabric interconnect supports multiple traffic classes over a lossless Ethernet fabric from the

blade through the interconnect. Significant savings in total cost of ownership (TCO) come from an FCoE-optimized

server design in which NICs, host bus adapters (HBAs), cables, and switches can be consolidated.

Red Hat Gluster Storage

Red Hat Gluster Storage is open, software-defined scale-out storage that easily manages unstructured data for

physical, virtual, and cloud environments. Combining both file and object storage with a scale-out architecture

(Figure 1), it is designed to cost-effectively store and manage petabyte-scale data even as the amount of data

continues to grow. It also delivers a continuous storage fabric across physical, virtual, and cloud resources so that

organizations can transform their big, semistructured, and unstructured data from a burden into an asset.


Figure 1. Red Hat Gluster Storage Scales Up and Out

Built on the industry-leading Red Hat Enterprise Linux (RHEL) operating system, Gluster Storage offers cost-

effective and highly available storage without scale or performance compromises (Table 1). Organizations can use

it to avoid storage silos by enabling global access to data through multiple file and object protocols. And it works

transparently with Cisco UCS C240 M4 servers.

Table 1. Red Hat Gluster Storage Features

Single global namespace Aggregates disk and memory resources into a single trusted storage pool

Replication Supports synchronous replication within a data center and asynchronous replication for disaster recovery

Snapshots Helps ensure data protection through clusterwide filesystem snapshots that are user accessible for easy recovery of files

Elastic hashing algorithm Eliminates performance bottlenecks and single points of failure because it does not have any metadata server layer

Easy online management ● Web-based management console

● Powerful and intuitive command-line interface (CLI) for Linux administrators

● Monitoring (Nagios based)

● Capability to expand and shrink storage capacity without downtime

Industry-standard client support ● Network File System (NFS), Server Message Block (SMB) for file-based access

● OpenStack Swift for object access

● GlusterFS native client for highly parallelized access

System Specifications

This 100-TB solution includes Cisco UCS C240 M4 Rack Servers, Red Hat Gluster Storage, and Cisco UCS

6248UP 48-Port Fabric Interconnects. For the purpose of running the performance tests, the components of the

solution were configured to the specifications described in this section. Figure 2 provides an overview of the

system.


Figure 2. Solution Diagram

Component Configuration

Table 2 describes how the components of the solution were configured for the tests.

Table 2. Component Configuration

Component Configuration

Cisco UCS C240 M4 Rack Server 4 Cisco UCS C240 M4 servers, each configured with

● 12 x 6-TB 7200-rpm disks

● 2 Intel Xeon processor E5-2660 v3 CPUs

● 256 GB of RAM (sixteen 16-GB DDR4 at 2133 MHz)

● 1 Cisco UCS VIC 1227 dual-port 10-Gbps Enhanced Small Form-Factor Pluggable (SFP+)

● 1 Cisco 12-Gbps SAS RAID controller with a 4-GB flash-backed write cache (FBWC)

● 2 internal 2.5-inch SATA solid-state disk (SSD) drives

Operating system Red Hat Enterprise Linux 6.6

Red Hat Gluster Storage software Red Hat Gluster Storage 3.0.4

Connectivity 2 Cisco UCS 6248UP 48-Port Fabric Interconnects supporting 1- and 10-Gbps SFP+

RAID Configuration

The RAID configuration includes the following:

● Each Cisco UCS C240 M4 server includes 12-disk RAID 6 arrays for the Gluster Storage bricks, created

using the LSI RAID controller graphical configuration utility.

● The RAID controller cache is set to write back with battery-backed write cache.

● Each RAID 6 array includes one virtual disk (60 TB), and each virtual disk has a capacity of 55 TB after

formatting with XFS.

Appendix A shows the StorCLI output results.

Network Configuration

The network is configured as follows:

● Each Cisco UCS C240 M4 server includes a Cisco UCS VIC 1227 adapter.


● Each VIC 1227 adapter has two 10-Gbps ports, with one port connected to fabric interconnect A and the

other port connected to fabric interconnect B. With this configuration, each Cisco UCS C240 M4 has an

active-passive 2 x 10-Gbps connection to each fabric interconnect.

Red Hat Gluster Storage Installation and Configuration

Installation and configuration of Red Hat Gluster Storage consists of several steps:

1. Install the Red Hat Gluster Storage 3.0.4 ISO image on one of the two internal SSDs in the Cisco UCS C240

M4 servers. This release is based on RHEL 6.6 and GlusterFS 3.6. For step-by-step installation and

configuration guidance for Gluster Storage, visit https://access.redhat.com/site/documentation/en-

US/Red_Hat_Storage/.

2. Install RHEL 7.1 on the client servers, following instructions in the administration guide for installing the native

client in Red Hat Gluster Storage.

3. Create storage bricks, using the rhs-server-init.sh script on all nodes.

4. Make sure that the glusterd daemon is running on each of the Red Hat Gluster Storage servers (gluster5,

gluster6, gluster7, and gluster8), and then use the gluster peer probe command to create the trusted storage

cluster from the gluster1 server:

# gluster peer probe gluster6



5. Confirm that all the storage servers are in a connected state using the gluster peer status command:

# gluster peer status

6. Create the distributed replicated (two-way) volume from the gluster1 server. (See Appendix B for the full

volume-creation script.)

7. Start the volume, and look at the status:

The rhs-server-init.sh script does the following:

● Creates a physical volume.

● Creates a logical volume.

● Places the XFS file system on the logical volume.

● Runs a tuned performance profile for Red Hat Gluster Storage virtualization.

https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/


https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Native_Client.html

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Native_Client.html


# gluster volume start gv0

# gluster volume info

# gluster volume status

8. Confirm that all the storage servers are in a connected state.

9. Mount the GlusterFS volume gvol0 on client 1:

# mkdir /mnt/gv0

# mount -t glusterfs gluster1:/gv0 /mnt/gv0

10. Repeat these commands on the other 15 clients.

Note: Even though the same Red Hat Gluster Storage server is being used for mounting the volumes, the native

client protocol has built-in load balancing. The clients use the mount server initially to get the volume information,

and after that they contact the individual storage servers directly to access the data. The data request does not

have to go through the mount server.


Test Methodology and Results

The test environment was set up to include industry-standard tools to measure read-write performance and to

provide benchmarking for performance across a specific cluster and for performance of distributed storage.

Driver Setup

Eight Cisco UCS B200 M4 servers were used to transfer I/O to the 100-TB Gluster Storage cluster. Each server

was equipped as follows:

● Two Intel Xeon processor E5-2600 series CPUs

● 256 GB of RAM

● One Cisco VIC 1240 capable of 40 Gbps of throughput

The Cisco UCS 5100 Series chassis housing the Cisco UCS B200 M4 servers includes two 8-port Cisco UCS

2208XP Fabric Extender I/O modules connected to the two Cisco UCS 6248UP fabric interconnects.

To transfer I/O, each Cisco UCS B200 M4 server also ran a hypervisor and two virtual switches, with a dedicated

10 Gigabit Ethernet network interface. Each virtual machine was running RHEL 7.1 and the I/O benchmark tools

discussed in the following sections.

Sequential I/O Benchmarking: IOzone Tool

IOzone is a file-system benchmark tool that is useful for performing a broad file-system analysis of a computer

platform. In this case, IOzone was used to test the sequential read-write performance of the GlusterFS replicated

volume. IOzone’s cluster mode option -+m is particularly well-suited for distributed storage testing because testers

can start many worker threads from a variety of client systems in parallel, targeting the GlusterFS volume. The 2x2

distributed replicated volume was tested using the following command with a total of 128 threads running across 16

client systems (8 threads per client):

# iozone -+m ${IOZONE_CONFIG_FILENAME} -i ${IOZONE_TEST} -C -w -+n -s ${IOZONE_FILESZ} -r

${IOZONE_RECORDSZ} -+z -c -e -t ${TEST_THREADS}

The following parameters were used in the IOzone command line:

● -+m specifies the cluster testing mode.

● IOZONE_CONFIG_ FILENAME is the IOzone configuration file for the cluster mode. The file lists the client

host names, the associated GlusterFS mount point, and the location of the iozone utility.

● The IOZONE_TEST parameter varied to cover the sequential-read and sequential-write test cases.

● IOZONE_FILESZ is set to 8 GB. 8-GB file transfers are used as representative of the workload for the

content cloud reference architecture testing.

● IOZONE_RECORDSZ varied between 64 KB and 16 MB, using record sizes that were powers of two. This

range of record sizes is intended to characterize the effect of the record or block size (in client file requests)

on I/O performance.

● TEST_THREADS specifies the number of threads or processes that are active during the measurement.

128 threads are used across 16 client systems (8 threads per client).

Figure 3 shows sequential read throughput of up to 3.8 GBps and up to 2.3-GBps replicated write results for typical

sequential I/O sizes.


Figure 3. Sequential Throughput: IOzone Tool

Performance Benchmarking Across a Cluster: SmallFile Tool

The SmallFile benchmark is a Python-based small-file distributed POSIX workload generator that can be used to

measure performance for a variety of small-file and file metadata-intensive workloads across an entire cluster. The

SmallFile benchmark was used to complement the IOzone benchmark, which is primarily used for measuring the

performance of large-file workloads. Although the SmallFile benchmark kit supports multiple operations, only the

create operation was used to create files and write data to them using varying file sizes ranging from 10 KB to

2 GB.

SmallFile benchmark parameters included the following:

● The create operation is used to create a file and write data to it.

● Eight threads are running on each of the 16 clients (client 1 through client 16).

● The file size is 10 KB, and each thread processes 100,000 files.

● Each thread pauses for 10 microseconds before starting the next file operation, and the response time is

the file operation duration, measured to microsecond resolution.

● For files larger than 1 MB, a record size of 1024 KB is used to determine how much data is transferred in a

single write system call.

The SmallFile command line is as follows:

# python /root/smallfile_cli.py --operation create --threads 8 --file-size

10 --files 100000 --top $path --network-sync-dir /mnt/lockvol/smf-shared

--pause 10 --host-set client1,client2 … client16 --response-times Y


The benchmark returns the number of files processed per second, and it returns the rate that the application

transferred data in MBps.

SmallFile test results show that up to 1530 tiny files per second were created simultaneously, and that write

throughput for the larger files was up to 2.1 GBps (Figure 4). The graph in Figure 4 reflects the trade-off in

throughput (files per second) and transfer rate (MBps) when transferring small files compared large files.

Figure 4. File Create Operations: SmallFile Tool

Random I/O Performance: Flexible I/O Tool

Flexible I/O (FIO) is an industry-standard storage performance benchmark that has been used primarily for single-

server testing. However, with the recent addition of client and server options, it can also be used in a distributed

storage environment. The front end and back end of FIO can be run separately, with the FIO server generating an

I/O workload on the system under test while being controlled from another system. The --server option is used to

launch FIO in a request listening mode. The --client option is used to pass a list of test servers along with the FIO

profile to define the workload to be generated.

FIO was used to determine the random I/O performance using smaller block sizes (ranging from 4 to 32 KB).

Appendix C lists the FIO profile used on each client driver for the random test.

The cluster accommodated about 3000 random read I/O operations per second (IOPS) at typical sizes with a

latency of about 11 milliseconds (ms), as shown in Figure 5.


Figure 5. Random Read Tests: FIO Tool

The solution achieves about 2000 random write IOPS at typical sizes, as shown in Figure 6. Latency is less than

18 ms using the write cache on the array controllers on the Cisco UCS C3160 Rack Server.

Figure 6. Random Write Tests: FIO Tool


Conclusion

Companies facing scenarios that require support for hundreds of terabytes of unstructured data can use a

combination of Cisco UCS C-Series and Red Hat Gluster Storage 100-TB technologies to create a scalable, high-

throughput solution that can meet these demands while remaining easy to manage. Test results using FIO,

SmallFile, and IOzone show remarkable performance and indicate that this solution can handle storage-intensive

enterprise workloads in a dense data center footprint.

For More Information

● Cisco UCS C240 M4 LFF Rack Server

● Cisco UCS 6200 Series Fabric Interconnects

● Red Hat Gluster Storage

● IOzone test

● Large Unstructured Data Storage in a Small Datacenter Footprint: Cisco UCS C3160 and Red Hat Gluster

500-TB Solution

Appendix A: StorCLI Output

[root@gluster5 ~]# /opt/MegaRAID/storcli/storcli64 /c0 show all

Generating detailed summary of the adapter, it may take a while to complete.

Controller = 0

Status = Success

Description = None

Basics :

======

Controller = 0

Model = Cisco 12G SAS Modular Raid Controller

Serial Number = SV50239847

Current Controller Date/Time = 10/07/2015, 15:38:41

Current System Date/time = 10/07/2015, 08:38:43

SAS Address = 578da6e715b66710

PCI Address = 00:0a:00:00

Mfg Date = 01/09/15

Rework Date = 00/00/00

Revision No = 05C

Version :

=======

Firmware Package Build = 24.5.0-0020

Firmware Version = 4.250.00-3632

http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c240-m4-rack-server/datasheet-c78-732455.html

http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-6200-series-fabric-interconnects/data_sheet_c78-675245.html


http://www.iozone.org/

http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/whitepaper_c11_734975.html

http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/whitepaper_c11_734975.html


Bios Version = 6.19.05.0_4.16.08.00_0x06080500

NVDATA Version = 3.1406.00-0079

Ctrl-R Version = 5.06-0004

Boot Block Version = 3.02.00.00-0001

Driver Name = megaraid_sas

Driver Version = 06.803.01.00-rh1

Bus :

===

Vendor Id = 0x1000

Device Id = 0x5D

SubVendor Id = 0x1137

SubDevice Id = 0xDB

Host Interface = PCIE

Device Interface = SAS-12G

Bus Number = 10

Device Number = 0

Function Number = 0

Pending Images in Flash :

=======================

Image name = No pending images

Status :

======

Controller Status = Optimal

Memory Correctable Errors = 0

Memory Uncorrectable Errors = 0

ECC Bucket Count = 0

Any Offline VD Cache Preserved = No

BBU Status = 0

Support PD Firmware Download = No

Lock Key Assigned = No

Failed to get lock key on bootup = No

Lock key has not been backed up = No

Bios was not detected during boot = No

Controller must be rebooted to complete security operation = No

A rollback operation is in progress = No

At least one PFK exists in NVRAM = No

SSC Policy is WB = No

Controller has booted into safe mode = No


Supported Adapter Operations :

============================

Rebuild Rate = Yes

CC Rate = Yes

BGI Rate = Yes

Reconstruct Rate = Yes

Patrol Read Rate = Yes

Alarm Control = Yes

Cluster Support = No

BBU = Yes

Spanning = Yes

Dedicated Hot Spare = Yes

Revertible Hot Spares = Yes

Foreign Config Import = Yes

Self Diagnostic = Yes

Allow Mixed Redundancy on Array = No

Global Hot Spares = Yes

Deny SCSI Passthrough = No

Deny SMP Passthrough = No

Deny STP Passthrough = No

Support more than 8 Phys = Yes

FW and Event Time in GMT = No

Support Enhanced Foreign Import = Yes

Support Enclosure Enumeration = Yes

Support Allowed Operations = Yes

Abort CC on Error = Yes

Support Multipath = Yes

Support Odd & Even Drive count in RAID1E = No

Support Security = Yes

Support Config Page Model = Yes

Support the OCE without adding drives = Yes

Support EKM = Yes

Snapshot Enabled = No

Support PFK = Yes

Support PI = Yes

Support LDPI Type1 = No



Support Ld BBM Info = No

Support Shield State = Yes

Block SSD Write Disk Cache Change = Yes

Support Suspend Resume BG ops = Yes

Support Emergency Spares = No

Support Set Link Speed = Yes


Support Boot Time PFK Change = No

Support JBOD = Yes

Disable Online PFK Change = No

Support Perf Tuning = Yes

Support SSD PatrolRead = Yes

Real Time Scheduler = Yes

Support Reset Now = Yes

Support Emulated Drives = Yes

Headless Mode = Yes

Dedicated HotSpares Limited = No

Point In Time Progress = Yes

Extended LD = No

Boot Volume Supported = No

Support Uneven span = No

Support Config Auto Balance = No

Support Maintenance Mode = No

Support Diagnostic results = Yes

Support Ext Enclosure = No

Support Sesmonitoring = No

Support SecurityonJBOD = No

Support ForceFlash = No

Support DisableImmediateIO = No

Support DrvActivityLEDSetting = No

Support FlushWriteVerify = No

Support CPLDUpdate = No

Support ForceTo512e = Yes

Support discardCacheDuringLDDelete = No

Support JBOD Write cache = No

Supported PD Operations :

=======================

Force Online = Yes

Force Offline = Yes

Force Rebuild = Yes

Deny Force Failed = No

Deny Force Good/Bad = No

Deny Missing Replace = No

Deny Clear = No

Deny Locate = No

Support Power State = Yes

Set Power State For Cfg = No

Support T10 Power State = No

Support Temperature = Yes

NCQ = Yes


Support Max Rate SATA = No

Supported VD Operations :

=======================

Read Policy = Yes

Write Policy = Yes

IO Policy = Yes

Access Policy = Yes

Disk Cache Policy = Yes

Reconstruction = Yes

Deny Locate = No

Deny CC = No

Allow Ctrl Encryption = No

Enable LDBBM = No

Support FastPath = Yes

Performance Metrics = Yes

Power Savings = No

Support Powersave Max With Cache = No

Support Breakmirror = Yes

Support SSC WriteBack = No

Support SSC Association = Yes

Support VD Hide = No

Support VD Cachebypass = No

Support VD discardCacheDuringLDDelete = No

Advanced Software Option :

========================

-----------------------------------------------------

Adv S/W Opt Time Remaining Mode

-----------------------------------------------------

MegaRAID FastPath Unlimited Factory Installed

MegaRAID SafeStore Unlimited Secured

MegaRAID RAID6 Unlimited Secured

MegaRAID RAID5 Unlimited Secured

-----------------------------------------------------

Safe ID = XS66VHB2XEGSM7VFM4D5IUCJ473SFKCDW4GWIW2Z

HwCfg :

=====

ChipRevision = C0

BatteryFRU = N/A


Front End Port Count = 0

Backend Port Count = 8

BBU = Present

Alarm = On

Serial Debugger = Present

NVRAM Size = 32KB

Flash Size = 32MB

On Board Memory Size = 4095MB

CacheVault Flash Size = 16.0 GB

TPM = Absent

Upgrade Key = Present

On Board Expander = Absent

Temperature Sensor for ROC = Present

Temperature Sensor for Controller = Absent

Upgradable CPLD = Absent

Current Size of CacheCade (GB) = 0

Current Size of FW Cache (MB) = 3534

ROC temperature(Degree Celsius) = 58

Policies :

========

Policies Table :

==============

------------------------------------------------

Policy Current Default

------------------------------------------------

Predictive Fail Poll Interval 300 sec

Interrupt Throttle Active Count 16

Interrupt Throttle Completion 50 us

Rebuild Rate 30 % 30%

PR Rate 30 % 30%

BGI Rate 30 % 30%

Check Consistency Rate 30 % 30%

Reconstruction Rate 30 % 30%

Cache Flush Interval 4s

------------------------------------------------

Flush Time(Default) = 4s

Drive Coercion Mode = 1GB

Auto Rebuild = On

Battery Warning = On

ECC Bucket Size = 15


ECC Bucket Leak Rate (hrs) = 24

Restore HotSpare on Insertion = Off

Expose Enclosure Devices = On

Maintain PD Fail History = Off

Reorder Host Requests = On

Auto detect BackPlane = SGPIO/i2c SEP

Load Balance Mode = Auto

Security Key Assigned = Off

Disable Online Controller Reset = Off

Use drive activity for locate = Off

Boot :

====

BIOS Enumerate VDs = 1

Stop BIOS on Error = Off

Delay during POST = 0

Spin Down Mode = None

Enable Ctrl-R = Yes

Enable Web BIOS = No

Enable PreBoot CLI = No

Enable BIOS = Yes

Max Drives to Spinup at One Time = 2

Maximum number of direct attached drives to spin up in 1 min = 10

Delay Among Spinup Groups (sec) = 12

Allow Boot with Preserved Cache = Off

High Availability :

=================

Topology Type = None

Cluster Permitted = No

Cluster Active = No

Defaults :

========

Phy Polarity = 0

Phy PolaritySplit = 0

Strip Size = 64kB

Write Policy = WT

Read Policy = None

Cache When BBU Bad = Off

Cached IO = Off

VD PowerSave Policy = Controller Defined


Default spin down time (mins) = 30

Coercion Mode = 1 GB

ZCR Config = Unknown

Max Chained Enclosures = 2

Direct PD Mapping = No

Restore Hot Spare on Insertion = No

Expose Enclosure Devices = Yes

Maintain PD Fail History = No

Zero Based Enclosure Enumeration = No

Disable Puncturing = No

EnableLDBBM = No

DisableHII = No

Un-Certified Hard Disk Drives = Allow

SMART Mode = Mode 6

Enable LED Header = No

LED Show Drive Activity = No

Dirty LED Shows Drive Activity = No

EnableCrashDump = No

Disable Online Controller Reset = No

Treat Single span R1E as R10 = No

Power Saving option = Enabled

TTY Log In Flash = Yes

Auto Enhanced Import = Yes

BreakMirror RAID Support = Yes

Disable Join Mirror = No

Enable Shield State = Yes

Time taken to detect CME = 60 sec

Capabilities :

============

Supported Drives = SAS, SATA

Boot Volume Supported = NO

RAID Level Supported = RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50,

RAID60, PRL 11, PRL 11 with spanning, SRL 3 supported,

PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span

Enable JBOD = No

Mix in Enclosure = Allowed

Mix of SAS/SATA of HDD type in VD = Allowed

Mix of SAS/SATA of SSD type in VD = Allowed

Mix of SSD/HDD in VD = Not Allowed

SAS Disable = No

Max Arms Per VD = 32

Max Spans Per VD = 8

Max Arrays = 128


Max VD per array = 64

Max Number of VDs = 64

Max Parallel Commands = 928

Max SGE Count = 60

Max Data Transfer Size = 8192 sectors

Max Strips PerIO = 42

Max Configurable CacheCade Size(GB) = 0

Min Strip Size = 64 KB

Max Strip Size = 1.0 MB

Scheduled Tasks :

===============

Consistency Check Reoccurrence = 168 hrs

Next Consistency check launch = 10/10/2015, 03:00:00

Patrol Read Reoccurrence = 168 hrs

Next Patrol Read launch = 10/10/2015, 03:00:00

Battery learn Reoccurrence = 672 hrs

Next Battery Learn = 10/30/2015, 18:00:00

OEMID = LSI

Drive Groups = 1

TOPOLOGY :

========

-------------------------------------------------------------------------

DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace

-------------------------------------------------------------------------

0 - - - - RAID6 Optl N 54.569 TB dsbl N dflt N

0 0 - - - RAID6 Optl N 54.569 TB dsbl N N dflt N

0 0 0 0:1 6 DRIVE Onln N 5.456 TB dsbl N N dflt -












-------------------------------------------------------------------------


DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID

DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded

Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active

PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign

DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present

Virtual Drives = 1

VD LIST :

=======

----------------------------------------------------------

DG/VD TYPE State Access Consist Cache sCC Size Name

----------------------------------------------------------

0/0 RAID6 Optl RW Yes RAWBD - 54.569 TB

----------------------------------------------------------

Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded

Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|

R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|

AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled

Check Consistency

Physical Drives = 12

PD LIST :

=======

-----------------------------------------------------------------------

EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp

-----------------------------------------------------------------------

0:1 6 Onln 0 5.456 TB SAS HDD N N 4 KB ST6000NM0014 U












-----------------------------------------------------------------------


EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup

DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare

UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface

Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info

SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign

UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded

CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Cachevault_Info :

===============

------------------------------------

Model State Temp Mode MfgDate

------------------------------------

CVPM03 Optimal 31C - 2015/01/06

------------------------------------

[root@gluster5 ~]# /opt/MegaRAID/storcli/storcli64 /c0 show

Generating detailed summary of the adapter, it may take a while to complete.

Controller = 0

Status = Success

Description = None

Product Name = Cisco 12G SAS Modular Raid Controller

Serial Number = SV50239847

SAS Address = 578da6e715b66710

PCI Address = 00:0a:00:00

System Time = 10/07/2015 08:38:50

Mfg. Date = 01/09/15

Controller Time = 10/07/2015 15:38:48

FW Package Build = 24.5.0-0020

BIOS Version = 6.19.05.0_4.16.08.00_0x06080500

FW Version = 4.250.00-3632

Driver Name = megaraid_sas

Driver Version = 06.803.01.00-rh1

Vendor Id = 0x1000

Device Id = 0x5D

SubVendor Id = 0x1137

SubDevice Id = 0xDB

Host Interface = PCIE

Device Interface = SAS-12G

Bus Number = 10


Device Number = 0

Function Number = 0

Drive Groups = 1

TOPOLOGY :

========

-------------------------------------------------------------------------

DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace

-------------------------------------------------------------------------

0 - - - - RAID6 Optl N 54.569 TB dsbl N N dflt N

0 0 - - - RAID6 Optl N 54.569 TB dsbl N N dflt N













-------------------------------------------------------------------------

DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID

DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded

Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active

PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign

DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present

Virtual Drives = 1

VD LIST :

=======

----------------------------------------------------------

DG/VD TYPE State Access Consist Cache sCC Size Name

----------------------------------------------------------

0/0 RAID6 Optl RW Yes RAWBD - 54.569 TB

----------------------------------------------------------

Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|dgrd=Degraded

Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|B=Blocked|Consist=Consistent|


R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack|

AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled

Check Consistency

Physical Drives = 12

PD LIST :

=======

-----------------------------------------------------------------------

EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp

-----------------------------------------------------------------------













-----------------------------------------------------------------------

EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup

DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare

UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface

Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info

SeSz-Sector Size|Sp-Spun|U-Up|D-Down|T-Transition|F-Foreign

UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded

CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded

Cachevault_Info :

===============

------------------------------------

Model State Temp Mode MfgDate

------------------------------------

CVPM03 Optimal 31C - 2015/01/06

------------------------------------


Appendix B: Gluster Volume-Creation Script

[root@gluster5 images]# cat gluster-volume-create-c240

gluster volume create gv0 replica 2 gluster5:/rhs/brick1/gv0

gluster6:/rhs/brick1/gv0 gluster7:/rhs/brick1/gv0 gluster8:/rhs/brick1/gv0

Appendix C: FIO Profile Used on Each Client Driver for the Random Test

WriteRand.FIO

[global]

numjobs=2

iodepth=1

ioengine=libaio

direct=1

directory=/fio/

group_reporting

fsync_on_close=1

runtime=300

[test]

name=randwrite

rw=randwrite

bs=[4k|8k|16k|32k]

size=512g

ReadRand.FIO

[global]

numjobs=2

iodepth=1

ioengine=libaio

direct=1

directory=/fio/

group_reporting

runtime=300

[test]

name=randread

rw=randread

bs=[4k|8k|16k|32k]

size=512g


About Red Hat

Red Hat is the world's leading provider of open source software solutions, using a community-powered approach to

reliable and high-performing cloud, Linux, middleware, storage, and virtualization technologies. Red Hat also offers

award-winning support, training, and consulting services. As a connective hub in a global network of enterprises,

partners, and open source communities, Red Hat helps create relevant, innovative technologies that liberate

resources for growth and prepare customers for the future of IT.

Printed in USA C11-736051-00 10/15

Documents

Deploy Large Unstructured Data Storage in a Small Data Center ... - · PDF fileMount the GlusterFS volume gvol0 on client 1: # mkdir /mnt/gv0 # mount -t glusterfs gluster1:/gv0 /mnt/gv0