24
Technical white paper HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server HP Converged Infrastructure with MapR M7 for Apache Hadoop and HBase Table of contents Executive summary ...................................................................................................................................................................... 2 HP SL4500 Series .......................................................................................................................................................................... 3 MapR M7 overview ........................................................................................................................................................................ 3 MapR M7 Edition ........................................................................................................................................................................ 5 Deployment of MapR services ............................................................................................................................................... 6 High-availability considerations ................................................................................................................................................. 6 Pre-deployment considerations ................................................................................................................................................ 6 Operating system...................................................................................................................................................................... 7 Computation .............................................................................................................................................................................. 7 Memory ....................................................................................................................................................................................... 7 Storage ........................................................................................................................................................................................ 7 Network ....................................................................................................................................................................................... 8 Data movement into and out of the MapR cluster ............................................................................................................ 9 Switches ...................................................................................................................................................................................... 9 HP Insight Cluster Management Utility ................................................................................................................................... 10 Server selection ........................................................................................................................................................................... 11 Reference Architectures ............................................................................................................................................................ 16 Single Rack Mapr M7 configuration ..................................................................................................................................... 16 Medium Cluster (Two-Rack) MapR M7 configuration ...................................................................................................... 19 Multi-Rack Cluster MapR M7 configuration ....................................................................................................................... 21 Edge nodes ................................................................................................................................................................................... 23 Summary ....................................................................................................................................................................................... 23 For more information ................................................................................................................................................................. 24

HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

  • Upload
    trandat

  • View
    222

  • Download
    0

Embed Size (px)

Citation preview

Page 1: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper

HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server HP Converged Infrastructure with MapR M7 for Apache Hadoop and HBase

Table of contents Executive summary ...................................................................................................................................................................... 2

HP SL4500 Series .......................................................................................................................................................................... 3

MapR M7 overview ........................................................................................................................................................................ 3

MapR M7 Edition ........................................................................................................................................................................ 5

Deployment of MapR services ............................................................................................................................................... 6

High-availability considerations ................................................................................................................................................. 6

Pre-deployment considerations ................................................................................................................................................ 6

Operating system ...................................................................................................................................................................... 7

Computation .............................................................................................................................................................................. 7

Memory ....................................................................................................................................................................................... 7

Storage ........................................................................................................................................................................................ 7

Network ....................................................................................................................................................................................... 8

Data movement into and out of the MapR cluster ............................................................................................................ 9

Switches ...................................................................................................................................................................................... 9

HP Insight Cluster Management Utility ................................................................................................................................... 10

Server selection ........................................................................................................................................................................... 11

Reference Architectures ............................................................................................................................................................ 16

Single Rack Mapr M7 configuration ..................................................................................................................................... 16

Medium Cluster (Two-Rack) MapR M7 configuration ...................................................................................................... 19

Multi-Rack Cluster MapR M7 configuration ....................................................................................................................... 21

Edge nodes ................................................................................................................................................................................... 23

Summary ....................................................................................................................................................................................... 23

For more information ................................................................................................................................................................. 24

Page 2: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

2

Executive summary

HP and Apache Hadoop allow you to derive new business insights from Big Data by providing a platform to store, manage and process data at scale. However, Hadoop is complex to deploy, configure, manage and monitor. This white paper provides several performance optimized configurations for deploying MapR M7 distribution of Apache Hadoop clusters of varying sizes on HP infrastructure that provide a significant reduction in complexity and increase in value and performance.

The configurations are based on the MapR M7 edition and the HP ProLiant SL4540 Gen8 server platform. The configurations included in this document have been jointly designed and developed by HP and MapR to provide Hadoop solutions that efficiently balance performance, storage and cost.

The configurations are built to support varying Big Data workloads. HP ProLiant SL4540 servers pack maximum performance, productivity and cost effectiveness in ultra-dense configurations. In fact, recent testing in April 2013 shows that HP ProLiant SL4540 holds the world’s fastest 10TB Hadoop TeraSort result.1

The HP ProLiant SL4540 servers offer breakthrough economics, density, and simplicity – and support up to a maximum of 27 servers with a total of 1.62 PB raw capacity in a single expansion 42U rack. In effect, the SL4540 Gen8 server’s 3x15 configuration provides customers 67% more storage capacity and 33% more compute density at a 42U rack level when compared to 2U rack mount servers deployed today.

Most 2U 12 LFF drive rack mount servers are deployed for Hadoop with 6 core processors in order to maintain the 1:1 core to spindle ratio. Because the SL4540 Gen8 server’s 3x15 configuration can provide 15 LFF drives per server, customers can take advantage of the 8 core processors to drive Hadoop workloads and maintain the 1:1 core to spindle ratio needed. As a consequence, SL4540 Gen8 servers deployed as worker nodes provide the following savings compared to 2U 12-drive rack mount worker nodes:

• 40% less Space

• 47% less Energy

• 41% less Cables

HP Big Data solutions provide best-in-class performance and availability, with integrated software, services, infrastructure, and management – all delivered as one proven solution as described at hp.com/go/hadoop. In addition to the benefits described above, the solution in this white paper also includes the following features that are unique to HP:

• For servers, the HP ProLiant SL4540 Gen8 servers include:

– The HP Smart Array P420i controller which provides increased2 I/O throughput performance resulting in a significant performance increase for I/O bound Hadoop workloads (a common use case) and the flexibility for the customer to choose the desired amount of resilience in the Hadoop Cluster with either JBOD or various RAID configurations.

– 3x15 configuration of the HP ProLiant SL4540 Gen8 with two sockets, 8 core processors and the Intel C600 Series Chipset consists of three compute nodes and a total of forty-five (45) large form factor (LFF) 3.5" hard disk drives (HDD) in the chassis. The HP ProLiant SL4540 is a dual socket Gen8 server, with a choice of different Intel Xeon processors, up to 192GB of memory and one PCIe slot for expansion per node. Every compute node also has its own dedicated networking ports. The product also supports InfiniBand interconnect. For more details see the quickspecs for the product.

– The HP iLO Management Engine on the servers is a comprehensive set of embedded management features, including HP Integrated Lights-Out 4 (iLO 4), Agentless Management, Active Health System, and Intelligent Provisioning which reduces node and cluster level administration costs for Hadoop.

• For networking, the HP 5900 Switch Series is a family of high-density ultra-low latency top-of-rack (ToR) data center switches. HP 5900 switches are suited for deployment at the server access layer of large enterprise data centers or for deployment at the data center core layer of medium-sized enterprises. The HP 5900AF-48XG-4QSFP+ Switch has 48x 10-Gigabit SFP+ ports with four QSFP+ 40-Gigabit ports for ultra-high capacity connections. The HP 5900AF enables customers to scale their server-edge 10GbE ToR deployments to new heights with high-density 48x10GbE ports delivered in a 1U design. The high server port density is backed by 4x40GbE uplinks to ensure availability of needed bandwidth for demanding applications. Redundant power and replaceable fan trays deliver highest availability.

• For management, HP Insight Cluster Management Utility (CMU) provides push-button scale out and provisioning with industry leading provisioning performance (deployment of 800 nodes in 30 minutes), reducing deployments from days to hours. In addition, CMU provides real-time and historical infrastructure and Hadoop monitoring with 3D visualizations. This allows customers to easily characterize Hadoop workloads and cluster performance thus reducing complexity and

1 The HP ProLiant SL4540 Cluster had 23% faster throughput per U than the previous benchmark HP achieved using an HP ProLiant DL380p Cluster. Both tests

were performed within a single 42U rack. 2 Compared to the previous generation of Smart Array controllers

Page 3: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

3

improving system optimization which leads to improved performance and reduced cost. HP Insight Management and HP Service Pack for ProLiant allow for easy management of firmware and the server.

All of these features reflect HP’s balanced building blocks of servers, storage and networking, along with integrated management software and bundled support.

In addition, this white paper has been created to assist in the rapid design and deployment of the MapR M7 software on HP infrastructure for clusters of various sizes. It is also intended to concretely identify the software and hardware components required in a solution in order to simplify the procurement process. The recommended HP Software, HP ProLiant servers, and HP Networking switches and their respective configurations have been carefully tested with a variety of I/O, CPU, network, and memory bound workloads. The configurations included provide the best value for optimum MapReduce and HBase computational performance.

Target audience: This document is intended for decision makers, system and solution architects, system administrators and experienced users who are interested in reducing the time to design or purchase an HP and MapR solution. An intermediate knowledge of the Apache Hadoop and scale out infrastructure is recommended. Those already possessing expert knowledge about these topics may proceed directly to Pre-deployment considerations.

HP SL4500 Series

The HP SL4500 Series is an excellent choice for running Hadoop and includes the HP ProLiant SL4540 Gen8 server. This recommended configuration document will focus on the 3 node configuration of HP ProLiant SL4540 Gen8. It consists of three compute nodes and a total of forty-five (45) large form factor (LFF) 3.5" hard disk drives (HDD) in the chassis. Each server has exclusive access to fifteen (15) large form factor (LFF) 3.5" hard disk drives (HDD) in the storage section. The HP ProLiant SL4540 is a dual socket Gen8 server, with a choice of different Intel Xeon processors, up to 192GB of memory and one PCIe slot for expansion per node. Every compute node also has its own dedicated networking ports.

The HP ProLiant SL4540 Gen8 server offers the advancements and benefits of the Gen8 platform:

• HP SmartMemory: memory performance improvements and enhanced functionality as listed for Gen8 in the HP SmartMemory QuickSpecs at: http://h18000.www1.hp.com/products/quickspecs/14225_div/14225_div.html.

• HP Smart Array RAID controllers: An HP Dynamic Smart Array B120i SATA RAID Controller is located on the server node and controls the two small form factor (SFF/2.5") hard drives on the server node itself, and an HP Smart Array P420i controller is located in the I/O Module at the back of the SL4540 enclosure and controls the large form factor (LFF/ 3.5") hard drives in the chassis.

• HP Agentless Management: With HP iLO Management Engine in every HP ProLiant Gen8 server, the base hardware monitoring and alerting capability is built into the system (running on the HP iLO chipset) and starts working the moment that a power cord and an Ethernet cable is connected to the server.

• HP Intelligent Provisioning includes everything needed for system setup and deployment. Firmware, drivers, and tools to get systems online are embedded on a NAND flash chip on the server motherboard (thus SmartStart CDs or firmware DVDs are not necessarily needed).

For more information see hp.com/servers/sl4540.

MapR M7 overview

MapR is a complete distribution for Apache Hadoop that includes HBase, Pig, Hive, Mahout, Cascading, Sqoop, Flume and more (see Figure 1). MapR’s distribution is compatible with Hadoop (MapReduce, HDFS and HBase).

MapR makes Hadoop a reality for the enterprise. MapR's distribution for Hadoop is designed to not only meet core enterprise needs but also provide exceptional performance.

Page 4: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

4

Figure 1. MapR’s software overview

MapR provides these distinct advantages:

• Lights Out Data Center capability – MapR provides high availability across the entire stack including storage, MapReduce, NFS layers and more.

– In the storage layer, the No NameNode architecture provides high availability with self-healing and support for multiple, simultaneous failures, with no additional hardware whatsoever.

– In the MapReduce layer, MapR’s JobTracker HA makes JobTracker failures transparent to applications – the currently running tasks continue to execute during the failover process.

– In the NFS layer, MapR automatically manages virtual IP addresses and balances them between the nodes so that failures are transparent to clients that are reading and writing data via NFS.

– In addition, these capabilities are combined with both data protection (snapshots) and disaster recovery (mirroring) providing protection against user errors and enabling greater resiliency.

– Automated rolling upgrades eliminate the need to bring the cluster down to move to the latest software release of MapR.

• Ease of Use – MapR provides the easiest interface to integrate other enterprise software with Hadoop.

– The NFS layer allows reading and writing directly to a file in the cluster, enabling near real-time streaming of data into a MapR cluster.

– On the administration side, the MapR Control System provides a management console to set the frequency of the snapshots, number of replications, mirroring policies, and to better understand user defined cluster health criteria.

– Furthermore, with provision for logical partitioning of the physical cluster it is extremely easy to share the cluster across different users, groups and applications.

– Performance – MapR delivers faster performance using direct block device I/O, optimized format on disk, automatic and transparent client-side compression, and Lockless architecture designed for large clusters. It provides:

• Faster execution of MapReduce and HBase job

• Increased throughput in MB/sec.

Page 5: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

5

• Optimized shuffle

• Automatic and transparent client-side compression

• Network overhead and footprint reduction

MapR M7 Edition

The M7 edition of MapR provides ease of use, dependability and performance advantages for NoSQL and Apache Hadoop applications. MapR M7 has removed the operational and performance issues of the standard Apache distribution of HBase. M7 unifies the storage and processing of files as well as tables in a single platform while maintaining compatibility with HDFS and HBase API’s (see Figure 2). As a result, MapR’s existing management, access and data protection capabilities above now apply to applications supported by HBase.

Figure 2. MapR’s M7 simplified architecture

With M7, there are no region servers, additional processes, or any redundant layer between the application and the data residing in the cluster. M7’s zero administration approach includes automatic region splits and self-tuning with no downtime required for any operation including schema changes.

MapR's implementation of the HBase API provides enterprise-grade high availability (HA), data protection, and disaster recovery features for tables on a distributed Hadoop cluster. MapR tables can be used as the underlying key-value store for Hive, or any other application requiring a high-performance, high-availability key-value datastore. Because MapR tables are API-compatible with HBase, many legacy HBase applications can continue to run without modification.

MapR has extended HBase shell to work with MapR tables in addition to Apache HBase tables. Similar to development for Apache HBase, the simplest way to create tables and column families in MapR-FS, and put and get data from them, is to use HBase shell. MapR tables can be created from the MapR Control System (MCS) user interface or from the Linux command line, without the need to coordinate with a database administrator. You can treat a MapR table just as you would a file, specifying a path to a location in a directory, and the table appears in the same namespace as your regular files. You can also create and manage column families for your table from the MCS or directly from the command line.

Page 6: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

6

Deployment of MapR services

MapR provides the aforementioned features via services that run on nodes throughout the cluster. Services that manage the cluster and coordinate the jobs (“control services”) run on a small number of nodes that the user can designate; the services responsible for the data processing and storage (“worker services”) run on most or all nodes in the cluster. The Reference Architectures we provide in this document are prescriptive as to which nodes the control and worker services are deployed to. By mapping the services onto specific nodes within the HP infrastructure for clusters of varying sizes we have simplified your deployment.

Table 1. Control Services Summary

Service Package Distribution across nodes

Container Location Database (CLDB) mapr-cldb 1-3

WebServer mapr-webserver 1 or more

ZooKeeper mapr-zookeeper 1, 3, or 5

JobTracker mapr-jobtracker 1-3

Table 2. Worker Services Summary

Service Package Distribution across nodes

FileServer mapr-fileserver Most or all nodes

TaskTracker mapr-tasktracker Most or all nodes

NFS mapr-nfs Most or all nodes

High-availability considerations

The following are some of the high availability features considered in this reference configuration:

• OS availability and reliability – For the reliability of the server, the OS disk is configured in RAID 0+1 configuration thus preventing failure of the system from OS hard disk failures.

• Network reliability – The reference configuration uses two HP 5900 switches for redundancy, resiliency and scalability in using Intelligent Resilient Framework (IRF) bonding. We recommend using redundant power supplies.

• Power supply – To ensure the servers and racks have adequate power redundancy we recommend that each server have a backup power supply, and each rack have at least two Power Distribution Units (PDUs).

Pre-deployment considerations

There are a number of key factors you should consider prior to designing and deploying a Hadoop Cluster. The following subsections articulate the design decisions in creating a balanced baseline configuration for the reference architectures. The rationale provided includes the necessary information for you to take the configurations and modify them to suit a particular custom scenario.

Table 3. Overview of Functional Components and Configurable Value

Functional Component Value

Operating System Improves Availability and Reliability

Computation Ability to balance Price with Performance

Memory Ability to balance Price with Capacity and Performance

Page 7: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

7

Functional Component Value

Storage Ability to balance Price with Capacity and Performance

Network Ability to balance Price with Performance and Availability

Operating system

MapR supports 64-bit Red Hat Enterprise Linux (RHEL) 5.4 or later, CentOS 5.4 or later, SUSE Linux Enterprise Server (SLES) 11 or later and Ubuntu 9.04 or later as choices for the operating system.

Note

HP recommends using a 64-bit operating system to avoid constraining the amount of memory that can be used on worker nodes. 64-bit Red Hat Enterprise Linux 6.1 or greater is recommended due to better ecosystem support, more comprehensive functionality for components such as RAID controllers and compatibility with HP Insight CMU. The Reference Architectures listed in this document were tested with 64-bit Red Hat Enterprise Linux 6.2.

Note

MapR requires Java JDK 6 or JDK 7 (not JRE). MapR supports both Oracle JDK and Open JDK.

Computation

The processing or computational capacity of a MapR M7 cluster is determined by the aggregate number of MapReduce slots available across all nodes. MapReduce slots are configured on a per node basis. Employing Hyper-Threading improves process scheduling, allowing you to configure more MapReduce slots. Refer to the Storage section to see how I/O performance issues arise from sub-optimal disk to core ratios (too many slots and too few disks). For CPU bound workloads we recommend buying processors with faster clock speeds to remove the bottleneck.

MapR automatically tunes the cluster for most purposes. MapR determines the default number of map and reduce slots on each node based on the amount of memory not already allocated to MapR services. In certain circumstances, manual tuning of MapR cluster might be desired and defaults can be overridden on each node by either specifying the maximum number of map and reduce slots on each node or by defining a formula to calculate the maximum number of map and reduce slots based on the memory, CPUs, and disks on the node.

Memory

Use of Error Correcting Memory (ECC) is a practical requirement for MapR M7 and is standard on all HP ProLiant servers. Memory requirements differ between the management services and the worker services. For the worker services, sufficient memory is needed to manage the TaskTracker and FileServer services in addition to the sum of all the memory assigned to each of the MapReduce slots. If you have a memory bound MapReduce Job we recommend that you increase the amount of memory on all the nodes running worker services.

Best practice It is important to saturate all the memory channels available to ensure optimal use of the memory bandwidth. For example, on a two socket processor with three memory channels that supports two DIMMs each for a total of six (6) DIMMs per installed processor or a grand total of twelve (12) DIMMs for the server, one would typically fully populate the channels with 8GB DIMMs resulting in a configuration of 96GB of memory per server.

Storage

Fundamentally, Hadoop is designed to achieve performance and scalability by moving the compute activity to the data. It does this by distributing the Hadoop jobs to nodes close to their data, ideally running the tasks against data on local disks.

Page 8: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

8

Best practice Given the architecture of Hadoop, the data storage requirements for the worker nodes are best met by direct attached storage (DAS) in a Just a Bunch of Disks (JBOD) configuration and not as DAS with RAID or Network Attached Storage (NAS).

There are several factors to consider and balance when determining the number of disks a node requires.

• Storage capacity – The number of disks and their corresponding storage capacity determines the total amount of the raw storage capacity for your cluster. We recommend Large Form Factor (3.5”) disks due to their lower cost and larger storage capacity compared with the Small Form Factor (2.5”) disks. See the best practice below for alternate recommendations for I/O bound workloads.

• Redundancy – MapR file system ensures that a certain number of block copies are consistently available. This number is configurable as a volume property, which is typically set to three. If a MapR M7 node goes down, it will replicate the blocks that had been on that server onto other servers in the cluster to maintain the consistency of the number of block copies. For example, if the NICs (Network Interface Cards) on a server with 12 TB of data fail, that 12 TB of data will be replicated onto other servers, generating 12 TB of traffic within the cluster. The failure of a non-redundant ToR (Top of Rack) switch will generate even more replication traffic. Make sure your network, whether 1GbE or 10GbE, has the capacity to handle block replication for your server configurations in case of failure.

• I/O performance – Each node has a certain number of MapReduce slots available for processing Hadoop tasks. Each slot typically operates on one block of data at a time. The more disks you have, the less likely it is that you will have multiple tasks accessing a given disk at the same time and it will also result in more tasks being able to run against node-local data.

Best practice Customers can choose to use SATA or SAS MDL (Nearline 7200 RPM) disks in a MapR M7 cluster. SAS MDL disks are preferred by customers who are looking for disks that are more efficient at driving higher I/O throughput across a larger amount of disks delivering to customers a higher performing cluster. SAS MDL disks are recommended over SAS ENT (10K and 15K SAS drives) disks in a MapR M7 cluster because the SAS ENT disks deliver only marginally higher performance at a higher cost with lower storage capacity.

Network

Configuring only a single Top of Rack (ToR) switch per rack introduces a single point of failure for each rack. In a multi-rack system such a failure will result in a flood of network traffic as MapR file system rebalances storage, and in a single-rack system such a failure brings down the whole cluster. Consequently, configuring two ToR switches per rack is recommended for all production configurations as it provides an additional measure of redundancy. This can be further improved by configuring link aggregation between the switches. The most desirable way to configure link aggregation is by bonding one link from a server to one ToR switch with a second link from the same server to the other ToR switch in its rack. When done properly, this allows the bandwidth of both links to be used. If either of the switches fail, the servers will still have full network functionality, but with the performance of only a single link. Not all switches have the ability to do link aggregation from individual servers to multiple switches, however, the HP 5900AF-48XG-4QSFP+ switch supports this through HP’s Intelligent Resilient Framework (IRF) technology. In addition, switch failures can be further mitigated by incorporating dual power supplies for the switches. The MapR M7 can be configured to be aware of the server and rack topology within the cluster. This is beneficial in that it allows Hadoop to take rack affinity into account when placing block data on Hadoop Worker nodes. MapR recommends 1-2 10GbE NICs per node to satisfy the disk throughput requirements.

Larger clusters with three or more racks will benefit from having ToR switches connected by 40GbE uplinks to core aggregation switches. During the map phase of Hadoop jobs that utilize the Hadoop file system, the majority of tasks reference data on the server that executes the task (node-local). For those tasks that must access data remotely, the data is usually on other servers in the same rack (rack-local). Only a small percentage of map tasks need to access data from remote racks. Although the amount of remote-rack accesses increases for larger clusters, it is expected to put a relatively small load on the ToR and core switches. However during the shuffle phase of Hadoop jobs, the network traffic can be significant for certain workloads e.g. ETL jobs and the ToR switch should be able to handle the traffic without dropping packets.

Best practice Each reduce task can concurrently request data from a default of 12 map output files, so there is the possibility that servers will be delivered more data than their network connections can handle. This will result in dropped packets and can lead to a collapse in traffic throughput. This is why we recommend ToR switches with deep buffering.

Page 9: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

9

Data movement into and out of the MapR cluster

It is best to isolate the MapR M7 cluster on its own private network in order to ensure that external network traffic does not affect the performance of the MapR network. However, one still needs to ensure that data can be moved into and out of the MapR cluster from external networks. MapR can be mounted by another system as a network file share (NFS). In order to enable the import and export of data to and from a MapR cluster, we recommend that you multi-home the MapR nodes you plan to NFS mount from external networks.

Switches

MapR clusters contain two types of switches, namely Aggregation switches and Top of Rack switches. Top of Rack switches route the traffic between the nodes in each rack and Aggregation switches route the traffic between the racks.

Top of Rack (ToR) switches (HP 5900AF-48XG-4QSFP+)

The HP 5900AF-48XG-4QSFP+ 10GbE high-density, ultra-low latency, top-of-rack (ToR) switch provides IRF Bonding and sFlow which simplifies the management, monitoring and resiliency of the customer’s Hadoop network. This model has 48x 10-Gigabit / Gigabit SFP+ ports with four QSFP+ 40-Gigabit ports for ultra-high capacity connections. The high performance 10 GbE networking provides cut-through and nonblocking architecture that delivers industry-leading low latency (~1 microsecond) for very demanding enterprise applications; the switch delivers a 1.28 Tbps switching capacity and 952.32 Mpps packet forwarding rate in addition to incorporating 9 MB of packet buffers.

For more information on the HP 5900AF-48XG-4QSFP+ switch, please see http://h17007.www1.hp.com/us/en/products/switches/HP_5900_Switch_Series/index.aspx

The configuration for the HP 5900AF-48XG-4QSFP+ switch is provided below. The HP 5900 switch can also be used for connecting the SL4540 Shared Network Port LOM iLOs from the individual servers. (Direct connection of the iLO port on the SL4540 to the 5900 will not work, because the 5900 ports will not support 100Mbs.) This will require an HP X120 1G SFP RJ45 T Transceiver (JD089B) (1 GbE transceiver) to be installed in the 5900 switch for each server. This connectivity option will also require that the iLO configuration on the servers be changed to utilize the NIC 1, 1GbE port on the SL4540 server node (3 per chassis), instead of the iLO module connector located in the SL4540 chassis. For additional information on how to configure the iLOs go to hp.com/go/ilo. Another option for iLO connectivity is to add an HP 5120 EI switch and connect the iLO connectors from each chassis to that switch.

Note For customers expecting a complete physical isolation between the iLO traffic from the servers as well as the management traffic, for their switches using the Out of Band interface (dedicated RJ-45 port), HP recommends to build a separate “out of band” network. By building this separate network, all the management traffic follows a completely different path which will never be impacted by any action executed on the data traffic coming from the servers (applications). A switch like the 5120 EI can be used for this type of “out of band network”, with 10/100/1000 RJ-45 connectivity (48 ports: JE069A or 24 ports: JE066A) to connect iLO ports.

Figure 3. HP 5900AF-48XG-4QSFP+ Top of Rack (ToR) switch

Table 4. HP 5900AF-48XG-4QSFP+ Single Switch options

Qty Description

1 HP 5900AF-48XG-4QSFP+ Switch

2 HP 58x0AF 650W AC Power Supply

2 HP 5830AF-48G Back(power)-Front(prt) Fan Tray

1 HP X240 40G QSFP+ QSFP+ 1m DAC Cable

Page 10: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

10

HP Insight Cluster Management Utility

HP Insight Cluster Management Utility (CMU) is an efficient and robust hyperscale cluster lifecycle management framework and suite of tools for large Linux clusters such as those found in High Performance Computing (HPC) and Big Data environments. A simple graphical interface enables an “at-a-glance” view of the entire cluster across multiple metrics, provides frictionless scalable remote management and analysis, and allows rapid provisioning of software to all the nodes of the system. Insight CMU makes the management of a cluster more user friendly, efficient, and error free than if it were being managed by scripts, or on a node-by-node basis. Insight CMU offers full support for iLO 2, iLO 3, iLO 4 and LO100i adapters on all HP ProLiant servers in the cluster.

Note HP Insight CMU allows you to easily correlate MapR M7 metrics with cluster infrastructure metrics, such as CPU Utilization, Network Transmit/Receive, Memory Utilization, Power Consumption, Ambient Node and Rack temperature and I/O Read/Write. This allows you to characterize your MapR M7 workloads and optimize the system thereby improving the performance of the MapR M7 Cluster. CMU TimeView Metric Visualizations will help you understand, based on your workloads, whether your cluster needs more memory, a faster network or processors with faster clock speeds. In addition, Insight CMU also greatly simplifies the deployment of MapR M7, with its ability to create a Golden Image from a Node and then deploy that image to up to 4000 nodes. Insight CMU is able to deploy 800 nodes in 30 minutes.

Insight CMU is highly flexible and customizable, offers both a graphical user interface (GUI) and a command-line interface (CLI) interface, and is being used to deploy a range of software environments, from simple compute farms to highly customized, application-specific configurations. Insight CMU is available for HP ProLiant and HP BladeSystem servers with Linux operating systems, including Red Hat Enterprise Linux, SUSE Linux Enterprise, CentOS, and Ubuntu. Insight CMU also includes options for monitoring graphical processing units (GPUs) and for installing GPU drivers and software.

For more information, please see hp.com/go/cmu.

Table 5. HP Insight CMU options

Qty Description

1 HP Insight CMU 1yr 24x7 Flex Lic

1 HP Insight CMU 1yr 24x7 Flex E-LTU

1 HP Insight CMU 3yr 24x7 Flex Lic

1 HP Insight CMU 3yr 24x7 Flex E-LTU

1 HP Insight CMU Media

Page 11: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

11

Figure 4. HP Insight CMU Interface – real-time view

Figure 5. HP Insight CMU Interface – Time View

Server selection

In a typical MapR cluster, most (or all) nodes run worker services that are dedicated to data processing and storage, and a smaller number of nodes run control services that provide cluster coordination and management. Depending on the size of the cluster, a MapR M7 deployment consists of one or more nodes running control services and one or more nodes running worker services. In small to medium size clusters, a single node may perform data processing as well as management.

Page 12: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

12

Note MapR is able to run both the control services and worker services on the same server resulting in a fully homogeneous server deployment. The Reference Architecture will utilize this configuration to make maximum use of the server resources.

This section specifies which servers to use and the rationale behind it. The Reference Architectures section will provide topologies for the deployment of control and worker services across the nodes for clusters of varying sizes.

The HP ProLiant SL4540 Gen8 (3x15) (4.3U) is an excellent choice as the server platform for the MapR nodes. For ease of management we recommend a homogenous server infrastructure for your nodes. The 3 node configuration of the HP ProLiant SL4540 Gen8 consists of three compute nodes and a total of forty-five (45) large form factor (LFF) 3.5" hard disk drives (HDD) in the chassis (15 per node) and six (6) small form factor (SFF) 2.5” hard drives in the front of the chassis (two per node). The HP ProLiant SL4540 is a dual socket Gen8 server, with a choice of different Intel Xeon processors, up to 192GB of memory and one PCIe slot for expansion per node. Every node also has its own dedicated networking ports.

Figure 6. HP ProLiant SL4540 Gen8 (3 x 15) Server

Processor configuration

The configuration for each server node in the SL4540 features two sockets with 8 core processors and the Intel C600 Series chipset which provide 16 physical or 32 Hyper-Threaded cores per server. MapR manages the amount of work each server is able to undertake via the amount of MapReduce slots configured for that server. The more cores and memory available to the server, the more MapReduce slots can be configured for the server (see the Computation section for more detail). We recommend that Hyper-Threading be turned on.

Drive configuration

The HP Smart Array P420i controller provides increased3 I/O throughput performance resulting in a significant performance increase for I/O bound Hadoop workloads (a common use case). Redundancy is built into the Hadoop and MapR architecture and thus there is no need for RAID schemes to improve redundancy on the nodes as it is all coordinated and managed by MapR. Drives should use a Just a Bunch of Disks (JBOD) configuration, which can be achieved with the HP Smart Array P420i controller by configuring each individual disk as a separate RAID 0 volume. Additionally array acceleration features on the P420i should be turned off for the RAID 0 data volumes. The node design includes a total of 17 drives per node, 2 are used for the operating system and 15 are used for data.

Best practice The two 500GB SATA MDL disks in the front of the SL4540 are configured as RAID 1 using the HP Smart Array B120i Controller for mirrored OS. This provides additional measures of redundancy for the nodes. The OS drives should not be partitioned for MapR file system use to avoid degraded I/O performance. We recommend allocating 15 2TB SAS MDL hard drives to MapR file system.

Performance The HP Smart Array P420i controller provides two port connectors per controller with each containing 4 SAS links. The controller drives the 15 disks in the chassis per node. The 2 disks in the front per node are controlled by the B120i SATA

3 Compared to the previous generation of Smart Array controllers

Page 13: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

13

RAID controller. For a performance oriented solution, we recommend SAS MDL drives as they offer a significant read and write throughput performance enhancement over SATA disks.

Core to disk ratio The more drives a server contains, the more efficiently it can service I/O requests because it reduces the likelihood of multiple threads contending for the same drive which can result in interleaved I/O and degraded performance.

Memory configuration

Servers running the worker node processes should have sufficient memory for the MapR File System (which caches both files and tables) and for the amount of MapReduce Slots configured on the server. The Intel Xeon E5-2470 has 3 memory channels per processor. When configuring memory, one should always attempt to populate all the memory channels available to ensure optimum performance.

Best practice It is important to saturate all the memory channels available to ensure optimal use of the memory bandwidth. For example, on a two socket server with three memory channels that supports two DIMMs each for a total of six (6) DIMMs per installed processor or a grand total of twelve (12) DIMMs for the server, one would typically fully populate the channels with 8GB DIMMs resulting in a configuration of 96GB of memory per server.

See Figure 7 for the memory slots to populate when using 8GB memory DIMMS.

Figure 7. SL4540 8GB DIMM memory configuration recommendation

Network configuration

We recommend using two 10GbE NICs. These can be bonded to appear as a single network interface to the operating system or they can be configured on separate subnets. MapR will take advantage of multiple NICs on separate subnets for both availability and throughput performance. In addition, in the reference configurations later on in the document you will notice that we use two IRF Bonded switches. In order to ensure the best level of redundancy we recommend cabling NIC 1 to Switch 1 and NIC 2 to Switch 2.

ProLiant SL4540 Gen8 server configuration The ProLiant SL4540 Gen8 node should have the following base configuration:

• Dual Eight-Core Xeon E5-2470 2.3 GHz Processors with Hyper-Threading

• 15 x 2TB 3.5” SAS SC MDL 7.2K RPM disks (higher density disks are also available)

• 96 GB DDR3 Memory

• 2 x 10GbE Ethernet NICs

• 2 x P420i Smart Array Controllers

Best practice Customers have the option of purchasing a second power supply for additional power redundancy. This is especially appropriate for single rack clusters where the loss of a node represents a noticeable percentage of the cluster.

Page 14: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

14

Table 6. The HP ProLiant SL4540 Gen8 (3 x 15) Server Configuration

Qty Description

SL4540 base

1 HP SL4540 (3 x15) Gen8 Chassis

4 HP 750W CS Gold Hot Plug Power Supply Kit

1 ProLiant SL4540 Gen8 HW Support

1 server node (order 3 nodes per SL4540 chassis)

1 HP 3xSL4540 Gen8 Tray Node Svr

1 HP SL4540 Gen8 E5-2470 Kit FIO Kit

1 HP SL4540 Gen8 E5-2470 Kit

12 HP 8GB 2Rx4 PC3L-10600R-9 Kit

15 HP 2TB 6G SAS 7.2K 3.5in SC MDL HDD

2 HP 500GB 6G SATA 7.2k 2.5in SC MDL HDD

1 HP 10G IO Module

2 HP RAID Mez Ctrllr p420i FIO Kit

2 HP 12in Super Cap for Smart Array

2 HP 2GB FBWC for P-Series Smart Array

Server software

Each node contains different MapR software packages depending on the services planned on the node. MapR is a complete Hadoop distribution, but not all services are required. Every Hadoop MapReduce installation requires JobTracker and TaskTracker services to manage Map/Reduce tasks. In addition, MapR requires the ZooKeeper service to coordinate the cluster, and at least one node must run the CLDB service. MapR distribution includes an Apache HBase compatible table functionality directly in MapR file system and no additional HBase services need to be installed. You can use MapR tables exclusively or work in a mixed environment with Apache HBase tables. The WebServer service is required if the browser-

based MapR Control System will be used. Configuring a cluster for HA involves redundant instances of specific services, as well as a correct configuration of the MapR NFS service. In order to use MapR Metrics, you have to set up a MySQL database where metrics data will be logged. MySQL is not included in the MapR distribution for Apache Hadoop, and you need to download and install it separately. Consider running the MySQL server on a machine external to the cluster to prevent the MySQL server’s resource needs from affecting services on the cluster.

Please see the following link for more information on installing and configuring the planned MapR services: mapr.com/doc/display/MapR/Installing+MapR+Software

Page 15: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

15

Table 7. MapR Node Software

Software Description

RHEL 6.2 Recommended Operating System

Oracle JDK 1.6.0

Java Development Kit

MapR FileServer

The MapR service that manages disk storage for MapR-FS on each node

CLDB The CLDB service that coordinates data storage services among MapR FileServer services, on select nodes

TaskTracker The TaskTracker service for MapReduce that manages task execution, on each node (Only if running MapReduce)

JobTracker The JobTracker service for MapReduce that coordinates the execution of MapReduce jobs, on select nodes (Only if running MapReduce)

ZooKeeper The ZooKeeper service that enables high availability (HA) and fault tolerance for MapR clusters by providing coordination, on select nodes

WebServer Runs the MapR Control System and provides the MapR Heatmap, on select nodes

Metrics The metrics service that provides optional real-time analytics data on Hadoop job performance. If used, the Metrics service is required on all JobTracker and WebServer nodes.

NFS Provides read-write MapR Direct Access NFS access to the cluster, with full support for concurrent read and write access, on select nodes

HBase Client Provides access to tables in MapR-FS on an M7 Edition cluster via HBase APIs. Required on all nodes that will access table data in MapR-FS, typically all TaskTracker nodes and edge nodes for accessing table data

HBase Master The HBase master service manages the region servers that make up HBase table storage, on select nodes, only if Apache HBase tables are used. Not needed if Apache HBase compatible MapR tables are used as they are implemented directly within MapR-FS

HBase RegionServer

The HBase region servers that make up HBase table storage, only if Apache HBase tables are used. Not needed if Apache HBase compatible MapR tables are used as they are implemented directly within MapR-FS

Pig Pig is a high-level data-flow language and execution framework, to be installed on nodes where it is used

Hive Hive is a data warehouse that supports SQL-like ad hoc querying and data summarization, to be installed on nodes where it is used

Flume Flume is a service for aggregating large amounts of log data, to be installed on nodes where it is used

Oozie Oozie is a workflow scheduler system for managing Hadoop jobs, to be installed on nodes where it is used

HCatalog HCatalog is a table and storage management service for data created using Apache Hadoop, to be installed on nodes where it is used. HCatalog has been merged into the Apache Hive project and will be included in future Hive releases.

Cascading Cascading is an application framework for analyzing and managing big data, to be installed on nodes where it is used

Mahout Mahout is a set of scalable machine-learning libraries, to be installed on nodes where it is used

Sqoop Sqoop is a tool for transferring bulk data between Hadoop and relational databases, to be installed on nodes where it is used

Page 16: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

16

Reference Architectures

This section provides a progression of configurations from single rack to full scale-out MapR M7 cluster configurations. Best practices for each of the components within the configurations specified have been articulated earlier in this document.

Single Rack Mapr M7 configuration

The Single Rack MapR M7 configuration provides a starting point for a MapR M7 cluster configuration. The configuration reflects the following components.

Rack enclosure

The rack contains nine HP ProLiant SL4540 Gen8 chassis with 27 server nodes and two HP 5900AF-48XG-4QSFP+ switches within a 42U rack. This leaves 1U open for optional KVM switch.

Power and cooling

In planning for large clusters, it is important to properly manage power redundancy and distribution. To ensure the servers and racks have adequate power redundancy we recommend that each server has a backup power supply, and each rack has at least two Power Distribution Units (PDUs). There is an additional cost associated with procuring redundant power supplies. This is less important for larger clusters as the inherent failover redundancy within the MapR M7 will ensure there is less impact.

Best practice For each SL4540 chassis, there are four power supplies and we recommend that two power supplies be connected to two different PDUs. Furthermore, the PDUs in the rack can each be connected to a separate data center power line to protect the infrastructure from a data center power line failure.

Additionally, distributing the server power supply connections evenly to the in-rack PDUs, as well as distributing the PDU connections evenly to the data center power lines ensures an even power distribution in the data center and avoids overloading any single data center power line. When designing a cluster, check the maximum power and cooling that the data center can supply to each rack and ensure that the rack does not require more power and cooling than is available.

Network As previously described in the Switches section, two HP 5900AF-48XG-4QSFP+ switches are specified for performance and redundancy. The HP 5900AF-48XG-4QSFP+ includes up to four 40GbE uplinks which can be used to connect the switches in the rack into the desired network. Keep in mind that if IRF bonding is used, it requires 2 40GbE ports per switch, which would leave 2 40GbE ports on each switch for uplinks.

ProLiant SL4540 Gen8 servers The ProLiant SL4540 servers in the rack act as nodes in the MapR cluster. The architecture of MapR software allows virtually any service to run on any node, or nodes, to provide a high-availability, high-performance cluster. The basic requirements of a node are not different for control service or for worker service nodes resulting in a homogeneous server configuration. At the same time, the service assignment to the nodes can affect the performance of the cluster. For good cluster performance, some of the control services such as CLDB and ZooKeeper services need to be run on separate nodes. To provide high availability (HA), it is recommended to configure multiple control services in the rack. One should have at least 3 ZooKeeper, 2 to 3 CLDB, 2 to 3 JobTracker, and 2 WebServer services to provide HA functionality in the cluster to eliminate single points of failure.

The diagram below (see Figure 8) shows the recommended configuration for a Single Rack MapR M7 cluster which includes HA functionality. There are 3 ZooKeeper, 2 CLDBs and 2 JobTrackers configured to provide high availability. 2 WebServer services, along with the Metrics services, are placed on the JobTracker nodes. NFS services can be placed on most of the nodes as needed. The redundant control services, i.e. CLDB, ZooKeeper, and JobTracker services, have been distributed across separate chassis to provide HA in case of a single chassis failure.

Page 17: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

17

Figure 8. Single Rack MapR M7 Configuration – Rack Level View

The details of various node services in the M7 Single Rack cluster are as follows:

Nodes 1 and 4

Nodes 1 and 4 have the following services:

• Container Location Database (CLDB) service

• FileServer service

• NFS service

• TaskTracker service (If MapReduce is used)

Nodes 2 and 5

Nodes 2 and 5 have the following services:

• JobTracker service (If MapReduce is used)

• WebServer service

• Metrics service

• FileServer service

• NFS service

• TaskTracker service (If MapReduce is used)

Nodes 6, 9, and 12

Nodes 6, 9 and 12 have the following services:

• ZooKeeper service

• FileServer service

• NFS service

• TaskTracker service (If MapReduce is used)

Note MapR will automatically reduce the number of map and reduce slots on nodes running control services, if you use the default settings. If setting these values manually, you should reduce the number of map and reduce slots in order to ensure that the worker services do not consume the resources required for the control services.

Page 18: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

18

Rest of the nodes

After the nodes running the control services are configured, there are up to 20 nodes in the rack that can be configured to run worker services exclusively. Keep in mind that it is not mandatory to fully populate the rack with nodes running worker services; however, both performance and storage capacity will improve as more nodes are added. The following worker services run on the remaining nodes:

• FileServer

• Network File Share (NFS)

• TaskTracker (If MapReduce is used)

Figure 9. Single Rack MapR M7 Configuration – MapR Services

Page 19: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

19

Medium Cluster (Two-Rack) MapR M7 configuration

The MapR M7 Medium Cluster design assumes the Single Rack design is already in place and extends its scalability. The Single Rack design ensures the required amount of control services are in place for scale out. For a medium cluster configuration, one simply adds one more rack to the Single Rack design. This section reflects the design of the racks for medium cluster configuration.

Rack enclosures

The first rack configuration is same as the Single Rack configuration. The second rack also contains nine HP ProLiant SL4540 Gen8 chassis with 27 server nodes and two HP 5900AF-48XG-4QSFP+ switches within a 42U rack, but the services are configured differently as described below. This leaves 1U open for optional KVM switch.

Network As previously described in the Switches section, two HP 5900AF-48XG-4QSFP+ switches are specified for performance and redundancy. The HP 5900AF-48XG-4QSFP+ includes up to four 40GbE uplinks which can be used to connect the switches in the rack into the desired network. Keep in mind that if IRF bonding is used, it requires 2 40GbE ports per switch, which would leave 2 40GbE ports on each switch for uplinks.

ProLiant SL4540 Gen8 servers The first rack configuration is the same as the Single Rack configuration. All the 27 ProLiant SL4540 Gen8 (3 x 15) server nodes in the second rack are configured with worker services, as all required management services are already configured in the Single Rack RA.

Figure 10. Medium Cluster MapR M7 Configuration – Rack Level View

The details of various node services in the MapR M7 Medium cluster are as follows:

Nodes 1 and 4 of Rack 1

Nodes 1 and 4 of Rack 1 have the following services:

• Container Location Database (CLDB) service

• FileServer service

• NFS service

• TaskTracker service (If MapReduce is used)

Nodes 2 and 5 of Rack 1

Nodes 2 and 5 of Rack 1 have the following services:

• JobTracker service (If MapReduce is used)

• WebServer service

• Metrics service

• FileServer service

Page 20: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

20

• NFS service

• TaskTracker service (If MapReduce is used)

Nodes 6, 9, and 12 of Rack 1

Nodes 6, 9 and 12 of Rack 1 have the following services:

• ZooKeeper service

• FileServer service

• NFS service

• TaskTracker service (If MapReduce is used)

Note MapR will automatically reduce the number of map and reduce slots on nodes running control services, if you use the default settings. If setting these values manually, you should reduce the number of map and reduce slots in order to ensure that the worker services do not consume the resources required for the control services.

Rest of the SL4540 3x15 servers in Rack 1 and Second Rack

After the nodes running the control services are configured, remaining nodes in the first rack and the second rack can be configured to run worker services exclusively. Both performance and storage capacity will improve as more nodes and more racks are added. The following worker services run on the remaining nodes of Rack 1 and Rack 2:

• FileServer

• Network File Share (NFS)

• TaskTracker (If MapReduce is used)

Figure 11. Medium Cluster MAPR M7 Configuration– MapR Services

Page 21: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

21

Multi-Rack Cluster MapR M7 configuration

The MapR M7 Multi-Rack Cluster configuration extends the scalability further to large clusters. To meet the additional scale out requirements, one more CLDB service and one more JobTracker service are added to the configuration. In addition, the nodes with control services are now exclusively used for control services use and worker services will not be placed on these nodes. To provide the highest availability, the redundant control services, i.e., CLDB, ZooKeeper, and JobTracker services, have been distributed across 3 separate racks. After the third rack, all additional racks will simply have nodes with worker services.

Rack enclosure All racks contain nine HP ProLiant SL4540 Gen8 chassis with 27 server nodes and two HP 5900AF-48XG-4QSFP+ switches within a 42U rack, but the services are configured differently, as described below. This leaves 1U open for optional KVM switch.

Network

As previously described in the Switches section, two HP 5900AF-48XG-4QSFP+ switches are specified for performance and redundancy. The HP 5900AF-48XG-4QSFP+ includes up to four 40GbE uplinks which can be used to connect the switches in the rack into the desired network. Keep in mind that if IRF bonding is used, it requires 2 40GbE ports per switch, which would leave 2 40GbE ports on each switch for uplinks.

ProLiant SL4540 Gen8 servers

The first 3 racks will have a few server nodes (Nodes 1, 4, and 7 – across separate chassis) with control services and they are now exclusively used for control services, i.e., they will not be used for any worker services. The rest of the nodes in the first 3 racks and all 27 server nodes in each additional rack are configured with worker services.

Note Nodes with CLDB service will still have FileServer service as CLDB needs a container to store its metadata. In a large cluster (100 nodes or more) you may consider creating CLDB-only nodes to ensure high performance. This configuration provides additional control over the placement of the CLDB data, for load balancing, fault tolerance, or high availability (HA). Setting up CLDB-only nodes involves restricting the CLDB volume to its own topology and making sure all other volumes are on a separate topology. For details on isolating CLDB nodes, please see the following link: mapr.com/doc/display/MapR/Isolating+CLDB+Nodes

Figure 12. Large cluster MapR M7 Configuration – Rack Level View

Page 22: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

22

The details of various node services in the MapR M7 Large Cluster are as follows:

Node 1 of Rack 1, Rack 2, and Rack 3

Node 1 of first 3 racks has only the following services:

• Container Location Database (CLDB) service

• FileServer

Node 4 of Rack 1, Rack 2, and Rack 3

Node 4 of first 3 racks has the following services:

• JobTracker service (If MapReduce is used)

• WebServer service

• Metrics service

Node 7 of Rack 1, Rack 2, and Rack 3

Node 7 of first 3 racks has only the following service:

• ZooKeeper service

Rest of the SL4540 3x15 servers in all Racks

After the nodes running the control services are configured, remaining nodes in the first rack and the rest of the racks can be configured to run worker services exclusively. Both performance and storage capacity will improve as more nodes and more racks are added. The following worker services run on the remaining nodes:

• FileServer

• Network File Share (NFS)

• TaskTracker (If MapReduce is used)

Figure 13. Large Cluster MAPR M7 Configuration– MapR Services

Page 23: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

23

Edge nodes

Edge nodes are servers that are typically multi-homed into both the MapR private network as well as a network accessible by users. These are optional in the MapR cluster. Multi-homing allows the edge nodes to run applications that can submit jobs to the MapR cluster and have the application interfaces accessible to users within the user network. The edge nodes can be used as a staging area where data from the other systems located in the external network is stored temporarily before being migrated to the MapR filesystem. Another example of how an edge node can be leveraged would be to install and configure application tools such as Datameer and Platfora, etc. This provides users with tools that interact with MapR to allow users to analyze and visualize their data. Applications that run on edge nodes typically do not require a large amount of Direct Attached Storage and thus are well suited to the 1U HP ProLiant DL360p Gen8.

HP ProLiant DL360p Gen8

The HP ProLiant DL360p Gen8 (1U) is an excellent choice as an Edge Node for MapR M7 clusters.

Figure 14. HP ProLiant DL360p Gen8 Server

The following base configuration is popular for Edge Nodes:

• Dual Six-Core Intel E5-2667 2.9 GHz Processors

• Smart Array P420i Controller

• Eight 900GB SFF SAS 10K RPM disks

• 64 GB DDR3 Memory

• 4 x 1GbE NICs

Table 8. The HP ProLiant DL360p Gen8 Server Configuration

Qty Description

1 HP DL360p Gen8 8-SFF CTO Chassis

1 HP DL360p Gen8 E5-2667 FIO Kit

1 HP DL360p Gen8 E5-2667 Kit

8 HP 8GB 1Rx4 PC3-12800R-11 Kit

4 HP 900GB 6G SAS 10K 2.5in SC ENT HDD

1 HP Ethernet 1GbE 4P 331FLR FIO Adapter

1 HP 512MB FBWC for P-Series Smart Array

2 HP 460W CS Gold Hot Plug Power Supply Kit

1 HP 1U SFF BB Gen8 Rail Kit

1 ProLiant DL36x(p) HW Support

Summary

HP and MapR allow you to derive new business insights from Big Data by providing a platform to store, manage and process data at scale. However, designing and ordering Hadoop Clusters can be both complex and time consuming. This white paper provides several reference configurations for deploying clusters of varying sizes of the MapR M7 edition on HP infrastructure. These configurations leverage HP’s balanced building blocks of servers, storage and networking, along with integrated management software and bundled support. In addition, this white paper has been created to assist in the rapid design and deployment of the MapR M7 edition on HP infrastructure for clusters of various sizes.

Page 24: HP Reference Architecture for MapR M7 on HP ProLiant ... · HP Reference Architecture for MapR M7 on HP ProLiant ... included in this document have been jointly designed and developed

Technical white paper | HP Reference Architecture for MapR M7 on HP ProLiant SL4540 Gen8 Server

For more information

MapR, mapr.com

HP Solutions for Apache Hadoop, hp.com/go/hadoop

HP Insight Cluster Management Utility (CMU), hp.com/go/cmu

HP ProLiant SL4540 Gen8, hp.com/servers/sl4540

HP ProLiant servers, hp.com/go/proliant

HP Enterprise Software, hp.com/go/software

HP Networking, hp.com/go/networking

HP Integrated Lights-Out (iLO), hp.com/servers/ilo

HP Product Bulletin (QuickSpecs), hp.com/go/quickspecs

HP Services, hp.com/go/services

HP Support and Drivers, hp.com/go/support

HP Systems Insight Manager (HP SIM), hp.com/go/hpsim

To help us improve our documents, please provide feedback at hp.com/solutions/feedback.

Sign up for updates

hp.com/go/getupdated

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for

HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as

constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates.

4AA4-8663ENW, September 2013