Lenovo Big Data Reference Architecture for IBM BigInsights for Apache Hadoop

Dan Kangas (Lenovo) Ajay Dholakia (Lenovo) Jesse Chen (IBM) Stewart Tate (IBM)

Last update: 15 February 2017

Configuration Reference Number: BDABGNTXX53

Uses powerful, versatile new Lenovo® System x3650 M5 server

Leading security and reliability Innovative energy efficiency and performance

A software platform for discovering, analyzing, and visualizing data from disparate sources

Helps quickly build and deploy custom analytics to capture insight from data

ii Lenovo Big Data Reference Architecture for IBM BigInsights

Table of Contents 1 Introduction ............................................................................................... 1

2 Business problem and business value ................................................... 2

2.1 Business problem .................................................................................................... 2

2.2 Business value ......................................................................................................... 2

3 Reference architecture use ...................................................................... 4

4 Requirements ............................................................................................ 6

4.1 Functional requirements........................................................................................... 6

4.2 Non-functional requirements .................................................................................... 6

5 IBM BigInsights for Apache Hadoop Overview ...................................... 7

5.1 What’s new in IBM BigInsights 4.1 for Apache Hadoop ........................................... 8

6 Architectural overview ............................................................................. 9

7 Component model .................................................................................. 10

7.1 New capabilities in IBM BigInsights for Apache Hadoop ........................................ 11

8 Operational model .................................................................................. 15

8.1 Hardware description ............................................................................................. 15 8.1.1 Lenovo System x3650 M5 Server ............................................................................................. 15 8.1.2 Lenovo System x3550 M5 Server ............................................................................................. 16 8.1.3 Lenovo RackSwitch G8052 ....................................................................................................... 16 8.1.4 Lenovo RackSwitch G8272 ....................................................................................................... 17 8.1.5 Lenovo RackSwitch G8332 - Cross-Rack Switch ..................................................................... 18

8.2 Cluster nodes ......................................................................................................... 19 8.2.1 Management nodes ................................................................................................................... 19 8.2.2 Data nodes ................................................................................................................................ 21 8.2.3 Edge nodes ................................................................................................................................ 23

8.3 Systems management node .................................................................................. 24 8.3.1 Data network .............................................................................................................................. 26 8.3.2 Hardware management network ............................................................................................... 26

iii Lenovo Big Data Reference Architecture for IBM BigInsights

8.3.3 Multi-rack network...................................................................................................................... 27

8.4 Power considerations ............................................................................................. 29

8.5 Predefined cluster configurations ........................................................................... 30

9 Deployment considerations ................................................................... 33

9.1 Rack considerations ............................................................................................... 33

9.2 Increasing cluster performance .............................................................................. 34

9.3 Designing for lower cost ......................................................................................... 34

9.4 Designing for high ingest rates ............................................................................... 35

9.5 Estimating disk space ............................................................................................ 36

9.6 Scaling considerations ........................................................................................... 37

9.7 High availability considerations .............................................................................. 38 9.7.1 Networking considerations ........................................................................................................ 38 9.7.2 Hardware availability considerations ......................................................................................... 38 9.7.3 Storage availability ..................................................................................................................... 38 9.7.4 Software availability considerations ........................................................................................... 39

10 Acknowledgements ................................................................................ 40

11 Resources ............................................................................................... 41

12 Document History ................................................................................... 43

1 Lenovo Big Data Reference Architecture for IBM BigInsights

1 Introduction This document describes the reference architecture for the Big Data Solution from IBM. It provides a predefined and optimized hardware infrastructure for IBM BigInsights for Apache Hadoop, which is a distribution of Apache Hadoop with added value capabilities that are specific to IBM. BigInsights for Apache Hadoop (BigInsights V4.1) is a logical progression of IBM InfoSphere BigInsights V3. This reference architecture provides the planning, design considerations, and best practices for implementing IBM BigInsights with Lenovo products.

The Lenovo and IBM teams worked together on this document, and the reference architecture that is described herein was validated by Lenovo and IBM.

The predefined configuration provides a baseline configuration for a big data solution, which can be modified based on the specific customer requirements, such as lower cost, improved performance, and increased reliability.

The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing, and implementing the big data solution with Lenovo hardware. It is assumed that you are familiar with Apache Hadoop components and capabilities. For more information about Hadoop, see “Resources” on page 44.

.

2 Lenovo Big Data Reference Architecture for IBM BigInsights

2 Business problem and business value This section describes the business problem that is associated with big data environments and the value that is offered by the IBM BigInsights solution that uses Lenovo hardware.

2.1 Business problem By 2012, the world had generated 2.5 million terabytes (TB) of data, a level that is expected to increase to 44 zettabytes (44 trillion GB) by 2020. In all, 90% of the data in the world today was created in the last two years alone. This data comes from everywhere, including sensors that are used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone global positioning system (GPS) signals. This data is big data.

Big data spans the following dimensions:

• Volume: Big data comes in one size: large. Enterprises are awash with data, easily amassing terabytes and even petabytes of information.

• Velocity: Often time-sensitive, big data must be used as it is streaming into the enterprise to maximize its value to the business.

• Variety: Big data extends beyond structured data, including unstructured data of all varieties, such as text, audio, video, click streams, and log files.

Big data is more than a challenge; it is an opportunity to find insight into new and emerging types of data to make your business more agile. Big data also is an opportunity to answer questions that, in the past, were beyond reach. Until now, there was no effective way to harvest this opportunity. Today, IBM’s platform for big data uses such technologies as the real-time analytics processing capabilities of stream computing and the massive scale-out capabilities of Hadoop to open the door to a world of possibilities.

As part of the IBM platform for big data, IBM InfoSphere Streams allow you to capture and act on all of your business data, all of the time, just in time.

2.2 Business value IBM BigInsights for Apache Hadoop brings the power of Apache Hadoop to the enterprise. Apache Hadoop is the open source software framework that is used to reliably manage large volumes of structured and unstructured data. BigInsights enhances this technology to withstand the demands of your enterprise, adding administrative, provisioning, and security features, along with best-in-class analytical capabilities from IBM Research. The result is a more developer-compatible and user-compatible solution for complex, large-scale analytics.

How can businesses process tremendous amounts of raw data in an efficient and timely manner to gain actionable insights? By using IBM BigInsights for Apache Hadoop, organizations can run large-scale, distributed analytics jobs on clusters of cost-effective server hardware. This infrastructure can be used to tackle large data sets by breaking up the data into chunks and coordinating the processing of the data across a massively parallel environment. When the raw data is stored across the nodes of a distributed cluster, queries and analysis of the data can be handled efficiently, with dynamic interpretation of the data format at read time. The bottom line is that businesses can finally embrace massive amounts of untapped data and mine that data for valuable insights in a more efficient, optimized, and scalable way.

3 Lenovo Big Data Reference Architecture for IBM BigInsights

IBM BigInsights for Apache Hadoop that is deployed on Lenovo System x servers with Lenovo networking components provides superior performance, reliability, and scalability. The reference architecture supports entry through high-end configurations and the ability to easily scale as the use of big data grows. A choice of infrastructure components provides flexibility in meeting varying big data analytics requirements.

4 Lenovo Big Data Reference Architecture for IBM BigInsights

3 Reference architecture use The Lenovo Big Data Reference Architecture for IBM BigInsights for Apache Hadoop represents a well-defined starting point for architecting a IBM BigInsights for Apache Hadoop hardware and software solution and can be modified to meet client requirements.

When reviewing the potential of using System x with IBM BigInsights for Apache Hadoop, use this reference architecture paper as part of an overall assessment process with a customer.

When working on a big data proposal with a client, you can go through several phases and activities as outlined in the following list and in Table 1.

• Discover the client’s technical requirements and usage (hardware, software, data center, workload, user data, and high availability).

• Analyze the client’s requirements and current environment. • Exploit with proposals based on Lenovo hardware and IBM software.

Table 1. Client technical discovery, analysis, and exploitation

Discover Analyze Exploit

New applications

• Determine data storage requirements, including user data size and compression ratio.

• Determine high availability requirements.

• Determine customer corporate networking requirements, such as networking infrastructure and IP addressing.

• Determine whether data node OS disks require mirroring.

• Determine disaster recovery requirements including backup/recovery and multisite disaster recover requirements.

• Determine cooling requirements, such as airflow and BTU requirements.

• Determine workload characteristics, such as Apache YARN or HBase.

• Identify cluster management strategy, such as node firmware and OS updates.

• Identify a cluster rollout strategy, such as node hardware and software deployment.

• Propose IBM BigInsights for Apache Hadoop cluster as the solution to big data problems.

• Use the Lenovo System x M5 architecture for easy scalability of storage and memory.

Existing

applications

• Determine data storage requirements and existing shortfalls.

• Determine memory requirements and existing

• Propose a nondisruptive and lower risk solution.

• Propose a Proof-of-Concept (PoC) for the

5 Lenovo Big Data Reference Architecture for IBM BigInsights

Discover Analyze Exploit

shortfalls.

• Determine throughput requirements and existing bottlenecks.

• Identify system utilization inefficiencies.

next server deployment.

• Propose an BigInsights cluster as a solution to big data problems.

• Use System x M5 architecture for easy scalability of storage and memory.

Data center

health

• Determine server sprawl.

• Determine electrical, cooling, space headroom.

• Identify inefficiency concerns.

• Propose a scalable BigInsights cluster.

• Propose lowering data center costs with energy efficient System x servers.

6 Lenovo Big Data Reference Architecture for IBM BigInsights

4 Requirements The functional and non-functional requirements for this reference architecture are desribed in this section.

4.1 Functional requirements A big data solution supports the following key functional requirements:

• Various application types, including batch and real-time analytics • Industry-standard interfaces so that applications can work with IBM BigInsights • Real-time streaming and processing of data • Various data types and databases • Various client interfaces • Large volumes of data

4.2 Non-functional requirements Customers require their big data solution to be easy, dependable, and fast. The following non-functional requirements are key:

• Easy:

o Ease of development o Easy management at scale o Advanced job management o Multi-tenancy

• Dependable:

o Data protection with snapshot and mirroring o Automated self-healing o Insight into software/hardware health and issues o High availability (HA) and business continuity

• Fast:

o Superior performance o Scalability

• Secure:

o Strong authentication and authorization o Kerberos support o Data confidentiality and integrity

7 Lenovo Big Data Reference Architecture for IBM BigInsights

5 IBM BigInsights for Apache Hadoop Overview IBM BigInsights for Apache Hadoop is a complete Hadoop platform and is designed to enhance in-Hadoop

analytics for data scientists and analysts. It offers three new modules and runs in IBM Open Platform which is

IBM’s implementation of ODP listed below:

• IBM BigInsights Analyst, which will feature sophisticated SQL support and a spreadsheet-style

interface for exploring and visualizing big data. Big SQL is IBM’s industry-standard SQL query

interface and SQL processing engine for data stored in HDFS, Hive, or HBase. BigSheets enables

analysts to work with big data without writing any code. If you want to explore what’s available today

in Big SQL or BigSheets, start with the BigInsights Overview tutorial on Hadoop Dev.

• IBM BigInsights Data Scientist, which will include a new machine-learning engine with various

algorithms, such as Decision Trees, PageRank and Clustering. It will also provide native support for

open source R statistical computing.

• IBM BigInsights Enterprise Management, which will introduce new management tools for allocating

resources and optimizing workflows so organizations can more easily scale their Hadoop platform to

large numbers of users and clusters. These tools will provide multi-tenancy and multi-instance support

in a cluster.

In addition, IBM also announced that it has become a founding member of the Open Data Platform (ODP)

Initiative, a new industry association announced today to help drive collaboration, innovation, and

standardization across Hadoop and big data technologies. ODP core efforts will initially focus on Apache

Hadoop (including HDFS, YARN, and MapReduce) and Apache Ambari. IBM also includes Apache Spark in

its’ platform to support new computing engines and additional analytic applications. The BigInsights modules

also run on Open Data Platform. For more details, see the IBM press release or visit this IBM web site.

8 Lenovo Big Data Reference Architecture for IBM BigInsights

Figure 1 shows the overview of the IBM BigInsights platform and the BigInsights value-add services.

Figure 1. IBM BigInsights Overview

5.1 What’s new in IBM BigInsights 4.1 for Apache Hadoop This is a list of new features in IBM BigInsights 4.1:

• Access to all data, whether in Hive, HBase or HDFS within a single query (Big SQL)

• Improved high availability, greater performance and even richer SQL (Big SQL)

• Ability to manipulate and visualize data with a spreadsheet-like interface that now includes web tooling for business users (BigSheets)

• Automated prediction through machine learning algorithms in R

• Deeper insight through advanced analytics, including text and geospatial

• Enhanced text analytics that can infer context and relationships from text

• BigSheets supports important use cases for data science teams

9 Lenovo Big Data Reference Architecture for IBM BigInsights

6 Architectural overview From an infrastructure design perspective, basic Hadoop version 2 has two core components: Hadoop Distributed File System (HDFS) for storing data and MapReduce parallel processing framework for processing data. The IBM BigInsights reference architecture solution has three server roles:

• Management nodes: Nodes that are implemented on System x3550 M5 servers. These nodes encompass BigInsights daemons that are related to managing the cluster and coordinating the distributed environment.

• Data nodes: Nodes that are implemented on System x 3650 M5 servers. These nodes encompass daemons that are related to storing data and accomplishing work within the distributed environment.

• Edge nodes: Nodes that act as a boundary between the BigInsights cluster and the outside (client) environment.

Figure 2 shows the architecture overview of the IBM BigInsights reference architecture that uses Lenovo hardware. HBase related services are optional and only needed when customers need to use a NoSQL Database.

Figure 2. IBM BigInsights architecture overview

10 Lenovo Big Data Reference Architecture for IBM BigInsights

7 Component model Figure 3 illustrates the component model for the BigInsights Reference Architecture.

Figure 3. BigInsights Reference Architecture component model

HDFS is the file system in which Hadoop stores data. HDFS provides a distributed file system that spans all the nodes within a Hadoop cluster, linking the files systems on many local nodes to make one big file system with a single namespace.

HDFS has three associated daemons:

• NameNode: Runs on a management node and is responsible for managing the HDFS namespace and access to the files stored in the cluster.

• HA NameNode: Standby NameNode that can be used to provide automated failover. • DataNode: Runs on all data nodes and is responsible for managing the storage that is used by HDFS

across the BigInsights Hadoop Cluster.

BigInsights 4.1 comes with two options for MapReduce, scheduling and resource management. These are Apache Hadoop YARN and IBM Platform Symphony. YARN (Yet Another Resource Negotiator) is one of the key features in the second-generation Hadoop 2 version of the Apache Software Foundation's open source distributed processing framework. Platform Symphony is a high performance distributed computing and high-throughput data access framework through which Hadoop understands jobs and assigns work to servers within the BigInsights Hadoop cluster. It is a big data analytics product widely used in large scale grid computing environments.

The Apache Hadoop YARN MapReduce has two associated daemons:

• YARN ResourceManager: Manages and arbitrates resources among all the applications in the system.

11 Lenovo Big Data Reference Architecture for IBM BigInsights

• YARN NodeManager: It is the per-machine framework agent who is responsible for containers, monitoring their resource usage (cpu, memory, disk, network) and reporting the same to the ResourceManager/Scheduler.

HBase is a schemaless, No-SQL database that is implemented within the Hadoop environment and is included in BigInsights. HBase has its own set of daemons that run on management nodes and data nodes. The HBase daemons are in addition to the management node and data node daemons of HDFS.

HBase has two daemons:

• HMaster: The HBase master daemon. It is responsible for monitoring the HBase cluster and is the interface for all metadata changes.

• HRegionServer: The HBase region daemon. It runs on all data nodes. The HRegionServer daemon is responsible for managing and serving HBase regions. Within HBase, a region is the basic unit of distribution of an HBase table, allowing a table to be distributed across multiple servers within a cluster.

Use care when considering running MapReduce workloads in a cluster that is also running HBase. MapReduce jobs can use significant resources and can have a negative impact on HBase query performance and service-level agreements (SLAs). Some utilities, such as IBM BigSQL, are able to effectively collocate MapReduce and HBase workloads within the same cluster. We recommend giving careful consideration before running MapReduce jobs (beyond those related to HBase utilities) on a cluster that requires low-latency responses to HBase queries.

ZooKeeper is centralized daemon that enables synchronization and coordination across the bigdata cluster; it runs on management nodes too.

Additionally, IBM BigInsights uses Apache Ambari, an open framework, to help administrators to maintain servers, manage services and HDFS components, and manage data nodes within the BigInsights cluster. Apache Ambari is an open framework for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides an intuitive and easy-to-use Hadoop management web UI backed by its collection of tools and APIs that simplify the operation of Hadoop clusters. Ambari metrics services runs on a management node.

7.1 New capabilities in IBM BigInsights for Apache Hadoop

IBM BigInsights for Apache Hadoop has several new capabilities.

• Big SQL: Big SQL enables analysts to leverage IBM's strength in SQL engines to provide ANSI SQL access to data across any system from Hadoop, via JDBC or ODBC - seamlessly whether that data exists in Hadoop or a relational database. This means that developers familiar with the SQL programming language can access data in Hadoop without having to learn new languages or skills.

With Big SQL, all of your big data is SQL accessible. It presents a structured view of your existing data, using an optimal execution strategy, given available resources. Big SQL can leverage MapReduce parallelism when needed for complex data sets and avoid it when it hinders, using direct access for smaller, and low-latency queries. Big SQL offers the following capabilities: o Low-latency queries enabled by massively parallel processing (MPP) technology o Query rewrite optimization and cost-based optimizer

12 Lenovo Big Data Reference Architecture for IBM BigInsights

o Integration of both Hive and HBase data sources o Unparalleled support for ANSI SQL Standard o Federated query access to IBM DB2®, Oracle, Teradata and ODBC sources Big SQL supports the most common use cases for modernizing and building next-generation logical data warehouses: o Offload data and workloads from existing data warehouses o Move rarely used data out of high-cost data warehouses by creating query able archives in

Hadoop o Enable rapid prototyping of business intelligence reports o Support rapid adoption of Hadoop by using existing SQL skills, without compromising on data

security.

• BigSheets: BigSheets makes Do It Yourself Analytics into a reality for Analysts by going beyond structured database management into unstructured data management. Seeing the whole picture will help all levels of business make better decisions.

BigSheets provides a web-based, spreadsheet-style view into collections of files in Hadoop. Users can perform data transformations, filtering and visualizations at massive scale. No coding is required because BigSheets translates the spreadsheet actions into MapReduce to leverage the computational resources of the Hadoop cluster. This helps analysts discover value in data quickly and easily. BigSheets is an extension of the mashup paradigm that: o Integrates gigabytes, terabytes, or petabytes of unstructured data from web-based repositories o Collects a wide range of unstructured web data stemming from user-defined seed URLs o Extracts and enriches that data using the unstructured information management architecture you

choose (LanguageWare, OpenCalais, etc.) o Let users explore and visualize this data in specific, user defined contexts (such as ManyEyes). Some of the BigSheets benefits include: o Provides business users with a new approach to keep pace with data escalation. By taking the

structure to the data, this helps mine petabytes of data without additional storage requirements. o BigSheets provides business users with a new approach that allows them to break down data into

consumable, situation-specific frames of reference. This enables organizations to translate untapped, unstructured, and often unknown web data into actionable intelligence.

o Leverage all the compute resources of the Hadoop cluster to drive insights and visualizations with BigSheets right on the cluster—no extraction required

13 Lenovo Big Data Reference Architecture for IBM BigInsights

Figure 4 shows the value of Big SQL and BigSheets.

Figure 4. Value of Big SQL and BigSheets

• Big R: For Data Scientists, the IBM BigInsights Data Scientist module includes Big R.

Big R enables data scientists to run native R functions to explore, visualize, transform and model big data right from within the R environment. Data scientists can now run scalable machine learning algorithms with a wide class of algorithms and growing R-like syntax for new algorithms & customize existing algorithms. BigInsights for Apache Hadoop running Big R can use the entire cluster memory, spill to disk and run thousands of models in parallel. Big R provides a new processing engine enables automatic tuning of machine learning performance over massive data sets in Hadoop clusters. Big R can be used for comprehensive data analysis, hiding some of the complexity of manually writing MapReduce jobs. Benefits of Big R include: o End-to-end integration with open source R o Transparent execution on Hadoop o Seamless access to rich and scalable machine learning algorithms provided in Big R o Text analytics to extract meaningful information from unstructured data

• Text Analytics: A sophisticated text analytics capability unique to BigInsights allows developers to easily build high-quality applications able to process text in multiple written languages, and derive insights from large amounts of native textual data in various formats.

• Enterprise Management: For Administrators, the IBM BigInsights Enterprise Management module provides a comprehensive web-based interface included in BigInsights simplifies cluster management,

14 Lenovo Big Data Reference Architecture for IBM BigInsights

service management, job management and file management. Administrators and users can share the same interface, launching applications and viewing a variety of configurable reports and dashboards.

• Built-in Security: BigInsights was designed with security in mind, supporting Kerberos authentication and providing data privacy, masking and granular access controls with auditing and monitoring functions to help ensure that the environments stays secure.

15 Lenovo Big Data Reference Architecture for IBM BigInsights

8 Operational model This section describes the operational model for the IBM BigInsights reference architecture. To show the operational model for different sized customer environments, three different models are provided for supporting different amounts of data. Throughout the document, these models are referred to as starter rack, full rack, and multi-rack configuration sizes. The multi-rack is three times larger than the full rack. A IBM BigInsights deployment consists of cluster nodes, networking, power, and racks. The predefined configurations can be implemented as is or modified based on specific customer requirements, such as lower cost, improved performance, and increased reliability. Key workload requirements, such as the data growth rate, sizes of datasets, and data ingest patterns help in determining the proper configuration for a specific deployment. A best practice when an IBM BigInsights cluster infrastructure is designed is to conduct the proof of concept testing by using representative data and workloads to ensure that the proposed design works.

8.1 Hardware description This reference architecture uses Lenovo System x3650 M5 and x3550 M5 servers and Lenovo RackSwitch G8052 and G8272 top of rack switches.

8.1.1 Lenovo System x3650 M5 Server The Lenovo System x3650 M5 server (as shown in Figure 5) is an enterprise class 2U two-socket versatile server that incorporates outstanding reliability, availability, and serviceability (RAS), security, and high-efficiency for business-critical applications and cloud deployments. It offers a flexible, scalable design and simple upgrade path to 26 2.5-inch hard disk drives (HDDs) or solid-state drives (SSDs), or 14 3.5-inch HDDs with doubled data transfer rate through 12 Gbps serial-attached SCSI (SAS) internal storage connectivity and up to 1.5 TB of TruDDR4 Memory. On-board it provides four standard embedded Gigabit Ethernet ports and two optional embedded 10 Gigabit Ethernet ports without occupying PCIe slots.

Figure 5. Lenovo System x3650 M5

Combined with the Intel® Xeon® processor E5-2600 v4 product family, the Lenovo x3650 M5 server offers an even higher density of workloads and performance that lowers the total cost of ownership (TCO) per virtual machine. Its pay-as-you-grow flexible design and great expansion capabilities solidify dependability for any kind of virtualized workload with minimal downtime.

The x3650 M5 server provides internal storage density of up to 87.6 TB in a 2U form factor with its impressive array of workload-optimized storage configurations. It also offers easy management and saves floor space and power consumption for most demanding storage virtualization use cases by consolidating storage and server

16 Lenovo Big Data Reference Architecture for IBM BigInsights

into one system.

The reference architecture recommends the storage-rich System x3650 M5 model for the following reasons:

• Storage capacity: The nodes are storage-rich. Each of the 14 3.5-inch drives has raw capacity up to 6 TB and each of two 2.5-inch drives has raw capacity of 1.8 TB for a total of 87.6 TB per node and over 1 petabyte per rack.

• Performance: This hardware supports the latest Intel Xeon processors and TruDDR4 Memory. • Flexibility: Server hardware uses embedded storage, which results in simple scalability (by adding

nodes). • More PCIe slots: Up to 8 PCIe slots are available if rear disks are not used, and up to 2 PCIe slots if

both Rear 3.5-inch HDD Kit and Rear 2.5-inch HDD Kit are used. They can be used for network adapter redundancy and increased network throughput.

• Better power efficiency: Innovative power and thermal management provides energy savings. • Reliability: Lenovo is first in the industry in reliability and has exceptional uptime with reduced costs.

For more information, see the following Lenovo System x3650 M5 website :

Lenovo x3650 M5 with V4 CPU.

8.1.2 Lenovo System x3550 M5 Server The Lenovo System x3550 M5 server (as shown in Figure 6) is a cost- and density-balanced 1U two-socket rack server. The x3550M5 features a new, innovative, energy-smart design with up to two Intel Xeon processors of the high-performance E5-2600 v4 product family processors a large capacity of faster, energy-efficient TruDDR4 Memory, up to twelve 12Gb/s SAS drives, and up to three PCI Express (PCIe) 3.0 I/O expansion slots in an impressive selection of sizes and types. The server’s improved feature set and exceptional performance is ideal for scalable cloud environments.

Figure 6: Lenovo System x3550 M5

For more information, see the following Lenovo System x3550 M5 website:

Lenovo x3550 M5 with V4 CPU.

8.1.3 Lenovo RackSwitch G8052 The Lenovo System Networking RackSwitch G8052 (as shown in Figure 7) is an Ethernet switch that is designed for the data center and provides a virtualized, cooler, and simpler network solution. The Lenovo RackSwitch G8052 offers up to 48 1 GbE ports and up to 4 10 GbE ports in a 1U footprint. The G8052 switch is

17 Lenovo Big Data Reference Architecture for IBM BigInsights

always available for business-sensitive traffic by using redundant power supplies, fans, and numerous high-availability features.

Figure 7. Lenovo RackSwitch G8052

Lenovo RackSwitch G8052 has the following characteristics:

• A total of 48 1 GbE RJ45 ports • Four standard 10 GbE SFP+ ports • Low 130W power rating and variable speed fans to reduce power consumption

For more information, see this website:

lenovopress.com/tips0813

8.1.4 Lenovo RackSwitch G8272 Designed with top performance in mind, Lenovo RackSwitch G8272 is ideal for today’s big data, cloud, and optimized workloads. The G8272 switch offers up to 72 10 Gb SFP+ ports in a 1U form factor and is expandable with four 40 Gb QSFP+ ports. It is an enterprise-class and full-featured data center switch that delivers line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. Redundant power and fans and numerous HA features equip the switches for business-sensitive traffic.

The G8272 switch (as shown in Figure 8) is ideal for latency-sensitive applications, such as client virtualization. It supports Lenovo Virtual Fabric to help clients reduce the number of I/O adapters to a single dual-port 10 Gb adapter, which helps reduce cost and complexity. The G8272 switch supports the newest protocols, including Data Center Bridging/Converged Enhanced Ethernet (DCB/CEE) for support of FCoE and iSCSI and NAS.

Figure 8: Lenovo RackSwitch G8272

The enterprise-level Lenovo RackSwitch G8272 has the following characteristics:

18 Lenovo Big Data Reference Architecture for IBM BigInsights

• 48 x SFP+ 10GbE ports plus 6 x QSFP+ 40GbE ports • Support up to 72 x 10Gb connections using break-out cables • 1.44 Tbps non-blocking throughput with low latency (~ 600 ns) • Up to 72 1Gb/10Gb SFP+ ports • OpenFlow enabled allows for easily created user-controlled virtual networks

For further information on the G8272 switch visit this link: Lenovo RackSwitch G8272

8.1.5 Lenovo RackSwitch G8332 - Cross-Rack Switch The RackSwitch™ G8332 (as shown in Figure 9) is a 40 Gigabit Ethernet (GbE) switch that is designed for the data center, providing speed, intelligence, and interoperability on a proven platform. It is an ideal aggregation class switch for connecting multiple RackSwitch G8264 class switches with their 40Gb uplink ports.. RackSwitch G8332 provides line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. Hot-swappable, redundant power and fans, along with numerous high-availability features, enable the RackSwitch G8332 to be available for business-sensitive traffic. 16 ports of 40 Gb Ethernet with QSFP+ transceivers are provided while each port can optionally opearte as four 10Gb ports using the 4x 10 Gb SFP+ break-out cable.

Figure 9 - Lenovo RackSwitch G8332

For further information on the G8332 switch visit this link:

Lenovo RackSwitch G8332

19 Lenovo Big Data Reference Architecture for IBM BigInsights

Cluster nodes The IBM BigInsights reference architecture is implemented on a set of nodes that make up a cluster. An IBM BigInsights cluster consists of four types of nodes: data nodes, management nodes, a systems management node and edge Nodes. Data nodes use System x3650 M5 servers with locally attached storage. Management nodes, edge nodes and systems management nodes use System x3550 M5 servers.

Data nodes run data (worker) services for storing and processing data.

Management nodes run management (control) services for coordinating and managing the cluster.

The edge node acts as a boundary between the IBM BigInsights cluster and the outside (client) environment.

The systems management node runs hardware alerting software and provides a dedicated node for hardware troubleshooting and firmware maintenance.

8.1.6 Management nodes Management nodes encompass the following HDFS, HBase and BigInsights management daemons:

• NameNode • HA NameNode • YARN ResourceManager • HMaster • Ambari Server

Table 2 lists the recommended components for a management node. Management nodes can be customized according to client needs.

Table 2. Management node configuration

Component Management node configuration

System System x3550 M5

Processor 2x Intel Xeon processor E5-2650 v4 2.2 GHz 12-core

Memory - base 128 GB – 8x 16 GB 2133 MHz RDIMM (minimum)

Disk (OS / local storage)

• 4 TB drives: 2 x 4TB NL SATA 3.5 inch • 6 TB drives: 2 x 6TB NL SATA 3.5 inch

HDD controller ServeRAID M5210 SAS/SATA Controller

Hardware storage protection RAID 1 hardware mirroring of two disk drives

Hardware management network adapter Integrated 1GBaseT Adapter

Data network adapter 10Gb Ethernet (bonded)

SSD and PCIe flash storage can be used to provide improved I/O performance.

20 Lenovo Big Data Reference Architecture for IBM BigInsights

A BigInsights Hadoop cluster requires between one and six management nodes, depending on the client’s environment. We recommend a minimum of one (1) Management node for Development and Test clusters (when performance is not a factor). If performance is a concern, we recommend a minimum of three (3) Management nodes. If Big SQL is used, we recommend a minimum of four (4) Management nodes. For HA Deployment, we recommend six (6) Management nodes.

Table 3 specifies the number of required management nodes. In this table, the columns that contain node information represent BigInsights Hadoop services that are housed across cluster management nodes.

Table 3. Cluster required management nodes

Environment Development

Environment

Production/Test

Environment(<20 data nodes)

Large Production/Test

Environment (>=20 data nodes)

HA Deployment

Required management nodes

1 41 6 1

Node 1

Ambari (PostgreSQL) PostgreSQL Knox Journal Node Zookeeper Hive Resource Manager HBase Master Oozie Job history server Big SQL Headnode Big SQL Scheduler Hive Server (MySQL) MySQL metastore NameNode2

Ambari MySQL server Knox Journal Node ZooKeeper3 LDAP

Ambari MySQL server

1 If there are multiple racks, management nodes should be separated across racks.

2 In a single management node configuration, place the Standby NameNode on a data node to enable recoverability of the HDFS namespace if a failure of the management node occurs.

3 There is no fixed approach to the number of ZooKeeper and greater than five instances is certainly possible. However, we recommend an odd number of ZooKeeper instances. In some failure modes, odd numbers of ZooKeeper instances permit the ZooKeeper quorum to exist with a fewer number of surviving instances.

21 Lenovo Big Data Reference Architecture for IBM BigInsights

Node 2

Resource Manager Name Node (standby) HBase Master Journal Node Zookeeper Oozie Hive Server2 Solr

Resource Manager Name Node (standby) Journal Node Zookeeper Oozie

Node 3

Name Node Job history server Journal Node Zookeeper Flume

Resource Manager (standby) Name Node Job history server Journal Node Zookeeper

Node 4

BigInsights HomePage Big SQL Headnode Big SQL Scheduler, DSM Server MySQL metastore (Hive) BigSheets Service Text Analytics Service BigR connector service

Big SQL Headnode Big SQL Scheduler HBase Master (standby) Hive Server2 MySQL metastore (Hive)

Node 5

Big SQL Headnode (Standby) Big SQL Scheduler (Standby) HBase Master Hive Server2 Journal Node Zookeeper

Node 6 Knox

If you plan to scale up the cluster significantly, it might be best to separate out each of these functions from the beginning, even if the starting configuration is smaller and requires fewer management nodes. The number of Management Nodes can be customized based on specific needs.

8.1.7 Data nodes Table 4 lists the recommended system components for data nodes.

Table 4. Data node configuration

Component Data node configuration

System System x3650 M5

Processor 2 x Intel Xeon processor E5-2680 v4 2.4GHz 14-core

22 Lenovo Big Data Reference Architecture for IBM BigInsights

Component Data node configuration

Memory - base 512 GB: 16 x 32GB 2400MHz RDIMM

Disk (OS) • 2x 2.5” HDD or SSD

Disk (data) • 4 TB drives: 14 x 4TB NL SATA 3.5 inch (56 TB Total) • 6 TB drives: 14 x 6TB NL SATA 3.5 inch (84 TB Total) • 8 TB drives: 14 x 8TB NL SATA 3.5 inch (96 TB Total)

HDD controller OS: ServeRAID M1215 SAS/SATA Controller

HDFS: N2215 SAS/SATA HBA

Hardware storage protection None (JBOD). By default, IBM BigInsights maintains a total of three copies of data stored within the cluster. The copies are distributed across data servers and racks for fault recovery.

Hardware management network adapter

Integrated 1GBaseT Adapter

Data network adapter 10Gb Ethernet (bonded)

The Intel® Xeon® processor E5-2650 v4 is recommended to provide sufficient performance. A minimum of 128 GB of memory is recommended for most MapReduce workloads with 256 GB or more recommended for HBase, Spark, and memory-intensive MapReduce workloads. Two sets of disks are used: one set of disks is for the operating system and the other set of disks is for data. For the operating system disks, SSD is recommended which can be used by both OS and bigdata shuffle, this will greatly improve the performance, RAID 1 or 0 can be chosen based on the needs.OS can also be installed on one of the data disk and leave both SSD for shuffle.

Each data node in the reference architecture has an internal directly attached storage. External storage is not used in this reference architecture. Available data space assumes the use of Hadoop replication with three copies of the data, and 25% capacity reserved for efficient file system operation and to allow time to increase capacity, if needed.

In situations where higher storage capacity is required, the main design approach is to increase the amount of data disk space per node. Using 6 TB drives instead of 4 TB drives increases the total per node data disk capacity from 56 TB to 84 TB, a 50% increase.

When increasing data disk capacity, there is a balance between performance and throughput. For some workloads, increasing the amount of user data that is stored per node can decrease disk parallelism and negatively affect performance. Increasing drives sizing also affects rebuilding and repopulating the replicas if there is a disk or node failure. Higher density disks or nodes results in higher rebuild times. Drives that are larger than 4 TB are not recommended based on the balance of capacity and performance. In this case, higher capacity can be achieved by increasing the number of nodes in the cluster.

For higher IO throughout, the data node can be configured with 24 2.5-inch SAS drives, which have less

23 Lenovo Big Data Reference Architecture for IBM BigInsights

storage capacity but much higher IO throughout.

For the HDD controller, just a bunch of disks (JBOD) is the best choice for a IBM BigInsights cluster. It provides excellent performance and, when combined with the Hadoop default of 3x data replication, also provides significant protection against data loss. The use of RAID with data disks is discouraged because it reduces performance and the amount data that can be stored. Data nodes can be customized according to client needs.

The number of data nodes that are required within a BigInsights cluster depends on the client requirements. Such requirements might include the size of a cluster, the size of the user data, the data compression ratio, workload characteristics, and data ingest.

A minimum of three data nodes are required as Hadoop has three copies of data by default. Three data nodes should be used for test or POC environments only. A minimum of five data nodes are required for production environment if there are data node failures.

8.1.8 Edge nodes The edge node acts as a boundary between the BigInsights cluster and the outside (client) environment. The edge node is used for data ingest, which refers to routing data into the cluster through the data network of the reference architecture. Edge nodes can be System x3550 M5 servers, other System x servers, or other client-provided servers.

Table 5 lists the recommended components for an edge node. Edge nodes can be customized according to client needs.

Table 5. Edge node configuration

Component Management node configuration

System System x3550 M5

Processor 2 x Intel Xeon processor E5-2650 v4 2.2 GHz 12-core

Memory - base 128 GB – 8x 16 GB 2133 MHz RDIMM (minimum)

Disk (OS / local storage)

• 4 TB drives: 2 x 4TB NL SATA 3.5 inch • 6 TB drives: 2 x 6TB NL SATA 3.5 inch

HDD controller ServeRAID M5210 SAS/SATA Controller

Hardware storage protection OS storage on 2x drives that are mirrored by using RAID 1 hardware mirroring.

Application storage on 2x drives in JBOD or RAID 1 hardware mirroring configuration.

Hardware management network adapter Integrated 1GBaseT Adapter

Data network adapter 10Gb Ethernet (bonded)

24 Lenovo Big Data Reference Architecture for IBM BigInsights

With the design of the System x3550 M5 management node, the same configuration can be used as an edge node. When you use this configuration as an edge node, the first port on each Broadcom dual-port 10GbE network adapter connects back to the G8272 switch at the top of the node’s home rack. The second port on each Broadcom dual-port 10GbE network adapter connects to the client’s data network. This edge node design serves as a ready-made platform for extract, transform, and load (ETL) tools, such as IBM DataStage®.

Although a BigInsights cluster can have multiple edge nodes, depending on applications and workload, not every cluster rack needs to be connected to an edge node. However, every data node within the BigInsights cluster must be a cluster data network IP address that is routable from within the corporate data network. As gateways into the BigInsights cluster, you must properly size edge nodes to ensure that they do not become a bottleneck for accessing the cluster, for example, during high volume ingest periods.

Important: The number of edge nodes and the edge node server physical attributes that are required depend on ingest volume and velocity. Because of physical space constraints within a rack, adding an edge node to a rack can displace a data node.

In low volume/velocity ingest situations (< 1 GB/hr), the Ambari management node can be used as an edge node. InfoSphere DataStage and InfoSphere Data Click servers can also function as edge nodes. When using InfoSphere DataStage or other ETL software, consult an appropriate ETL specialist for server selection.

In Proof-of-Concept (PoC) situations, the edge node can be used to isolate both cluster networks (data and administrative/management) from the customer corporate network.

8.2 Systems management node The systems management node provides a dedicated node for hosting hardware alerting software, and facilitating hardware troubleshooting and firmware maintenance.

Systems management node uses the Lenovo XClarity™ Administrator, which is a centralized resource management solution that reduces complexity, speeds up response, and enhances the availability of Lenovo® server systems and solutions.

The Lenovo XClarity Administrator provides agent-free hardware management for Lenovo’s System x® rack servers and Flex System™ compute nodes and components, including the Chassis Management Module (CMM) and Flex System I/O modules.

Figure 10 shows the Lenovo XClarity administrator interface in which Flex System components and rack servers are managed and are seen on the dashboard. Lenovo XClarity Administrator is a virtual appliance that is quickly imported into a virtualized environment server configuration.

25 Lenovo Big Data Reference Architecture for IBM BigInsights

Figure 10: XClarity Administrator interface

Also, the open source provisioning software, xCAT, provides a scalable distributed computing management and provisioning tool that provides a unified interface for hardware control, discovery, and operating system deployment. It can be used to facilitate or automate the management of cluster nodes. For more information about xCAT, see the Resources section.

8.3 Networking Regarding networking, the reference architecture specifies two networks: a data network and an administrative or management network. Figure 11 shows the networking configuration for IBM BigInsights .

26 Lenovo Big Data Reference Architecture for IBM BigInsights

Figure 11: IBM BigInsights network configuration

8.3.1 Data network The data network is a private cluster data interconnect among nodes that is used for data access, moving data across nodes within a cluster, and importing data into the IBM BigInsights cluster. The IBM BigInsights cluster typically connects to the customer’s corporate data network. The BigInsights cluster typically connects to the client’s corporate data network by using one or more edge nodes. These edge nodes can be System x 3550 M5 servers, other System x servers, or other client-specified server. Edge nodes act as interface nodes between the BigInsights cluster and the outside client environment (for example, data ingested from a corporate network into a cluster). Not every rack has an edge node connection to a client network. Data can be ingested into the cluster via edge nodes or via parallel ingest.

Two top of rack switches are required for the data network that is used by BigInsights. Two 1 GbE or 10 GbE switches can be used (a 1 Gb Ethernet switch is sufficient for some workloads). The recommended 1 GbE switch is the Lenovo RackSwitch G8052. The 10 Gb Ethernet switch can provide extra I/O bandwidth for better performance. The recommended 10 GbE switch is the Lenovo System Networking RackSwitch™ G8272.

The two Broadcom 10 GbE ports of each node are link aggregated to the recommended G8272 rack switch for better performance and improved HA. The data network is configured to use a virtual local area network (VLAN).

8.3.2 Hardware management network The hardware management network is a 1 GbE network that is used for in-band operating system administration and out-of-band hardware management. In-band administrative services, such as SSH or Virtual Network Computing (VNC) that is running on the host operating system enables cluster nodes to be

27 Lenovo Big Data Reference Architecture for IBM BigInsights

administered. Using the integrated management modules II (IMM2) within the System x3650 M5 server, out-of-band management enables the hardware-level management of cluster nodes, such as node deployment or basic input/output system (BIOS) configuration.

Hadoop has no dependency on the IMM2. Based on customer requirements, the administration links and management links can be segregated onto separate VLANs or subnets. The administrative or management network is typically connected directly to the customer’s administrative network. When the in-band administrative services on the host operating system are used, IBM BigInsights is configured to use the data network only.

The reference architecture requires one 1 Gb Ethernet top-of-rack switch for the hardware management network. Administrators also can access all of the nodes in the cluster through the customer admin network, as shown in Figure 11 on page 267. This rack switch for the hardware management network is connected to each of the nodes in the cluster by using two physical links (one for in-band operating system administration and one link for out-of-band IMM2 hardware management). On the nodes, the administration link connects to port 1 on the integrated 1 GBaseT adapter and the management link connects to the dedicated IMM2 port.

8.3.3 Multi-rack network The data network in the predefined reference architecture configuration consists of a single network topology. Appropriate other Lenovo RackSwitch G8332 core switches per cluster is required for cross racks communication. In this case, the second Broadcom 10 GbE port can be connected to the second Lenovo RackSwitch G8272. The over-subscription ratio for G8272 is 1:2.

Figure 12 shows how the network is configured when the IBM BigInsights cluster is installed across more than one rack. The data network is connected across racks by two aggregated 40 GbE uplinks from each rack’s G8272 switch to a core G8332 switch.

28 Lenovo Big Data Reference Architecture for IBM BigInsights

Figure 12: IBM BigInsights cross rack network configuration4

A 40GbE switch is recommended for interconnecting the data network across multiple racks. Lenovo System Networking RackSwitch G8332 is the recommended switch. A best practice is to have redundant core switches for each rack to avoid a single point of failure. Within each rack, the G8052 switch can optionally be configured to have two uplinks to the G8272 switch to allow propagation of the administrative or management VLAN across cluster racks through the G8332 core switch. For large clusters, the Lenovo System Networking RackSwitch G8332 is recommended because it provides a better cost value per 40 Gb port than the G8332. Many other cross rack network configurations are possible and might be required to meet the needs of specific deployments or to address clusters larger than three racks.

If the solution is initially implemented as a multi-rack solution, or if the system grows by adding racks, the nodes that provide management services are distributed across racks to maximize fault tolerance.

4. One G8272 can be used but two G8272s are configured for HA.

29 Lenovo Big Data Reference Architecture for IBM BigInsights

8.4 Power considerations Within racks, switches and management nodes have redundant power feeds with each power feed connected from a separate protocol data unit (PDU). Data nodes have a single power feed, and the data node power feeds should be connected so that all power feeds within the rack are balanced across the PDUs.

Figure 13 shows power connections within a full rack with three management nodes.

Figure 13: Power connections

30 Lenovo Big Data Reference Architecture for IBM BigInsights

8.5 Predefined cluster configurations The intent of the predefined configurations is to ease initial sizing for customers and to show example starting points for three different-sized workloads.

• The starter rack configuration consists of three data nodes and a pair of rack switches.

• The full rack configuration (a rack fully populated) consists of 17 nodes, managment nodes, and a pair of rack switches.

• The multi-rack contains a total of 53 data nodes and 4 master nodes spread across the 3 racks for improved performance, a pair of G8272 data switches and a G8052 systems management switch in each rack. Two cross-rack G8332 switches are also provided, and a systems management node

The configuration is not limited to these sizes, and any number of data nodes is supported.

The following table lists predefined configurations for the IBM BigInsights reference architecture. The table also lists the amount of space for data and the number of nodes that each predefined configuration provides. Storage space is described in two ways: the total amount of raw storage space when 4 TB or 6 TB drives (raw storage) are used and the amount of space for the data that the customer has (available data space). Available data space assumes the use of Hadoop replication with three copies of the data and 25% capacity that is reserved for intermediate data (scratch storage). The estimates that are listed in Table 6 do not include extra space that is freed up by using compression because compression rates can vary widely based on file contents.

Table 6. Cluster configurations

Starter rack Full rack Multi-rack

Raw storage (4 TB) 168 TB 952 TB 3,136 TB

Available data space (4 TB) 42 TB 238 TB 784 TB

Raw storage (6TB) 251 TB 1428 TB 4,704 TB

Available data space (6TB) 63 TB 357 TB 1,176 TB

Number of Data Nodes 3 17 56

Number of Management Nodes 4 4 4

Number of Racks 1 1 3

Number of 10 GbE cables 14 42 120

Number of 1 GbE cables 14 42 120

Figure 14 shows an overview of the architecture in three different one-rack sized clusters without network redundancy: a starter rack, a half rack, and a full rack.

Figure 15 on page 32 shows a multi-rack-sized cluster without network redundancy.

31 Lenovo Big Data Reference Architecture for IBM BigInsights

Full Rack Starter Rack

(17 Data Nodes) (3 Data Nodes)

Figure 14: Starter rack and full rack IBM BigInsights predefined configurations5

5 The network configuration HA is enabled by default. If the customer does not need network HA then only one G8272 switch is needed for

each configuration.

32 Lenovo Big Data Reference Architecture for IBM BigInsights

Multi-Rack (53 Data Nodes)

Figure 15: Multi-rack IBM BigInsights predefined configuration

33 Lenovo Big Data Reference Architecture for IBM BigInsights

9 Deployment considerations This section describes other considerations for deploying the IBM BigInsights solution.

The predefined configurations represent a baseline configuration that can be implemented as is or modified based on specific client requirements, such as lower cost, improved performance, and increased reliability.

When you consider modifying the predefined configuration, you must understand key aspects of how the cluster will be used. In terms of data, you must understand the current and future total data to be managed, the size of a typical data set, and whether access to the data will be uniform or skewed. In terms of ingest, you must understand the volume of data to be ingested and ingest patterns, such as regular cycles over specific time periods and bursts in ingest. Consider also the data access and processing characteristics of common jobs and whether query-like frameworks, such as IBM BigSQL, are used.

When designing a BigInsights cluster infrastructure, we recommend conducting the necessary testing and proof of concepts against representative data and workloads to ensure that the proposed design will achieve the necessary success criteria. The following sections provide information about customizing the predefined configuration. When considering customizations to the predefined configuration, work with a systems architect who is experienced in designing BigInsights cluster infrastructures.

9.1 Rack considerations Within a rack, data nodes occupy 2U of space and management nodes, and rack switches occupy 1U of space.

A one-rack BigInsights implementation comes in three sizes: Starter rack, half rack, and full rack. These three sizes allow for easy ordering. However, reference architecture sizing is not rigid and supports any number of data nodes with the appropriate number of management nodes. Table 7 on page 33 describes the node counts.

Table 7. Rack configuration node counts

Rack configuration size Number of data nodes6 Number of management nodes7

Starter rack 3 1 or 3

Full rack with management nodes 18 8 1, 3, or 5

Full rack, no management nodes 20 0

6 Maximum number of data nodes per full rack based on network switches, management nodes, and data nodes. Adding edge nodes to the rack can displace additional data nodes.

7 The number of management notes depends on development or the production/test environment type. For more information about selecting the correct number of management nodes, see “Management nodes” on page 13.

8 A full rack with one management nodes can accommodate up to 19 data nodes.

34 Lenovo Big Data Reference Architecture for IBM BigInsights

A BigInsights implementation can be deployed as a multirack solution. If the system is initially implemented as a multirack solution or if the system grows by adding more racks, to maximize fault tolerance, distribute the cluster management nodes across racks.

In the reference architecture for BigInsights, a fully populated predefined rack with two G8272 switch and one G8052 switch can support up to 17 data nodes. However, the total number of data nodes that a rack can accommodate can vary based on the number of top-of-rack switches and management nodes that are required for the rack within the overall solution design. The number of data nodes can be calculated by the following equation:

Maximum number data nodes = (42U - (# 1U Switches + # 1U Management Nodes)) / 2

Edge nodes: This calculation does not consider edge nodes. Based on the client’s choice of edge node, proportions can vary. Every two 1U edge nodes displace one data node, and every one 2U displaces one data node.

9.2 Increasing cluster performance To increase cluster performance, you can increase data node memory or use a high-performance job scheduler, such as IBM Platform Symphony, within the MapReduce framework. Often, improving performance comes at increased cost. Therefore, you must consider the cost/benefit trade-offs of designing for higher performance.

In the BigInsights predefined configuration, data node memory can be increased to 1024 GB by using sixteen 64GB RDIMMs.

The impact of MapReduce shuffle file and other temporary file I/O on data node performance can be workload dependent. In some cases, data node performance can be increased by utilizing solid-state disk (SSD) for MapReduce shuffle files and other temporary files. The Lenovo Big Data Reference Architecture for IBM BigInsights data nodes utilize the N2215 12 Gbps HBA. This HBA provides expanded bandwidth to exploit the performance-enhancing characteristics of placing MapReduce shuffle files and other temporary files on SSD. When considering the use of SSD, it is important to ensure consistency in SSD to HDD capacity proportions across all BigInsights cluster data nodes.

9.3 Designing for lower cost There are two key modifications that can be made to lower the cost of a BigInsights reference architecture solution. When lower-cost options are considered, it is important to ensure that customers understand the potential lower performance implications of a lower-cost design. A lower-cost version of the IBM BigInsights reference architecture can be achieved by using lower-cost node processors and lower-cost cluster data network infrastructure.

The node processors can be substituted with the Intel Xeon E5-2630 v4 2.2GHz 10-core. This processor requires 2133 MHz RDIMMs, which can also lower the per-node cost of the solution.

The use of a lower-cost network infrastructure can significantly lower the cost of the solution, but can also have a substantial negative effect on intra-cluster data throughput and cluster ingest rates. To use a lower-cost network infrastructure, use the following substitutions to the predefined configuration:

35 Lenovo Big Data Reference Architecture for IBM BigInsights

• Within each node, substitute the Broadcom 10GbE dual SFP+ network adapter with the extra ports on the integrated 1GBaseT adapters within the System x3650 M5 server.

• Within each rack, substitute the Lenovo RackSwitch G8272 top-of-rack switch with the Lenovo RackSwitch G8052.

• Within each cluster, substitute the Lenovo RackSwitch G8332 core switch with the Lenovo RackSwitch G8272.

Although the network wiring schema is the same as the schema that is described in “Networking”, the media types and link speeds within the data network changed. The data network within a rack that connects the cluster nodes to the lower-cost option G8052 top-of rack-switch is now based on two aggregated 1GBaseT links per node. The physical interconnect between the admin/management networks and the data networks within each rack is now based on two aggregated 1GBaseT links between the administrative or management network G8052 switch and the lower cost data network G8052 switch. Within a cluster, the racks are interconnected through two aggregated 10 GbE links between the substitute G8052 data network switch in each rack and a lower cost G8272 core switch.

9.4 Designing for high ingest rates Ingesting data into a BigInsights cluster is accomplished by using edge nodes that are connected to the cluster data network switches within each rack (Lenovo RackSwitch G8272). For more information about cluster networking, see “Networking” on page 25. For more information about edge nodes, see “Edge nodes” on page 23.

Designing for high ingest rates is difficult. It is important to have a full characterization of the ingest patterns and volumes. The following questions provide guidance to key factors that affect the rates:

• On what days and at what times are the source systems available or not available for ingest? • When a source system is available for ingest, what is the duration for which the system remains

available? • Do other factors affect the day, time, and duration ingest constraints? • When ingests occur, what is the average and maximum size of ingest that must be completed? • What factors affect ingest size? • What is the format of the source data (structured, semi-structured, or unstructured)? Are there any

data transformation or cleansing requirements that must be achieved during ingest?

To increase the data ingest rates, consider the following points:

• Ingest data with MapReduce job, which helps to distribute the I/O load to different nodes cross the cluster.

• Ingest during cluster load is not high, if possible. • Compressing data is a good option in many cases, which reduces the I/O load to disk and network. • Filter and reduce data in earlier stage saves more costs.

If the client is using or planning to use ETL software for ingest, such as IBM DataStage, consult the appropriate ETL specialist, such as an IBM DataStage architect, to help size the appropriate edge node configuration.

The key to successfully addressing a high ingest rate is to ensure that the number and physical attributes of edge nodes are sufficient for the throughput and processor needs for ingest and the ETL needs.

36 Lenovo Big Data Reference Architecture for IBM BigInsights

9.5 Estimating disk space When you are estimating disk space within a IBM BigInsights Hadoop cluster, consider the following points:

• For improved fault tolerance and performance, the Hadoop file system replicates data blocks across multiple cluster data nodes. By default, the file system maintains three replicas.

• Compression ratio is an important consideration in estimating disk space and can vary greatly based on file contents. If the customer’s data compression ratio is unavailable, assume a compression ratio of 2.5:1.

• To ensure efficient file system operation and to allow time to add more storage capacity to the cluster if necessary, reserve 25% of the total capacity of the cluster.

Assuming the default three replicas maintained by the Hadoop file system, the raw data disk space, and the required number of nodes can be estimated by using the following equations:

Total raw data disk space = (User data, uncompressed) * (4 / compression ratio)

Total required data nodes = (Total raw data disk space) / (Raw data disk per node)

You should also consider future growth requirements when estimating disk space.

Based on these sizing principals, Table 8 on page 37 shows an example for a cluster that must store 500 TB of uncompressed user data. The example shows that the IBM BigInsights cluster needs 800 TB of raw disk to support 500 TB of uncompressed data. The 800 TB is for data storage and does not include operating system disk space. A total of 15 nodes are required to support a deployment of this size.

37 Lenovo Big Data Reference Architecture for IBM BigInsights

Table 8. Example of storage sizing with 4TB drives

Description Value

Size of uncompressed user data 500 TB

Compression ratio 2.5:1

Size of compressed data 200 TB

Storage multiplication factor 4

Raw data disk space needed for IBM BigInsights cluster 800 TB

Storage needed for IBM BigInsights Hadoop 3x replication 600 TB

Reserved storage for headroom 200 TB

Raw data disk per node (with 4 TB drives) 56 TB

Minimum number of nodes required (800/56) 15

Note: A 2.5:1 compression ratio is an estimate based on measurements taken in a controlled environment. Compression results vary based on data and compression libraries used. IBM cannot guarantee compression results or compressed data storage amounts. Improved estimates can be calculated by testing customer data using appropriate compression libraries.

9.6 Scaling considerations The Hadoop architecture is linearly scalable. When the capacity of the existing infrastructure is reached, the cluster can be scaled out by adding more data nodes and, if necessary, management nodes. As the capacity of existing racks is reached, new racks can be added to the cluster. Some workloads might not scale linearly.

When you design a new BigInsights reference architecture implementation, future scale out is a key consideration in the initial design. You must consider the two key aspects of networking and management. Both of these aspects are critical to cluster operation and become more complex as the cluster infrastructure grows.

The networking model that is described in the section “Networking” on page 25 is designed to provide robust network interconnection of racks within the cluster. As more racks are added, the predefined networking topology remains balanced and symmetrical. If there are plans to scale the cluster beyond one rack, initially design the cluster with multiple racks, even if the initial number of nodes might fit within one rack. Starting with multiple racks will enforce proper network topology and prevent future reconfiguration and hardware changes.

Also, as the number of nodes within the cluster increases, many of the tasks of managing the cluster also increase, such as updating node firmware or operating systems. Building a cluster management framework as part of the initial design and proactively considering the challenges of managing a large cluster will pay off significantly in the end.

Proactive planning for future scale out and the development of cluster management framework as a part of

38 Lenovo Big Data Reference Architecture for IBM BigInsights

initial cluster design provides a foundation for future growth that will minimize hardware reconfigurations and cluster management issues as the cluster grows.

9.7 High availability considerations When a IBM BigInsights cluster on System x is implemented, consider availability requirements as part of the final hardware and software configuration. Typically, Hadoop is considered a highly reliable solution. Hadoop and IBM BigInsights best practices provide significant protection against data loss. Generally, failures can be managed without causing an outage. There is redundancy that can be added to make a cluster even more reliable. Some consideration must be given to hardware and software redundancy.

9.7.1 Networking considerations Optionally, a second redundant switch can be added to ensure HA of the hardware management network. The hardware management network does not affect the availability of the Hadoop file system functionality, but it might affect the management of the cluster; therefore, availability requirements must be considered.

To support HA in the network, link aggregation is used between the 10 Gb ports of a server network adapter and the top-of-rack switch. Virtual Aggregation Groups (vLAG) can be used between redundant switches.

If 1 Gbps data network links are used, it is recommended that more than one link is used per node to increase throughput.

9.7.2 Hardware availability considerations The redundancy of each individual data node is not necessary with IBM BigInsights . Hadoop 3x replication provides built-in redundancy and makes loss of data unlikely. If Hadoop best practices are used, an outage from a data node loss is extremely unlikely as the workload can be dynamically re-allocated. The loss of a data node cannot stop workload processing; workload is automatically re-allocated to another data note.

Multiple management nodes are recommended so that if there is a failure, function can be moved to an operational management node. Having multiple management nodes does not resolve the issue of the NameNode being a single point of failure. For more information, see “Software availability considerations”.

Within racks, switches and nodes must have redundant power feeds with each power feed connected from a separate PDU.

9.7.3 Storage availability If default 3x replication is not sufficient for availability requirements, RAID is another option for insuring against data loss. HDFS disks can be set up with RAID, but performance is negatively affected. In most cases, HDFS 3x replication provides more than sufficient protection. Also, IBM General Parallel File System (GPFS™), included within BigInsights 4.1 Enterprise Management, overcomes the potential for NameNode failures by not depending on stand-alone services to manage the distributed file system metadata. GPFS also has the added benefits of being POSIX-compliant, having more robust tools for online management of underlying storage, point-in-time snapshot capabilities, and off-site replication.

39 Lenovo Big Data Reference Architecture for IBM BigInsights

9.7.4 Software availability considerations Operating system availability is provided by using mirrored drives for the operating system.

NameNode HA is recommended and can be achieved by using three management nodes. Active and standby nodes communicate with a group of separate daemons called JournalNodes to keep their state synchronized. When any namespace modification is performed by the active NameNode, it durably logs a record of the modification to most of these JournalNodes. The standby NameNode can read the edits from the JournalNodes and is constantly watching them for changes to the edit log. As the standby Node sees the edits, it applies them to its own namespace.

40 Lenovo Big Data Reference Architecture for IBM BigInsights

10 Acknowledgements This reference architecture document has benefited very much from the detailed and careful review comments provided by Brian Finley and Ron Kunkel of Lenovo.

41 Lenovo Big Data Reference Architecture for IBM BigInsights

11 Resources For more information, see the following resources:

• Lenovo System x3650 M5 (IBM BigInsights Data Node): o Product page: shop.lenovo.com/us/en/systems/servers/racks/systemx/x3650-m5/ o Lenovo Press product guide: Lenovo System x3650 M5 V4

• Lenovo System x3550 M5 (IBM BigInsights Management Node): o Product page: shop.lenovo.com/us/en/systems/servers/racks/systemx/x3550-m5/ o Lenovo Press product guide: Lenovo System x3550 M5 V4

• Lenovo RackSwitch G8052 (1GbE Switch): o Product page: shop.lenovo.com/us/en/systems/browsebuy/%20rackswitch-g8052.html o Lenovo Press product guide: lenovopress.com/tips0813

• Lenovo RackSwitch G8272 (10GbE Switch): o Product page: http://shop.lenovo.com/us/en/systems/networking/ethernet-rackswitch/g8272/ o Lenovo Press product guide: lenovopress.com/tips1267

• Lenovo RackSwitch G8332 (40GbE Switch): o Product page: http://shop.lenovo.com/us/en/systems/networking/ethernet-rackswitch/g8332/ o Lenovo Press product guide: Lenovo RackSwitch G8332

• Lenovo XClarity Administrator: o Product page: http://shop.lenovo.com/us/en/systems/software/systems-management/xclarity/ o Lenovo Press product guide: lenovopress.com/tips1200

• IBM: o IBM BigInsights 4.1

• IBM Analytics Internet: http://www.ibm.com/software/data/infosphere/biginsights

• IBM Knowledge Center: http://www-01.ibm.com/support/knowledgecenter/SSPT3X_4.1.0/com.ibm.swg.im.infosphere.biginsights.welcome.doc/doc/welcome.html

• IBM BigInsights Twitter: https://twitter.com/Hadoop_Dev

• IBM HadoopDev blog posts: https://developer.ibm.com/hadoop/blog/2015/08/25/ibm-presents-an-all-new-open-platform-for-apache-hadoop-on-intel-and-power-platforms-and-biginsights-4-1-on-intel/

o IBM Spectrum Scale (formerly GPFS)

• IBM Internet: http://www-03.ibm.com/systems/storage/spectrum/scale/index.html

• IBM Information Center : http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.infocenter.doc/infocenter.html

42 Lenovo Big Data Reference Architecture for IBM BigInsights

o IBM Platform Computing

• IBM Internet: http://www.ibm.com/systems/technicalcomputing/platformcomputing/index.html

• Enhancing IBM BigInsights with IBM Platform Computing and GPFS http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=SA&subtype=WH&htmlfid=DCW03045USEN

• Open source software: o Hadoop: hadoop.apache.org o Spark: http://spark.apache.org o Flume: flume.apache.org o HBase: hbase.apache.org o Hive: hive.apache.org o Hue: gethue.com o Impala: rideimpala.com o Oozie: oozie.apache.org o Mahout: mahout.apache.org o Pig: pig.apache.org o Sentry: sentry.incubator.apache.org o Sqoop: sqoop.apache.org o Whirr: whirr.apache.org o ZooKeeper: zookeeper.apache.org o Parquet: parquet.apache.org

• xCat: xcat.sourceforge.net

43 Lenovo Big Data Reference Architecture for IBM BigInsights

12 Document History Versoin Date Notes

1.0 11/2/2015 Initial release version

1.1 1/20/2017 Update for x3550/x3650 M5 with Intel Xeon E5 v4 CPU

1.2 2/15/2017 Updated management node OS/local HDD count in table 2

44 Lenovo Big Data Reference Architecture for IBM BigInsights

Trademarks and special notices © Copyright Lenovo 2015.

References in this document to Lenovo products or services do not imply that Lenovo intends to make them available in every country.

Lenovo, the Lenovo logo, ThinkCentre, ThinkVision, ThinkVantage, ThinkPlus and Rescue and Recovery are trademarks of Lenovo.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others.

Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used Lenovo products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-Lenovo products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by Lenovo. Sources for non-Lenovo list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. Lenovo has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-Lenovo products. Questions on the capability of non-Lenovo products should be addressed to the supplier of those products.

All statements regarding Lenovo future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local Lenovo office or Lenovo authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in Lenovo product announcements. The information is presented here to communicate Lenovo’s current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard Lenovo benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models.

Any references in this information to non-Lenovo websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this Lenovo product and use of those websites is at your own risk.