NoSQL DB Benchmarking with high performance Networking solutions

© 2013 Mellanox Technologies 1

NoSQL DB Benchmarking with high performance Networking solutions

WBDB, Xian, July 2013

© 2013 Mellanox Technologies 2

Leading Supplier of End-to-End Interconnect Solutions

Host/Fabric SoftwareICs Switches/GatewaysAdapter Cards Cables

Comprehensive End-to-End InfiniBand and Ethernet Portfolio

Virtual Protocol Interconnect

StorageFront / Back-EndServer / Compute Switch / Gateway

56G IB & FCoIB 56G InfiniBand10/40/56GbE & FCoE 10/40/56GbE

Fibre Channel

Virtual Protocol Interconnect

© 2013 Mellanox Technologies 3

Motivation to Accelerate Data Analytics

Data Analysis Requires Faster Network• Hadoop Map Reduce Framework is a network

intensive workload- Mapped data is shuffled between nodes in the cluster

• Data Replication - A high availability event triggers Multi-Tera of data

movement

Provide Higher Data Value• Expose SSD’s low latency capabilities• Better server/CPU utilization

* Data Source: Intersect360 Research, 2012, IT and Data scientists survey

Big Data Applications Require High Bandwidth and Low Latency Interconnect

© 2013 Mellanox Technologies 4

Cassandra Database enables update capabilities Latency factors

• Commit-log settings• Workload

Cassandra, Update Latency

© 2013 Mellanox Technologies 5

Cassandra Database Read Latency factors

• Media used• Workload

Cassandra, Read Latency

© 2013 Mellanox Technologies 6

5 Nodes in the Ring 64GB RAM

• 8 x 8GB DDR3 1333MHz 2 x E5-2670

• 8 Cores per socket 5 x Seagate® Constellation® ES SATA 6Gb/s 2TB Hard Drive

• 7200 RPM NIC: Mellanox Technologies MT27500 Family [ConnectX-3]

• 10Gb Ethernet• FW_VER=2.11.500

Switch SX1036 OS: RH 6.3

• MLNX_OFED_LINUX-1.5.3 Apache Cassandra 1.1.12, 2 seeds

System Used for Cassandra Benchmark

© 2013 Mellanox Technologies 7

SSDs Become De-Facto standard in HDFS deployment• Read capability is a critical factor for application performance

E-DFSIO, Part of Intel’s HiBench test suite, profiles aggregated throughput on the cluster• 1GbE network impede any performance benefit from SSD deployment

Unlocking the Power of SSDs In Hadoop Environment

E-DFSIO, Showing the Power of SSD @ HDFS

© 2013 Mellanox Technologies 8

Updates are made to server memory• Extreme low latency for HBase- Java GC policy hurting on large throughput

HBase Benchmarking, Update Latency

© 2013 Mellanox Technologies 9

Hitting the media capabilities

HBase Benchmarking, Read Latency

© 2013 Mellanox Technologies 10

4 Region servers, 1 Master, 3 Zookeeper quorum servers 64GB RAM

• 8 x 8GB DDR3 1333MHz 2 x E5-2670

• 8 Cores per socket 5 x Seagate® Constellation® ES SATA 6Gb/s 2TB Hard Drive

• 7200 RPM NIC: Mellanox Technologies MT27500 Family [ConnectX-3]

• 10Gb Ethernet• FW_VER=2.11.500

Switch SX1036 OS: RH 6.3

• MLNX_OFED_LINUX-1.5.3 Apache Hbase 0.94.9, Zookeeper 3.4.5, Apache Hadoop 1.1.2

System Used for HBase Benchmarks

© 2013 Mellanox Technologies 11

EMC 1000-Node Analytic Platform Accelerates Industry's Hadoop Development 24 PetaByte of physical storage Mellanox VPI Solutions

Test Drive Your Big Data

2X Faster Hadoop Job Run-TimeHadoopAcceleration

High Throughput, Low Latency, RDMA Critical for ROI

© 2013 Mellanox Technologies 12

The Great Things in Hadoop Distributed File System

• HDFS is a block storage solution• Block size can be modified to provide efficient solutions for very large files• Inherent reliability, no need for high end storage solution to make sure data is there!• Tuned for Hadoop work loads, write one and read many

© 2013 Mellanox Technologies 13

The Less Great Things in HDFS

It’s hard to manage the different settingto get the right nodes into the right capabilities.

Ingress and extraction of data requires additional tools.

Small files or latency sensitiveDefault 3x ReplicationMetadata Server Failure

© 2013 Mellanox Technologies 14

Local Disks – The Common Practice

© 2013 Mellanox Technologies 15

Other Distributed Storage Solution for Hadoop, Really?!

© 2013 Mellanox Technologies 16

OrangeFS as Hadoop Storage Solution

© 2013 Mellanox Technologies 17

Lustre as Hadoop Storage Solution

Source: Map/Reduce on Lustre, Hadoop Performance in HPC Environments, Nathan Rutman, Senior Architect, Networked Storage Solutions, Xyratex

© 2013 Mellanox Technologies 18

CEPH as Hadoop Storage Solution

Generating lot of Interest since the Ceph kernel client was pulled into Linux kernel 2.6.34• Object-based parallel file system• Scalable metadata server• Each file can specify it’s own striping strategy and object size• Automatic rebalancing of data with minimal data movement• Hadoop module for integrating Ceph has been in development since 0.12 release

Benchmarks on Ceph is still WIP• We are currently working on using running benchmarks on Ceph – Stay tuned!!

© 2013 Mellanox Technologies 19

Thank You