© 2009 VMware Inc. All rights reserved
Hadoop as a Service
Hadoop on Virtualization Hadoop World, December 2011 Jun Ping Du Richard McDougall VMware, Inc.
2
Cloud: Big Shifts in Simplification and Optimization
2. Dramatically Lower Costs
to redirect investment into
value-add opportunities
3. Enable Flexible, Agile IT Service Delivery
to meet and anticipate the
needs of the business
1. Reduce the Complexity
to simplify operations and maintenance
3
Infrastructure, Apps and now Data…
Private Public
Build Run
Manage
Simplify Infrastructure With Cloud
Simplify App Platform Through PaaS Next Trend:
Simplify Data
4
Trend 1/3: New Data Growing at 60% Y/Y
Source: The Information Explosion , 2009
medical(imaging,(sensors(
cad/cam,(appliances,(videoconfercing,(digital(movies(
digital(photos(
digital(tv(
audio(
camera(phones,(rfid(
satellite(images,(games,(scanners,(twi8er(
Exabytes of information stored 20 Zetta by 2015 1 Yotta by 2030 Yes, you are part of the yotta generation…
5
Trend 2/3: Big Data – Driven by Real-World Benefit
6
Trend 3/3: Value from Data Exceeds Hardware Cost
! Value from the intelligence of data analytics now outstrips the cost of hardware • Hadoop enables the use of lower cost hardware
• Hardware cost halving every 18mo
Big Iron: $40k/CPU
Commodity Cluster: $1k/CPU
Value
Cost
7
SQLCluster
Unified Big Data Infrastructure
Hadoop Cluster
Private Public
Big SQL
Three Big Reasons to Virtualize Hadoop: 1. Simplify Hardware
Hadoop NoSQL
DSS Cluster
NoSQL Cluster
! Trend is “not just hadoop” for big data • Hadoop is often combined with other
technologies: Big SQL, NoSQL etc,… • Unify the infrastructure platform for all
! Common Hardware Base • Eliminate the hardware/driver/testing phase • Use existing team for ordering, diagnosis,
capacity management of hardware farm
8
Three Big Reasons to Virtualize Hadoop: 2. Rapid Provisioning
I WANT MY HADOOP CLUSTER NOW!
! Instant Cluster Provisioning • Provision Hadoop Clusters instantly • Automatable using provisioning
engines/scripts: e.g. whir
9
Three Big Reasons to Virtualize Hadoop: 3. Leverage Capabilities
! Increase Utilization • Hadoop cluster only uses resources it needs
• Extra resources can be used by other applications when not in use
! Eliminate single points of failure • Use vSphere HA for Namenode and Jobtracker
! Use VM Isolation • Create separate clusters with defensible security
• Enables multiple-versions of Hadoop on the same infrastructure
• Extends to Hadoop and Linux Environments
! Leverage Resource Management • Control/assign resources through resource pools
• E.g. Use spare cycles for Hadoop Processing through priority control
10
What? Hadoop in a VM? Really?
Actually, Hadoop performs well in a virtual machine
11
Performance Test: Cluster Configuration
AMAX ClusterMax 2X X5650, 96 GB 12X SATA 500 GB Mellanox 10 GbE adapter
Mellanox10 GbE switch
12
Cluster Configuration ! Hardware • AMAX ClusterMax, 7 nodes
• 2X X5650 2.67 GHz hex-core, 96 GB memory
• 12X SATA 500 GB 7200 RPM (10 for Hadoop data), EXT4
• Mellanox ConnectX VPI (MT26418), 10 GbE
• Mellanox Vantage 6048, 10 GbE
! OS/Hypervisor • RHEL 6.1 x86_64 (native and guest)
• ESX 5.0 RTM with devel Mellanox driver
! VMs (HT off/on) • 1 VM: 92000 MB, (12/24) vCPUs, 10 PRDM disks
• 2 VMs: 46000 MB, (6/12) vCPUs, 5 PRDM disks
• 4 VMs (HT on only): • 2 small: 18400 MB, 5 vCPUs, 2 disks • 2 large: 27600 MB, 7 vCPUs, 3 disks
13
Hadoop Configuration Distribution • Cloudera CDH3u0
• Based on Apache open-source 0.20.2
Parameters • dfs.datanode.max.xcievers=4096
• dfs.replication=2
• dfs.block.size=134217728
• io.file.buffer.size=131072
• mapred.child.java.opts=”-Xmx2048m -Xmn512m” (native)
• mapred.child.java.opts=”-Xmx1900m -Xmn512m” (virtual)
! Network topology • Hadoop uses info for reliability and performance
• Multiple VMs/host: Each host is a “rack”
14
Benchmarks ! Derived from test apps included in distro ! Pi • Direct-exec Monte-Carlo estimation of pi
• # map tasks = # logical processors
• 1.68 T samples
! TestDFSIO • Streaming write and read
• 1 TB
• More tasks than processors
! Terasort • 3 phases: teragen, terasort, teravalidate
• 10B or 35B records, each 100 Bytes (1 TB, 3.5 TB)
• More tasks than processors
• CPU, networking, and storage I/O
π ~ 4*R/(R+G) = 22/7
15
Performance of Hadoop for Several Workloads
0
0.2
0.4
0.6
0.8
1
1.2 R
atio
to N
ativ
e
1 VM
2 VMs
Ratio of time taken – Lower is Better
16
Architecting Hadoop as a Service using Virtualization
! Goals • Make it fast and easy to provision new Hadoop Clusters on Demand
• Leverage virtual machines to provide isolation (esp. for Multi-tenant)
• Optimize Hadoop’s performance based on virtual topologies
• Make the system reliable based on virtual topologies
! Leveraging Virtualization • Elastic scale in/out
• Use high-availability to protect namenode/job tracker
• Resource controls and sharing: re-use underutilized memory, cpu
• Prioritize Workloads: limit or guarantee resource usage in a mixed environment
17
Provisioning
! Leverage the vSphere APIs to auto-deploy a cluster • Whirr, HOD, or custom using ruby, chef, etc,…
! Use linked-clones to rapidly fork many nodes
18
Fast Provisioning
! From a “seed” node to a cluster
Thin Provisioning Linked Clone
60GB => 3.5GB� ~6 second�
19
SAN, NAS or Local Disk?
! Shared Storage: SAN or NAS • Easy to provision
• Automated cluster rebalancing
! Hybrid Storage • SAN for boot images, VMs, other
workloads • Local disk for HDFS
• Scalable Bandwidth, Lower Cost/GB
Host
Had
oop
Oth
er V
M
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
Host
Had
oop
Oth
er V
M
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
Host
Had
oop
Had
oop
Oth
er V
M
20
Enable Automatic Rack awareness through vSphere
! Important to robust hadoop cluster
! Automatic network topology detect — an important vSphere feature
! Rack script is generated automatically
21
Multi-tenant: share cluster or not
! Shared big cluster VS. Isolated small clusters
High performance Large scale
Pre-job provisioning
Secure Flexible
Post-job provisioning
Combination – as customers’ requirement are different
22
Elastic Hadoop Cluster�
! Traditional hadoop cluster • Easy to scale out
• Fast-provision new hadoop nodes and join into existing cluster
• Hard to scale in
! Elastic hadoop cluster
While (ClusterIsTooLarge) {
choose node k;
kill (node k);
wait (k’s data block is recovered);
if necessary, hadoop.rebalance();
}
NN JT
…
…
Elastic node
Normal node
TaskTracker
DataNode
23
Replica Placement�
! Second Replica • Different rack
• Rack-awareness required
! Third Replica • Same rack, different physical host
• Nodes share host (in virtualized
environment)
24
Demo
25
Performance
! Create more smaller VMs • Makes Hadoop scale better
• Allows for easier/faster adjustment of packing of VMs across hosts by vSphere (including through DRS)
! Sizing/Configuration of storage is critical • Plan on ~50Mbytes/sec of bandwidth per core
• SANs are typically configured by default for IOPS, not Bandwidth
• Ensure SAN ports/switch topology allows required aggregate bandwidth
• Performance of the backend storage should be tested/sized
• Local disks will give ~100-140MBytes/sec per disk: pick correct controller
26
Summary
! Hadoop does work well in a virtual environment ! Plan a virtual cluster, enable other big-data solutions on the same
infrastructure ! Leverage the recipes to automate your configuration and
deployment