SEIZE THE DATA. 2015h41382. · Edit Bios (Set HP Power Profile to Maximum Performance): System Configuration > BIOS/Platform Configuration (RBSU) > Power Management > HP Power Profile

© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.1 SEIZE THE DATA. 2015

SEIZE THE DATA. 2015


Sizing and Deploying Vertica ClustersIn both Virtual and Physical environments

Alex Jackson & Sumeet Keswani / August 11, 2015

SEIZE THE DATA. 2015Agenda

Hardware validation

Cookbooks

Cloud + virtualized best practices

Hardware tuning

What’s next


Hardware validationV*Perf Testing Utilities


VCPUPerf

Good Bad


VIOPerf**Make sure you run on data directory**

#/opt/vertica/bin/vioperf /data


VNetPerf


Hardware configuration guidesDL380 Gen9 & DL360 Gen9


Large clusters

HP DL380 24sff Gen9

Tried and true building block

• Proven in the field as the best platform for HP Vertica

• Designed for large & high performance clusters

• Scales from TB to PB without issue

• 24x 10k RPM drives (+2 for OS/Catalog)

• 256-512G RAM

• 20 – 28 cores at 2.4-2.8 GHz

• 10G Network (Bonded to 20+G)


Small clusters

HP DL360 8sff Gen9

Under 20T high performance workloads

• Designed for smaller workloads where DL380 is overkill

• 3 nodes cluster holds 1-10T

• Scalable to 100s of T

• 16 cores at 3.2GHz

• 128 – 256G RAM

• 10x 15k rpm drives

• 10G network


Cloud + Virtualized Best PracticesAWS & WMWare


HP Vertica on AWS/public clouds

• AMI on AWS marketplace (license : BYOL)

• Uses Red Hat Enterprise Linux 7.0 Update 1

(runs better on virtualization/cloud, significant performance improvements on the previous AMI )

• Always use Cluster Placement Groups

(guarantees locality of nodes)

• Use enhanced networking (SRIOV) (10 G networking)

• Vertica control network uses udp pt-to-pt

• i2, c3, c4, r3 (8x and 4x) instance types

(use EBS optimized)

• Use additional attached EBS volumes for data.

(ext4 file system, RAID-0 for performance)

AWS Marketplace AMI


Typical users/uses of Vertica in AWS

• Deploy quickly

• AWS is used for proof of concept, before people purchase hardware

• Understanding scale and sizing for typical workloads

• Typical production deployments are 3 to 8 node clusters (5TB to 30TB). few as large clusters as 100 nodes 300 TB (compressed)

• Most use dedicated instances for cost (and performance) reasons

• Evaluate your workloads to see if they are appropriate for AWS or Virtualization

• A high concurrency, low latency workload is not ideal for AWS and/or VM/Clouds


Typical challenges on AWS

• Network (UDP reliability). Use enhanced networking (recommended)

• Application connectivity to/through the Cloud can also be problematic or unreliable

• Disk I/O can be typically on the low end. Configure as RAID-0 with multiple volumes

• We have seen kernel performance issues with respect to interaction of the linux kernel with the hypervisor. Newer kernels work much better since there is better integration with hypervisors

• Depending on user sophistication, there is some management overhead with respect to backup, disaster recovery etc.

SR-IOV (10G) | MB/s (sent) | MB/s (rec)

-------------------|-------------|-------------

udp-throughput | 913.157 | 884.541

tcp-throughput | 981.079 | 980.484

8 Vol(RAID-0)| counter name ----------|-----------------------------Write | 990 MB/sReWrite | 474+474(MB-read+MB-write)/sRead | 1040 MB/sSkipRead | 23215 seeks/s


HP Vertica on virtualized infrastructure/private clouds

• We have started to provide guidance on how to use HP Vertica on “private clouds”. We started with VMWare since that is one of the most often used platforms. Fewer users use OpenStack

• We recommend aggressive monitoring of instances/nodes and “baselining” resource performance (I/O , Network) when using Virtualized infrastructure

• Typically tend to be 3 to 8 node clusters (10TB to 30TB clusters)

• Biggest challenges relate to over-provisioning the virtualized/shared infrastructure. espNetwork or CPU starvation

• Users that have SAN storage (not recommended) with Virtualized Infrastructure typically see high I/O wait and their cluster is typically disk I/O bound. Some sophisticated users use dedicated SAN, and PV-SCSI to get the most out of their SAN and Vertica Cluster.


Configuration recommendations

HP Vertica on virtualized infrastructure/private clouds

• Expectations: Performance on VM cannot be same as that of similarly sized bare metal. There are additional layers for memory, I/O and CPU which consume time (latency), resources

• We recommend using Thick provisioned volumes, “Independent and Persistent mode”. All writes go directly to disk

• Do not rely on snapshots as a backup. Don't run the cluster from snapshot volumes

• Reserve/Pre-Allocate Guest Memory. Disable swapping of VM memory and Don't use memory ballooning

• VMotion or similar live movement of VMs within your Virtual Data Center is not recommended

• An IOP generated in a VM must pass thru extra layers of processing that enable all useful features of Virtualization. This can impact query performance. Configure your storage for low latency. Align your VMFS partitions

• Vertica K-safety in VM environments does not protect against underlying hardware failure (only protects against VM failure)

− You must rely on your virtual data centers features for this redundancy

− Use Vertica Fault Groups feature to express your underlying hardware configuration to get some level of High Availability


Hardware tuningCPU, network, operating system, memory & storage


CPU tuning

Disable CPU scaling

On dl380 Gen9:Edit Bios (Set HP Power Profile to Maximum Performance):

System Configuration > BIOS/Platform Configuration (RBSU) > Power Management > HP Power Profile > [Maximum Performance]

Edit /etc/grub.conf file, add the following parameters to the kernel line

On other systems:Disable all C-States, Sleep-States, CPU Package States, and set any power options to High Performance


Network tuning


Operating System tuning

Memory and storage tuning

Set deadline as IO Scheduler

Disable Hugepages

Set Readahead on data array(s) to 8192 (for dl380)

Set Readahead on all other array(s) to 2048


What’s next


What’s next

High Density Cookbook

15+ LFF drives, 2 E5-26XXv3 CPUs, 256-512G RAM

Vertica for SQL on Hadoop Cookbook

Compute Nodes - 1 or 2 E5-26XXv3 CPU, 128G-256G RAM

Storage Nodes (HDFS) – 20+ LFF High Density drives


Q&A


Documents

SEIZE THE DATA. 2015h41382. · Edit Bios (Set HP Power Profile to Maximum Performance): System Configuration > BIOS/Platform Configuration (RBSU) > Power Management > HP Power Profile