© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 1

Aerospike aer . o . spike [air-oh- spahyk] noun, 1. tip of a rocket that enhances speed and stability

IN-MEMORY NOSQL, Now OPEN SOURCE!

1MTPS ON AMAZON @ $1.68/HR

RAJKUMAR IYER

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 2

AGENDA

■Share knowledge we gained while trying to get best out of Amazon running Aerospike database.

■List out steps and Demonstrate 1 Million TPS on a single instance applying the above principle in Amazon

■Keep it Interactive

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 3

INSTANCE TYPES

Instance Name Comment

R3.* Memory Intensive

C3.* Compute Intensive

M3.* Balance between R and C

T* A very low end beginner machine

M1.* Previous Generation. Low network.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 4

NETWORK TYPES

Instance Type Network Type

*.large, *.xlarge Moderate

*.2xlarge Moderate - High

*.4xlarge High

*.8xlarge 10 Gbps

Moderate - Upper bound of 100MBps)

High - Ranges from 300Mbps to 1.8Gbps

10 Gigabit   - Upper bound of 8.8 Gbps)

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 5

OLD GENERATION INSTANCES

■m1.* and t1.* has low to moderate network.

■Are being retired.

■Reliability Guarantee are lowest of all.

■Can be used for developer activity.

■Not for running high performance service.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 6

NEW GENERATION INSTANCES

■C3.* R3.* and I3.* has Moderate to 10 Gbps network Capability

■*.4xlarge and above instances are shared with lot less other instances.

■Reliability Guarantee is on higher side. Barring Availability zone failure.

■Can be used for Real Service.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 7

VIRTUAL MACHINE TYPE

■PV *.8xlarge though rated at 10Gbps they upper bounded at 2.2 Gpbs (iperf result)

■PV overhead at high load is around 40% of entire CPU utilization.

■HVM has much better CPU utilization and network utilization.

■HVM nearly outperform PV instance by 3x. On apple-to-apple comparison of C3.8xlarge HVM gave 215ktps vs 85ktps on PV.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 8

NETWORK PACKET PROCESSING

■Soft Interrupt Processing (% si in top) is one of the largest CPU drains in the client-server network based system.

■In AWS a NIC is 2 queues as exposed. Interrupt for which can be only be serviced on two cores. This causes core bottleneck at high throughput rates.

■Single NIC peaks at 250ktps processing. Trick is to use multiple NICs

■Interrupt coalescing improves CPU utilization a little but did not translate into better processing.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 9

NETWORK PACKET PROCESSING

■Receive Packet Steering to spread IRQ processing over multiple cores. Single NIC with RPS enabled can push up to 800ktps with interrupts spread over 8 cores.echo ff | sudo tee /sys/class/net/eth0/queues/rx-[0-1]/rps_cpus

■ENIs provides a way of adding multiple (virtual) NICs to an instance. IRQ processing for each spread over different core is optimal.echo 1 | sudo tee /proc/irq/<irq number>/smp_affinity

■Using ENIs with private IPs is free of cost.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 10

MISCELLANEOUS

■Having instances in the same PLACEMENT GROUP lets instances have low-latency, full bisection 10 Gbps network.

■Instances have to be in a VPC to be in same placement group

■Shared vs Dedicated tenancy in the new generation instances did not seem to matter as we experimented.

■To avoid CPU stealing, avoid over user of CPU. Typically running with large number of threads.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 11

SETUP INSTANCES

■Setup Security group with following ports open.3000-3003 [Internode communication]8081 [Aerospike Monitoring Console]22 [ssh to the host ]

■Setup c3.8xlarge HVM instances in a VPC. Set it up with Placement Group

■Attach 3 additional ENI to the server instance. Setup RPS on each one of them.

■Setup 5 c3.2xlarge instances to run as a client. Put them in same VPC and Placement group as server instance.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 12

SETUP AEROSPIKE SERVER

■On server instance. Run afterburner.sh (cd /opt/aerospike/bin; sudo ./afterburner.sh) to set the optimal number of threads on each Aerospike server.

■Configure the in-memory namespace to use 32GB of RAM (/etc/aerospike/aerospike.conf)

■Start the Aerospike Serversudo service aerospike start

■Start the Aerospike Monitoring Console on one of the client instances.sudo service amc start

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 13

FIRE THE LOAD

■Load the server with data using Java benchmark client.

cd <java client>/benchmarks./run_benchmarks -z 64 -n test -w I  -o S:10 -b 10 -l 23 -k 10000000 -latency 5,1 -h YOUR_AWS_INTERNAL_IP

■Run the clients with 100% read load. Point each client to a separate private IP on different ENIs.

cd <java client>/benchmarks./run_benchmarks -z 64 -n test -w RU,100  -o S:10 -b 10 -l 23 -k 10000000 -latency 5,1 -h YOUR_AWS_INTERNAL_IP

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 14

RESULT

4 interfaces with proper irq affinity and 8 client processes on 4 client instances, we were able to get up to 960ktps on a single c3.8xlarge server instance.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 15

CLUSTER SETUP

■4 node Aerospike cluster with data in memory (RAM)

■40 Million unique objects in cluster.

■100 bytes record size; 10 bins of 10 bytes each.

■Replication Factor 2 with synchronous replication for immediate consistency.

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 16

INSTANCES

Instance Type Comment

m1.large Previous Generation

m3.xlarge Current Generation without Enhanced networking

r3.2xlarge Current Generation with Enhanced Networking

r3.large Current Generation low end

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 17

NODES vs TPS

© 2014 Aerospike, Inc. All rights reserved. Confidential. | 1Mtps On Amazon @ $1.68/hr– September 4, 2014 | 18

COST ANALYSIS