Bcache and Aerospike

Bcacheand

Aerospike

LSPEDec 20, 2014

Anshu PrateekDevops, Aerospike

Agenda

● Bcache● What is bcache?

– Features

● Why bcache?● How to use bcache?● Why bcache with Aerospike?

● Anshu Prateek● Aerospike Devops Lead● Ex - Yahoo! Search Operations● http://about.me/anshuprateek● anshu@aerospike.com

bcache

● Block – cache● cache in the Linux kernel's block layer● Starting Kernel >=3.10 (June 2013)

bcache?

● Kent Overstreet● http://bcache.evilpiepirate.org● https://github.com/g2p/bcache-tools.git● GNU General Public License (GPL)

What is bcache?

● creates hybrid volumes and provides performance improvements

● one or more fast storage devices, such as flash-based solid-state drives (SSDs), to act as a cache for one or more slower storage devices, such as hard disk drives

What is bcache?

Caching methods

● Write-through [default]● Write-pass● Write-around

Features

● A single cache device can be used to cache an arbitrary number of backing devices, and backing devices can be attached and detached at runtime, while mounted and in use (they run in passthrough mode when they don't have a cache).

● (But not multiple caching device for a single backing device)

Features

● Recovers from unclean shutdown - writes are not completed until the cache is consistent with respect to the backing device (Internally, bcache doesn't distinguish between clean and unclean shutdown).

● Detects and bypasses sequential IO (with a configurable threshold, and can be disabled).

Features

● Throttles traffic to the SSD if it becomes congested, detected by latency to the SSD exceeding a configurable threshold (useful if you've got one SSD for many disks).

● Readahead on cache miss (disabled by default).

Features

● Highly efficient writeback implementation; dirty data is always written out in sorted order, and if writeback_percent is enabled background writeback is smoothly throttled with a PD controller to keep around that percentage of the cache dirty.

● Very high performance b+ tree - bcache is capable of around 1M iops on random reads, if your hardware is fast enough.

Why bcache?

● BIG HDD– But slow

● Fast SSD– Smaller, comparatively costlier..

– But cheaper than RAM!

Why bcache?

● Cloud providers– AWS – Direct attached ephemeral SSD

– GCE – Local attached SSD (ephemeral)

– Persistent● AWS EBS● GCE SSD

Why bcache?

● Persistence (in cloud)

- Is network attached

- hence, slower compared to direct attached● Desktop / laptop / servers

Why bcache?

● Performance of direct attached (ephemeral)+

● Persistence of storage

How to..?

● Kernel >=3.10– Fedora, Ubuntu, Amazon Linux

● SSD– Direct attached is better

● Bigger/slower HDD/Network SSD

How to..?

● Install bcache tools● Unmount caching and backing device● Wipe existing FS (backup!)

– wipefs -a /dev/xvdb

● make-bcache -B /dev/xvdf -C /dev/xvdb● Done!

How to..?

● /sys/fs/bcache● /sys/block/bcache0

How to w/ Aerospike?

● Caching Mode

Default Write through

● Caching device– 2 or more

● Backing device– Same size as caching device

How to w/ Aerospike?

● Tuning● By default, bcache does not cache sequential

IO, just the random reads and writes that SSDs excel at. For Aerospike, we recommend enabling caching of sequential IO as well.

● Disable congestion throttling on bcache to avoid cache misses.

How to w/ Aerospike?

● We recommend using a one to one mapping between backing and caching devices to avoid cache misses. Avoid configuring a single caching device with multiple backing devices.

● Though bcache documentation shows way of creating and attaching caching and backing devices in a single command (make-bcache -B /dev/backing -C /dev/caching), we have seen problems when using this single command in cloud providers due to known bugs. Hence, we suggest not using this currently and instead use two invocations to setup the backing device and the caching device. These bugs have been fixed in the latest bcache but the fixed kernel is not available for all distros yet.

How to w/ Aerospike?

● http://www.aerospike.com/docs/operations/plan/ssd/bcache/asbcache/

Why w/ Aerospike?

● Cloud providers– No persistence w/ performance

● So far..– In-memory backed with persistence

– Costly!

● Bcache provides persistence w/ performance– At upto 80% the cost saving!!

Why w/ Aerospike?

Why w/ Aerospike?

Why w/ Aerospike?

Questions..?