+

Redis {beyond the basics}

Presented by

Steph ☺

+Outline

■ Revisiting Redis

■ Keeping data safe

■ Persistence

■ Replication

■ Replacing Failed Master

■ Transaction

■ Reducing memory use

■ Ziplist

■ Intlist

■ Sharding

■ Scaling

+Redis

■ In-memory remote database

■ Advanced key-value store

■ Data-structure server

■ Offers

■ High Performance

■ Replication

■ Unique data model

+Snapshotting

■ Data is taken as it exists and is written to the disk

■ Point-in-time copy of in-memory data

■ Backup & transfer to other server

■ Written to file in “dbfilename” stored in “dir”

■ Until next snapshot is taken, last snapshot can be lost if redis crashes

{Persistence}

+Snapshotting

Initiating Snapshots■ bgsave

■ SAVE

■ Save lines save 60 10000

■ SHUTDOWN / TERM

■ SYNC

{Persistence}

+Snapshotting

■ How often to perform an automatic snapshot

■ Accept writes on failure

■ Snapshot compression

■ What to name the snapshot on the disk

+Append Only File (AOF)

■ Copies incoming write commands as it happens

■ Records data changes at the end of the backup file

■ Data set could be recovered with replaying AOF

■ “append only yes”

■ “appendfsyc always”

■ Limited by disk performance

{Persistence}

+Append Only File (AOF)

■ Option to use AOF

■ Occurrence of sync writes to disk

■ Option to sync during AOF compaction

■ Occurrence of AOF compaction

+Replication

■ Method where other servers receive an updated copy of the data as its being written

■ Replicas can service read queries

■ Single master database sends writes out to multiple slave databases

■ Set operations can take seconds to finish

+Replication

■ Configuring for replication■ On master, ensure that the path and filename are writable

by redis process

■ Enable slaving : slaveof host port

■ In a running system, redis can be stopped slaving or connect to a different master

■ New / Transfer connection: slaveof host port

■ Stop data update: SLAVEOF no one

+Replication {SLAVE to MASTER Connection}

+Replication{Redis Master-Slave Replica Tree}

+Replacing Failed Master

■ What will we do in case of system failure?

■ Scenario■ Machine A – Redis Master, Machine B – Redis Slave

■ Machine A loses network connectivity

■ Machine C has Redis, but no copy of data

■ Solution A■ Make a fresh snapshot using Machine B using SAVE

■ Copy snapshot to Machine C

■ Start Redis on Machine C

■ Tell Machine B to be a slave of Machine C

{Scenario and Solution}

+Replacing Failed Master{Sequence of Commands}

+Replacing Failed Master

■ What will we do in case of system failure?

■ Solution B■ Use Machine B (Slave) as Master

■ Create a new Slave (maybe Machine C)

■ Update client configuration to read/write to proper servers

■ (optional) update server configuration if restart is needed

{Scenario and Solution}

+Transactions

■ Begin transaction with MULTI

■ Execute commands with EXEC

■ Delayed execution with multi/exec can improve performance■ Holds off sending commands until all of them are known

■ When all of the commands are known, MULTI is sent by client

+Transactions

■ Pipelining■ Send multiple commands at once

■ Wait for all replies

■ Reduces number of network roundtrips that the client waits for

+Reducing Memory Use

■ Method of reducing memory use

■ Ziplist – compact storage and unstructured representation of LISTs HASHes and ZSETs

■ Intset – compact representation of SET

■ As structures grow beyond limits, they are converted back to their original data structure type

■ Manipulating compact versions can become slow as they grow

{Short Structures}

+Ziplist

■ Basic configuration for the 3 data types are similar

■ *-max-ziplist-value – max number of items to be encoded as ziplist

■ If limits are exceeded, redis will convert the list/hash/zset into non-ziplist structure

+ZIPLIST - LIST

+Intset

+Sharded Structures

■ Sharding – takes data, partitions it to smaller pieces and sends data to different locations depending on which partition the data is assigned to

■ Sharding LISTs – uses LUA scripting

■ Sharding ZSETs – zset operations on shards violate how quickly zsets perform, sharding is not useful on zsets

+Sharded Structures

■ Sharding HASHes■ Method of partitioning data must be chosen

■ Hash’s keys can be used as source of info for sharding

■ To partition keys:

■ Calculate hash function on the key

■ Calculate number of shards needed depending on number of keys we want to fit in one shard and the total number of keys

■ Resulting number of shards along with hash value will be used to find out which shard we’ll use

+Scaling

■ In using small structures, make sure max ziplist is not too large

■ Use structures that offer good performance for the types of queries we want to perform

■ Compress large data sent to redis for caching to reduce network reads and writes

■ Use pipelining and connection pooling

{read capacity}

+Scaling

■ Increase total read throughput using read only slave servers■ Always remember to WRITE TO THE MASTER

■ Writing on SLAVE will cause an error

■ Redis Sentinel■ Mode where redis server binary doesn’t act like the typical one

■ Watches behavior and health of master(s) and slave(s)

■ Intended to offer automated failover

{read capacity}

+Scaling

■ Make sure to check all methods to reduce read data volume

■ Make sure larger pieces of unrelated functionality are moved to different servers

■ Aggregate writes in local memory before writing to redis

■ Consider using locks or LUA when limitations such as watch/multi/exec are encountered

■ When using AOF, keep in mind that the disk needs to keep up with the volume we’re writing

{memory capacity}

+Scaling

■ Presharding for growth■ Run multiple redis servers on your machine (listen on diff. ports)

■ Use multiple redis database on your database server

{write capability}

+Scaling

■ Scenario : machines have enough memory to hold index, but we need to execute more queries that server can handle

■ Use : SUNIONSTORE, SINTERSTORE, SDIFFSTORE, ZINTERSTORE, and/or ZUNIONSTORE

■ Since we “read” from slave, set : slave-read-only no

{complex queries}

+Reference

■ Carlson, Josiah. (2013) Redis in Action. Shelter Island, NY: Manning Publications

+

Thank You ☺