HA with RelStorage and Postgres

Harder, better, faster, stronger.

HA with RelStorage and Postgres

abstract @ PLOG 2014simone.deponti@abstract.it /

HA?From the not-very-realiable Wikipedia page about it:

A = Ut / Tt

Where Ut is the uptime and Tt the total time

Three rules of HA1. Eliminate single points of failure2. Have a reliable failover3. Detect failures as they occur

It’s a short way to HA

Two elephants in a cluster1. Use PostgreSQL’s native Streaming

Replication2. Means explicit master/slave roles at

any given time3. Manual failover (slave promotion) or

use repmgr

repmgr1. Developed by phoenicians

2ndQuadrant2. Works in addition to streaming

replication3. Acts as watchdog and can take

automatic actions (run bash scripts)4. You run it on the slave node

(https://github.com/2ndQuadrant/repmgr)

How it works1. Continuously and compulsively

checks twitter that the master is alive

2. If the master is unreachable for more than N seconds, runs a bash script

It also offers convenient command line tools to check status of cluster, promote nodes, syncronize.

repmgr’s gotchas1. Create a database to store

replication info (do not follow the bad example of the documentation)

2. The suggested wal_keep_segments setting is too high, will use up to 78GB, can be reduced with -w option

3. Use custom promote and follow scripts

4. Launch daemon with with --monitoring-history

Fail scenarios1. One node has a catastrophic failure

(easy)2. There is a total network outage

(when the network goes up again, you have a split brain)

3. There is network partitioning (similar to above, can be worst)

repmgr and streaming replication do no perform too well in cases 2 and 3

Our work is never over1. Always notify when a failure is

detected2. Investigate ASAP, even if automatic

action was taken3. Have the slave try to exclude the

master upon promotion

Example of #3:The slave upon promotion contacts all the clients and tells them to avoid talking to master

RelStorage and PostgreSQLIt has a smal issue with connections IDLE IN TRANSACTION.

Check with:

SELECT datname, usename, query_start, state_change, state, queryFROM pg_stat_activity;

It’s not fatal, but might result in locks during backups.

Simone Depontisimone.deponti@abstract.it