Replication in Postgresql - PGDay · • What is Streaming Replication and what is its advantages? Async vs Sync, Hot standby etc. • How to configurateMaster and Standby Servers?

Built-in Physical and Logical Replication in Postgresql

Fırat Güleç

- CompanyHepsiexpress

- NowInfrastructure & Database ManagerMember of Postgresql EuropeContributing to Open Source Community

- Next TalksAustria PGDay 2019, September 6th, 2019

Agenda• What is Replication?• Why do we need Replication?• How many replication layers do we have?• Understanding milestones of built-in Database Physical Replication.• What is the purpose of replication? and How to rescue system in case of failover?• What is Streaming Replication and what is its advantages? Async vs Sync, Hot standby etc.• How to configurate Master and Standby Servers? And What is the most important parameters?

Example of topoloji.• What is Cascading Replication and how to configurate it? Live Demo on Terminal.• Quorum Commit for Sync Replication etc.• What is Logical Replication coming with PostgreSQL 10? And What is its advantages?• What is the purpose of Logical replication?• How to set up Logical Replication and What are its benefits?• Limitations of Logical Replication• Logical Replication vs Physical Replication in detail.• 10 Questions quiz and giving some gifts to participants according to their success.

What is Replication?

Master Slave

Slave

Slave

Slave

Slave

Slave

Slave

Slave

Slave

Replication Layers

Application

Database logical

Database physical

Operating System

Hardware SAN

NAS

DRBD

Streaming Replication

Logical Replication

Slony

LondistePgpool II

Bucardo

High Availability, Load Balancing, and Replication Feature Matrix

https://www.postgresql.org/docs/11/different-replication-solutions.html

https://www.postgresql.org/docs/11/different-replication-solutions.html

Pyhsical Replication

The purpose of Streaming Replication?

High availability

App Server

Client

Master Slave

The purpose of Streaming Replication?

Load Balancing

App Server

Client

Master Slave

%50 %50

Database Physical Replication Milestones

File based from 8.3


9.0

Synchronous since 9.1

Quorum commit since

10.0

Architecture of Streaming Replication

Master Slave

WAL

WAL sender

WAL receiver

WAL Record

16 MB

pg_wal

WALArchievedirectory

Database Physical

File based from 8.3


9.0


Quorum commit since

10.0

File based Replication from Postgresql 8.3

Master Slave

WAL

WAL sender

WAL receiver

WAL Record

16 MB

pg_wal


WAL 16 MB

Database Physical

File based from 8.3


9.0


Quorum commit since

10.0


Master Slave

WAL

WAL sender

WAL receiver

WAL Record

16 MB

pg_wal


Record-based log shippingThe primary and standby servers so that they are as similar as possible

1- Major PostgreSQL release levels is not possible

2- 32-bit to a 64-bit system will not work.

pg_wal

WAL

Database Physical

File based from 8.3


9.0


Quorum commit since

10.0


Master Slave

WAL

WAL sender

WAL receiver

WAL Record

16 MB

1- Async vs Sync2- Hot Standby or Warm Standby?

readwrite read

pg_wal


Database Physical

File based from 8.3


9.0


Quorum commit since

10.0

Quorum Commit for Sync Replication

• 10.0: Quorum Commit

Master Slave 2

Slave 3

Slave 1

sync

10.0synchronous_standby_names ANY 2 (s1, s2, s3)

sync

sync10.0synchronous_standby_names FIRST 2 (s1, s2, s3)

6 Steps for Streaming Replication

Master Slave

1. Replica user is created

2. Configuration for Master

3. Authentication

4. Configuration for Slave

5. Synchronization

6. Create Recovery.conf

Hot Standy - postgresql.conf

• wal_level determines how much information is written

wal_level=‘minimal’wal_level=‘replica’wal_level=‘logical’

max_wal_senders

• max_wal_senders= specifies the maximumnumber of concurrent connections

Master

Slave

Slave

Slave

wal_keep_segments

wal_keep_segments

Master Slave

WAL WAL

WAL sender

WAL receiver

WAL Record

16 MB

pg_wal


1-Replication User for Master

• sudo -u postgres psqlNext, create a new user and role with the following command:• postgres=#CREATE USER replica REPLICATION LOGIN ENCRYPTED

PASSWORD ‘*******';

postgres=#\du• You should see the following output:

2-Hot Standby Configuration for Master

in postgresql.conf

• wal_level=replica• wal_keep_segment=20• max_wal_sender=3• archieve_mode=on• archive_command = 'test ! -f /var/lib/postgresql/pg_log_archive/%f

&& cp %p /var/lib/postgresql/pg_log_archive/%f'

3-pg_hba.conf configuration for Master

For authentication:

host replication replica 10.90.82.61/32 md5

4-Hot standy configuration for slave

In Postgresql.conf

• hot standby=on

Below configuration in case of fail over

• archive_mode = on• archive_command = 'test ! -f /var/lib/postgresql/pg_log_archive/%f && cp

%p /var/lib/postgresql/pg_log_archive/%f‘• wal_keep_segment=20• max_wal_sender=3

5-Syncronize Data from Master Server to Slave ServerOn the slave server, stop the postgresql service:

• sudo systemctl stop postgresql and move existing data folder.• pg_basebackup -h 10.90.82.31 -D /var/lib/postgresql/11/main/ -P -U

replica --wal-method=fetch

Master Slave

Transfering……

Data file directory

/var/lib/postgresql/11/main

10.90.82.31 10.90.82.61

6-Recovery.conf file on standby

Datafile Directory/var/lib/postgresql/11/main

Test Replication

Commandpsql -x -c "select * from pg_stat_replication;"

Streaming Replication is DONE

Cascading Postgresql Replication

Master

Slave

Slave

Slave

Slave

Slave

Slave

Upstream Slaves

Downstream Slaves

ReplicationCascading Replicaiton

hot_standby=on

Syncronization

1.

2.

3.

async

4. recovery.conf

max_standby_archive_delay for standby

Logical Replication

Logical replication

Master Slave

Streaming replication

Master Slave

DB A

DB B

DB A

DB B

DB A

DB B

DB AEmployee

What is Logical Replication?

Master Slave


Master SlaveRecovery Mode

Create indexCreate tableDrop tableCreate functionInsertDelete

SelectHot Standby

SelectWarm Standby

New Instance

Create indexCreate tableDrop tableCreate functionInsertDelete

Select

What is Logical Replication?Logical replication

Master Slave


Master Slave

PostgreSQL 9.5 PostgreSQL 9.5

PostgreSQL 11 PostgreSQL 12

PostgreSQL 9.6

PostgreSQL 10

PostgreSQL 10

PostgreSQL 11

What is Logical Replication?Logical replication

Expected use cases of Logical replication

Streaming replication Client

1. High availability 2. Load Balancing

Master Standby

1. Analytical Purpose

Master A

Master B

DB C

2. Sharing a subnet of database

Expected use cases of Logical replication

Streaming replication Client

1. High availability 2. Load Balancing

Master Standby

Master A Master B

3. Online Upgrade

PostgreSQL 10 PostgreSQL 11

App

DumpSchema

Logical Replication

Logical replication

Master Slave


Master Slave

WAL sender

WAL receiver

WAL Streaming

WAL sender

Publication Subscription

Apply Worker

WAL

WAL DecodingDecoded message Logical

Replication Launcher

triggers

Architecture

What is Publication&Subscription?


MasterSlave SlaveStreaming

ReplicationCascading Replication

Slave A

Slave B

Publication 2

Publication 1

Subscription 2Master

Subscription 1

Pub 3

Slave C

Subs 3Users Name

Users Name Surname

Async vs SyncLogical replicationLogical

Replication Slot 1

Logical Replication Slot 2

Logical replication


MasterSlave Slave


Cascading Replication

DB A

Sub 1Pub 1

Surname

DB B

Address

Pub 2Sub 2

Logical replication


MasterSlave Slave


Cascading Replication

Server A

Surname

Server B

Surname

Bi directional replication not allowed

Logical Replication Limitations in 11.0

• does not replicate schema and DDL • does not replicate sequences • does not replicate Large Objects• Replication is only possible from base tables to base tables

Logical replication with PostgreSQL 11

Master10.90.82.30

wal_level = logical1.

3.

WAL

6.

Publication 1

repluser

CREATE TABLE user (user_name text PRIMARY KEY, full_name text);

CREATE PUBLICATION Publication1 for table user;4.

CREATE ROLE repluser WITH LOGIN PASSWORD 'admin123' REPLICATION ;

user

GRANT SELECT ON public.delivery TO repluser;

GRANT USAGE ON SCHEMA public TO repluser;

5.

7.

Slave 10.90.82.31

Connected to db with postgres\c database12.

Pg_hba.conf host all repluser 10.90.82.31/0 md5

Connected to db with postgres\c database21.

2.

CREATE SUBSCRIPTION Subscribe1 CONNECTION 'host=10.90.82.31 dbname=database1 user=repluserpassword=admin123' PUBLICATION Publication 1 ;

Subscribe1

CREATE TABLE user (user_name text PRIMARY KEY,full_name text);

user

3.

Logical Replication is DONE

Topology(Logical & Pyhsical Replication)

App Server

Client

Master Slave

readwrite read

SlaveCascading ReplicationStreaming Replication

read

Reporting Server

SlaveLogical Replication

Analytical purposes

App Server

readwrite

CPU & Memory

Monitoring on Terminal

1- pg_stat_replication2- pg_replication_slots3- pg_publication_tables

1- pg_subscription2- pg_stat_subscription3- pg_subscription_rel

Master Slave(Logical) Slave(Physical)

1- pg_stat_wal_receiver

https://kahoot.it/

https://kahoot.it/

With an innovative service approach,We are shaping delivery and transportation system in

Turkey

www.hepsiexpress.com

Hepsiexpress & Horizon 2020

ISA: Innovative logistics Solutions and data

framework for the on-demand economy

We are looking for partners!

2018 Applications

Focus Areas for New Projects

• Logistics Research & Innovation• Route Optimisation• Natural Language Processing• Open Source Softwares

Thank you

Fırat Güleç

Built-in Physical and Logical Replication in Postgresql� Fırat GüleçAgendaWhat is Replication?Replication LayersHigh Availability, Load Balancing, and Replication Feature MatrixPyhsical ReplicationThe purpose of Streaming Replication?The purpose of Streaming Replication?Database Physical Replication MilestonesArchitecture of Streaming ReplicationDatabase PhysicalFile based Replication from Postgresql 8.3Database PhysicalStreaming ReplicationDatabase PhysicalStreaming ReplicationDatabase PhysicalQuorum Commit for Sync Replication6 Steps for Streaming ReplicationHot Standy - postgresql.confmax_wal_senderswal_keep_segmentswal_keep_segments1-Replication User for Master2-Hot Standby Configuration for Master3-pg_hba.conf configuration for Master4-Hot standy configuration for slave5-Syncronize Data from Master Server to Slave Server6-Recovery.conf file on standbyTest ReplicationCascading Postgresql Replicationmax_standby_archive_delay for standbyLogical ReplicationLogical replicationFoliennummer 36Foliennummer 37Expected use cases of Logical replication Expected use cases of Logical replication Logical replicationWhat is Publication&Subscription?Logical replicationLogical replicationLogical Replication Limitations in 11.0 Logical replication with PostgreSQL 11Topology(Logical & Pyhsical Replication)Monitoring on Terminalhttps://kahoot.it/Foliennummer 49Hepsiexpress & Horizon 2020 Thank you

Documents

Replication in Postgresql - PGDay · • What is Streaming Replication and what is its advantages? Async vs Sync, Hot standby etc. • How to configurateMaster and Standby Servers?