Scaling postgres

/

Scaling Postgres

Denish PatelDatabase Architect

https://twitter.com/DenishPatel

Wednesday, September 18, 13



1Wednesday, September 18, 13

OmniTI


OmniTI

1

• Helping customersnavigate explosive growthwith technology.


OmniTI

1


100MM+ users

$1B+ gross online sales


OmniTI

1


100MM+ users


Open and closed source thought leaders, experts and authors


OmniTI

1


100MM+ users


Open and closed source thought leaders, experts and authors



Talk Outline


Talk Outline

• Scalability

• Database Scaling needs, costs, methods

• Scaling Postgres

• Vertically

• Horizontally

• Obstacles to Scalability

• Beyond Postgres



What is Scalability?


What is Scalability?

3

A service is said to be scalable if when we increase the resources in a system, it results in

increased performance in a manner proportional to resources added.



Why to scale databases?



• Support a higher volume of users












• Provide better performance for existing users





• Store a larger volume of data






• Improve system availability






• Improve system availability

• Geographic dispersion




Why is database scalability so hard?





• Search



• Search

• Concurrency



• Search

• Concurrency

• Consistency



• Search

• Concurrency

• Consistency

• Speed



• Search

• Concurrency

• Consistency

• Speed

• Cost



• Search

• Concurrency

• Consistency

• Speed

• Cost

• Cost of hardware



• Search

• Concurrency

• Consistency

• Speed

• Cost


• Cost deployment effort



• Search

• Concurrency

• Consistency

• Speed

• Cost


• Cost deployment effort

• Cost Ongoing maintenance



Scaling Needs & Methods





• Data growth



• Data growth

• Read requests



• Data growth

• Read requests

• Write requests



• Data growth

• Read requests

• Write requests

• Vertical Scaling

• Horizontal Scaling



• Data growth

• Read requests

• Write requests





• Data growth

• Read requests

• Write requests





Vertical Scaling (Scale up)





• Pros



• Pros

• Simple to implement



• Pros


• Ease of maintenance



• Pros



• Cons



• Pros



• Cons




• Pros



• Cons


• Sometimes SPOFs



Horizontal Scaling (Scale Out)







• Pros



• Pros

• Cheaper in hardware cost



• Pros


• Flexibility



• Pros


• Flexibility

• Higher fault tolerance



• Pros


• Flexibility


• Cons



• Pros


• Flexibility


• Cons

• Complex to implement



• Pros


• Flexibility


• Cons


• Expensive to maintain



• Pros


• Flexibility


• Cons



• Bigger footprint in the Data Center



• Pros


• Flexibility


• Cons



• Bigger footprint in the Data Center

• No built in support in databases



Spec’ing Hardware


Spec’ing Hardware

• CPU


Spec’ing Hardware

• CPU

• 8+ cores


Spec’ing Hardware

• CPU

• 8+ cores

• RAM


Spec’ing Hardware

• CPU

• 8+ cores

• RAM

• 64GB+


Spec’ing Hardware

• CPU

• 8+ cores

• RAM

• 64GB+

• Disks


Spec’ing Hardware

• CPU

• 8+ cores

• RAM

• 64GB+

• Disks

• SSDs


Spec’ing Hardware

• CPU

• 8+ cores

• RAM

• 64GB+

• Disks

• SSDs

• RAID 10


Spec’ing Hardware

• CPU

• 8+ cores

• RAM

• 64GB+

• Disks

• SSDs

• RAID 10

• Network


Spec’ing Hardware

• CPU

• 8+ cores

• RAM

• 64GB+

• Disks

• SSDs

• RAID 10

• Network

• min Gigbit, 10Gigbit



Tune Postgres/memory parameters



• shared_buffers



• shared_buffers

• effective_cache_size



• shared_buffers


• checkpoint_completion_target



• shared_buffers



• checkpoint_segments



• shared_buffers




• max_connections



• shared_buffers




• max_connections

• work_mem



• shared_buffers




• max_connections

• work_mem

• maintenance_work_mem



• shared_buffers




• max_connections

• work_mem

• maintenance_work_mem

http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server





Tune Postgres/logging parameters







• logging_collector => 'on'




• log_destination => 'stderr'





• log_filename => 'postgresql-%Y-%m-%d_%H%M%S.log'






• log_line_prefix => '%m [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '







• log_min_duration_statement => 1000ms







• log_min_duration_statement => 1000ms

• log_autovacuum_min_duration => '0'










• log_lock_waits => 'on'




• log_temp_files => '0'





• log_checkpoints => 'on'






• log_connections => 'on'







• log_disconnections => 'on'








• log_min_error_statement => 'warning'









• log_min_messages => 'warning'









• log_min_messages => 'warning'

• log_statement => 'ddl'



Optimize Queries/report slow queries







Optimize Queries/Extensive monitoring


Optimize Queries/Extensive monitoring



Optimize Queries /Explain Analyze



explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1) Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision)) Heap Fetches: 0 Buffers: shared hit=923073 read=3848 Total runtime: 4297.405 ms



explain (analyze,buffers) select col1,col2 from demo_ios where col2 between 0.01 and 0.02; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------- Index Only Scan using idx_demo_ios on demo_ios (cost=0.00..35330.93 rows=993633 width=16) (actual time=58.100..3250.589 rows=1000392 loops=1) Index Cond: ((col2 >= 0.01::double precision) AND (col2 <= 0.02::double precision)) Heap Fetches: 0 Buffers: shared hit=923073 read=3848 Total runtime: 4297.405 ms



Optimize Queries /track functions





• track_functions = pl # none, pl, all




• reload online




• reload online

• select * from pg_stat_user_functions;




• reload online





• reload online


funcid | schemaname | funcname | calls | total_time | self_time




• reload online


funcid | schemaname | funcname | calls | total_time | self_time



Partitioning


Partitioning

• As table size grows, queries eventually slows down, even with indexing


Partitioning


• Allows data added, removed and queried fast


Partitioning



• Partitioning pruning queries


Partitioning



• Partitioning pruning queries

• Manage partitions



Partitioning /Postgres





• Postgres partitioning




• Trigger based




• Trigger based

• Rule based




• Trigger based

• Rule based

• Lack of built-in Postgres partition management




• Trigger based

• Rule based


• Postgres partition management extension: pg_partman




• Trigger based

• Rule based



• http://pgxn.org/dist/pg_partman/doc/pg_partman.html


http://pgxn.org/dist/pg_partman/doc/pg_partman.html




• Trigger based

• Rule based




• Tomorrow’s session at 11:30AM (When Postgres can’t....)





Partitioning /functional



Configuration Data Transaction data Session data

Configuration Tools Reporting Tools Monitoring Tools

Web Applications Other Applications






• Partition data based on functionality







• Separate Postgres clusters








• Start with Separate schemas









• No relationship between data










• Help to spread the load across server










• Help to spread the load across server

• Less complex compare to sharding!



pgbouncer


pgbouncer


pgbouncer


pgbouncer


pgbouncer

• A lightweight connection pooler


pgbouncer


• Helps to reduce # of newly created connections on DB server


pgbouncer



• Abstracts DBs from App


pgbouncer




• Helps to instrument smooth and easy failover


pgbouncer





• Connection pooling Options


pgbouncer






• Session, Transaction, Statement pooling options


pgbouncer






• Session, Transaction, Statement pooling options

• Beware! Transaction pooling doesn’t support prepared transactions



Caching


Caching


Caching

• Memcached


Caching

• Memcached

• Open source, High-performance distributed memory object caching system


Caching

• Memcached


• Speeds up dynamic web applications by alleviating database load.


Caching

• Memcached



• An in-memory key-value store for small chunks of arbitrary data


Caching

• Memcached




• Redis


Caching

• Memcached




• Redis

• Open source, advanced key-value store.


Caching

• Memcached




• Redis

• Open source, advanced key-value store.

• Works with an in-memory & persistent dataset



Replication /built-in



• Cluster Level Replication (Binary)




• Streaming Replication





• WAL-only replication






• Hybrid replication



Master







Master

Failover • Cluster Level Replication (Binary)






Master

Failover

Read Salve 1







Master

Failover

Read Salve 1







Master

Failover

Read Salve 1







Master

Failover

Read Salve 1

Streaming Replication







Master

Failover

Read Salve 1

PITR!








Master

Failover

Read Salve 1

PITR!

PITR!








Master

Failover

Read Salve 1

PITR!

PITR!

PITR!Streaming Replication







Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in

• Allows to open replicated database in read-only mode

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

• All or none

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

• All or none

• Doesn’t allow write on replicated database

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

• All or none


• Doesn’t work across major version

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

• All or none



• Postgres 9.2 (primary) does NOT replicate to 9.3 (secondary)

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

• All or none




• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)

Master

Failover

Read Salve 1

PITR!

PITR!








• Pros:

• Built-in


• Cons:

• All or none




• Postgres 9.2.1 (primary) can replicate to 9.2.4 (secondary)

Master

Failover

Read Salve 1

PITR!

PITR!


https://wiki.postgresql.org/wiki/Binary_Replication_Tools












Read Salve 1



Read Salve 1

DW System



Read Salve 1

DW System



Read Salve 1

DW System

postgres_fdw



Read Salve 1

DW System

postgres_fdw



Read Salve 1

DW System

postgres_fdw



• postgres_fdwRead

Salve 1

DW System

postgres_fdw



• postgres_fdw

• Postgres 9.3 feature

Read Salve 1

DW System

postgres_fdw



• postgres_fdw


• Allows to access data stored in external PostgreSQL servers

Read Salve 1

DW System

postgres_fdw



• postgres_fdw



• cross version queries

Read Salve 1

DW System

postgres_fdw



• postgres_fdw




• Postgres 9.3 could query Postgres 9.1

Read Salve 1

DW System

postgres_fdw



• postgres_fdw





• Application

Read Salve 1

DW System

postgres_fdw



• postgres_fdw





• Application

• Run query remotely on slave db

Read Salve 1

DW System

postgres_fdw



• postgres_fdw





• Application

• Run query remotely on slave db

• Data warehouse data refreshes

Read Salve 1

DW System

postgres_fdw



Replication /third-party tools







• Table level Replication Tools (Trigger based)




• Slony




• Slony

• Bucardo




• Slony

• Bucardo

• Mimeo: http://pgxn.org/dist/mimeo/




• Slony

• Bucardo


• Pros:




• Slony

• Bucardo


• Pros:

• Allows to open replicated database in read-write mode




• Slony

• Bucardo


• Pros:


• Allows table/database level replication




• Slony

• Bucardo


• Pros:



• Allows rolling upgrade




• Slony

• Bucardo


• Pros:




• cross version replication is allowed




• Slony

• Bucardo


• Pros:





• Multi-master replication




• Slony

• Bucardo


• Pros:






• Cons:




• Slony

• Bucardo


• Pros:






• Cons:

• Complicated to setup




• Slony

• Bucardo


• Pros:






• Cons:

• Complicated to setup

• Unknown territory



Sharding


Sharding


Sharding


Sharding

• Sharding is the process of splitting up your data so it resides in different tables or often different physical databases.


Sharding


• Application aware sharding


Sharding


• Application aware sharding

• Application transparent sharding



Application aware sharding






http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram









• http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram












• Postgres allows “logical” shards through Schema












• Postgres allows “logical” shards through Schema

• Easy to move to “physical” shard later











Application transparent sharding


Application transparent sharding



Sharding Challenges


Sharding Challenges

• Reliability


Sharding Challenges

• Reliability

• Distributed queries


Sharding Challenges

• Reliability


• Cross-shard join


Sharding Challenges

• Reliability



• Auto-increment key management


Sharding Challenges

• Reliability




• Choosing shard key


Sharding Challenges

• Reliability




• Choosing shard key

• Shard schemes



Obstacles for Scaling Postgres





• Postgres table bloat




• FKs relationships





• Insufficient logging






• Insufficient Caching







• Insufficient Monitoring and Metrics








• ORMs








• ORMs

• Single Point of Failure








• ORMs

• Single Point of Failure

• Lack of communications between teams



Beyond Postgres


Beyond Postgres


Beyond Postgres

• Avoid serialization in application code


Beyond Postgres


• Feature Flags


Beyond Postgres


• Feature Flags

• Browse only mode (Read only mode)


Beyond Postgres


• Feature Flags


• Don’t use database for Queuing


Beyond Postgres


• Feature Flags



• RabbitMQ


Beyond Postgres


• Feature Flags



• RabbitMQ

• Reconsider options for Full Text Search


Beyond Postgres


• Feature Flags



• RabbitMQ


• tsearch provided by Postgres


Beyond Postgres


• Feature Flags



• RabbitMQ


• tsearch provided by Postgres

• Solr, Lucene



Further reading . . .


Further reading . . .

• Scalable Internet Architectures - Theo Schlossnagle

• Web Operations: Keeping the Data On Time - John Allspaw , Jesse Robbins

• PostgreSQL 9.0 High Performance - Greg Smith



References


References


References

• http://www.postgresql.org/docs/

• http://www.circonus.com

• http://pgbouncer.projects.pgfoundry.org/doc/usage.html


• http://dalibo.github.io/pgbadger/

• https://github.com/omniti-labs/omnipitr


• http://memcached.org

• http://redis.io

• http://postgres-xc.sourceforge.net/docs/1_1/


http://www.postgresql.org/docs/

http://www.postgresql.org/docs/

http://www.circonus.com

http://www.circonus.com

http://pgbouncer.projects.pgfoundry.org/doc/usage.html

http://pgbouncer.projects.pgfoundry.org/doc/usage.html



http://dalibo.github.io/pgbadger/

http://dalibo.github.io/pgbadger/

https://github.com/omniti-labs/omnipitr

https://github.com/omniti-labs/omnipitr





http://memcached.org

http://memcached.org

http://redis.io

http://redis.io

Thanks

• PostgresOpen Conference Committee

• OmniTI

• You!!


Technology

Scaling postgres