Scaling with Postgres

Scaling with Postgres

Robert TreatPercona Performance Conference

2009

Wednesday, April 22, 2009

Disclaimer

Every “really big” site uses custom solutions. Don’t expect to get there using out of the box software. You don’t have to reinvent the wheel.

But you may have to build the car.


Who am I ?


Who am I ?

OmniTI

TB+ databases

top 100 websites

Oracle/MySQL/Postgres

Not necessarily in that order :-)


Scaling Databases 101


Scaling Databases 101Implementation



Tuning

Optimize Queries

Optimize Schema



Tuning

Optimize Queries

Optimize Schema

Throw Hardware At It

Caching



Tuning

Optimize Queries

Optimize Schema


Caching

Replication



Tuning

Optimize Queries

Optimize Schema


Caching

Replication

FederationWednesday, April 22, 2009

Implementation

Normalization

Foreign Keys

Relational Modeling


Implementation

Normalization

Foreign Keys

Relational Modeling

Mostly IrrelevantWednesday, April 22, 2009

Implementation


Implementation Monitoring

nagios, munin, etc..

check_postgres


Implementation Monitoring

nagios, munin, etc..

check_postgres

Trending

cacti, noit

watch your hardware!


Tuning


TuningGet the Basics Right

http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server






TuningGet the Basics Right


Someday

Xlogs, Checkpoints

Vacuum Strategy

Costs and other Minutia






Optimize Queries

explain

explain analyze

explain analyze verbose


Optimize Queries

writing queries tuned to your databases planner


Optimize Schema


Optimize Schema

indexing

partial, functional

fix data types

add/remove tables

de-normalize (maybe)


Why is this important?

Postgres is good at scaling vertically

Flawed database schema hides the real problems


Why is this unimportant?

Optimizing is often a trade off of time vs. money

It is not an ultimate solution



Find your bottlenecks

Memory / RAM

CPU

I/O, DISKS

usually the biggest problem


Can you take me higher?

32 Cores

128 GB+ RAM

Spindles out the wazoo

tablespaces


Caching


CachingInternal

materialized views


CachingInternal

materialized views

External

memcached, et al.

static files


Oh, the places you’ll go

Nothing Fancy or Exciting

Solid Engineering

Understanding needs / limits


Oh, the places you’ll go

Nothing Fancy or Exciting

Solid Engineering

Understanding needs / limits

Top 1000+ WebsiteWednesday, April 22, 2009

ReplicationIgnore the hype

Triggers won’t destroy your site

Scripting solves a lot of config issues

Telling your DBA to do his !@#$ job solves config issues too

Add-on is FUD for Modular


Replication

Truth in advertising

Oracle does it better

MySQL does it simpler


Replication


Replication

Scaling Reads

Master, many slaves

Slony, Londiste, Bucardo, PGPoolII


Replication

Scaling Reads

Master, many slaves

Slony, Londiste, Bucardo, PGPoolII

Scaling Writes

Mostly impossible

Bucardo, PGPoolII


What is Federation?

A process for eliminating performance and/or scaleability as a requirement for

your database


Federation


Federation

MySQL software has no federation advantage


Federation

MySQL software has no federation advantage

MySQL community has large federation advantage


Federation

Things to consider

Horizontal Partitioning

Application Based Partitioning

BASE

golconde, gearman


El Fin

www.xzilla.netwww.omniti.com

www.postgresql.us


http://www.xzilla.net

http://www.xzilla.net

http://www.omniti.com

http://www.omniti.com

http://www.postgresql.us

http://www.postgresql.us

Postgres BOF

Tonight @ 7:30 PM

Location @ Camino Real


Technology

Scaling with Postgres