P90 X Your Database!!

1

Friday, March 25, 2011

Your Database!!Denish Patel

Database ArchitectOmniTi Computer Consulting Inc.

[email protected]

1


mailto:[email protected]

mailto:[email protected]

2


Who am I ?• Denish Patel

• Database Architect

• 4+ Years with OmniTi

• Expertise

• Heterogeneous Databases

• PostgreSQL, Oracle, MySQL

• Very Large databases

• Performance tuning

• Application level

• SQL tuning

• Database and OS parameters

• Objects tuning2


Who am I ?• Denish Patel

• Database Architect

• 4+ Years with OmniTi

• Expertise

• Heterogeneous Databases

• PostgreSQL, Oracle, MySQL

• Very Large databases

• Performance tuning

• Application level

• SQL tuning

• Database and OS parameters

• Objects tuning2

OmniTi is hiring!


3


Agenda•Why bloat ?

•Issues

•Identify bloat

•Tables

•Indexes

•Bloat removal toolkit

•Monitoring and trending

3


4


why bloat?

•MVCC

•ACID

•Updates & Deletes

4


5


MVCC method-1

5


MVCC method-1

5


MVCC method-1!

!"#$%&'

()*+&$

'

,+-.$'

/+&+' (0/1,2'

34..-+56''

7$89$"&'

,31:7'

;<='

'!:723,'

/2;2,2'

72;2>,''

><??!,'

5


6


MVCC method-2

6


MVCC method-2

6


!

!"#$%&

'"("&

)*'+!,&

!-+./&

012&

&3./,-!&

',0,!,&

/,0,4!&&

41553!&

MVCC method-2

6


7


Issues•Unnecessary costly I/O operations

•Poor database tables access time

•Increase maintenance time

•Waste of resources

•RAM

•CPU

•Disk

7


8


Prevent Bloat• autovacuum =on

• vacuum manually between batch updates

• Truncate table

• Free Space Map

• <= PostgreSQL 8.3

• max_fsm_pages

• max_fsm_tables

• >=PostgreSQL 8.4

• auto sizing free space map

8


9


BEWARE!•Bug: long running auto-vacuum workers

•PostgreSQL Version Affected

•8.3.10

•8.3.11

•8.3.12

•8.4.4

•8.4.5

9


BEWARE!•Bug: long running auto-vacuum workers

•PostgreSQL Version Affected

•8.3.10

•8.3.11

•8.3.12

•8.4.4

•8.4.5

9

Upgrade to latest release !!


10


Measure Bloat

•Bloat Report

•pg_bloat_report (OmniTI Labs)

•Nagios Monitoring

•Check_postgres_bloat

10


11


pg_bloat_reportBloat Report Tables

1. omniti.hits 1969348 of 4255960 pages wasted (46.3%), 15 GB of 32 GB.

2. omniti.clients 279979 of 370509 pages wasted (75.6%), 2187 MB of 2895 MB.

Bloat Report Indexes

1. omniti.hits_pk 51275 of 71075 pages wasted (72.1%), 401 MB of 555 MB.

2. omniti.clients_tz 47550 of 71271 pages wasted (66.7%), 371 MB of 557 MB.11


12


check_postgres_bloat

Notification Type: PROBLEMHost: omniti

State: CRITICALDate/Time: Wed Feb 16 11:24:56 UTC 2011

Additional Info: POSTGRES_BLOAT CRITICAL: DB "omniti" (host:omniti)

(port=5432) table omniti.hits rows:23995164 pages:2858893 shouldbe:2278176 (1.3X)

wasted size:4757233664 (4 GB)

12


13


bloat removal toolkit

• Offline - with locking

• Vacuum

• Vacuum Full

• Cluster

• Cluster vs Vacuum Full

• Online - with minimal locking

• compact_table

• pg_reorg

13


14


14


15


vacuum•Auto vacuum

•VACUUM [verbose|analyze] table

•Reclaims storage occupied by dead tuples

•Reclaimed space available to re-use

•Normally free space doesn’t return to OS

•No exclusive table lock

15


16


vacuum full• VACUUM FULL table

• Compact table based on dead rows

• Starting PostgreSQL 9.0 , it’s rewrite entire table and indexes (like CLUSTER)

• Reclaimed space available to re-use

• Space returned to OS

• Table Exclusive lock

• Expensive operation

16


17


17


18


cluster•CLUSTER [verbose] table_name

[USING index_name]

•Create reorganized copy of table based on index

•Rebuild indexes too

•Requires Exclusive Lock on table

•Expensive Operation

18


19


cluster vs vacuum full

Order based on IndexIt’s just rewrite of table without

any specific order

Table needs index Doesn’t need index

Doesn’t bloat indexBloat indexes (Pre PostgreSQL

9.0)

Overall operation Faster Overall Operation Slower (Pre

PostgreSQL 9.0)

Recommended Not recommended

19


20


20


21


compact_table

•Developed by OmniTi

•Reorganize rows to empty pages at the end based on ctid

•ctid - (page number, tuple number)

•When we update a row, the row's ctid changes, because the update creates a new version of the row and leaves the old version behind

21


22


compact_tablepostgres=# select ctid,* from bar;

ctid | a | b -------+---+--- (0,1) | 1 | 1 (0,2) | 2 | 1

(2 rows)

postgres=# update bar set a=3 where a=2;UPDATE 1

postgres=# select ctid,* from bar; ctid | a | b

-------+---+--- (0,1) | 1 | 1 (0,3) | 3 | 1

(2 rows)

postgres=# vacuum verbose bar;..

INFO: "bar": found 1 removable, 2 nonremovable row versions in 1 out of 1 pagesDETAIL: 0 dead row versions cannot be removed yet.

VACUUM22


23


compact_table

23


compact_tablepostgres=# truncate table bar;

TRUNCATE TABLEpostgres=# insert into bar select generate_series(1,1000);

INSERT 0 1000postgres=# delete from bar where a %2 = 0;

DELETE 500postgres=# select max(ctid) from bar;

max -------- (4,95)(1 row)

postgres=# vacuum verbose bar;INFO: "bar": found 0 removable, 500

nonremovable row versions in 5 out of 5 pagesVACUUM23


24


compact_tablepostgres=# begin;

BEGINpostgres=# update bar set a=a where ctid>='(3,0)';

UPDATE 161postgres=# update bar set a=a where ctid>='(3,0)';





UPDATE 0postgres=# commit;

COMMITpostgres=# vacuum verbose bar;

.

.INFO: "bar": truncated 5 to 3 pages

.VACUUM24


25


compact_table

•Pretty safe update process

•No additional changes required

•Slow process

•Generate Index bloat

•Under development

•Not tested in production

•Future consideration?25


26


26


27


pg_reorg• Developed by NTT OSS Center

• Reorganize table with very short lock

• Needs Primary Key

• Not Null Unique Key works!

• CLUSTER online

• ORDER BY Key

• VACUUM FULL Online

• Rewrite without ORDER BY

• Build and Install like other Contrib Module

27


28

!

!"#$%&'(

)"'($"($*+,($-./*0(

( 1&2*#$(

( 3,4-$*(

( 5*/*$*(

6*,/-7(

8,$%"&-/(

9"#:(

;-./*(

;*+,(

)"'((

;-./*(

;-#'*$(

;-./*(

3,4-$*(

9"#:(($-./*(

<%$=(

8,*#-$%"&-/(

)"'(

!<-,,%&'(

<%$=(>*#7(

2="#$()"?:(

(


pg_reorg

28

!

!"#$%&'(

)"'($"($*+,($-./*0(

( 1&2*#$(

( 3,4-$*(

( 5*/*$*(

6*,/-7(

8,$%"&-/(

9"#:(

;-./*(

;*+,(

)"'((

;-./*(

;-#'*$(

;-./*(

3,4-$*(

9"#:(($-./*(

<%$=(

8,*#-$%"&-/(

)"'(

!<-,,%&'(

<%$=(>*#7(

2="#$()"?:(

(


29


pg_reorg Optionspg_reorg --no-order -table test -d omniti

-n -- no-orderDo online VACUUM FULL.

-o columns [,...]--order-by=columns [,...]Do online CLUSTER ordered by specified columns.

-t table--table=table

-Z--no-analyze

-T seconds--wait-timeout=seconds

29


30


pg_reorg in action$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test

LOG: (query) SET statement_timeout = 0LOG: (query) SET search_path = pg_catalog, pg_temp, publicLOG: (query) SET client_min_messages = warningLOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclassLOG: (param:0) = public.testLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.conflicted_triggers($1)LOG: (param:0) = 7441636LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test)LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)')LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')LOG: (query) COMMIT

30





SET variables

30





SET variables

Place Triggers

30


31


pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE

LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'), true)LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = 'virtualxid' AND pid <> pg_backend_pid()LOG: (query) DELETE FROM reorg.log_7441636LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_idLOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')LOG: (query) COMMITLOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1LOG: (param:0) = 7441636LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id)

LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000 31




LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000

CTASCreate Indexes

31




LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000

CTASCreate Indexes

Apply Operational

logs

31


32


pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 0

LOG: (query) SELECT reorg.reorg_swap($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SELECT reorg.reorg_drop($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTEDLOG: (query) ANALYZE publoc.testLOG: (query) COMMIT

32






Lock table and

apply operational logs

last time

32






Lock table and


last time

Swapping

32






Lock table and


last time

Swapping

Drop work and temp tables

Analyze new table

32


33


pg_reorg in Production

•Database Size 540GB

•OLTP database 2-3K transactions/sec

•Largest pre-rebuild table Size was 127GB

•Took 5.75 hours to rebuild

•Reclaimed 52GB

•No outage reported for website!

33


34


pg_reorg in Production

34

Table Size Before Reorg

Pg_Reorg Execution time

Bloat Removed

35GB 24m 10sec 13GB

27GB 116m 8 Sec 6GB

31GB 64m 32sec 4GB

44GB 128m 44sec 3GB


35


35


36


36


37


Test Case

• Database : PostgreSQL 9.0.2

• OS: Solaris 11

• RAM : 32GB

• work_mem=16MB

• Table Name : Users

• Table Size : 1180 MB

• Indexes Size : 256 MB

• Rows : 6M

postgres=# \d users Table "public.users"

Column | Type | Modifiers --------------+--------------------------+----------- user_id | integer | not null

name | text | creation_tsz | timestamp with time zone |

active | boolean | clicks | double precision |

Indexes: "users_pkey" PRIMARY KEY, btree (user_id) "users_creation_tsz" btree (creation_tsz)

- DELETE 3M ROWS- UPDATE 1M ROWS37


38


Comparison

CLUSTER VACUUM FULL PG_REORG

Execution Time: 2 min

Table Size: 594MBIndex Size: 128 MB

Execution Time:1 min 50 Sec


Execution Time: 1 min 4 Sec


offline offline online

Built - in Built-in External Tool

38


39


Other Tips

•Remove unused indexes

•Reindex indexes as needed

•Partitioned table

•Detach child table -> Cluster child table -> Attach

•Avoid vacuum full (Pre PostgrSQL 9.0)

39


40


Monitoring

•Circonus

•Nagios monitoring

•Bloat report

40


41


trending

•Cacti

•Circonus

41


42


trending

42


trending

42


43


thanks!

•OmniTi

•Robert Treat

•Depesz

•Joshua (End Point)

•PGEast Conference Committee

43


44


References•http://www.postgresql.org/docs/9.0/

•http://reorg.projects.postgresql.org/pg_reorg.html

•https://labs.omniti.com/pgtreats/trunk/tools/compact_table

•http://blog.endpoint.com/2010/09/reducing-bloat-without-locking.html

44


http://www.postgresql.org/docs/9.0/

http://www.postgresql.org/docs/9.0/

https://labs.omniti.com/pgtreats/trunk/tools/compact_table




45


open discussion

•When PostgreSQL will have built in features to rebuild table and index online ?

•Is it worth to change MVCC handling to avoid bloating?

45


46


46

Questions?

•Email : [email protected]

•Twitter : http://twitter.com/#!/denishpatel


http://twitter.com/denishpatel

http://twitter.com/denishpatel

Documents

P90 X Your Database!!