P90 X Your Database!!

Friday, March 25, 2011

Your Database!!Denish Patel

Database ArchitectOmniTi Computer Consulting Inc.

denish@omniti.com

Who am I ?• Denish Patel

• Database Architect

• 4+ Years with OmniTi

• Expertise

• Heterogeneous Databases

• PostgreSQL, Oracle, MySQL

• Very Large databases

• Performance tuning

• Application level

• SQL tuning

• Database and OS parameters

• Objects tuning2

Who am I ?• Denish Patel

• Database Architect

• 4+ Years with OmniTi

• Expertise

• Heterogeneous Databases

• PostgreSQL, Oracle, MySQL

• Very Large databases

• Performance tuning

• Application level

• SQL tuning

• Database and OS parameters

• Objects tuning2

OmniTi is hiring!

Agenda•Why bloat ?

•Issues

•Identify bloat

•Tables

•Indexes

•Bloat removal toolkit

•Monitoring and trending

why bloat?

•MVCC

•ACID

•Updates & Deletes

MVCC method-1

MVCC method-1!

!"#$%&'

()*+&$

,+-.$'

/+&+' (0/1,2'

34..-+56''

7$89$"&'

,31:7'

'!:723,'

/2;2,2'

72;2>,''

><??!,'

MVCC method-2

!"#$%&

)*'+!,&

!-+./&

&3./,-!&

',0,!,&

/,0,4!&&

41553!&

MVCC method-2

Issues•Unnecessary costly I/O operations

•Poor database tables access time

•Increase maintenance time

•Waste of resources

•RAM

•CPU

•Disk

Prevent Bloat• autovacuum =on

• vacuum manually between batch updates

• Truncate table

• Free Space Map

• <= PostgreSQL 8.3

• max_fsm_pages

• max_fsm_tables

• >=PostgreSQL 8.4

• auto sizing free space map

BEWARE!•Bug: long running auto-vacuum workers

•PostgreSQL Version Affected

•8.3.10

•8.3.11

•8.3.12

•8.4.4

•8.4.5

BEWARE!•Bug: long running auto-vacuum workers

•PostgreSQL Version Affected

•8.3.10

•8.3.11

•8.3.12

•8.4.4

•8.4.5

Upgrade to latest release !!

Measure Bloat

•Bloat Report

•pg_bloat_report (OmniTI Labs)

•Nagios Monitoring

•Check_postgres_bloat

pg_bloat_reportBloat Report Tables

1. omniti.hits 1969348 of 4255960 pages wasted (46.3%), 15 GB of 32 GB.

2. omniti.clients 279979 of 370509 pages wasted (75.6%), 2187 MB of 2895 MB.

Bloat Report Indexes

1. omniti.hits_pk 51275 of 71075 pages wasted (72.1%), 401 MB of 555 MB.

2. omniti.clients_tz 47550 of 71271 pages wasted (66.7%), 371 MB of 557 MB.11

check_postgres_bloat

Notification Type: PROBLEMHost: omniti

State: CRITICALDate/Time: Wed Feb 16 11:24:56 UTC 2011

Additional Info: POSTGRES_BLOAT CRITICAL: DB "omniti" (host:omniti)

(port=5432) table omniti.hits rows:23995164 pages:2858893 shouldbe:2278176 (1.3X)

wasted size:4757233664 (4 GB)

bloat removal toolkit

• Offline - with locking

• Vacuum

• Vacuum Full

• Cluster

• Cluster vs Vacuum Full

• Online - with minimal locking

• compact_table

• pg_reorg

vacuum•Auto vacuum

•VACUUM [verbose|analyze] table

•Reclaims storage occupied by dead tuples

•Reclaimed space available to re-use

•Normally free space doesn’t return to OS

•No exclusive table lock

vacuum full• VACUUM FULL table

• Compact table based on dead rows

• Starting PostgreSQL 9.0 , it’s rewrite entire table and indexes (like CLUSTER)

• Reclaimed space available to re-use

• Space returned to OS

• Table Exclusive lock

• Expensive operation

cluster•CLUSTER [verbose] table_name

[USING index_name]

•Create reorganized copy of table based on index

•Rebuild indexes too

•Requires Exclusive Lock on table

•Expensive Operation

cluster vs vacuum full

Order based on IndexIt’s just rewrite of table without

any specific order

Table needs index Doesn’t need index

Doesn’t bloat indexBloat indexes (Pre PostgreSQL

Overall operation Faster Overall Operation Slower (Pre

PostgreSQL 9.0)

Recommended Not recommended

compact_table

•Developed by OmniTi

•Reorganize rows to empty pages at the end based on ctid

•ctid - (page number, tuple number)

•When we update a row, the row's ctid changes, because the update creates a new version of the row and leaves the old version behind

compact_tablepostgres=# select ctid,* from bar;

ctid | a | b -------+---+--- (0,1) | 1 | 1 (0,2) | 2 | 1

(2 rows)

postgres=# update bar set a=3 where a=2;UPDATE 1

postgres=# select ctid,* from bar; ctid | a | b

-------+---+--- (0,1) | 1 | 1 (0,3) | 3 | 1

(2 rows)

postgres=# vacuum verbose bar;..

INFO: "bar": found 1 removable, 2 nonremovable row versions in 1 out of 1 pagesDETAIL: 0 dead row versions cannot be removed yet.

VACUUM22

compact_table

compact_tablepostgres=# truncate table bar;

TRUNCATE TABLEpostgres=# insert into bar select generate_series(1,1000);

INSERT 0 1000postgres=# delete from bar where a %2 = 0;

DELETE 500postgres=# select max(ctid) from bar;

max -------- (4,95)(1 row)

postgres=# vacuum verbose bar;INFO: "bar": found 0 removable, 500

nonremovable row versions in 5 out of 5 pagesVACUUM23

compact_tablepostgres=# begin;

BEGINpostgres=# update bar set a=a where ctid>='(3,0)';

UPDATE 161postgres=# update bar set a=a where ctid>='(3,0)';

UPDATE 0postgres=# commit;

COMMITpostgres=# vacuum verbose bar;

.INFO: "bar": truncated 5 to 3 pages

.VACUUM24

compact_table

•Pretty safe update process

•No additional changes required

•Slow process

•Generate Index bloat

•Under development

•Not tested in production

•Future consideration?25

pg_reorg• Developed by NTT OSS Center

• Reorganize table with very short lock

• Needs Primary Key

• Not Null Unique Key works!

• CLUSTER online

• ORDER BY Key

• VACUUM FULL Online

• Rewrite without ORDER BY

• Build and Install like other Contrib Module

!"#$%&'(

)"'($"($*+,($-./*0(

( 1&2*#$(

( 3,4-$*(

( 5*/*$*(

6*,/-7(

8,$%"&-/(

;-./*(

;-#'*$(

;-./*(

3,4-$*(

9"#:(($-./*(

8,*#-$%"&-/(

!<-,,%&'(

<%$=(>*#7(

2="#$()"?:(

pg_reorg

!"#$%&'(

)"'($"($*+,($-./*0(

( 1&2*#$(

( 3,4-$*(

( 5*/*$*(

6*,/-7(

8,$%"&-/(

;-./*(

;-#'*$(

;-./*(

3,4-$*(

9"#:(($-./*(

8,*#-$%"&-/(

!<-,,%&'(

<%$=(>*#7(

2="#$()"?:(

pg_reorg Optionspg_reorg --no-order -table test -d omniti

-n -- no-orderDo online VACUUM FULL.

-o columns [,...]--order-by=columns [,...]Do online CLUSTER ordered by specified columns.

-t table--table=table

-Z--no-analyze

-T seconds--wait-timeout=seconds

pg_reorg in action$ /opt/pgsql/bin/pg_reorg -e -t public.test --order-by=test_id -U postgres -d test

LOG: (query) SET statement_timeout = 0LOG: (query) SET search_path = pg_catalog, pg_temp, publicLOG: (query) SET client_min_messages = warningLOG: (query) SELECT * FROM reorg.tables WHERE relid = $1::regclassLOG: (param:0) = public.testLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.conflicted_triggers($1)LOG: (param:0) = 7441636LOG: (query) CREATE TYPE reorg.pk_7441636 AS (test_id integer)LOG: (query) CREATE TABLE reorg.log_7441636 (id bigserial PRIMARY KEY, pk reorg.pk_7441636, row pulbic.test)LOG: (query) CREATE TRIGGER z_reorg_trigger BEFORE INSERT OR DELETE OR UPDATE ON pulbic.test FOR EACH ROW EXECUTE PROCEDURE reorg.reorg_trigger('INSERT INTO reorg.log_7441636(pk, row) VALUES( CASE WHEN $1 IS NULL THEN NULL ELSE (ROW($1.test_id)::reorg.pk_7441636) END, $2)')LOG: (query) SELECT reorg.disable_autovacuum('reorg.log_7441636')LOG: (query) COMMIT

SET variables

Place Triggers

pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL SERIALIZABLE

LOG: (query) SELECT set_config('work_mem', current_setting('maintenance_work_mem'), true)LOG: (query) SELECT reorg.array_accum(virtualtransaction) FROM pg_locks WHERE locktype = 'virtualxid' AND pid <> pg_backend_pid()LOG: (query) DELETE FROM reorg.log_7441636LOG: (query) CREATE TABLE reorg.table_7441636 WITH (oids=false) TABLESPACE pg_default AS SELECT * FROM ONLY public.test ORDER BY test_idLOG: (query) SELECT reorg.disable_autovacuum('reorg.table_7441636')LOG: (query) COMMITLOG: (query) SELECT indexrelid, reorg.reorg_indexdef(indexrelid, indrelid) FROM pg_index WHERE indrelid = $1LOG: (param:0) = 7441636LOG: (query) CREATE UNIQUE INDEX index_201337719 ON reorg.table_7441636 USING btree (test_id)

LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000 31

LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000

CTASCreate Indexes

LOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 1000

CTASCreate Indexes

Apply Operational

pg_reorg in actionLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SET LOCAL statement_timeout = 100LOG: (query) LOCK TABLE public.test IN ACCESS EXCLUSIVE MODELOG: (query) RESET statement_timeoutLOG: (query) SELECT reorg.reorg_apply($1, $2, $3, $4, $5, $6)LOG: (param:0) = SELECT * FROM reorg.log_7441636 ORDER BY id LIMIT $1LOG: (param:1) = INSERT INTO reorg.table_7441636 VALUES ($1.*)LOG: (param:2) = DELETE FROM reorg.table_7441636 WHERE (test_id) = ($1.test_id)LOG: (param:3) = UPDATE reorg.table_7441636 SET (test_id, name, lat, lon, city, country, state_abr) = ($2.test_id, $2.name, $2.lat, $2.lon, $2.city, $2.country, $2.state_abr) WHERE (test_id) = ($1.test_id)LOG: (param:4) = DELETE FROM reorg.log_7441636 WHERE id <= $1LOG: (param:5) = 0

LOG: (query) SELECT reorg.reorg_swap($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTED

LOG: (query) SELECT reorg.reorg_drop($1)LOG: (param:0) = 7441636LOG: (query) COMMITLOG: (query) BEGIN ISOLATION LEVEL READ COMMITTEDLOG: (query) ANALYZE publoc.testLOG: (query) COMMIT

Lock table and

apply operational logs

last time

Lock table and

last time

Swapping

Lock table and

last time

Swapping

Drop work and temp tables

Analyze new table

pg_reorg in Production

•Database Size 540GB

•OLTP database 2-3K transactions/sec

•Largest pre-rebuild table Size was 127GB

•Took 5.75 hours to rebuild

•Reclaimed 52GB

•No outage reported for website!

pg_reorg in Production

Table Size Before Reorg

Pg_Reorg Execution time

Bloat Removed

35GB 24m 10sec 13GB

27GB 116m 8 Sec 6GB

31GB 64m 32sec 4GB

44GB 128m 44sec 3GB

Test Case

• Database : PostgreSQL 9.0.2

• OS: Solaris 11

• RAM : 32GB

• work_mem=16MB

• Table Name : Users

• Table Size : 1180 MB

• Indexes Size : 256 MB

• Rows : 6M

postgres=# \d users Table "public.users"

Column | Type | Modifiers --------------+--------------------------+----------- user_id | integer | not null

name | text | creation_tsz | timestamp with time zone |

active | boolean | clicks | double precision |

Indexes: "users_pkey" PRIMARY KEY, btree (user_id) "users_creation_tsz" btree (creation_tsz)

- DELETE 3M ROWS- UPDATE 1M ROWS37

Comparison

CLUSTER VACUUM FULL PG_REORG

Execution Time: 2 min

Table Size: 594MBIndex Size: 128 MB

Execution Time:1 min 50 Sec

Execution Time: 1 min 4 Sec

offline offline online

Built - in Built-in External Tool

Other Tips

•Remove unused indexes

•Reindex indexes as needed

•Partitioned table

•Detach child table -> Cluster child table -> Attach

•Avoid vacuum full (Pre PostgrSQL 9.0)

Monitoring

•Circonus

•Nagios monitoring

•Bloat report

trending

•Cacti

•Circonus

trending

thanks!

•OmniTi

•Robert Treat

•Depesz

•Joshua (End Point)

•PGEast Conference Committee

References•http://www.postgresql.org/docs/9.0/

•http://reorg.projects.postgresql.org/pg_reorg.html

•https://labs.omniti.com/pgtreats/trunk/tools/compact_table

•http://blog.endpoint.com/2010/09/reducing-bloat-without-locking.html

open discussion

•When PostgreSQL will have built in features to rebuild table and index online ?

•Is it worth to change MVCC handling to avoid bloating?

Questions?

•Email : denish@omniti.com

•Twitter : http://twitter.com/#!/denishpatel

P90 X Your Database!!

Documents

The Global Crop Protection Database - Agrobase- · PDF fileThe Global Crop Protection Database ... registration number, company name, active ingredients, ... Egypt 6 x x x x x partially

P90 -- Encargado Del Entrenamiento

Paint Database Guide - X-RitePaint Database Guide Database Preparation Refer to the X-Rite Database preparation and prepare all requested samples Before starting the measurement process,

P90-3S-P90-3S(4PP)GB(0114) - Stuart Group

I éò RgPéò P90~96

FE-LEEM/PEEM P90 Series - SPECS · FE-LEEM/PEEM P90 Series Schematic illustration of the electron optics of FE-LEEM P90 Sample Contrast Aperture Gun Screen Electrostatic Lens Magnetic

NPR of AMERICA, INCACURA P2 ~ P7 AUDI P8 ~ P11 BUICK P90 ~ P153 CADILLAC P90 ~ P153 CHRYSLER P12 ~ P51 CHEVROLET P90 ~ P153 DAEWOO P52 ~ P53 DAIHATSU P54 ~ P55 DODGE P12 ~ P51

P20 THE LAB — P90

P90 x fitness guide

Manual Nikon Coolpix P90 Es

P90 PALLET RACKING SYSTEM - Dexion Store · 2016. 5. 30. · The Dexion team at CSI. P90 Pallet Racking System 04 P90 Pallet Racking System We manufacture an extensive range of components

Informix 9.x Database Design

P90 holman turner

2 Mississippi Queen Humbucker Sized P90’s, Braided

Magpul 2012 Catalog p90-119

ZENworks Mobile Management 3.0.x Database Maintenance€¦ · ZENworks Mobile Management 3.0.x Database Maintenance Database Maintenance 4 Database Maintenance Database cleanup and

Arcam-versterkers A90, P90 en P90/3 Nederlands · A90/P90 2 A90/P90 3 English Using this handbook This handbook has been designed to give you all the information you need to install,

P90 HE Manual 1 Cover - universosniperairsoft.com

Beachbody P90 Diet Guide

RS-P90 - Pioneer Latin · Optical Digital Reference System Universal Digital Preamp/equalizer Owner’s Manual RS-P90 Optical Digital Reference System Universal Digital Preamp