Addressing vendor weaknesses in user space (Robert Treat)

Addressing Vendor Weaknesses in User-Space

ROBERT TREAT,OmniTI

Highload++ 2011

@robtreat2xzilla.net

+Robert Treat1

Monday, October 3, 11

Who Am I?

OMNTI - Internet Scalability ConsultantsLead Database Operations

Who Am I?

“Large Scale”

Who Am I?

“Large Scale”

High TransactionsTB+ Data

Who Am I?

“Large Scale”

High TransactionsTB+ Data

Mission Critical

Who Am I?

Database Operations @OMNTI

PostgresMySQLOracle& More

Postgres for Scalability

Traditional RDBMSHighly ExtensibleRuns Everywhere

Talks To Everything“BSD” Licensed

15+ Years DevelopmentOpen Development Community

The Bloat Problem

Data Footprint Can Be Critical To Performance

The Bloat Problem

Size On Disk Affects The Needs OfRAM, Disk Speed, Storage

The Bloat Problem

“Bloat” is unused, wasted disk space, used by the database,

but not needed for actual data storage

The Bloat Problem

“Bloat” is unused, wasted disk space, taken up by the database,

but not needed for actual data storage

MVCC Architecture

Multiversion Concurrency Control (MVCC) allows Postgres to offer high concurrency even during significant database read/write activity. MVCC specifically offers behavior where "readers never block writers, and writers never block readers".

MVCC Architecture

• Oracle

• MySQL (InnoDB)

• Informix

• Firebird

• MSSQL (optional)

MVCC Architecture

• Oracle

• MySQL (InnoDB)

• Informix

• Firebird

• MSSQL (optional)

• CouchDB

“Bloat” Manifests Differently, But Is Common

• MongoDB (deletes, some updates)

• dump/restore

• mongod --repair

• db.runCommand( { compact : 'mycollectionname' } )

• Lucene (updates)

• Hadoop / HDFS (small files)

Postgres MVCC Architecture

• Implemented Postgres 6.5• 1999, Vadim Mikheev

• MVCC Unmasked • http://momjian.us/main/writings/pgsql/mvcc.pdf

• Postgres maintains global transaction counters

• Keeps track of transaction counter per row for• creating transaction• removing transaction

• Using these counters, Postgres allows different transactions to see different rows, based on visibility rules.

• Postgres maintains global transaction counters

• Keeps track of transaction counter per row for• creating transaction• removing transaction

• Using these counters, Postgres allows different transactions to see different rows, based on visibility rules.

Transaction Reading An Old RowDoesn’t Block Transaction Writing A Row

MVCC Architecture

user_id X42Create 32 Expire

INSERT

MVCC Architecture

INSERT

DELETEuser_id X42Create 32 Expire 38

MVCC Architecture

user_id X69Create 43 Expire 56

OLD(delete)

NEW(insert)

UPDATE

MVCC Architecture

<~~ DEAD ROW

<~~ VISIBLE ROW

Clean Up / Bloat

MVCC Architecture

<~~ DEAD ROW

<~~ VISIBLE ROW

Clean Up / Bloat

Speed Up SQL Commands ByDealing With Clean Up Later

How Postgres Deals With Bloat

• Heap-Only-Tuples (HOT)• On-The-Fly, Per Page Cleanup• Marks Given Row’s Space Reusable• Update Only

How Postgres Deals With Bloat

• Heap-Only-Tuples (HOT)• On-The-Fly, Per Page Cleanup• Marks Given Row’s Space Reusable• Update Only

• VACUUM• Non-Blocking Bulk Cleanup• Removes End-Of-File Pages• “autovacuum” Process Monitors Tables

Problems With Automatic Cleanup

• HOT• Update Only• Doesn’t Work With Changing Index Data

Problems With Automatic Cleanup

• HOT• Update Only• Doesn’t Work When Changing Index Data

• VACUUM• Must Wait For Long Transactions To Complete• Costs I/O, Can Only Work So Fast• Can’t Remove Non End-Of-File Pages • Leaves A “High Water Mark”

Dealing With Bloat - The Hard Way

• VACUUM FULL / CLUSTER• The Good

• Reclaims All “Dead Rows”

Dealing With Bloat - The Hard Way

• VACUUM FULL / CLUSTER• The Good

• Reclaims All “Dead Rows”

• The Bad• Exclusive Lock• Rewrite All Data In Tables• Needs Working Space• Heavy I/O

Monitoring Your Bloat

• check_postgres.pl• Nagios plugin• Compares physical size to row size estimates• http://bucardo.org/wiki/Check_postgres

• “bloat report”• Script to measure table/index bloat• Compares physical size to row size estimates• http://labs.omniti.com/labs/pgtreats/browser/trunk/tools/

Dealing With Bloat In Userspace

• Solving MVCC Bloat Is A “Hard Problem”• Even a good solution would be hard to implement in core

Dealing With Bloat In Userspace

• Solving MVCC Bloat Is A “Hard Problem”• Even a good solution would be hard to implement in core

• Can we build a tool in user space?• Develop solution quicker• Easier to deploy and maintain • Provide a prototype for future development

Dealing With Bloat Redux

• Updating A Row Rewrites Data To New Location

• Use Vacuum To Mark Old Rows “Reusable”

• Use Vacuum To Mark Old Rows “Reusable”• Update Row To Rewrite Data At “Front” Of Page

• Use Vacuum To Mark Old Rows “Reusable”• Update Row To Rewrite Data At “Front” Of Page• Use Vacuum To Reclaim Space From End Of File

• Put A Script On It• https://labs.omniti.com/pgtreats/trunk/tools/compact_table

• “Compact Table”• Requires Lots of Time, I/O• Often Causes Heavy Index Bloat• Heavy Concurrency Bloats Faster Than We Can Recover It

Dealing With Bloat For Real!

• Enter “pg_reorg”

• Enter “pg_reorg”• Vacuum / Cluster Replacement

• Enter “pg_reorg”• Vacuum / Cluster Replacement• Command Line Tool

• Enter “pg_reorg”• Vacuum / Cluster Replacement• Command Line Tool• Online Table Rewrite

• Uses Minimal Locking

• Uses Minimal Locking• Developed By NTT

• Uses Minimal Locking• Developed By NTT• BSD Licensed• C Code• http://pgfoundry.org/projects/reorg/

How pg_reorg Works

• Create a log table for changes• Create triggers on the old table to log changes (I/U/D)• Create a new table with a copy of all data in old table• Create all indexes on the new table• Apply all changes from the log table to the new table• Modify the system catalogs information about table files• Drop old table, leaving new table in it’s place

How pg_reorg Works

• Create a log table for changes• Create triggers on the old table to log changes• Create a new table with a copy of all data in old table• Create all indexes on the new table• Apply all changes from the log table to the new table• MODIFY THE SYSTEM CATALOGS INFORMATION ABOUT THE TABLE FILES (!!!)• Drop old table, leaving the new table in it’s place

Open Source Code

The Power Is In Your Hands

Look At CodeExamine the SQL

(User Space Is Really Visible)

What Does Testing Look Like?

Create Some Tables, Create Artificial Bloat,

run pg_reorg

What Does Testing Look Like?

Create Some Tables, Create Artificial Bloat,

run pg_reorg

Test In “Prod”

Find Some Bloated Tables,Make Backup Of Tables,

Cross Fingers, pg_reorg

Test In “Prod”

Find Some Bloated Tables,Make Backup Of Tables,

Cross Fingers, pg_reorg

Eventually You Have To Use ItOn Something That Matters

pg_reorg In The Real World

• Production Database (OLTP) • 540GB Size• 2000 TPS (off-peak time, multiple statements)• Largest Table (pre-reorg) 127GB

• Rebuild Stats• 5.75 Hours To Rebuild• Reclaimed 52GB Disk Space • No outages reported for Website/API’s

Return Of The Jedi

“your overconfidence is your weakness.”

-Luke Skywalker

“your faith in your friends is yours.”

-Emperor Palpatine

Sometimes You Can Have Both

Trust in NTT’s Code == faith in friends

Success in production == overconfidence

When Good pg_reorgs Go Bad!

WARNING: unexpected attrdef record found for attr 61 of rel orders

WARNING: 1 attrdef record(s) missing for rel orders

WARNING: unexpected attrdef record found for attr 61 of rel orders

WARNING: 1 attrdef record(s) missing for rel orders

Yes, On A Production SystemYes, Trying To Take 1000’s of Orders Per Second

create table test ( a int4, b int4 default 2112, c bool);

Postgres internals track defaults / constraints based on column position “2”, not column name “b”

If you drop column “a” and then do pg_reorg, column “c” is now column “2”, and default 2112 is on boolean

This Is Fair - pg_reorg hacks the system tables68

Basic Fix: Drop All Defaults And Recreate

Alternative Fix: Hack System Catalogs Some More

Haven’t we had enough system catalog hacking

for now?

“now, if you'll excuse me, I'll go away and have a heart attack.”

What Next?

Report Problem To Mailing ListSubmit A Patch

Ultimately The Problem Is FixedEveryone’s Happy?

Hackers Discussion

Postgres Development Community Is Funny

Sometimes Hard To Get Them To Recognize Problems

Not Everyone See Online Rebuild As A Big Problem

Hackers Discussion

Postgres Development Community Is Funny

Sometimes Hard To Get Them To Recognize Problems

Not Everyone See Online Rebuild As A Big Problem

In All The Fairness, Not Everyone Has This Problem

Hackers Discussion

Hackers Meeting 2011, Discussion On Internal Queuing System

Could Be Used As Underlying Basis For On-Line Rebuilding

Until Then...

pg_reorg Is A Great Tool!Best Option For Difficult Situation

Just Be Careful!

Highload++NTT

OmniTIPostgres Community

Momjian, Depesz, Patel, Kocoloski

xzilla.net@robtreat2

+ Robert Treat

THANKS!

Addressing vendor weaknesses in user space (Robert Treat)

Technology

Addressing Structural Weaknesses of the Kurdistan Region ......7 Addressing Structural Weaknesses of the Kurdistan Region’s Economy The areas where weaknesses persist include: 1

Chapter 6 Weaknesses Exploited

Chapter 1 lon weaknesses

Presentació pública Treat or Treat

13A strengths vs weaknesses

Weaknesses As a Writer

Friendship weaknesses and friends

Assessing Strengths and Weaknesses

Discovering Mac OS X Discovering Mac OS X Weaknesses Weaknesses

Investor Presentation - Dicker Data Data... · 2019-04-29 · Vendor 3 Vendor 3 Vendor 3 Vendor 3 Vendor 4 Vendor 4 Vendor 4 Vendor 4 Vendor 4 Vendor 4 Vendor 5 Vendor 5 Vendor 5

Vibration Switch Weaknesses

THE RESPONSIBILITIES, STRENGTHS, AND WEAKNESSES OF … · THE RESPONSIBILITIES, STRENGTHS, AND WEAKNESSES OF ... Section and the responsibilities, strengths, and weaknesses of Kitchen

Vendor to Vendor Migrations - PROSTEP INC.webinarsbyprostep.com/.../07/Vendor-to-Vendor-Migrations.pdf · - The importance of using neutral software in vendor to vendor migrations

Vendor Evaluation by configuring Vendor Ratings and Vendor Scorecard:

Vendor Processing -- Vendor Managementfinance.columbia.edu/.../Vendor_Processing_Vendor_Management_TRAIN.pdfTraining Guide Vendor Processing -- Vendor Management Page iii Table of

Understanding Vendor Risk And Analyzing the SSAE No. 16 · Weaknesses in SSAE No. 16 • The auditee (not you or even the auditor) will identify the review areas ‒ Loan application

Increase your knowledge! - teachchildrenesl.com · Trick Treat Trick Treat Trick Treat Treat Trick Treat Trick Treat Trick Trick Treat Trick Treat Trick Treat Back of Cards Print

Vendor Name Vendor Address Vendor Contact Vendor Phone

Identifying Community Strengths & Weaknesses

PA MENU - malmaison.com · HANDS & FEET/ Nails have their own DNA, their own specific strengths and weaknesses – just like hair and skin. It is impossible to treat all nails the