Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd. [email protected]

Partition, Compress, Archive and PurgeKeep Your System on the Road

David KurtzGo-Faster Consultancy Ltd.

[email protected]

www.go-faster.co.uk

Who Am I?

• Oracle Database Specialist– Independent consultant

• Performance tuning– PeopleSoft ERP– Oracle RDBMS

• Book– www.psftdba.com

• OakTable•

UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge

©2013 www.go-faster.co.uk 2

http://www.psftdba.com/

It depends…

• The answer to all questions should be deemed to be implicitly prefixed ‘it depends’ if they are not already explicitly so prefixed. The reasons for this include, but are not limited to:– There may be exceptions where the answer is either

false or not completely true– There may be exceptions to the exceptions and so on ad

infinitum.– The question may not explicitly scope all the conditions

upon which the answer depends

©2013 www.go-faster.co.uk 3UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge

PeopleSoft HR, T&L, GP @ Morrisons

• 130,000 employees– 100,000 in stores

• Scheduled hours– Clock in/out – T&L matches reported hours with schedules to

determine payable time– Legacy T&L feed from Manufacturing and

Distribution.– Payable time passed to Global Payroll.





0.91Tb/yr

Two Challenges

• Performance– Big Tables

• (for OLTP)• Eg payroll accumulator

table 2.4 billion rows!

– Poor/Variable Performance• Various processes and

reports suffering • Critical Processes

• Database Size– >4TB– Growth ≈ 1TB/year– Backup

• Size/Contention

– Cloning• Dev/Test environments



Throw Away

• When we say the ‘database is slow’, often:– It is fetching data from disk, reading it, and deciding

it isn’t something it wants and throwing it away.– It would be better if we had not spent time and

resource fetching it in the first place

• The fastest way to do anything is not to do it at all.– Cary Millsap



4 Techniques

• Partition– Breaking tables and indexes into managable pieces

• Compress– Squeezing more data into the same space

• Archive– Get the data out of the live application tables and put

it somewhere else.

• Purge– Expunge the data from the database



PARTITIONING



Partitioning



Partitioning decisions driven by Application

• Global Payroll– ‘streamed’ processing– RANGE on EMPLID to match streaming

• Financials General Ledger– FISCAL_YEAR & ACCOUNTING_PERIOD– Monthly/Quarterly/Annual partitions.

• Other Financials Modules?– Less Obvious



Partitioning is a Pre-requisite

• Links logical data value to physical location of data in the database.

• Time-based partitioning– Future / Current / Historical

• Partition-wise operations– Compress historical partitions– Archive by partition exchange– Purge by dropping whole partitions.– Less Redo. Faster.



Delete rows –v- Drop Partition?

• Free space is left in data block.

• Data block remain part of table/index

• Still scanned by queries.

• Still backed up by RMAN

• No longer in a database object. So not referenced.

• Free space is available in tablespace.– Still backed up

• If the space at end of the datafile it can be trimmed off by resize.– No longer backed up



Managing Partitioning in PeopleSoft

• Partitioning strategies– Described own meta-data tables

• PL/SQL Utility package – generate DDL scripts.– Driven by PeopleTools tables and our own meta-data

• Application Designer from PT8.51 will preserve some existing partitioning

– See Managing Oracle Table Partitioning in PeopleSoft Applications with GFC_PSPART Package• http://www.go-faster.co.uk/gfc_pspart.manual.pdf



http://www.go-faster.co.uk/gfc_pspart.manual.pdf



Partitioning PeopleSoft at Morrisons

• Global Payroll– 46 tables range on EMPLID, matching 32 streams– List sub-partition largest tables by pay period (CAL_RUN_ID).– 24 archive tables, range partitioned on pay period into tax-year partitions.– 2 custom GP reporting tables

• Schedules & Time and Labor– 10 tables in weekly range partitions – of which Reported Time table hash sub-partitioned

• Audit Tables– 28 monthly range partitions on audit timestamp

• Security Tables– Separate partitions for different business units



Partitioning Trick

• Timezone Table– Small, but accessed VERY frequently by T&L– 1 partition for GMT only

• Only 1 row

– 1 partition for CET only• Only 1 row

– 1 partition for the rest– Less CPU to read first row in block than second

etc.UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge


Licencing of Partitioning

• Oracle RDBMS Enterprise Edition only– And then separately licenced

• Also a cost option on other database platforms.

• No licence implication for PeopleSoft.– Partitioning does not invalidate any aspect of

your support.



COMPRESSION



Oracle RDBMS Compression

• Basic Compression– Not separately licenced– Only option in 10g

• Advanced Compression• Out of scope of this presentation

– Licenced option– Available in 11g

• Hybrid Columnar Compression (HCC)• Also out of scope of this presentation.

– Exadata and ZFS filesystem only.



Simple Compression

• Block Dictionary Encoding– Data values stored only stored once in array in

data block.– Pointer from row of data to array of values.

• Shorter Rows, more rows per block, fewer Blocks, less I/O

• More CPU to read row.



Table Compression

• Specified at table or partition level with DDL– Sub-partition level inherits from partition level

• Compression only occurs – On creation of the object

• ALTER TABLE … COMPRESS

– Direct path insert into object– CPU overhead during compression and read

• Decompression occurs – during any other DML



Index Compression

• Apply to all or none of index • Index remains compressed

– DML does not decompress– Performance overhead

• We have not compressed many live application indexes.

• We have compressed all archive tables.

• Table compression does not affect index size.



What do I compress?

• Depends on application, business, and partitioning strategy.– Audit data partitions: as soon as month ends– Payroll result data: when pay period is closed

for 1 month– T&L: after 6 months



Compression Gotcha!

• During – ALTER TABLE PARTITION … MOVE …

COMPRESS.

• Local index partitions can momentarily become unusable before they go back to being usable.

• This can cause application errors– Choose when to compress carefully, or do it in an

outage!UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge


When can I compress my data?

• When it is no longer updated by application.– Because DML uncompresses it– But that may be well be you can archive/purge.

• Written custom PL/SQL utility package– Meta-data rules to determine when partition no

longer updated.– Needs knowledge of business/application.– Incrementally compress static partitions.UKOUG PeopleSoft 2013 - Partition,

Compress, Archive, Purge©2013 www.go-faster.co.uk 26

How much compression?

• It depends on your data!• We have see a factor 2x – 5x on tables

– (50%-80% reduction in size)

• Indexes are less compressible– Indexes with more columns can be more

compressible



PL/SQL Utility Package

• GFC_ARCH_MGMT– DBA run management utility– Moves partitions to correct tablespace– Compresses/Decompresses partitions per

specification in meta-data.



ARCHIVE & PURGE



Archive & Purge

• Archive:– Move data from live

application table to another table.

– In this case another to table in the same database.

– Read-only access provided to archive tables.

• Purge:– Expunge data from the

database.



Archive/Purge

• For each table or set of tables– Define a policy that specifies when data is

archived or purged.• Compression too, but that has no functional impact.

• Discuss with business– Check legal requirements

• PAYE 3 years, Company Accounts 6 years



‘But my data will disappear’

• That may be a good thing– If you keep data you have to look after it

responsibly.– Pay for the disk to store it/back it up.

• Read-only access via PS/Query tool to archive data.

• A very few components and reports access archive tables– Development cost of customisation



Example Archive/Purge Policies

– GP: archive 2 rolling years, purge after 8 tax years

– Schedules, purge at 56 weeks– T&L, purge at 108 weeks– Audit: various:

• Archive after 3 months, 1, 3 year• Purge 6 months, 1 year, 2 years, never



Archive Method

• Partition Management– Exchange/Drop whole

Partitions• Time based partitioning

– Avoid rewriting data– Avoids redo

• Faster

– Compression must match

• Some archive table indexes not compressed.

• Row-by-Row– Non-time based

partitioning• GP

– Can be done on-line– Can compress data

during rewrite– Different partitioning

strategy in archive and live.



PeopleTools Archive Utility

• Row-by-Row processing only– Slow and can’t reclaim space– Can’t compress data during

copy

• Additional key columns in archive table– Enlarges archive tables– Causes performance problems

with index access

• Still need to drop empty partitions.

• Generic– Meta-data driven

• Sets of tables to archive• Corresponding archive

tables• GP set delivered



PL/SQL Archive/Purge Utility

• Custom PL/SQL Package– Driven by same meta-data as partitioning utility– Archive by partition exchange– Purge by dropping partition.– Called from custom Application Engine

• So archiving can be done via application.

– Writes • To PeopleSoft Message Log• Various Oracle trace/log files



‘No-Archive’ Rules

• Can only archive 99% of partitions– Affects some Schedule tables– Utility that does partition exchanges retains

some rows by DML



Custom Archive Utility

• Based on delivered PeopleSoft Archive Utility

• We only use it for GP archiving• Copies data in direct path

– Faster– Can compress during copy

• Calls PL/SQL utility to drop partitions to save deleting data.



‘PSARCH’ Schema Design

• 2nd schema to hold archive tables– Only tables archived by partition exchange– Same name as live tables– Same local indexes as live

• Benefits– SYSADM cannot update/delete from these tables.

• Drawbacks– Views to make them accessible to PS/Query.– Can’t manage indexes through Application Designer.



Tablespace Design

• No impact on performance – Because all on same tier of disk

• Do not keep objects with dissimilar purge policies in same tablespaces– You can drop empty tablespaces and recover space.

• Periodic tablespaces make sense without purge policy

• With a purge policy might just have one tablespace.



Wrong Tablespace Design

• Before archive policy– Weekly T&L and Schedule partitions in same monthly

tablespaces

• Archive Policy– T&L: 108 weeks– Schedules: 56 weeks

• Schedules populated first, T&L later.– Schedules tend to be nearer start of tablespace– Schedules purged first leaving hole than cannot be

trimmed off.UKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge


Free Space Fragmentation

• Free Space left in middle of tablespace – Dropping objects– Compressing objects can leave holes

• Less Severe since Oracle 9i– Uniform extent sizes– Bitmap space map





Oracle RMAN Backup

• Backs up block with SCN– If an object is dropped blocks still have SCN– Empty blocks can still be backed up.

• Can only reduce backup size by trimming free space from end of data files.



Defragmentation of Free Space

• PL/SQL utility to rebuild objects nearest end of data file into spaces in the middle of the tables– Needs to be done during an application outage.

• Frees space at end of data file that can be trimmed off.– Thus reducing size of database and backup.



Oracle Flashback Database

• SQL Rollback– We are used to the idea of rolling back transaction

that have not been committed.• Flashback database

– Generates additional flashback logging– Rollback committed transaction and DDL

• but not through a drop/resize data file

– Whole database option• Highly effective in testing archive/purgeUKOUG PeopleSoft 2013 - Partition, Compress, Archive, Purge


Future: Oracle 11g

• Automatic Partition Creation• Range-Range partitioning• Advanced Compression

– Licenced Option– Supports DML

• Active Data-Guard– Off-load query activity to standby database



PROGRESS







0.91Tb/yr

0.48Tb/yr



Achievements

• Database Size– 4.5TB => <2.25TB (with GP purge)

• Still have to purge old schedule and T&L data

• Expected Performance Improvements– GP calculation and other processing– T&L processing– Financial Reports



Conclusion

• It depends…



QUESTIONS?Nullius in verba



http://en.wikipedia.org/wiki/Royal_Society

Documents

Partition, Compress, Archive and Purge Keep Your System on the Road David Kurtz Go-Faster Consultancy Ltd. [email protected]