MySQL optimisation Percona LeMug.fr

MySQL Optimization(A couple of different ways.)

Morgan Tockermorgan@percona.com

1Tuesday, March 8, 2011

★ InnoDB?★ MyISAM?★ XtraDB?★ MySQL 4.x?★ MySQL 5.0.x?★ MySQL 5.1?★ MySQL 5.5?★ Something else?

Show of Hands

Tuesday, March 8, 2011

About this presentation

★ Q: “Would you like to talk about MySQL optimization?”★ A: Sure.

★ Optimization can be made at many levels.★ I’m going to show 2-3 parts of the puzzle:

✦ Query Optimization.✦ How InnoDB works.✦ (Time Permitting) Our work at Percona on improving MySQL.

It all starts with EXPLAIN

★ Bookmark this manual page:http://dev.mysql.com/doc/refman/5.1/en/using-explain.html

★ It is the best source for anyone getting started.

Examples come from:

★ IMDB database loaded into InnoDB tables (~5G).

★ Download it and import it for yourself using imdbpy2sql.py:http://imdbpy.sourceforge.net/

First Example

Find the Title Bambi

ALL means tablescan

In this case a sort is required because of the order by

Anticipated number of rows to be examined

Additional filtering may be possible before passing to sort.

Aha! Now add an index:

We must revisit...

Using = for comparison, but not primary key lookup.

Size of the index used (in bytes)

Anticipated number of rows to be examined dropped considerably.

Identified title as a candidate index, chose to use it.

Other ways of accessing

Better type of ‘const’. We can stop when we find one row.

At most one matching row.

In InnoDB the primary key is often much faster than all other keys.

Ignore the time with EXPLAIN. Only look at the time for a query.

Type is range. BETWEEN, IN() and < > are also ranges.

Number of rows to be examined has increased - we are not specific enough.

Why’s that a range?

★ We're looking for titles between BambA and BambZ*★ When we say index in MySQL, we mean trees.

✦ That is, B-Tree/B+Tree/T-Tree.✦ Pretend they're all the same (for simplification).✦ There's no radically different indexing methods in MySQL

unless you play storage engine Bingo**.

12* In reality the range is a little wider** The memory storage engine supports hash indexes

What’s that?

Could this be a range?

No, we can’t traverse.

Do we head left or right here?

LIKE ‘Z%’

LIKE ‘T%’

LIKE ‘The %’

MySQL is (reasonably) smart.

★ It dynamically samples the data to choose which is the better choice - or in some cases uses static statistics*.

★ This helps the optimizer choose:✦ Which indexes will be useful.✦ Which indexes should be avoided.✦ Which is the better index when there is more than one.

19 * To refresh statistics run ANALYZE TABLE table_name;Tuesday, March 8, 2011

Why avoid indexes?

★ B-Trees work like humans search a phone book;✦ Use an index if you want just a few rows.✦ Scan cover-to-cover if you want a large percentage.

★ We benchmarked this on a different schema:

Why avoid indexes (cont.)

Table scan has a relatively fixed cost (red line).

The index has completely different effectiveness depending on how much it can filter.

Hopefully MySQL switches at the right point.

What you should take away:

★ Data is absolutely critical.✦ Development environments should contain sample data

exported from production systems.★ Input values are absolutely critical.

✦ Between two seemingly identical queries, execution plans may be very different.

22See also: http://www.mysqlperformanceblog.com/2009/10/16/how-not-to-find-unused-indexes/

Improve this Query

This number of rows is a guess. It keeps changing between examples.

We’re Spoiled for Choice.

Index on production_year

Might work if...

Index on title(50)

Comparing the two:

★ mysql> EXPLAIN SELECT * from title WHERE title = 'Pilot' AND production_year BETWEEN 2006 and 2009\G

Composite Indexes.

★ What is better?✦ INDEX py_t (production_year, title)✦ INDEX t_py (title, production_year)

Index on py_t

http://www.mysqlperformanceblog.com/2010/01/09/getting-around-optimizer-limitations-with-an-in-list/

Index on t_py

Recommendations

★ Index over multiple columns if it can improve filtering. i.e.✦ GOOD: Only some pilots made between 2006-2009.✦ BAD: All pilots made between 2006-2009.

Recommendations (cont.)

★ Don't know what order to specify the columns?✦ RULE: Think how to filter the fastest. Use that order left to

right.✦ EXCEPTION: If there's a range (><, BETWEEN, %). Those

always go to the RIGHT.

33 http://www.mysqlperformanceblog.com/2010/01/09/getting-around-optimizer-limitations-with-an-in-list/

How InnoDB Works

“Numbers everyone should know”

L1 cache reference 0.5 nsBranch mispredict 5 nsL2 cache reference 7 nsMutex lock/unlock 25 nsMain memory reference 100 nsCompress 1K bytes with Zippy 3,000 nsSend 2K bytes over 1 Gbps network 20,000 nsRead 1 MB sequentially from memory 250,000 nsRound trip within same datacenter 500,000 nsDisk seek 10,000,000 nsRead 1 MB sequentially from disk 20,000,000 nsSend packet CA->Netherlands->CA 150,000,000 ns

See: http://www.linux-mag.com/cache/7589/1.html and Google http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf

About Disks.

★ 10,000,000 ns = 10ms = 100 operations/second.✦ This is about the average for a 7200RPM drive.

★ The actual time has dramatic variation.✦ The variation is because disks are mechanical. ✦ We can much write faster sequentially than randomly.

[default] Everything is buffered!

★ When you write to a file, here’s what happens in the Operating System:

Block 9, 10, 1, 4, 200, 5.

Block 1, 4, 5, 9, 10, 200

What happens to this buffer if we loose power?

The OS provides a way!

★ $ man fsync

Synopsis#include <unistd.h>int fsync(int fd);int fdatasync(int fd);

Descriptionfsync() transfers ("flushes") all modified in-core data of (i.e., modified buffer cache pages for) the file referred to by the file descriptor fd to the disk device (or other permanent storage device) where that file resides. The call blocks until the device reports that the transfer has completed. It also flushes metadata information associated with the file (see stat(2)).

Hint: MyISAM just writes to the OS buffer and has no durability.

http://thunk.org/tytso/blog/2009/03/15/dont-fear-the-fsync/Tuesday, March 8, 2011

Knowing this:

★ InnoDB wants to try and reduce random IO.★ It can not (safely) rely on the operating system’s write

buffering and be ACID compliant.✦ .. and InnoDB algorithms have to compensate.

Basic Operation (High Level)

Log Files

SELECT * FROM CityWHERE CountryCode=ʼAUSʼ

Buffer PoolTablespace

Log Files

Basic Operation (cont.)

Log Files

UPDATE City SET name = 'Morgansville'

WHERE name = 'Brisbane' AND CountryCode='AUS'

Log Files

Why don’t we update?

★ This is an optimization!✦ The log file IO is sequential and much cheaper than live

updates.✦ The IO for the eventual updates to the tablespace can be

optimized as well. ★ Provided that you saved enough to recover, this

shouldn’t matter should it?

More on Logs...

★ Logs are only used during recovery.✦ Not even read when we need to write down

dirty pages!★ To figure out which pages need to be evicted we have two

lists - the flush list and the LRU.★ Log activities are all assigned a LSN (log sequence

number).

Log Files

Log Files, Checkpoints, etc.

★ Most database systems work this way:✦ In Oracle the transaction logs are called “Redo Logs”.

★ The background process of syncing dirty pages is normally referred to as a “Checkpoint”.

★ InnoDB has fuzzy checkpointing.

Log Writing

★ You can change increase innodb_log_file_size. This will allow InnoDB to “smooth out” background IO for longer.

✦ Tip: Optionally you could change innodb_log_files_in_group as well. Be aware that your effective log file is innodb_log_file_size * innodb_log_files_in_group

Log Writing (cont.)

★ You can also change innodb_flush_log_at_trx_commit to 0 or 2 to reduce the durability requirements of this write.✦ Requires less flushing - particularly helpful on systems

without writeback caches!★ innodb_log_buffer_size may also help buffer

changes for longer before writing to the logs.✦ Very workload dependent - tends to be more helpful for

writing big TEXT/BLOB changes.

In summary

★ On commit, the log has to be flushed to guarantee changes.

✦ Nothing else has to be done.★ What visibility do we have into these operations?★ How do we decide how much background work to do per

second?★ What happens if we fall behind in background work?

Enhancements via Percona

Terminology

Oracle Product License Percona Equivalent Product

License

MySQL Server GPL Percona Server GPL

The InnoDB Storage Engine (Plugin edition)

GPL The XtraDB Storage Engine

InnoDB Hot Backup Commercial XtraBackup GPL

(GPL = Completely free for you to use. Support not included.)

Performance Improvements

Improved Buffer Pool Scalability

Improved IO Path + adaptive checkpointing

Improved Rollback Segment Scalability*Separate purge

thread

Data dictionary memory consumption controls

Increased number of undo slots*

Faster page checksums*

Support for different page sizes*

* Changes on disk format (not backwards compatible)

Insert buffer controls

Completely disable the query cache.

Remove excess fcntl calls

Per session configuration of innodb_flush_log_at_trx_commit

Separate location of double write buffer*

Strip comments before using query cache

Transaction logs larger than 4G supported.

Improved Buffer Pool Scalability

★ Additional patches to what is already available in the InnoDB Plugin.✦ Splits the buffer pool mutex into smaller mutexes:

• Flush list mutex• LRU mutex• Free mutex• hash mutex

Data Dictionary control

★ Once an InnoDB table is opened it is never freed from the in-memory data dictionary (which is unlimited in size).

★ XtraDB introduces:✦ --innodb_dict_size_limit - a configuration item in

bytes.✦ innodb_dict_tables - a status variable for number

entries in the cache.

Undo Slots

★ In the built-in InnoDB, the number of undo slots is limited to 1024.✦ This means the number of open transactions is limited to

1023. See: http://bugs.mysql.com/bug.php?id=26590✦ Some statements require 2 undo slots.

★ In XtraDB, this is expanded to 4072 with --innodb_extra_undoslots=1.

53 Warning: This is binary format incompatible!Tuesday, March 8, 2011

Rollback Segments

★ In XtraDB, it’s also possible to have more than one rollback segment.✦ Each segment contains undo slots.

★ Configuration is --innodb-extra-rsegments=N★ This has the added effect of reducing mutex contention

on the rollback segment:✦ “Mutex at 0×1b3e3e78 created file trx/trx0rseg.c line 167″

http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_extra_rseghttp://www.mysqlperformanceblog.com/2009/10/14/tuning-for-heavy-writing-workloads/

Warning: This is binary format incompatible!Tuesday, March 8, 2011

Fast Checksums

★ The InnoDB page checksum computation is slower than it needs to be.

★ XtraDB has the option to use a faster checksum format.

Different Page Sizes

★ XtraDB now has support for different page sizes - 4K, 8K, 16K.

Separate Purge Thread

★ Cleans up a long history list length faster:

57See: http://www.mysqlperformanceblog.com/2009/10/14/tuning-for-heavy-writing-workloads/

Usability Enhancements

Show contents of the buffer pool

Import / export of innodb_file_per_table tables

Import / export of buffer pool contents

Transactional Replication

Better handling of corrupted tables

Store buffer pool in shared memory segment

Save index statistics between restarts

Advise in processlist when waiting on Query cache mutex.

Improved slow query log

User / Index / Table statistics

Disable automatic statistics regeneration

Show data dictionary

Deadlock counter

Show Temporary Tables

Log connection errors

Retain query response time distribution.

Show Buffer Pool Contents

mysql> SELECT d.*,round(100*cnt*16384/(data_length+index_length),2) fit FROM (SELECT schema_name,table_name,count(*) cnt,sum(dirty),sum(hashed) FROM INNODB_BUFFER_POOL_PAGES_INDEX GROUP BY schema_name,table_name ORDER BY cnt DESC LIMIT 20) d JOIN TABLES ON (TABLES.table_schema=d.schema_name AND TABLES.table_name=d.table_name);+-------------+---------------------+---------+------------+-------------+--------+| schema_name | table_name | cnt | sum(dirty) | sum(hashed) | fit |+-------------+---------------------+---------+------------+-------------+--------+| db | table1 | 1699133 | 13296 | 385841 | 87.49 || db | table2 | 1173272 | 17399 | 11099 | 98.42 || db | table3 | 916641 | 7849 | 15316 | 94.77 || db | table4 | 86999 | 1555 | 75554 | 87.42 || db | table5 | 32701 | 7997 | 30082 | 91.61 || db | table6 | 31990 | 4495 | 25681 | 102.97 || db | table7 | 1 | 0 | 0 | 100.00 |+-------------+---------------------+---------+------------+-------------+--------+7 rows in set (26.45 sec)

Source: http://www.mysqlperformanceblog.com/2010/03/26/tables-fit-buffer-poo/* Command may differ slightly between versions.

Save Buffer Pool Contents

★ Export the contents of the buffer pool to a file called ‘ib_lru_dump’ in the data directory:✦ SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*!XTRA_LRU_DUMP*/;

★ Restored ib_lru_dump:✦ SELECT * FROM information_schema.XTRADB_ADMIN_COMMAND /*!XTRA_LRU_RESTORE*/;

60Note: Not the actual contents - it takes 8 bytes to remember the address of a 16K page.

Import/Export tables

★ Because --innodb-file-per-table still has information (data dictionary, undo) in the global tablespace you can’t just back it up by itself.

★ With a new setting, --innodb_expand_import=1, this is no longer the case.

★ Tip: The import/export still has to be done with XtraBackup. Documentation available here:http://www.percona.com/docs/wiki/percona-xtradb:patch:innodb_expand_import

The Slow Query Log

$ mysql -e “SET GLOBAL log_slow_verbosity = ‘full’;”$ tail /var/log/mysql.slow..# Time: 100924 13:58:47# User@Host: root[root] @ localhost []# Thread_id: 10 Schema: imdb Last_errno: 0 Killed: 0# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608 Rows_affected: 0 Rows_read: 1# Bytes_sent: 131 Tmp_tables: 1 Tmp_disk_tables: 1 Tmp_table_sizes: 25194923# InnoDB_trx_id: 1403# QC_Hit: No Full_scan: Yes Full_join: No Tmp_table: Yes Tmp_table_on_disk: Yes# Filesort: Yes Filesort_on_disk: Yes Merge_passes: 5# InnoDB_IO_r_ops: 1064749 InnoDB_IO_r_bytes: 17444847616 InnoDB_IO_r_wait: 26.935662# InnoDB_rec_lock_wait: 0.000000 InnoDB_queue_wait: 0.000000# InnoDB_pages_distinct: 65329SET timestamp=1285336727;select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;

$ tail /var/log/mysql.slow..# Time: 100924 13:58:47# User@Host: root[root] @ localhost []# Query_time: 399.563977 Lock_time: 0.000110 Rows_sent: 1 Rows_examined: 46313608SET timestamp=1285336727;select STRAIGHT_JOIN count(*) as c, person_id FROM cast_info FORCE INDEX(person_id) INNER JOIN title ON (cast_info.movie_id=title.id) WHERE title.kind_id = 1 GROUP BY cast_info.person_id ORDER by c DESC LIMIT 1;

MySQL Server

Percona Server

User Statistics

MySQL Server

Percona Server

( Not Possible )

(Related) Xtrabackup Features

★ Report on fragmentation of indexes:★ $ xtrabackup --stats --tables=art.link* --datadir=/mnt/data/mysql/...table: art/link_out104, index: PRIMARY, space id: 12, root page 3leaf pages: recs=25958413, pages=497839, data=7492026403 bytes, data/pages=91%...

http://www.mysqlperformanceblog.com/2009/09/14/statistics-of-innodb-tables-and-indexes-available-in-xtrabackup/

Basic Operation (again.)

Log Files

Set innodb_buffer_pool_size to “50-80%” of memory.

Log Files

Increase the size of the log files (innodb_log_file_size and innodb_log_files_in_group)

Log Files

Set innodb_flush_log_at_trx_commit=2 if durability is not as important.

Log Files

Typically use innodb_flush_method=O_DIRECT if using a Hardware RAID controller.

Log Files

(Possibly) Move the log files to separate spindles (sequential IO)

Log Files

(Possibly) Move the log files to separate spindles (sequential IO)

If innodb_log_waits > 0 the buffer being filled (innodb_log_buffer_size) before writing to the log files may be too small.

The End.

★ Questions?

MySQL optimisation Percona LeMug.fr

Technology

MySQL In the Cloud - Percona€¦ · MySQL In the Cloud Migraon, Best Prac

Synchronous Replication For MySQL with Percona XtraDB Cluster

MySQL Geographic Features in - Percona

MySQL Monitoring - Percona · MySQL Monitoring Finding Meaning in Beautiful Graphs Peter Zaitsev CEO, Percona Oracle Open World September 20, 2016. 2 About Percona ... Or cloud hosted

Running MySQL on Linux - Percona · MySQL Versions • Newer versions scale better • MySQL 5.6 is most scalable • Percona Server brings even more improvements • Now GA! •

MySQL In the Cloud - Percona MySQL In the Cloud ? 4 © 2017 Percona ... AWS EC2, S3, EBS IaaS ... Source: 39 © 2017 Percona High

Percona xtrabackup - MySQL Meetup @ Mumbai

Implementing MySQL Security Dublin 2017 Percona Live ... · #PerconaLive @RonaldBradford @bytebot Implementing MySQL Security Features Ronald Bradford, Colin Charles Percona Live

Percona – The Database Performance Experts · Peter Zaitsev, CEO, Percona February 24, 2016 Percona Technical Webinars Using Grafana for MySQL Monitoring

Deploying MySQL in Production - Percona...2 © 2017 Percona Deploying MySQL in Production • Installation • Configuration (OS and MySQL) • Backups • Monitoring • Before you

Integrate Percona MySQL...EventTracker: Integrate Percona MySQL 3 Introduction Percona MySQL is an open source software company specializing in . MySQL support, consulting, managed

Percona Live: Various MySQL High Availability (HA) Solutions

At Rest Encryption with MySQL & Vault - Percona...Tate McDaniel Percona Senior MySQL DBA with 15 years in the industry, 2 years at Percona. Expert at query tuning, schema design, and

Choosing a MySQL High Availability Solution - Percona · Marcos Albe, Percona Inc. Live Webinar June 2017 Choosing a MySQL High Availability Solution

Percona Software & ServicesUpdate · Percona Server for MySQL MariaDB Percona XtraDBCluster GaleraCluster for MySQL MariaDBGaleraCluster MongoDB Percona Server for MongoDB Amazon

How MySQL and NoSQL Coexist - Percona · How MySQL and NoSQL Coexist Matt Yonkovit - Percona. DIAMOND SPONSORSHIPS THANK YOU TO OUR ... Oracle …

Best Practices for MySQL Scalability - Percona

Percona tool kit for MySQL DBA's

Percona Toolkit for Effective MySQL Administration

Turbocharge your MySQL - Percona€¦ · Turbocharge your MySQL analytics with ElasticSearch Guillaume Lefranc Data & Infrastructure Architect, Productsup GmbH Percona Live Europe