View
1.050
Download
2
Category
Tags:
Preview:
Citation preview
Performance Schema in MySQL Danil Zburivsky
MySQL DBA and Team Lead at Pythian
Tuesday, October 23, 12
About myself• MySQL DBA and Team
Lead at Pythian
• Managing dozens of customers and thousands of MySQL servers
• http://www.pythian.com/news/author/zburivsky/
• @zburivsky
Tuesday, October 23, 12
Debugging MySQL Performance
Tuesday, October 23, 12
MySQL is a great database, but instrumentation sucks
SHOW ENGINE INNODB STATUS
Slow query log
OS STATSSHOW
PROCESSLIST
Tuesday, October 23, 12
I think there is a problem with database...
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
• Collect data during incident
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
• Collect data during incident
• Collect more data
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
• Collect data during incident
• Collect more data
• Collect as much data as you can!
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
• Collect data during incident
• Collect more data
• Collect as much data as you can!
• Try to make sense of it
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
• Collect data during incident
• Collect more data
• Collect as much data as you can!
• Try to make sense of it
• Tune settings, SQL, hardware. Did it help?
Tuesday, October 23, 12
I think there is a problem with database...
• Identify timeframe (use trending tools)
• Collect data during incident
• Collect more data
• Collect as much data as you can!
• Try to make sense of it
• Tune settings, SQL, hardware. Did it help?
• I think there is a problem with database...
Tuesday, October 23, 12
Tools that might help you
• pt-stalk (http://www.percona.com/doc/percona-toolkit/2.1/pt-stalk.html)
• innotop (http://code.google.com/p/innotop/)
• Or you write your own stuff
Tuesday, October 23, 12
What do “Big Boys” have?
• Oracle: SQL-traces, kernel events timings, tons of books on performance tuning
• SQL Server: SQL profiler, Data Management Views, Extended Events,tons of books on performance tuning
Tuesday, October 23, 12
Performance schema in MySQL 5.5
Tuesday, October 23, 12
The Idea
• Executing a query breaks down to hundreds of smaller tasks
• There are background tasks as well
• We want to instrument it all to know where server is spending time
Tuesday, October 23, 12
Implementation
• Instrumentation: measuring when event begins and ends
• Implemented in MySQL code on server end storage engine level
• Can be enabled/disabled or customized
Tuesday, October 23, 12
Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)
Tuesday, October 23, 12
Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)
Tuesday, October 23, 12
Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)
Tuesday, October 23, 12
Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)
Tuesday, October 23, 12
Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)
Tuesday, October 23, 12
Schema and a storage engine+----------------------------------------------+| Tables_in_performance_schema |+----------------------------------------------+| cond_instances || events_waits_current || events_waits_history || events_waits_history_long || events_waits_summary_by_instance || events_waits_summary_by_thread_by_event_name || events_waits_summary_global_by_event_name || file_instances || file_summary_by_event_name || file_summary_by_instance || mutex_instances || performance_timers || rwlock_instances || setup_consumers || setup_instruments || setup_timers || threads |+----------------------------------------------+17 rows in set (0.00 sec)
Tuesday, October 23, 12
Instruments+------------------------------------------------------------+| NAME |+------------------------------------------------------------+| wait/synch/mutex/sql/PAGE::lock || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_sync || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_active || wait/synch/mutex/sql/TC_LOG_MMAP::LOCK_pool |..... || wait/synch/rwlock/sql/LOCK_grant || wait/synch/rwlock/sql/LOGGER::LOCK_logger |
......| wait/synch/cond/sql/COND_thread_count | | wait/synch/cond/sql/COND_thread_cache |..... || wait/io/file/sql/binlog || wait/io/file/sql/binlog_index |+------------------------------------------------------------+
Tuesday, October 23, 12
Consumers
SELECT * FROM setup_consumers;+----------------------------------------------+---------+| NAME | ENABLED |+----------------------------------------------+---------+| events_waits_current | YES || events_waits_history | YES || events_waits_history_long | YES || events_waits_summary_by_thread_by_event_name | YES || events_waits_summary_by_event_name | YES || events_waits_summary_by_instance | YES || file_summary_by_event_name | YES || file_summary_by_instance | YES |+----------------------------------------------+---------+8 rows in set (0.00 sec)
Tuesday, October 23, 12
What are top wait events in my server?
SELECT COUNT_STAR, SUM_TIMER_WAIT, AVG_TIMER_WAIT FROM events_waits_summary_by_instance ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+---------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+---------------------------------------------+-----------------------+---------------+---------|| wait/io/file/innodb/innodb_data_file | 3347631 | 13547848609831118 | 4046995803 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69535441 | 2000129499580014 | 28764173 || wait/synch/mutex/innodb/buf_pool_mutex | 800909600 | 433331246983734 | 541048 || wait/synch/mutex/innodb/log_sys_mutex | 181356697 | 420610626124392 | 2319245 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3060704 | 232839801243104 | 76073936 || wait/io/file/innodb/innodb_data_file | 84259 | 163336572508164 | 1938505946 || wait/synch/rwlock/innodb/btr_search_latch | 192529196 | 161285917556962 | 837721 || wait/synch/mutex/sql/LOCK_open | 285022231 | 154822982324406 | 543196 || wait/synch/mutex/innodb/kernel_mutex | 403820715 | 130291049763820 | 322645 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285017708 | 121061567338910 | 424751 |+---------------------------------------------+------------+-------------------+----------------+10 rows in set (0.03 sec)
Tuesday, October 23, 12
innodb_buffer_pool_instances=10
SELECT * FROM events_waits_summary_by_instance ORDER BY SUM_TIMER_WAIT DESC LIMIT 10;+-----------------------------------------------+------------+-------------------+----------------+| EVENT_NAME | COUNT_STAR | SUM_TIMER_WAIT | AVG_TIMER_WAIT |+-----------------------------------------------+------------+-------------------+----------------+| wait/io/file/innodb/innodb_data_file | 3600080 | 14130028463169872 | 3924920685 || wait/synch/rwlock/innodb/index_tree_rw_lock | 69313641 | 2045091134703870 | 29504886 || wait/synch/mutex/innodb/trx_doublewrite_mutex | 6432834 | 592011487365760 | 92029654 || wait/synch/mutex/innodb/log_sys_mutex | 183016201 | 459928639055248 | 2513048 || wait/synch/rwlock/innodb/index_tree_rw_lock | 3239574 | 277212671866944 | 85570717 || wait/io/file/innodb/innodb_data_file | 400484 | 219787308922610 | 548804219 || wait/synch/rwlock/innodb/btr_search_latch | 190363676 | 183251774563990 | 962640 || wait/synch/mutex/sql/LOCK_open | 285022522 | 144911560909258 | 508421 || wait/synch/mutex/mysys/THR_LOCK::mutex | 285018012 | 130317830152708 | 457226 || wait/synch/mutex/innodb/kernel_mutex | 401734769 | 126815854421630 | 315670 |+-----------------------------------------------+------------+-------------------+----------------+10 rows in set (0.02 sec)
buf_pool_mutex is no longer in top 10 wait events
Tuesday, October 23, 12
Which files are being accessed the most?
mysql> SELECT SUM_NUMBER_OF_BYTES_READ, SUM_NUMBER_OF_BYTES_WRITE FROM file_summary_by_instance ORDER BY SUM_NUMBER_OF_BYTES_READ+SUM_NUMBER_OF_BYTES_WRITE DESC LIMIT 10;+------------------------------+--------------------------+---------------------------+| FILE_NAME |SUM_NUMBER_OF_BYTES_READ | SUM_NUMBER_OF_BYTES_WRITE |+------------------------------+--------------------------+---------------------------+| /var/lib/mysql/ibdata1 | 166920192 | 363649155072 || /var/lib/mysql/WIR.ibd | 180310720512 | 114136367104 || /var/lib/mysql/EMIWPR.ibd | 51803815936 | 71776468992 || /var/lib/mysql/EWAPR.ibd | 38596755456 | 54104932352 || /var/lib/mysql/OFPR.ibd | 24171593728 | 33964572672 || /var/lib/mysql/OFIR.ibd | 38002688000 | 14265303040 || /var/lib/mysql/EQPO.ibd | 12718882816 | 21407596544 || /var/lib/mysql/FIR.ibd | 19646103552 | 10337140736 || /var/lib/mysql/EIR.ibd | 14708899840 | 12814794752 || /var/lib/mysql/FPR.ibd | 6360350720 | 9764585472 |+------------------------------+--------------------------+---------------------------+10 rows in set (0.01 sec)
Tuesday, October 23, 12
Threads table(performance_schema) > SELECT * FROM threads;+-----------+----------------+----------------------------------------+| THREAD_ID | PROCESSLIST_ID | NAME |+-----------+----------------+----------------------------------------+| 0 | 0 | thread/sql/main || 12 | 0 | thread/innodb/srv_lock_timeout_thread || 14 | 0 | thread/innodb/srv_monitor_thread || 16 | 0 | thread/innodb/srv_purge_thread || 13 | 0 | thread/innodb/srv_error_monitor_thread || 2 | 0 | thread/innodb/io_handler_thread || 1 | 0 | thread/innodb/io_handler_thread || 9 | 0 | thread/innodb/io_handler_thread || 25 | 8 | thread/sql/one_connection || 3 | 0 | thread/innodb/io_handler_thread || 15 | 0 | thread/innodb/srv_master_thread || 10 | 0 | thread/innodb/io_handler_thread || 5 | 0 | thread/innodb/io_handler_thread || 8 | 0 | thread/innodb/io_handler_thread || 7 | 0 | thread/innodb/io_handler_thread || 17 | 0 | thread/sql/signal_handler || 4 | 0 | thread/innodb/io_handler_thread || 6 | 0 | thread/innodb/io_handler_thread |+-----------+----------------+----------------------------------------+18 rows in set (0.00 sec)
Tuesday, October 23, 12
Average wait per thread (I)
SELECT e.THREAD_ID, e.EVENT_NAME, MAX(e.AVG_TIMER_WAIT)FROM events_waits_summary_by_thread_by_event_name e LEFT JOIN threads t ON t.THREAD_ID = e.THREAD_IDWHERE event_name LIKE 'wait/synch/mutex/innodb/%' AND t.NAME='thread/sql/one_connection'AND AVG_TIMER_WAIT > 0GROUP BY t.thread_id, e.EVENT_NAMEORDER BY t.thread_id, MAX(e.AVG_TIMER_WAIT)
Tuesday, October 23, 12
Average wait per thread (II)
+-----------+-------------------------------------------------------+-------------------------+| THREAD_ID | EVENT_NAME | MAX(AVG_TIMER_WAIT) |+-----------+-------------------------------------------------------+-------------------------+| 27 | wait/synch/mutex/innodb/mutex_list_mutex | 49463 || 27 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83247 |
.....
.....| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 115681 || 27 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 168537 || 27 | wait/synch/mutex/innodb/ibuf_mutex | 282400 || 27 | wait/synch/mutex/innodb/log_sys_mutex | 1690694 |----------------------------------------------------------------------------------------------| 28 | wait/synch/mutex/innodb/mutex_list_mutex | 49504 || 28 | wait/synch/mutex/innodb/flush_list_mutex | 52375 || 28 | wait/synch/mutex/innodb/log_flush_order_mutex | 62040 |
.....
.....| 28 | wait/synch/mutex/innodb/autoinc_mutex | 63012 || 28 | wait/synch/mutex/innodb/dict_sys_mutex | 181203 || 28 | wait/synch/mutex/innodb/ibuf_mutex | 222376 || 28 | wait/synch/mutex/innodb/innobase_share_mutex | 239514 || 28 | wait/synch/mutex/innodb/ibuf_pessimistic_insert_mutex | 266684 || 28 | wait/synch/mutex/innodb/log_sys_mutex | 1718503 |
Tuesday, October 23, 12
Average wait per thread (III)
+-----------+--------------------------------------------------+-------------------------+| THREAD_ID | EVENT_NAME | MAX(ebt.AVG_TIMER_WAIT) |+-----------+--------------------------------------------------+-------------------------+| 27 | wait/synch/mutex/innodb/ibuf_mutex | 55872 || 27 | wait/synch/mutex/innodb/flush_list_mutex | 59495 || 27 | wait/synch/mutex/innodb/mutex_list_mutex | 59499 || 27 | wait/synch/mutex/innodb/autoinc_mutex | 64136 |
.....
.....| 27 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 88242 || 27 | wait/synch/mutex/innodb/buf_pool_mutex | 101598 || 27 | wait/synch/mutex/innodb/kernel_mutex | 148332 || 27 | wait/synch/mutex/innodb/log_sys_mutex | 4671112 |-----------------------------------------------------------------------------------------| 28 | wait/synch/mutex/innodb/ibuf_mutex | 42886 || 28 | wait/synch/mutex/innodb/mutex_list_mutex | 56646 || 28 | wait/synch/mutex/innodb/flush_list_mutex | 58105 || 28 | wait/synch/mutex/innodb/autoinc_mutex | 66873 |
....
....| 28 | wait/synch/mutex/innodb/purge_sys_bh_mutex | 83545 || 28 | wait/synch/mutex/innodb/ibuf_bitmap_mutex | 87697 || 28 | wait/synch/mutex/innodb/buf_pool_mutex | 101365 || 28 | wait/synch/mutex/innodb/kernel_mutex | 149135 || 28 | wait/synch/mutex/innodb/log_sys_mutex | 4829134 |
innodb_change_buffering = none [Don’t try this at home! ]
Tuesday, October 23, 12
Picoseconds, shmicosecond! ps_helper to rescue!
• http://www.markleith.co.uk/ps_helper/
• Great examples of using performance_schema
• Useful tools for converting time, bytes, paths into human readable format
Tuesday, October 23, 12
Example: top_global_io_consumers_by_bytes_usage
(ps_helper) > SELECT event_name, total_read, avg_read, total_written, avg_written,avg_latency FROM top_global_io_consumers_by_bytes_usage;+-------------------------+------------+-----------+---------------+-------------+-------------+| event_name | total_read | avg_read | total_written | avg_written | avg_latency |+-------------------------+------------+-----------+---------------+-------------+-------------+| innodb/innodb_data_file | 10.76 GiB | 16.00 KiB | 16.05 GiB | 30.90 KiB | 10.96 ms || innodb/innodb_log_file | 68.00 KiB | 11.33 KiB | 1.03 GiB | 369.66 KiB | 2.33 ms || myisam/dfile | 197.79 KiB | 8.60 KiB | 187.04 KiB | 5.84 KiB | 9.89 µs || sql/FRM | 142.28 KiB | 491 bytes | 4.50 KiB | 242 bytes | 3.66 µs || sql/ERRMSG | 43.68 KiB | 14.56 KiB | 0 bytes | 0 bytes | 15.42 µs || myisam/kfile | 13.22 KiB | 226 bytes | 5.96 KiB | 24 bytes | 8.26 µs || mysys/charset | 17.83 KiB | 17.83 KiB | 0 bytes | 0 bytes | 10.68 µs || sql/partition | 1.06 KiB | 32 bytes | 0 bytes | 0 bytes | 4.38 µs || sql/pid | 0 bytes | 0 bytes | 6 bytes | 6 bytes | 34.78 µs || sql/global_ddl_log | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 3.69 µs || archive/data | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.35 µs || sql/dbopt | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 4.55 µs || sql/casetest | 0 bytes | 0 bytes | 0 bytes | 0 bytes | 28.08 µs |+-------------------------+------------+-----------+---------------+-------------+-------------+13 rows in set (0.00 sec)
Tuesday, October 23, 12
No free lunch: PFS overhead
• Instrumentation doesn’t come for free
• In CPU-bound workloads overhead can be ~15-20%
• In IO-bound — 5%-8%
• Significant improvements in 5.6
Tuesday, October 23, 12
Performance schema in MySQL 5.6
Tuesday, October 23, 12
What’s new in 5.6?
• performance_schema enabled by default!
• Less overhead: 5%-10% for CPU-bound workload
• Statements, Stages, Actors and Objects
Tuesday, October 23, 12
Actors: filter events by user
(performance_schema) > SELECT * FROM setup_actors;+------+------+------+| HOST | USER | ROLE |+------+------+------+| % | % | % |+------+------+------+1 row in set (0.00 sec)
Tuesday, October 23, 12
Objects: filter out events by database/table
(performance_schema) > SELECT * FROM setup_objects;+-------------+--------------------+-------------+---------+-------+| OBJECT_TYPE | OBJECT_SCHEMA | OBJECT_NAME | ENABLED | TIMED |+-------------+--------------------+-------------+---------+-------+| TABLE | mysql | % | NO | NO || TABLE | performance_schema | % | NO | NO || TABLE | information_schema | % | NO | NO || TABLE | % | % | YES | YES |+-------------+--------------------+-------------+---------+-------+4 rows in set (0.00 sec)
Tuesday, October 23, 12
StatementsSELECT * FROM events_statements_summary_by_digest\G
*************************** 4. row *************************** DIGEST: f60b0866d7cbd78d39caf5fb52f43bfb DIGEST_TEXT: SELECT * FROM `table_io_waits_summary_by_index_usage` ORDER BY `SUM_TIMER_WAIT` DESC LIMIT ? COUNT_STAR: 1 SUM_TIMER_WAIT: 511539000 MIN_TIMER_WAIT: 511539000 AVG_TIMER_WAIT: 511539000 MAX_TIMER_WAIT: 511539000 SUM_LOCK_TIME: 72000000 SUM_ERRORS: 0 SUM_WARNINGS: 0 SUM_ROWS_AFFECTED: 0 SUM_ROWS_SENT: 5 SUM_ROWS_EXAMINED: 91SUM_CREATED_TMP_DISK_TABLES: 0 SUM_CREATED_TMP_TABLES: 0 SUM_SELECT_FULL_JOIN: 0 SUM_SELECT_FULL_RANGE_JOIN: 0 SUM_SELECT_RANGE: 0 SUM_SELECT_RANGE_CHECK: 0 SUM_SELECT_SCAN: 1 SUM_SORT_MERGE_PASSES: 0 SUM_SORT_RANGE: 0 SUM_SORT_ROWS: 5 SUM_SORT_SCAN: 1 SUM_NO_INDEX_USED: 1 SUM_NO_GOOD_INDEX_USED: 0 FIRST_SEEN: 2012-10-16 21:50:13 LAST_SEEN: 2012-10-16 21:50:13
Tuesday, October 23, 12
Summary. Pros.
• Provides insight into internal server metric
• Flexible
• Relational model allows to build your own views on top of this data
Tuesday, October 23, 12
Summary. Cons.
• Steep learning curve: not very well documented
• Overhead for CPU-bound loads can be significant
Tuesday, October 23, 12
Thank you!
Tuesday, October 23, 12
Recommended