Performance Schema for MySQL troubleshooting

<Insert Picture Here>

Performance Schema for MySQL troubleshooting

Sveta SmirnovaSenior Principal Technical MySQL Support Engineer

Content

• History of Performance Schema• Tables for DBA• Tables for developers• Other tables• Tools• Performance and tests• Options• Information sources• Conclusion

History of Performance Schema

• First version: in MySQL 5.5• 17 tables• Useful mostly for developers of MySQL code• Tools for

– Mutexes– Locks

• Required good knowledge of MySQL code

Kinds of tables

• Settings– _setup– _instances

• Events– events_waits_

• Digests• History• Other

Version 5.6 turned its face to DBA

• More features• 52 tables• New tables, very useful

for DBA• Knowledge of MySQL

source code is not a requirement anymore

*That's me talking at Devconf 2012 about how I am, as MySQL Support engineer, is happy with new features in Performance Schema

*

Tables for DBA

• events_statements_*• events_stages_*• Connection

events_statements_*

• Statements– statement/sql

• statement/sql/delete• statement/sql/select

• Commands– COM_PING, COM_QUIT, ...– statement/com

• statement/com/Ping• statement/com/Quit

• Errors– statement/sql/error– statement/com/Error

events_statements_*: which queries finished with an error

●

● mysql> select THREAD_ID, substr(SQL_TEXT, 1, 20),● MYSQL_ERRNO from events_statements_history_long where ● MYSQL_ERRNO != 0;● ++++● | THREAD_ID | substr(SQL_TEXT, 1, 20) | MYSQL_ERRNO |● ++++● | 18 | select from * event_ | 1064 |● | 18 | select * from event | 1146 |● | 18 | select * from event | 1146 |● | 18 | select THREAD_ID, SQ | 1146 |● ++++● 4 rows in set (0.00 sec)●

●

events_statements_*:queries which need to be optimized

●

● mysql> select THREAD_ID as TID, substr(SQL_TEXT, 1, 20) ● as SQL_TEXT, ROWS_SENT as RS, ROWS_EXAMINED as RE from ● events_statements_history_long where ROWS_EXAMINED > ● ROWS_SENT * 10 limit 5;● +++++● | TID | SQL_TEXT | RS | RE |● +++++● | 18 | select THREAD_ID, SQ | 4 | 147 |● | 18 | select THREAD_ID, su | 4 | 148 |● | 18 | select THREAD_ID, su | 4 | 152 |● | 18 | select THREAD_ID, su | 4 | 153 |● | 18 | select THREAD_ID, su | 1 | 154 |● +++++● 5 rows in set (0.00 sec)

events_statements_*: what also is worth attention

• CREATED_TMP_DISK_TABLES• CREATED_TMP_TABLES• SELECT_FULL_JOIN • SELECT_RANGE_CHECK• SELECT_SCAN• SORT_MERGE_PASSES• SORT_SCAN• NO_INDEX_USED• NO_GOOD_INDEX_USED

events_statements_*: ps_helper view

• http://www.markleith.co.uk/ps_helper/• View: statement_analysis• View: statements_with_runtimes_in_95th_percentile• View: statements_with_temp_tables• View: statements_with_sorting• View: statements_with_full_table_scans• View: statements_with_errors_or_warnings

http://www.markleith.co.uk/ps_helper/

event_stages_*

• Same information which you see in table INFORMATION_SCHEMA.PROCESSLIST or SHOW PROCESSLIST output– init– executing– Opening tables– ...

• Replacement of SHOW PROFILE• Only server-level• No information from storage engine in this table!

event_stages_*:«Sending data» for more than 10 seconds

●

● mysql> select events_stages_history_long.event_name,● sql_text, ● events_stages_history_long.timer_wait/1000000000000 ● wait_s from events_stages_history_long join ● events_statements_history_long on ● (events_stages_history_long.nesting_event_id = ● events_statements_history_long.event_id) where ● events_stages_history_long.EVENT_NAME like '%Sending ● data' and rows_sent < 10000000 and ● events_stages_history_long.timer_wait > 10*1000000000000 ● order by events_stages_history_long.timer_wait desc\G● ************************ 1. row ************************● event_name: stage/sql/Sending data● sql_text: insert into test.t2 select * from test.t2 ● wait_s: 243.5235● 1 rows in set (0.01 sec)●

event_stages_*: other operations which can run slow

• Everything, related to temporary tables– EVENT_NAME LIKE 'stage/sql/%tmp%'

• Everything, related to locks– EVENT_NAME LIKE 'stage/sql/%lock%'

• Everything in state «Waiting for»– EVENT_NAME LIKE 'stage/%/Waiting for%'

• Frequently met issues (from my Support experience)– EVENT_NAME='stage/sql/end'– EVENT_NAME='stage/sql/freeing items'– EVENT_NAME='stage/sql/Sending data'– EVENT_NAME='stage/sql/cleaning up'– EVENT_NAME='stage/sql/closing tables'

event_stages_*: longest queries

●

● mysql> select eshl.event_name, sql_text,● eshl.timer_wait/1000000000000 wait_s from ● events_stages_history_long eshl join ● events_statements_history_long esthl on ● (eshl.nesting_event_id = esthl.event_id) where ● eshl.timer_wait > 10*1000000000000\G● ************************ 1. row ************************● event_name: stage/sql/copy to tmp table● sql_text: alter table t2 engine=innodb● wait_s: 186.8122● ************************ 2. row ************************● event_name: stage/sql/Waiting for table metadata lock● sql_text: insert into t2 select * from t2 LIMIT 10● wait_s: 46.6250● 2 rows in set (0.01 sec)●

event_stages_*: joins

• NESTING_EVENT_ID– Statement– Wait– Stage

• EVENT_ID

events_statements

EVENT_ID

events_stages

NESTING_EVENT_ID

events_stages

NESTING_EVENT_ID

events_stages

NESTING_EVENT_ID

Connection Tables: accounts

●

● mysql> select user, host, current_connections as cur, ● total_connections as total from accounts;● +++++● | user | host | cur | total |● +++++● | foo | localhost | 0 | 3 |● | root | localhost | 1 | 3 |● | NULL | NULL | 14 | 17 |● +++++● 3 rows in set (0.01 sec)●

Connection Tables: users, hosts

●

● mysql> select user, current_connections as cur, ● total_connections as total from users;● ++++● | user | cur | total |● ++++● | root | 1 | 3 |● | NULL | 14 | 17 |● | foo | 0 | 3 |● ++++● 3 rows in set (0.00 sec)● mysql> select host, current_connections as cur, ● total_connections as total from hosts;● ++++● | host | cur | total |● ++++● | NULL | 14 | 17 |● | localhost | 1 | 6 |● ++++● 2 rows in set (0.01 sec)

Connection Attribute Tables

●

● mysql_init(&mysql);● mysql_options(&mysql,MYSQL_OPT_CONNECT_ATTR_RESET, 0);●

●

● mysql_options4(&mysql,MYSQL_OPT_CONNECT_ATTR_ADD,● "program", "Devconf2013");● mysql_options4(&mysql,MYSQL_OPT_CONNECT_ATTR_ADD, ● "author", "Sveta Smirnova");● mysql_options4(&mysql,MYSQL_OPT_CONNECT_ATTR_ADD, ● "session", "MySQL Performance Schema");●

●

● mysql_real_connect(&mysql, "127.0.0.1", "root", "",● "test", 13000, NULL, 0);

Connection Attribute Tables

●

● mysql> select ATTR_NAME, ATTR_VALUE from ● performance_schema.session_account_connect_attrs where ● processlist_id != @@pseudo_thread_id;● +++● | ATTR_NAME | ATTR_VALUE |● +++● | _os | Linux |● | _client_name | libmysql |● | _pid | 4729 |● | program_name | Devconf2013 |● | _platform | x86_64 |● | session | MySQL Performance Schema |● | author | Sveta Smirnova |● | _client_version | 5.6.12 |● +++● 8 rows in set (0.01 sec)

Connection Attribute Tables: foreigners prohibited!

●

● mysql> select PROCESSLIST_ID as PID, ATTR_NAME, ● ATTR_VALUE from session_account_connect_attrs where ● attr_name='program_name';● ++++● | PID | ATTR_NAME | ATTR_VALUE |● ++++● | 9 | program_name | mysql |● | 13 | program_name | Devconf2013 |● ++++● 2 rows in set (0.00 sec)

Connection Attribute Tables: foreigners prohibited!

●

● mysql> select PROCESSLIST_ID as PID, ATTR_NAME, ● ATTR_VALUE from session_account_connect_attrs where ● attr_name='program_name' union select PROCESSLIST_ID as ● PID, 'program_name' as ATTR_NAME, ● sum(if(attr_name='program_name', 1, 0)) as ATTR_VALUE ● from session_account_connect_attrs group by ● processlist_id having(ATTR_VALUE=0);● ++++● | PID | ATTR_NAME | ATTR_VALUE |● ++++● | 9 | program_name | mysql |● | 13 | program_name | Devconf2013 |● | 21 | program_name | 0 |● ++++● 3 rows in set (0.01 sec)●

●

host_cache

• Content of DNS cache• Errors from

– Name server– Connection– Authentication– max_connect_errors, max_user_errors, etc.

• Your first assistant in case of connection issue

threads

• Two kinds of THREADS– Background– Foreground

• Fields– THREAD_ID

• Internal thread id– PROCESSLIST_ID

• id, observable in the SHOW PROCESSLIST output– NAME

• Instrument– PARENT_THREAD_ID

• Internal id of the parent thread– PROCESSLIST_*

• Only for для FOREGROUND threads

threads

●

● mysql> select * from threads where type = 'foreground'\G● ************************ 1. row ************************● THREAD_ID: 16● NAME: thread/sql/one_connection● TYPE: FOREGROUND● PROCESSLIST_ID: 1● PROCESSLIST_USER: root● PROCESSLIST_HOST: localhost● PROCESSLIST_DB: performance_schema● PROCESSLIST_COMMAND: Query● PROCESSLIST_TIME: 0● PROCESSLIST_STATE: Sending data● PROCESSLIST_INFO: select * from threads where type = 'foreground'● PARENT_THREAD_ID: 1● ROLE: NULL● INSTRUMENTED: YES● 1 row in set (0.00 sec)●

threads

●

● mysql> select name from threads where type='background';● ++● | name |● ++● | thread/sql/main |● | thread/innodb/io_handler_thread |● | thread/innodb/io_handler_thread |● | thread/innodb/io_handler_thread |● | thread/innodb/io_handler_thread |● | thread/innodb/io_handler_thread |● | thread/innodb/io_handler_thread |● | thread/innodb/srv_lock_timeout_thread |● | thread/innodb/srv_error_monitor_thread |● | thread/innodb/srv_monitor_thread |● | thread/innodb/srv_master_thread |● | thread/innodb/srv_purge_thread |● | thread/innodb/page_cleaner_thread |● | thread/sql/signal_handler |● ++● 14 rows in set (0.00 sec)●

events_waits_*

• EVENT_NAME– wait/synch/rwlock/innodb/dict_operation_lock

• SOURCE– Line of the source code

• OPERATION– Kind of operation: read, lock, write

event_waits_*

●

● mysql> select e.EVENT_NAME, e.SOURCE, e.OPERATION, ● t.PROCESSLIST_INFO from events_waits_current e join ● threads t using(thread_id) where type='foreground' and ● processlist_id != 1\G● ************************ 1. row ************************● EVENT_NAME:● wait/synch/cond/sql/Item_func_sleep::cond● SOURCE: item_func.cc:4212● OPERATION: timed_wait● PROCESSLIST_INFO: select sleep(100) from t1● 1 row in set (0.01 sec)●

wait/synch/cond/sql/Item_func_sleep::cond

●

● $ cat n sql/item_func.cc | head n 4220 | tail n 35● 4186● 4187 /**● 4188 Wait for a given condition to be signaled.● 4189● 4190 @param cond The condition variable to wait on.● 4191 @param mutex The associated mutex.● 4192● 4193 @remark The absolute timeout is preserved across ● calls.● 4194● 4195 @retval return value from mysql_cond_timedwait● 4196 */● 4197●


●

● 4198 int Interruptible_wait::wait(mysql_cond_t● *cond, mysql_mutex_t *mutex)● 4199 {● 4200 int error;● 4201 struct timespec timeout;● 4202● 4203 while (1)● 4204 {● 4205 /* Wait for a fixed interval. */● 4206 set_timespec_nsec(timeout, ● m_interrupt_interval);● 4207● 4208 /* But only if not past the absolute ● timeout. */● 4209 if (cmp_timespec(timeout, m_abs_timeout) > 0)● 4210 timeout= m_abs_timeout;●


● ● 4212 error= mysql_cond_timedwait(cond, mutex, ● &timeout);● 4213 if (error == ETIMEDOUT || error == ETIME)● 4214 {● 4215 /* Return error if timed out or connection ● is broken. */● 4216 if (!cmp_timespec(timeout, m_abs_timeout) || ● !m_thd>is_connected())● 4217 break;● 4218 }● 4219 /* Otherwise, propagate status to the caller. ● */● 4220 else●

Query statistics

●

● mysql> UPDATE performance_schema.threads SET ● instrumented = 'NO'; ● Query OK, 15 rows affected (0.04 sec)● Rows matched: 15 Changed: 15 Warnings: 0●

● Open new connection●

● mysql> truncate events_waits_history_long; Query OK, 0 rows affected (0.00 sec)●

● In new connection●

● mysql2> create temporary table norepl_t1 engine=myisam● select amount, price, money, id_product from test;● Query OK, 262144 rows affected (4.76 sec)● Records: 262144 Duplicates: 0 Warnings: 0

Query events_waits_history_long

●

● mysql> select e.EVENT_NAME, e.SOURCE, e.OPERATION,● count(*) as cnt from events_waits_history_long e join ● threads t using(thread_id) where type='foreground' and ● processlist_id not in (1, @@pseudo_thread_id) group by ● e.EVENT_NAME, e.SOURCE, e.OPERATION order by cnt desc\G● ************************ 1. row ************************● EVENT_NAME: wait/synch/mutex/innodb/lock_mutex● SOURCE: lock0lock.cc:5529● OPERATION: lock● cnt: 1428● ************************ 2. row ************************● EVENT_NAME: wait/synch/mutex/innodb/lock_mutex● SOURCE: lock0lock.cc:6362● OPERATION: lock● cnt: 1428


●

● ************************ 3. row ************************● EVENT_NAME: wait/synch/mutex/innodb/trx_sys_mutex● SOURCE: lock0lock.cc:5530● OPERATION: lock● cnt: 1428● ************************ 4. row ************************● EVENT_NAME: wait/synch/mutex/innodb/trx_mutex● SOURCE: lock0lock.cc:2133● OPERATION: lock● cnt: 1423● ************************ 5. row ************************● EVENT_NAME: wait/io/table/sql/handler● SOURCE: handler.cc:2627● OPERATION: fetch● cnt: 1423


●

● ************************ 6. row ************************● EVENT_NAME: wait/synch/mutex/innodb/lock_mutex● SOURCE: lock0lock.cc:6050● OPERATION: lock● cnt: 1421● ************************ 7. row ************************● EVENT_NAME: wait/synch/mutex/innodb/trx_sys_mutex● SOURCE: trx0sys.ic:431● OPERATION: lock● cnt: 1421● ************************ 8. row ************************● EVENT_NAME: wait/synch/mutex/innodb/buf_pool_mutex● SOURCE: buf0buf.ic:887● OPERATION: lock● Cnt: 6● ...

Which kind of events can we examine?

• setup_instruments.NAME – wait/io/file

• Operations with files– wait/io/socket– wait/io/table/sql/handler– wait/lock/table/sql/handler– wait/synch/cond

• InnoDB, MyISAM, sql– wait/synch/mutex

• sql, mysys, storage engines– wait/synch/rwlock/

• sql, InnoDB, MyISAM

ps_helper

• All VIEWs work for MySQL 5.5– latest_file_io– top_io_by_file– top_io_by_thread– top_global_consumers_by_avg_latency– top_global_consumers_by_total_latency– top_global_io_consumers_by_latency– top_global_io_consumers_by_bytes_usage

• There are few views for 5.6 which use digest tables

*_instances tables

• file_instances– Opened files

• socket_instances– Connections

• cond_instances• rwlock_instances

– select * from rwlock_instances where \ READ_LOCKED_BY_COUNT > 0;

– select * from rwlock_instances where \ WRITE_LOCKED_BY_THREAD_ID > 0;

• mutex_instances– LOCKED_BY_THREAD_ID

Digests

• events_stages_*• events_statements_*• events_waits_*• file_*• objects_*• socket_*• table_io_waits_*• table_lock_waits_*

Digests: events_stages_summary_*

• events_stages_summary_by_account_by_event_name– Helps to find an account which performs problematic queries

• events_stages_summary_by_host_by_event_name• events_stages_summary_by_user_by_event_name

– Same, but sorted by host and user name

• events_stages_summary_by_thread_by_event_name– Easy to find out what makes troubles on your server right now– Since statistics is saved for some time you can find it and after the

problem stopped to show up

• events_stages_summary_by_global_by_event_name– Global stats by event name– Does not indicate user, host, account and thread

Digests: events_statements_summary_*

• events_statements_summary_by_account_by_event_name• events_statements_summary_by_host_by_event_name • events_statements_summary_by_user_by_event_name• events_statements_summary_by_thread_by_event_name• events_statements_summary_global_by_event_name

– Same as stages, but stats are taken from tables events_statements_*

• events_statements_summary_by_digest– Stats by digest field:

• 42b93d481e96b9c9b4049b9407900194• Query written as SELECT fname FROM tname WHERE fname = ?

– For example, you can find all statements which create temporary tables by querying this table

Digests: events_waits_summary_*

• events_waits_summary_by_account_by_event_name• events_waits_summary_by_host_by_event_name• events_waits_summary_by_thread_by_event_name• events_waits_summary_by_user_by_event_name • events_waits_summary_global_by_event_name

– Similar to events_stages_* digests

• events_waits_summary_by_instance – By OBJECT_INSTANCE_BEGIN field

Other digests

• file_summary_by_event_name – Does not show file name!

• file_summary_by_instance• objects_summary_global_by_type • socket_summary_by_event_name• socket_summary_by_event_name• socket_summary_by_instance

– By OBJECT_INSTANCE_BEGIN field

• table_io_waits_summary_by_index_usage• table_io_waits_summary_by_table• table_lock_waits_summary_by_table

Digests

• WHERE COUNT_STAR > 0• Sort or query by an operation you are interested in• Sort by COUNT_STAR

Performance

Performance: version 5.5

• Performance Schema is OFF by default• Noticeable performance issues

– Up to 7% in case of RO load– Up to 20% in case of RW load– Numbers based on tests by Dimitri Kravtchuk

(http://dimitrik.free.fr/blog/archives/2010/05/mysql-performance-using-performance-schema.html)

• No performance loss if turned off

http://dimitrik.free.fr/blog/archives/2010/05/mysql-performance-using-performance-schema.html

Performance: version 5.6

• Performance Schema is ON by default• Performance loss can happen, but not big

– Not more than 5% for most setups, likely near 0– Maximum up to 10% in case if all instrumentations are turned ON– Numbers based on tests by Dimitri Kravtchuk

(http://dimitrik.free.fr/blog/archives/2012/06/mysql-performance-pfs-overhead-in-56.html)

• global_instrumentation– Minimal overhead

• Detailed instrumentation– Noticeable overhead

• History tables– minimal overhead

http://dimitrik.free.fr/blog/archives/2012/06/mysql-performance-pfs-overhead-in-56.html

How P_S uses OS and hardware resources

• Memory– Allocated at the server startup– Freed when MySQL server is stopped– Uses arrays instead of linked lists– mysql> show engine performance_schema status;++++| Type | Name | Status |++++...| performance_schema | performance_schema.memory | 68024616 |++++

• CPU– Depends from number of instruments– More instruments — higher load

Options

What, where and when to setup

• At compile time• At the server startup

– Options in my.cnf– All options are static

• Runtime– setup_* tables

• What can you tune?– Look for tables documentation

Configuration options

• performance_schema = ON|OFF– Is it On or Off?

• performance_schema_%_size– Size of history tables– Size of instrumented objects

• performance_schema_max_%_classes– Maximum number of cond|fle|io|% instruments

• performance_schema_max_%_instances– Maximum number of cond|fle|io|% objects

Configuration options

• performance_schema_consumer_TABLE_NAME– performance_schema_consumer_events_stages_current– performance_schema_consumer_events_waits_current– ...

• Turns instrumentations On of Off– OFF, FALSE, 0– ON, TRUE, 1

• setup_consumers table– update setup_consumers set enabled='no' \where name='events_stages_current';

Tables setup_actors and setup_objects

• setup_actors– Which user threads to monitor– DELETE , then INSERT– UPDATE not allowed– insert into setup_actors values('%', 'sveta', '%');

• Only for user sveta

• setup_objects– Which objects to monitor– update setup_objects set enabled='no' \where object_schema='%';

– insert into setup_objects values \('TABLE', 'test', 't1', 'YES', 'YES');

setup_instruments table

• Detailed setup of instruments• 549 instruments in the standard distribution*• update setup_instruments set enabled='no';• update setup_instruments set enabled='yes' \where name like 'statement%';

*Written at June, 2013. Subject to change.

Timers

●

● Values for your machine●

● mysql> select * from performance_timers;● +++++● | TIMER_NAME | TIMER_FREQUENCY | TIMER_RESOLUTION | TIMER_OVERHEAD |● +++++● | CYCLE | 2592796019 | 1 | 18 |● | NANOSECOND | 1000000000 | 1 | 45 |● | MICROSECOND | 1000000 | 1 | 48 |● | MILLISECOND | 1037 | 1 | 54 |● | TICK | 103 | 1 | 547 |● +++++● 5 rows in set (0.00 sec)●

● How to tune:● mysql> update setup_timers set timer_name='tick' \● where name = 'stage';

What happens inside Performance Schema?

●

● mysql> show global status like 'perf%';● +++● | Variable_name | Value |● +++● | Performance_schema_accounts_lost | 0 |● | Performance_schema_cond_classes_lost | 0 |● | Performance_schema_cond_instances_lost | 0 |● | Performance_schema_digest_lost | 0 |● ...●

● If Value is not null — your *_size options are too small

What happens inside Performance Schema?

• SHOW ENGINE PERFORMANCE_SCHEMA STATUS;• Contains information about memory usage• Table_name.attribute• (Internal_buffer).attribute• *.size, *.row_size

– Not-configurable, for example, size of a table row

• *.count, *.row_count– Configurable with help of options

• *.memory– size * count– events_waits_history_long.memory– performance_schema.memory

Where to find information?

• http://www.markleith.co.uk/ps_helper/• http://www.drdobbs.com/database/detailed-profiling-of-sql-activity-in-my/240154959

• http://marcalff.blogspot.ru• http://dimitrik.free.fr/blog/• http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html

http://www.markleith.co.uk/ps_helper/

http://www.drdobbs.com/database/detailed-profiling-of-sql-activity-in-my/240154959

http://marcalff.blogspot.ru/

http://dimitrik.free.fr/blog/

http://dev.mysql.com/doc/refman/5.6/en/performance-schema.html

Conclusion

• Performance schema — wonderful tool for a DBA when she needs to troubleshoot performance issue

• You can configure it online: without server restart• Allows very detailed setup• Always tune it for your own needs!• Don't instrument everything: use it for operations you

are interested in only

?

THANK YOU!

The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Technology

Performance Schema for MySQL troubleshooting