Jeremy Glick, MySQL DBASan Francisco MySQL Meetup

Who am I?

Jeremy Glick

• 6 Years MySQL DBA• Chicago / Sacramento• Organizer, Chicago MySQL Meetup

Andrew Moore

• Remote DBA @ Percona• Organizer, SW UK MySQL Meetup

Agenda• Why log• How to log• Audit plugins• Elasticsearch ELK• Demos

Logs

[timestamp]: [some useful data]

Why log?

Why log?

Why log?

Why log?

How [not] to log● General log● Slow log● Binary log● Sniff network● In schema● init_connect

MySQL Pluggable Audit Interface• Available since 5.5.3• Audit interface notifies plugin of

– General log messages– Error log messages– Query results sent to client

* https://dev.mysql.com/doc/refman/5.6/en/audit-plugins.html

MySQL Pluggable Audit Interface

• Custom plugin• Most popular open source plugins

– McAfee– Percona– MariaDB

Installing PluginsCLI:INSTALL PLUGIN plugin_name SONAME='shared_lib_name.so'

my.cnf: (RECOMMENDED)plugin-load=plugin_name=shared_lib_name.so

Startup:mysqld –plugin-load='plugin_name'='shared_lib_name.so'

Installing PluginsForce:plugin_name=FORCE_PLUS_PERMANENT

Configuration Options: (vary from plugin to plugin)• Filtering• Sync/performance• File|syslog[ng]

McAfee Audit Plugin• Available for 5.1+

– Binary hooking• Great community support• Most filtering options• JSON output• Socket and file options

McAfee Audit Plugin• Install may require generation of

offsets

./offset-extract.sh /path/to/mysqld /path/to/mysqld.debug

McAfee Audit Plugin• Filtering

– audit_record_cmds– audit_record_objects– audit_whitelist_users– audit_whitelist_cmds

McAfee Audit Plugin{"msg-type":"activity", "date":"1425967153721", "thread-id":"2", "query-id":"17", "user":"root", "priv_user":"root", "host":"localhost", "ip":"", "cmd":"select", "objects":[{"db":"test","name":"people","obj_type":"TABLE"}], "query":"select * from people"}

McAfee Audit Plugin{"msg-type":"activity", "date":"1425967153721", "thread-id":"2", "query-id":"17", "user":"root", "priv_user":"root", "host":"localhost", "ip":"", "cmd":"select", "objects":[{"db":"test","name":"people","obj_type":"TABLE"}], "query":"select * from people"}

{"msg-type":"activity", "date":"1425968812525", "thread-id":"3", "query-id":"29", "user":"root", "priv_user":"root", "host":"localhost", "ip":"", "cmd":"select", "objects": [{"db":"test","name":"people_vw","obj_type":"VIEW"}, {"db":"test","name":"people","obj_type":"TABLE"}], "query":"select * from people_vw"}

Percona Audit Plugin• Available for 5.5.3+• Ships with Percona Server• Drop in replacement for

Oracle's plugin• Limited filtering• JSON, XML, CSV output

Percona Audit Plugin• audit_log_strategy

– ASYNCHRONOUS– PERFORMANCE– SEMISYNCHRONOUS– SYNCHRONOUS

Percona Audit Plugin

● Support for syslog– audit_log_handler = FILE|SYSLOG

Percona Audit Plugin<AUDIT_RECORD NAME="Query" RECORD="20_2015-03-10T06:49:49" TIMESTAMP="2015-03-10T06:53:55 UTC" COMMAND_CLASS="select" CONNECTION_ID="3" STATUS="0" SQLTEXT="select * from people" USER="root[root] @ localhost []" HOST="localhost" OS_USER="" IP=""/>

Percona Audit Plugin<AUDIT_RECORD NAME="Query" RECORD="32_2015-03-10T06:49:49" TIMESTAMP="2015-03-10T06:55:35 UTC" COMMAND_CLASS="select" CONNECTION_ID="4" STATUS="0" SQLTEXT="select * from people_vw" USER="root[root] @ localhost []" HOST="localhost" OS_USER="" IP=""/>

MariaDB Audit Plugin

• Available for 5.5+• Expanded Audit API• Included by default

MariaDB Audit Plugin

• Table level events• CSV output• Syslog• Plain text passwords

– < 5.5.42 (1.2.0)

MariaDB Audit Plugin

• Filtering– server_audit_events– server_audit_excl_users– server_audit_incl_users

MariaDB Audit Plugin20150310 03:07:25,localhost.localdomain,root,localhost,3,10,QUERY,test,'select * from people',0

MariaDB Audit Plugin20150310 03:11:18,localhost.localdomain,root,localhost,3,9,READ,test,people,20150310 03:11:18,localhost.localdomain,root,localhost,3,9,QUERY,test,'select * from people_vw',0

ReplicationMcAfee

• Slaves log replication events by default*Whitelist blank user “{}” to prevent

Percona and MariaDB• Not logged

Best practices• Secure data

– OS level not logged• Utilize log rotation

Best practices• Sequential logging lives away from

random access• Use FS with journalling to be crash

safe(r)• Synchronizing writes to disk hurts a lot

Log File Storage• Secure storage (encryption)• Sign logs to ensure not altered• Set permissions• Store offsite (encrypted of course)• Store on read only media

Log Aggregation● Proprietary

– Oracle Audit Vault– McAfee DAM– Splunk

● Open Source– Elasticsearch ELK Stack

Elasticsearch ELK

Elasticsearch

● Full text and analytics● Apache Lucene● RESTful web interface● Schema-free JSON

documents

Elasticsearch

● Index = table● Document = row

Logstash

● Centralize logs● Supports many input types● Filtering

Logstash

● Output plugins available– Nagios and Nagios_nsca– XMPP (hipchat, slack, etc.)– Pager duty

/etc/logstash/logstash.conf

input { file { path => "/var/log/mysql/audit.log" type => "mysql-audit" }

filter { do_something

}

output { elasticsearch { cluster => "logstash"

host => elasticsearch1}

}

Kibana

● Browser based dashboards● Real-time search and

analytics● Seamless integration with

Elasticsearch

Elasticsearch ELK

Demo

● Audit Plugin Performance● ELK