High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances

Santa Clara, California | April 23th – 25th, 2018

ProxySQL TutorialHigh Performance & High Availability

Proxy for MySQL

With a GPL license!

2

Who we are

René Cannaò

ProxySQL Founder

Derek Downey

Director of OSDB, Pythian

3

Agenda● What is ProxySQL?● Features● Fundamentals● What is new?● Internals● Admin● Clustering● Rules● Mirroring● Failover● New protocols● GTID casual reads

4

Other sessions

Consistent Reads Using ProxySQL and GTID

Tuesday April 24, 2018 @3:50PM

Automatic Failovers with Kubernetes using Orchestrator, ProxySQL and Zookeeper

Tuesday April 24, 2018 @3:50PM

Overview of ProxySQL

6

What is ProxySQL

The MySQL data stargate

7

Main motivations

empower the DBAs

improve operation

understand and improve performance

create a proxy layer to shield the database

High performance and High Availability

ProxySQL FeaturesSome of the most interesting features:

● on-the-fly rewrite of queries

● caching reads outside the database server

● connection pooling and multiplexing

● complex query routing and read/write split

● load balancing

● real time statistics

● monitoring

● Data masking

● Multiple instances on same ports

ProxySQL Features (2)Some of the most interesting features:

● High Availability and Scalability

● seamless failover

● firewall

● query throttling

● query timeout

● query mirroring

● runtime reconfiguration

● Scheduler

● Support for Galera/PXC and Group Replication

What’s new in ProxySQL 1.4.0Native Support for Group Replication

Multiple regex engines

Better performance

Support for millions of users

Support for tens of thousands of database servers

Better queries control

http://www.proxysql.com/blog/releasing-proxysql-140

What’s new in ProxySQL 1.4.7

Native ProxySQL Clustering Solution

Support for ClickHouse as a backend

SQLite network

Support for millions of shards

What’s new in ProxySQL 2.0 (dev)… all in 1.4.x plus:

Support for Aurora

SSL support for frontend

SSLv1.2

Native Support for Galera

Causal reads using GTID

Multi-threaded connections reset

Hostgroups and Query Routing

All backends are grouped into hostgroups

Hostgroups have logical functionalities

Hostgroups example #1

HostGroup0 (HG0): Write mastersHostGroup1( HG1): Read slaves

Read/Write split

Hostgroups example #2HG0: main write masters

HG1: main read slaves

HG2: reporting slaves

HG3: ad-hoc queries slaves

HG4: data warehouse write masters

HG5: data warehouse read slaves

HG6: remote site servers

HG7: test servers

HG8 : mirror for traffic on HG0

HG9 : mirror for traffic on HG1

Basic designHG0

HG2

HG1

ProxySQL

APP1

APP2

APP3

Basic designHG0

HG2

HG1

APPAPP

ProxySQL

Very low latency usingUnix Domain Socket (?)

One ProxySQL per host

Silos approach

Clustered ProxySQL Architecture

DB DBDB

APP

DB

PROXYSQL

DB DB

APPPROXYSQL

APPPROXYSQL

APPPROXYSQL

PROXYSQL PROXYSQL

DBMANAGER

PROXYSQL

Clustered Proxy Layer

Clustered ProxySQL at scaleTested with:

● 8 app servers with 3k clients’ connections each (24k total)● 4 middle layer proxysqls processing 4k connections each from local

proxysqls (16k total)● 256 backends/shard (meaning 256 routing rules) processing 600

connections each (150k total)

Single ProxySQL was tested with up to 950k connections

At today, ProxySQL is able to process up to 750k QPS

ProxySQL Internals

ProxySQL Modules in 1.3.xHG0

HG1

APP1 ProxySQL

Queries cache

Query Processor

APP2

APP3

Thre

ad #

1

Thre

ad #

2

Thre

ad #

3

Users Auth

ConnectionPool

HostgroupManager

Admin

Monitoring

ProxySQL Modules in 1.4.xHG0

HG1

APP1 ProxySQL

Queries cache

Query Processor

APP2

APP3Th

read

#1

Thre

ad #

2

Thre

ad #

3

Users AuthConnection

Pool

HostgroupManager

Admin

MonitoringClickHouse Server

ClickHouse Auth

Cluster

SQLite3 Server

Thre

ad #

3 id

les

Thre

ad #

2 id

les

Thre

ad #

1 id

les Purge Thread

Queries ProcessorBased on Queries Rules

Defines what to cache

Defines the hostgroup target

Timeout/delay

Firewall

Mirroring

Rewrite queries

Queries rulesComplex rules to match incoming traffic:

● regex on query

● regex on digest text

● username

● schemaname

● Source IP address

● Bind IP address/port

● digest

Rules can be chained

Queries Cache and RewriteCaching on the wire

Internal key/value storage

In memory only

Pattern based

Expired by timeout

Rewrite on the wire

Regex match/replace on query on digest text

Optionally cached or mirrored

Users Authentication

Credentials stored in the proxy

User login always possible (even without backends)

Max connections

Login credentials are encrypted

Extremely scalable, up to millions of users:

https://www.percona.com/live/e17/sessions/scaling-million-databases-000webhost-lt

Hostgroups Manager

Management of servers

Track servers status

Tightly integrated with the connections pool

Connections Pool

Reduced the overhead of creating new connections, and are recycled when not in use

One to many connections

Multiplexing & maximum connections

Auto-reconnect and automatic re-execution of queries

Failover management

Purge Thread

Unsafe connections are reset instead of being dropped

Drastically reduce the need to establish new connections

In ProxySQL 2.0 : each worker thread perform most of the reset

Auto-reconnect and re-execution

Automatic detection of failures

Graceful handling

Auto-reconnect when possible

Pause until a backend becomes available

Re-execution of queries

Multiplexing

Reduce the number of connections against mysqld (configurable)

Many clients connections (tens of thousands) can use few backend connections (few hundreds)

Tracks connection status (transactions, user variables, temporary tables, etc)

Order by waiting time

Multiplexing (see it in web UI)

Multiplexing (see it in web)

More metrics on web UI



Monitoring Module

It monitors backends and collects metrics

Monitors replication lag and shun hosts

Monitors read_only (and super_read_only and innodb_read_only in 2.0) variables

(replication hostgroups)

Ping and terminates unresponsive nodes

Monitoring ModuleBuilt in support for asynchronous replication:

second behind master and pt-heartbeat

Support for any type of clustering solution via external scripts

(Galera/PXC , and any heterogeneous replication setup)

In ProxySQL 1.4.0, support for Group Replication

In ProxySQL 2.0, support for Galera

Admin Module

Admin InterfaceAllows runtime configuration

Exports internal statuses

It uses MySQL protocol

Configuration possible from any client/tool using MySQL API

Covered during the tutorial

Admin InterfaceDiskMemoryRuntime

Runtime

Memory

DiskConfig

Try it!Source code on GitHub:

https://github.com/sysown/proxysql/

Forum:

https://groups.google.com/forum/#!forum/proxysql

Tutorials on:

http://www.proxysql.com

https://github.com/sysown/proxysql/

https://groups.google.com/forum/#!forum/proxysql

http://www.proxysql.com

Demo Environment

47

DemoCode on GitHub:https://github.com/renecannao/ProxySQL-Tutorial-PLSC2018

Some hosts provided:

Username: plsc

Passwords: proxysql

ProxySQL Cluster

ProxySQL ClusterNew in 1.4.2 – experimental (not fully completed, subject to change)

Support for MySQL Query Rules, MySQL Users, MySQL Servers and ProxySQL Servers

A group of instances constantly check each other for changes in configuration

A configuration change applied in a proxy is pulled by the other proxies

Allow to keep multiple ProxySQL instances with the same configuration(s)

ProxySQL Cluster

ProxySQL ClusterConflict resolution is timestamp based• No order• Last win

No quorum (in the roadmap)

Highly scalable: Core nodes vs satellite nodes

52

ProxySQL Cluster - Configmysql> SET admin-cluster_username='radmin';

mysql> SET admin-cluster_password='radmin';

mysql> LOAD ADMIN VARIABLES TO RUNTIME;

mysql> SAVE ADMIN VARIABLES TO DISK;

mysql> INSERT INTO proxysql_servers (hostname) VALUES

('proxysql1'),('proxysql2'),('proxysql3');

mysql> LOAD PROXYSQL SERVERS TO RUNTIME;

mysql> SAVE PROXYSQL SERVERS TO DISK;

mysql> SHOW VARIABLES LIKE 'admin-cluster%';+---------------------------------------------------+--------+| Variable_name | Value |+---------------------------------------------------+--------+| admin-cluster_username | radmin || admin-cluster_password | radmin || admin-cluster_check_interval_ms | 1000 || admin-cluster_check_status_frequency | 10 || admin-cluster_mysql_query_rules_diffs_before_sync | 3 || admin-cluster_mysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_users_diffs_before_sync | 3 || admin-cluster_proxysql_servers_diffs_before_sync | 3 || admin-cluster_mysql_query_rules_save_to_disk | true || admin-cluster_mysql_servers_save_to_disk | true || admin-cluster_mysql_users_save_to_disk | true || admin-cluster_proxysql_servers_save_to_disk | true |+---------------------------------------------------+--------+12 rows in set (0.00 sec)

53

ProxySQL Cluster - MonitoringStats_proxysql_servers_checksums

Stats_proxysql_servers_metrics

ProxySQL ClusterDEMO

MySQL Query rules

56

Query rewriting overview

Sql-aware

Regex-based

flexible

57

Query rewriting examples

read/write splitting

Data masking

Sharding

58

Query rewriting flow

Match Related Tablesmysql_users

• active• fast_forward(bypass)

mysql_query_rules• username• schemaname• flagIN/OUT• client_addr• proxy_addr• proxy_port• digest• match_digest• match_pattern• negate_match_pattern

Action related fieldsflagOUTreplace_patterndestination_hostgroupcache_ttltimeoutretriesdelaymirror_flagOUTmirror_hostgroupOK_msgerror_msglogmultiplexapply

active

3 layers for ProxySQL tablesDiskMemoryRuntime

Runtime

Memory

DiskConfig

PrecautionsConfirm the query hits with stats_mysql_query_rulesdouble check rules and rewrite resultshold off on writing rules to disk until you're sure they are working as expectedhave a rollback plan

• load from disk, • then load from memory to runtime

test select query rules and rewrites on a slaves firsttest mutable rules on a mirror or other throw away db.

63

Example - Simple R/Wmysql> SELECT username, default_hostgroup FROM mysql_users\G

*************************** 1. row ***************************

username: root

default_hostgroup: 0

1 row in set (0.00 sec)

mysql> SELECT match_digest, destination_hostgroup FROM

mysql_query_rules\G

*************************** 1. row ***************************

match_digest: ^SELECT.*FOR UPDATE

destination_hostgroup: 0

*************************** 2. row ***************************

match_digest: ^SELECT


2 rows in set (0.00 sec)

Query Rewriting with ProxySQLDEMO

Click to add text

Mirroring with ProxySQLClick to add text

66

What is mirroring?

67

What is mirroring?

68

Why mirror queries?

Validate performance on a different server using different hostgroups.

Validate performance of query rewrite or schema change

Pre-fetch slave replication (Replication Booster)

69

How to mirrormysql>SHOW CREATE TABLE mysql_query_rules\G

*********************** 1. row ************************

table: mysql_query_rules

Create Table: CREATE TABLE mysql_query_rules (

rule_id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL,

active INT CHECK (active IN (0,1)) NOT NULL DEFAULT 0,

username VARCHAR,

schemaname VARCHAR,

flagIN INT NOT NULL DEFAULT 0,

*snip*

mirror_flagOUT INT UNSIGNED,

mirror_hostgroup INT UNSIGNED,

error_msg VARCHAR,

*snip*)

70

Mirroring flow

71

Mirroring - example #1

mysql> SELECT username, destination_hostgroup, mirror_hostgroup,

mirror_flagOUT FROM mysql_query_rules WHERE

username='plam_mirror'\G

****************** 1. row **********************

username: pl_mirror


mirror_hostgroup: 4

mirror_flagOUT: NULL

72

Mirroring - example #2

Mirroring DemoClick to add text

74

Mirroring caveats

Can be dropped: mysql-mirror_max_concurrency,

mysql-mirror_max_queue_length

Can be executed out of order

No error handling of failed queries

Failover with ProxySQL

76

Failover highlights

improve failover time as perceived by the application

prevent errors sent to the application

perform transparent database failovers: gracefully redirecting traffic without the application knowing

existing applications do not have to be rewritten to autoreconnect since connections are not lost from failovers

77

Failover scenario - without ProxySQL

78

Failover scenario - without ProxySQL

79

Failover scenario - with ProxySQL

80


81


82


83

Failover without ProxySQL (10 minute outage)

https://www.nylas.com/blog/growing-up-with-mysql/

84

Failover with ProxySQL (10 second outage)

https://www.nylas.com/blog/growing-up-with-mysql/

85

Failover - MySQL Serversmysql> SELECT hostgroup_id, hostname, status FROM mysql_servers

WHERE hostname IN ('mysql1, 'mysql2’)\G

*************************** 1. row ***************************

hostgroup_id: 0

hostname: mysql1

status: ONLINE

*************************** 2. row ***************************

hostgroup_id: 1

hostname: mysql2

status: ONLINE

2 rows in set (0.00 sec)

86

Failover - Replication Hostgroupsmysql> SELECT * FROM mysql_replication_hostgroups\G

*************************** 1. row ***************************

writer_hostgroup: 0

reader_hostgroup: 1

comment:

1 row in set (0.00 sec)

87

ProxySQL Failover ProTIP

ProxySQL does not handle promotion or re-slaving

External process needed, such as MHA or Orchestrator

Failover DemoClick to add text

New FeaturesClick to add text

ClickHouse ServerNew in 1.4.3 – experimental (not fully completed, subject to change)

Support for ClickHouse: https://clickhouse.yandex/

Fast, really fast data analytics using MySQL client

Users management

Supports only one backendSupports only DDL and SELECT (no INSERT)

https://clickhouse.yandex/

ClickHouse Server

ClickHouse Server

SQLite3 ServerNew in 1.4.3 – experimental (not fully completed, subject to change)

Support for SQLite3 : https://www.sqlite.org/

Fully transactional, in-memory

Same users as MySQL backends

A lot of features in the roadmap

https://www.sqlite.org/

You-Name-It Server

What other service should become part of ProxySQL ?

You-Name-It Server

What other service should become part of ProxySQL ?

Causal read with GTIDFeature introduced in 2.0

Ensures that reads sent to slave returns what the client has written on master

Requires GTID and MySQL 5.7+

97

ProxSQL Read / Write Split

98

ProxySQL Binlog Reader

99

ProxySQL Binlog Reader

100

We offer support

Subscriptions

Consulting

Professional services

102

Rate My Session

Thank You!

Documents

High Performance & High Availability Proxy for MySQL ......complex query routing and read/write split load balancing real time statistics monitoring Data masking Multiple instances