MySQL Cluster is the distributed, shared-nothing version of MySQL. It’s typically used for applications that need any combination of high availability, real-time performance, and scaling of reads and writes. After a brief introduction to the technology, its uses, and the new features added in MySQL Cluster 7.3, this session focuses on the very latest developments happening in MySQL Cluster 7.4. As you’d expect from a real-time, scalable, distributed, in-memory database, performance continues to be a top priority, as do simplicity of use and robustness. Come hear firsthand what’s being done to make sure MySQL Cluster continues to dominate in mission-critical, high-performance applications.
Citation preview
1. MySQL Cluster Dive into the Latest Developments Copyright
2014, Oracle and/or its affiliates. All rights reserved. | Andrew
Morgan @andrewmorgan clusterdb.com Bernhard Ocklin
[email protected] 2nd October 2014
2. Safe Harbor Statement The following is intended to outline
our general product direction. It is intended for information
purposes only, and may not be incorporated into any contract. It is
not a commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality
described for Oracles products remains at the sole discretion of
Oracle. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
3. Todays Database Requirements Extreme Write Scalability
Real-Time User Experience Rock Solid Reliability Rapid Service
Innovation Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
4. Todays Database Requirements Transactional Integrity OLTP +
Real-Time Analytics Extreme Write Scalability Real-Time User
Experience Standards & Skillsets Rock Solid Reliability Rapid
Service Innovation Copyright 2014, Oracle and/or its affiliates.
All rights reserved. |
5. Whos Using MySQL Cluster Copyright 2014, Oracle and/or its
affiliates. All rights reserved. |
6. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | MySQL Cluster Overview Auto-Sharding, Multi-Master ACID
Compliant, OLTP + Real-Time Analytics HIGH SCALE, READS + WRITES
Shared nothing, no Single Point of Failure Self Healing + On-Line
Operations 99.999% AVAILABILITY In-Memory Optimization + Disk-Data
Predictable Low-Latency, Bounded Access Time REAL-TIME Key/Value +
Complex, Relational Queries SQL + Memcached + JavaScript + Java +
HTTP/REST & C++ SQL + NoSQL Open Source + Commercial Editions
Commodity hardware + Management, Monitoring Tools LOW TCO
7. MySQL Cluster Architecture MySQL Cluster Data Nodes Clients
Application Layer Data Layer Copyright 2014, Oracle and/or its
affiliates. All rights reserved. |
8. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | MySQL Cluster Scaling MySQL Cluster Data Nodes Clients
Application Layer Data Layer
9. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
10. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | MySQL Cluster HA MySQL Cluster Data Nodes Clients
Application Layer Data Layer
11. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
12. Scale On-Line Operations Backup Evolve Copyright 2014,
Oracle and/or its affiliates. All rights reserved. | Upgrade
13. MySQL Cluster 7.3: 1.2 Billion UPDATEs per Minute NoSQL C++
API, flexaSynch benchmark 30 x Intel E5-2600 Intel Servers, 2
socket, 64GB ACID Transactions, with Synchronous Replication
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
| 25 20 15 10 5 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Millions
of UPDATEs per Second MySQL Cluster Data Nodes
14. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
15. NoSQL Access to MySQL Cluster data Apps Apps Apps Apps Apps
Apps Apps Apps Apps Apps Apps Apps JPA Cluster JPA PHP Perl Python
Ruby JDBC Cluster J JS Apache Memcached MySQL JNI Node.JS mod_ndb
ndb_eng NDB API (C++) MySQL Cluster Data Nodes Copyright 2014,
Oracle and/or its affiliates. All rights reserved. |
16. Cluster & Memcached Schema-Free key value key value Key
Value town:maidenhead SL6 generic table Copyright 2014, Oracle
and/or its affiliates. All rights reserved. | Application view SQL
view
17. Cluster & Memcached - Configured Schema key value
prefix key value Prefix Table Key-col Val-col policy town: map.zip
town code cluster Copyright 2014, Oracle and/or its affiliates. All
rights reserved. | Config tables town ... code ... maidenhead ...
SL6 ... map.zip Application view SQL view
18. Real-Time Analytics - Adaptive Query Localization Scale
Complex Queries across Shards JOINs pushed down to data nodes
Executed in parallel Returns single result set to MySQL Opens Up
New Use-Cases Real-time analytics Recommendations engines
Fraud-detection Copyright 2014, Oracle and/or its affiliates. All
rights reserved. | mysqld Data Nodes mysqld A Q L Data Nodes DONT
COMPROMISE FUNCTIONALITY TO SCALE-OUT !!
19. Query Execution Time Seconds 87.23 seconds MySQL Cluster
7.1 MySQL Cluster 7.2 Copyright 2014, Oracle and/or its affiliates.
All rights reserved. | Real-World Test Case 100 90 80 70 60 50 40
30 20 10 0 Web-Based Content Management System JOINs 11-tables,
33.5k rows Returns 2k rows, 19 columns per row 1.26 seconds 70x
Faster
20. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Foreign Key Support MySQL 5.6 Auto-Installer NoSQL
JavaScript for node.js
21. MySQL Cluster 7.3: Foreign Keys Brings MySQL Cluster to a
broader range of workloads Packaged apps, custom projects Adds
powerful functionality while reducing complexity App logic &
data model Enabled by default Enforced for SQL & NoSQL APIs
On-line add and drop Copyright 2014, Oracle and/or its affiliates.
All rights reserved. |
22. MySQL Cluster 7.3: Auto-Installer Copyright 2014, Oracle
and/or its affiliates. All rights reserved. | Fast configuration
Auto-discovery Workload optimized Repeatable best practices Specify
Workload Auto- Discover Define Deploy Topology
23. MySQL Cluster 7.3: Connection Thread Scalability Increases
throughput of each connection to the data layer More client threads
can use each connection by splitting mutexes Reduces configuration
complexity Increases cluster scalability Each connection consumes
one of 256 node ids Up to 7.5x higher performance per connection
SQL & NoSQL interfaces Copyright 2014, Oracle and/or its
affiliates. All rights reserved. | MySQL Cluster Data Nodes MySQL
Cluster Data Nodes With Connection Thread Scalability
24. MySQL Cluster 7.3: Connection Thread Scalability DBT2
Benchmark Single MySQL Server Single Data Node 128 client
connections Copyright 2014, Oracle and/or its affiliates. All
rights reserved. | 8.5x
25. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Node.js NoSQL API Native JavaScript access to MySQL
Cluster End-to-End JavaScript: browser to the app & DB Storing
and retrieving JavaScript objects directly in MySQL Cluster
Eliminate SQL transformation Implemented as a module for node.js
Integrates Cluster API library within the web app Couple high
performance, distributed apps, with high performance distributed
database Optionally routes through MySQL Server V8 JavaScript
Engine MySQL Cluster Node.js Module MySQL Cluster Data Nodes
Clients
26. MySQL Cluster 7.4.1 DMR Available Now! Copyright 2014,
Oracle and/or its affiliates. All rights reserved. |
27. MySQL Cluster 7.4.1 DMR Better performance and operational
simplicity Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Performance gain over 7.3 47% (Read-Only) 38%
(Read-Write) Faster maintenance operations Nodal & Rolling
restarts Upgrades
28. Asynchronous replication between MySQL Clusters
Active-Active Update anywhere Conflict detection Application
notified through exception tables Can opt to have conflicts
resolved automatically Auto-conflict-resolution Conflicting
transaction and dependent ones are rolled-back No changes to
application schema Active-Active Geo-Replication Copyright 2014,
Oracle and/or its affiliates. All rights reserved. |
29. Spend $40 Add $100 Copyright 2014, Oracle and/or its
affiliates. All rights reserved. | What is a conflict?
John.balance==$100 John.balance-=$40 John.balance==$60
John.balance==$200 John.balance==$100 John.balance+= $100
John.balance==$200 John.balance==$600 $60 $200
30. MySQL Cluster 7.4.1 DMR Detects conflicting inserts/updates
Entire transactions (and dependent ones) rolled back All conflicts
are handled before switching primary Later in MySQL Cluster 7.4
Conflicting deletes Rolling back of transactions that read
conflicted data Copyright 2014, Oracle and/or its affiliates. All
rights reserved. | Handling of Conflicts
31. Detecting Conflicts - Reflected GCI Primary store logical
timestamp (GCI) against updated row Window for conflict opens GCI
replicated with updated row to Secondary The same row and GCI is
replicated back (reflected) from Secondary to Primary after it has
been applied Closing window for conflict Primary checks every event
originating from the Secondary to ensure it isnt for a conflictable
row Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Primary Secondary John.balance==$100 John.balance-=$40
John.balance==$60 John.balance==$200 John.balance==$100
John.balance+= $100 John.balance==$200 John.balance==$600 Spend $40
Add $100 $60 $200
32. How to Use Conflict Detection/Resolution Copyright 2014,
Oracle and/or its affiliates. All rights reserved. | Decide which
tables need protecting For each table, specify what to do on
conflicts Just record in exception table Application or DBA acts on
content Rollback the conflicting row Rollback the conflicting
transaction
33. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Restart Times What operations benefit? Restarting data
node with locally checkpointed data Major improvement Restarting
data node which must recover data from peer Major improvement
Further speedups to come in 7.4.X (greater parallelization)
Upgrade/rolling restarts Major improvement Cluster shutdown and
restart Minor improvement
34. Enhanced Memory Reporting mysql> CREATE DATABASE
clusterdb;USE clusterdb; mysql> CREATE TABLE simples (id INT NOT
NULL AUTO_INCREMENT PRIMARY KEY) ENGINE=NDB; mysql> select
node_id AS node, fragment_num AS frag, fixed_elem_alloc_bytes
alloc_bytes, fixed_elem_free_bytes AS free_bytes,
fixed_elem_free_rows AS spare_rows from memory_per_fragment where
fq_name like '%simples%';
+------+------+-------------+------------+------------+ | node |
frag | alloc_bytes | free_bytes | spare_rows |
+------+------+-------------+------------+------------+ | 1 | 0 |
131072 | 5504 | 172 | | 1 | 2 | 131072 | 1280 | 40 | | 2 | 0 |
131072 | 5504 | 172 | | 2 | 2 | 131072 | 1280 | 40 | | 3 | 1 |
131072 | 3104 | 97 | | 3 | 3 | 131072 | 4256 | 133 | | 4 | 1 |
131072 | 3104 | 97 | | 4 | 3 | 131072 | 4256 | 133 |
+------+------+-------------+------------+------------+ Copyright
2014, Oracle and/or its affiliates. All rights reserved. | See how
much memory a table is using
36. Enhanced Memory Reporting Check how well
partitioned/sharded a table is mysql> CREATE TABLE simples (id
INT NOT NULL AUTO_INCREMENT, species VARCHAR(20) DEFAULT "Human",
PRIMARY KEY(id, species)) engine=ndb PARTITION BY KEY(species);
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
| // Add some data mysql> select node_id AS node, fragment_num
AS frag, fixed_elem_alloc_bytes alloc_bytes, fixed_elem_free_bytes
AS free_bytes, fixed_elem_free_rows AS spare_rows from
ndbinfo.memory_per_fragment where fq_name like '%simples%';
+------+------+-------------+------------+------------+ | node |
frag | alloc_bytes | free_bytes | spare_rows |
+------+------+-------------+------------+------------+ | 1 | 0 | 0
| 0 | 0 | | 1 | 2 | 196608 | 11732 | 419 | | 2 | 0 | 0 | 0 | 0 | |
2 | 2 | 196608 | 11732 | 419 | | 3 | 1 | 0 | 0 | 0 | | 3 | 3 | 0 |
0 | 0 | | 4 | 1 | 0 | 0 | 0 | | 4 | 3 | 0 | 0 | 0 |
+------+------+-------------+------------+------------+
37. Oracle MySQL HA & Scaling Solutions Copyright 2014,
Oracle and/or its affiliates. All rights reserved. | MySQL
Replication MySQL Fabric Oracle VM Template Oracle Clusterware
Solaris Cluster Windows Cluster DRBD MySQL Cluster App
Auto-Failover Data Layer Auto-Failover Zero Data Loss MySQL 5.7
MySQL 5.7 Platform Support All All Linux Linux Solaris Windows
Linux All Clustering Mode Master + Slaves Master + Slaves
Active/Pas sive Active/Passi ve Active/P assive Active/Pas sive
Active/P assive Multi- Master Failover Time N/A Secs Secs + Secs +
Secs + Secs + Secs + < 1 Sec Scale-out Reads Cross-shard
operations N/A N/A N/A N/A N/A N/A Transparent routing For HA
Shared Nothing Storage Engine InnoDB+ InnoDB+ InnoDB+ InnoDB+
InnoDB+ InnoDB+ InnoDB+ NDB Single Vendor Support 37
38. When to Consider MySQL Cluster Scalability demands Sharding
for write performance? Latency demands Cost of each millisecond?
Uptime requirements Cost per minute of downtime? Failure versus
maintenance? Application agility Developer languages and
frameworks? SQL or NoSQL? Copyright 2014, Oracle and/or its
affiliates. All rights reserved. |
39. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
40. MySQL Cluster Auto-Installer Copyright 2014, Oracle and/or
its affiliates. All rights reserved. | Fast configuration
Auto-discovery Workload optimized Repeatable best practices Specify
Workload Auto- Discover Define Deploy Topology
41. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. |
42. Enhancing DevOps Agility, Reducing Downtime Copyright 2014,
Oracle and/or its affiliates. All rights reserved. | MySQL Cluster
Manager Automated Management Start / Stop node or whole cluster
On-Line Scaling On-Line Reconfiguration On-Line Upgrades On-Line
Backup & Restore Import Running Cluster Self-Healing Node
monitoring Auto-recovery extended to SQL + mgmt nodes HA Operations
Cluster-wide configuration consistency Persistent configurations HA
Agents
43. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Upgrade using MCM MySQL Cluster Data Nodes Application
Layer Data Layer upgrade cluster --package=7.3 mycluster;
44. MySQL Cluster Manager 1.3.2 GA Available Now! Copyright
2014, Oracle and/or its affiliates. All rights reserved. |
45. MySQL Cluster Manager 1.3.2 GA Unmanaged production Cluster
mcm> create cluster --import mcm> import config [--dryrun]
mcm> import cluster[--dryrun] Cluster now managed by MCM
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
| Import a running Cluster into MCM
46. Oracle University MySQL Training Services Prepare Your
Organization to Enable Reliable and High-Performance Web-Based
Database Applications Top Courses for Administrators and Developers
Top Certifications Copyright 2014, Oracle and/or its affiliates.
All rights reserved. | Training and team skill have the most
significant impact on overall performance of technology and success
of technology projects. - IDC, 2013 Premier Support customers
eligible to save 20% on learning credits. Benefits Expert-led
training to support your MySQL learning needs Flexibility to train
in the classroom or online Hands-on experience to gain real world
experience Key skills needed for database administrators and
developers MySQL for Beginners MySQL for Database Administrators
MySQL Performance Tuning MySQL Cluster NEW - Register Your
Interest! MySQL and PHP - Developing Dynamic Web Applications MySQL
for Developers MySQL Developer Techniques MySQL 5.6 Database
Administrator MySQL 5.6 Developer To find out more about available
MySQL Training & Certification offerings, go to:
education.oracle.com/mysql RECENTLY RELEASED !!ALL NEW!! MySQL
Cluster Training To Register your interest to influence the
schedule on this newly released course go to
education.oracle.com/mysql and click on the MySQL Cluster
Course
47. Copyright 2014, Oracle and/or its affiliates. All rights
reserved. | Next Steps Learn More www.mysql.com/cluster Authentic
MySQL Curriculum: http://oracle.com/education/mysql Try it Out
dev.mysql.com/downloads/cluster/ Let us know what you think
bugs.mysql.com forums.mysql.com/list.php?25