Upload
suresh81c
View
221
Download
0
Embed Size (px)
Citation preview
8/10/2019 Nocoug 201311 Mysql Bigdata
1/47
Using MySQL for Big Data AdvantageIntegrate for Insight
Sastry [email protected] Copyright 2013, Oracle and/
8/10/2019 Nocoug 201311 Mysql Bigdata
2/47
Agenda The rise of Big Data & Hadoop
MySQL in the Big Data Lifecy
MySQL Solutions for Big Data
Q&A
Copyright 2013, Oracle and/or
8/10/2019 Nocoug 201311 Mysql Bigdata
3/47
The following is intended to outline our general product direction. It
intended for information purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any material, code, or functionality
and should not be relied upon in making purchasing decision. Th
development, release, and timing of any features or functionality described for
Oracle s products remains at the sole discretion of Oracle.
Safe Harbor Statement
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
8/10/2019 Nocoug 201311 Mysql Bigdata
4/47
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
McKinsey Glo
MySQL Enterprise Monitor 2.2MySQL Cluster 7.1MySQL Cluster Manager 1.0MySQL Workbench 5.2MySQL Database 5.5MySQL Enterprise Backup 3.5
MySQL Enterprise Monitor 2.3MySQL Cluster Manager 1.1
DRIVING MySQLINNOVATION
All GA!
MySQL Enterprise Backup 3.7
Oracle VM Template for MySQLEnterprise Edition
MySQL Enterprise OracleCertificationsMySQL Windows Installer
MySQL Enterprise Security
MySQL Enterprise Scalability
MySQL Database 5.6 DMR*
MySQL Cluster 7.2 DMR
MySQL Labs!
(
early and often
)
All GA!
MySQL Clus
MySQL Clus
MySQL Utili
MySQL Migr
MySQL EnteMySQL Ente
MySQL Data
MySQL Clus
MySQL Data
A BEM
*Development Milestone Release
2010 2011
All
8/10/2019 Nocoug 201311 Mysql Bigdata
5/47
Filesystems, Files and LogsRedo, Undo, Data, Index, Binary, Error, Query and Slow
Pluggable Storage Engines Architecture
ConnectorsNative C API, JDBC, ODBC, .Net, PHP, Ruby, Python, VB, Perl
Connection Pool Authentication Thread Reuse Connection Limits Check Memory
Enterprise ManagementServices and Utilities
Backup & RecoverySecurity
ReplicationCluster
PartitioningInstance Manager
Information_SchemaMySQL Workbench
SQL InterfaceDDL, DML, StoredProcedures, Views,
Triggers, Etc..
ParserQuery Translation,Object Privileges
Optimizer Access Paths,
Statistics
Pluggable Storage EnginesMemory, Index and Storage Management
InnoDB MyISAM Cluster Etc Partners Comm
MySQL ServerClients and Apps
2010 Oracle Corporation Proprietary and Confidential
8/10/2019 Nocoug 201311 Mysql Bigdata
6/47
Cloud
Web & Enterprise OEM & ISV
Industry Leaders Rely on MySQL
http://images.google.com/imgres?imgurl=http://www.infinitytechnology.com/uploadedImages/Home/TechFocus_News/AlcatelLucent_Hor_2col_lrg.gif&imgrefurl=http://www.infinitytechnology.com/phonesystems.aspx&h=251&w=720&sz=11&hl=en&start=4&sig2=urYvX47BOQRqvADY_UMA0w&um=1&tbnid=pA91nzVKrUFImM:&tbnh=49&tbnw=140&ei=94SgSOaRCaqkpATZu-jgBw&prev=/images?q=alcatel+lucent&um=1&hl=en&suggon=0&rls=com.microsoft:en-us&sa=N8/10/2019 Nocoug 201311 Mysql Bigdata
7/47
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
McKinsey G
Scales to 48 CPU Threads Up to 230% performance gain over MySQL 5.5
IMPROVED PERFORMANCE AND SCALABILITY
Better transactional throughput and availabilityIMPROVED INNODB
Better query exec times and diagnostics for query tuning and debuggingIMPROVED OPTIMIZER
Higher performance, availability and data integrity
IMPROVED REPLICATION
Better Instrumentation, User/Application level statistics and monitoring
IMPROVED PERFORMANCE SCHEMA
Fast, Key Value access with full ACID compliance, better developer agility
New! NoSQL ACCESS TO INNODB
MySQL 5.6: In Summary
8/10/2019 Nocoug 201311 Mysql Bigdata
8/47
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
McKinsey Glo
Multi-Threaded Slaves Binary Log Group Commit
Optimized Row-Based Replication
PERFORMANCE
Global Transaction Identifiers Replication Failover & Admin Utilities Crash Safe Slaves
FAILOVER & RECOVERY
Replication Event ChecksumsDATA INTEGRITY
Time Delayed Replication Remote Binlog Backup
Informational Log Events
DEV/OPS AGILITY
MySQL 5.6: Best Replication Features Ever
8/10/2019 Nocoug 201311 Mysql Bigdata
9/47
Leading Use-Case, On-Line Retail
Users
Browsing
RecommendationsProf
PurchHist
Web Logs:Pages Viewed
Comments Posted
Social media updatesPreferences
Brands Liked
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Telephony Stream
8/10/2019 Nocoug 201311 Mysql Bigdata
10/47
BI Solutions
ACQUIRE
ORGANIZANALYZE
DECIDE
MySQL in the Big Data Lifecycle
8/10/2019 Nocoug 201311 Mysql Bigdata
11/47
*Leading Hadoop Vendor
Copyright 2013, Oracle and/or
Download the MySQL Guide to Big Data:http://www.mysql.com/why-mysql/white-papers/mysql-and-hadoop-guide-to-big-data-integra
8/10/2019 Nocoug 201311 Mysql Bigdata
12/47
NoSQLInterfaceMySQL Da
MySQL Clu
MySQL in the Big Data Lifecycle
ANALYZE
DECIDE
ORGANIZE
ACQUIRE
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
8/10/2019 Nocoug 201311 Mysql Bigdata
13/47
MySQL NoSQL InterfaceDesign Goals: Fast, Flexible and Safe
Fully transactional / ACID
Blazing FastKey / Value
Queries
NoSQLacross
Dat
Combined with Schema Flexibility: Online DDL
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
8/10/2019 Nocoug 201311 Mysql Bigdata
14/47
Memcached is in-memory key-value storefor small dataIt is one of the most widely used In-Memory cache implementations for socialnetwork websitesMemcached has a simple and openprotocol as opposed to a rich client bound
to a specific language, andimplementation makes it portable across awide variety of languages andenvironments
How Memcached is used with MySQL separately
MySQL
clientSimple protocol
Set/get/add/delete
SQL query
Save/retrieve copy
8/10/2019 Nocoug 201311 Mysql Bigdata
15/47
InnoDB as a Key Value store
Combine the best of the NoSQL world and SQL worldMemcached listens on specific ports as the front end,directs requests directly to InnoDBSimple commands, much smaller network transmitpackagesPersistent storage from InnoDBIndex on the key column
Full ACID complianceBypass Optimizer and QP layer of MySQL anddirectly access the storage engineDual access of data (SQL and Memcached)
8/10/2019 Nocoug 201311 Mysql Bigdata
16/47
Key-value accessBypasses SQL pa
Implemented via:- Memcached plu- Memcached m
InnoDB API- Use existing M- Shared process
latency
MySQL 5.6: NoSQL Interface to InnoDBMemcached API
Copyright 2013, Oracle and/or
8/10/2019 Nocoug 201311 Mysql Bigdata
17/47
Performance
Up to 9x Higher SET / INSERT Throughpu
0
10000
20000
30000
40000
50000
60000
70000
80000
8 32 128 512
T P S
Client Connections
MySQL 5.6: NoSQL Benchmarking
Memcached API
SQL
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
M SQL Cl M l i l N SQL I f
8/10/2019 Nocoug 201311 Mysql Bigdata
18/47
MySQL Cluster: Multiple NoSQL Interfaces
Mix&
Match
Copyright 2013, Oracle and/or
8/10/2019 Nocoug 201311 Mysql Bigdata
19/47
NoSQL C++ API, flexaSynch benchmark 30 x Intel E5-2600 Intel Servers, 2 socket, 64GB ACID Transactions, with Synchronous Replication
0
5
10
15
20
25
2 4 6 8 10 12 14 16 18 20 22 24 26 28 30
M i l l i o n s o
f U P D A T E s p e r
S e c o n
d
MySQL Cluster Data Nodes
1.2 Billion UPDATEs per Minute
Copyright 2013, Oracle and/or
8/10/2019 Nocoug 201311 Mysql Bigdata
20/47
ImportApache SqoMySQL Ha
MySQL in the Big Data Lifecycle
ANALYZE
DECIDE ACQUIRE
ORGANIZE
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
8/10/2019 Nocoug 201311 Mysql Bigdata
21/47
Apache Sqoop
Apache TLP, part of Hadoop project Developed by Cloudera
Bulk data import and export Between Hadoop (HDFS) and external data stores
JDBC Connector architecture Supports plug-ins for specific functionality
Fast Path Connector developed for MySQL
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
I ti g D t
8/10/2019 Nocoug 201311 Mysql Bigdata
22/47
Transactional
Data
HDFSStorage
Sqoop Job
Map
Map
Map
Map
Hadoop Cluster
Sqoop Import
Gather Metadata
Submit Map Only JobImporting Data
Copyright 2013, Oracle and/or
8/10/2019 Nocoug 201311 Mysql Bigdata
23/47
Hadoop Applier: Design
8/10/2019 Nocoug 201311 Mysql Bigdata
24/47
Hadoop Applier: Design
Uses MySQL replication techniques forreal time integration
Binlog API uses Binary Log to rapidly
fetch new data from a running servervia the replication protocol MySQL Binlog comprised of events,
each event represents a databasechange
Hadoop Applier receives the eventsusing the Binlog API, and writes thechanges into a file in HadoopDistributed File System
Other tools in Hadoop Ecosystem,suchas Apache Hive, can then consumethis data
8/10/2019 Nocoug 201311 Mysql Bigdata
25/47
Hadoop Applier: Implementation
8/10/2019 Nocoug 201311 Mysql Bigdata
26/47
Hadoop Applier: Implementation
Replicates rows inserted into a table in MySQL to Hadoop Distributed File Syste
Uses an API provided by libhdfs, a C library to manipulate files in HDFS
The library comes pre-compiled with Hadoop Distributions
Connects to the MySQL master (or reads the binary log generated by MySQL) toFetch the row insert events occurring on the masterDecode these events, extracting data inserted into each field of the rowSeparate the data by the desired field delimiters and row delimitersUse content handlers to get it in the format required
Append it to a text file in HDFS
8/10/2019 Nocoug 201311 Mysql Bigdata
27/47
Integration with HIVE
Hive runs on top of Hadoop. Install HIVE onthe hadoop master node
Set the default datawarehouse directorysame as the base directory into whichHadoop Applier writesCreate similar schema's on both MySQL &
HiveTimestamps are inserted as first field in
HDFS filesData is stored in 'datafile1.txt' by defaultThe working directory is
base_dir/db_name.db/tb_name
SQL Query
CREATE TABLE t (iINT);
CREtimeINT[RO
DELSTOTEX
d h
8/10/2019 Nocoug 201311 Mysql Bigdata
28/47
Mapping Between MySQL and HDFS Schema
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
8/10/2019 Nocoug 201311 Mysql Bigdata
29/47
MySQL in the Big Data Lifecycle
AnalyzeExport DDecide
ANALYZE
DECIDE
Copyright 2013, Oracle and/or
A l Bi D
8/10/2019 Nocoug 201311 Mysql Bigdata
30/47
Analyze Big Data
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Export Data
8/10/2019 Nocoug 201311 Mysql Bigdata
31/47
Export Data
HDStor
Sqoop Job
Map
Map
Map
Map
Hadoop Cluster
Sqoop Export
Gather Metadata
Submit Map Only Job
Transactional
& AnalyticsData
Copyright 2013, Oracle and/or
M SQL R ti D t b f BI
8/10/2019 Nocoug 201311 Mysql Bigdata
32/47
MySQL Reporting Database for BI
Copyright 2013, Oracle and/or
MySQL Operational Database for Web
8/10/2019 Nocoug 201311 Mysql Bigdata
33/47
MySQL Operational Database for Web
Copyright 2013, Oracle and/or
l d
8/10/2019 Nocoug 201311 Mysql Bigdata
34/47
Oracle PremierLifetime Support
Oracle ProductCertifications/Integrations
MySQL EnterpriseHigh Availability
MySQL EnterpriseSecurity
MySQL EnterpriseScalability
MySQL EntBacku
MySQL EnterpriseMonitor/Query Analyzer
MySQL
Data Analysis: MySQL Enterprise EditionHighest Levels of Security, Performance and Availability
MySQL EnterpriseAudit
Copyright 2013, Oracle and/or
MySQL Enterprise Monitor with Query Analyze
8/10/2019 Nocoug 201311 Mysql Bigdata
35/47
MySQL Enterprise Monitor with Query Analyze Tune Analytical Queries Enhance DevOps Agility
Copyright 2013, Oracle and/or
Scaling Security and Data Protection
8/10/2019 Nocoug 201311 Mysql Bigdata
36/47
Scaling, Security and Data Protection
MySQL Enterprise Scalability
MySQL Enterprise Security
MySQL Enterprise Audit
MySQL Enterprise Backup
Copyright 2013, Oracle and/or
MySQL Enterprise Backup
8/10/2019 Nocoug 201311 Mysql Bigdata
37/47
MySQL Enterprise Backup Online Backup for InnoDB Full, Incremental, Partial Backups (scriptable interface Compression Point in Time, Full, Partial Recovery options Metadata on status, progress, history
Unlimited Database Size Cross-Platform
Windows, Linux, Unix
Certified with Oracle Secure Backup
MEB Backup
Files
mysqlbackup
Ensures quick , on l ine backup and recovery of your MySQ
M SQL Enterprise Sec rit
8/10/2019 Nocoug 201311 Mysql Bigdata
38/47
PAM (Pluggable Authentication Modules)
Access external authentication methods Standard interface (Unix, LDAP, others) proxied and non-proxied users
Windows
Access native Windows services Authenticate users already logged intoWindows (Windows Active Directory)
Pluggable Authentication API
MySQL Enterprise SecurityMySQL External Authentication
Integr ates MySQL with existing security infrastructures and SO
5 5 MySQL Enterprise Scalability
8/10/2019 Nocoug 201311 Mysql Bigdata
39/47
MySQL default thread-handling excellent
performance, can limit scalability as connections grow MySQL Thread Pool improves sustainedperformance/scale as user connections grow
5.5 MySQL Enterprise ScalabilityMySQL Thread Pool
Thread Pool
8/10/2019 Nocoug 201311 Mysql Bigdata
40/47
Thread Pool
MySQL Enterprise Audit
8/10/2019 Nocoug 201311 Mysql Bigdata
41/47
Out-of-the-box logging of connections, logins, queryactivity across all or specific MySQL serversUser defined policies, filtering and log rotationDynamically enabled, disabled: no server restartXML-based audit stream per Oracle audit specificatio
Easily implemented via MySQL 5.5 Audit APIMySQL 5.5.28 and higher
Get it here: support.oracle.com and edelivery.oracle.comAdd s regu la tory com pl iance to MySQL appl icat ions
MySQL Enterprise AuditPolicy-based Auditing for MySQL Applications
Oracle Premier Support for MySQL
8/10/2019 Nocoug 201311 Mysql Bigdata
42/47
Straight from the Source Largest Team of MySQL Experts Backed by MySQL Developers Forward Compatible Hot Fixes MySQL Maintenance Releases MySQL Support in 29 Languages 24/7/365 Unlimited Incidents Knowledge Base MySQL Consultative Support
pp y QRely on The Experts - Get Unique Benefits
"The MySQL support service hessential in helping us with troand providing recommendatioproduction cluster, Thanks."
-- Carlos Morales Playf
O
Copyright 2013, Oracle and/or
Summary
8/10/2019 Nocoug 201311 Mysql Bigdata
43/47
Summary
MySQL + Hadoop: widely deployed solution
Best of both worlds SQL + NoSQL Access
Tools and expertise to support you
End to end Oracle solutions for Big Data
Integrate for Insight
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Next Steps
8/10/2019 Nocoug 201311 Mysql Bigdata
44/47
Next Steps
Download the Guidehttp://www.mysql.com/why-mysql/white-
papers/mysql_wp_hadoop.php
Try Out MySQL 5.6http://www.mysql.com/downloads/mysql/
Engage MySQL Consultinghttp://www.mysql.com/consulting/
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
Thank you!
8/10/2019 Nocoug 201311 Mysql Bigdata
45/47
Thank you!
Copyright 2013, Oracle and/or its affiliates. All rights reserved.
8/10/2019 Nocoug 201311 Mysql Bigdata
46/47
Copyright 2013, Oracle and/or
8/10/2019 Nocoug 201311 Mysql Bigdata
47/47