Upload
dangthu
View
218
Download
3
Embed Size (px)
Citation preview
© 2014 IBM Corporation
Analyzing SQL on Hadoop
3.03.0
Big!
SQLUday Kale ([email protected])Big SQL Development, IBM InfoSphere BigInsights
18-Sep-2014
Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
Agenda
Why SQL on Hadoop?
SQL-on-Hadoop landscape
Big SQL 3.0• What is it?• SQL capabilities• Architecture
Conclusion
3
Why SQL for Hadoop?
Hadoop is designed for any data• Doesn't impose any structure• Extremely flexible
At lowest levels is API based• Requires strong programming
expertise
• Steep learning curve• Even simple operations can be
tedious
Yet many, if not most, use cases deal with structured data!• e.g. aging old warehouse data into queriable archive
Why not use SQL in places its strengths shine?• Familiar widely used syntax• Separation of what you want vs. how to get it• Robust ecosystem of tools
Pre-Processing Hub Query-able Archive Exploratory Analysis
Information Integration
Data Warehouse
StreamsReal-time processing
BigInsightsLanding zone
for all data
Data Warehouse
BigInsights Can combine with
unstructured information
Data Warehouse
1 2 3
SQL-on-Hadoop landscape
The SQL-on-Hadoop landscape is changing rapidly!
They all have their different strengths and weaknesses
Many, including Big SQL, draw their basic designs on Hive…
Agenda
Why SQL on Hadoop?
SQL-on-Hadoop landscape
Big SQL 3.0• What is it?• SQL capabilities• Architecture
Conclusion
6
BigInsights Enterprise Edition
Connectivity and Integration
StreamsNetezza
Text processing engine and library
JDBC
Flume
Infrastructure Jaql
Hive
Pig
HBase
MapReduce
HDFS
ZooKeeperLucene
Adaptive MapReduce
Oozie
Text compression
Enhanced security
Flexible scheduler
Optional IBM and partner offerings
Analytics and discovery “Apps”
DB2
BigSheets
Web Crawler
Distrib file copy
DB export
Boardreader
DB import
Ad hoc query
Machine learning
Data processing
. . .
Administrative and development tools
Web console
• Monitor cluster health, jobs, etc. • Add / remove nodes• Start / stop services• Inspect job status• Inspect workflow status• Deploy applications • Launch apps / jobs • Work with distrib file system•Work with spreadsheet interface•Support REST-based API • Create / view alerts • . . .
Big R
Eclipse tools
• Text analytics• MapReduce programming• Jaql, Hive, Pig development• BigSheets plug-in development• Oozie workflow generation
Integrated installer
Open Source IBM IBM
Cognos BI
Big SQL
Accelerator for machine data analysis
Accelerator for social data analysis
Guardium DataStageData Explorer
Sqoop
HCatalogGPFS –FPOGPFS –FPO
Solr
Big SQL 3.0 – At a glance
Application Portability & Integration
Data shared with Hadoop ecosystem
Comprehensive file format support
Superior enablement of IBM software
Enhanced by Third Party software
Performance
Modern MPP runtime
Powerful SQL query rewriter
Cost based optimizer
Optimized for concurrent user throughput
Results not constrained by memory
Federation
Distributed requests to multiple data sources within a single SQL statement
Main data sources supported:DB2 LUW, DB2/z, Teradata, Oracle,
Netezza
Enterprise Features
Advanced security/auditing
Resource and workload management
Self tuning memory management
Comprehensive monitoring
Rich SQLComprehensive SQL Support
IBM SQL PL compatibility
Creating tables in Big SQL
Big SQL syntax is derived from Hive's syntax with extensions
create hadoop table users( id int not null primary key, office_id int null, fname varchar(30) not null, lname varchar(30) not null, salary timestamp(3) null, constraint fk_ofc foreign key (office_id) references office (office_id))row format delimited fields terminated by '|'stored as textfile;
Creating tables in Big SQL
Big SQL syntax is derived from Hive's syntax with extensions
create hadoop table users( id int not null primary key, office_id int null, fname varchar(30) not null, lname varchar(30) not null, salary timestamp(3) null, constraint fk_ofc foreign key (office_id) references office (office_id))row format delimited fields terminated by '|'stored as textfile; Hadoop Keyword
• Big SQL requires the HADOOP keyword• Big SQL has internal traditional RDBMS table support
• Stored only at the head node• Does not live on HDFS• Supports full ACID capabilities• Not usable for "big" data
• The HADOOP keyword identifies the table as living on HDFS
Hadoop Keyword• Big SQL requires the HADOOP keyword• Big SQL has internal traditional RDBMS table support
• Stored only at the head node• Does not live on HDFS• Supports full ACID capabilities• Not usable for "big" data
• The HADOOP keyword identifies the table as living on HDFS
Creating tables in Big SQL
Big SQL syntax is derived from Hive's syntax with extensions
create hadoop table users( id int not null primary key, office_id int null, fname varchar(30) not null, lname varchar(30) not null, salary timestamp(3) null, constraint fk_ofc foreign key (office_id) references office (office_id))row format delimited fields terminated by '|'stored as textfile; Nullability Indicators
• Enforced on read• Used by query optimizer for smarter rewrites
Nullability Indicators• Enforced on read• Used by query optimizer for smarter rewrites
Creating tables in Big SQL
Big SQL syntax is derived from Hive's syntax with extensions
create hadoop table users( id int not null primary key, office_id int null, fname varchar(30) not null, lname varchar(30) not null, salary timestamp(3) null, constraint fk_ofc foreign key (office_id) references office (office_id))row format delimited fields terminated by '|'stored as textfile;
Constraints• Unenforced• Useful as documentation and to drive query builders• Used by query optimizer for smarter rewrites
Constraints• Unenforced• Useful as documentation and to drive query builders• Used by query optimizer for smarter rewrites
Table types
Big SQL supports many of the "standard" Hadoop storage formats• Text delimited• Text delimited sequence files• Binary delimited sequence files• Parquet• RC• ORC• Avro
Each has different features/advantages/disadvantages
Custom file formats may be supported as well via custom java classes
Populating Big SQL tables
There are a number of ways to populate tables
Tables can be defined against existing data• All validation is performed at query time
Rows can be directly inserted into tables
• Data is validated and converted to storage format • Only suitable for testing• Produces one physical data file per call to INSERT
create external hadoop table csv_data( c1 int not null primary key, c2 varchar(20) null)row format delimited fields terminated by ','stored as textfilelocation '/user/bob/csv_data'
insert into t1 values (5, 'foo'), (6, 'bar'), (7, 'baz')
Populating Big SQL tables (cont.)
Tables can be populated from other tables
Tables can be created from other tables• Great way to convert between storage types or partition data
insert into top_sellersselect employee_id, rank() over (order by sales) from ( select employee_id, sum(sales) sales from product_sales group by employee_id )limit 10;
create hadoop table partitioned_salespartitioned by (dept_id int not null)stored as rcfileasselect emp_id, prod_id, qty, cost, dept_id from sales
Populating Big SQL tables (cont.)
The LOAD HADOOP is used to populate Hadoop tables from an external data source
• Statement runs on the cluster – cannot access data at the client• Nodes of the cluster ingest data in parallel• Performs data validation during load• Performs data conversion (to storage format) during load
Supports the following sources of data• Any valid Hadoop URL (e.g. hdfs://, sftp://, etc.)
• JDBC data sources (e.g. Oracle, DB2, Netezza, etc.)
Loading from URL
Data may be loaded from delimited files read via any valid URL• If no URI specified is provided, HDFS is assumed:
Example loading via SFTP:
Just remember LOAD HADOOP executes on the cluster• So file:// will be local to the node chosen to run the statement
LOAD HADOOP USING FILE URL '/user/biadmin/mydir/abc.csv' INTO TABLE T1WITH SOURCE PROPERTIES( 'field.delimiter'=',', 'date.time.format'=''yyyy-MM-dd-HH.mm.ss.S')
LOAD HADOOP USING FILE URL sftp://biadmin:[email protected]:22/home/biadmin/mydir'
LOAD HADOOP USING FILE URL file:///path/to/myfile/file.csv'INTO TABLE T1
Loading from JDBC data source
A JDBC URL may be used to load directly from external data source• Tested internally against Oracle, Teradata, DB2, and Netezza
It supports many options to partition the extraction of data • Providing a table and partitioning column• Providing a query and a WHERE clause to use for partitioning
Example usage:
LOAD USING JDBC CONNECTION URL 'jdbc:db2://myhost:50000/SAMPLE' WITH PARAMETERS ( user = 'myuser', password='mypassword' ) FROM TABLE STAFF WHERE "dept=66 and job='Sales'" INTO TABLE staff_sales PARTITION ( dept=66 , job='Sales') APPEND WITH LOAD PROPERTIES (bigsql.load.num.map.tasks = 1) ;
SQL capabilities
Leverage IBM's rich SQL heritage, expertise, and technology• Modern SQL:2011 capabilities• DB2 compatible SQL PL support
– SQL bodied functions– SQL bodied stored procedures– Robust error handling– Application logic/security encapsulation– Flow of control logic
The same SQL you use on your data warehouse should run with few or no modifications
SQL capability highlights
Full support for subqueries• In SELECT, FROM, WHERE and HAVING• Correlated and uncorrelated • Equality, non-equality subqueries • EXISTS, NOT EXISTS, IN, ANY, SOME, etc.
All standard join operations• Standard and ANSI join syntax
• Inner, outer, and full outer joins• Equality, non-equality, cross join support• Multi-value join (WHERE (c1, c2) = …)• UNION, INTERSECT, EXCEPT
SELECT s_name, count(*) AS numwaitFROM supplier, lineitem l1, orders, nationWHERE s_suppkey = l1.l_suppkey AND o_orderkey = l1.l_orderkey AND o_orderstatus = 'F' AND l1.l_receiptdate > l1.l_commitdate AND EXISTS ( SELECT * FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey ) AND NOT EXISTS ( SELECT * FROM lineitem l3 WHERE l3.l_orderkey = l1.l_orderkey AND l3.l_suppkey <> l1.l_suppkey AND l3.l_receiptdate > l3.l_commitdate ) AND s_nationkey = n_nationkey AND n_name = ':1' GROUP BY s_nameORDER BY numwait desc, s_name;
SELECT s_name, count(*) AS numwaitFROM supplier, lineitem l1, orders, nationWHERE s_suppkey = l1.l_suppkey AND o_orderkey = l1.l_orderkey AND o_orderstatus = 'F' AND l1.l_receiptdate > l1.l_commitdate AND EXISTS ( SELECT * FROM lineitem l2 WHERE l2.l_orderkey = l1.l_orderkey AND l2.l_suppkey <> l1.l_suppkey ) AND NOT EXISTS ( SELECT * FROM lineitem l3 WHERE l3.l_orderkey = l1.l_orderkey AND l3.l_suppkey <> l1.l_suppkey AND l3.l_receiptdate > l3.l_commitdate ) AND s_nationkey = n_nationkey AND n_name = ':1' GROUP BY s_nameORDER BY numwait desc, s_name;
SQL capability highlights (cont.)
Extensive analytic capabilities• Grouping sets with CUBE and ROLLUP• Standard OLAP operations
•
•
• Analytic aggregates
LEAD LAG RANK DENSE_RANK
ROW_NUMBER RATIO_TO_REPORT FIRST_VALUE LAST_VALUE
CORRELATION COVARIANCE STDDEV VARIANCE
REGR_AVGX REGR_AVGY REGR_COUNT REGR_INTERCEPT
REGR_ICPT REGR_R2 REGR_SLOPE REGR_XXX
REGR_SXY REGR_XYY WIDTH_BUCKET VAR_SAMP
VAR_POP STDDEV_POP STDDEV_SAMP COVAR_SAMP
COVAR_POP NTILE
Architected for performance
Architected from the ground up for low latency and high throughput
MapReduce replaced with a modern MPP architecture• Compiler and runtime are native code (not java)• Big SQL worker daemons live directly on cluster
– Continuously running (no startup latency)– Processing happens locally at the data
• Message passing allows data to flow directly between nodes
Operations occur in memory with the ability to spill to disk
• Supports aggregations and sorts larger than available RAM
InfoSphere BigInsights
Big SQL
SQL MPP Runtime
Data Sources
Parquet CSV Seq RC
Avro ORC JSON Custom
SQL-basedApplication
IBM Data Server Client
Big SQL 3.0 – Architecture
23
Management Node
Big SQLMaster Node
Management Node
Big SQLScheduler
Big SQLWorker Node
JavaI/O
FMP
NativeI/O
FMP
HDFS Data Node
MRTask Tracker
Other ServiceHDFS
Data HDFSData HDFS
Data
TempData
UDF FMP
Compute Node
Database Service
Hive Metastore
Hive Server
Big SQLWorker Node
JavaI/O
FMP
NativeI/O
FMP
HDFS Data Node
MRTask Tracker
Other ServiceHDFS
Data HDFSData HDFS
Data
TempData
UDF FMP
Compute Node
Big SQLWorker Node
JavaI/O
FMP
NativeI/O
FMP
HDFS Data Node
MRTask Tracker
Other ServiceHDFS
Data HDFSData HDFS
Data
TempData
UDF FMP
Compute Node
DDLFMP
UDF FMP
*FMP = Fenced mode process
Scheduler Service
The Scheduler is the main RDBMS↔Hadoop service interface
Interfaces with Hive metastore for table metadata• Compiler ask it for some "hadoop" metadata, such as partitioning
columns
Acts like the MapReduce job tracker for Big SQL• Big SQL provides query predicates for scheduler to perform
partition elimination• Determines splits for each “table” involved in the query• Schedules splits on available Big SQL nodes
(favoring scheduling locally to the data)• Serves work (splits) to I/O engines• Coordinates “commits” after INSERTs
Management Node
Big SQLMaster Node
Big SQLScheduler
DDLFMP
UDF FMP
Mgmt Node
Database Service
Hive Metastore
Big SQLWorker Node
JavaI/O
FMP
NativeI/O
FMP
HDFS Data Node
MRTask TrackerUDF
FMP
I/O Processing
Native I/O FMP• The high-speed interface for common file formats• Delimited, Parquet, RC, Avro, and Sequencefile
Java I/O FMP• Handles all other formats via standard Hadoop/Hive API’s
Both perform multi-threaded direct I/O on local data
The database engine understands storage format capabilities• Projection list is pushed into I/O format• Predicates are pushed as close to the data as
possible (into storage format, if possible)• Predicates that cannot be pushed down are
evaluated within the database engine
The database engine is only aware of which nodesneed to read
• Scheduler directs the readers to their portion of work
Big SQLWorker Node
JavaI/O
FMP
NativeI/O
FMP
HDFS Data Node
MRTask Tracker
Other ServiceHDFS
Data HDFSData HDFS
Data
TempData
UDF FMP
Compute Node
Conclusion
Today, it seems, performance numbers are the name of the game
But in reality there is so much more…• How rich is the SQL?• How difficult is it to (re-)use your existing SQL?• How secure is your data?• Is your data still open for other uses on Hadoop?• Can your queries span your enterprise?• Can other Hadoop workloads co-exist in harmony?• …
With Big SQL 3.0 performance doesn't mean compromise
Questions?
Useful links• BigInsights Information Center• Big SQL technology preview (cloud demo)• BigInsights Trial VM Image• “IBM Analytics for Hadoop” Big Data Service on Bluemix
https://ace.ng.bluemix.net/#/solutions/solution=big_data
• Big SQL Tutorial and videos
https://developer.ibm.com/hadoop/docs/tutorials/big-sql-hadoop-tutorial/
https://developer.ibm.com/hadoop/videos/category/sql-on-hadoop-live-events/
http://www.ibm.com/support/knowledgecenter/SSPT3X_3.0.0/com.ibm.swg.im.infosphere.biginsights.tut.doc/doc/tut_intro_bigsql.html
https://developer.ibm.com/bluemix/2014/08/26/hands-on-with-hadoop-in-minutes/
• BigInsights Quickstart Edition (includes Big SQL)– Tutorial Video:
https://developer.ibm.com/hadoop/videos/getting-started-biginsights-quick-start-edition-single-node/
PerformanceQuery rewrites
• Exhaustive query rewrite capabilities• Leverages additional metadata such as constraints and nullability
Optimization• Statistics and heuristic driven query optimization• Query optimizer based upon decades of IBM RDBMS experience
Tools and metrics• Highly detailed explain plans and query diagnostic tools• Extensive number of available performance metrics
SELECT ITEM_DESC, SUM(QUANTITY_SOLD), AVG(PRICE), AVG(COST)
FROM PERIOD, DAILY_SALES, PRODUCT, STORE
WHERE
PERIOD.PERKEY=DAILY_SALES.PERKEY AND
PRODUCT.PRODKEY=DAILY_SALES.PRODKEY AND
STORE.STOREKEY=DAILY_SALES.STOREKEY AND
CALENDAR_DATE BETWEEN AND
'01/01/2012' AND '04/28/2012' AND
STORE_NUMBER='03' AND
CATEGORY=72
GROUP BY ITEM_DESC
Access plan generationQuery transformation
Dozens of query transformations
Hundreds or thousands of access plan options
Store
Product
Product Store
NLJOIN
Daily SalesNLJOIN
Period
NLJOIN
Product
NLJOIN
Daily Sales
NLJOIN
Period
NLJOIN
Store
HSJOIN
Daily Sales
HSJOIN
Period
HSJOIN
Product
StoreZZJOIN
Daily Sales
HSJOIN
Period
30
Comparing Big SQL and Hive 0.12 for Ad-Hoc Queries
*Based on IBM internal tests comparing IBM Infosphere Biginsights 3.0 Big SQL with Hive 0.12 executing the "1TB Classic BI Workload" in a controlled laboratory environment. The 1TB Classic BI Workload is a workload derived from the TPC-H Benchmark Standard, running at 1TB scale factor. It is materially equivalent with the exception that no update functions are performed. TPC Benchmark and TPC-H are trademarks of the Transaction Processing Performance Council (TPC). Configuration: Cluster of 9 System x3650HD servers, each with 64GB RAM and 9x2TB HDDs running Redhat Linux 6.3. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Results as of April 22, 2014
Big SQL is up to 41x faster than
Hive 0.12
Big SQL is up to 41x faster than
Hive 0.12
31
Comparing Big SQL and Hive 0.12 for Decision Support Queries
* Based on IBM internal tests comparing IBM Infosphere Biginsights 3.0 Big SQL with Hive 0.12 executing the "1TB Modern BI Workload" in a controlled laboratory environment. The 1TB Modern BI Workload is a workload derived from the TPC-DS Benchmark Standard, running at 1TB scale factor. It is materially equivalent with the exception that no updates are performed, and only 43 out of 99 queries are executed. The test measured sequential query execution of all 43 queries for which Hive syntax was publically available. TPC Benchmark and TPC-DS are trademarks of the Transaction Processing Performance Council (TPC).
Configuration: Cluster of 9 System x3650HD servers, each with 64GB RAM and 9x2TB HDDs running Redhat Linux 6.3. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Results as of April 22, 2014
Big SQL 10x faster than Hive 0.12
(total workload elapsed time)
Big SQL 10x faster than Hive 0.12
(total workload elapsed time)
How many times faster is Big SQL than Hive 0.12?
* Based on IBM internal tests comparing IBM Infosphere Biginsights 3.0 Big SQL with Hive 0.12 executing the "1TB Modern BI Workload" in a controlled laboratory environment. The 1TB Modern BI Workload is a workload derived from the TPC-DS Benchmark Standard, running at 1TB scale factor. It is materially equivalent with the exception that no updats are performed, and only 43 out of 99 queries are executed. The test measured sequential query execution of all 43 queries for which Hive syntax was publically available. TPC Benchmark and TPC-DS are trademarks of the Transaction Processing Performance Council (TPC).
Configuration: Cluster of 9 System x3650HD servers, each with 64GB RAM and 9x2TB HDDs running Redhat Linux 6.3. Results may not be typical and will vary based on actual workload, configuration, applications, queries and other variables in a production environment. Results as of April 22, 2014
32
Queries sorted by speed up ratio (worst to best)
Max Speedup 74xMax Speedup 74x
Avg Speedup 20xAvg Speedup 20x