Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
© 2013 SAP AG. All rights reserved. 2Public
Agenda
Welcome
Agenda• Introduction to Dobler Consulting• SAP IQ Roadmap – What to Expect• Q&A
Presenters• Courtney Claussen - SAP IQ Product Management • Peter Dobler - CEO Dobler Consulting
Closing
© 2013 SAP AG. All rights reserved. 3Public
Introduction to Dobler Consulting
Dobler Consulting is a leading information technology and database services company, offering a broad spectrum of services to their clients; acting as your Trusted Adviser, Provide License Sales, Architectural Review and Design Consulting, Optimization Services, High Availability review and enablement, Training and Cross Training, and lastly ongoing support and preventative maintenance. Founded in 2000, the Tampa consulting firm specializes in SAP/Sybase, Microsoft SQL Server, and Oracle.
Visit us online at www.doblerconsulting.com, or contact us at 813 322 3240, or [email protected].
© 2013 SAP AG. All rights reserved. 4Public
Your Data is Your DNA, Dobler Consulting Focus Areas
Strategic Database Consulting
DBA Managed Services
Database Training Programs
SAP VAR D&T
Cross-Platform Expertise SAP Sybase® SQL Server® Oracle®
© 2013 SAP AG. All rights reserved. 5Public
What’s Ahead
ISUG-TECH Annual Conference April 14-17, Atlanta
• Register at http://my.isug.com/conference/registration
• Early Bird ending 2/28/14 (free hotel room with registration)
SAPPHIRENOW Annual Conference June 3-5, Orlando
• Come visit our kiosk in the exhibition hall
Courtney Claussen / SAP IQ Product Management
February 27, 2014
SAP IQ Roadmap Dobler Events Webinar
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 2
Disclaimer
This presentation outlines our general product direction and should not be relied on in making a
purchase decision. This presentation is not subject to your license agreement or any other agreement
with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to
develop or release any functionality mentioned in this presentation. This presentation and SAP's
strategy and possible future developments are subject to change and may be changed by SAP at any
time for any reason without notice. This document is provided without a warranty of any kind, either
express or implied, including but not limited to, the implied warranties of merchantability, fitness for a
particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this
document, except if such damages were caused by SAP intentionally or grossly negligent.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 3
Table of contents
Product Overview
SAP IQ product description
Strategic asset in SAP’s database portfolio
SAP IQ architecture
Product Road Map
Shared nothing
Extreme data volume
Enhanced administration
Summary
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 5
SAP IQ product description High performance analytics with low TCO
10x – 1000x faster queries over
traditional DBMS
Supports heavy ad-hoc loads with
1000s of users concurrently
Store & analyze terabytes to
petabytes of structured &
unstructured data
Performance & scalability
Best-of-breed partners certified
with SAP IQ
Tighter integration with SAP
HANA, BusinessObjects BI
Supports 3rd party BI tools and
analytic applications
Integrated ecosystem
#1 disk-backed column DB for
reporting and analytics
Platform of algorithms, services &
APIs for big data analysis
Highly flexible deployment options;
supports all major hardware & OS
platforms
Trusted open analytics
server
See Appendix for abbreviations
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 6
SAP IQ in SAP’s database portfolio Strategic asset
SAP HANA Data Platform
SAP HANA
SAP
SQL Anywhere
Mobile
and embedded
database
Transactional
database with low
TCO
SAP ASE SAP IQ
Petabyte scale
analytics database
with low TCO
SAP ESP,
Replication Server,
PowerDesigner & EIM
Unified
EIM platform for
real-time
Open for
partners
See Appendix for abbreviations
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 7
Ecosystem
SAP IQ architecture A comprehensive analytics platform
Most mature
column store
Comprehensive
lifecycle tiering
MPP queries, virtual marts,
and user scaling
High-speed
loads
Structured and
unstructured store
Comprehensive ANSI
SQL with OLAP
Built-in full-text
search
In-database analytics with
MapReduce and simulator Web 2.0 APIs
Big Data open
source APIs
Optimized BI and EIM to
model and replicate
Development and
administration tools Predictive analytics
Packaged ILM
applications
Bradmark,
Symantec,
Whitesands, Quest,
and ZEND
SAS, SPSS, KXEN,
Fuzzy Logix,
Zementis
BMMSoft,
SOLIX, and PBS
SAP PowerDesigner,
SAP Replication Server,
SAP Data Services,
Panopticon
Application
services
DBMS Hadoop and
R
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 9
2009
Very large database (VLDB) platform foundation
2011
MapReduce API
2011
PlexQ MPP Foundation
2010
Full text search; Web 2.0 API
2009
In-database analytics
2013
• Extended
storage for
SAP HANA
• Shared
nothing
architecture
SAP Sybase IQ 16 SP08 and SP09 Path to Extreme Scale
2014
New generation
column store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 10
SAP IQ 16.0 SP08 and SP09 Planned improvements through 2014
Shared Nothing
• DAS (Direct Attached Storage) Cache
• DAS Files
• DAS Files elasticity
Enhanced Administration
• Point in time recovery
• Integrated system management
Extreme Data Volume
• Extended storage for HANA
• Distributed load
• RLV (row level version) store on Multiplex
SAP IQ
Enhanced XLDB
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 11
SAP IQ 16 Key building blocks
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 12
SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
• Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 13
SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
• Distributed load
• Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 14
SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
• Distributed load
• HANA extended tables • PITR – Point in Time Recovery
• Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 15
SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
• DAS Cache extends data affinity
• DAS Files “shared nothing” architecture with elasticity
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
• Distributed load
• HANA extended tables • PITR – Point in Time Recovery
• Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 16
SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
• DAS Cache extends data affinity
• DAS Files “shared nothing” architecture with elasticity
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
• Distributed load
• HANA extended tables • PITR – Point in Time Recovery
• Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 17
SAP IQ 16 SP08 and SP09 Key building blocks – new features highlighted in green boxes
• DAS Cache extends data affinity
• DAS Files “shared nothing” architecture with elasticity
SAP IQ 16: engine
Re
silie
nt M
ultip
lex g
rid a
rch
itectu
re
We
b b
ase
d a
dm
inis
tratio
n a
nd
mo
nito
ring
Shared store
Loading engine
N-bit and tiered indexing
Text search
Web-enabled analytics
Info
rma
tion
lifecycle
ma
na
ge
me
nt
Communications and security
Schema
Query engine
Column store
Hash partitioned tables
Distributed queries with data affinity
Database, column, and wire encryption
LDAP, Kerberos authentication, and RBAC
Re-engineered petascale disk store
In-memory, write-optimized delta store
Fully parallel
bulk loading
• Distributed load
• HANA extended tables • PITR – Point in Time Recovery
• RLV on Multiplex
• Integrated system management
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 19
High Speed Interconnect to Shared Store
Scale out Scale out
Elastic Logical Server
IQ Server
Data affinity
IQ Server
Data affinity
IQ Server
Data affinity
IQ Server
Data affinity
IQ Server
Data affinity
Elastic Logical Server
Scale out Scale out
IQ 16.0 – “Shared Everything” Multiplex with Data Affinity
SAP IQ Multiplex
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 20
IQ Server
Data affinity
IQ Server
Data affinity
IQ Server
Data affinity
Shared Store
DAS cache extends data affinity
• DAS (direct attached storage) Cache is a
mechanism for maintaining one or more
copies of primary data in direct attached
storage on different physical servers.
• The DAS cache is populated in a pull model
as new versions of data are accessed
• The DAS cache can offload bandwidth from
the SAN for subsequent data access
DAS cache DAS cache DAS cache
SAP IQ Multiplex
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 21
Multiplex with high speed interconnect
IQ Server
DAS Files – IQ adopts a “shared nothing” architecture
• DAS (direct attached storage) files are primary
direct attached storage devices that are being
shared across the cluster using the high-
speed interconnect.
• Primary data may exist in only one DAS file.
• These can be used in place of the SAN for
primary storage
• When data is loaded, it will be automatically
written to the appropriate DAS file that the
data has affinity to.
• With DAS files, DAS cache will still have value
for dealing with data access that cannot be
affinitized.
DAS Files DAS Files DAS Files
DAS
cache DAS
cache
DAS
cache
IQ Server IQ Server
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 22
DAS Files – mirrored for write safety
• DAS files are managed within a logical server
• Primary files are mirrored on a different server for data redundancy
• Optionally, user may create a DAS file without a redundant copy
Logical Server (Server_1, Server_2, Server_3, Server_4)
Server_1 Server_2 Server_3 Server_4
L1 M4 L2 M2 L3 M3 L4 M1
Primary Mirror
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 23
High speed interconnect
DAS Files elasticity
The IQ server redistributes data when nodes or files are added or removed
IQ
Server_3
NEW
IQ
Server_4
IQ
Server_2
IQ
Server_1
Empty
High speed interconnect
NEW
IQ
Server_4
IQ
Server_3
IQ
Server_1
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 24
High speed interconnect
DAS Files – does not preclude a shared store
• SAP Sybase IQ continues its flexible
architecture.
• An IQ database may be comprised of both
DAS DBSpaces and shared DBSpaces.
• Shared store could be used for data that is
needed within each node, without duplication.
Optional shared store
DAS Files DAS Files DAS Files
DBFile DBFile DBFile
DAS
cache
DAS
cache
DAS
cache
IQ Server IQ Server IQ Server
Extreme Data • Extended storage for SAP HANA
• Distributed load
• RLV (row level version) store on Multiplex
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 26
SAP IQ is the preferred solution for warm and cold data archiving
capabilities for SAP BW, Business Suite, and HANA
NLS (Near Line Store) for SAP BW
• Optimized NLS data transfer throughput using IQ Loader functionality
• SAP HANA and IQ share the same columnar paradigm, and similar data compression rates
• Ready for large data volumes
• Suitable for ad-hoc queries with long history
• Minimum administration effort
ILM for SAP Business Suite
• Fast archive index read w/o additional secondary DB indexes
• Increased search capabilities
• Faster archive I/O – fewer layers (software, network, storage hardware)
• ERP archive files as well as archive indexes stored in SAP IQ
Warm archive for HANA using smart data access (SDA)
• Using SDA, HANA customers can access IQ as a federated store
• Store warm data in IQ, and real time data in HANA
• Push query processing down to IQ
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 27
HANA extended storage brings integrated warm data management to
SAP HANA
NLS Store for SAP BW
ILM for SAP Business Suite
Warm archive for HANA, federated via SDA
Extended storage for SAP HANA
• Handle all data seamlessly via HANA tables with Calc View support
• Transformative ROI – enable new insights from data processing at speed and scale not possible before
• Steady performance across data volumes, variety, velocity with scalable/elastic capacity
• Practical for both on-premise and hosted models
• Native Big Data solution allows customers to gain real-time insights by cost-effectively managing and analyzing ALL enterprise data
Cold/warm data archival
Warm Data Management for SAP HANA
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 28
HANA extended storage Requirements from our customers
• Manage data cost effectively, yet with desired performance based on SLAs
• Application defines which data is “hot”, and which data is “warm”
• Handle very large data sets – terabytes to petabytes
• Update and query all data seamlessly via HANA extended tables – IQ store is transparent to
applications
• Provide a native Big Data solution that covers most enterprise use cases without Hadoop
Table
Table
Queries and
updates initiated
from HANA
SAP HANA
SAP IQ
Extended table (access via HANA)
29 © 2014 SAP AG or an SAP affiliate company. All rights reserved.
BW Process and Data Management
SAP HANA
Master Data, InfoCubes, DataStore Objects (DSO)
SAP IQ Near Line
Store (NLS) SAP Extractors, Data Services, SLT
Replication
SAP and non-SAP data
sources
NL
S
inte
rfac
e
Persistent
Staging Area
(PSA)
Corporate
Memory
BW on HANA with extended storage in IQ
SAP IQ
Reads
Bulk load
Smart Data Access (SDA) SDA SDA
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 30
DISTRIBUTED LOADS SCALED OUT DATA LOADING
Value proposition
Optimize performance of massive bulk loads
Exploit large memory and core footprints
Architectural considerations
SAP IQ base table data ingestion is distributed.
Bulk loads scale out for the best possible load
performance.
Load speed should scale almost linearly when more
cores / nodes are added.
User may control whether a load gets distributed or not.
Graphical plans for loads will be enhanced for
distributed loads.
Multiplex
Shared Store
Client
IQ Server IQ Server IQ Server IQ Server
Logical server
Bulk load
to IQ table
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 31
ROW LEVEL VERSION DELTA STORE FOR MULTIPLEX HIGH VELOCITY DATA LOADING ON IQ CLUSTER
Multiplex
IQ Writer
Server
IQ Store
Memory RLV
store
Clients
High velocity,
concurrent writes
IQ Writer
Server
Memory RLV
store
IQ
Reader
Server
periodic
merge
periodic
merge
Clients
High velocity,
concurrent writes
Value proposition
Continuous analytics over operational data
High velocity, concurrent data modifications
across all writer nodes in a Multiplex
Exploit large memory and core footprints in a
clustered deployment
Architectural considerations
Each writer node in a Multiplex may manage its
own RLV store
All RLV stores are fully recoverable with
dedicated transaction logs on each writer node
Query engine sees a consistent view of data
across RLV stores and main IQ store
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 33
POINT IN TIME RECOVERY
Point in time recovery:
• Database backup performed at 6:00pm on Monday
• Disk failure occurs on Thursday at 3:00pm
• User wants to recover database state on Thursday at noon: • Restore Monday backup
• Replay transaction log from time of Monday backup until Thursday at noon – apply committed transactions, and roll back uncommitted transactions
Backup at
midnight on
Monday
Monday Tuesday Wednesday Thursday
3:00pm Sunday
Server crashed
and database is
corrupt
Thursday
Database transaction log U = Update B = Backup
D = Delete C = Commit
I = Insert K = Checkpoint
Op I D U C I D K B I U K D C I K I C I D C K U I C Crash
2a 6a 11a 4p 3a 2p 5p 6p 11p 1a 8a 3p 9p 3a 10a 7p 11p 4a 10a 1p 8p 9a 11a 1p 3p Time
Recover to noon
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 34
INTEGRATED SYSTEMS MANAGEMENT
SAP Control Center (SCC) supports
up to 100 servers of:
SAP ASE
SAP IQ
SAP Replication Server (SRS)
SAP Event Stream Processor (ESP)
SAP Mobile Platform (SMP)
* Emerging in future:
SAP HANA
SAP Real-Time Data Platform (RTDP)
SAP SQL Anywhere (SA)
Support up to 250 servers
SCC >Admin >Monitor
>Alert
HANA*
RTDP*
SA*
SMP
ESP
SRS
IQ
ASE
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 35
SAP CONTROL CENTER
PLANNED ARCHITECTURAL IMPROVEMENTS
• Creation of a new datacenter management
feature which will easily allow an
administration group to identify the overall
health of the servers within the datacenter
and quickly drill into those servers.
• Provide integration of the datacenter features
with existing landscape components
registered to the LMDB
• Modify the existing Flex based management
console to UI5 to enable a better user
experience for ‘on board’ features.
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 36
Landscape view
Show nodes or server statuses, for example SCC Heat Chart:
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 37
RS HANA ASE
RS IQ ASE
ESP HANA
SA ASE
Topology view
Show connectivity or relationship, for example SCC Topology View (future and current):
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 39
Summary
SAP IQ is a strategic technology asset
for SAP’s database portfolio, and
continues its innovative roadmap
with:
Shared Nothing MPP – Optional shared
nothing MPP configuration supports cloud
deployments with more elastic
architecture.
Extreme Data Volume – New extended
store feature for HANA, distributed loads,
and RLV store on Multiplex address the
extreme data volume and velocity
requirements of big data.
Enhanced Administration – Improved
management features help today’s
database administrators perform their jobs
more efficiently and confidently.
This is the current state of planning and may be changed by SAP at any time.
See Appendix for abbreviations
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 40
SAP IQ Resources Where to get information on SAP IQ
• SAP IQ SCN Developer Site:
http://scn.sap.com/community/developer-center/analytic-server
• Free Express and Enterprise Edition Downloads
https://www.sap.com/iq-downloads
• “How to” Tutorials
http://scn.sap.com/docs/DOC-43300
• SAP IQ 16 (video):
https://www.youtube.com/watch?v=Bbcl8uY3tj4
• SAP IQ for Big Data(video):
https://www.youtube.com/watch?v=XRNgVPxiyVw
• SAP IQ as Near-line Storage (video):
https://www.youtube.com/watch?v=FdtdAhTQYSg
• SAP IQ for BusinessObjects (video):
http://www.youtube.com/watch?v=A4hR0n2y8zQ
© 2014 SAP AG or an SAP affiliate company. All rights reserved.
Thank you
Courtney Claussen, Director of Product Management, SAP IQ
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 42
Acronym Full Text
API Application Programming Interface
ANSI American National Standards Institute
BI Business Intelligence
DAS Direct Attached Storage
EDW Enterprise Data Warehouse
EIM Enterprise Information Management
XLDB Extreme Large Database
ILM Information Lifecycle Management
LDAP Lightweight Directory Access Protocol
PAM Linux Pluggable Authentication Module
MPP Massively Parallel Processing
NLS Near Line Storage
Acronym glossary (1 of 2)
© 2014 SAP AG or an SAP affiliate company. All rights reserved. 43
Acronym Full Text
OS Operating System
RTDP Real-Time Data Platform
RDBMS Relational Database Management System
RBAC Role Based Access Control
RLV Row Level Versioned
ASE SAP Adaptive Server Enterprise
ERP SAP Enterprise Resource Planning
ESP SAP Event Stream Processor
SAN Storage Area Network
SMP Symmetric multiprocessing (slide 21); SMP is also the acronym for ‘Service Marketplace’
TCO Total Cost of Ownership
Acronym glossary (2 of 2)