SAP HANA Data Tiering Options · • Supported for both HANA on-premise and HANA-as-a-Service (HaaS) • Available for any HANA application • Complements, without replacing, other

PUBLIC

SAP HANA Product Management, SAP

April 2019

SAP HANAData Tiering Options

2PUBLIC© 2019 SAP SE or an SAP affiliate company. All rights reserved. ǀ

The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP.

Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service

or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related

document, or to develop or release any functionality mentioned therein.

This presentation, or any related document and SAP's strategy and possible future developments, products and or platforms directions and

functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this

presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality. This presentation is provided

without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a

particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP

assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross

negligence.

All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from

expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates,

and they should not be relied upon in making purchasing decisions.

Disclaimer


Problem:

▪ manage a continuous data growth in the HANA in-memory

database

▪ dependency between database growth and storage costs

▪ data growth impacts system performance

Solution:

▪ decouple data location from fixed storage layer

▪ storage layers differ in the costs and performance

▪ scale and store data with the best cost/performance ratio

The Data Growth Challenge

Trend: Organizations collect increasingly more and more information about their business to control their daily operations in real-time

Scale-up

Switch to scale-out

Add in-memory nodes

Data

Vo

lum

e

Hard

wa

re C

ost

Time


Technology Overview

Data Tiering Options in SAP HANA

▪ Hot Store

▪ Warm Store

▪ Cold Store

Application View on SAP HANA Data Tiering

Data Lifecycle Management

▪ SAP HANA DWF/DLM 2.0 SP04 + SP05

Agenda


• Mission-critical data

• For real-time processing and real-time analytics.

• Data is retained in-memory of the SAP HANA database.

• Data with reduced performance SLAs, which is less frequently accessed.

• Stored on a lower cost storage tier, managed as a unified part of the SAP

HANA database.

• Voluminous data for sporadic or very limited access.

• Stored on low cost storage tiers, like disk or Hadoop, managed separately

from the SAP HANA database, but still accessible at any time.

Solution: Decouple data location from a fixed storage layer.

Hot Store

Warm Store

Cold Store

SAP HANA Data Tiering

Hot

Data

Warm

Data

Cold

Data


SAP HANA Data TieringTechnology Overview

Hot Store

Persistent Memory extends

the in-memory storage

capacity for hot data in SAP

HANA.

Warm Store

Native Storage Extension (NSE) is an intelligent,

built-in disk extension for the SAP HANA in-

memory database. It is the primary warm store

option for HANA on-premise and HANA Service.

Extension node and dynamic tiering will continue

to be offered.

Cold Store

SAP HANA cold data tiering provides persistence capabilities

for HANA cold data in external data stores, like HDFS, Azure

Data Lake and SAP Big Data Services.

DRAM

Native Storage Extension

Extension Node

Dynamic Tiering

Spark Controller

Hadoop

SAP IQ

Persistent Memory

Nearline

Storage

(SAP BW)

Hot StorePersistent Memory


SAP HANA Native Support for Persistent MemoryOfficially Supported in SAP HANA 2.3 (April 2018)

Intel® Optane™ DC persistent memory available since 2nd April 2019

Benefit

Process more data

in real-time at a lower TCO with

improved business continuity

Persistent Memory

non-volatile

Data Reliability

faster starts

Higher Capacity

than DRAMTransforming

the memory hierarchy

Larger memory capacity with high

performance (vs. DRAM & lower tier

storage)

Lower TCO data storage hierarchy

Faster start time delivers less downtime

Co-innovation with Intel® leads to first

fully optimized major DBMS platform

Early Adoption Program with key

partners/customers ongoing

First major DBMS vendor to officially support Intel Optane DC persistent memory!

sap.com/persistent-memory

12.5xImprovement in startup time*

*Internal Benchmark measured with a 6TB dataset in SAP HANA

> 4 TBIncreased total memory capacity per CPU


App Direct Mode

• Non-volatile – Data is retained across a restart.

• DAX filesystem maps persistent memory to

memory volumes.

• Applications must be adapted & optimized.

Memory Mode

• Volatile – Data will not survive a restart.

• Pool of volatile memory, combining DRAM and

Persistent Memory.

• Transparent to applications – no changes

necessary.

Operating ModesIntel® Optane™ DC Persistent Memory

Volatile Memory Pool

Other Vendor’s Solutions

DRAMas cache

DCPMM

Direct Access (DAX)Volatile MP

SAP HANA

DRAM DCPMM

Memory Mode App Direct Mode


SAP HANA Persistent MemoryMemory Distribution

SAP HANA controls what is placed in Persistent Memory and what remains in DRAM.

Column Store Main moves to Persistent Memory

• More than 95% of data in most HANA systems.

• Loading tables at startup becomes obsolete.

• Lower TCO, larger capacity.

No changes to the persistence.

Volatile data structures remain in DRAM.

StorageDATALOG

DRAM

Volatile data

Working Memory

Persistent

MemoryNon-volatile data


Memory ConfigurationExample: 2 sockets, largest DRAM and largest PMEM configurations

1,536 GB

6,144 GB

3,072 GB

7,680 GB 3,072 GB

Storage

DRAM

Persistent

Memory

128 128

128 128

128

512512

128

512 512

512 512

128 128

128 128

128

512512

128

512 512

512 512

Actual configuration and ratios between DRAM,

PMEM and CPU depend on application sizing.

Intel® Xeon™

Cascade Lake

Storage

DRAM

128 128

128 128

128

128128

128

128 128

128 128

128 128

128 128

128

128128

128

128 128

128 128

Intel® Xeon™

Cascade Lake


Data Tiering

Hot and warm data are stored on

persistent memory.

• Keep more data in HOT.

• Dedicated warm data store becomes

obsolete.

Extension Node

Extension nodes allow to overload a node

by factor 4.

• No performance impact for warm data

stored on extension node.

• Less I/O on persistence.

Lower TCO, higher capacity

PMEM replaces DRAM for data storage.

DRAM, PMEM and CPU requirements

must be carefully evaluated!

Use Cases for Persistent MemoryCapacity & Data Volume Management

DRAM

PMEM

warm data

store

warm

data

Master

Persistence

Extension

(DRAM only)

frequent

LOAD /

UNLOAD

Extension

(w/ PMEM)Worker

Warm StoreNative Storage Extension


Native Storage Extension (NSE) value proposition and use cases

• Value proposition:

• Increase HANA data capacity at low TCO

• Deeply integrated warm data tier, with full HANA functionality

• Will support all HANA data types and data models

• Simple system landscape

• Scalable with good performance

• Supported for both HANA on-premise and HANA-as-a-Service (HaaS)

• Available for any HANA application

• Complements, without replacing, other warm data tiering solutions (extension nodes, dynamic tiering)

• Use cases:

• Any customer built or SAP built HANA application that is challenged by growing data volumes

• S/4HANA data aging (NSE is an evolution of “paged attributes”)

• BW team currently uses extension nodes, but may evaluate NSE in the future


Hot “column loadable” data:

• HANA is an in-memory database and loads all data into memory

for fast processing.

• Data is “column loadable” and resides completely in memory.

NSE allows the user to specify that certain data is “page loadable”

Warm “page loadable” data:

• Less frequently accessed data may be specified as “page loadable”.

• “Page loadable” data is loaded into memory in granular units of pages as

required for query processing.

• NSE will reduce memory footprint for “page loadable” data. Data is partly in

memory, and partly on disk.

• Query performance on warm data may be reduced compared to hot data.

• Data may be converted between “column loadable” and “page loadable”.

HANA Memory

Persistence

Working Memory

Hot Warm

Buffer

Cache


Example DDL

Table:

CREATE COLUMN TABLE T (C1 INT, C2 VARCHAR(10))

PAGE LOADABLE;

Partition:

CREATE COLUMN TABLE T (C1 INT)

PARTITON BY RANGE (C1)

(PARTITION 0 <= VALUES < 10 PAGE LOADABLE,

PARTITION OTHERS COLUMN LOADABLE);

Column:

CREATE COLUMN TABLE T (C1 INT, C2 VARCHAR(10)

PAGE LOADABLE);

Convert table to page loadable:

ALTER TABLE T PAGE LOADABLE IMMEDIATE CASCADE;

• Data may be specified as “page loadable” at table, partition, and

column level.

• Data may be converted between “page loadable” and “column

loadable”.

• NSE supports range, range-range partitioning.

Native Storage ExtensionSpecifying data as “page loadable”

Partitioned Table

NSE P_3

S_F

P_4

S_G S_H

P_5

Memory P_2

S_D S_E

P_1

S_A S_B S_C


NSE manages “page loadable” warm data in the HANA database with expanded disk capacity and an

intelligent buffer cache to transfer pages of data between memory and disk.

Native Storage Extension Native warm data tier of the SAP HANA database

SAP HANA

Persistence

Working Memory

Hot Data

SAP HANA with NSE

Persistence

Warm Data

Working Memory

Buffer

Cache Hot Data

Capacity

Hot data

in memoryHot data

in memory

+

Warm data

on disk


Native Storage ExtensionSizing and Limitations (on-premise)

Limitations with SPS04:

• Only scale-up landscapes are supported.

• Maximum data volume:

• 4 x data volume in memory, or

• 10 TB

Persistence layer

Working memory (1TB)

Hot data (1TB)

Example:

Server: 2.0 TB

Data volume: 1.0 TB

Expand database capacity with warm data

that is 4x the size of hot data

Option 2: Allocate from

existing memory

Database size: 4.0 TB

Pe

rsis

ten

ce

Warm data (3.2 TB)

Ma

in M

em

ory Working memory (0.8

TB)

Hot data (0.8 TB)

Buffer cache (0.4 TB)

Option 1: Add memory

Database size: 5.0 TB

Pe

rsis

ten

ce

Warm data (4 TB)

Ma

in M

em

ory Working memory (1 TB)

Hot data (1 TB)

Buffer cache (0.5 TB)

𝑠𝑖𝑧𝑒(𝑏𝑢𝑓𝑓𝑒𝑟 𝑐𝑎𝑐ℎ𝑒) =𝑠𝑖𝑧𝑒 (𝑤𝑎𝑟𝑚 𝑑𝑎𝑡𝑎)

8


Native Storage ExtensionTooling

SAP HANA Cockpit

• Configure buffer cache size (on-premise only; HaaS will

configure this for the user)

• Configure tables, columns, and partitions as “page loadable”

• Monitor buffer cache usage and capacity

• Report on resident memory status for page loadable data

• Includes rule-based “recommendation engine” to monitor user

data access patterns.

• Based on statistics, the engine will advise user on which tables,

columns, or partitions would benefit from being converted to

“page loadable”

Data Lifecycle Manager (planned Q4/2019):

• DLM will allow user to convert tables, columns, and table

partitions between “column loadable” and “page loadable”

SQL Analyzer (DB Explorer)

• Visualized query plan will display when warm data is accessed

from NSE in order to satisfy the query

Warm StoreExtension Node, Dynamic Tiering


Feature description

• HANA node in the scale-out landscape is reserved for warm-data

storage and processing

• Supports all HANA operations and data management features

• Allows larger data footprint of up to 200% of the node DRAM size

• HANA persistent memory is supported

New Features:

• Benefits from new partitioning and scale-out features in SPS04:

• range-hash partitioning scheme

• “pinning” tables on fixed HANA nodes

• partition grouping

What’s New in SPS 04?SAP HANA Extension Node

HANA Database

Extension

Node

Worker

Node

Scale-Out

Paged

Warm Data

Hot DataWarm

data


What’s New in SPS 04?SAP HANA Dynamic Tiering

Feature description

• Addition of TIMESTAMP data type for multistore tables

• Asynchronous table replicas of slowly changing dimension

tables maintained consistently across the HANA and

dynamic tiering servers

Benefits

• 7-digit TIMESTAMP support for multistore tables rounds out basic data type support in SAP HANA dynamic tiering

• Asynchronous table replicas improves cross-store query performance by allowing SQL JOIN operations to be executed close to the data

SAP HANA with Dynamic Tiering

SAP HANA

node

DT server

Disk Store

T1 T1_REPATR

T2

local join


Customers should start with the built-in Native Storage Extension for warm data. Depending on SLAs, customers may choose extension

node (performance) or dynamic tiering (data volume) as alternative options. We don’t recommend to mix multiple warm store options in

one landscape, due to complexity reasons.

Warm Store Options – Getting Started

Costs Performance

Data Volume

Extension Node


Dynamic Tiering

Cold StoreSpark Controller / Hadoop


What’s New in SPS 04?HANA Spark Controller

New Features in SPS04:

▪ For DLM scenarios, you can now install SAP HANA spark controller on Amazon Elastic MapReduce (EMR)

▪ Support of newer Apache Spark versions: 2.3.x and 2.4.0

▪ Support of newer Hadoop Distributions: CDH 6.1.0, MapR 6.1, HDP 2.6 and HDP 3.0

SAP HANA 2 SPS03

SDAIn-memory

Spark

Adapter

DLM

HANA Clients

HANA Spark Adapter

protocol

Spark Cluster

HDP, MapR, CDH, SCP BDS, Azure HDInsight

HANA Spark Controller SPS04

Hot Data

Cold Data

(HDFS, ADLS)

DLM profiles

DLM Views

Application ViewData Tiering


Which Data Tier Should I Use?

Native HANASuite on HANA

S/4HANABW on HANA

BW/4HANA

Extension Node


Near-Line Storage, SAP IQ

DWF/DLM with

Spark Controller

Extension

Node

HANA

Database

HDFS, ADLS

BW NLS,

BW/4 DTO w/ IQ

Data Aging

(via NSE)

Extended Store Dynamic

Tiering

ILM Store

w/ IQ

BW NLS,

BW/4 DTO

External

Store

In-Memory PMEM PMEM

Extension

Node

PMEM

ILM/

Archiving

NSE

Data Lifecycle ManagementSAP HANA DWF/DLM 2.0 SP04 + SP05


What’s New?SAP HANA DWF/DLM 2.0 SP04 + SP05

▪ SAP data warehousing foundation is a separate software

release, but with the same shipment date as HANA 2.0

SPS04

▪ SAP DWF includes the Data Lifecycle Management tool

(DLM)

▪ SAP DWF 2.0 SPS05 release is based on the XSA software

stack and supports:

▪ Dynamic tiering multistore tables

▪ Generating groups of tables that are common in their data sets

(same columns) and move them together to another storage

location

▪ Defining “nominal key” to enable the relocation of data for tables

without a primary key, which is mandatory to relocate data

Contact information

Robert Waywell: [email protected]

Andreas Schuster: [email protected]

Daniel Felsmann (DLM): [email protected]

SAP HANA Product Management

[email protected]

Thank you

mailto:[email protected]




© 2019 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of

SAP SE or an SAP affiliate company.

The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its

distributors contain proprietary software components of other software vendors. National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or

warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials.

The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty

statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional

warranty.

In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or

any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation,

and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platforms, directions, and

functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason

without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or

functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ

materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they

should not be relied upon in making purchasing decisions.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered

trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names

mentioned are the trademarks of their respective companies.

See www.sap.com/copyright for additional trademark information and notices.

www.sap.com/contactsap

Follow us

https://www.sap.com/copyright

https://www.sap.com/registration/contact.html

https://www.linkedin.com/company/sap

https://www.youtube.com/user/SAP

https://twitter.com/sap

https://www.facebook.com/SAP

Documents

SAP HANA Data Tiering Options · • Supported for both HANA on-premise and HANA-as-a-Service (HaaS) • Available for any HANA application • Complements, without replacing, other