Planning and Fast-Start-Solutions for Implementation Guide ......This planning guide will focus on the available Fast-Start-Solutions for SAP HANA on Power. Hardware setup for Fast-Start-Solution

Fast-Start-Solutions for SAP HANA on Power Overview and Comparison Marc Stephan Tauchert

IBM Deutschland Research & Development GmbH Created on November 06, 2019 – Version 0.0 Last modified on March 17, 2020 – Version 1.0 © Copyright IBM Corporation, 2020

Planning and Implementation Guide

2

IBM Systems Solution for SAP HANA on POWER and IBM System Storage © Copyright IBM Corporation, 2020

Edition Notice and Version Information © Copyright IBM Corporation 2020. All Rights Reserved. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. All trademarks or registered trademarks mentioned herein are the property of their respective holders. IBM Corporation Systems and Technology Group Route 100 Somers, NY 10589 Edition Notice: this is version 1.0 of this document. Focus: SAP HANA Scale-up and Scale-out solutions Target:

• SAP HANA 2.0

• POWER9

Doc Version Changes

0.0 Initial Version 0.1 First Publication

1.0 Mapping for H922 and H924 models

Note: Before using this information and the product it supports, be sure to read the general information under ”

Copyrights and Trademarks” on page 11 as well as “Disclaimer and Special Notices” on page 12.

3


Preface Running SAP HANA on IBM Power Systems offers customers a consistent platform for their HANA-based and traditional applications, best-in-class performance, resilience for critical workloads, and most flexible infrastructure. Existing IT assets - servers, storage, as well as skills and operation procedures - can easily be (re-)used leveraging the SAP HANA Tailored Data Center Integration (TDI) concept, instead of enforcing additional investment into dedicated SAP HANA appliances. This paper will show how a HANA can be started 2 to 18 times faster exploiting different IBM technologies.

About This Document This document is intended for architects and specialists planning an SAP HANA® on POWER® deployment. It describes the design considerations for hardware, networking, and software components of the SAP HANA on POWER solution stack. This guide does not replace existing SAP HANA documentation and sizing guides. It serves as a supplement to the existing SAP HANA documentation and SAP Sizing methods to provide specific guidance on how to meet all SAP requirements when running SAP HANA on IBM Power Systems™, IBM System Storage®, IBM PowerVM®, and Linux Operating Systems. IBM employees can access the ISICC SAP HANA on Power Systems Community (IBM only) for up-to-date materials complementary to this guide. The most recent document version can be downloaded from IBM TechDocs: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102502 Feel free to provide feedback and change requests for this document via email to [email protected].

About the Author Marc-Stephan Tauchert is a System Specialist at IBM Germany. In his 25+ years’ experience in designing and implementing SAP Solutions on IBM Power Systems, he has covered various topics and roles. As Technical Sales Consultant and SAP Solution Architect, he supports customer situations regarding the connectivity of SAP Applications and Infrastructure, SAP sizing, and hybrid solutions. He is an expert in SAP Database and Application performance, including SAP HANA.

https://w3-connections.ibm.com/communities/service/html/communitystart?communityUuid=e4d11bf2-8632-4a00-ba19-d767a98a2cf3

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102502

mailto:[email protected]

4


Table of Contents

Introduction ............................................................................................................................................. 5

Hardware setup for Fast-Start-Solution used .......................................................................................... 5

Process and test overview ....................................................................................................................... 5

Architecture and setup instructions ........................................................................................................ 6

Fast-Start-Solution with TMPFS: .......................................................................................................... 6

Fast-Start-Solution with vPMEM ......................................................................................................... 7

Fast-Start-Solution with Rapid-Cold-Start ........................................................................................... 7

Fast-Start-Solution with Internal NVMe PCIe cards and NVMe SAN Storage ..................................... 8

Scenario comparison between the different Fast-Start-Solutions .......................................................... 9

Application memory comparison ............................................................................................................ 9

Performance differences between the Fast-Start solution variants ....................................................... 9

Referenced documents ..........................................................................................................................11

Copyrights and Trademarks ...................................................................................................................11

Disclaimer and Special Notices ..............................................................................................................12

Figures Figure 2: Scheme of tmpfs solution design ............................................................................................. 6

Figure 3: Scheme of vPMEM design ........................................................................................................ 7

Figure 3: Scheme of the Rapid-Cold-Start solution ................................................................................. 8

Tables Table 1: Different scenario's / trigger's for start and stop actions .......................................................... 9

Table 2: Memory values .......................................................................................................................... 9

Table 3: Startup times, stop times and their benefits ...........................................................................10

5


Introduction This planning guide supports to architecture and implement new technologies on Power for SAP HANA environments. This technical document is based on experimental tests done at SAP in Walldorf in cooperation with the ISICC. This planning guide will focus on the available Fast-Start-Solutions for SAP HANA on Power.

Hardware setup for Fast-Start-Solution used This section lists the used hardware but does not necessarily define all requirements for every client’s production SAP HANA instances. The test has been conducted on a Power 9 system 9040-MR9 which has the following hardware features:

- 48 cores

- 8192 GB memory

- 3 x PCIe2 16Gb 2-Port Fibre Channel Adapter

- 4 x PCIe3 3.2TB NVMe Flash Adapter II x8

- 1 x SAS RAID Controller, PCIe3 x8, Quad-port 6Gb

- 2 x PCIe2 4-Port (10GbE SFP+ & 1GbE RJ45) Adapter

Following Logical Partitions are defined: - 2 x Virtual I/O Server with 1 dedicated core and 4 GB main memory

- 1 x SAP HANA database server with 28 dedicated cores and 512 to 4196 GB main memory

- 1 x SAP Netweaver Application Server with 16 dedicated cores and 512 GB main memory

Software and Firmware levels:

- The firmware level of the Power 9 system should equal or greater VM940_FW940.00 (27)

(used in this comparison test)

- Firmware level of the PCIe3 3.2TB NVMe Flash Adapter II x8 adapters should be MN14MN14

- The Virtual I/O Servers based on the version 2.2.6.31.

- The SAP Netweaver Application Server based on SLES 12 SP4 and SAP Netweaver Kernel 7.49

- The SAP HANA database server has the OS version SLES15 SP1 and the SAP HANA version

2.00.044.00.1571081837

SAP HANA DatabaseContent

- SAP Netweaver 7.50

- Data volume of 7.8 billion records which maps to 3 TB main memory usage by SAP

HANA.

Process and test overview For all different Fast-Start-Solutions the same start and stop method are processed. On each Fast-Start-Solution all different start states of the system are tested and compared. All test

6


results are based on multiple start and stop actions. All result measurements in this document are normalized‚ average values. The start and stop measurement times are extracted from the SAP HANA indexserver trace file. The verification about a performance impact on the application side between the different Fast-Start-Solutions, were based on Online Analytical Processing (OLAP) workload without performance difference. The SAP HANA database size was 3 TB after loading all data into memory.

Architecture and setup instructions

This chapter provides information about the implementation and setup of the different Fast-Start-Solutions. This documentation will only provide an overview and will not replace the regular configuration and implementation guides of these Fast-Start-Solutions.

Fast-Start-Solution with TMPFS:

This solution based on temporary filesystems which are defined on the operating system level. These temporary filesystems are working like RAM based filesystems. For each NUMA node one temporary filesystem has to be created and mounted to a specific HANA directory. The Figure 1 shows a scheme which part of SAP HANA will be stored in the temporary filesystems.

Figure 1: Scheme of tmpfs solution design

Detailed setup instructions can be found in the “SAP HANA Administration Guide for SAP HANA Platform - SAP HANA Fast Restart Option” at: https://help.sap.com/viewer/6b94445c94ae495c83a19646e7c3fd56/2.0.04/en-US/ce158d28135147f099b761f8b1ee43fc.html

SAP HANA Working Area

Row Store

Column Store

Delta

Main

DRAM of LPAR

tmpfsFilesystems

inMemory

(per numa nodeone temporary

filesystem)

https://help.sap.com/viewer/6b94445c94ae495c83a19646e7c3fd56/2.0.04/en-US/ce158d28135147f099b761f8b1ee43fc.html

https://help.sap.com/viewer/6b94445c94ae495c83a19646e7c3fd56/2.0.04/en-US/ce158d28135147f099b761f8b1ee43fc.html

7


Fast-Start-Solution with vPMEM

The vPMEM solution is very similar to the tmpfs solution for SAP HANA Fast-Start option. The main difference between the vPMEM and the tmpfs solution is the placement of the memory. The tmpfs Fast-Start-Solution using the main memory of the LPAR, while the vPMEM Fast-Start-Solution is using the persistent memory which is defined for the LPAR. In total the summary of main memory and persistent memory have the same size like the tmpfs or the Rapid-Cold-Start solution. The Figure 2 shows the scheme of architecture for vPMEM with SAP HANA on Power.

Figure 2: Scheme of vPMEM design

More details about the configuration and sizing for virtual persistent memory can be found here: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102502

Fast-Start-Solution with Rapid-Cold-Start

This solution is not based on a technology using main memory compared to vPMEM and tempfs. Instead it is using NVMe devices to optimize the read performance of the attached I/O components. Figure 3 shows the design of the Rapid-Cold-Start-Solution.

SAP HANA Working Area

Row Store

Column Store

Delta

Main

DRAM of LPAR

vPMEMFilesystem

inMemory

(per numa node one vPMEM filesystem)

PMEM of LPAR

8


Figure 3: Scheme of the Rapid-Cold-Start solution

Detailed setup documentation can be found here: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102502

Fast-Start-Solution with Internal NVMe PCIe cards and NVMe SAN Storage

From previous measurements traditional internal and external Storage attachments are added in the comparison. The detailed setup documentation for SAN attachments is part of the SAP TDI process. The instructions on how to setup HANA based on internal discs can be found here: http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP102502

Software Raid 0/dev/md0

LVM Stripping/dev/data_vg/data_lv

NVMe PCI cards SVC Storage

Software Raid 1/dev/md1

XFS Filesystem for /hana/data



9


Scenario comparison between the different Fast-Start-Solutions During the lifetime of a SAP HANA system there could be a lot of different trigger for start and stop the SAP HANA and the underlying Operating System. The following table (Table 1) gives an overview about these different trigger’s and the impact for the different Fast-Start-Solutions.

Scenario tmpfs vPMEM Rapid-Cold-Start

Internal NVMe Cards

FS9100 NVMe

Storage

SAP HANA start (initial)

No No Yes Yes Yes

SAP HANA start after stop

Yes Yes Yes Yes Yes

SAP HANA restart Yes Yes Yes Yes Yes

SAP HANA service restart

Yes Yes Yes Yes Yes

Operating system reboot

No Yes Yes Yes Yes

Operating system update/service patch

No Yes Yes Yes Yes

System restart No No Yes Yes Yes Table 1: Different scenario's / trigger's for start and stop actions

Application memory comparison The application performance differences between the different Fast-Start-Solutions goes to zero. The following table (Error! Reference source not found.) shows the memory configuration and memory usage by SAP HANA on the tested scenarios.

Standard tmpfs vPMEM NVMe

Total memory 4091 GB 4091 GB 4006 GB1 4091 GB

No Persistent memory 4091 GB 4091 GB 511 GB 4091 GB

Persistent memory 0 GB 0 GB 3495 GB 0 GB

Memory by SAP HANA 2921 GB 2923 GB 2924 GB 2924 GB Table 2: Memory values

Performance differences between the Fast-Start solution variants The following table contains the measurement data of the startup and stop tests for the benefit validation of the different Fast-Start methods for SAP HANA. This table considered only the startup and stop behavior of SAP HANA. The start and stop times for LPAR operations or system operations are not included in this table.

1 The complete memory based on the addition of vPMEM and main memory configured in the HMC for the specific test state of the LPAR differ due to the need of 100% equal distribution needs and can be ignored.

10


The time values are based on HH:MM:SS. The startup and stop time information’s are extracted from the index server trace files of the SAP HANA environment. All tests are running up to minimum 3 times. The result values in the table below are average values. It is important to emphasize that the factors the startup is improved has no correlation to the runtime and heavily depends on the amount of data residing inside HANA. Hence, it is recommended to validate if the targeted Business application already supports SAP HANA Native Storage Extension (NSE) to first start reducing the amount of memory loaded and hence the overall memory consumption.

Standard tmpfs vPMEM Rapid-Cold

Start times after System Power Off

Benefit against Standard

00:35:45 00:29:00

×1,2

00:31:02

×1,2

00:05:19

×6,5

Start times after LPAR down


00:32:13 00:28:51

×1,1

00:02:27

×13

00:05:00

nearly ×6,5

Start times after SAP HANA stop

(Table load = 5 parallel)


00:31:44 00:02:22

×13

00:02:24

×13

00:04:59

nearly ×6,5

Start times after SAP HANA stop

(Table load = 12 parallel)


00:31:05 00:01:41

×18

00:01:42

×18

00:04:20

×7

Stop times


00:01:38 00:01:00

×1,6

00:01:01

×1,6

00:01:32

×1,1

Table 3: Startup times, stop times and their benefits

Based on older measurements moving to a FS9100 model a startup time improvement of factor 2-3x can be expected at minimum. For the PCIe NVMe local disks 5x startup improvements are easily to be achieved.

Mapping to H922 and H924 models In 2019 IBM release new H922 and H924 bundles documented on IBM Redbooks. Depending on the size they come with 3 or 5 NVMe cards preconfigured providing protection from a single card failure (if higher protection is required adding one card will provide protection from two card failures.

https://www.redbooks.ibm.com/abstracts/redp5498.html?Open

11


Note: It is mandatory to setup monitoring and alerting in case a card fails to instantly take corrective actions.

Figure 4 NVMe module distribution for 3x3.2 NVMe cards

Figure 5 NVMe module distribution for 5x3.2TB NVMe cards

Referenced documents SAP HANA on IBM Power Systems and IBM System Storage - Guides

Copyrights and Trademarks © Copyright 2020 IBM Corporation. All Rights Reserved. IBM Corporation Systems and Technology Group

Raid1 -SAP HANA Data

Raid1 -SAP HANA Log

Raid 0 for /hana/dataRaid 0 for /hana/data

1/0.5 TB SAP HANA Configuration on 3 NVMe cards

Raid 0 for /hana/logRaid 0 for /hana/log

Raid1 -OS

Raid 0 for /Raid 0 for /

2 TB SAP HANA Configuration on 5 NVMe cards

Raid1 -SAP HANA Data

Raid1 -SAP HANA Log

Raid 0 for /hana/dataRaid 0 for /hana/data

Raid 0 for /hana/logRaid 0 for /hana/log

Raid1 -OS

Raid 0 for /Raid 0 for /


12


Route 100 Somers, NY 10589 Neither this documentation nor any part of it may be copied or reproduced in any form or by any means or translated into another language, without the prior consent of the IBM Corporation. IBM makes no warranties or representations with respect to the content hereof and specifically disclaim any implied warranties of merchantability or fitness for any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date. Edition Notice: 2020 This is version 1.0 of this document. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. A current list of IBM trademarks is available on the web at: Copyright and trademark information. Adobe and PostScript are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries. Intel, Intel Xeon, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc., in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is the registered trademark of Linus Torvalds in the United States, and other countries. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. SAP HANA, SAP NetWeaver are trademarks or registered trademarks of SAP Corporation in the United States, other countries, or both. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other company, product or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind. Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide home pages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.

Disclaimer and Special Notices This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems.

http://www.ibm.com/legal/copytrade.shtml

13


Furthermore, some measurement may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs ANY INFORMATION HEREIN IS PROVIDED “AS IS” WITHOUT WARRANTY OR INDEMNIFICATION OF ANY KIND BY IBM AND DO NOT ANY EXPRESS OR IMPLIED, WARRANTIES OR CONDITIONS OF MERCHANTABILITY, FITNESS OR USAGE FOR PARTICULAR PURPOSE AND ANY WARRANTY OR CONDITION OF NON-INFRINGEMENT.

Documents

Planning and Fast-Start-Solutions for Implementation Guide ......This planning guide will focus on the available Fast-Start-Solutions for SAP HANA on Power. Hardware setup for Fast-Start-Solution