Sap Hana on Rhel on Ibm Xserver Using Gpfs

RED HAT ENTERPRISE LINUX FOR SAP HANA ON IBM: INSTALLATION GUIDE

August 2014

Version 2.1

www.redhat.comCopyright 2014 Red Hat, Inc. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, and GlusterFS are trademarks of Red Hat, Inc., registered in the U.S. and other countries. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Java is the registered trademark of Oracle America, Inc. in the United States and other countries. All other trademarks are the property of their respective owners.

Red Hat Enterprise Linux for SAP HANA: Installation Guide | August 2014 2

Disclaimer This document has been carefully crafted to assist with setup of SAP HANA on Red Hat Enterprise Linux 6 on IBM X-Series Servers and is subject to change without notification and will not cover the issues encountered in every customer situation. It should be used only in conjunction with the official product literature from Red Hat and IBM. The information contained in this document has not been submitted to any formal Red Hat or IBM test and is distributed AS IS.

All statements regarding Red Hat or IBM future direction and intent are subject to change or withdrawal without

notice, and represent goals and objectives only.

Altering the configuration by adding or removing software components will result in a non-certified configuration and void warranty and support.

This document is for educated service personnel only. If you are not familiar with the described system, we will ask you to restrain from trying to apply what is described herein you could void the preloaded system installation and void the SAP certified configuration. This will void the warranty and support of said machine. Please contact the Red Hat at the SAP Linux Labs or IBM SAP International Competence Center (ISICC) to get enrolled for education prior to installing an IBM Workload Optimized Solution for SAP HANA appliance.

In case of issues with the SAP HANA appliance, the customer is asked to open a SAP Help Desk request (OSS ticket) first andforemost. Only by following this path, can we ensure the proper configuration of the IBM Workload Optimized Solution for SAP HANA. If the customer would open a Red Hat or IBM support ticket for the system, he might be requested to perform system upgrades to firmware or software to the latest available levels which might not be supported with the SAP HANA appliance. If identified as a hardware or file system issue, the ticket will be forwarded to the IBM support team and handled appropriately. Although this may be contrary to standard IBM Support processes, it is the approved and accepted support process for all SAPAppliances including the SAP HANA appliance.

Table of ContentsSetting up Single Node and Cluster Configurations with GPFS and SAP HANA......3

Installing the nodes First steps.................................................................................................................4Manual Installation of Red Hat Enterprise Linux for SAP HANA...........................................................4Network configuration............................................................................................................................8

GPFS Deployment.....................................................................................................................................10Installation of the GPFS packages and configuration files...................................................................10Preparing the nodes and building the GPFS kernel modules..............................................................13Configuring GPFS................................................................................................................................14

Installation of SAP HANA..........................................................................................................................17Unattended Installation using Kickstart.....................................................................................................18

Kickstart RHEL for SAP HANA............................................................................................................19Installing GPFS using the scripts from Kickstart..................................................................................22Installing SAP HANA using the scripts from Kickstart..........................................................................22

References and Notes..................................................................................................23Changelog................................................................................................................................................. 24


SETTING UP SINGLE NODE AND CLUSTER CONFIGURATIONS WITH GPFS AND SAP HANA

This document describes the installation of Red Hat Enterprise Linux (RHEL) for SAP HANA on a standalone IBM X-Series Server as well as a cluster of IBM X-Series Servers for using SAP HANA in combination with the cluster file system GPFS. For GPFS, also single-node configurations are regarded as cluster; they just consist of one node. There are two installation modes described, a manual installation as well as an automatic installation using Kickstart.

General recommendations for setting up RHEL 6 servers for running SAP HANA can be found in the following SAP Notes:

SAP Note 2009879 - SAP HANA Guidelines for Red Hat Enterprise Linux (RHEL) Operating SystemSAP Note 2013638 - SAP HANA DB: Recommended OS settings for RHEL 6.5

SAP Note 2001528 - Linux: SAP HANA Database SPS 08 revision 80 (or higher) on RHEL 6 or SLES 11

SAP Note 1084263 - 1084263 - Cluster File System: Use of GPFS on Linux

In this guide, a single-node configuration and a three-node cluster configuration are described step by step. The host name used for the single node setup is hanamachine01; for the three-node setup the hostnames used are hanamachine0{1,2,3}. In general, all machines used are IBM System X-Series3850 X6 which are certified according to the IBM Quick Start Guide. [1] (See the References and Notes section at the end of this document.)

The initial network configuration is as follows:

Host name hanamachine03 hanamachine02 hanamachine01

MAC-Address (i350) 40:f2:e9:30:11:c0 40:f2:e9:30:11:0c 40:f2:e9:30:08:5c

IP-Address (i350) 10.68.204.178 10.68.206.124 10.68.206.123

Netmask (i350) 255.255.252.0 255.255.252.0 255.255.252.0

Gateway (i350) 10.68.204.1 10.68.204.1 10.68.204.1


Installing the nodes First stepsFirst, install Red Hat Enterprise Linux for SAP HANA on all machines. The RAID configuration was already done by the OEM. Please refer to the Implementation Guide [9] for details.

Connect the remote CD/DVD image to the Integrated Memory Module (IMM).

Boot the machine and select F12 for the boot menu.

Choose the boot option IMM1:CD/DVD - IMM Remote Mount for booting from the virtual installationDVD.

Manual installation of Red Hat Enterprise Linux for SAP HANAFor general information on installing Red Hat Enterprise Linux please refer to the Red Hat Enterprise Linux 6 Installation Guide [2].

When the black screen pops up choose the entry, Red Hat Enterprise Linux.


When the GRUB boot screen appears press a key before the countdown runs out. You should now see the following boot selection menu:

Navigate to Install system with basic video driver and press [Enter]. Install the operating system on the first hard drive /dev/sda with about 1 TB in size. Choose minimal installation and configure the primary network adapter according to the settings

above. Accept the default partitioning scheme (Use all space). If connection to the Red Hat Customer Portal or a local Red Hat Satellite server is possible please

register the system there as described in the Red Hat Enterprise Linux 6 Installation Guide [2] and subscribe the system to the RHEL for SAP HANA channel.

If connection to the RH Customer Portal is not possible, check if the Installaton DVD is already mounted. If this is not the case, mount it, e. g. to /cdrom. After mounting the Installation DVD, create the file /etc/yum.repos.d/rhel-dvd.repo with the following content:

[RHEL6-DVD] name=dvd baseurl=file:/// gpgcheck=1enabled=1

Wipe out all cached yum data with:

# yum clean all

Now install the required packages with:

# yum -y groupinstall base infiniband# yum -y install gtk2 libicu xulrunner ntp sudo tcsh libssh2 \rsyslog expect cairo iptraf krb5-workstation krb5-libs.i686 \nfs-utils lm_sensors libtool-ltdl xauth openssl098e gcc gcc-c++ \ kernel-headers kernel-devel PackageKit-gtk-module libcanberra-gtk2 \cpufrequtils libunwind unixODBC infiniband-diags perftest qperf \ opensm ibutils lsscsi ipmitool tuned icedtea-web

Install the RHEL for SAP HANA compatibility package. You can find it in the child channel. Alternatively, you can use the RHEL for SAP HANA ISO image which should have been provided as part of the installation package. Mount it via loopback using:

# mount -o ro,loop /mnt

Change to the /mnt/Packages directory and install the compat-sap-c++ package:

# yum localinstall compat-sap-c++[tab]

Disable SELinux. SAP HANA is currently not compatible with SELinux:

# sed -i s/^SELINUX=*/SELINUX=disabled/g \ /etc/sysconfig/selinux


A precise clock is crucial for database applications, especially in a clustered environment. Therefore NTP should be configured on all HANA nodes: Modify /etc/ntp.conf to:

server .fqdn

Turn NTP off, set the hardware clock and turn it on again:

# service ntpd stop# ntpdate# service ntpd start# chkconfig ntpd on

The ntpdate service script adjusts the time according to the ntp server every time when the system comes up. This happens before the regular ntp service is started and ensures an exact system time even if the time deviation is too big to be compensated for by the ntp service. For details please refer to note [10].

# echo .fqdn >> /etc/ntp/step-tickers# chkconfig ntpdate on

Disable the following services to avoid overhead:

# chkconfig kdump off# chkconfig abrtd off# chkconfig abrt-cpp off# chkconfig rhsmcertd off# chkconfig NetworkManager off# chkconfig opensm off

Make sure that some essential services are enabled:

# chkconfig network on # chkconfig ntpd on # chkconfig rdma on# chkconfig tuned on

Create some symbolic links required for running SAP HANA:

# ln -s /usr/lib64/libssl.so.0.9.8e /usr/lib64/libssl.so.0.9.8# ln -s /usr/lib64/libssl.so.1.0.1e /usr/lib64/libssl.so.1.0.1# ln -s /usr/lib64/libcrypto.so.0.9.8e /usr/lib64/libcrypto.so.0.9.8# ln -s /usr/lib64/libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1

The usage of Transparent Hugepages (THP) is not supported when running SAP HANA and IBM GPFS. Therefore THP should be disabled permanently. To disable THP on boot, add the option, transparent_hugepage=never at the end of all lines stating with kernel in the file /boot/efi/EFI/redhat/grub.conf.

Transparent Hugepages can be disabled on the running system with the following command:

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

To tune the systems for best performance, activate the tuned with the latency-performance profile:

# tuned-adm profile latency-performance# service tuned start


To prevent fork-bombs, RHEL restricts the maximum amount of processes per user. For hardware with 8 IvyTown sockets and larger, the maximum number of processes for the sapsys group isn't sufficient. The symptom is that the installation of HANA fails in this case. The solution is to create the file, /etc/security/limits.d/99-sap.conf

@sapsys soft nproc unlimited

Keep in mind that this can be a potential security issue. Disable core file creation when processes crash by adding the following to

/etc/security/limits.d/99-sap.conf:

* soft core 0* hard core 0

Network configurationFor a deep dive in networking using RHEL, please refer to the Red Hat Enterprise Linux 6 Deployment Guide Part III - Networking [3].The layout the network configuration is based upon at least one of three network segments:

The primary network connection for things like administrator access The network for the gpfsnode0n bonded interfaces used for GPFS communication

In case of a single-node configuration the IP address of gpfsnode01 is set to 127.0.1.1/8

The network for the communication used by SAP HANA itself, hananode0n In case of a single-node configuration the IP address of hananode01 is set to

127.0.2.1/8 First of all, we will have to determine which network interface will be used for the primary network.

In this case, it is an Intel i350 type which is connected to the internal network infrastructure and is detected as eth4. To make sure you choose the connected one, you can run:

# ethtool -p eth4 30

The interface LED blinks for 30 seconds and can be identified easily. Configure the primary network interface. Modify /etc/sysconfig/network-scripts/ifcfg-

eth4 to:DEVICE=eth4BOOTPROTO=static IPADDR= GATEWAY= PREFIX=HWADDR=ONBOOT=yes TYPE=Ethernet DNS1=DNS2= IPV6INIT=no USERCTL=no


Configure the bonding interfaces. In this case we have Mellanox Infiniband cards in Ethernet mode.The interfaces to be grouped in this example are eth{0,2} and eth{1,3}. The files /etc/sysconfig/network-scripts/ifcfg-eth{0..3} look like this:

DEVICE=eth{0..3} USERCTL=no ONBOOT=yes HWADDR= SLAVE=yesMASTER=bond{0,1}

Now we can define the bonded interfaces. These are defined in /etc/sysconfig/network-scripts/ifcfg-bond{0,1}:

DEVICE=bond{0,1}IPADDR= NETWORK= NETMASK=USERCTL=no BOOTPROTO=noneONBOOT=yes MTU=9000

We use jumbo frames (MTU=9000) here to maximize performance. Please check with your NetworkManagement team to make sure jumbo frames are enabled on all switches etc.

Bonding needs a kernel module with the appropriate options. To load it when activating bonding, create the file /etc/modprobe.d/bonding.conf:

alias bond0 bonding options bond0 mode=802.3ad xmit_hash_policy=layer3+4 miimon=100 alias bond1 bonding options bond1 mode=802.3ad xmit_hash_policy=layer3+4 miimon=100

The option mode=802.3ad activates channel aggregation to maximize performance. Please checkwith your Network Management team to make sure that it is possible to use this mode, otherwise use a mode that is supported in the network environment.

Check the file /etc/udev/rules.d/70-persistent-net.rules for proper mapping betweenthe network devices and MAC-addresses.

Check the file /etc/sysconfig/network. It should contain:

NETWORKING=yes HOSTNAME=

The HOSTNAME entry must be the short one, not FQDN. Otherwise, SAP HANA refuses to operate. You can get it by running hostname -s.

Make sure that the DNS domain names and DNS servers are correctly configured in /etc/resolv.conf:

search nameserver nameserver


A working name resolution is required at all times. Using DNS in a clustered environment can be dangerous because if the DNS service fails, the cluster would be inoperable. To avoid that the cluster fails to operate in case the DNS servers aren't reachable, the best practice is to make sure that /etc/nsswitch.conf contains the following line:

hosts: files dns

Then list all the cluster members in the file /etc/hosts on each node. In our three-node example this would be:

192.168.206.123 hanamachine01 hanamachine01.example.com192.168.123.102 gpfsnode01 gpfsnode01192.168.234.102 hananode01 hananode01192.168.206.124 hanamachine02 hanamachine02.example.com192.168.123.101 gpfsnode02 gpfsnode02192.168.234.101 hananode02 hananode02192.168.204.178 hanamachine03 hanamachine03.example.com192.168.123.103 gpfsnode03 gpfsnode03192.168.234.103 hananode03 hananode03

In a single-node environment, just the first node would be used:

192.168.206.123 hanamachine01 hanamachine01.example.com127.0.1.1 gpfsnode01 gpfsnode01127.0.2.1 hananode01 hananode01

To ensure that the traffic inside the cluster of a multi-node configuration is piped through the internal HANA network in order to gain throughput, a route to each node must be set on each node [4]. In our three-node example we would set them in /etc/sysconfig/network-scripts/route-bond1:

GATEWAY0=0.0.0.0NETMASK0=255.255.255.255ADDRESS0=192.168.206.123GATEWAY1=0.0.0.0NETMASK1=255.255.255.255ADDRESS1=192.168.206.124GATEWAY2=0.0.0.0NETMASK2=255.255.255.255ADDRESS2=192.168.204.178


GPFS deploymentThe following steps have been taken from the Implementation Guide for SAP HANA Appliance [5]. The guide is intended for a different platform and the steps were adapted to RHEL for SAP HANA. Please refer to this document to obtain further information.

Installation of the GPFS packages and configuration filesThe minimum maintenance level of GPFS for running RHEL for SAP HANA on IBM HANA appliances is 3.5.0-17.In general, you need to download the packages required for running GPFS directly from IBM or install them from your local repository or custom channel. This procedure must be repeated on every node in the cluster.

First, install the base package:

# yum -y install gpfs.base

Ensure that the base package is updated to the latest release and install additional packages required for running GPFS:

# yum -y update gpfs.base# yum -y install gpfs.docs gpfs.gpl gpfs.msg.en_US

We need a path extension for the GPFS tools and several directories. Create the file /etc/profile.d/gpfs.csh:

# setenv PATH ${PATH}:/usr/lpp/mmfs/bin

Create the file /etc/profile.d/gpfs.sh:

# export PATH=$PATH:/usr/lpp/mmfs/bin

Now set the permissions appropriately:

# chmod 644 /etc/profile.d/gpfs.csh# chmod 644 /etc/profile.d/gpfs.sh

The following directories need to be created: the first one for temporary data, the second one for the GPFS configuration files:

# mkdir -p /tmp/GPFSdump# mkdir -p /var/mmfs/config

A number of configuration files are required to start GPFS successfully. The following configuration is an example. You may see more or fewer block devices on your

machine depending on its configuration. Every %nsd-entry requires one dedicated block device. The first block device, e.g. /dev/sda must not be used because it is normally used to hold the operating system.

The first configuration file on node n (see /etc/hosts above which host name corresponds to which node) is /var/mmfs/config/disk.list.data.gpfsnode0n

%nsd: device=/dev/sdb nsd=data01node0nservers=gpfsnode0n usage=dataAndMetadata failureGroup=100npool=system %nsd: device=/dev/sdc nsd=data02node0nservers=gpfsnode0n


usage=dataAndMetadata failureGroup=100npool=system

In this example the two disks dedicated to GPFS are /dev/sd{b,c}. The second configuration file required is /var/mmfs/config/nodes.list.

In a three-node environment the file should contain the following:

gpfsnode01gpfsnode02gpfsnode03

If there is just a single-node configuration the file should contain:

gpfsnode01

The third and last configuration file /var/mmfs/config/nodes.cluster holds the quoruminformation. In a three-node cluster we use:

gpfsnode01:quorumgpfsnode02:quorumgpfsnode03:quorum

If there is just a single node configuration, we omit the other nodes here:

gpfsnode01:quorum

Preparing the nodes and building the GPFS kernel modules For the compilation of the GPFS kernel module, some additional packages must be installed if not

already done:

# yum -y install gcc gcc-c++ kernel-headers kernel-devel

To avoid a reboot of the machine set the PATH variables required manually:

# export PATH=$PATH:/usr/lpp/mmfs/bin

Wipe the signature of the disks. WARNING: make sure to use the correct devices to prevent data loss:

# dd if=/dev/zero of=/dev/sdb bs=1G count=1 # dd if=/dev/zero of=/dev/sdc bs=1G count=1

GPFS accesses the other nodes using SSH. Therefore SSH keys need to be set up on all nodes. For a three-node cluster, generate a RSA-key and distribute it to all other nodes and

localhost for SSH-access of GPFS:

# NODES=(hanamachine0{1,2,3} gpfsnode0{1,2,3} hananode0{1,2,3})# ssh-keygen -t dsa -N "" -f ~/.ssh/id_dsa# for node in ${NODES[@]}; do

ssh-copy-id -i /root/.ssh/id_dsa.pub $nodedone


For a single-node configuration, generate a RSA-key and distribute it just to localhost for SSH-access of GPFS:

# NODES=(hanamachine01 gpfsnode01 hananode01)# ssh-keygen -t dsa -N "" -f ~/.ssh/id_dsa# for node in ${NODES[@]}; do

ssh-copy-id -i /root/.ssh/id_dsa.pub $nodedone

Now we can build the kernel module and the portability layer. Keep in mind that after every kernel update this process must be repeated:

# cd /usr/lpp/mmfs/src# make Autoconfig# make World# make InstallImages

Configuring GPFSThe majority of the configuration steps are only done on the master node. In this example, it is hanamachine01 which represents node 2 and therefore corresponds to gpfsnode02.

For a three-node cluster, create the GPFS cluster on the master node with:

# mmcrcluster -n /var/mmfs/config/nodes.cluster -p gpfsnode01 \-s gpfsnode02 -r /usr/bin/ssh -R /usr/bin/scp -C HANAcluster -A

For a single-node configuration, just include this node with:

# mmcrcluster -n /var/mmfs/config/nodes.cluster -p gpfsnode01 \-r /usr/bin/ssh -R /usr/bin/scp -C HANAcluster -A

On a three-node configuration assign a GPFS server license on the master node to all nodes in thecluster:

# mmchlicense server --accept -N gpfsnode01,gpfsnode02,gpfsnode03

If there are more than three servers, use the first three as quorum nodes. The quorum nodes need server licenses. The remaining nodes must get the FPO license assigned:

# mmchlicense fpo --accept -N gpfsnode0X,gpfsnode0Y,gpfsnode0Z

On a single-node configuration, just assign the node itself:

# mmchlicense server --accept -N gpfsnode01

Create the file /var/mmfs/config/disk.list.data.fs On the three-node cluster, create a combined disk file on the master node with:

#rm -f /var/mmfs/config/disk.list.data.fs# for i in {1..3}; doscp gpfsnode0$i:/var/mmfs/config/disk.list.data.gpfsnode0$i \/var/mmfs/configcat /var/mmfs/config/disk.list.data.gpfsnode0$i >> \/var/mmfs/config/disk.list.data.fsdone


On a single-node configuration, create a disk file for the node itself:

# rm -f /var/mmfs/config/disk.list.data.fs# cat /var/mmfs/config/disk.list.data.gpfsnode01 >> \/var/mmfs/config/disk.list.data.fs

On every node, create the GPFS disks with:

# mmcrnsd -F /var/mmfs/config/disk.list.data.gpfsnode0n

Adjust various GPFS settings on the master node:

# mmchconfig dataStructureDump=/tmp/GPFSdump,pagepool=16G,\maxMBpS=2048,maxFilesToCache=4000,nsdInlineWriteMax=1M,\readReplicaPolicy=local,enableLinuxReplicatedAIO=yes,\restripeOnDiskFailure=yes

Important: There must be no tabs or spaces between the comma-separated arguments of mmchconfig. Otherwise, the command doesn't work and GPFS will come up with a wrong configuration.

On the master node, start GPFS:

# mmstartup -a

Create the GPFS file system. On the master node for a multi-node cluster use the following command:

# mmcrfs sapmntdata -F /var/mmfs/config/disk.list.data.fs -A yes \-B 1M -N 3000000 -v no -m 2 -M 3 -r 2 -R 3 -j hcluster \--write-affinity-depth 1 -s failureGroupRoundRobin \--block-group-factor 1 -Q yes -T /sapmnt

On a single-node configuration, create the file system with the following:

# mmcrfs sapmntdata -F /var/mmfs/config/disk.list.data.fs -A yes \-B 1M -N 3000000 -v no -m 1 -M 3 -r 1 -R 3 -Q yes -T /sapmnt

On the master node, create the filesets needed for SAP HANA:

# mmcrfileset sapmntdata hanadata -t "Data Volume for HANA database"# mmcrfileset sapmntdata hanalog -t "Log Volume for HANA database"# mmcrfileset sapmntdata hanashared -t \"Shared Directory for HANA database"

On the master node, mount the GPFS file system with:

# mmmount /sapmnt -a

On the master node, link the filesets into the file system:

# mmlinkfileset sapmntdata hanadata -J /sapmnt/data# mmlinkfileset sapmntdata hanalog -J /sapmnt/log# mmlinkfileset sapmntdata hanashared -J /sapmnt/shared

On the master node, check if the filesets have been properly created:

# mmlsfileset sapmntdata

Define quotas. This can be done using a helper script to be obtained from IBM:

# sh ./saphana-quota-calculator.sh -a

GPFS should now be running on all nodes. To check, the following can be used:

# df -hT # -> /sapmnt# mmdf /dev/sapmntdata


Some permissions need to be set. On the master node, set these to:

# chmod -R 755 /sapmnt/shared # chmod -R 777 /sapmnt/data # chmod -R 777 /sapmnt/log

Installation of SAP HANAThe last step is the installation of the database which is done on the master node exclusively. For details on installing SAP HANA please refer also the SAP HANA Server Installation and Update Guide [6]. The installation will be done in unattended mode. The following things are required for this step:

A password file called /root/password.xml. It has the following appearance:

5WqX59!,

5WqX59!,

5WqX59!,

redhat

Replace the values between the brackets in the XML file with the following values according to

the requirements of the customer:

password

sapadm_password

system_user_password

root_password

The archive of the SAP HANA database software IMDB_SERVER.SAR which can be obtained from the SAP Service Marketplace. Copy the archive to the /tmp directory.

The archive utility SAPCAR_.exe which is also available from the SAP Service Marketplace. Copy the archive utility to the local /root/bin directory.

Extract the archive with:

# cd /tmp# //SAPCAR_.exe -xvsf IMDB_SERVER.SAR

Change the directory to:

# cd /tmp/SAP_HANA_DATABASE


For stability reasons, the memory usage of SAP HANA should be limited. The recommendation is as follows: Memory 256 MiB: max_mem = 90%

# TOT_MEM=$(echo $(cat /proc/meminfo | grep MemTotal | \# awk '{print $2}')/1024 | bc | cut -d. -f1)# if [ $TOT_MEM -le 262144 ]; then MAX_MEM=$(echo $TOT_MEM*0.8 | bc | cut -d. -f1)else MAX_MEM=$(echo $TOT_MEM*0.9 | bc | cut -d. -f1)fi

For the installation of SAP HANA, the customer can choose the SID and Instance number (options --sid= and number=). On our three-node cluster example, we can now install the software in unattended mode with:

# LD_PRELOAD=/opt/rh/SAP/lib64/compat-sap-c++.so \cat /root/password.xml | /tmp/SAP_HANA_DATABASE/hdblcm \ --read_password_from_stdin=xml --sid=ANA --number=00 \ --sapmnt=/sapmnt/shared --datapath=/sapmnt/data/ANA \ --logpath=/sapmnt/log/ANA \ --shell=/bin/bash --hostname=hanamachine01 -b --system_usage=test \ --autostart=1 --components=server,client --timezone=UTC \ --max_mem=$MAX_MEM \ --addhosts=hanamachine02:role=worker,hanamachine03:role=worker \ --internal_network=192.168.234.102/24 \ --home=/usr/sap/ANA/home --userid=1100 --groupid=79 \ --certificates_hostmap=hananode01=hanamachine02.wdf.sap.corp,\ hananode02=hanamachine01.wdf.sap.corp,\hananode03=hanamachine03.wdf.sap.corp --vm=/usr/bin/java

On the single-node configuration, SAP HANA can be installed as follows:

# LD_PRELOAD=/opt/rh/SAP/lib64/compat-sap-c++.so \cat /root/password.xml | /tmp/__installer.HDB/hdblcm --read_password_from_stdin=xml --sid=ANA --number=00 \ --sapmnt=/sapmnt/shared --datapath=/sapmnt/data/ANA \ --logpath=/sapmnt/log/ANA \ --shell=/bin/bash --hostname=hanamachine01 -b --system_usage=test \ --autostart=1 --components=server,client --timezone=UTC \--max_mem=\$MAX_MEM --vm=/usr/bin/java

After a couple of minutes, SAP HANA should be up and running on all nodes. This can be observedon each node with:

# su - anaadm# watch -n1 HDB info


Unattended Installation using KickstartTo perform an unattended installation, for each node a Kickstart file needs to be prepared. A Kickstart file is an answer file which is passed over to anaconda, the system installer of RHEL for SAP HANA. It contains allthe answers a user would have made to the installer if he had chosen an interactive installation process. In addition, the Kickstart file can also contain %pre and %post scripts for automating the configuration process, e.g. the creation and modification of configuration files or license agreements before the installation. In this example the Kickstart file also installs scripts for installing and configuring GPFS and SAPHANA. Please refer also to notes [7] and [8].

Kickstart RHEL for SAP HANA Begin with the first steps described in the first section of this guide. Start booting with the virtual

DVD. When the black screen with the countdown pops, up press a key. You are now delegated to the

installation menu. Navigate to Install system with basic video driver and press a to append the

following to the kernel command line: hanamachine01 (single-node):

ks=http://10.18.98.198/hanamachine01_single.cfg \ksdevice=40:f2:e9:30:05:58 ip=10.68.204.181 netmask=255.255.252.0 \ gateway=10.68.204.1


hanamachine01 (multi-node):

ks=http://10.18.98.198/hanamachine01.cfg \ksdevice=40:f2:e9:30:08:5c ip=10.68.206.123 netmask=255.255.252.0 \ gateway=10.68.204.1

hanamachine02:

ks=http://10.18.98.198/hanamachine02.cfg \ksdevice=40:f2:e9:30:11:0c ip=10.68.206.124 netmask=255.255.252.0 \ gateway=10.68.204.1

hanamachine03:

ks=http://10.18.98.198/hanamachine03.cfg \ ksdevice=40:f2:e9:30:11:c0 ip=10.68.204.178 netmask=255.255.252.0 \ gateway=10.68.204.1

Press [Enter] to start the machine. Sometimes, due to network problems it cannot access the network during the first couple of attempts.

Check for the URL of the Kickstart file and press the tab key. As soon the OK button is highlighted,press [Space] to confirm. If the problem persists, wait a few seconds and press enter again. Also, due to these problems the installer cannot see the URL of the RHEL distribution.Move the cursor using the [Tab] key to URL and choose OK.

As the URL to the distribution, enter ftp://10.18.98.198/pub/rhel6/dvd here.


Now you are asked to confirm the license agreement.If the Reject button is pressed, the installation process will be stopped and the machine is rebooted.

All file systems are created and all packages and package groups listed in the Kickstart file in the %packages section are installed now using Kickstart.

Installing GPFS using the scripts from KickstartIn general, GPFS must be installed on all three nodes. After reboot of all machines of the cluster right after the installation, you can see three scripts in the /root directory:

All nodes: /root/install.gpfs.sh All nodes: /root/configure.gpfs.sh Master node: /root/install.hana.sh

Install GPFS on all three nodes using:

# /root/install.gpfs.sh

Check the output for errors or warnings. Create the cluster on the master node gpfsnode02.

On the master node launch:

# /root/configure.gpfs.sh

On all other nodes launch:

# /root/configure.gpfs.sh

Create the file system on the master node with:

# /root/create.gpfs.sh

Now check the installation on each node with:

# df -hT# mmdf /dev/sapmntdata

Installing SAP HANA using the scripts from Kickstart The last step is to install SAP HANA on the master node using:

# /root/install.hana.sh

After a couple of minutes, SAP HANA should be up and running on all nodes. This can be observedon each node with:

# su - anaadm# watch -n1 HDB info


REFERENCES AND NOTES

Number Remarks

[1] http://www-947.ibm.com/support/entry/portal/docdisplay?lndocid=MIGR-5087035

[2] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Installation_Guide/index.html

[3] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.html#part-Networking

[4] Verbal message from F. Bausch on July 02, 2014[5] http://www-

03.ibm.com/support/techdocs/atsmastr.nsf/5cb5ed706d254a8186256c71006d2e0a/f1359b7fd3cce99786257ca500214577/$FILE/IBM%20-%20SAP%20HANA%20Implementation%20Guide%20X6-1.7.73-9.pdf

[6] http://www.saphana.com/servlet/JiveServlet/download/1031-13-14351/SAP_HANA_Server_Installation_Guide_en.pdf

[7] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Installation_Guide/index.html#pt-Advanced_installation_options

[8] http://fedoraproject.org/wiki/Anaconda/Kickstart

[9] http://www-03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/TD106247

10 https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s1-Configure_ntpdate_Servers.html


ChangelogDate Version Author Remarks

18.06.2014 1.0 Jrgen Blume Initial version of the document

20.06.2014 1.1 Jrgen Blume Correct usage of mmchconfig

25.06.2014 1.2 Jrgen Blume Some typos fixed

25.06.2014 1.3 Jrgen Blume Generic host names

30.06.2014 1.4 Jrgen Blume Added single-node instructions

02.07.2014 1.5 Jrgen Blume Added reference section and routes

08.07.2014 1.6 Jrgen Blume Modified the disclaimer

04.07.2014 1.7 Joachim Kunze Editor changes

07.07.2014 1.8 Frank Danapfel General updates

09.07.2014 1.9 Jrgen Blume Minor updates

11.08.2014 2.1 Joachim Kunze Minor updates


Setting up Single Node and Cluster Configurations with GPFS and SAP HANAInstalling the nodes First stepsManual installation of Red Hat Enterprise Linux for SAP HANANetwork configuration

GPFS deploymentInstallation of the GPFS packages and configuration filesPreparing the nodes and building the GPFS kernel modulesConfiguring GPFS

Installation of SAP HANAUnattended Installation using KickstartKickstart RHEL for SAP HANAInstalling GPFS using the scripts from KickstartInstalling SAP HANA using the scripts from Kickstart

References and NotesChangelog

Documents

Sap Hana on Rhel on Ibm Xserver Using Gpfs