Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system

TIBCO Software Inc.

Global Headquarters

3307 Hillview Avenue

Palo Alto, CA 94304

Tel: +1 650-846-1000

Toll Free: 1 800-420-8450

Fax: +1 650-846-1005

www.tibco.com

TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of your business through analytical insights. Thousands of customers around the globe rely on us to build compelling experiences, energize operations, and propel innovation. Learn how TIBCO makes digital smarter at www.tibco.com.

Configuring a TIBCO Enterprise Message ServiceTM Fault Tolerant Environment on AWS with Zadara Storage This document provides the steps for configuring and testing TIBCO Enterprise Message Service Fault Tolerance with a Linux operating environment on AWS utilizing Zadara Storage

Version 1.0 October 2019 Initial Document

©2019 TIBCO Software Inc. All Rights Reserved. 2

Copyright Notice COPYRIGHT© 2019 TIBCO Software Inc. All rights reserved.

Trademarks TIBCO, the TIBCO logo, and TIBCO Enterprise Message Service are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA


Table of Contents

Configuring a TIBCO Enterprise Message ServiceTM Fault Tolerant Environment on AWS with Zadara Storage .......................................................................................................................................... 1

1 Overview ................................................................................................................................ 5 1.1 Zadara Virtual Private Storage Array Support .................................................................................. 5 1.2 Linux Kernel Support ...................................................................................................................... 5 1.3 EMS Version Support ...................................................................................................................... 6 1.4 Prerequisites .................................................................................................................................. 6

2 EC2 Instance Creation in AWS................................................................................................. 7

3 Zadara Virtual Private Storage Array Configuration ............................................................... 10

4 Configuring the AWS EC2 Instances ...................................................................................... 14 4.1 Software updates ......................................................................................................................... 14 4.2 Configuring NFS ............................................................................................................................ 14

4.2.1 IPsec installation and Configuration .......................................................................................... 14 4.2.2 Create the Zadara NFS Mount Configuration ............................................................................ 15 4.2.3 NFS Mount via Fstab ................................................................................................................. 16 4.2.4 Linux Kernel Changes ................................................................................................................ 16

5 Enterprise Message Service Installation and Configuration ................................................... 18 5.1 EMS Installation ............................................................................................................................ 18 5.2 EMS Configuration ........................................................................................................................ 18

5.2.1 Stores.conf ............................................................................................................................... 18 5.2.2 Factories.conf ........................................................................................................................... 19 5.2.3 Tibemsd.conf ............................................................................................................................ 19 5.2.4 Starting the EMS Instances ....................................................................................................... 21

6 Testing EMS Fault Tolerance on AWS with Zadara Storage .................................................... 22 6.1 EMS Client Application Setup ........................................................................................................ 22 6.2 Performing the EMS Fault Tolerant Test Cases .............................................................................. 23

6.2.1 EMS Process Failure Test .......................................................................................................... 23 6.2.2 Network Failure Test ................................................................................................................ 26 6.2.3 System Failure Test ................................................................................................................... 30


Table of Figures FIGURE 1 - CREATE AWS EC2 INSTANCE .............................................................................................................................. 7 FIGURE 2 - CONFIGURE EC2 INSTANCE DETAILS ..................................................................................................................... 8 FIGURE 3 - SECURITY GROUP INPUTS ................................................................................................................................... 8 FIGURE 4 - NEW EC2 INSTANCES ........................................................................................................................................ 9 FIGURE 5 - VPSA GUI ................................................................................................................................................... 10 FIGURE 6 - CONFIGURING A SERVER IN THE VPSA ................................................................................................................. 11 FIGURE 7 - CREATE SHARED VOLUME EXAMPLE .................................................................................................................... 12 FIGURE 8 – EXAMPLE OF A CREATED VOLUME ...................................................................................................................... 13 FIGURE 9 - ATTACHED SERVERS TO THE VPSA...................................................................................................................... 13 FIGURE 10 - MNT-NFS.MOUNT SERVICE EXAMPLE.................................................................................................................. 16 FIGURE 11 - FSTAB EXAMPLE ........................................................................................................................................... 16 FIGURE 12 - SYSCTL CHANGE ........................................................................................................................................... 17 FIGURE 13 - STORES.CONF EXAMPLE ................................................................................................................................. 19 FIGURE 14 - FACTORIES CONFIGURATION EXAMPLE ............................................................................................................... 19 FIGURE 15 - STARTING AN EMS INSTANCE EXAMPLE ............................................................................................................. 21 FIGURE 16 - CREATE THE SYNC QUEUE ............................................................................................................................... 23 FIGURE 17 - RUNNING THE TIBJMSMSGPRODUCERPERF APPLICATION ....................................................................................... 24 FIGURE 18 - THE STANDBY EMS INSTANCE BECOMING ACTIVE ON SERVER2 ................................................................................ 25 FIGURE 19 - PURGE THE SYNC QUEUE FOR TIBEMSADMIN ....................................................................................................... 26 FIGURE 20 - DROP_NFS.SH SCRIPT .................................................................................................................................... 27 FIGURE 21 - RUNNING DROP_NFS.SH SCRIPT ....................................................................................................................... 28 FIGURE 22 - DISK WRITE ERROR ON SERVER1 ...................................................................................................................... 28 FIGURE 23 - SERVER2 BECOMING ACTIVE AFTER NETWORK FAILURE .......................................................................................... 29 FIGURE 24 - AWS COMPUTE ENGINE/EC2 INSTANCES PAGE................................................................................................... 30 FIGURE 25 – STANDBY EMS INSTANCE RECOVERING FROM SYSTEM FAILURE OF PRIMARY ............................................................... 31


1 Overview

The purpose of the document is to provide a guide to install, configure, and run TIBCO Enterprise Message ServiceTM (EMS) in a fault-tolerant configuration on Amazon Web Service (AWS) utilizing a Zadara Storage System solution for shared storage. In addition, the document will provide the steps and expected results for testing EMS F/T on AWS with Zadara.

The document will outline:

• Setting up Linux EC2 instance on AWS • General outline for setting up the Zadara VPSA for the shared file system • Setting up the NFS4 mount on the Linux AWS EC2 instance • Installing and configuring EMS for F/T • Tuning EMS for the AWS environment • Running tests for:

o EMS process failure o Network failure between the AWS EC2 instance running EMS and the shared

storage o Accidental reboot of the AWS EC2 instance from the AWS Dashboard

1.1 Zadara Virtual Private Storage Array Support

Currently, Amazon Web Services provides EFS as a shared storage platform capable of supporting the shared storage requirements required for TIBCO Enterprise Message Service. However, there are alternatives. Zadara provides an Enterprise Storage as a Service Solution (STaaS). Zadara provides a fully managed on premise or in the cloud storage solution that meets the shared storage requirements for EMS when utilizing NFSv4. See https://www.zadara.com and http://guides.zadarastorage.com/vpsa-guide/1711/ for details on accessing a Zadara Virtual Private Storage Array (VPSA). Note: This document does not provide guidance on creating or accessing the VPSA, but will provide guidance on configuring a NAS/NFSv4 for EMS.

1.2 Linux Kernel Support

This document covers the installation and configuration of EMS on Amazon’s Linux2 kernel, which is based on Linux kernel 4.14.143-118. All Linux distributions that are materially equivalent to the above listed Linux version are supported. See https://docs.tibco.com/pub/ems/8.5.0/TIB_ems_8.5.0_readme.txt?id=1 for details.


1.3 EMS Version Support

EMS version 8.5 or later should be used for EMS server installations running in a AWS environment. Any available EMS hot fixes should also be applied. The Enterprise Edition or the Community Edition of EMS can be used. Download the EE from edelivery.tibco.com , and the CE at https://www.tibco.com/products/tibco-enterprise-message-service.

1.4 Prerequisites

The following prerequisites must be meet before configuring EMS on AWS:

• An AWS Account is required. An AWS Account can be created/accessed at https://console.aws.amazon.com.

• The TIBCO Enterprise Message Service Software version 8.5.0 has been downloaded and is available to be uploaded to the AWS.

• The Zadara VPSA onboarding has been completed, and the VPSA is available for configuration. See section 1.1 for details.

• The reader of this document is familiar with the following concepts: o The use of AWS Console for configuring EC2 instances o TIBCO EMS installation and configuration o Linux configuration o NFSv4 o Zadara VPSA o Configuration of IPsec for encryption


2 EC2 Instance Creation in AWS

Three AWS Linux EC2 instances are required. The following steps will outline creating the EC2 instances in the AWS console. Amazon Linux2 was used for the EC2 instances. Other operating systems or versions can be used, but are not covered in this document. For the following examples, the AWS region US-EAST is used. Note: TIBCO Enterprise Message Service 8.5 is currently not supported on CentOS/RHEL 8 distributions.

To create a new AWS EC2 Linux instance, use the following:

• In the AWS console, select Services, EC2, and then Launch Instance to create a new Amazon EC2 instance. Three EC2 instances are needed: two for EMS F/T and one for the client application.

Figure 1 - Create AWS EC2 Instance

• Select Amazon Linux 2 AMI (HVM), SSD Volume Type - ami-0b69ea66ff7391e80 (64-bit x86) for the AMI

• Select the appropriate size instance (vCPUs and Memory) for the environment • Click on Next: Configure Instance Details • Leave the number instances to 1. Use the default VPC, and defaults for the other settings,

except for the subnet. Select a subnet in the AWS Region Zone where the EC2 instance will be created. The EC2 instances used for the EMS Servers should be in separate zones.


Figure 2 - Configure EC2 Instance Details

• Click Next: Storage • Configure the Volume Type and the Size in GB. These will depend on what else is be

installed other than EMS. If just EMS, 20GB is sufficient. • Click on Next: Add Tags, and then Next: Configure Security Group • Either use a new security group, or an existing security group. Ensure the security group

used has rules to support SSH, TCP, NFS, and ICMP-IPv4 at the minimum. Must consider Source, Port Range, Protocol, and Type.

Figure 3 - Security Group Inputs

• Click on Review and Launch • Repeat the above steps for the two other EC2 instances


Figure 4 - New EC2 Instances

• Once the EC2 Linux instances have started, Putty (on Windows) or ssh on Mac/Linux/UNIX can be used to access the EC2 instances using the public IP address created by AWS.

• Click on the EC2 instance, and then on the Connect button. The Connect to Your Instance instructions will be displayed. Follow the instructions to connect to the EC2 instance.

• Once completed, note the Name, Internal, and External IP Address of the new compute engines.


3 Zadara Virtual Private Storage Array Configuration

The Zadara Virtual Private Storage Array (VPSA) must be configured before completing the setup of the AWS EC2 instances for EMS. See section 1.1 on setting up access to the Zadara Virtual Private Storage Array (VPSA). Zadara will provide the IP Address of the VPSA, as part of the onboarding procedure. This will also be the IP address used on the AWS EC2 instances for the NFS mount. Once there is access to the Zadara VPSA, follow this section to create the basic configuration for the Zadara VPSA with NFSv4 for use as the shared storage for EMS.

• Log into the VPSA using the username and password configured during the initial VPSA setup with Zadara. In the following examples, EMSAWS is the name of the VPSA. Click on Drives under resources, and verify there is at least one drive to configure.

Figure 5 - VPSA GUI

• Next, click on Servers under Resources. Click on Add and then Manual. This will need to be done for the two AWS EC2s used as the EMS servers in AWS. Provide:

o The name of the AWS EC2 instance used for EMS server o The OS type (Linux) o The internal IP Address of the AWS EC2 instance o Click on Enable IPsec if end to end encryption is required (recommended). o Click on Create

• The newly created server should look similar to the following example.


Figure 6 - Configuring a Server in the VPSA

• Create the volume in the VPSA for the NFSv4 mount • Click on Volumes under Resources, and then on Create, and then click on Create NAS

Share. Provide the: o The name for the new volume o The capacity for the volume (in GiB). Must be below the capacity of the drive o Export name for the NFS server. Note the export name, a this will also be used as

part of the NFS mount on the AWS EC2 instances. o Click on encrypted, if data at rest encryption is required. If IPsec was enabled

previously, it recommended to enable encryption at rest also o Ensure 512 KiB read ahead is selected o The rest of the values can be at their defaults if desired o Click on submit


Figure 7 - Create Shared Volume Example

• Once the new volume is created, click on the details for the newly created volume • Ensure that the all settings defined previously have been enabled


Figure 8 – Example of a created Volume

• Click on Servers from the top section of the VPSA GUI, while still on volumes • Next, click on Attach to Server(s), and click on the two servers previously created, and click

submit. This will allow the two AWS EC2 instances to be attached to the VPSA via the NFS mount.

• Click on Servers in the details for the volume. The two EC2 instances should be listed.

Figure 9 - Attached Servers to the VPSA

• The VPSA is now ready for use as the shared storage device.


4 Configuring the AWS EC2 Instances

The individual AWS EC2 instances can now be configured. The following sections will outline additional software and changes needed for EMS installation and configuration.

4.1 Software updates

Additional software is required to install and configure EMS and NFSv4 on the EC2 instances created on AWS. Not all software needs to be installed all machines, but it simplifies the installation.

• Log into each of the newly created AWS EC2 instances. Follow the AWS documentation for using ssh to connect to each EC2.

• There are several packages to install, but all can be installed at once. Use o sudo yum install -y nfs-utils java-devel unzip to install all packages o sudo yum update -y to update all packages

• Repeat on the other EC2s

4.2 Configuring NFS

On the two AWS EC2 instances which will be used as the EMS servers, NFSv4 must be configured and the shared storage mounted. If IPsec is being used, it must be installed and configured first. If IPsec is not being used, sections 4.2.1 and 4.2.2 can be skipped.

4.2.1 IPsec installation and Configuration IPsec will provide “end to end” encryption of all EMS messages between the EMS server and the Zadara VPSA. Follow this section to install and configure IPsec.

• An additional software package must be downloaded and installed. Openswan is utilized to connect to Zadara VPSAs over IPsec. Use sudo yum install –y openswan to install openswan

• Add the following to /etc/ipsec.conf:

conn zadara-VPSA-NFS type=transport auto=start authby=secret ike=aes128-sha1-modp1024 phase2=esp phase2alg=aes128-sha1 ikev2=never dpddelay=5 dpdtimeout=5 dpdaction=restart pfs=no


salifetime=1h rekey=yes rekeymargin=9m keyingtries=%forever # Always try to connect to VPSA ikelifetime=3h left=<Internal IP Address of the EC2 instance> leftprotoport=tcp right=<IP Address of the VPSA> rightprotoport=tcp/2049

• The file /etc/ipsec.secrets needs to contain the shared secret. This is the IPSEC key from the VPSA:

o From the Main Dashboard panel of the VPSA on the left hand side, click on System, then Settings.

o In the main screen, click on Security. The IPSEC key is at the bottom of the screen. This is the PSK.

o Edit /etc/ipsec.sec, and modify as shown below:

################### <IP Address of the EC2 instance> <IP Address of the VPSA>: PSK "<PSK>" ###################

For example: 10.150.0.3 172.24.140.200: PSK "D12340348GF34E26A481D74A1F456789"

• Start IPsec: sudo systemctl restart ipsec • To start after reboots: sudo systemctl enable ipsec • To verify IPsec, use: sudo ipsec whack –status - The output should contain information

that the connection to the Zadara VPSA has been established. Note: If IPsec is being used, Section 4.2.1 must be completed on both AWS instances where the EMS servers will run before continuing! See https://support.zadarastorage.com/hc/en-us/articles/213024926-How-to-enable-IPsec-on-Linux-instances for additional information.

4.2.2 Create the Zadara NFS Mount Configuration By default, with Linux systemd, the IPsec service may stop before the Zadara NFS storage can be completely dismounted. This will cause time-outs for several minutes while trying to stop the EC2 instance. To ensure the NFS dismount occurs before IPsec is stopped on a more consistent basis, an NFS mount service must be created.

Note: If IPsec is not being used, see section 4.2.3 for mounting the Zadara Storage via fstab.

• Create /usr/lib/systemd/system/mnt-nfs.mount. Note: The following example shows 172.32.160.200:/export/emsaws as the NFS Server, and /mnt/nfs as the mount point. Change accordingly. If /mnt/nfs is not the mount point, then the name of the mount service must also be changed. For example, if the mount point


is /opt/aws, then the name of the mount file would be opt-aws.mount. DO NOT modify the options shown below!

[Unit] Description=Zadara NFS /mnt/nfs

Requires=ipsec.service After=ipsec.service

[Mount] What=172.32.160.200:/export/emsaws/ Where=/mnt/nfs

Type=nfs Options=nfsvers=4.0,soft,retrans=2,actimeo=1,timeo=150,_netdev

[Install]

WantedBy=multi-user.target

Figure 10 - mnt-nfs.mount service example

• Enable the new mount service to run on startup: sudo systemctl enable mnt-nfs.mount • Create a new mount point for the storage: sudo mkdir /mnt/nfs • Mount the shared storage using sudo systemctl start mnt-nfs.mount

4.2.3 NFS Mount via Fstab If IPsec is not being used, creating a mount service is not required, and the Zadara Storage can be mounted via /etc/fstab. The following example is an example of the Zadara Storage being installed via fstab. As with the mount service, DO NOT modify the options!

172.32.160.200:/export/emsaws/ /mnt/nfs nfs nfsvers=4.0,soft,retrans=2,actimeo=1,timeo=150,_netdev

Figure 11 - Fstab example

• Create a new mount point for the storage: sudo mkdir /mnt/nfs • Mount the shared storage using sudo mount /mnt/nfs

4.2.4 Linux Kernel Changes The Linux kernel by default can keep the tcp_keepalives for up to twenty minutes. This can have a delayed affect on EMS fail-over. To shorten this time, the Linux kernel property tcp_retries2 can be modified. To modify this property, do the following:

• Edit /etc/sysctl.conf, and add the following value: net.ipv4.tcp_retries2 = 4 • Following is an example of /etc/sysctl.conf with the change:

sysctl settings are defined through files in # /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/. # # Vendors settings live in /usr/lib/sysctl.d/.


# To override a whole file, create a new file with the same in # /etc/sysctl.d/ and put new settings there. To override # only specific settings, add a file with a lexically later # name in /etc/sysctl.d/ and put new settings there. # # For more information, see sysctl.conf(5) and sysctl.d(5). # Update to shorten tcp keep alives net.ipv4.tcp_retries2 = 4

Figure 12 - Sysctl change

• Reboot the AWS EC2 instance. After the reboot, check: o The shared file system has mounted o That tcp_retries2 =4. Use cat /proc/sys/net/ipv4/tcp_retries2 to verify

Note: Resolve any issues before continuing!


5 Enterprise Message Service Installation and Configuration

This section will outline the installation and configuration of EMS on the AWS EC2 instances.

5.1 EMS Installation

EMS 8.5 was used for all testing, and is recommended. Either the Enterprise or Community edition of EMS may be used. Install EMS on all of the of the AWS EC2 instances. Nothing specific or custom is required to the base configuration of EMS. Follow the EMS 8.5 Installation guide for installing EMS.

Once EMS is installed, use the following to configure EMS on all three EC2s.

• EMS 8.5 may have been installed as the root user. If EMS was installed as the root user, change the ownership to the username created by AWS, and used to login into the AWS EC2 instances.

• Use sudo chown –R <username>:<username> /opt/tibco to change the ownership.

On the EMS server EC2s:

• Create /opt/tibco/ems/8.5/bin/logs directory

On one of the EMS server EC2s:

• Create the configuration directory on the shared storage:

mkdir –parents /mnt/nfs/tibco/cfgmgmt/ems/data/datastore

• Copy /opt/tibco/ems/8.5/samples/config/*.conf /mnt/nfs/tibco/cfgmgmt/ems/data/

5.2 EMS Configuration

There are specific configuration changes which must be made to provide better write performance and reliability of EMS F/T. This section will discuss these changes. See the EMS User Guide for additional information on setting or the use of, any properties discussed. Note: EMS 8.5 no longer pre-configures the main tibemsd.conf configuration file as part of the installation. All configuration modifications discussed must be completed. Copy tibemsd.conf to tibemsd1.conf and tibemsd2.conf.

5.2.1 Stores.conf In stores.conf, modify/add the following:

The file_minimum=xxGB can be added to each data store. By adding this property, EMS will pre-allocate the space on the shared storage the data store. Note: This is an optional setting, and is not required, but will enhance the message write throughput on disk. The minimum should be at least 1GB. Expect the initial startup of EMS to take longer as it creates and allocates the space for the store file.


The file_crc=true should be added, if not already present. This enables EMS to check for data integrity of the data store. This is the default, but ensures it is set.

The following is an example of stores.conf with the changes.

[$sys.failsafe] type=file file=sync-msgs.db mode=sync file_minimum=2GB file_crc=true [sync2] type=file file=sync2-msgs.db mode=sync file_crc=true

Figure 13 - Stores.conf example

5.2.2 Factories.conf The EMS client reconnect properties must be set to enable the EMS client to reconnect to the EMS server in the event of an EMS server failure in an F/T configuration. The reconnect properties can be defined in a number of ways, including in the Java/C code, TIBCO application’s configuration file, and/or through the connection factory when they are used. The default values may be too low in AWS to reliably allow the EMS client to reconnect to the EMS server after a fail-over, especially with network or system failure. It is recommended to set the reconnect_attempt_count to 100, and the reconnect_attempt_delay to 5000. With these values, the EMS client will attempt to reconnect 100 times, every 5 seconds.

The following example shows the values for the FTConnectionFactory in factories.conf. Note: In the following example for the url, <server1> is 10.150.0.7 and the <port1> is 7224, and <server2> is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.

[FTConnectionFactory] type = generic url = tcp://10.150.0.5:7224,tcp://10.150.0.3:7224 reconnect_attempt_count=100 reconnect_attempt_delay=5000

Figure 14 - Factories Configuration Example

5.2.3 Tibemsd.conf The tibemsd.conf for both EMS Servers needs to be updated for multiple properties.

These include:

• Location of all configuration files – The location must be on the shared storage device.

######################################################################## # Configuration files. ########################################################################


users = "/mnt/nfs/tibco/cfgmgmt/ems/data/users.conf" groups = "/mnt/nfs/tibco/cfgmgmt/ems/data/groups.conf" topics = "/mnt/nfs/tibco/cfgmgmt/ems/data/topics.conf" queues = "/mnt/nfs/tibco/cfgmgmt/ems/data/queues.conf" acl_list = "/mnt/nfs/tibco/cfgmgmt/ems/data/acl.conf" factories = "/mnt/nfs/tibco/cfgmgmt/ems/data/factories.conf" routes = "/mnt/nfs/tibco/cfgmgmt/ems/data/routes.conf" bridges = "/mnt/nfs/tibco/cfgmgmt/ems/data/bridges.conf" transports = "/mnt/nfs/tibco/cfgmgmt/ems/data/transports.conf" tibrvcm = "/mnt/nfs/tibco/cfgmgmt/ems/data/tibrvcm.conf" durables = "/mnt/nfs/tibco/cfgmgmt/ems/data/durables.conf" channels = "/mnt/nfs/tibco/cfgmgmt/ems/data/channels.conf" stores = "/mnt/nfs/tibco/cfgmgmt/ems/data/stores.conf" ######################################################################## # Persistent Storage.

# # store: directory to store persistent messages. ######################################################################## store = "/mnt/nfs/tibco/cfgmgmt/ems/data/datastore"

• Log File location – The location must be on the local disk if the AWS EC2 instance. The following example locates in the /opt/tibco/ems/8.5/bin/logs directory.

logfile = "/opt/tibco/ems/8.5/bin/logs/tibemsd1.log"

Server and Client Heartbeat and timeout values – These properties determine how long the client/server listen for the heartbeat from the the client/server, before disconnecting. The values shown below have been tested, and work well on AWS with the Zadara NFS storage. The connection_timeout can be increased if necessary, but the heartbeat must remain at 5.

server_heartbeat_client = 5 server_timeout_client_connection = 30 client_heartbeat_server = 5 client_timeout_server_connection = 30

Enabling exiting disk error property – New property since EMS version 8.4. This property defines to EMS to exit when there is a disk error reading/writing to a device. It must be enabled. This property will help prevent “Dual Active Server” conditions, sometimes seen in networked storage devices.

always_exit_on_disk_error = enable

FT properties – Normal properties for the defining the peer EMS server instance, heartbeat between instances, and etc. must be set. Ensure the ft_reconnect_timeout is set to the value shown.

ft_reconnect_timeout = 120 Increase the Maximum Message Memory to 1024 or greater, depending on how much RAM is allocated to the AWS EC2 instance.


max_msg_memory = 1024

Define a value for destination_backlog_swapout. This will help limit excessive reads to the shared disk. A minimum of 10000 is recommended. If the queues, will persistent a larger number of messages, increase the size.

destination_backlog_swapout = 10000

5.2.4 Starting the EMS Instances Once the configuration files are updated, EMS can be started. The –forceStart parameter is optional. Start both instances, taking note of which instance is the active EMS instance. Note: It will take ~ 5 minutes for the first EMS instance to become active due to the creation of the data stores on the shared storage. Note: Leave the both windows to the EMS server instances open. This will be needed for the testing.

Figure 15 - Starting an EMS Instance Example


6 Testing EMS Fault Tolerance on AWS with Zadara Storage

Once EMS has been started on both of the AWS EC2 instances, the failover testing can be performed. This section will outline several test cases, including EMS Server process failure, machine failure, and network failure. Tests are performed using queues with persistence set. This guarantees that the shared file system will be accessed during the tests.

6.1 EMS Client Application Setup

The third AWS EC2 instance is used to run the test application. EMS is shipped with sample Java applications which can be used for the testing. The tibjmsMsgProducerPerf utility should be used for the testing. All sample Java applications are located in /opt/tibco/ems/8.5/samples/java. Use the following to setup the environment:

• Ensure the version Java 1.8 development environment is installed. • Install EMS version 8.5 on the third AWS EC2 instance following the EMS installation

procedures. • After the installation of EMS is completed:

o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o javac *.java – This should compile all Java apps in the directory

• Ensure that at least one of the EMS server instances is running (both should be running) • Use the TIBCO EMS Administration Tool to create the EMS Queue sync utilizing the

$sys.failsafe data store. This is required for testing with a synchronous data store: o cd to $TIBCO_HOME/ems/8.5/bin o ./tibemsadmin tcp://<server>:port – of the active EMS instance

Note: In the following examples, <server> is the IP address is 10.150.0.5 and the <port> is 7224. Substitute with the appropriate values for the environment.


Figure 16 - Create the Sync Queue

6.2 Performing the EMS Fault Tolerant Test Cases

Three different tests should be performed: 1. EMS Process failure – Active EMS instance is stopped 2. Network failure – Network failure between the Active EMS Server machine, and the

Zadara VPSA 3. System failure – Accidental restart of the AWS EC2 running the Active EMS server

instance

This section will outline how to run these three tests, and what the expected results should be. Note: All test cases must be run from the third virtual machine where the Java sample app was compiled.

6.2.1 EMS Process Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during an EMS server process failover. Two EMS server instances will be running in a F/T configuration, while messages are being sent. The active EMS instance will be stopped, the stand-by EMS instance should take over, and continue processing messages until the EMS Java completes publishing messages. Note: In the following examples, <server1> the IP address is 10.150.0.5 and the <port1> is 7224, and <server2> the IP address is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.


6.2.1.1 Running the Process Failure Test Three ssh terminal sessions are needed for this test; one for server1, one for server2, and one for the client Start EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance. From the client, start the Java application

o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –

factory FTConnectionFactory –delivery PERSISTENT –connections 10 –threads 8 –count 20000 –size 1024 –queue sync

Figure 17 - Running the tibjmsMsgProducerPerf Application

Immediately kill/stop the EMS instance on server1, with cntrl-C The standby EMS instance on server2 will become active, and recover all messages. It should be possible to stop and start the EMS instances a few times while the Java test application is running. The number of recovered messages will increase.


Figure 18 - The Standby EMS Instance becoming active on Server2

After the Java application completes, run tibemsadmin tcp://10.150.0.3:7224 (or from server1, if it is active), to verify that there are 20000 messages in the sync queue. Restart the EMS instance on server1, and stop the EMS instance on server2. EMS on server1 should become active, and recover all 20K messages with no errors. Use tibemsadmin and purge the sync queue in preparation for the next test.


Figure 19 - Purge the Sync queue for tibemsadmin

• Stop and restart EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance.

6.2.1.2 Expected Results The Java test application should complete, with a slight pause during failover but should resume sending messages once the failover is complete. No messages should be lost. There should not be more than 20K messages, and there should never be less than 20K. Depending on the number of messages that must be recovered, the fail-over should be very short, possibly less than 1 second. Complete fail-over from one server to the other with the 20K messages should be under 10 seconds.

6.2.2 Network Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during a network failure between the active EMS server instance, and the Zadara VPSA. Two EMS server instances will be running in a F/T configuration, while messages are being sent. The TCP/NFS port will be blocked between the active EMS instance and the Zadara VPSA via iptables. The active EMS instance should get a write error, and exit, allowing the stand-by EMS instance to gain the locks on the EMS data stores, and take over. The EMS Java application should continue processing messages until it completes. Note: In the following examples, <server1> IP address is 10.150.0.5 and the <port1> is 7224, and <server2> IP address 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.


6.2.2.1 Running the Network Failure Test Four ssh terminal sessions are needed for this test; two for server1, one for server2, and one for the client A script will be needed to block the NFS port (2049) on server1 while the Java app is publishing messages. The following figure shows the drop_nfs.sh script. Cut and past the following to create the script. The script must be created in the second ssh terminal window on server1.

# # Script to get the current iptables definitions, drop NFS port, then restore the original table defi #nitions. # echo " Saving existing IP table definitions" echo "" sudo iptables-save >iptables_save # # Drop the NFS port 2049 # date echo " Dropping NFS 2049 port" echo "" sudo iptables -A INPUT -p TCP --dport 2049 -j DROP sudo iptables -A OUTPUT -p TCP --dport 2049 -j DROP # # Sleep for 3 minutes # echo " Sleeping for 3 minutes..." echo "" sleep 3m # # Restore original IP table definitions" # echo " Restoring original IP table definitions" echo "" sudo iptables -F sudo iptables-restore <iptables_save # date echo "Done."

Figure 20 - drop_nfs.sh script

From the client, start the Java application o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –


From the second ssh terminal window on server1, run sudo drop_nfs.sh


Figure 21 - Running drop_nfs.sh script

• The active EMS instance on server1 should terminate with a disk write error:

Figure 22 - Disk Write Error on Server1

The standby EMS instance on server2 should determine EMS on server1 is no longer producing a heartbeat, will attempt to become active. This should only take a few seconds for server2 to become active, depending on the amount of data to be recovered. There may be other warnings, depending on how long it takes for server2 to obtain the locks.


Figure 23 - Server2 becoming Active after network failure

After the Java application completes, run tibemsadmin tcp://10.150.0.3:7224 (or on server1, if it is active), to verify that there is a minimum of 20000 messages in the sync queue. Restart the EMS instance on server1, and stop the EMS instance on server2. EMS on server1 should become active, and recover all 20K messages with no errors. While still in tibemsadmin, purge the sync queue in preparation for the next test. Stop and restart EMS on server1 and server2 in the foreground. EMS on server1 should be the active EMS instance.

6.2.2.2 Expected Results The Java test application should complete, pausing during the failover, but should resume sending messages once the failover is complete. No messages should be lost. There can be more than 20K messages, depending on the number of connections/threads, but there should never be less than 20K. Depending on the number of messages that must be recovered, the fail-over should only take a few seconds. Note: It was observed that on occasion, the NFS mount on Zadara does not reconnect after the default iptables configuration was restored, and a reboot of the server1 EC2 instance was required. However, server2 still recovers all messages, and continues. No messages were lost. Note: In some circumstances, the following error may occur with the network failure, if a file_minimum for a data store has been set.


SEVERE ERROR: Error occurred while processing Message record id=xxxxxx (get record) - Unable to reconstruct message: 74 - IO failed

This can sometimes occur when there is a file_minimum set, and the file is only partially filled. This is not an error, and can be ignored. All messages will be recovered. If this is seen, restart the tibemsd server with the –forceStart option as shown in Figure 15.

6.2.3 System Failure Test This test verifies that an EMS client continues to function correctly, with no message loss during a system failure on the AWS EC2 instance running the active EMS server instance. This is not a normal occurrence. However, it is possible to accidentally restart the virtual machine from the AWS console. Two EMS server instances will be running in a F/T configuration, while messages are being sent. From the AWS console, the EC2 instance where the active EMS instance is running will be restarted. The stand-by EMS instance should be able to gain the locks on the EMS data stores, and take over. The EMS Java application should continue processing messages until it completes. Note: In the following examples, <server1> IP address is 10.150.0.5 and the <port1> is 7224, and <server2> IP address is 10.150.0.3 and the <port2> is 7224. Substitute with the appropriate values for the environment.

6.2.3.1 Running the System Failure Test Three ssh terminal sessions are needed for this test; one from each of the virtual machines. The AWS console must also be available, and be on the AWS/EC2 Instances dashboard page. Note: In the examples below, Test Instance is server1. Click on the instance, Actions

Figure 24 - AWS Compute Engine/EC2 Instances page

• From the client, start the Java application: o cd to $TIBCO_HOME/ems/8.5/samples/java o . ./setup.sh o java tibjmsMsgProducerPerf –server tcp://10.150.0.5:7224, tcp://10.150.0.3:7224 –



• From the AWS dashboard, click on the server1 instance, then click on the Actions tab, then on Instance Status, and finally on Reboot. Verify to reboot the server. This will reboot the server1 AWS EC2 instance.

• The ssh terminal window to the server1 AWS EC2 instance should immediately terminate, and the stand-by EMS instance should recover all messages, and become active within a few seconds, depending on the number of messages to be recovered. There may be other warnings, depending on how long it takes for server2 to obtain the locks.

Figure 25 – Standby EMS instance recovering from system failure of primary

• After the Java application completes, run tibemsadmin tcp://server2:7224 (or to the active EMS instance), to verify that there is a minimum of 20000 messages in the sync queue.

• Restart the EMS instance on the restarted AWS EC2 instance, and stop the currently active EMS instance on the second AWS EC2 instance. EMS should become active on the restarted instance, and recover all 20K messages with no errors.

• Use tibemsadmin to verify, then purge the sync queue. • Stop EMS on both AWS EC2 instances. • This concludes the tests, so all processes, terminal windows, and AWS EC2 instances can

be stopped.


6.2.3.2 Expected Results The Java test application should complete, pausing during failover, but should resume sending messages once the failover is complete. No messages should be lost. There can be more than 20K messages, depending on the number of connections/threads, but there should never be less than 20K messages. Depending on the number of messages that must be recovered, the fail-over should take less than a minute. There may also be additional warning regarding the clients trying to reconnect, depending on how many messages that must be recovered. Note: In some circumstances, the following error may occur with an abrupt system shutdown, if a file_minimum for a data store has been set. SEVERE ERROR: Error occurred while processing Message record id=xxxxxx (get record) - Unable to reconstruct message: 74 - IO failed

This can sometimes occur when there is a file_minimum set, and the file is only partially filled. This is not an error, and can be ignored. All messages will be recovered. If this is seen, restart the tibemsd server with the –forceStart option as shown in Figure 15.

Documents

Configuring a TIBCO Enterprise Message Service Storage · 2019-10-11 · 4.2.3 NFS Mount via Fstab ... • General outline for setting up the Zadara VPSA for the shared file system