IBM Cloud Orchestrator Version 2.4: Capacity Planning

IBM® Cloud and Smarter Infrastructure Software

IBM Cloud Orchestrator Version 2.4: Capacity Planning, Performance, and Management Guide

Document version 2.4.0

IBM Cloud Orchestrator Performance Team

© Copyright International Business Machines Corporation 2015. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM Cloud Orchestrator Version 2.4:

Capacity Planning, Performance, and Management Guide

iii

CONTENTS

Contents .............................................................................................................................. iii

List of Figures ...................................................................................................................... vi

Author List ......................................................................................................................... viii

Revision History .................................................................................................................. ix

1 Introduction ............................................................................................................ 10

2 IBM Cloud Orchestrator 2.4 Overview .................................................................. 11

2.1 Functional Overview.................................................................................. 11

2.2 Architectural Overview .............................................................................. 13

3 Performance Overview ......................................................................................... 15

3.1 Sample Benchmark Environment ............................................................. 15

3.2 Key Performance Indicators ..................................................................... 16

3.2.1 Concurrent User Performance ............................................................... 17

3.2.2 Provisioning Performance ...................................................................... 19

4 Performance Benchmark Approaches .................................................................. 21

4.1 Monitoring and Analysis Tools .................................................................. 21

4.2 Infrastructure Benchmark Tools ................................................................ 22

4.3 Cloud Benchmarks .................................................................................... 22

5 Capacity Planning Recommendations .................................................................. 23

5.1 Cloud Capacity Planning Spreadsheet ..................................................... 23

5.2 IBM Cloud Orchestrator Management Server Capacity Planning ........... 24

5.3 Provisioned Virtual Machines Capacity Planning ..................................... 25

6 Cloud Configuration Recommendations ............................................................... 29

iv

6.1 OpenStack Keystone Worker Support ...................................................... 29

6.2 Disabling the IWD Service ........................................................................ 29

6.3 IBM Workload Deployer Configuration ..................................................... 30

6.4 Virtual Machine IO Scheduler Configuration ............................................ 30

6.5 Advanced Configuration and Power Interface Management ................... 30

6.6 Java Virtual Machine Heap Configuration ................................................ 31

6.7 Database Configuration ............................................................................ 32

6.8 Database Management ............................................................................. 32

6.8.1 DBMS Versions ...................................................................................... 32

6.8.2 Automatic Maintenance .......................................................................... 32

6.8.3 Operating System Configuration (Linux) ................................................ 33

6.9 Database Hygiene Overview .................................................................... 33

6.9.1 Database Backup Management ............................................................. 33

6.9.2 Database Statistics Management ........................................................... 36

6.9.3 Database Reorganization ....................................................................... 36

7 Summary Cookbook.............................................................................................. 38

7.1 Base Installation Recommendations ........................................................ 38

7.2 Post Installation Recommendations ......................................................... 39

7.3 High Scale Recommendations ................................................................. 39

Appendix A: IBM Cloud Orchestrator Monitoring Options ................................................ 40

A.1 OpenStack Monitoring .............................................................................. 40

A.2 IBM Cloud Orchestrator Monitoring .......................................................... 41

A.3 Infrastructure Monitoring ........................................................................... 42

Appendix B: OpenStack Keystone Monitoring ................................................................. 45



v

B.1 PvRequestFilter ......................................................................................... 45

B.2 Enabling PvRequestFilter ......................................................................... 45

References ........................................................................................................................ 47

vi

LIST OF FIGURES

Figure 1: Revision History ................................................................................................................ ix

Figure 2: IBM Cloud Orchestrator Benefits Estimator .................................................................... 11

Figure 3: ICO Cloud Marketplace View .......................................................................................... 12

Figure 4: ICO Architecture Reference Topology ............................................................................ 13

Figure 5: ICO Sample Benchmark Environment ............................................................................ 15

Figure 6: Benchmark Data Model Population ................................................................................ 18

Figure 7: Load Driving (User) Scenarios ........................................................................................ 19

Figure 8: Provisioning Performance in a Closed System ............................................................... 20

Figure 9: Monitoring and Analysis Tools ........................................................................................ 21

Figure 10: Infrastructure Benchmark Tools .................................................................................... 22

Figure 11: ICO Management Server Capacity Planning ................................................................ 24

Figure 12: Capacity Planning Tool: Inquiry Form ........................................................................... 25

Figure 13: Capacity Planning Tool: User Demographic Information .............................................. 26

Figure 14: Capacity Planning Tool: Systems and Storage ............................................................. 26

Figure 15: Capacity Planning Tool: System and Workload Options ............................................... 27

Figure 16: Capacity Planning Tool: Virtual Machine Requirements ............................................... 27

Figure 17: Planning Tool: Confirmation Screen ............................................................................. 27

Figure 18: Planning Tool: System Summary .................................................................................. 28

Figure 19: Keystone Worker Configuration .................................................................................... 29

Figure 20: IWD Configuration ........................................................................................................ 30

Figure 21: Modifying the IO Scheduler........................................................................................... 30

Figure 22: Java Virtual Machine Heap Change Sets ..................................................................... 31

Figure 23: Database Configuration Change Sets .......................................................................... 32

Figure 24: DBMS Versions ............................................................................................................ 32

Figure 25: Database Automatic Maintenance Configuration .......................................................... 33

Figure 26: Database Backup with Compression Command ........................................................... 33

Figure 27: Database Offline Backup Restore ................................................................................. 33

Figure 28: Database Online Backup Schedule .............................................................................. 34

Figure 29: Database Incremental Backup Enablement .................................................................. 34

Figure 30: Database Online Backup Manual Restore .................................................................... 34

Figure 31: Database Online Backup Automatic Restore ................................................................ 34

Figure 32: Database Log Archiving to Disk .................................................................................... 34

Figure 33: Database Log Archiving to TSM ................................................................................... 35



vii

Figure 34: Database Roll Forward Recovery: Sample A................................................................ 35

Figure 35: Database Roll Forward Recovery: Sample B................................................................ 35

Figure 36: Database Backup Cleanup Command .......................................................................... 35

Figure 37: Database Backup Automatic Cleanup Configuration .................................................... 35

Figure 38: Database Statistics Collection Command ..................................................................... 36

Figure 39: Database Statistics Collection Table Iterator ................................................................ 36

Figure 40: Database Reorganization Commands .......................................................................... 36

Figure 41: Database Reorganization Table Iterator ....................................................................... 37

Figure 42: Base Installation Recommendations ............................................................................. 38

Figure 43: Post Installation Recommendations .............................................................................. 39

Figure 44: High Scale Recommendations...................................................................................... 39

Figure 45: OpenStack Ceilometer Metrics ..................................................................................... 40

Figure 46: OpenStack Ceilometer Core Metrics ............................................................................ 41

Figure 47: Infrastructure Core Metrics ........................................................................................... 44

Figure 48: Keystone Monitoring PvRequestFilter Format .............................................................. 45

Figure 49: Keystone Monitoring PvRequestFilter Sample Output .................................................. 45

Figure 50: Keystone Monitoring Log Messages Example .............................................................. 46

Figure 51: Keystone Monitoring Statistics Example ....................................................................... 46

viii

AUTHOR LIST

This paper is the team effort of a number of cloud performance specialists comprising the IBM Cloud Orchestrator performance team. Additional recognition goes out to the entire IBM Cloud Orchestrator and OpenStack development teams.

Mark Leitch (primary contact for this paper) IBM Toronto Laboratory Nate Rockwell IBM USA Tiarnán Ó Corráin IBM Ireland

Amadeus Podvratnik Marc Schunk IBM Boeblingen Laboratory

Alessandro Chiantera Andrea Tortosa Massimo Marra Michele Licursi Paolo Cavazza Sandro Piccinini IBM Rome Laboratory



ix

REVISION HISTORY

Date Version Revised By Comments

March 12th, 2015 Draft MDL Initial version for review.

April 1st, 2015 2.4.0 MDL Incorporated review comments.

Figure 1: Revision History

10

1 Introduction Capacity planning involves the specification of the various components of an installation to meet customer requirements, often with growth or timeline considerations. A key aspect of capacity planning for cloud, or virtualized, environments is the specification of sufficient physical resources to provide the illusion of infinite resources in an environment that may be characterized by highly variable demand. This document will provide an overview of capacity planning for the IBM Cloud Orchestrator (ICO) Version 2.4. In addition, it will offer management best practices to achieve a well performing installation that demonstrates service stability.

ICO Version 2.4 offers end to end management of service offerings across a number of cloud technology offerings including VMware, Kernel-based Virtual Machine (KVM), IBM PowerVM, and IBM System z. A key implementation aspect is integration with OpenStack, the de facto leading open virtualization technology. OpenStack offers the ability to control compute, storage, and network resources through an open, community based architecture.

In this document we will provide an ICO 2.4 overview, including functionality, architecture, and performance. We will then offer the capacity planning recommendations, including considerations for hardware configuration, software configuration, and cloud maintenance best practices. A summary “cookbook” is provided to manage installation and configuration for specific instances of ICO.

Note: This document is considered a work in progress. Capacity planning recommendations will be refined and updated as new ICO releases are available. While the paper in general is considered suitable for all ICO Version 2.4 releases, it is best oriented towards ICO Version 2.4.0.1. In addition, a number of references are provided in the References section. These papers are highly recommended for readers who want detailed knowledge of ICO server configuration, architecture, and capacity planning.

Note: Some artifacts are distributed with this paper. The distributions are in zip format. However Adobe protects against files with a “zip” suffix. As a result, the file suffix is set to “zap” per distribution. To use these artifacts, simply rename the distribution to “zip” and

process as usual.



11

2 IBM Cloud Orchestrator 2.4 Overview An overview of ICO Version 2.4 will be provided from the following perspectives:

1. Functional

2. Architectural

2.1 Functional Overview

The basic functional capability of ICO involves the management of cloud computing resources for dynamic data centers. In a nutshell, ICO offers infrastructure, platform, and orchestration services that make it possible to lower the cost of service delivery (both in terms of time and skill) while delivering higher degrees of standardization and automation. In order to determine the benefits of deploying ICO in business terms, the IBM Cloud Orchestrator Benefits Estimator (URL) is available. A screenshot of the estimator is provided below.

Figure 2: IBM Cloud Orchestrator Benefits Estimator

https://roianalyst.alinean.com/ibm_bva/AutoLogin.do?d=804434328733361758

12

A more detailed cloud marketplace view of the ICO solution follows.

Figure 3: ICO Cloud Marketplace View

The core functional capabilities of ICO include the following.

Workflow Orchestration. The Business Process Manager (BPM) component offers a standard library as well as a graphical editor for workflow orchestration. Overall, this provides a powerful mechanism for complex and custom business process in the cloud context.

Pattern Management. The IBM Workload Deployer (IWD) offers sophisticated pattern support for deploying multi node applications that may consist of complex middleware. Once again, graphical editor support for pattern management is provided.

Service Management. Service management options are available in the ICO Enterprise edition. It provides a set of management utilities to further facilitate business process management.

Not shown in the diagram is a Scalable Web Infrastructure to facilitate cloud self service offerings. For more information please consult the ICO knowledge center (URL). In addition, the ICO resource center is available (URL).

http://www-01.ibm.com/support/knowledgecenter/SS4KMC_2.4.0/com.ibm.sco.doc_2.4/t_working_with_ss.html

http://www-03.ibm.com/software/products/en/ibm-cloud-orchestrator



13

2.2 Architectural Overview

The following diagram shows the reference deployment topology for ICO. A description of the reference topology follows.

Figure 4: ICO Architecture Reference Topology

The core of the reference topology is based on a core set of virtual machines:

Deployment Server. The installation or deployment service for the ICO instance(s).

Central Server 1. This server hosts the DB2 Database Management System (DBMS). The performance of the DBMS is critical to the overall solution and is dealt with extensively in Section 6.

Central Server 2. This is essentially the “super node” for the ICO instance. This server hosts OpenStack Keystone, providing identity, token, catalog, and policy services. It also hosts Business Process Manager (BPM), the primary mechanism for driving business process workflows. The most critical aspect of this server is managing Keystone and BPM, as described in Section 6.

Central Server 3. This server hosts the IBM Workload Deployer pattern engine. Performance configuration of this component is described in Section 6.

Associated with these core server virtual machines are a number of region servers. Region servers may represent a specific cluster or geographic zone of cloud compute nodes. Sample compute nodes are shown for VMware, KVM, and PowerVM, with associated communication paths. For example, for VMware the VMware community driver is used to drive the operation of the VMware cluster. For KVM, the OpenStack control node is used to coordinate the KVM instance.

Given this is a virtual implementation, some considerations should be kept in mind:

In general, it is more difficult to manage performance in a virtual environment due to the additional hypervisor management overhead and system configuration.

Device parallelism via dedicated storage arrays/LUNs is preferred. Sample approaches, from most impactful to least impactful, are provided below.

o Separate data stores for “managed from” and “managed to” environments.

14

o Spread data stores across several physical disks to maximize storage capability.

o Separate data stores for image templates and provisioned images.

o Employ the “deadline” or “noop” scheduler algorithm for management server

and provisioned VMs (see Section 6.4).

o Optimize base storage capability (i.e. SSD with “VMDirectPath” enablement for VMware). Servers where this may be critical, due to their dependency on disk IO capabilities, are Central Server 1 and the VMware vCenter instances.

Network optimization, for example 10GbE adoption. In addition, segment customer networks to an acceptable level to reduce address lookup impact.



15

3 Performance Overview There are two distinct aspects of cloud performance:

1. Performance of the ICO management server itself. This is the primary focus of this section.

2. Performance of the provisioned server instances. This is more of a capacity planning statement, and is covered in Section 5.3.

We will provide a general overview of the Key Performance Indicators (KPIs) for the ICO management server. The following sections will describe the general benchmark environment, and the associated KPIs.

3.1 Sample Benchmark Environment

The following figure shows a sample configuration that has been used for ICO benchmarks.

Figure 5: ICO Sample Benchmark Environment

The environment is characterized by the following features, broken down in terms of the ICO management server (aka “managed from”) and the associated cloud (aka “managed to”).

Managed from:

o Server configuration:

16

4/5 HS22V Blades with 2 x 4 cores Intel Xeon x5570 2.93 GHz.

8 physical cores per blade, 16 logical cores when hyper-threading is enabled.

72 GB RAM per blade.

2 x Redundant 10G Ethernet Networking (Janice HSSM).

2 x Redundant 8G FC Network (Qlogic FC SM).

o Storage configuration: 1 x DS3400 with 4 Exp with 12 Disk 600 GB SAS 10K each (48 x 600 GB = 28.8 TB raw).

Managed to:

o Server configuration:

Tens of HS22V Blades with 2 x 6 cores Intel Xeon x5670 2.93 GHz.

12 physical cores per blade, 24 logical cores when hyper-threading is enabled.

72 GB RAM per blade.

2 x Redundant 10G Ethernet Networking (Janice HSSM).

2 x Redundant 8G FC Network (Qlogic FC SM).

o Storage configuration: 1 x Storwize v7000 with 3 Exp with 12 Disks 2 TB NL-SAS 7.2k each (36 x 2 TB = 72 TB raw).

o Storage access has been configured to use the multi-path access granted by Storwize. In particular, VMware ESXi servers have been configured to use all of the 8 active paths to access LUNs using a round robin policy.

3.2 Key Performance Indicators

The following Key Performance Indicators are managed for ICO through a set of comprehensive benchmarks.

1. Concurrent User Performance, comprising:

a. Average response time for ICO pages related to administrative tasks.

b. Average response time for ICO pages related to end user tasks.

2. Provisioning throughput, comprising:

a. Provisioning throughput for a vSys with a single part.

b. Average service time for provisioned VMs.

3. LAMP (Linux, Apache, MySQL, Python) stack performance, comprising:

a. vApp deployment time.

b. vApp stop time.

c. vApp deletion time.

4. Bulk windows stack performance comprising vSys with multiple parts (15 VMs) provisioning time.

A key aspect of the benchmarks is they are run with associated background workloads and for a long duration (e.g. weeks or months). The rationale behind this is very simple: to run benchmarks that closely emulate the customer experience and will drive “real world”



17

results (versus overly optimistic lab based results). We will describe the concurrent user and provisioning throughput KPIs in more detail.

3.2.1 Concurrent User Performance

ICO User Interface performance is established through concurrent user benchmark tests. In order to understand the applicability of such a benchmark, it is important to understand what is meant by a concurrent user. Consider:

P = total population for an instance of ICO (including cloud administrators, end users, etc.).

C = the concurrent user population for an instance of ICO. Concurrent users are considered to be the set of users within the overall population P that are actively managing the cloud environment at a point in time (e.g. administrator operations in the User Interface, provisioning operations, etc.).

In general, P is a much larger value than C (i.e. P >> C). For example, it is not unrealistic that a total population of 200 users may have a concurrent user population of 40 users (i.e. 20%).

For the concurrent user workload driven for ICO, there are three sets of criteria that drive the benchmark:

1. Load driving parameters.

2. Data population.

3. Load driving (user) scenarios.

Load Driving Parameters

The following load driving parameters apply.

1. User transaction rate control. The frequency that simulated users drive actions against the back end is managed via loop control functions. Closed loop simulation approaches are used where a new user will enter the system only when a previous user completes. Through the closed loop system, steady state operations under load may be driven.

2. Think times. Think times are the “pause” between user operations, meant to simulate the behavior of a human user. The think time interval used is [100%,300%] (meaning, the replay via the load driver is up to three times the rate of the scenario recording rate).

3. Bandwidth throttling. In order to simulate low speed or high latency lines, bandwidth throttling is employed for some client workloads. The throttle is set to a value that represents a moderate speed ADSL connection (cable/DSL simulation setting of 1.5 Mbps download, 384 Kbps upload).

Data Population Parameters

The benchmark is run against a data model that represents a large scale customer environment. The following table shows a sample configuration where the system is populated with data to represent a large number of users, active Virtual System instances, and active Virtual Machines existing prior to ICO installation. Through this approach, the workload for managing the solution is representative of some customer environments.

18

Benchmark Parameter Value

Cloud Administrators 1

Cloud Domains 11

Tenants 200

Users 1000

Hypervisor Types 2

(KVM, VMware)

Cloud Groups 1

Environment Profile 1

Image Templates 40

(20Linux, 20 Windows)

vSys Patterns 20 + 1

(20 Linux vSys patterns, 1 bulk Windows pattern)

vApp Patterns 1

(LAMP vApp for VMware domain)

Flavors 5

(1 flavor for RHEL, 3 flavors for Windows, 1 flavor for vApp)

Active vSys instances 20

(1 per Linux vSys Pattern)

Standalone (Unmanaged) VMs 400

(10 per image template 200 Linux, 200 Windows)

Figure 6: Benchmark Data Model Population

Load Driving (User) Scenarios

The concurrent user population (i.e. C) is broken down into the following user profile distribution and scenarios.

User Profile Scenario per User

Number of Users: 20 (50% overall)

User Type: End User

Task Type: VM Provisioning

Activity: vSys with single part (Linux) provisioning through Self-Service Catalog (SSC) offering on VMware.

1. Login. 2. Provision vSys single part using SSC

offering. 3. Wait until available. 4. Go to the vSys instance details page. 5. Delete vSys using SSC offering. 6. Wait until deletion complete. 7. Logout. 8. Enter next cycle according to arrival rate.


User Type: End User

Task Type: User Management

Activity: End user operations through Self-Service Catalog (SSC) offering.

1. Login. 2. Submit SSC offering "Create User in VM",

selecting one of the VMs belonging to one of the pre-populated vSys.

3. Wait until done. 4. Submit SSC offering "Delete User in VM",

selecting the same VM. 5. Wait until done. 6. Logout. 7. Enter next cycle according to arrival rate.



19


User Type: Administrator

Task Type: Monitoring

Activity: Administrative operations through the IBM Workload Deployer user interface.

1. Login. 2. List hypervisors. 3. Select a hypervisor. 4. List VMs in hypervisor. 5. Show all instances. 6. Go to "My Requests". 7. Sort the requests by status. 8. View the trace log. 9. Logout.

Number of Users: 1 (2.5% overall)

User Type: End User

Task Type: Provisioning

Activity: vApp (LAMP) provisioning through IBM Workload Deployer user interface on VMware.

1. Login. 2. Provision vApp using the IWD UI. 3. Wait until available. 4. Stop vApp using the IWD UI. 5. Wait until done. 6. Delete vApp using the IWD UI. 7. Wait until deletion complete. 8. Logout. 9. Enter next cycle according to arrival rate.

Number of Users: 1 (2.5% overall)

User Type: End User

Task Type: Provisioning

Activity: vSys with multiple parts (bulk Windows) provisioning through Self-Service Catalog (SSC) offering on VMware.

1. Login. 2. Provision vSys bulk Windows using SSC

offering. 3. Wait until available. 4. Go to vSys instance details page. 5. Delete vSys bulk Windows using SSC

offering. 6. Wait until deleted. 7. Logout. 8. Enter next cycle according to arrival rate.

Figure 7: Load Driving (User) Scenarios

In overall terms, 55% of the load driving activities are driving Virtual Machine provisioning scenarios. The remaining 45% of scenarios are general administration and management tasks. For the active workload, the user operations meet the following response time thresholds.

Administrative page response times: 90% of pages < 10s, 100% of pages < 15s.

End user operations: 90% of pages < 2s, 100% of pages < 5s.

3.2.2 Provisioning Performance

Cloud provisioning is enormously complex in performance terms. Hardware configuration, user workloads, image properties, and a multitude of other factors combine to determine overall capability. ICO provisioning performance is typically measured via a closed system, defined as an isolated system where we can demonstrate a constant sustained provisioning workload. In order to achieve this, as requests complete within the system, new requests are initiated.

20

Figure 8: Provisioning Performance in a Closed System

The performance systems running ICO workloads literally run for months. These systems are treated like customer systems with 24x7 operations and field ready maintenance approaches in place (as described in Section Error! Reference source not found.). In terms of provisioning performance, the following are sample statistics a long run scenario driven for a number of weeks, once a period of operational stability has been reached based on the recommendations provided in this paper.

Number of systems provisioned: > 1,000,000 VMs.

Provisioning rate (average): > 400 VMs/hour.

Service times (average): 2 minutes 38 seconds (VMware non linked clones).

Workflow capability: On the order of 300 workflows per hour (generally short running workflows under a minute in duration).

Success rate: > 99.99%

Given this is sustained, continuous workload, higher peak workloads are, of course, possible. The success rate is considered especially noteworthy.



21

4 Performance Benchmark Approaches As part of cloud management and capacity planning, it is valuable to manage cloud benchmarks. Value propositions include:

Understanding the capability of the cloud infrastructure (and potentially poorly configured or underperforming components of the infrastructure).

Understanding the base capability of the ICO implementation and associated customization.

Understanding the long term performance stability of the system.

We will describe basic system monitoring approaches, infrastructure benchmarks, and cloud benchmarks.

4.1 Monitoring and Analysis Tools

The following table shows the core recommended monitoring and analysis tools.

Tool Description

pdcollect ICO log collection tool. Documentation and recommended invocation: ICO Product Knowledge Center

esxtop VMware performance collection tool. Documentation: URL Recommended invocation: esxtop -b -a -d 60 -n <number_of_samples> > <output file>

nmon nmon is a comprehensive system monitoring tool for the UNIX platform. It is highly useful for understanding system behavior. Documentation: URL Sample invocation: nmon -T -s <samplerate> -c <iterations> -F <output file>

Note: On Windows systems, Windows perfmon may be used.

db2support Database support collection tool. Documentation: URL Recommended invocation: db2support <result directory> -d <database> -c -f -s -l

DBMS Snapshots

DBMS snapshot monitoring can offer insight into SQL workload, and in particular expensive SQL statements. Documentation: URL

WAIT Java WAIT monitoring can provide a non-invasive view of JVM performance through accumulated Java cores and analytic tools. Documentation and recommended invocation: URL

Figure 9: Monitoring and Analysis Tools

http://www-01.ibm.com/support/knowledgecenter/SS4KMC_2.4.0/com.ibm.sco.doc_2.4/c_pdcollect.html

http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.monitoring.doc%2FGUID-D89E8267-C74A-496F-B58E-19672CAB5A53.html

http://nmon.sourceforge.net/pmwiki.php

http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp?topic=%2Fcom.ibm.db2.luw.admin.trb.doc%2Fdoc%2Ft0020808.html

http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/index.jsp?topic=%2Fcom.ibm.db2.luw.admin.cmd.doc%2Fdoc%2Fr0001945.html

https://wait.ibm.com/

22

4.2 Infrastructure Benchmark Tools

The following table shows some recommended infrastructure benchmark tools.

Tool Description

iometer I/O subsystem measurement and characterization tool for single and clustered systems. Documentation: URL Recommended invocation: dynamo /m <client host name or ip>

iperf TCP and UDP measurement and characterization tool that reports bandwidth, delay, jitter, and datagram loss. Documentation: URL Recommended server invocation: iperf –s Recommended client invocation #1: iperf -c <server host name or ip> Recommended client invocation #2: iperf -c <server host name or ip> -R

UnixBench UNIX measurement and characterization tool, with reference benchmarks and evaluation scores. Documentation: URL Recommended invocation: ./Run

Figure 10: Infrastructure Benchmark Tools

4.3 Cloud Benchmarks

Cloud benchmarks should be based on enterprise utilization. Sample benchmarks that are easy to manage include the following.

1. Single VM deployment times.

2. Small scale concurrent VM deployment times (e.g. 10 requests in parallel).

3. REST API response times.

It is recommended to establish a small load driver, record a baseline, and then use these small benchmarks as a standard to assess ongoing cloud health. More complex benchmarks, including client request monitoring approaches, may of course be established.

For OpenStack specific benchmarks, OpenStack Rally may be leveraged (see the References section for further detail). In addition, the Open Systems Group is involved in cloud computing benchmark standards. A report, including the IBM CloudBench tool, is available in the References section.

http://sourceforge.net/projects/iometer/

https://code.google.com/p/iperf/

https://code.google.com/p/byte-unixbench/



23

5 Capacity Planning Recommendations We will provide capacity planning recommendations through three approaches.

Static planning via a spreadsheet approach.

Capacity planning for the ICO management server (aka the “managed from” infrastructure).

Capacity planning for the provisioned Virtual Machines (aka the “managed to” infrastructure).

5.1 Cloud Capacity Planning Spreadsheet

In order to provide a desired hardware and software configuration for an ICO implementation, a wide range of parameters must be understood. The following questions are usually relevant.

1. What operations are expected to be performed with ICO?

2. What are the average and peak concurrent user workloads?

3. What is the enterprise network topology?

4. What is the expected workload for provisioned virtual servers, and how do they map to the physical configuration?

5. For the provisioned servers:

a. What is the distribution size?

b. What are the application service level requirements?

A capacity planning spreadsheet is attached to this paper (“ICO Capacity Planning Profile

v2.4.0.xlsx”). The spreadsheet may be used to provide a cloud profile for further sizing activities (e.g. a capacity planning activity in association with the document authors).

24

5.2 IBM Cloud Orchestrator Management Server Capacity Planning

The ICO management server requirements are documented in the ICO Knowledge Center (URL). The summary table is repeated here for discussion purposes.

Server & Configuration Processor (vCPUs)

Memory (GB)

Free Storage (GB)

Deployment Server

Minimum 1 4 117

Recommended 2 8 117

Central Server 1

Minimum 2 vCPUs 6 GB 100 GB

Recommended 4 vCPUs 12 GB 200 GB

Central Server 2



Central Server 3



SA Application Manager

Minimum n/a n/a n/a


Neutron Network Server



Region Server: VMware



Region Server: KVM, Power, z/VM



KVM Compute Node


Recommended Application Specific

Figure 11: ICO Management Server Capacity Planning

While further qualifiers are available in the Knowledge Center, some comments apply.

In general, the recommended vCPU and memory allocations should be met.

To determine the ratio of virtual to physical CPUs, monitoring of the production system is required. For performance verification, a 1:1 mapping is used.

http://www-01.ibm.com/support/knowledgecenter/SS4KMC_2.4.0/com.ibm.sco.doc_2.4/c_hw_prerequisites.html



25

For the physical mapping, it is important to distinguish between “real” cores and hyper threaded (HT) cores. External benchmarks suggest an HT core may yield 30% of the capability of a “real” core.

The recommended storage amounts are highly subjective. For example, the minimum recommendations are sufficient for performance verification systems driven for months (with some minor exceptions).

5.3 Provisioned Virtual Machines Capacity Planning

Managing cloud workloads is typically driven as a categorization exercise where workload “sizes” are used to determine the overall capacity requirements. A capacity planning tool is available for managing the cloud workload sizes (URL). We will provide an overview of using this tool.

The first step is to provide any relevant business value. In the absence of a defined opportunity, simple “not applicable” entries may be given (per the sample below). Once submitted, you must accept the usage agreement which will bring up the demographic page.

Figure 12: Capacity Planning Tool: Inquiry Form

http://www.ibm.com/partnerworld/wps/sizing/protect/sizingguide/RunGuide?guide_id=sgq26454833270303102

26

The demographic page simply asks for generic information about the submitter.

Figure 13: Capacity Planning Tool: User Demographic Information

When “Continue” is selected, then the systems and storage page is provided.

Figure 14: Capacity Planning Tool: Systems and Storage



27

Then the target system and associated utilization and Virtual Machine requirements are selected. Note for the utilization we select 20% headroom to support peak cloud workloads.

Figure 15: Capacity Planning Tool: System and Workload Options

At this point, the virtual machine requirements may be selected. Note a number of entries may be added.

Figure 16: Capacity Planning Tool: Virtual Machine Requirements

A confirmation screen is then provided to finalize the capacity planning request.

Figure 17: Planning Tool: Confirmation Screen

The summary capacity planning recommendation is then provided. The summary details the compute node, CPU, memory, and storage requirements based on the selected configuration and associated workloads.

28

Figure 18: Planning Tool: System Summary



29

6 Cloud Configuration Recommendations The ICO 2.4 offering provides suitable configuration as part of the default installation. However, there are some specific configuration aspects that may improve the capability. The configuration points follow.

1. OpenStack Keystone worker support.

2. Disabling the IWD service.

3. IBM Workload Deployer configuration.

4. Virtual Machine IO scheduler.

5. Advanced Configuration and Power Interface (ACPI) management.

6. Java Virtual Machine heap.

7. Database configuration.

8. Database management.

6.1 OpenStack Keystone Worker Support

The initial SCO 2.3 offering contained a Keystone implementation that is characterized by a single execution thread instance. For ICO 2.4, improvements have been made to exploit multiple concurrent Keystone workers. This change offers advantages when Keystone exhibits high request latency, or is seen to consume a significant amount of a virtual CPU (e.g. > 80%). In order to exploit this support, it is necessary to revise the configuration to exploit multiple workers. Further detail on this is provided below.

With the Keystone Worker improvement in place, the following configuration change will allow a pool of four public workers, and four administrative workers. This will permit increased concurrency, at the expense of virtual CPU consumption. As a result, the virtual CPU allocation should be increased based on monitoring data. In the “4+4” worker example below, it is expected to increase the virtual CPU allocation on the order of two to four virtual CPUs.

Location: (Central Server 2)

/etc/keystone/keystone.conf

# The number of worker processes to serve the public WSGI application

# (integer value).

public_workers=4

# The number of worker processes to serve the admin WSGI application

#(integer value).

admin_workers=4

Figure 19: Keystone Worker Configuration

6.2 Disabling the IWD Service

The IWD service consumes significant resources across the ICO management stack. In the event the service is not required, it should be disabled.

30

6.3 IBM Workload Deployer Configuration

The IWD component offers a number of configuration options. One specific option provides the ability to control a polling interval to refresh cloud information. Based on the size of the cloud, this configuration option should be changed.

Location: (Central Server 3)

/opt/ibm/rainmaker/purescale.app/private/expanded/ibm/rainmaker.vmsupport-

4.0.0.1/config/vmpublish.properties

Original: RuntimeInterval=12000

Recommended: RuntimeInterval=30000

Figure 20: IWD Configuration

6.4 Virtual Machine IO Scheduler Configuration

Each Linux instance has an IO scheduler. The intent of the IO scheduler is to optimize IO performance, potentially by clustering or sequencing requests to reduce the physical impact of IO. In a virtual world, however, the operating system is typically disassociated from the physical world through the hypervisor. As a result, it is recommended to alter the IO scheduler algorithm so that it is more efficient in a virtual deployment, with scheduling delegated to the hypervisor.

The default scheduling algorithm is typically “cfq” (completely fair queuing). Alternative and recommended algorithms are “noop” and “deadline”. The “noop” algorithm, as expected, does as little as possible with a first in, first out queue. The “deadline” algorithm is more

advanced, with priority queues and age as a scheduling consideration. System specific benchmarks should be used to determine which algorithm is superior for a given workload. In the absence of available benchmarks, we would recommend the “deadline” scheduler be used.

The following console output shows how to display and modify the IO scheduler algorithm for a set of block devices. In the example, the “noop” scheduler algorithm is set. Note to

ensure the scheduler configuration persists, it should be enforced via the operating system configuration (e.g. /etc/rc.local).

Figure 21: Modifying the IO Scheduler

6.5 Advanced Configuration and Power Interface Management

The Advanced Configuration and Power Interface (ACPI) operating system support may exhibit high virtual CPU utilization and offers limited value in virtual environments. It is recommended to disable ACPI on the ICO “managed from” nodes through the following steps.

1. Disabling “kacpid”.



31

To switch off the kernel ACPI daemon, edit “/etc/grub.conf” and append "acpi=off" to

the kernel boot command line. For example: title Red Hat Enterprise Linux (2.6.32-431.el6.x86_64)

root (hd0,0)

kernel /boot/vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=e1131bc1-bdbc-4b2e-

9ae7-d540b32b1f35

initrd /boot/initramfs-2.6.32-431.el6.x86_64.img

becomes: title Red Hat Enterprise Linux (2.6.32-431.el6.x86_64)

root (hd0,0)

kernel /boot/vmlinuz-2.6.32-431.el6.x86_64 ro root=UUID=e1131bc1-bdbc-4b2e-9ae7-

d540b32b1f35 acpi=off

initrd /boot/initramfs-2.6.32-431.el6.x86_64.img

2. Disabling the user-space acpi daemon. To disable user space ACPI on managed-from nodes: chkconfig acpid off

3. Reboot the nodes.

6.6 Java Virtual Machine Heap Configuration

The default Java Virtual Machine (JVM) heap sizes are intended to be economical. However, in the presence of sufficient available memory, it is recommended to increase the heap allocation. The three change sets below are recommended for application. They apply to Central Server 3 and, in particular, the IBM Workload Deployer instance. The IWD instance should be restarted once the changes are complete.

Location: /opt/ibm/rainmaker/purescale.app/config/overrides.config

Original: /config/zso/jvmargs = ["-Xms1024M","-Xmx1024M"]

Recommended: /config/zso/jvmargs = ["-Xms1536M","-Xmx1536M"]

Location: /etc/rc.d/init.d/iwd-utils

Original: sed -i -e 's/3072M/1024M/g' $ZERO_DIR/config/overrides.config

Recommended: sed -i -e 's/3072M/1536M/g' $ZERO_DIR/config/overrides.config

Location: /opt/ibm/rainmaker/purescale.app/config/zero.config

Original: "-Xms1024M","-Xmx1024M"

Recommended: "-Xms1536M","-Xmx1536M"

Figure 22: Java Virtual Machine Heap Change Sets

32

6.7 Database Configuration

ICO is deployed with a DB2 database. The performance of the database is critical to the overall capability of the solution. The following database configuration changes are recommended for a base ICO 2.4 installation.

Type Configuration

Configuration For each relevant database set: STMT_CONC = LITERALS

LOCKTIMEOUT = 60 NUM_IOCLEANERS = AUTOMATIC

NUM_IOSERVERS = AUTOMATIC

AUTO_REORG = ON

For example: db2 UPDATE DB CFG FOR OPENSTAC USING LOCKTIMEOUT 60

Foreign Key Modification

An OpenStack foreign key should be modified to enable cascading deletes. Please apply the “ICO_MODIFY_FKEY.sh” script provided with this paper.

Figure 23: Database Configuration Change Sets

6.8 Database Management

Generally speaking, the “out of the box “database configuration will achieve good results for both large and small installations. The following recommendations are primarily in the area of database maintenance.

6.8.1 DBMS Versions

The following DBMS versions are recommended. All versions should be 64 bit.

Version Notes

DB2 10.5 fp5 or later The minimum recommended fixpack level is 10.5 fp3a.

Figure 24: DBMS Versions

6.8.2 Automatic Maintenance

DB2 offers a number of automatic maintenance options. Automatic statistics collection (aka runstats) is considered a basic and necessary configuration setting, and is enabled for the product by default. Two other recommended configuration settings follow. It is expected these configuration settings will be enabled by default in future versions of the products.

1. Real time statistics. The default runstats configuration generally collects statistics at two hour intervals. The real time statistics option provides far more granular statistics collection, essentially generating statistics as required at statement compilation time.

2. Automatic reorganization. Many customers ignore database reorganization and system performance starts to decline. This can be especially critical in the cloud space. The recommendation is to enable automatic reorganization support so it is self managed by the DBMS. Further discussion of database reorganization is covered in section 6.9.3.



33

The following commands may be used to enable these automatic maintenance options. At the time of this writing, they are conditionally recommended. Each of these options has runtime impact and should be monitored to ensure there is no unnecessary system impact. In order to facilitate this, they should only be enabled once the system has been established and monitored. In addition, automatic reorganization is dependent on the definition of a maintenance window (see the DB2 Knowledge Center for more detail).

update db cfg for OPENSTAC using AUTO_STMT_STATS ON

update db cfg for OPENSTAC using AUTO_REORG ON

Figure 25: Database Automatic Maintenance Configuration

6.8.3 Operating System Configuration (Linux)

The product installation guides have comprehensive instructions for Operating System pre-requisites and configuration. However, on Linux systems improper configuration is common, so we will highlight specific issues.

The first configuration point to check is the file system ulimit for the maximum number of open files allowed for a process (i.e. nofiles). The value for this kernel limit should be either “unlimited” or “65536”. The DB2 reference for this configuration setting is available here.

In addition, the kernel semaphore and message queue specifications should be correct. These configuration settings are a function of the physical memory available on the machine. The DB2 reference for these configuration settings is available here.

6.9 Database Hygiene Overview

The following steps will be described for database hygiene overview:

1. Database backup management.

2. Database statistics management.

3. Database reorganization.

4. Database archive management.

5. Database maintenance automation.

Steps make reference to recommended scheduling frequencies. The general purpose “cron” scheduling utility may be used to achieve this. However, other scheduling utilities may also be used. The key aspect of a cron’ed activity is it is scheduled at regular intervals (e.g. nightly, weekly) and typically does not require operator intervention. Designated maintenance windows may be used for these activities.

6.9.1 Database Backup Management

It is recommended that nightly database backups be taken. The following figures offer a sample database offline backup (utilizing compression), along with a sample restore.

backup db <dbname> user <user> using <password> to <backup directory> compress

Figure 26: Database Backup with Compression Command

restore db <dbname> from <backup directory> taken at <timestamp> without

prompting

Figure 27: Database Offline Backup Restore

Online backups may be utilized as well. The following figure provides commands that comprise a sample weekly schedule. With the given schedule, the best case scenario is a restore requiring one image to restore (Monday failure using the Sunday night backup). The worst case scenario would require four images (Sunday + Wednesday + Thursday +

http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/topic/com.ibm.db2.luw.qb.server.doc/doc/r0052441.html?resultof=%22%75%6c%69%6d%69%74%22%20

http://pic.dhe.ibm.com/infocenter/db2luw/v10r1/topic/com.ibm.db2.luw.qb.server.doc/doc/c0057140.html

34

Friday). An alternate approach would be to utilize a full incremental backup each night to make the worst case scenario two images. The tradeoffs for the backup approaches are the time to take the backup, the amount of disk space consumed, and the restore dependencies. A best practice can be to start with nightly full online backups, and introduce incremental backups if time becomes an issue.

(Sun) backup db <dbname> online include logs use tsm

(Mon) backup db <dbname> online incremental delta use tsm

(Tue) backup db <dbname> online incremental delta use tsm

(Wed) backup db <dbname> online incremental use tsm

(Thu) backup db <dbname> online incremental delta use tsm

(Fri) backup db <dbname> online incremental delta use tsm

(Sat) backup db <dbname> online incremental use tsm

Figure 28: Database Online Backup Schedule

Note to enable incremental backups, the database configuration must be updated to track page modifications, and a full backup taken in order to establish a baseline.

update db cfg for OPENSTAC using TRACKMOD YES

Figure 29: Database Incremental Backup Enablement

To restore the online backups, either a manual or automatic approach may be used. For the manual approach, you must start with the target image, and then revert to the oldest relevant backup and move forward to finish with the target image. A far simpler approach is to use the automatic option and let DB2 manage the images. A sample of each approach is provided below, showing the restore based on the Thursday backup.

restore db <dbname> incremental use tsm taken at <Sunday full timestamp>

restore db <dbname> incremental use tsm taken at <Wednesday incremental

timestamp>

restore db <dbname> incremental use tsm taken at <Thursday incremental delta

timestamp>

Figure 30: Database Online Backup Manual Restore

restore db <dbname> incremental auto use tsm taken at <Thursday incremental

delta timestamp>

Figure 31: Database Online Backup Automatic Restore

In order to support online backups, archive logging must be enabled. The next subsection provides information on archive logging, including the capability to restore to a specific point in time using a combination of database backups and archive logs.

Database Log Archiving

A basic approach we will advocate is archive logging with the capability to support online backups. The online backups themselves may be full, incremental (based on the last full backup), and incremental delta (based on the last incremental backup). In order to enable log archiving to a location on disk, the following command may be used.

update db cfg for <dbname> using logarchmeth1 DISK:/path/logarchive

Figure 32: Database Log Archiving to Disk



35

Alternatively, in order to enable log archiving to TSM, the following command may be used1.

update db cfg for <dbname> using logarchmeth1 TSM

Figure 33: Database Log Archiving to TSM

Note that a “logarchmeth2” configuration parameter also exists. If both of the log archive

method parameters are set, each log file is archived twice (once per log archive method configuration setting). This will result in two copies of archived log files in two distinct locations (a useful feature based on the resiliency and availability of each archive location).

Once the online backups and log archive(s) are in effect, the recovery of the database may be performed via a database restore followed by a roll forward through the logs. Several restore options have been previously described. Once the restore has been completed, roll forward recovery must be performed. The following are sample roll forward operations.

rollforward <dbname> to end of logs

Figure 34: Database Roll Forward Recovery: Sample A

rollforward <dbname> to 2012-02-23-14.21.56 and stop

Figure 35: Database Roll Forward Recovery: Sample B

It is worth noting the second example recovers to a specific point in time. For a comprehensive description of the DB2 log archiving options, the DB2 Knowledge Center should be consulted (URL). A service window (i.e. stop the application) is typically required to enable log archiving.

Database Backup Cleanup

Unless specifically pruned, database backups may accumulate and cause issues with disk utilization or, potentially, a stream of failed backups. If unmonitored backups begin to fail, it may make disaster recovery near impossible in the event of a hardware or disk failure. A simple manual method to prune backups follows.

find /backup/DB2 -mtime +7 | xargs rm

Figure 36: Database Backup Cleanup Command

A superior approach is to let DB2 automatically prune the backup history and delete your old backup images and log files. A sample configuration is provided below.

update db cfg for OPENSTAC using AUTO_DEL_REC_OBJ ON

update db cfg for OPENSTAC using NUM_DB_BACKUPS 21

update db cfg for OPENSTAC using REC_HIS_RETENTN 180

Figure 37: Database Backup Automatic Cleanup Configuration

It is also generally recommended to have the backup storage independent from the database itself. This provides a level of isolation in the event volume issues arise (e.g. it ensures that a backup operation will not fill the volume hosting the tablespace containers, which could possibly lead to application failures).

1 The log archive methods (logarchmeth1, logarchmeth2) have the ability to associate configuration

options with them (logarchopt1, logarchopt2) for further customization.

https://www-01.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.kc.doc/welcome.html

36

6.9.2 Database Statistics Management

As discussed in the previous “Automatic Maintenance” section, database statistics ensure that the DBMS optimizer makes wise choices for database access plans. The DBMS is typically configured for automatic statistics management. However, it may often be wise to force statistics as part of a nightly or weekly database maintenance operation. A simple command to update statistics for all tables in a database is the “reorgchk” command.

reorgchk update statistics on table all

Figure 38: Database Statistics Collection Command

One issue with the reorgchk command is it does not enable full control over statistics capturing options. For this reason, it may be beneficial to perform statistics updates on a table by table level. However, this can be a daunting task for a database with hundreds of tables. As a result, the following SQL statement may be used to generate administration commands on a table by table basis.

select 'runstats on table ' || STRIP(tabschema) || '.' || tabname || ' with

distribution and detailed indexes all;' from SYSCAT.TABLES where tabschema in

('CIRnnnnn','GLEnnnnn','NOAnnnnn','SCEnnnnn','KSDB');

Figure 39: Database Statistics Collection Table Iterator

6.9.3 Database Reorganization

Over time, the space associated with database tables and indexes may become fragmented. Reorganizing the table and indexes may reclaim space and lead to more efficient space utilization and query performance. In order to achieve this, the table reorganization command may be used. Note, as discussed in the previous “Automatic Maintenance” section, automatic database reorganization may be enabled to reduce the requirement for manual maintenance.

The following commands are examples of running a “reorg” on a specific table and its associated indexes. Note the “reorgchk” command previously demonstrated will actually have a per table indicator of what tables require a reorg. Using the result of “reorgchk” per

table reorganization may be achieved for optimal database space management and usage.

reorg table <table name> allow no access

reorg indexes all for table <table name> allow no access

Figure 40: Database Reorganization Commands

It is important to note there are many options and philosophies for doing database reorganization. Every enterprise must establish its own policies based on usage, space considerations, performance, etc. The above example is an offline reorg. However it is possible to also do an online reorg via the “allow read access” or “allow write access” options. The “notruncate” option may also be specified (indicating the table will not be truncated in order to free space). The “notruncate” option permits more relaxed locking and

greater concurrency (which may be desirable if the space usage is small or will soon be reclaimed). If full online access during a reorg is required, the “allow write access” and “notruncate” options are both recommended.



37

Note it is also possible to use our table iteration approach to do massive reorgs across hundreds of tables as shown in the following figure. The DB2 provided snapshot routines and views (e.g. SNAPDB, SNAP_GET_TAB_REORG) may be used to monitor the status of reorg operations.

select 'reorg table ' || STRIP(tabschema) || '.' || tabname || ' allow no

access;' from SYSCAT.TABLES where tabschema in


select 'reorg indexes all for table ' || STRIP(tabschema) || '.' || tabname ||

' allow no access;' from SYSCAT.TABLES where tabschema in


Figure 41: Database Reorganization Table Iterator

38

7 Summary Cookbook The following tables provide a cookbook for the solution implementation. The cookbook approach implies a set of steps the reader may “check off” as completed to provide a stepwise implementation of the ICO solution. The recommendations will be provided in three basic steps:

1. Base installation recommendations.

2. Post installation recommendations.

3. High scale recommendations.

All recommendations are provided in tabular format. The preferred order of implementing the recommendations is in order from the first row of the table through to the last.

7.1 Base Installation Recommendations

The base installation recommendations are considered essential to a properly functioning ICO instance. All steps should be implemented.

Identifier Description Status

B1 Perform the base ICO installation, ensuring the recommended configuration described in Section 5.2 is achieved.

A central DB2 server should be used (i.e. the region servers should not manage a local DBMS unless there are compelling geographic considerations). Where possible it is recommended to install the DBMS on bare metal, or in a DBA managed pool, to facilitate performance management.

B2 Enable the OpenStack Keystone worker support (Section 6.1).

B3 Disable the IWD service, if possible (Section 6.2).

B4 If the IWD service is required, optimize the IWD component (Section 6.3).

B5 Configure the Linux IO scheduler (Section 6.4).

B6 Disable the ACPI management (Section 6.5).

B7 Ensure the Java heaps are optimized (Section 6.6).

B8 Configure the central database (Section 6.7).

Figure 42: Base Installation Recommendations



39

7.2 Post Installation Recommendations

The post installation recommendations will provide additional throughput and superior functionality. All steps should be implemented.


P1 Perform a set of infrastructure and ICO benchmarks to determine the viability of the installation (see Sections 4.2 and 4.3).

P2 Implement the database statistics maintenance activity per Section 6.9.2.

P3 Implement the database reorg maintenance activity per Section 6.9.3.

P4 Implement a suitable backup and disaster recovery plan comprising regular backups of all critical server components (including the database and relevant file system objects). Guidelines are provided in the ICO Knowledge Center (URL).

Figure 43: Post Installation Recommendations

7.3 High Scale Recommendations

The high scale recommendations should be incorporated once the production installation wants to support the high water mark for scalability. All steps may be optionally implemented over time based upon workload.


S1 Apply the latest ICO fixpack.

S2 Monitor the performance of the installation (Section 4.1) and adjust the management server to the recommended installation values (Section 5.2) as appropriate.

S3 Optimize Central Server 1 (DBMS) performance. A basic way to achieve this is to have dedicated, high performance storage allocated to the database containers and logs.

Figure 44: High Scale Recommendations

https://www-01.ibm.com/support/knowledgecenter/SS4KMC_2.4.0/com.ibm.sco.doc_2.4/gs/aat_wwapplia.html?lang=en

40

APPENDIX A: IBM CLOUD

ORCHESTRATOR MONITORING OPTIONS

Monitoring is important to understand and ensure the health of any cloud solution. A number of monitoring approaches are available for ICO. The solutions are described via the following summary sections, broken down into three categories.

1. OpenStack monitoring via Ceilometer.

2. ICO monitoring via IBM BPM.

3. Infrastructure monitoring via IBM Tivoli Monitoring (ITM) and third party solutions.

A separate appendix is provided that is specific to OpenStack Keystone monitoring.

A.1 OpenStack Monitoring

OpenStack monitoring is provided via the Ceilometer component. Ceilometer offers a comprehensive and customizable infrastructure, including support for event and threshold management. Note while Ceilometer is not enabled as part of the base ICO 2.4 distribution, it is a constituent of the OpenStack Grizzly base, with continued enhancement in subsequent OpenStack releases.

Ceilometer provides three distinct types of metrics:

1. Cumulative: counters that accumulate or increase over time.

2. Gauge: counters that offer discrete, point in time values.

3. Delta: differential counters showing change rates.

A vast array of metrics is provided by Ceilometer. An easy way to interactively derive the set of available metrics is to query Ceilometer directly (see the sample below). In addition, the Ceilometer documentation provides the default set, with associated attributes (URL).

ceilometer meter-list -s openstack

Figure 45: OpenStack Ceilometer Metrics

http://docs.openstack.org/developer/ceilometer/measurements.html

42

warehouse (relevant information available in the References section)2. The performance data warehouse may be enabled via “autotracking”, which will enable both custom KPIs as well as the default total time KPIs. The core KPIs to understand BPM capability are:

BPM processes executed per second.

Average service times per BPM process.

It is important to note that given Ceilometer provides a general plugin and distribution infrastructure, it may be combined with the ICO monitoring solution. A sample approach for managing these monitoring points follows.

1. Derive a BPM plugin to retrieve raw times from the BPM performance data warehouse (PDWDB) database. The preferred method is the provided REST interface (versus direct database access).

2. Perform calculations based on the raw data. For example, converting a series of milestones into performance KPIs, or calculating statistical quantities (e.g. standard deviation, harmonic mean).

3. Push the results to Ceilometer as the meter distribution mechanism.

4. Read the results via the Ceilometer REST API and display in the visualization tool of your choice.

A.3 Infrastructure Monitoring

Infrastructure monitoring can address the operating system and hypervisor health of the cloud. Available tools include IBM Tivoli Monitoring (ITM) or the open source offering Nagios. For example, ITM v6.2 provides the follow infrastructure monitoring agents (for reference, see URL).

1. IBM Tivoli Monitoring Endpoint.

2. Linux OS.

3. UNIX Logs.

4. UNIX OS.

5. Windows OS.

6. i5/OS®.

7. IBM Tivoli Universal Agent.

8. Warehouse Proxy.

9. Summarization and Pruning.

10. IBM Tivoli Performance Analyzer.

2 It is worth noting that BPM is built on IBM WebSphere and as a result, WebSphere monitoring

capabilities also apply.

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3fp1/itm623FP1_install06.htm?path=3_1_3_0_4_0#itm_over



43

Critical KPIs to monitor at the infrastructure level are summarized in the following table (VMware is provided as a representative hypervisor sample).

Component Meters

Operating System CPU utilization including kernel,

user, IO wait, and idle times.

Disk utilization including read/write request and byte rates.

Network utilization including incoming and outgoing packet and byte rates.

Volume free space across the central and region servers. Special attention should be paid to the Virtual Image Library on Central Server 2 to ensure the “/home/library” space is well managed.

DBMS: ITM for DB2 (URL) Application IO activity workspace.

Application lock activity workspace.

Application overview workspace.

Buffer Pool workspace.

Connection workspace.

Database workspace.

Database Lock Activity workspace.

Historical Summarized Capacity Weekly workspace.

Historical Summarized Performance Weekly workspace.

Locking Conflict workspace.

Tablespace workspace.

Application Server: ITCAM Agent for

WebSphere Applications (URL)

WebSphere Agent Summary workspace.

Application Server Summary workspace.

J2EE: ITCAM Agent for J2EE (URL) Application Health Summary

workspace.

HTTP: ITCAM Agent for HTTP Servers

(URL)

Web Server Agent workspace.

Hypervisor: ITM for Virtual Environments

(URL)

Server workspace.

CPU workspace.

Disk workspace.

Memory workspace.

Network workspace.

Resource Pools workspace.

Virtual Machines workspace.

Hypervisor: VMware esxtop sample CPU:

Run(%RUN), Wait (%WAIT), Ready (%RDY), Co-Stop (%CSTP).

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itmfd.doc/main_db2.htm

http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcamfapps_ad.doc_721/itcam_71_was_tema_help/kyn_overview.html

http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcamfapps_ad.doc_721/itcam_71_j2ee_tema_help/kyj_overview.html

http://publib.boulder.ibm.com/infocenter/tivihelp/v24r1/topic/com.ibm.itcamfapps_ad.doc_721/itcam_71_http_tema_help/kht_overview.html

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.tivoli.itmvs.doc_7.1/vmware/fac_aboutthisdoc.html

44

Network: Dropped packets (%DRPTX, %DRPRX).

IO: Latency (DAVG, KAVG), Queue length (QUED)

Memory: Memory reclaim (MCTLSZ), Swap (SWCUR, SWR/s, SWW/s),

Figure 47: Infrastructure Core Metrics



45

APPENDIX B: OPENSTACK KEYSTONE

MONITORING

The Keystone component is critical to overall performance of IBM Cloud Orchestrator. For example, if one component saturates Keystone, the overall throughput of the system will be impacted. This is magnified by the fact that Keystone has only a single execution thread instance. In order to understand Keystone performance, the best method is to look at the requests and responses via a proxy such as the IaaS Gateway. This provides the ability to see requests that are dropped before being processed by Keystone.

We will describe an approach for monitoring Keystone via the PvRequestFilter.

B.1 PvRequestFilter

The PvRequestFilter was designed to output request and response data into the Keystone log. When enabled it prints the data as warning messages, so it is not necessary to turn up the default debug level to generate the log messages.

The format of the messages is as follows. All fields except “<duration>” are printed out for both requests and responses. The duration of the request is printed only for the response.

WARNING [REQUEST|RESPONSE] <millisecond timestamp to identify request>

<REMOTE_ADDR>:<REMOTE_PORT> <REQUEST_METHOD> <RAW_PATH_INFO> [<duration>]

Figure 48: Keystone Monitoring PvRequestFilter Format

Sample output follows.

2014-07-21 17:16:56.509 22811 WARNING keystone.contrib.pvt_filter.request [-]

REQUEST 2014-07-21_17:16:56.509 172.18.152.103:1278 GET /v3/users


RESPONSE 2014-07-21_17:16:56.509 172.18.152.103:1278 GET /v3/users 0.276294


REQUEST 2014-07-21_17:16:56.807 172.18.152.103:1278 GET /v3/domains


RESPONSE 2014-07-21_17:16:56.807 172.18.152.103:1278 GET /v3/domains 0.017691


REQUEST 2014-07-21_17:16:56.839 172.18.152.103:1279 GET

/v3/users/e92b94d7068843ef98d664521bd9c983/projects


RESPONSE 2014-07-21_17:16:56.839 172.18.152.103:1279 GET

/v3/users/e92b94d7068843ef98d664521bd9c983/projects 0.028558

Figure 49: Keystone Monitoring PvRequestFilter Sample Output

B.2 Enabling PvRequestFilter

The process to enable PvRequestFilter follows.

1. Log onto Central Server 2.

46

2. Extract the distribution provided with this paper (keystoneStats.zap).

3. Install the filter and backup the existing configuration: ./deployKeystoneFilter.sh

4. Make the following changes to the “/etc/keystone/keystone.conf” file. Note: Reversing step 2 will disable the filter.

a. Add the following lines just above line starting with "[filter:debug]". [filter:pvt]

paste.filter_factory =

keystone.contrib.pvt_filter.request:PvtRequestFilter.factory

b. Add "pvt" to three of the pipeline statements: [pipeline:public_api]

pipeline = access_log sizelimit url_normalize token_auth

admin_token_auth xml_body json_body simpletoken ec2_extension

user_crud_extension pvt public_service

[pipeline:admin_api]



s3_extension crud_extension pvt admin_service

[pipeline:api_v3]



s3_extension pvt service_v3

c. Restart the keystone service. service openstack-keystone restart

d. Validate that the “/var/log/keystone/keystone.log” is producing the appropriate log messages (sample below).

e. Update the “hosts.table” file to reflect your environment.

f. Run the workload or scenario for analysis.

g. Generate the statistics for the request and response data in the

“keystone.log” file (sample below): ./keystoneStats.sh /var/log/keystone/keystone.log > results

Figure 50: Keystone Monitoring Log Messages Example

Figure 51: Keystone Monitoring Statistics Example



47

REFERENCES

IBM Cloud Orchestrator and Related Component References

IBM Cloud Orchestration Knowledge Center

ICO 2.4 Knowledge Center IBM Cloud Orchestrator Resource Center

ICO Resource Center IBM Cloud Orchestrator Version 2.4: Security Hardening Guide

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO85 SmartCloud Orchestrator Version 2.3: Capacity Planning, Performance, and Management Guide

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO7P

SmartCloud Orchestrator Version 2.3: Security Hardening Guide http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO7W

IBM Cloud Orchestrator Version 2.3: Database Movement Cookbook http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO8T

IBM SmartCloud Orchestrator: Offline-backup approach using Tivoli Storage Manager for Virtual Environments http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO7Q

IBM Business Process Manager V8.0 Performance Tuning and Best Practices http://www.redbooks.ibm.com/redpapers/pdfs/redp4935.pdf IBM Business Process Manager Performance Data Warehouse

http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r5m0/topic/com.ibm.wbpm.admin.doc/topics/managing_performance_servers.html

IBM Tivoli Monitoring Information Center http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3fp1/welc

ome.htm IBM DB2 10.5 Knowledge Center DB2 10.5 Knowledge Center

OpenStack References

OpenStack Performance Presentation (Folsom, Havana, Grizzly) http://www.openstack.org/assets/presentation-media/openstackperformance-v4.pdf OpenStack Ceilometer http://docs.openstack.org/developer/ceilometer OpenStack Rally https://wiki.openstack.org/wiki/Rally

https://www-01.ibm.com/support/knowledgecenter/SS4KMC_2.4.0/com.ibm.sco.doc_2.4/kc_welcome-ico.html?lang=en

http://www-03.ibm.com/software/products/en/ibm-cloud-orchestrator

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO85

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO7P

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO7W

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO8T

http://www.ibm.com/software/ismlibrary?NavCode=1TW10SO7Q

http://www.redbooks.ibm.com/redpapers/pdfs/redp4935.pdf



http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3fp1/welcome.htm

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.itm.doc_6.2.3fp1/welcome.htm

https://www-01.ibm.com/support/knowledgecenter/SSEPGG_10.5.0/com.ibm.db2.luw.kc.doc/welcome.html

http://www.openstack.org/assets/presentation-media/openstackperformance-v4.pdf

http://docs.openstack.org/developer/ceilometer

https://wiki.openstack.org/wiki/Rally

48

Hypervisor References

Performance Best Practices for VMware vSphere™ 5.0 http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf Performance Best Practices for VMware vSphere™ 5.1 http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.1.pdf VMware: Troubleshooting virtual machine performance issues VMware Knowledge Base VMware: Performance Blog http://blogs.vmware.com/vsphere/performance Linux on System x: Tuning KVM for Performance KVM Performance Tuning Kernel Virtual Machine (KVM): Tuning KVM for performance http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaattuning_pdf.pdf PowerVM Virtualization Performance Advisor Developer Works PowerVM Performance IBM PowerVM Best Practices

http://www.redbooks.ibm.com/redbooks/pdfs/sg248062.pdf

Benchmark References

Report on Cloud Computing to the OSG Steering Committee, SPEC Open Systems Group, https://www.spec.org/osgcloud/docs/osgcloudwgreport20120410.pdf

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.0.pdf

http://www.vmware.com/pdf/Perf_Best_Practices_vSphere5.1.pdf

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1008360

http://blogs.vmware.com/vsphere/performance

http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaattunkickoff.htm

http://pic.dhe.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaattuning_pdf.pdf

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/PowerVM%20Virtualization%20Performance%20Advisor

http://www.redbooks.ibm.com/redbooks/pdfs/sg248062.pdf

https://www.spec.org/osgcloud/docs/osgcloudwgreport20120410.pdf



49

®

© Copyright IBM Corporation 2015 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to:

IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation 4205 South Miami Boulevard Research Triangle Park, NC 27709 U.S.A. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. If you are viewing this information softcopy, the photographs and color illustrations may not appear.

50

Trademarks

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at http://www.ibm.com/legal/copytrade.shtml.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other

countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Other company, product, or service names may be trademarks or service marks of others.

Documents

IBM Cloud Orchestrator Version 2.4: Capacity Planning