IBM zEnterprise System · PDF filemake any such changes without ... which involve transactions moving money into and out ... different OpCodes and options that you can execute by typing

IBM Systems & Technology Group

IBM zEnterprise® System: SAP Enqueue Replication into System z Coupling Facility Performance Test Report ©Copyright IBM Corporation 2013 All Rights Reserved Page 1 of 27

IBM zEnterprise System: SAP Enqueue Replication into System z Coupling Facility Performance Test Report

Authors: Seewah Chan Harald Duvenbeck Dr. Paul Lekkas Rose L. Manz Wolfgang Reichert Michael Sheets Stefan Sell

Document Owner: Rose L. Manz

Version: 1.01 Date: February 13, 2013



Reminders Neither this document nor any part of it may be copied or reproduced in any form or by any means or translated into another language without the prior consent of the IBM Corporation. IBM makes no warranties or representations with respect to the content hereof and specifically disclaim any implied warranties of merchantability or fitness of any particular purpose. IBM assumes no responsibility for any errors that may appear in this document. The information contained in this document is subject to change without any notice. IBM reserves the right to make any such changes without obligation to notify any person of such revision or changes. IBM makes no commitment to keep the information contained herein up to date. The results shown are based on specific workloads run in a somewhat controlled environment. However, these tests were not intended to be definitive performance measurements. The actual throughput that any user will experience will vary considerably from these results. Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml. SAP, all SAP logos, SAP R/3, SAP ERP, mySAP, mySAP.com, xApps, xApp, ABAP, BAPI, SAP NetWeaver and all SAP product and service names mentioned herein are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. Feedback Please send comments, questions or suggestions for changes to Rose Manz at [email protected] . Acknowledgements The authors would like to recognize the contributions of the following reviewers: John Langer - Project Leader, IBM Systems & Technology Group, Poughkeepsie, NY Barbara Weiler - Performance Analyst, IBM Systems & Technology Group, Poughkeepsie, NY

http://www.ibm.com/legal/copytrade.shtml

mailto:[email protected]



Table of Contents 1 INTRODUCTION................................................................................................................ 4 2 WORKLOADS ................................................................................................................... 5

2.1 DAY POSTING WORKLOAD.............................................................................................. 5 2.2 STANDALONE ENQUEUE TEST......................................................................................... 6

3 CONFIGURATIONS........................................................................................................... 7 3.1 HARDWARE ENVIRONMENT ............................................................................................. 7 3.2 SOFTWARE ENVIRONMENT ............................................................................................. 8

4 TEST SCENARIOS AND RESULTS................................................................................ 11 4.1 TEST SCENARIOS......................................................................................................... 12 4.2 DAY POSTING MEASUREMENT RESULTS ........................................................................ 13 4.3 STANDALONE ENQUEUE TEST RESULTS ........................................................................ 14

5 ANALYSIS ....................................................................................................................... 15 5.1 SYSTEM PERFORMANCE............................................................................................... 15 5.2 ENQUEUE PERFORMANCE ............................................................................................ 19 5.3 COUPLING FACILITY PERFORMANCE.............................................................................. 22

6 CONCLUSIONS............................................................................................................... 25 7 APPENDIX ....................................................................................................................... 26 8 REFERENCES................................................................................................................. 27

Figures: Figure 1: Diagram of Enqueue in Coupling Facility ............................................................................................ 4 Figure 2: Hardware Environment .......................................................................................................................... 8 Figure 3: zEC12 LPAR Mapping.......................................................................................................................... 11 Figure 4: CPU Utilization...................................................................................................................................... 15 Figure 5: ITR.......................................................................................................................................................... 16 Figure 6: End User Response Time.................................................................................................................... 17 Figure 7: ETR in terms of Dialog Steps per Second ......................................................................................... 18 Figure 8: ETR in terms of Postings per Hour .................................................................................................... 18 Figure 9: Total Enqueue Requests ..................................................................................................................... 19 Figure 10: Average SAP Enqueue Lock Time per Request.............................................................................. 20 Figure 11: SAP Enqueue Requests/second as measured in Standalone Enqueue Tests ............................ 21 Figure 12: Average Coupling Facility Note Pad Service Sync Request Time ................................................ 22 Figure 13: Coupling Facility Utilization .............................................................................................................. 23 Figure 14: Coupling Facility Storage Used ........................................................................................................ 24 Tables: Table 1: Day Posting Measurement Results....................................................................................................... 13 Table 2: Standalone Enqueue Test Results........................................................................................................ 14 Table 3: ITR and Savings Calculations ............................................................................................................... 26



1 Introduction The joint IBM® and SAP®

development team introduced a new solution that further enhances the availability characteristics of one of the most business-critical components of an SAP system, the SAP enqueue server. There are two SAP enqueue replication mechanisms available on z/OS®, the traditional TCP/IP - based replication that requires a replication server on a separate LPAR and the newly introduced SAP Enqueue Replication into System z® Coupling Facility. With the new replication mechanism, an SAP enqueue server running on z/OS can now exploit the System z Coupling Facility (CF) as persistent storage for replication of its enqueue table. This new design point eliminates the need to send replication requests to an enqueue replication server on a remote host; therefore, eliminates the need for a separate SAP enqueue replication server. In case of a failover, the enqueue server can be restarted manually or through automation software on another LPAR within the sysplex and will automatically read the replication data from the CF. A recently introduced z/OS feature (APAR OA38450) was designed and implemented specifically with SAP enqueue services in mind; the XCF Note Pad service. This service is exploited by the SAP enqueue server to store the actual SAP replication information in the CF. This special catalog structure SYSXCF_NPCATALOG is necessary to enable the Note Pad functionality along with additional structure definitions that contain the actual SAP replication information.

Figure 1: Diagram of Enqueue in Coupling Facility. ENQ: SAP Enqueue Server; REPL: SAP Enqueue Replication Data

The advantages of this solution are simplicity and robustness. The high availability configurations, as well as the automation policy are simplified. Since the Coupling Facility is



accessible from every LPAR in the sysplex, the enqueue server can be (re-)started anywhere, as illustrated in Figure 1. The high load test in an SAP Banking environment showed superior robustness and throughput characteristics. This new implementation requires a small footprint in the Coupling Facility and results in lower overall resource consumption. This paper will describe the performance of the new enqueue replication data in the Coupling Facility solution compared to standard TCP/IP based replication server. The System z SAP Performance and Development teams performed the tests at the IBM Systems & Technology Group Systems Solution Development Laboratory in Poughkeepsie, NY USA. The System z SAP Performance Lab in Poughkeepsie, NY USA executed the SAP Banking Day Posting workload [1] and the Boeblingen Development Lab performed the standalone enqueue tests.

2 Workloads Two separate and distinct tests were used to measure the performance benefit of the Enqueue Replication into System z Coupling Facility. The tests used the SAP Day Posting workload and the SAP enqt tool [2]. The workloads are described in the following sections.

2.1 Day Posting Workload As part of IBM’s continuous testing and measuring of its products, IBM has included testing and measuring SAP’s core banking application. For several years, IBM has gained experience with the banking application as it has evolved. This paper focuses on the Day Posting business scenarios, which involve transactions moving money into and out of accounts, displays, and some on-line activities. A posting is a deposit or withdrawal from a customer’s account. Typical examples of a posting are a payment out of an account or a deposit into an account. Postings can occur anytime, day or night, especially as more Internet banking takes place. However, most postings take place during first shift. While they can occur every day, they are most frequent on business days. The workload was developed by SAP to simulate customer environments. The workload consists of interactive “users” going through repetitive cycles of 15 dialogue steps. The vast majority, about 86% of postings, are executed as mass postings through an adaptor via the SAP Business Application Programming Interface (BAPI®). This is intended to reflect the fact that most postings occur through electronic data interchanges (EDI) from other financial institutions. For example, one bank sending a group of deposits and withdrawals to another bank. There are other individual payments posted as well as several other types of transactions done via dialogue processing. In a workload loop, 175 postings are executed. The whole cycle uses many accounts to avoid creating “hot spots” in the data. The key metric is postings per hour.



2.2 Standalone Enqueue Test . The purpose of the standalone enqueue test is to measure enqueue rates that are not influenced by any SAP system activity. The tests use the standard SAP standalone enqueue test tool called enqt that is delivered with the SAP kernel. The program enqt initiates client connections to the SAP Enqueue Server and triggers SAP enqueue lock operations. The program can be found in the /usr/sap/<SID>/SCS<instance number>/exe directory. Refer to the command line help of the program for more information about how to start it, as well as the different OpCodes and options that you can execute by typing enqt –help on the command line. For this test, the following OpCodes were used:

OpCode Description 8 Live Monitoring of Enqueue Statistics

10 Emulate SD Benchmark Load The tests are executed with the enqt tool without the SAP system running. To generate an enqueue load on the system, the following command with OpCode 10 was run on the SAP application servers:

enqt pf=<central services profile> 10 300 This enqt command generates an enqueue workload that is similar to an SAP SD benchmark workload during a time interval of 300 seconds. For every test case the number of parallel invocations of enqt is increased up to the point where no further increase in the enqueue rate could be observed. At a maximum, 36 enqt instances per AIX machine were started in parallel. The enqueue rate is measured by running the following enqt command on the LPAR where the enqueue server was running. This command using OpCode 8 prints out enqueue statistics every 5 seconds and shows the accumulated enqueue Ops/min rate for different time intervals:

enqt pf=<enq_server_profile> 8 5 The maximum Ops/min rate that is reported as a 1-minute average from a test run is used as the enqueue rate. Using the reported test results, divide the Ops/min by 60 to obtain the enqueue requests per second.



3 Configurations A test environment was designed at the IBM Systems & Technology Group Systems Solution Development Laboratory in Poughkeepsie, NY USA, which simulated a customer’s SAP banking system in System z sysplex environment. The following sections discuss the hardware and software environments used for the SAP Day Posting and Enqueue Standalone tests. The hardware environment illustrated in Figure 2 was used for all tests.

3.1 Hardware Environment System z Database Server: Tests were performed on a zEC12 (2827-HA1) with 1.6 TB of memory. The system was configured with two z/OS LPARs each with 6 CPs and 140GB of memory. The z/OS systems were connected to two Coupling Facilities (CF) at CF level 18. Both CFs were configured with 2 dedicated ICFs and 64GB memory. Each z/OS system (S30 and S31) had three ICP channels connected to a local CF and two 12x IFB3 CIB channels connected to an external (referred to as remote) CF. The 12x IFB3 links were 10 meters in length. One CF is always local to a z/OS image and the other is a remote CF to that image. For the tests, the enqueue server ran on the z/OS image called S31; CF32 was the local CF; CF30 was the remote CF. Figure 3 illustrates the LPAR mapping of the zEC12 used in the test. Storage: The DASD used in the measurements consisted of two sets of dual frame IBM TotalStorage® DS8300 Model 2107-922/92E servers with 15K RPM disks configured as RAID 5. Each 2107-922/92E coupled unit has 128 GB regular cache, 4 GB non-volatile storage (NVS). Application Servers: 25 POWER7® model PS701 blades with 128GB memory were used as SAP application servers. Network: All network connections used 10 Gigabit Ethernet. The application servers were connected via a 10 Gigabit Ethernet switch to the zEC12 via eight short range (SR) ports using Open System Adapters (OSA) OSA-Express4S 10 GbE SR adapters.



zEC12 2827-HA1 with 1.6TB Memory

System z Database ServerTwo z/OS LPARs each with 6 dedicated

CPs and 140GB central memoryTwo internal coupling facilities each with

2 ICFs and 64GB central memoryCFCC Level 1812x IFB3 ChannelsICP Channels

Software:z/OS 1.13DB2 10

10Gbe Network Switch

P7 Blade Centers

Application Servers:Power7 Model PS701 128GB memorySoftware:DB2 Connect v9.7.0.6SAP NW kernel 721.38AIX 7.1 TL1 SP4

IBM System TotalStorageDS8300 2107-922/92E128GB Cache 4GB non-volatile storage (NVS)

FICON Express8S

OSA-Express4S 10GbE SR adapters

SAP Enqueue Replication Services on System zHardware Landscape

Figure 2: Hardware Environment

3.2 Software Environment z/OS

z/OS release 01.13.00 (R1.13) with APAR OA38450 installed [3] CFLEVEL 18 DB2® 10 dated June 2012 [4,5] Note: The z/OS TCP/IP profile parameter UDPQUEUELIMIT should be set, thereby limiting the number of UDP packets that can be queued.



Coupling Facility Structure Definitions

STRUCTURE NAME(IXCNP_SAPSB750) SIZE(204800) PREFLIST(CF30,CF32) STRUCTURE NAME(SYSXCF_NPCATALOG) SIZE(36000) PREFLIST(CF30,CF32)

AIX® AIX 7.1 TL1 SP4 (7100-01-04-1216) IBM DB2 Connect™

Informational tokens are “DB2 v9.7.0.6”, “s120629”, “IP23414”, and Fix Pack “6”. DB2 Connect driver version 09.07.0006

SAP Application Levels

SAP NetWeaver® 7.1

Release Patch Level Name Component

200 0007 FSAPPL SAP Banking Services

2005_1_710 0004 ST-PI SAP Solution Tools Plug - In

2006_1_710 0013 PI_BASIS PI_BASIS 2006_1_710

700 0007 SEM-BW SEM-BW 700: Add-On Installation

700 0017 FINBASIS Financial Basis

710 0013 SAP_ABAP Cross-Application Component

710 0013 SAP_BASIS SAP Basis Component

710 0013 SAP_BW SAP NetWeaver BI 7.1

720 114 SAP_Kernel SAP Kernel SAP z/OS Central Services (including Enqueue)

Release PatchLevel Name Component

721 0038 SAP_Kernel SAP Kernel



SAP Profile Parameters The new SAP profile parameters for Enqueue Replication Data into the Coupling Facility are:

enque/server/replication_local = true enque/server/replication_dll = repstzoscf.so

The implementation and profile parameter details are documented in SAP CSS note: Note 1753638 - z/OS: Enqueue Replication into System z Coupling Facility [6]



4 Test Scenarios and Results The project used a zEC12 processor and executed the ENQ test scenarios with the Day Posting workload in a 2-member data sharing configuration. For the test landscape, each scenario used two z/OS LPARs and two CF’s in the same zEC12 CEC with each z/OS LPAR connected to a “local” CF via ICP links and to a “remote” CF via CIB links.

zEC12 LPAR Mapping

zEC12 2827-HA1 with 1.6 TB memory

S306 CPs

140GB memory

DB2

S316 CPs

140GB memory

DB2

ENQ server

CF302 ICFs

CF Level 18

CF322 ICFs

CF Level 18

ICP Channels ICP Channels

12x IFB3 CIB

12x IFB3 CIB

Figure 3: zEC12 LPAR Mapping



4.1 Test Scenarios Four scenarios were defined for the Day Posting and Standalone Enqueue tests. They were:

Scenario A allows measuring the theoretical maximum enqueue rate that can be achieved on the given hardware. For SAP production systems it does not represent a valid high availability (HA) configuration since enqueue locks will be lost if an outage occurs. Scenario B is the traditional replication mechanism where the enqueue replication server is started on a different LPAR than the enqueue server and where both communicate via TCP/IP. Scenarios C and D use the new functionality in z/OS 1.13 and the SAP 7.21 Kernel where the SAP enqueue server writes its replication information directly into the z/OS Coupling Facility storage.

The four measurement scenarios executed to illustrate the SAP Enqueue Replication into System z Coupling Facility performance using the Day Posting Workload are documented in Section 4.2, Table 1 as:

Scenario A) Enqueue services running on USS without USS replication server Scenario B) Enqueue services running on USS with replication server Scenario C) Enqueue services running on USS with ENQ CF replication data on local

Coupling Facility Scenario D) Enqueue services running on USS with ENQ CF replication data on remote

Coupling Facility Three measurement scenarios were performed using the standalone enqueue tool. These tests are documented in Section 4.3, Table 2 as:

Scenario A) Enqueue Server without replication Scenario B) Enqueue Server with TCP/IP replication Scenario C) Enqueue Server with ENQ CF replication data on remote Coupling Facility



4.2 Day Posting Measurement Results

Scenario A Scenario B Scenario C Scenario D Runid S21114M1 S21110M1 S21112M1 S21113M2

Description ENQ on USS w/o replication

ENQ on USS with replication

ENQ on Local Coupling Facility

ENQ on Remote Coupling Facility

DBserver LPAR 2x zEC12-6w 2x zEC12-6w 2x zEC12-6w 2x zEC12-6w

Data Sharing 2way 2way 2way 2way

# of users 4,800 4,800 4,800 4,800

Resp Time (sec) 1.393 1.590 1.507 1.527

ETR (DS/sec) 418.19 411.47 414.53 413.39

Posting per hour 17,563,980 17,281,740 17,410,260 17,362,380

CPU util. (S30) 64.07% 66.07% 63.89% 62.22%

CPU util. (S31) 85.19% 86.57% 86.51% 87.86%

Avg CPU util. 74.63% 76.32% 75.20% 75.04%

ITR (DS/sec) 560.35 539.14 551.24 550.89

Total Lock Time (usec) 65,371,097 1,062,831,375 937,503,463 967,635,299

Total ENQ requests 9,986,761 9,986,793 9,986,710 9,986,709

Avg SAP ENQ Lock Time/req (usec) 6.5 106.4 93.9 96.9

Avg CF ENQ service (usec) n/a n/a 3.3 8.3

CF util. (CF30) 30.5% 30.1% 30.4% 31.9%

CF util. (CF32) 21.1% 20.6% 22.1% 20.8%

Avg CF utilization 25.8% 25.4% 26.3% 26.4%

CF30 storage used (MB) 43,248 43,248 43,248 43,484

CF30 storage used (MB) 35,312 35,312 35,548 35,312

Total Storage Used (MB) 78,560 78,560 78,796 78,796

Configuration ENQ on USS

w/o repl (1 local CF + 1 remote

CF)

ENQ on USS w/repl (1 local CF + 1 remote

CF)

ENQ on local CF (1 local CF +

1 remote CF)

ENQ on remote CF (1 local CF +

1 remote CF)

Table 1: Day Posting Measurement Results



4.3 Standalone Enqueue Test Results Scenario A Scenario B Scenario C

Description ENQ w/o Replication

ENQ w/TCPIP Based Replication

ENQ w/CF Based Replication

(Remote CF) SAP enqueue requests/second 47,500 28,333 34,166 SAP Profile Parameters Enqueue Table Size 102,400 102,400 102,400 enque/server/max_requests [7] 5,000 65,000 5,000 enque/ni_queue_size 100 (default) 8,000 100 (default)

Table 2: Standalone Enqueue Test Results As a note of interest on the TCP/IP based scenario, the enque/ni_queue_size needed to increase from its default of 100 to 8000 in both the SAP central services instance profile and the SAP enqueue replication instance profile. Low values for this parameter have been known to cause network connection suspensions between the enqueue server and replication server. This makes the ENQ replication on CF an attractive HA solution, since the potential loss of failover capability is eliminated.



5 Analysis This section describes the system, enqueue, and coupling facility performance based on the analysis of the series of tests that were executed. For discussion purposes, percent ranges are used to represent ITR and CPU savings. A detailed account of the ITR and savings calculations can be found in the Appendix.

5.1 System Performance In our two member data-sharing configuration, the database server load is symmetrically split between the two z/OS LPARs (S30 and S31). However, S31 also provides the additional ENQ service support, which is about 14% to 17% of the total database server load. As observed from the graph in Figure 4, S31 CPU utilization is higher than S30. This difference is mainly due to the additional enqueue server on S31. Enqueue replication incurs a slight 2% to 4% CPU cost regardless of whether replication is implemented via the conventional TCP/IP protocol or using Coupling Facility (CF) structures. Using the new ENQ on CF implementation can provide a CPU savings by eliminating the active ENQ replication server used by the conventional TCP/IP protocol implementation. This can be clearly observed by comparing the S30 CPU utilization for “ENQ on USS with Replication” where there is a 2% higher CPU utilization than the “ENQ on USS without replication” scenario. This difference is due to the active enqueue replication server activity.

CPU Utilization Per z/OS

64.07% 62.22%63.89%66.07%

87.86%85.19% 86.57% 86.51%

75.04%75.20%76.32%74.63%

0%

25%

50%

75%

100%

Enq on USS withoutreplication

Enq on USS withreplication

Enq on LocalCoupling Facility

Enq on RemoteCoupling Facility

CPU

Util

izat

ion

S30 CPU - DB Only S31 CPU - DB and Enq Server Average CPU Utilization

Figure 4: CPU Utilization



LSPR methodology [8] using Internal Throughput Rate (ITR) is better suited for CPU utilization comparison for our enqueue test scenarios as illustrated in Figure 5. The ITR can be compared across all scenarios. ENQ on USS with replication has a 4% ITR costs (lower ITR) than ENQ on USS without replication. Quantifiable CPU savings in the form of increased ITR are shown over the conventional ENQ with replication on USS for both the local and remote ENQ configurations since the ITR cost for this replication method is only 2%. Enqueue replication incurs a CPU cost regardless of the implementation. However, the added CPU cost associated with ENQ on CF is only half that of the traditional ENQ with replication on USS. The CPU savings are achieved primarily through the elimination of the enqueue replication server plus its associated network traffic processing. For enqueue replication in the CF, there are a CPU and response time improvements using local vs. remote CF. However, local CF is not a traditional customer HA configuration. Nevertheless, both remote and local ENQ on CF cases have shown quantifiable CPU savings and improved response time over the traditional ENQ with replication on USS.

Internal Throughput Rate (ITR)

550.89551.24539.14560.35

0

100

200

300

400

500

600





ITR

(D

ialo

g St

ep /

Seco

nd)

Figure 5: ITR



Besides CPU cost, enqueue replication also incurs response time degradation due to enqueue lock delays as illustrated in Figure 6. However, the response time degradation is noticeably smaller for ENQ on CF than ENQ on USS with replication. For ENQ on CF, the difference between local versus remote CF is only 0.02 seconds.

End User Response Time

1.3931.5271.5071.590

0.000.200.400.600.801.001.201.401.601.80





End

Use

r Res

pons

e Ti

me

(sec

)

Figure 6: End User Response Time



Since the same user load is executed for all scenarios in the ENQ study, better response time translates into better External Throughput Rate (ETR). The two graphs in Figure 7 and Figure 8 illustrate ETR in two metrics - Dialog Steps per Second and Postings per Hour.

External Throughput Rate (ETR)Dialog Steps/Second

411.47 414.53 413.39418.19

050

100150200250300350400450500





ETR

(D

ialo

g St

ep /

Seco

nd)

Figure 7: ETR in terms of Dialog Steps per Second

External Throughput Rate (ETR)Postings Per Hour

17,281,740 17,410,260 17,362,38017,563,980

02,000,0004,000,0006,000,0008,000,000

10,000,00012,000,00014,000,00016,000,00018,000,00020,000,000





ETR

Post

ings

Per

Hou

r

Figure 8: ETR in terms of Postings per Hour



5.2 Enqueue Performance As discussed, overall ITR improved by approximately 2%. The actual ITR improvement for the enqueue replication using the newly introduced Enqueue replication data into the Coupling Facility feature versus the traditional TCP/IP replication server is in the 13% to 15% range since the enqueue server is 14% to 17% of the total database server load. The total number of enqueue lock requests was collected at the end of each measured scenario. We observed a consistent number of lock requests for all scenarios as illustrated in Figure 9. This demonstrates that the posting workload has relatively equivalent enqueue request activity in all our scenarios.

Total Enqueue Requests

9,986,761 9,986,7099,986,7109,986,793

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000





Enqu

eue

Req

uest

s

Figure 9: Total Enqueue Requests



Enqueue replication causes delays in enqueue lock processing. As observed in Figure 10, the enqueue lock delays are in microseconds, which is minuscule for workloads when dialog step response time is in seconds. Without any enqueue replication, the enqueue server is a single point of failure in the SAP system landscape; its inadvertent failure could cause a prolonged system outage. Therefore, the longer enqueue lock delay with enqueue replication is a small price to pay for the enhanced HA aspects. Comparing the newly introduced Enqueue Replication into the System z Coupling Facility to the traditional TCP/IP replication server, the average enqueue lock time is reduced by 10% as shown in Figure 10.

Average SAP Enqueue Lock Time

106.493.9 96.9

6.5

0

20

40

60

80

100

120





Ave

rage

SA

P En

queu

e Lo

ck T

ime

per R

eque

st

(usec)

Figure 10: Average SAP Enqueue Lock Time per Request



As demonstrated by the standalone enqueue tests and illustrated in Figure 11, the SAP enqueue rates improved approximately 20% when using the new enqueue on Coupling Facility replication services. For availability reasons it is recommended that customers run their SAP system with SAP enqueue replication. With enqueue replication enabled, SAP systems can recover from an outage without any loss of SAP enqueue locks. Improving enqueue replication request times helps to improve the maximum throughput rate the SAP enqueue server can deliver.

SAP Enqueue Requests/Second

47,500

28,33334,166

0

10,000

20,000

30,000

40,000

50,000

ENQ w/o Replication ENQ w/TCPIP BasedReplication

ENQ w/CF BasedReplication (Remote CF)

SAP

Enqu

eue

Req

uest

s / s

econ

d

Figure 11: SAP Enqueue Requests/second as measured in Standalone Enqueue Tests



5.3 Coupling Facility Performance The Enqueue Replication into System z Coupling Facility exploits the System z Coupling Facility new feature called the XCF Note Pad catalog structure [3]. Therefore, CF performance has direct bearing on the new enqueue replication implementation. As measured on the zEC12 processor in our study, synchronous enqueue CF request time to the local CF is 3.3 microseconds whereas it is 8.3 microseconds to the remote CF as illustrated in Figure 12. This difference in service time is primarily due to the difference in the coupling link technology.

Average CF Note Pad Service Sync Requests

8.3

3.3

0

1

2

3

4

5

6

7

8

9

10

Enq on Local Coupling Facility Enq on Remote Coupling Facility

Not

e Pa

d Se

rvic

e Sy

nc R

eque

sts

/T

ime

(use

cs)

Figure 12: Average Coupling Facility Note Pad Service Sync Request Time

Although the synchronous CF Note Pad service time is noticeably better for the local CF than the remote CF, overall ITR and response time deltas between them are rather small (discussed previously in section 5.1). For customer environments, enqueue replication data on local CF is more suitable for test and development systems. For production environments, it is not recommended to keep the local CF enqueue replication data and the enqueue server on the same physical CEC, as this creates a single point of failure.



Coupling Facility Utilization

30.5% 31.9%30.4%30.1%20.8%21.1% 20.6% 22.1%

26.4%26.3%25.4%25.8%

0%

25%

50%

75%

100%





CF

Util

izat

ion

CF30 Utilization CF32 Utilization Average CF Utilization

Figure 13: Coupling Facility Utilization

The CF resources consumed by the enqueue CF requests are rather small - total CF utilization with the ICF's increased by less than 1.5% with a very small CF memory footprint increase of 236MB as illustrated in Figure 13 and Figure 14.



As illustrated in Figure 14, the Coupling Facility storage used for the enqueue replication only increased by 236MB regardless of configuration. Prior to the enqueue Note Pad structure being introduced, the CFRM policy consisted of database and system structures for a total CF storage requirement of 78,560MB. For sizing guidelines on the enqueue replication Note Pad structure, see the pdf attachment of the SAP CSS Note 1753638 - z/OS: Enqueue Replication into System z Coupling Facility. .

Coupling Facility Storage Used

43,248 43,248 43,48443,24835,54835,31235,312 35,312

78,560 78,560 78,796 78,796

0

10,000

20,00030,000

40,000

50,000

60,000

70,00080,000

90,000

100,000





CF

Stor

age

MB

CF30 Storage (MB) in Use CF32 Storage (MB) in Use Total CF Storage in Use (MB)

Figure 14: Coupling Facility Storage Used



6 Conclusions Customers, who are already using enqueue replication on USS and are in a parallel sysplex, should seriously consider switching to the new replication mechanism that uses the System z Coupling Facility. From a performance perspective, when comparing enqueue replication data on the Coupling Facility to the traditional TCP/IP enqueue replication server, the anticipated benefits are:

• Enqueue CPU savings ranging from 13% to 15% • Approximately 2% savings in ITR • 10% improvement in enqueue lock times • Minimal impact on the Coupling Facility Utilization • Small Coupling Facility memory requirement

To recap the design, the enqueue replication mechanism for z/OS no longer requires a replication server. In the past, the standard mechanism for ensuring high availability for the enqueue server was running an enqueue replication server on a remote host. The Enqueue Server would replicate its lock data via TCP/IP to an enqueue replication server. The replication server wrote the data received from the enqueue server into a replication table. In case of a failover, the enqueue server moved to the host where the replication server was running and then read the data from the replication table to reestablish its enqueue table.

This new alternative solution is based on the fact that several z/OS hosts (LPARs) can be organized in a z/OS sysplex and share data via the System z Coupling Facility (CF). Instead of sending requests to an enqueue replication server on a remote host, the enqueue server writes the replication data directly to the CF, thereby eliminating the need for a separate USS based replication server. In case of a failover scenario, the enqueue server can be restarted manually or via automation on any LPAR within the sysplex and read the replication data directly from the CF. The advantages of this solution are simplicity and robustness. Our enqueue tests show that the new enqueue replication service on the Coupling Facility has significant advantages compared to alternative scenarios:

- replication is active at all times ensuring full failover capability - standard SAP profile parameter settings can be used - maximum enqueue rates are higher than with TCP/IP-based replication

The high load tests that were conducted in the SAP Banking environment showed superior robustness and throughput characteristics. This new implementation requires a lower overall resource consumption while improving performance and reliability for enqueue replication services.



7 Appendix The below table represents the calculations derived from the SAP Enqueue Replication into System z Coupling Facility performance test using the Day Posting workload.

ETR (DS/sec)

CPU util.

(S30)

CPU util.

(S31)

Avg CPU util.

ITR (DS/sec)

ENQ Server:

CPU Portion of DB Server

ENQ REPL on CF

Savings

ITR Savings

Compared to TCPIP

replication

ITR Delta: ENQ

ReplicationCost

Enq on USS without replication

418.19 64.07% 85.19% 74.63% 560.35 14.15% n/a n/a 0.0% (base)

Enq on USS with replication

411.47 66.07% 86.57% 76.32% 539.14 n/a 0.0% (base)

0.0% (base) 3.79%

Enq on Local Coupling Facility

414.53 63.89% 86.51% 75.20% 551.24 15.04% 14.92% 2.24% 1.63%

Enq on Remote Coupling Facility

413.39 62.22% 87.86% 75.04% 550.89 17.08% 12.76% 2.18% 1.69%

Table 3: ITR and Savings Ca



8 References [1] SAP AG 2011. SAP for Banking on System z Reference Architecture http://www.sdn.sap.com/irj/sdn/db2?rid=/library/uuid/a00e4718-314f-2b10-19a6-a76f257addaf. [2] SAP ENQT Tool provided with every SAP kernel release. [3] IBM Corp 2012. APAR OA38450. http://publibz.boulder.ibm.com/z/OSlib/pdf/OA38450.pdf. [4] IBM Corp. 2010. DB2 10 for z/OS http://www.ibm.com/software/data/db2/z/OS/. [5] IBM Corp. 2010. DB2 10 for z/OS Technical Overview http://www.redbooks.ibm.com/redbooks/pdfs/sg247892.pdf. [6] SAP AG 2012. SAP note 1753638: z/OS Enqueue Replication into System z Coupling Facility. [7] SAP AG 2012. SAP note 920979: Out of memory in standalone enqueue server. [8] IBM Corp. 2012. Large Systems Performance Reference. Document Nbr SC28-1187-16. https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?OpenDocument. [9] IBM Corp. 2012. IBM System zEnterprise, System Storage, and DB2 10 for z/OS: SAP Banking Services 7.0 150 Million Accounts Measurements http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101978.

http://www.sdn.sap.com/irj/sdn/db2?rid=/library/uuid/a00e4718-314f-2b10-19a6-a76f257addaf

http://publibz.boulder.ibm.com/z/OSlib/pdf/OA38450.pdf

http://www.ibm.com/software/data/db2/z/OS/

http://www.redbooks.ibm.com/redbooks/pdfs/sg247892.pdf

https://www-304.ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprindex?OpenDocument

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101978

Documents

IBM zEnterprise System · PDF filemake any such changes without ... which involve transactions moving money into and out ... different OpCodes and options that you can execute by typing