s 2528 Ga 091442

8/12/2019 s 2528 Ga 091442

http://slidepdf.com/reader/full/s-2528-ga-091442 1/20

© 2008 IBM Corporation

Understanding and MonitoringParallel SysplexPerformance

Speaker Name: Glenn Anderson, IBM Training

TrademarksThe following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.

The following are trademarks or registered trademarks of other companies.

* All other products may be trademarks or registered trademarks of their respective companies.

Notes:Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will varydepending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that anindividual user will achieve throughput improvements equivalent to the performance ratios stated here.IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.

All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costsand performance characteristics will vary depending on individual customer configurations and conditions.This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult yourlocal IBM business contact for information on the product or services available in your area.

All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any otherclaims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.

Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom.Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its

subsidiaries in the United States and other countries.UNIX is a registered trademark of The Open Group in the United States and other countries.Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency, which is now part of the Office of Government Commerce.

For a complete list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:

*, AS/400®, e business(logo)®, DBE, ESCO, eServer, FICON, IBM®, IBM (logo)®, iSeries®, MVS, OS/390®, pSeries®, RS/6000®, S/30, VM/ESA®, VSE/ESA, WebSphere®,xSeries®, z/OS®, zSeries®, z/VM®, System i, System i5, System p, System p5, System x, System z, System z9®, BladeCenter®

Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is notsignificant within its relevant market.

Those trademarks followed by ® are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.

08/07/08 (c) Copyright IBM Corporation 2008 1

8/12/2019 s 2528 Ga 091442


SoftwareCoupling Facility Control Code

z/OS and subsystemsXCF

XES

Architecture

Structures (Lock, List andCache)

Commands

Hardware

Coupling Facility

CPs (shared , dedicated)

Links

high speed linksStorage Size

Coupl ing Technology

CF

What about the Sysplex Components inPerformance ?

LOCKS DATA

BUFFERSLOCKSDATA

BUFFERS

MVS Sysplex

Services

Coupling

Technology

REQUESTS REQUESTS

Multi-Systems

- serialization

-

modifieddata

MVS

S/390

MVS

S/390

MicrosecondsMicroseconds

Milliseconds

NanosecondsNanoseconds

DATABASE

MANAGERS

DATABASE

MANAGERS

Caches

Lock Bits

Global Buffer Directory

Parallel Sysplex Environment


8/12/2019 s 2528 Ga 091442


Coupling Facility Technology

Shared ICF

DELTED_NAMEsupport

Initial CF

HW andFunctions

Cache Vector indexComparison

Enhancement

Dynamic Alter CICS TSQSystem Logger

Level 0

Level 1

List KEY andList cursor

Enhancements

Level 1

Register Name List andBatch Unlock

Level 2

Level 2

StructureEnhancements

(255/Structure &1023 Structure/CF)

Level 2

IMS SharedMessage Queue

Base

Level 3

DB2 CACHEDuplexingDynamic ICFExpansion

Level 5Level 6

Level 7

Level 9

List StructureEnhancementsMulti-SystemEnclaves

Dynamic CF

DispatchICF Support

Level 4

IC ChannelICBTPF support

Level 8

Dynamic ICFExpansion intoshared ICFsSystem ManagedRebuild

Level 10 - z900

Level 11 - SM Duplexing

Level 12 - z990 64-bit CFCC AddressibilityLevel 13 - DB2 castout enhancementsLevel 14 - CFCC

Dispatcher RestructureLevel 15 - Increasing tasks RMF data

enhancements

Tuner's View of the Coupling Facility

3 different types of structure

LIST (and "Serialized List")

LOCK

CACHE

Each exploiter has a different implementation

Beware of general concepts like "good or bad"

Use the performance data provided by the 'exploiter

Make a general review of workloads and rates

Are the CF accesses equivalent?

Are the "service times" equivalent?

Are there any indicators of delay of type links or subchannels?

What are the CPU indicators?


8/12/2019 s 2528 Ga 091442


CF Request Types

Synchronous immediate

Lock type accessSynchronous non-immediate

Cache type access (transfers of data <4K*)

Asynchronous

Cache type access (transfers of data >4K *)

CF Request Types...

Synchronous immediate

Delay Time and Service time are included in CPU busy time

Synchronous non-immediate

Service time included in CPU busy time (if not converted)

Can be converted to Asynch request

can be queued on a subchannel if no subchannel available

Asynchronous

Can be queued on a subchannel if no subchannel available orredriven if no path available


8/12/2019 s 2528 Ga 091442


CFCECLink

Path

z/OS

z/OS

Software (z/OS, XES)

Subchannels

Link Buffers

Link

Path

2

1

CPs

Elements of the CF Model

Compat Mode - 2 buffers per link

Peer Mode - 7 buffers per link

CF: Measurements Basic

z/OS CF

CP

a

bc Service Time

DelayTime

Link/Pathd

Async

Sync

Subchannel

SUBCHANNEL ACTIVITY

-------------------------------------------------------------------------------------

----------- REQUESTS ----------- ------------------ DELAYED REQUESTS --------------

# -SERVICE TIME(MIC)- # % OF ------ AVG TIME(MIC) ------

REQ AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

SYNC 247394 38.7 22.3 LIST/CACHE 13 0.0 15.4 6.5 0.0

ASYNC 688869 145.6 392.5 LOCK 0 0.0 0.0 0.0 0.0

CHANGED 6 INCLUDED IN ASYNC TOTAL 13 0.0

UNSUCC 0 0.0 0.0

ab -d

c

08/07/08 (c) Copyright IBM Corporation 2008 9-1

8/12/2019 s 2528 Ga 091442


8/12/2019 s 2528 Ga 091442


CF Partition Options - What are theDifferences?

Standalone CF

dedicated processor(s)

best for productionshared processor(s)

test or non-data-sharing production

dynamic CF dispatching

competition from other partitions

ICF (Internal Coupling Facility)

dedicated or shared processor(s)

see above

optional shared processor(s) expansionavailability considerations

addressed by CF duplexing

Internal Coupling Facility (ICF)

CFz/OS z/OS z/OS

PR/SMIC

z/OS Spare CPs can be used as CF CPs

CF CPs ONLY run Coupling Facility Microcode

CF MIPS excluded from software costs

Can run as IC Channel or accessed via CouplingLinks


8/12/2019 s 2528 Ga 091442


System Managed CF Structure Duplexing

z/OS receives CF exploiter's update request and splits it into two distinct requests, onedestined for each of two structures

Two CF's stay in synch by signal exchange over a CF to CF coupling link

Return command responses to z/OS system merged into a single consolidated response

Works for either synch or asynch and is transparent to exploiter

1.Request in

7.Response out

XESExploiter

z/OS

2a.Split req out

2b.Split req out

6a. Response6b. Response

4.Update 4.Update

CF01 CF02

3a+b. "Ready to start"

5a+b. "Ready to end"

(split/merge)

------------------------------------------------------------------------------------------------------

COUPLING FACILITY NAME = CF2

------------------------------------------------------------------------------------------------------ STRUCTURE NAME = DSNDB1G_LOCK1 TYPE = LOCK STATUS = ACTIVE PRIMARY

# REQ -------------- REQUESTS ------------- ------------- DELAYED REQUESTS -----------

SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF --- AVG TIME(MIC) ---

NAME AVG/SEC REQ ALL AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

JA0 179K SYNC 163K 91.5 108.5 137.6 NO SCH 0 0.0 0.0 0.0 0.0

99.28 ASYNC 15K 8.5 483.9 474.0 PR WT 0 0.0 0.0 0.0 0.0CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 47K 26.1 24.4 99.5 6.4

------------------------------------------------------------------------------------------------------

COUPLING FACILITY NAME = CF3

------------------------------------------------------------------------------------------------------

STRUCTURE NAME = DSNDB1G_LOCK1 TYPE = LOCK STATUS = ACTIVE SECONDARY

# REQ -------------- REQUESTS ------------- ------------ DELAYED REQUESTS ------------

SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF --- AVG TIME(MIC)----


JA0 177K SYNC 162K 91.4 103.8 147.8 NO SCH 0 0.0 0.0 0.0 0.0

98.52 ASYNC 15K 8.6 473.6 464.9 PR WT 177K 100 5.4 15.7 5.4

CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 131K 73.7 15.1 56.2 11.1

SUBCH

SUBCH

RSV SEND

PR WT SEND

PR CMPCF2

CF3

Queue time Service time

RMF- DB2 Lock, all requests duplexed


8/12/2019 s 2528 Ga 091442


Performance Issues for Duplexing

z/OS V1R8 SYSPLEX UTCPLXJ8 DATE 07/24/2006 INTERVAL 015.00.000 CONVERTED TO z/OS V1R8 RMF TIME 11.30.00 CYCLE 01.000 SECONDS

------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY NAME = CF1 ------------------------------------------------------------------------------------------------------------------------------ CF TO CF ACTIVITY ------------------------------------------------------------------------------------------------------------------------------ # REQ ----------- REQUESTS ----------- -------------- DELAYED REQUESTS -------------- PEER TOTAL -- CF LINKS -- # -SERVICE TIME(MIC)- # % OF ------ AVG TIME(MIC) ------ CF AVG/SEC TYPE USE REQ AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

CF2 18424K CBP 1 SYNC 18424K 14.3 4.2 SYNC 0 0.0 0.0 0.0 0.0 20471 CFP 3

CF3 5531K CFP 6 SYNC 5531K 15.4 4.0 SYNC 0 0.0 0.0 0.0 0.0 6145.2

System-managed duplexing performance issues:Cost to initiate and receive messages will increase for writes

Response time for updates will increase

CF utilization and link utilization will go up

Remember, you are gated by the slowest resource

Some estimates on the cost of duplexing

RMF provides a separate report on CF to CF link activity

Costs Storage z/OS CPU CF Links (time) CF CPU

User Managed (DB2 GBPs) 2x 2x 2x 2x

System Managed (Lock) 2x 4x 5x 8x

System Managed (List) 2x 3x 4x 6x

Estimate of amount of structure activity to be duplexed:Cache: 1% to 100% (typically 20%), Lock: 100%, List: near 100%

CF Coupling Link Performance

9672R64 to C04

2064114 to 1xx

20843xx to 3xx

2094109 to 2xx

SYNC 100 - 175 15 - 30 10 -25 7-20

ASYNC 500 - 1500 150 - 450 100-350 80-300

Range to account for amount of data being transferred

Low end - no data (ex. GRSLOCK)

High end - largest data transfer allowed ( SYNC - 4K, ASYNC - 64K)

Assumes fastest CF link technology available on that processor and a

well-tuned sysplex

What is the best possible response to expect

Optimization hints for managing CF linksMake sure you have enough, at least 2 per CPC to each CF

If using EMIF sharing, keep subchannel BUSY COUNTS < 10%

Check CF RMF reports for indicators of path busy conditions that canshow the necessity to add more

CFR links are reconfigurable on CFs C04/C05 and higher

Only one partition allowed in candidate list


8/12/2019 s 2528 Ga 091442


RMF Coupling Technology Support

Historical (Postprocessor)

Based on RMF SMF Data

Provides information for capacity planning and performanceanalysis

Supports:Coupling Facilities

Cross-System Coupling Facility (XCF)

Inter System Channel Activity

Real-Time (Monitor III)

Provides information for problem determination andperformance assurance

Who uses and/or is delayed for a parallel sysplex component

Supports:Coupling Facilities

XCF

CF: Postprocessor Report Content

CP CP...

RMF CF Report:CF=AA

Processor Summary

Storage Summary

Structure Summary

Structure Activity Section

Requests Rates

Service & Delay Times

Lock Contention

Cache Statistics

Usage Summary

Subchannel Activity SectionRequests Rates

Service & Delay Times

Path/Subchannel Busy

CF=AA

SYS1

SYS2

Channel Path Activity Report

IS Link Utilization

Shows information necessary tounderstand the utilization andexploitation of the CF !

Summary information gives anoverview,

Structure information goes into detail

Subchannel Activity shows CF usagesummarized by each system

Complemented by Channel Path

Activity Reports from each system


8/12/2019 s 2528 Ga 091442


Coupling Facility Usage Report (1)

C O U P L I N G F A C I L I T Y A C T I V I T Y PAGE 1 z/OS V1R8 SYSPLEX UTCPLXJ8 DATE 07/24/2006 INTERVAL 015.00.000 CONVERTED TO z/OS V1R8 RMF TIME 11.30.00 CYCLE 01.000 SECONDS

------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY NAME = CF2 TOTAL SAMPLES(AVG) = 897 (MAX) = 899 (MIN) = 893 ------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY USAGE SUMMARY ------------------------------------------------------------------------------------------------------------------------------ STRUCTURE SUMMARY

------------------------------------------------------------------------------------------------------------------------------

% OF % OF % OF AVG LST/DIR DATA LOCK DIR REC/ STRUCTURE ALLOC CF # ALL CF REQ/ ENTRIES ELEMENTS ENTRIES DIR REC TYPE NAME STATUS CHG SIZE STOR REQ REQ UTIL SEC TOT/CUR TOT/CUR TOT/CUR XI'S

LIST DSNDB1G_SCA ACTIVE 40M 0.7 27542 0.1 N/A 30.60 12K 108K N/A N/A PRIM 216 1906 N/A N/A IXCPLEX_PATH2 ACTIVE 47M 0.8 14915K 47.1 N/A 16572 10K 10K N/A N/A 1 37 N/A N/A

LIST CSLRMGR_PROD ACTIVE 25M 0.4 52357 0.2 N/A 58.17 54K 10K 256 N/A PRIM 16K 3590 0 N/A ISTGENERIC ACTIVE 9M 0.1 40945 0.1 N/A 45.49 26K 547 4 N/A PRIM 11K 34 0 N/A SYSARC_DFHSM_RCL ACTIVE 5M 0.1 1490 0.0 N/A 1.66 5005 4973 16 N/A PRIM 364 352 0 N/A

LOCK DSNDB1G_LOCK1 ACTIVE 73M 1.2 1642K 5.2 N/A 1824.8 128K 0 8389K N/A PRIM 6137 0 1877 N/A DSNDB2G_LOCK1 ACTIVE 17M 0.3 0 0.0 N/A 0.00 42K 0 2097K N/A SEC 0 0 0 N/A

CACHE DSNDB1G_GBP0 ACTIVE 29M 0.5 2944 0.0 N/A 3.27 64K 1491 N/A 0 PRIM 14K 1107 N/A 0 SYSIGGCAS_ECS ACTIVE 1M 0.0 608798 1.9 N/A 676.44 142 134 N/A 0 48 48 N/A 0 SYSZWLM_B52A2084 ACTIVE 20M 0.3 101298 0.3 N/A 112.55 3831 7657 N/A 0 PRIM 289 699 N/A 0 SYSZWLM_WORKUNIT ACTIVE 20M 0.3 0 0.0 N/A 0.00 3831 7662 N/A 0 SEC 0 0 N/A 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------- ------ ------- ----- ------ ------- STRUCTURE TOTALS 3G 46.4 31643K 100 N/A 35159

Coupling Facility Usage Report (2)

1 C O U P L I N G F A C I L I T Y A C T I V I T Y PAGE 5 z/OS V1R8 SYSPLEX UTCPLXJ8 DATE 07/24/2006 INTERVAL 015.00.000 CONVERTED TO z/OS V1R8 RMF TIME 11.30.00 CYCLE 01.000 SECONDS

------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY NAME = CF2 TOTAL SAMPLES(AVG) = 897 (MAX) = 899 (MIN) = 893 ------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY USAGE SUMMARY ------------------------------------------------------------------------------------------------------------------------------ STORAGE SUMMARY ------------------------------------------------------------------------------------------------------------------------------

ALLOC % OF CF ------- DUMP SPACE ------- SIZE STORAGE % IN USE MAX % REQUESTED

TOTAL CF STORAGE USED BY STRUCTURES 2820M 46.4 TOTAL CF DUMP STORAGE 879M 14.5 0.0 0.5 TOTAL CF STORAGE AVAILABLE 2378M 39.1 ------- TOTAL CF STORAGE SIZE 6077M

ALLOC % ALLOCATED SIZE

TOTAL CONTROL STORAGE DEFINED 6077M 60.9 TOTAL DATA STORAGE DEFINED 0K 0.0

PROCESSOR SUMMARY ------------------------------------------------------------------------------------------------------------------------------

COUPLING FACILITY 2094 MODEL S38 CFLEVEL 14

AVERAGE CF UTILIZATION (% BUSY) 31.8 LOGICAL PROCESSORS: DEFINED 3 EFFECTIVE 3.0


8/12/2019 s 2528 Ga 091442


Coupling Facility List Structure Report 1

1 C O U P L I N G F A C I L I T Y A C T I V I T Y PAGE 37 z/OS V1R8 SYSPLEX UTCPLXJ8 DATE 07/24/2006 INTERVAL 015.00.000 CONVERTED TO z/OS V1R8 RMF TIME 11.30.00 CYCLE 01.000 SECONDS ------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY NAME = CF2 ------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY STRUCTURE ACTIVITY ------------------------------------------------------------------------------------------------------------------------------

STRUCTURE NAME = DSNDB1G_SCA TYPE = LIST STATUS = ACTIVE PRIMARY # REQ -------------- REQUESTS ------------- -------------- DELAYED REQUESTS ------------- SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF ---- AVG TIME(MIC) ----- NAME AVG/SEC REQ ALL AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

JA0 5295 SYNC 1334 4.8 64.9 37.8 NO SCH 0 0.0 0.0 0.0 0.0 5.88 ASYNC 3961 14.4 222.2 1728.2 PR WT 2588 48.9 7.2 73.5 3.5 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 1290 24.4 39.4 269.2 9.6 DUMP 0 0.0 0.0 0.0 0.0

JB0 7334 SYNC 2635 9.6 62.2 36.0 NO SCH 0 0.0 0.0 0.0 0.0 8.15 ASYNC 4699 17.1 196.5 523.7 PR WT 3637 49.6 5.4 13.8 2.7 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 1845 25.2 30.4 190.3 7.6 DUMP 0 0.0 0.0 0.0 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------------------------------ TOTAL 27542 SYNC 11K 39.8 50.6 32.8 NO SCH 2 0.0 4966 6908 0.4 30.60 ASYNC 17K 60.2 220.4 912.7 PR WT 13K 48.8 5.0 33.3 2.4 CHNGD 3 0.0 PR CMP 6401 23.2 38.4 590.6 8.9 DUMP 0 0.0 0.0 0.0 0.0

STRUCTURE NAME = IXCPLEX_PATH2 TYPE = LIST STATUS = ACTIVE # REQ -------------- REQUESTS ------------- -------------- DELAYED REQUESTS ------------- SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF ---- AVG TIME(MIC) ----- NAME AVG/SEC REQ ALL AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

JA0 2223K SYNC 0 0.0 0.0 0.0 NO SCH 10 0.0 622.3 356.2 0.0 2470 ASYNC 2223K 14.9 113.6 487.5 PR WT 0 0.0 0.0 0.0 0.0 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 0 0.0 0.0 0.0 0.0 DUMP 0 0.0 0.0 0.0 0.0

JB0 2679K SYNC 0 0.0 0.0 0.0 NO SCH 294 0.0 524.6 1150 0.1 2977 ASYNC 2679K 18.0 99.5 356.4 PR WT 0 0.0 0.0 0.0 0.0 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 0 0.0 0.0 0.0 0.0 DUMP 0 0.0 0.0 0.0 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------------------------------ TOTAL 14915K SYNC 0 0.0 0.0 0.0 NO SCH 950 0.0 264.7 775.8 0.0 16572 ASYNC 15M 100 103.8 378.4 PR WT 0 0.0 0.0 0.0 0.0 CHNGD 0 0.0 PR CMP 0 0.0 0.0 0.0 0.0 DUMP 0 0.0 0.0 0.0 0.0


STRUCTURE NAME = DSNDB1G_LOCK1 TYPE = LOCK STATUS = ACTIVE PRIMARY # REQ -------------- REQUESTS ------------- -------------- DELAYED REQUESTS ------------- SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF ---- AVG TIME(MIC) ----- EXTERNAL REQUEST NAME AVG/SEC REQ ALL AVG STD_DEV REQ REQ /DEL STD_DEV /ALL CONTENTIONS

JA0 661K SYNC 630K 38.3 66.3 30.8 NO SCH 1 0.0 35.0 0.0 0.0 REQ TOTAL 722K 735.0 ASYNC 32K 1.9 203.1 512.5 PR WT 661K 100 4.6 20.4 4.6 REQ DEFERRED 1717 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 478K 72.3 8.7 84.7 6.3 -CONT 1461 -FALSE CONT 150

JB0 913K SYNC 903K 55.0 65.9 31.3 NO SCH 34 0.0 61.5 183.0 0.0 REQ TOTAL 948K 1015 ASYNC 10K 0.6 225.0 520.3 PR WT 913K 100 4.1 19.3 4.1 REQ DEFERRED 3704 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 670K 73.3 8.1 49.1 5.9 -CONT 2141 -FALSE CONT 311 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------------------------------ TOTAL 1642K SYNC 1587K 96.6 66.1 31.0 NO SCH 37 0.0 58.5 175.5 0.0 REQ TOTAL 1760K 1825 ASYNC 55K 3.4 202.6 467.4 PR WT 1642K 100 4.3 19.6 4.3 REQ DEFERRED 8646 CHNGD 0 0.0 PR CMP 1199K 73.0 8.4 66.5 6.2 -CONT 6792 -FALSE CONT 543

STRUCTURE NAME = IRLMLOCK1 TYPE = LOCK STATUS = ACTIVE PRIMARY # REQ -------------- REQUESTS ------------- -------------- DELAYED REQUESTS ------------- SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF ---- AVG TIME(MIC) ----- EXTERNAL REQUEST NAME AVG/SEC REQ ALL AVG STD_DEV REQ REQ /DEL STD_DEV /ALL CONTENTIONS

JA0 198K SYNC 189K 7.4 69.3 34.1 NO SCH 2 0.0 18.5 16.3 0.0 REQ TOTAL 222K 220.4 ASYNC 8905 0.3 279.2 517.0 PR WT 197K 99.2 5.0 22.5 5.0 REQ DEFERRED 1065 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 148K 74.4 9.7 85.2 7.2 -CONT 1054 -FALSE CONT 619

JB0 558K SYNC 513K 20.0 70.6 32.7 NO SCH 70 0.0 32.0 15.1 0.0 REQ TOTAL 652K 619.8 ASYNC 45K 1.8 225.7 369.5 PR WT 555K 99.5 4.6 17.3 4.6 REQ DEFERRED 3808 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 413K 74.1 9.5 61.0 7.0 -CONT 3689 -FALSE CONT 1447 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------------------------------ TOTAL 2560K SYNC 1753K 68.5 65.5 32.4 NO SCH 119 0.0 29.0 16.9 0.0 REQ TOTAL 2879K 2844 ASYNC 807K 31.5 146.1 218.5 PR WT 2552K 100 3.6 11.9 3.5 REQ DEFERRED 31K CHNGD 0 0.0 PR CMP 1857K 72.6 18.2 149.9 13.2 -CONT 17K -FALSE CONT 9137

Coupling Facility Lock Structure Report


8/12/2019 s 2528 Ga 091442



STRUCTURE NAME = DSNDB1G_GBP0 TYPE = CACHE STATUS = ACTIVE PRIMARY # REQ -------------- REQUESTS ------------- -------------- DELAYED REQUESTS ------------- SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF ---- AVG TIME(MIC) -----


JA0 430 SYNC 426 14.5 29.3 18.8 NO SCH 0 0.0 0.0 0.0 0.0 0.48 ASYNC 4 0.1 251.0 106.9 PR WT 0 0.0 0.0 0.0 0.0 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 0 0.0 0.0 0.0 0.0 DUMP 0 0.0 0.0 0.0 0.0

JB0 803 SYNC 757 25.7 38.5 23.5 NO SCH 0 0.0 0.0 0.0 0.0 0.89 ASYNC 46 1.6 213.5 103.4 PR WT 0 0.0 0.0 0.0 0.0 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 0 0.0 0.0 0.0 0.0 DUMP 0 0.0 0.0 0.0 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------------------------------ TOTAL 2944 SYNC 2876 97.7 25.8 18.2 NO SCH 0 0.0 0.0 0.0 0.0 -- DATA ACCESS --- 3.27 ASYNC 68 2.3 203.9 101.1 PR WT 0 0.0 0.0 0.0 0.0 READS 164 CHNGD 0 0.0 PR CMP 0 0.0 0.0 0.0 0.0 WRITES 928 DUMP 0 0.0 0.0 0.0 0.0 CASTOUTS 295 XI'S 258 STRUCTURE NAME = SYSZWLM_B52A2084 TYPE = CACHE STATUS = ACTIVE PRIMARY # REQ -------------- REQUESTS ------------- -------------- DELAYED REQUESTS ------------- SYSTEM TOTAL # % OF -SERV TIME(MIC)- REASON # % OF ---- AVG TIME(MIC) ----- NAME AVG/SEC REQ ALL AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

JA0 25817 SYNC 155 0.2 88.8 64.9 NO SCH 8 0.0 169.9 44.9 0.1 28.69 ASYNC 26K 25.3 157.0 471.8 PR WT 13K 49.8 5.7 16.3 2.9 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 7118 27.6 86.7 348.1 23.9 DUMP 0 0.0 0.0 0.0 0.0

JB0 24976 SYNC 273 0.3 96.4 67.8 NO SCH 62 0.2 146.9 164.0 0.4 27.75 ASYNC 25K 24.4 157.9 627.8 PR WT 12K 49.9 5.0 6.2 2.5 CHNGD 0 0.0 INCLUDED IN ASYNC PR CMP 6917 27.7 73.3 251.6 20.3 DUMP 0 0.0 0.0 0.0 0.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ------------------------------------------------------------------------------------------------------------------------------ TOTAL 101K SYNC 692 0.7 96.7 69.5 NO SCH 126 0.1 562.1 3045 0.7 -- DATA ACCESS --- 112.6 ASYNC 101K 99.3 149.4 696.4 PR WT 50K 49.9 5.1 10.0 2.5 READS 50857 CHNGD 1 0.0 PR CMP 28K 27.2 68.4 682.2 18.6 WRITES 50181 DUMP 0 0.0 0.0 0.0 0.0 CASTOUTS 0 XI'S 0

Coupling Facility Cache Structure Report

1 C O U P L I N G F A C I L I T Y A C T I V I T Y PAGE 125 z/OS V1R8 SYSPLEX UTCPLXJ8 DATE 07/24/2006 INTERVAL 015.00.000 CONVERTED TO z/OS V1R8 RMF TIME 11.30.00 CYCLE 01.000 SECONDS ------------------------------------------------------------------------------------------------------------------------------ COUPLING FACILITY NAME = CF2 ------------------------------------------------------------------------------------------------------------------------------ SUBCHANNEL ACTIVITY ------------------------------------------------------------------------------------------------------------------------------ # REQ ----------- REQUESTS ----------- ------------------ DELAYED REQUESTS -------------- SYSTEM TOTAL -- CF LINKS -- PTH # -SERVICE TIME(MIC)- # % OF ------ AVG TIME(MIC) ------ NAME AVG/SEC TYPE GEN USE BUSY REQ AVG STD_DEV REQ REQ /DEL STD_DEV /ALL

JA0 6102K CBP 1 1 2964 SYNC 2877K 46.9 35.7 LIST/CACHE 46 0.0 578.2 477.0 0.0 6780.6 CFP 3 3 ASYNC 3198K 135.6 516.6 LOCK 12 0.0 25.0 17.1 0.0 SUBCH 28 28 CHANGED 8 INCLUDED IN ASYNC TOTAL 58 0.0 UNSUCC 0 0.0 0.0 JB0 5862K CBP 1 1 4311 SYNC 1806K 64.9 35.1 LIST/CACHE 1219 0.0 631.9 2385 0.2 6513.4 CFP 3 3 ASYNC 4025K 132.0 415.0 LOCK 111 0.0 40.5 102.0 0.0 SUBCH 28 28 CHANGED 494 INCLUDED IN ASYNC TOTAL 1330 0.0 UNSUCC 0 0.0 0.0 JC0 3603K CBP 1 1 5303 SYNC 704564 61.9 38.4 LIST/CACHE 1202 0.0 1076 6567 0.4 4002.9 CFP 3 3 ASYNC 2870K 147.4 585.0 LOCK 62 0.0 41.2 100.3 0.0 SUBCH 28 28 CHANGED 523 INCLUDED IN ASYNC TOTAL 1264 0.0 UNSUCC 0 0.0 0.0 JE0 2486K CBP 1 1 1530 SYNC 521887 56.7 40.5 LIST/CACHE 115 0.0 255.8 521.1 0.0 2761.9 CFP 3 3 ASYNC 1910K 157.0 351.1 LOCK 2 0.0 10.5 3.5 0.0 SUBCH 28 28 CHANGED 80 INCLUDED IN ASYNC TOTAL 117 0.0 UNSUCC 0 0.0 0.0 JF0 4206K ICP 8 8 0 SYNC 137625 18.6 50.9 LIST/CACHE 1183 0.0 43.5 96.8 0.0 4673.8 SUBCH 56 56 ASYNC 3937K 120.3 253.2 LOCK 0 0.0 0.0 0.0 0.0 CHANGED 594 INCLUDED IN ASYNC TOTAL 1183 0.0 UNSUCC 0 0.0 0.0 JH0 1857K CBP 1 1 0 SYNC 127855 16.7 48.5 LIST/CACHE 438 0.0 275.4 416.6 0.1 2063.8 CFP 2 2 ASYNC 1704K 121.9 247.9 LOCK 2 0.0 13.0 1.4 0.0 SUBCH 21 21 CHANGED 8 INCLUDED IN ASYNC TOTAL 440 0.0 UNSUCC 0 0.0 0.0 J80 4537K ICP 8 8 0 SYNC 1679K 33.7 34.5 LIST/CACHE 7 0.0 409.6 280.7 0.0 5041.0 SUBCH 56 56 ASYNC 2850K 104.8 215.5 LOCK 6 0.0 12.8 7.9 0.0 CHANGED 1 INCLUDED IN ASYNC TOTAL 13 0.0 UNSUCC 0 0.0 0.0 J90 1820K CBP 3 3 0 SYNC 545768 69.0 22.5 LIST/CACHE 0 0.0 0.0 0.0 0.0 2022.0 CFP 4 4 ASYNC 1270K 148.9 842.9 LOCK 0 0.0 0.0 0.0 0.0 SUBCH 49 49 CHANGED 0 INCLUDED IN ASYNC TOTAL 0 0.0 UNSUCC 0 0.0 0.0

Subchannel Activity


8/12/2019 s 2528 Ga 091442


CF: How to Read the Reports

1. Check CF Service Times and Delay Times

Examine Service TimesBy System, By Structure

Examine Delay Times

If Service Times seem too high2. Check Path and Subchannel Busy

Cross check with Link Utilization in Channel Path Activity Report

3. Check CF Utilization

4. Structure details:

Lock Structures - Lock Contention

Cache Structures - Read/Write Statistics

List Structures - List Elements used at end of interval

5. More information:

XCF:see RMF XCF Report

GRS: see RMF XCF and Enqueue Reports

Postprocessor

DurationReport

OverviewReport

andRecord

RMF CF Online Monitoring

Monitor III Real-Time

CFSYS

CP...CF=AA

SYS1

SYS2

CP

CF=BB

CFACT

CFOVER


8/12/2019 s 2528 Ga 091442


CF Overview Report: CFOVER

ERB3CFO RMF V1R8 CF Overview - UTCPLXHD Line 1 of 10

Command ===> Scroll ===> PAGE

Samples: 120 Systems: 3 Date: 07/24/06 Time: 11.30.00 Range: 100 Sec

---- Coupling Facility ----- ---- Processor ----- Request -- Storage --

Name Type Model Level Util% Defined Effect Rate Size Avail

CF1 2096 S07 14 24.6 4 3.8 27107.1 6077M 2500M

CF2 2094 S38 14 31.8 3 3.0 35159.4 6077M 2378M

CF3 2064 212 13 28.4 4 4.0 15887.1 6078M 3408M

CF Systems Report: CFSYS

RMF V1R8 CF Systems - UTCPLXJ8


CF Name System Subch -- Paths -- -- Sync --- ------- Async -------

Delay Avail Delay Rate Avg Rate Avg Chng Del

% % Serv Serv % %

CF1 JA0 0.0 6 2.7 185.4 64 3145 200 0.0 0.2

JB0 0.0 4 22.3 42.8 68 1007 187 0.1 5.5

JC0 0.0 4 36.9

JE0 0.0 6 1.6

JF0 0.0 6 3.2

JG0 0.0 3 1.5

JH0 0.0 3 2.6

J80 0.0 2 0.0

J90 0.0 6 1.4


8/12/2019 s 2528 Ga 091442


ERB3CFS RMF V1R8 CF Systems - UTCPLXJ8 Line1 of 6

Command ===> Scroll ===> CSR


CF Name System Subch -- Paths -- -- Sync --- ------- Async -------

Delay Avail Delay Rate Avg Rate Avg Chng Del

% % Serv Serv % %

CF1 SYS1 5.2 8 1.2 5.5 2111 2.2 4333 2.1 5.4

SYS2 10.3 2 15.4 5.6 2333 2.3 4555 1.2 5.6

SYS3 1.1 2 0.1 5.5 2111 2.2 4333 3.0 5.4

SYS4 0.3 2 0.2 5.6 2333 2.3 4555 2.4 5.6

SYS5 4.8 2 1.2 5.6 2333 2.3 4555 0.0 5.6

SYS6 7.3 2 5.7 5.6 2333 2.3 4555 0.0 5.6

CF2 SYS1 0.5 8 0.5 5.5 2111 2.2 4333 1.0 5.4

SYS2 10.3 2 15.4 5.6 2333 2.3 4555 1.2 5.6

SYS3 1.1 2 0.1 5.5 2111 2.2 4333 3.0 5.4

SYS4 0.3 2 0.2 5.6 2333 2.3 4555 2.4 5.6

SYS5 4.8 2 1.2 5.6 2333 2.3 4555 0.0 5.6

SYS6 7.3 2 5.7 5.6 2333 2.3 4555 0.0 5.6

RMF Coupling Facility - Subchannels and Paths

Details for System : JAB0

Coupling Facility : CF1

Subchannels Generated : 14

In Use : 14

Max : 14

Path IDs : C0 80

Press Enter to return to the Report panel.

F1=Help F2=SplitScr F3=End F6=RMFHelp F9=SwapScr

F12=Return

CFSYS Details via Cursor Sensitivity

CF On-line Report Sample

ERB3CFS RMF V1R8 CF Systems - UTCPLXJ8 Line1 of 34

Command ===> Scroll ===> PAGE


CF: CF1 Type ST System --- Sync --- --------- Async --------

Rate Avg Rate Avg Chng Del

Structure Name Serv Serv % %

COUPLE_CKPT1 LIST *ALL 11.1 2222 5.5 4444 1.1 5.5

SYS1 5.5 2111 2.2 4333 1.3 5.4

SYS2 5.6 2333 2.3 4555 1.0 5.6

COUPLE_CKPT2 LIST *ALL 11.1 2222 5.5 4444 2.1 5.5

SYS3 5.5 2111 2.2 4333 2.0 5.4

SYS4 5.6 2333 2.3 4555 2.3 5.6

IRLMLOCK1 LOCK *ALL 11.1 2222 5.5 4444 3.0 5.5

SYS1 5.5 2111 2.2 4333 2.9 5.4

SYS2 5.6 2333 2.3 4555 3.1 5.6

DSNDB1G CACHE *ALL 3.2 1111 2.2 2222 0 1.2

SYS1 3.2 1111 2.2 2222 0 1.2

TESTCACHE CACHE A *ALL 0 0 0 0 0 0

SYS1 0 0 0 0 0 0

RMF CF Activity - Structure Details

Details for structure IRLMLOCK1 in coupling facility CF1

Press Enter to return to the Report panel.

LOCK information for system JAB0

Structure Size : 20M Connection Name: IRLMLOCKP00C0001

List Entries Total : 4096 Jobname : IRLM

Current : 256 Status : Active

Lock Entries Total : 4096 ASID : 109

Current : 256 CF Level : 2

Contention (%) : 80

False Contention (%) : 20

IRLMLOCK1


8/12/2019 s 2528 Ga 091442


What's it Cost Me?

Host CPU effect

Varies based on

Portion of workload involved in data sharing Access rate to shared data

Type of hardware for Host, CF and CF links

Number of systems

Typical system-level effects

Resource Sharing: 3% versus single image

Data sharing primary production application: 5% to 10%

Individual Transaction/Job effects - can have widevariation

Do I Need to Add More Coupling Links?

Each CF link has 2 (7 in peer mode) hardware buffers

z/OS assigns a subchannel to each buffer

XES must obtain a subchannel to start a CF operation

If all subchannels are busy ...

Lock - XES spins (SYNC DELAYED REQUEST)

Sync cache/list - XES changes to async (CHANGED)

Async cache/list - XES queues (ASYNC DELAYED REQ)

Warning sign: >10% TOTAL DELAYED REQUESTS

Investigate cause of delaysSubchannel utilization >30% - add or upgrade links

Not directly reported by RMF but easily calculated

Exploiter issues burst of CF ops

XCF message size negotiation - create new structure

DB2 GBP castout - tune frequency and/or castout ownership


8/12/2019 s 2528 Ga 091442


8/12/2019 s 2528 Ga 091442


XCF Path Statistics Report

TOTAL SAMPLES = 30 XCF PATH STATISTICS

-------------------------------------------------------------------------------------------------------------------------------

OUTBOUND FROM SYS5 INBOUND TO SYS5

--------------------------------------------------------------------------------- ------------------------------------------

T FROM/TO T FROM/TO

TO Y DEVICE, OR TRANSPORT REQ AVG Q FROM Y DEVICE, OR REQ BUFFERS

SYSTEM P STRUCTURE CLASS OUT LNGTH AVAIL BUSY RETRY SYSTEM P STRUCTURE IN UNAVAIL

SYS1 S IXCSIG1 DEFLT4K 25 0.00 25 0 0 SYS1 S IXCSIG1 31 0 S IXCSIG2 DEFAULT 718 0.00 718 0 0 S IXCSIG2 2,946 0

S IXCSIG3 DEFLARG 1 0.00 1 0 0 S IXCSIG3 16 0

S IXCSIG4 DEFLT12K 204 0.00 204 0 0 S IXCSIG4 198 0

S IXCSIG5 DEFAULT 1,316 0.00 1,316 0 0 S IXCSIG5 3,073 0 S IXCSIG6 DEFLT36K 176 0.00 176 0 0 S IXCSIG6 17 0

C 0B61 TO 0BA9 DEFAULT 552 0.00 552 0 0 C 0BA2 TO 0B6A 326 0

C 0160 TO 01A0 DEFAULT 617 0.00 617 0 0 C 0110 TO 01F0 299 0

SYS2 S IXCSIG1 DEFLT4K 1,728 0.00 1,728 0 0 SYS2 S IXCSIG1 522 0

S IXCSIG2 DEFAULT 1,439 0.00 1,439 0 0 S IXCSIG2 13,264 0 S IXCSIG3 DEFLARG 12 0.00 12 0 0 S IXCSIG3 14 0

S IXCSIG4 DEFLT12K 1,147 0.00 1,147 0 0 S IXCSIG4 895 0 S IXCSIG5 DEFAULT 2,209 0.00 2,209 0 0 S IXCSIG5 4,538 0

S IXCSIG6 DEFLT36K 188 0.00 188 0 0 S IXCSIG6 137 0

C 0B51 TO 0BA9 DEFAULT 934 0.00 934 0 0 C 0BA2 TO 0B5A 85 0

C 0150 TO 01A0 DEFAULT 423 0.00 423 0 0 C 0110 TO 01E0 21 0

SYS3 S IXCSIG1 DEFLT4K 2212920 0.07 2209656 3,264 0 SYS3 S IXCSIG1 749,787 0

S IXCSIG2 DEFAULT 2847630 0.23 2847614 16 0 S IXCSIG2 9142336 0

S IXCSIG3 DEFLARG 0 0.00 0 0 0 S IXCSIG3 15 0S IXCSIG4 DEFLT12K 730,206 0.10 730,025 181 0 S IXCSIG4 148,564 0

S IXCSIG5 DEFAULT 5123192 0.33 5123164 28 0 S IXCSIG5 35,150 0S IXCSIG6 DEFLT36K 1,991 0.00 1,991 0 0 S IXCSIG6 7,609 0

C 0B41 TO 0BA9 DEFAULT 73,416 0.03 73,408 8 0 C 0BA2 TO 0B4A 173,828 0

C 0140 TO 01A0 DEFAULT 76,525 0.00 76,516 9 0 C 0110 TO 01D0 170,868 0

---------- ----------

TOTAL 11,440,428 TOTAL 11,219,955

CF

StructuresCTCs

High values heremay indicate

insufficient paths

Values here mean therewas insufficient buffers

for input requests

Proportion of Traffic

DEFAULT = 78%

DEFLT4K = 13%

DEFLT12K = 4%

(remainder)= 5%

XCF Usage by System Report

% of big messages thatsuffered performance

degradation

XCF USAGE BY SYSTEM

REMOTE SYSTEMS LOCAL

OUTBOUND FROM SYS5 INBOUND TO SYS5 SYS5

----- BUFFER ----- ALL

TO TRANSPORT BUFFER REQ % % % % PATHS REQ FROM REQ REQ TRANSPORT REQ

SYSTEM CLASS LENGTH OUT SML FIT BIG OVR UNAVAIL REJECT SYSTEM IN REJECT CLASS REJECT

SYS1 DEFLT12K 12,220 204 94 6 0 0 0 0 SYS1 7,391 0 DEFLT12K 0

DEFLT36K 36,796 176 100 0 0 0 0 0 DEFLT36K 0

DEFLT4K 4,028 25 0 100 0 0 0 0 DEFLT4K 0

DEFLARG 62,464 1 0 100 0 0 0 0 DEFLARG 0

DEFAULT 956 5,405 0 100 0 0 0 0 DEFAULT 0

SYS2 DEFLARG 62,464 12 100 0 0 0 0 0 SYS2 19,039 0

DEFLT12K 12,220 1,147 98 2 0 0 0 0

DEFLT36K 36,796 188 100 0 0 0 0 0

DEFLT4K 4,028 1,728 0 100 0 0 0 0

DEFAULT 956 16,260 0 100 0 0 0 0SYS3 DEFLT4K 4,028 2212920 0 100 0 0 0 0 SYS3 11178K 0

DEFAULT 956 8456121 0 100 0 0 0 0

DEFLARG 62,464 0 0 0 0 0 0 0

DEFLT36K 36,796 1,991 100 0 0 0 0 0

interesting to knowhow much smallerbuffers could be

# of messages migratedbecause no signal paths

available

# of messages rejectedbecause no buffers

were available


8/12/2019 s 2528 Ga 091442


WLM Courses from IBM Education

Basic z/OS Tuning Using the Workload Manager(WLM)

ES5454.5 Days, Hands-on Lab Exercises

Advanced z/OS Performance: WLM, Sysplex,Unix Services, Web

ES851

4.5 Days

ibm.com/training

Documents

s 2528 Ga 091442