IBM Impact Session CICS Transaction Gateway and CICS in a HA environment english-v4

13/04/2023 PAGE 1

3070 CICS TG and CICSin a High Availability

Environment

A customer experience with CA-SILCA

IBM IMPACT 2014 Conference 27 April- 1 May LAS VEGAS

Sylvie Constans Manager of the CICS & IMS team at CA-SILCA

13/04/2023 PAGE 2


CA-SILCA and the groupe Crédit Agricole SA

The activities of the group Crédit Agricole SA are organized in 4 business lines. Retail bank in France and World Wide Asset management Specialised financial services Corporate and investment bank

CA-SILCA is the IT subsidiary of the groupe Crédit Agricole SA.

Is located in France (region of Paris)

13/04/2023 PAGE 3


CA-SILCA: Overview

Founded in 2005 , groups at the beginning the IT productions of its 3 founding members

Currently we have more than 40 customers only subsidiaries

Center of expertise for the group Operation services of IT applications Buildings infrastructure services telephony, network,

office automation Services for providing workstations for employees

13/04/2023 PAGE 4


CA-SILCA: Overview

Some keys figures

3 PetaB of storage for the servers 36 000 workstations 33 000 mail boxes 550 hosted web sites 4 000 logical servers 16 000 phone lines

A new data center (5000 m2) composed of 2 sites

13/04/2023 PAGE 5


CA-SILCA: Mainframe infrastructure

4 ZEC12 located in a 2-site environment (bi-site) separated by 10 kms

2 CPC active /passive in each site.

Each LPAR on a active CPC has its image on the passive CPC of the other site.

2827-731 models 60.000 Mips Primary disk array (active

data) in one site Secondary disk array in

the second site

Third site for data replication

13/04/2023 PAGE 6


CA-SILCA: Mainframe activity

9 customers (59 LPARs, 60 000 Mips) CA-SILCA manages the system environment of 5

customers o 400 CICS (currently migrating from CICSTS4.1 to

CICSTS 5.1) o 45 CICS TG (version 8.1)o 75 DB2 (DB2 V10 migration in CM mode in 2014)o 60 WebSphere MQ (version 7.1)o 15 IMS (migration from IMS V11 to IMS V13 in 2014)o 40 LPAR z/OS 1.13 (z/OS2.1 migration planned in

2015)o 6 sysplexeso Coupling facilities on each CPC

13/04/2023 PAGE 7

LCL: Le Crédit Lyonnais


Our main customer is LCL (retail bank) Founded in Lyon in 1863 1925 agencies, 6 millions customers in France 70 divisions of private banking (150 000 customers) Retail bank for professionals

Its technical environment 130 CICS, 40 CICS TG, 25 DB2, 25 Websphere MQ

o Some applications in a non HA TOR/AOR architecture

o Benefits from a High Availability Environment for e-banking and intranet applications

13/04/2023 PAGE 8


LCL : At the beginning

The agencies are connected to the mainframe (Datacenter) based on their location

13/04/2023 PAGE 9



Local DB2

local VSAM

TOR1

AOR1

Région Parisienne

LPAR1

LocalDB2l

local VSAM

TOR2

AOR2

Région Centre

LPAR2

Local DB2

local VSAM

TOR3

AOR3

Région Sud Est

LPAR3

13/04/2023 PAGE 10


LCL : Current architecture

13/04/2023 PAGE 11



In 1995, the mainframes were grouped in the region of Paris on 2 sites. The agencies are still connected based on their location

Several production problems led LCL to think about a High Availability architecture In the early 2000s a sysplex was implemented with the

help of IBM 9 production LPARs and one DEV

o Implementation of RLS/SMSVSAMo DB2 Data sharing

First on the DEV LPAR (mono partition) to validate the cost

13/04/2023 PAGE 12



At the same time, a major project of merging data is started Regional files become national DB2 regional databases become national

TORs are accessed with generic resources

A 3-tier architecture is implemented WAS are connected to the SNA servers The applications must comply with this architecture Very few 3270 applications left 4 LPARs are dedicated to this architecture

o 2 for network purposes, 2 for applications

13/04/2023 PAGE 13


LCL architecture: the start of High Availability

RES2

LP2

TOR ProdTOR Pilote

TOR Prod

AOR Pilote

RES1

LP1

TOR ProdTOR Pilote

TOR Prod

AOR Pilote

AOR ProdAOR Prod AOR Prod AOR Prod

Ferme WASPilote

Ferme WASde Production

Passerelles SNA

LU6.2 LU6.2LU6.2

LU6.2 LU6.2LU6.2

TCP/IP TCP/IP

LU6.2 LU6.2

VTAM VTAM

AOR Prod

13/04/2023 PAGE 14



Pilot Some agencies are connected to the pilot WAS They have access to the pilot CICS TG and the pilot CICS We can deploy new versions of programs without

impacting all of production The CICS pilots have one specific load library ahead in

the DFHRPL Production

The rest

13/04/2023 PAGE 15


LCL architecture : the start of High Availability

The routing of transactions to the AORs is managed by the dynamic routing program (DFHDYP) we have customised

At the end of 2003, ETU9XLOP , the dynamic routing program of CICSPlex SM (TS2.2) was implemented Because our routing program didn’t satisfed us entirely

Round robin algorithm Simplistic

Number of transactions/ day : 4 Million (TOR+AOR) Implementation in goal mode, uses the service class

definitionso Provides average response time and not a

percentage These CICS are clones No affinities between transactions

13/04/2023 PAGE 16



Implementation of shared TSQ servers Implementation of named counters servers

Give each application a unique id in the Sysplex

The DEV environments have the same architecture(1 TOR, 2 AORs) To be sure not to generate affinities between

transactions during the development of applications

13/04/2023 PAGE 17



After one year we decided to stop using EYU9XLOP Because of the failover of one LPAR : business

impacto CICS has response time heavy degradation in a

LPAR (DB2)o CICSPlex SM continues to route to this LPAR

The services classes were probably not correctly set

The CICSPlex SM/WLM delay in reacting was too long

o The LPAR fails, we have to do an IPLo The remaining LPAR couldn’t handle the workloado The WAS fail to handle the incoming requests

We decided to rewrite our routing program to better fit our needs

13/04/2023 PAGE 18


LCL architecture : the start of High Availability

What we learned about this architecture Having only one LPAR is not sufficient in case of failover

during the day We have to restart the critical applications first

We decided to have 4 application LPARs. In case of failover only 25% of the workload has to be

dispatched onto the 3 others

The SNA servers have been replaced by CICS TGs on z/OS We noticed some affinities between LU6.2 connections

and TORs Loose coupling between WAS and CICS (logical names

instead of applids)

White paper :ftp://public.dhe.ibm.com/software/htp/cics/tserver/v32/library/WSW14020-USEN-00_systemz_harmony_0324A.pdf

13/04/2023 PAGE 19



At that time, CICS TGs were seen as black boxes (prior to version 7.0) No statistics available

At first CICS TG architecture is mapped on the existing CICS one Multi channel

No standard monitoring available with the tools on the z platform Introscope (Wily Technologie) was implemented on one

CTG o Only one CICS TG because of the overhead

A dashboard was implemented with the help of the vendoro To monitor the JCA pools activityo To monitor the activity and the CICS response time

13/04/2023 PAGE 20


The tools: Introscope (Wily Technologie)

13/04/2023 PAGE 21


Monitoring tools

D From D + 2

CICSPA . Analyze the CICS and CTG SMF records(DB2, WMQ informations)

MAINVIEW CICS/DB2/WMQ. Real time vision. System administration and tuning

CICSPA. Analyze the CICS and CTG SMF records(DB2, WMQ informations)

SAS / METROLOGY figures. Monthly Consolidation

MAINVIEW CICS/DB2/WMQ. Real time vision. System Administration and tuning. Analyse with 3 tools, LPAR by LPAR. History (LCL) < 1day (TP DB2 : 15 mn)

INTROSCOPE

Chiffres SAS / METROLOGIE.Aggregated SMF records. Aggregation on 1 hour. Monthly consolidation

D + 1

CICSPA. Analyze the CICS and CTG SMF records(DB2, WMQ informations)

No problem analysis

No problem analysis

Limited problem analysis (impossible for

DB2/LCL)

Problem analysis in 5 minutes Trend analysis

Problem analysis

Trend analysis

Problem analysis

Trend analysis

INTROSCOPE

BEFORE

CURRENT

13/04/2023 PAGE 22


The tools: CICS Performance Analyzer (IBM)

CICS Performance Analyzer allows us to do: Performance analysis by exploiting SMF110(1) records Tuning for our CICS TGs ( SMF111 records since V7.0)

o Do we have enough connection managers?o Workers information are taken from the

DFHXCURMo We would like to have « cross domain » informations

in batch reports• Peak numbers of connection managers, number

of requests, CICS response time, Daemon response time…

• A RFE has been raised n°46252: you can vote for it !!

13/04/2023 PAGE 23


LCL: Current architecture

2 sysplex distributors : algorithm in round robin (can backup each other)

CICS TG on z/OS using port sharing (dedicated by business) DB2 datasharing, RLS, Websphere MQ sharing group, shared

TSQ servers, named counters servers 4 application LPARs 2100 transactions/sec in peak hour (Tuesday morning) 95 million transactions per day 30 ms average response time

13/04/2023 PAGE 24


LCL : Current architecture

4 coupling facilities for the production sysplex for LCL 1 on each of the active CPC 2 external coupling facilities (passive CPCs )

o For the DB2 and RLS lock structures 11 Gb memory each Use of duplexing

o Expensive, only for the DB2 group buffer pools o Save 20% CPU on IRLM since the suppression of

duplexing for DB2 lock structure Automatic rebuild for the other structures Double failure not handled: loss at the same time of DB2

and its lock structure

13/04/2023 PAGE 25

LCL: Current architecture


Sysplex Distributor

System P Production servers

System P Pilot servers

13/04/2023 PAGE 26


Transaction routing CICS TG – TOR : DFHXCURM

The routing between CICS TGs and TORs is performed by a customised DFHXCURM The ECI request provides

o The sysplex distributor addresso The CICS TG port number to be joinedo A logical server name( we are independent if we

need to add/suppress a TOR)o The program name to be executedo The transaction id (best practice)

Integrates a routing table which takes into account the following criteriao LPAR on which the CICS TG is runningo A set of target TORs with « handicap » (local TORs

are preferred)

13/04/2023 PAGE 27


Transaction routing CICS TG – TOR : DFHXCURM

Part of the routing table integrated in the module

SYSA XCURMTAB TYPE=SYSTEM,SYSNAME=SYSA XCURMTAB TYPE=GROUP,GROUPID=ZZPL1, TARGET=((PL11,0),(PL12,05)) XCURMTAB TYPE=GROUP,GROUPID=ZZPL9, TARGET=((PL91,0),(PL92,05)) XCURMTAB TYPE=ENDSYS SYSB XCURMTAB TYPE=SYSTEM,SYSNAME=SYSB XCURMTAB TYPE=GROUP,GROUPID=ZZPL1, TARGET=((PL12,0),(PL11,05)) XCURMTAB TYPE=GROUP,GROUPID=ZZPL9, TARGET=((PL92,0),(PL91,05)) XCURMTAB TYPE=ENDSYS

13/04/2023 PAGE 28


Transaction routing TOR - AOR: DFHDYP

The request is routed to the less busy AOR The program knows the number of sessions defined

between the TOR and each AOR Counts the current number of tasks between the TOR

and each AOR

We can isolate one AOR or several AORs from routing to lighten an LPAR

We can route a transaction or a set of transactions to a AOR or a set of AORs thanks to a configuration file In case of affinity

The CSMI transaction is forbidden for routing In order to be able to tune and analyze performance Can’t be set disabled in case of problems

13/04/2023 PAGE 29


Exploitation

Monthly IPL for all the LPARs

For LCL only 2 out of 4 LPARs are eligible for batch processing

During the IPL of one LPAR, batch is starting on the second one OPCplex Schedule environment

For each IPL a change request is associated

Our changes (that we declare) can be IPL dependent (ie migration) If the IPL is delayed, we know all the changes

associated

13/04/2023 PAGE 30


Exploitation

All the CICS in the same LPAR are stopped/started once a week Start = auto but overriden as cold start after analyzing

the DFHGCD (if previous stop OK) To be sure not to forget changes done dynamically

CICS TGs are stopped/started one after another every midnight To suppress the affinity between the CICS TGs and the

TORs : CTG_PIPE_REUSE= ALL To avoid memory problems

13/04/2023 PAGE 31


Migration Strategy (1/2)

Use of aliases for libraries and symbols : member IEASYM in the z/OS parmlib Transparent for the developers : no JCL modification

to do for compiling Example :

o CICSTS.CIC.SDFHAUTH : alias (used by developers)o Points for instance at CICSTS.CIC10.SDFHAUTHo CICSTS.CIC&VERCIC..SDFHAUTH : is referenced in

the z/OS parmlibo SYMDEF(&VERCIC.=‘10')

13/04/2023 PAGE 32


Migration Strategy (2/2)

A new level is generated for the migration (ie 15) Definitions hard coded in the parmlib for the new libraries

to coexist with the 2 CICS versions (LINKLIST,LPA, APF) CICSTS.CIC15.SDFHLINK CICSTS.CIC&VERCIC..SDFHLINK (with &VERCIC =

« 10 ») Taken into account during the IPL

Allows us to not migrate all the CICS of the same partition at one time

The new CICS procedure has the CICS libraries hardcoded The symbol is set to the new level once the last CICS of the

last production LPAR has been migrated

REXX procedures have been written (we provide the name of the CICS to be migrated) Generates the CICS files Assembles the PLTxx, SIT.. tables Creates the new DFHCSD (one CSD by version) Creates the new CICS procedure

13/04/2023 PAGE 33


Failover (1/2)

The continuity of service is only ensured for the components in high availability Failover of a LPAR

o The sysplex distributor will route the requests to one of the CICS TGs running on the 3 LPARs left

o If DB2 had retained locks, it will be automatically restarted on another LPAR to release locks, then stops

Failover of CICS TGo The sysplex distributor will send the request to

another CICS TG listening on the same port Failover of TOR

o DFHXCURM detects the error ‘NO CICS ‘ and routes the request to the TOR of another LPAR

13/04/2023 PAGE 34


Failover(2/2)

Failover of AORo DFHDYP can’t send requests to this CICS anymore;

no more connection is available, scans the AORs left and sends the request to the less busy AOR

Failover of DB2

o DB2 abend : DB2 is restarted by ARMo DB2 frozen: the number of current tasks inscreases

(sessions TOR /AOR) DFHDYP sends the request to another LPAR

13/04/2023 PAGE 35


IT contingency plan

Every year, we perform 2 IT contingency plans for our customers (one for each site) We isolate one site

o The LPARs are restarted on the »passive » CPC of the other site

o Activation of CBUo Depending on the preference of our customers, their

production can run on this CPC for the weekend or the whole week

o We must have the keys for the products for the « passive » CPC if they are depending on serial number

13/04/2023 PAGE 36


High Availability : the pros and the cons

Pros Continuity of service in case of unavailability of

components CICS and CICS TG migrations in production during the

service Cons

It’s a real project, not only CICS Not magic

If there is a application problem (loop, lock on data) it is propagated to the whole CICSplex

13/04/2023 PAGE 37


LCL Architecture: what next

Planning CICSPlex SM implementation To use CICSTS5.1 new functionalities Lack of assembler skill: use the routing program of

CICSPlex SMo The algorithm has been enhanced, use of data

spaceso Service classes specified in percentile

WUI (Web User Interface) : centralized administration of CICS (SPOC)

But we would like to keep the same flexibility for migration in service

We don’t use CICS Explorer yet Must be installed on virtual servers: Citrix The flow should be opened from the IP addresses of

these servers to the mainframe

13/04/2023 PAGE 38



Questions ?

Technology

IBM Impact Session CICS Transaction Gateway and CICS in a HA environment english-v4