Upgrading CA Workload Automation ESP Edition Agents at Verizon

Upgrading CA Workload Automation ESP Edition Agents at Verizon

MFX07S #CAWorld

Mainframe

Cedric Persaud, Verizon

Associate Director

2 © 2014 CA. ALL RIGHTS RESERVED.

© Verizon 2014 All Rights Reserved Information contained herein is provided AS IS and subject to change without notice. All trademarks used herein are property of their respective owners.


Abstract

Verizon Telecommunications made CA Workload

Automation ESP Edition (CA WA ESP) the standard tool

for workload management across the Wireline

organization for both mainframe and distributed

systems using the CA WA ESP agent technology. In this

session, they will share insights on how they

successfully upgraded to the newer version of the CA

WA ESP agent across the enterprise for all of their

distributed applications. They will also review their

processes and procedures used to upgrade ESP, as well

as share lessons learned and best practices.

Cedric Persaud

Verizon

Associate Director



Agenda

COMMUNICATION AND NOTIFICATIONS 5

CONDUCTING ACTUAL UPGRADE TO R11.3 6

LESSONS LEARNED & BEST PRACTICES 8

ISSUES ENCOUNTERED 7

VERIZON ENVIRONMENT 1

REASONS FOR UPGRADING TO R11.3 AND BENEFITS 2

PREPARING AND PLANNING FOR UPGRADE 4

OVERVIEW OF UPGRADE EFFORT 3

SUMMARY & CLOSING REMARKS 9

Q&A 10



Verizon Environment - Overview

Heterogonous Environment (Distributed & Mainframe) – Mainframe applications using CA WA ESP : Billing, Provisioning, etc.

– Distributed applications using CA WA ESP agent: Payroll, Finance, Billing, etc.

– 212 distributed applications using CA WA ESP

1,450 CA WA ESP agents in Distributed Environment (August 2014)

Within past 3 years, Verizon has consolidated data centers and merged CA WA ESP masters. – Currently use 13 production masters & 3 test/dev masters

CA WA ESP agents with CA WA ESP Plug-Ins – SAP

– PeopleSoft

– Remote Execution Agent (REA)



Verizon Environment – Jobs Processed

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

4,752,697

4,284,502 4,522,348 4,434,071

3,942,083 3,756,169

3,904,927 3,911,250 3,861,952

CA WA ESPx Total Jobs Run



Verizon Environment - Architecture Overview

Remote

Execution

Agent Windows

AIX HPUX Solaris

CA WA ESP Workstation 3270

LINUX

SAP PeopleSoft

CA WA ESP Proxy

CA WA ESP Proxy CA WA ESP Proxy

CA WA ESP Master

Cloud servers IBM z/OS

Sysplex

I5 Tandem

CA WA ESP System Agents

CA WA ESP Application Integration

Agents(Plug-ins)



Verizon Environment - CA WA ESPx Infrastructure

Freehold Data Center

Fairland Data Center

Freehold

Mainframe

Fairland

Mainframe

ESPMBHC1

ESPMSAC

ESPMTPA

ESPMFDC2

ESPMOME

ESPMSTE (non-prod)

ESPMDEV (non-prod)

ESPMDEVL (dev)

CA WA ESPx Agents

CA WA ESPx Plug-in

Mainframes & CA WA ESP Masters



Verizon Environment – Organization and My Team

Verizon IT Cloud Services (ITCS)

– Responsible for delivering & managing IT systems, hardware, and

infrastructure across Verizon’s strategic data centers.

Production Control Organization – Supports all scheduling activities for all applications and platforms

– CA WA ESP Administration team responsible for supporting CA WA ESP on mainframe

Open Systems Scheduling (My Team) – 12 FTEs support & manage CA WA ESPx

– Administrators of CA WA ESPx and associated components

– Responsible for deploying new agents and scheduling new batch jobs

– Responsible for providing Tier 2 support of production jobs

– Interface with CA to address CA WA ESPx bugs and/or enhancements



Reasons for Upgrading and Benefits

Reasons for Upgrading from R7 to R11.3 – R7 agent reaching EOL (end of service life): March 2015

– Improved features and scheduling benefits for customers

– R11.3 is compatible with newer operating systems and platforms

Benefits of R11.3 Agent – R11.3 Agent supports Windows 8/Windows 2012 operating systems

– R11.3 adds more detailed logging options to assist in problem resolution

– Performance and optimization improvements

– Many other enhanced features such as:

File watchers with re-trigger option

Event sensors to monitor whether events are loaded



Overview of Upgrade

CA Recommended to Upgrade to R11.3

Started Upgrade in 3Q13 (September/October)

Target Completion Date: 4Q14

Inventory of Agents – Utilized Verizon’s standard inventory tools with discovery capability

– CMDB (HP)

– Discovery Dependency Mapping Inventory tool (HP)

Scope of Project – 1900 R7 agents identified and reviewed across footprint

– 900 R7 agents deemed inactive & removed from environment

– 1,000 R7 agents (or 125 applications) eligible and in scope for upgrade



Preparing and Planning Upgrade

Tested and “Soaked” R11.3 Agent – Installed and performed comprehensive testing of R11.3 agent

No issues found

– Verified compatibility issues with CA WA ESP Master on mainframe (R11.3)

No issues found

– Ensured R11.3 had same functionality as R7 agent

No CA WA ESPx applications or job procedures had to be changed

Packaged 11.3 for Mass Deployment – Verizon has standard packages used for Unix and Windows

– Standard deployment tools used to distribute R11.3 agent

Tivoli (IBM)

System Center Configuration Manager (Microsoft)



Communication and Notification to Users

Users of CA Workload Automation are Verizon Internal Application Teams.

Sought Concurrence and “Buy-In” from Application Teams – Conducted numerous meetings and planning sessions with application

teams.

– Reviewed the benefits and features of R11.3

– Explained R7 EOL as key driver for upgrading to R11.3

– Assured application teams upgrade will be minimally disruptive

– No application code or CA WA ESPx job changes required for upgrade

– Partnered with application teams to perform functional testing of R11.3 agent in test/development environment.



Conducting Upgrade

Prioritized Applications for Upgrade – Initially selected applications running on an CA WA ESP master

targeted for merger/consolidation.

Adhered to Verizon’s Change Management Policy – Submitted change records 3 weeks prior to actual upgrade

Upgrade Steps Performed (average duration per server = 1hr) – Suspended active Jobs/Events (scripted)

– Shutdown agent

– Removed old R7 package (scripted)

– Deployed R11.3 package to target server (scripted)

– Restarted/Recycled agent (scripted)

– Resumed Jobs/Events (scripted)

– Validated change



Issues Encountered with Upgrade

“Agent Notify” issue due to lost connectivity with CA WA ESP master

Out of disk space – upgrade requires more space on server

Discovered additional/unknown servers requiring upgrade

Incomplete removal of R7 package on Windows – Stale files and programs left by CA uninstall program

Server auto-reboot issue on R11.3 package for HP-UX

Default shell issue with R11.3 and R7 (/bin/sh vs. /bin/ksh) – R7 jobs used /bin/ksh. Whereas, R11.3 package defaulted to /bin/sh

– After upgrade, some CA WA ESPx jobs failed because they needed to run using /bin/ksh.



Lessons Learned and Best Practices

Notify application teams to test CA WA ESP agent plug-ins early in development environment.

Partner with CA to perform agentparm.txt file gap analysis & audit.

It’s a good idea to clean-up/remove unused CA WA ESP agents prior to upgrade.

Ensure support has login access to server prior to upgrade

Ensure agent install packages are built per CA’s standards and work with CA to validate.



Summary & Closing Remarks CA Workload Automation Upgrade is a Worthwhile Effort

Results

Improved benefits and features of R11.3 for application teams

Upgrading provides opportunity to “sanitize” the CA WA ESP

environment.

Verizon customers are satisfied with CA’s support

We are pleased with CA’s responsiveness.

Satisfied with CA’s resolution of technical issues and product

enhancements.

Excellent training offered by CA for Verizon staff and

customers.



For More Information

To learn more about Mainframe, please

visit:

http://bit.ly/1wbiPkl

Insert appropriate screenshot and text overlay from following “More Info Graphics” slide here;

ensure it links to correct page Mainframe







For Informational Purposes Only

© 2014 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.

This presentation provided at CA World 2014 is intended for information purposes only and does not form any type of warranty. Content provided in this presentation has not been reviewed for accuracy and is based on information provided by CA Partners and Customers.

Terms of this Presentation

Technology

Upgrading CA Workload Automation ESP Edition Agents at Verizon