Still Manually Managing Batch Workloads to Completion? A Risk Too Far?Kelly Vogt

Compuware a BMC Company

kelly.vogt@compuware.com

5 November 2020

Session 3AO

Dedicated to the Real Batch Heroes…Hint: It’s Not the Bleedin’ Officers!

Agenda

Your Batch StrategyHave you ever thought of it like this?

Manual OperationIt’s not difficult until it is…

Proper InitiationWalking the tightrope

How to Continuously Achieve Maximum ThroughputSettling for “good enough” is not good enough

Your Batch StrategyHave you ever thought of it like this?

Every Shop Has a Batch Strategy

• What is a Batch Strategy?

• Processes and procedures

• Roles and responsibilities

• Technical and domain knowledge

• Standards and conventions

• Job schedulers, system exits and tools

• And all the products that make up your batch environment

• Should be responsive to your changing needs

• Does it optimize your batch processing?

• Does it optimize your operational costs?

• Is it retirement proof?

How Agile Is It???

Every Shop Has a Batch Strategy

• Some are “hard coded”… Set it up and let what happens, happen…

• Some are heavily scriptedSee This… do That… What if we don’t have a script for This?!

• Most are hugely dependent upon long-experienced, skilled and knowledgeable individuals

Manual OperationIt’s not difficult until it is…

JES2 Is a Manually Operated Environment• After 50+ years, batch is still manually

operated

• Operation of JES2 is not for faint-hearted

• Much technical knowledge

• Juggling and plate-spinning skills

• Domain knowledge

• Constant attention to details

• Problem resolution requires experience to keep batch moving timely

• It’s a craft and your operators are masters of their craft

And Then You Get to This Level of Complexity!

43 Batch Job Classes!… and you can have a lot more…

Common Job-Related Commands$A A Release all jobs

$A J Release held jobs

$ADD JOBCLASS Add new job class

$ADD SRVCLASS Add new perm Service class element

$C J Cancel a job

$D A Display active jobs

$D INITDEF Display logical initiator currently defined

$D INTRDR Display current INTRDR initialization statement

$D I Display Initiators

$D JOBCLASS Display job class characteristics

$D JOBDEF Display characteristics assigned to jobs

$D JOBPRTY Display current job priority characteristics

$D J Display information about specific jobs

$D M J Send a message (to a job)

$D N Display job queue information

$D Q Display the number of jobs queued

$D RDI Display the status of JES2

$DEL JOBCLASS Delete a job class

$E J Restart a job

$G A Release a job at another node

$G C Cancel a job on another node

$G D Display a job on another node

$G H Hold a job on another node

$H A Hold all jobs

$H J Hold specific job

$P I Stop an Initiator

$P J Purge a job

$P XEQ Prevent JES2 & WLM initiators from selecting work

$S I Start an Initiator

$S J Initiate a batch job into execution

$S SRVCLASS Start JES2 processing for a given Service Class

$S XEQ Allow JES2 and WLM initiators to select work

$T INTRDR Control internal readers

$T I Control Initiators

$T JOBCLASS Set job class characteristics

$T..Qaff Which member a particular job class is active on

$T..XEQCount Max number of jobs executed in the MAS by job class

$T..XEQMEMBER Max number of jobs that can execute concurrently

$T JOBDEF Set job processing characteristics

$T JOBPRTY Set job scheduling characteristics

$T SRVCLASS Modify JES2 information for specified service class

$T J Change job class, scheduling priority or affinity

$TJ..SRVCLASS Change WLM Service Class for the Job specified

$TJ..Class Assign a new Class for the Job(s) specified

$TJ..Priority Assign a new Priority for the Job(s) specified

$TJ..Sysaff Change System affinity for the Job(s) specified

$Z I Halt initiator activity

MVS RESET Change Service Class, Quiesce or Job Reclassify

RESET...SRVCLASS Assign new Service Class to Job(s)/Address Spaces

RESET...QUIESCE Quiesce target Job or Address Space

RESET...RESUME Reclassify a Job or Address Space

People Do Make Mistakes…

Proper InitiationWalking the tightrope

Job Classes

• Job classes are used to segregate jobs… to classify work into lists

• Historically 36 single character job classes A-Z, 0-9

• Now 8-character classes available… up to 512 in total!

• For every job class there is at least one ,often more initiators assigned

Production

sales

night

sludge

dev2

dba

lab

Turn center

What Classes Don’t Do…

They don’t work together• Each job queue is independent from

other job queues

• Jobs in queues may be prioritized, but irrespective of jobs in other queues

• Job queues effectively race against each other to get initiated

Initiators

“JOHN”

AAAA

BB

B

BB

C

BB

BB

Z

BB

D,E,FI

BB

F,E,D

“BOB”

“TOM”

MB

B

E,F,D

BB

BB

BB

BB

P,KJ

L

N Q

NR

BB

BB

S,U

BB

BB

V,X,G

BB

H

O

T,U

• INITs pull jobs based on classes assigned to each initiator

• Single class

• Multi-class

• Round robin

• WLM-managed INITs

What Initiators Do…

Selects and prepares jobs for execution• Converts and interprets JCL

readying it for execution

• Dataset allocation – or not!

• Load and transfer control to job step program (application)

• Inter-job step activity

• WLM manages performance

Initiation Levels

• Over-Initiation – attempting to execute too much work concurrently – batch drags

• System efficiency is impacted• Programs wait for CPU dispatch

• Processor cache misses

• More CPU time required to accomplish the same work

• Excess job classes and all that entails puts you here!

Over-Initiation

Commands increasing initiation levels

• $A A• $SI• $SJ• $S SRVCLASS• $S XEQ• $T… Xaff• $T… XEQ Count• $T… XEQMEMBER• $TJ… SRVCLASS• RESET… SRVCLASS

• Broad brush strokes with operator commands can easily over-commit available capacity, especially in small environments

• Optimizing the CPU requires constant monitoring and intelligent adjustment

• Prioritizing job access to the CPU must be thoughtfully done

Initiation Levels

• Under-Initiation – running too little work

• Executing tasks run fast, but you could have run more• Wasted opportunity

• Possibly risking SLAs

• Is your machine oversized?!

Under-Initiation

Commands decreasing initiation levels

• $C J

• $HA

• $HJ

• $PI

• $PJ

• $P XEQ

• $T… Xaff

• $T… XEQ Count

• $T… XEQMEMBER

• $TJ… SRVCLASS

• RESET… QUIESCE

• Broad brush strokes with operator commands easily starve your batch operation for work, wasting CPU time

• Optimizing the CPU requires constant monitoring and adjusting

• You are actually only one job step change away from upsetting the balance

Proper Initiation

• Proper Initiation – where all work runs at machine speed

• Batch rips through the system

• Maximum throughput achieved

• You’re in the “sweet spot”

• Processor efficiency is optimized so jobs use less CPU than when over-initiated

How to Consistently Achieve Maximum ThroughputSettling for “good enough” is not good enough

How Well Are You Doing Today?

Are you over/under-initiating?

Is someone watching:

• Batch service class performance?

• System utilization?

• WLM metrics?

Are you starting/stopping initiators?

• Changing initiator classes?

• Manual or automation?

Do Your Job Priorities Align with Their Importance to the Business?How do you ensure the most urgent job is the next to initiate?

How often do you use $SJ commands?

• Do you change job queue priority?

How often do you change job WLM service class?

Do people call to ask: “When will my job start?”

Business Value1) Critical2) High3) Medium4) Low5) Discretionary

How to Consistently Achieve Maximum Throughput

Continuously observe and adjust…

• z/OS monitor system utilization

• WLM performance

• Decide whether to increase/decrease/maintain INITs

~ When He’s Not Doing That ~

Continuously scan the job queues

• Reorder each queue by job’s business importance

Repeat constantly… all day… all night… & all weekend…

How to Consistently Achieve Maximum Throughput

Decades of technical and domain knowledge, deep experience and well-practiced expertise

What will you do when they retire?

AUTOMATE!

But you each have our own superheroes

Their best effort keeps it going

Mainframe Batch Maturity Model

1 Reactive

Jobs are submitted via scheduler and managed by operations

2 Proactive

JCL standards are enforced, jobs prioritized via job classes and executed by initiators; service-level objectives prevalent

3 Automated

Service-level management is automated across complete batch process

4 Optimized

Batch demand is managed to reduce cost while maintaining service levels

5 Modernized

Consolidated real-time control and visualization of your complete end-to-end batch process

Console Operations has been automatedStorage Management has been automatedNetwork Operation has been automated

WHY NOT BATCH?

Batch Maturity Modelcompuware.com/batch-maturity-wp

kelly.vogt@compuware.com

Q

U

E

S

T

I

O

N

S

Please submit your session feedback!

• Do it online at http://conferences.gse.org.uk/2020/feedback/3AO

• This session is 3AO

GSE UK Conference 2020 Charity

• The GSE UK Region team hope that you find this presentation and others that follow useful and help to expand your knowledge of z Systems.

• Please consider showing your appreciation by kindly donating a small sum to our charity this year, NHS Charities Together. Follow the link below or scan the QR Code:

http://uk.virginmoneygiving.com/GuideShareEuropeUKRegion