Upload
jemima-conley
View
219
Download
3
Embed Size (px)
Citation preview
Best Practices in Batch TuningBest Practices in Batch TuningEmpowering IT Staff to Reduce CPU and Run-Time in the Empowering IT Staff to Reduce CPU and Run-Time in the
EnterpriseEnterprise
Software Engineering of America www.seasoft.com
Agenda
• Did You KnowDid You Know
• Presentation GoalPresentation Goal
• Importance of Batch TuningImportance of Batch Tuning
• Common Problems EncounteredCommon Problems Encountered
• Solving the Problem Solving the Problem
• Remediation OptionsRemediation Options
• SummarySummary
Software Engineering of America www.seasoft.com
Did You Know!
The Center on Aging & Work / Workplace Flexibility has used U.S. Census data to calculate that an average of 4.6 adults will turn 65 each minute in 2007.
In 2025, an average of 8.0 adults will turn 65 each minute. (McNamara, 2006).
In 2012, there could be as many as 21 million vacant jobs, but just 17 million workers to fill those posts.
Computing Technology Industry Association
Inc.
Flexible schedules 19%
Part-time work options 12%Contracting retired employees 9%Increased compensation 9%
Training or professional 9%development programs
Bonuses for staying on 8%
Job sharing 5%
Delayed retirement plans 4%
Other 4%
No steps taken 60%
Are you taking any steps to retain IT pros who are nearing
traditional retirement age?
Source: ComputerWorld Vital Signs survey of 233 IT professional, May-June 2007. Multiple responses allowed.
Where have all the Baby Boomers gone!
Software Engineering of America www.seasoft.com
Ford – Launched project Everest to replace mainframe-based Ford Supplier Network procurement system with new ERP-based system. After 4 years and multiple problems, Everest is scrapped. Cost: $400M
McDonald’s - Launched project Innovate to replace multiple custom-built applications (including mainframe accounting systems) with global, ERP-based platform. After a year of R&D and spiraling costs, Innovate is killed. Cost: $170M
Innovate eats McDonalds’ lunchInnovate eats McDonalds’ lunch
Ford fails to climb EverestFord fails to climb Everest
Did You Know! Who Said the Mainframe is DEAD!
http://www.roughtype.com/archives/2006/01/down_the_drain.php
Software Engineering of America www.seasoft.com
A recent survey* in the UK showed that among 50 mid-to-large size enterprises using mainframes…
• 93% still depend on the mainframe to run mission-critical applications
• 50% say it is too risky to migrate off mainframe
• 28% tried and failed to migrate off the mainframe
• 12% had limited success with a migration off the mainframe
• 8% had tried and succeeded with a migration off the mainframe
*Source: Mainframe still reufusing to die. Survey, Ireland News, 2/28/07
Did You Know! Who Said the Mainframe is DEAD!
Software Engineering of America www.seasoft.com
Overall Presentation Goal
Explore the key factors that often limit the Explore the key factors that often limit the focus on batch tuning, and discuss the focus on batch tuning, and discuss the technology framework and methods that technology framework and methods that effectively assist performance and capacity effectively assist performance and capacity staff as well as empower other key staff staff as well as empower other key staff members in the identification and resolution members in the identification and resolution of batch tuning opportunities.of batch tuning opportunities.
Discuss methods for identifying the major sources Discuss methods for identifying the major sources of batch inefficiencies that recover the greatest of batch inefficiencies that recover the greatest batch elapsed time, CPU and I/O savings quickly, batch elapsed time, CPU and I/O savings quickly, and with minimum effort.and with minimum effort.
Software Engineering of America www.seasoft.com
Why Is Batch Tuning Still Important?
• Corporate growth and new business needs have caused Corporate growth and new business needs have caused the introduction of complex applicationsthe introduction of complex applications
• Most Data Centers are operating 24x7 with online Most Data Centers are operating 24x7 with online availability nearly 100%. To accommodate this, the Batch availability nearly 100%. To accommodate this, the Batch window process must be reducedwindow process must be reduced
• More than 50% of critical data processing is done via the More than 50% of critical data processing is done via the batch processbatch process
• Batch processing continues to grow while batch windows Batch processing continues to grow while batch windows continue to shrink continue to shrink
Software Engineering of America www.seasoft.com
Common Problems of Batch Tuning Practice?
• New forms of z/OS transactions have lead the performance New forms of z/OS transactions have lead the performance analyst to concentrate efforts on WLManalyst to concentrate efforts on WLM
• Lack of efficient JCL coding practices and Standards Lack of efficient JCL coding practices and Standards EnforcementEnforcement
• Serial design of most batch jobs, along with I/O intensity, Serial design of most batch jobs, along with I/O intensity, significantly contributes to the elapsed runtime of batch jobssignificantly contributes to the elapsed runtime of batch jobs
Software Engineering of America www.seasoft.com
Common Problems of Batch Tuning Practice?
• Existing Performance Analysis Tools:Existing Performance Analysis Tools:– Can be extremely complex to Use Can be extremely complex to Use – Complicate data analysis with a multiplicity of Complicate data analysis with a multiplicity of
reports/options/softwarereports/options/software– Fail to consider batch workload characterization/profilingFail to consider batch workload characterization/profiling– Ignore JCL construct factorsIgnore JCL construct factors
• The Cost of RemediationThe Cost of Remediation
• Batch jobs, although operating correctly, have not Batch jobs, although operating correctly, have not benefited from advanced technology, coding practices or benefited from advanced technology, coding practices or processing methods. processing methods.
•The typical thinking is: The typical thinking is: “If it ain't broke, don't fix it“If it ain't broke, don't fix it”!.”!.
Software Engineering of America www.seasoft.com
Batch Tuning – Areas to Investigate
OTHERSData not usedExcessive backupsParallelism (Job Split)Redundant Sort Job classes schemeOperator involvement (Automation)Job Priority
DATA IN MEMORY (DIM)DATA IN MEMORY (DIM)VIO
Hiperbatch
BatchPipes
VLF, modules and Execs
Data Base Buffers
Sort Parameters (DFSORT Hipersorting)
VSAM Index and/or Data, BLSR
OPTIMIZE I/OOPTIMIZE I/OBlock sizes (connect time)
Use cache where possible
Level of granularity of serialization (Reduce ENQ time)
Buffering
SYTEM & TECHNOLOGYExploit new addressing scheme
64 bitsWorkload ManagerDASD Technology (ESCON Channels, Stripping)Coupling Facility (JES) TMM
DB2 & IMS Subsystems
Software Engineering of America www.seasoft.com
Unsystematic Approach
Inaccurate Metrics
Unrepresentative Workload
Inappropriate Evaluation Technique
Ignoring Significant JCL and JOBS Design Factors
Improper presentation of results.
GoalGoal MetricsMetrics SelectSelectWorkloadWorkload AnalyzeAnalyze Imp.Imp.
ChangesChanges ResultsResults
Iteration
1
2
3
4
5
6
Common Problems of Batch Tuning Practice?
Software Engineering of America www.seasoft.com
GoalGoal MetricsMetrics SelectSelectWorkloadWorkload AnalyzeAnalyze Imp.Imp.
ChangesChanges ResultsResults
IterationIteration
The Challenge
How to Identify specific opportunities by other staff members, e.g. Development, Operations, Production Control
How to minimize data correlation and analysis
How to Model the Batch Timeline and Workload Characterization
How to streamline the remediation phase, minimizing cost and maximizing payoff
Software Engineering of America www.seasoft.com
Batch Performance Analysis Dimensions
JCL InefficienciesJCL Inefficiencies
Copy SyndromeCopy Syndrome
Deployment CycleDeployment Cycle Target SystemTarget System
System wide focus of performance System wide focus of performance analysisanalysis
Applications complexitiesApplications complexities Performance analysis is a highly Performance analysis is a highly
iterative process and it is difficult to iterative process and it is difficult to implement a practice focused on batch implement a practice focused on batch workloadworkload
Problem data for the batch job Problem data for the batch job population is practically impossible for a population is practically impossible for a specialist to gather and maintain specialist to gather and maintain
Users and application support rely on Users and application support rely on infrastructure for performance studiesinfrastructure for performance studies
Traditional Perf. Analysis
DevelopersDevelopers
Perf. & Capacity Perf. & Capacity Planning AnalystsPlanning Analysts
Sys. ProgrammersSys. Programmers
Production Control Production Control
Batch KnowledgebaseBatch Knowledgebase
Software Engineering of America www.seasoft.com
A Batch Performance tool should provide:
• Conservative savings in the 15% to 20% range in CPU MIPS and Conservative savings in the 15% to 20% range in CPU MIPS and elapsed runtime. Large organizations should see millions in elapsed runtime. Large organizations should see millions in savings and a return on investment measured in months. savings and a return on investment measured in months.
• Target is Application/System based versus Source code based. Target is Application/System based versus Source code based. This means lower complexity and fast time to value.This means lower complexity and fast time to value.
• Knowledge Base should provide the combined expertise of a Knowledge Base should provide the combined expertise of a Capacity Analyst and a Production Control Analyst.Capacity Analyst and a Production Control Analyst.
• Recommendations should be precise, prioritized, and easy to Recommendations should be precise, prioritized, and easy to understand.understand.
Software Engineering of America www.seasoft.com
A Batch Performance tool should provide:
• Majority of changes are JCL related and they typically follow the Majority of changes are JCL related and they typically follow the 80/20 rule. 80% or more of the benefits are seen in 20% or less of 80/20 rule. 80% or more of the benefits are seen in 20% or less of the changes.the changes.
• Deliverables should not only facilitate fine tuning of what needs to Deliverables should not only facilitate fine tuning of what needs to run but help identify what can be eliminated. The savings in run but help identify what can be eliminated. The savings in eliminating redundant sorts, or the creation of files that are not eliminating redundant sorts, or the creation of files that are not used, is a 100% efficiency gain for those components eliminated!used, is a 100% efficiency gain for those components eliminated!
• Should not be overhead intensive. Analysis should be based on Should not be overhead intensive. Analysis should be based on history.history.
• The approach should be proactive and embrace a strategy to The approach should be proactive and embrace a strategy to preserve the integrity of the changes. This is much more resource preserve the integrity of the changes. This is much more resource efficient than automating the same change every time an inefficient efficient than automating the same change every time an inefficient component is executed.component is executed.
Software Engineering of America www.seasoft.com
• Jobs Sorting Already-Sorted Data• Jobs Performing Unnecessary Passes Over Data• Jobs Executing Slow Utilities• Jobs with Inefficient DFDSS/FDR Executions • Jobs with Inefficient DB2 Utility Execution• Jobs Failing During Execution • Unnecessary Passes Over Data
CPU Savings
Software Engineering of America www.seasoft.com
• Data Created But Not Referenced• Data Sets with Inefficient Block-Size• VSAM Inefficiency Matrix• VSAM Inefficient NSR Buffering• VSAM Candidates for Batch LSR• DFSMS-compressed Data Set Inefficiency Matrix• DFSMShsm Data Set Migration Thrashing • Inefficient Cobol Compile Options
CPU Savings
Software Engineering of America www.seasoft.com
• Jobs with Allocation Delays• Jobs Waiting for Initiator• Jobs with Excessive Program Run-Time• Jobs Delayed Prior to Init. Selection Phase • Data Set with ENQ Conflicts• Sort Inefficiencies Matrix
Elapsed Time Savings
Software Engineering of America www.seasoft.com
• Data Sets with Inefficient Block-Size• Over-Allocated DASD Data Sets• Data Created But Not Used• VSAM Inefficiency Matrix
DISK Space Savings
Software Engineering of America www.seasoft.com
Workload Selection
Workload Selection– Network Flowchart and Workload Selection– Network Flowchart and Critical Path Analysis in a Batch StudyCritical Path Analysis in a Batch Study
A critical Path is that sequence of activities (Jobs) that determine the total elapsed time for a given batch workload to reach a defined point. It identifies which of the activities in a batch workload govern the timing of work during the batch Window.
Software Engineering of America www.seasoft.com
Workload Selection
Workload Selection– Network Flowchart and Critical Workload Selection– Network Flowchart and Critical Path Analysis in a Batch StudyPath Analysis in a Batch Study
A systematic and reliable study requires a consistent compilation of data from comparable runs. Ex1: Daily Jobs, plot data from a month, (20 samples) Weekly Jobs, plot the data from three months (13 samples) Verify if the end of month run implies extra work
Average job execution times from number of batch cycles Watch Deviation ! Produce exceptional reports after each batch window Trend analysis and monthly reports Modeling the effect of changes. Focus on new CP is path is projected to change Identify Steps that can be moved to another Job (backups, reorgs, etc.)
1. IBM Redbook SYSTEM/390 MVS Parallel Sysplex Batch Performance
Software Engineering of America www.seasoft.com
Workload Selection
Workload Selection– Network Flowchart and Critical Workload Selection– Network Flowchart and Critical Path Analysis in a Batch StudyPath Analysis in a Batch Study
Software Engineering of America www.seasoft.com
Job Analysis Example #1
The problem:
TEMPORARY DATASET CREATED BUT NOT REFERENCED AFTERWARD
Exactly where it occurs:
JOB GZDTP###
PGM=TBR017 STEP= SEL0301 PROCSTEP=
EXEC=000.11.50 CPU=0.02.49 EXCP#=353K
DD=SYSUT2 MODE=OUTPUT ACCESS=QSAM EXCP#=238K
DSN=SYS10020.T235730.RA000.ZMZL0200.R0149276
How to correct it:
DATA IS WRITTEN INTO A TEMPORARY DATASET BUT DOES NOT SEEM TO BE USED AFTERWARDS. THE CREATION OF THE DATA CAN PROBABLY BE ELIMINATED
Software Engineering of America www.seasoft.com
Job Analysis Example #2
The problem: IDCAMS REPRO EXECUTION FOLLOWED BY COMPLEMENTARY SORTExactly where it occurs: JOB ZMZL0200
PGM=IDCAMS STEP=STEP03 PROCSTEP=EXEC=000.04.01 CPU=0.00.49 EXCP#=62KDD=UNLOAD MODE=OUTPUT ACCESS=QSAM DSN=DM.MPROD.DATA33
EXCP#=31kPGM=ICEMAN STEP=SM03 PROCSTEP=EXEC=000.02.09 CPU=0.00.27 EXCO#=69KDD=SORTING MODE: INPUT ACCESS=BSAMD DSN=DM.MPROD.DATA33
EXCP#=31KHow to correct it:
A SINGLE SORT STEP CAN USUALLY REPLACE THE TWO STEPS OF IDCAMS REPRO AND SORT. IMPLEMENTING THE SINGLE SORT STEP WILL PROVIDE SIGNIFICANT SAVINGS IN ELAPSED TIME, CPU TIME, AND I/O ACTIVITY
Software Engineering of America www.seasoft.com
What About DB2?
• Improve DB2 Application EfficiencyImprove DB2 Application Efficiency
• Increase DB2 ParallelismIncrease DB2 Parallelism
• Improve Efficiency of DB2 HousekeepingImprove Efficiency of DB2 Housekeeping
• Eliminate DB2 ContentionsEliminate DB2 Contentions
Software Engineering of America www.seasoft.com
DB2 Analysis Example #1The problem:
A READ-ONLY DB2 THREAD DOES NOT EXPLOIT DB2 QUERY PARALLELISM
Exactly where it occurs:JOB=MDIKIG04 STEP=STEP020 PROCSTEP= PGM=IKJEFT1B
DB2-ID=DBP2 PLAN=MDPPROD TYPE=TSO-BACKGROUNDPACKAGE: VKB_DVP2.COLPROD.MDIKIGSTART-OF-THREAD: 2010/01/18 20.28.09END-OF-THREAD: 2010/01/18 21.16.10
How to correct it:THE APPLICATION PERFORMS EXTENSIVE READ-ONLY OPERATIONS, BUT DB2 HAS NOT USED THE OPTION TO PROCESS THE QUERY IN PARALLEL. PARALLEL DB2 OPERATIONS CAN SIGNIFICANTLY REDUCE THE ELAPSED TIME OF EXTENSIVE QUERIES. IF THIS IS AN IMPORTANT THREAD THAT MUST RUN FASTER, CONSIDER ENABLING DB2 PARALLELISM FOR THIS APPLICATION.
Software Engineering of America www.seasoft.com
The problem:
A DB2 THREAD HAS ENCOUNTERED INTERNAL LIMITATIONS DURING RID LIST PROCESSING.
Exactly where it occurs:
JOB=MDIKIG04 STEP=STEP020 PROCSTEP= PGM=IKJEFT1B
DB2-ID=DBP2 PLAN=MDPPROD TYPE=TSO-BACKGROUND
PACKAGE: VKB_DVP2.COLPROD.MDIKIG
START-OF-THREAD: 2010/01/12 19.08.02
END-OF-THREAD: 2010/01/13 22.11.10
RID LIST PROCESSING: #SUCCESSES=12 #FAILURES= 237
DB2 Analysis Example #2
Software Engineering of America www.seasoft.com
How to correct it:
RID LIST PROCESSING CAN HELP REDUCE I/O RESOURCE CONSUMPTION AND ELAPSED TIME. HOWEVER, IF THERE IS NOT ENOUGH RID LIST STORAGE, RESOURCES ARE ACTUALLY WASTED (THE APPLICATION HAS ALREADY CONSUMED RESOURCES AND, UPON FAILURE, MUST REVERT TO A TABLE SPACE SCAN.)
IT IS SUGGESTED THAT THE DB2 ADMINISTRATOR CONSIDER THE FOLLOWING:
1. IF THE PACKAGE WAS BOUND WITH A SMALL AMOUNT OF DATA: THE ACCESS PATH THAT WAS CHOSEN MIGHT BE INEFFICIENT AND MIGHT HAVE CAUSED THE FAILURE. RUNSTATS AND REBIND MAY SOLVE THE PROBLEM.
2. CHANGING THE APPLICATION PATH TO INHIBIT RID LIST PROCESSING (LIST PREFETCH ACCESS PATH).
3. REVIEWING THE APPLICATION LOGIC.
DB2 Analysis Example #2
Software Engineering of America www.seasoft.com
A PC-based Graphical Analysis Facility (GAF) provides a set of pre-defined Excel spreadsheets of some Global reports
Graphical Analysis Facility (GAF)
JOBHISTJOBHIST SUBGAFSUBGAF
Hlq.. REP%ID%.D%DATE%.T%TIME%Hlq.. REP%ID%.D%DATE%.T%TIME% My Computer (.TXT file)My Computer (.TXT file)
ExcelExcel
Software Engineering of America www.seasoft.com
Remediation Techniques and Options (EXAMPLE)
Success Factors Cost Factors
Reduce MIPS
Consumed Lower
HW/SW Cost
Software Engineering of America www.seasoft.com
Remediation Techniques and Options
• Implement gradual changes using a systematic approach Implement gradual changes using a systematic approach (minimize the level of risk) (minimize the level of risk)
• Consider JCL refurbishing (< cost)Consider JCL refurbishing (< cost)
• Focus improvement on the Jobs in the Critical PathFocus improvement on the Jobs in the Critical Path
• Fine-tune the base modelFine-tune the base model
• Coordinate the implementation of changesCoordinate the implementation of changes
• Verify results (adjust base line), start againVerify results (adjust base line), start again
Software Engineering of America www.seasoft.com
Summary
• Tuning the Batch environment can result in significant CPU Tuning the Batch environment can result in significant CPU and elapsed time savingsand elapsed time savings
• The traditional approach to batch tuning makes it difficult to The traditional approach to batch tuning makes it difficult to identify batch application inefficiencies, hinders participation identify batch application inefficiencies, hinders participation by application development teams, and limits the by application development teams, and limits the effectiveness of resultseffectiveness of results
• Identifies and deciphers the batch inefficiencies so that Identifies and deciphers the batch inefficiencies so that application development and support personnel are application development and support personnel are empowered to locate and correct batch issues. empowered to locate and correct batch issues.
Interested in learning more: Schedule an on-site meeting and live demonstration
for your entire IT staff
FOR MORE INFORMATION PLEASE CONTACT: FOR MORE INFORMATION PLEASE CONTACT:
E-mail: [email protected]
Tel: 800.272.7322
Web: www.seasoft.com