Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
IBM SystemsTechnical Eventsibm.com/training/events
Making it Difficult for Those IMS
Problems to Hide
[email protected] 14:00-15:00 Lindner Hotel,Lange Kievitstraat 125, 2018 Antwerpen, België
Jeff Maddix IMS Worldwide Technical Support
Guide Share Europe November 9, 2017
• IMS BeNELux User group Meeting
© Copyright IBM Corporation 2017.
Please note
• IBM’s statements regarding its plans, directions, and intent are subject to change or
withdrawal without notice at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product
direction and it should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment,
promise, or legal obligation to deliver any material, code or functionality. Information
about potential future products may not be incorporated into any contract.
• The development, release, and timing of any future features or functionality described for
our products remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks
in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as
the amount of multiprogramming in the user’s job stream, the I/O configuration,
the storage configuration, and the workload processed. Therefore, no assurance can be
given that an individual user will achieve results similar to those stated here.
2
© Copyright IBM Corporation 2017.
Notice and disclaimers• Copyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without
written permission from IBM.
• U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
• Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date
of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is
distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information,
including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to
the terms and conditions of the agreements under which they are provided.
• IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless,
our warranty terms apply.”
• Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.
• Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those
customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may
vary.
• References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all
countries in which IBM operates or does business.
• Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials
and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual
participant or their specific situation.
• It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and
interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with
such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
3
© Copyright IBM Corporation 2017.
Notice and disclaimers cont.
• Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other
publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of
performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should
be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such
third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including
but not limited to, the implied warranties of merchantability and fitness for a particular, purpose.
• The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents,
copyrights, trademarks or other intellectual property right.
• IBM, the IBM logo, ibm.com, AIX, BigInsights, Bluemix, CICS, Easy Tier, FlashCopy, FlashSystem, GDPS, GPFS, Guardium,
HyperSwap, IBM Cloud Managed Services, IBM Elastic Storage, IBM FlashCore, IBM FlashSystem, IBM MobileFirst, IBM Power
Systems, IBM PureSystems, IBM Spectrum, IBM Spectrum Accelerate, IBM Spectrum Archive, IBM Spectrum Control, IBM
Spectrum Protect, IBM Spectrum Scale, IBM Spectrum Storage, IBM Spectrum Virtualize, IBM Watson, IBM Z, IBM z Systems, IBM
z13, IMS, InfoSphere, Linear Tape File System, OMEGAMON, OpenPower, Parallel Sysplex, Power, POWER, POWER4, POWER7,
POWER8, Power Series, Power Systems, Power Systems Software, PowerHA, PowerLinux, PowerVM, PureApplica- tion, RACF,
Real-time Compression, Redbooks, RMF, SPSS, Storwize, Symphony, SystemMirror, System Storage, Tivoli, WebSphere, XIV, z
Systems, z/OS, z/VM, z/VSE, zEnterprise and zSecure are trademarks of International Business Machines Corporation, registered in
many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.
• Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Java and all Java-based trademarks and
logos are trademarks or registered trademarks of Oracle and/or its affiliates.
4
© Copyright IBM Corporation 2017. 5
Make it Difficult for those IMS Problems to Hide - Agenda
� Section 1: z/OS Setup Recommendations
� Section 2: IMS Setup Recommendations
� Section 3: Preservation of Standard Documentation
� Section 4: Manual Dump Intervention
� Section 5: Recommended SLIP traps
� Section 6: Bonus Material - Situation Specific Diagnostics
© Copyright IBM Corporation 2017. 6
Section 1.
z/OS Setup Recommendations
© Copyright IBM Corporation 2017. 7
z/OS Setup RecommendationsMaximize System Trace Table Size
� System Trace Table used to record many varieties of system events such as SRB dispatches, supervisor calls, IO and machine interrupts, mode changes,…
• Advantages:
• Extremely valuable for a large variety of problem types such at ABENDs, loops, and performance issues
• Setup
� Set z/OS System Trace table size to 3M or larger
� Default size is only 1M
–z/OS max size is 9G
� The z/OS command “TRACE ST,3M” can be specified in the z/OS COMMNDxx SYS1.PARMLIB member.
� For further detail, See manuals z/OS MVS Systems Commands, and z/OS MVS Diagnosis: Tools and Service Aids
© Copyright IBM Corporation 2017. 8
Use the z/OS Display command to see the system trace table settings:
– DISPLAY TRACE
IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479
SSSSYYYYSSSSTTTTEEEEMMMM SSSSTTTTAAAATTTTUUUUSSSS IIIINNNNFFFFOOOORRRRMMMMAAAATTTTIIIIOOOONNNN
ST=(ON,3000K,12000K)ST=(ON,3000K,12000K)ST=(ON,3000K,12000K)ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF AS=ON BR=OFF EX=ON MO=OFF AS=ON BR=OFF EX=ON MO=OFF AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) MT=(ON,1000K) MT=(ON,1000K) MT=(ON,1000K)
z/OS Setup RecommendationsMaximize System Trace Table Size
• Considerations:
� System Trace Table preferred central storage is page fixed storage
� Supply a reasonable value to the nnnM or nG parameter after considering the available central storage and the actual storage required for system trace
� Supplying a large value for nnnM or nG might cause a shortage of pageable storage in the system
© Copyright IBM Corporation 2017. 9
z/OS Setup RecommendationsIncrease Master Trace Table Size
� Master trace maintains a table of the most recently issued system console messages. Included in the dump when TRT SDATA option is specifies
• Advantages:
� Faster insight into events leading up to the dump creation
• Setup
�Set z/OS Master Trace Table size to 1000K (Def only 24K)
� Specify “MT SIZE(1000K)" in the SCHEDxx member of SYS1.PARMLIB
�See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning Guide/ Ref, and z/OS MVS System Commands manuals for complete details
© Copyright IBM Corporation 2017. 10
z/OS Setup RecommendationsIncrease Master Trace Table Size
� Increase master trace table size
• Use the z/OS Display command to ensure the master trace table is active:
– DISPLAY TRACE
IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479
SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION
ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K)MT=(ON,1000K)MT=(ON,1000K)MT=(ON,1000K)
• Considerations:
� Ensure Master Trace Table is large enough to span most error time frames
� Uses Subpool 229 Key 0 High Private Pageable Storage of the master scheduler address space
© Copyright IBM Corporation 2017. 11
z/OS Setup RecommendationsUse the Common Storage Tracker
�Use the Common Storage Tracker
• Advantages:
• SVC dumps (or RMF reports) will provide CSA/ECSA ownership information with jobname, time, and requesting module information.
• Great for common storage outage (storage leaks) problems and fragmentation analysis.
• Setup
� Use the DIAGxx SYS1.PARMLIB member to contain the request. Specify DIAG=xx in the IPL system parameters
–Example: DIAGxx member: VSM TRACK CSA(ON) SQA(ON)
� See z/OS MVS Diagnosis: Tools and Service Aids, z/OS MVS Initialization and Tuning Guide/ Reference, and z/OS MVS System Commands manuals for complete details.
© Copyright IBM Corporation 2017. 12
z/OS Setup RecommendationsUse the Common Storage Tracker
�Use the Common Storage Tracker
• Display the Common Storage Tracker status by using the following z/OS command:
• DISPLAY DIAG
• Considerations:
� Negligible Performance impact. Table stored in ESQA
IGV007I 11.44.05 DIAG DISPLAY 713
VSM TRACK CSA(ON) SQA(ON)
VSM TRACE GET(OFF) FREE(OFF)
VSM ALLOWUSERKEYCSA(YES)
VSM BESTFITCSA(NO)
VSM USEZOSV1R9RULES(YES)
TRAPS NAME()
CBLOC
VIRTUAL24()
VIRTUAL31()
REUSASID(NO)
AUTOIPL SADMP(NONE) z/OS(NONE)
© Copyright IBM Corporation 2017. 13
z/OS Setup RecommendationsTune CHNGDUMP MAXSPACE
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump.
• Advantages:
• Higher likelihood that SVC dumps will be captured in their entirety without worry of "Partial Dump".
• Setup
• Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate CHNGDUMP command at IPL time.
– Example: CD SET,SDUMP,MAXSPACE=5000M - Default size is 500M
– See z/OS MVS Diagnosis: Tools and Service Aids, z/OS MVS System Commands, and z/OS Initialization and Tuning Guide for additional details.
• Continued on next page
© Copyright IBM Corporation 2017. 14
z/OS Setup RecommendationsTune CHNGDUMP MAXSPACE
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump.
Display the MAXSPACE setting using the following z/OS Command:
D DUMP,OPTIONS
SDUMP- ADD OPTIONS
(ALLPSA,NUC,SQA,LSQA,RGN,LPA,TRT,CSA,SWA,SUMDUMP,
Q=NO,GRSQ,XESDATA),BUFFERS=00000000K,
MAXSPACE=00005000M,MSGTIME=99999 MINUTES,
MAXSNDSP=015 SECONDS,AUXMGMT=ON ,DEFERTND=NO
•Continued on next page
© Copyright IBM Corporation 2017. 15
z/OS Setup RecommendationsTune CHNGDUMP MAXSPACE
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump
• Considerations:
• MAXSPACE is the total virtual storage used by SDUMP
• SDUMP data capture is serialized so only one dump can be written at a time. However, z/OS releases the serialization after data capture, so another dump can be started before the first one gets written out (especially if something causes a delay in writing the first dump out to DASD). So MAXSPACE virtual storage might have to be temporarily "shared" by more than one dump.
� Ensure central storage and/or local page datasets are large enough to contain their normal peak load plus additional SVC dumps!
� See z/OS MVS Initialization and Tuning Guide for additional details.
© Copyright IBM Corporation 2017. 16
�Section 2.
IMS Setup Recommendations
© Copyright IBM Corporation 2017. 17
IMS Setup RecommendationsUse FMTO Option D
� Specify IMS control region EXEC parm value: FMTO=D
• Produces an SDUMP for terminating and non-terminating errors
� Non-terminating errors include:
– IMS dynamic allocation failures
– Some ESAF (External Subsystem Attach Facility) failures
• Allows use of SYSMDUMP as a backup dump in case of SDUMP failures
© Copyright IBM Corporation 2017. 18
IMS Setup RecommendationsSYSMDUMP DD for IMS Control Regions
� The SYSMDUMP specification will be used as a backup dump by IMS in the event
that SDUMP processing should fail.
� Specify SYSMDUMP DD in JCL of:
• IMS CTL Region
• IMS DLI/SAS Region
• IMS DBRC Region
� The following dump options should be specified in the SYS1.PARMLIB(IEADMR00)
member to ensure that adequate areas of z/OS storage are dumped to diagnose the
problem under most circumstances:
• SDATA=(CSA,RGN,SQA,SUM,TRT,ALLNUC,LPA) )
© Copyright IBM Corporation 2017. 19
IMS Setup RecommendationsSYSMDUMP DD, Continued
� SYSMDUMP DD Specification
• DISP=MOD should be specified because it is possible for multiple tasks to
generate dumps during the processing of an error.
GDG Example
//SYSMDUMP DD DISP=(MOD,CATLG),DSN=IMSAP05.SYSAIMS.SYSMDUMP(+1),
// DCB=(LRECL=4160,RECFM=FB,BLKSIZE=29120),UNIT=3390,
// SPACE=(CYL,(750,100),RLSE),DATACLAS=dtcls,VOL=SER=volser
� Testing the SYSMDUMP function can be done on a test system by temporarily setting the SDUMP to NODUMP via the CHNGDUMP command. This forces a SYSMDUMP to be created
• CD SET,SDUMP,NODUMP
• Then modify IMS out to test the SYSMDUMP function : F IMSTEST,DUMP
– IMS will terminate with ABENDU0020, the SDUMP should fail, and a modify dump will be produced to the SYSMDUMP dataset
© Copyright IBM Corporation 2017. 20
IMS Setup RecommendationsSYSMDUMP DD, Continued
Display the SYSMDUMP settings using the following z/OS Command:
D DUMP,OPTIONS
IEE857I 16.01.50 DUMP OPTION
SYSMDUMP- ADD OPTIONS
(NUC,SQA,LSQA,SWA,TRT,RGN,LPA,CSA,SUM,ALLNUC)
© Copyright IBM Corporation 2017. 21
IMS Setup RecommendationsSYSUDUMP DD
• The SYSUDUMP specification will be used by IMS dependent regions for failure events
• Application dump formatter tools rely on the SYSUDUMP DD statement to be present in the JCL
• Specify SYSUDUMP DD in JCL of IMS Dependent Regions
• The following dump options should be specified in the z/OS SYS1.PARMLIB(IEADMP00) member to ensure that adequate areas of z/OS storage are dumped
• SDATA=(CB,ERR,SUM) PDATA=(JPA,LPA,PSW,REGS,SA,SPLS)
• Display SYSUDUMP options using z/OS D DUMP,O command
IEE857I 16.23.15 DUMP OPTION
SYSUDUMP- ADD OPTIONS SDATA=(CB,ERR,SUM), PDATA=(SA,REGS,LPA,JPA,PSW,SPLS)
© Copyright IBM Corporation 2017. 22
IMS Setup RecommendationsIMS Table Traces Activated
� Turn on the IMS Dispatcher, Scheduler, DLI, and Lock traces to capture general system diagnostics and database access history
• Specify the following options in the IMS PROCLIB member: DFSVSMxx
• DISP=ON, SCHD=ON, DL/I=ON, LOCK=ON
• Or, use IMS command:
• /TRA SET ON TABLE nnnn
Where nnnn is alternately = DISP, SCHD, DLI, and LOCK
• These traces have relatively low CPU overhead
• SCHD being the lowest and DLI being the highest
� For non-production environments, when recreating failures, run with table traces on for the functional areas being tested
© Copyright IBM Corporation 2017. 23
IMS Setup RecommendationsIMS Table Traces Activated to External Trace
� IMS external tracing allows for IMS trace table output to be placed on IMS trace datasets rather than on the IMS OLDS when:
• DFSVSMxx PROCLIB member with nnnn=OUT is specified rather than nnnn=ON
• DISP=ON, SCHD=ON, DL/I=OUT, LOCK=OUT
• The IMS TRACE command specifies OPTION LOG
• /TRA SET ON TABLE nnnn OPTION LOG
� Use of the external trace dataset replaces sending output to the OLDS for better performance
• External trace dataset logging can add 1% as measured at SVL
• External trace datasets are allocated in the following order:
1) DASD JCL - DFSTRA01 and DFSTRA02 DDNAMES
2) DASD MDA- DFSTRA01 and DFSTRA02 Dynamic Allocation Members
3) TAPE MDA - DFSTRA0T Dynamic Allocation Member
4) IMS OLDS - If none of the above are found
© Copyright IBM Corporation 2017. 24
IMS Setup RecommendationsExternal Trace Environment
� IMS external tracing specifications
•DFSTRA01 and DFSTRA02DSORG SequentialRECFM VBLRECL 4004BLKSIZE 20024 (LRECLx n +4)
-Use automation to archive filled trace dataset off to trace archive GDG datasets on each external trace dataset switch to retain the trace data across the problem recreate
DFS2864I EXTERNAL TRACE DATA SET xxxxxxxx FULL - SWITCHING TO yyyyyyyy imsid
- and for IMS ABEND Scenario –
DFS3896I ATTEMPTING TO PURGE OLDS BUFFERS
© Copyright IBM Corporation 2017. 25
IMS Setup RecommendationsIMS Table Traces Display Status
� Display IMS Table Trace status
Command: /DIS TRACE TABLE
TABLE TRACE LOGGING ENTRIES VOLUME SYS3
DL/I ON 2268 SYS3
DL/I ON MEDIUM SYS3
LOCK ON MEDIUM SYS3
PI OFF OFF SYS3
LATC OFF OFF 3024 OFF SYS3
DISP ON OFF 1260 MEDIUM SYS3
SCHD ON OFF 630 MEDIUM SYS3
SUBS OFF OFF 504 OFF SYS3
DLOG ON OFF 756 MEDIUM SYS3
© Copyright IBM Corporation 2017. 26
IMS Setup RecommendationsExternal Trace Environment
� Display IMS external tracing specification
Command: /DIS TRACE XTRC
IMS ACTIVE TRACES
IMS EXTERNAL TRACE IS TRACING TO DISK
XTRC DDNAME ALLOC STATUS STATUS
DFSTRA01 DYNA ALLOCATED CLOSED
DFSTRA02 DYNA ALLOCATED ACTIVE
DFSTRA0T DYNA UNALLOCATED CLOSED
© Copyright IBM Corporation 2017. 27
�Section 3.
Preservation of Standard Documentation
© Copyright IBM Corporation 2017. 28
Preservation of Standard Documentation
�Consider implementing normal operating procedures to preserve the following documentation near the time of error
z/OS Console (SYSLOG)
SYS1.LOGREC
IMS OLDS/SLDS
JES JOBLOG of jobs related to failure
Any Dumps Produced
For elusive problems, these items might be requested a month or more after the problem has been reported.
© Copyright IBM Corporation 2017. 29
Ideal Time Periods for Syslog, Logrec, SLDS, and Joblog Preservation
z/OS Console (SYSLOG), Logrec, SLDS and Joblogs
• The ideal time frame:
�Back to the last IMS restart
� The moderate time frame:
�24 hours
• The minimum time frame:
�Two IMS system checkpoint intervals
© Copyright IBM Corporation 2017. 30
Dump Preservation
All dumps created near the time of IMS failure should be retained
SYS1.DUMP Datasets should be examined
� Multiple dumps may be created
� Keep all dumps at time of failure regardless of the subsystem
SYSMDUMP for the IMS Control, DLI/SAS, and DBRC regions need to be examined in case of primary SYS1.DUMP failures.
� Save these datasets if a dump was produced
� If GDG is used for SYSMDUMP copies, examine in ISPF for tracks used
SYSUDUMP should be saved for affected IMS dependent regions
© Copyright IBM Corporation 2017. 31
�Section 4.
Manual Dump Intervention
© Copyright IBM Corporation 2017. 32
Additional Manual Dump Intervention
�While IMS produces SDUMPs for most internal errors without human intervention, IMS Wait/Loop or partial loss of function conditions will require operator intervention to produce an SVC dump
�IMS wait/loops or performance issues can be caused by interaction with many address
spaces including:
CTL DLI/SAS DBRC IRLM CQS
OM RMSCIODBA
cctlMPP
(DBDC,
DBCTL,
DCCTL)
IFP BMP
IMS Connect
JMP JBP
APPC VTAMTCPIP
RRS
MQ Series
Non-IBM
Tools
IBM Tools
IXGLOGR
DB2
ODBM
WLMOther?
**
******
*
*
ap thr ap thr ap thr ap thrap thr ap thr
thrap
Subsys *
Subsys *
•Note: All z/OS dumps have a maximum job limit of 15 per dump. Omission of any interrelated address space or data area adds to the possibility that the dump may not be sufficient to solve the problem
*
© Copyright IBM Corporation 2017. 33
IMS Dump Techniques
IMS SVC Dumps can be requested using various techniques
• z/OS SYS1.PARMLIB IEADMCxx
• DUMP Command Parmlib
• z/OS SYS1.PARMLIB IEASLPxx
• SLIP Command Parmlib
• Manual z/OS DUMP command with paste from notepad
• Customized JCL can be built and submitted
• Sysplex Manager tool
• Others?....
• Note: Modify dumps (F IMS,DUMP) of IMS are no longer the preferred method to obtain wait/loop dump documentation
© Copyright IBM Corporation 2017. 34
IEADMCxx, z/OS SYS1.PARMLIB
� DUMP Command Parmlib Member
• Create SYS1.PARMLIB members called IEADMCxx for each customized dump command
• Advantages
• Can be used to customize IMS dumps prior to error event.
• Simple operator interface
• Disadvantages
• All Jobs must be active or no dump is produced on the system on which the dump command is entered. Remote systems without a jobname match will dump the DUMPSRV address space
• Avoid issues:
• Use wildcards * (multiple chars) and ? (single char) and similar naming conventions can be used to specify the jobnames
• Edit the Parmlib member jobnames prior to issuing the command
• See z/OS MVS Initialization and Tuning Guide and z/OS System Commands
© Copyright IBM Corporation 2017. 35
� Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters:
IEADMCxx Example for IMS
JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)
Where: Where: Where: Where:
j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.
j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.j3 = IMS DLI Region Jobname. j4 = IRLM Region Jobname.
j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2
j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.
Note: It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down.
Jobnames IMSPCTLA , IMSPDLIA, IMSPDRCA IMSPRLMA CICSP1A CICSP2A CICSP3A could be reduced to the following wildcards in the JOBNAME statement
- JOBNAME=(IMSP*,CICSP*)
© Copyright IBM Corporation 2017. 36
IEADMCxx DUMP Activation
� To request a dump from the IEADMCI1 parmlib member, enter the following z/OS commands:
• DUMP TITLE=(DUMP OF IMS and CCTL Regions ),PARMLIB=I1
A dump dataset will be created on the z/OS image from which the dump command is entered
© Copyright IBM Corporation 2017. 37
IMS SYSPLEX Dump Considerations
�IMS SYSPLEX implementations need to consider the possibility that a hang or problem on one IMSplex member may be due to a problem originating from another member, particularly in the area of IRLM locking
�Problems such as IMS Wait/Loops or partial loss of function conditions which require intervention to produce an SVC dump, should include SVC dumps from other members of the IMSplex
�Challenge:
Ensure that a dump is taken for all necessary address spaces on each system.
© Copyright IBM Corporation 2017. 38
� Create SYS1.PARMLIB member called IEADMCI2 containing the following DUMP parameters:
Sysplex IEADMCxx Example
Jobnames IMSPCTLA , IMSPDLIA, IMSPDRCA IMSPRLMA CICSP1A CICSP2A CICSP3A could be reduced
to the following wildcards in the JOBNAME and REMOTE statements
- JOBNAME=(IMSP*,CICSP*) REMOTE=(SYSLIST=(*(‘IMSP*’,’CICSP*’) ),SDATA,DSPNAME))
Note: The DSPNAME, XESDATA, COUPLE and REMOTE parms are added in SYSPLEX
environments
JOBNAME=(j1,j2,j3,j4,j5,j6,j7),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE),
DSPNAME=('j4'.*,'XCF*'.*),
REMOTE=(SYSLIST=(*('j1','j2','j3','j4’,’j5’,’j6’,’j7’),
SDATA,DSPNAME))
Where:
j1 = IMS Control Region Jobname j2 = DBRC Region Jobname
j3 = IMS DLI Region Jobname j4 = IRLM Region Jobname
j5 = CCTL Region 1 J6 = CCTL Region 2
j7 = CCTL Region 3
© Copyright IBM Corporation 2017. 39
Sysplex IEADMCxx DUMP Activation
� To request a dump from the IEADMCI2 parmlib member, enter the following z/OS command:
• DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=I2
� One dump dataset will be created on each z/OS image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs
© Copyright IBM Corporation 2017. 40
�Section 5.
Recommended SLIP Traps
© Copyright IBM Corporation 2017. 41
�Recommended SLIP Traps:CTL/DLI Region “Out of Storage”
�IMS control region “out of storage” conditions (ABEND80A ABEND878) can result in ABEND40D (Recursive errors in z/OS RTM while processing the S80A or S878 caused z/OS to end the address space before taking a dump of the address space)
• Problem: Unable to diagnose problem because the dump dataset doesn’t contain private storage for the failing region.
• Solution: Set the following four SLIP traps to capture the initial “out of storage” prior to RTM gaining control
© Copyright IBM Corporation 2017. 42
�Recommended SLIP Traps:CTL/DLI Region “Out of Storage”, Continued
SLIP SET,C=80A,J=jobname, SLIP SET,C=80A,J=jobname, SLIP SET,C=80A,J=jobname, SLIP SET,C=80A,J=jobname,
SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),
ID=xxxx,END ID=xxxx,END ID=xxxx,END ID=xxxx,END
SLIP SET,C=878,J=jobname, SLIP SET,C=878,J=jobname, SLIP SET,C=878,J=jobname, SLIP SET,C=878,J=jobname,
SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),
ID=xxxx,END ID=xxxx,END ID=xxxx,END ID=xxxx,END
WhereWhereWhereWhere
jobname = Alternatively, Jobname of the IMSjobname = Alternatively, Jobname of the IMSjobname = Alternatively, Jobname of the IMSjobname = Alternatively, Jobname of the IMS
CTL region,CTL region,CTL region,CTL region,
= Jobname of the IMS DLI/SAS region= Jobname of the IMS DLI/SAS region= Jobname of the IMS DLI/SAS region= Jobname of the IMS DLI/SAS region
Enter the following two SLIP traps for both the IMS
Control and IMS DLI/SAS regions (4 slips total)
© Copyright IBM Corporation 2017. 43
�Recommended SLIP Traps:IRLM Users Application System Timeout
� Application lockouts are detected by IRLM and MSG DXR162I is issued.
DXR162I irlmx CYCLE NUMBER nnnnnnnn PROCESSED FOR TIMEOUT
Explanation: A TIMEOUT candidate has been detected on at least one IRLM. If an
incompatible blocker was found, the IMS TIMEOUT EXIT has been driven to
present WAITER/BLOCKER information
Set the following SLIP to capture a dump for this event
SLIP SET,A=(SVCD),MSGID=DXR162I,
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE),
JL=(IMSP*,CI1?SA??),ID=M162,DSPNAME=('IMSPRLM?'.*,'XCF*'.*),
REMOTE=(JOBLIST,DSPNAME,SDATA),
END
© Copyright IBM Corporation 2017. 44
�Section 6 – Bonus Material
IMS Situation Specific Diagnostics
�DBRC
�DBCTL
�Database
�Data Communications
�APPC
�MSC
�CQS (Shared Queues)
�ESAF
© Copyright IBM Corporation 2017. 45
�Situation Specific Diagnostics:DBRC Related Problems
� DBRC Related Problems
• Obtain a listing of the DBRC RECONs prior to repair attempts
• Use Recover Control Utility (DSPURX00) - LIST.RECON
command
• If the RECON dataset listing cannot be obtained due to size, then
alternatively obtain a subsystem listing
• Use Recover Control Utility (DSPURX00) - LIST.SUBSYS ALL
command
• LIST.RECON is preferred
© Copyright IBM Corporation 2017. 46
�Situation Specific Diagnostics:DBRC Related Problems, Continued
� Obtain before and after copies of the RECONs
• JCL Example for Recon dataset DUMP to a file
using DF/DSS (Data Facility/Data Set Service)
//JJJJJJJJ JOB ( )
//* DUMP RECON TO FILE
//STEP1 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=*
//DASD DD DSN=hhhhhhhh.RECON.BACKUP,DISP=(NEW,CATLG),
// UNIT=3390,SPACE=(CYL,(5,5),RLSE)
//SYSIN DD *
DUMP DS(INCLUDE(IMS.RECON*)) -
OUTDD(DASD)
/*
© Copyright IBM Corporation 2017. 47
�Situation Specific Diagnostics:DBRC Related Problems, Continued
� JCL Example for Recon RESTORE from file using DFDSS
//JJJJJJJJ JOB ( )
//* TO DOWNLOAD TAPE LOADED WITH DFSDSS
//* NEED TO USE RENUNC TO RENAME TO YOUR HLQ
//* TAPE MAP WILL NOT SHOW LRECL OR BLKSIZE....
//S01RSTR EXEC PGM=ADRDSSU
//SYSPRINT DD SYSOUT=*
//IN DD DISP=SHR,DSN=hhhhhhhh.RECON.BACKUP
//DASD DD UNIT=3390,DISP=OLD,VOL=SER=SYS285
//SYSIN DD *
RESTORE DATASET(INCLUDE(**)) -
INDDNAME(IN) -
OUTDDNAME(DASD) -
CANCELERROR -
CATALOG -
REPLACE -
IMPORT -
TGTALLOC(SOURCE) -
RENAMEUNCONDITIONAL(hhhhhhhh) -
WAIT(2,2)
/*
© Copyright IBM Corporation 2017. 48
�Situation Specific Diagnostics:DBRC Related Problems, Continued� DBRC Related Problems, Continued
• Use z/OS command D GRS,CONTENTION on each system sharing the
RECON to determine if the RECON is held at the exclusion of other waiters
D GRS,CONTENTION
S=SYSTEMS DSPURI01 RECON1.IMSBP1
SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS
BT21 IBA2FDBR 0124 009AF7F8 EXCLUSIVE OWN
BT23 IBA1FDBR 0090 009AF1B0 EXCLUSIVE WAIT
BT21 IBA4FDBR 05E0 009AF7F8 EXCLUSIVE WAIT
BT26 IBA4DBRC 05DE 009F15B8 EXCLUSIVE WAIT
BT21 IBA1DBRC 00D7 009FF118 EXCLUSIVE WAIT
BT23 IBA2DBRC 0600 009F15C8 EXCLUSIVE WAIT
� If so, dump the owning address space
DUMP COMM=(dump title)
R nn,JOBNAME=(j1),SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ),END
© Copyright IBM Corporation 2017. 49
�Situation Specific Diagnostics:DBCTL Related Problems
� DBCTL Related Problems
• Issue IMS /DISPLAY ACTIVE and /DISPLAY CCTL commands, multiple times
• /DIS ACTIVE
• /DIS CCTL ALL
• /DIS CCTL cctlname
• Set the AP portion of the CICS trace to level 1-2
• Set the FC (FILE CONTROL) portion of the CICS trace to level 1-2
• Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions, especially the CCTL regions
© Copyright IBM Corporation 2017. 50
�Situation Specific Diagnostics:Database Problems
�For problems in the database arena, obtain
�Damaged database dataset
• Database Image copy of broken database in state prior to damage
• Image Copy of logically related databases
• IMS SLDS from all data sharing IMS subsystems
• Save to last good database image copy of damaged database
• If possible, and not already set, use IMS command and save output:
• /TRA SET ON TABLE nnnn OPTION LOG
• Where nnnn alternately = DLI and LOCK with external trace dataset recommended
• Joblog and SYSOUT from the Pointer Checker broken database
• Joblog and SYSOUT from batch jobs that accessed the damaged database
© Copyright IBM Corporation 2017. 51
�Situation Specific Diagnostics:Database Problems, Continued
�For problems in the database arena, obtain (Cont...)
• DBD for broken databases
• PSBs that access the broken DB
• List.RECON and LIST.HISTORY DBD(broken database)
• SMF 60 (VSAM Volume Data Set Updated), 62 (VSAM Component or Cluster Opened), and 64 (VSAM Component or Cluster Status) 79 (IRLM) records from all data sharing systems
-Back to last good image copy of broken database
• Any dumps produced related to the broken database
• z/OS Syslog
• IMS CTL Joblog
• IMS DLI Joblog
© Copyright IBM Corporation 2017. 52
�Situation Specific Diagnostics:Database Problems, Continued
�For problems in the database arena, obtain (continued)
• For VSAM datasets, close the broken KSDS and run IDCAMS commands:
� VERIFY� EXAMINE ITEST NODTEST
� EXAMINE NOITEST DTEST� LISTCAT ALL
� If the dataset cannot be closed, run IDCAMS commands:
� EXAMINE ITEST NODTEST
� EXAMINE NOITEST DTEST� EXAMINE ITEST NODTEST
� EXAMINE NOITEST DTEST
� LISTCAT ALL
� If the data set is open and being updated, then an EXAMINE may reflect what appears to be a problem because
the data set is in a state of flux. Therefore, if you run the EXAMINE commands twice and the errors are the same,
there this helps identify the problem
© Copyright IBM Corporation 2017. 53
�Situation Specific Diagnostics:DC Related Problems
� Problems involving Data Communications
• Issue IMS DISPLAY NODE x command
• /DIS NODE nodename
• Turn on IMS NODE trace. Data will be captured in the IMS TYPE6701 log record. Print records using IMS utility programs DFSERA10/DFSERA30
• /TRA SET ON NODE nodename
• Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS NODE trace. Note: GTF must be active with the USR option
• F NET,TRACE,TYPE=BUF,ID=nodename
• F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG)
• Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM
© Copyright IBM Corporation 2017. 54
�Situation Specific Diagnostics:APPC Related Problems
� Problems involving APPC
• Turn on the IMS LUMI Trace to the external trace dataset
•
• /TRACE SET ON TABLE LUMI OPTION LOG
• /TRACE SET ON LUNAME XXXXXXX INPUT
• /TRACE SET ON LUNAME XXXXXXX OUTPUT
• (XXXXXXX is the partner LU)
• Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS LUMI trace. Note: GTF must be active with the USR option
•
• F NET,TRACE,TYPE=BUF,ID=nodename
• F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG)
• Turn on the Transaction trace for TPPCB DL/1 calls
• /TRACE SET ON TRAN tttttttt
• tttttttt is the transaction name
• IMS TYPE6701 log records will be produced
© Copyright IBM Corporation 2017. 55
�Situation Specific Diagnostics:APPC Related Problems
� Problems involving APPC , (cont...)
• Turn on the z/OS APPC component trace.
• TRACE CT,ON,200M,COMP=SYSAPPC
• nn,OPTIONS=(GLOBAL),END
• When the problem has been recreated, stop the CTRACE
• TRACE CT,OFF,COMP=SYSAPPC
• APPC component trace sends its trace buffers to a SYS1.DUMP dataset
• The following IPCS commands can be used to format the trace:
• CTRACE COMP(SYSAPPC) SHORT - One line summary entries
• CTRACE COMP(SYSAPPC) FULL - Detailed trace entry formatting
– Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM and the APPC regions
© Copyright IBM Corporation 2017. 56
�Situation Specific Diagnostics:MSC Related Problems
� If you use IMS MSC (Multiple Systems Coupling) and experience a related
problem
• Take a SVC dump of the coupled IMS regions (minimally the CTL
regions, but the problem may reside in any IMS related region as
discussed earlier)
• In addition, ensure that the VTAM and/or TCPIP address space is
included. Do this as close to the problem time as possible, prior to
attempting to fix the problem
• Save the IMS OLDS for both coupled systems from the time of the
message creation
• Issue DISPLAY LINK ALL and DISPLAY LINK ALL MODE commands to
show the status and queue counts for the logical link and physical link
information for partner
© Copyright IBM Corporation 2017. 57
�Situation Specific Diagnostics:MSC Related Problems, Cont.
MSC related problems, continued
If using VTAM, Turn on the VTAM Internal Trace and VTAM Buffer trace for both coupled systems.
Start GTF with BLOK=4M, and TRACE=USRP,USR=(FE1,FEF,FF1),END
using a 1000 Cylinder GTF Trace dataset.
– F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG),BFRNUM=8
– F NET,TRACE,TYPE=BUF,ID=nodename
• The MSC LINK trace should also be turned on for both coupled systems (link# or ALL is acceptable for the link specification)
– TRACE SET ON LINK link# LEVEL 4 MODULE ALL
� IMS TYPE6701 records containing TM control blocks for each message at key points
� Note: The options on this page will affect performance, so use only if the prior dump and Query command analysis indicate it is necessary
© Copyright IBM Corporation 2017. 58
�Situation Specific Diagnostics:MSC Related Problems, Cont.
MSC related problems, continued
Issue the following SPOC commands at 1 minute intervals (5 times)
Example:
//RXSPOC3 JOB ('A=W005'),BATCH,CLASS=K,MSGLEVEL=(1,1),
// TIME=1440,USER=USRT001,PASSWORD=xxxxxxxx
/*ROUTE PRINT STLVM3/MADDIX
//SPOC EXEC PGM=CSLUSPOC,
// PARM=('IMSPLEX=PLEX1,ROUTE=IMS1,F=BYCOL')
//* PARM=('IMSPLEX=PLEX1,F=BYRSC')
//* PARM=('IMSPLEX=PLEX1,F=WRAP')
//STEPLIB DD DISP=SHR,DSN=IMSBLD.I12RTSMM.ARESLIB
//SYSPRINT DD SYSOUT=*,
// DCB=(RECFM=FB,LRECL=121,BLKSIZE=2420,DSORG=PS)
//SYSIN DD *
QUERY MSLINK NAME(LNK12V02) SHOW(ALL)
QUERY MSLINK NAME(LNK12V02) SHOW(STATISTICS)
UPDATE MSLINK NAME(LNK12V02) START(STATISTICS) OPTION(RESET)
© Copyright IBM Corporation 2017. 59
�Situation Specific Diagnostics:CQS Additional Manual Dump Intervention
� IMS/CQS environment additional dump considerations include:
• Structure dumps – Primary, Overflow, Lock entries
• CQS Region
• Other CQS Clients with their related CQS regions
• Above regions from the other IMSplex members
• z/OS Logger
© Copyright IBM Corporation 2017. 60
� Create two SYS1.PARMLIB members:
�Situation Specific Diagnostics:CQS - IEADMCxx Example with Structures
SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA –––– IMS PartnersIMS PartnersIMS PartnersIMS Partners
JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE),
DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=DSPNAME=(‘j4'.*,'XCF*'.*),REMOTE=
(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),(SYSLIST=(*('j1','j2','j3','j4','j5‘,’j6’,’j7’,’j8’,’j9’),
SDATA,DSPNAME)) SDATA,DSPNAME)) SDATA,DSPNAME)) SDATA,DSPNAME))
Where: Where: Where: Where:
j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname j1 = IMS CTL Region Jobname j3 = DBRC Region Jobname
j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname j2 = IMS DLI Region Jobname j4 = IRLM Region Jobname
j5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Region
j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM
j9 = Other CQS Client Regionj9 = Other CQS Client Regionj9 = Other CQS Client Regionj9 = Other CQS Client Region
It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down.
© Copyright IBM Corporation 2017. 61
� SYS1.PARMLIB member for structures and z/OS logger
�Situation Specific Diagnostics:CQS - IEADMCxx Example with Structures
SYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIB
JJJJOOOOBBBBNNNNAAAAMMMMEEEE====((((IIIIXXXXGGGGLLLLOOOOGGGGRRRR)))),,,,DDDDSSSSPPPPNNNNAAAAMMMMEEEE====((((''''IIIIXXXXGGGGLLLLOOOOGGGGRRRR''''....****)))),,,,
SSSSDDDDAAAATTTTAAAA====((((AAAALLLLLLLLNNNNUUUUCCCC,,,,LLLLPPPPAAAA,,,,PPPPSSSSAAAA,,,,RRRRGGGGNNNN,,,,SSSSQQQQAAAA,,,,TTTTRRRRTTTT,,,,CCCCSSSSAAAA,,,,GGGGRRRRSSSSQQQQ,,,,XXXXEEEESSSSDDDDAAAATTTTAAAA,,,,CCCCOOOOUUUUPPPPLLLLEEEE)))),,,,
STRLIST=STRLIST=STRLIST=STRLIST=
(STRNAME=(STRNAME=(STRNAME=(STRNAME=imsmsgq01imsmsgq01imsmsgq01imsmsgq01,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),
STRNAME=STRNAME=STRNAME=STRNAME=imsmsgq01oflwimsmsgq01oflwimsmsgq01oflwimsmsgq01oflw,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER)
, , , ,
STRNAME=STRNAME=STRNAME=STRNAME=z/OSlogqmsg01z/OSlogqmsg01z/OSlogqmsg01z/OSlogqmsg01,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,
ADJ=CAPTURE))ADJ=CAPTURE))ADJ=CAPTURE))ADJ=CAPTURE))
Where:Where:Where:Where:
imsmsgq01.................is the main structure nameimsmsgq01.................is the main structure nameimsmsgq01.................is the main structure nameimsmsgq01.................is the main structure name
imsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure name
z/OSlogqmsg01..............is the associated logger z/OSlogqmsg01..............is the associated logger z/OSlogqmsg01..............is the associated logger z/OSlogqmsg01..............is the associated logger
structurestructurestructurestructure
© Copyright IBM Corporation 2017. 62
�Situation Specific Diagnostics:CQS - IEADMCxx DUMP Activation
� To request a dump from the IEADMCIA and IEADDMCIB parmlib members, enter the following z/OS command:
DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)DUMP TITLE=(IMS Partners/Structures),PARMLIB=(IA,IB)
One combined dump dataset will be created on the z/OS image from which the command is entered.
One dump dataset will be created on each image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs for PARMLIB member IA.
NOTE: z/OS logger support only needs the z/OS logger address space from the system which experiences the problem.
© Copyright IBM Corporation 2017. 63
�Situation Specific Diagnostics:CQS - z/OS Log Stream
�The merged z/OS log stream can be used to examine CQS log
records
• IEBGENER can be used along with the default log stream
subsystem exit routine, IXGSEXIT, to copy the log records at
time of failure for later analysis
© Copyright IBM Corporation 2017. 64
�Situation Specific Diagnostics:CQS - z/OS Log Stream, JCL Example
//CQSCPYLG JOB USERID,USERID,MSGLEVEL=1,CLASS=K
//*********************************************************************
//* This job copies a CQS log stream to a dataset (max 32K / record) *
//* *
//* - Replace the DSN on the SYSUT1 card with your CQS logstream *
//* name. *
//* *
//* - Replace the DSN on the SYSUT2 card with your desired output *
//* dataset name. You may also need to adjust the space *
//* allocations, depending on the size of your logstream. *
//* *
//* Changes: *
//* 02/24/97 ben Creation *
//*********************************************************************
//STEP1 EXEC PGM=IEBGENER,REGION=1024K
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN=SYSLOG.MSGQ01.LOG,
// SUBSYS=(LOGR,IXGSEXIT),
// DCB=(BLKSIZE=32760)
//SYSUT2 DD DSN=CQS.LOG.COPY,
// DISP=(NEW,KEEP,DELETE),
// VOL=SER=USER05,
// SPACE=(CYL,(2,10)),
ƒ // UNIT=SYSDA
© Copyright IBM Corporation 2017. 65
�Situation Specific Diagnostics:CQS - SRDS
� Save the most recent CQS SRDS (Structure Recovery Data Set) for each structure dumped
• Use IDCAMS repro if LRECL is acceptable (less than 32761)
• If LRECL greater that 32761, use DFSDSS Copy - DFSMS DFDSS ADRDSSU utility
© Copyright IBM Corporation 2017. 66
�Situation Specific Diagnostics:ESAF Interface Problems
� Problems involving the ESAF (External Subsystem Attach Facility)
• Turn on the IMS SUBS TRACE to the external trace dataset
• IMS command: /TRA SET ON TABLE SUBS OPTION LOG
• Take a SVC dump of the related regions before and after the recreate attempt.
� Ensure that pertinent regions for the affected subsystem (db2mstr, db2dbm1, mqseries) are included with the other related IMS regions
� Note: IMSplex partner dumps are probably not required for ESAF problems, unless they are also experiencing the problem.
© Copyright IBM Corporation 2017. 67
Making it Difficult for those IMS Problems to Hide -Summary
� z/OS Setup Recommendations
� IMS Setup Recommendations
� Preservation of Standard Documentation
� Manual Dump Intervention
� Recommended SLIP traps
� Situation Specific Diagnostics
© Copyright IBM Corporation 2017. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
IBM Systems Technical Events – ibm.com/training/events
68
Your opinion matters
Your feedback about this session is very important to us.
Submit a survey at:
ibmtechu.com
© Copyright IBM Corporation 2017. Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.
IBM Systems Technical Events – ibm.com/training/events
69
IBM Systems
Technical Events
community
Join today
bit.ly/IBMTechUconnect
view event
highlights
talk to tech
experts
connect with
attendees
read training
articles
© Copyright IBM Corporation 2017.
Continue growing your IBM skills
70
ibm.com/training
provides a comprehensive
portfolio of skills and career
accelerators that are designed
to meet all your training needs.
If you can’t find the training that is right for you
with our Global Training Providers, we can help.
Contact IBM Training at [email protected]
© Copyright IBM Corporation 2017.
Thank
ibm.com/training/systems