Upload
ibm-ims
View
724
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
IBM Software Group
Jeff MaddixIBM Silicon Valley [email protected]
Making It Hard for Those IMS
Problems to Hide
Session Number 3262A
1
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
2
Please note
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
3
Making it Hard for those IMS problems to Hide - Agenda
� Section 1: MVS Setup Recommendations
� Section 2: IMS Setup Recommendations
� Section 3: Preservation of Standard Documentation
� Section 4: Manual Dump Intervention
� Section 5: Situation Specific Diagnostics
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
4
Section 1.
MVS Setup Recommendations
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
5
MVS Setup RecommendationsMaximize System Trace Table Size
�Maximize System Trace Table Size.
� Advantages:
� Extremely valuable for a large variety of problem types such at ABENDs, loops, and performance issues.
�Setup
� Set MVS System Trace table size to 999K or 3M in z/OS 1.9 or higher
–Default size is only 1M
–z/OS 1.9 max size is 9G
� The MVS command "TRACE ST,999K" or “TRACE ST,3M” can be specified in the MVS COMMNDxx SYS1.PARMLIB member.
� See the z/OS MVS Systems Commands Manual for further detail
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
6
�Maximize System Trace Table Size.
� Use the MVS Display command to ensure the system trace table is active:
–DISPLAY TRACE
IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479
SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION
ST=(ON,3000K,12000K)ST=(ON,3000K,12000K)ST=(ON,3000K,12000K)ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K)
�Considerations:
� System Trace Table is page fixed storage.
� Supply a reasonable value to the nnnM or nG parameter after considering the available central storage and the actual storage required for system trace.
� Supplying a large value for nnnM or nG might cause a shortage of pageablestorage in the system.
MVS Setup RecommendationsMaximize System Trace Table Size
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
7
MVS Setup RecommendationsIncrease Master Trace Table Size � Increase master trace table size.
�Advantages:
� Master trace maintains a table of the most recently issued operator messages.
� Allows view of external events at the time of failure.
� Setup
� Set MVS Master Trace Table size to 1000K
–Default size is only 24K
� "TRACE MT,1000K" can be specified in the SCHEDxx member of SYS1.PARMLIB.
� See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning
Guide/ Ref, and z/OS MVS System Commands manuals for complete details
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
8
MVS Setup RecommendationsIncrease Master Trace Table Size
� Increase master trace table size
�Use the MVS Display command to ensure the master trace table is active:
–DISPLAY TRACE
IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479 IEE843I 13.17.07 TRACE DISPLAY 479
SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION SYSTEM STATUS INFORMATION
ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF ST=(ON,3000K,12000K) AS=ON BR=OFF EX=ON MO=OFF MT=(ON,1000K)MT=(ON,1000K)MT=(ON,1000K)MT=(ON,1000K)
�Considerations:
� Ensure Master Trace Table is large enough to span most error time frames
� Uses Subpool 229 Key 0 High Private Pageable Storage of the master scheduler address space
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
9
MVS Setup RecommendationsUse the Common Storage Tracker
�Use the Common Storage Tracker
� Advantages:
� SVC dumps (or RMF reports) will provide CSA/ECSA ownership information with jobname, time, and requesting module information.
� Great for common storage outage (storage leaks) problems and fragmentation analysis.
�Setup
� Use the DIAGxx SYS1.PARMLIB member to contain the request. Specify
DIAG=xx in the IPL system parameters or use the SET DIAG=xx operator command.
–Example: DIAGxx member: VSM TRACK CSA(ON) SQA(ON)
� See z/OS MVS Diagnosis: Tools and Service Aids, z/OS Initialization and Tuning
Guide/ Reference, and z/OS MVS System Commands manuals for complete details.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
10
MVS Setup RecommendationsUse the Common Storage Tracker
�Use the Common Storage Tracker
� Display the Common Storage Tracker status by using the following MVS command:
�DISPLAY DIAG
�Considerations:
� Negligible Performance impact. Table stored in ESQA
IGV007I 11.44.05 DIAG DISPLAY 713
VSM TRACK CSA(ON) SQA(ON)
VSM TRACE GET(OFF) FREE(OFF)
VSM ALLOWUSERKEYCSA(YES)
VSM BESTFITCSA(NO)
VSM USEZOSV1R9RULES(YES)
TRAPS NAME()
CBLOC
VIRTUAL24()
VIRTUAL31()
REUSASID(NO)
AUTOIPL SADMP(NONE) MVS(NONE)
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
11
MVS Setup RecommendationsTune CHNGDUMP MAXSPACE
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC
dump.
�Advantages:
� Higher likelihood that SVC dumps will be captured in their entirety without worry of "Partial Dump".
�Setup
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC dump.
� Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate
CHNGDUMP command at IPL time.
–Example: CD SET,SDUMP,MAXSPACE=3000M - Default size is 500M
�3000M is standard for large multi-address space SVC Dumps.
–See z/OS MVS Diagnosis: Tools and Service Aids, z/OS MVS System
Commands, and z/OS Initialization and Tuning Guide for additional details.
�Continued on next page
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
12
MVS Setup RecommendationsTune CHNGDUMP MAXSPACE
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC
dump.
Display the MAXSPACE setting using the following MVS Command:
D DUMP,OPTIONS
SDUMP- ADD OPTIONS
(ALLPSA,NUC,SQA,LSQA,RGN,LPA,TRT,CSA,SWA,SUMDUMP,
Q=YES,GRSQ,XESDATA),BUFFERS=00000000K,
MAXSPACE=00003000M,MSGTIME=99999 MINUTES,
MAXSNDSP=015 SECONDS,AUXMGMT=ON ,DEFERTND=NO
•Continued on next page
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
13
MVS Setup RecommendationsTune CHNGDUMP MAXSPACE
� Ensure adequate CHNGDUMP MAXSPACE is specified to hold the internal SVC
dump.
�Considerations:
�MAXSPACE is the total virtual storage used by SDUMP
�SDUMP data capture is serialized so only one dump can be written at a time.
However, MVS releases the serialization after data capture, so another dump can be started before the first one gets written out (especially if something causes a delay in writing the first dump out to DASD). So MAXSPACE virtual storage might have to be temporarily "shared" by more than one dump.
� Ensure central storage and/or local page datasets are large enough to contain their normal peak load plus additional SVC dumps!
� See MVS Initialization and Tuning Guide for additional details.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
14
MVS Setup RecommendationsAutomatic Dump Dataset Allocation
� Ensure automatic dump dataset allocation is in place.
�Advantages:
� SVC dumps will be allocated to the correct size without worry of "Partial Dump".
�Setup
� Use the COMMNDxx SYS1.PARMLIB member to issue the appropriate DUMPDS commands to set this up.
–Example: DUMPDS NAME=, DUMPDS ADD, and DUMPDS ALLOC=ACTIVE
–See z/OS MVS Diagnosis: Tools and Service Aids and z/OS MVS Systems Commands manuals for complete details.
�Considerations:
� Ensure assigned storage class has enough space to ensure adequate SVC dump storage requirements
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
15
�Section 2.
IMS Setup Recommendations
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
16
IMS Setup RecommendationsUse FMTO Option D
� Specify IMS control region EXEC parm value: FMTO=D
�Produces an SDUMP for terminating and non-terminating errors
� Non-terminating errors include:
– IMS dynamic allocation failures.
– Some ESAF (External Subsystem Attach Facility) failures.
�Allows use of SYSMDUMP as a backup dump in case of SDUMP failures.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
17
IMS Setup RecommendationsSYSMDUMP DD for IMS Control Regions
� The SYSMDUMP specification will be used as a backup dump by IMS in the event
that SDUMP processing should fail.
� Specify SYSMDUMP DD in JCL of:
�IMS CTL Region.
�IMS DLI/SAS Region.
�IMS DBRC Region.
� The following dump options should be specified in the SYS1.PARMLIB(IEADMR00)
member to ensure that adequate areas of MVS storage are dumped to diagnose the
problem under most circumstances:
� SDATA=(CSA,RGN,SQA,SUM,TRT,NUC,LPA)
�The SYSMDUMP SDATA options can be added via the CHNGDUMP command
�CD SET,SYSMDUMP=(CSA,RGN,SQA,SUM,TRT,NUC,LPA)
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
18
IMS Setup RecommendationsSYSMDUMP DD, Continued
� SYSMDUMP DD Specification
�DISP=MOD should be specified because it is possible for multiple tasks to
generate dumps during the processing of an error. We probably will need to see
the first dump produced.
GDG Example
//SYSMDUMP DD DISP=(MOD,CATLG),DSN=IMSSP05.IBA1IMS.SYSMDUMP(+1),
// DCB=(LRECL=4160,RECFM=FB,BLKSIZE=29120),UNIT=3390,
// SPACE=(CYL,(750,100),RLSE),DATACLAS=dtcls,VOL=SER=volser
� Testing the SYSMDUMP function can be done on a test system by temporarily setting the SDUMP to NODUMP via the CHNGDUMP command. This forces a SYSMDUMP to be created.
�CD SET,SDUMP,NODUMP
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
19
IMS Setup RecommendationsSYSUDUMP DD
� The SYSUDUMP specification will be used by IMS dependent regions for failure events.
�Many application dump formatter tools rely on the SYSUDUMP DD statement to be present in the JCL.
� Specify SYSUDUMP DD in JCL of:
�IMS Dependent Regions.
� The following dump options should be specified in the MVS SYS1.PARMLIB(IEADMP00) member to ensure that adequate areas of MVS storage are dumped.
� SDATA=(CB,ERR,SUM) PDATA=(JPA,LPA,PSW,REGS,SA,SPLS)
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
20
IMS Setup RecommendationsIMS Table Traces Activated
� Turn on the IMS Dispatcher, Scheduler, DLI, and Lock traces to capture general system diagnostics and database access history
�Specify the following options in the IMS PROCLIB member: DFSVSMxx
� DISP=ON, SCHD=ON, DL/I=ON, LOCK=ON
�Or, use IMS command:
� TRA SET ON TABLE nnnn
� Where nnnn is alternately = DISP, SCHD, DLI, and LOCK
�These traces have relatively low CPU overhead
� SCHD being the lowest and DLI being the highest.
� For non-production environments, when recreating failures, run with traces for the functional areas being tested
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
21
IMS Setup RecommendationsExternal Trace Environment
� IMS external tracing allows for IMS trace table output to be placed on IMS trace datasets rather than on the IMS OLDS when:
� DFSVSMxx PROCLIB member with nnnn=OUT is specified rather than nnnn=ON
�DISP=ON, SCHD=ON, DL/I=OUT, LOCK=OUT
� The IMS TRACE command specifies OPTION LOG
�/TRA SET ON TABLE nnnn OPTION LOG
� Increases IMS system throughput should external tracing be desired versus sending trace data to the IMS OLDS.
� External trace dataset logging adds less than 1% to the system assuming IO subsystem isn’t constrained
� External trace datasets are allocated in the following order:
1) DASD JCL - DFSTRA01 and DFSTRA02 DDNAMES
2) DASD MDA- DFSTRA01 and DFSTRA02 Dynamic Allocation Members
3) TAPE MDA - DFSTRA0T Dynamic Allocation Member
4) IMS OLDS - If none of the above are found
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
22
IMS Setup RecommendationsExternal Trace Environment
� IMS external tracing specifications
• DFSTRA01 and DFSTRA02DSORG SequentialRECFM VBLRECL 4004BLKSIZE 20024 (LRECLx n +4)
-Set up unique GDG with - minimum 24 hour limit for archive
- ideal to last IMS restart or longer
hlq.imsid.TRACE.ARCHIVE(0)
-Use automation to archive filled trace dataset off to trace archive GDG datasets on external trace dataset switch
DFS2864I EXTERNAL TRACE DATA SET xxxxxxxx FULL - SWITCHING TO yyyyyyyy. imsid
- and for IMS ABEND Scenario –
DFS3896I ATTEMPTING TO PURGE OLDS BUFFERS
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
23
IMS Setup RecommendationsExternal Trace Environment
� Display IMS external tracing specification
Command: /DIS TRACE XTRC.
IMS ACTIVE TRACES
IMS EXTERNAL TRACE IS TRACING TO DISK
XTRC DDNAME ALLOC STATUS STATUS
DFSTRA01 DYNA ALLOCATED CLOSED
DFSTRA02 DYNA ALLOCATED ACTIVE
DFSTRA0T DYNA UNALLOCATED CLOSED
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
24
�Section 3.
Preservation of Standard Documentation
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
25
Preservation of Standard Documentation
�Consider implementing normal operating procedures to preserve the following documentation near the time of error.
�MVS Console (SYSLOG)
�SYS1.LOGREC
�IMS OLDS/SLDS
�JES JOBLOG of jobs related to failure
�Any Dumps Produced
�For elusive problems, these items might be requested a month or more after the
problem has been reported.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
26
Ideal Time Periods for Syslog, Logrec, SLDS, and Joblog Preservation
�MVS Console (SYSLOG), Logrec, SLDS and Joblogs
–The ideal time frame:
�Back to the last IMS restart.
�MVS Console from the prior clean execution (For comparison)
�The moderate time frame:
�24 hours of MVS Console messages.
�The minimum time frame:
�Two IMS system checkpoint intervals.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
27
Dump Preservation
� All dumps created near the time of IMS failure should be retained.
� SYS1.DUMP Datasets should be examined.
� Multiple dumps may be created.
� Keep all dumps at time of failure regardless of the subsystem.
� SYSMDUMP for the IMS Control, DLI/SAS, and DBRC regions need to be examined in case of primary SYS1.DUMP failures.
� Save these datasets if a dump was produced.
� If GDG is used for copies, examine in ISPF for tracks used
� SYSUDUMP should be saved for affected IMS dependent regions.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
28
�Section 4.
Manual Dump Intervention
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
29
Additional Manual Dump Intervention
�While IMS produces SDUMPs for most internal errors without human intervention, IMS Wait/Loop or partial loss of function conditions will require intervention to produce an SVC dump
�IMS hangs can be caused by interaction with many address spaces including:
CTL DLI/SAS DBRC IRLM CQS
OM RMSCIODBA
cctlMPP
(DBDC,
DBCTL,
DCCTL)
IFP BMP
IMS Connect
JMP JBP
APPC VTAMTCPIP
RRS
MQ
Series
Non-IBM
Tools
IBM Tools
IXGLOGR
DB2ODBM
WLM
Other?
**
******
*
*
ap thr ap thr ap thr ap thrap thr ap thr
thrap
Subsys *
Subsys *
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
30
How Much Data to Dump?
� Due to the complex interactions between the many address spaces, it is sometimes difficult to determine exactly where thesource of the problem lies without a dump of the associated address spaces and their associated data areas.
� Omission of any interrelated address space or data area adds to the possibility that the dump may not be sufficient to solve the problem.
�MVS dumps have a maximum job total of 15 per dump
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
31
IMS Dump Techniques
IMS SVC Dumps can be requested using various techniques
�MVS SYS1.PARMLIB IEADMCxx
� DUMP Command Parmlib
�MVS SYS1.PARMLIB IEASLPxx
�SLIP Command Parmlib
�Manual MVS DUMP command
�Customized JCL can be built and submitted
�Others?....
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
32
IEADMCxx, MVS SYS1.PARMLIB
� DUMP Command Parmlib Member
� Create SYS1.PARMLIB members called IEADMCxx for each customized dump command.
�Advantages
– Can be used to customize IMS dumps prior to error event.
– Simple operator interface.
�Disadvantages
– All Jobs must be active or no dump is produced on the system on which the
dump command is entered. Remote systems without a jobname match will dump the DUMPSRV address space.
– Avoid issues:
–Use wildcards * (multiple chars) and ? (single char) and similar naming conventions can be used to specify the jobnames.
–Edit the Parmlib member jobnames prior to issuing the command
� See z/OS MVS Initialization and Tuning Guide and z/OS System Commands
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
33
� Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters:
IEADMCxx Example for IMS
JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),JOBNAME=(j1,j2,j3,j4,j5,j6,j7),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ)
Where: Where: Where: Where:
j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.j1 = IMS Control Region Jobname. j2 = DBRC Region Jobname.
j3 = IMS DLI Region Jobname. j4 = IRLM Region j3 = IMS DLI Region Jobname. j4 = IRLM Region j3 = IMS DLI Region Jobname. j4 = IRLM Region j3 = IMS DLI Region Jobname. j4 = IRLM Region JobnameJobnameJobnameJobname....
j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2j5 = IMS CCTL Region 1. j6 = IMS CCTL Region 2
j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.j7 = IMS CCTL Region 3.
Note: It is recommended to use wildcards if possible to ensure jobname matches to allow
for cases where one or more regions might already be down.
Example: Using the following naming conventions, the JOBNAME statement could be
reduced: IMSPCTLA, IMSPDLIA, IMSPDRCA IMSPRLMA CICSP1A CICSP2A CICSP3A
JOBNAME=(IMSP*A,CICSP?A)
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
34
IEADMCxx DUMP Activation
� To request a dump from the IEADMCI1 parmlib member, enter the following MVS commands:
� DUMP TITLE=(DUMP OF IMS and CCTL Regions ),PARMLIB=(I1)
A dump dataset will be created on the MVS image from which the dump command is entered
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
35
IMS SYSPLEX Dump Considerations
�IMS SYSPLEX implementations need to consider the possibility that a hang or problem on one IMSplex member may be due to a problem originating from another member, particularly in the area of IRLM locking
�Problems such as IMS Wait/Loops or partial loss of function conditions which require intervention to produce an SVC dump, should include SVC dumps from other members of the IMSplex
�Challenge:
� Ensure that a dump is taken for all necessary address spaces on each system.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
36
� Create SYS1.PARMLIB member called IEADMCI1 containing the following DUMP parameters:
Sysplex IEADMCxx Example
It is recommended to use wildcards if possible to ensure jobname matches to allow for
cases where one or more regions might already be down.
Note: The XESDATA, COUPLE and REMOTE parms are for use in SYSPLEX environments
JOBNAME=(j1,j2,j3,j4,j5,j6,j7),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE),
DSPNAME=('j4'.*,'XCF*'.*),
REMOTE=(SYSLIST=(*('j1','j2','j3','j4’,’j5’,’j6’,’j7’),
SDATA,DSPNAME))
Where:
j1 = IMS Control Region Jobname j2 = DBRC Region Jobname
j3 = IMS DLI Region Jobname j4 = IRLM Region Jobname
j5 = CCTL Region 1 J6 = CCTL Region 2
j7 = CCTL Region 3
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
37
Sysplex IEADMCxx DUMP Activation
� To request a dump from the IEADMCI1 parmlib member, enter the following MVS command:
� DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=(I1)
� A dump dataset will be created on each MVS image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
38
Storage as a Moving Target
� Storage can be a moving target while being dumped, especially if multiple address spaces interact. Here are some considerations:
�If the MVS sdump CHNGDUMP SET, SDUMP=(Q=Y) command , is specified, all address spaces that are not exempt from system-wide nondispatchability are set nondispatchable while global storage (CSA, SQA) is dumped.
–By quiescing most system activity during SVC dump global storage capture, the dump will contain a more consistent image however, your system activity will nearly cease during global storage capture.
– This option is generally not recommended for this reason.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
39
Storage as a Moving Target, Continued
� Storage can be a moving target, continued…
•In all cases (Q=Y or Q=N), dumped address spaces are made nondispatchable while global storage and the private region storage is being dumped for those address spaces
�There is no intentional dispatching coordination between requested address spaces in a dump.
�All private storage is dumped through scheduling of SRBs to the requested address spaces.
�This SRB activity is asynchronous.
� Multiple SRBs to multiple address spaces will run independentlyand loosely in parallel
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
40
�Section 5.
IMS Situation Specific Diagnostics
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
41
�Situation Specific Diagnostics:Control Region Wait/Hang
� Control Region Wait/Hang
� Take a SVC dump using one of the methods discussed earlier in the presentation.
� Consider dumping related regions
�If a partial wait occurs, consider taking two of the above mentioned SVC Dumps to identify normal activity versus abnormal activity.
�Note: Modify dumps of IMS are no longer the preferred method to obtain wait documentation.
�IMS Control Region
�IMS DLI/SAS Region
�DBRC Region
�IRLM Region
�CQS
�Operations Mgr
�Resource Mgr
�Structure Call Interface
�TCPIP
�Websphere
�ESAF - DB2, MQSeries, others
�Other Regions ???
�Other IMSplex members with
all their related regions
�ODBM
� = Can't do without
�Troublesome IMS Dep Regs
�CCTL Regions
�ODBA
�IXGLOGRC
�RRS
�APPC
�WLM
�VTAM
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
42
�Situation Specific Diagnostics:Control Region Wait/Hang, Continued
JOBNAME=(isa?ims,isa?dls,isa?irlm,ci1?sa??),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE),
DSPNAME=('ISA?IRLM'.*,'XCF*'.*),
REMOTE=(SYSLIST=(
SP1A('isa?ims','isa?d*','isa?irlm','ci1?sa??'),
SP1C('isa?ims','isa?d*','isa?irlm','ci1?sa??'),
SP1E('isa?ims','isa?d*','isa?irlm','ci1?sa??'),
SP1F('isa?ims','isa?d*','isa?irlm','ci1?sa??'),
SDATA,DSPNAME))
� IEADMCxx Parmlib (IEADMCA1) example for a slow moving performance problem or hang in a DBCTL data sharing environment with IRLM
� Dump Command to activate the above dump:
� DUMP TITLE=(IMS/CCTL SYSPLEX Dumps),PARMLIB=(A1)
� One dump will be created on each of the specified sysplex members
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
43
�Situation Specific Diagnostics:Application System Timeout
� Application lockouts are detected by IRLM and MSG DXR162I is issued.
DXR162I irlmx CYCLE NUMBER nnnnnnnn PROCESSED FOR TIMEOUT
Explanation: A TIMEOUT candidate has been detected on at least one IRLM. If an
incompatible blocker was found, the IMS TIMEOUT EXIT has been driven to
present WAITER/BLOCKER information
Set the following SLIP to capture a dump for this event
SLIP SET,A=(SVCD),MSGID=DXR162I,
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ,XESDATA,COUPLE),
JL=(ISA*,CI1?SA??),ID=M162,DSPNAME=('ISA?IRLM'.*,'XCF*'.*),
REMOTE=(SYSLIST=(SP1A,SP1C,SP1E,SP1F),JOBLIST,DSPNAME,SDATA),
END
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
44
�Situation Specific Diagnostics:IMS Loop
� If the IMS Control, DLI, DBRC or Dependent Region enters a loop state:
�Turn on the MVS System Trace Table with BR=ON
�TRACE ST,3M,BR=ON - (size greater than 1m in z/OS 1.9 and above)
�If possible, and not already set, use IMS command: /TRA SET ON TABLE nnnn
� Where nnnn alternately = DISP, SCHD, DLI, LOCK, and LATCH.
� Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions.
� Note: IMSplex partner dumps are probably not required for loop problems, unless they are also looping.
� Reset the MVS system trace table and IMS trace tables to their original settings.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
45
�Situation Specific Diagnostics:DBRC Related Problems
� DBRC Related Problems
�Obtain a listing of the DBRC RECONs prior to repair attempts.
� Use Recover Control Utility (DSPURX00) - LIST.RECON
command.
� If the RECON dataset listing cannot be obtained due to size,
then alternatively obtain a subsystem listing (LIST.RECON is
preferred).
� Use Recover Control Utility (DSPURX00) - LIST.SUBSYS
ALL command
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
46
�Situation Specific Diagnostics:DBRC Related Problems, Continued� DBRC Related Problems, Continued
�Use D GRS,CONTENTION command on each system sharing the RECON to
determine if the RECON is held at the exclusion of other waiters
D GRS,CONTENTION
S=SYSTEMS DSPURI01 RECON1.IMSBP1
SYSNAME JOBNAME ASID TCBADDR EXC/SHR STATUS
BT21 IBA2FDBR 0124 009AF7F8 EXCLUSIVE OWN
BT23 IBA1FDBR 0090 009AF1B0 EXCLUSIVE WAIT
BT21 IBA4FDBR 05E0 009AF7F8 EXCLUSIVE WAIT
BT26 IBA4DBRC 05DE 009F15B8 EXCLUSIVE WAIT
BT21 IBA1DBRC 00D7 009FF118 EXCLUSIVE WAIT
BT23 IBA2DBRC 0600 009F15C8 EXCLUSIVE WAIT
� If so, dump the owning address space
DUMP COMM=(dump title)
R nn,JOBNAME=(j1),SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,ALLNUC,LPA,GRSQ),END
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
47
�Situation Specific Diagnostics:DBRC Related Problems, Continued
� Obtain before and after copies of the RECONs
�JCL Example for Recon dump to file
//JJJJJJJJ JOB ( )
//* DUMP RECON TO FILE
//STEP1 EXEC PGM=ADRDSSU,REGION=0M
//SYSPRINT DD SYSOUT=*
//DASD DD DSN=hhhhhhhh.RECON.BACKUP,DISP=(NEW,CATLG),
// UNIT=3390,SPACE=(CYL,(5,5),RLSE)
//SYSIN DD *
DUMP DS(INCLUDE(IMS.RECON*)) -
OUTDD(DASD)
/*
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
48
�Situation Specific Diagnostics:DBRC Related Problems, Continued
� JCL Example for Recon restore from file
//JJJJJJJJ JOB ( )
//* TO DOWNLOAD TAPE LOADED WITH DFSDSS
//* NEED TO USE RENUNC TO RENAME TO YOUR HLQ
//* TAPE MAP WILL NOT SHOW LRECL OR BLKSIZE....
//S01RSTR EXEC PGM=ADRDSSU
//SYSPRINT DD SYSOUT=*
//IN DD DISP=SHR,DSN=hhhhhhhh.RECON.BACKUP
//DASD DD UNIT=3390,DISP=OLD,VOL=SER=SYS285
//SYSIN DD *
RESTORE DATASET(INCLUDE( ** )) -
INDDNAME(IN) -
OUTDDNAME(DASD) -
CANCELERROR -
CATALOG -
REPLACE -
IMPORT -
TGTALLOC(SOURCE) -
RENAMEUNCONDITIONAL(hhhhhhhh) -
WAIT(2,2)
/*
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
49
�Situation Specific Diagnostics:DBCTL Related Problems
� DBCTL Related Problems
� Issue IMS /DISPLAY ACTIVE and /DISPLAY CCTL commands, multiple times
� /DIS ACTIVE
� /DIS CCTL ALL
� /DIS CCTL cctlname
� Set the AP portion of the CICS trace to level 1-2
� Set the FC (FILE CONTROL) portion of the CICS trace to level 1-2
�Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions, especially the CCTL regions
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
50
�Situation Specific Diagnostics:Database Problems
�For problems in the database arena, obtain
�Damaged database dataset
�Database Image copy of broken database in state prior to damage
�Image Copy of logically related databases
�IMS SLDS from all data sharing IMS subsystems
� Save to last good database image copy of damaged database
�If possible, and not already set, use IMS command and save output:
� /TRA SET ON TABLE nnnn OPTION LOG
– Where nnnn alternately = DLI and LOCK
�SYSOUT from the Pointer Checker broken database
�SYSOUT from batch jobs that accessed the damaged database
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
51
�Situation Specific Diagnostics:Database Problems, Continued
�For problems in the database arena, obtain (Cont...)
�DBD for broken databases
�PSBs that access the broken DB
�List.RECON and LIST.HISTORY DBD(broken database)
�SMF 60 (VSAM Volume Data Set Updated), 62 (VSAM Component or Cluster Opened), and 64 (VSAM Component or Cluster Status) records from all data sharing systems
� Back to last good image copy of broken database
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
52
�Situation Specific Diagnostics:Database Problems, Continued
�For problems in the database arena, obtain (continued)
�For VSAM datasets, close the broken KSDS and run IDCAMS commands:
� VERIFY
� EXAMINE ITEST NODTEST
� EXAMINE NOITEST DTEST� LISTCAT ALL
� If the dataset cannot be closed, run IDCAMS commands:
� EXAMINE ITEST NODTEST� EXAMINE NOITEST DTEST
� EXAMINE ITEST NODTEST
� EXAMINE NOITEST DTEST� LISTCAT ALL
� If the data set is open and being updated then an EXAMINE may reflect what appears to be a problem because the
data set is in a state of flux. Thus if you run the EXAMINEs twice and the errors are the same, there is probably a
problem. If the errors are different, then there is no problem.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
53
�Situation Specific Diagnostics:ESAF Interface Problems
� Problems involving the ESAF (External Subsystem Attach Facility)
� Turn on the IMS SUBS TRACE to the external trace dataset
ƒIMS command: /TRA SET ON TABLE SUBS OLDS
� Take a SVC dump of the related regions before and after the recreate attempt.
�Ensure that pertinent regions for the affected subsystem (db2mstr, db2dbm1, mqseries) are included with the other related IMS regions
�Note: IMSplex partner dumps are probably not required for ESAF problems, unless they are also experiencing the problem.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
54
�Situation Specific Diagnostics:DC Related Problems
� Problems involving Data Communications
�Issue IMS DISPLAY NODE x command.
� /DIS NODE nodename
�Turn on IMS NODE trace. Data will be captured in the IMS TYPE6701 log record. Print records using IMS utility programs DFSERA10/DFSERA30.
� /TRA SET ON NODE nodename
�Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS NODE trace. Note: GTF must be active with the USR option.
� F NET,TRACE,TYPE=BUF,ID=nodename
� F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG)
�Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
55
�Situation Specific Diagnostics:APPC Related Problems
� Problems involving APPC
� Turn on the IMS LUMI Trace to the external trace dataset.
� /TRACE SET ON TABLE LUMI OPTION LOG
� /TRACE SET ON LUNAME XXXXXXX INPUT
� /TRACE SET ON LUNAME XXXXXXX OUTPUT
– (XXXXXXX is the partner LU)
�Turn on the VTAM Buffer Trace and VTAM Internal Trace to complement the IMS LUMI trace. Note: GTF must be active with the USR option.
� F NET,TRACE,TYPE=BUF,ID=nodename
� F NET,TRACE,TYPE=VTAM,MODE=EXT,OPT=(API,PIU,MSG)
�Take a SVC dump using one of the methods discussed earlier in the presentation with related IMS regions - especially VTAM and the APPC regions.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
56
�Situation Specific Diagnostics:APPC Related Problems, Continued
� Problems involving APPC (cont...)
�Turn on the program trace for TPPCB DL/1 calls.
� /TRACE SET ON PROGRAM pppppppp
– pppppppp is the program name of the application.
� Turn on the MVS APPC component trace.
� TRACE CT,ON,200M,COMP=SYSAPPC
– nn,OPTIONS=(GLOBAL),END
� When the problem has been recreated, stop the CTRACE.
� TRACE CT,OFF,COMP=SYSAPPC
� APPC component trace sends its trace buffers to a SYS1.DUMP dataset.
� The following IPCS commands can be used to format the trace:
� CTRACE COMP(SYSAPPC) SHORT - One line summary entries
� CTRACE COMP(SYSAPPC) FULL - Detailed trace entryformatting
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
57
�Situation Specific Diagnostics:CQS Additional Manual Dump Intervention
� IMS/CQS environment additional dump considerations include:
�Structure dumps – Primary, Overflow, Lock entries
�CQS Region
�Other CQS Clients with their related CQS regions
�Above regions from the other IMSplex members
�MVS Logger
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
58
� Create two SYS1.PARMLIB members:
�Situation Specific Diagnostics:CQS - IEADMCxx Example with Structures
SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA SYS1.PARMLIB IEADMCIA –––– IMS PartnersIMS PartnersIMS PartnersIMS Partners
JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),JOBNAME=(j1,j2,j3,j4,j5,j6,j7,j8,j9),
SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE), SDATA=(CSA,PSA,RGN,SQA,SUM,TRT,GRSQ,XESDATA,COUPLE),
DSPNAME=(DSPNAME=(DSPNAME=(DSPNAME=(‘‘‘‘j4'.*,'XCF*'.*),REMOTE=j4'.*,'XCF*'.*),REMOTE=j4'.*,'XCF*'.*),REMOTE=j4'.*,'XCF*'.*),REMOTE=
(SYSLIST=(*('j1','j2','j3','j4','j5(SYSLIST=(*('j1','j2','j3','j4','j5(SYSLIST=(*('j1','j2','j3','j4','j5(SYSLIST=(*('j1','j2','j3','j4','j5‘‘‘‘,,,,’’’’j6j6j6j6’’’’,,,,’’’’j7j7j7j7’’’’,,,,’’’’j8j8j8j8’’’’,,,,’’’’j9j9j9j9’’’’),),),),
SDATA,DSPNAME)) SDATA,DSPNAME)) SDATA,DSPNAME)) SDATA,DSPNAME))
Where: Where: Where: Where:
j1 = IMS CTL Region j1 = IMS CTL Region j1 = IMS CTL Region j1 = IMS CTL Region JobnameJobnameJobnameJobname j3 = DBRC Region j3 = DBRC Region j3 = DBRC Region j3 = DBRC Region JobnameJobnameJobnameJobname
j2 = IMS DLI Region j2 = IMS DLI Region j2 = IMS DLI Region j2 = IMS DLI Region JobnameJobnameJobnameJobname j4 = IRLM Region j4 = IRLM Region j4 = IRLM Region j4 = IRLM Region JobnameJobnameJobnameJobname
j5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Regionj5 = IMS CQS Region j6 = APPC Region
j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM j7 = APPC Scheduler j8 = VTAM
j9 = Other CQS Client Regionj9 = Other CQS Client Regionj9 = Other CQS Client Regionj9 = Other CQS Client Region
It is recommended to use wildcards if possible to ensure jobname matches to allow for cases where one or more regions might already be down.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
59
� SYS1.PARMLIB member for structures and MVS logger
�Situation Specific Diagnostics:CQS - IEADMCxx Example with Structures
SYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIBSYS1.PARMLIB IEADMCIB
JOBNAME=(IXGLOGR),DSPNAME=('IXGLOGR'.*), JOBNAME=(IXGLOGR),DSPNAME=('IXGLOGR'.*), JOBNAME=(IXGLOGR),DSPNAME=('IXGLOGR'.*), JOBNAME=(IXGLOGR),DSPNAME=('IXGLOGR'.*),
SDATA=(ALLNUC,LPA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA,COUPLE), SDATA=(ALLNUC,LPA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA,COUPLE), SDATA=(ALLNUC,LPA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA,COUPLE), SDATA=(ALLNUC,LPA,PSA,RGN,SQA,TRT,CSA,GRSQ,XESDATA,COUPLE),
STRLIST=STRLIST=STRLIST=STRLIST=
(STRNAME=(STRNAME=(STRNAME=(STRNAME=imsmsgq01imsmsgq01imsmsgq01imsmsgq01,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER), ,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),
STRNAME=STRNAME=STRNAME=STRNAME=imsmsgq01oflwimsmsgq01oflwimsmsgq01oflwimsmsgq01oflw,LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER),LOCKE,(LISTNUM=ALL,ADJ=CAPTURE,EDATA=UNSER)
, , , ,
STRNAME=STRNAME=STRNAME=STRNAME=mvslogqmsg01mvslogqmsg01mvslogqmsg01mvslogqmsg01,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,,LOCKE,ACC=NOLIM,(LISTNUM=ALL,EDATA=UNSER,
ADJ=CAPTURE))ADJ=CAPTURE))ADJ=CAPTURE))ADJ=CAPTURE))
Where:Where:Where:Where:
imsmsgq01.................is the main structure nameimsmsgq01.................is the main structure nameimsmsgq01.................is the main structure nameimsmsgq01.................is the main structure name
imsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure nameimsmsgq01oflw.............is the overflow structure name
mvslogqmsg01..............is the associated logger structurmvslogqmsg01..............is the associated logger structurmvslogqmsg01..............is the associated logger structurmvslogqmsg01..............is the associated logger structureeee
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
60
�Situation Specific Diagnostics:CQS - IEADMCxx DUMP Activation
� To request a dump from the IEADMCIA and IEADDMCIB parmlib members, enter the following MVS command:
DUMP TITLE=(IMS Partners/DUMP TITLE=(IMS Partners/DUMP TITLE=(IMS Partners/DUMP TITLE=(IMS Partners/Structures),PARMLIBStructures),PARMLIBStructures),PARMLIBStructures),PARMLIB=(IA,IB)=(IA,IB)=(IA,IB)=(IA,IB)
One combined dump dataset will be created on the MVS image from which the command is entered.
One dump dataset will be created on each image in the SYSPLEX matching the REMOTE specifications for the JOBNAMEs for PARMLIB member IA.
�NOTE: MVS logger support only needs the MVS logger address space from the system which experiences the problem.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
61
�Situation Specific Diagnostics:CQS - MVS Log Stream
�The merged MVS log stream can be used to examine CQS log
records.
�IEBGENER can be used along with the default log stream
subsystem exit routine, IXGSEXIT, to copy the log records at
time of failure for later analysis
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
62
�Situation Specific Diagnostics:CQS - MVS Log Stream, JCL Example
//CQSCPYLG JOB USERID,USERID,MSGLEVEL=1,CLASS=K
//*********************************************************************
//* This job copies a CQS log stream to a dataset (max 32K / record) *
//* *
//* - Replace the DSN on the SYSUT1 card with your CQS logstream *
//* name. *
//* *
//* - Replace the DSN on the SYSUT2 card with your desired output *
//* dataset name. You may also need to adjust the space *
//* allocations, depending on the size of your logstream. *
//* *
//* Changes: *
//* 02/24/97 ben Creation *
//*********************************************************************
//STEP1 EXEC PGM=IEBGENER,REGION=1024K
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT1 DD DSN=SYSLOG.MSGQ01.LOG,
// SUBSYS=(LOGR,IXGSEXIT),
// DCB=(BLKSIZE=32760)
//SYSUT2 DD DSN=CQS.LOG.COPY,
// DISP=(NEW,KEEP,DELETE),
// VOL=SER=USER05,
// SPACE=(CYL,(2,10)),
ƒ // UNIT=SYSDA
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
63
�Situation Specific Diagnostics:CQS - SRDS
� Save the most recent CQS SRDS (Structure Recovery Data
Set) for each structure dumped
�Use IDCAMS repro if LRECL is acceptable (less than 32761)
�If LRECL greater that 32761, use DFSMS ADRDSSU utility
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
64
�Situation Specific Diagnostics:MSC Related Problems
� If you use IMS MSC (Multiple Systems Coupling) and experience a related
problem
�Take a SVC dump of the coupled IMS regions (minimally the CTL
regions, but the problem may reside in any IMS related region as
discussed earlier)
� In addition, ensure that the VTAM and/or TCPIP address space is
included. Do this as close to the problem time as possible, prior to
attempting to fix the problem
� Save the IMS OLDS for both coupled systems from the time of the
message creation
�Issue DISPLAY LINK ALL and DISPLAY LINK ALL MODE commands to
show the status and queue counts for the logical link and physical link
information for partner
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
65
�Situation Specific Diagnostics:MSC Related Problems, Cont.
MSC related problems, continued
If the problem is recreatable,
� If using VTAM, Turn on the VTAM Internal Trace for both coupled systems
– F NET,TRACE,TYPE=VTAM,OPT=(API,PIU,MSG),DSPSIZE=5,SIZE=999
� Using these options, the VIT will be created in a VTAM dataspace. When the problem has been recreated, the dump parameters should also include the VTAM dataspace:
– DSPNAME=('NET'.ISTITDS1)
� The MSC LINK trace should also be turned on for both coupled systems (link# or ALL is acceptable for the link specification)
– TRACE SET ON LINK link# LEVEL 4 MODULE ALL
� IMS TYPE6701 records containing TM control blocks for each message at key points
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
66
�Situation Specific Diagnostics:CTL/DLI Region “Out of Storage”
�IMS control region “out of storage” conditions (ABEND80A ABEND878) can result in ABEND40D (Recursive errors in MVS RTM)
• Problem: Unable to diagnose problem because the dump dataset doesn’t contain private storage for the failing region.
• Solution: Set the following four SLIP traps to capture the initial “out of storage” prior to RTM gaining control
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
67
�Situation Specific Diagnostics:CTL/DLI Region “Out of Storage”, Continued
SLIP SET,C=80A,J=SLIP SET,C=80A,J=SLIP SET,C=80A,J=SLIP SET,C=80A,J=jobnamejobnamejobnamejobname, , , ,
SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),
ID=ID=ID=ID=xxxx,ENDxxxx,ENDxxxx,ENDxxxx,END
SLIP SET,C=878,J=SLIP SET,C=878,J=SLIP SET,C=878,J=SLIP SET,C=878,J=jobnamejobnamejobnamejobname, , , ,
SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),SDATA=(CSA,PSA,SUM,RGN,SQA,TRT,ALLNUC,LPA),
ID=ID=ID=ID=xxxx,ENDxxxx,ENDxxxx,ENDxxxx,END
WhereWhereWhereWhere
jobnamejobnamejobnamejobname = Alternatively, = Alternatively, = Alternatively, = Alternatively, JobnameJobnameJobnameJobname of the IMSof the IMSof the IMSof the IMS
CTL region,CTL region,CTL region,CTL region,
= = = = JobnameJobnameJobnameJobname of the IMS DLI/SAS regionof the IMS DLI/SAS regionof the IMS DLI/SAS regionof the IMS DLI/SAS region
Enter the following two SLIP traps for both the IMS
Control and IMS DLI/SAS regions (4 slips total)
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
68
Making Hard for those IMS Problems to Hide -Summary
� MVS Setup Recommendations
� IMS Setup Recommendations
� Preservation of Standard Documentation
� Manual Dump Intervention
� Situation Specific Diagnostics
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
Acknowledgements and Disclaimers
Availability. References in this presentation to IBM products, programs, or services do not imply that they will be available in all
countries in which IBM operates.
The workshops, sessions and materials have been prepared by IBM or the session speakers and reflect their own views. They are provided for informational purposes only, and are neither intended to, nor shall have the effect of being, legal or other guidance or
advice to any participant. While efforts were made to verify the completeness and accuracy of the information contained in thispresentation, it is provided AS-IS without warranty of any kind, express or implied. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this presentation or any other materials. Nothing contained in this presentation is intended to,
nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. Nothing contained in these
materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.
© Copyright IBM Corporation 2012. All rights reserved.
– U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
IBM, the IBM logo, ibm.com, IMS, MVS, z/OS, DB2, CICS, and WebSphere are trademarks or registered trademarks of International
Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on
their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law
trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law
trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at
www.ibm.com/legal/copytrade.shtml. Other company, product, or service names may be trademarks or service marks of others.
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
70
Communities
• On-line communities, User Groups, Technical Forums, Blogs,
Social networks, and more
– Find the community that interests you …
• Information Management bit.ly/InfoMgmtCommunity
• Business Analytics bit.ly/AnalyticsCommunity
• Enterprise Content Management
bit.ly/ECMCommunity
• IBM Champions
– Recognizing individuals who have made the most outstanding
contributions to Information Management, Business
Analytics, and Enterprise Content Management communities
• ibm.com/champion
IBM Corporation, 2012c
Making it Hard for those IMS Problems to Hide
71
Thank You! Your Feedback is Important!
• Access SmartSite to complete your session
surveys
– Any web or mobile browser at iodsmartsite.com
– SmartSite applications for iPhone, iPad, Droid and
Blackberry
– Any SmartSite kiosk onsite