Fault Analyser

1 Fault Analyzer Mentor Workshop © 2003 IBM Corporation November 4, 2003

zSeries AD Tools

IBM Fault Analyzer

Version 4

Program Number 5655-L25

Mentor Workshop


zSeries AD Tools

Agenda

Introduction

Fault Analyzer Files, Setup, and Options

Real-Time Fault Analysis Report

Using the Online Interface

Fault Re-analysis

CICS, DB2, & IMS Support

Preparing programs for Fault Analysis

Using Options

Hints & Tips


zSeries AD Tools

What IS Fault Analyzer ?

Fault Analyzer is a tool that helps you determine the cause of an application abendso you can more quickly identify and resolve

the problem

Delivers information about an application when it has abended, to help you assess:What happened, and why?What program?What line of source code?What source variables were involved?


zSeries AD Tools

Languages and Environments Supported

COBOL

PL/I

Assembler

C/C++

z/OS

OS/390

LE (Language Environment)

CICS

IMS

DB2

MQSeries

UNIX System Services


zSeries AD Tools

What Functions does Fault Analyzer have ?

Real-Time AnalysisAutomatic generation of an analysis report when

an application abendsCaptures and saves detailed information about

the application and its environment

Online InterfaceView an analysis reportPerform a ReanalysisManage saved abend entries

Interactive ReanalysisPoint-and-shoot navigation of an abendApply program source data

Batch ReanalysisRun a batch job to generate a new analysis reportApply program source data


zSeries AD Tools

Real-Time Analysis … how it works

1. An application abends. The system intercepts the abend and calls one of the Fault Analyzer system exits. The exit invokes FA.

2. FA reads options that control whether it will analyze the abend, how to process, and which Fault History file to use Installation options are

specified for the system Options can be overridden

for a job step or online region

Side Files

z/OS

FaultAnalyzer

ApplicationFault History File

FaultEntry

Compiler Listings

Options

FA Invocation Exit

Abend

AnalysisReport

(batch application)

Real-Time Analysis

Side Files


zSeries AD Tools

z/OS

FaultAnalyzer


FaultEntry

Compiler Listings

Options

FA Invocation Exit

Abend

AnalysisReport

(batch application)

Real-Time Analysis


3. Fault Analyzer examines application programs and their environment

4. FA gets source information for application programs by reading compiler listings and side files Compiler listings and side

files are stored in PDS or PDSE data sets

Multiple data sets can be scanned

If a listing / side file is not found, the report will not show source information for that program


zSeries AD Tools

z/OS

FaultAnalyzer


FaultEntry

Compiler Listings

Options

FA Invocation Exit

Abend

AnalysisReport

(batch application)

Real-Time Analysis


5. A new Fault Entry is written to a Fault History File. Options control which History File is used. The entry contains: Info about the application The Analysis Report A “mini-dump” of the

application (this enables reanalysis)

6. The Analysis Report is written to SYSOUT (batch jobs only)

7. SYSMDUMP output is suppressed You can specify FA options if

you want to get SYSMDUMP output


zSeries AD Tools

View a Real-Time Analysis Report

For all abends, FA stores the Real-Time Analysis report in a Fault Entry

Use the online interface to view the report from the Fault Entry

For batch jobs, FA also sends the report to SYSOUTDefault is SYSOUT=*

DD name is IDIREPRT

FaultAnalyzer Online

Interface

ISPF

Fault History Files

FaultEntry

SYSOUT

AnalysisReport Printed

output

For batch abends only

For any abend


zSeries AD Tools

The Real-Time Analysis Report

Synopsis and the suspected cause of a failure abending program executing line of source

code active variables

Explanation of error messages / codes

File and Database detail

Detailed information about programs:

Values of all program variables / storage compile, link, load, and entry point information

System information: z/OS, CICS, DB2, IMS


zSeries AD Tools

Interactive Reanalysis

Online screens to research abend information

Apply source information, even if the source info was not available during Real-Time analysis

Options can be specified to control processing and locate source listings

If a compiler listing does not not already exist, it can be re-created after the abend

You do NOT have to recreate the abend

FaultAnalyzer

Online Interface

Fault History File

FaultEntry

Compiler Listings

Side Files

OptionsISPF

CompilerProgram Source

Optional

The Fault Entry was saved by FA when the application abended

The Fault Entry was saved by FA when the application abended

For any abend


zSeries AD Tools

Interactive Reanalysis

Synopsis Any program in the call chain Program variables and their values File and database detail CICS, DB2, and IMS info Browse program source listing Browse storage

Point and Shoot navigation

Info shown in the Real-time report

Quickly navigate to:


zSeries AD Tools

Batch Reanalysis

Batch utility to generate a new analysis report

Apply source information, even if the source info was not available during Real-Time analysis

Fault Analyzer generates JCL

JCL can be edited, submitted, and/or saved

Fault History File

FaultEntry

Compiler Listings

Side Files

ISPF

CompilerProgram Source

Optional

Use the FA online interface to generate and submit JCL for the batch reanalysis job

Use the FA online interface to generate and submit JCL for the batch reanalysis job

Fault Analyzer Batch

Reanalysis Job

AnalysisReport

(SYSOUT)


zSeries AD Tools

How Fault Analyzer gets Source Information

Fault Analyzer looks for program source info during: Real-Time Analysis Re-analysis

To get source information about programs, Fault Analyzer scans PDS or PDSE data sets containing: Compiler Listings (the report output of compilers) IDILANGX Side Files

Produced by processing a compiler listing through utility program IDILANGX. It is smaller than a compiler listing

COBOL SYSDEBUG Side Files Produced by COBOL compilers when the TEST(…,

…,SEPARATE) compiler option is specified

If a program listing is not found, the Analysis Report is produced without source information for that program


zSeries AD Tools

How Fault Analyzer gets Source Information

Fault Analyzer looks for a member name that matches1. the program’s entry point name. If not found, then … 2. the CSECT name

If multiple versions of compiler listings or side files are found, FA compares: the timestamp in each listing / side file - - - against - - - the timestamp in the load module

the matching version of the listing / side file is used If no version has a matching timestamp, then:

• The compiler listing / side file with a timestamp nearest the load module timestamp is used

• A warning message is written to the Analysis Report

Multiple data sets can be searched Test, QA, Production, and other compiler listings and

side files can all be searched together


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

Fault Analyzer Setup

To get the most from Fault Analyzer, there should be:

One or more Fault History Files (required)

Processes for generating and storing Compiler Listings or Side Files

Fault Analyzer Options customized for your system


zSeries AD Tools

Fault History Files

Fault History Files are used by Fault Analyzer to store abend information

File type is PDSE (recommended) or PDS Initialized with utility program IDIUTIL

One entry is stored for an abend

Each entry includes: Information about the

abenda copy of the Real-Time

Analysis reporta "minidump" of the

application

Fault History File

Fault Entries

FaultAnalyzer

Application

Options

FA Invokation Exit

Abend

Analysis ReportCompiler Listings / Side Files


zSeries AD Tools

Fault History Files

A History File can be set up to wrap around (recommended)

"Duplicate" entries can be suppressed to save spacean abend is a duplicate if it has the same:

program name, compile timestamp, abend code, and offset

controlled by the age of an entry for example… no duplicates stored for 1 hour after

the first abend

How long does a fault entry stay in the history file ? It depends on:

the size of the file (how many entries it can hold) application size how frequently applications abend


zSeries AD Tools

Fault History Files

One or Multiple History Files can be usedconsider creating separate files for:

production vs. test• So test abends won’t cause production info to roll off

by department or application group• For security, if program or file information is a concern

To automatically select a Fault History File, code a Fault Analyzer “Analysis Control user exit”Choose a Fault History File based on Jobname, Tran ID,

User ID, etc.Typically done by the Installer / Systems Programmer


zSeries AD Tools

Fault Analyzer Options

FA uses default options customized for your system

You can also specify options to override the defaults, to tell Fault Analyzer:where program source listings are stored the name of the Fault History File to use to include or exclude certain applications for processingTo collect a SYSMDUMP or SYSUDUMPThe level of detail to show in Analysis ReportsAnd more

FA accepts override options during:Real-time Analysis

Specify options in JCLRe-analysis

Specify your options interactively


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

Real Time Analysis Report

Can be viewed one of two ways: View any Real-Time report with the Online Interface

displayed from the Fault History File For a batch job, view the generated report

FA dynamically allocates DD “IDIREPRT “

Contents: Synopsis Event Summary Event Details

Point of Failure Associated Messages Open Files Associated Storage Areas

Abend Job Information Fault Analyzer Options in Effect


zSeries AD Tools

Real-Time Analysis Report ******************************************************************************** * IBM Fault Analyzer for z/OS and OS/390 V3R1M0 (UQ73759 2003/02/17) * * * * (C) Copyright IBM Corp. 2000, 2002. All rights reserved. * ******************************************************************************** JOBNAME: DBSTOUTY SYSTEM ABEND: 0CB DEMOMVS 2003/06/08 13:06:13 Page 001 <H1> I B M F A U L T A N A L Y Z E R S Y N O P S I S A system abend 0CB occurred in module SAM1 program SAM1 at offset X'67C'. A program interruption code 000B (Decimal-Divide Exception) is associated with this abend and indicates that: The divisor was zero in a signed decimal division. The cause of the failure was program SAM1 in module SAM1. The COBOL source code that immediately preceded the failure was: Source List Line # Stmt # ------ ------ 000108 n/a COMPUTE UNIT-PRICE ROUNDED = 000109 n/a ORD-SALE-AMOUNT / ORD-NUM-ITEMS The COBOL source code for data fields involved in the failure: Source List Line # Stmt # ------ ------ 000040 n/a 05 ORD-NUM-ITEMS PIC 9(5). 000042 n/a 05 ORD-SALE-AMOUNT PIC 9(5)V99. 000057 n/a 05 UNIT-PRICE PIC S9(5)V99 COMP-3. Data field values at time of abend: ORD-NUM-ITEMS = 00000 *** Cause of error *** ORD-SALE-AMOUNT = 0002000 UNIT-PRICE = X'0000100C'


zSeries AD Tools

Real-Time Analysis Report

<H1> I B M F A U L T A N A L Y Z E R E V E N T S U M M A R Y The following events are presented in chronological order. Event Fail Module Program EP Source List # Type Point Name Name Name Line # Stmt # Offset Module Origin -- ------------ ----- -------- -------- -------- ------ ------ -------- -------------------------------------------- 1 Abend S0CB ***** SAM1 SAM1 SAM1 000108 n/a 67C DNET074.ADLAB.LOAD <H1> I B M F A U L T A N A L Y Z E R E V E N T D E T A I L S ================================================================================ <H2> EVENT 1 OF 1: ABEND S0CB ******************************************************************************** *********************** P O I N T O F F A I L U R E ********************** ******************************************************************************** Abend Code. . . . . . . . . : S0CB Program Interruption Code . : 000B (Decimal-Divide Exception) The divisor was zero in a signed decimal division. Source List Line # Stmt # ------ ------ COBOL Source Code . . . . . : 000108 n/a COMPUTE UNIT-PRICE ROUNDED = 000109 n/a ORD-SALE-AMOUNT / ORD-NUM-ITEMS Source List Line # Stmt # ------ ------ Data Field Declarations . . : 000040 n/a 05 ORD-NUM-ITEMS PIC 9(5). 000042 n/a 05 ORD-SALE-AMOUNT PIC 9(5)V99. 000057 n/a 05 UNIT-PRICE PIC S9(5)V99 COMP-3.


zSeries AD Tools

Real-Time Analysis Report Data Field Values . . . . . : ORD-NUM-ITEMS = 00000 *** Cause of error *** ORD-SALE-AMOUNT = 0002000 UNIT-PRICE = X'0000100C' The listing or side file used for the above was found in DNET074.ADLAB.LISTING(SAM1). Load Module Name. . . . . . : DNET074.ADLAB.LOAD(SAM1) At Address. . . . . . . . : 19300418 Load Module Length. . . . : X'1BE8' Link-Edit Date and Time . : 2003/06/08 13:00:03 Program and Entry Point Name: SAM1 At Address. . . . . . . . : 19300418 (Module SAM1 offset X'0') Program Length. . . . . . : X'FA0' Program Language. . . . . : COBOL (Compiled using IBM Enterprise COBOL for z/OS and OS/390 V3 R2 M0 on 2003/06/08 at 13:00:03) Service Level . . . . . . : n/a Machine Instruction . . . . : FD72D120D128 DP 288(8,R13),296(3,R13) At Address. . . . . . . . : 19300A94 (Program SAM1 offset X'67C') AMODE . . . . . . . . . . : 31 Failing Operand . . . . . : Second operand First Operand Address . . : 0000D150 (204464 bytes of storage addressable) First Operand Length. . . : 8 First Operand Storage . . : 00000000 0020000F *........* Second Operand Address. . : 0000D158 (204456 bytes of storage addressable) Second Operand Length . . : 3 Second Operand Storage. . : 00000F *...* Program Status Word (PSW) . : 078D1000 99300A9A General Purpose Registers: R0: 0000D168 (204440 bytes of storage addressable) R1: 19393DA0 (Module SAM1 program SAM1 FILE SECTION BLF=0000 + X'0', symbol ORDER-RECORD, source line # 29) R2: 193009F6 (Module SAM1 program SAM1 + X'5DE', source line # 98) R3: 19384038 (5423048 bytes of storage addressable)


zSeries AD Tools

Real-Time Analysis Report R4: 193805E0 (5437984 bytes of storage addressable) R5: 99300A40 (Module SAM1 program SAM1 + X'628', source line # 104) R6: 193AFFB0 (Module SAM1 program SAM1 FILE SECTION BLF=0001 + X'0', symbol REPORT-LINE, source line # 48) R7: 19393DA0 (Module SAM1 program SAM1 FILE SECTION BLF=0000 + X'0', symbol ORDER-RECORD, source line # 29) R8: 193840B8 (Module SAM1 program SAM1 WORKING-STORAGE SECTION BLW=0000 + X'0', symbol PROGRAM-WORK-FIELDS, source line # 52) R9: 19380448 (5438392 bytes of storage addressable) R10: 19300540 (Module SAM1 program SAM1 + X'128') R11: 19300850 (Module SAM1 program SAM1 + X'438') R12: 19300514 (Module SAM1 program SAM1 + X'FC') R13: 0000D030 (204752 bytes of storage addressable) R14: 800328F0 (Module IGZEQBL + X'2170') R15: 00B1A0D0 (Module IGG019AA + X'0') -------------------------------------------------------------------------------- <H3> Associated Messages CEE3211S The system detected a decimal-divide exception (System Completion Code=0CB). CEE3211S CEE3211S The system detected a decimal-divide exception (System Completion Code=0CB). Explanation: Your program attempted to perform a mathematical operation where, in decimal division, the divisor is zero or the quotient exceeds the specified data-field size. See a Principles of Operation manual for a full list of decimal-divide exceptions. Programmer Response: You can use a condition handling routine to correct the data values and resume the application. System Action: The thread is terminated. Symbolic Feedback Code: CEE34B --------------------------------------------------------------------------------


zSeries AD Tools


<H3> Open Files File Name . . . . . . . . . : ORDERS Data Set Name . . . . . . : DNET074.ADLAB.DATA File Attributes . . . . . : ORGANIZATION=SEQUENTIAL, ACCESS MODE=SEQUENTIAL, RECFM=FIXED Last I/O Function . . . . : READ Open Status . . . . . . . : INPUT File Status Code. . . . . : 0 Previous Record . . . . . : Record data length 80 Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393D50 F0F0F0F0 F34040F9 4040F0F1 F0F540D1 *00003 9 0105 J* 19393D60 +10 D6C8D5E2 D6D54040 40F0F0F0 F2F040F0 *OHNSON 00020 0* 19393D70 +20 F0F0F2F0 F0F04040 40404040 40404040 *002000 * 19393D80 +30 40404040 40404040 40404040 40404040 * * Line 19393D90 same as above Current Record. . . . . . : Record data length 80 Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393DA0 F0F0F0F0 F44040F9 4040F0F1 F0F340C4 *00004 9 0103 D* 19393DB0 +10 C1E5C9E2 40404040 40F0F0F0 F0F040F0 *AVIS 00000 0* 19393DC0 +20 F0F0F2F0 F0F0404C 7E7E7DF0 F0F0F0F0 *002000 <=='00000* 19393DD0 +30 7D40C9D5 407BC9E3 C5D4E240 C3C1E4E2 *' IN #ITEMS CAUS* 19393DE0 +40 C5E240E2 F0C3C240 C9D540E2 C1D4F140 *ES S0CB IN SAM1 * Next Record . . . . . . . : Record data length 80 Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393DF0 F0F0F0F0 F54040F9 40407C50 7C5040E2 *00005 9 @&@& S* 19393E00 +10 E3D6E4E3 40404040 40F0F0F0 F0F140F0 *TOUT 00001 0* 19393E10 +20 F0F0F1F0 F0F0404C 7E7E7D7C 507C507D *001000 <=='@&@&'* 19393E20 +30 40C9D540 C9E3C5D4 C3D6C4C5 40C3C1E4 * IN ITEMCODE CAU* 19393E30 +40 E2C5E240 E2F0C3F7 40C9D540 E2C1D4F2 *SES S0C7 IN SAM2*


zSeries AD Tools

Real-Time Analysis Report File Name . . . . . . . . . : SUMMARY Data Set Name . . . . . . : DNET074.DBSTOUTY.JOB09683.D0000103.? File Attributes . . . . . : ORGANIZATION=SEQUENTIAL, ACCESS MODE=SEQUENTIAL, RECFM=FIXED Last I/O Function . . . . : WRITE Open Status . . . . . . . : OUTPUT File Status Code. . . . . : 0 No record buffers available. -------------------------------------------------------------------------------- <H3> Associated Storage Areas FILE SECTION at Address 19393DA0 (File ORDERS): Source (Starting at Line # 000029) EBCDIC Value Hex Value ------------------------------------------------------------------------ ------------------ ---------------------------- 01 ORDER-RECORD. 05 ORD-NUMBER PIC 9(5). *00004 * F0F0F0F0 F4 05 FILLER REDEFINES ORD-NUMBER. 10 ORDER-REC-COMMENT PIC X. *0 * F0 10 FILLER PIC X(4). *0004 * F0F0F0F4 05 FILLER PIC XX. * * 4040 05 ORD-LOCATION-CODE PIC 9. *9 * F9 05 FILLER PIC XX. * * 4040 05 ORD-ITEM-CODE PIC 9(4). *0103 * F0F1F0F3 05 FILLER PIC X. * * 40 05 ORD-CUST-ID PIC X(10). *DAVIS * C4C1E5C9 E2404040 4040 05 ORD-NUM-ITEMS PIC 9(5). *00000 * F0F0F0F0 F0 05 FILLER PIC X. * * 40 05 ORD-SALE-AMOUNT PIC 9(5)V99. *0002000 * F0F0F0F2 F0F0F0 FILE SECTION at Address 193AFFB0 (File SUMMARY): Source (Starting at Line # 000048) EBCDIC Value Hex Value ------------------------------------------------------------------------ ------------------ ---------------------------- 01 REPORT-LINE PIC X(80). *JOHNSON DB99* D1D6C8D5 E2D6D540 40404040 C *9 - BLUE * F9406040 C2D3E4C5 40404040 4 *20 $ 20.00 * F2F04040 405B4040 4040F2F0 4 * $ 1.00 * 405B4040 404040F1 4BF0F040 4 * * 40404040 40404040 40404040 4


zSeries AD Tools

Real-Time Analysis Report WORKING-STORAGE SECTION at Address 193840B8: Source (Starting at Line # 000052) EBCDIC Value Hex Value ------------------------------------------------------------------------ ------------------ ---------------------------- 01 PROGRAM-WORK-FIELDS. 05 WS-START PIC X(29) VALUE '<WORKING-STORAGE-STARTS-HERE>'. *<WORKING-STORAGE* 4CE6D6D9 D2C9D5C7 60E2E3D6 D *-STARTS-HERE> * 60E2E3C1 D9E3E260 C8C5D9C5 6 05 TOTAL-SALES-AMOUNT PIC S9(7)V99 COMP-3. *..... * 00002375 0C 05 NUMBER-OF-ORDERS PIC S9(7) COMP-3. *.... * 0000003C 05 UNIT-PRICE PIC S9(5)V99 COMP-3. *.... * 0000100C 05 TOT-DISC-AMOUNT PIC S9(5)V99 COMP-3. *.... * 00000000 05 RETURNED-DESCRIP PIC X(15). *DB999 - BLUE * C4C2F9F9 F9406040 C2D3E4C5 4 01 REPORT-TEXT. 05 DETAIL-HDR-1. 10 FILLER PIC X(36) VALUE 'CUST ID ITEM DESCRIP QUANT '. *CUST ID ITEM* C3E4E2E3 40C9C440 40404040 C * DESCRIP QUA* 40C4C5E2 C3D9C9D7 40404040 4 *NT * D5E34040 10 FILLER PIC X(23) VALUE 'SALE AMOUNT UNIT PRICE'. *SALE AMOUNT UNI* E2C1D3C5 40C1D4D6 E4D5E340 4 *T PRICE * E340D7D9 C9C3C5 05 DETAIL-HDR-2. 10 FILLER PIC X(36) VALUE '---------- --------------- ----- '. *---------- ----* 60606060 60606060 60604040 6 *----------- ---* 60606060 60606060 60606040 4 *-- * 60604040 10 FILLER PIC X(23) VALUE '----------- ----------'. *----------- ---* 60606060 60606060 60606040 4 *------- * 60606060 606060 05 DETAIL-LINE. 10 RPT-CUST-ID PIC X(10). *JOHNSON * D1D6C8D5 E2D6D540 4040 10 FILLER PIC XX VALUE SPACES. * * 4040 10 RPT-ITEM-DESCRIP PIC X(15). *DB999 - BLUE * C4C2F9F9 F9406040 C2D3E4C5 4 10 FILLER PIC XX VALUE SPACES. * * 4040 10 RPT-QUANTITY PIC ZZZZ9. * 20 * 404040F2 F0 10 FILLER PIC XXX VALUE SPACES. * * 404040 10 RPT-SALE-AMOUNT PIC $ZZ,ZZ9.99. *$ 20.00 * 5B404040 40F2F04B F0F0 10 FILLER PIC XX VALUE SPACES. * * 4040 10 RPT-UNIT-PRICE PIC $ZZ,ZZ9.99. *$ 1.00 * 5B404040 4040F14B F0F0


zSeries AD Tools


05 SUMMARY-LINE-1. 10 FILLER PIC X(21) VALUE 'ORDERS PROCESSED = '. *ORDERS PROCESSED* D6D9C4C5 D9E240D7 D9D6C3C5 E * = * 4040407E 40 10 RPT-NUM-ORDERS PIC ZZZZ9. *..... * 00000000 00 05 SUMMARY-LINE-2. 10 FILLER PIC X(14) VALUE 'TOTAL SALES = '. *TOTAL SALES = * E3D6E3C1 D340E2C1 D3C5E240 7 10 RPT-TOTAL-SALES PIC $Z,ZZZ,ZZ9.99. *............. * 00000000 00000000 00000000 0 01 PROGRAM-SWITCHES. 05 END-OF-ORDERS PIC X. *. * 10 NOTE: BLL=0000 has not been assigned an address. <H1> I B M F A U L T A N A L Y Z E R A B E N D J O B I N F O IBM Fault Analyzer Abend Job Information: Abend Date. . . . . . . . : 2003/06/08 Abend Time. . . . . . . . : 13:06:13 System Name . . . . . . . : DEMOMVS Job Type. . . . . . . . . : Batch Job Name. . . . . . . . . : DBSTOUTY Job Step Name . . . . . . : RUNSAM1 Job Class . . . . . . . . : A EXEC Program Name . . . . : SAM1 User ID . . . . . . . . . : DNET074 Data Sets: DDname Data Set or Path Name -------- -------------------------------------------- STEPLIB DNET074.ADLAB.LOAD


zSeries AD Tools

Real-Time Analysis ReportEvent-Related Application Programs: The following list of event-related application programs is sorted by module link-edit date/time and program compilation date/time in reverse chronological order. Module Link-Edit Program Compilation Name Date Time Name Date Time -------- ---------- -------- -------- ---------- -------- SAM1 2003/06/08 13:00:03 SAM1 2003/06/08 13:00:03 Execution Environment: Operating System. . . . . : z/OS V1R4M0 Data Facility Product . . : DFSMS z/OS V1R3M0 Job Entry Subsystem . . . : JES2 z/OS 1.4 Language Environment. . . : V1 R4.0 CPU Model . . . . . . . . : 2064 <H1> I B M F A U L T A N A L Y Z E R O P T I O N S IBM Fault Analyzer Options in Effect: DETAIL(MEDIUM) LISTSEARCH(BEST) MAXMINIDUMPPAGES(256) Data Sets: The following Fault Analyzer data set or path names were either preallocated, specified via DATASETS options, or provided as defaults. DDname Data Set or Path Name -------- -------------------------------------------- IDIBOOKS FAULTANL.V3R1.SIDIBOOK IDICACHE FAULTANL.V3R1.CACHE IDIDOC FAULTANL.V3R1.SIDIDOC1 IDIHIST FAULTANL.V3R1.HIST IDILCOB DNET074.ADLAB.LISTING


zSeries AD Tools


IDIMAPS FAULTANL.V3R1.SIDIMAPS Fault Analyzer was invoked via the LE CEEEXTAN exit (IDIXCEE) and used a maximum of 0.81 megabytes of allocated storage for analysis.

End of Fault Analyzer report (2003/06/08 13:06).

When this analysis report was generated, Fault Analyzer found source listings for the programs

The report starting on the next page shows the same application abending in the same way, but the listings were not available to Fault Analyzer


zSeries AD Tools

Real-Time Report – Without Source

******************************************************************************** * IBM Fault Analyzer for z/OS and OS/390 V3R1M0 (UQ73759 2003/02/17) * * * * (C) Copyright IBM Corp. 2000, 2002. All rights reserved. * ******************************************************************************** JOBNAME: DBSTOUTN SYSTEM ABEND: 0CB DEMOMVS 2003/06/08 13:05:18 Page 001 <H1> I B M F A U L T A N A L Y Z E R S Y N O P S I S A system abend 0CB occurred in module SAM1 program SAM1 at offset X'67C'. A program interruption code 000B (Decimal-Divide Exception) is associated with this abend and indicates that: The divisor was zero in a signed decimal division. The abend was caused by machine instruction FD72D120D128 (DP). Most recently referenced data items: Data Item . . . . . . . . : BLF=0000+019 At Address. . . . . . . : 19393DB9 Length. . . . . . . . . : X'5' Data Item Storage . . . : F0F0F0F0 F0 *00000* Data Item . . . . . . . . : BLF=0000+01F At Address. . . . . . . : 19393DBF Length. . . . . . . . . : X'7' Data Item Storage . . . : F0F0F0F2 F0F0F0 *0002000* NOTE: Source code information could not be presented because the search for a compiler listing or side-file was unsuccessful for program SAM1.


zSeries AD Tools


<H1> I B M F A U L T A N A L Y Z E R E V E N T S U M M A R Y The following events are presented in chronological order. Event Fail Module Program EP Source List # Type Point Name Name Name Line # Stmt # Offset Module Origin -- ------------ ----- -------- -------- -------- ------ ------ -------- -------------------------------------------- 1 Abend S0CB ***** SAM1 SAM1 SAM1 n/a n/a 67C DNET074.ADLAB.LOAD <H1> I B M F A U L T A N A L Y Z E R E V E N T D E T A I L S ================================================================================ <H2> EVENT 1 OF 1: ABEND S0CB ******************************************************************************** *********************** P O I N T O F F A I L U R E ********************** ******************************************************************************** Abend Code. . . . . . . . . : S0CB Program Interruption Code . : 000B (Decimal-Divide Exception) The divisor was zero in a signed decimal division. Most recently referenced data items: Data Item . . . . . . . . : BLF=0000+019 At Address. . . . . . . : 19393DB9 Length. . . . . . . . . : X'5' Data Item Storage . . . : F0F0F0F0 F0 *00000* Data Item . . . . . . . . : BLF=0000+01F At Address. . . . . . . : 19393DBF Length. . . . . . . . . : X'7' Data Item Storage . . . : F0F0F0F2 F0F0F0 *0002000* NOTE: Source code information could not be presented because the search for a compiler listing or side-file was unsuccessful for program SAM1.


zSeries AD Tools


Load Module Name. . . . . . : DNET074.ADLAB.LOAD(SAM1) At Address. . . . . . . . : 19300418 Load Module Length. . . . : X'1BE8' Link-Edit Date and Time . : 2003/06/08 13:00:03 Program and Entry Point Name: SAM1 At Address. . . . . . . . : 19300418 (Module SAM1 offset X'0') Program Length. . . . . . : X'FA0' Program Language. . . . . : COBOL (Compiled using IBM Enterprise COBOL for z/OS and OS/390 V3 R2 M0 on 2003/06/08 at 13:00:03) Service Level . . . . . . : n/a Machine Instruction . . . . : FD72D120D128 DP 288(8,R13),296(3,R13) At Address. . . . . . . . : 19300A94 (Program SAM1 offset X'67C') AMODE . . . . . . . . . . : 31 Failing Operand . . . . . : Second operand First Operand Address . . : 0000D150 (204464 bytes of storage addressable) First Operand Length. . . : 8 First Operand Storage . . : 00000000 0020000F *........* Second Operand Address. . : 0000D158 (204456 bytes of storage addressable) Second Operand Length . . : 3 Second Operand Storage. . : 00000F *...* Program Status Word (PSW) . : 078D1000 99300A9A General Purpose Registers: R0: 0000D168 (204440 bytes of storage addressable) R1: 19393DA0 (Module SAM1 program SAM1 FILE SECTION BLF=0000 + X'0') R2: 193009F6 (Module SAM1 program SAM1 + X'5DE') R3: 19384038 (5423048 bytes of storage addressable) R4: 193805E0 (5437984 bytes of storage addressable) R5: 99300A40 (Module SAM1 program SAM1 + X'628') R6: 193AFFB0 (Module SAM1 program SAM1 FILE SECTION BLF=0001 + X'0') R7: 19393DA0 (Module SAM1 program SAM1 FILE SECTION BLF=0000 + X'0') R8: 193840B8 (Module SAM1 program SAM1 WORKING-STORAGE SECTION BLW=0000 + X'0') R9: 19380448 (5438392 bytes of storage addressable)


zSeries AD Tools


R10: 19300540 (Module SAM1 program SAM1 + X'128') R11: 19300850 (Module SAM1 program SAM1 + X'438') R12: 19300514 (Module SAM1 program SAM1 + X'FC') R13: 0000D030 (204752 bytes of storage addressable) R14: 800328F0 (Module IGZEQBL + X'2170') R15: 00B1A0D0 (Module IGG019AA + X'0') -------------------------------------------------------------------------------- <H3> Associated Messages CEE3211S The system detected a decimal-divide exception (System Completion Code=0CB). CEE3211S CEE3211S The system detected a decimal-divide exception (System Completion Code=0CB). Explanation: Your program attempted to perform a mathematical operation where, in decimal division, the divisor is zero or the quotient exceeds the specified data-field size. See a Principles of Operation manual for a full list of decimal-divide exceptions. Programmer Response: You can use a condition handling routine to correct the data values and resume the application. System Action: The thread is terminated. Symbolic Feedback Code: CEE34B -------------------------------------------------------------------------------- <H3> Open Files File Name . . . . . . . . . : ORDERS Data Set Name . . . . . . : DNET074.ADLAB.DATA File Attributes . . . . . : ORGANIZATION=SEQUENTIAL, ACCESS MODE=SEQUENTIAL, RECFM=FIXED Last I/O Function . . . . : READ


zSeries AD Tools


Open Status . . . . . . . : INPUT File Status Code. . . . . : 0 Previous Record . . . . . : Record data length 80 Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393D50 F0F0F0F0 F34040F9 4040F0F1 F0F540D1 *00003 9 0105 J* 19393D60 +10 D6C8D5E2 D6D54040 40F0F0F0 F2F040F0 *OHNSON 00020 0* 19393D70 +20 F0F0F2F0 F0F04040 40404040 40404040 *002000 * 19393D80 +30 40404040 40404040 40404040 40404040 * * Line 19393D90 same as above Current Record. . . . . . : Record data length 80 Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393DA0 F0F0F0F0 F44040F9 4040F0F1 F0F340C4 *00004 9 0103 D* 19393DB0 +10 C1E5C9E2 40404040 40F0F0F0 F0F040F0 *AVIS 00000 0* 19393DC0 +20 F0F0F2F0 F0F0404C 7E7E7DF0 F0F0F0F0 *002000 <=='00000* 19393DD0 +30 7D40C9D5 407BC9E3 C5D4E240 C3C1E4E2 *' IN #ITEMS CAUS* 19393DE0 +40 C5E240E2 F0C3C240 C9D540E2 C1D4F140 *ES S0CB IN SAM1 * Next Record . . . . . . . : Record data length 80 Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393DF0 F0F0F0F0 F54040F9 40407C50 7C5040E2 *00005 9 @&@& S* 19393E00 +10 E3D6E4E3 40404040 40F0F0F0 F0F140F0 *TOUT 00001 0* 19393E10 +20 F0F0F1F0 F0F0404C 7E7E7D7C 507C507D *001000 <=='@&@&'* 19393E20 +30 40C9D540 C9E3C5D4 C3D6C4C5 40C3C1E4 * IN ITEMCODE CAU* 19393E30 +40 E2C5E240 E2F0C3F7 40C9D540 E2C1D4F2 *SES S0C7 IN SAM2* File Name . . . . . . . . . : SUMMARY Data Set Name . . . . . . : DNET074.DBSTOUTN.JOB09682.D0000103.? File Attributes . . . . . : ORGANIZATION=SEQUENTIAL, ACCESS MODE=SEQUENTIAL, RECFM=FIXED Last I/O Function . . . . : WRITE Open Status . . . . . . . : OUTPUT File Status Code. . . . . : 0 No record buffers available.


zSeries AD Tools


<H3> Associated Storage Areas Task Global Table (TGT) at Address 19380448: Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19380448 00000000 00000000 00000000 00000000 *................* Lines 19380458-19380478 same as above 19380488 +40 00000000 00000000 F3E3C7E3 00000000 *........3TGT....* 19380498 +50 06000000 64020260 19380100 000077FC *.......-........* 193804A8 +60 193805A8 00000002 00000129 00000000 *...y............* 193804B8 +70 00000000 0002D1C0 00000000 00000000 *......J{........* 193804C8 +80 1930CBE8 0000015C 00000000 00000000 *...Y...*........* 193804D8 +90 00000000 00000001 E2E8E2D6 E4E34040 *........SYSOUT * 193804E8 +A0 C9C7E9E2 D9E3C3C4 00000000 00000000 *IGZSRTCD........* 193804F8 +B0 00000000 00000000 00000000 00000000 *................* Lines 19380508-19380518 same as above 19380528 +E0 00000000 00000000 19300514 00000000 *................* 19380538 +F0 19380590 193803F0 19300640 00000000 *.......0... ....* 19380548 +100 19300418 19300558 19380588 1930053C *...........h....* 19380558 +110 19380588 193840B8 00000000 00000000 *...h.. .........* 19380568 +120 00000000 00000000 19384038 193840B8 *.......... ... .* 19380578 +130 00000000 19393DA0 193AFFB0 19381500 *................* 19380588 +140 193805E0 193806F8 40000000 00000000 *...\...8 .......* 19380598 +150 00000000 19380810 00000001 *............ * FILE SECTION BLF=0000 at Address 19393DA0 (File ORDERS): Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 19393DA0 F0F0F0F0 F44040F9 4040F0F1 F0F340C4 *00004 9 0103 D* 19393DB0 +10 C1E5C9E2 40404040 40F0F0F0 F0F040F0 *AVIS 00000 0* 19393DC0 +20 F0F0F2F0 F0F0404C 7E7E7DF0 F0F0F0F0 *002000 <=='00000* 19393DD0 +30 7D40C9D5 407BC9E3 C5D4E240 C3C1E4E2 *' IN #ITEMS CAUS* 19393DE0 +40 C5E240E2 F0C3C240 C9D540E2 C1D4F140 *ES S0CB IN SAM1 * 19393DF0 +50 F0F0F0F0 F54040F9 40407C50 7C5040E2 *00005 9 @&@& S* 19393E00 +60 E3D6E4E3 40404040 40F0F0F0 F0F140F0 *TOUT 00001 0* 19393E10 +70 F0F0F1F0 F0F0404C 7E7E7D7C 507C507D *001000 <=='@&@&'* 19393E20 +80 40C9D540 C9E3C5D4 C3D6C4C5 40C3C1E4 * IN ITEMCODE CAU* 19393E30 +90 E2C5E240 E2F0C3F7 40C9D540 E2C1D4F2 *SES S0C7 IN SAM2*


zSeries AD Tools


19393E40 +A0 F0F0F0F0 F64040F9 4040F0F1 F0F440E2 *00006 9 0104 S* 19393E50 +B0 C5D5D540 40404040 40F0F0F2 F5F040F0 *ENN 00250 0* 19393E60 +C0 F2F0F0F0 F0F04040 40404040 40404040 *200000 * 19393E70 +D0 40404040 40404040 40404040 40404040 * * Line 19393E80 same as above 19393E90 +F0 00000000 00000000 00000000 00000000 *................* Lines 19393EA0-19394D90 same as above FILE SECTION BLF=0001 at Address 193AFFB0 (File SUMMARY): Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 193AFFB0 D1D6C8D5 E2D6D540 40404040 C4C2F9F9 *JOHNSON DB99* 193AFFC0 +10 F9406040 C2D3E4C5 40404040 40404040 *9 - BLUE * 193AFFD0 +20 F2F04040 405B4040 4040F2F0 4BF0F040 *20 $ 20.00 * 193AFFE0 +30 405B4040 404040F1 4BF0F040 40404040 * $ 1.00 * 193AFFF0 +40 40404040 40404040 40404040 40404040 * * 193B0000 +50 00000000 00000000 00000000 00000000 *................* Lines 193B0010-193B0AF0 same as above 193B0B00 +B50 00000000 00000000 47F0F028 00C3C5C5 *.........00..CEE* 193B0B10 +B60 000001B8 00000014 47F0F001 98CEAC00 *.........00.q...* 193B0B20 +B70 193B0BBE 00000000 00000000 00000000 *................* 193B0B30 +B80 90ECD00C 4110F038 98EFF04C 07FF0000 *..}...0.q.0<....* 193B0B40 +B90 193B0B08 00000000 193B1180 193B0BB6 *................* 193B0B50 +BA0 193B0B08 193B0DFA 193B14A8 193B0BD2 *...........y...K* 193B0B60 +BB0 00104001 00000008 E2C1D4F2 40404040 *.. .....SAM2 * 193B0B70 +BC0 F2F0F0F3 F0F6F0F8 F1F2F5F9 F1F2F0F3 *2003060812591203* 193B0B80 +BD0 F0F2F0F0 04740000 0000076C A4E87CCC *0200.......%uY@.* 193B0B90 +BE0 20000000 4000000D 05000002 08000000 *.... ...........* 193B0BA0 +BF0 00008000 00000013 0000001B 00008200 *..............b.* 193B0BB0 +C00 40404040 0004E2C1 D4F24040 40400500 * ..SAM2 ..* 193B0BC0 +C10 0001193B 13F80000 0000FFFF FFB2193B *.....8..........*

. . . skip a few lines . . . 193B0F40 +F90 47F0B34C D2039164 A030F230 D1087007 *.0.<K.j...2.J...* 193B0F50 +FA0 960FD10B FC32D108 A0D2F822 802ED109 *o.J...J..K8...J.* 193B0F60 +FB0 F223D108 700A960F D10AFA22 D108802E *2.J...o.J...J...* 193B0F70 +FC0 F3428029 D10896F0 802D58B0 C02847F0 *3...J.o0....{..0* 193B0F80 +FD0 B1FA5820 91644A20 A0265020 91645820 *....j.½...&.j...* 193B0F90 +FE0 A0285920 916447B0 B21AD20B 6000A091 *....j.....K.-..j* 193B0FA0 +FF0 D202600C C00458B0 C02847F0 B24C5820 *K.-.{...{..0.<..*


zSeries AD Tools


WORKING-STORAGE SECTION BLW=0000 at Address 193840B8: Address Offset Hex EBCDIC -------- --------- ----------------------------------- ------------------ 193840B8 4CE6D6D9 D2C9D5C7 60E2E3D6 D9C1C7C5 *<WORKING-STORAGE* 193840C8 +10 60E2E3C1 D9E3E260 C8C5D9C5 6E000023 *-STARTS-HERE>...* 193840D8 +20 750C0000 003C0000 100C0000 0000C4C2 *..............DB* 193840E8 +30 F9F9F940 6040C2D3 E4C54040 40000000 *999 - BLUE ...* 193840F8 +40 C3E4E2E3 40C9C440 40404040 C9E3C5D4 *CUST ID ITEM* 19384108 +50 40C4C5E2 C3D9C9D7 40404040 40D8E4C1 * DESCRIP QUA* 19384118 +60 D5E34040 E2C1D3C5 40C1D4D6 E4D5E340 *NT SALE AMOUNT * 19384128 +70 40E4D5C9 E340D7D9 C9C3C560 60606060 * UNIT PRICE-----* 19384138 +80 60606060 60404060 60606060 60606060 *----- ---------* 19384148 +90 60606060 60604040 60606060 60404060 *------ ----- -* 19384158 +A0 60606060 60606060 60604040 60606060 *---------- ----* 19384168 +B0 60606060 6060D1D6 C8D5E2D6 D5404040 *------JOHNSON * 19384178 +C0 4040C4C2 F9F9F940 6040C2D3 E4C54040 * DB999 - BLUE * 19384188 +D0 40404040 4040F2F0 4040405B 40404040 * 20 $ * 19384198 +E0 F2F04BF0 F040405B 40404040 40F14BF0 *20.00 $ 1.0* 193841A8 +F0 F0D6D9C4 C5D9E240 D7D9D6C3 C5E2E2C5 *0ORDERS PROCESSE* 193841B8 +100 C4404040 7E400000 000000E3 D6E3C1D3 *D = .....TOTAL* 193841C8 +110 40E2C1D3 C5E2407E 40000000 00000000 * SALES = .......* 193841D8 +120 00000000 00000000 D5 *........N * NOTE: BLL=0000 has not been assigned an address. <H1> I B M F A U L T A N A L Y Z E R A B E N D J O B I N F O IBM Fault Analyzer Abend Job Information: Abend Date. . . . . . . . : 2003/06/08 Abend Time. . . . . . . . : 13:05:18 System Name . . . . . . . : DEMOMVS Job Type. . . . . . . . . : Batch Job Name. . . . . . . . . : DBSTOUTN Job Step Name . . . . . . : RUNSAM1 Job Class . . . . . . . . : A


zSeries AD Tools


EXEC Program Name . . . . : SAM1 User ID . . . . . . . . . : DNET074 Data Sets: DDname Data Set or Path Name -------- -------------------------------------------- STEPLIB DNET074.ADLAB.LOAD Event-Related Application Programs: The following list of event-related application programs is sorted by module link-edit date/time and program compilation date/time in reverse chronological order. Module Link-Edit Program Compilation Name Date Time Name Date Time -------- ---------- -------- -------- ---------- -------- SAM1 2003/06/08 13:00:03 SAM1 2003/06/08 13:00:03 Execution Environment: Operating System. . . . . : z/OS V1R4M0 Data Facility Product . . : DFSMS z/OS V1R3M0 Job Entry Subsystem . . . : JES2 z/OS 1.4 Language Environment. . . : V1 R4.0 CPU Model . . . . . . . . : 2064 <H1> I B M F A U L T A N A L Y Z E R O P T I O N S IBM Fault Analyzer Options in Effect: DETAIL(MEDIUM) LISTSEARCH(BEST) MAXMINIDUMPPAGES(256)


zSeries AD Tools


Data Sets: The following Fault Analyzer data set or path names were either preallocated, specified via DATASETS options, or provided as defaults. DDname Data Set or Path Name -------- -------------------------------------------- IDIBOOKS FAULTANL.V3R1.SIDIBOOK IDICACHE FAULTANL.V3R1.CACHE IDIDOC FAULTANL.V3R1.SIDIDOC1 IDIHIST FAULTANL.V3R1.HIST IDIMAPS FAULTANL.V3R1.SIDIMAPS Fault Analyzer was invoked via the LE CEEEXTAN exit (IDIXCEE) and used a maximum of 0.70 megabytes of allocated storage for analysis. End of Fault Analyzer report (2003/06/08 13:05).


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

The Online Interface

Use the ISPF interface (in TSO) to:

View reports

Perform reanalysis

Display and delete fault entries

FaultAnalyzer

Fault History Files

FaultEntries

Compiler Listings

Options

Side Files

Options

ISPF


zSeries AD Tools

The Online Interface

Name of a Fault History File (or View)Overtype the

name to open a different file

Available line commands are shown here… and entered here


zSeries AD Tools

1. Select from a list of History Files you have used beforeSo you don’t have to remember the file names

2. Select from a list of Fault Entries you have recently accessed

Some helpful things to do on the FILE menu

4. Select from a list of “Views”A View is a shortcut a a History File, orA view can be a concatenation of History filesThey can be defined to Fault Analyzer (usually by a Systems Programmer)


zSeries AD Tools

1-2. Add or remove blank lines from the displayAdd for easier reading, remove to conserve screen space

3-6. Add or remove detail and help info

Some helpful things to do on theVIEW menu

8. Customize columns shown in the list of Fault EntriesAdd or remove columns from the listSame as the COLS command


zSeries AD Tools

1. Look up a messageBookmanager lookup of messages received form the system

2. Copy to a fileCopy the entire scrollable area to a sequential file or PDS member

Some helpful things to do on theSERVICES menu


zSeries AD Tools

1. View the Fault Analyzer User’s Guide online

2. Get info about Fault Analyzer version and maintenance level

Some helpful things to do on theHELP menu


zSeries AD Tools

Line Commands

? Query summary info for a Fault Entry

V View (browse) a Real-Time Report

I Interactive Reanalysis

B Batch Reanalysis

D Delete an Entry


zSeries AD Tools

Line Commands

Use the V line command to view a Real-Time report

Navigation is similar to a Browse session

Use Collapse and Expand for easier navigation

Enter


zSeries AD Tools

Line Commands

Use the ? line command to see more information about a fault entry

Enter


zSeries AD Tools

Fault Analyzer Commands

Commands

Use help (PF1) to get descriptions

COLS Brings up the Column Configuration panel

COPY dsname Copies the current display to a file

FIND argument Finds argument in the list of Fault Entries

LOOKUP message-id Looks up a message

MATCH column argument Displays only entries that match argument

REFRESH Refreshes the display to show new entries

VIEWS Displays a list of Views that can be selected


zSeries AD Tools

Working with the list of Entries

Two ways to find the entry you are looking for:

Use the FIND command to locate an entry

Use the MATCH command to limit the list of displayed entries


zSeries AD Tools

Locate a fault entry with the Find command

Use the Find command to locate any character string

Enter


zSeries AD Tools

Locate a fault entry with the Match command

Use the Match command to display only entries that match your criteria

The * wildcard can be used anywhere in the search argument

Enter Now only jobs that start

with D are displayed

Now only jobs that start with D are displayed

A Match command is entered …


zSeries AD Tools

Multiple Match commands

Enter the Match command more than once to specify multiple criteria

Entries must meet ALL of your Match conditions

… and then another Match command is entered.

Now only entries are shown that match both: Job/Tran = D* and User_ID = DNET074

Now only entries are shown that match both: Job/Tran = D* and User_ID = DNET074

Enter


zSeries AD Tools

Match All command

The MATCH ALL command displays all entries again

Enter

All entries are shown

All entries are shown


zSeries AD Tools

The COLS command

Columns can be customized

You can also get here from the View menu

Settings are stored in your profile

Enter


zSeries AD Tools

Column Customization

Specify which columns you want to see. PF3 to return.

Type zero to remove a column from your display

Type zero to remove a column from your display

Enter a non-zero number to make a column appear

Enter a non-zero number to make a column appear

PF8


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

Why do Re-analysis ?

The abending program's source listing was not available to Fault Analyzer when the abend occurred In the Real-Time Report, we got:

Apply program source informationSo with Reanalysis, we get:

Also, interactive Reanalysis lets you navigate online and view program source

A system abend 0C7 occurred in module IDISCBL1 CSECT IDISCBL1 at offset X'3D0'. The abend was caused by machine instruction FD73D0B8D0A8(DP)

A system abend 0C7 occurred in module IDISCBL1 CSECT IDISCBL1 at offset X'3D0'. The abend was caused by machine instruction FD73D0B8D0A8(DP)

The cause of the failure was program IDISCBL1. The COBOL source code that immediately preceded the failure was: Source Line # ------ 000051 DIVIDE NUMBERX BY ERROR_COUNT GIVING BAD_RESULT

The cause of the failure was program IDISCBL1. The COBOL source code that immediately preceded the failure was: Source Line # ------ 000051 DIVIDE NUMBERX BY ERROR_COUNT GIVING BAD_RESULT


zSeries AD Tools

Interactive Reanalysis Quick Tour

Walkthrough: Start an Interactive Reanalysis session andstep through basic navigation

Set up Fault Analyzer to automatically prompt for options when Interactive Reanalysis is started

‘I’ command to start Re-analysis

Specify options to tell Fault Analyzer where to find program Source Listings

Step through the basic navigation features


zSeries AD Tools

Interactive Reanalysis Tour (1 of 25)

The I line command starts an Interactive Reanalysis session…

Enter


zSeries AD Tools


interactive reanalysis options

Any FA option can be specified. Use the “datasets” option to tell FA where to find compiler listings


“Y” here makes this screen appear automatically whenever you start an Interactive Reanalysis

You can get to this screen the first time from the “Options” pull-down menu, then select “Interactive Reanalysis Options”

“Y” here makes this screen appear automatically whenever you start an Interactive Reanalysis

You can get to this screen the first time from the “Options” pull-down menu, then select “Interactive Reanalysis Options”

PF3


zSeries AD Tools


main navigation screen… go to the program listing

“Point and shoot” fields are highlighted.

Use tab and Enter to navigate

“Point and shoot” fields are highlighted.

Use tab and Enter to navigate

Enter


zSeries AD Tools


program listing … return to the main navigation screen

Browse program source information from the compiler listing here

Use standard browse commands and PF Keys

Browse program source information from the compiler listing here

Use standard browse commands and PF Keys

PF3


zSeries AD Tools


main navigation screen … go to synopsis information

1 to select Synopsis1 to select Synopsis

Enter


zSeries AD Tools


synopsis shows high-level information about the abend

Abend explanationAbend explanation

ProgramProgram

Executing statementExecuting statement

Variables used by the executing statementVariables used by the executing statement

PF3


zSeries AD Tools


main navigation screen … go to Events information

select Eventsselect Events

Enter


zSeries AD Tools


events summary … go to program (event) detail

The Event Summary shows the call chain and the abending program.

Any program or Event can be examined for more detail.

The Event Summary shows the call chain and the abending program.

Any program or Event can be examined for more detail.

Enter


zSeries AD Tools


program (event) detail … go to abend code description

The abend code is a Point and Shoot field

The abend code is a Point and Shoot field

Enter


zSeries AD Tools


a description of the abendPF3


zSeries AD Tools


program (event) detail … page forward PF8


zSeries AD Tools


program (event) detail … point-and-shoot to a Address

Tab to go to an address

Many of the addresses are Point and Shoot fields

Tab to go to an address

Many of the addresses are Point and Shoot fields

Enter


zSeries AD Tools


storage display and browse

Type an address and Enter to go to any available storage…

Type an address and Enter to go to any available storage…

… or tab to any valid address value and Enter to go there

… or tab to any valid address value and Enter to go there

PF3


zSeries AD Tools


back to program (event) detail… page forward

Registers are Point and Shoot fields

Registers are Point and Shoot fields

PF8


zSeries AD Tools


more program (event) detail… page forward

Descriptions of the areas pointed to by each register

Descriptions of the areas pointed to by each register

The SQLCA is shown (DB2 programs only)

The SQLCA is shown (DB2 programs only)

PF8


zSeries AD Tools


end of program detail … go to Associated Storage Areas

Tab to select Storage Areas associated with the program

Tab to select Storage Areas associated with the program

Enter


zSeries AD Tools


Associated Storage Areas … go over to the right

Variables defined to the program are shown here in Associated Storage Areas

Use normal Browse commands and PF Keys to navigate: - Up, Down, Left, Right - Top, Bottom, Max - Find

Variables defined to the program are shown here in Associated Storage Areas

Use normal Browse commands and PF Keys to navigate: - Up, Down, Left, Right - Top, Bottom, Max - Find

PF11


zSeries AD Tools


Associated Storage Areas (on the right side of the display)

Variable values are shown in character and hex

Variable values are shown in character and hex

PF3


zSeries AD Tools


back to program (event) detailPF3


zSeries AD Tools


back to events summaryPF3


zSeries AD Tools


back to main navigation screen … get System-Wide Info

select System-Wide Informationselect System-Wide Information

Enter


zSeries AD Tools


system-wide info … get DB2 info

On this screen there may be links for: - DB2 Information - CICS Information - IMS Information

On this screen there may be links for: - DB2 Information - CICS Information - IMS Information

select DB2 Informationselect DB2 Information

Enter


zSeries AD Tools


DB2 informationPF3


zSeries AD Tools


back to system-wide infoPF3


zSeries AD Tools


back to the main navigation screen


zSeries AD Tools

Interactive Reanalysis – Analyze an Abend

Walkthrough: Use Interactive Reanalysis to determine the cause of an abend

‘I’ command to start Re-analysis


Navigate to understand why the application abended


zSeries AD Tools

Analyze an Abend (1 of 19)


Enter


zSeries AD Tools




“Y” here makes this screen appear automatically when you start an Interactive Reanalysis.Get to this screen the first time from the “Options” pull-down menu, then select “Interactive Reanalysis Options”

“Y” here makes this screen appear automatically when you start an Interactive Reanalysis.Get to this screen the first time from the “Options” pull-down menu, then select “Interactive Reanalysis Options” PF3


zSeries AD Tools


go to Synopsis

Debug Clues:

Abended in program SAM2 because of a data exception

What is Fault Analyzer’s Synopsis?

select Synopsisselect Synopsis

Enter


zSeries AD Tools


page forward to see the active variables

Debug Clues:

Abended in program SAM2 while executing an ADD statement because of a data exception

Here’s a clue. What can cause a data exception?

Here’s a clue. What can cause a data exception?

PF8


zSeries AD Tools


return to the menu

Debug Clues:

Abended in program SAM2 while executing an ADD statement because of a data exception

Let’s go look at the bad variable

What variable contained the bad data?

What variable contained the bad data?

PF3


zSeries AD Tools


Debug Clues:

Abended in program SAM2 because:

ORD-ITEM-CODE had bad numeric data

Let’s go look at the bad variable, ORD-ITEM-CODE

select Eventsselect Events

Enter


zSeries AD Tools


go to detail for program SAM1

What is the relationship between programs SAM1 and SAM2?

What is the relationship between programs SAM1 and SAM2?

Enter

Debug Clues:





zSeries AD Tools


go to the bottom of the program detail displayEnter

Debug Clues:





zSeries AD Tools


Enter

Debug Clues:




What will we see here?What will we see here?


zSeries AD Tools


find the variable name or valueEnter

Debug Clues:





zSeries AD Tools


return to Program Detail

It is in Linkage Section. What does that mean?

It is in Linkage Section. What does that mean?

Debug Clues:


ORD-ITEM-CODE had bad numeric data (it is part of ORDER-RECORD)

Here is the the bad variable, ORD-ITEM-CODE

PF3

The bad dataThe bad data


zSeries AD Tools


return to the Events listPF3

Debug Clues:


ORD-ITEM-CODE has bad numeric data (it is part of ORDER-RECORD), which was passed from a calling program

Let’s go look at the passed data


zSeries AD Tools


go to detail for the calling program

The bad data was passed from a calling program. SAM1 called SAM2.

The bad data was passed from a calling program. SAM1 called SAM2.

Enter

Debug Clues:


ORD-ITEM-CODE has bad numeric data (it is part of ORDER-RECORD), which was passed from a calling program

Let’s go look at the passed data


zSeries AD Tools


detail for program SAM1 … go to the bottom

Debug Clues:


ORD-ITEM-CODE has bad numeric data (it is part of order-record), which was passed from a calling program

SAM1 called SAM2

Let’s go look at the passed dataEnter

What passed variable contained the bad data?

What passed variable contained the bad data?

clueclue


zSeries AD Tools


go to see variables and storage for this program

Debug Clues:



SAM1 called SAM2

SAM1 passed bad data in ORDER-RECORD

Let’s go look at ORDER-RECORD

Enter


zSeries AD Tools


return to program detail

The bad dataThe bad data

It is in File Section. What does that mean?

It is in File Section. What does that mean?

What is the file DD name?

What is the file DD name?

Debug Clues:



SAM1 called SAM2


Here is ORDER-RECORDPF3


zSeries AD Tools


go to Orders file info

Debug Clues:


ORD-ITEM-CODE has bad numeric data (it is part of order-record)

SAM1 called SAM2


The bad data was read from file ORDERS

Enter


zSeries AD Tools


page forward to see current record

Debug Clues:



SAM1 called SAM2



What is the full name of the ORDERS file?

What is the full name of the ORDERS file?

PF8


zSeries AD Tools


Debug Clues:



SAM1 called SAM2



Here is the bad data in the record

Here is the bad data in the record


zSeries AD Tools

Program SAM1

File DNET074.ADWORK.DATA

Analyze an Abend … what happened:

1. SAM1 read a record from an input file that contained bad data in the ORD-ITEM-CODE field

2. Program SAM1 called SAM2, passing the bad data

3. SAM2 abended when it tried to calculate using the bad data

Program SAM2

Abend


zSeries AD Tools

Batch Reanalysis

What is BATCH reanalysis?

A Fault Analyzer utility that runs as a batch job and generates an Analysis report

Why do Batch reanalysis?

To generate a new Analysis report

Possibly to:Apply compiler listing informationCreate a printable report for a CICS abendGenerate a new reanalysis report that you can print

Or download and email it to someone


zSeries AD Tools

Batch Reanalysis

Walkthrough: Batch Reanalysis

‘B’ command to start Re-analysis


Navigate to understand why the application abended


zSeries AD Tools

Batch Reanalysis

The B line command starts a Batch Reanalysis session…

Enter


zSeries AD Tools

Batch Reanalysis

batch options can be specified

PF3


zSeries AD Tools

Batch Reanalysis

JCL is generated … submit and/or save

Enter

Use the Edit ‘Create’ or ‘Replace’ command if you want to save the JCL in your own file

Use the Edit ‘Create’ or ‘Replace’ command if you want to save the JCL in your own file


zSeries AD Tools

Batch Reanalysis

report generated by batch reanalysis


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

CICS Transaction Support

Additional information available for CICS applicationsCICS control blocksTransaction storageLast Screen BufferCICS Trace TableCICS abend code explanation

View in Real-Time report or with Reanalysis


zSeries AD Tools

CICS Transaction Analysis



zSeries AD Tools


.

Enter


zSeries AD Tools


.

Enter

Control Blocks:CommareaCSATCAEISEUISTCTTE


zSeries AD Tools


.

Enter

Transaction Storage


zSeries AD Tools


.

Enter

Last CICS 3270 Screen Buffer


zSeries AD Tools


.

CICS Trace Table

Enter


zSeries AD Tools


get the CICS Abend Code description … first go to Events


zSeries AD Tools


select the abend event


zSeries AD Tools


select the abend code


zSeries AD Tools


abend description


zSeries AD Tools

DB2 Analysis

Additional information available for DB2 applicationsDB2 SubsystemPlan informationOwner and Authorization IDsLast Executed SQL StatementHost VariablesSQL Communications AreaSQLCode and SQLState explanations



zSeries AD Tools

DB2 Analysis

.Enter


zSeries AD Tools

DB2 Analysis

.Enter


zSeries AD Tools


page forward through DB2 information

DB2 subsystem, plan, IDs

DB2 subsystem, plan, IDs

Last SQL statementLast SQL statement

Host variablesHost variables

PF8


zSeries AD Tools

DB2 Analysis

Select SQLCode

SQL Communications AreaSQL Communications Area

Enter


zSeries AD Tools

DB2 Analysis

SQLCode explanationPF3


zSeries AD Tools

DB2 Analysis

Select SQLStateEnter


zSeries AD Tools

DB2 Analysis

SQLState explanation


zSeries AD Tools

IMS Analysis

Additional information available for IMS applications IMS SubsystemRegion TypePSB IOPCBDatabase PCBs

Segment Key feedback area Call trace

Call counts IMS abend code explanations



zSeries AD Tools

IMS Analysis

select System-Wide InformationEnter


zSeries AD Tools

IMS Analysis

select IMS informationEnter

Information about database datasets (DLI region)

Information about database datasets (DLI region)


zSeries AD Tools

IMS Analysis

page forward through IMS information

connection infoconnection info

Database PCB

Database PCB

PF8


zSeries AD Tools

IMS Analysis


Key feedback areaKey feedback area

Call traceCall trace

Second Database PCB

Second Database PCB

PF8


zSeries AD Tools

IMS Analysis


Call countsCall counts

PF8


zSeries AD Tools

IMS Analysis


Call countsCall counts


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

Getting your Application Ready

Fault Analyzer reads compiler listings and/or side files to get source information

Specific compiler options are needed

Save compiler listings or side files to a PDS or PDSE or be able to recompile your program to re-create a

listing


zSeries AD Tools

Generating Source info for Fault Analyzer

There are three ways to store Program Source data that Fault Analyzer will use:

1. Save the compiler listing for assembler, save the SYSADATA file instead

2. or process the compiler listing through FA utility program IDILANGX to create an IDILANGX “side file” It has the same info as the compiler listing, but takes less

space

3. or (for COBOL only), save the SYSDEBUG side file generated by the compiler specify the TEST(… , … ,SEPARATE) compiler option to

create a SYSDEBUG file


zSeries AD Tools

Save source info… Method 1

Program Source

Link EditCompiler

Compile and link the application program

Save the compiler listing in a PDS member

Use FA options to point FA to the listings PDS

Use different Listings files for different compilers

Load module

LOADLIB

FaultAnalyzer

Application

FA Invokation Exit

AbendCompiler Listings

PDS

Fault Analyzer options:DATASETS(IDILCOB(…),IDILCOBO(…), IDIADATA(…),IDILPLI(…),IDILPLIE,IDIC(…))

SYSP

RIN

T D

D


zSeries AD Tools


Program Source

Link EditCompilerCompile and link the application program

Process the listing with the IDILANGX utility

Save the output of IDILANGX in a PDS member

Use FA options to point FA to the IDILANGX side file

Load module

LOADLIB

FaultAnalyzer

Application

FA Invokation Exit

AbendIDILANGX

side file PDS

Fault Analyzer options:DATASETS(IDILANGX(…))

SYSPRINT DD Temporary

data set

IDILANGX


zSeries AD Tools


Program Source

Link EditCOBOL

Compiler

For COBOL only (not OS/VS COBOL or COBOL II)

Specify the TEST compiler option with SEPARATE

Save the generated SYSDEBUG file in a PDS member

FA uses the dataset name stored in the load module to locate the side file

Load module

LOADLIB

FaultAnalyzer

Application

FA Invokation Exit

AbendSYSDEBUG Side File

PDSSY

SDEB

UG

DD

TEST(… , … ,SEPARATE) compiler option


zSeries AD Tools

Options to write info needed by FA in the listing

Required Compiler Options

If a compiler listing or IDILANGX side file will be used, these compiler options must be specified:

ASSEMBLER ESD LIST or ADATA

C/C++ AGGREGATE LIST SOURCE XREF

COBOL LIST MAP SOURCE XREF

PL/I AGGREGATE ATTRIBUTES ESD LIST MAP SOURCE STMT XREF

Don’t use OFFSET LONGNAME

These COBOL options are NOT needed if source info is stored in a SYSDEBUG side file instead of a listing

These COBOL options are NOT needed if source info is stored in a SYSDEBUG side file instead of a listing


zSeries AD Tools

COBOL Compile JCL – save Listing

Store the listing in a PDS or PDSE


Specify compiler options needed by FA

Specify compiler options needed by FA

Sample COBOL compile JCL to generate and save a compiler listing

SourceSource


zSeries AD Tools

PLI Compile JCL – save Listing



Ensure that compiler options needed by FA are all on

Ensure that compiler options needed by FA are all on

Sample PLI compile JCL to generate and save a compiler listing

SourceSource


zSeries AD Tools

Listings Datasets

Store Listings in a PDS or PDSE dataset

COBOL SYSPRINT: RECFM=FBA LRECL=133

PLI SYSPRINT: RECFM=VBA LRECL=125 minimum

Enterprise PLI SYSPRINT: RECFM=VBA LRECL=137

Assembler SYSADATA:

RECFM=VB LRECL=8188


zSeries AD Tools

COBOL Compile JCL – save SYSDEBUG

The compiler “TEST” option with SEPARATE is specified

The compiler “TEST” option with SEPARATE is specified

The SYSPRINT DD is the listing

The SYSPRINT DD is the listing

The SYSDEBUG DD is the side file

The SYSDEBUG DD is the side file

In this example, Source and Symbolics data are stored in the side file.

Sample COBOL compile JCL to generate and save a SYSDEBUG file

SourceSource


zSeries AD Tools

COBOL Compiler SYSDEBUG Dataset

Contains Symbolics information generated by the compiler

It is created only if TEST with SEPARATE option is specified

It is saved as a PDS or PDSE member The member name is the same as program ID

Format: Record format FB Logical record length between 80 and 1024


zSeries AD Tools

How FA locates SYSDEBUG datasets

FA locates SYSDEBUG datasets differently than compiler listings and IDILANGX side files

The compiler stores the name of the SYSDEBUG dataset in the load module

If the file is copied or renamed after the compile, FA will not be able to locate it

Load Library

Load Module ‘SAM1’

‘MY.LIST(SAM1)’

‘MY.LIST’

member ‘SAM1’

SYSDEBUG PDS


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

Fault Analyzer Options

System-wide options for your system are customized as part of the Fault Analyzer installation process stored in PARMLIB member IDICNF00 by default

You can specify options to override the defaults: For a batch job step For online functions

To override Batch job step options: Specify in JCL

To override Online options: Specify from the OPTIONS pull-down menu for

Interactive Reanalysis Batch Reanalysis


zSeries AD Tools

The “Datasets” Option

The DATASETS option specifies data sets that are to be dynamically allocated by Fault Analyzer

It is most commonly used to specify the name of files where program compiler listings are stored


zSeries AD Tools

Some other useful Fault Analyzer Options

RETAINDUMPFault Analyzer is to retain the SYSABEND dump, SYSMDUMP, or SYSUDUMP

INCLUDEReal-time analysis should be performed if an abend occurs

EXCLUDEReal-time analysis should not be performed if an abend occurs

DETAIL

The level of detail that should be included in the report as either SHORT, MEDIUM, or LONG

PREFERREDFOR-MATTINGWIDTH

The report line width for flowing paragraph text and hexadecimal storage formatting. Other parts of the report are not affected.

MAXMINIDUMP-PAGES

The maximum size of a minidump that will be stored in a Fault History file

NODUPControls if an entry for a duplicate abend is captured in a Fault History File


zSeries AD Tools

How to override Options – Batch Jobs

In most cases, no changes to run-time JCL are needed If you have to make changes to many batch jobs,

discuss with your Systems Programmer to see if default options or setup can be improved

To override default options, code an IDIOPTS DD statement in the step

Refer to a file containing options, or…

Refer to a file containing options, or…

…code options in-stream

…code options in-stream


zSeries AD Tools

How to override Options – Batch Jobs

Another way to override options is to preallocate files used by Fault Analyzer with DD statements.This has the same effect as specifying the DATASETS option

in IDIOPTS

Each DD statement must reference PDS or PDSE dataset(s)://IDIHIST DD … Fault History File//IDILC DD … C compiler listings//IDILCOB DD … COBOL compiler listings (not OS/VS)//IDILCOBO DD … OS/VS COBOL compiler listings//IDILPLI DD … PL/I compiler listings//IDIADATA DD … Assembler SYSADATA //IDILANGX DD … side files for all languages

If a job step does NOT abend, FA options are ignored


zSeries AD Tools

How to override Options – Reanalysis

Similar process for Interactive vs. Batch reanalysis

settings are remembered for your TSO ID

Enter

a. Select the Options menu

b. Select Batch or Interactive reanalysis

a. Select the Options menu

b. Select Batch or Interactive reanalysis

enter options…

enter options…

…or refer to a file containing

options

…or refer to a file containing

options


zSeries AD Tools

Tip: Use a file for options

If you have a lot of options to specify, put them in a file

Point to the file: in a batch step, with an //IDIOPTS DD statement for an online function, from the options panel


zSeries AD Tools

Options and Exits Setup

In addition to the default options, other customization is availableThese are typically set up by a Systems Programmer

Exit programs can be written to:Dynamically override options

For example, If your system has more than one Fault History File, an exit is used to select one during Real-Time analysis

Control report formatting, message lookup, and notification

User options module IDICNFUM can be used to:Point to a non-parmlib default options fileSpecify unique defaults for different groups or

applications


zSeries AD Tools

Agenda

Introduction




Fault Re-analysis



Using Options

Hints & Tips


zSeries AD Tools

Why didn’t Fault Analyzer find my compiler listing ?

Add an IDITRACE DD to your job to get trace information about why a particular compiler listing or side file was selected or rejected

//IDITRACE DD SYSOUT=*

Sample output:

Listing/Side-file search trace for COBMST5LE compile date 2002-12-17 time 16-11-10DA.WDBLANGX

Rejected - Member not foundRejected - Member not found

PATRICK.LISTING.COBOLAccepted - Timestamp valid, size test valid

- Timestamp date 2002-12-17 time 16.11.10DA.LISTING.COBOL

- Search not required. LISTSEARCH(BEST) satisfied- Search not required. LISTSEARCH(BEST) satisfied


zSeries AD Tools

Make a copy of your Fault Entry

Fault Entries will roll off of a Fault History file eventually You can keep a copy of a Fault Entry in your own file

First, allocate your own Fault History file Make it a PDS or PDSE, with the same attributes as the original Fault

History File (but probably a lot smaller!)

Method 1: just copy your Fault Entry from from the original Fault History file to yours The Fault Entry is a PDS member, the member name is the Fault ID Note: DO NOT delete or rename Fault Entries in the original file!

Method 2: use the IDIUTIL utility to “import” the entry This will also delete the entry from the original file


zSeries AD Tools

Take a “snapshot” from a program

IDISNAP is a program supplied with Fault Analyzer

To use it, code a call to program IDISNAP in your program

When IDISNAP is called, FA collects info and stores a Fault Entry Apart from the way in which Fault Analyzer is invoked, there is

no difference between this type of analysis and any other real-time analysis caused by an abend

After the snapshot is taken, control is restored to the calling program

COBOL example..

01 SNAP-PROGRAM PIC X(8) VALUE ’IDISNAP ’.PROCEDURE DIVISION...

CALL SNAP-PROGRAM...


zSeries AD Tools

Use VIEWs to access Fault History Files

If you have more than one Fault History File, “VIEW”s can be defined to simplify access

A VIEW can be: A shortcut to a Fault History File A concatenation of Fault History Files

Advantages: You don’t have to know full names of the Fault History Files A view can concatenate multiple files, so you don’t have to look in several files

one at a time

Access VIEWs from the FILE->LIST VIEWS menu

Some example VIEW names: PROD TEST CICS DEPT01 ALL

VIEWs can be set up by a Systems Programmer as part of FA installation


zSeries AD Tools

Look up Messages and Codes on IBM’s “LOOKAT” web site

www-1.ibm.com/servers/s390/os390/bkserv/lookat

Or www.ibm.com, and search for “LOOKAT”


zSeries AD Tools

Reference

The User’s Guide is available from the HELP menu from the Fault Analyzer online screens

Fault Analyzer manuals are available on the web (see next page)

Redbook: Introduction to the IBM Application Development Tools on z/OS and OS/390 http://www.redbooks.ibm.com/

and search for “AD Tools” describes the IBM Application Development Tools for z/OS and

OS/390 and includes scenarios that show how to use the tools to recognize, locate, and fix errors in application programs

IBM Fault Analyzer IBM File Manager IBM Debug Tool IBM Application Monitor IBM Workload Simulator


zSeries AD Tools

Fault Analyzer on the web

http://www.ibm.com/software/awdtools• then click on IBM Application Development Tools

Browse / Download product manuals

Documents

Fault Analyser