SOA Testing IMS applicationsRegression testing of Z/OS, IMS DB2 and MQ at the SCC by replaying production workload
Process Summary
Extract 01 and 03 logrecs for a time period
Create Input file from 01 and 03 data
Sort the input file
Transfer input file to the SCC
Run the test
Input file
Each input record and it’s responses are grouped together consecutively on the log
00 control record• contains information about the message, userid, lterm,etc. Also contains the
elapsed time since the last message
01 record, contains the input message. • May be segmented over multiple messages
03 record contains a ‘response snippet’• A bit of the response to be compared for success• An indication of the expected response time• Includes Length and offset• One per response• May be 0 or more than one per input message
Log extract Process
Extract program
ProductionIMS LOGS
Apply optional
filters Sort
ExtractFile
Take this file to the flash copy test systems
Extract Program – LIVSOAL
Read IMS SLDS directly
• Produces smallest amount of data
• Use for normal testing
• Each system extracted separately, then merged with a sort
Produce from IMSPI extract
• Extract and merge 01 03 recs into a DASD dataset
• Run the extract program against the DASD dataset
• Useful for rerunning, developing, etc
• Still requires sort step afterwards
Distributed Environment - Simplified
MQ
JVM
JVM
JVM
JVM
AIX
MQ
OTMA MPR
MPR
MPR
MPR
DB2
MPR
IMS
Z/OS
WAS
Regression testing SOA applications
Subtask
Subtask
Subtask
Subtask
Subtask
Driver
MQ
OTMA MPR
MPR
MPR
MPR
DB2
MPR
IMS
Extractfile
Z/OS
Address Space
Driver Program LIVSOAD
Uses subtasking to drive MQ at the required message rate
Mainline controls pacing and subtask scheduling
Subtasks issue MQPUT, MQGET with wait option.
Waits on message correlid, like our real SOA systems
Driver Program LIVSOAD
Finds available input queues dynamically
Accepts commands via sysin or modify
Coordinates startup and shutdown of subtasks
Reads the input file
Creates the input message from segments
Build the MQIIH in front
POSTs subtask to do the MQ calls
Subtask Program LIVSOAS1
•Connects to MQ, creates reply queue• First time only
•Sends Message to IMS using MQPUT1• Uses the original userid authority• Uses the same commit mode• Uses the same MQ input queue for a particular TPIPE• 3270 messages round robined
•Waits for response with MQGET• Wait time factored on original response time• Waits on a specific correlid response message
•Compares the response received to the expected messages•Accumulates response times•Waits for more work•Disconnects from MQ on termination
Error Processing
•Count of errors maintained• Terminate after a maximum numbers
•MQ errors• Completion and reason codes written to syslog
•Comparison Errors• Snaps output record and snippets.
•All subtasks busy• Wait 1 second and retry• Repeat for up to 30 seconds before termination• Number of subtasks is adjustable• But beware of MQ IDBACK=20 – can reassemble/zap table, or set with
online command =MVMA SET SYSTEM IDBACK(500)
Checkpoint Processing
Replay log data until stop is triggered• By end of log data• By MAXMSGS processed• By MAXERRS reached• By ELAPSED minutes• By STOP command
Write the next record STCK to checkpoint file
Reposition here on next restart
Aim is only to replay a piece of log data once
Execution Parameters
Command Values SYSIN Modify Notes
MAXTASKS 1-100 Yes No Number of subtasks
PCTRATE 1-999 Yes Yes Percentage of the original message rate to send
MINRATE 0-999 Yes Yes Minimum rate override
MAXRATE 0-999 Yes Yes Maximum rate override
ELAPSED 0-999 Yes Yes Elapsed time for test
MAXMSGS 0-99999 Yes Yes Max messages for test
MAXERRS 0-99999 Max errors before term
ABEND N/A No Yes Terminates with dump
QMGR 8char Yes No Queue manager name
STGCLASS 8char Yes No Storage class name of
IMS as defined to MQ
STRTTIME 6char Yes No Time to start
STRTDATE 8char Yes No Date to start
PLEXNAME 8char Yes No Stop Accidents!!
Log extract Parameters
Command Values Notes
STRTDATE yyyymmdd Start date to extract
STRTTIME hhmmss Start time to extract
ENDTDATE yyyymmdd END date to extract
ENDTIMETIME hhmmss END time to extract
If not specified, reads from start to end of supplied log
Paired date and time must be both specified, or omitted
Log extract Filtering
Parameters Values Notes
T= TRAN name Can be generic
U= userid Optional, can be generic
M= lterm Optional, can be generic
O= 1-500 Compare offset, defaults to 0
L= 0,255 Length of comparison, default 100,
C N/A Uppercase compare data
X N/A Exclude
Examples T=ABC,M=CSQ*,O=12,L=10T=ABC,O=12,L=10,U
Testing Results
About 95% of transactions worked OK ‘out of the box’, OTMA+3270
With filtering, now 99.8% work
Not all transactions replay OK
SOA/OTMA workload is generally OK
Some transactions ‘call out’ via MQ to off mainframe – not available at SCC
Some transactions had a date/time in first 100 bytes – change the compare offset/length for these trans
Not all 3270 works
EDIT=UC and no MFS – gets uppercased when it didn’t previously
DRD could easily change to EDIT=ULC3270 traffic was not the reason for writing – a bonus if it worked
Testing Approach
Get a clean run, excluding stuff we can’t get to workRC=0 means function worked without errorsTune the comparison offset for some transactionsIMSGEN to remove EDIT=UC, or online command with DRDExclude transactions we cannot get to workRun the test before a changeRun the test after a changeNeed to pull off and compare IMF data for performance profile comparisons
Testing Notes
Startup mainview on SYSA/B
Start up summary trace and MWAIT for all transactions
Start up MQ MVMA and MVM1
Nobble JCL for CHIN
Make sure DB2s are started and connected, MSC links started, regions up
Start with a small, slow sample.
Expect ‘first time’ timeouts
Can run overnight, set STRTDATE and STRTTIME in SYSIN. Job will wait until that time to begin test
Testing Notes
Jobs on IV.MASTER.JOB(SOATESTn)
1 – log extract IMSA2 – Log extract IMSC3 – SORT SOA file4 - DUMP/TERSE SOA FILE5 - FTP GET/UNTERSE SOA FILE at SCC6 - Execute driver program7 – Terse IMF logrecords to take back to real systems8 – Restore IMF logrecs9 – LOAD IMS logrecs into SAS
Enhancements?
Do a before/after comparison job for IMF log data
Report the offset of compare failures
Summary report – list by highest errors
Can we do anything for stored procs?
Extract program – Use DBRC API, Dynamic allocation of SLDS, SORT in one step
IMS Connect
OTMACI