Upload
norma-ramsey
View
212
Download
0
Embed Size (px)
Citation preview
8 Copyright IBM Corporation, 2009. All Rights Reserved.This publication may refer to products that are not currently available in your country. IBM makes no commitment to make available any products referred to herein.
Agenda Key: 55CHSession Number: 540038
Performance Analysis using iDoctor Job Watcher
Dawn [email protected]
2© 2009 IBM Corporation
IBM iDoctor for i
a rapid snapshot taker that allows real-time, detailed analysis
an IBM i exclusive!Patented technology!
Job Watcher
4© 2009 IBM Corporation
Job Watcher
the goals
Broaden the user base for Performance Investigationenable Operators, Programmers, IS Managementas well as Performance Specialists, Consultants
Simplify and automate processes
Provide quick, immediate access to collected data
Provide more analysis options
6© 2009 IBM Corporation
Job Watcher Performance Data Collector
What is Job Watcher
a snapshot approach a non-intrusive collector collects one or multiple jobs/tasks or all jobs/tasks harvests job details such as waits, call stacks by default optionally retrieve SQL, activation group, communication details analysis can start after just two snapshots
More…
8© 2009 IBM Corporation
Job Watcher Graphical User Interface
What is Job Watcher cont…
Windows Graphical Client application wizard for constructing the collection options viewer for managing the collections viewer for analyzing collections (during and post) currently a 106 predefined graphs setup a monitor to have JW running continuously
user defined query and user defined graph facilitiesuser can create queries and graphssave, categorize and transfer
data viewer can also be used against other DB files like collection services
9© 2009 IBM Corporation
Job Watcher
Starting Job Watcher
GUI startup wizard or green screen command select active jobs/threads/tasks from a GUI list
even jobs on a job queue set the collection options:
interval durationwatch durationwatch name and librarywatch descriptionwhat details to collectoptionally set a trigger to control when the watch should
start or what program to call if specified conditions are met during a watch (a trigger wizard is provided)
11© 2009 IBM Corporation
iDoctor vs. IBM i 6.1 Performance Tools
When to use iDoctor vs. the new tools?
General positioning:
• Basic, high-level performance management – use new tools
• Advanced, detailed analysis – continue to use iDoctor
October 2008 IBM Systems Magazine article comparing iDoctor vs. IBM i 6.1 Performance Tools http://www.ibmsystemsmag.com/ibmi/october08/trends/21990p1.aspx
Job Watcher
12© 2009 IBM Corporation
Job Watcher
waiting
Is there a problem?What potential for improvement is there?What areas would we have to address to improve performance?
What if a job's run/wait signature looked like......
running
running waiting
the first step in detailed Performance Investigation
14© 2009 IBM Corporation
Job Watcher
What if the job's run/wait signature looked like......
running waiting
DASDexample #1
Blocksexample #2
CPUqexample #3
- easy to use and non-intrusive to jobs being watched.- allows real-time, summarized or detailed views of how jobs spend time
16© 2009 IBM Corporation
Job Watcher
snapshot #1 snapshot #2
OK if idle wait (like Key/Think time)
But if a long DASD request or a block? (and the current wait has not ended yet)
current wait
current wait can span snapshot intervals
17© 2009 IBM Corporation
Job Watcher
snapshot #1 snapshot #2
the current wait “bucket values” should be investigated for “bad waits” also
for example, a large “bad wait” bucket time and count suggest many small duration waits are occurring
current wait
or they can be of very short duration
current wait bucket totals
19© 2009 IBM Corporation
T I M E
JW – use rule/action and the lurk facility with history
the problem occurs
satisfied Job Watcher "rule" starts up DB recording
start investigation
JW coverage
dump reduction start investigation
21© 2009 IBM Corporation
Job Watcher
Job Watchera real-time, high-performance, snapshot sampler
created to fill the gap between
Collection Services
- and -
the short duration, high data volumes, long dump/analysis times of PEX Trace
Job Watcher in a nutshell
23© 2009 IBM Corporation
Once installed, we are ready to begin.
25© 2009 IBM Corporation
Connect to your system.
27© 2009 IBM Corporation
Launching into the world of Job Watcher
29© 2009 IBM Corporation
Job Watcher Demos
1. Starting a Job Watcher collectionSetting up a Job Watcher monitor
2. Job Watcher analysisObject Lock Contention
3. Setup a collection using the trigger option4. Job Watcher analysis and Situational Analysis
30© 2009 IBM Corporation
Demo #1
Starting a Job Watcher CollectionSetting up a Job Watcher Monitor
31© 2009 IBM Corporation
33© 2009 IBM Corporation
35© 2009 IBM Corporation
37© 2009 IBM Corporation
39© 2009 IBM Corporation
41© 2009 IBM Corporation
43© 2009 IBM Corporation
45© 2009 IBM Corporation
47© 2009 IBM Corporation
49© 2009 IBM Corporation
51© 2009 IBM Corporation
53© 2009 IBM Corporation
55© 2009 IBM Corporation
57© 2009 IBM Corporation
59© 2009 IBM Corporation
Working with Job Watcher Monitors
Click on “Job Watcher” then select “Monitors”.
Monitors are in the tree-view similar to collections.
61© 2009 IBM Corporation
63© 2009 IBM Corporation
65© 2009 IBM Corporation
67© 2009 IBM Corporation
69© 2009 IBM Corporation
Refresh button
71© 2009 IBM Corporation
73© 2009 IBM Corporation
All existing collections will be displayed
75© 2009 IBM Corporation
And many hours later….
77© 2009 IBM Corporation
Demo #2
Job Watcher Data AnalysisObject Lock Contention
79© 2009 IBM Corporation
81© 2009 IBM Corporation
83© 2009 IBM Corporation
85© 2009 IBM Corporation
87© 2009 IBM Corporation
89© 2009 IBM Corporation
91© 2009 IBM Corporation
Demo #3
Job Watcher Data Analysis
92© 2009 IBM Corporation
Customer supplied examples of the problem
223:33:225/4/200523:33:205/4/2005327500FRGJOB
223:36:585/4/200523:36:565/4/2005327472FRGJOB
223:45:125/4/200523:45:105/4/2005327466FRGJOB
321:45:235/4/200521:45:205/4/2005327451FRGJOB
321:45:235/4/2020521:45:205/4/2005327447FRHJOB
321:54:065/4/200521:54:035/4/2005327455FRHJOB
321:54:065/4/200521:54:035/4/2005327461FRGJOB
419:59:295/4/200519:59:255/4/2005327410FRGJOB
522:44:215/4/200522:44:165/4/2005327422FRGJOB
523:06:135/4/200523:06:085/4/2005327432FRHJOB
623:06:135/4/200523:06:075/4/2005327450FRGJOB
1123:46:375/4/200523:46:265/4/2005327446FRGJOB
secondsEndedEndedStartedStartedNumName
ElapseTimeDateTimeDateJobJob
Nightly processing log – 05/04/2005
94© 2009 IBM Corporation
96© 2009 IBM Corporation
98© 2009 IBM Corporation
100© 2009 IBM Corporation
102© 2009 IBM Corporation
Fast path to JW supplied graphs
Collection management
options
Collection details, system info.
Graph scope Available graphs
104© 2009 IBM Corporation
106© 2009 IBM Corporation
108© 2009 IBM Corporation
Start with a search for the job referenced…
110© 2009 IBM Corporation
Start with a search for the job referenced…
Enter job name and time range
112© 2009 IBM Corporation
114© 2009 IBM Corporation
116© 2009 IBM Corporation
Select an interval of a candidate and go for a run/wait signature…
118© 2009 IBM Corporation
Waiter’s run/wait signature
120© 2009 IBM Corporation
Legend on/off button
122© 2009 IBM Corporation
124© 2009 IBM Corporation
Menu Buttons Interval Arrows
126© 2009 IBM Corporation
128© 2009 IBM Corporation
Drill down for call stack, holder, etc. or go for a collection overview
130© 2009 IBM Corporation
132© 2009 IBM Corporation
Holder’s run/wait signature
134© 2009 IBM Corporation
Who is using CPU?
136© 2009 IBM Corporation
138© 2009 IBM Corporation
Why is job# 328268 burning so much CPU?
140© 2009 IBM Corporation
142© 2009 IBM Corporation
List other jobs that are also engaged in QQQTSORT…
144© 2009 IBM Corporation
146© 2009 IBM Corporation
148© 2009 IBM Corporation
Conclusion:
• Using Job Watcher we were able to determine that there were a number of jobs running sorts consuming high CPU. This caused jobs to wait excessively in CPUQ resulting in seizes to be held longer causing our job to wait for a seize much longer than expected.
• DBMON and/or Visual Explain should be used to review the SQL statements and the access methods being used in an effort to understand the reasons for the sort.
149© 2009 IBM Corporation
In Summary
The Job Watcher Approach
allows real-time and post-watch inspection of run/wait components
provides results for day to day use by coders, testers, and operations as well as for presentations in the executive boardroom
requires less skill to get started in performance investigation and makes the skilled, experienced performance investigator more productive
150© 2009 IBM Corporation
Demo #4
Job Watcher Data Analysis
151© 2009 IBM Corporation
Job Watcher analysis demo
• The investigation of a very large, busy 595 during a slowdown
• Use some of the newer iDoctor features
• Starting with Situation Analysis!
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
247© 2009 IBM Corporation
What have we learned so far about the data…
• Long DASD response experienced by jobs (mostly related to writes).
• Long DASD write response is slowing DBCLOSE operations, which in turn causes many users to experience long seize wait contention.
• The DBCLOSE operation is forcing index and data space pages to DASD (is this the last updater forces all the file indexes and data space pages out scenario?).
• Other significant overview wait categories (Disk Space Contention, Abnormal Contention and Disk Op-Start Contention) suggest pressure on DASD control blocks, stress in the create/destroy/extends of DASD space and waiting on system lists/queues to get a turn – all events closely related to file opens/closes, users signing on/off, new job creation (JW misses job fragments and short-lived jobs – so SQE could be a factor).
• There are lots of new jobs (inspection of initiations/termination – and the increase in TDE count from beg to end of collection. (see extra credit query example).
• Bursts of IO requests that JW can’t see (but would be caught in CSI reports along with DASD unit statistics to allow comparison of JW system oriented DASD response times to CSI hardware DASD response times).
248© 2009 IBM Corporation
And about Job Watcher features…
• Situation Analysis
• Alternate View support, Two graph support, Variable width and Normalize buttons, Legend options, Table support, Record Quick View
• Drill downs -- to call stacks, call stack program occurrence reports, object occurrence reports and Current wait details
• The iDoctor SQL editor
• Extra Credit -- a look at the JW Thread wait time signature that ran and collected the data for/during this collection
• Extra Credit -- a user defined query to count new the TDE’s created per interval
249© 2009 IBM Corporation
And about Current Waits…
• Job Watcher captures TDE accumulative data counters at interval snapshots … including the current wait condition.
JOB100010
JOB100009
JOB100008
JOB100007
JOB100006
Time - - - - - - - - - - - - - - - - - - - - ->
Interval Snapshot 19 20 21
Current Wait Reported at snapshot The actual Wait duration
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
© 2009 IBM Corporation
259© 2009 IBM Corporation
Visit the web site for more information
261© 2009 IBM Corporation
© 2008 IBM Corporation
®
262262 © 2008 IBM Corporation
Trademarks and Disclaimers8 IBM Corporation 1994-2009. All rights reserved.References in this document to IBM products or services do not imply that IBM intends to make them available in every country.Trademarks of International Business Machines Corporation in the United States, other countries, or both can be found on the World Wide Web at http://www.ibm.com/legal/copytrade.shtml.
Adobe, Acrobat, PostScript and all Adobe-based trademarks are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, other countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.
IT Infrastructure Library is a registered trademark of the Central Computer and Telecommunications Agency which is now part of the Office of Government Commerce.
ITIL is a registered trademark, and a registered community trademark of the Office of Government Commerce, and is registered in the U.S. Patent and Trademark Office.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Cell Broadband Engine and Cell/B.E. are trademarks of Sony Computer Entertainment, Inc., in the United States, other countries, or both and are used under license therefrom.
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
Information is provided "AS IS" without warranty of any kind.
The customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.
Information concerning non-IBM products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by IBM. Sources for non-IBM list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. IBM has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-IBM products. Questions on the capability of non-IBM products should be addressed to the supplier of those products.
All statements regarding IBM future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only.
Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in IBM product announcements. The information is presented here to communicate IBM's current investment and development activities as a good faith effort to help with our customers' future planning.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.
Prices are suggested U.S. list prices and are subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography.