View
107
Download
5
Category
Preview:
Citation preview
Java Agent Implementation Guide Release 9.5
CA Application Performance Management
This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to as the Documentation) is for your informational purposes only and is subject to change or withdrawal by CA at any time.
This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. This Documentation is confidential and proprietary information of CA and may not be disclosed by you or used for any purpose other than as may be permitted in (i) a separate agreement between you and CA governing your use of the CA software to which the Documentation relates; or (ii) a separate confidentiality agreement between you and CA.
Notwithstanding the foregoing, if you are a licensed user of the software product(s) addressed in the Documentation, you may print or otherwise make available a reasonable number of copies of the Documentation for internal use by you and your employees in connection with that software, provided that all CA copyright notices and legends are affixed to each reproduced copy.
The right to print or otherwise make available copies of the Documentation is limited to the period during which the applicable license for such software remains in full force and effect. Should the license terminate for any reason, it is your responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed.
TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION AS IS WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, LOST INVESTMENT, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED IN ADVANCE OF THE POSSIBILITY OF SUCH LOSS OR DAMAGE.
The use of any software product referenced in the Documentation is governed by the applicable license agreement and such license agreement is not modified in any way by the terms of this notice.
The manufacturer of this Documentation is CA.
Provided with Restricted Rights. Use, duplication or disclosure by the United States Government is subject to the restrictions set forth in FAR Sections 12.212, 52.227-14, and 52.227-19(c)(1) - (2) and DFARS Section 252.227-7014(b)(3), as applicable, or their successors.
Copyright 2013 CA. All rights reserved. All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.
CA Technologies Product References
This document references the following CA Technologies products and features:
CA Application Performance Management (CA APM)
CA Application Performance Management ChangeDetector (CA APM ChangeDetector)
CA Application Performance Management ErrorDetector (CA APM ErrorDetector)
CA Application Performance Management for CA Database Performance (CA APM for CA Database Performance)
CA Application Performance Management for CA SiteMinder (CA APM for CA SiteMinder)
CA Application Performance Management for CA SiteMinder Application Server Agents (CA APM for CA SiteMinder ASA)
CA Application Performance Management for IBM CICS Transaction Gateway (CA APM for IBM CICS Transaction Gateway)
CA Application Performance Management for IBM WebSphere Application Server (CA APM for IBM WebSphere Application Server)
CA Application Performance Management for IBM WebSphere Distributed Environments (CA APM for IBM WebSphere Distributed Environments)
CA Application Performance Management for IBM WebSphere MQ (CA APM for IBM WebSphere MQ)
CA Application Performance Management for IBM WebSphere Portal (CA APM for IBM WebSphere Portal)
CA Application Performance Management for IBM WebSphere Process Server (CA APM for IBM WebSphere Process Server)
CA Application Performance Management for IBM z/OS (CA APM for IBM z/OS)
CA Application Performance Management for Microsoft SharePoint (CA APM for Microsoft SharePoint)
CA Application Performance Management for Oracle Databases (CA APM for Oracle Databases)
CA Application Performance Management for Oracle Service Bus (CA APM for Oracle Service Bus)
CA Application Performance Management for Oracle WebLogic Portal (CA APM for Oracle WebLogic Portal)
CA Application Performance Management for Oracle WebLogic Server (CA APM for Oracle WebLogic Server)
CA Application Performance Management for SOA (CA APM for SOA)
CA Application Performance Management for TIBCO BusinessWorks (CA APM for TIBCO BusinessWorks)
CA Application Performance Management for TIBCO Enterprise Message Service (CA APM for TIBCO Enterprise Message Service)
CA Application Performance Management for Web Servers (CA APM for Web Servers)
CA Application Performance Management for webMethods Broker (CA APM for webMethods Broker)
CA Application Performance Management for webMethods Integration Server (CA APM for webMethods Integration Server)
CA Application Performance Management Integration for CA CMDB (CA APM Integration for CA CMDB)
CA Application Performance Management Integration for CA NSM (CA APM Integration for CA NSM)
CA Application Performance Management LeakHunter (CA APM LeakHunter)
CA Application Performance Management Transaction Generator (CA APM TG)
CA Cross-Enterprise Application Performance Management
CA Customer Experience Manager (CA CEM)
CA Embedded Entitlements Manager (CA EEM)
CA eHealth Performance Manager (CA eHealth)
CA Insight Database Performance Monitor for DB2 for z/OS
CA Introscope
CA SiteMinder
CA Spectrum Infrastructure Manager (CA Spectrum)
CA SYSVIEW Performance Management (CA SYSVIEW)
Contact CA Technologies
Contact CA Support
For your convenience, CA Technologies provides one site where you can access the information that you need for your Home Office, Small Business, and Enterprise CA Technologies products. At http://ca.com/support, you can access the following resources:
Online and telephone contact information for technical assistance and customer services
Information about user communities and forums
Product and documentation downloads
CA Support policies and guidelines
Other helpful resources appropriate for your product
Providing Feedback About Product Documentation
If you have comments or questions about CA Technologies product documentation, you can send a message to techpubs@ca.com.
To provide feedback about CA Technologies product documentation, complete our short customer survey which is available on the CA Support website at http://ca.com/docs.
Contents 7
Contents
Chapter 1: Introduction to the Java Agent 19
About Introscope and Java Agents ............................................................................................................................. 19
Planning a Java Agent Deployment ............................................................................................................................ 20
Install and Evaluate the Default Functionality .................................................................................................... 20
Determine Configuration Requirements ............................................................................................................. 21
Define a Baseline Agent Profile with Appropriate Configuration Properties ...................................................... 21
Evaluate Agent Performance Overhead .............................................................................................................. 22
Validate and Deploy the Agent Configuration .................................................................................................... 22
Deploying the Java Agent ........................................................................................................................................... 22
Chapter 2: Installing and Configuring the Java Agent 23
Before You Install the Agent ...................................................................................................................................... 23
Select the method for installing the Java agent ......................................................................................................... 24
Install the Java Agent Interactively ..................................................................................................................... 24
Install the Java Agent Silently .............................................................................................................................. 26
Install Manually Using Installation Archives........................................................................................................ 30
About the Java Agent Directory Structure ................................................................................................................. 30
Configuring Detection of Synthetic Transactions ....................................................................................................... 32
Using the TagScript Utility ................................................................................................................................... 34
Java 7 Autoprobe ....................................................................................................................................................... 35
How to instrument applications ................................................................................................................................. 35
Configuring the Application Server to Start the Java Agent ....................................................................................... 36
Configure Apache Tomcat to use the Java agent ................................................................................................ 36
Configure JBoss to Use the Java Agent ............................................................................................................... 37
Configure Oracle WebLogic to Use the Java Agent ............................................................................................. 40
Configure WebLogic with JRockit JVM to Use the Java Agent ............................................................................ 45
Configure WebLogic with JRockit JVM to View Socket Metrics .......................................................................... 45
Configure IBM WebSphere to Use the Java Agent .............................................................................................. 46
Configure Oracle Application Server to use the Java agent ................................................................................ 57
Configure GlassFish to Use the Java Agent ......................................................................................................... 57
Configure SAP Netweaver to Use the Java Agent ............................................................................................... 58
Configuring the connection to the Enterprise Manager ............................................................................................ 59
Connect to the Enterprise Manager Using a Direct Socket Connection ............................................................. 59
Connect to the Enterprise Manager with HTTP tunneling .................................................................................. 60
Configure a proxy server for HTTP tunneling ...................................................................................................... 61
Connect to the Enterprise Manager with HTTPS tunneling ................................................................................ 61
8 Java Agent Implementation Guide
Connect to the Enterprise Manager over SSL ..................................................................................................... 62
Configure Agent Load Balancing ......................................................................................................................... 63
Upgrade Multiple Agent Types................................................................................................................................... 63
Uninstalling the Java agent ........................................................................................................................................ 64
Uninstalling the Java agent from z/OS ................................................................................................................ 65
Chapter 3: Configuring Agent Properties 67
How to modify communication with Enterprise Manager ......................................................................................... 67
How to configure backup Enterprise Managers and failover properties ................................................................... 67
How to enable and use additional GC metrics ........................................................................................................... 69
How to enable and configure thread dumps ............................................................................................................. 69
How to Configure an Agent to Collect Distribution Statistic Metrics ......................................................................... 72
Examples of Distribution Statistics Metrics ......................................................................................................... 73
Chapter 4: AutoProbe and ProbeBuilding Options 75
AutoProbe and ProbeBuilding Overview.................................................................................................................... 75
Unsupported Instrumentation Method .............................................................................................................. 76
Configuring ProbeBuilding ......................................................................................................................................... 76
Full or typical tracing options .............................................................................................................................. 76
Dynamic ProbeBuilding ....................................................................................................................................... 77
Configure Dynamic ProbeBuilding ...................................................................................................................... 79
Dynamic Instrumentation Impacts Performance for IBM JDK ............................................................................ 80
Dynamic ProbeBuilding Versus Dynamic Instrumentation ................................................................................. 80
ProbeBuilding Class Hierarchies .......................................................................................................................... 81
Removing line numbers in bytecode ................................................................................................................... 84
Chapter 5: ProbeBuilder Directives 85
ProbeBuilder Directives Overview ............................................................................................................................. 85
Components traced by the default PBDs ............................................................................................................ 86
Default PBD Files ................................................................................................................................................. 87
Default PBD Files From Previous Releases .......................................................................................................... 89
Default PBL files .................................................................................................................................................. 90
Default Tracer Groups and Toggles Files ............................................................................................................. 90
Turning tracer groups on or off ........................................................................................................................... 99
Adding classes to a tracer group ....................................................................................................................... 100
EJB Naming ........................................................................................................................................................ 102
Using the IntroscopeAgent.profile, PBLs, and PBDs together .................................................................................. 103
Applying ProbeBuilder Directives ............................................................................................................................. 103
Using JVM AutoProbe ....................................................................................................................................... 103
Using the ProbeBuilder Wizard or command-line ProbeBuilder ...................................................................... 104
Contents 9
Instrumenting with new and changed PBDs ..................................................................................................... 104
Creating custom tracers ........................................................................................................................................... 106
Using a custom BlamePointTracer tracer for common metrics ........................................................................ 106
Directive names and arguments used in tracer syntax ..................................................................................... 107
Commonly used tracer names and examples ................................................................................................... 109
Advanced single-metric tracers......................................................................................................................... 111
Skip directives ................................................................................................................................................... 114
Counting object instances ................................................................................................................................. 114
Turning on InstrumentPoint directives ............................................................................................................. 115
Combining custom tracers ................................................................................................................................ 115
Instrumenting and Inheritance ......................................................................................................................... 115
Java Annotations ............................................................................................................................................... 116
Using Blame Tracers to mark blame points.............................................................................................................. 117
Blame Tracers .................................................................................................................................................... 117
High agent CPU overhead from deep nested frontend transactions ................................................................ 118
Custom FrontendMarker Directive ................................................................................................................... 118
Blame Tracers in standard PBDs ....................................................................................................................... 119
Boundary Blame and Oracle Backends ............................................................................................................. 119
Chapter 6: Java Agent Naming 121
Understanding the Java Agent name ....................................................................................................................... 121
How the Agent Determines its Name ............................................................................................................... 122
How Introscope resolves agent naming conflicts ............................................................................................. 123
Agent naming considerations for clustered applications ......................................................................................... 124
Specifying an agent name using a Java system property ......................................................................................... 124
Specifying an agent name using a system property key .......................................................................................... 125
Obtaining an agent name from the application server ............................................................................................ 125
Application servers that support agent naming ................................................................................................ 125
Automatic agent naming .......................................................................................................................................... 126
Automatic agent naming and renamed agents ................................................................................................. 127
Advanced automatic agent naming options ..................................................................................................... 127
Enabling cloned agent naming in clustered environments ...................................................................................... 129
Cloned agent naming scenario .......................................................................................................................... 129
Configuring unique names for application instances ........................................................................................ 129
Application triage map and the agent name ............................................................................................................ 130
Chapter 7: Java Agent Monitoring and Logging 131
Configuring agent connection metrics ..................................................................................................................... 131
Socket metrics .......................................................................................................................................................... 132
Restricting socket and SSL metric collection ..................................................................................................... 132
Fine-tuning socket and SSL metric collection .................................................................................................... 133
10 Java Agent Implementation Guide
SSL, NIO, and Socket Tracing in the Application Triage Map ............................................................................ 133
Change the Component Name in the Application Triage Map ......................................................................... 134
Turning off socket and SSL metric collection .................................................................................................... 135
Backwards Compatibility ................................................................................................................................... 135
Configuring logging options ..................................................................................................................................... 136
Running the agent in verbose mode ................................................................................................................. 136
Redirecting agent output to a file ..................................................................................................................... 137
Changing the name or location of the agent logfile .......................................................................................... 138
Agent log files and automatic agent naming .................................................................................................... 138
Rolling up logs by date or size ........................................................................................................................... 139
Managing ProbeBuilder Logs ................................................................................................................................... 140
Command-line ProbeBuilder and ProbeBuilder Wizard log name and location ............................................... 140
AutoProbe log name and location .................................................................................................................... 140
Chapter 8: Configuring LeakHunter and ErrorDetector 141
LeakHunter ............................................................................................................................................................... 141
How LeakHunter works ..................................................................................................................................... 142
What LeakHunter tracks in Java ........................................................................................................................ 142
What LeakHunter does not track ...................................................................................................................... 143
System and version requirements .................................................................................................................... 143
Enabling and disabling LeakHunter .......................................................................................................................... 144
Configuring LeakHunter properties .......................................................................................................................... 144
Ignoring collections that cause poor performance ........................................................................................... 146
Running LeakHunter ................................................................................................................................................. 147
Identifying potential leaks with collection IDs ......................................................................................................... 147
LeakHunter log file ................................................................................................................................................... 148
Potential leak first identified log entry ............................................................................................................. 148
Identified potential leak stops leaking log entry ............................................................................................... 149
Identified potential leak is leaking again log entry ........................................................................................... 150
LeakHunter timeout log entry ........................................................................................................................... 150
Using LeakHunter ..................................................................................................................................................... 150
ErrorDetector ........................................................................................................................................................... 151
Types of errors .................................................................................................................................................. 151
How ErrorDetector works ................................................................................................................................. 152
Enabling ErrorDetector in the Java Agent ................................................................................................................ 152
Configuring ErrorDetector options ........................................................................................................................... 153
Advanced error data capture ................................................................................................................................... 154
Defining new error types.......................................................................................................................................... 154
ExceptionErrorReporter .................................................................................................................................... 154
MethodCalledErrorReporter ............................................................................................................................. 155
ThisErrorReporter ............................................................................................................................................. 155
Contents 11
HTTPErrorCodeReporter ................................................................................................................................... 156
Using error tracer directives with caution ........................................................................................................ 156
Using ErrorDetector ................................................................................................................................................. 156
Chapter 9: Configuring Boundary Blame 157
Understanding Boundary Blame .............................................................................................................................. 157
Using URL groups ..................................................................................................................................................... 157
Defining keys for URL groups ............................................................................................................................ 158
Defining membership of each URL group ......................................................................................................... 158
Define the name for a URL group ..................................................................................................................... 159
Advanced naming techniques for URL groups (optional) ................................................................................. 159
Running the URLGrouper .................................................................................................................................. 163
Using Blame tracers ................................................................................................................................................. 163
Chapter 10: Configuring Transaction Trace Options 165
New Mode of Transaction Tracing ........................................................................................................................... 165
Configuring the Agent to Use Legacy Mode Transaction Tracing ..................................................................... 166
Controlling automatic Transaction Tracing behavior ............................................................................................... 168
Transaction Trace component clamp ................................................................................................................ 168
Transaction trace sampling ............................................................................................................................... 169
Agent Heap Sizing ............................................................................................................................................. 169
Cross-Process Transaction Tracing ........................................................................................................................... 170
Extending transaction trace data collection ............................................................................................................. 170
About User ID data ............................................................................................................................................ 170
About servlet request data ............................................................................................................................... 171
Configuring Agent to collect additional transaction trace data ........................................................................ 171
Configuring Component Stall Reporting .................................................................................................................. 173
Downstream Subscriber Component Stalls ....................................................................................................... 173
Upstream Inherited Component Stalls ............................................................................................................. 174
Disabling the capture of stalls as Events ........................................................................................................... 174
Chapter 11: Configuring the Introscope SQL Agent 175
The SQL Agent overview .......................................................................................................................................... 175
The SQL Agent files ................................................................................................................................................... 176
SQL statement normalization ................................................................................................................................... 176
How poorly written SQL statements create metric explosions......................................................................... 176
SQL statement normalization options .............................................................................................................. 178
Turn Off Statement Metrics ..................................................................................................................................... 184
SQL metrics............................................................................................................................................................... 184
12 Java Agent Implementation Guide
Chapter 12: Enabling JMX Reporting 187
Java Agent JMX Support ........................................................................................................................................... 187
Introscope Support for WebLogic JMX Metrics ................................................................................................ 188
Default JMX Metric Conversion Process .................................................................................................................. 188
Using Primary Key Conversion to Streamline JMX Metrics ...................................................................................... 189
Managing Metric Volume with JMX Filters .............................................................................................................. 190
JMX filters for WebLogic ................................................................................................................................... 191
Configure WebSphere and WebLogic to Use JMX Reporting ................................................................................... 191
Enable JSR-77 Data and View JMX Metrics on WAS ................................................................................................. 192
Chapter 13: Configuring Platform Monitoring 195
Platform Monitors .................................................................................................................................................... 195
Enabling platform monitors on Windows Server 2003 ............................................................................................ 196
Enable Platform Monitors on AIX ............................................................................................................................. 196
Disable Platform Monitors ....................................................................................................................................... 197
Configure Permissions to Access Platform Monitors on HP-UX ............................................................................... 197
Troubleshoot Platform Monitoring .......................................................................................................................... 197
Troubleshooting platform monitoring on Windows ......................................................................................... 198
Chapter 14: Integrating CA APM with CA LISA 199
How to Integrate CA APM with CA LISA ................................................................................................................... 199
Install CA LISA .................................................................................................................................................... 200
Configure CA LISA Tracers ................................................................................................................................. 204
Verify the CA APM and CA LISA Integration ...................................................................................................... 204
Chapter 15: Integrating CA APM Cloud Monitor with CA APM 207
How to Integrate CA APM Cloud Monitor In Your CA APM Deployment ................................................................. 208
Download and Install the CA APM Cloud Monitor Agent ................................................................................. 208
Validate the CA APM Cloud Monitor Agent Connection ................................................................................... 209
How to Limit Data .................................................................................................................................................... 210
Appendix A: Java Agent Properties 213
Configuring the IntroscopeAgent.profile location ................................................................................................... 213
Command-line property overrides ........................................................................................................................... 214
Agent failover ........................................................................................................................................................... 215
introscope.agent.enterprisemanager.connectionorder ................................................................................... 215
introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds ............................................................ 216
Agent HTTP tunneling .............................................................................................................................................. 217
Agent HTTP tunneling for proxy servers ........................................................................................................... 217
Contents 13
Agent HTTPS tunneling...................................................................................................................................... 219
Agent memory overhead ......................................................................................................................................... 220
introscope.agent.reduceAgentMemoryOverhead ............................................................................................ 221
Agent metric aging ................................................................................................................................................... 221
Configuring agent metric aging ......................................................................................................................... 222
Agent metric clamp .................................................................................................................................................. 225
introscope.agent.metricClamp ......................................................................................................................... 225
Agent naming ........................................................................................................................................................... 226
introscope.agent.agentAutoNamingEnabled .................................................................................................... 227
introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds .................................................... 227
introscope.agent.agentAutoRenamingIntervalInMinutes ................................................................................ 228
introscope.agent.agentName ........................................................................................................................... 228
introscope.agent.agentNameSystemPropertyKey ............................................................................................ 229
introscope.agent.disableLogFileAutoNaming ................................................................................................... 229
introscope.agent.clonedAgent .......................................................................................................................... 230
introscope.agent.customProcessName ............................................................................................................ 230
introscope.agent.defaultProcessName ............................................................................................................. 231
introscope.agent.display.hostName.as.fqdn .................................................................................................... 231
Agent recording (business recording) ...................................................................................................................... 231
introscope.agent.bizRecording.enabled ........................................................................................................... 232
Agent thread priority ................................................................................................................................................ 232
introscope.agent.thread.all.priority .................................................................................................................. 233
Agent to Enterprise Manager connection ................................................................................................................ 233
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT ................................................................. 233
introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT ................................................................. 234
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT .................................................. 234
introscope.agent.enterprisemanager.transport.tcp.local.ipaddress.DEFAULT ................................................ 235
introscope.agent.enterprisemanager.transport.tcp.local.port.DEFAULT ......................................................... 235
Application triage map ............................................................................................................................................. 235
introscope.agent.appmap.enabled ................................................................................................................... 236
introscope.agent.appmap.metrics.enabled ...................................................................................................... 236
introscope.agent.appmap.queue.size ............................................................................................................... 237
introscope.agent.appmap.queue.period .......................................................................................................... 237
introscope.agent.appmap.intermediateNodes.enabled................................................................................... 238
Application Triage Map and Catalyst Integration ..................................................................................................... 238
Configure the Ability to Send Information ........................................................................................................ 238
Configure a List of Available Networks ............................................................................................................. 239
Application triage map business transaction POST parameters .............................................................................. 240
introscope.agent.bizdef.matchPost .................................................................................................................. 240
Known limitations ............................................................................................................................................. 241
Application triage map managed socket configuration ........................................................................................... 242
introscope.agent.sockets.managed.reportToAppmap ..................................................................................... 243
14 Java Agent Implementation Guide
introscope.agent.sockets.managed.reportClassAppEdge................................................................................. 243
introscope.agent.sockets.managed.reportMethodAppEdge ............................................................................ 244
introscope.agent.sockets.managed.reportClassBTEdge ................................................................................... 244
introscope.agent.sockets.managed.reportMethodBTEdge .............................................................................. 245
AutoProbe ................................................................................................................................................................ 245
introscope.autoprobe.directivesFile ................................................................................................................. 245
introscope.autoprobe.enable ........................................................................................................................... 246
introscope.autoprobe.logfile ............................................................................................................................ 246
Bootstrap Classes Instrumentation Manager ........................................................................................................... 247
introscope.bootstrapClassesManager.enabled ................................................................................................ 247
introscope.bootstrapClassesManager.waitAtStartup ....................................................................................... 247
CA CEM Agent Profile Properties ............................................................................................................................. 248
introscope.autoprobe.directivesFile ................................................................................................................. 248
introscope.agent.remoteagentconfiguration.allowedFiles .............................................................................. 249
introscope.agent.remoteagentconfiguration.enabled ..................................................................................... 250
introscope.agent.decorator.enabled ................................................................................................................ 250
introscope.agent.decorator.security ................................................................................................................ 251
introscope.agent.cemtracer.domainconfigfile.................................................................................................. 251
introscope.agent.cemtracer.domainconfigfile.reloadfrequencyinminutes ...................................................... 252
introscope.agent.distribution.statistics.components.pattern .......................................................................... 253
Configure the Session ID Collection .................................................................................................................. 253
ChangeDetector configuration ................................................................................................................................. 254
introscope.changeDetector.enable................................................................................................................... 254
introscope.changeDetector.agentID ................................................................................................................. 254
introscope.changeDetector.rootDir .................................................................................................................. 255
introscope.changeDetector.isengardStartupWaitTimeInSec............................................................................ 255
introscope.changeDetector.waitTimeBetweenReconnectInSec....................................................................... 255
introscope.changeDetector.profile ................................................................................................................... 256
introscope.changeDetector.profileDir .............................................................................................................. 256
introscope.changeDetector.compressEntries.enable ....................................................................................... 257
introscope.changeDetector.compressEntries.batchSize .................................................................................. 257
Cross-process tracing in WebLogic Server ............................................................................................................... 257
introscope.agent.weblogic.crossjvm ................................................................................................................. 258
Cross-process transaction trace ............................................................................................................................... 258
introscope.agent.transactiontracer.tailfilterPropagate.enable ........................................................................ 258
Dynamic instrumentation ........................................................................................................................................ 259
introscope.autoprobe.dynamicinstrument.enabled ......................................................................................... 259
autoprobe.dynamicinstrument.pollIntervalMinutes ........................................................................................ 260
introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs .............................................................. 260
introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo .................................................................. 260
introscope.agent.remoteagentdynamicinstrumentation.enabled ................................................................... 261
introscope.autoprobe.dynamicinstrument.pollIntervalMinutes ...................................................................... 261
Contents 15
ErrorDetector ........................................................................................................................................................... 261
introscope.agent.errorsnapshots.enable .......................................................................................................... 262
introscope.agent.errorsnapshots.throttle ........................................................................................................ 262
introscope.agent.errorsnapshots.ignore. ............................................................................................. 262
Extensions ................................................................................................................................................................ 263
introscope.agent.extensions.directory ............................................................................................................. 263
introscope.agent.common.directory ................................................................................................................ 263
GC Monitor ............................................................................................................................................................... 264
introscope.agent.gcmonitor.enable.................................................................................................................. 264
Java NIO .................................................................................................................................................................... 264
Channels ............................................................................................................................................................ 265
NIODatagramTracing metrics ............................................................................................................................ 265
Restricting Java NIO metrics ............................................................................................................................. 266
JMX ........................................................................................................................................................................... 270
introscope.agent.jmx.enable ............................................................................................................................ 270
introscope.agent.jmx.ignore.attributes ............................................................................................................ 271
introscope.agent.jmx.name.filter ..................................................................................................................... 271
introscope.agent.jmx.name.jsr77.disable ......................................................................................................... 272
introscope.agent.jmx.name.primarykeys ......................................................................................................... 273
introscope.agent.jmx.excludeStringMetrics ..................................................................................................... 274
LeakHunter ............................................................................................................................................................... 274
introscope.agent.leakhunter.collectAllocationStackTraces .............................................................................. 275
introscope.agent.leakhunter.enable ................................................................................................................. 276
introscope.agent.leakhunter.leakSensitivity..................................................................................................... 276
introscope.agent.leakhunter.logfile.append .................................................................................................... 277
introscope.agent.leakhunter.logfile.location .................................................................................................... 277
introscope.agent.leakhunter.timeoutInMinutes .............................................................................................. 278
introscope.agent.leakhunter.ignore. ............................................................................................... 278
Logging ..................................................................................................................................................................... 279
log4j.logger.IntroscopeAgent ............................................................................................................................ 280
log4j.appender.logfile.File................................................................................................................................. 281
log4j.logger.IntroscopeAgent.inheritance ........................................................................................................ 281
log4j.appender.pbdlog.File ............................................................................................................................... 282
log4j.appender.pbdlog ...................................................................................................................................... 282
log4j.appender.pbdlog.layout ........................................................................................................................... 283
log4j.appender.pbdlog.layout.ConversionPattern ............................................................................................ 283
log4j.additivity.IntroscopeAgent.inheritance ................................................................................................... 284
Metric count ............................................................................................................................................................. 284
introscope.ext.agent.metric.count ................................................................................................................... 285
Multiple inheritance ................................................................................................................................................. 285
introscope.autoprobe.hierarchysupport.enabled ............................................................................................ 286
introscope.autoprobe.hierarchysupport.runOnceOnly .................................................................................... 286
16 Java Agent Implementation Guide
introscope.autoprobe.hierarchysupport.pollIntervalMinutes .......................................................................... 287
introscope.autoprobe.hierarchysupport.executionCount ................................................................................ 287
introscope.autoprobe.hierarchysupport.disableLogging .................................................................................. 288
introscope.autoprobe.hierarchysupport.disableDirectivesChange .................................................................. 288
Platform monitoring ................................................................................................................................................. 288
introscope.agent.platform.monitor.system ...................................................................................................... 289
Remote configuration .............................................................................................................................................. 289
introscope.agent.remoteagentconfiguration.enabled ..................................................................................... 289
introscope.agent.remoteagentconfiguration.allowedFiles .............................................................................. 290
Security ..................................................................................................................................................................... 290
introscope.agent.decorator.security ................................................................................................................ 290
Servlet header decorator ......................................................................................................................................... 291
introscope.agent.decorator.enabled ................................................................................................................ 291
Socket metrics .......................................................................................................................................................... 291
introscope.agent.sockets.reportRateMetrics ................................................................................................... 292
introscope.agent.io.socket.client.hosts ............................................................................................................ 292
introscope.agent.io.socket.client.ports ............................................................................................................ 293
introscope.agent.io.socket.server.ports ........................................................................................................... 293
SQL Agent ................................................................................................................................................................. 293
introscope.agent.sqlagent.normalizer.extension ............................................................................................. 294
introscope.agent.sqlagent.normalizer.regex.matchFallThrough ...................................................................... 295
introscope.agent.sqlagent.normalizer.regex.keys ............................................................................................ 296
introscope.agent.sqlagent.normalizer.regex.key1.pattern .............................................................................. 296
introscope.agent.sqlagent.normalizer.regex.key1.replaceAll .......................................................................... 297
introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat ................................................................... 297
introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive ..................................................................... 298
introscope.agent.sqlagent.sql.artonly .............................................................................................................. 298
introscope.agent.sqlagent.sql.rawsql ............................................................................................................... 299
introscope.agent.sqlagent.sql.turnoffmetrics .................................................................................................. 299
introscope.agent.sqlagent.sql.turnofftrace ...................................................................................................... 299
SSL communication .................................................................................................................................................. 300
introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT ................................................................. 300
introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT ................................................................. 301
introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT .................................................. 301
introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT ........................................................ 302
introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT ................................................. 302
introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT .......................................................... 302
introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT ................................................... 303
introscope.agent.enterprisemanager.transport.tcp.ciphersuites.DEFAULT ..................................................... 303
Stall metrics .............................................................................................................................................................. 303
introscope.agent.stalls.thresholdseconds ........................................................................................................ 303
introscope.agent.stalls.resolutionseconds ....................................................................................................... 304
Contents 17
Thread dumps .......................................................................................................................................................... 304
introscope.agent.threaddump.enable .............................................................................................................. 305
introscope.agent.threaddump.deadlockpoller.enable ..................................................................................... 305
introscope.agent.threaddump.deadlockpollerinterval ..................................................................................... 306
introscope.agent.threaddump.MaxStackElements .......................................................................................... 306
Transaction tracing ................................................................................................................................................... 307
introscope.agent.bizdef.turnOff.nonIdentifying.txn ......................................................................................... 307
introscope.agent.transactiontracer.parameter.httprequest.headers .............................................................. 308
introscope.agent.transactiontracer.parameter.httprequest.parameters ........................................................ 308
introscope.agent.transactiontracer.parameter.httpsession.attributes ............................................................ 309
introscope.agent.transactiontracer.userid.key ................................................................................................. 309
introscope.agent.transactiontracer.userid.method ......................................................................................... 310
introscope.agent.transactiontrace.componentCountClamp ............................................................................ 311
introscope.agent.crossprocess.compression .................................................................................................... 312
introscope.agent.crossprocess.compression.minlimit...................................................................................... 313
introscope.agent.crossprocess.correlationid.maxlimit ..................................................................................... 314
introscope.agent.transactiontracer.sampling.enabled ..................................................................................... 314
introscope.agent.transactiontracer.sampling.perinterval.count ...................................................................... 315
introscope.agent.transactiontracer.sampling.interval.seconds ....................................................................... 315
introscope.agent.transactiontrace.headFilterClamp ........................................................................................ 316
introscope.agent.ttClamp ................................................................................................................................. 317
URL grouping ............................................................................................................................................................ 317
introscope.agent.urlgroup.keys ........................................................................................................................ 318
introscope.agent.urlgroup.group.default.pathprefix........................................................................................ 318
introscope.agent.urlgroup.group.default.format ............................................................................................. 318
WebSphere PMI ....................................................................................................................................................... 319
introscope.agent.pmi.enable ............................................................................................................................ 320
introscope.agent.pmi.enable.alarmManagerModule ....................................................................................... 320
introscope.agent.pmi.enable.beanModule ...................................................................................................... 321
introscope.agent.pmi.enable.cacheModule ..................................................................................................... 321
introscope.agent.pmi.enable.connectionPoolModule ..................................................................................... 322
introscope.agent.pmi.enable.hamanagerModule ............................................................................................ 322
introscope.agent.pmi.enable.j2cModule .......................................................................................................... 323
introscope.agent.pmi.enable.jvmpiModule...................................................................................................... 323
introscope.agent.pmi.enable.jvmRuntimeModule ........................................................................................... 324
introscope.agent.pmi.enable.objectPoolModule ............................................................................................. 324
introscope.agent.pmi.enable.orbPerfModule .................................................................................................. 325
introscope.agent.pmi.enable.schedulerModule ............................................................................................... 325
introscope.agent.pmi.enable.servletSessionsModule ...................................................................................... 326
introscope.agent.pmi.enable.systemModule ................................................................................................... 326
introscope.agent.pmi.enable.threadPoolModule ............................................................................................. 327
introscope.agent.pmi.enable.transactionModule ............................................................................................ 327
18 Java Agent Implementation Guide
introscope.agent.pmi.enable.webAppModule ................................................................................................. 328
introscope.agent.pmi.enable.webServicesModule .......................................................................................... 328
introscope.agent.pmi.enable.wlmModule........................................................................................................ 329
introscope.agent.pmi.enable.wsgwModule ..................................................................................................... 329
introscope.agent.pmi.filter.objrefModule ........................................................................................................ 330
WLDF metrics ........................................................................................................................................................... 330
introscope.agent.wldf.enable ........................................................................................................................... 330
Appendix B: Alternative Methods for Instrumentation 331
Java Agent Deployment on Other Application Servers ............................................................................................ 331
Configure Sun ONE to Use AutoProbe ..................................................................................................................... 332
Configure Oracle to Use AutoProbe ......................................................................................................................... 333
Configure WebLogic Server ...................................................................................................................................... 334
Configuring HTTP servlet tracing .............................................................................................................................. 335
Create an AutoProbe Connector File ....................................................................................................................... 335
Running the AutoProbe Connector for a JVM ................................................................................................... 336
Example: Use Xbootclasspath to Instrument WAS ........................................................................................... 338
About running ProbeBuilder manually ..................................................................................................................... 339
Configure AutoProbe for WebSphere for z/OS ........................................................................................................ 339
Appendix C: Using the PBD Generator 343
About the CA PBD Generator ................................................................................................................................... 343
Configuring the PBD Generator ................................................................................................................................ 344
Required PBD Generator parameters ............................................................................................................... 344
Using the PBD Generator ......................................................................................................................................... 344
Appendix D: Using the Network Interface Utility 347
Determine Network Interface Names ...................................................................................................................... 347
Index 349
Chapter 1: Introduction to the Java Agent 19
Chapter 1: Introduction to the Java Agent
This section contains the following topics:
About Introscope and Java Agents (see page 19) Planning a Java Agent Deployment (see page 20) Deploying the Java Agent (see page 22)
About Introscope and Java Agents
CA Introscope is an enterprise application performance management solution that enables you to monitor complex web applications in production environments 24x7, detect problems before they affect your customers, and resolve these issues quickly and collaboratively. A key part of the architecture for this solution is the low overhead agent.
The agent is a data gathering component of Introscope that collects detailed performance information about applications and the computing environment as transactions are executed. The Java agent collects this information from applications and resources running on Java Virtual Machines (JVMs) and sends the information to the Enterprise Manager for further processing.
The Java agent inserts probes into the bytecode of the components that make up the JVM that the application uses. Inserting the probes into the bytecode is part of the instrumentation process that enables the monitoring of your applications.
Instrumenting an application also requires tracers that are defined in ProbeBuilder Directive (PBD) files. The instructions or directives in PBD files identify the application components to monitor. The tracers identify the metrics the agent should collect from which probes as the application runs in the JVM. You can control what is monitored by changing the PBD files to suit your environment.
When you install the Java agent, several default PBD files are deployed to enable default monitoring of your environment. You can modify the default monitoring to achieve the balance of visibility and performance that you require. After the application is instrumented, the Java agent collects the data you are interested in and reports the data to the Enterprise Manager. The Enterprise Manager then processes and stores the data for real-time and historical reporting. You can then view and work with the collected data using the Introscope Workstation to create alerts or take responsive action.
Planning a Java Agent Deployment
20 Java Agent Implementation Guide
For application management, the key activities are:
Deploy agents to monitor the performance and availability of application servers.
Test, adjust, and optimize monitoring of application components.
Customize the agent profile to control agent operations as needed.
Create application- or agent-specific metric groupings, dashboards, alerts, and actions
Investigate, triage, and diagnose application issues.
Planning a Java Agent Deployment
When planning a deployment, your goal is to develop the right balance between agent overhead and the visibility of application performance. Low overhead agents allow for real-time monitoring of all transactions in production environments. Keeping overhead low helps performance and availability of critical applications and server resources. However, keeping overhead low does not provide enough information to diagnose problems when they occur. Therefore, it is common to deploy and adjust the agent configuration throughout the lifecycle of an application. In addition, monitoring more components when an application is being developed or tested, then reducing the components after an application is released to production.
Install and Evaluate the Default Functionality
The first step in deploying agents involves installing and evaluating the default agent configuration. The default agent configuration demonstrates data collection for many common components of the application and the computing environment. The default agent configuration also includes several features enabled and other, less frequently used features, disabled.
Your goal is to evaluate the depth and breadth of data collection provided by default and become familiar with Introscope and how to monitor applications. After you are familiar with the performance metrics the agent provides by default, you can customize the agent to collect data, as needed.
As you evaluate the default performance, keep in mind that when the agent collects more metrics, it consumes more system resources. If the agent collects fewer metrics, you have less visibility into potential problems. As you refine the agent configuration, try to strike a balance between the depth of data collection and an acceptable level of performance. The appropriate level of instrumentation typically depends on where the agent is deployed. For example, an agent monitoring within a test environment is typically configured to collect large numbers of metrics. An agent on a production server, however, is typically configured to deliver essential information.
Planning a Java Agent Deployment
Chapter 1: Introduction to the Java Agent 21
Determine Configuration Requirements
Before you deploy the agent, determine your data collection requirements. This information lets you tailor the data collection behaviors of the agent, and evaluate the impact on overhead through alternative configurations of the agent.
Introscope can be used throughout the lifecycle of an application. For example, from development through testing, load verification, staging, and into production. At each stage in the lifecycle, the monitoring goals, environmental constraints, and service level expectations are likely to be different. To address these differences, you configure the agent to behave differently for the type of environment that is monitored.
Your goal is to determine the appropriate trade-off between the visibility of performance details and resource overhead. Also consider the optimum level of visibility at a reasonable cost for the environment being monitored.
In preproduction application environments, such as development, you typically configure a higher level of data collection to provide deeper visibility into the application performance. In production or high-volume transaction environments, you typically reduce the metrics reported to control agent overhead. Depending on your requirements, you can also configure agent properties to control specific agent behavior. For example, track the maximum number of metrics collected and the removal of old metrics.
For your environment, determine the appropriate level of visibility and acceptable performance overhead, so you can configure the agent to match your requirements.
Define a Baseline Agent Profile with Appropriate Configuration Properties
After you identify the type of application environment to monitor, you can create a "candidate" agent configuration. Most agent operations are controlled using properties in the agent profile (IntroscopeAgent.profile). For example, the IntroscopeAgent.profile file defines ProbeBuilder Directive and ProbeBuilder List files the agent uses. The files listed in the agent profile then control the specific metrics that the agent gathers. The IntroscopeAgent.profile file also provides properties that enable or disable specific features or tune operations such as polling intervals.
Depending on your configuration and environment, you can adjust the properties in the agent profile to evaluate the impact of each change. For example, you can start with a default agent profile that does not have ChangeDetector enabled. Later, you can enable ChangeDetector properties in the profile and evaluate agent performance after the change before adding any other features.
Deploying the Java Agent
22 Java Agent Implementation Guide
Evaluate Agent Performance Overhead
When evaluating an agent configuration, verify that the metrics collected provide sufficient visibility into application performance and availability. In addition, verify that the volume of metrics cannot impose an unacceptable load on the operating environment. The agent cannot report more metrics than are necessary to identify and localize performance and availability problems.
You can effectively evaluate agent performance by understanding the performance characteristics of the application. For example, you can load test your application before and after implementing default monitoring to verify impact.
To extend data collection in a controlled way, verify agent operation and overhead before and after implementing changes. For example, only add monitoring support for one application at a time, so you can evaluate the performance of each add-on individually.
Validate and Deploy the Agent Configuration
After verifying that the candidate agent configuration provides the visibility required, deploy the configuration across that environment.
You can deploy a validated configuration by installing the following configuration items to the target environment:
IntroscopeAgent.profile file
modified or custom PBD files
Deploying the Java Agent
Deploying the agent involves the following high level steps:
1. Install the agent on the target computer.
2. Configure the application server (see page 35) to instrument your applications.
3. Configure the startup script (see page 36) or Java command for the application server to include the agent and the location of the agent profile.
4. Configure the agent connection (see page 59) to the Enterprise Manager.
5. Restart the application server to instrument the application and start data collection.
6. Configure the IntroscopeAgent.profile (see page 67) if you want to change any properties set during installation or want to modify data collection or other agent operations.
Chapter 2: Installing and Configuring the Java Agent 23
Chapter 2: Installing and Configuring the Java Agent
This section contains the following topics:
Before You Install the Agent (see page 23) Select the method for installing the Java agent (see page 24) About the Java Agent Directory Structure (see page 30) Configuring Detection of Synthetic Transactions (see page 32) Java 7 Autoprobe (see page 35) How to instrument applications (see page 35) Configuring the Application Server to Start the Java Agent (see page 36) Configuring the connection to the Enterprise Manager (see page 59) Upgrade Multiple Agent Types (see page 63) Uninstalling the Java agent (see page 64)
Before You Install the Agent
Before you install the agent, do the following steps:
1. Review the Java agent deployment process.
2. Verify that you have a supported version of the application server.
Note: The server where you plan to install the agent must have a supported version of the JVM available locally. For application server and JVM requirements, see the Compatibility Guide.
3. Verify that you have a supported version of the JVM. If you are unable to use a supported version, use a previous version:
Use the 8.x Java agent for Java 1.4.x.
Use the 7.2 Java agent for Java 1.3.x.
4. Verify that the Introscope Enterprise Manager and Workstation components are installed. Note the host name and port number for the Enterpris
Recommended