APM_9.5--Java Agent Implementation Guide

Java Agent Implementation Guide Release 9.5

CA Application Performance Management

This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to as the Documentation) is for your informational purposes only and is subject to change or withdrawal by CA at any time.

This Documentation may not be copied, transferred, reproduced, disclosed, modified or duplicated, in whole or in part, without the prior written consent of CA. This Documentation is confidential and proprietary information of CA and may not be disclosed by you or used for any purpose other than as may be permitted in (i) a separate agreement between you and CA governing your use of the CA software to which the Documentation relates; or (ii) a separate confidentiality agreement between you and CA.

Notwithstanding the foregoing, if you are a licensed user of the software product(s) addressed in the Documentation, you may print or otherwise make available a reasonable number of copies of the Documentation for internal use by you and your employees in connection with that software, provided that all CA copyright notices and legends are affixed to each reproduced copy.

The right to print or otherwise make available copies of the Documentation is limited to the period during which the applicable license for such software remains in full force and effect. Should the license terminate for any reason, it is your responsibility to certify in writing to CA that all copies and partial copies of the Documentation have been returned to CA or destroyed.

TO THE EXTENT PERMITTED BY APPLICABLE LAW, CA PROVIDES THIS DOCUMENTATION AS IS WITHOUT WARRANTY OF ANY KIND, INCLUDING WITHOUT LIMITATION, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NONINFRINGEMENT. IN NO EVENT WILL CA BE LIABLE TO YOU OR ANY THIRD PARTY FOR ANY LOSS OR DAMAGE, DIRECT OR INDIRECT, FROM THE USE OF THIS DOCUMENTATION, INCLUDING WITHOUT LIMITATION, LOST PROFITS, LOST INVESTMENT, BUSINESS INTERRUPTION, GOODWILL, OR LOST DATA, EVEN IF CA IS EXPRESSLY ADVISED IN ADVANCE OF THE POSSIBILITY OF SUCH LOSS OR DAMAGE.

The use of any software product referenced in the Documentation is governed by the applicable license agreement and such license agreement is not modified in any way by the terms of this notice.

The manufacturer of this Documentation is CA.

Provided with Restricted Rights. Use, duplication or disclosure by the United States Government is subject to the restrictions set forth in FAR Sections 12.212, 52.227-14, and 52.227-19(c)(1) - (2) and DFARS Section 252.227-7014(b)(3), as applicable, or their successors.

Copyright 2013 CA. All rights reserved. All trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.

CA Technologies Product References

This document references the following CA Technologies products and features:

CA Application Performance Management (CA APM)

CA Application Performance Management ChangeDetector (CA APM ChangeDetector)

CA Application Performance Management ErrorDetector (CA APM ErrorDetector)

CA Application Performance Management for CA Database Performance (CA APM for CA Database Performance)

CA Application Performance Management for CA SiteMinder (CA APM for CA SiteMinder)

CA Application Performance Management for CA SiteMinder Application Server Agents (CA APM for CA SiteMinder ASA)

CA Application Performance Management for IBM CICS Transaction Gateway (CA APM for IBM CICS Transaction Gateway)

CA Application Performance Management for IBM WebSphere Application Server (CA APM for IBM WebSphere Application Server)

CA Application Performance Management for IBM WebSphere Distributed Environments (CA APM for IBM WebSphere Distributed Environments)

CA Application Performance Management for IBM WebSphere MQ (CA APM for IBM WebSphere MQ)

CA Application Performance Management for IBM WebSphere Portal (CA APM for IBM WebSphere Portal)

CA Application Performance Management for IBM WebSphere Process Server (CA APM for IBM WebSphere Process Server)

CA Application Performance Management for IBM z/OS (CA APM for IBM z/OS)

CA Application Performance Management for Microsoft SharePoint (CA APM for Microsoft SharePoint)

CA Application Performance Management for Oracle Databases (CA APM for Oracle Databases)

CA Application Performance Management for Oracle Service Bus (CA APM for Oracle Service Bus)

CA Application Performance Management for Oracle WebLogic Portal (CA APM for Oracle WebLogic Portal)

CA Application Performance Management for Oracle WebLogic Server (CA APM for Oracle WebLogic Server)

CA Application Performance Management for SOA (CA APM for SOA)

CA Application Performance Management for TIBCO BusinessWorks (CA APM for TIBCO BusinessWorks)

CA Application Performance Management for TIBCO Enterprise Message Service (CA APM for TIBCO Enterprise Message Service)

CA Application Performance Management for Web Servers (CA APM for Web Servers)

CA Application Performance Management for webMethods Broker (CA APM for webMethods Broker)

CA Application Performance Management for webMethods Integration Server (CA APM for webMethods Integration Server)

CA Application Performance Management Integration for CA CMDB (CA APM Integration for CA CMDB)

CA Application Performance Management Integration for CA NSM (CA APM Integration for CA NSM)

CA Application Performance Management LeakHunter (CA APM LeakHunter)

CA Application Performance Management Transaction Generator (CA APM TG)

CA Cross-Enterprise Application Performance Management

CA Customer Experience Manager (CA CEM)

CA Embedded Entitlements Manager (CA EEM)

CA eHealth Performance Manager (CA eHealth)

CA Insight Database Performance Monitor for DB2 for z/OS

CA Introscope

CA SiteMinder

CA Spectrum Infrastructure Manager (CA Spectrum)

CA SYSVIEW Performance Management (CA SYSVIEW)

Contact CA Technologies

Contact CA Support

For your convenience, CA Technologies provides one site where you can access the information that you need for your Home Office, Small Business, and Enterprise CA Technologies products. At http://ca.com/support, you can access the following resources:

Online and telephone contact information for technical assistance and customer services

Information about user communities and forums

Product and documentation downloads

CA Support policies and guidelines

Other helpful resources appropriate for your product

Providing Feedback About Product Documentation

If you have comments or questions about CA Technologies product documentation, you can send a message to [email protected].

To provide feedback about CA Technologies product documentation, complete our short customer survey which is available on the CA Support website at http://ca.com/docs.

Contents 7

Contents

Chapter 1: Introduction to the Java Agent 19

About Introscope and Java Agents ............................................................................................................................. 19

Planning a Java Agent Deployment ............................................................................................................................ 20

Install and Evaluate the Default Functionality .................................................................................................... 20

Determine Configuration Requirements ............................................................................................................. 21

Define a Baseline Agent Profile with Appropriate Configuration Properties ...................................................... 21

Evaluate Agent Performance Overhead .............................................................................................................. 22

Validate and Deploy the Agent Configuration .................................................................................................... 22

Deploying the Java Agent ........................................................................................................................................... 22

Chapter 2: Installing and Configuring the Java Agent 23

Before You Install the Agent ...................................................................................................................................... 23

Select the method for installing the Java agent ......................................................................................................... 24

Install the Java Agent Interactively ..................................................................................................................... 24

Install the Java Agent Silently .............................................................................................................................. 26

Install Manually Using Installation Archives........................................................................................................ 30

About the Java Agent Directory Structure ................................................................................................................. 30

Configuring Detection of Synthetic Transactions ....................................................................................................... 32

Using the TagScript Utility ................................................................................................................................... 34

Java 7 Autoprobe ....................................................................................................................................................... 35

How to instrument applications ................................................................................................................................. 35

Configuring the Application Server to Start the Java Agent ....................................................................................... 36

Configure Apache Tomcat to use the Java agent ................................................................................................ 36

Configure JBoss to Use the Java Agent ............................................................................................................... 37

Configure Oracle WebLogic to Use the Java Agent ............................................................................................. 40

Configure WebLogic with JRockit JVM to Use the Java Agent ............................................................................ 45

Configure WebLogic with JRockit JVM to View Socket Metrics .......................................................................... 45

Configure IBM WebSphere to Use the Java Agent .............................................................................................. 46

Configure Oracle Application Server to use the Java agent ................................................................................ 57

Configure GlassFish to Use the Java Agent ......................................................................................................... 57

Configure SAP Netweaver to Use the Java Agent ............................................................................................... 58

Configuring the connection to the Enterprise Manager ............................................................................................ 59

Connect to the Enterprise Manager Using a Direct Socket Connection ............................................................. 59

Connect to the Enterprise Manager with HTTP tunneling .................................................................................. 60

Configure a proxy server for HTTP tunneling ...................................................................................................... 61

Connect to the Enterprise Manager with HTTPS tunneling ................................................................................ 61

8 Java Agent Implementation Guide

Connect to the Enterprise Manager over SSL ..................................................................................................... 62

Configure Agent Load Balancing ......................................................................................................................... 63

Upgrade Multiple Agent Types................................................................................................................................... 63

Uninstalling the Java agent ........................................................................................................................................ 64

Uninstalling the Java agent from z/OS ................................................................................................................ 65

Chapter 3: Configuring Agent Properties 67

How to modify communication with Enterprise Manager ......................................................................................... 67

How to configure backup Enterprise Managers and failover properties ................................................................... 67

How to enable and use additional GC metrics ........................................................................................................... 69

How to enable and configure thread dumps ............................................................................................................. 69

How to Configure an Agent to Collect Distribution Statistic Metrics ......................................................................... 72

Examples of Distribution Statistics Metrics ......................................................................................................... 73

Chapter 4: AutoProbe and ProbeBuilding Options 75

AutoProbe and ProbeBuilding Overview.................................................................................................................... 75

Unsupported Instrumentation Method .............................................................................................................. 76

Configuring ProbeBuilding ......................................................................................................................................... 76

Full or typical tracing options .............................................................................................................................. 76

Dynamic ProbeBuilding ....................................................................................................................................... 77

Configure Dynamic ProbeBuilding ...................................................................................................................... 79

Dynamic Instrumentation Impacts Performance for IBM JDK ............................................................................ 80

Dynamic ProbeBuilding Versus Dynamic Instrumentation ................................................................................. 80

ProbeBuilding Class Hierarchies .......................................................................................................................... 81

Removing line numbers in bytecode ................................................................................................................... 84

Chapter 5: ProbeBuilder Directives 85

ProbeBuilder Directives Overview ............................................................................................................................. 85

Components traced by the default PBDs ............................................................................................................ 86

Default PBD Files ................................................................................................................................................. 87

Default PBD Files From Previous Releases .......................................................................................................... 89

Default PBL files .................................................................................................................................................. 90

Default Tracer Groups and Toggles Files ............................................................................................................. 90

Turning tracer groups on or off ........................................................................................................................... 99

Adding classes to a tracer group ....................................................................................................................... 100

EJB Naming ........................................................................................................................................................ 102

Using the IntroscopeAgent.profile, PBLs, and PBDs together .................................................................................. 103

Applying ProbeBuilder Directives ............................................................................................................................. 103

Using JVM AutoProbe ....................................................................................................................................... 103

Using the ProbeBuilder Wizard or command-line ProbeBuilder ...................................................................... 104

Contents 9

Instrumenting with new and changed PBDs ..................................................................................................... 104

Creating custom tracers ........................................................................................................................................... 106

Using a custom BlamePointTracer tracer for common metrics ........................................................................ 106

Directive names and arguments used in tracer syntax ..................................................................................... 107

Commonly used tracer names and examples ................................................................................................... 109

Advanced single-metric tracers......................................................................................................................... 111

Skip directives ................................................................................................................................................... 114

Counting object instances ................................................................................................................................. 114

Turning on InstrumentPoint directives ............................................................................................................. 115

Combining custom tracers ................................................................................................................................ 115

Instrumenting and Inheritance ......................................................................................................................... 115

Java Annotations ............................................................................................................................................... 116

Using Blame Tracers to mark blame points.............................................................................................................. 117

Blame Tracers .................................................................................................................................................... 117

High agent CPU overhead from deep nested frontend transactions ................................................................ 118

Custom FrontendMarker Directive ................................................................................................................... 118

Blame Tracers in standard PBDs ....................................................................................................................... 119

Boundary Blame and Oracle Backends ............................................................................................................. 119

Chapter 6: Java Agent Naming 121

Understanding the Java Agent name ....................................................................................................................... 121

How the Agent Determines its Name ............................................................................................................... 122

How Introscope resolves agent naming conflicts ............................................................................................. 123

Agent naming considerations for clustered applications ......................................................................................... 124

Specifying an agent name using a Java system property ......................................................................................... 124

Specifying an agent name using a system property key .......................................................................................... 125

Obtaining an agent name from the application server ............................................................................................ 125

Application servers that support agent naming ................................................................................................ 125

Automatic agent naming .......................................................................................................................................... 126

Automatic agent naming and renamed agents ................................................................................................. 127

Advanced automatic agent naming options ..................................................................................................... 127

Enabling cloned agent naming in clustered environments ...................................................................................... 129

Cloned agent naming scenario .......................................................................................................................... 129

Configuring unique names for application instances ........................................................................................ 129

Application triage map and the agent name ............................................................................................................ 130

Chapter 7: Java Agent Monitoring and Logging 131

Configuring agent connection metrics ..................................................................................................................... 131

Socket metrics .......................................................................................................................................................... 132

Restricting socket and SSL metric collection ..................................................................................................... 132

Fine-tuning socket and SSL metric collection .................................................................................................... 133


SSL, NIO, and Socket Tracing in the Application Triage Map ............................................................................ 133

Change the Component Name in the Application Triage Map ......................................................................... 134

Turning off socket and SSL metric collection .................................................................................................... 135

Backwards Compatibility ................................................................................................................................... 135

Configuring logging options ..................................................................................................................................... 136

Running the agent in verbose mode ................................................................................................................. 136

Redirecting agent output to a file ..................................................................................................................... 137

Changing the name or location of the agent logfile .......................................................................................... 138

Agent log files and automatic agent naming .................................................................................................... 138

Rolling up logs by date or size ........................................................................................................................... 139

Managing ProbeBuilder Logs ................................................................................................................................... 140

Command-line ProbeBuilder and ProbeBuilder Wizard log name and location ............................................... 140

AutoProbe log name and location .................................................................................................................... 140

Chapter 8: Configuring LeakHunter and ErrorDetector 141

LeakHunter ............................................................................................................................................................... 141

How LeakHunter works ..................................................................................................................................... 142

What LeakHunter tracks in Java ........................................................................................................................ 142

What LeakHunter does not track ...................................................................................................................... 143

System and version requirements .................................................................................................................... 143

Enabling and disabling LeakHunter .......................................................................................................................... 144

Configuring LeakHunter properties .......................................................................................................................... 144

Ignoring collections that cause poor performance ........................................................................................... 146

Running LeakHunter ................................................................................................................................................. 147

Identifying potential leaks with collection IDs ......................................................................................................... 147

LeakHunter log file ................................................................................................................................................... 148

Potential leak first identified log entry ............................................................................................................. 148

Identified potential leak stops leaking log entry ............................................................................................... 149

Identified potential leak is leaking again log entry ........................................................................................... 150

LeakHunter timeout log entry ........................................................................................................................... 150

Using LeakHunter ..................................................................................................................................................... 150

ErrorDetector ........................................................................................................................................................... 151

Types of errors .................................................................................................................................................. 151

How ErrorDetector works ................................................................................................................................. 152

Enabling ErrorDetector in the Java Agent ................................................................................................................ 152

Configuring ErrorDetector options ........................................................................................................................... 153

Advanced error data capture ................................................................................................................................... 154

Defining new error types.......................................................................................................................................... 154

ExceptionErrorReporter .................................................................................................................................... 154

MethodCalledErrorReporter ............................................................................................................................. 155

ThisErrorReporter ............................................................................................................................................. 155

Contents 11

HTTPErrorCodeReporter ................................................................................................................................... 156

Using error tracer directives with caution ........................................................................................................ 156

Using ErrorDetector ................................................................................................................................................. 156

Chapter 9: Configuring Boundary Blame 157

Understanding Boundary Blame .............................................................................................................................. 157

Using URL groups ..................................................................................................................................................... 157

Defining keys for URL groups ............................................................................................................................ 158

Defining membership of each URL group ......................................................................................................... 158

Define the name for a URL group ..................................................................................................................... 159

Advanced naming techniques for URL groups (optional) ................................................................................. 159

Running the URLGrouper .................................................................................................................................. 163

Using Blame tracers ................................................................................................................................................. 163

Chapter 10: Configuring Transaction Trace Options 165

New Mode of Transaction Tracing ........................................................................................................................... 165

Configuring the Agent to Use Legacy Mode Transaction Tracing ..................................................................... 166

Controlling automatic Transaction Tracing behavior ............................................................................................... 168

Transaction Trace component clamp ................................................................................................................ 168

Transaction trace sampling ............................................................................................................................... 169

Agent Heap Sizing ............................................................................................................................................. 169

Cross-Process Transaction Tracing ........................................................................................................................... 170

Extending transaction trace data collection ............................................................................................................. 170

About User ID data ............................................................................................................................................ 170

About servlet request data ............................................................................................................................... 171

Configuring Agent to collect additional transaction trace data ........................................................................ 171

Configuring Component Stall Reporting .................................................................................................................. 173

Downstream Subscriber Component Stalls ....................................................................................................... 173

Upstream Inherited Component Stalls ............................................................................................................. 174

Disabling the capture of stalls as Events ........................................................................................................... 174

Chapter 11: Configuring the Introscope SQL Agent 175

The SQL Agent overview .......................................................................................................................................... 175

The SQL Agent files ................................................................................................................................................... 176

SQL statement normalization ................................................................................................................................... 176

How poorly written SQL statements create metric explosions......................................................................... 176

SQL statement normalization options .............................................................................................................. 178

Turn Off Statement Metrics ..................................................................................................................................... 184

SQL metrics............................................................................................................................................................... 184


Chapter 12: Enabling JMX Reporting 187

Java Agent JMX Support ........................................................................................................................................... 187

Introscope Support for WebLogic JMX Metrics ................................................................................................ 188

Default JMX Metric Conversion Process .................................................................................................................. 188

Using Primary Key Conversion to Streamline JMX Metrics ...................................................................................... 189

Managing Metric Volume with JMX Filters .............................................................................................................. 190

JMX filters for WebLogic ................................................................................................................................... 191

Configure WebSphere and WebLogic to Use JMX Reporting ................................................................................... 191

Enable JSR-77 Data and View JMX Metrics on WAS ................................................................................................. 192

Chapter 13: Configuring Platform Monitoring 195

Platform Monitors .................................................................................................................................................... 195

Enabling platform monitors on Windows Server 2003 ............................................................................................ 196

Enable Platform Monitors on AIX ............................................................................................................................. 196

Disable Platform Monitors ....................................................................................................................................... 197

Configure Permissions to Access Platform Monitors on HP-UX ............................................................................... 197

Troubleshoot Platform Monitoring .......................................................................................................................... 197

Troubleshooting platform monitoring on Windows ......................................................................................... 198

Chapter 14: Integrating CA APM with CA LISA 199

How to Integrate CA APM with CA LISA ................................................................................................................... 199

Install CA LISA .................................................................................................................................................... 200

Configure CA LISA Tracers ................................................................................................................................. 204

Verify the CA APM and CA LISA Integration ...................................................................................................... 204

Chapter 15: Integrating CA APM Cloud Monitor with CA APM 207

How to Integrate CA APM Cloud Monitor In Your CA APM Deployment ................................................................. 208

Download and Install the CA APM Cloud Monitor Agent ................................................................................. 208

Validate the CA APM Cloud Monitor Agent Connection ................................................................................... 209

How to Limit Data .................................................................................................................................................... 210

Appendix A: Java Agent Properties 213

Configuring the IntroscopeAgent.profile location ................................................................................................... 213

Command-line property overrides ........................................................................................................................... 214

Agent failover ........................................................................................................................................................... 215

introscope.agent.enterprisemanager.connectionorder ................................................................................... 215

introscope.agent.enterprisemanager.failbackRetryIntervalInSeconds ............................................................ 216

Agent HTTP tunneling .............................................................................................................................................. 217

Agent HTTP tunneling for proxy servers ........................................................................................................... 217

Contents 13

Agent HTTPS tunneling...................................................................................................................................... 219

Agent memory overhead ......................................................................................................................................... 220

introscope.agent.reduceAgentMemoryOverhead ............................................................................................ 221

Agent metric aging ................................................................................................................................................... 221

Configuring agent metric aging ......................................................................................................................... 222

Agent metric clamp .................................................................................................................................................. 225

introscope.agent.metricClamp ......................................................................................................................... 225

Agent naming ........................................................................................................................................................... 226

introscope.agent.agentAutoNamingEnabled .................................................................................................... 227

introscope.agent.agentAutoNamingMaximumConnectionDelayInSeconds .................................................... 227

introscope.agent.agentAutoRenamingIntervalInMinutes ................................................................................ 228

introscope.agent.agentName ........................................................................................................................... 228

introscope.agent.agentNameSystemPropertyKey ............................................................................................ 229

introscope.agent.disableLogFileAutoNaming ................................................................................................... 229

introscope.agent.clonedAgent .......................................................................................................................... 230

introscope.agent.customProcessName ............................................................................................................ 230

introscope.agent.defaultProcessName ............................................................................................................. 231

introscope.agent.display.hostName.as.fqdn .................................................................................................... 231

Agent recording (business recording) ...................................................................................................................... 231

introscope.agent.bizRecording.enabled ........................................................................................................... 232

Agent thread priority ................................................................................................................................................ 232

introscope.agent.thread.all.priority .................................................................................................................. 233

Agent to Enterprise Manager connection ................................................................................................................ 233

introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT ................................................................. 233

introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT ................................................................. 234

introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT .................................................. 234

introscope.agent.enterprisemanager.transport.tcp.local.ipaddress.DEFAULT ................................................ 235

introscope.agent.enterprisemanager.transport.tcp.local.port.DEFAULT ......................................................... 235

Application triage map ............................................................................................................................................. 235

introscope.agent.appmap.enabled ................................................................................................................... 236

introscope.agent.appmap.metrics.enabled ...................................................................................................... 236

introscope.agent.appmap.queue.size ............................................................................................................... 237

introscope.agent.appmap.queue.period .......................................................................................................... 237

introscope.agent.appmap.intermediateNodes.enabled................................................................................... 238

Application Triage Map and Catalyst Integration ..................................................................................................... 238

Configure the Ability to Send Information ........................................................................................................ 238

Configure a List of Available Networks ............................................................................................................. 239

Application triage map business transaction POST parameters .............................................................................. 240

introscope.agent.bizdef.matchPost .................................................................................................................. 240

Known limitations ............................................................................................................................................. 241

Application triage map managed socket configuration ........................................................................................... 242

introscope.agent.sockets.managed.reportToAppmap ..................................................................................... 243


introscope.agent.sockets.managed.reportClassAppEdge................................................................................. 243

introscope.agent.sockets.managed.reportMethodAppEdge ............................................................................ 244

introscope.agent.sockets.managed.reportClassBTEdge ................................................................................... 244

introscope.agent.sockets.managed.reportMethodBTEdge .............................................................................. 245

AutoProbe ................................................................................................................................................................ 245

introscope.autoprobe.directivesFile ................................................................................................................. 245

introscope.autoprobe.enable ........................................................................................................................... 246

introscope.autoprobe.logfile ............................................................................................................................ 246

Bootstrap Classes Instrumentation Manager ........................................................................................................... 247

introscope.bootstrapClassesManager.enabled ................................................................................................ 247

introscope.bootstrapClassesManager.waitAtStartup ....................................................................................... 247

CA CEM Agent Profile Properties ............................................................................................................................. 248

introscope.autoprobe.directivesFile ................................................................................................................. 248

introscope.agent.remoteagentconfiguration.allowedFiles .............................................................................. 249

introscope.agent.remoteagentconfiguration.enabled ..................................................................................... 250

introscope.agent.decorator.enabled ................................................................................................................ 250

introscope.agent.decorator.security ................................................................................................................ 251

introscope.agent.cemtracer.domainconfigfile.................................................................................................. 251

introscope.agent.cemtracer.domainconfigfile.reloadfrequencyinminutes ...................................................... 252

introscope.agent.distribution.statistics.components.pattern .......................................................................... 253

Configure the Session ID Collection .................................................................................................................. 253

ChangeDetector configuration ................................................................................................................................. 254

introscope.changeDetector.enable................................................................................................................... 254

introscope.changeDetector.agentID ................................................................................................................. 254

introscope.changeDetector.rootDir .................................................................................................................. 255

introscope.changeDetector.isengardStartupWaitTimeInSec............................................................................ 255

introscope.changeDetector.waitTimeBetweenReconnectInSec....................................................................... 255

introscope.changeDetector.profile ................................................................................................................... 256

introscope.changeDetector.profileDir .............................................................................................................. 256

introscope.changeDetector.compressEntries.enable ....................................................................................... 257

introscope.changeDetector.compressEntries.batchSize .................................................................................. 257

Cross-process tracing in WebLogic Server ............................................................................................................... 257

introscope.agent.weblogic.crossjvm ................................................................................................................. 258

Cross-process transaction trace ............................................................................................................................... 258

introscope.agent.transactiontracer.tailfilterPropagate.enable ........................................................................ 258

Dynamic instrumentation ........................................................................................................................................ 259

introscope.autoprobe.dynamicinstrument.enabled ......................................................................................... 259

autoprobe.dynamicinstrument.pollIntervalMinutes ........................................................................................ 260

introscope.autoprobe.dynamicinstrument.classFileSizeLimitInMegs .............................................................. 260

introscope.autoprobe.dynamic.limitRedefinedClassesPerBatchTo .................................................................. 260

introscope.agent.remoteagentdynamicinstrumentation.enabled ................................................................... 261

introscope.autoprobe.dynamicinstrument.pollIntervalMinutes ...................................................................... 261

Contents 15

ErrorDetector ........................................................................................................................................................... 261

introscope.agent.errorsnapshots.enable .......................................................................................................... 262

introscope.agent.errorsnapshots.throttle ........................................................................................................ 262

introscope.agent.errorsnapshots.ignore. ............................................................................................. 262

Extensions ................................................................................................................................................................ 263

introscope.agent.extensions.directory ............................................................................................................. 263

introscope.agent.common.directory ................................................................................................................ 263

GC Monitor ............................................................................................................................................................... 264

introscope.agent.gcmonitor.enable.................................................................................................................. 264

Java NIO .................................................................................................................................................................... 264

Channels ............................................................................................................................................................ 265

NIODatagramTracing metrics ............................................................................................................................ 265

Restricting Java NIO metrics ............................................................................................................................. 266

JMX ........................................................................................................................................................................... 270

introscope.agent.jmx.enable ............................................................................................................................ 270

introscope.agent.jmx.ignore.attributes ............................................................................................................ 271

introscope.agent.jmx.name.filter ..................................................................................................................... 271

introscope.agent.jmx.name.jsr77.disable ......................................................................................................... 272

introscope.agent.jmx.name.primarykeys ......................................................................................................... 273

introscope.agent.jmx.excludeStringMetrics ..................................................................................................... 274

LeakHunter ............................................................................................................................................................... 274

introscope.agent.leakhunter.collectAllocationStackTraces .............................................................................. 275

introscope.agent.leakhunter.enable ................................................................................................................. 276

introscope.agent.leakhunter.leakSensitivity..................................................................................................... 276

introscope.agent.leakhunter.logfile.append .................................................................................................... 277

introscope.agent.leakhunter.logfile.location .................................................................................................... 277

introscope.agent.leakhunter.timeoutInMinutes .............................................................................................. 278

introscope.agent.leakhunter.ignore. ............................................................................................... 278

Logging ..................................................................................................................................................................... 279

log4j.logger.IntroscopeAgent ............................................................................................................................ 280

log4j.appender.logfile.File................................................................................................................................. 281

log4j.logger.IntroscopeAgent.inheritance ........................................................................................................ 281

log4j.appender.pbdlog.File ............................................................................................................................... 282

log4j.appender.pbdlog ...................................................................................................................................... 282

log4j.appender.pbdlog.layout ........................................................................................................................... 283

log4j.appender.pbdlog.layout.ConversionPattern ............................................................................................ 283

log4j.additivity.IntroscopeAgent.inheritance ................................................................................................... 284

Metric count ............................................................................................................................................................. 284

introscope.ext.agent.metric.count ................................................................................................................... 285

Multiple inheritance ................................................................................................................................................. 285

introscope.autoprobe.hierarchysupport.enabled ............................................................................................ 286

introscope.autoprobe.hierarchysupport.runOnceOnly .................................................................................... 286


introscope.autoprobe.hierarchysupport.pollIntervalMinutes .......................................................................... 287

introscope.autoprobe.hierarchysupport.executionCount ................................................................................ 287

introscope.autoprobe.hierarchysupport.disableLogging .................................................................................. 288

introscope.autoprobe.hierarchysupport.disableDirectivesChange .................................................................. 288

Platform monitoring ................................................................................................................................................. 288

introscope.agent.platform.monitor.system ...................................................................................................... 289

Remote configuration .............................................................................................................................................. 289

introscope.agent.remoteagentconfiguration.enabled ..................................................................................... 289

introscope.agent.remoteagentconfiguration.allowedFiles .............................................................................. 290

Security ..................................................................................................................................................................... 290

introscope.agent.decorator.security ................................................................................................................ 290

Servlet header decorator ......................................................................................................................................... 291

introscope.agent.decorator.enabled ................................................................................................................ 291

Socket metrics .......................................................................................................................................................... 291

introscope.agent.sockets.reportRateMetrics ................................................................................................... 292

introscope.agent.io.socket.client.hosts ............................................................................................................ 292

introscope.agent.io.socket.client.ports ............................................................................................................ 293

introscope.agent.io.socket.server.ports ........................................................................................................... 293

SQL Agent ................................................................................................................................................................. 293

introscope.agent.sqlagent.normalizer.extension ............................................................................................. 294

introscope.agent.sqlagent.normalizer.regex.matchFallThrough ...................................................................... 295

introscope.agent.sqlagent.normalizer.regex.keys ............................................................................................ 296

introscope.agent.sqlagent.normalizer.regex.key1.pattern .............................................................................. 296

introscope.agent.sqlagent.normalizer.regex.key1.replaceAll .......................................................................... 297

introscope.agent.sqlagent.normalizer.regex.key1.replaceFormat ................................................................... 297

introscope.agent.sqlagent.normalizer.regex.key1.caseSensitive ..................................................................... 298

introscope.agent.sqlagent.sql.artonly .............................................................................................................. 298

introscope.agent.sqlagent.sql.rawsql ............................................................................................................... 299

introscope.agent.sqlagent.sql.turnoffmetrics .................................................................................................. 299

introscope.agent.sqlagent.sql.turnofftrace ...................................................................................................... 299

SSL communication .................................................................................................................................................. 300

introscope.agent.enterprisemanager.transport.tcp.host.DEFAULT ................................................................. 300

introscope.agent.enterprisemanager.transport.tcp.port.DEFAULT ................................................................. 301

introscope.agent.enterprisemanager.transport.tcp.socketfactory.DEFAULT .................................................. 301

introscope.agent.enterprisemanager.transport.tcp.truststore.DEFAULT ........................................................ 302

introscope.agent.enterprisemanager.transport.tcp.trustpassword.DEFAULT ................................................. 302

introscope.agent.enterprisemanager.transport.tcp.keystore.DEFAULT .......................................................... 302

introscope.agent.enterprisemanager.transport.tcp.keypassword.DEFAULT ................................................... 303

introscope.agent.enterprisemanager.transport.tcp.ciphersuites.DEFAULT ..................................................... 303

Stall metrics .............................................................................................................................................................. 303

introscope.agent.stalls.thresholdseconds ........................................................................................................ 303

introscope.agent.stalls.resolutionseconds ....................................................................................................... 304

Contents 17

Thread dumps .......................................................................................................................................................... 304

introscope.agent.threaddump.enable .............................................................................................................. 305

introscope.agent.threaddump.deadlockpoller.enable ..................................................................................... 305

introscope.agent.threaddump.deadlockpollerinterval ..................................................................................... 306

introscope.agent.threaddump.MaxStackElements .......................................................................................... 306

Transaction tracing ................................................................................................................................................... 307

introscope.agent.bizdef.turnOff.nonIdentifying.txn ......................................................................................... 307

introscope.agent.transactiontracer.parameter.httprequest.headers .............................................................. 308

introscope.agent.transactiontracer.parameter.httprequest.parameters ........................................................ 308

introscope.agent.transactiontracer.parameter.httpsession.attributes ............................................................ 309

introscope.agent.transactiontracer.userid.key ................................................................................................. 309

introscope.agent.transactiontracer.userid.method ......................................................................................... 310

introscope.agent.transactiontrace.componentCountClamp ............................................................................ 311

introscope.agent.crossprocess.compression .................................................................................................... 312

introscope.agent.crossprocess.compression.minlimit...................................................................................... 313

introscope.agent.crossprocess.correlationid.maxlimit ..................................................................................... 314

introscope.agent.transactiontracer.sampling.enabled ..................................................................................... 314

introscope.agent.transactiontracer.sampling.perinterval.count ...................................................................... 315

introscope.agent.transactiontracer.sampling.interval.seconds ....................................................................... 315

introscope.agent.transactiontrace.headFilterClamp ........................................................................................ 316

introscope.agent.ttClamp ................................................................................................................................. 317

URL grouping ............................................................................................................................................................ 317

introscope.agent.urlgroup.keys ........................................................................................................................ 318

introscope.agent.urlgroup.group.default.pathprefix........................................................................................ 318

introscope.agent.urlgroup.group.default.format ............................................................................................. 318

WebSphere PMI ....................................................................................................................................................... 319

introscope.agent.pmi.enable ............................................................................................................................ 320

introscope.agent.pmi.enable.alarmManagerModule ....................................................................................... 320

introscope.agent.pmi.enable.beanModule ...................................................................................................... 321

introscope.agent.pmi.enable.cacheModule ..................................................................................................... 321

introscope.agent.pmi.enable.connectionPoolModule ..................................................................................... 322

introscope.agent.pmi.enable.hamanagerModule ............................................................................................ 322

introscope.agent.pmi.enable.j2cModule .......................................................................................................... 323

introscope.agent.pmi.enable.jvmpiModule...................................................................................................... 323

introscope.agent.pmi.enable.jvmRuntimeModule ........................................................................................... 324

introscope.agent.pmi.enable.objectPoolModule ............................................................................................. 324

introscope.agent.pmi.enable.orbPerfModule .................................................................................................. 325

introscope.agent.pmi.enable.schedulerModule ............................................................................................... 325

introscope.agent.pmi.enable.servletSessionsModule ...................................................................................... 326

introscope.agent.pmi.enable.systemModule ................................................................................................... 326

introscope.agent.pmi.enable.threadPoolModule ............................................................................................. 327

introscope.agent.pmi.enable.transactionModule ............................................................................................ 327


introscope.agent.pmi.enable.webAppModule ................................................................................................. 328

introscope.agent.pmi.enable.webServicesModule .......................................................................................... 328

introscope.agent.pmi.enable.wlmModule........................................................................................................ 329

introscope.agent.pmi.enable.wsgwModule ..................................................................................................... 329

introscope.agent.pmi.filter.objrefModule ........................................................................................................ 330

WLDF metrics ........................................................................................................................................................... 330

introscope.agent.wldf.enable ........................................................................................................................... 330

Appendix B: Alternative Methods for Instrumentation 331

Java Agent Deployment on Other Application Servers ............................................................................................ 331

Configure Sun ONE to Use AutoProbe ..................................................................................................................... 332

Configure Oracle to Use AutoProbe ......................................................................................................................... 333

Configure WebLogic Server ...................................................................................................................................... 334

Configuring HTTP servlet tracing .............................................................................................................................. 335

Create an AutoProbe Connector File ....................................................................................................................... 335

Running the AutoProbe Connector for a JVM ................................................................................................... 336

Example: Use Xbootclasspath to Instrument WAS ........................................................................................... 338

About running ProbeBuilder manually ..................................................................................................................... 339

Configure AutoProbe for WebSphere for z/OS ........................................................................................................ 339

Appendix C: Using the PBD Generator 343

About the CA PBD Generator ................................................................................................................................... 343

Configuring the PBD Generator ................................................................................................................................ 344

Required PBD Generator parameters ............................................................................................................... 344

Using the PBD Generator ......................................................................................................................................... 344

Appendix D: Using the Network Interface Utility 347

Determine Network Interface Names ...................................................................................................................... 347

Index 349


Chapter 1: Introduction to the Java Agent

This section contains the following topics:

About Introscope and Java Agents (see page 19) Planning a Java Agent Deployment (see page 20) Deploying the Java Agent (see page 22)

About Introscope and Java Agents

CA Introscope is an enterprise application performance management solution that enables you to monitor complex web applications in production environments 24x7, detect problems before they affect your customers, and resolve these issues quickly and collaboratively. A key part of the architecture for this solution is the low overhead agent.

The agent is a data gathering component of Introscope that collects detailed performance information about applications and the computing environment as transactions are executed. The Java agent collects this information from applications and resources running on Java Virtual Machines (JVMs) and sends the information to the Enterprise Manager for further processing.

The Java agent inserts probes into the bytecode of the components that make up the JVM that the application uses. Inserting the probes into the bytecode is part of the instrumentation process that enables the monitoring of your applications.

Instrumenting an application also requires tracers that are defined in ProbeBuilder Directive (PBD) files. The instructions or directives in PBD files identify the application components to monitor. The tracers identify the metrics the agent should collect from which probes as the application runs in the JVM. You can control what is monitored by changing the PBD files to suit your environment.

When you install the Java agent, several default PBD files are deployed to enable default monitoring of your environment. You can modify the default monitoring to achieve the balance of visibility and performance that you require. After the application is instrumented, the Java agent collects the data you are interested in and reports the data to the Enterprise Manager. The Enterprise Manager then processes and stores the data for real-time and historical reporting. You can then view and work with the collected data using the Introscope Workstation to create alerts or take responsive action.

Planning a Java Agent Deployment


For application management, the key activities are:

Deploy agents to monitor the performance and availability of application servers.

Test, adjust, and optimize monitoring of application components.

Customize the agent profile to control agent operations as needed.

Create application- or agent-specific metric groupings, dashboards, alerts, and actions

Investigate, triage, and diagnose application issues.


When planning a deployment, your goal is to develop the right balance between agent overhead and the visibility of application performance. Low overhead agents allow for real-time monitoring of all transactions in production environments. Keeping overhead low helps performance and availability of critical applications and server resources. However, keeping overhead low does not provide enough information to diagnose problems when they occur. Therefore, it is common to deploy and adjust the agent configuration throughout the lifecycle of an application. In addition, monitoring more components when an application is being developed or tested, then reducing the components after an application is released to production.

Install and Evaluate the Default Functionality

The first step in deploying agents involves installing and evaluating the default agent configuration. The default agent configuration demonstrates data collection for many common components of the application and the computing environment. The default agent configuration also includes several features enabled and other, less frequently used features, disabled.

Your goal is to evaluate the depth and breadth of data collection provided by default and become familiar with Introscope and how to monitor applications. After you are familiar with the performance metrics the agent provides by default, you can customize the agent to collect data, as needed.

As you evaluate the default performance, keep in mind that when the agent collects more metrics, it consumes more system resources. If the agent collects fewer metrics, you have less visibility into potential problems. As you refine the agent configuration, try to strike a balance between the depth of data collection and an acceptable level of performance. The appropriate level of instrumentation typically depends on where the agent is deployed. For example, an agent monitoring within a test environment is typically configured to collect large numbers of metrics. An agent on a production server, however, is typically configured to deliver essential information.



Determine Configuration Requirements

Before you deploy the agent, determine your data collection requirements. This information lets you tailor the data collection behaviors of the agent, and evaluate the impact on overhead through alternative configurations of the agent.

Introscope can be used throughout the lifecycle of an application. For example, from development through testing, load verification, staging, and into production. At each stage in the lifecycle, the monitoring goals, environmental constraints, and service level expectations are likely to be different. To address these differences, you configure the agent to behave differently for the type of environment that is monitored.

Your goal is to determine the appropriate trade-off between the visibility of performance details and resource overhead. Also consider the optimum level of visibility at a reasonable cost for the environment being monitored.

In preproduction application environments, such as development, you typically configure a higher level of data collection to provide deeper visibility into the application performance. In production or high-volume transaction environments, you typically reduce the metrics reported to control agent overhead. Depending on your requirements, you can also configure agent properties to control specific agent behavior. For example, track the maximum number of metrics collected and the removal of old metrics.

For your environment, determine the appropriate level of visibility and acceptable performance overhead, so you can configure the agent to match your requirements.

Define a Baseline Agent Profile with Appropriate Configuration Properties

After you identify the type of application environment to monitor, you can create a "candidate" agent configuration. Most agent operations are controlled using properties in the agent profile (IntroscopeAgent.profile). For example, the IntroscopeAgent.profile file defines ProbeBuilder Directive and ProbeBuilder List files the agent uses. The files listed in the agent profile then control the specific metrics that the agent gathers. The IntroscopeAgent.profile file also provides properties that enable or disable specific features or tune operations such as polling intervals.

Depending on your configuration and environment, you can adjust the properties in the agent profile to evaluate the impact of each change. For example, you can start with a default agent profile that does not have ChangeDetector enabled. Later, you can enable ChangeDetector properties in the profile and evaluate agent performance after the change before adding any other features.

Deploying the Java Agent


Evaluate Agent Performance Overhead

When evaluating an agent configuration, verify that the metrics collected provide sufficient visibility into application performance and availability. In addition, verify that the volume of metrics cannot impose an unacceptable load on the operating environment. The agent cannot report more metrics than are necessary to identify and localize performance and availability problems.

You can effectively evaluate agent performance by understanding the performance characteristics of the application. For example, you can load test your application before and after implementing default monitoring to verify impact.

To extend data collection in a controlled way, verify agent operation and overhead before and after implementing changes. For example, only add monitoring support for one application at a time, so you can evaluate the performance of each add-on individually.

Validate and Deploy the Agent Configuration

After verifying that the candidate agent configuration provides the visibility required, deploy the configuration across that environment.

You can deploy a validated configuration by installing the following configuration items to the target environment:

IntroscopeAgent.profile file

modified or custom PBD files

Deploying the Java Agent

Deploying the agent involves the following high level steps:

1. Install the agent on the target computer.

2. Configure the application server (see page 35) to instrument your applications.

3. Configure the startup script (see page 36) or Java command for the application server to include the agent and the location of the agent profile.

4. Configure the agent connection (see page 59) to the Enterprise Manager.

5. Restart the application server to instrument the application and start data collection.

6. Configure the IntroscopeAgent.profile (see page 67) if you want to change any properties set during installation or want to modify data collection or other agent operations.

Chapter 2: Installing and Configuring the Java Agent 23

Chapter 2: Installing and Configuring the Java Agent

This section contains the following topics:

Before You Install the Agent (see page 23) Select the method for installing the Java agent (see page 24) About the Java Agent Directory Structure (see page 30) Configuring Detection of Synthetic Transactions (see page 32) Java 7 Autoprobe (see page 35) How to instrument applications (see page 35) Configuring the Application Server to Start the Java Agent (see page 36) Configuring the connection to the Enterprise Manager (see page 59) Upgrade Multiple Agent Types (see page 63) Uninstalling the Java agent (see page 64)

Before You Install the Agent

Before you install the agent, do the following steps:

1. Review the Java agent deployment process.

2. Verify that you have a supported version of the application server.

Note: The server where you plan to install the agent must have a supported version of the JVM available locally. For application server and JVM requirements, see the Compatibility Guide.

3. Verify that you have a supported version of the JVM. If you are unable to use a supported version, use a previous version:

Use the 8.x Java agent for Java 1.4.x.

Use the 7.2 Java agent for Java 1.3.x.

4. Verify that the Introscope Enterprise Manager and Workstation components are installed. Note the host name and port number for the Enterpris

Documents

APM_9.5--Java Agent Implementation Guide