J2EE

DIAGNOSING J2EE PERFORMANCE PROBLEMSTHROUGHOUT THE APPLICATION LIFECYCLE

ABSTRACT

Many large-scale, complex enterprise applications are now built and

deployed using the J2EE architecture. However, many of these

applications suffer from poor performance and scalability because the

focus of the development process is on functionality, while performance

and scalability are dealt with as an afterthought.

This paper will present techniques for delivering high-performance

applications to production, managing and measuring the performance of

applications, and diagnosing the toughest J2EE problems throughout the

application lifecycle. It will examine the various types of performance

issues at each stage of the lifecycle and which diagnostic tools and

techniques can best resolve them.

Abstract ………………………………………………………………………………2

Challenges for J2EE Applications and Performance ……………………………3

Performance Evaluation and Diagnostic NeedsThroughout the Application Lifecycle …………………………………………3

Typical J2EE Application Performance Problems ………………………………6

Diagnosing Performance Problems with Mercury Diagnostics for J2EE ……7

J2EE Diagnostics for J2EE for Mercury LoadRunner andMercury Performance Center …………………………………………………8

Mercury Diagnostics for J2EE for Mercury Business Availability …………8

J2EE Deep Diagnostics …………………………………………………………9

Mercury Diagnostics Profiler for J2EE ……………………………………………10

Summary ……………………………………………………………………………11

TABLE OF CONTENTS

Challenges for J2EE Applications and Performance

Today’s enterprises are choosing to build business applications that leverage the power, portability, and

rapid development of Java technology. J2EE offers many advantages to developers, but introduces new

challenges for the development, performance diagnosis, tuning, deployment, and management of

applications. Successful delivery and management of typically complex J2EE-based applications require

evaluating performance throughout the entire application lifecycle.

An application may perform well in the development and QA environment, but fail to scale or exhibit

transient performance problems in production. It is important to understand the impact of the

infrastructure in which the application runs and the behavior of the many application components as

they interact under load. From the diagnostic perspective, it is important to be able to isolate the

problem by tier of the application architecture and by application component, and to have progressive

drill-down visibility into J2EE performance problems, the J2EE environment, and into the actual code,

with sufficient detail to determine the root cause of the problems.

There are additional factors that can increase the difficulties of application delivery and management.

The deployment lifecycle for many J2EE applications is compressed, due to increased pressures for

quick time-to-market. Boundaries between development, QA, deployment, and production stages and

IT groups are blurred. Centralized IT organizations may be managing hundreds of applications, without

in-depth J2EE experience.

Many applications have not been sufficiently architected for performance and scalability, with thorough

consideration of design and usage patterns, and adequate attention to planning and testing

performance against well-defined service objectives. J2EE scalability capabilities, while extensive, do

not substitute for such efforts. Later in the lifecycle, applications may be pushed into production to

meet deadlines, with insufficient validation of performance or scalability and inadequate tools to see

into the application’s internal behavior. This makes it difficult and expensive to fix problems. Getting to

the root of performance problems in the complex, distributed, dynamic J2EE environment is truly a

challenge.

Performance Evaluation and Diagnostic Needs Throughout the Application Lifecycle

During the application lifecycle, there are many IT stakeholders in application performance. They include

application architects, developers, load testers and engineers in QA, application support staff, and site

operations in production. They share common requirements for end-user application performance, but

also have specific roles and needs unique to their environments.

• In the design and specification phases, a topic outside the scope of this diagnostic paper,

performance and scalability issues need to be considered and incorporated, and specific business

objectives need to be set.

J2EE

WWW.MERCURY.COM 3

J2EE

• In the development and unit-testing phase, application profiling can be useful to validate

performance and functionality. Developers should test the performance of not just application

components, but also performance of the end-to-end integrated application (as it would be tested by

QA and/or used in production) against identified performance criteria. This step allows development

teams to find problems earlier in the lifecycle by preparing the application to be load tested.

• During QA cycles, load testing typically follows integrated functional and regression testing. A

complete application, including all interfaces with external systems, should be fully load tested prior

to software release. Objectives include scalability and capacity estimation under load that realistically

represent expected live usage, along with visibility into the internal performance behavior of the

application and actionable data on bottlenecks. This should include transaction breakdown of

latencies for each J2EE tier and method, along with additional specific root-cause diagnostic

information.

• Most try using profiling tools for root-cause diagnostics and fail. Traditional profiling tools used at the

developer desktop cannot be used in QA during load testing for performance diagnosis, due to the

high overhead they impose. What is needed is a profiling technology that can work in a development

environment to find and fix common problems in applications earlier in the lifecycle. Such a tool

should allow development teams to release a solid application into QA that is ready to be load tested

and where most obvious performance bugs have been found and addressed. J2EE diagnostic tools

used in QA need to be designed for load and should be integrated with load testing tools to boost

productivity, test effectively, and facilitate collaboration between application owners and QA teams to

triage, diagnose, and solve problems under load conditions.

• During staging or pre-production, deployment teams should determine the performance and

scalability of the application in the specific IT environment and configuration that will serve live users,

verify projected peak load capability, and tune the application and infrastructure to meet its designed

transaction throughput and response-time service levels, keeping the business objectives in mind.

Load testing and performance diagnostic tools that can optimize both application and infrastructure

and drill down to solve problems are essential during this stage.

• In live production, operations and application managers need to continuously monitor the health of

the entire system, measure the performance of the application, and, if something starts to go wrong,

quickly alert, triage, and isolate the problem area. Once the problem area is identified, the

appropriate teams should be able to collaborate to pinpoint the root cause of the problem. It is

important to not only proactively monitor end-user business processes, but also to diagnose issues

that happen with real users using the systems.

WWW.MERCURY.COM 4

J2EE

Let’s see an example of how a typical collaborative effort on diagnostics might look using Mercury

solutions for a new application being developed. The above diagram shows this process. The four key

stakeholders are shown as the four circles — each representing the business owner, the application

owners, QA, and production operations. The application lifecycle is shown by dotted arrows on the

outside, and the interactions of the various roles in the lifecycle are shown as solid arrows for each

stage. The Mercury lifecycle solution facilitates collaboration between team members by providing a

single technology across the lifecycle and role-based web dashboards that can drastically reduce the

mean time to problem resolution.

Throughout the entire application lifecycle, J2EE-specific visibility and diagnostic capabilities should be

integrated with and complementary to multi-platform, multi-protocol tools for functional testing, load

testing, and application performance management. Common tools and measurements help facilitate

communication, while team members need specific features and capabilities suited to their roles.

WWW.MERCURY.COM 5

Figure 1: Mercury Diagnostics is a single solution that enables developers to find problems earlier in the lifecycle, QA teams to beempowered with the right data set to share with development to fix problems under load, and production operations teams to managethe application 24x7, and triage and diagnose issues as they occur.

MARKET REQUIREMENTS/

GOALS

BUSINESSOWNER

DEVELOP

APPLICATIONOWNER

APPLICATIONREADY

MERCURY DIAGNOSTICS FOR J2EE

FUNCTIONALAPPLICATION

(1 USER)

LOAD-READYAPPLICATION

(5-20 USERS)

BUSINESS-READY

APPLICATION

(50,000 USERS)

QA

DEPLOY

PRODUCTIONOPERATIONS

PRODUCTION-READY

APPLICATION

MERCURY DIAGNOSTICS PROFILER

MERCURY BUSINESS AVAILABILITY CENTER DIAGNOSTICS

QA/TESTING

MERCURY LOADRUNNER/MERCURY PERFORMANCE CENTER DIAGNOSTICS

J2EE

Typical J2EE Application Performance Problems

There are many problems that can surface during the application lifecycle. For J2EE applications, the

user’s experience of performance is affected by many external network infrastructure factors that are

independent of application behavior. A broad set of monitors that can cover the complex environment

is needed to triage problems quickly. From the J2EE application diagnostic perspective, it is essential to

be able to capture and correlate specific external parameters, such as method arguments, that can

drive performance problems in a J2EE method or sequence of transactions. Specific latencies and

parameter captures are also frequently required to identify problems at J2EE interfaces to external

systems, such as backend databases, legacy systems, and packaged software.

Within the J2EE environment, some of the most common problems include:

1. Code problems:

a. Slow methods

• Consistently slow methods

• Intermittently slow methods, related to specific user/data values driving problematic application

behavior

b. Memory problems, including memory thrashing and memory leaks

c. Coding practices, such as using exceptions as a means to transfer control in the application,

processing large result sets, etc.

d. Synchronization problems, including both under-synchronization and over-synchronization for

locks and threads

2. Application server configuration problems:

a. JDBC connection pool size

b. JVM heap size

c. Thread pool sizes

3. Architecture and design problems, with a wide range of issues, such as:

a. Data-marshalling problems resulting from filtering at the wrong tier

b. Serial execution resulting from over synchronization in design of custom code

An effective diagnostic solution must provide capabilities and techniques to triage and identify the root

cause of each of these common problems, regardless of when they emerge, from development through

production. Capturing application performance data requires the ability to tune up the data capture as

you go from identification and triage to root-cause diagnostics. This is best achieved with lifecycle

technology without having to install new tools to collect greater details for root-cause diagnostics.

WWW.MERCURY.COM 6

J2EE

Diagnosing Performance Problems with Mercury Diagnostics for J2EE

Mercury Diagnostics for J2EE™ provides the industry’s first top-down, end-to-end lifecycle approach to

seamlessly manage, monitor, diagnose, and resolve critical problems in J2EE applications – in both pre-

and post-production environments. It can be implemented alone or integrated with Mercury

LoadRunner®, Mercury Performance Center™, and Mercury Business Availability Center™ offerings.

Mercury Diagnostics for J2EE provides a common, consistent foundation of shared assets, metrics,

scripts, and monitors, as well as specific solutions for the differing requirements of application delivery

and application management.

Mercury Diagnostics for J2EE uses a combination of JMX, aggregation, and unique Total Trace

technology, which can capture data to diagnose all of the problems previously described. It is the only

solution that can capture, under load, necessary and sufficient information to get to the root cause for

the full range of J2EE performance problems. Unlike other techniques, Total Trace allows capturing of

every event (including arguments) at low overheads. Other transaction tracing techniques may use a

threshold for collecting a fixed number of sampled events regardless of whether they are related to the

business process in question or not. This could lead to capturing of sampled events that might be

totally unrelated to the problem at hand.

Another common technique used for tracing a transaction is to “tag” a server request or HTTP URL

request with a specific header. This header then can be “followed” through the application to

provide traceability across methods. Most common implementations of such techniques ignore the

business process name that a specific method belongs to and are limited in their ability to trace

across JVM boundaries.

WWW.MERCURY.COM 7

Figure 2: Mercury Diagnostics for J2EE combines J2EE-specific diagnostics with application delivery and application managementsolutions.

BEAIBMSAP

ORACLE...

MERCURY IT GOVERNANCE CENTER

MONITOR • TRIAGE • DIAGNOSE

APPLICATION MANAGEMENT

MERCURY BUSINESS AVAILABILITY CENTER

END USER, APPLICATION, SYSTEM LEVEL AND

SERVICE LEVEL MANAGEMENT

APPLICATION DELIVERYAPPLICATION DEVELOPMENT

MERCURY DIAGNOSTICS PROFILER

MERCURY QUALITY CENTER

MERCURY PERFORMANCE CENTER

TEST MANAGEMENTFUNCTIONAL TESTING LOAD TESTING

APPLICATION LIFECYCLE

APPLICATION PLATFORMS - BEA, IBM, MICROSOFT, ORACLE, SAP, SIEBEL

MERCURY DIAGNOSTICS

J2EE

For example, if a business

transaction were to interact with

several JVMs hosted on remote

machines, it would be critical to

trace the business transaction

across the different JVM

boundaries and provide

diagnostics data without losing

the business context of the

transaction. The screen shot to

the right shows how the remote

calls crossing JVM boundaries are

distinguished with a “Remote

Calls” layer and critical data such

as SQL queries and application

exception information.

Mercury Diagnostics for J2EE for Mercury LoadRunner and Mercury Performance Center

Mercury Diagnostics for J2EE is fully integrated with Mercury LoadRunner and Mercury Performance

Center to expose and diagnose the most common J2EE performance problems under load testing.

Mercury Diagnostics for J2EE first correlates end-user problems to the web transaction tier for fast

isolation and resolution, then pinpoints the infrastructure, application, or database problems. It analyzes

the TCP/IP stack to break down a transaction’s end-to-end response time from an end-user perspective

to determine whether poor transaction time is caused by external network issues or by a specific server

tier. It then guides users through an intuitive top-down analysis process across multiple application

layers, tracing business performance problems from the end user all the way to the problematic

component.

In addition, Mercury Diagnostics for J2EE shows performance metrics for the typical J2EE tiers such as

JSP/Servlets, EJBs, JMS, JDBC, and JNDI across a clustered environment in a single dashboard,

allowing you to isolate and identify slow components, detect memory leaks, and more.

Mercury Diagnostics for J2EE for Mercury Business Availability

Mercury Diagnostics for J2EE is fully integrated with Mercury Business Availability Center to maximize the

performance and availability of J2EE applications by providing a unified solution for your operations and

application support teams to quickly monitor, diagnose, and optimize production applications. It provides

teams with visibility across user, business process, application server, application platform suites such as

portal servers, and system tiers down to the component and method level. Ultimately, high-performing,

mission-critical applications protect service-level agreements (SLAs) and the bottom line.

WWW.MERCURY.COM 8

Figure 3: Tracing remote method invocation across JVMs.

J2EE

The solution provides the

industry’s broadest monitoring

capabilities that allow 24x7

monitoring of the entire

application infrastructure,

including the J2EE environment,

from a single web-based console.

Agentless monitoring for

enterprise infrastructure

components can optimize the

total cost of systems

management. In addition, it can

facilitate capacity planning by

ensuring that purchases of hardware

and software are optimized.

Mercury Business Availability Center provides correlation of end-user performance with root cause in

the infrastructure and application layers. Its J2EE-specific diagnostics identify problems involving EJB

components, methods, and JDBC calls with SQL statements. Mercury Diagnostics for J2EE integrated

with Mercury Business Availability Center provides the same transaction breakdown capabilities as

Mercury LoadRunner and Mercury Performance Center, using aggregated data capture, while optimized

for low-overhead, real-time monitoring, triage, and problem isolation.

Unlike other implementations that might need inclusion of client-side scripting code in web application

pages to measure the response time from the browser, Mercury’s diagnostics solution uses industry-

leading scripting technology to capture this information from an end-user perspective with synthetic

scripts. A similar noninvasive technique is also employed to get diagnostics information for real users

(actual people using the production systems as opposed to synthetic scripts) for production usage.

J2EE Deep Diagnostics

Complex problems, such as finding the cause of thread deadlocks or objects that aren’t de-allocated

and removed by Java Garbage Collection, are responsible for the majority of the time and cost spent in

diagnosing the root cause of performance problems.

J2EE Deep Diagnostics uses Mercury’s Total Trace data capture, which captures every event in a

selected area of the application with low overhead, by providing carefully tuned, predefined sets of

byte-code instrumentation and a GUI for custom selection and automatic application of narrowly

targeted instrumentation. In production, a Mercury Business Availability Center alert can trigger a deep

WWW.MERCURY.COM 9

Figure 4: Transaction breakdown for real user for Portal Server environment showing the specific portletperformance.

J2EE

diagnostic data capture for offline analysis. This stops the common frustration of persistently recurring

production performance problems that cannot be reproduced in a test bed. The same J2EE Deep

Diagnostics works with Mercury LoadRunner when performance problems under load need better

characterization for effective bug definition and rapid fixes.

J2EE Deep Diagnostics provides multi-layer correlation of HTTP, servlets, JSPs, EJBs/objects, methods,

and SQL calls. It correlates these internal component measurements with JMX and OS metrics to show

problem details. It delivers graphical views of transaction traces, call chains, memory leaks and

thrashing problems, synchronization details of locks held and threads blocked, latency charts, and

source code views of problem methods. J2EE Deep Diagnostics captures details of specific instances

of method invocations, including parameters passed and individual latencies, to pinpoint the source of

intermittent problems caused by specific user data values or usage patterns.

For further examples and details on how to solve synchronization problems, intermittent slow methods,

and other J2EE performance problems using Mercury Diagnostics for J2EE, contact your Mercury

representative.

Mercury Diagnostics Profiler for J2EE

Mercury Diagnostics Profiler for J2EE extends the application lifecycle solution into the development

environment. Developers can use the same lifecycle probe technology used for load testing and

production to find and fix common application problems earlier in the lifecycle. This also enables

applications to be readied for load testing. Often, when QA groups start to load test the application,

they find that the application is not ready for load testing. Specifically, the application breaks or doesn’t

scale beyond two to five users. Lack of application readiness for load testing and production wastes

time and effort.

Developers are often very interested in application performance data that is actionable. Mercury

Diagnostics Profiler not only provides developers the ability to solve the most common problems

discussed earlier, but also to find specific problems such as application exceptions being thrown,

slowest SQL queries, memory leaks in primitive object types, JAVA objects, and JAVA collections. The

following screen shots show how a developer can understand if and specifically what exceptions are

being thrown in the application at run time. These could migrate to production as painful timebombs if

left undetected earlier.

For diagnosing problems in JAVA collections, it’s not only important to understand which collections are

growing fastest and which are the largest, but also to understand the contained class and where the

collection is being allocated. For example, is the contained class in a collection a critical resource such

WWW.MERCURY.COM 10

as DB connection or socket or a less

important Integer object? Figure 5

shows how Mercury Diagnostics

Profiler can provide this visibility to

help find and solve more problems

earlier in the lifecycle.

Summary

With Mercury Diagnostics for J2EE,

enterprises can improve the readiness

of their J2EE applications before they

are placed into production, and

dramatically accelerate problem

identification, isolation, and resolution.

Mercury Diagnostics for J2EE allows

companies to reduce their costs of

operations by providing a business

process view into the application and

by isolating and resolving problems

before business users and the bottom

line are impacted. The solution can also

help companies optimize their existing

infrastructure and avoid unnecessary

hardware or software expenditures.

Choosing the appropriate application

profilers, application delivery tools, and

application management tools is critical

to guaranteeing application quality and

ensuring that SLAs can be met.

Mercury Diagnostics for J2EE is the

industry’s first, most complete

application delivery and application

management suite for the J2EE

ecosystem that improves the quality,

performance, and scalability of J2EE

applications across the entire

application lifecycle.

J2EE

WWW.MERCURY.COM 11

Figure 5: Exception stack traces.

Figure 6: JAVA collection memory diagnostics.

© 2006 Mercury Interactive Corporation. Patents pending. All rights reserved. Mercury Interactive, Mercury the Mercury logo, Mercury Diagnostics for J2EE, Mercury Diagnostics Profiler, Mercury LoadRunner, Mercury Performance Center,and Mercury Business Availability Center are trademarks or registered trademarks of Mercury Interactive Corporation in the United States and/or other foreign countries. All other company, brand, and product names are marks of theirrespective holders. WP-1355-0406

Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business outcome of IT.WWW.MERCURY.COM