product direction. It is intended for information€¦ · WebLogic Server Performance Tuning Jonathan Leung FMW APAC Performance Specialist ... WebLogic Server Core Tuning ... •

2 | © 2011 Oracle Corporation – Proprietary and Confidential

The following is intended to outline our general

product direction. It is intended for information

purposes only, and may not be incorporated into

any contract. It is not a commitment to deliver any

material, code, or functionality, and should not be

relied upon in making purchasing decisions.

The development, release, and timing of any

features or functionality described for Oracle’s

products remain at the sole discretion of Oracle.

Extreme (WLS/Java) Performance WorkshopWebLogic Server Performance Tuning

Jonathan Leung

FMW APAC Performance Specialist

Copyright © 2012, Oracle and/or its affiliates. All rights reserved.

Agenda

• WebLogic Server Core Tuning

• Web Application Tuning

• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning

• Application Tuning

• Timeout Tuning

• Diagnosing Performance Issues


WebLogic Server Core TuningNetwork-related Tunables

• WebLogic Server 10.3.2+

• The maximum number of threads is 4.

• WebLogic Server 10.3.1 (and below)

• # of muxer threads = # of cpus + 1

• On larger systems, with multicore CPUs, could

easily have over 100 muxer threads

• Higher CPU utilization due to fat lock

contention while synchronizing

• Limit the # of Muxer threads:

-Dweblogic.SocketReaders=4


WebLogic Server Core TuningNetwork Tuning

• Always verify that WebLogic is using NativeIO.

The following message indicates it is in use

• <Aug 7, 2012 8:32:10 AM CDT> <Info> <Socket>

<BEA-000446> <Native IO Enabled.>

• Be careful when switching from 32bit to 64bit

JVM. The 32bit WebLogic Server download does

not contain the 64bit native libraries needed for

the Socket Muxer.


WebLogic Server Core TuningExalogic Optimizations

• Turn on ―ExalogicOptimizationsEnabled‖ flag under

Domain->Configuration->General

• This will enable the following core optimizations

• Pure java NIO Socket Muxer

• Scattered Reads

• Gathered Writes

• JSSE as the SSL Provider

• Lockless Request Manager

• More aggressive self-tuning thread algorithm


WebLogic Server Core TuningMonitor Channel Connections

• Ensure we don’t have too many or uneven

tcp connection counts across managed

servers


WebLogic Server Core TuningThreads

• Monitor Stuck Threads, Queue Length,

Throughput


WebLogic Server Core TuningSTUCK Threads

• WLS monitors its threads to detect threads where

a single request is taking “too long” to process.

• “Too long” is defined by Stuck Thread Max Time (600 s)

• What does WLS do when it detects such a thread?

• Marks the thread as STUCK

• WLS logs a warning message with a stack trace of the

thread at that instant in time

• Increments the stuck thread count of the appropriate

Work Manager

• Takes Failure Action if count >= Stuck Thread Count


WebLogic Server Core TuningSTUCK Threads (cont’d)

• WebLogic Server will NOT kill a STUCK thread

• A STUCK thread is not necessarily stuck

• Just means it is long-running

• Just means that at the point in time detected, the logged

stack trace is where the thread was currently executing

• It may be doing something else immediately before or after!

• When you encounter a STUCK thread

• Take multiple thread dumps at regular intervals

• Looking at the stack trace of the STUCK thread and its

progress over time is key to diagnosing the issue.

• WLDF can be configured to automatically create an

incident that will include the series of thread dumps

• Oracle SOA Suite domains do this OOTB


WebLogic Server Core TuningWork Manager Tuning

• Create Work Managers to avoid thread

starvation, only as needed.

• Fair Share work manager is typically what you

want

• Monitor

• Pending Requests

• Max Wait Time

• Current Wait Time


WebLogic Server Core TuningWork Manager Tuning (cont’d)


WebLogic Server Core TuningWork Manager Constraint Tuning

• Max Threads Constraint

• Limit the number of concurrent threads

processing requests for the Work Manager

• Use to configure MDB concurrency

• Default is 16

• Capacity Constraint

• Limits the number of total requests (currently

processing and queued) to avoid resource

issues. When the capacity is reached

• HTTP requests get an HTTP 503 response

• RMI clients get a QueueThrottleException response


WebLogic Server Core TuningWork Manager Constraint Tuning (cont’d)

• Min Threads Constraint

• Guarantees thread availability to process

requests, thus avoiding thread starvation

• Use only if absolutely required

• Keep value very low (e.g., 3)

• Min Threads Constraint work is immediately

scheduled if constraint not yet met

• Setting too high affects work for other work

managers


WebLogic Server Core TuningStandalone Java Clients

• Prefer wlfullclient.jar

• Replacement for weblogic.jar

• Use WebLogic JarBuilder utility to generate

• Clients that use WLST need a full install

• If jar size is critical, use wlthint3client.jar

• wlclient.jar uses the Java ORB

• Silently converts t3 URLs to use IIOP.

• Generally slower than wlfulllclient.jar

• Not as scalable for multi-threaded clients

• wlthint3client.jar replaces wlclient.jar

• Uses t3 runtime faster, better scalability!


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning



Web Application TuningJSP and Servlet Change Detection Tuning

• WLS can automatically detect and reload

Servlets/JSP that have changed

• Servlet Reload Check Seconds (servlet-reload-check-secs)

• Entire web application is reloaded when change detected!

• JSP Page Check Seconds (pageCheckSeconds)

• JSP page is recompiled and reloaded

• Default Behavior

• Development Mode, check every second

• Production Mode, never check

• Values in weblogic.xml override defaults

• Be careful when moving from Development to Production


Web Application TuningJSP Compilation Tuning

• By default, JSP files are compiled on demand

• Can cause poor response time for first access

• To avoid this initial overhead, use WLS tools

to precompile JSP files prior to deployment.

Precompile JSPs from the command line:

java weblogic.appc hrWeb.war


Web Application TuningJSP Compilation Tuning (cont’d)

• Can force WLS to never look for JSP source

files by using JspClassServlet

• Precompile the JSP files

• Map jsp extension to JspClassServlet

• No need to package JSP files in WAR

<servlet>

<servlet-name>JSPClassServlet</servlet-name>

<servlet-class>weblogic.servlet.JSPClassServlet</servlet-class>

</servlet>

<servlet-mapping>

<servlet-name>JSPClassServlet</servlet-name>

<url-pattern>*.jsp</url-pattern>

</servlet-mapping>


Web Application TuningJSP and Servlet Output Buffer Tuning

• Web Container buffers output while building

response

• Data flushed to client once buffer is full

• Cannot forward request once data sent to client

• Set buffer size accordingly

• ServletResponse.setBufferSize(int)

• <@page buffer=―64kb‖ @>

• WLS default buffer size is 12kb

• Avoid calling flush() and close() on

ServletOutputStream


Web Application TuningHttpSession Replication Tuning

• WLS replicates only session’s new/changed

attributes

• Keep session objects as small as possible

• Better to store lots of attributes rather than one

• Put only Serializable objects in the session

• Failure to do so will prevent replication

• Monitor primary distributions across cluster

• Uneven distributions likely means the Load

Balancer/Proxy isn’t properly distributing the load


Web Application TuningHttpSession Replication Tuning (cont’d)


Web Application TuningHttpSession Replication Tuning (cont’d)

• Make sure requests with existing sessions

stick to server where primary is located

• Remote session ―fetch‖ and secondary relocation

will have significant performance implications

• WebLogic Web Server Plug-ins provide sticky

routing

• No need for HWLB sticky sessions in front of web

servers

• When not using Web Servers with WL Plug-in,

HWLB should always use Sticky Sessions

• Active Cookie Persistence is recommended


Web Application TuningExalogic Session Replication Optimizations

• Turn on ―ExalogicOptimizationsEnabled‖ flag under Domain-

>Configuration->General

• This will automatically enable Lazy Deserialization

• Create Replication Channel

• Environment->Servers-><server>->Protocols->Channels

• Enable the use of this replication channel

• Cluster->Configuration->Replication

• Cluster Replication Ports

• Environment->Servers-><serverName>->Configuration-

>Cluster

• Configure replication port range (Eg: 8001-8010)


Web Application TuningOffload HTTP Sessions to Coherence*Web

• WLS performance can be impacted if too many

sessions remain in memory

• Consider Coherence*Web for apps with lots of

users, large sessions, or long session timeouts

• Offloads sessions into a Data Grid

• Easy to scale by increasing the number of

Coherence nodes in the Grid

• Allows sharing sessions across applications


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning



JDBC TuningConnection Pool Sizing

• Creating DB connections is time consuming

• Ideally, set Min = Max to avoid creating connections on

demand

• If DB connections are limited, set Min to handle normal load

and max to handle peak loads, enable pool shrinking

• Most applications need only one connection per DB

per concurrent request being processed

• Typically, that means one per execute thread.

• Consider <max-threads-constraint> to prevent contention

with limited numbers of DB connections


JDBC TuningConnection Pool Sizing (cont’d)

• Monitor Data Source

statistics in the Console

• Active Connections High

Count

• Waiting on Connection

High Count

• Wait Seconds High

Count

• Tune accordingly

• Maximum Capacity

• Connection Reserve

Timeout (default: 10 sec)


JDBC TuningStatement Cache Tuning

• WLS caches JDBC

Statement objects

• Cache is per connection

• # of cached statements =

# of conns * cache size

• Tune according to

hit/miss ratio

• WLS Console


JDBC TuningStatement Cache Tuning (etc.)

• Databases also cache

parsed SQL statements

• Oracle DB caches SQL

in Shared Pool

• SQL cache shared

across all connections

• Work with your DBA to

optimize database

parsing of SQL


JDBC TuningExalogic Optimizations

• SDP-enabled JDBC thin driver

• JDBC URL: Replace “(PROTOCOL =

TCP)” with “(PROTOCOL = SDP)”

• Requires Oracle DB (Exadata) with

• SDP Listener configured (listener.ora)

• Use ActiveGridlink data source instead of Multi

Datasource


JDBC TuningMiscellaneous Tuning

• Connection Creation Retry Seconds (default = 0)

• If database is not available for a Data Source, WLS will fail to

boot if set to zero—the default!

• Set to a non-zero value to allow the server to boot and

periodically retry to create the connection pool

• Seconds to Trust Idle Pool Connection (default = 10 s)

• Works in conjunction with Test Connections On Reserve

• Can reduce connection testing queries significantly

• Consider LLR for JTA transactions

• Only helps with XA transactions involving multiple resources

• Last Logging Resource (non-XA) must be a database


JDBC TuningMiscellaneous Tuning (cont’d)

• Remove Infected Connections Enabled (default = on)

• Used to clean up connections where app accessed

underlying physical connection object

• Can improve performance to disable when using physical

connections to access driver-specific extensions

• Use only when safe!!!

• App doesn’t retain reference to the Connection, Statements,

or ResultSet objects

• App doesn’t change the state of the connection

• Effective database user ID

• Auto commit state


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning



EJB TuningStateless Session Beans

• The defaults are typically good enough

• initial-beans-in-free-pool = 0

• max-beans-in-free-pool = 1000

• There is a cost with creating a new instance in

the pool and thus tuning the initial-beans-in-

free-pool might be beneficial if the runtime

cost is too much

• This may increase the time to startup but reduces

the runtime cost



• Tuning based on Pool Miss Ratio

• Pool Miss Ratio = (Pool Total Miss Count / Pool

Total Access Count) * 100

• These values are available as MBeans (console,

wlst, jmx)

• A high pool miss ratio indicates an improperly

tuned pool where it doesn’t find an instance in the

pool




EJB TuningMDB Tuning

• Maximum MDB concurrency determined by:

• Is MDB using a custom work manager with a max-

threads-constraint?

• Yes

• max-threads-constraint < max-beans-in-free-pool?

• Yes : Max concurrency = max-threads-constraint

• No : Max concurrency = max-beans-in-free-pool

• No

• Is max-beans-in-free-pool > 16?

• Yes: Max concurrency = 16

• No: Max concurrency = max-beans-in-free-pool


EJB TuningMDB Tuning (cont’d)

• To handle errors in MDBs, you should:

• For transactional delivery, use setRollbackOnly() to

force the message back onto the source queue

• For non-transactional delivery, raise a

RuntimeException to prevent message

acknowledgement.

• EJB spec requires destroying the MDB instance

• Use weblogic.ejb.NonDestructiveRuntimeException to

prevent WLS from destroying the MDB instance

• Use Foreign JMS Server for remote destinations


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning



JMS TuningInter-Application Messaging Patterns

Producing Applicatio

n

Consuming

Application

Conceptual Model


n

SAF

Consuming

Application

Push Model


n

Consuming

Application

Pull Model

When you have control over the consuming application,

prefer the Pull Model as it is simpler and more efficient


JMS TuningUnderstanding JMS Transactions

• JMS supports two types of transactions

• Transacted Sessions – local transaction only

involving JMS operations on a single JMS Session

• JTA Transactions – global JTA transaction that can

include other transactional (i.e., XA) resources

• Transacted Sessions and JTA transactions

are mutually exclusive

• JTA transaction must use createSession (false, …)

• Messages sent within a transaction are

acknowledged on commit


JMS TuningChoosing a Persistent Store Type

• WebLogic Server supports two types of

persistent stores for messages

• File Store

• Database Store

• File Store is generally faster

• Database Store is generally easier for HA

• JMS Store must be highly available for failover

• File Store shared disk (e.g., SAN)

• Database Store database is accessible


JMS TuningJMS Wrappers

• WLS JMS Wrappers provide:

• JMS Connection, Session, & Producer pooling

• Testing/Re-establishment of JMS connection

• Automatic transaction enlistment for XA providers

• To use:

• Map Connection Factory & Destination to app using:

• <resource-ref> entries in deployment descriptors, or

• @Resource annotations with dependency injection

• Write normal JMS code to send a message:

• Create new connection, session, and producer for each request

• Close objects at the end of each request


JMS TuningJMS Checklist

• Consumers must be able to keep up with

producers over extended period of time

• Avoid intra-cluster message routing for the

best performance and scalability

• Use JMS Wrappers to send messages in

server-side apps

• Prefer asynchronous consumers

• Use MDBs for consumers running in a WLS server

• Use Message Listeners for standalone consumers


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning



Application TuningApplication Performance Problems

• Most application performance issues are

typically contention-related:

• Overuse of synchronization

• Insufficient resources (e.g., database connections)

• Overloaded backend systems

• Load testing can identify most contention

• Take thread dumps at regular intervals (5-10 secs)

• Attach a profiler such as JRockit Mission

Control or Yourkit


Application TuningCoding for Performance

• Cache JNDI InitialContext, Data Source, JMS

Connection Factory and Destination objects

• Use the Service Locator pattern to cache lookups, or

• Use Dependency Injection

• Use Coherence to cache frequently used data

• Can be used explicitly in your application

• Can be used as a JPA Level 2 cache


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning



Timeout TuningSetting Network Timeouts

• Set appropriate CONNECT and READ

timeouts to prevent threads hanging indefinitely

• RMI - No default timeouts when using JNDI

• Set connect and read timeouts using JNDI InitialContext

properties

• weblogic.jndi.WLContext.REQUEST_TIMEOUT – This is

the connect timeout

• weblogic.jndi.WLContext.RMI_TIMEOUT – This is the

read timeout

• Future versions will use:

• weblogic.jndi.WLContext.CONNECT_TIMEOUT

• weblogic.jndi.WLContext.RESPONSE_READ_TIMEOUT


Timeout TuningSetting Network Timeouts (cont’d)

• HTTP – set timeouts for outbound calls

• Set programmatically

• URLConnection.setConnectTimeout(int milliseconds)

• URLConnection.setReadTimeout(int milliseconds)

• If using default weblogic.net.http implementation,

default values can be set using system properties

• -Dweblogic.http.client.defaultConnectTimeout=3000

• -Dweblogic.http.client.defaultReadTimeout=30000


Timeout TuningSetting Network Timeouts (cont’d)

• JDBC – set timeouts to prevent database

issues from hanging threads in the server

• Set Oracle Thin Driver properties

• oracle.net.CONNECT_TIMEOUT=3000

• oracle.jdbc.ReadTimeout=30000

• WebLogic Console

• Set Invocation Timeout Seconds in order to avoid

scenarios where a managed server might be

physically disconnected from the network and yet

the Admin Server tries to contact it causing threads

to hang indefinitely

• DomainConfigurationGeneralAdvanced


Timeout TuningRJVM Heartbeat Tuning

• RJVM – logical, persistent connection between

two WebLogic endpoints

• Connection health is determined via heartbeats

• By default, heartbeats sent every 60 seconds (Period Length)

• By default, connection is declared unhealthy and closed after

4 missed heartbeats (Idle Periods Until Timeout)

• By default, can take 240 seconds (60*4) to detect an

unhealthy connection and potentially fail over

• Way too long, should be tuned down

• Idle Periods Until Timeout minimum is 4

• Tune down Period Length instead

• ServerConfigurationTuningAdvanced


Agenda



• JDBC Tuning

• EJB/MDB Tuning

• JMS Tuning


• Timeout Tuning


Copyright © 2012, Oracle and/or its affiliates. All rights reserved. `

Diagnosing Performance IssuesThread Dumps

• Take multiple thread dumps at regular intervals

• Lots of ways to get thread dumps

• kill -3 <pid> (best)

• jstack <pid> (Hotspot)

• jrcmd <pid> print_threads (JRockit)

• Ctrl-Break on Windows (only foreground process)

• Visual VM

• WLST


Diagnosing Performance IssuesThread Dumps (cont’d)

• Use ThreadLogic to analyze thread dumps

(shameless plug)

• It can analyze and provide recommendations

automatically based on common patterns

• It can do a ―diff‖ of multiple thread dumps over time

and provide recommendations

• http://java.net/projects/threadlogic


http://java.net/projects/threadlogic





Diagnosing Performance IssuesHeap Dumps

• A common cause of performance problems is frequent,

long GC pauses

• Likely due to heavy memory usage or memory leak

• Heap dump is a memory snapshot at a point in time

• Several ways to get an HPROF heap dump

• jmap –dump:format=b,file=heapdump.hprof <pid> (Hotspot)

• jrcmd <pid> hprofdump filename=heapdump.hprof (JRockit)

• jextract corefile [zipfile] (IBM)

• Add the following to the JVM options first :

• -Xdump:system:defaults:request=exclusive+prepwalk

• -XX:+HeapDumpOnOutOfMemoryError


Diagnosing Performance IssuesHeap Dumps (cont’d)

• Various tools to analyze HPROF heap dumps

• Eclipse Memory Analyzer Tool (MAT)

• Visual VM

• Java Heap Analysis Tool (JHAT)

• YourKit

• These tools can show

• Which objects make up the heap

• Compare and ―diff‖ multiple heap dumps to detect memory

leaks

• MAT is probably the most popular and easiest to use






Diagnosing Performance IssuesNetwork Performance Troubleshooting

• Best way to troubleshoot a potential network

issues is with a packet capture

• To capture the data

• tcpdump (Unix)

• Make sure to output to a file

• tcpdump –i bond0 -w file.pcap

• Wireshark (Windows)

• Use Wireshark to visually analyze the packet capture


Diagnosing Performance IssuesNetwork Performance Troubleshooting (cont’d)

• tcptrace

• neat utility that can analyze a packet capture and

provide statistics such as packet loss and

retransmissions in a nicely laid out format

• Prebuilt Windows binary exists as well (uses cygwin)

• Fiddler

• Very useful Web Debugging HTTP proxy that

intercepts browser request/responses








Diagnosing Performance IssuesOther Useful Profiling Tools

• JRockit Mission Control

• OOTB Flight Recorder has very minimal performance

overhead and can be used in production

• Analyze GC behavior, potential latency issues

• Visual VM

• YourKit

• AD4J


Diagnosing Performance IssuesOther Useful Profiling Tools (cont’d)

• OS Tools

• truss/strace – trace system calls.

• Very useful to see whether performance issues are at

the lowest level

• top, ps, prstat, vmstat, mpstat, iostat, netstat, etc. –

Monitor performance at a system level.

• Useful for monitoring CPU, disk I/O, NFS statistics,

network I/O, virtual memory statistics, processor

statistics, and process statistics (such as how much

CPU is consumed per thread)


Questions?


Documents

product direction. It is intended for information€¦ · WebLogic Server Performance Tuning Jonathan Leung FMW APAC Performance Specialist ... WebLogic Server Core Tuning ... •