Analysis bottleneck in J2EE application

BEA Systems KoreaByungwook Cho

([email protected])

Analysis WebLogic Hang up & slow down

Agenda

What is slowdown & hang up?1

Slowdown in WAS2

Slowdown in other resources3

Common mistake in J2ee developer4

What is the slowdown & hang up?

• Slowdown– System runs very slowly like a stop– Sometimes, system gets a lot of resources

(CPU,Memory)

• Hang up– System stops – System has no response

What is the slowdown & hang up

• To solve this problem

Find the reason

Solve it!!

With experience,tool and test

Find the problemDevelopment env : Stress Test(Load Runner,MS Stress -free)

Production : Itself is a problem

What is the slowdown & hang up

User AP

WAS

JVM DBMSWeb Server

Network

Client

< Basic J2ee architecture >

Legacy

Slow down in WAS & User AP


User AP

WAS

JVM DBMSWeb Server

Network

Client

Legacy


• Thread Pooling

Thread Pool

Idle Thread(Waiting)

request

Thread Pool

Working Thread(Handling request)

request

response

Thread Pool

Working Thread


• Basic structure of WAS

WebServerOR BrowserOR Client

Dispatcher

Request Queue

ThreadPool

ConnectionPool

Thread Queue

JMS Thread Queue

APP1 Thread Queue

APP2 Thread Queue

Other ResourcesConnector

Working Thread

IdleThread


• Java Thread State

Runnable

newstart

BlockedSleep/done sleep

Wait/notify

Suspend/Resume

Block on IO/IO complete

dead

stop

<< Thread State Diagram >>

- Runnable : running- Suspended,Locked : waiting on monitor entry / waiting on condition /MW- Wait : Object.Wait()


• Thread dump– Snapshot of java ap (X-ray)– We can recognize thread running

tread state by “continuous thread dump”


• Getting Thread Dump– Unix : kill –3 pid , Windows : Ctrl + Break– Get thread dump about 3~5 times between 3~5secs※ Linux : pstree -pan

Threads

Time

Working thread

Thread dump


• Thread dump– It displays all of thread state by “THREAD STACK”

"ExecuteThread: '42' for queue: 'default'" daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68]

at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427) at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344) at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51) at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56)

Thread name Thread id (signature)Thread State

Program stack of this thread

Sun JVM


• How to analysis thread dump

MYTHREAD_RUN(){ call methodA();} methodA(){

//doSomething(); call methodB();} methodB(){

//call methodC(); }

methodC(){

// doSomething for(;;){// infinite loop } }

“MYTHREAD” runnable at …MYTHREAD_RUN(): “MYTHREAD” runnable at …methodA() at …MYTHREAD_RUN(): “MYTHREAD” runnable at …methodB()at …methodA()at …MYTHREAD_RUN(): “MYTHREAD” runnable at …methodC()at …methodB()at …methodA()at …MYTHREAD_RUN():

“MYTHREAD” runnable at …methodC()at …methodB()at …methodA()at …MYTHREAD_RUN():

계속 이모양이반복됨

Source codeThread dumps


• CASE 1. Lock contention– In the java application java thread wait for lock in

synchronized method– Occurred in multi threaded program– Reduce frequency of synchronized method– Synchronized block must be implemented to be end

as soon as possible (※ IO? )

public void synchronized methodA(Object param){ //do something while(1){}}

< sample code >


• CASE 1. Lock contention

Thread1 Thread3

Thread2

Thread4

Sychronized MethodA

Threads

Time

Thread1 – That has a lock

Thread 2.3.4 – waiting for lock


• CASE 1. Lock contention : Thread dump example

"ExecuteThread: '12' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055ae20 nid=23 lwp_id=3722788 waiting for monitor entry [0x2fb6e000..0x2fb6d530]

:at java.lang.ClassLoader.loadClass(ClassLoader.java:255)

:at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Source)at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:252)- locked <0x329fcf50> (a java.lang.Class)

"ExecuteThread: '13' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0055bde0 nid=24 lwp_id=3722789 waiting for monitor entry [0x2faec000..0x2faec530]

at org.apache.axis.utils.XMLUtils.getSAXParser(XMLUtils.java:247)- waiting to lock <0x329fcf50> (a java.lang.Class) :

"ExecuteThread: '15' for queue: 'weblogic.kernel.Default'" daemon prio=10 tid=0x0061dc20 nid=26 lwp_id=3722791 waiting for monitor entry [0x2f9ea000..0x2f9ea530]

at org.apache.axis.utils.XMLUtils.releaseSAXParser(XMLUtils.java:283)- waiting to lock <0x329fcf50> (a java.lang.Class)at


• CASE 2. Dead lock– CWC “Circular waiting condition”– Commonly it makes WAS hang up– Remove CWC by changing lock direction– Some kind of JVM detects dead lock automatically

sychronized methodA(){ call methodB();}sychronized methodB(){ call methodC();}sychronized methodC(){ call methodA();}

< sample code >


• CASE 2. Dead lock

Thread1

Sychronized MethodA

Thread2

Sychronized MethodB

Thread3

Sychronized MethodC

Threads

Time

Thread 1

Thread 2

Thread 3

Circular waiting condition


• CASE 2. Dead lockFOUND A JAVA LEVEL DEADLOCK:----------------------------"ExecuteThread: '12' for queue: 'default'": waiting to lock monitor 0xf96e0 (object 0xbd2e1a20, a weblogic.servlet.jsp.JspStub), which is locked by "ExecuteThread: '5' for queue: 'default'""ExecuteThread: '5' for queue: 'default'": waiting to lock monitor 0xf8c60 (object 0xbd9dc460, a weblogic.servlet.internal.WebAppServletContext), which is locked by "ExecuteThread: '12' for queue: 'default'" JAVA STACK INFORMATION FOR THREADS LISTED ABOVE:------------------------------------------------Java Stack for "ExecuteThread: '12' for queue: 'default'":========== at weblogic.servlet.internal.ServletStubImpl.destroyServlet(ServletStubImpl.java:434) - waiting to lock <bd2e1a20> (a weblogic.servlet.jsp.JspStub) at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2377) at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207) : Java Stack for "ExecuteThread: '5' for queue: 'default'":========== at weblogic.servlet.internal.WebAppServletContext.removeServletStub(WebAppServletContext.java:2370) at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:207) at weblogic.servlet.jsp.JspStub.prepareServlet(JspStub.java:154) - locked <bd2e1a20> (a weblogic.servlet.jsp.JspStub) at weblogic.servlet.internal.ServletStubImpl.getServlet(ServletStubImpl.java:368) : Found 1 deadlock.========================================================

Deadlock auto detect in Sun JVM


• CASE 2. Dead lock

"ExecuteThread-6" (TID:0x30098180, sys_thread_t:0x39658e50, state:MW, native ID:0xf10) prio=5at oracle.jdbc.driver.OracleStatement.close(OracleStatement.java(Compiled Code))at weblogic.jdbc.common.internal.ConnectionEnv.cleanup(ConnectionEnv.java(Compiled :

"ExecuteThread-8" (TID:0x30098090, sys_thread_t:0x396eb890, state:MW, native ID:0x1112) prio=5at oracle.jdbc.driver.OracleConnection.commit(OracleConnection.java(Compiled Code))at weblogic.jdbcbase.pool.Connection.commit(Connection.java(Compiled Code)) :

sys_mon_t:0x39d75b38 infl_mon_t: 0x39d6e288: oracle.jdbc.driver.OracleConnection@310BC380/310BC388: owner "ExecuteThread-8" (0x396eb890) 1 entry 1)

Waiting to enter: "ExecuteThread-10" (0x3977e2d0) "ExecuteThread-6" (0x39658e50)

sys_mon_t:0x39d75bb8 infl_mon_t: 0x39d6e2a8: \ oracle.jdbc.driver.OracleStatement@33AA1BD0/33AA1BD8: owner "ExecuteThread-6" (0x39658e50) 1 entry 2)

Waiting to enter: "ExecuteThread-8" (0x396eb890

Deadlock trace in AIX JVM


• CASE 3. Wait for IO Response– Situation in waiting for IO response (DB,Network)– Commonly occurred caused by DB locking or slow

response

Threads

Time

Wait for IO response


• CASE 3. Wait for IO Response

"ExecuteThread: '42' for queue: 'default'" daemon prio=5 tid=0x3504b0 nid=0x34 runnable [0x9607e000..0x9607fc68] at java.net.SocketInputStream.socketRead(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:85) at oracle.net.ns.Packet.receive(Unknown Source) at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.net.ns.NetInputStream.read(Unknown Source) at oracle.jdbc.ttc7.MAREngine.unmarshalUB1(MAREngine.java:730) at oracle.jdbc.ttc7.MAREngine.unmarshalSB1(MAREngine.java:702) at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:373) at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1427) at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:911) at oracle.jdbc.driver.OracleStatement.doExecuteQuery(OracleStatement.java:1948) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:2137) at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:404) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:344) at weblogic.jdbc.pool.PreparedStatement.executeQuery(PreparedStatement.java:51) at weblogic.jdbc.rmi.internal.PreparedStatementImpl.executeQuery(PreparedStatementImpl.java:56) at weblogic.jdbc.rmi.SerialPreparedStatement.executeQuery(SerialPreparedStatement.java:42) at com.XXXX 생략 at ……..


• CASE 4. High CPU usage– When monitoring CPU usage by top,glance. WAS

process consumes a lot of CPU - almost 90~100%– Find it using tools below

• Sun : prstat,pstack,thread dump• AIX : ps –mp,dbx,thread dump• HP : glance,thread dump• See a document “How to find bottleneck in J2EE

application “ in http://www.j2eestudy.co.kr

http://www.j2eestudy.co.kr/


• CASE 4. High CPU usage– Example in HP UX

HP glance

1. Start “glance”2. Press “G” and enter the

WAS PID3. Find the TID that

consumes a lot of CPU resource

4. Get a thread dump5. Find the thread in Thread

dump that has a matched LWID with this TID

6. And find the reason and fix it


• How to prevent slow down & hang up– Use clustering (use more than 2 WAS instance)– Make “Dedicate thread group” for each work– Plz make “Good Application that has no risk!!”


• Summarize– Thread dump is snapshot of Java Application.– If u meet a slowdown situation, Get a thread dump.– Analysis thread dump.– And fix it..

Slow down in other resources


User AP

WAS

JVM DBMSWeb Server

Network

Client

Legacy


• Slow down in JVM– JVM memory layout– Minor GC,Full GC


• Slow down in JVM– Full GC takes a lot of time and it freeze the WAS.– In the high load, after full GC finished, request can be

gathered suddenly.– Reduce un-needed memory usage– JVM Memory tuning is needed


• Slow down in WebServer– Webserver doesn’t make a slow down almost time– Use Keep-Alive between WebServer and Client※ Apache Keep Alive

http://httpd.apache.org/docs-2.0/mod/core.html#keepalive






• Slow down in network– Get a enough bandwidth– Tuning TCP/IP Kernel parameter in OS some times

it brings slowdown • Tuning buffer size• Connection abort secs etc.• This parameter is provided from WAS vendor※ WebLogic TCP/IP Kernel Parameterhttp://e-docs.bea.com/wls/docs81/plugins/plugin_params.html#1143055

http://e-docs.bea.com/wls/docs81/plugins/plugin_params.html#1143055












• Slow down in DBMS– Tune DBMS (It’s not a issue of this session)– We can profiling execute time of SQL

※ http://eclipse.new21.org/phpBB2/viewtopic.php?printertopic=1&t=380&start=0&postdays=0&postorder=asc&vote=viewresult

※ http://www.j2eestudy.co.kr/qna/bbs_read.jsp?table=j2ee&db=qna0104&id=5&searchBy=subject&searchKey=sql&block=0&page=0

Common mistake in J2ee developer


• Java Programming• Servlet/JSP Programming• JDBC Programming• EJB Programming


• Common mistake in Java Programming– Do not use String class in complex string operation– Close socket and file after use it– Use synchronized block appropriately


• Common mistake in Servlet/JSP programming– JSP buffer size

• <% page buffer=“128kb” %>

– Member variable • <%! Parameter param; %>• Use synchronized mehtod

– Out of memory in file uploading


• Common mistake in JDBC programming– Close stmt, pstmt when using Connection Pooling– Out of memory in Bigsize query result

• Do not save the result in memory (Vector etc..)

– Connection Leak

conn = getConnection();try{ do something}catch(Exception e){ dosomething(); }}finally{ if(stmt!=null) stmt.close(); if(conn!=null) conn.close();}


• Common mistake in EJB programming– Study and use EJB in appropriate place– Remove EJB after use it (SFSB)– Reduce JNDI look up

• Caching Home Interface

– Do not use!! Rollback & Commit in CMT.– Do not!! use hot deploy in production mode

Thanx

[email protected]

Technology

Analysis bottleneck in J2EE application