1 Copyright © 2013, Oracle and/or its affiliates. All ...€¦ · Therap Services, LLC •Documentation and Communication Software for MR/DD •EHR for the DD industry is the closest

Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 12 of the corporate presentation template1

JVM Diagnostics: Java Profiling in Production Environments

Mojahedul Hoque Abul Hasanat, Therap Services

Michael Avrahamov, Oracle

Avi Huber, Oracle

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3

The preceding is intended to outline our general product direction. It is intended

for information purposes only, and may not be incorporated into any contract.

It is not a commitment to deliver any material, code, or functionality, and should

not be relied upon in making purchasing decisions. The development, release,

and timing of any features or functionality described for Oracle’s products

remains at the sole discretion of Oracle.


Program Agenda

JVM issues in the production environment

JVM Diagnostics - Java profiling through sampling

Three common scenarios

Therap Services - Customer use case

JVM Diagnostics is an integral part of EM


Oracle Enterprise Manager 12c Overview

Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6

Oracle Enterprise Manager 12c

Complete Cloud

Lifecycle Management

Integrated Cloud

Stack Management

Business-Driven Application

Management

Self-Service IT Simple and Automated Business Driven| |


JVM Issues in the Production Environment



Intermittent slow response time

Time outs

Heap size grows over time

Long GC runs

High CPU

Out of memory exceptions

System freeze

Crashes

Problems you will see in production that you didn’t see in QA



Limited access to host and AppServer

– No direct access to cmd line, log files, config files

– Need someone else to change log level, run a

thread or heap dump

Application cannot be stopped or put at risk

– Can’t restart the JVM

– Can’t simply try a solution

It is harder to diagnose a problem in production

The problem happened earlier, can’t reproduce at will

Can’t run the code in a debugger


JVM issues in the production environmentStill, you must solve these issues

Search logs

Restart server

Try to reproduce in a QA environment

Increase host resources

Add instrumentation (log messages)

Long time to resolution

Application downtime

Repeat problems



Search logs

Restart server






Repeat problems

Buy a BCI product

Long pre-production testing

Restart application for deployment

See high overhead

Change configuration


Performance impact

High Cost

No results? Get a third party diagnostics product



Search logs

Restart server






Repeat problems

Buy a BCI product

Long pre-production testing

Restart application for deployment

See high overhead

Change configuration


Performance impact

High Cost

Or – start using JVM Diagnostics… today!


JVM Diagnostics - Java Profiling Through Sampling


JVM Diagnostics

Thread Sampling

– Use JVMTI to sample the heap

– Categorize threads by wait states

– Examine local members to add thread context

– Store active thread data in EM repository

Heap Analysis

– Facilitate heap dump collection and

visualization

– Analyze heap dump files

– Live heap analysis

Profiling through sampling


JVM DiagnosticsArchitecture

Java Container

JAM Agent WAR

Java Container

JAM Agent WAR

EM Cloud Control Domain

JVMD ManagerEnterprise Manager 12c

Historical Real-time

Enterprise Manager

Repository

JVM Diagnostics Manager

EJBHTTP(s)

Port

JVM

JVMD Agent WAR

Enterprise Manager

Console

Application

Database

Monitored Application


Live Demo

Three scenarios:

End users experience slow response time

The application seems to be slower today

We have some memory problems


Therap Service – Customer use case

JVMD @ Therap ServicesHow JVMD Plays a Vital Role in Therap Applications

Mojahedul Hoque Abul Hasanat

CTO, Therap Services

Therap Services, LLC

• Documentation and Communication Software for MR/DD• EHR for the DD industry is the closest for describing us

• Niche segment in the health sector

• Improve quality of life for people with DD by improving efficiency of delivery through communication

• SaaS business model

• 150K+ active users

• 1000+ providers in 48 states

• State customers• Extensive usage for DD in DHS ND and DHHS NE

• 150+ employees

• Based in CT, dev center in Bangladesh

JVMD @ Therap - OOW 2013 19

The Application

• The application is our business

• 1M+ lines of code

• 60+ modules

• 1M+ sustained HTTP requests/hour

• 30K+ peak requests/minute

• 6000+ concurrent users

• Based on JEE and the Spring Framework

• Hibernate

• Seam

• GRAILS


Delivery Platform

• 2 identical sites in two states

• Primary hosts (per site):

• 4 WebLogic application servers in cluster

• 1 Memory based data server (in-house, java)

• 1 Oracle database server

• 1 NetApp storage (SAN)

• 1 F5 Load balancer

• Supporting hosts

• Use Dyn for site high availability

• Data replication with Oracle Golden Gate


What Matters

• Availability

• Application is used 24x7

• Application use is critical to the business of our customers

• Performance

• A user needs to spend as little time as possible in our application

• Most users use it daily, multiple times

• Data integrity

• Fast development turnaround


How JVMD Helped Us


The log4j bottleneck

• During load testing, we could not increase load beyond a certain point

• CPU load was low

• JVMD showed us something that we could hardly believe

• Many threads were contending for lock for writing to the log file

• The contention only shows up at high loads

• Used JVMD heavily to find the best logging backend and the best

configuration


log4j…


log4j…


log4j...


Logging Bottleneck

• 2000 logs/s – log4j with sync appender

• 4000 logs/s – logback with sync appender

• 8000 logs/s – logback with async appender

• 12000 logs/s – log4j v2 with sync appender


Unexpected Top Method

• Noticed a JMS listener in the top method list

• In production!

• Did not show up during synthetic load testing

• We forgot to add a “message selector” on the listener


Top Method…


The Slow Library

• A library call for producing JSON showed on the top method list

• JSON is needed for AJAX

• It was totally unexpected

• The library was old and inefficient

• Replaced it with a newer and more efficient library


The Slow Library…


The Slow Library…


The Slow Library…


In-efficient Network Write

• Initially discovered in production through JVMD

• There were instances of high network waits

• Methods a certain module in the application showed up in the top list during

the high network wait periods

• Discovered a 3 level loop that writes data

• Further inspection through JProfiler confirmed it


In-efficient Network Write…


In-efficient Network Write…


Automatic Thread Snapshots

• Previously, relied on kill -3

• Manual, missed dumps at crucial moments

• Now, JVMD takes thread snapshots when an abnormal thread state is

reached on any WebLogic server

• Combined with auto-restart from WebLogic, eliminated unplanned downtime






Drilldown from RUEI

Drilldown from BTM

Drilldown to DB Diag.

Drill-up from SQL Diag.

ECID search

Middleware Diagnostics Advisor

Incident and Alerts

Corrective Actions

Dashboard regions

Access management

EM workflows and integrations:

Cloud Application Foundation -

WebLogic, Coherence

GlassFish, iAS,

SOA - OSB, SOA infrastructure

IDM, EBIEE

Oracle Apps - Fusion Apps, Ebiz,

PeopleSoft, Retek, …

Diagnostics for the Oracle stack:


Q&A


Enterprise Manager Resources Oracle.com: http://www.oracle.com/enterprisemanager

– Enterprise Manager Customer References

OTN: http://www.oracle.com/technetwork/oem

– Demos on Demand

– Enterprise Manager Forums

Enterprise Manager Training from Oracle University

– Oracle Enterprise Manager 12c Learning Library

– Oracle Enterprise Manager Cloud Control 12c: Install and Upgrade

– Using Oracle Enterprise Manager Cloud Control 12c Ed 1

– Oracle Enterprise Manager 12c: Overview Bundle Self-Study

– Oracle Enterprise Manager 12c: Management Bundle Self-Study

Partners

– List of Enterprise Manager Specialized Partners

– Oracle Enterprise Manager OPN Knowledge Zone

– Oracle Enterprise Manager OPN Specialization

– Oracle Enterprise Manager Application Quality Management OPN Specialization

– IOUG Oracle Enterprise Manager Special Interest Group

Social Media

– Twitter, Facebook, YouTube, Linkedin, Blog

http://www.oracle.com/enterprisemanager

http://www.oracle.com/us/corporate/customers/customersearch/index.html?xProductServices=12/44

http://www.oracle.com/technetwork/oem

http://www.oracle.com/technetwork/oem/em12c-screenwatches-512013.html

https://forums.oracle.com/forums/category.jspa?categoryID=70

https://apex.oracle.com/pls/apex/f?p=44785:141:104974085335854::::P141_PAGE_ID,P141_SECTION_ID:16,92

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D73244GC10

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D74510

http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getCourseDesc?dc=D74511

https://solutions.oracle.com/scwar/sc/index.html?canned_search=PP_SPEC_OENTM

https://competencycenter.oracle.com/opncc/spec_status.cc?spec_id=18

https://competencycenter.oracle.com/opncc/spec_status.cc?spec_id=16

http://www.ioug.org/p/cm/ld/fid=148&gid=440

http://twitter.com/

https://www.facebook.com/oracleenterprisemanager

http://www.youtube.com/OracleEnterpriseMgr

http://www.linkedin.com/groups?about=&gid=2688744

http://blogs.oracle.com/oem/



Documents

1 Copyright © 2013, Oracle and/or its affiliates. All ...€¦ · Therap Services, LLC •Documentation and Communication Software for MR/DD •EHR for the DD industry is the closest