GlassFish Community Feedback

GlassFish usage in Rakuten, Inc.

About me

Name: Makito Hashiyama(@capyogu)

Age: 29

Role: Team manager / In charge of APIs for Rakuten Ichiba

Like: GlassFish, Tomcat, KVS(memcached, Coherence), GAE,

Android…

E-mail: makito.hashiyama@mail.rakuten.com,

hashiyaman@gmail.com

What is Rakuten? E-commerce and Internet company based in Tokyo, Japan

B2B2C e-commerce platform

E-Commerce

eBook

Travel

Other services & businesses

Rakuten Institute of Technology

Development center

Head Office / Regional Headquarters

Head Office

How We Are Using GlassFish

Usage

One of core API service for Rakuten Ichiba

Require high availability

Environment Use on production environment

GlassFish version 3.1.2.2

Our API Oracle

Coherence

GlassFish

Clusters

External

APIs

SOAP/

REST

Client side

Reference implementation of Java EE

Only GlassFish supported JAX-WS as standard in 2007

It was an advantage to evaluate new features earlier

Easy to manage with low cost We need to manage huge cluster without stopping

Cost saving(Weblogic -> GlassFish)

Benefits gleaned from Glassfish

What Worked with GlassFish

Community support(create patches)

https://java.net/jira/browse/GLASSFISH-5200

(If use jvmRoute JSESSIONID cookie is not secure even in

HTTPS)

https://java.net/jira/browse/GRIZZLY-1333

(NetworkAddressValidator will fail when passed property

substitution values)

We have contributed patched to GlassFish community.

Improvement to handle a huge traffic

Rakuten Super Sale

Biggest online sales in Japan

A lot of doorbuster deals(It causes a huge amount of traffic)

Performance bottleneck

External APIs called by our API were slow down

We needed to improve the system at the peek time

Our API

External

APIs delay Slow down Client side

Request

Task

Queue

Worker Thread

Worker Thread

Worker Thread

GlassFish

Worker Thread

Worker Thread

External

APIs

delay

delay

delay

delay

delay

CPU load

was high

Improvement to handle a huge traffic

Client

side

Request

Task

Queue

Worker Thread

Worker Thread

Worker Thread

GlassFish

External

APIs

(1)According to vmstat, ‘run queue’ was very high

(2)Decrease worker threads to keep ‘run queue’ low

(3)As a result, latency increased but throughput was improved

Improvement to handle a huge traffic

Client

side

As a result…

Our API could process over 12,000 transactions / minute

The result showed the high reliability and availability of

GlassFish

Improvement to handle a huge traffic

Resolve issues & challenge to upper goal

Some issues

Instance down due to the full of task queue

Unknown exception on server.log

Challenge to upper goal

org.glassfish.flashlight.impl.client.ReflectiveClientInvoker

java.lang.reflect.InvocationTargetException

CAUSE: java.lang.NullPointerException

id=101

target=org.glassfish.web.admin.monitor.HttpServiceStatsProvider@193e1fc

method=public void org.glassfish.web.admin.monitor.HttpServiceStatsProvider.

connectionAcceptedEvent(java.lang.String,int,java.lang.String)

paramNames=[listenerName, connection, address]

probeIndices=[0, 1, 2]

useProbeArgs=true

hasComputedParams=false