Java ee7 with apache spark for the world's largest credit card core systems, and beyond

Oct 28, 2017

Hirofumi Iwasaki

Arshal Ameen

Rakuten Card Co., Ltd.

2

Hirofumi Iwasaki @HirofumiIwasaki

Group Manager

Technology Strategy Group, System Strategy Department,

Rakuten Card Co., Ltd.

Career

Planning, designing & implementation of huge enterprise systems for financial,

manufacturing and public systems with Java EE in Japan over 18 years.

Opus, Lectures, etc.

Conferences: OOW 2017, 2014, JavaOne 2017, 2015, 2014, Java Day Tokyo

2014-2015, Rakuten Tech Conference 2013-2016, etc.

3

Part 1

Java EE 7 with Apache Spark for the

World’s Largest Credit Card Core

Systems

Old Core Systems and Issues to

Solve

Strategies and Designs of new Core

Systems

Development and Results

Part 2

New Core Systems and Beyond

– Java EE 8 and Payara

4

Top-level credit card

company in Japan

Core of Rakuten eco

systems.

3rd position of total

transaction volume in 2016.

Growing rapidly.

5

Core Systems

Web Systems

External Systems

Intra Systems

6

Mainframe

Old architecture – over 20 years

High cost structure

Capacity and performance

limitation – no scale out

Low maintainability with piled

programs and old architecture

database "NDB"

Risk against vendor locked-in

Limitation of the security for the

significant data

7

Business

Operations

Development

8

Old New

Cannot scale-out Apply scale-out enabled

architecture, with Oracle RAC

and clustered WebLogic server.

Low connectivity to other

systems

Apply Java EE and latest

protocol.

Less security management on

data

Apply Oracle database

security options.

No latest auto testing

environment

Introduce latest auto testing

environment.

9

Old New

No local development Apply Java EE and Oracle DB

for local dev.

Hard to understand because

of its old architecture

Apply latest Java EE for its

basement.

Poor version control systems Introduce git server and issue

track systems.

No development community Apply Java EE and join open

community.

10

Old New

Poor automated operations Introduce Jenkins and

automations.

Manual error monitoring Include Zabbix monitoring to

cover the new core system.

Difficult to pin-point cause of

error

Use standard Java tools: stack

traces, Flight Recorder, etc.

Tons of unused codes Apply automated source code

analyzing tool.

11

Phase of the improvement – 3.0

1.0 Initial phase

2.0 In-house

development

3.0 Standardization

4.0Data Optimized

Outsource based,

just started.

Vendor locked-in.

In-house

development,

differentiate with

lower costs and

faster delivery.

Standardized

system

architecture, both

for hardware and

software.

Overwhelming

differentiation,

with enabling

architecture for

customer centric

service.

Achieved Next

Current Standard

Architecture

12

2013 2017

Web systems

ExpandJava EE 6

Core

Systems

Java EE 7

13

Oracle Exalogic

+ Exadata + ZFS Servers

13

Mainframe

Old New

Core

Systems

1414

Oracle Exalogic

+ Exadata + ZFS Servers

Oracle Cloud Machine

(On premise private cloud)

For temporarily

request spiking

Low-Cost

Temp

Resource

New

Core

Systems

15

Financial de-facto standard

Java EE compliant.

Matured, from 1997.

Financial de-facto standard

ISO/IEC 9075 SQL compliant

Matured, from 1983.

COBOL

Network

DB

App Server

Database

Old New

WebLogic Server

Oracle Database

16

Hardware, OS, app

server independent,

vendor free.

Mainframe,

Japanese COBOL,

vendor locked-in

Old New

WidflyPayara

WebLogic

hp-ux

AIXSolarisLinux

Windows

macOS

WebSphere

Java EE 7

17

Japanese COBOL

Source code

Source code

Custom made

source code

converter

Convert from Japanese

COBOL to Java EE

Keep original core

business logic

Java EE

18

From Web Systems,

For New Logic

COBOLFrom Old System,

converted to Java

Ease of migration, resource re-use

Introduce power of Java EE

Introduce converter from YPS to Java

“Dual Source Architecture”

Japanese

COBOL

Japanese source code

Almost abandoned

No books, no community

Old New

Java EE

19

BIG

-IP

Real-time Servers

(WebLogic)

Batch Servers

(Spark & Java)

Façade

Rich clients Façade

Façade

Intranet

External

Intra

Exadata

Mail

Form

BIG

-IP

Façade

BIG

-IPExternal

customers

Scheduler

Core

Busin

ess L

ogic

AP

Is

Operation

terminal

Web

bro

wse

r

Old NewJava EE

Java

20

New Database

ISAM

VSAMNDB

Java

Business Logics

Japanese COBOL

business logics

Common Module

Data Accessor Common Module

Database Accessor

Migrate

Web

Database

Old New

21

New Database

- Record Key

(Unique)

- Record Key

(Not Unique)

A_RDB_TABLE

----------------------------

- PRIMARY KEY

- OTHER COLUMN

Add unique

index.

Add index

only.

Old New

ISAM/VSAM

22

3. Run tests

on staging environment

2. Execute auto testing

on several times

1. Register auto test scenarios

Automatic testing

using latest IBM

Rational test software.

Regression test

enabled when

something changed.

Reduce error

possibilities on

production release.

Testing

Server

23

Time

Sequential

Parallel

24

Cluster Node

Cluster Node

Cluster Node

Cluster Node

Cluster Node

Cluster Node

Bootstrap

Share

d M

em

ory

Scheduler

25

1. Making business logic parallel

Independent processing

2. I/O

Data transferred over network

3. Data ordering

Shuffles

26

Good for

Heavy processing

Independent input data records

One input, multiple outputs

Unordered data

Not so great for

Little processing

Dependencies between data records

Merging multiple data sources

27

vs.

28

Next Phase

1.0 Initial phase

2.0 In-house

development

3.0 Standardization

4.0Data Optimized

Outsource based,

just started.

Vendor locked-in.

In-house

development,

differentiate with

lower costs and

faster delivery.

Standardized

system

architecture, both

for hardware and

software.

Overwhelming

differentiation,

with enabling

architecture for

customer centric

service.

Achieved Next

Current Standard

Architecture

29

Arshal Ameen

Web System Development Team Manager

Core Engineer and trainer

Carrier

Java EE expert

App developer (iOS, Android and Win).

Opus, Lectures, etc.

Conferences: JavaOne 2015, 2014, Oracle OpenWorld 2014, Rakuten Tech Conf 2016-2013, JJUG Tokyo(2014)

30

Part 1

Java EE 7 with Apache Spark for the

World’s Largest Credit Card Core

Systems

Old Core Systems and Issues to

Solve

Strategies and Designs of new Core

Systems

Development and Results

Part 2

New Core Systems and Beyond

– Moving on to Payara

31

32

Drop in replacement for glassfish(currently orphaned)

Enhancements to GlassFish, bug fixes, patches to GlassFish server and

dependent libraries

Quarterly community releases

Open source @ www.github.com/payara (RakutenCard is already a code

contributor)

Synched with Glassfish SVN

33

GlassFishPAYARA

EXPERTISE

34

Feature GlassFish Payara

Releases/Upgrades No plan Quarterly

Patches None after Glassfish4.0 Monthly (Licensed)

Quarterly (Community)

Security Fixes Infrequent Instant (Licensed)

ASAP (Community)

Support None ✓

Caching Shoal Shoal/Jcache/Payara scales

Clustering None Hazelcast

Admin command recorder None ✓

SQL logging/Monitoring

logging

None ✓

Healthcheck service None ✓

35

Feature Weblogic Payara

Releases/Upgrades Delayed Quarterly

Security Patches 2016 Security bugs

2 resolved

18 pending

2016 Security bugs

All resolved

Support Cost $$$$ 18750$/6 Processor

Caching OracleWebCache (buggy) Shoal/Jcache/Payara scales

Clustering Vendor specific Hazelcast

Server start time ~10 mins ~ 12 sec

Open source No ✓

Critical Hotfixes (~1hr

support)

No ✓

36

37

Need to setup a clustered domain with auto session replication ????

asadmin create-domain --nopassword --portbase 10000 example-domain

asadmin start-domain example-domain

asadmin -p 10048 create-cluster sample-cluster

asadmin -p 10048 set-hazelcast-configuration --enabled true --dynamic true –

multicastport 5000 --target sample-cluster

asadmin -p 10048 set sample-cluster-config.availability-service.web-container

availability.availability-enabled=true

asadmin -p 10048 set sample-cluster-config.availability-service.web-container

availability.persistence-type=hazelcast

Save this as script and create ∞ domains in a jiffy(technically 33.33ps)

Schedule automatic backups

Restore from a backup or create new from backup

38

39

40

41

42

healthcheck-cpool Connection Pool

healthcheck-cpu CPU usage

healthcheck-gc Garbage collection

healthcheck-heap Heap memory

healthcheck-threads Threads

healthcheck-machinemem Machine storage

asadmin>

Step1: Enable

healthcheck-configure --enabled=true --dynamic=true

Step 2: Configure frequency

healthcheck-configure-service --serviceName=healthcheck-cpu --time=5

--unit=MINUTES

Step3: Configure threshold

healthcheck-configure-service-threshold --thresholdCritical=95 --

thresholdWarning=75 --thresholdGood=60

43

JDBC connection pools

HTTP connection pool

GC activity

Stuck threads

44

45

Hazelcast

Hazelcast’s JCache cached In-memory

WAN replication across multiple DCs

Provides resilience and high availability

Disaster recovery – multiple copies of sessions across

cluster for HA

High Density Memory Store scales

Standardized

46

• Standard Java API for Caching

JSR107

• API and CDI binding

• Supported by Many Cache Providers

• Hazelcast, EhCache, Coherence, Infinispan,

Apache Ignite

• Built in to Payara Server

Uses Hazelcast

47

• In‐Memory DataGrid(IMDG)

• Clustering and highly scalable

data distribution solution forJava

• Distributed Data Structures for Java

• Distributed Hashtable(DHT) and more

48

• Open source(ApacheLicense)

• Superlight,simple,no‐dependency

• Distributed/partioned implementation of map,queue,set,list,lock

and executor service

• Transactional(JCAsupport)

• Dynamic clustering,backup,fail‐over

• Partion/Share data across cluster

• Send/receive messages

• Scaling and load balancing

• Process in parallel on many JVM

49

50