Java without the Jitter - Azul Systems, Inc. · 2020-01-11 · Most old dead space can be reclaimed without moving it [e.g. CMS] track dead space in lists, and reuse it in place But

©2012 Azul Systems, Inc.

Java without the Jitter

Gil Tene, CTO & co-Founder, Azul Systems

1Thursday, June 21, 12


About me: Gil Tene

co-founder, CTO @Azul Systems

Have been working on “think different” GC approaches since 2002

Created Pauseless & C4 core GC algorithms (Tene, Wolf)

A Long history building Virtual & Physical Machines, Operating Systems, Enterprise apps, etc... * working on real-world trash compaction issues, circa 2004



About Azul

We make scalable Virtual Machines

Have built “whatever it takes to get job done” since 2002

3 generations of custom SMP Multi-core HW (Vega)

Now Pure software for commodity x86 (Zing)

“Industry firsts” in Garbage collection, elastic memory, Java virtualization, memory scale

V

C4



About Azul Supporting mission-critical deployments around



The Zing platform



Zing

A JVM for Linux/x86 servers

ELIMINATES Garbage Collection as a concern for enterprise applications

Decouples scale metrics from response time concerns

Transaction rate, data set size, concurrent users, heap size, allocation rate, mutation rate, etc.

Leverages elastic memory for resilient operation



The Zing PlatformFocused on application benefits

If you want to:

F Improve response times

F Increase transaction rates

F Increase concurrent users

F Forget about GC pauses

F Elastically grow and shrink...

F Gain production visibility



Java runtimes have some common limitations

Responsiveness

Scale and Complexity

Rigid, non-elastic, and inefficient

Sensitivity to load, fragility

Poor production time visibility

These are “pre-existing conditions”



Java limitations are getting in the way

Java is not making effective use of industry trends

New hardware with lots of cores, lots of memory

Virtualization

Cloud infrastructure

Elasticity

All we seem to be doing is placing more and more tiny instances on growing infrastructure



Memory use How many of you use heap sizes of:

F more than ½ GB?

F more than 1 GB?

F more than 2 GB?

F more than 4 GB?

F more than 10 GB?

F more than 20 GB?

F more than 50 GB?



Reality check: servers in 2012

Retail prices, major web server store (US $, May 2012)

Cheap (≈ $1/GB/Month), and roughly linear to ~1TB

10s to 100s of GB/sec of memory bandwidth

16 vCore, 96GB server ≈ $5K



32 vCore, 1TB server ≈ $35K



How much memory do applications need?

“640KB ought to be enough for anybody”

WRONG!

So what’s the right number?6,400K?64,000K?640,000K?6,400,000K?64,000,000K?

There is no right number

Target moves at 50x-100x per decade

“I've said some stupid things and

some wrong things, but not that.

No one involved in computers

would ever say that a certain

amount of memory is enough for

all time …” - Bill Gates, 1996



“Tiny” application history

100KB apps on a ¼ to ½ MB Server

10MB apps on a 32 – 64 MB server

1GB apps on a 2 – 4 GB server

??? GB apps on 256 GB

Assuming Moore’s Law means:

“transistor counts grow at ≈2x every ≈18 months”

It also means memory size grows ≈100x every 10 years

2010

2000

1990

1980

“Tiny”: would be “silly” to distribute

Application Memory Wall



Framing the discussion:Garbage Collection at modern server scales

Modern Servers have 10s and 100s of GB of memory

Each modern x86 core (when actually used) produces garbage at a rate of ¼ - ½ GB/sec +

Servers are dramatically “larger” than [typical] JVMs

Monolithic stop-the-world operations inside JVMs (such as Stop-the-world GC) are the root cause of current responsiveness and scale limitations in Java applications



Some GC terminology



A Basic Terminology example:What is a concurrent collector?

A Concurrent Collector performs garbage collection work concurrently with the application’s own execution

A Parallel Collector uses multiple CPUs to perform garbage collection



A Concurrent Collector performs garbage collection work concurrently with the application’s own execution

A Parallel Collector uses multiple CPUs to perform garbage collection

Classifying a collector’s operation

An Incremental collector performs a garbage collection operation or phase as a series of smaller discrete operations with (potentially long) gaps in between

A Stop-the-World collector performs garbage collection while the application is completely stopped

Mostly means sometimes it isn’t (usually means a different fall back mechanism exists)



What’s common to allprecise GC mechanisms?

Identify the live objects in the memory heap

Reclaim resources held by dead objects

Periodically relocate live objects

Examples:

Mark/Sweep/Compact (common for Old Generations)

Copying collector (common for Young Generations)



History of GC: Delaying the inevitable

Delay tactics focus on getting “easy empty space” firstThis is the focus for the vast majority of GC tuning

Most objects die young [Generational]So collect young objects only, as much as possible

But eventually, some old dead objects must be reclaimed

Most old dead space can be reclaimed without moving it [e.g. CMS] track dead space in lists, and reuse it in place

But eventually, space gets fragmented, and needs to be moved

Much of the heap is not “popular” [e.g. G1, “Balanced”]A non popular region will only be pointed to from a small % of the heap

So compact non-popular regions in short stop-the-world pauses

But eventually, popular objects and regions need to be compacted



The typical combosin commercial server JVMS

Young generation usually uses a copying collector

Young generation is usually monolithic, stop-the-world

Old generation usually uses Mark/Sweep/Compact

Old generation may be STW, or Concurrent, or mostly-Concurrent, or Incremental-STW, or mostly-Incremental-STW



HotSpot™ ParallelGCCollector mechanism classification

Monolithic Stop-the-world copying NewGen

Monolithic Stop-the-world Mark/Sweep/Compact OldGen



HotSpot™ ConcMarkSweepGC (aka CMS)Collector mechanism classification

Monolithic Stop-the-world copying NewGen (ParNew)

Mostly Concurrent, non-compacting OldGen (CMS)Mostly Concurrent marking

Mark concurrently while mutator is running

Track mutations in card marks

Revisit mutated cards (repeat as needed)

Stop-the-world to catch up on mutations, ref processing, etc.

Concurrent Sweeping

Does not Compact (maintains free list, does not move objects)

Fallback to Full Collection (Monolithic Stop the world).Used for Compaction, etc.



HotSpot™ G1GC (aka “Garbage First”) Collector mechanism classification

Monolithic Stop-the-world copying NewGen

Mostly Concurrent, OldGen markerMostly Concurrent marking

Stop-the-world to catch up on mutations, ref processing, etc.

Tracks inter-region relationships in remembered sets

Stop-the-world mostly incremental compacting old gen Objective: “Avoid, as much as possible, having a Full GC…”

Compact sets of regions that can be scanned in limited time

Delay compaction of popular objects, popular regions

Fallback to Full Collection (Monolithic Stop the world).Used for compacting popular objects, popular regions, etc.



We need to solve the right problems

Focus on the causes of the “Application Memory Wall”Scale is artificially limited by responsiveness

Responsiveness must be unlinked from scaleHeap size, Live Set size, Allocation rate, Mutation rate

Responsiveness must be continually sustainableCan’t just delay or ignore “rare” events

Eliminate all Stop-The-World FallbacksAt modern server scales, any STW fall back is a failure


One way to deal with Monolithic-STW GC



Azul’s “C4” Garbage Collector Continuously Concurrent Compacting Collector

Concurrent, compacting new generation

Concurrent, compacting old generation

Concurrent guaranteed-single-pass markerOblivious to mutation rate

Concurrent ref (weak, soft, final) processing

Concurrent CompactorObjects moved without stopping mutator

References remapped without stopping mutator

Can relocate entire generation (New, Old) in every GC cycle

No stop-the-world fallbackAlways compacts, and always does so concurrently



Sample responsiveness improvement

๏ SpecJBB + Slow churning 2GB LRU Cache๏ Live set is ~2.5GB across all measurements๏ Allocation rate is ~1.2GB/sec across all measurements



Sustainable Throughput:The throughput achieved while safely maintaining service levels

UnsustainableThroughout



Instance capacity test: “Fat Portal”HotSpot CMS: Peaks at ~ 3GB / 45 concurrent users

* LifeRay portal on JBoss @ 99.9% SLA of 5 second response times



Instance capacity test: “Fat Portal”C4: still smooth @ 800 concurrent users



Some fun with jHiccup



Incontinuities in Java platform execution

0"

200"

400"

600"

800"

1000"

1200"

1400"

1600"

1800"

0" 200" 400" 600" 800" 1000" 1200" 1400" 1600" 1800"

Hiccup&Dura*on&(msec)&

&Elapsed&Time&(sec)&

Hiccups"by"Time"Interval"

Max"per"Interval" 99%" 99.90%" 99.99%" Max"

0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=1665.024&

0"

200"

400"

600"

800"

1000"

1200"

1400"

1600"

1800"


&

&

Percen*le&

Hiccups"by"Percen@le"Distribu@on"



A classic look at response time behavior

Key Assumption: Response time is a function of load

source: IBM CICS server docuementation, “understanding response times”

Average?

Max?

Median?

90%?

99.9%



Response time over time

When we measure behavior over time, we often see:

source: ZOHO QEngine White Paper: performance testing report analysis

“Hiccups”



jHiccup

A tool for capturing and displaying platform hiccups

Records any observed non-continuity of the underlying platform

plots results in simple, consistent format

Simple, non-intrusive

As simple as adding the word “jHiccup” to your java launch line

% jHiccup java myflags myApp

Adds a background thread that samples time @ 1000/sec

Open Source

released to the public domain, creative commons CC0



Telco App Example

0"

20"

40"

60"

80"

100"

120"

140"

0" 500" 1000" 1500" 2000" 2500"



Hiccups&by&Time&Interval&


0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

0"

20"

40"

60"

80"

100"

120"

140"


&

&

Percen*le&

Hiccups&by&Percen*le&Distribu*on&

Hiccups"by"Percen?le" SLA"

Optional SLA

plotting

Max Time per

interval

Hiccup

duration at

percentile

levels



Examples



Idle App on Busy System

0"

10"

20"

30"

40"

50"

60"

0" 100" 200" 300" 400" 500" 600" 700" 800" 900"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=49.728&

0"

10"

20"

30"

40"

50"

60"


&

&

Percen*le&


Idle App on Quiet System

0"

5"

10"

15"

20"

25"

0" 100" 200" 300" 400" 500" 600" 700" 800" 900"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=22.336&

0"

5"

10"

15"

20"

25"


&

&

Percen*le&




Idle App on Quiet System

0"

5"

10"

15"

20"

25"

0" 100" 200" 300" 400" 500" 600" 700" 800" 900"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=22.336&

0"

5"

10"

15"

20"

25"


&

&

Percen*le&


Idle App on Dedicated System

0"

0.05"

0.1"

0.15"

0.2"

0.25"

0.3"

0.35"

0.4"

0.45"

0" 100" 200" 300" 400" 500" 600" 700" 800" 900"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=0.411&

0"

0.05"

0.1"

0.15"

0.2"

0.25"

0.3"

0.35"

0.4"

0.45"


&

&

Percen*le&




EHCache: 1GB data set under load

0"

500"

1000"

1500"

2000"

2500"

3000"

3500"

4000"

0" 500" 1000" 1500" 2000"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=3448.832&

0"

500"

1000"

1500"

2000"

2500"

3000"

3500"

4000"


&

&

Percen*le&




Telco Application

0"

200"

400"

600"

800"

1000"

1200"

1400"

1600"

1800"

0" 200" 400" 600" 800" 1000" 1200" 1400" 1600" 1800"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=1665.024&

0"

200"

400"

600"

800"

1000"

1200"

1400"

1600"

1800"


&

&

Percen*le&




FiServ Pricing Application

0"

100"

200"

300"

400"

500"

600"

700"

0" 2000" 4000" 6000" 8000" 10000" 12000" 14000" 16000"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=636.928&

0"

100"

200"

300"

400"

500"

600"

700"


&

&

Percen*le&

Hiccups"by"PercenCle"DistribuCon"



Obviously, we can use it forcomparison purposes



Oracle HotSpot CMS, 4GB in a 18GB heap

0"

5000"

10000"

15000"

20000"

25000"

30000"

35000"

40000"

45000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=40173.568&

0"

5000"

10000"

15000"

20000"

25000"

30000"

35000"

40000"

45000"


&

&

Percen*le&


Oracle HotSpot CMS, 1GB in an 8GB heap

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=13156.352&

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"


&

&

Percen*le&




Oracle HotSpot ParallelGC, 1GB in 8GB heap

0"

1000"

2000"

3000"

4000"

5000"

6000"

0" 500" 1000" 1500" 2000" 2500" 3000"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=5439.488&

0"

1000"

2000"

3000"

4000"

5000"

6000"


&

&

Percen*le&


Oracle HotSpot G1 ,1GB in 8GB heap

0"

1000"

2000"

3000"

4000"

5000"

6000"

0" 500" 1000" 1500" 2000" 2500" 3000"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=5144.576&

0"

1000"

2000"

3000"

4000"

5000"

6000"


&

&

Percen*le&




Fun with jHiccup




0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=13156.352&

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"


&

&

Percen*le&


Zing 5, 1GB in an 8GB heap

0"

5"

10"

15"

20"

25"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=20.384&

0"

5"

10"

15"

20"

25"


&

&

Percen*le&





0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=13156.352&

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"


&

&

Percen*le&



0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"Max=20.384&

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"


&

&

Percen*le&




Java GC tuning is “hard”…Examples of actual command line GC tuning parameters:

Java -Xmx12g -XX:MaxPermSize=64M -XX:PermSize=32M -XX:MaxNewSize=2g

-XX:NewSize=1g -XX:SurvivorRatio=128 -XX:+UseParNewGC

-XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0

-XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled

-XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=12

-XX:LargePageSizeInBytes=256m …

Java –Xms8g –Xmx8g –Xmn2g -XX:PermSize=64M -XX:MaxPermSize=256M

-XX:-OmitStackTraceInFastThrow -XX:SurvivorRatio=2 -XX:-UseAdaptiveSizePolicy

-XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled

-XX:+CMSParallelRemarkEnabled -XX:+CMSParallelSurvivorRemarkEnabled

-XX:CMSMaxAbortablePrecleanTime=10000 -XX:+UseCMSInitiatingOccupancyOnly

-XX:CMSInitiatingOccupancyFraction=63 -XX:+UseParNewGC –Xnoclassgc …



The complete guide toZing GC tuning

java -Xmx40g



What to expect with Zingin the low latency world

Assuming you don’t have 1000 threads competeing for 10 cores...

“Easily” get your application to <10msec worst case

With some minor tuning, 1-2msec worst case

We can go to below 1 msec worst case...

May require heavy tuning/tweaking

Mileage WILL vary




0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%"

Max=13156.352&

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"


&

&

Percen*le&



0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"Max=20.384&

0"

2000"

4000"

6000"

8000"

10000"

12000"

14000"


&

&

Percen*le&




Oracle HotSpot Zing 5.2

0"

100"

200"

300"

400"

500"

600"

700"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=636.928&

0"

100"

200"

300"

400"

500"

600"

700"


&

&

Percen*le&


0"

100"

200"

300"

400"

500"

600"

700"

0" 500" 1000" 1500" 2000" 2500" 3000" 3500"





0%" 90%" 99%" 99.9%" 99.99%" 99.999%" 99.9999%"

Max=54.528&0"

100"

200"

300"

400"

500"

600"

700"


&

&

Percen*le&




Q & Ahttp://www.azulsystems.com

http://www.azulsystems.com/dev_resources/jhiccup


http://www.azylsystems.com








Documents

Java without the Jitter - Azul Systems, Inc. · 2020-01-11 · Most old dead space can be reclaimed without moving it [e.g. CMS] track dead space in lists, and reuse it in place But