Embed Size (px)
Citation preview
DENISE UNTERWURZACHER • CONFLUENCE ENGINEER • ATLASSIAN
Trash Talk!How to Reduce Downtime by Tuning Garbage
Collection
G A R B A G E C O L L E C T I O N B A S I C S
A C A U T I O N A RY TA L E
T R O U B L E S H O O T I N G
Agenda
T U N I N G B E S T P R A C T I C E S
The defaults are sane
The JVM is built to react adaptively
The knock-on effects can be hard
to predict
Don’t mess with it
EAC - extranet.atlassian.com
Node 3Node 2Node 15.5gb
VM
5.5gb
VM5.5gb
VM
8s
50s
45s
Garbage Collection Basics
Eden Tenured
Young Generation
Old Generation
Survivor+
Survivor TenuredEden
Survivor TenuredEden
Survivor TenuredEden
‘Stop the world’ - stops the
application running to collect garbage
Collects garbage while the
application runs
Serial/Parallel Concurrent
Single or multi-threaded
Serial/Parallel
Good for small heaps
‘Stop the world’ full GCs
G1GC
Needs a larger heap
Still some pauses
No full GCs
Young generation collections are never
concurrent.
Pause is inevitable.
Pause types
To-space allocation
Full GC
Young generation
Mark/remark/cleanup
Serial/parallel, concurrent failover
All collectors
Concurrent
Concurrent
To-space allocation-XX:G1ReservePercent=20
BenchmarkingNumber of pausesLength of pausesTime between pauses
<1s
Garbage collection is hard
Fixes can make things worse
Benchmarking is essential
Lessons learned
Benchmarking and troubleshooting tools
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
Raw verbose logs
GC Viewer
GC Viewer
1GB heap
GC Viewer
Young generation
GC Viewer
Tenured generation
GC Viewer
Server times
GC Viewer
Pause times
GC Viewer
GC times
GC Viewer
< 0.10 s
GC Viewer
GC Viewer
Full GCs
GC Viewer
GC Viewer
GC Viewer
GC Viewer
GC Viewer
When GC fails
OutOfMemory Error - Java Heap Space
Long pauses
Bad tuning - new generation too small
Heap too large
GC times
Long pauses - allocation failure
To-spaceallocationfailure
Full GC
Long pauses - unable to free enough space
Solutions
Long pauses
Heap too large
OOMEsIncrease memory by small increments.
Decrease memory.
Diagnose, benchmark, test.
Best Practices
BenchmarkingPlanChangeMeasureRepeat
Tuning Goals
Footprint
Throughput
LatencyMaximum GC pause times
Time spent serving application requests
The size of the heap
Footprint
Throughput
Pause times
Low pause times + high throughput
Use G1GC and a large heap
Footprint
Throughput
Pause times
Low pause times + low footprint
Use G1GC and a medium sized heap
Footprint
Throughput
Pause times
Low pause times + low footprint
Use G1GC and a medium sized heap
Footprint
Throughput
Pause times
High throughput + low footprint
Use a parallel collector and a small heap
Footprint
Throughput
Pause times
High throughput + low footprint
Use a parallel collector and a small heap
JVM auto-tuning
3. Footprint
2. Throughput
1. LatencyMaximum GC pause times
Time spent serving application requests
The size of the heap
Overtuning
Increasing the heap too much
Parameters that stop the heap auto-
tuning
-XX:MaxNewSize=2g -XX:NewRatio=3
Hard tuning Soft tuning
Young Generation
-XX:GCTimeLimit=10 -XX:GCTimeRatio=9
Hard tuning Soft tuning
-XX:MaxGCPauseMillis=500
Throughput
-XX:SurvivorRatio=4 -XX:G1ReservePercent=20
Hard tuning Soft tuning
Survivor
Tuning is hard. Avoid it.
If you have to tune, tread carefully.
Have a plan.
Benchmark&Change&Measure& Repeat.
What you can do when you get home
Be minimal!
Think about goals
Review current state
Where to get help with GC
Atlassian Support
Oracle’s docs
Atlassian’s docshttps://confluence.atlassian.com/x/aACCGw
http://tinyurl.com/caq6y7h
https://support.atlassian.com
Thank you!
DENISE UNTERWURZACHER • CONFLUENCE ENGINEER • ATLASSIAN
