Embed Size (px)
Citation preview
Are Your V8 Garbage Collection Logs Speaking to You? Joyee Cheung, Alibaba Cloud
What Prompted us to Look into themNode.js at Alibaba
What Prompted Us to Look Into ThemWhy GC Logs?
v Java is the mainstream at Alibabav Chose Node.js for productivity, but it’s meaningless without availabilityv Usually not the bottleneck, but with a huge traffic, every optimization counts
v Garbage collection is important in long-running, data-intensive serversv High CPU usage: hurts responsivenessv High memory usage: affects other processesv Memory leaks: misled by developers, crashes the process(OOM)v Java developers know their VM and its GCv …
v More about Node.js at Alibabav https://www.linux.com/blog/can-nodejs-scale-ask-team-alibaba
What Prompted Us to Look Into ThemWhy GC Logs?
v Tools from V8/Chromium are more about optimizing frontend applications
CPU profiles aremore about function calls,less about objects
v Heap snapshots and timelines are more about allocations, less about collections
v Lack of high-level overview of the heap, too much details
Hard to use in staging/production environment
What Prompted Us to Look Into ThemWhy GC Logs?
v V8 provides garbage collection logs v Not documented, not available in the devtools, need to use command line argsv Higher-level than heap profiles, lower-level than the number of memory usuagev Help verify the fixes by displaying patterns of GCv Not silver bullet, but nice to have
v This talk is based on the versions of V8 used by Node v4 & v6 LTSv LTS plan stablizes the V8 versions for each LTS and its GC logs
An Introduction to V8 GCWhat is Garbage Collection
An Introduction to V8 GCHow Most GCs are triggered
An Introduction to V8 GCStop-The-World, Parallel, Concurrent GC
An Introduction to V8 GCGenerational GC
An Introduction to V8 GCHeap Organization
An Introduction to V8 GCHeap Limits(Default)
v deps/v8/src/heap/heap.h: Heap::Heap()
16MB1MB1.4GB0.7GB
512MB
v Can be configured via command line argumentsv --max_old_space_size, --max_semi_space_size, --max_executable_space_size
An Introduction to V8 GCNew Generation – Scavenge GC
An Introduction to V8 GCOld Generation – Mark-Sweep/Compact GC
An Introduction to V8 GCGC Pauses in V8
How to Get The GC Logs
v Plenty of options (node --v8-options| grep gc)v We’ll be focusing on
v --trace_gcv Print a line briefly describing when, how, what and why for each GC
v --trace_gc_verbosev Combined with --trace_gc, summarizes about each space and the
heap after each GCv --trace_gc_nvp
v Print name-value pairs describing how each step of GC goesv In case you do want to read the code
v deps/v8/src/heap, start with gc-tracer.cc
How to Read The GC Logs
v --trace_gcWHERE: [PID:Isolate]
WHEN: Time since the process has started
WHAT: Type of GC
HOW MUCH:Size of all objects(Size of memory allocated from OS)
HOW LONG:GC pause/Time spent on external memory
WHY
How to Read The GC Logs
v --trace_gc_verbose && --trace_gc
Memory V8 allocated from the OS
All memory reserved for this space (insidededicated pages)
Maintained by external C++ code
Adjust heap growing factor and the allocation limit that triggers the next incremental marking
How to Read The GC Logs
v --trace_gc_nvp ms=Mark-Sweep, s=Scavenge
What We’ve Found
v Tons of Scavenge GC in New Space eating up the CPUv Cache misses caused by inappropriate caching strategiesv Excessive deep clones/object merging
v Large Object Space keeps growingv Serialization/deserialization of huge RPC payload
v Code Space keeps growingv Bugs in templating engines
v Old Space keeps growingv Closures in long connections
Tools
v Our parserv Up on NPMv For people who already have
applications running in production
https://github.com/aliyun-node/v8-gc-log-parser
http://alinode.aliyun.com/blog(translation coming soon)
Visualization coming soon!
Tools
v --trace_gc_object_statsv Feed the output to https://mlippautz.github.io/v8-heap-stats/
v Requires Node >= v7v More about the memory usuage, less about GC
v --track_gc_object_statsv Combined with --log_timer_events, writes a log file to diskv Can be viewed with chrome://tracing (Incomplete???)v Pending PR: https://github.com/nodejs/node/pull/9304
An Introduction to V8 GCAccurate GC
