-
7/25/2019 Node Heavy CPU Then Crash
1/30
Marvel - Node Statistics
QUERY
FILTERING
Pinned elasticsearch1-elasticsearch-3-vm
Development Trial an hour ago to 2 minutes ago refreshed every
1d () Dashboards ()
OS CPU
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
JVM HEAP USAGE (%)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
LOAD (1M)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
5
10
15
OS CPU
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
Elasticsearch getsmanually shut downhere.
Elasticsearch startsbecoming unresponsive.
https://es.docketalarm.com/_plugin/marvel/kibana/index.htmlhttps://es.docketalarm.com/_plugin/marvel/kibana/index.html
-
7/25/2019 Node Heavy CPU Then Crash
2/30
OS MEMORY USAGE (%)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
OS LOAD (1M)
21:5006-30
22:0006-30
22:1006-30
22:2006-30
22:3006-30
22:4006-30
0
5
10
15
OS CPU SYS
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
OS CPU STEAL
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
OS SWAP USED
21:50 22:00 22:10 22:20 22:30 22:40
0 B
2 B
-
7/25/2019 Node Heavy CPU Then Crash
3/30
06-30 06-30 06-30 06-30 06-30 06-30
JVM HEAP USED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
16 GB
32 GB
JVM GC YOUNG DURATION (TIME %)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
JVM GC OLD DURATION (TIME %)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
GC YOUNG DURATION (TIME %)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
25
50
75
100
JVM GC YOUNG COUNTS
10
20
30
40
There is 31GB of heapmemory assigned toElasticsearch.
-
7/25/2019 Node Heavy CPU Then Crash
4/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
JVM GC OLD DURATION (TIME %)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
025
50
75
100
JVM GC OLD COUNTS
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.5
1.0
1.5
2.0
2.5
INDICES SEARCH SHARD QUERY RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.5
1.0
1.5
2.0
2.5
INDICES TOTAL SHARD SEARCH QUERY TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
10
20
30
40
INDICES SEARCH SHARD FETCH RATE
0.5
1.0
1.5
-
7/25/2019 Node Heavy CPU Then Crash
5/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
INDICES TOTAL SEARCH SHARD FETCH TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.000
0.0020.004
0.006
0.008
INDICES INDEXING RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
5
10
15
INDICES TOTAL INDEXING TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.05
0.10
0.15
INDICES INDEXING THROTTLING TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
INDICES DELETE RATE
0.03
0.04
-
7/25/2019 Node Heavy CPU Then Crash
6/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.010.02
INDICES GET REQUESTS
INDICES PERCOLATE REQUESTS RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
INDICES TOTAL PERCOLATE TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
INDICES PERCOLATE QUERIES
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
INDICES STORE
INDICES FIELD DATA
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
3 GB
There is a modestincrease in field datamemory, but onlymaybe
300MB. Thatdoes not seem to be somuch as to crash thesystem.
-
7/25/2019 Node Heavy CPU Then Crash
7/30
INDICES FILTER CACHE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
INDICES LUCENE MEMORY
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
2 GB
4 GB
6 GB
8 GB
INDICES ID CACHE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
2 B
INDICES PERCOLATION SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
2 B
INDICES COMPLETION SIZE
21:50 22:00 22:10 22:20 22:30 22:40
0 B
2 GB
4 GB
6 GB
I have a pretty bigcompletion index, but itseems pretty
steady.
-
7/25/2019 Node Heavy CPU Then Crash
8/30
06-30 06-30 06-30 06-30 06-30 06-30
INDICES INDEX WRITER MEMORY
21:5006-30
22:0006-30
22:1006-30
22:2006-30
22:3006-30
22:4006-30
0 B
4 MB
8 MB
12 MB
INDICES VERSION MAP MEMORY
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
16 KB
32 KB
48 KB
INDICES INDEX WRITER MAX MEMORY
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
INDICES QUERY CACHE MEMORY
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
2 B
Replay History
INDICES ALLOCATED
Legend Primary Replica Relocating Initializing Unassigned
Primary Unassigned
Replica
Nodes Indices
elasticsearch1-elasticsearch-3-vm .marvel-2016.06.28
-
7/25/2019 Node Heavy CPU Then Crash
9/30
10.240.0.4:9300 0
.marvel-2016.06.29
0
.marvel-kibana
1
2
da-prod7
0
1
INDICES CURRENT MERGES
21:5006-30
22:0006-30
22:1006-30
22:2006-30
22:3006-30
22:4006-30
0 B
128 MB
256 MB
384 MB
512 MB
INDICES TOTAL REFRESH TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.000
0.025
0.050
0.075
0.100
INDICES FLUSH COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.01
0.02
0.03
0.04
INDICES OPEN SEARCH CONTEXTS
0
20
40
60
80
-
7/25/2019 Node Heavy CPU Then Crash
10/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
INDICES TOTAL WARMER TIME
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.000
0.001
0.002
0.003
0.004
INDICES SEGMENT COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
50
100
150
200
PARENT CIRCUIT BREAKER ESTIMATED SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
3 GB
PARENT CIRCUIT BREAKER TRIP COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
PARENT CIRCUIT BREAKER LIMIT
8 GB
16 GB
24 GB
-
7/25/2019 Node Heavy CPU Then Crash
11/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
FIELD DATA CIRCUIT BREAKER ESTIMATED SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
3 GB
FIELD DATA CIRCUIT BREAKER TRIP COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
FIELD DATA CIRCUIT BREAKER LIMIT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
8 GB
16 GB
24 GB
REQUEST CIRCUIT BREAKER ESTIMATED SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
64 MB
128 MB
192 MB
REQUEST CIRCUIT BREAKER TRIP COUNT
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
12/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.000.25
REQUEST CIRCUIT BREAKER LIMIT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
4 GB
8 GB
12 GB
16 GB
FIELD DATA SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
3 GB
FIELD DATA EVICTION RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
FILTER CACHE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
1 GB
2 GB
FILTER CACHE EVICTION RATE
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
13/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.250.50
QUERY CACHE MEMORY
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
2 B
QUERY CACHE EVICTION RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
QUERY CACHE CACHE HIT RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
QUERY CACHE MISS RATE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
PROCESS CPU
1.00 K
-
7/25/2019 Node Heavy CPU Then Crash
14/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
250
500750
PROCESS OPEN FILE DESCRIPTORS
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
250
500
7501.00 K
PROCESS THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
50
100
150
SEARCH THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
5
10
15
SEARCH THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
SEARCH THREAD POOL OPS PER SEC
-
7/25/2019 Node Heavy CPU Then Crash
15/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
34
SEARCH THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
50
100
150
SEARCH THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
5
10
15
INDEX THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
INDEX THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
INDEX THREAD POOL OPS PER SEC
-
7/25/2019 Node Heavy CPU Then Crash
16/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.01
0.02
0.03
0.04
INDEX THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
INDEX THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
BULK THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
BULK THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
17/30
BULK THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
BULK THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.5
1.0
1.5
2.0
2.5
BULK THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
GET THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
GET THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
18/30
GET THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.000
0.005
0.010
0.015
0.020
GET THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
GET THREAD POOL LARGEST THREAD COUNT
GENERIC THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
GENERIC THREAD POOL REJECTED
21:50 22:00 22:10 22:20 22:30 22:40
0.00
0.25
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
19/30
06-30 06-30 06-30 06-30 06-30 06-30
GENERIC THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
10
20
30
40
50
60
GENERIC THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
GENERIC THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
10
20
30
40
MANAGEMENT THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
4
5
6
MANAGEMENT THREAD POOL REJECTED
0.25
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
20/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
MANAGEMENT THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.5
1.0
1.5
2.0
MANAGEMENT THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
MANAGEMENT THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
12
3
4
5
6
SUGGEST THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
SUGGEST THREAD POOL REJECTED
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
21/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.000.25
SUGGEST THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.1
0.2
0.3
0.4
0.5
0.6
SUGGEST THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
SUGGEST THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
8
10
PERCOLATE THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
PERCOLATE THREAD POOL REJECTED
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
22/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.250.50
PERCOLATE THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
PERCOLATE THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.500.75
1.00
PERCOLATE THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
WARMER THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
34
5
WARMER THREAD POOL REJECTED
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
23/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
WARMER THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.5
1.0
1.5
2.0
WARMER THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
WARMER THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
45
FLUSH THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
4
5
FLUSH THREAD POOL REJECTED
1.00
-
7/25/2019 Node Heavy CPU Then Crash
24/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
FLUSH THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.2
0.4
0.6
0.8
FLUSH THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
FLUSH THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
4
5
REFRESH THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
4
REFRESH THREAD POOL REJECTED
-
7/25/2019 Node Heavy CPU Then Crash
25/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
REFRESH THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.0
0.2
0.4
0.6
0.8
REFRESH THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
REFRESH THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
4
OPTIMIZE THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
26/30
OPTIMIZE THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
OPTIMIZE THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
OPTIMIZE THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
OPTIMIZE THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
SNAPSHOT THREAD POOL THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
-
7/25/2019 Node Heavy CPU Then Crash
27/30
SNAPSHOT THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
SNAPSHOT THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
SNAPSHOT THREAD POOL QUEUE SIZE
21:5006-30
22:0006-30
22:1006-30
22:2006-30
22:3006-30
22:4006-30
0.00
0.25
0.50
0.75
1.00
SNAPSHOT THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
LISTENER THREAD POOL THREAD COUNT
21:50 22:00 22:10 22:20 22:30 22:40
0
1
2
3
4
5
-
7/25/2019 Node Heavy CPU Then Crash
28/30
06-30 06-30 06-30 06-30 06-30 06-30
LISTENER THREAD POOL REJECTED
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
LISTENER THREAD POOL OPS PER SEC
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.05
0.10
0.15
LISTENER THREAD POOL QUEUE SIZE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0.00
0.25
0.50
0.75
1.00
LISTENER THREAD POOL LARGEST THREAD COUNT
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
1
2
3
4
5
DISK READ RATE (BYTES)
16 MB
32 MB
-
7/25/2019 Node Heavy CPU Then Crash
29/30
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
DISK WRITE PER (BYTES)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
32 MB
64 MB
96 MB
DISK FREE SPACE
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
64 GB
128 GB
DISK IOPS
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
100
200
300
400
500
DISK READ IOPS
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
50
100
150
200
250
DISK WRITE IOPS
100
150
200
250
-
7/25/2019 Node Heavy CPU Then Crash
30/30
ADD A ROW
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
050
NETWORK HTTP CONNECTION OPENED (PER SEC)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0
2
4
6
NETWORK TRANSPORT BYTES SENT (PER SEC)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
4 MB
8 MB
12 MB
NETWORK TRANSPORT BYTES RECEIVED (PER SEC)
21:50
06-30
22:00
06-30
22:10
06-30
22:20
06-30
22:30
06-30
22:40
06-30
0 B
8 MB
16 MB
24 MB
