Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Jagadeesh Devaraj & Ramesh Nataraja
SER2905BE
#VMworld SER2905BE
Uncovering ESXTOP
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
2#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1 What is ESXTOP
2 ESXTOP Screen
3 CPU, Memory, Power, Storage, VM screen
4 Storage adapter, Storage device, VSAN screen
5 Network
6 Data collection
3#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
What is esxtop?
• ESXTOP is a built-in command line tool that gives real time information on
– Resources used (cpu, memory, network, storage, power)
– Understand the resource fault trending
– Help in hardware scalability assessment
4
How do you run ESXTOP?
Run esxtop from the ESXi console with command
“esxtop”
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
esxtop screen
Key strokes to navigate
5
CPU c
Memory m
Power Management p
Storage VM v Network n
Storage Device u
Storage Adapter d
vSAN x
Interrupts i
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘c’ for CPU
6
Current timeUptime of the hostNumber of userworlds runningNumber of VMs runningNumber of vCPUs assigned to all running VMs
CPU load average in
1 minute, 5 minutes, and 15 minutes
based on 6-second samples
The percentage CPU usage per Physical CPU, and its average over all Physical CPUsThe percentage of unhalted CPU cycles per Physical CPU, and its average over all
Physical CPUsID of the worldID of the groupName of the sched groupNumber of worlds in the groupThe percentage physical CPU time accounted to the world.
%USED = %RUN + %SYS - %OVRLPThe percentage of time the virtual machine is consuming CPU resourcesThe percentage of time spent by system services on behalf of the world
The percentage of time the world spent in wait state.
The world is waiting for some vmkernel resourceRepresents the total percentage of time the World spent in a blocked state waiting for events. Amount of time the virtual machine was ready to run, waiting in a queue to be scheduled on the PCPUThe percentage of time the vCPU world is in an idle loop.
The percentage of time spent by system services on behalf of other worldsAmount of time an SMP vm was ready to run, but
incurred delay due to co-vCPU scheduling contentionPercentage of time the world was ready to run but deliberately wasn't
scheduled because that would violate the "CPU limit" settings
The percentage of time the world is waiting for the vmkernel swapping memory
%SWPWT time is included in the %WAIT time
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘m’ for Memory
7
PMEM /MB - This line displays the physical memory statistics for the server
total is for the total amount of physical memory in the server
vmk is for the amount of physical memory being used by the vmkernel
other is for the amount of physical memory being used by everything other than the vmkernel
free is for the amount of physical memory that is free
VMKMEM/MB - This line displays the machine memory statistics for the vmkernel
managed is for the total amount of physical memory managed by the vmkernel
minfree is for the minimum amount of physical memory that the vmkernel would like to keep free
rsvd is for the amount of physical memory that is currently reserved
ursvd is for the amount of physical memory that is
currently unreserved and
state is for the memory state as reported by the vmkernel.
high(300%)|clear(64%)|soft(32%)|hard(16%)|low(0%)
NUMA /MB - This line displays the ESXi NUMA statistics.
This line is only displayed if ESXi is running on a NUMA
server
The first statistic is the total amount of physical memory in
the NUMA node that is managed by ESXi.
The second statistic, that is displayed within round brackets, is the amount of physical memory in the node that is currently free
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘m’ for Memory
PSHARE/MB This line displays the ESXi page-sharing
statistics.
shared - is for the amount of physical memory that is
being shared
common - is for the amount of physical memory that is
common across World(s)
saving - is for the amount of physical memory that is
saved due to page-sharing
SWAP /MB This line displays the ESXi swap usage statistics
curr is for the current swap usage
rclmtgt is for where ESXi expects the reclaimed memory to be,
using swapping and compression
r/s is for the rate at which memory is swapped in by ESXi from
disk
w/s is for the rate at which memory is swapped to disk by the
ESXi Server
ZIP /MB - This line displays the ESXi memory compression
statistics
zipped - is for the total compressed physical memory
saved - is for the saved memory by compression
MEMCTL/MB This line displays the memory balloon statistics
curr is for the total amount of physical memory reclaimed using
the vmmemctl modules
target is for the total amount of physical memory ESXi would
like to reclaim using the vmmemctl modules
max is for the maximum amount of physical memory ESXi can
reclaim using the vmmemctl modules
#SER2905BE CONFIDENTIAL 8
VMworld 2017 Content: Not fo
r publication or distri
bution
‘m’ for Memory
Resource Pool ID / World idResource Pool nameThe amount of physical memory allocated to a vmThe amount of machine memory the ESXi VMkernel wants to allocate to vmThe amount of machine memory consumed by a vmThe amount of physical memory ESXi VMkernel wants to allocate to a vmThe working set estimate for a vmThe write working set estimate for a vmCurrent swap usage by vmWhere ESXi expected the Swap usage by the vm to be
Rate at which memory is being swapped in by ESXi from disk for the vmRate at which vm memory is being swapped to disk by ESXiRate at which memory is faulted from host cacheRate at which memory is written to the host cache from various sourcesCurrent space overhead for the user worldCurrent space overhead for vmMaximum space overhead that may
be incurred by vm
#SER2905BE CONFIDENTIAL 9
VMworld 2017 Content: Not fo
r publication or distri
bution
‘p’ for Power Management
10
This screen displays CPU Power utilization statistics. On this screen statistics are arranged per PCPU
-
-.
Power Usage The current total power usage in Watts
Power Cap The total power cap in Watts
%USED The percentage of PCPU nominal frequency that was used since
the last screen update
%UTIL The raw PCPU utilization, that is the percentage of real time that
PCPU was not idle
%Cx The percentage of time the PCPU spent in C-State 'x'
%Px The percentage of time the PCPU spent in P-State 'x'.
%Tx The percentage of time the PCPU spent in T-State 'x'
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘p’ for Power Management
When to use Power Management?
11
Sluggish Application
Performance on a vm
Verify if there is a discrepancy
between PCPU Used% and PCPU Util%
Verify if there is CPU contention
(high %RDY)
Divide %RDY value by the number of
vCPUs assigned
Isolate if excess vCPU is
the cause of low application performance
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
This screen displays VM-centric storage statistics. On this screen, statistics are aggregated on a per-resource-poolbasis by default. One VM has one corresponding resource pool, so they are equivalent to per-VM statistics
12
‘v’ for Storage VM
The GroupID of VM/Resource PoolThe name of VMThe name of the virtual deviceThe number of vscsi devicesThe number of commands issued per secondThe number of read commands issued per secondThe number of write commands issued
per secondThe megabytes read per secondThe megabytes written
per second
The average latency (millisecs)
per readThe average latency (millisecs)
per write
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘u’ for Storage Device
13
Name of the storage devicePath nameWorld IDPartition IDCurrent device queue depth of the storage deviceWorld queue depthNo. of commands in the vmkernel that are
currently activeNo. of commands in the ESXi VMkernel that
are currently queued
Percentage of the queue depth used by
vmkernel active commands
%USD = ACTV / QLEN * 100%
Ratio of vmkernel active commands plus vmkernel
queued commands to queue depthNo. of commands issued per secondNo. of read commands issued per
second
No. of write commands issued per
secondMegabytes read per secondMegabytes written per second
Average device latency per
command in milliseconds
Average ESXi VMkernel latency
per command in milliseconds
Average guest operating system latency
per command in milliseconds
GAVG = KAVG + DAVG
Average queue read latency per read
operation in milliseconds
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘d’ for Storage Adapter
14
This screen displays server-wide storage utilization statistics. On this screen statistics are aggregated per
storage adapter by default
The name of the storage adapterThe storage pathnameThe number of pathsThe number of commands
completed per secondThe number of read commands
completed per second
The number of write commands
completed per secondThe megabytes read per secondThe megabytes written per
second
The average device latency
(millisecs) per command
The average vmkernel
latency (millisecs) per
commandThe average Guest OS latency (millisecs) per commandThe average queue latency (millisecs) per command
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘x’ for vSAN
15
VSAN Enabled Whether VSAN is currently enabled on the host
ROLE The name of VSAN DOM Role
READS/s Number of read operations completed per second
MBREAD/s Megabytes read per second
WRITES/s Number of write operations completed per second
MBWRITE/s Megabytes written per second
RECOWR/s Number of recovery write operations completed per second
MBRECOWR/s Megabytes written for recovery per second
SDLAT Standard deviation of latency in millisecs for read, write and recovery write
AVGLAT Average latency in millisecs for read, write and recovery write
This screen displays VSAN statistics
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
‘n’ for Network
16
PORT-ID The virtual network device port id
UPLINK Y, implies the corresponding port is uplink. N, implies it is not
UP Y, implies the corresponding link is up. N, implies it is not
SPEED The link speed in MegaBits per second
FDUPLX Y, implies the corresponding link is operating at full duplex. N, implies it is not
USED-BY The virtual network device port user
TEAM-PNIC The physical NIC name for the team uplink
DNAME The virtual network device name
PKTTX/s The number of packets transmitted per second
PKTRX/s The number of packets received per second
MbTX/s The MegaBits transmitted per second
MbRX/s The MegaBits received per second
PSZTX The average packet size transmitted in bytes
PSZRX The average packet size received in bytes
%DRPTX The percentage of transmit packets dropped
%DRPRX The percentage of receive packets dropped
ACTN/s Number of actions per second
PKTTXMUL/s The number of multicast packets transmitted per second
PKTRXMUL/s The number of multicast packets received per second
PKTTXBRD/s The number of broadcast packets transmitted per second
PKTRXBRD/s The number of broadcast packets received per second
#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
Data Collection
ESXTOP results could be collected using Batch mode. The output can be redirected to a .csv file, this file could be opened through MS Excel or Perfmon tool
Below is the syntax for generating esxtop in batch mode
-b : for batch mode -d : for delay -n : for no. of iterations-a : for collection of all metrics
In the above example esxtop will log all metrics for 500 seconds and compress the results in zipped format to the specified location
17#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution
Questions?
19#SER2905BE CONFIDENTIAL
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution