Critical Path Analysis through Hierarchical Distributed Virtualized Environments
using Host Kernel TracingHani Nemati
May 10, 2018
Polytechnique Montréal
Laboratoire DORSAL
POLYTECHNIQUE MONTREAL – Hani Nemati
Introduction
2The figure retrieved from: https://mad-owl.deviantart.com/art/Chibi-Luke-found-something-193450025
Tracing is
?
POLYTECHNIQUE MONTREAL – Hani Nemati
Introduction
2The figure retrieved from: https://mad-owl.deviantart.com/art/Chibi-Luke-found-something-193450025
Tracing is
POLYTECHNIQUE MONTREAL – Hani Nemati
AgendaIntroduction
● Research update and research motivation
New Investigations● Host-based Execution-graph Construction (HEC)● Critical Path Analysis through hierarchical virtualized
environments ● Proposed Algorithm ● Usecases ● Demo● Overhead Analysis of HEC and existing critical path
analysis
Conclusion and in-progress 3
POLYTECHNIQUE MONTREAL – Hani Nemati
Virtual Machine Hierarchy Motivation
4
VMM VMM VMM
Virtual Machines
Host Operating SystemAPP
Hardware
Guest OS
Guest OS
Guest OS
Maps virtual re
sources
to physical resource
Resource may be
time-shared or
partitioned
POLYTECHNIQUE MONTREAL – Hani Nemati
Virtual Machine Hierarchy Motivation
5
KVM KVM KVM
Virtual Machines
APP
Hardware
LinuxLinux as Host OS
KVM as Hypervisor
Arbitrary Guest OS
POLYTECHNIQUE MONTREAL – Hani Nemati
Virtual Machine Hierarchy Motivation
6
KVM KVM KVM
Virtual Machines
Hardware
Host OS: Linux
WindowsMac OS
Old Kernel
Limited Resources
KVM
RestrictedAccess
KVM KVM KVM
Virtual Machines
Hardware
Host OS: Linux
WindowsMac OS
Old Kernel
Limited Resources
KVM
RestrictedAccess
POLYTECHNIQUE MONTREAL – Hani Nemati 7
1) Install a tracer on each VM2) Trace them
3) Sync the traces
Motivation
POLYTECHNIQUE MONTREAL – Hani Nemati
Hierarchical Virtualized Environments
Investigation
8
KVM
Virtual Machines
Host OS: Linux
Hardware
KVMHost OS: Linux
VMM
Guest OSVMM
Nested VM OS
Nested VM OSNested VM Level - L2
VM Level - L1
Host Level - L0
POLYTECHNIQUE MONTREAL – Hani Nemati 9
Is there any method that preferably limits its data collection to
the physical host level?
Motivation
POLYTECHNIQUE MONTREAL – Hani Nemati
virtFlow features Investigation
11
KVM
Virtual Machines
Host OS: Linux
Hardware
KVMHost OS: Linux
VMM
Guest OSVMM
Nested VM OS
Nested VM OS
Hierarchal vCPU view for VMRunning StatesWait States
vCPU view for Tracecompass
Nested VM vCPU view for Tracecompass
POLYTECHNIQUE MONTREAL – Hani Nemati
virtFlow features Investigation
12
KVM
Virtual Machines
Host OS: Linux
Hardware
KVMHost OS: Linux
VMM
Guest OSVMM
Nested VM OS
Nested VM OS
Hierarchal Process view for VMRunning StatesWait States
Process view for Tracecompass
Nested VM vCPU view for Tracecompass
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
13
KVM
Virtual Machines
Host OS: Linux
Hardware
KVMHost OS: Linux
VMM
Guest OSVMM
Nested VM OS
Nested VM OS
virtFlow features Critical Path Analysis through Hierarchical Virtualized Environments
Critical Path Analysis for VM
Nested VM vCPU view for Tracecompass
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
14
Virtual Machine
Hardware
KVMHost OS: Linux
Guest OS
Virtual Machine
Hardware
KVMHost OS: Linux
Guest OS
Virtual Machine
Hardware
KVMHost OS: Linux
Guest OS
Distributed Virtualized Environments
POLYTECHNIQUE MONTREAL – Hani Nemati
Motivation
15
virtFlow features Critical Path Analysis through Distributed Virtualized Environments
Critical Path Analysis for VM
Nested VM vCPU view for Tracecompass
Virtual Machine
Hardware
KVM
Linux
Guest OS
Virtual Machine
Hardware
KVM
Linux
Guest OS
Virtual Machine
Hardware
KVM
Linux
Guest OS
POLYTECHNIQUE MONTREAL – Hani Nemati
Motivation
15
virtFlow features Critical Path Analysis through Distributed Virtualized Environments
Critical Path Analysis for VM
Nested VM vCPU view for Tracecompass
Virtual Machine
Hardware
KVM
Linux
Guest OS
Virtual Machine
Hardware
KVM
Linux
Guest OS
Virtual Machine
Hardware
KVM
Linux
Guest OS
POLYTECHNIQUE MONTREAL – Hani Nemati
Containers within Virtualized Environments
Investigation
16
KVM
Virtual Machines
Host OS: Linux
Hardware
KVMHost OS: Linux
Docker Engine
Guest OS
Container Container
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
17
KVM KVM KVM
Virtual Machines
Hardware
Host OS: Linux
WindowsMac OS
Old Kernel
Limited Resources
KVM
RestrictedAccess
KVM KVM KVM
Virtual Machines
Hardware
Host OS: Linux
WindowsMac OS
Old Kernel
Limited Resources
KVM
RestrictedAccess
Containers within Virtualized Environments
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
17
KVM KVM KVM
Virtual Machines
Hardware
Host OS: Linux
WindowsMac OS
Old Kernel
Limited Resources
KVM
RestrictedAccess
KVM KVM KVM
Virtual Machines
Hardware
Host OS: Linux
WindowsMac OS
Old Kernel
Limited Resources
KVM
RestrictedAccess
Containers within Virtualized Environments
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
18
VM Analysis through Hierarchical Virtualized Environments
qemu-thread
ControlFlow view
vCPU viewvCPU 0
Nested vCPU viewvCPU 0
1 2 3 4 5
1 sched_switch(in=qemu_thread)2 inj_virq(vec=timer)3 vm_entry(vcpu0, cr3#0)
4 vm_exit(reason=12)5 sched_switch(out=qemu_thread)
Timerrootnon-root
blockrunning
unknown
MethodologyNested vCPU view
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
19
MethodologyNested vCPU view
VM Analysis through Hierarchical Virtualized Environments
qemu-thread
ControlFlow view
vCPU viewvCPU 0
Nested vCPU viewvCPU 0
1
1 sched_switch(in=qemu_thread)2 inj_virq(vec=disk)3 vm_entry(vcpu0, cr3#1)4 vm_exit(reason=24)5 vm_entry(vcpu0, cr3#2)
timerrootnon-root
blockrunning
unknown
2 3
disk
4 5 6 7 8 9
timerrootnon-root_l1unknown
disk non-root_l2
6 vm_exit(reason=12)7 vm_entry(vcpu0, cr3#1)8 vm_exit(reason=12)9 sched_switch(out=qemu_thread)
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigations
20
Critical Path AnalysisLinux Advance Packaging Tool
1) apt-get downloads and reads cached packages2) apt-get installs the packages along with downloaded dependencies3) The installation of man-pages
What is going on here ?
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigations
21
Critical Path Analysis Undesirable parallelism
waits for disk
waits for another process
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigations
22
Critical Path Analysis Network Intensive VM – IMS Network
Waits for DNS server
Waits for user to response
Waits for user to response
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigation
23
Existing Critical Path Analysis in TraceCompass
Critical Path Analysis
Host-based Execution-graph Construction
Preemption State
POLYTECHNIQUE MONTREAL – Hani Nemati
Investigations Overhead Analysis
CPA : Existing Critical Path Analysis in TraceCompass HEC: Host-based Execution-graph Construction
24
POLYTECHNIQUE MONTREAL – Hani Nemati
InvestigationsHow to try these new features?
● Access to Host only
● Run LTTng on Host with my new added tracepoint (vcpu_enter_guest)
● Clone TraceCompass from github (incubator)● Open vCPU block View of TraceCompass (XML view) ● Open vProcess block View of TraceCompass (XML view)● Open Nested VM vCPU Block View of TraceCompass (XML view)● Open Nested VM vProcess Block View of TraceCompass (XML view)● Use Execution Flow Analysis of TraceCompass
25
POLYTECHNIQUE MONTREAL – Hani Nemati
Conclusion and in-progress
Inferences
● Wait Analysis of process inside VM and Nested VM● A process is waiting for
● A Disk Block request to finish● A Network packet to receive ● Another process● A Timer to fire● Other devices
● Critical Path Analysis of process inside VM and Nested VM
26