Critical Path Analysis through Hierarchical Distributed ...amdls.dorsal.polymtl.ca/system/files/10May2018.pdf · Critical Path Analysis through Hierarchical Distributed Virtualized

Critical Path Analysis through Hierarchical Distributed Virtualized Environments

using Host Kernel TracingHani Nemati

May 10, 2018

Polytechnique Montréal

Laboratoire DORSAL

POLYTECHNIQUE MONTREAL – Hani Nemati

Introduction

2The figure retrieved from: https://mad-owl.deviantart.com/art/Chibi-Luke-found-something-193450025

Tracing is

?


Introduction

2The figure retrieved from: https://mad-owl.deviantart.com/art/Chibi-Luke-found-something-193450025

Tracing is


AgendaIntroduction

● Research update and research motivation

New Investigations● Host-based Execution-graph Construction (HEC)● Critical Path Analysis through hierarchical virtualized

environments ● Proposed Algorithm ● Usecases ● Demo● Overhead Analysis of HEC and existing critical path

analysis

Conclusion and in-progress 3


Virtual Machine Hierarchy Motivation

4

VMM VMM VMM

Virtual Machines

Host Operating SystemAPP

Hardware

Guest OS

Guest OS

Guest OS

Maps virtual re

sources

to physical resource

Resource may be

time-shared or

partitioned



5

KVM KVM KVM

Virtual Machines

APP

Hardware

LinuxLinux as Host OS

KVM as Hypervisor

Arbitrary Guest OS



6

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess

POLYTECHNIQUE MONTREAL – Hani Nemati 7

1) Install a tracer on each VM2) Trace them

3) Sync the traces

Motivation


Hierarchical Virtualized Environments

Investigation

8

KVM

Virtual Machines

Host OS: Linux

Hardware

KVMHost OS: Linux

VMM

Guest OSVMM

Nested VM OS

Nested VM OSNested VM Level - L2

VM Level - L1

Host Level - L0


Is there any method that preferably limits its data collection to

the physical host level?

Motivation


virtFlow

Motivation


virtFlow features Investigation

11

KVM

Virtual Machines

Host OS: Linux

Hardware

KVMHost OS: Linux

VMM

Guest OSVMM

Nested VM OS

Nested VM OS

Hierarchal vCPU view for VMRunning StatesWait States

vCPU view for Tracecompass

Nested VM vCPU view for Tracecompass


virtFlow features Investigation

12

KVM

Virtual Machines

Host OS: Linux

Hardware

KVMHost OS: Linux

VMM

Guest OSVMM

Nested VM OS

Nested VM OS

Hierarchal Process view for VMRunning StatesWait States

Process view for Tracecompass



Investigation

13

KVM

Virtual Machines

Host OS: Linux

Hardware

KVMHost OS: Linux

VMM

Guest OSVMM

Nested VM OS

Nested VM OS

virtFlow features Critical Path Analysis through Hierarchical Virtualized Environments

Critical Path Analysis for VM



Investigation

14

Virtual Machine

Hardware

KVMHost OS: Linux

Guest OS

Virtual Machine

Hardware

KVMHost OS: Linux

Guest OS

Virtual Machine

Hardware

KVMHost OS: Linux

Guest OS

Distributed Virtualized Environments


Motivation

15

virtFlow features Critical Path Analysis through Distributed Virtualized Environments



Virtual Machine

Hardware

KVM

Linux

Guest OS

Virtual Machine

Hardware

KVM

Linux

Guest OS

Virtual Machine

Hardware

KVM

Linux

Guest OS


Motivation

15

virtFlow features Critical Path Analysis through Distributed Virtualized Environments



Virtual Machine

Hardware

KVM

Linux

Guest OS

Virtual Machine

Hardware

KVM

Linux

Guest OS

Virtual Machine

Hardware

KVM

Linux

Guest OS


Containers within Virtualized Environments

Investigation

16

KVM

Virtual Machines

Host OS: Linux

Hardware

KVMHost OS: Linux

Docker Engine

Guest OS

Container Container


Investigation

17

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess



Investigation

17

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess



Investigation

18

VM Analysis through Hierarchical Virtualized Environments

qemu-thread

ControlFlow view

vCPU viewvCPU 0

Nested vCPU viewvCPU 0

1 2 3 4 5

1 sched_switch(in=qemu_thread)2 inj_virq(vec=timer)3 vm_entry(vcpu0, cr3#0)

4 vm_exit(reason=12)5 sched_switch(out=qemu_thread)

Timerrootnon-root

blockrunning

unknown

MethodologyNested vCPU view


Investigation

19

MethodologyNested vCPU view

VM Analysis through Hierarchical Virtualized Environments

qemu-thread

ControlFlow view

vCPU viewvCPU 0

Nested vCPU viewvCPU 0

1

1 sched_switch(in=qemu_thread)2 inj_virq(vec=disk)3 vm_entry(vcpu0, cr3#1)4 vm_exit(reason=24)5 vm_entry(vcpu0, cr3#2)

timerrootnon-root

blockrunning

unknown

2 3

disk

4 5 6 7 8 9

timerrootnon-root_l1unknown

disk non-root_l2

6 vm_exit(reason=12)7 vm_entry(vcpu0, cr3#1)8 vm_exit(reason=12)9 sched_switch(out=qemu_thread)


Investigations

20

Critical Path AnalysisLinux Advance Packaging Tool

1) apt-get downloads and reads cached packages2) apt-get installs the packages along with downloaded dependencies3) The installation of man-pages

What is going on here ?


Investigations

21

Critical Path Analysis Undesirable parallelism

waits for disk

waits for another process


Investigations

22

Critical Path Analysis Network Intensive VM – IMS Network

Waits for DNS server

Waits for user to response

Waits for user to response


Investigation

23

Existing Critical Path Analysis in TraceCompass

Critical Path Analysis

Host-based Execution-graph Construction

Preemption State


Investigations Overhead Analysis

CPA : Existing Critical Path Analysis in TraceCompass HEC: Host-based Execution-graph Construction

24


Investigations

Demo


InvestigationsHow to try these new features?

● Access to Host only

● Run LTTng on Host with my new added tracepoint (vcpu_enter_guest)

● Clone TraceCompass from github (incubator)● Open vCPU block View of TraceCompass (XML view) ● Open vProcess block View of TraceCompass (XML view)● Open Nested VM vCPU Block View of TraceCompass (XML view)● Open Nested VM vProcess Block View of TraceCompass (XML view)● Use Execution Flow Analysis of TraceCompass

25


Conclusion and in-progress

Inferences

● Wait Analysis of process inside VM and Nested VM● A process is waiting for

● A Disk Block request to finish● A Network packet to receive ● Another process● A Timer to fire● Other devices

● Critical Path Analysis of process inside VM and Nested VM

26


Questions?

[email protected]

https://github.com/Nemati

Documents

Critical Path Analysis through Hierarchical Distributed ...amdls.dorsal.polymtl.ca/system/files/10May2018.pdf · Critical Path Analysis through Hierarchical Distributed Virtualized