Host Hypervisor Trace Mining for Virtual Machine …Host Hypervisor Trace Mining for Virtual Machine Workload Characterization Hani Nemati, Vahid Azhari, and Michel Dagenais Dec 6,

Host Hypervisor Trace Mining for Virtual Machine Workload Characterization

Hani Nemati, Vahid Azhari,

and Michel Dagenais

Dec 6, 2018

EMU

Experiments Conclusion Feature extractionPreviously on VM AnalysisIntroduction

Outline

POLYTECHNIQUE MONTREAL – Hani Nemati & Vahid Azhari1

Introduction Previously on VM Analysis

Feature Extraction

VM Clustering

Conclusion &Future Work

1 2 3 4 5

Agent-less

Restricted Access

Old Kernel

Limited Resources

K-MEANS

CPU Intensive

Disk Intensive

Network Intensive

New Tracepoint

Nested VM

Wait Analysis

Critical Path

Virtual Interrupt

CPU Metrics

Disk Metrics

Network Metrics

Trace mining

VM Clustering

vCPU features

Process Clustering


VM Architecture


VMM VMM VMM

Virtual Machines

Host Operating SystemAPP

Hardware

Guest OS

Guest OS

Guest OS


Motivation


KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess

KVM KVM KVM

Virtual Machines

Hardware

Host OS: Linux

WindowsMac OS

Old Kernel

Limited Resources

KVM

RestrictedAccess

POLYTECHNIQUE MONTREAL – Hani Nemati & Vahid Azhari

AStep 01-Monitoring

Fine-grain resource Monitoring Retrieve guest state from VMCS Analyzing VM interference Disk View, vCPU view , Netrowk view

10 Dec 2015

4


Previously on VM Analysis


BStep 02- Overcommitment

Resource Overcommitment Detection CPU Running State Analysis CPU overcommitment Analysis Memory overcommitment Analysis

5 May 2016




CStep 03 -Nested VM

Any level Nested VM State Analysis Nested VM vCPU state analysis Nested VM overcommitment detection Virtualization overhead analysis

12 Dec 2016




DStep 04 -Wait Analysis

vCPU idle State Analysis Virtual interrupt injection Analysis Analyzing idle State of VM eBPF based VM analysis

5 May 2017




E

Step 05 -Critical Path

Execution Path Analysis of VM Contention of Disk Cap on virtual resources VM boot-up comparison

7 Dec 2017




FStep 06 -Distributed VM

Hierarchical Distributed Critical Path Critical Path for Nested VM Critical Path for distributed VMs

10 May 2018FStep 06 -Distributed VM


10 May 2018FStep 06 -Distributed VM


10 May 2018



A

B

C

D

E

F

Step 01-Monitoring

Step 03 -Nested VM

Step 02- Overcommitment

Step 05 -Critical Path

Step 06 -Distributed VM

Step 04 -Wait Analysis

Fine-grain resource Monitoring

Any level Nested VM State Analysis

Execution Path Analysis of VM

Hierarchical Distributed Critical Path

vCPU idle State Analysis

Resource Overcommitment Detection

10 May 2018

7 Dec 2017

12 Dec 2016

5 May 2017

5 May 2016

10 Dec 2015

Previously on VM Analysis Experiments Conclusion Feature extractionPreviously on VM AnalysisIntroduction



Automation


How about

Automation?

11

VM Clustering Architecture



Hybrid Clouds

Public Clouds

Private Clouds

Agent-less tracing LTTng

Data Collection Feature Extraction Tracecompass

VM clustering KMEANS

VM Data

Report Engine

12


Disk MetricsExperiments Conclusion Feature extractionPreviously on VM AnalysisIntroduction

4

Control Flow view

vCPU thread

IO thread

main thread futex futexuserspace idle futex futexuserspaceidle idle

userspace userspaceidle

idle futex preadv futexUser space

User space idle

userspacesyscallidle

Disk latency

Wait for handling disk request

inject_disk_interruptvm_exit =301

2

3

4 5

1

2

3

4

5

VM exits from guest mode to host mode to handle disk request

Qemu main thread submits the disk request

Qemu IO thread handles the disk request

Two metrics: wait for handling disk request and disk latency

Virtual disk interrupt is injected to VM

13


CPU MetricsExperiments Conclusion Feature extractionPreviously on VM AnalysisIntroduction

4

1

2

3

Virtual timer interrupt is injected to VM

A VM process wakes up another VM process

Virtual task interrupt is injected to VM

1

2

3

Control Flow view

vCPU thread

vCPU thread


idle idle


Wait for task

idle userspace

userspace

inject_timer_interrupt inject_task_interrupt

try_to_wakeup

Wait for timer

14


Network MetricsExperiments Conclusion Feature extractionPreviously on VM AnalysisIntroduction

4

1

2

3

A packet is received by vhost-net

Wait for a packet to receive

Virtual network interrupt is injected to VM

Control Flow view

vCPU thread

vhost-net


idle idle


inject_network_interrupt1 3

userspace

Wait for packet to receive

2

15

Experimental Evaluation



k-meansClustering

VM

VMVM

VM

VM

VMVM VM

VM

VM

VMVM VM

VMVMVM

VM

VM

VM VM

VMVM

Resource Metrics

Resource Metrics

16




CPUIntensive

17




CPUIntensive

18




CPUIntensive

19




CPUIntensive

20




CPUIntensive

421




CPUIntensive

419422




DiskIntensive

423




NetworkIntensive

424




1

2

3

4

5

How toTry

Access to Host only

Run LTTng on Host with my new added tracepoint (vcpu_enter_guest)

Clone TraceCompass from github (incubator:https://github.com/Nemati/org.eclipse.tracecompass.incubator.git )

Git checkout vm2

Clone lamiminer from github (https://github.com/azharivs/lamiminer.git)

425

CPU intensive group has high timer interrupt and high CPU usage

Disk intensive group has high disk interrupt and high task interrupt

Conclusions



1

2

3

4

Clustering VM based on injected interrupts to VM

Network intensive group has high network interrupt and high CPU usage

Questions



azhari

vs@gm

ail.co

m

http:/

/githu

b.com

/azhar

[email protected]

http://github.com/Nemati

427

Documents

Host Hypervisor Trace Mining for Virtual Machine …Host Hypervisor Trace Mining for Virtual Machine Workload Characterization Hani Nemati, Vahid Azhari, and Michel Dagenais Dec 6,