VM Analysis – Episode 4

Wait analysis of virtualized environments using host kernel tracing

Hani Nemati

May 5, 2017

Polytechnique Montréal

Laboratoire DORSAL

POLYTECHNIQUE MONTREAL – Hani Nemati

Agenda

Introduction● Research update and research motivation

New Investigations● Wait analysis of virtualized environments using host kernel tracing

● Sate of the art

● Proposed Algorithm

● Demo

● KVM-Tool for eBPF

Conclusion and in-progress

POLYTECHNIQUE MONTREAL – Hani Nemati

Available Trace-Points in different layers

HardwareHardware

Host Kernel Host KernelKVM.KOKVM.KO

QemuQemu

GuestKernel

VM 1

QemuQemu

GuestKernel

VM 2

QemuQemu

GuestKernel

VM n

Guest Kernel TraceGuest Kernel Trace

Qemu TraceQemu Trace

KVM TraceKVM Trace

Host Kernel TraceHost Kernel Trace

Hardware PMCHardware PMC

Previously on “VM Analysis”

POLYTECHNIQUE MONTREAL – Hani Nemati

Available Trace-Points in different layers

HardwareHardware

Host Kernel Host KernelKVM.KOKVM.KO

QemuQemu

GuestKernel

VM 1

QemuQemu

GuestKernel

VM 2

QemuQemu

GuestKernel

VM n

KVM TraceKVM Trace

Host Kernel TraceHost Kernel Trace

Previously on “VM Analysis”

POLYTECHNIQUE MONTREAL – Hani Nemati

Resource View for VM without tracing the VM

Previously on “VM Analysis”

POLYTECHNIQUE MONTREAL – Hani Nemati

Previously on “VM Analysis”

VirtFlow: Execution Flow Analysis of Virtual Machine

POLYTECHNIQUE MONTREAL – Hani Nemati

Two Nested VMs and One VM are preempting each other

Previously on “VM Analysis”

POLYTECHNIQUE MONTREAL – Hani Nemati

MotivationWhy the VM is waiting?

POLYTECHNIQUE MONTREAL – Hani Nemati

MotivationLet's use the Critical Flow view of Trace Compass?

POLYTECHNIQUE MONTREAL – Hani Nemati

InvestigationsMethodology Vec from kvm_inj_virq

CR3 from vcpu_enter_guest

Vec from kvm_inj_virqCR3 from vcpu_enter_guestCR3 from vcpu_enter_guest

If (Vec == (Block I/O irq)) {Block State = Block I/O State

} else if (Vec == (network irq)) {Block State = Network State

}

POLYTECHNIQUE MONTREAL – Hani Nemati

InvestigationsMethodology

If (Vec == 239) {Block State = Timer

} else if (Vec == 251) {Block State = Task

}

Vec from kvm_inj_virqCR3 from vcpu_enter_guest

CR3 from vcpu_enter_guest

Vec from kvm_inj_virqCR3 from vcpu_enter_guest

POLYTECHNIQUE MONTREAL – Hani Nemati

Investigations

Demo

POLYTECHNIQUE MONTREAL – Hani Nemati

InvestigationsWhat do you need to test this project?

● Access to Host only

● Run LTTng on Host with my new added tracepoint (vcpu_enter_guest)

● Clone TraceCompass from my github (virtFlow)● https://github.com/Nemati

● Open Resource View of TraceCompass

POLYTECHNIQUE MONTREAL – Hani Nemati

Investigations

One More Thing ...

KVM-ToolsFor

eBPF

POLYTECHNIQUE MONTREAL – Hani Nemati

POLYTECHNIQUE MONTREAL – Hani Nemati

Conclusion and in-progress

Inferences

● Wait Analyzing of process inside VM● A process is waiting for

● A Block request to finish● A network packet to receive ● Another process● A timer to fire

What you will see in Episode 5

● Wait Analyzing of process inside Nested VM

POLYTECHNIQUE MONTREAL – Hani Nemati

Questions?

Hani.nemati@polymtl.ca

https://github.com/Nemati