Comparison between xen and kvm - Indico · 2018. 11. 17. · KVM seen by system admin rpms:...

XEN and KVM in INFN production systems and a comparison between them

Riccardo VeraldiAndrea ChiericiINFN - CNAFHEPiX Spring 2009

Outline

xenkvmTest descriptionBenchmarksConclusions

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea 2

What we have now: Xen

Open source “industry standard” for virtualizationIncluded in all recent linux distributionsSupports para and hvm approachesWidely used in HEP communityWon’t go into details during presentation

What could be the future: KVMKernel-based Virtual MachineOpen sourceincluded in latest linux kernels: implemented as a module

A user space program uses /dev/kvm interface to set up VMs (qemu-kvm)

Supports hvm approachRather new to HEP communityQumranet now owned by Red Hat

Foreseen boost in development

KVM virtio

Direct access to hardwareHard disk (/dev/vdx)Ethernet interface

Support from kernel 2.6.25Supported since RHEL 5.3 (virtio backport)

kernel 2.6.18-128.1.1 Virtio drivers must be on the VM, not on the hostEGEE sw can’t currently benefit from this

KVM seen by system adminrpms: basically the kernel module and a modified version of qemu (qemu-kvm)Network configuration to be done by hand in order to get public IP

bridge-utils and tunctl rpms were used to set-up “tap” interfaceScript developed to automate the process

A modified init.d script to configure software bridge has been developed at CNAF

No VM configuration file direct supportVMs are launched via a standard UNIX command + command line optionslibvirt

KVM seen by system admin

No direct interface to manage VMs by command line (e.g. “xm”)

Libvirt supportVirt-manager workingMonitor interface

Very powerful, lots of options availableqemu already well documented, support available on-line

KVM introduced at CNAF

We use quattor to configure and install grid nodes and virtual machines

Profiles describe machine configurationNetwork boot for installation working

Keeps booting from network, need to restart with “-boot c”

No need for any modification, just like xen-hvm

Qualitative testCNAF is running LHCb tier2 site entirely on xen VMs (2 CEs, 1 SE)Changed one Computing Element with a KVM machine

Has been working for more that 3 weeks flawlesslyQuattor machine profile unmodified, no effort for sysadmin

CMS secondary squid server installed on the same host with same result (2 weeks ago)Running a KVM in production since 2 months for INFN-CA backups. Works flawlesslyKVM executed (and live migrated!) a win7 VMHardware used: 1 node, dual E5420, 16GB ram, sata disks via Areca controller

Quantitative test: description

Need some measures to understand what is the best solutionTested 3 classic parameters

CPU → hep-spec06 (v1.1), PI test for cpu distributionNetwork → iperf (v2.0.4)Disk access → bonnie++ (v1.94)

Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline

Quantitative test: descriptionHardware used: 1 blade, dual E5420, 16GB ram, 10k sas disk via LSI logic raid controller (raid0)Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a fileXen-hvm VM specs: 1 vcpu, 2GB ram, disk on a file, “netfront” network driverKVM VM specs: 1 vcpu, 2GB ram, disk on a file, e1000 network driver emulation (non-virtio tests)Host OS: SL 5.2 x86_64, kernel 2.6.18-92.1.22.el5VM OS: SLC 4.5 i386, kernel 2.6.9-67.0.15.EL.cernVM OS: SLC 5.3 x86_64, kernel 2.6.18-128.1.1 for kvm virtio testsKVM version: 83Xen version: 3.2.1

Benchmarks: HEP-Spec06

Riccardo.Veraldi@cnaf.infn.it - HEPiX Spring 2009 Umea

1 4 7 8

‐Spe

Number of concurrent VMs

XEN vs. KVM on dual Intel E5420, single performance measure

Xen‐hvm Xen‐para KVM E5420

1vm 4vm 7vm 8vm

‐Spe

VMs vs. CPU

kvm xen‐hvm xen‐para E5420

E5410 E5420 E5420kvm E5420xen‐hvm E5420xen‐para E5430

‐Spe

8VMs aggregate vs. CPUs

Virtualization Technology

% loss from non emulated CPU (E5420, 8vm)

E5420kvm 3,42E5420xen-hvm 4,55E5420xen-para 2,02E5410 vs. E5420 4,07

Benchmarks: PI test

System tools to measure CPU usage are not reliable enoughCalculating PI with n arbitrary ciphers

Algorithm based on Numerical RecipesWritten in CCompiled with options

gcc -O3 -ffast-math -finline-limit=1000

Returns the number of CPU cyclesAn external function included into the code calculates the number of CPU cycles, using rdtsc IA32 registry

Benchmarks: PI test

2.1E+12

2.2E+12

2.3E+12

2.4E+12

2.5E+12

2.6E+12

2.7E+12

2.8E+12

2.9E+12

3.1E+12

host guest

CPU cycles

KVM host vs. guest

difference < 2%

Benchmarks: PI test

• 8 core• Balanced Performance• Every guest uses one core at 100%

2.1E+12

2.2E+12

2.3E+12

2.4E+12

2.5E+12

2.6E+12

2.7E+12

2.8E+12

2.9E+12

3.1E+12

guest singolo

guest1 guest2 guest3 guest4 guest5 guest6 guest7 guest8

CPU cycles

8 concurrent guest

Benchmarks: PI test

• Good balancing on guest• Max difference 14%, average ~5%

CPU cycles

17 concurrent guests

Benchmarks: Iperf

1vm 2vm 4vm 8vm host

Mbits/sec

iperf ‐w256k ‐P 5 ‐t 900

KVM Network Performance, e1000

Benchmarks: Iperf

kvm e1000 kvm virtio xen‐para xen‐hvm pv

Mbits/sec

iperf ‐w256k ‐P5 ‐t 900

KVM vs XEN, 1vm, network performance

Benchmarks: bonnie++, file vs lvm

100000

120000

140000

160000

Host 1 kvm file 1 kvm lvm 1 xen‐para file 1 xen‐para lvm 1 xen‐hvm file 1 xen‐hvm lvm

bonnie++ on a single VM

2GB Ram, 4GB data set, 1vm comparison

seq out block seq out rewrite seq in block

Benchmarks: bonnie++

100000

120000

1 kvm vm 4 kvm vm 8 kvm vm 1 xen‐para 4 xen‐para 8 xen‐para 1 xen‐hvm 4 xen‐hvm 8 xen‐hvm

2GB ram, 4GB data set, single

Benchmarks: bonnie++, kvm virtio

100000

120000

140000

Host hdb vdb

Bonnie++ on a single vm

32GB data set, lvm partition, single

Benchmarks: bonnie++

100000

120000

seq output block seq output rewrite seq input block

2GB ram, 4GB data set, 8vm aggregate

Conclusions

KVM proved good stability and reliabilityNo problems on running production machines for more than 4 weeksCPU performances are extremely good

Modern CPU virtualization technologies help!Network performances are fair with e1000, good with virtioDisk I/O seems the most problematic aspect

Other solutions have problems tooRequires sysadmins only a small effort

Even if looking promising, right now xen is the most performing solution

Future work

qemu snapshot featuresBackup and cloning of nodes

high-level VM managersOvirt, enomalism, ganeti

Bibliography and links

XenXen repository

KVMKvm repository

QuattorHEP-SPECNumerical Recipes: second edition in CComputer Systems: A Programmer's Perspective

Comparison between xen and kvm - Indico · 2018. 11. 17. · KVM seen by system admin rpms:...

Documents

Optimizing VM images for OpenStack with KVM/QEMU Chet Burgess Sr. Director, Engineering Metacloud

Securely integrating with QEMU - KVM · 2016-08-30 · KVM Forum 2016: securely integrating with QEMU Secrets: creation Password inline as clear text (insecure)-object secret,id=sec0,data=letmein

Yabusame: Postcopy Live Migration for Qemu/KVM€¦ · Yabusame: Postcopy Live Migration for Qemu/KVM Takahiro Hirofuchi, AIST Isaku Yamahata, VALinux Systems Japan K.K. KVM Forum

Low latency edge computing with QEMU/KVM: Challenges …€¦ · Low latency edge computing with QEMU/KVM: Challenges and future ... • Follows the original implementation of Jan

hypervisor to QEMU/KVM Painless switch from proprietary · 2016. 8. 30. · Switch options (low level) KVM + QEMU More devices supported QEMU is dumb and simple, does not make decisions

EntropyPoisoningfromtheHypervisor - MIT · QEMU can be run with KVM[9] or Xen acceleration, where QEMU does not perform the translations and simply allows the other virtualizer to

DevOps with Vagrant and KVM/qemu Hiroshi Miura (@miurahr ... · Vagrant-KVM vs. -libvirt Features Vagrant-KVM Vagrant-libvirt KVM Yes Yes Xen N.A. Plan Remote N.A. Experimental

Optimizing VM images for OpenStack with KVM/QEMU

- KVM€¦ · Knut Omang, KVM Forum/LinuxCon Aug 24, 2016 Experience in using Qemu/KVM as a tool to develop software for a

Evaluating storage APIs for QEMU - Indico · 2010-04-29 · The V-Word QEMU is used by Xen and KVM for I/O but... – this is not a virtualization talk Let's just think of QEMU as

KVM and CPU feature enablement - QEMU€¦ · KVM and CPU feature enablement Eduardo Habkost Developer Conference 2014. Agenda • Basic concepts • Existing

Dave Gilbert - KVM and QEMU

VIRTUALIZATION ON MULTICORE MPUS · KVM/QEMU –Overview •KVM/QEMU–open source virtualization technology based on the Linux kernel •KVM is a Linux kernel module •QEMU is a

Guest operating system debugging - KVM€¦ · Logging Printk, debugfs, ... Printf, logfiles, -d (tcg only) ... (via guest memory) Tracing KGTP, strace, dtrace .. e.g. qemu + kvm

QEMU 2.0 and Beyond - events.static.linuxfound.org · KVM Xen Android SDK (fork ... Minor forks like qemu-kvm and qemu-dm A few cases where major forks almost happened Tremendous

Using Intel Virtualization Technology (Intel VT) with ......Alternatively, use yum -y groupinstall Virtualization. This will install qemu-kvm qemu-img virt-manager libvirt libvirt-python

Wanos on QEMU/KVM and virt-manager …wanos.co/misc/wiki/documents/kvm/wanos_howto_kvm-virt...Wanos on QEMU/KVM and virt-manager Comprehensive guide for a complete lab This document

Live Block Device Operations in QEMU - FOSDEM › 2018 › schedule › event › vai... · KVM/QEMUvirtualizationcomponents Linux: KVM–/dev/kvm VM1 QEMU VM2 QEMU libvirtd OpenStack

Escape From The Docker-KVM-QEMU Machine-ENGconference.hitb.org/hitbsecconf2016ams/materials/D1T1 - Shengpin… · Escape From The Docker-KVM-QEMU Machine Shengping Wang, Xu Liu Qihoo

LCA14: LCA14-303: Virtualization update (KVM, Xen, QEMU 64-bit)