Comparison between xen and kvm - Indico · 2018. 11. 17. · KVM seen by system admin rpms: basically the kernel module and a modified version of qemu (qemu-kvm) Network configuration

XEN and KVM in INFN production systems and a comparison between them

Riccardo VeraldiAndrea ChiericiINFN - CNAFHEPiX Spring 2009

Outline

xenkvmTest descriptionBenchmarksConclusions

[email protected] - HEPiX Spring 2009 Umea 2

What we have now: Xen

Open source “industry standard” for virtualizationIncluded in all recent linux distributionsSupports para and hvm approachesWidely used in HEP communityWon’t go into details during presentation


What could be the future: KVMKernel-based Virtual MachineOpen sourceincluded in latest linux kernels: implemented as a module

A user space program uses /dev/kvm interface to set up VMs (qemu-kvm)

Supports hvm approachRather new to HEP communityQumranet now owned by Red Hat

Foreseen boost in development


KVM virtio

Direct access to hardwareHard disk (/dev/vdx)Ethernet interface

Support from kernel 2.6.25Supported since RHEL 5.3 (virtio backport)

kernel 2.6.18-128.1.1 Virtio drivers must be on the VM, not on the hostEGEE sw can’t currently benefit from this


KVM seen by system adminrpms: basically the kernel module and a modified version of qemu (qemu-kvm)Network configuration to be done by hand in order to get public IP

bridge-utils and tunctl rpms were used to set-up “tap” interfaceScript developed to automate the process

A modified init.d script to configure software bridge has been developed at CNAF

No VM configuration file direct supportVMs are launched via a standard UNIX command + command line optionslibvirt


KVM seen by system admin

No direct interface to manage VMs by command line (e.g. “xm”)

Libvirt supportVirt-manager workingMonitor interface

Very powerful, lots of options availableqemu already well documented, support available on-line


KVM introduced at CNAF

We use quattor to configure and install grid nodes and virtual machines

Profiles describe machine configurationNetwork boot for installation working

Keeps booting from network, need to restart with “-boot c”

No need for any modification, just like xen-hvm


Qualitative testCNAF is running LHCb tier2 site entirely on xen VMs (2 CEs, 1 SE)Changed one Computing Element with a KVM machine

Has been working for more that 3 weeks flawlesslyQuattor machine profile unmodified, no effort for sysadmin

CMS secondary squid server installed on the same host with same result (2 weeks ago)Running a KVM in production since 2 months for INFN-CA backups. Works flawlesslyKVM executed (and live migrated!) a win7 VMHardware used: 1 node, dual E5420, 16GB ram, sata disks via Areca controller


Quantitative test: description

Need some measures to understand what is the best solutionTested 3 classic parameters

CPU → hep-spec06 (v1.1), PI test for cpu distributionNetwork → iperf (v2.0.4)Disk access → bonnie++ (v1.94)

Compared Xen (para-virtualized and hvm) with KVM, using non virtualized machine as a baseline


Quantitative test: descriptionHardware used: 1 blade, dual E5420, 16GB ram, 10k sas disk via LSI logic raid controller (raid0)Xen-para VM specs: 1 vcpu, 2 GB ram, disk on a fileXen-hvm VM specs: 1 vcpu, 2GB ram, disk on a file, “netfront” network driverKVM VM specs: 1 vcpu, 2GB ram, disk on a file, e1000 network driver emulation (non-virtio tests)Host OS: SL 5.2 x86_64, kernel 2.6.18-92.1.22.el5VM OS: SLC 4.5 i386, kernel 2.6.9-67.0.15.EL.cernVM OS: SLC 5.3 x86_64, kernel 2.6.18-128.1.1 for kvm virtio testsKVM version: 83Xen version: 3.2.1


Benchmarks: HEP-Spec06

[email protected] - HEPiX Spring 2009 Umea

7.5

8

8.5

9

9.5

10

10.5

11

1 4 7 8

HEP

‐Spe

c06

Number of concurrent VMs

XEN vs. KVM on dual Intel E5420, single performance measure

Xen‐hvm Xen‐para KVM E5420

12



0

10

20

30

40

50

60

70

1vm 4vm 7vm 8vm

HEP

‐Spe

c06

VMs vs. CPU

kvm xen‐hvm xen‐para E5420

13



0

10

20

30

40

50

60

70

E5410 E5420 E5420kvm E5420xen‐hvm E5420xen‐para E5430

HEP

‐Spe

c06

8VMs aggregate vs. CPUs

14


Virtualization Technology

% loss from non emulated CPU (E5420, 8vm)

E5420kvm 3,42E5420xen-hvm 4,55E5420xen-para 2,02E5410 vs. E5420 4,07


Benchmarks: PI test

System tools to measure CPU usage are not reliable enoughCalculating PI with n arbitrary ciphers

Algorithm based on Numerical RecipesWritten in CCompiled with options

gcc -O3 -ffast-math -finline-limit=1000

Returns the number of CPU cyclesAn external function included into the code calculates the number of CPU cycles, using rdtsc IA32 registry


Benchmarks: PI test


2.1E+12

2.2E+12

2.3E+12

2.4E+12

2.5E+12

2.6E+12

2.7E+12

2.8E+12

2.9E+12

3E+12

3.1E+12

host guest

CPU cycles

KVM host vs. guest

difference < 2%

17

Benchmarks: PI test


• 8 core• Balanced Performance• Every guest uses one core at 100%

2.1E+12

2.2E+12

2.3E+12

2.4E+12

2.5E+12

2.6E+12

2.7E+12

2.8E+12

2.9E+12

3E+12

3.1E+12

guest singolo

guest1 guest2 guest3 guest4 guest5 guest6 guest7 guest8

CPU cycles

8 concurrent guest

18

Benchmarks: PI test


• Good balancing on guest• Max difference 14%, average ~5%

1E+12

2E+12

3E+12

4E+12

5E+12

6E+12

7E+12

CPU cycles

17 concurrent guests

19

Benchmarks: Iperf


0

100

200

300

400

500

600

700

800

900

1000

1vm 2vm 4vm 8vm host

Mbits/sec

iperf ‐w256k ‐P 5 ‐t 900

KVM Network Performance, e1000

in

out

20

Benchmarks: Iperf


0

100

200

300

400

500

600

700

800

900

1000

kvm e1000 kvm virtio xen‐para xen‐hvm pv

Mbits/sec

iperf ‐w256k ‐P5 ‐t 900

KVM vs XEN, 1vm, network performance

in

out

21

Benchmarks: bonnie++, file vs lvm


0

20000

40000

60000

80000

100000

120000

140000

160000

Host 1 kvm file 1 kvm lvm 1 xen‐para file 1 xen‐para lvm 1 xen‐hvm file 1 xen‐hvm lvm

K/sec

bonnie++ on a single VM

2GB Ram, 4GB data set, 1vm comparison

seq out block seq out rewrite seq in block

22

Benchmarks: bonnie++


0

20000

40000

60000

80000

100000

120000

1 kvm vm 4 kvm vm 8 kvm vm 1 xen‐para 4 xen‐para 8 xen‐para 1 xen‐hvm 4 xen‐hvm 8 xen‐hvm

2GB ram, 4GB data set, single


23

Benchmarks: bonnie++, kvm virtio


0

20000

40000

60000

80000

100000

120000

140000

Host hdb vdb

K/sec

Bonnie++ on a single vm

32GB data set, lvm partition, single


24

Benchmarks: bonnie++


0

20000

40000

60000

80000

100000

120000

K/sec

seq output block seq output rewrite seq input block

2GB ram, 4GB data set, 8vm aggregate

25

Conclusions

KVM proved good stability and reliabilityNo problems on running production machines for more than 4 weeksCPU performances are extremely good

Modern CPU virtualization technologies help!Network performances are fair with e1000, good with virtioDisk I/O seems the most problematic aspect

Other solutions have problems tooRequires sysadmins only a small effort

Even if looking promising, right now xen is the most performing solution


Future work

qemu snapshot featuresBackup and cloning of nodes

high-level VM managersOvirt, enomalism, ganeti


Bibliography and links

XenXen repository

KVMKvm repository

QuattorHEP-SPECNumerical Recipes: second edition in CComputer Systems: A Programmer's Perspective


http://www.xen.org/

http://www.gitco.de/repo

http://www.qumranet.com/

http://www.lfarkas.org/linux/packages/centos

http://www.quattor.org/

https://twiki.cern.ch/twiki/bin/view/FIOgroup/TsiBenchHEPSPEC

http://www.nr.com

http://csapp.cs.cmu.edu/

Documents

Comparison between xen and kvm - Indico · 2018. 11. 17. · KVM seen by system admin rpms: basically the kernel module and a modified version of qemu (qemu-kvm) Network configuration