© 2014 VMware Inc. All rights reserved.

Internship @ VMwareMemory management inside ESXi

Julien Freche

September 5, 2014

CONFIDENTIAL 2

Introduction

• GISTRE 2014 – LSE – ACU

• Strongly interested in kernel programming

• Previous internship @ Aldebaran Robotics

• R&D – ESX Platform – VMkernel - Resource Management

CONFIDENTIAL 3

Table of content

1. Company presentationa) VMwareb) Products

2. Team presentationa) ESX Platformb) VMkernel & RM

3. Subject of the internship

3. Memory management inside ESXi

a) Overcommitmentb) Resource distribution

4. Technical realizationa) Guest statsb) Memory Schedulerc) Benchmark & results

CONFIDENTIAL 4

Company Presentation

CONFIDENTIAL 5

Company presentation - VMware

• HQ: Palo Alto, California

• Founded in 1998

• World leader in virtualization solutions

• First company to successfully virtualize the x86 architecture

• It offers servers and desktop solutions

CONFIDENTIAL 6

Company presentation - VMware

• 14,300 employees worldwide

• ~5000 in Palo Alto

• CEO: Pat Gelsinger

• 150,000+ Customers

• 98% of the Fortune 500

CONFIDENTIAL 7

Company presentation - Products

• Desktop– Workstation: hosted hypervisor that runs on Windows and Linux

– Fusion: hosted hypervisor for Mac OS X

• Infrastructure as a service (IaaS)– vCloud: public cloud - virtual machines migration from a private

cloud.

CONFIDENTIAL 8

Company presentation - Products

• Server– ESXi: the bare metal hypervisor

– vCenter: software that allows to manage multiple ESXi

– vSphere: software suite that includes ESXi and vCenter

– NSX: network virtualization engine

– Virtual SAN: Virtual data storage integrated with vSphere

CONFIDENTIAL 9

Team Presentation

CONFIDENTIAL 10

Team Presentation – ESX Platform

• ESX Platform: engineers that work on ESXi.– VMkernel team– Virtual Machine monitor team– User world team

CONFIDENTIAL 11

Team Presentation – VMKernel

• VMKernel team: responsible for the kernel of ESXi– Resource Management Team

– Core Team

– Hardware Team

• Resource Management Team– Responsible for CPU and Memory management.

– Philip Moltmann: Mentor during my internship

– Igor Braslavsky: Manager during my internship

– Xavier Deguillard: EPITA – GISTR 2012

CONFIDENTIAL 12

Subject of the internship

CONFIDENTIAL 13

Subject of the internship

Use statistics from the guest OS to:

• Reclaim memory without hurting the VM

• Make smarter scheduling decisions

CONFIDENTIAL 14

Memory management inside ESXi

CONFIDENTIAL 15

Memory management - Overcommitment

Allocate more memory to the Virtual Machines than physically available on the host.

CONFIDENTIAL 16

Memory management - Overcommitment

Technics:

• Allocation on demand,

• Memory sharing,

• Ballooning,

• Memory compression,

• Swapping

CONFIDENTIAL 17

Memory management - Sharing

Use only one physical pages for virtual pages with the same content.

Pages are marked as read only.

CONFIDENTIAL 18

Memory management - Balloning

A driver inside the guest will allocate pages, making them unusable for any other purpose in the guest OS but usable by the hypervisor.

CONFIDENTIAL 19

Memory management – Swapping and Zipping

Swap a page to disk at the hypervisor level.

Try to compress a page instead of swapping it when possible.

CONFIDENTIAL 20

Memory management – Resource Tree

Memory is balanced using a resource tree.

• R: Reservation or min

• L: Limit or max

• S: Shares

CONFIDENTIAL 21

Memory management - Active

• Estimate the number of pages used by a VM

• Based on a mathematical model

• Only used when overcommitting memory.

Drawbacks:

• Locked pages may not be accounted as active.

• Evolve slowly: over 5min to adjust to a new workload.

• Only decrease under high memory pressure.

CONFIDENTIAL 22

Technical Realization

CONFIDENTIAL 23

Technical realization

• Benchmark framework

• Guest statistics analysis and retrieval

• Design of a new memory scheduler

CONFIDENTIAL 24

Technical realization – Guest stats

• lol

CONFIDENTIAL 25

Technical realization – Benchmark framework

• Benchmark framework

• Guest statistics analysis and retrieval

• Design of a new memory scheduler

CONFIDENTIAL 26

Technical realization – Memory Scheduler

• Benchmark framework

• Guest statistics analysis and retrieval

• Design of a new memory scheduler

CONFIDENTIAL 27

Technical realization – Results

• Specjbb 1-5-1

CONFIDENTIAL 28

Technical realization – Results

– vdi

CONFIDENTIAL 29

Technical realization – Results

– +22

CONFIDENTIAL 30

Technical realization – Results

– +15

CONFIDENTIAL 31

Technical realization – Future

• Make it production ready

• Set balloon target automatically

CONFIDENTIAL 32

Special thanks to

• Xavier Deguillard for his recommendation

• Philip Moltmann, for being my mentor,

• Igor Braslavsky, my manager, and the whole resource management team

• Renaud Voltz, former EPITA and LSE student,

• EPITA and the LSE for this awesome work environment, and all the teachers and people I had the pleasure to work with,

• To my family and my friends for their support

CONFIDENTIAL 33

The end

Thanks for you attention.