Logical Domains

USE IMPROVE EVANGELIZE

Logical Domains (LDoms)Liam MerwickLDoms DeveloperSun Microsystems – SPARC Platform S/W Group

Page: 2


Contents

• Background• Features• Architecture• Components• Future Roadmap• Configuration Examples• Q & A

Page: 3


Background

Page: 4


Consolidation

• Conventional Unix server utilisation is 7 to 15%

• Data center compression> Goal: Increase utilisation of equipment

10%

20%

10% Utilisation

Time

Page: 5


Solutions from SunHard Partitions Virtual Machines OS Virtualisation Resource Mgmt.

Server

OS

App

Multiple OSes Single OSTrend to flexibility Trend to isolation

Dynamic System Domains Solaris Containers

(Zones + SRM)

Solaris Containers

for Linux Applications

Solaris Trusted Extensions

Solaris Resource Manager

(SRM)

SPARC Logical Domains

Solaris xVM

VMware ESX

Microsoft Virtual Server

CalendarServer Database Web

ServerSunRayServer

AppServerDatabaseMail

ServerWeb

ServerFile

ServerIdentityServer

AppServer Database

Page: 6


Virtualisation for SPARC platforms

• New “sun4v” architecture

sun4u code

Solaris Xupdate

(genunix)

CPU “Z”

Solaris X (sun4v)

SPARC CPU

SPARC hypervisor

Solaris X(genunix)

sun4vinterfaceUS-Z CPU

code

OperatingSystem

Platform

Page: 7


Virtual Machine for SPARC

• Thin software layer between OS and platform hardware

• Hypervisor + sun4v interface• Virtualises machine HW and isolates OS from

register-level• Delivered with platform not OS• Not itself an OS

SPARC hardware

Hypervisor

Solaris

UserApp

sun4v virtual machine

stableinterface“sun4v”

UserApp

OpenBoot

UserApp

Page: 8


PlatformHardware

LDom A

Logical Domains Technology

• Virtualisation and partitioning of machine resources> Each domain is a full virtual machine, with a dynamically reconfigurable sub-

set of machine resources, and its own independent OS> Protection & isolation via SPARC hardware and LDoms Hypervisor firmware

LDom B LDom C LDom D

I/O I/O

Memory

CPUCPU CPU CPU CPU CPU CPU CPU

Memory Memory Memory

Linux FreeBSD

LDomsHypervisor

OS Environment

of choice

Page: 9


Features

Page: 10


Fundamentals

• Each virtual machine should appear as an entirely independent machine> own kernel, patches, tuning parameters> own user accounts, administrators> own disks> own network interfaces, MAC & IP addresses> Start, stop and reboot independently of each other

• Paravirtualised OS

Page: 11


Features

• Add and remove cpus while OS is running• Protection and isolation via SPARC hardware and

LDoms Hypervisor firmware• Service domains for I/O

> direct I/O access

Page: 12


Features – UltraSPARC-T1 specific

• Up to 32 LDoms per box> CPU threads individually assignable to different LDoms

• Memory assignable to with an 8KB granularity• Two PCI busses independently assignable to the

same or different LDoms.• Guest Domains can be configured, started and

stopped independently> Without requiring a power-cycle of machine

Page: 13


Features – UltraSPARC-T2 specific

• 64 threads => up to 64 domains> more domains per box than with UltraSPARC-T1

• On-chip NIU assignable to any domain.

Page: 14


Architecture

Page: 15


Architecture

• Logical Domains overview• Hypervisor• LDoms Manager• Virtualised I/O• Dynamic Reconfiguration (DR)

Page: 16


Hypervisor Support

• Hypervisor software is responsible for maintaining separation between domains> Using extensions built into a sun4v CPU

• Also provides Logical Domain Channels (LDCs) so that domains can communicate with each other> Mechanism by which domains can be virtually networked

with each other, or provide services to each other

Page: 17


LDoms Manager

• One Manager per host HV> Controls Hypervisor and all its LDoms

• Exposes control interface via CLI• Maps Logical Domains to physical resources

> Constraint engine> Heuristic binding of LDoms to resources

> Assists with performance optimisation> Assists in event of failures / blacklisting

Page: 18


Direct I/O

Hypervisor

I/OBridge

Nexus Driver

/pci@B

Device Driver

/pci@B/qlc@6

PCIRoot

I/O MMU

Logical Domain

Virtual Nexus I/FHyper Privileged

Privileged

Hardware

PCI-Express

Logical Domain owns PCI root and tree

• Traditional model> Existing drivers and devices

continue to work

App

App

App

Page: 19


Virtualised I/O

Logical Domain A

Hypervisor

I/OBridge

Nexus Driver/pci@B

Device Driver/pci@B/qlc@6

PCIR

oot

I/O MMU

Service Domain

Virtual Nexus I/FHyper Privileged

Privileged

Hardware

PCI B

Domain Channel

App

Virtual DeviceDriver

AppApp

App

Virtual DeviceService

Page: 20


Logical Domain 2

AppApp

App App

vDiskclientdriver

Logical Domain 3

AppApp

App App

vDiskclientdriver

Logical Domain 1

AppApp

App App

vDiskclientDriver

Virtual (Block) Disk device

Hyper-visor

Service Domain

Virtual SAN 1

Virtual SAN 2

I/O Bridge

FC-ALI/F

vDiskServer

vDiskServer

Device-Driver

Page: 21


Logical Domain 2

AppApp

App App

V-EtherDriver

Logical Domain 3

AppApp

App App

V-EtherDriver

Logical Domain 1

AppApp

App App

V-EtherDriver

Virtual Ethernet device

Hyper-visor

I/O Bridge

V-EtherSwitch

Service Domain

GbEther I/F

V-EtherSwitch

Virtual LAN 1: 192.168.0/24

Virtual LAN 2: 63.24/16

Device-Driver

Page: 22


Dynamic Reconfiguration (DR)

• Ability to dynamically grow or shrink compute capacity of an LDom on demand

• No need to re-boot Solaris• Simply add / remove:

> CPUs> Memory (future)> I/O (future)

• Improve utilisation by balancing resources between LDoms

Page: 23


LDoms Manager

Logical Domain 2

AppApp

App

OS DRStack

Logical Domain 3

AppApp

App

OS DRStack

Logical Domain 1

AppApp

App App

OS DRStack

Hyper-visor

LDoms Manager

Service Domain

HV DRStack

Page: 24


Example: vCPU reconfiguration

V

AppApp

Logical Domain 1

Hyper-visor

V

P P

V

AppApp

Logical Domain 1

Hyper-visor P P

V

AppApp

Logical Domain 2

P

V

Example command line operations:% ldm remove-vcpu 1 Logical_Domain_1% ldm add-vcpu 1 Logical_Domain_2

Page: 25


Components

Page: 26


Components – Hardware & Software

• Hardware> UltraSPARC-T1 (Niagara) family of CMT processors

> T1000 Netra CP3060 T6300> T2000 Netra T2000

> UltraSPARC-T2 (Niagara2) family of CMT processors> T5120 T5220 T6320

• Operating System> OpenSolaris (build 70+ recommended)> Solaris 10 11/06 (U3) or later> Linux 2.6.23

Page: 27


Components – Hardware & Software

• Firmware> 6.4.6 or later> 6.5.x and 7.0.x now available

• LDoms Manager Software> 1.0.1 (Solaris package - available as Free download)> 1.0 (For use with 6.4.x firmware)

• OS can and will be released/updated independently of Firmware/LDoms Manager

Page: 28


Future Roadmap

Page: 29


Latest Release: LDoms 1.0.1

• Rebooting of Control, Service and IO domains> Without bringing down guest domains

• Support for new platforms• Updates to Firmware and LDoms Manager package• Upgrade to Solaris 10 8/07 (U4) recommended

> Independent of LDoms 1.0.1

Page: 30


Key Features on Roadmap• Warm Migration

> Suspend and resume

• Live Migration

• Hybrid I/O

• Dynamic Reconfiguration> Virtual and Physical I/O> Memory

• Booting/Installing from DVD/ISO

• Using ZFS/SVM volumes

Page: 31


Configuration Examples

Page: 32


Configuring a machine to run LDoms• Make resources available (remove from control domain)

primary#> ldm set-mau 2 primary

primary#> ldm set-vcpu 8 primary

primary#> ldm set-mem 8g primary

• Configure services

primary#> ldm add-vcc port-range=5000-5100 primary-vcc0 primary

primary#> ldm add-vds primary-vds0 primary

primary#> ldm add-vsw net-dev=e1000g0 primary-vsw0 primary

Page: 33


Adding another domain (I)

• Add disk deviceprimary#> ldm add-vdsdev /dev/dsk/c1t0d0s2 vol1@primary-vds0

• Adding resourcesprimary#> ldm create domain1

primary#> ldm add-mem 2g domain1

primary#> ldm add-vcpu 4 domain1

primary#> ldm add-vdisk vdisk1 vol1@primary-vds0 domain1

primary#> ldm add-vnet vnet1 primary-vsw0 domain1

• Show domain constraintsprimary#> ldm list -l domain1

Page: 34


Adding another domain (II)

• Booting the newly created domain

primary#> ldm bind domain1

primary#> ldm start domain1

• Connect to the console

primary#> telnet localhost 5000

Page: 35


LDoms Community

• Newly Created community

• Website> http://www.opensolaris.org/os/communites/ldoms

• Alias> [email protected]

http://www.opensolaris.org/os/communites/ldoms

mailto:[email protected]

Page: 36


More information

• BigAdmin Portal> http://www.sun.com/bigadmin/hubs/ldoms/

• Sun Blueprint> Beginners Guide to LDoms: Understanding and Deploying

Logical Domains> http://www.sun.com/blueprints/0207/820-0832.pdf

• Training Course> Solaris 10 Logical Domains Administration (SA-345-S10)> http://uk.sun.com/training/catalog/courses/SA-345-S10.xml

http://www.sun.com/bigadmin/hubs/ldoms/

http://www.sun.com/blueprints/0207/820-0832.pdf

http://uk.sun.com/training/catalog/courses/SA-345-S10.xml


Liam MerwickSun [email protected]://blogs.sun.com/merwick

mailto:[email protected]

http://blogs.sun.com/merwick