Virtualization: an old concept transforming datacenters in new ways

technology brief

Abstract.............................................................................................................................................. 2 Introduction......................................................................................................................................... 2 Layers of virtualization in the datacenter................................................................................................. 2 Virtualization within software layers ....................................................................................................... 4

Multitasking OS ............................................................................................................................... 4 Virtual machines .............................................................................................................................. 4 Cluster file systems ........................................................................................................................... 6

Virtualization using hardware within the platform .................................................................................... 7 Processor virtualization ..................................................................................................................... 7

Protected ring layers ..................................................................................................................... 7 Newest virtualization extensions and protected ring layers ................................................................ 8

Storage RAID................................................................................................................................... 9 Virtualization in the network layers ...................................................................................................... 10

Network partitioning ...................................................................................................................... 10 Platform collection .......................................................................................................................... 10

HP direction for future ........................................................................................................................ 11 Virtualized resources using industry-standard platforms....................................................................... 11 Automated resource management .................................................................................................... 12

Conclusion........................................................................................................................................ 13 Glossary........................................................................................................................................... 14 For more information.......................................................................................................................... 15 Call to action .................................................................................................................................... 15

Abstract Virtualizing compute or storage resources in the datacenter is becoming increasingly important for improved flexibility, availability, and productivity. Even though it is gaining prominence and being incorporated more often in the datacenter, virtualization is not a new concept. This paper defines virtualization and provides several examples to show how resources are already being virtualized within the network layers, in the platform hardware layers, and in the software layers of the datacenter. HP is working internally and with external working groups to ensure that the next steps in virtualization will allow servers to be easily pooled, shared, or re-provisioned, based on the needs of the business.

Introduction The terms virtual and virtualization are seen throughout the IT industry—virtual machines, virtual memory, virtual network, virtual storage, CPU virtualization, and so on. Furthermore, other terms such as utility computing, grid computing, and Adaptive Enterprise refer to computing strategies that incorporate strategic goals of virtualizing compute resources. While these terms often refer to new products and may be presented as new technologies, the concept of virtualization is not new.

In this technology brief, the term virtualization refers to abstracting, or masking, a physical resource to make it appear different logically than it is physically. All forms of virtualization share at least two common elements: First, the virtualization technology abstracts a physical resource to make it appear logically different than its physical reality. This abstraction allows resources to be pooled and/or shared. Second, there is always a management layer or control point for the virtualized resources. As virtualization technologies multiply in the datacenter, it is important for IT managers to understand how these various control points might affect security and how each control layer interacts with other management control layers.

Layers of virtualization in the datacenter Every IT datacenter is comprised of layers of resources, as illustrated in Figure 1:

• Software layers—including operating systems, business applications, and software applications such as virtual machines

• Server platform hardware—including processors, memory, local I/O, and distributed I/O connectivity hardware

• Network or system interconnect layer—including the network managers and switches for storage area networks (SANs) and local area networks (LANs)

• Power and cooling infrastructure for the datacenter

2

Figure 1. Layers of resources in the datacenter architecture: most already include virtualization technologies

Network Layer

Platform HardwareLayer

Infrastructure Layer

Platform Software Layer

Memory

Local Interconnect (PCI)

Distributed Network Connection

Cluster (RDMA/IB) LAN (Ethernet) SAN (iSCSI, FC)

Direct-attachedstorage

BIOS PlatformManager

Network Manager

Cluster, LAN, SAN Endpoints(NIC, HBA, HCA)

Firmware

Power Cooling

KVMCPU

System software (applications, OS)

I/O Slots

Switch

The ultimate goal of IT administrators is to build an adaptive datacenter that can share any IT resource so that utilization is optimized and supply automatically meets demand—within all the architecture layers. Many components within these layers already have the capability to be virtualized resources.

As more and more components and layers within the datacenter are virtualized, the IT administrator gains:

• Increased flexibility because resources can be moved as needed • Increased scalability because resources can be scaled up or down based on changing workload

demands • Improved resiliency by simplifying backup, failover, and disaster recovery solutions • Reduced total cost of ownership as physical resources are used to their fullest potential

With those benefits, virtualization also brings challenges, including:

• Increased management complexity. Every time a specific resource is virtualized, another control point or layer of management is added, regardless of the layer in which the abstraction is occurring. Therefore, it is critical that virtual resources use an efficient management structure. Furthermore, as the IT administrator establishes more virtualized resources, there becomes a need for a single, unified management layer throughout the datacenter to reduce overhead in managing multiple types of virtualized resources.

• Unknown performance levels and models for meeting performance levels. Virtualizing a resource involves management overhead that can reduce raw hardware performance in some cases. In addition, IT administrators need to make hardware and software decisions based on meeting business objectives or application performance levels. As more of the datacenter becomes virtualized, it will be important to understand service-level agreements (SLAs) and to build a service-

3

oriented architecture that includes robust monitoring and metering practices that ensure SLAs are being met.

• Increased acquisition costs for management and hardware infrastructure. Although the end result may reduce costs because of increased utilization, there may be large initial acquisition costs to establish a virtualized solution.

• Uncertain software licensing models. As virtualization solutions become more prevalent, vendors will need to address how their software is licensed and paid for. Today’s software is normally licensed based on the physical hardware on which the software is operating. However, as software instances are established on virtual hardware, the cost structures will need to comprehend those changes.

Virtualization within software layers Numerous examples exist of ways that the platform software layers (applications and operating systems) provide virtualized solutions in today’s datacenter. For example, a preemptive multitasking OS shares the processor across several different applications, making it appear as if all are running simultaneously. Virtual machines are an extension of this concept, in which software abstracts a server’s physical resources (processors, memory, and I/O) to share the hardware across multiple instances of operating systems, making it appear as if operating systems are running simultaneously on different hardware. Clustered file systems also use a software layer to share a common file system across multiple platforms, making it appear as if the storage devices are all using a single file system.

Multitasking OS Multitasking operating systems have been widely available in the PC world since IBM and Microsoft released OS/2 in 1987. Multitasking operating systems provide one of the earliest examples of how software layers (in this case, the OS kernel) can manage the hardware so that it is shared across applications, providing flexibility to the end user.

A preemptive multitasking OS determines how long (based on time or priority levels) any single application can use the processor. If it is a time-based function, the OS allows one application to run for a defined time period. Then the OS preempts that application off the processor, saving all its state and task information in special registers. The OS then allows a second application to use the processor for this same amount of time. The OS will continue to preempt each application after its allotted time, saving the state information each time, and returning to that state information when each application resumes operation. Thus, the OS virtualizes the processor so that multiple applications share its resources. This increases flexibility by allowing the user to run multiple applications “simultaneously.” Although at first, performance levels between standard DOS and multitasking operating systems were compared and questioned, this type of virtualization has become essential to most users. The incurred overhead is minimal compared to the flexibility that the end-user gains.

Virtual machines Today, one of the most common examples of software abstraction is the use of virtual machine software to consolidate multiple operating systems onto a single server. Virtual machines are also known as “soft” or “logical” partitions of a server. Virtual machine software abstracts the physical resource and creates one or more virtual instances of that physical resource. In other words, virtual machine software can create multiple “virtual machines,” each with its own virtual drives, virtual NICs, virtual host-bus adapters, virtual processor, etc. Each physical resource is shared among these multiple virtual machines. Each virtual machine can load its own OS and applications (Figure 2) and is isolated from the actions of any other OS.

A virtual machine uses a separate software layer—sometimes referred to as a hypervisor or virtual machine monitor—to abstract the computer’s hardware. It acts as a supervisor, ensuring security and

4

cleanliness of the shared resources (such as making sure that all buffers are flushed and state information is saved). It also manages the access of each guest OS (that runs in the virtual machine instance) to the physical hardware resources.

Prominent examples of virtual machine software include Microsoft® Virtual Server, VMware ESX Server and GSX Server, and Xensource Xen (open-source) software. Virtual machine software can use either para-virtualization or full virtualization techniques. Para-virtualization requires the guest OS to be modified before it can run in a virtual machine instance. Modifying the OS simplifies the work of the virtualization layer. Xen uses this technique. Full virtualization refers to techniques such as those that Microsoft-Virtual Server uses, in which an un-modified OS runs in the virtual machine instance. VMware uses a hybrid technique in which the VMware kernel performs a binary-translation of a portion of the guest OS code that must operate at the most privileged level of processor instructions.

Figure 2. A software layer abstracts the physical resources so that each instance of the OS and application(s) appears to have its own NIC, processor, disk, and memory, when in fact these are “virtual” instances.

HP is adding value to virtual machine solutions by providing a centralized management resource for virtual machines. The HP ProLiant Essentials Virtual Machine Management (VMM) Pack provides a single management and control point for virtual machines established using Microsoft Virtual Server, VMware ESX Server, or VMware GSX Server. VMM Pack integrates with HP Systems Insight Manager (SIM), providing the ability to manage both physical and virtual resources from a user interface that is consistent with HP SIM. It allows IT administrators to associate the virtual machine to the physical host and to identify how resources are being consumed in a virtual or physical machine. For example, IT administrators can easily identify virtual machines or host servers reaching high CPU, memory, or disk utilization levels. VMM Pack also simplifies management by providing a way for the IT administrator to control remotely the virtual machine functions (such as start, stop, copy, move, and backup).

VMM Pack provides the foundation for automating physical to virtual migrations through the HP ProLiant Essentials Server Migration Pack. The Server Migration Pack enables the functionality to migrate virtual machines and their operating systems from physical (P) to virtual (V) platforms (P2V), from virtual to physical (V2P), or from one virtual machine to another (V2V).

5

Cluster file systems Cluster file systems are another common example of how application software is used to abstract physical resources such as hard disk drives. For example, clustered servers are often connected to a SAN that contains many disk drives and storage arrays (Figure 3). To simplify access to data stored on the SAN, cluster file system management software is added to each server node. In this way, each server accesses a common file system that is distributed across multiple servers. The client accessing the data is unaware of whether the data is spread across multiple arrays or multiple disk drives: It simply requests the data, and the cluster file system software layer performs any necessary translations to acquire the data from the correct location in the SAN. Furthermore, any server in the cluster can write to or read from any file in the SAN—all while maintaining a consistent view of the data, even if it is being viewed by multiple servers in the cluster.

Cluster file systems reduce the challenges that system administrators or programmers have in managing multiple file systems across a cluster and also improve the storage utilization efficiency by allowing disk capacity to be shared among multiple servers. Examples of cluster file system software include HP StorageWorks Scalable File Share, PolyServe® Matrix Server, and Red Hat Global File System.

Figure 3. Example of a clustered file system

6

Virtualization using hardware within the platform The platform hardware level of the datacenter architecture consists of initiators (such as servers and clients) and target devices (such as storage arrays and printers). This section reviews some examples of how the components within server platforms—such as processors, memory, storage, and so on—provide virtualization. For example:

• Processor virtualization through the use of protected ring layers in x86 processors and through new virtualization extensions

• Storage RAID and RAID memory technology

Even though the virtualization functions reside in the hardware layers, they are not independent of software. Each virtualization function also requires some software infrastructure.

Processor virtualization Because all x86 processors are based on designs originally developed decades ago, there are many cases in which the processor hardware performs some level of masking or translation to enable added functionality. These functions have been developed in hardware to:

• Provide backward compatibility for operating systems and applications (for example, 16-bit mode versus 32-bit mode versus 64-bit mode)

• Enable multitasking operating systems to preempt applications (for example, protected ring layers) • Enable the hardware resources to more easily run multithreaded code without adding additional

execution units (for example, HyperThreading technology)

Protected ring layers An x86 processor includes four different “ring” layers which differentiate the type of instructions that the processor can execute, depending on what software is calling the instruction (Figure 4). In a typical multitasking OS, the OS operates in “ring 0,” in which it has full access to all the processor and platform resources, such as memory mapping. This is the most privileged level, also referred to as kernel mode. Applications typically operate in “ring 3,” sometimes referred to as user mode, in which functions such as memory mapping are restricted to keep one application from adversely affecting another application. Originally, ring layers 1 and 2 were designed to house elements such as device drivers and the OS file system; however, these layers are not typically used today. The device drivers now typically reside with the OS in ring 0, and the file system resides with the application in ring 3 or with the OS in ring 0. The protected ring layers in the processor hardware work hand-in-hand with the OS to enable a multitasking OS to preempt an application off the processor, providing the appearance that multiple applications are operating simultaneously.

7

Figure 4. In a typical multitasking OS, the application resides in ring 3, while the OS kernel and device drivers reside in ring 0.

Application

OS Kernel/DD Ring 0

Ring 3

Typical Multitasking OS

ProcessorPlatform hardware layer

Software layersRing 2 (typically not used)

Ring 1 (typically not used)

Newest virtualization extensions and protected ring layers AMD and Intel have recently announced an expansion of this protected ring layer concept with their hardware-based AMD Pacifica technology and Intel Virtualization Technology. These virtualization extensions are designed to accommodate virtual machine monitors more efficiently and to reduce some of the software overhead that such a virtualization layer requires.

Virtual machine software such as Microsoft Virtual Server and VMWare ESX Server adds an additional software layer—the virtualization layer—into the software stack. Because the virtualization layer must be able to control all the physical resources of the system, just as the OS normally does (with privileged instructions), the virtualization layer normally operates in ring 0 and moves the OS into ring 1 or ring 3 (Figure 5). However, this can create a conflict and potential excessive system fault when the OS issues a privileged instruction but it is not operating at the privileged ring 0 level. The result is that the virtual machine monitor must act as a fault handler, executing many lines of code to determine what the OS has requested and handling the request as an intermediary between the OS and the hardware.

8

Figure 5. Today, a typical virtual machine environment allows the OS to operate in ring 1 or ring 3 of the processor privileged instructions.

Application

OS Kernel/DD Ring 0

Ring 3

Typical Multitasking OS

Processor

Virtualization layer Ring 0

Ring 3

Multitasking OS in aVirtual server environment

Processor

Ring 2

Ring 1 Guest OSKernel/ DD

Ring 2

Ring 1

Guestapplication

Guestapplication

Guest OSKernel/DD

With virtualization extensions, software such as virtual machine monitors will be able to run in a more privileged ring layer. This will allow guest operating systems to run in their normal ring 0 layer. In addition, AMD Pacifica and Intel Virtualization Technology are developing hardware support for new instructions that are specific to virtual machine monitors. The combination of these two factors (higher-privilege ring level and new hardware instructions) will provide a more efficient way for the virtual machine monitors to virtualize the hardware: It will simplify the way the virtual machine monitor preempts an instance of an OS, in the same way that an OS preempts an application today in a multitasking OS environment.

Storage RAID For years, the computer industry has used arrays of disk drives to improve read/write performance, increase availability, and provide fault tolerance for disk drive subsystems. In one of its most basic forms, an array controller manages the physical disk drives and “combines” multiple physical disk drives into a larger, single, logical disk drive or logical unit number (LUN) from the standpoint of the OS. Storage RAID technology took this basic concept of a logical drive and added exclusive-OR engines to calculate parity data and then store that data on a portion of the physical drive(s). In the event of a physical drive failure, the OS is unaware that anything has changed in the logical drive: The RAID controller can recreate the data from parity information even though a physical drive is no longer accessible. Thus, storage arrays and RAID technology in disk drives present a virtualized or abstracted view of the physical drives to the OS. The array controller manages that virtualization process.

9

Virtualization in the network layers Two common ways that virtualization occurs today within the network layers of a datacenter use partitioning techniques, such as in virtual local area networks (VLANs) and platform collection techniques.

Network partitioning Ethernet VLANs allow a network administrator to define logical groups of any network-attached end devices including printers, storage, or compute nodes (for example, servers, workstations, or clients). The administrator can provide separate virtual LANs that logically isolate one group of nodes from another—for example, setting up all servers, switch ports, or users related to marketing on one VLAN and setting up all accounting servers on a different VLAN. Even though both may exist on the same physical network, the VLAN controller in the switch masks the physical attributes of the network by keeping the two VLANS logically isolated.

On the other hand, a VLAN can also define virtual domains that communicate as if they were on the same physical network, even if they are physically located on different Ethernet segments (Figure 6). The use of virtual segments allows multiple physical segments to behave logically as if they were a single network domain. These same techniques are used for other virtualized networks such as Fibre-Channel storage area networks zones and InfiniBand Pkey (partition key) networks.

Figure 6. Example of logical group of network components across physically separate network segments.

Platform collection More recently, network resources are being virtualized through the use of platform collection technologies, such as those developed by Troika Networks, Inc., (acquired by QLogic Corporation in 2005), and TopSpin Communications, Inc. In platform collection technologies, a single unified fabric connects multiple servers to Ethernet networks and/or SANs (Figure 7).

10

Figure 7. Representation of a platform collection fabric

Host Host Host

Switchfabric

Switchfabric

SANAddressMapping

Virtual -Physical

EthernetAddress Mapping

Virtual-Physical

EthernetEthernetFibre ChannelFibre Channel

For example, a platform collection fabric can use a hardware-based switch device to virtualize the Ethernet (IP) or SAN (Fibre Channel) addresses to the servers. The switch device typically performs all translations of the actual, physical addresses in the LAN or SAN to the virtual addresses that are presented to the servers by means of the switch fabric. Thus, the platform collection fabric virtualizes the LAN or SAN network resources: Even though a server appears to be connecting with a specific local I/O endpoint such as an individual NIC, in reality the endpoint is determined by the fabric and could be distributed across any of the physical NICs. In other words, this technique presents a number of servers as a “super” endpoint on a traditional Ethernet or SAN fabric.

HP direction for future The previous examples show that virtualization is applied in many different ways in today’s datacenter. HP is working toward a next-generation datacenter architecture that will encompass even more virtualization technologies to create datacenters in which resources are optimized and supply automatically meets demand. Even the infrastructure resources—such as power and cooling—are being investigated as resources that can be moved or partitioned in response to changing business requirements. To meet this goal, HP believes that two key components are needed: virtualized hardware resources and simplified, automated management of those resources.

Virtualized resources using industry-standard platforms HP is developing key technologies in both hardware and system software that will simplify virtualizing industry-standard servers. Thus, a server may be identified by its logical or virtual identity (the applications and OS it is running and the type of LAN and SAN connections it has) rather than by its physical identity. HP is evaluating options for virtualizing these I/O connections to make it easier to move a logical identity from one physical resource to another (Figure 8).

11

Figure 8. HP vision for transforming physical resources to virtual and adaptive resources

OS

Hosted virtualization layer(VMware GSX

Microsoft Virtual Server)

“Bare metal” virtualization layer

(VMware ESX)

vm1 vm2 vm3 vm1 vm2 vm3

EthernetEthernet

Virtualization assistance

FibreChannel

FibreChannel

“software identity”

“I/O identity”

Logical serverLogicalserver

FibreChannel

FibreChannel

EthernetEthernet

vm = virtual machine

As a result of servers having a logical identity that can be moved or re-configured as needed, IT administrators will be able to improve:

• Resource utilization by deploying resources where they are needed • Disaster recovery and failover scenarios by seamlessly transferring to different physical resources

HP will use its engineering expertise gained through developing the ProLiant and Integrity server lines to provide world-class virtualization solutions. Furthermore, HP plans to develop ProLiant virtualization solutions that will be based on industry standards and will be consistent with existing ProLiant BL, DL, and ML architectures.

Automated resource management As shared and virtualized resources continue to proliferate in the datacenter, it is extremely important that these layers of complexity can be managed efficiently, for example by using a unified resource management control model. HP is poised to deliver such a management resource by expanding the roles of HP Systems Insight Manager and HP Openview so that they can provide coordinated management functions for all logical resources. It is expected that HP SIM and ProLiant Essentials software tools will supply the automated management needed for monitoring hardware and for provisioning or reprovisioning servers. In addition, HP plans to expand the role of HP Openview to monitor and deliver computing resources, allowing the next-generation datacenter to meet business objectives such as service level agreements.

12

Conclusion Virtualization is a concept that is being widely touted now because of the pressing need for computing resources that are truly flexible and adaptive. However, virtualization is not a new concept. It is simply taking a resource, masking physical characteristics, and dividing it among multiple users (scale-in virtualization) or conglomerating multiple resources into a single, larger resource (scale-out virtualization). Multiple examples exist of ways in which virtualization already exists in the layers of the datacenter—in the platform software, platform hardware, and networking infrastructure layers.

HP is working with internal development teams and with working groups from industry-standard bodies to ensure that the next-generation datacenter will be equipped with new levels of virtualization, especially in the area of virtualizing industry-standard server hardware.

13

14

Glossary The following terms are used in the text of this document.

Term Definition

Adaptive enterprise

HP term for its initiatives to enable virtualization and automation throughout its portfolio of products.

CPU Central processing unit, or microprocessor

Grid computing A computing architecture that shares the processing and storage resources of many separate computers to solve large-scale computation problems. It differs from a cluster in that the compute resources are often connected through the Internet.

Hypervisor Another name for a virtual machine monitor, or the software layer that manages the process of creating virtual machines in a physical server. Hypervisor software ensures that only one virtual resource controls a physical resource at any given time and ensures security and cleanliness of the shared resources.

Initiator Initiators, such as servers and clients, initiate requests on the network.

LUN Logical unit number. Originally applied to an address for an individual disk drive within a SCSI storage array, the term now commonly refers not to the address of an individual disk drive but to the address of an individual volume, or virtual drive, within a storage array.

Target device Target devices, such as storage arrays and printers, are devices that wait until contacted by an initiator before performing functions.

Utility computing “Utility computing is a service provisioning model in which a service provider makes computing resources and infrastructure management available to the customer as needed, and charges them for specific usage rather than a flat rate.” Definition from WhatIs.com website, at: http://whatis.techtarget.com/definition/0,,sid9_gci904539,00.html

Virtual LAN Virtual local area network

Virtual machine A virtual machine provides a faithful implementation of a physical processor’s hardware running in a protected and isolated environment. Virtual machine is a term that is often used interchangeably with virtual server.

Virtual machine monitor

Software layer that runs directly on the server hardware to enable virtual machines. Also known as a virtualization layer or a hypervisor.

For more information For additional information, refer to the resources listed below.

Resource description Web address

AMD website

Processor virtualization information (AMD Virtualization™ AMD-V ™)

www.amd.com/us-en/Processors/ProductInformation/0,,30_118_8796,00.html

HP website

Industry-Standard Server Technology Papers

ProLiant servers

ProLiant Essentials (VMM Pack)

www.hp.com/servers/technology

www.hp.com/go/proliant

www.hp.com/servers/proliantessentials

Intel® website

Processor virtualization information

www.intel.com/technology/computing/vptech/

Call to action Please send comments about this paper to: TechCom@HP.com.

© 2006 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

AMD, AMD Opteron, and combinations thereof are trademarks of Advanced Micro Devices, Inc.

Intel and Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

TC060109TB, 1/2006