Consolidation of a Performance Sensitive Application: Virtualizing Electronic Sports League's Gaming Infrastructure Breakout Session PO3435 Bob AlbersSreeram

Consolidation of a Performance Sensitive Application: Virtualizing Electronic Sports League's Gaming Infrastructure

Breakout Session PO3435

Bob Albers Sreeram SammetaI/O Usage Architect, Intel Senior Systems Engineer, Intel2008-08-26

Server Applications:

• Can We Virtualize* Everything?

• New Technologies Allow Us To Virtualize* More

• Question The Assumptions

• ROI Delivered

Agenda

*In this context “virtualize” means to use a VMM or hypervisor

Can We Virtualize Everything?Can We Virtualize Everything?

Perception: “Non-virtualize-able” Applications

Mission Critical

Transaction Latency Sensitive

CPU Intensive: Single Thread vs. Multi-Thread

Memory Intensive: Size/Throughput/Latency

I/O Intensive: Disk/Network; Throughput/Latency

Perception: Virtualization impacts all of the above

Therefore: These types of applications “can’t be virtualized” because the tradeoffs are too severe

Perception or Reality?

Electronic Sports League

Largest online gaming community in Europe

>844,000 active users*

Mission Critical Game Servers

Transaction Latency Sensitive

Single Thread CPU Intensive

Can We Virtualize Everything?Can We Virtualize Everything?

“Can’t Be Virtualized” but …

ESL motivated to try: sprawl, cost, power, OpEx

Internet

Game Servers

…

Firewall

* Source: ESL web site 2008-08-12

ESL Proof of Concept (PoC) Overview

Hypothesis: virtualization of gaming servers may be possible if …

We use the latest technologies

Intel® Xeon® Processor 7400-based servers with Intel VMDq NICs

VMware ESX* 3.5U1 & NetQueue

Test network latency impact of virtualization in the Intel lab

Private testing @ ESL lab and public testing on the Internet with real ESL members

New Technologies Allow Us To Virtualize MoreNew Technologies Allow Us To Virtualize More

Prove it!

ESL Network

vSwitch1 vSw0

VM1

VMn

Svc Cons

10 GbE VMDq 1 GbE

Firewall

InternetInternet

Force 10 S50N

PoC Server Hardware

Intel® Xeon® Processor 7400 Series (Dunnington)

HW and SW compatible with existing Quad-core 7300 based platforms

Performance boost from 6-core with new 16 MB L3 cache

Energy efficient boost from 45nm high-k process technology

Enhanced hardware assist features for virtualization

Available September ’08

32 slots256 GB max

32 GB as tested

Configurable PCI Express*

ESB2 I/O Bridge

4 sockets with 6-cores each

7400 Series4x1066 MHz

FBDFBDFBDFBDFBD

FBDFBDFBDFBDFBD

FBDFBDFBDFBDFBDFBDFBDFBDFBDFBD

Intel 7300


Importance of I/O Tuned For Virtualization

VMM overhead

Switching load

Interrupt bottleneck

VM1 VM2 VMn

Virtual NIC Virtual NIC Virtual NIC…

VirtualizationHypervisor

NIC

Source: Intel. Throughput measures receive side (Rx) I/O performance of 10GbE LAN.

LAN

Th

rou

gh

pu

t (G

B)

0.0

2.0

4.0

6.0

10.0

8.0

4.0

Result: NIC performance can be up to ~60% under utilized

Unused I/O

capacity

w/o VMDq

Source: Intel.


Optimizing I/O For Virtualization

VM1 VM2 VMn…

VMware with NetQueue

NIC with VMDq

LAN

Virtual NIC Virtual NIC Virtual NIC

Th

rou

gh

pu

t (G

B)

0.0

2.0

4.0

6.0

10.0

w/o VMDq

8.0

4.0

Source: Intel.

9.59.2

• 2x throughput• Near native 10GbE

w/ VMDq w/ VMDq Jumbo Frames

Tests measure Wire Speed Receive (Rx) Side Performance With VMDq on Intel® 82598 10 Gigabit Ethernet Controller


VMDq & NetQueue

Optimize switching

Load balance interrupts

PoC Software

VMware ESX* 3.5U1 Virtual Center 2.5 VMDq & NetQueue

enabled (16 queues on one 10G port)

1 vCPU per VM 2GB memory per VM

Windows Server* 2003 32-bit

CounterStrike* 1.6


Intel® Xeon® Processor 7400 Series based Server

VMware® ESX 3.5 U1

Counter Strike 1.6

Windows Server 2003

32 bit

VM1

Counter Strike 1.6

Windows Server 2003

32 bit

VMn

Enough theory…

let’s move on to engineering & testing!

Key Performance Metric: In-Game Transaction Latency

In-game transaction latency = round-trip network latency + game server processing time

= >20mS typical or 10mS best case on Internet

=~ 5mSec best case for LAN connected players

In -G a m e L a te n c y (m S e c )

0

5

10

15

20

25

B e s t L A NB e s t In te rn e tT yp . In te rn e t

Question The AssumptionsQuestion The Assumptions

How much does virtualization increase round-trip Network Latency?

Source: ESL

Source: ESL observations. Actual performance may vary.

40-200 bytes UDP

2000 bytes UDP

Player Game Server

Round-Trip Network Latency Test

Virtualization increases latency; ~double

VMDq latency is near-native @ 0.13mS for 64-byte packets

Negligible since In-Game Transaction Latency is 5mS best case

Question The AssumptionsQuestion The Assumptions

N e tp e rf 2 .4 .4 U D P Late n cy te s t w ith 8 p ara lle l s tre am s

0 .0 0

0 .0 5

0 .1 0

0 .1 5

0 .2 0

0 .2 5

0 .3 0

6 4 2 5 6 1 0 2 4

P ac k e t S ize (b y te s)

Avg

. lat

ency

(ms)

N a ti ve

V M D q

N oV M D q

Network Latency w VMDq is near Native!

Negligible impact to In-Game Latency!

Source: Intel Lab. Performance measured using the netperf benchmark running on Intel® Xeon® processors 7300 (2.93 GHz). Actual performance may vary.

Source: Intel lab

ESL Game Testing

Private ESL & Public Internet testing revealed no impact on In-Game Transaction Latency!

ROI DeliveredROI Delivered "Playing on virtualized gameservers running on Intel and VMWare

technologies gives professional gamers no disadvantages compared

with playing on a non virtualized server. Everything ran smoothly and I

did not notice anything unusual. A perfect setup for professional

gaming."

—Navid Javadi aka mousesports|Kapio

Source: ESL Lab. Performance Source: ESL Lab. Performance measured using esxtop & power meter measured using esxtop & power meter with ESL reference s/w stack running with ESL reference s/w stack running on Intel® Xeon® processor 7400 on Intel® Xeon® processor 7400 Series based platform. Power Series based platform. Power measurement is wall power for the measurement is wall power for the entire server as tested including 8 SAS entire server as tested including 8 SAS drives. Actual performance may vary.drives. Actual performance may vary.

ESL virtual game servers on Intel® Xeon® Processor 7400 Series based platform

020406080

100

24 32 36 40

# of VMs

CPU

(%)

620640660680700720740

Pow

er (W

atts)

Source: ESL lab

ESL Game Testing Results

ROI DeliveredROI Delivered

"The new Six-Core Intel® Xeon® 7400 series processors

(“Dunnington”) were completely overwhelming in all terms. The Intel Xeon MP servers with Intel VMDq technology enable us to efficiently run our servers with reduced costs and without any

negative impacts."

—Bjoern Metzdorf

Director Information TechnologyElectronic Sports League

Power savings of $648,000 per year for each group of 1000 servers converted!

Source: ESL Lab. Performance measured using esxtop & power meter with reference s/w stack running on Intel® Xeon® processors 7400. Power savings calculated based on 24x7x365 usage, ESL actual power rate & Yahoo $/€ exchange rate as of 2008-08-12. Power numbers and cost savings noted here are for server wall power only. No consideration of cooling power costs and other potential savings is included here. Actual performance and savings may vary.

Before After

Server 1P C2D 4P Xeon 7400

Cores 2 24

Game server processes

3 per CPU core

3 per VM; 4.5 per CPU core

VMs per box - 36

Game server processes per box

6 108

Users per box 72 1296

CPU Utilization 60-80% 75%

Power per physical server

350W 710W

Consolidation Ratio

18 1

Annual direct power cost

$731,000 per 1000 servers

$83,000 for 56 servers

Summary

Server Applications:

• Can We Virtualize Everything?

• New Technologies Allow Us To Virtualize More

• Question The Assumptions

• ROI Delivered

Do you have “non-virtualize-able” applications?

Evaluate the latest technologies.

Prove it!

Additional sources of information on this topic:

Intel Virtualization Developer Community http://softwarecommunity.intel.com/articles/eng/1424.htm

Intel Virtualization Technology http://www.intel.com/technology/virtualization/index.htm

Intel® Xeon® Processor 7300 Series http://www.intel.com/performance/server/xeon_mp/virtualization.htm?iid=SEARCH

More information on Intel® VT for Connectivity: www.intel.com/go/vtc

More information on Intel Networking solutions: www.intel.com/network

Intel® Virtualization Technology for Connectivity Tech Brief http://softwarecommunity.intel.com/isn/downloads/virtualization/pdfs/20137_LAD_VTc_Tech_Brief_r04.pdf

Virtual Machine Device Queues White Paper http://www.intel.com/technology/platform-technology/virtualization/VMDq_whitepaper.pdf

VMware ESX Server 3 Configuration Guide – Update 2 for ESX 3.5 http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_3_server_config.pdf Search for NetQueue/VMDq configuration info

Intelligent Queueing Technologies for Virtualization: Intel-VMware white paper http://download.intel.com/network/connectivity/products/whitepapers/Intel-VMware_VMDq_wp_May08.pdf

Intel 4-Processor Server System S7000FC4UR http://www.intel.com/products/server/systems/4-processor-s7000fc4ur/4-processor-s7000fc4ur-overview.htm

We want to hear from you…..

Join the conversation on Intel’s Open Port: The Server Room

Bring your questions to the “Ask an Expert” discussion forum

Share Best Practices with IT influencers

Engage with Intel on your DataCenter challenges

Check out -- communities.intel.com/community/server

Contact Info :Wm. Hank Lea:

[email protected] Hilliker:

[email protected]

Q&A

Legal DisclaimerINFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS.

Intel may make changes to specifications and product descriptions at any time, without notice.

All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.

Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Dunnington and other code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user

Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance.

Intel, Intel Inside, Intel Xeon and the Intel logo are trademarks of Intel Corporation in the United States and other countries.

*Other names and brands may be claimed as the property of others.

Copyright © 2008 Intel Corporation.

Risk Factors

This presentation contains forward-looking statements that involve a number of risks and uncertainties. These statements do not reflect the potential impact of any mergers, acquisitions, divestitures, investments or other similar transactions that may be completed in the future. The information presented is accurate only as of today’s date and will not be updated. In addition to any factors discussed in the presentation, the important factors that could cause actual results to differ materially include the following: Demand could be different from Intel's expectations due to factors including changes in business and economic conditions, including conditions in the credit market that could affect consumer confidence; customer acceptance of Intel’s and competitors’ products; changes in customer order patterns, including order cancellations; and changes in the level of inventory at customers. Intel’s results could be affected by the timing of closing of acquisitions and divestitures. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by the timing of new Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including product offerings and introductions, marketing programs and pricing pressures and Intel’s response to such actions; Intel’s ability to respond quickly to technological developments and to incorporate new features into its products; and the availability of sufficient supply of components from suppliers to meet demand. The gross margin percentage could vary significantly from expectations based on changes in revenue levels; product mix and pricing; capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; excess or obsolete inventory; manufacturing yields; changes in unit costs; impairments of long-lived assets, including manufacturing, assembly/test and intangible assets; and the timing and execution of the manufacturing ramp and associated costs, including start-up costs. Expenses, particularly certain marketing and compensation expenses, vary depending on the level of demand for Intel's products, the level of revenue and profits, and impairments of long-lived assets. Intel is in the midst of a structure and efficiency program that is resulting in several actions that could have an impact on expected expense levels and gross margin. Intel's results could be impacted by adverse economic, social, political and physical/infrastructure conditions in the countries in which Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Intel's results could be affected by adverse effects associated with product defects and errata (deviations from published specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust and other issues, such as the litigation and regulatory matters described in Intel's SEC reports. A detailed discussion of these and other factors that could affect Intel’s results is included in Intel’s SEC filings, including the report on Form 10-Q for the quarter ended June 28, 2008.

Documents

Consolidation of a Performance Sensitive Application: Virtualizing Electronic Sports League's Gaming Infrastructure Breakout Session PO3435 Bob AlbersSreeram