BCO2891-Taking VMware vSphere Storage APIs - Data Protection to the Limit Pushing the Backup Performance Envelope_Final_US.pdf

Taking VMware vSphere Storage APIs - Data Protection to the Limit: Pushing the Backup Performance Envelope

Abdul Rasheed, Symantec

George Winter, Symantec

Roger Andersson, Cisco

INF-BCO2891

#vmworldinf

INF-BC02891 - Pushing the Backup Performance Envelope 2

INF-BC02891 Pushing the Backup Performance Envelope

George Winter Technical Product Manager, Symantec

Abdul Rasheed Technical Marketing Manager, Symantec

Taking vSphere Storage APIs for Data Protection to the Limit

Roger Andersson Director, Technical Marketing, Cisco

Agenda

Goal of Performance Benchmark 1

Intro to vStorage APIs for Data Protection 2

Performance Benchmark Environment 3

vStorage APIs for Data Protection Best Practices 4

Benchmark Results 5

3 INF-BC02891 - Pushing the Backup Performance Envelope

Goal of Performance Benchmark

4 4 INF-BC02891 - Pushing the Backup Performance Envelope

Goal of Performance Benchmark

Evaluate performance characteristics of VADP

• Adjust the variables for maximum backup throughput • Adjust the variables for minimum intrusion to ESXi

Test drive underlying infrastructure performance

Obtain real world performance numbers

Develop VADP best practice guidelines


An Introduction: vStorage APIs for Data Protection


vStorage APIs for Data Protection (VADP)

• Supports ESX 3.5 U2 (and later)

Introduced with vSphere 4

Change Block Tracking (CBT) provides true incremental backups

• Requires no backup software on virtual machines

Very simple implementation


VADP Backup Process

8

VMware ESXi

Datastore

Backup Server

INF-BC02891 - Pushing the Backup Performance Envelope

ESXi Host

VADP Backup Process

9

VMware ESXi

Datastore

Backup Server

VADP Snapshot of Target VM(s)


ESXi Host

VADP Backup Process

10

VMware ESXi

Datastore

Backup Server VM Backup

Sent to Server


ESXi Host

VADP Backup Process

11

VMware ESXi

Datastore

Backup Server

Snapshot Released


ESXi Host

VADP Backup Snapshot Process

12

A Closer Look


Detailed Look at the VADP Snapshot Process

• During backup VADP creates snapshot

– VSS provider flushes OS buffers within VM

– Snapshot created, VMDKs are frozen

– Redo log created – all writes redirected to redo log

– VM data copied to backup storage

– Redo log applied to original VMDKs

– Snapshot released - backup completed

• Why does this matter? – Each step involves significant I/O

– Reducing snapshots per Datastore improves backup performance & reliability

– Incremental backups are quick and reduces snapshot impact

13

VMware ESXi


VMware ESXi

VADP SAN Transport

14

Hypervisor level snapshots are taken for virtual machines

Virtual Machine data is streamed directly from storage to backup server

Backup server processes and stores data in disk, deduplicated disk, tape or cloud

100% true offhost backup

Suited for enterprise

Fibre Channel and iSCSI storage


VMware ESXi

VADP Network Block Device (NBD) Transport

15


Virtual Machine data is streamed via ESXi VMkernel ports to backup server


Simplest to implement

Fibre Channel, iSCSI, DAS, NFS

May strain VMkernel if not planned well


VMware ESXi

VADP Hotadd Transport

16


Virtual Machine snapshots are attached to a dedicated Proxy VM that streams data to backup server


Careful planning provides offhost backups

Throttle using traffic shaping

Fibre Channel, iSCSI, NFS, limited DAS

Proxy VM


Benchmark Environment



• FlexPod specifications for VMware vSphere – NetApp FAS and V-Series storage – Cisco UCS 6248

– VMware vSphere 5

• Symantec NetBackup 5220 VMware Ready Backup Appliance – VMware vStorage API support built-in – Inline Symantec V-Ray capable deduplication

• Labs sponsored by Datalink



19

Cisco UCS Blades

1x vCenter, 6x ESXi

Nexus

NetBackup 5220 NetApp FAS 6080 NetApp V-Series 6080

Block Storage

8Gb Fibre Channel

10Gb Ethernet



20

Cisco UCS Blades

1x vCenter, 6x ESXi

Nexus


Block storage

8Gb Fibre Channel

10Gb Ethernet


Cisco UCS • 2 x 6248 Fabric Interconnects • 4 x B200-M3 Server (ESXi 5.0) • 2 x B230-M2 Server (ESXi 5.0) • 1 x B440-M2 Server (vCenter 5.0) • 2 x 2208 IO Modules

Cisco UCS B-Series Blade Servers and UCS Manager

Cisco Nexus Family Switches

NetApp FAS OnCommand Software Suite

Features Standard, pre-validated, converged

platform Virtualized and non-virtualized

environments

Flexible: One platform scales up or out to fit most workloads Add Applications and workloads

Benefits Flexibility Built-in data center efficiencies Reduced risk Secure Multi-tenancy

Joint Cisco and NetApp Solution for Virtualized Infrastructure and Cloud

• UCS designed from ground up for next generation data center

• Tightly integrates x86 servers,

adapters and LAN/SAN connectivity

• Policy-Driven Integrated Management across Blade and Rack servers

• UCS Virtual Interface Adapter

• I/O Bandwidth Capacity

• Extended Memory Technology

• Policy-Based Infrastructure Management

CPU

Mem

ory

VM VM VM

VM

VM VM

VM

VM VM

VM

• Virtualizes more Applications • Increase VM density • Increase visibility and

management

Infrastructure Management

This is an upcoming product—Feature set for first release subject to change

Unified Management at Scale 10,000 Blade and Rack Servers

Coming in 2H 2012

UCS Manager UCS Manager Data Center 1

UCS Manager UCS Manager Data Center 2

UCS Manager Data Center 3

UCS Central

• Unifies management of multi UCS domains

• Leverages UCS Manager technology

• Simplify global operations with centralized inventory, faults, logs and server console

• Delivers global policies, service profiles, ID pools and templates

Foundation for high availability, disaster recovery and workload mobility

• Model based API for large scale automation

UCS Central

Best CPU Performance

Best Virtualization Performance

Best Cloud Computing Performance

Best Enterprise Application Performance

Best Enterprise Middleware Performance

Best HPC Performance

VMmark 2.0 Overall B200 M2

VMmark 2.1 2-socket Blade B200 M2

VMmark 1.x 2 –socket Blade B230 M1

VMmark 1.x Overall C460 M1

VMmark 1.x 2-socket B200 M1


VMmark 1.x Overall C460 M1

VMmark 1.x Blade Server B440 M1


VMmark 2.1 Overall C460 M2

VMmark 2.1 Two–node 4-socket C460

M2

VMmark 2.1 4-socket C460 M2

SPECompLbase2001 2-socket B200 M2

SPECompMbase2001 2-socket B230 M2


SPECompMbase2001 4-socket C460 M2



LinPack 2-socket B200 M2

LS-Dyna 4-socket C460 M1



Oracle E-Business Suite Medium Model Payroll

Batch B200 M2

Oracle E-Business Suite Xtra Large Model Payroll B200

M3

Oracle E-Business Suite Medium Model Payroll

Batch B200 M2

Oracle E-Business Suite Medium Model

Order-to-Cash B200 M2

Oracle E-Business Suite Large Model

Order-to-Cash B200 M3

Oracle E-Business Suite Ex-large Model Payroll Batch

B200 M2

SPECjbb2005 2-socket C260 M2

SPECjbb2005 2-socket B230 M2



SPECjbb2005 X86 2-socket B200 M2

SPECjbb2005 X86 4-socket C460 M1

SPECjAppServer2004 2-node B230 M1



SPECjAppServer2004 1-node 2-socket C250 M2

SPECfp_rate_base2006 2-socket

C260 M2

SPECint_rate_base2006 2-socket

C260 M2

SPECint_rate2006 X86 4-socket

C460 M2

SPECint_rate_base2006 X86 4-socket C460 M1

SPECint_rate_base2006 X86 2-socket B200 M2


SPECfp_rate_base2006 X86 2-socket B200 M2


SPECfp_rate_base2006 X86 4-socket C460 M1

SPECfp_rate_base2006 X86 2-socket B200 M1

SPECjEnterprise2010 Overall B440 M1

SPECjEnteprise2010 2-node B440 M2

Cisco UCS Benchmarks that held world record performance records as of date of publication

Oracle E-Business Suite Xtra Large Model Payroll Batch

B230 M2




VMmark 2.1 2-socket B200 M3

TPC-C Oracle DB 11g & OEL

C250 M2

TPC-H 1000GB Microsoft SQL Server

C460 M2

SPECjbb2005 X86 2-socket C220 M3

SPECfp_rate_base2006 X86 2-socket C220 M3

SPECint_rate_base2006 X86 2-socket C220 M3

SPECfp_base2006 X86 2-socket

C220 M3

TPC-H 100GB VectorWise

C250 M2

TPC-H 300GB VectorWise

C250 M2


26

Cisco UCS Blades

1x vCenter, 6x ESXi

Nexus


Block Storage

8Gb Fibre Channel

10Gb Ethernet

NetApp Storage • Dual head FAS 6080 cluster • 8 DS14MK4 storage trays • 15k RPM, 300GB FC drives • Hosting storage for ESXi

hosts 1 and 4



27

Cisco UCS Blades

1x vCenter, 6x ESXi

Nexus


Block Storage

8Gb Fibre Channel

10Gb Ethernet


NetApp Storage • Dual head V-Series

6080 cluster • 8 DS14MK4 storage

trays • Block Storage • Hosting storage for

ESXi hosts 2,3,4 & 6


28

Cisco UCS Blades

1x vCenter, 6x ESXi

Nexus


Block Storage

8Gb Fibre Channel

10Gb Ethernet


Symantec NetBackup 5220 • VADP access via 2x10GigE • VADP access via 2x8Gb FC • Built-in V-Ray deduplication

Backup System Used for Benchmark N

etBa

ckup

522

0

• Dual Intel E5620 CPUs • 24GB RAM • 4TB Dedupe storage • Connectivity • 2x8Gb Fibre Channel • 2x10Gb Ethernet

• Typical power consumption: < 415W


vStorage API for Data Protection Best Practices


VMware ESXi

VADP SAN Transport


Virtual Machine data is streamed directly from storage to backup server


100% true offhost backup

Suited for enterprise

Fibre Channel and iSCSI storage

Cisco UCS

NetApp Storage

NetBackup 5220


Cisc

o an

d N

etAp

p Fl

exPo

d

VMware ESXi

VADP Network Block Device (NBD) Transport


Virtual Machine data is streamed via ESXi VMkernel ports to backup server


Simplest to implement

Fiber Channel, iSCSI, DAS, NFS

May strain VMkernel if not planned well

Cisco UCS

NetApp Storage

NetBackup 5220


Cisc

o an

d N

etAp

p Fl

exPo

d

Intel Case Study: 10Gb Ethernet for Virtualization and file transfer

33

Theoretical max

Everyday copy workloads

ESXi management traffic cap

Source: Intel case study, Maximizing Gigabit performance for file transfer and virtualization

Single threaded streams cannot deliver!


Intel Case Study: 10Gb Ethernet for Virtualization and file transfer

34

Theoretical max

Esxi management traffic cap

8 streams

3-4 streams sufficient to

meet management

traffic cap

Source: Intel case study, Maximizing Gigabit performance for file transfer and virtualization


Network impact of NBD on ESXi host

35

Single stream

2 streams 3 streams

Diminishing returns on improving sustained throughput after 3-4 streams because of traffic cap


CPU impact of NBD on ESXi host

36

Single stream 2 streams 3 streams

Average 5% CPU utilization per stream for entire host


VADP Performance Characteristics

• Single ESXi host – NBD Performance Per Stream

37

# of streams: 1 2 3 4

100 MB/Sec

100 MB/sec

75 MB/Sec

60 MB/Sec 55

MB/Sec

4 Stream Aggregate

Throughput = 220 MB/sec

Based on Cisco/VMware/NetBackup

Benchmark Testing



• Single ESXi host – NBD Performance Per Stream

38


100 MB/Sec

100 MB/sec

75 MB/Sec

60 MB/Sec 55

MB/Sec

4 Stream Aggregate

Throughput = 220 MB/sec

Based on Cisco/VMware/NetBackup

Benchmark Testing

220 MB/sec Creates VMkernel strain on ESXi host



• Four Separate ESXi hosts – Creating One Stream Per ESXi

39

100 MB/Sec

100 MB/sec

Possible Network Performance

100 MB/Sec

100 MB/Sec

100 MB/Sec




• Four Separate ESXi hosts – Creating One Stream Per ESXi

40

100 MB/Sec

100 MB/sec

Possible Network Performance

100 MB/Sec

100 MB/Sec

100 MB/Sec

Aggregate Throughput = 400 MB/sec

400 MB/sec Creates Less Strain on VMkernel per

ESXi host



Best Practices in using NBD and 10Gb

• Limit 3-4 streams from a given ESXi system

• The VMkernel (management) port may stay in the default VM Network, rest of the bandwidth complements VM traffic

• Automated in NetBackup using VMware Resource Limits

41

Limit streams per ESXi Illustration


Tip for NetBackup Users!

Best Practices in using NBD and 10Gb

• Backup VMs from multiple ESXi systems to fill the pipe to backup server

– 1 backup from 10 ESXi hosts is better than 10 backups from a single ESXi host!

– Effective utilization of backup server – Concurrency = Smaller backup

window – Lower strain on each ESXi host

– Automated in NetBackup using VMware Intelligent Policies

42

Distribute over multiple ESXi hosts

VMware ESXi

VMware ESXi

VMware ESXi



VMware ESXi

VADP Hotadd Transport


Virtual Machine snapshots are attached to a dedicated Proxy VM that streams data to backup server


Careful planning provide offhost backups

Throttle using traffic shaping

Fibre Channel, iSCSI, NFS, limited DAS

Proxy VM Cisco UCS NetBackup 5220

NetApp Storage


Cisc

o an

d N

etAp

p Fl

exPo

d

Best Practices for Hotadd Transport

• Proxy VMs need to access the datastore of VMs being protected

• Recommended to have a Proxy VM at each ESXi system

• Proxy VMs can undergo vMotion or VMware HA

• Proxy VM Operating System – SuSE Enterprise Server for VMware – SuSE Enterprise Server – Microsoft Windows

• Proxy VMs do not need to be backed up, exclude them using Intelligent Policy in NetBackup

44

Proxy VM Considerations

VMware ESXi VMware ESXi


Proxy VM 1

Proxy VM 3 Proxy VM 4

Proxy VM 2


Tips for NetBackup Users!

Best Practices for Hotadd Transport

• Offload deduplication at the source

• Deduplicated data sent to backup server, reduces network utilization

• NetBackup lets you select deduplication location at Proxy VM level for flexibility

45

Scale-out deduplication



Proxy VM 1

Proxy VM 3 Proxy VM 4

Proxy VM 2



Hotadd with Deduplication at Proxy: Impact on Virtual Machine Network in ESXi

46

Datastore: Single stream

~200 MB/sec Average

Datastore: 2 streams

~175 MB/sec Average

Network: Single stream

~1 MB/sec Average


~1 MB/sec Average

Up to 99% bandwidth savings!


Hotadd with Deduplication at Proxy: Impact on ESXi CPU

47

Datastore: Single stream

~200 MB/sec Average

Datastore: 2 streams

~175 MB/sec Average

CPU: Single stream 13% Average


13% Average


Benchmark Results


0 200 400 600 800 1000 1200

Performance Advantage: NetBackup 7.5 and NetBackup 5220 for VMware

VMware Consolidated Backup

Backup Throughput – MB/sec

B-Mark 1: 63 MB/sec


0 200 400 600 800 1000 1200




B-Mark 1: 63 MB/sec

50

B-Mark 2: 600 MB/sec VADP – Cisco / NetBackup


0 200 400 600 800 1000 1200




B-Mark 1: 63 MB/sec

51



VADP – Cisco / NetApp FlexPod / NetBackup B-Mark 3: 1340 MB/sec

0 200 400 600 800 1000 1200




B-Mark 1: 63 MB/sec

52




0.27 TB/hr

0 200 400 600 800 1000 1200




B-Mark 1: 63 MB/sec

53




0.27 TB/hr

2.1 TB/hr

0 200 400 600 800 1000 1200

Performance Advantage: NetBackup 7.5 for VMware



B-Mark 1: 63 MB/sec

54




0.27 TB/hr

2.1 TB/hr

4.8 TB/hr

Single VADP host!

0 200 400 600 800 1000 1200

Performance Advantage: NetBackup 7.5 for VMware



B-Mark 1: 63 MB/sec

55




270 VMs

2100 VMs

4800 VMs

Single VADP host!

Weekend full backups, daily incremental backups, 100GB VMs, 60GB data

NetBackup SE Interlock 2012

Restore Performance

• Restore process involves more I/O than backup process – Disks (vmdk’s) must be first created as target of restore – Type of vmdk can impact restore speed and I/O required

• Single restore typically won’t saturate restore path – Typically restore performance about half of backup performance – As with backup – balance restores across ESX or Datastore

• Restore performance highly dependent on VMDK format – Thin – faster restores when data is small

percent of VMDK reserved size

– Thick – faster when VMDK nearly full

56

Thank you!

Copyright © 2012 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

57

George Winter, Symantec Corporation Abdul Rasheed, Symantec Corporation Roger Andersson, Cisco Systems, Inc.


FILL OUT A SURVEY

EVERY COMPLETE SURVEY IS ENTERED INTO DRAWING FOR A

$25 VMWARE COMPANY STORE GIFT CERTIFICATE

Taking VMware vSphere Storage APIs - Data Protection to the Limit: Pushing the Backup Performance Envelope

Abdul Rasheed, Symantec

George Winter, Symantec

Roger Andersson, Cisco

INF-BCO2891

#vmworldinf

Documents

BCO2891-Taking VMware vSphere Storage APIs - Data Protection to the Limit Pushing the Backup Performance Envelope_Final_US.pdf