Upload
alvin-sutton
View
244
Download
12
Tags:
Embed Size (px)
Citation preview
Advanced XenServer TrainingCCAT
Q1 - 2013
© 2012 Citrix | Confidential – Do Not Distribute
Coursework Agenda
• XenServer Overview• XenServer Components Deep-Dive
ᵒ Lab 1: Navigating Dom0, Logs and the CLI
• XenServer Storage Primitives (Caveman-ish?)ᵒ Lab 2: Playing with Storage
• XenServer Networkingᵒ Lab 3: NICs, NICs and more NICs
• XenServer Troubleshootingᵒ Lab 4: How do I fix this stuff?
Hypervisor OverviewWhat is XenServer?
© 2012 Citrix | Confidential – Do Not Distribute
Topics Covered
• Comparisons to other hypervisorsᵒ VMware vSphere 5.1ᵒ Microsoft Hyper-V 2012
• Real world applications for XenServerᵒ XenAppᵒ XenDesktopᵒ PVS
Hypervisor OverviewXenServer vs. The Competition
© 2012 Citrix | Confidential – Do Not Distribute
Market Share and Growth Comparison
Company Hypervisor Platform
2011 Growth Current Estimated Market Share
Microsoft Hyper-V 62% 25%
Citrix XenServer 25% 15 – 20%
VMware ESX 21% 50%
© 2012 Citrix | Confidential – Do Not Distribute
VMware vSphere
• Current Version – 5.1
• Consists of two components: VMware ESX and vCenter Server ᵒ ESXi is the hypervisor and installs on bare metal hardware. ᵒ vCenter Server provides centralized management and allows administrators to
configure and monitor ESXi hosts, provision virtual machines, storage, networking, and much more.
• Management mechanism: vSphere Client ᵒ Windows application that acts as a single pane of glass to manage either a standalone
ESXi host directly or an entire datacenter though vCenter.
© 2012 Citrix | Confidential – Do Not Distribute
vSphere Architecture diagram
© 2012 Citrix | Confidential – Do Not Distribute
vSphere VM Architecture diagram
© 2012 Citrix | Confidential – Do Not Distribute
Hyper-V
• Current Version – Hyper-V 2012
• Originally released in Server 2008
• Bundled with Windows Server licenses
• Behind VMware, Hyper-V retains the most market share in server virtualization
• Consists of single component: Windows Server with Hyper-V roleᵒ Windows Server is the base hypervisor
• Management mechanism: SCVMMᵒ SCVMM (System Center Virtual Machine Manager) provides centralized management
and allows administrators to configure and monitor hosts, provision virtual machines, storage, networking, etc.
ᵒ Can tie in directly with MOM/SCOM and Systems Management Server
© 2012 Citrix | Confidential – Do Not Distribute
Hyper-V Architecture diagram
© 2012 Citrix | Confidential – Do Not Distribute
Hyper-V VM Architecture diagram
© 2012 Citrix | Confidential – Do Not Distribute
Why is Hyper-V important?
• Bundled in with Windows Server 2012, Hyper-V provides free virtualization, and enterprise features at no additional cost
• Bridging functionality gap with VMware
• Tight integration with Windows infrastructure
• Microsoft has had much success with midmarket companies (<1000 users), winning 30 – 40% of the time
© 2012 Citrix | Confidential – Do Not Distribute
XenServer
• Current Version – 6.1
• Consists of single component: XenServer Coreᵒ CentOS-based hypervisor
• Management mechanism: XenCenterᵒ XenCenter console which is Windows only is the management tool. It directly connects
to the current Pool master in order to send commands to the rest of the XS pool ᵒ Has APIs for monitoring and management (To be discussed in more detail later).
© 2012 Citrix | Confidential – Do Not Distribute
XenServer VM Architecture diagram
© 2012 Citrix | Confidential – Do Not Distribute
Limitations Comparison
Parameter Hyper-V 2012 XenServer 6.1 vSphere 5.1
Host Cores 320 160 160
RAM 4 TB 1 TB 2 TB
VMs 1024 150 (60 w/HA) 512
vCPUs 900 2048 (25/core)
Cluster/Pool VMs 4000 1024 (600/SR) 4000
Hosts 64 16 32
VM CPUs 64 16 64
RAM 1 TB 128 GB 1 TB
NUMA Yes Host Only Yes
© 2012 Citrix | Confidential – Do Not Distribute
Feature Comparison
Feature Hyper-V 2012 XenServer 6.1 vSphere 5.1
Incremental Backups Yes Yes (VM Prot & Reco) Yes
NIC Teaming Yes Yes Yes
Integrated HA Yes Yes Yes
OS App Monitoring Yes No API-Only
Failover Prioritization Yes Yes Yes
Affinity & Rules Yes API-Only (SCVMM) Yes
Cluster-Aware Updating Yes 50/50 (Pool Upgrade is not very smart)
Yes
Hypervisor OverviewXenServer 6.1 - What’s new?
-Thanks to Ryan McClure for donating content
© 2012 Citrix | Confidential – Do Not Distribute
XenServer 6.1 – Key features
• Link Aggregation Control Protocol (LACP) support
• Storage XenMotion
• Integrated StorageLink (iSL) support for EMC VNX series arrays
© 2012 Citrix | Confidential – Do Not Distribute
XenServer 6.1 – Virtual Appliances
• Demo Linux Virtual Appliance
• Workload Balancing 6.1.0 Virtual Appliance
• vSwitch Controller Virtual Appliance
• Web Self Service 1.1.2 Virtual Appliance
• Citrix License Server VPX v11.10
• Citrix XenServer Conversion Managerᵒ Allows up to 500 VM conversions from VMware vSphere to XenServer
© 2012 Citrix | Confidential – Do Not Distribute
What’s New? – Management and Support
• Storage XenMotion
• Live VDI Migration
• Emergency Network Reset
• XenServer Conversion Manager
• Enhanced Guest OS Support
• Supported VM Count Increased to 150
© 2012 Citrix | Confidential – Do Not Distribute
• LACP Support
• VLAN Scalability Improvements
• IPv6 Guest Support
• Updated OvS
What’s New? – Networking
© 2012 Citrix | Confidential – Do Not Distribute
Storage XenMotion and VDI Migration
Storage XenMotion Live VDI Migration
© 2012 Citrix | Confidential – Do Not Distribute
XenServer 6.1 – Feature(s) that didn’t get lime light
• Performance Monitoring Enhancements Supplemental Packᵒ Extended XenServer monitoring available (stored in RRD – Round Robin Databases)
Metric AvailabilityAverage physical CPU usage over all CPUs (%) NewTime physical CPU spent in [C,P]-state (%) NewHost memory reclaimed by dynamic memory control (B) New
Maximum host memory that could be reclaimed by dynamic memory control (B) New
Host aggregate, physical network interface traffic received/sent (B/s) New
I/O throughput read/write/total per PBD (MiB/s) NewAverage I/O latency (milliseconds) NewI/O average queue size NewNumber of I/O requests currently in flight NewIOPS, read/write/total (requests/s) NewTime spent waiting for I/O (%) New
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Roadmap
• Sarasotaᵒ Dom0 Disaggregationᵒ Increased Scalabilityᵒ Fewer Reboots due to Hotfixesᵒ vGPU
• Jupiterᵒ Storage QoS
• Tallahassee – Private Releaseᵒ NetScaler SDX Updateᵒ XenDesktop Interoperability and
Usability Enhancementsᵒ Hardware Monitoring and Reporting
• Clearwaterᵒ Windows 8 Guest Supportᵒ UI Enhancements
• Naplesᵒ User Experience and Third Party
Enhancementsᵒ VHDX Disk Format Support
© 2012 Citrix | Confidential – Do Not Distribute
Additional Resources
• CTX118447 XenServer Administrator’s Guide• CTX118448 XenServer Installation Guide• CTX118449 XenServer Virtual Machine Installation Guide• CTX118451 XenServer Release Notes• CTX119472 XenConvert Guide • CTX118450 XenServer Software Development Guide
Advanced XenServer TrainingXenServer Under The Hood
© 2012 Citrix | Confidential – Do Not Distribute
Agenda
• Core Componentsᵒ Hardwareᵒ Hypervisorᵒ Dom0ᵒ DomU
• Beyond Single Serverᵒ Resource Poolsᵒ HAᵒ Meta Dataᵒ XenMotion/Live Migration
• Workload Balancing
XenServer OverviewCore Components
© 2012 Citrix | Confidential – Do Not Distribute
Terminology to Remember
• Host
• Resource Pool
• Pool Master
• Dom0
• XAPI (XENAPI)
• XenServer Tools
• Live VDI Migration
• Storage XenMotion
© 2012 Citrix | Confidential – Do Not Distribute
Behind The Scenes
• Native 64-bit bare metal hypervisor – Is this true?
• Cent OS
• Based on Linux 2.6.32 kernel
• Drivers are SUSE Linux Enterprise Server v11 SP1
• Xen hypervisor and Domain 0 manage physical server resources among virtual machines
© 2012 Citrix | Confidential – Do Not Distribute
Legacy Virtualization Architecture
UserApps
UserApps
Hardware
HALT SAFE HALT
Legacy Hypervisor
“Binary Translation”Slow, expensive software
emulation layer
© 2012 Citrix | Confidential – Do Not Distribute
UserApps
UserApps
Hardware
HALTHYPERCALL
HALT
XenServer
VT/AMD-V
Paravirtualized guests make high-speed calls
directly to the hypervisor
Paravirtualization
• Relies on “enlightened” operating systems
• Kernel and I/O paths know they are being virtualized
• Cooperation provides best performance
© 2012 Citrix | Confidential – Do Not Distribute
Hardware-Assisted Virtualization
UserApps
UserApps
Hardware
HALTHYPERCALL
HALT
XenServer
VT/AMD-V
Other guests benefit from hardware-accelerated call
translation
• Hardware-assist allows high performance without emulation
© 2012 Citrix | Confidential – Do Not Distribute
Understanding Architectural Components
The Xen hypervisor and Domain 0 manage physical server resources among virtual machines
© 2012 Citrix | Confidential – Do Not Distribute
Understanding the Hardware Component
Hardware layer contains the physical server components, including memory, CPU and storage
© 2012 Citrix | Confidential – Do Not Distribute
Understanding the Hypervisor Component
Xen hypervisor is a thin layer of software that runs right on top of the hardware
© 2012 Citrix | Confidential – Do Not Distribute
Understanding the Domain 0 Component
Domain 0, or the Control Domain, is a Linux VM that manages the network and storage I/O of all guest VMs
© 2012 Citrix | Confidential – Do Not Distribute
Dom0 Resourcing
• Linux-based guests• virtual network interfaces• virtual disks• virtual CPUs
• Windows-based guests• virtual network interfaces• virtual disks
• Resource Control• Processor, network, disk I/O priority• Machine Power-States
© 2012 Citrix | Confidential – Do Not Distribute
Why This Model Kinda Sucks
• Control Requests - XS Console / Direct CLIᵒ Queuingᵒ Confusion
• Dom0 Overburdening
• Distributed Domino Effect
• Limited space due to pre-allocation
• Log Bug
• Not built to Scale out of the Box ᵒ Dom0 Memoryᵒ Dom0 CPU
© 2012 Citrix | Confidential – Do Not Distribute
Why This Model Kinda Sucks
• Control Requests - XS Console / Direct CLIᵒ Queuingᵒ Confusion
• Dom0 Overburdening
• Distributed Domino Effect
• Limited space due to pre-allocation
• Log Bug
• Not built to Scale out of the Box ᵒ Dom0 Memoryᵒ Dom0 CPU
© 2012 Citrix | Confidential – Do Not Distribute
DOM0 Tuning
• Supports 4 vCPUs (Default 1 Prior to 6.0)ᵒ More is betterᵒ IRQBALANCE (http://irqbalance.org/)
• Memory Tweaking (Default 752)ᵒ 2940M Tested for 130/16.25 VMsᵒ Warning: Single Server!
• XenHeap (5.5 Only)ᵒ Set xenheap_megabytes to (12 + max-vms/10)ᵒ Xc.Error("creating domain failed: hypercall 36 fail: 12: Cannot allocate memory (ret -
1)")
© 2012 Citrix | Confidential – Do Not Distribute
Understanding the DomU Linux VM Component
Linux VMs include paravirtualized kernels and drivers
© 2012 Citrix | Confidential – Do Not Distribute
Understanding the DomU Windows VM Component
Windows VMs use paravirtualized drivers to access storage and network resources through Domain 0
© 2012 Citrix | Confidential – Do Not Distribute
Real-time Resource Adjustment
• Linux-based guestsᵒ Add/remove virtual network interfacesᵒ Add virtual disksᵒ Add/remove virtual CPUs
• Windows-based guestsᵒ Add/remove virtual network interfacesᵒ Add/remove virtual disks
• Resource QoS Controlᵒ Processor, network, disk I/O priority
© 2012 Citrix | Confidential – Do Not Distribute
Why This is Cool
• Paravirtualization is key to performanceᵒ Hypervisor and OS cooperation provides best performance
• Utilizes hardware-assisted virtualizationᵒ Fully supports Intel VT and AMD-V enabled processorsᵒ Hardware-assist allows high performance without emulation
• Future: Paravirtualization with Hardware-Assist
• Benchmarks vs running on nativeᵒ Linux can expect .5 - 4% overheadᵒ Windows can expect 2 - 7% overhead
• VMware (EULA prevents publishing)ᵒ 5-10% Overhead seen on internal testing
© 2012 Citrix | Confidential – Do Not Distribute
Lessons Learned: The other DOM - DOMU
• We can only push around 350 Mb/s through a single Windows VM.
• We can 1 to 1.1 Gb/s on a single Linux VM • To push more traffic through a single Windows
VM: ᵒ Manually add additional netback processes
• xen-netback.netback_max_groups=<#Procs> under /boot/extlinux.conf in section labeled xe-serial just after the assignment console=hvc0.
ᵒ Manually pinning vCPUs to specific VMs• xe vm-param-set uuid=<vm-uuid> VCPUs-
params:mask=<CPUs to pin>
XenServer OverviewBeyond Single Server
© 2012 Citrix | Confidential – Do Not Distribute
Management Architecture Comparison
VMWare XenServer
Traditional ManagementArchitecture
• Single backend management server
DistributedManagement Architecture
• Clustered management layer
© 2012 Citrix | Confidential – Do Not Distribute
Resource Pools
• Join multiple physical servers into one logical pool
• Enables shared configurations, including storage
• Required for Automated VM Placement and XenMotion
Xen Hypervisor
Xen Hypervisor
Xen Hypervisor
© 2012 Citrix | Confidential – Do Not Distribute
High Availability
Xen HypervisorXen HypervisorXen Hypervisor
Shared Storage
Prot
ecte
dPr
otec
ted
Prot
ecte
dD
o no
t re
star
t
© 2012 Citrix | Confidential – Do Not Distribute
Xen HypervisorXen HypervisorXen Hypervisor
Shared Storage
XenMotion - Live Migration
© 2012 Citrix | Confidential – Do Not Distribute
XenMotion – Live VM Movement
• XenMotion allows minimal downtime movement of VMs between physical systems
• Generally 150-200ms of actual “downtime”
• Most of the downtime is related to network switch moving IP traffic to new port
© 2012 Citrix | Confidential – Do Not Distribute
XenMotion CPU Requirements
• XenMotion requires systems XS have similar CPUs
• Must be the same manufacturer
• Must be the same type
• Can be different speed
• Example Xeon 51xx series chipsᵒ Could have a mix of 5130 and 5120 chips
© 2012 Citrix | Confidential – Do Not Distribute
• Systems verify correct storage and network setup on destination server• VM Resources Reserved on Destination Server
Pre-Copy Migration: Round 1
Source Virtual Machine Destination
© 2012 Citrix | Confidential – Do Not Distribute
Pre-Copy Migration: Round 1
• While source VM is still running XenServer copies over memory image to destination server• XenServer keeps track of any memory changes during this process
© 2012 Citrix | Confidential – Do Not Distribute
Pre-Copy Migration: Round 1
© 2012 Citrix | Confidential – Do Not Distribute
Pre-Copy Migration: Round 1
© 2012 Citrix | Confidential – Do Not Distribute
• After first pass most of the memory image is now copied to the destination server• Any memory changes during initial memory copy are tracked
Pre-Copy Migration: Round 1
© 2012 Citrix | Confidential – Do Not Distribute
• XenServer now does another pass at copying over changed memory
Pre-Copy Migration: Round 2
© 2012 Citrix | Confidential – Do Not Distribute
Pre-Copy Migration: Round 2
© 2012 Citrix | Confidential – Do Not Distribute
• Xen still tracks any changes during the second memory copy• Second copy moves much less data• Also less time for memory changes to occur
Pre-Copy Migration: Round 2
© 2012 Citrix | Confidential – Do Not Distribute
Pre-Copy Migration: Round 2
© 2012 Citrix | Confidential – Do Not Distribute
• Xen will keep doing successive memory copies until minimal differences between source and destination
Pre-Copy Migration
© 2012 Citrix | Confidential – Do Not Distribute
• Source VM is paused and last bit of memory and machine state copied over• Master unlocks storage from source system and locks to destination system• Destination VM is unpaused and attached to storage and network resources• Source VM resources cleared
XenMotion: Final
© 2012 Citrix | Confidential – Do Not Distribute
The Concept of MetaData
• The metadata contains information about the VMs:ᵒ Name, description, Universally Unique Identifier (UUID), VM configuration, and
information about the use of resources on the host or Resource Pool such as Virtual Networks, Storage Repository, ISO Library.
• Most metadata is written when the VM, SR and network are created and is updated if you make changes to the VM configuration.
• If the metadata is corrupted, the resource pool and/or VMs may become unusable.
• Always backup metadata during:ᵒ Resource pool and/or host upgradeᵒ Before patchingᵒ Migration from one storage repository to another
© 2012 Citrix | Confidential – Do Not Distribute
Backing up the MetaData• It is not necessary to perform daily
exports of all the VM metadata.
• To export the VM metadata:1. Select Backup, Restore and
Update from the menu.2. Select Backup Virtual Machine
Metadata.3. If prompted, log on with root
credentials.4. Select the Storage Repository
where the VMs you want to back up are stored.
5. After the metadata backup is done, verify the successful completion on the summary screen.
6. In XenCenter, on the Storage tab of the SR selected in step 3, a new VDI should be created named Pool Metadata Backup.
© 2012 Citrix | Confidential – Do Not Distribute
Restoring MetaData
• Prerequisites: ᵒ Set up/re-attach the vDisk Storage Repository ᵒ Virtual Networks are set up correctly by using the same names
• From the console menu:1. Select Backup, Restore and Update from the menu.2. Select Restore Virtual Machine Metadata.3. If prompted, log on with root credentials.4. Select the Storage Repository to restore from.5. Select the Metadata Backup you want to restore.6. Select if you want to restore only VMs on this SR or all VMs in the pool.7. After the metadata restore is done, verify the summary screen for errors.8. Your VMs are now available in XenCenter and can be started at the new site.
© 2012 Citrix | Confidential – Do Not Distribute
Data Replication for DR
© 2012 Citrix | Confidential – Do Not Distribute
Simplifying Disaster Recovery
Xen Hypervisor
Xen Hypervisor
Xen Hypervisor
Shared Storage
Xen Hypervisor
Xen Hypervisor
Xen Hypervisor
Shared Storage
Production Site DR Site
Automated backup of VM metadata to SR
Replication of SR includes Virtual Disks and VM metadata
Attach replicated SR
Restore of VM metadata will recreate VMs
1
2
3
4
1
2
3
4
XenServer OverviewWorkload Balancing
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing
• Highly granular, continuous guest and host performance profiling
• Policy-driven workload balancing across XenServer pools and hosts
• Historical analysis and trending for planning purposes
• Reports available in XenCenter
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing – Critical Thresholds
• Components included in WLB evaluation:• CPU• Memory• Network Read• Network Write• Disc Read• Disk Write
• Optimization recommendation is being triggered if a threshold is reached
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing – Weights
• Weights are all about the recommended target host
• Simple interface to determine which factors are most relevant for the VM
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing
• Ideal placement recommendations at VM start-up, resume and live relocation
• Ideal VM relocation recommendations for host maintenance
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing – Placement Strategies
• Maximize Performance• Default setting• Spread workload evenly across all physical
hosts in a resource pool• The goal is to minimize CPU, memory, and
network pressure for all hosts
• Maximize Density• Fit as many virtual machines as possible
onto a physical host• The goal is to minimize the number of
physical hosts that must be online
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing Server
• All components can run on one (virtual) machine
• Multiple server deployment recommended for larger, multi-pool environments
• Data store can be hosted on existing DB platform
• Architecture look familiar?
Data Store
Data Collection Manager service
Web Service Host
Analysis Engine service
© 2012 Citrix | Confidential – Do Not Distribute
Workload Balancing - Components
• Workload Balancing Components• Data Collection Manager service• Analysis Engine service• Web Service Host• Data Store• XenServer• XenCenter
Data StoreXenServer
Resource PoolXenServer
Resource Pool
Data Collection Manager service
Web Service Host
XenCenter
Performance Metrics
Perform
ance Metric
s
Recommendations
Analysis Engine service
Hypervisor OverviewLab 1 – Intro Lab
© 2012 Citrix | Confidential – Do Not Distribute
In this lab… 30 minutes
• Create a XenServer Resource Pool
• XenServer Resource Pool Management (switching pool master, Dom0 tweaks)
• Upgrading from XenServer 6.0.2 to 6.1
• Backing up and Restoring Metadata
XenServer StorageDeep Dive
© 2012 Citrix | Confidential – Do Not Distribute
Agenda
• Introduction to XenServer Storage• SRs – Deep Dive• Xen Storage – Command line• Storage XenMotion Under The Hood• Debugging Storage Issues• Key Takeaways and References
© 2012 Citrix | Confidential – Do Not Distribute
Before we get started…
• Lets familiarize ourselves with the following terms:ᵒ HBAᵒ LUNᵒ Fabricᵒ Storage Processorᵒ SRᵒ VDIᵒ PBD and VBDᵒ Multipathingᵒ Fast Clone, Linked Cloneᵒ XenMotion and Storage XenMotion
XenServer StorageIntroduction to Storage
© 2012 Citrix | Confidential – Do Not Distribute
Disk Types in common use
• SATA (Serial ATA) disksᵒ Size up to 3 TB (4TB + due for release)ᵒ Spin speed 7200 RPMᵒ Transfer rate ~ 30-40 MB per secondᵒ IOPS 70-80ᵒ Pros: Inexpensive, large Storage. Cons: Relatively slow, long rebuild time
• SAS (Serial Attached SCSI) Disksᵒ Size up to 600 GB (900+ GB due for release)ᵒ Spin speed 15000 RPMᵒ Transfer rate ~ 80-120 MB per secondᵒ IOPS 160-180ᵒ Pros faster access time, redundancy Cons Higher cost per GB
• SSD (Solid state Disk)ᵒ Size 100GB upwards (enterprise class)ᵒ Extremely fast access speedsᵒ Transfer rate ~ up to full bus speedᵒ IOPS read > 40,000 write > 10,0000ᵒ Pros Extremely fast access times low power Cons Very expensive, Limited life.
© 2012 Citrix | Confidential – Do Not Distribute
Disk Sizing
Disk Size In MB Expected MB Diff Reported Size in windows
100 GB 100,000,000,000 107,374,182,400 7.37% 93.1 GB, 95,367 MB
1 TB 1,000,000,000,000 1,099,511,627,776 9.95% 931 GB, 953,674 MB
Latency TimeThe faster the disk spins, the shorter the wait for the computer. A hard drive spinning at 7,200 rotations per minute has an average latency of 4.2 thousandths of a second. This is approximately one-half the time it takes for the disk to turn one revolution.Latency ConsequencesWhile 4.2 thousandths of a second seems inconsequential, a computer might process many thousands of data requests. The computer's CPU is fast enough to keep up with the workload but the hard disk's rotational latency creates a bottleneck.
© 2012 Citrix | Confidential – Do Not Distribute
RAID (redundant array of independent disks)
• Way of presenting multiple disks so that they appear to the server as a single larger disk.
• Can also provide for redundancy in the event of a single/multiple disk failure(s)
• Has multiple types each with its advantages and disadvantages.
© 2012 Citrix | Confidential – Do Not Distribute
Raid Sets
Pros – cheapestCon – A failure of any disk will fail all of the data on the disk
Pros – a failure of any drive will still provide full access to dataOnly 1 additional write to secure dataCon - Double the amount of disks needed
© 2012 Citrix | Confidential – Do Not Distribute
Pros – Data is striped across multiple disks with parityCan support a single disk failure without data loss
Cons – when a disk fails, performance is reduced because the controller has to calculate the missing disks data.Full performance is not recovered until the failed disk is replaced and rebuilt.Write intensive applications can run slowly due to the overhead of calculating and writing the parity blocks
Pros – Data is striped across multiple disks with double parityCan support two disk failures without data loss
A single disk failure does not affect performance
Cons – Write intensive applications can run slower because of the overhead of calculating and writing two parity blocks
Additional 2 disks per raid set required to handle parity requirements.
© 2012 Citrix | Confidential – Do Not Distribute
Hybrid RAID types• RAID 0+1• RAID 10• RAID 50
© 2012 Citrix | Confidential – Do Not Distribute
IOPS (Input/Output Operations per second)
Measurement Description
Total IOPS Total number of I/O operations per second (when performing a mix of read and write tests)
Random Read IOPS Average number of random read I/O operations per second
Random Write IOPS Average number of random write I/O operations per second
Sequential Read IOPS Average number of sequential read I/O operations per second
Sequential Write IOPS Average number of sequential write I/O operations per second
IOPS are primarily a benchmarking figure that is used to find/compare the performance of storage solutions.
IOPS are affected by many factors – such as RAID type, stripe size, Read/Write ratios,Random/sequential balance, cache sizes on controller and disk.
Quoted maximum IOPS from manufactures seldom translate to real world applications.
© 2012 Citrix | Confidential – Do Not Distribute
What is a SAN?
• SAN (Storage Area Network)
• Dedicated network that provides access to consolidated, block level data storage.
• Can be a single device or Multiple Devices
• Does not provide any file level functionality
• However file systems that make use of storage on a SAN will allow for file locking sharing etc.
• Provide access to a pool of shared storage using Either ISCSI, Fibre Channel, FCOIP (fibre channel over IP) for connectivity.
• Is (ideally) on a separate network from other traffic.
© 2012 Citrix | Confidential – Do Not Distribute
© 2012 Citrix | Confidential – Do Not Distribute
NAS (Network Attached Storage)
A NAS differs from a SAN in that it offers remote Access to a filesystem as if it were a fileserver whereas a SAN offers remote access to Block level storage as if it was a hard disk controller.
NAS normally allows connection to its storage via NFS (Unix/Linux), SMB/CIFS (Windows), AFP (Apple)
Some NAS can be used for other protocols such as FTP, HTTP, HTTPS,NDMP, and various media streaming protocols.
A NAS Head (or Filer) can have single or multiple nodes A NAS essential replaces one or more fileservers
© 2012 Citrix | Confidential – Do Not Distribute
What are SAN/NAS used for
• Both provide access to SHARED storage.
• Both can provide high availability/fault tolerance by having multiple Disk Controllers / NAS Heads
• A SAN is used to provide BLOCK level storage to multiple Hosts.
• Connection to a SAN is Normally through Fibre channel or ISCSI (1GB/10GB) over Ethernet.
• A NAS is used to replace FILE servers by providing CIFS / NFS/FTP etc. features.
• Connection to a NAS is normally via Ethernet.
XenServer StorageSRs, SR types and more…
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Storage Objects
• Describes a particular storage target in which Virtual Disk Images (VDIs) are stored.
• Flexible—supports a wide variety of storage types.
• Centralized—easier to manage, more reliable with a XenServer pool.
• Must be accessible to each XenServer host.
© 2012 Citrix | Confidential – Do Not Distribute
Types of SRs supported in XenServer
• Block based – LVMᵒ Fiber Channel (FC)ᵒ iSCSIᵒ Local LVM
• File basedᵒ Ext (local)ᵒ NFSᵒ CIFS (ISOs only)
© 2012 Citrix | Confidential – Do Not Distribute102
Protocols
• NFS – inexpensive, easiest to implement, 1 or 10 Gb, overhead*
• iSCSI – fairly cheap and easy, HW and SW flavors, 1 or 10 Gb
• FCoE – pretty expensive and difficult, requires CNAs, 10 Gb only
• FC – most expensive and difficult to implement, 1/2/4/8/16** Gb• FC is most common in the enterprise, but 10 Gb+ networking is making
NFS and iSCSI more and more common
• In terms of performance, typically you will find that 8 Gb FC will outperform 10 Gb iSCSI or NFS
• Ask the customer if they have “storage tiers”ᵒ Ex: 1. EMC 8 Gb FC 2. HP 10 Gb iSCSI 3. NetApp 1 Gb NFS
© 2012 Citrix | Confidential – Do Not Distribute
Comparisons to other Hypervisors
Facet XenServer VMware vSphere Microsoft Hyper-V
VM Storage LVM, File-level File-level CSV, SMB 3.0 (CIFS)
Storage plugins Storagelink (integrated)
PSP, NMP, Vendor supported
N/A
Multipathing Yes Yes Yes
NFS Support Yes Yes No
© 2012 Citrix | Confidential – Do Not Distribute
xapi
kernel
dom0
Xen Hypervisor
Linuxstoragedevices
LinuxNetworkdevices
Storage Network Domainsdb
SM i/fscripts:
xenstore
Blktap/blkbackNetback
qemu-dm
vm consoles
XML/RPC; HTTP
XenAPIstreamingservices
Hardware devices
VM1
Shared memory pages
VM1
Shared memory pages
XenServer – under the hood
© 2012 Citrix | Confidential – Do Not Distribute
XenServer - under the hood
• I/O (Network and Storage) depends significantly on Dom0ᵒ This changes in the future releases of Xenserver (disaggregation of Dom0)
• All VMs use para-virtualized drivers (installed with Xen tools) to communicateᵒ Blkfront, netfront components in VMs send I/O traffic to Dom0’s Blkback, netbackᵒ Hence the SLOW performance and scalability limits with XenServer
© 2012 Citrix | Confidential – Do Not Distribute
The competition
• ESXi - file based architecture
• VMFS - clustered filesystem
• VMFS 5 - can be shared between 32 hosts
• Custom plugins can be installed based on shared filesystem
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Storage Objects
• XenServer Storage Objectsᵒ SRs, VDIs, PBDs and VBDs
• Virtual Disk Data Formatsᵒ File-based VHD, LVM and StorageLink
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Storage ObjectsVDIs, PBDs, VBDs
• Virtual Disk Images are a storage abstraction that is presented to a VM.
• Physical Block Devices represent the interface between a physical server and an attached SR.
• Virtual Block Devices are connector objects that allow mappings between VDIs and VMs.
© 2012 Citrix | Confidential – Do Not Distribute
PBD
PBD
PBD
SR
VDI
VDI
VDI VBD
VBD
VBDXenServer Host
XenServer Host
XenServer Host
Virtual Machine
Virtual Machine
XenServer Storage Objects
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Disk Data FormatsLogical Volume (LVM)-based VHDs
• The default XenServer block device-based storage inserts a Logical Volume manager on a disk. VDIs are represented as volumes within the Volume manager.
• Introduced LVHD in XenServer 5.5ᵒ Enhances LVM for SRsᵒ Hosts VHD files directly on LVM volumesᵒ Adds Advanced Storage features like Fast Cloning and Snapshotsᵒ Fast and simple upgradeᵒ Backwards compatible
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Disk Data FormatsFile-based VHD
• VM images are stored as thin-provisioned VHD format files on either a local non-shared file system (EXT type SR) or a shared NFS target (NFS type SR).
• What is VHD?ᵒ A Virtual Hard Disk (VHD) is a file formatted to be structurally identical to a physical
Hard Disk Drive.ᵒ Image Format Specification was created by Microsoft in June, 2005.
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Disk Data FormatsStorageLink (LUN per VDI)
• LUNs are directly mapped to VMs as VDIs by SR types that provide an array-specific plug-in (NetApp, Equallogic or StorageLink type SRs). The array storage abstraction therefore matches the VDI storage abstraction for environments that manage storage provisioning at an array level.
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Disk Data FormatsStorageLink Architecture
ᵒ XenServer calls direct to Array API‘s to provision and adjust storage on demand.
ᵒ Fully leverages array hardware capabilities.ᵒ Virtual disk drives are individual LUNs.ᵒ High performance storage model.ᵒ Only the server running a VM connects to the
individual LUN(s) for that VM.
© 2012 Citrix | Confidential – Do Not Distribute
LVM vs. StorageLink
Storage Repository
LUN
LVM Volume Group
LVMLogicalVolume
VHD header
VHD header
XenServer 6.1 iSCSI / FC
LUN
Storage Repository
XenServer 6.1 iSCSI / FC + integrated
StorageLink
LUN
VM Virtual Disk
LVMLogicalVolume
XenServer StorageCommand line vs. XenCenter
© 2012 Citrix | Confidential – Do Not Distribute
Xen Storage – command line vs. XenCenter
• Can you trust XenCenter info?ᵒ The answer is – NOᵒ XenCenter relies on the database integrity of XenServer resource poolᵒ Each entity of XenServer (Resource pool data, CPU, NIC and Storage) are stored in
XenServer’s built-in databaseᵒ From a Storage perspective:
• PBD• SR• VDI• VBD
ᵒ All the above entities are stored in each XenServer’s databaseᵒ And now…why should I remember this?
• If a VM fails to boot – the above information will come in handy during troubleshooting• If SR disappears from XenServer (oh yes it happens!), you will need these commands to
recreate the storage path
© 2012 Citrix | Confidential – Do Not Distribute
Xen Storage – Command line
• All Xen commands start with xe <Object>-<Action> <parameters>ᵒ For example:# xe vbd-list vm-name-label=WIN7The above command will list all the disks attached to the VM1. Object can be SR, VBD, VDI, PBD2. Action(s) can be create, delete, list, probe, scan etc.3. Parameters are specific to the command
• Entire list of Xen Commands can be found here
XenServer StorageStorage XenMotion
© 2012 Citrix | Confidential – Do Not Distribute
Storage XenMotion – under the hood
• New with XenServer 6.1
• Supported to migrate VMs across multiple resource pools
• Also supports Storage migration between local storage and shared storage
• Please keep in mind – its version 1.0
© 2012 Citrix | Confidential – Do Not Distribute
Storage XenMotion and VDI Migration
Storage XenMotion Live VDI Migration
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: Use cases
1. Upgrade a storage array
2. Upgrade a pool with VMs on local storage
3. Provide tiered storage arrays
4. Rebalance VMs between XenServer pools, or CloudStack clusters
“The Cloud” was the major use case we had in mind when designing this.
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: Bird’s-eye view
Cross-pool migration and VDI migration consist of the following:
1. Synchronously mirror VDIs between source and destination
2. Create new VM object on destination pool (new ref, same uuid)
3. When copy complete, migrate VM as usual
Note: VDI migrate implemented with “localhost” cross-pool migrate!
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: Limitations
• No more than 6 VDIs, and no more than 1 VM snapshot
• No more than 3 concurrent ops per host
• No VMs with PCI pass-through enabled
• The following are untested and unsupported:ᵒ HA and WLB on source or destination poolᵒ DVSC integrationᵒ VDI-migrate between multiple local SRs
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: Caveats
• Minimum network or storage throughput requirements? This is currently unknown, but we’ll begin investigations soon.
• Can’t check whether destination SR has space available for incoming VDIs – if you fill up an SR, your migration will fail to complete.
• Extra temporary storage is required on the source (and destination) SR, so you must be careful when migrating VMs off of a full SR.
• IO performance inside guest will be reduced during migration because of synchronous mirroring of VDIs.
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: API walkthrough
Host.migrate_receivehost:refnetwork:refoptions:map
Result = receive_token
VM.migrate_sendvm:refreceive_tokenlive:boolvdi_sr:mapvif_network:mapoptions:map
Result = None
VM.assert_can_migratevm:refdest:maplive:bool
vdi_sr:mapvif_network:mapoptions:map
Result = None
VDI.pool_migratevdi:refsr:refoptions:map
Result = vdi:ref
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: CLI walkthrough
• xe vm-migrateᵒ New params: remote-address, remote-username, remote-password, remote-network, vif, vdi
• Extends the original vm-migrate command• Bold params are required to enable cross-pool migration• vif and vdi map VIFs to target networks and VDIs to target SRs• remote-network specifies the network used for data transfer• Can use host/host-uuid to specify host on pool to send VM
• xe vdi-pool-migrateᵒ Params: uuid, sr-uuid
• uuid of target VDI• sr-uuid of destination SR
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: GUI walkthrough
© 2012 Citrix | Confidential – Do Not Distribute
Feature Overview: GUI walkthrough
© 2012 Citrix | Confidential – Do Not Distribute
Architecture: VDI operations
• For each VDI:ᵒ Snapshot VDI and synchronously mirror all subsequent writes to destination SRᵒ Copy the snapshot to destination SRᵒ Finally, compose those writes onto the snapshot on the destination SR
• Continue to mirror all new writes
• Each of these operations occurs sequentially for each VDI, but each VDI mirror continues until the VM migration is complete
• VM memory is copied only after final VDI compose is complete
VDI 1: snapshot & start mirroring
VDI 1: copy snapshots
VDI 2: snapshot & start mirroringVDI 2: copy snapshots
Copy VM memory
© 2012 Citrix | Confidential – Do Not Distribute
VDI mirroring in picturesSOURCE DESTINATION
mirror
copy
root
VM VM
no color = emptygradient = live
© 2012 Citrix | Confidential – Do Not Distribute
VMware Storage VMotion In Action
132
Source Destination
“Fast suspend/resume” VM to start running on new home and disks
4
Copy all remaining disk blocks
4
Copy VM home to new location
1
Delete original VM home and disks
5
Start changed block tracking
2
Pre-copy disk to destination(multiple iterations)
3
XenServer StorageDebugging Storage Issues
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting tips
• Check logs: /var/log/{xensource.log,SMlog,messages}ᵒ Note: xensource.log and messages both implemented with syslog, so they now have
consistent timestamps!
• xn command: CLI to xenopsdᵒ Try ‘xn help’ for documentation.
• tap-ctl commandᵒ Could be useful for diagnosing problems (can’t describe usage here)
© 2012 Citrix | Confidential – Do Not Distribute
Logging/Debugging
• All backend drivers use the Smlog to record storage eventsᵒ /var/log/Smlog
• Logs are rotated, same as system message log, xensource.log files
• In the python module util.py there is a helper log function -> util.SMlog(STRING)
• On retail edition, all python backend files are editable for logging/debugging purposes
• Some things to look for in SMLog (tail –f /var/log/SMLog):ᵒ Slow SR Probing: Check for breaks in the SMLog timestamps or for
SR_BACKEND_FAILURE_46, this is likely an issue with multipathingᵒ Storage Issue: Look for the word FAILURE within the log, this is likely an issue
communicating with the storage or a corrupted SRᵒ Metadata Corruption: Look for SR_BACKEND_FAILURE_18, this is likely an issue
with metadata being corrupted due to special characters being used in VM namesᵒ VDI Unavailable: This is typically caused by faulty clones/snapshots but could also
point to misbehaving Storage.
© 2012 Citrix | Confidential – Do Not Distribute
Debugging tips
• Use tail –f on any log file while actively debugging a system
• Correlate logs between xensource.log, SMLog and messages
• Verify IP settings, firewall config for any IP based storage connections
• Check status of dependent objects, such as PIFs, VIFs, PBDs etc...
XenServer StorageLab 2 – Storage Lab
© 2012 Citrix | Confidential – Do Not Distribute
In this lab… 90 minutes
• Create a local SR
• Create a shared SR with Integrated Storagelink
• Storage XenMotion Lab
• (Advanced Lab) Create shared SR using NetApp SIM
XenServer NetworkingFoundational Concepts
© 2012 Citrix | Confidential – Do Not Distribute
Agenda
• Networking Overview• XenServer Networking• XenServer 6.1 – What’s New?
ᵒ Link Aggregation (LACP)
• Debugging Networking Issues• Key Takeaways and References
© 2012 Citrix | Confidential – Do Not Distribute
Before we get started…
• Lets familiarize ourselves with the following terms:Common Enterprise Networking terminologies:ᵒ NICᵒ Switch – Coreᵒ Router, Hubᵒ Firewallᵒ DHCP, PXE, TFTP, TSBXenServer:ᵒ LACPᵒ VLANᵒ VIFᵒ PIF
© 2012 Citrix | Confidential – Do Not Distribute
Internet
Server Farm
Backbone Switch
Access Switch
Example Enterprise Switching Environment
Distribution Switch
Firewall
© 2012 Citrix | Confidential – Do Not Distribute
Example WAN Environment
© 2012 Citrix | Confidential – Do Not Distribute
Network Vendors
Vendor Components
Cisco Switches, ACE (load balancing), ASA (Firewall), Routers
Juniper Network Security Solutions
Brocade Switches, Routers
Citrix NetScalerF5River Bed
Access Gateway, WAN Accelerators,Load Balancing
XenServer Advanced TrainingIntroduction to Enterprise Networking
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Networking Conceptual Model
Xen Hypervisor
Control Domain(Dom 0)
Hardware
Virtual Machine
Network Card
Linux Driver
PIFvNIC
BridgeVIF
Netback Netfront
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Networking Configurations
Linux NIC Drivers
Linux Config Files
XenServer PoolDB
Network Card
XAPI
Command Line
XenCenter
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Switches
PIF (eth0)
VIF
VIF
VIF
Virtual Machine
Virtual Machine
XenServer Network Terminology
Network 0 (xenbr0)
Private(xapi1)
Network Card
© 2012 Citrix | Confidential – Do Not Distribute
Virtual Switches
PIF (eth1)
PIF (eth0)
VIF
VIF
VIF
Virtual Machine
Virtual Machine
XenServer Network Terminology
Network 1 (xenbr1)
Network 0 (xenbr0)
Network Card
Network Card
© 2012 Citrix | Confidential – Do Not Distribute
PIF (bond0)
PIF
VIF
VIF
Virtual Machine
Virtual Machine
XenServer Network Terminology
Network Card
Network Card
VIF
Bond 0+1 (xapi2)
PIF (eth0)
PIF (eth1)
© 2012 Citrix | Confidential – Do Not Distribute
VLAN 55
Virtual Switches
PIF
PIF
VIF
VIF
VIF
Virtual Machine
Virtual Machine
XenServer Network Terminology
Network 1
Network 0
VLAN 25
Network Card
Network Card
© 2012 Citrix | Confidential – Do Not Distribute
PIFs
VIF
VIF
VIF
Virtual Machine
Virtual Machine
XenServer Network Terminology
VLAN 25
Network Card
Network Card
VLAN 55
Bond0+1
© 2012 Citrix | Confidential – Do Not Distribute
Bonding Type (Balance SLB)
Virtual Machine
Network Card
Network Card
Virtual Machine
Bond
0:00 SEC0:10 SEC0:20 SEC0:30 SEC
Stacked Switches
Virtual Machine
© 2012 Citrix | Confidential – Do Not Distribute
What about faster iSCSI/NFS?
Virtual Machine
Network Card
Network Card
Virtual Machine
Dom0 with iSCSI or NFS software
Bond
0:00 SEC0:10 SEC0:20 SEC0:30 SEC
iSCSI/NFS
iSCSI/NFS
Advanced XenServer TrainingDistributed vSwitch
© 2012 Citrix | Confidential – Do Not Distribute
Open Virtual Switch for XenServer
VM
Hypervisor
VM
VM
VM
VM
Hypervisor
VM
VM
VM
VM
Hypervisor
Visibility· Resource control · Isolation · Security
VM
VM
• Open Source Virtual Switch maintained at www.openvswitch.org• Rich layer 2 feature set (in contrast to others on the market)• Ships with XenServer 5.6 FP1 as a post-install configuration option
© 2012 Citrix | Confidential – Do Not Distribute
Hypervisor
Distributed Virtual Switch Controller
HypervisorHypervisorHypervisor
DVS
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
DVS Controller is a XenServer Virtual Appliance that controls multiple Open vSwitches
© 2012 Citrix | Confidential – Do Not Distribute
Distributed Virtual Switch
HypervisorHypervisorHypervisor
Built-in policy-based ACLs move with VMs
DVS
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123
Virtual Interface (VIF) {MAC, IP} ACLspermit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 192.168.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit udp 172.16.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq domain permit tcp 10.0.0.0 0.0.0.255 10.20.0.0 0.0.0.255 eq 123
XenServer NetworkingLACP Support – Finally!!
© 2012 Citrix | Confidential – Do Not Distribute
Overview
• Existing bonding modes and related issues:• active-active (balance-slb)
• ARP table confusion for some switches (due to MAC flapping)• active-passive (active-backup)
• only one link used at a time• LACP (802.3ad)
• not supported• working only on Linux Bridge
• The modern way – LACP bonding mode:• Link aggregation on both server and switch side.• Sides communicate with LACP protocol.• Better load balancing.
© 2012 Citrix | Confidential – Do Not Distribute
NIC 1
VM 1
Virtual interface (VIF)
LACP Bond
NIC 2
App 1
App 2
VM traffic
Virtual interface (VIF)
VM 2
App 1
App 2
StackedSwitches
© 2012 Citrix | Confidential – Do Not Distribute
Architecture
• For given NICs, we bond the PIFs and set mode to LACP.
• Bond object is created on xapi level.
• Command bond-create executes vSwitch commands.
• LACP is configured on the switch ports.
• Switch and server exchange LACP frames.
• Switch chooses the active members.
• Switch and server balance their outgoing traffic independently.
© 2012 Citrix | Confidential – Do Not Distribute
Load balancing
• Hash-based traffic balancing:ᵒ vSwitch computes flow number (0-255) for each packetᵒ Each flow (hash) is assigned to an active linkᵒ Flows can be moved between links (re-balancing every 10s)
• Two hashing algorithms:ᵒ tcpudp_ports: based on IP and port of source and destination
• default algorithm for LACP• source MAC also taken into account• VM traffic from different applications can use more than one link
ᵒ src_mac: based on source MAC• vSwitch uses the same mechanism as for active-active• traffic from one VIF will not be split
© 2012 Citrix | Confidential – Do Not Distribute
Configuration
• Two new bond modes in XenCenter wizard:ᵒ LACP with load balancing based on IP and port of source and destinationᵒ LACP with load balancing based on source MAC address
• CLI commands:xe bond-create mode=lacp network-uuid=… pif-uuids=… \ properties:hashing-algorithm=src_mac
xe bond-param-set uuid=… \ properties:hashing_algorithm=tcpudp_ports
Create LACP bond
specify hashing algorithm
© 2012 Citrix | Confidential – Do Not Distribute
Switch configuration
• Switch must support 802.3ad (LACP)
• Choose unused Link Aggregation Group (LAG)
• For each NIC being bonded:ᵒ Find the switch port/unit connected to the NICᵒ Set LAG membership for the portᵒ Enable LACP for the port
• If necessary, bring up the LAG interface
• If required, configure the VLAN settings on the LAG
© 2012 Citrix | Confidential – Do Not Distribute
Side note: Static LAG
• Static LAG:ᵒ Ports are members of a Link Aggregation Group.ᵒ Ports do not have LACP set.ᵒ All links within the LAG are active.
• Static LAG is not LACP.ᵒ Use balance-slb, not LACP bond mode for static LAG.
© 2012 Citrix | Confidential – Do Not Distribute
Limitations and changes
• LACP for Linux Bridge not supported.If used anyway, note that it was renamed from 802.3ad to lacp.
• EtherChannel/Cisco Port Aggregation Protocol (PAgP) not supported.
• Switches must be stacked.
• Up to four NICs per bond for all bond modes.
• Only 2 NICs per bond supported for Linux Bridge.
• Improvements for bonding planned in near future:ᵒ Switching off dynamic load balancing (preventing MAC flapping)ᵒ Choice of active link for active-passive bonds
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting
• Frequent surprises:ᵒ Wiring misinformation – wrong ports might be aggregatedᵒ Switch rules – and it can decide to use just one link as activeᵒ No HCL – 802.3ad is a standardᵒ Longer set-up – more time might be required for LACP and DHCP ᵒ One ‘fat’ flow still will not be splitᵒ In ovs commands, ‘bond mode’ means ‘hashing algorithm’
• Mismatching settingsᵒ LACP only on server side – we should have active-active modeᵒ LACP only on switch side – you are on the mercy of the switch
© 2012 Citrix | Confidential – Do Not Distribute
Debugging
• CLI command: xe bond-param-list
• vSwitch commands: ᵒ ovs-vsctl list port ᵒ ovs-appctl bond/show bond0 (lacp_negotiated: false/true)ᵒ ovs-appctl lacp/show bond0 (actor and partner information, hashes)
• xensource.log – network daemon entries, including DHCP
• /var/log/messages – vSwitch entries, including hash shifting
XenServer NetworkingSR IOV and more
© 2012 Citrix | Confidential – Do Not Distribute
SR-IOV
• SR-IOV – VM to talk directly to a NIC rather than having to pass network traffic through Dom0
• SR-IOV virtualizes the network in hardware, rather than in software
• benefits and disadvantages:ᵒ good: there has generally a significant speed improvement in per-VM bandwidth (and
usually in latency) when using SR-IOV instead of Dom0-mediated networkingᵒ good: partitioningᵒ good: avoiding the switchᵒ bad: avoiding the switchᵒ bad: migration issues
© 2012 Citrix | Confidential – Do Not Distribute
SR-IOV Pre-requisites
• SR-IOV capable network device (i.e. Intel 82599 10 Gb-E Controller)
• The SR-IOV NIC cannot be used as the management. ᵒ A second physical NIC must be installed on the system
• iommu must be enabled on the XS hostᵒ /opt/xensource/libexec/xen-cmdline --set-xen iommu=1
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Networking – without SR-IOV
Xen Hypervisor
Control Domain(Dom 0)
Hardware
Virtual Machine
Network Card
Linux Driver
PIFvNIC
BridgeVIF
Netback Netfront
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Networking – with SR-IOV
Xen Hypervisor
Control Domain(Dom 0)
Hardware
Virtual Machine
Network Card
Linux Driver
PIFvNIC
BridgeVIF
Netback VF Driver
SR-IOV Aware Network Card
© 2012 Citrix | Confidential – Do Not Distribute175
Using Multi-Queue NICs
Driver Domain
Guest Domain
Backend Driver
Frontend Driver
Xen
Physical Driver
HardwareMQ NIC
I/O Channels
Incoming Pkt
IRQ
event6 8
Post grant on I/O channel
1
Map bufferpost buf on dev
queue
DMA5
UnMap buffer7
9Push into the network stack
gr3
2
• Advantage of multi-queue• Avoid data copy• Avoid software
bridgeOne RX queue
per guest
guestMAC addr
demux4
© 2012 Citrix | Confidential – Do Not Distribute
VLAN Overview
• A VLAN allows a network administrator to create groups of logically networked devices that act as if they are on their own independent network, even if they share a common infrastructure with other VLANs.
• Using VLANs, you can logically segment switched networks based on functions, departments, or project teams.
• You can also use a VLAN to geographically structure your network to support the growing reliance of companies on home-based workers.
• These VLANs allow the network administrator to implement access and security policies to particular groups of users.
© 2012 Citrix | Confidential – Do Not Distribute
VLAN Overview
© 2012 Citrix | Confidential – Do Not Distribute
VLAN in details
• A VLAN is a logically separate IP subnetwork.
• VLANs allow multiple IP networks and subnets to exist on the same switched network.
• For computers to communicate on the same VLAN, each must have an IP address and a subnet mask that is consistent for that VLAN.
• The switch has to be configured with the VLAN and each port in the VLAN must be assigned to the VLAN.
© 2012 Citrix | Confidential – Do Not Distribute
VLAN in details
• A switch port with a singular VLAN configured on it is called an access port.
• Remember, just because two computers are physically connected to the same switch does not mean that they can communicate.
• Devices on two separate networks and subnets must communicate via
a router (Layer 3), whether or not VLANs are used.
© 2012 Citrix | Confidential – Do Not Distribute
Dom0 – Limitations with VLAN
• Dom0 doesn’t support VLANsᵒ Management network needs to be on its on NICᵒ XenMotion cannot be separate from management NICᵒ iSCSI, NFS can be on VLANsᵒ Dom0 supports NIC bonding
XenServer NetworkingLab 3 – Networking Lab
© 2012 Citrix | Confidential – Do Not Distribute
In this lab… 30 minutes
• Create a LACP bond
• Create a distributed virtual switch
• Explore VIFs, PIFs and eth# interfaces
© 2012 Citrix | Confidential – Do Not Distribute
Resource Links
• XS Administrator’s Guide – revised section on bonding
• The standard:
http://standards.ieee.org/getieee802/download/802.1AX-2008.pdf
• Internal wiki page: http://confluence.uk.xensource.com/display/engp/Bonding+and+LACP
• Feature lead: Kasia Borońska ([email protected])
XenServer HealthMonitoring & Troubleshooting
© 2012 Citrix | Confidential – Do Not Distribute
Agenda
• Managing & Monitoring this wonderful piece of software• It’s broken, How do I fix it?• Show me Your Commands!
XenServer Monitoring & TroubleshootingManaging & Monitoring this wonderful piece of Software
© 2012 Citrix | Confidential – Do Not Distribute
Using Console Folder View
• Manage XenServer environment in your own way
• Selection styles:• SRs (Remote and Local)• Snapshots• Networks• VMs• Pools• Servers
• Custom Folders • Simple drag & drop functionality
© 2012 Citrix | Confidential – Do Not Distribute
Delegated Management (Web Self Service)
• Delegate VM level access to end-users
• View consolidated virtual machine guests from multiple resource pools
• Basic life cycle operations such as Start, Stop, Suspend and Reset on virtual machine
• Remote login (VNC for Linux Guests and RDP for Windows Guests) to the virtual machine guests
• Force Shutdown & Force Reboot (Only with XS Tools)
• Fully AD Integrated
© 2012 Citrix | Confidential – Do Not Distribute
Web Self Service Roles
XenServer Roles WSS Roles
Pool Admin WSS Admin
Pool Operator WSS Operator
VM AdminVM OperatorVM Power AdminRead Only
WSS User
No Role (Default) WSS User
© 2012 Citrix | Confidential – Do Not Distribute
Web Self Service OdditiesSharing Through Tags? What?
© 2012 Citrix | Confidential – Do Not Distribute
Web Self Service Oddities
• Authentication Can’t be changed – Better Make up your mind
• Integrated Auth is pretty annoying
• Not Really AD, XS AD!
• Process is still manual for AD?ᵒ Must Add every User unless auto-login is enabled
© 2012 Citrix | Confidential – Do Not Distribute
Legacy Performance Monitoring
• In Citrix XenServer 5.6+ you are no longer able to collect Host and VM performance data via the “legacy” API calls. Instead they direct you to use the web services via URL
© 2012 Citrix | Confidential – Do Not Distribute
RRDs (Round Robin Databases)
• RRDs are the OpenSource industry standard, high performance data logging and graphing system for time series data.
• XenServer’s Performance monitoring file is actually an RRDTool File, go figure.
• Download format is: http://<username>:<password>@<host>/rrd_updates?start=<secondssinceJan1,1970>ᵒ Use &host=true suffix to get RRD for a XS Host
• XML downloaded contains every VM's data. It is not possible to query a particular VM on its own or a particular parameter. ᵒ To differentiate, the 'legend' field is prefixed with the VM's uuid.
• Free RRDTool available at: http://oss.oetiker.ch/rrdtool/
© 2012 Citrix | Confidential – Do Not Distribute
TAAS (Previously XenoScope)
• XenServer Metrics:ᵒ Uses RRD Metricsᵒ Can be drilled down by domain, time frame, graph size. ᵒ Host memory usage Piechartᵒ Host Hardwareᵒ Pool hosts Overviewᵒ Pool Hardware Comparisonᵒ Installed Hotfixes Comparisonᵒ Running Virtual Machines Overviewᵒ CPU Flags Overview and Comparisonᵒ Recommendationsᵒ Server Log review tools
© 2012 Citrix | Confidential – Do Not Distribute
TAAS (Previously XenoScope)A Very Clean Interface (Also for XA/XD/NS)
© 2012 Citrix | Confidential – Do Not Distribute
TAAS (Previously XenoScope)A Very Clean Interface (Also for XA/XD/NS)
© 2012 Citrix | Confidential – Do Not Distribute
DoubleRev’s Management Pack (http://www.doublerev.com/ )
• Pros:ᵒ Automatic and agent less mass update of XenToolsᵒ DVD Manager ᵒ Storage Analyzer ᵒ Mass VM-Copyᵒ Re-convert templates to virtual machinesᵒ VM-Inventory Exportᵒ Seamless XS Console Integration
© 2012 Citrix | Confidential – Do Not Distribute
DoubleRev’s Management Pack (http://www.doublerev.com/ )
• Cons:ᵒ CLI Commandsᵒ Limited Monitoringᵒ Most of these features probably will come in Future XS Consoleᵒ Costs $$$
• Verdict:ᵒ Should only be considered if you are extremely lazy as most of these features are
nothing terribly difficult to accomplish
XenServer Monitoring & TroubleshootingIt’s Broken, How do I fix it?
© 2012 Citrix | Confidential – Do Not Distribute
Methodology For Recovery Overview
© 2012 Citrix | Confidential – Do Not Distribute
Step 1 - Is the Pool Master down?
• Then the Master server cannot be contacted and is likely non-functional.ᵒ If connection to the XenServer pool is not possible from XenCenterᵒ If issuing commands (xe host-list) at CLI of a surviving host returns “Cannot perform
operation as the host is running in emergency mode”
• Under some circumstances (such as power outages) there may have been multiple XenServer host failures in the pool. In this case, the pool master should be attempted to be recovered first.
© 2012 Citrix | Confidential – Do Not Distribute
Who’s the Pool Master Anyways?
• Determined through pool.conf located under /etc/xensourceᵒ Master for PoolMasterᵒ slave:<pool master IP> for all others
• Important Commands when in Emergency Mode:ᵒ xe pool-emergency-reset-masterᵒ xe pool-emergency-transition-to-master ᵒ Follow up with: xe pool-recover-slavesᵒ Check that default-SR is valid: xe pool-param-list
© 2012 Citrix | Confidential – Do Not Distribute
Step 2 - Recover Pool operations
If recovery of the pool master is not possible within a short timeframe, it will be necessary to promote a member server to a master to recover pool management and restart any failed Virtual Machines.
1. Select any running XenServer within the pool that will be promoted. 2. From the server’s command line, issue the following command: xe pool-emergency-
transition-to-master3. Once the command has completed, recover connections to the other member
servers using the following command: xe pool-recover-slaves4. Verify that pool management has been restored by issuing a test command at the CLI
(xe host-list)
© 2012 Citrix | Confidential – Do Not Distribute
Step 3 - Verify which XenServer(s) failed
This step verifies which XenServer(s) in the pool have failed
1. Issue the following command at the Command line of a surviving pool member: xe host-list params=uuid,name-label,host-metrics-live
2. Any servers listed as host-metrics-live = false have failed.
If it was necessary to recover the Pool Master with Step 2, the Master will now show as true.
3. Note down the first few characters of the UUID of any failed servers (or the UUID of the Master server if it originally failed).
© 2012 Citrix | Confidential – Do Not Distribute
Step 3 - Verify which XenServer(s) failed
ᵒ Result of xe host-list params=uuid,name-label,host-metrics-live
© 2012 Citrix | Confidential – Do Not Distribute
Step 4 - Verify which Virtual Machines have failed
• Regardless of whether a Master or Slave server fails, virtual machines running on other hosts continue to run.
1. Issue command at the CLI of a surviving server. Use the UUIDs from Step 3, type first 3 digits of the UUID, and press [tab] to complete: xe vm-list is-control-domain=false resident-on=UUID_of_failed_server
2. Note that the power state of these VMs is still “running” even though the server has failed. Step 5 will take care of this problem.
3. Repeat this step if necessary using the UUID of any other failed servers (including master).
© 2012 Citrix | Confidential – Do Not Distribute
Step 4 - Verify which Virtual Machines have failed
• Result of xe vm-list is-control-domain=false resident-on=UUID_of_failed_server
© 2012 Citrix | Confidential – Do Not Distribute
Step 5 - Reset power state on failed VMs
• To restart VMs after a host failure, it is necessary to reset their power state.
1. Issue the following command at the command line of a surviving server: xe vm-reset-powerstate resident-on= UUID_of_failed_server --force –multiple
ᵒ Alternately, you can reset VMs individually.
2. Verify that there are no VMs still listed as resident on the failed server by repeating step 4. The vm-list command should now return no results.
© 2012 Citrix | Confidential – Do Not Distribute
Step 6 - Restart VMs on another XenServer
1. Load XenCenter and verify that each VM that was originally running on the failed server is now marked as halted (Red icon next to the VM)
Note: VMs which have a home server assigned will not appear in XenCenter – Why?
ᵒ xe vm-param-set uuid=<uuid of vm to change> affinity=<uuid of new home server>
2. Restart each VM on a surviving pool member
© 2012 Citrix | Confidential – Do Not Distribute
Collecting Crash Dumps (Regular Style)
1. Configure System for Memory Dump: http://support.microsoft.com/kb/927069
2. Find the Universally Unique Identifier (UUID) of the VM from which you would like to collect memory dump using: xe vm-list
3. Based on the UUID of the virtual machine, find a domain ID in the current system. The Domain ID changes every time the VM starts: list_domains
4. Use the Domain ID (1st number listed) to crash the host using: /usr/lib/xen/bin/crash_guest <VM Domain>
© 2012 Citrix | Confidential – Do Not Distribute
Collecting Crash Dumps (Ninja Style)
• Crashdump redirection to dom0 filespaceᵒ When streaming PVS guests, where you cannot save the crashdump to the Pagefile
after a crash due to the network being disabled automaticallyᵒ When the root drive doesn't have enough space to store a Pagefile as large as the
guest's memory size
• Consider mounting either an iscsi lun or nfs, cifs filesystem under /crashdumps to store large dumps since root partition typically has <1.5GB available space
© 2012 Citrix | Confidential – Do Not Distribute
Collecting Crash Dumps (Ninja Style)
1. Open /opt/xensource/libexec/qemu-dm-wrapper with vi
2. Search for "SDL_DISABLE_WAITMAPPED“ and modify by adding the following at the line immediately following it:
ᵒ qemu_args.append("-priv")ᵒ qemu_args.append("-dumpdir")ᵒ qemu_args.append("/crashdumps")ᵒ qemu_args.append("-dumpquota")ᵒ qemu_args.append("1024")
3. Create the directory where the dumps will be stored: mkdir -p /crashdumps
© 2012 Citrix | Confidential – Do Not Distribute
Collecting Crash Dumps (Ninja Style)
4. Reboot the VM
5. use the crash_guest util to force the crashdump. If the condition leads to the BSOD, just wait for the streaming to finish and check the file at /crashdumps. Example:
Find dom ID: xe vm-list name-label="Windows Server 2003" params=dom-id
Result: dom-id ( RO) : 3To crash the domain id 3: /usr/lib/xen/bin/crash_guest 3
6. Copy the dumps outside the VM with a tool such as winscp.
© 2012 Citrix | Confidential – Do Not Distribute
VM Needs Imaginary Storage
• While working on-site with a client, XenServer decided to subscribe to the cult of Jacob Marley and refused to turn on VM saying ‘This VM needs storage that cannot be seen from that host’
© 2012 Citrix | Confidential – Do Not Distribute
VM Needs Imaginary Storage
• Caused by CIFS/DVD drives on XenServer and the machine where the ISOs/DVD resided failed.
1. xe vm-cd-eject –multiple (Gets rid of Faulty CD/DVD media)
© 2012 Citrix | Confidential – Do Not Distribute
File System on Control Domain Full
© 2012 Citrix | Confidential – Do Not Distribute
File System on Control Domain Full
• Result of the Dom0 Storage allocation becoming Full
• Typically caused by overgrown Log Filesᵒ XenServer logrotate feature can break - it cannot parse the size parameter.
1. Check Disk Space to validate issue: df –h
2. Check what’s causing the problem: find {/path/to/directory/} -type f -size +{size-in-kb}k -exec ls -lh {} \; | awk ‘{ print $9 “: ” $5 }’
ᵒ Specify the path and the size in kb
3. In order to determine root cause lower the log rotations and enable compression (/etc/logrotate.conf)
© 2012 Citrix | Confidential – Do Not Distribute
File System on Control Domain Full
4. In Case of Total Loss (XAPI died): xe-toolstack-restart
5. Typically this issue is due to TCPOffload freaking out causing unnecessary entries in logfiles.
6. Run Script to disable TCP/Checksum Offloadᵒ Disabling TCP Offload and checksum means that the main CPU handles all the
load which was previously handled directly by the network card
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Unable To See NICs
• The XenServers are experiencing problems detecting their network cards. The network cards are registered in the XenServer OS, however, they do not display in the XenServer console.
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Unable To See NICs
• Issue is caused by disconnecting the pool master while it has attached slave servers.
1. Log into the local XenServer, opening the file pool.conf located under /etc/xensource, deleting the contents of this file and replacing them with the word master.
2. Set the management console’s NIC to a valid IP/subnet
3. Run xe-toolstack-restart
4. Edit the pool.conf again to read slave:<pool master IP>
© 2012 Citrix | Confidential – Do Not Distribute
Frozen VM
• My VM(s) Froze and now XenCenter refuses to respond because it simply does not care about how troubled my life already is
© 2012 Citrix | Confidential – Do Not Distribute
Frozen VM
• This is typically caused by conflicting commands being sent through the Xen Center console resulting in erroneous metadata
• To force reboot VM (Like a Boss)
1. Find out the target VM UUID, type the command line : xe vm-list (Hint: Use | more when there are too many VMs)
2. Find your VM(s) UUID
3. Run the command: xe vm-reset-powerstate uuid=<Machine UUID> force=true
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Installation Fails
• I started installing XenServer on a bunch of machines, went out to a super relaxing lunch only to come back to my XenServer installation stuck at 49%
© 2012 Citrix | Confidential – Do Not Distribute
XenServer Installation Fails
• Some versions of XenServer do not include the firmware for the Qlogic QLA2100, QLA2200, QLA2300 and QLA2322 series NICs
1. Turn on the host, using the XenServer installation media
2. At the boot prompt, type: shell
3. You will now be presented with a command prompt. Type the following: rmmod qla2xxx; rmmod qlge
4. Type: Exit
5. Install XS normally
© 2012 Citrix | Confidential – Do Not Distribute
Commands CheatSheet
• XenServer Host and Dom0: xentop
• Hostᵒ xe-toolstack-restart to restart XAPIᵒ eject to eject physical CD from server.ᵒ cat /etc/xensource-inventory to see your host information.
• Networkᵒ ifconfig to see what interfaces are up for networking.ᵒ ethtool –p eth0 60 for make NIC flash for identification.ᵒ ethtool eth0 to check the status of the interface.ᵒ ethtool –i eth0 to check the driver type and version of NIC.
© 2012 Citrix | Confidential – Do Not Distribute
Commands CheatSheet
• Diskᵒ fdisk –l to view local disk information.ᵒ df –h to see how much space you have left in root disk.
• Multipathᵒ multipath -ll to view the current mulitpath topology as presented by control domain.
• VMᵒ xe vm-reboot vm=<VM Name> force=true to hard-reboot a VM.
• Logsᵒ xen-bugtool –yestoall to get the logs for support.ᵒ tail –f /var/log/messages to view events in messages log.
XenServer Monitoring & TroubleshootingLab 4 -Show Me Your Commands!
© 2012 Citrix | Confidential – Do Not Distribute
In this lab… 60 minutes
• Install Performance monitoring pack (new with XenServer 6.1)
• Configure Performance VM and run sample workloads (Discuss results)
• TAAS.CITRIX.COM – Upload server status reports
• (Advanced) – Break/Fix lab
Advanced XenServer TrainingQuestions, Yes?
Work better. Live better.