Architecture and Design - VMware Validated Design 4 · Architecture and Design ... VMware Validated Design Architecture and Design is intended for ... is tightly coupled with the

Architecture and DesignVMware Validated Design 4.0VMware Validated Design for Software-Defined DataCenter 4.0

Architecture and Design

VMware, Inc. 2

You can find the most up-to-date technical documentation on the VMware website at:

https://docs.vmware.com/

If you have comments about this documentation, submit your feedback to

[email protected]

Copyright © 2016, 2017 VMware, Inc. All rights reserved. Copyright and trademark information.

VMware, Inc.3401 Hillview Ave.Palo Alto, CA 94304www.vmware.com

https://docs.vmware.com/

mailto:[email protected]

http://pubs.vmware.com/copyright-trademark.html

Contents

About VMware Validated Design Architecture and Design 5

1 Architecture Overview 6

Physical Infrastructure Architecture 8

Pod Architecture 8

Pod Types 9

Physical Network Architecture 10

Availability Zones and Regions 18

Virtual Infrastructure Architecture 19

Virtual Infrastructure Overview 19

Network Virtualization Components 21

Network Virtualization Services 22

Cloud Management Platform Architecture 25

Logical Architecture of the Cloud Management Platform 26

Cloud Management Layer Elements 27

Cloud Management Platform Logical Architecture 28

Operations Architecture 29

Operations Management Architecture 29

Logging Architecture 33

Data Protection and Backup Architecture 36

Disaster Recovery Architecture 37

2 Detailed Design 39

Physical Infrastructure Design 40

Physical Design Fundamentals 40

Physical Networking Design 45

Physical Storage Design 54

Virtual Infrastructure Design 62

ESXi Design 65

vCenter Server Design 67

Virtualization Network Design 80

NSX Design 96

Shared Storage Design 118

Cloud Management Platform Design 137

vRealize Automation Design 138

vRealize Orchestrator Design 167

Operations Infrastructure Design 176

vRealize Operations Manager Design 177

VMware, Inc. 3

vRealize Log Insight Design 192

vSphere Data Protection Design 205

Site Recovery Manager and vSphere Replication Design 213

vSphere Update Manager Design 225


VMware, Inc. 4

About VMware Validated DesignArchitecture and Design

TheVMware Validated Design Architecture and Design document contains a validated model of theSoftware-Defined Data Center (SDDC) and provides a detailed design of each management componentof the SDDC stack.

Chapter 1 Architecture Overview discusses the building blocks and the main principles of each layerSDDC management layer.Chapter 2 Detailed Design provides the available design options according tothe design objective, and a set of design decisions to justify selecting the path for building each SDDCcomponent.

Intended AudienceVMware Validated Design Architecture and Design is intended for cloud architects, infrastructureadministrators and cloud administrators who are familiar with and want to use VMware software to deployin a short time and manage an SDDC that meets the requirements for capacity, scalability, backup andrestore, and extensibility for disaster recovery support.

Required VMware SoftwareVMware Validated Design Architecture and Design is compliant and validated with certain productversions. See VMware Validated Design Release Notes for more information about supported productversions.

VMware, Inc. 5

Architecture Overview 1The VMware Validated Design for Software-Defined Data Center (SDDC) enables an IT organization toautomate the provisioning of common repeatable requests and to respond to business needs with moreagility and predictability. Traditionally this has been referred to as IaaS, or Infrastructure as a Service,however the VMware Validated Design for Software-Defined Data Center extends the typical IaaSsolution to include a broader and more complete IT solution.

The VMware Validated Design architecture is based on a number of layers and modules, which allowsinterchangeable components be part of the end solution or outcome such as the SDDC. If a particularcomponent design does not fit a business or technical requirement for whatever reason, it should be ableto be swapped out for another similar component. The VMware Validated Designs are one way of puttingan architecture together. They are rigorously tested to ensure stability, scalability and compatibility.Ultimately, the system is designed in such a way as to ensure the desired IT outcome will be achieved.

Figure 1‑1. Architecture Overview

ServiceManagement

Portfolio Management

OperationsManagement

CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity

Fault Tolerance and Disaster

Recovery

Backup & Restore

Hypervisor

Pools of Resources

Virtualization Control

VirtualInfrastructure

Layer

Compute

Storage

Network

PhysicalLayer

Security

Replication Compliance

Risk

Governance

VMware, Inc. 6

Physical LayerThe lowest layer of the solution is the Physical Layer, sometimes referred to as the 'core', which consistsof three main components, Compute, Network and Storage. Inside the compute component sit the x86based servers that run the management, edge and tenant compute workloads. There is some guidancearound the physical capabilities required to run this architecture, however no recommendations on thetype or brand of hardware is given. All components must be supported on the VMware HardwareCompatibility guide.

Virtual Infrastructure LayerSitting on the Physical Layer components is the Virtual Infrastructure Layer. Within the VirtualInfrastructure Layer, access to the physical underlying infrastructure is controlled and allocated to themanagement and tenant workloads. The Virtual Infrastructure Layer consists primarily of the physicalhost's hypervisor and the control of these hypervisors. The management workloads consist of elements inthe virtual management layer itself, along with elements in the Cloud Management Layer, ServiceManagement, Business Continuity and Security areas.

Cloud Management LayerThe Cloud Management Layer is the "top" layer of the stack and is where the service consumptionoccurs. Typically through a UI or API, this layer calls for resources and then orchestrates the actions ofthe lower layers to achieve the request. While the SDDC can stand on its own without any other ancillaryservices, for a complete SDDC experience other supporting components are needed. The ServiceManagement, Business Continuity and Security areas complete the architecture by providing this support.

Service ManagementWhen building any type of IT infrastructure, portfolio and operations management play key roles incontinued day-to-day service delivery. The Service Management area of this architecture mainly focuseson operations management in particular monitoring, alerting and log management.

Business ContinuityTo ensure a system is enterprise ready, it must contain elements to support business continuity in thearea of data backup, restoration and disaster recovery. This area ensures that when data loss occurs, theright elements are in place to prevent permanent loss to the business. The design providescomprehensive guidance on how to operate backup and restore functions, along with run books detailinghow to fail over components in the event of a disaster.


VMware, Inc. 7

SecurityAll systems need to be inherently secure by design. This is to reduce risk and increase compliance whilestill providing a governance structure. The security area outlines what is needed to ensure the entireSDDC is resilient to both internal and external threats.

This chapter includes the following topics:

n Physical Infrastructure Architecture

n Virtual Infrastructure Architecture

n Cloud Management Platform Architecture

n Operations Architecture

Physical Infrastructure ArchitectureThe architecture of the data center physical layer is based on logical hardware pods, a leaf-and-spinenetwork topology, and zones and regions for high availability.

Pod ArchitectureThe VMware Validated Design for SDDC uses a small set of common building blocks called pods.

Pod Architecture CharacteristicsPods can include different combinations of servers, storage equipment, and network equipment, and canbe set up with varying levels of hardware redundancy and varying quality of components. Pods areconnected to a network core that distributes data between them. The pod is not defined by any hardphysical properties, as it is a standard unit of connected elements within the SDDC network fabric.

A pod is a logical boundary of functionality for the SDDC platform. While each pod usually spans onerack, it is possible to aggregate multiple pods into a single rack in smaller setups. For both small andlarge setups, homogeneity and easy replication are important.

Different pods of the same type can provide different characteristics for varying requirements. Forexample, one compute pod could use full hardware redundancy for each component (power supplythrough memory chips) for increased availability. At the same time, another compute pod in the samesetup could use low-cost hardware without any hardware redundancy. With these variations, thearchitecture can cater to the different workload requirements in the SDDC.

One of the guiding principles for such deployments is that VLANs are not spanned beyond a single podby the network virtualization layer. Although this VLAN restriction appears to be a simple requirement, ithas widespread impact on how a physical switching infrastructure can be built and on how it scales.


VMware, Inc. 8

Pod to Rack MappingPods are not mapped one-to-one to 19" data center racks. While a pod is an atomic unit of a repeatablebuilding block, a rack is merely a unit of size. Because pods can have different sizes, how pods aremapped to 19" data center racks depends on the use case.

n One Pod in One Rack. One pod can occupy exactly one rack.

n Multiple Pods in One Rack. Two or more pods can occupy a single rack, for example, onemanagement pod and one shared edge and compute pod can be deployed to a single rack.

n Single Pod Across Multiple Racks. A single pod can stretch across multiple adjacent racks. Forexample, a storage pod with filer heads and disk shelves can span more than one rack or a computepod that has more host then a single rack can support.

Note The mangement and the shared edge and compute pods can not span racks. This is due to NSXcontrollers and other virtual machines on a VLAN backed network migrating to a different rack where thatIP subnet is not available due to layer 2 termination at the Top of Rack switch.

Pod TypesThe SDDC differentiates between different types of pods including management pod, compute pod, edgepod, shared edge and compute pod, and storage pod. Each design includes several pods.

Figure 1‑2. Pods in the SDDC

ToR Switch

ToR Switch

Management pod(4 ESXi hosts)

Edge and Compute pod(4 ESXi hosts)

External connection


VMware, Inc. 9

Management PodThe management pod runs the virtual machines that manage the SDDC. These virtual machines hostvCenter Server, vSphere Update Manager, NSX Manager, NSX Controller, vRealize Operations Manager,vRealize Automation, vRealize Log Insight, and other management components. Because themanagement pod hosts critical infrastructure, consider implementing a basic level of hardwareredundancy for this pod.

Management pod components must not have tenant-specific addressing.

Shared Edge and Compute PodThe shared edge and compute pod runs the required NSX services to enable north-south routingbetween the SDDC and the external network, and east-west routing inside the SDDC. This shared podalso hosts the SDDC tenant virtual machines (sometimes referred to as workloads or payloads). As theSDDC grows, additional compute-only pods can be added to support a mix of different types of workloadsfor different types of Service Level Agreements (SLAs).

Compute PodCompute pods host the SDDC tenant virtual machines (sometimes referred to as workloads or payloads).An SDDC can mix different types of compute pods and provide separate compute pools for different typesof SLAs.

Storage PodStorage pods provide network-accessible storage using NFS or iSCSI. Different types of storage podscan provide different levels of SLA, ranging from just a bunch of disks (JBODs) using IDE drives withminimal to no redundancy, to fully redundant enterprise-class storage arrays. For bandwidth-intense IP-based storage, the bandwidth of these pods can scale dynamically.

Physical Network ArchitectureThe physical network architecture is tightly coupled with the pod-and-core architecture, and uses a Layer3 leaf-and-spine network instead of the more traditional 3-tier data center design.

Leaf-and-Spine Network ArchitectureA leaf-and-spine network is the core building block for the physical network in the SDDC.

n A leaf switch is typically located inside a rack and provides network access to the servers inside thatrack, it is also referred to as a Top of Rack (ToR) switch.

n A spine switch is in the spine layer and provides connectivity between racks. Links between spineswitches are typically not required. If a link failure occurs between a spine switch and a leaf switch,the routing protocol ensures that no traffic for the affected rack is sent to the spine switch that has lostconnectivity to that rack.


VMware, Inc. 10

Figure 1‑3. Leaf-and-Spine Physical Network Design

Leaf Leaf Leaf Leaf Leaf Leaf

Spine Spine Spine

Ports that face the servers inside a rack should have a minimal configuration, shown in the following high-level physical and logical representation of the leaf node.

Note Each leaf node has identical VLAN configuration with unique /24 subnets assigned to each VLAN.

Figure 1‑4. High-Level Physical and Logical Representation of a Leaf Node

Span

of V

LAN

s

Layer 3 ToR Switch

Layer 2

Layer 3

VLAN 1612 VLAN 1614 VLAN 1613VLAN 1611

VLAN Trunk (802.1Q)

vMotion VXLAN StorageMgmt172.16.11.0/24

DGW:172.16.11.253

172.16.12.0/24DGW:

172.16.12.253

172.16.13.0/24DGW:

172.16.13.253

172.16.14.0/24DGW:

172.16.14.253

Span

of V

LAN

s

vSphere Host (ESXi)

Routed uplinks (ECMP)

Network TransportYou can implement the physical layer switch fabric for a SDDC by offering Layer 2 transport services orLayer 3 transport services to all components. For a scalable and vendor-neutral data center network, useLayer 3 transport.


VMware, Inc. 11

Benefits and Drawbacks for Layer 2 Transport

In a design that uses Layer 2 transport, leaf switches and spine switches form a switched fabric,effectively acting like one large switch. Using modern data center switching fabric products such as CiscoFabricPath, you can build highly scalable Layer 2 multipath networks without the Spanning Tree Protocol(STP). Such networks are particularly suitable for large virtualization deployments, private clouds, andhigh-performance computing (HPC) environments.

Using Layer 2 routing has the following benefits and drawbacks:

n The benefit of this approach is more design freedom. You can span VLANs, which is useful forvSphere vMotion or vSphere Fault Tolerance (FT).

n The drawback is that the size of such a deployment is limited because the fabric elements have toshare a limited number of VLANs. In addition, you have to rely on a specialized data center switchingfabric product from a single vendor because these products are not designed for interoperabilitybetween vendors.

Benefits and Drawbacks for Layer 3 Transport

A design using Layer 3 transport requires these considerations:

n Layer 2 connectivity is limited within the data center rack up to the leaf switch.

n The leaf switch terminates each VLAN and provides default gateway functionality. That is, it has aswitch virtual interface (SVI) for each VLAN.

n Uplinks from the leaf switch to the spine layer are routed point-to-point links. VLAN trunking on theuplinks is not allowed.

n A dynamic routing protocol, such as OSPF, ISIS, or BGP, connects the leaf switches and spineswitches. Each leaf switch in the rack advertises a small set of prefixes, typically one per VLAN orsubnet. In turn, the leaf switch calculates equal cost paths to the prefixes it received from other leafswitches.

Using Layer 3 routing has the following benefits and drawbacks:

n The benefit is that you can chose from a wide array of Layer 3 capable switch products for thephysical switching fabric. You can mix switches from different vendors due to general interoperabilitybetween implementation of OSPF, ISIS or BGP. This approach is typically more cost effectivebecause it makes use of only the basic functionality of the physical switches.

n A design restriction, and thereby a drawback of using Layer 3 routing, is that VLANs are restricted toa single rack. This affects vSphere vMotion, vSphere Fault Tolerance, and storage networks.

Infrastructure Network ArchitectureA key goal of network virtualization is to provide a virtual-to-physical network abstraction.

To achieve this, the physical fabric must provide a robust IP transport with the following characteristics:

n Simplicity

n Scalability


VMware, Inc. 12

n High bandwidth

n Fault-tolerant transport

n Support for different levels of quality of service (QoS)

Simplicity and Scalability

Simplicity and scalability are the first and most critical requirements for networking.

Simplicity

Configuration of the switches inside a data center must be simple. General or global configuration suchas AAA, SNMP, syslog, NTP, and others should be replicated line by line, independent of the position ofthe switches. A central management capability to configure all switches at once is an alternative.

Scalability

Scalability factors include, but are not limited to, the following:

n Number of racks supported in a fabric.

n Amount of bandwidth between any two racks in a data center.

n Number of paths from which a leaf switch can select when communicating with another rack.

The total number of ports available across all spine switches and the oversubscription that is acceptabledetermine the number of racks supported in a fabric. Different racks may host different types ofinfrastructure, which can result in different bandwidth requirements.

n Racks with storage systems might attract or source more traffic than other racks.

n Compute racks, such as racks hosting hypervisors with workloads or virtual machines, might havedifferent bandwidth requirements than shared edge and compute racks, which provide connectivity tothe outside world.

Link speed and the number of links vary to satisfy different bandwidth demands. You can vary them foreach rack without sacrificing other aspects of the leaf-and-spine architecture.


VMware, Inc. 13

Figure 1‑5. Pod Network DesignSpineSwitch

SpineSwitch

SpineSwitch

ToR Switch

ToR Switch

ToR Switch

ToR Switch

Compute pods (19 ESXi hosts each)

Shared Edge andCompute pod(4 ESXi hosts)


External connection

ToR Switch

ToR Switch

L2

L3

The number of links to the spine switches dictates how many paths for traffic from this rack to anotherrack are available. Because the number of hops between any two racks is consistent, the architecture canutilize equal-cost multipathing (ECMP). Assuming traffic sourced by the servers carries a TCP or UDPheader, traffic distribution can occur on a per-flow basis.

Oversubscription

In leaf-and-spine topologies, oversubscription typically occurs at the leaf switch.

Oversubscription is equal to the total amount of bandwidth available to all servers connected to a leafswitch divided by the aggregate amount of uplink bandwidth.

oversubscription = total bandwidth / aggregate uplink bandwidth

For example, 19 servers with one 10 Gigabit Ethernet (10 GbE) port each create up to 190 Gbps ofbandwidth. In an environment with four 40 GbE uplinks to the spine (a total of 160 Gbps) a 1.2:1oversubscription results, as shown in Figure 1‑6.

1.2 (oversubscription) = 190 (total) / 160 (total uplink)


VMware, Inc. 14

Figure 1‑6. Oversubscription in the Leaf Layer

Leaf Leaf

Spine Spine Spine

19x10 GbE

Leaf

No Oversubscription

4x40 GbE

1.2 : 1

19x10 GbE

4x40 GbE

No Oversubscription

You can make more or less bandwidth available to a rack by provisioning more or fewer uplinks. Thatmeans you can change the available bandwidth on a per-rack basis.

Note The number of uplinks from a leaf switch to each spine switch must be the same to avoid hotspots.

For example, if a leaf switch has two uplinks to spine switch A and only one uplink to spine switches B, Cand D, more traffic is sent to the leaf switch by way of spine switch A, which might create a hotspot.

Fault Tolerance

The larger the environment, the more switches that make up the overall fabric and the greater thepossibility for one component of the data center switching fabric to fail. A resilient fabric can sustainindividual link or switch failures without widespread impact.


VMware, Inc. 15

Figure 1‑7. Compensation for a Link Failure

Х

Leaf L1 Leaf LN

Spine S1 Spine S2 Spine S3 Spine S4

LN via S1, S2,S3, S4

L1 via S1, S2,S3, S4

For example, if one of the spine switches fails, traffic between racks continues to be routed across theremaining spine switches in a Layer 3 fabric. The routing protocol ensures that only available paths arechosen. Utilizing more than two spine switches reduces the impact of a spine switch failure.

Multipathing-capable fabrics handle switch or link failures, reducing the need for manual networkmaintenance and operations. If a software upgrade of a fabric switch becomes necessary, theadministrator can take the node out of service gracefully by changing routing protocol metrics, which willquickly drain network traffic from that switch, freeing the switch for maintenance.

Depending on the width of the spine, that is, how many switches are in the aggregation or spine layer, theadditional load that the remaining switches must carry is not as significant as if there were only twoswitches in the aggregation layer. For example, in an environment with four spine switches, a failure of asingle spine switch only reduces the available capacity by 25%.

Quality of Service Differentiation

Virtualized environments carry different types of traffic, including tenant, storage and management traffic,across the switching infrastructure. Each traffic type has different characteristics and makes differentdemands on the physical switching infrastructure.

n Management traffic, although typically low in volume, is critical for controlling physical and virtualnetwork state.

n IP storage traffic is typically high in volume and generally stays within a data center.

For virtualized environments, the hypervisor sets the QoS values for the different traffic types. Thephysical switching infrastructure has to trust the values set by the hypervisor. No reclassification isnecessary at the server-facing port of a leaf switch. If there is a congestion point in the physical switchinginfrastructure, the QoS values determine how the physical network sequences, prioritizes, or potentiallydrops traffic.


VMware, Inc. 16

Figure 1‑8. Quality of Service (Differentiated Services) Trust Point

VM

Leaf

Trust or Set DSCP and CoS

Trust DSCP and CoS

Hypervisor

No Marking/Reclassification

Spine Spine Spine

Two types of QoS configuration are supported in the physical switching infrastructure.

n Layer 2 QoS, also called class of service.

n Layer 3 QoS, also called DSCP marking.

A vSphere Distributed Switch supports both class of service and DSCP marking. Users can mark thetraffic based on the traffic type or packet classification. When the virtual machines are connected to theVXLAN-based logical switches or networks, the QoS values from the internal packet headers are copiedto the VXLAN-encapsulated header. This enables the external physical network to prioritize the trafficbased on the tags on the external header.

Server Interfaces (NICs)If the server has more than one server interface (NIC) of the same speed, use two as uplinks with VLANstrunked to the interfaces.

The vSphere Distributed Switch supports many different NIC Teaming options. Load-based NIC teamingsupports optimal use of available bandwidth and supports redundancy in case of a link failure. Use two 10GbE connections for each server in combination with a pair of leaf switches. 802.1Q network trunks cansupport a small number of VLANs. For example, management, storage, VXLAN, vSphere Replication,and VMware vSphere vMotion traffic.


VMware, Inc. 17

Availability Zones and RegionsIn an SDDC, availability zones are collections of infrastructure components. Regions support disasterrecovery solutions and allow you to place workloads closer to your customers. Typically multipleavailability zones form a single region.

This VMware Validated Design uses two regions, but uses only one availability zone in each region. Thefollowing diagram shows how the design could be expanded to include multiple availability zones.

Figure 1‑9. Availability Zones and Regions

Region A: SFO Region B: LAX

AvailabilityZone

AvailabilityZone

AvailabilityZone

AvailabilityZone

AvailabilityZone

Availability ZonesEach availability zone is isolated from other availability zones to stop the propagation of failure or outageacross zone boundaries.

Together, multiple availability zones provide continuous availability through redundancy, helping to avoidoutages and improve SLAs. An outage that is caused by external factors (such as power, cooling, andphysical integrity) affects only one zone. Those factors most likely do not lead to an outage in other zonesexcept in the case of major disasters.

Each availability zone runs on its own physically distinct, independent infrastructure, and is engineered tobe highly reliable. Each zone should have independent power supply, cooling system, network, andsecurity. Common points of failures within a physical data center, like generators and cooling equipment,should not be shared across availability zones. Additionally, these zones should be physically separate sothat even uncommon disasters affect only a single availability zone. Availability zones are usually eithertwo distinct data centers within metro distance (latency in the single digit range) or two safety/fire sectors(data halls) within the same large scale data center.

Multiple availability zones (usually two) belong to a single region. The physical distance betweenavailability zones can be up to approximately 50 kilometers (30 miles), which offers low, single-digitlatency and large bandwidth by using dark fiber between the zones. This architecture allows the SDDCequipment across the availability zone to operate in an active/active manner as a single virtual datacenter or region.

You can operate workloads across multiple availability zones within the same region as if they were partof a single virtual data center. This supports an architecture with very high availability that is suitable formission critical applications. When the distance between two locations of equipment becomes too large,these locations can no longer function as two availability zones within the same region and need to betreated as separate regions.


VMware, Inc. 18

RegionsMultiple regions support placing workloads closer to your customers, for example, by operating oneregion on the US east coast and one region on the US west coast, or operating a region in Europe andanother region in the US.

Regions are helpful in several ways.

n Regions can support disaster recovery solutions: One region can be the primary site and anotherregion can be the recovery site.

n You can use multiple regions to address data privacy laws and restrictions in certain countries bykeeping tenant data within a region in the same country.

The distance between regions can be rather large. This design uses two example regions, one region isSan Francisco (SFO), the other region is Los Angeles (LAX).

Virtual Infrastructure ArchitectureThe virtual infrastructure is the foundation of an operational SDDC.

Within the virtual infrastructure layer, access to the physical underlying infrastructure is controlled andallocated to the management and tenant workloads. The virtual infrastructure layer consists primarily ofthe physical hosts' hypervisors and the control of these hypervisors. The management workloads consistof elements in the virtual management layer itself, along with elements in the cloud management layerand in the service management, business continuity, and security areas.

Figure 1‑10. Virtual Infrastructure Layer in the SDDC

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

Virtual Infrastructure OverviewThe SDDC virtual infrastructure consists of two regions. Each region includes a management pod and ashared edge and compute pod.


VMware, Inc. 19

Figure 1‑11. SDDC Logical Design

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OS

APP

OS

APP

OSAPP

OS

ESXi ESXi

APP

OSAPP

OS

APP

OSAPP

OS

Virtual InfrastructureManagement

NSXController

(Mgmt)

OtherManagementApplications

NSXEdge

(Mgmt)

NSXManager(Mgmt)

NSXManager

(Compute)

NSXEdge

(Compute)

NSXController(Compute)

ESXi ESXi ESXi ESXi ESXi ESXi

SDDCPayload

Virtual Infrastructure Compute Edge

NSX Transport Zone (Compute)

vDS (Compute) vDS (Mgmt)

NSX Transport Zone (Mgmt)

Shared Edge and Compute Cluster

Management Cluster

Managed by: Compute vCenter Server

Managed by: Management vCenter Server

Network: External(Internet/MPLS)

Network: Internal SDDCFabric (Spine/Leaf)

Management and Shared Edge and Compute Pod

vCenterServer(Mgmt)

vCenterServer

(Compute)

Management PodManagement pods run the virtual machines that manage the SDDC. These virtual machines host vCenterServer, vSphere Update Manager, NSX Manager, NSX Controller, vRealize Operations, vRealize LogInsight, vRealize Automation, Site Recovery Manager and other shared management components. Allmanagement, monitoring, and infrastructure services are provisioned to a vSphere cluster which provideshigh availability for these critical services. Permissions on the management cluster limit access to onlyadministrators. This limitation protects the virtual machines that are running the management, monitoring,and infrastructure services.


VMware, Inc. 20

Shared Edge and Compute PodThe shared edge and compute pod runs the required NSX services to enable north-south routingbetween the SDDC and the external network and east-west routing inside the SDDC. This pod also hoststhe SDDC tenant virtual machines (sometimes referred to as workloads or payloads). As the SDDC growsadditional compute-only pods can be added to support a mix of different types of workloads for differenttypes of SLAs.

Network Virtualization ComponentsVMware NSX for vSphere, the network virtualization platform, is a key solution in the SDDC architecture.The NSX for vSphere platform consists of several components that are relevant to the networkvirtualization design.

NSX for vSphere PlatformNSX for vSphere creates a network virtualization layer. All virtual networks are created on top of this layer,which is an abstraction between the physical and virtual networks. Several components are required tocreate this network virtualization layer:

n vCenter Server

n NSX Manager

n NSX Controller

n NSX Virtual Switch


VMware, Inc. 21

These components are separated into different planes to create communications boundaries and provideisolation of workload data from system control messages.

Data plane Workload data is contained wholly within the data plane. NSX logicalswitches segregate unrelated workload data. The data is carried overdesignated transport networks in the physical network. The NSX VirtualSwitch, distributed routing, and the distributed firewall are also implementedin the data plane.

Control plane Network virtualization control messages are located in the control plane.Control plane communication should be carried on secure physicalnetworks (VLANs) that are isolated from the transport networks that areused for the data plane. Control messages are used to set up networkingattributes on NSX Virtual Switch instances, as well as to configure andmanage disaster recovery and distributed firewall components on eachESXi host.

Management plane The network virtualization orchestration happens in the management plane.In this layer, cloud management platforms such as VMware vRealize®

Automation™ can request, consume, and destroy networking resources forvirtual workloads. Communication is directed from the cloud managementplatform to vCenter Server to create and manage virtual machines, and toNSX Manager to consume networking resources.

Network Virtualization ServicesNetwork virtualization services include logical switches, logical routers, logical firewalls, and othercomponents of NSX for vSphere.

Logical SwitchesNSX for vSphere logical switches create logically abstracted segments to which tenant virtual machinescan connect. A single logical switch is mapped to a unique VXLAN segment ID and is distributed acrossthe ESXi hypervisors within a transport zone. This allows line-rate switching in the hypervisor withoutcreating constraints of VLAN sprawl or spanning tree issues.

Universal Distributed Logical RouterThe NSX for vSphere Universal Distributed Logical Router is optimized for forwarding in the virtualizedspace (between VMs, on VXLAN- or VLAN-backed port groups). Features include:

n High performance, low overhead first hop routing.

n Scaling the number of hosts.

n Support for up to 1,000 logical interfaces (LIFs) on each distributed logical router.


VMware, Inc. 22

The Universal Distributed Logical Router is installed in the kernel of every ESXi host, as such it requires aVM to provide the control plane. The universal distributed logical router Control VM is the control planecomponent of the routing process, providing communication between NSX Manager and NSX Controllercluster through the User World Agent. NSX Manager sends logical interface information to the Control VMand NSX Controller cluster, and the Control VM sends routing updates to the NSX Controller cluster.

Figure 1‑12. NSX for vSphere Universal Distributed Logical Router

Region A

Universal Transit Logical Switch

Universal Distributed Logical Router

Universal Logical Switches

NSX N/S Edges

UDLRController

BGPECMP

BGPECMPBGP

ECMP

NSX N/S Edges

BGPECMP

Region B

ToRSwitches

ToRSwitches

Designated InstanceThe designated instance is responsible for resolving ARP on a VLAN LIF. There is one designatedinstance per VLAN LIF. The selection of an ESXi host as a designated instance is performedautomatically by the NSX Controller cluster and that information is pushed to all other hosts. Any ARPrequests sent by the distributed logical router on the same subnet are handled by the same host. In caseof host failure, the controller selects a new host as the designated instance and makes that informationavailable to other hosts.

User World AgentUser World Agent (UWA) is a TCP and SSL client that enables communication between the ESXi hostsand NSX Controller nodes, and the retrieval of information from NSX Manager through interaction withthe message bus agent.


VMware, Inc. 23

Edge Services GatewayWhile the Universal Logical Router provides VM to VM or east-west routing, the NSX Edge servicesgateway provides north-south connectivity, by peering with upstream Top of Rack switches, therebyenabling tenants to access public networks.

Logical FirewallNSX for vSphere Logical Firewall provides security mechanisms for dynamic virtual data centers.

n The Distributed Firewall allows you to segment virtual data center entities like virtual machines.Segmentation can be based on VM names and attributes, user identity, vCenter objects like datacenters, and hosts, or can be based on traditional networking attributes like IP addresses, portgroups, and so on.

n The Edge Firewall component helps you meet key perimeter security requirements, such as buildingDMZs based on IP/VLAN constructs, tenant-to-tenant isolation in multi-tenant virtual data centers,Network Address Translation (NAT), partner (extranet) VPNs, and user-based SSL VPNs.

The Flow Monitoring feature displays network activity between virtual machines at the application protocollevel. You can use this information to audit network traffic, define and refine firewall policies, and identifythreats to your network.

Logical Virtual Private Networks (VPNs)SSL VPN-Plus allows remote users to access private corporate applications. IPSec VPN offers site-to-siteconnectivity between an NSX Edge instance and remote sites. L2 VPN allows you to extend yourdatacenter by allowing virtual machines to retain network connectivity across geographical boundaries.

Logical Load BalancerThe NSX Edge load balancer enables network traffic to follow multiple paths to a specific destination. Itdistributes incoming service requests evenly among multiple servers in such a way that the loaddistribution is transparent to users. Load balancing thus helps in achieving optimal resource utilization,maximizing throughput, minimizing response time, and avoiding overload. NSX Edge provides loadbalancing up to Layer 7.

Service ComposerService Composer helps you provision and assign network and security services to applications in avirtual infrastructure. You map these services to a security group, and the services are applied to thevirtual machines in the security group.

Data Security provides visibility into sensitive data that are stored within your organization's virtualizedand cloud environments. Based on the violations that are reported by the NSX for vSphere Data Securitycomponent, NSX security or enterprise administrators can ensure that sensitive data is adequatelyprotected and assess compliance with regulations around the world.


VMware, Inc. 24

NSX for vSphere ExtensibilityVMware partners integrate their solutions with the NSX for vSphere platform to enable an integratedexperience across the entire SDDC. Data center operators can provision complex, multi-tier virtualnetworks in seconds, independent of the underlying network topology or components.

Cloud Management Platform ArchitectureThe Cloud Management Platform (CMP) is the primary consumption portal for the entire Software-DefinedData Center (SDDC). Within the SDDC, users use vRealize Automation to author, administer, andconsume VM templates and blueprints.

Figure 1‑13. Cloud Management Platform Conceptual Architecture

Compute

Business Group(s)

Rainpole End User 1Production Workload

Rainpole End User 2Test/Dev Workload

Fabric Group(s)

Cloud Management PortalVM Templates and Blueprints

Tenant Access

Administration of virtualand cloud resources

Internal Virtual Resources

Network Storage Compute

External Cloud Resources

Network Storage

Cloud Admin

Admin Access

Blueprints

App and ServicesAuthoring

Cloud Admin

App Authoring

The Cloud Management Platform consists of the following elements and components.

Table 1‑1. Elements and Components of the Cloud Management Platform

Element Components

Users n Cloud administrators. Tenant, group, fabric, infrastructure, service, and other administrators as definedby business policies and organizational structure.

n Cloud (or tenant) users. Users within an organization that can provision virtual machines and directlyperform operations on them at the level of the operating system.

Tools and supportinginfrastructure

VM templates and blueprints are the building blocks that provide the foundation of the cloud. VM templatesare used to author the blueprints that tenants (end users) use to provision their cloud workloads.


VMware, Inc. 25

Table 1‑1. Elements and Components of the Cloud Management Platform (Continued)

Element Components

Provisioninginfrastructure

On-premise and off-premise resources which together form a hybrid cloud.n Internal Virtual Resources. Supported hypervisors and associated management tools.n External Cloud Resources. Supported cloud providers and associated APIs.

Cloud managementportal

A portal that provides self-service capabilities for users to administer, provision and manage workloads.n vRealize Automation portal, Admin access. The default root tenant portal URL used to set up and

administer tenants and global configuration options.n vRealize Automation portal, Tenant access. Refers to a subtenant and is accessed using with an

appended tenant identifier.

Note A tenant portal might refer to the default tenant portal in some configurations. In this case, the URLsmatch, and the user interface is contextually controlled by the role-based access control permissions thatare assigned to the tenant.

Logical Architecture of the Cloud Management PlatformThe design of the Cloud Management Platform considers characteristics such as availability,manageability, performance, scalability, and security. To provide this it must deliver a comprehensive setof multi-platform and multi-vendor cloud services.

The Cloud Management Platform layer delivers the following multi-platform and multi-vendor cloudservices.

n Comprehensive and purpose-built capabilities to provide standardized resources to global customersin a short time span.

n Multi-platform and multi-vendor delivery methods that integrate with existing enterprise managementsystems.

n Central user-centric and business-aware governance for all physical, virtual, private, and public cloudservices.

n Architecture that meets customer and business needs, and is extensible.

This design considers the following characteristics.

Availability Indicates the effect a choice has on technology and related infrastructure toprovide highly-available operations and sustain operations during systemfailures. VMware vSphere High Availability will provide the required hostredundancy and tolerance of hardware failures where appropriate.

Manageability Relates to the effect a choice has on overall infrastructure manageability.


VMware, Inc. 26

Key metrics: Accessibility and the lifecycle of the infrastructure beingmanaged.

Performance Reflects whether the option has a positive or negative impact on overallinfrastructure performance. This architecture follows the VMware referencearchitecture sizing guidelines to provide certain performancecharacteristics.

Key metrics: Performance analysis and tuning of the database, Managerservice, Model Manager, portal Web site, and data collection.

Scalability Depicts the effect the option has on the ability of the solution to beaugmented to achieve better sustained performance within theinfrastructure.

Key metrics: Web site latency, network traffic, and CPU usage on thedatabase and web servers.

Security Reflects whether the option has a positive or negative impact on overallinfrastructure security.

Key metrics: Data confidentiality, integrity, authenticity, and non-repudiationof cloud automation components and the option's integration withsupporting and provisioning infrastructures.

Cloud Management Layer ElementsThe Cloud Management Platform elements include software and physical components providing portal-based functionality and service catalog, Infrastructure as a Service (IaaS) components to model andprovision virtualized workloads, and orchestration engine.

Table 1‑2. Cloud Management Platform Elements

Element Component

vRealize Automation virtual appliance n vRealize Automation Portal Web/Application Servern vRealize Automation PostgreSQL Databasen vRealize Automation service catalogn VMware Identity Manager

vRealize Automation IaaS components n vRealize Automation IaaS Web Servern vRealize Automation IaaS Manager Services

Distributed execution components n vRealize Automation Distributed Execution Managers.n Orchestratorn Workers

Integration components n vRealize Automation Agent machines

vRealize Orchestrator components n vRealize Orchestrator virtual appliances

Provisioning infrastructure n vSphere environmentn Other supported physical, virtual, or cloud environments.


VMware, Inc. 27

Table 1‑2. Cloud Management Platform Elements (Continued)

Element Component

Costing components n vRealize Business for Cloud Standard servern vRealize Business for Cloud Standard data collector

Supporting infrastructure n Microsoft SQL database environmentn Active Directory environmentn SMTPn NTP

Cloud Management Platform Logical ArchitectureIn this architecture, vRealize Automation and vRealize Orchestrator run on a VXLAN-backed network thatis fronted by the NSX Logical Distributed Router. An NSX Edge services gateway, acting as a loadbalancer, is deployed to provide load balancing services for the CMP components.

Figure 1‑14. vRealize Automation Logical Architecture for Region A

IWS 1

OS

BUS 1

OSMSSQL

OS

IMS

OS

DEM 1

OS

DEM 2

OS

vRO 1

OS

vRO 2

OS

SVR 2

OS

IWS 2

OS

IAS 1

OS

IAS 2

OS

IMS 2

OS

BUC 1

OS

Active

SVR 1

OS

Active Standalone Standalone Active Passive

Active Active Active Active


Mgmt-xRegion01-VXLAN

Mgmt-RegionA01-VXLAN

Region Independent Region Specific

NSX Edge Load

Balancer

NSX Edge Load

Balancer

Cluster

ESXi Resource Cluster(s) vCenter Server NSX Manager

Active Directory DNS SMTP

Access Network Admin Network

vRA End User

vRA End User

Cloud Admin

Abbreviations

vRA vRealize AutomationvRO vRealize OrchestratorDEM Distributed Execution ManagerDNS Domain Name SystemSVR vRA ApplianceIWS IaaS Web ServerIMS IaaS Manager ServiceIAS IaaS vSphere Proxy AgentBUS vRealize Business ServerBUC vRealize Business CollectorMSSQL Microsoft SQL

vSphere-Mgmt

Ext-Mgmt

Standalone



VMware, Inc. 28

Figure 1‑15. vRealize Automation Logical Architecture for Region B

IAS 1

OS

IAS 2

OS

BUC 2

OS

Active Active

Mgmt-RegionB01-VXLAN

Region Specific

ESXi Resource Cluster(s) vCenter Server NSX Manager

Active Directory DNS SMTP

Access Network Admin Network

vRA End User

vRA End User

Cloud Admin

Abbreviations

vRA vRealize AutomationDNS Domain Name SystemIAS IaaS vSphere Proxy AgentBUC vRealize Business Collector

vSphere-Mgmt

Ext-Mgmt


Standalone

Operations ArchitectureThe architecture of the operations management layer includes management components that providesupport for the main types of operations in an SDDC. You can perform monitoring, logging, backup andrestore, and disaster recovery.

Operations Management ArchitecturevRealize Operations Manager tracks and analyzes the operation of multiple data sources within theSoftware-Defined Data Center (SDDC) by using specialized analytics algorithms. These algorithms helpvRealize Operations Manager to learn and predicts the behavior of every object it monitors. Users accessthis information by using views, reports, and dashboards.


VMware, Inc. 29

Installation ModelsvRealize Operations Manager is available in two different deployment models: a preconfigured virtualappliance, or a Windows or Linux installable package. Select the installation method for your deploymentaccording to the following considerations:

n When you use the vRealize Operations Manager virtual appliance, you deploy the OVF file of thevirtual appliance once for each cluster node. You access the product to set up cluster nodesaccording to their role, and log in to configure the installation.

Use virtual appliance deployment to easily create vRealize Operations Manager nodes with pre-defined identical size.

n When you use the Windows or Linux installable package, you run the vRealize Operations Managerinstallation on each cluster node. You access the product to set up cluster nodes according to theirrole, and log in to configure the installation.

Use installable package deployment to create vRealize Operations Manager node with customidentical size.

ArchitecturevRealize Operations Manager contains functional elements that collaborate for data analysis and storage,and support creating clusters of nodes with different roles.

Figure 1‑16. vRealize Operations Manager Architecture

HIS

Global xDB FSDB xDB

HIS

Global xDB FSDB xDB FSDB

Master Node

Controller

Analytics

Persistence

Master Replica Node Data Node

Product/Admin UI

REST API/Collector

Product/Admin UI

REST API/Collector

Product/Admin UI

REST API/Collector

Remote Collector

Product/Admin UI

REST API/Collector


VMware, Inc. 30

Types of Nodes and ClustersFor high availability and scalability, you can deploy several vRealize Operations Manager instances in acluster where they can have either of the following roles.

Master Node Required initial node in the cluster. In large-scale environments, the masternode manages all other nodes. In small-scale environments, the masternode is the single standalone vRealize Operations Manager node.

Master Replica Node (Optional) Enables high availability of the master node.

Data Node Enables scale-out of vRealize Operations Manager in larger environments.Data nodes have adapters installed to perform collection and analysis. Datanodes also host vRealize Operations Manager management packs.

Larger deployments usually include adapters only on data nodes, not onthe master node or replica node

Remote Collector Node In distributed deployments, enables navigation through firewalls, interfaceswith a remote data source, reduces bandwidth across regions, or reducesthe load on the vRealize Operations Manager analytics cluster. Remotecollector nodes only gather objects for the inventory and forward collecteddata to the data nodes. Remote collector nodes do not store data orperform analysis. In addition, you can install them on a different operatingsystem than the rest of the cluster nodes.

The master and master replica nodes are data nodes with extended capabilities.

vRealize Operations Manager can form two types of clusters according to the nodes that participate in acluster.

Analytics clusters Tracks, analyzes, and predicts the operation of monitored systems.Consists of a master node, data nodes, and optionally of a master replicanode.

Remote collectorscluster

Only collects diagnostics data without storage or analysis. Consists only ofremote collector nodes.

Application Functional Components The functional components of a vRealize Operations Manager instance interact to provide analysis ofdiagnostics data from the data center and visualize the result in the Web user interface.


VMware, Inc. 31

Figure 1‑17. vRealize Operations Manager Logical Node Architecture

HIS

Global xDB FSDB xDB

Cap

acity

Perfo

rman

ce

Master Node

Controller

Analytics

Persistence

Product/Admin UI

REST API/Collector

The components of vRealize Operations Manager node perform these tasks:

Admin / Product UIserver

The UI server is a Web application that serves as both user andadministration interface.

REST API / Collector The Collector collects data from all components in the data center.

Controller The Controller handles the data flow the UI server, Collector, and theanalytics engine.

Analytics The Analytics engine creates all associations and correlations betweenvarious data sets, handles all super metric calculations, performs allcapacity planning functions, and is responsible for triggering alerts.

Persistence The persistence layer handles the read and write operations on theunderlying databases across all nodes.

FSDB The File System Database (FSDB) stores collected metrics in raw format.FSDB is available in all the nodes.

xDB (HIS) The xDB stores data from the Historical Inventory Service (HIS). Thiscomponent is available only on the master and master replica nodes.

Global xDB The Global xDB stores user preferences, alerts, and alarms, andcustomization that is related to the vRealize Operations Manager. Thiscomponent is available only on the master and master replica nodes.

Management PacksManagement packs contain extensions and third-party integration software. They add dashboards, alertdefinitions, policies, reports, and other content to the inventory of vRealize Operations Manager. You canlearn more details about and download management packs from VMware Solutions Exchange.


VMware, Inc. 32

Multi-Region vRealize Operations Manager DeploymentThe scope of the SDDC design covers multiple regions. Using vRealize Operations Manager acrossmultiple regions requires deploying an analytics cluster that is protected by Site Recovery Manager, anddeploying remote collectors in each region.

Logging ArchitecturevRealize Log Insight provides real-time log management and log analysis with machine learning-basedintelligent grouping, high-performance searching, and troubleshooting across physical, virtual, and cloudenvironments.

OverviewvRealize Log Insight collects data from ESXi hosts using the syslog protocol. It connects to other VMwareproducts, like vCenter Server, to collect events, tasks, and alarms data, and integrates with vRealizeOperations Manager to send notification events and enable launch in context. vRealize Log Insight alsofunctions as a collection and analysis point for any system capable of sending syslog data. In addition tosyslog data an ingestion agent can be installed on Linux or Windows servers or may come pre-installedon certain VMware products to collect logs. This agent approach is especially useful for customapplication logs and operating systems that don't natively support the syslog protocol, such as Windows.

Installation ModelsYou can deploy vRealize Log Insight as a virtual appliance in one of the following configurations:

n Standalone node

n Highly available cluster of one master and at least two worker nodes using an integrated loadbalancer (ILB)

The compute and storage resources of the vRealize Log Insight instances can scale-up as growthdemands.


VMware, Inc. 33

Cluster NodesFor high availability and scalability, you can deploy several vRealize Log Insight instances in a clusterwhere they can have either of the following roles:

Master Node Required initial node in the cluster. The master node is responsible forqueries and log ingestion. The Web user interface of the master nodeserves as a single pane of glass, presenting data from multiple sources inthe cluster in a unified display. All queries against data are directed to themaster, which in turn queries the workers as appropriate.

Worker Node Enables scale-out in larger environments. A worker node is responsible foringestion of logs. A worker node stores logs locally. If a worker node isdown, the logs on that worker becomes unavailable. You need at least twoworker nodes to form a cluster with the master node.

Integrated LoadBalancer (ILB)

Provides high availability (HA). The ILB runs on one of the cluster nodes. Ifthe node that hosts the ILB Virtual IP (VIP) address stops responding, theVIP address is failed over to another node in the cluster.

Architecture of a ClusterThe architecture of vRealize Log Insight enables several channels for HA collection of log messages.

Figure 1‑18. Cluster Architecture of vRealize Log Insight

vRealize Log Insight Clients(ESXi, vCenter Server, NSX

for vSphere, vRealize Automation)

vRealize Log Insight User

Interface

vRealize Log Insight Cluster

vRealize Operations

Manager Master Worker1 Worker2 WorkerN

Content Packs

Ingestion API Syslog

vRealize Log Insight clients connect to ILB VIP address, and use the Web user interface and ingestion(by using syslog or the Ingestion API) to send logs to vRealize Log Insight.

By default, the vRealize Log Insight collects data from vCenter Server systems and ESXi hosts. Forforwarding logs from NSX for vSphere, and vRealize Automation, use content packs which containextensions or provide integration with other systems in the SDDC.


VMware, Inc. 34

Authentication ModelsYou can configure vRealize Log Insight user authentication to utilize one or more of the followingauthentication models:

n Microsoft Active Directory

n Local Accounts

n VMware Identity Manager

Integration with vRealize Operations ManagerThe integration with vRealize Operations Manager provides data from multiple sources to a central placefor monitoring the SDDC. vRealize Log Insight sends notification events to vRealize Operations Manager.You can also launch vRealize Log Insight from the vRealize Operations Manager Web user interface.

ArchivingvRealize Log Insight supports data archiving on NFS shared storage that each vRealize Log Insight nodecan access.

BackupYou back up each vRealize Log Insight cluster using traditional virtual machine backup solutions that usevSphere Storage APIs for Data Protection (VADP) compatible backup software such as vSphere DataProtection.

Multi-Region vRealize Log Insight DeploymentThe scope of the SDDC design covers multiple regions. Using vRealize Log Insight in a multi-regiondesign can provide a syslog infrastructure in all regions of the SDDC. Using vRealize Log Insight acrossmultiple regions requires deploying a cluster in each region. vRealize Log Insight supports eventforwarding to other vRealize Log Insight deployments across regions in the SDDC. Implementing failoverby using vSphere Replication or disaster recovery by using Site Recovery Manager is not necessary. Theevent forwarding feature adds tags to log message that identify the source region and event filteringprevents looping messages between the regions.


VMware, Inc. 35

Figure 1‑19. Event Forwarding in vRealize Log Insight

Management/Compute

vCenter Server

vRealizeAutomation


Master Worker Worker

NSX

vRealize Operations

Manager

event forwarding

VMDK Storage

Archiving Storage

Management/Compute

vCenter Server



NSX

VMDK Storage

Archiving Storage

Region A Region B

Data Protection and Backup ArchitectureYou can use a backup solution, such as vSphere Data Protection, to protect the data of your SDDCmanagement components on the management and edge clusters, and of the tenant workloads that runon the compute clusters.

Data protection solutions provide the following functions in the SDDC:

n Back up and restore virtual machines.

n Organization of virtual machines into groups by VMware product.

n Store data according to company retention policies.

n Inform administrators about backup and restore activities through reports.

vSphere Data Protection instances in the two regions provide data protection for the products thatimplement the management capabilities of the SDDC. vSphere Data Protection stores backups of themanagement product virtual appliances on a shared storage allocation according to a defined schedule.


VMware, Inc. 36

Figure 1‑20. Dual-Region Data Protection Architecture

VMVMVMVM

VMVirtual

ApplianceVirtual

Appliance

VMVMVMVM

VMVirtual

ApplianceVirtual

Appliance

vSphere Data Protection

Management Cluster

Storage

Region A Region B

Protected

SharedStorage

vSphere Data Protection

Management Cluster

Storage

Protected

SharedStorage

Disaster Recovery ArchitectureYou use VMware Site Recovery Manager to implement disaster recovery for the workloads of themanagement products in the SDDC.

Elements of Disaster RecoveryDisaster recovery that is based on VMware Site Recovery Manager has the following main elements:

n Dual-region configuration. All protected virtual machines are located in Region A that is considered asthe protected region, and are recovered in Region B that is considered as the recovery region.In atypical Site Recovery Manager installation, the protected region provides business-critical data centerservices. The recovery region is an alternative infrastructure to which Site Recovery Manager canmigrate these services.

n Replication of virtual machine data.

n Array-based replication. When you use array-based replication, one or more storage arrays at theprotected region replicate data to peer arrays at the recovery region. To use array-basedreplication with Site Recovery Manager, you must configure replication first before you canconfigure Site Recovery Manager to use it.


VMware, Inc. 37

n Replication by using vSphere Replication. You deploy the vSphere Replication appliance andconfigure vSphere Replication on virtual machines independently of Site RecoveryManager. vSphere Replication does not require storage arrays. The replication source and targetstorage can be any storage device, including, but not limited to, storage arrays. You canconfigure vSphere Replication to regularly create and retain snapshots of protected virtualmachines on the recovery region.

n Protection groups. A protection group is a collection of virtual machines that Site RecoveryManager protects together. You configure virtual machines and create protection groups differentlydepending on whether you use array-based replication or vSphere Replication. You cannot createprotection groups that combine virtual machines for which you configured array-based replication withvirtual machines for which you configured vSphere Replication.

n Recovery plans. A recovery plan specifies how Site Recovery Manager recovers the virtual machinesin the protection groups that it contains. You can include a combination of array-based replicationprotection groups and vSphere Replication protection groups in the same recovery plan.

Disaster Recovery ConfigurationThe VMware Validated Design implements the following disaster recovery configuration:

n The following management applications are a subject of disaster recovery protection:

n vRealize Automation together with vRealize Orchestrator and vRealize Business

n Analytics cluster of vRealize Operations Manager

n The virtual infrastructure components that are not in the scope of the disaster recovery protection,such as vRealize Log Insight, are available as separate instances in each region.

Figure 1‑21. Disaster Recovery Architecture

(by using vSphere Replication)

Region A Non-Replicated

vRealize Log Insight

Region A Virtual Infrastructure - Management

vSphereNSX for vSphere

vSphere Data ProtectionSite Recovery Manager

Region B Non-Replicated

vRealize Log Insight

Region B Replicated

vRealize Automation (shadow)

vRealize Operations Manager (shadow)

(by using vSphere Replication)

SRM

Region A Replicated

SRMvRealize Automation

vRealize Operations Manager

Region B Virtual Infrastructure - Management

vSphereNSX for vSphere

vSphere Data ProtectionSite Recovery Manager


VMware, Inc. 38

Detailed Design 2The Software-Defined Data Center (SDDC) detailed design considers both physical and virtualinfrastructure design. It includes numbered design decisions and the justification and implications of eachdecison.

Each section also includes detailed discussion and diagrams.

Physical InfrastructureDesign

Focuses on the three main pillars of any data center, compute, storage andnetwork. In this section you find information about availability zones andregions. The section also provides details on the rack and podconfiguration, and on physical hosts and the associated storage andnetwork configurations.

Virtual InfrastructureDesign

Provides details on the core virtualization software configuration. Thissection has information on the ESXi hypervisor, vCenter Server, the virtualnetwork design including VMware NSX, and on software-defined storagefor VMware vSAN. This section also includes details on business continuity(backup and restore) and on disaster recovery.

Cloud ManagementPlatform Design

Contains information on the consumption and orchestration layer of theSDDC stack, which uses vRealize Automation and vRealize Orchestrator.IT organizations can use the fully distributed and scalable architecture tostreamline their provisioning and decommissioning operations.

OperationsInfrastructure Design

Explains how to architect, install, and configure vRealize OperationsManager and vRealize Log Insight. You learn how to ensure that servicemanagement within the SDDC is comprehensive. This section ties directlyinto the Operational Guidance section.

This chapter includes the following topics:

n Physical Infrastructure Design

n Virtual Infrastructure Design

n Cloud Management Platform Design

n Operations Infrastructure Design

VMware, Inc. 39

Physical Infrastructure DesignThe physical infrastructure design includes details on decisions for availability zones and regions and thepod layout within datacenter racks.

Design decisions related to server, networking, and storage hardware are part of the physicalinfrastructure design.

Figure 2‑1. Physical Infrastructure Design

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

n Physical Design Fundamentals

Physical design fundamentals include decisions on availability zones and regions and on pod types,pods, and racks. The ESXi host physical design is also part of the design fundamentals.

n Physical Networking Design

The physical network uses a leaf-and-spine network architecture.

n Physical Storage Design

This VMware Validated Design relies on both vSAN storage and NFS storage. The "Shared StorageDesign" section explains where the SDDC uses which type of storage and gives backgroundinformation. The focus of this section is physical storage design.

Physical Design FundamentalsPhysical design fundamentals include decisions on availability zones and regions and on pod types,pods, and racks. The ESXi host physical design is also part of the design fundamentals.


VMware, Inc. 40

Availability Zones and RegionsAvailability zones and regions are used for different purposes.

Availability zones An availability zone is the fault domain of the SDDC. Multiple availabilityzone scan provide continuous availability of an SDDC, minimizeunavailability of services and improve SLAs.

Regions Regions provide disaster recovery across different SDDC instances. Thisdesign uses two regions. Each region is a separate SDDC instance. Theregions have a similar physical layer design and virtual infrastructuredesign but different naming. For information on exceptions to this design,see the Business Continuity / Disaster Recovery Design chapter.

Note This design leverages a single availability zone for a one region deployment, and a singleavailability zone in each region in the case of a two region deployment.

The design uses the following regions. The region identifier uses United Nations Code for Trade andTransport Locations(UN/LOCODE) along with a numeric instance ID.

Region Region Identifier Region-specific Domain Name Region Description

A SFO01 sfo01.rainpole.local San Francisco, CA, USA based data center

B LAX01 lax01.rainpole.local Los Angeles, CA, USA based data center

Note Region Identifiers will vary based on the locations used in your deployment.

Table 2‑1. Availability Zones and Regions Design Decisions

Decision ID Design Decision Design Justification Design Implication

SDDC-PHY-001 Per region, a singleavailability zone that cansupport all SDDCmanagement componentsis deployed.

A single availability zone can support allSDDC management and computecomponents for a region. You can later addanother availability zone to extend andscale the management and computecapabilities of the SDDC.

Results in limited redundancy of theoverall solution. The singleavailability zone can become asingle point of failure and preventhigh-availability design solutions.

SDDC-PHY-002 Use two regions. Supports the technical requirement ofmulti-region failover capability as outlinedin the design objectives.

Having multiple regions will requirean increased solution footprint andassociated costs.

Pods and RacksThe SDDC functionality is split across multiple pods. Each pod can occupy one rack or multiple racks.The total number of pods for each pod type depends on scalability needs.


VMware, Inc. 41

Figure 2‑2. SDDC Pod Architecture

ToR Switch

ToR Switch

ToR Switch

ToR Switch




External connection

ToR Switch

ToR Switch

Table 2‑2. Required Number of Racks

Pod (Function)

Required Numberof Racks (for fullscale deployment)

MinimumNumber ofRacks Comment

Management podand shared edge andcompute pod

1 1 Two half-racks are sufficient for the management pod andshared edge and compute pod. As the number and resourceusage of compute VMs increase adding additional hosts to thecluster will be required, as such extra space in the rack shouldbe reserved for growth.

Compute pods 6 0 With 6 compute racks, 6 compute pods with 19 ESXi hostseach can achieve the target size of 6000 average-sized VMs.If an average size VM has two vCPUs with 4 GB of RAM,6000 VMs with 20% overhead for bursting workloads require114 hosts.

The quantity and performance varies based on the workloadsrunning within the compute pods.

Storage pods 6 0 (if using vSANfor ComputePods)

Storage that is not vSAN storage is hosted on isolated storagepods.

Total 13 1


VMware, Inc. 42

Table 2‑3. POD and Racks Design Decisions


SDDC-PHY-003

The management andthe shared edge andcompute pod occupy thesame rack.

The number of required compute resources for themanagement pod (4 ESXi servers) and shared edgeand compute pod (4 ESXi servers) are low and do notjustify a dedicated rack for each pod.

On-ramp and off-ramp connectivity to physicalnetworks (for example, north-south L3 routing on NSXEdge virtual appliances) can be supplied to both themanagement and compute pods through thismanagement/edge rack.

Edge resources require external connectivity tophysical network devices. Placing edge resources formanagement and compute in the same rack willminimize VLAN spread.

The design must includesufficient power and cooling tooperate the server equipment.This depends on the selectedvendor and products.

If the equipment in this entirerack fails, a second region isneeded to mitigate downtimeassociated with such an event.

SDDC-PHY-004

Storage pods canoccupy one or moreracks.

To simplify the scale out of the SDDC infrastructure,the storage pod to rack(s) relationship has beenstandardized.

It is possible that the storage system arrives from themanufacturer in dedicated rack or set of racks and astorage system of this type is accommodated for in thedesign.

The design must includesufficient power and cooling tooperate the storage equipment.This depends on the selectedvendor and products.

SDDC-PHY-005

Each rack has twoseparate power feeds.

Redundant power feeds increase availability byensuring that failure of a power feed does not bringdown all equipment in a rack.

Combined with redundant network connections into arack and within a rack, redundant power feeds preventfailure of equipment in an entire rack.

All equipment used mustsupport two separate powerfeeds. The equipment mustkeep running if one power feedfails.

If the equipment of an entirerack fails, the cause, such asflooding or an earthquake, alsoaffects neighboring racks. Asecond region is needed tomitigate downtime associatedwith such an event.

SDDC-PHY-006

Mount the computeresources (minimum of4 ESXi servers) for themanagement podtogether in a rack.

Mounting the compute resources for the managementpod together can ease physical datacenter design,deployment and troubleshooting.

None.

SDDC-PHY-007

Mount the computeresources for the sharededge and compute pod(minimum of 4 ESXiservers) together in arack.

Mounting the compute resources for the shared edgeand compute pod together can ease physicaldatacenter design, deployment and troubleshooting.

None.

ESXi Host Physical Design SpecificationsThe physical design specifications of the ESXi host list the characteristics of the hosts that were usedduring deployment and testing of this VMware Validated Design.


VMware, Inc. 43

Physical Design Specification Fundamentals

The configuration and assembly process for each system is standardized, with all components installedthe same manner on each host. Standardizing the entire physical configuration of the ESXi hosts iscritical to providing an easily manageable and supportable infrastructure because standardizationeliminates variability. Consistent PCI card slot location, especially for network controllers, is essential foraccurate alignment of physical to virtual I/O resources. Deploy ESXi hosts with identical configuration,including identical storage, and networking configurations, across all cluster members. Identicalconfigurations ensure an even balance of virtual machine storage components across storage andcompute resources.

Select all ESXi host hardware, including CPUs following the VMware Compatibility Guide.

The sizing of the physical servers for the ESXi hosts for the management and edge pods has specialconsideration because it is based on the VMware document VMware Virtual SAN Ready Nodes, as thesepod type use VMware vSAN.

n An average sized VM has two vCPUs with 4 GB of RAM.

n A standard 2U server can host 60 average-sized VMs on a single ESXi host.

Table 2‑4. ESXi Host Design Decisions


SDDC-PHY-008 Use vSAN Ready Nodes. Using a vSAN Ready Node ensuresseamless compatibility with vSAN duringthe deployment.

Might limit hardware choices.

SDDC-PHY-009 All nodes must have uniformconfigurations across agiven cluster.

A balanced cluster delivers morepredictable performance even duringhardware failures. In addition, performanceimpact during resync/rebuild is minimalwhen the cluster is balanced.

Vendor sourcing, budgeting andprocurement considerations foruniform server nodes will beapplied on a per cluster basis.

ESXi Host Memory

The amount of memory required for compute pods will vary depending on the workloads running in thepod. When sizing memory for compute pod hosts it's important to remember the admission control setting(n+1) which reserves one hosts resources for fail over or maintenance.

Note See the VMware vSAN 6.5 Design and Sizing Guide for more information about disk groups,including design and sizing guidance. The number of disk groups and disks that an ESXi host managesdetermines memory requirements. 32 GB of RAM is required to support the maximum number of diskgroups.

Table 2‑5. Host Memory Design Decision


SDDC-PHY-010 Set up each ESXi host in the managementpod to have a minimum 192 GB RAM.

The management and edge VMs in thispod require a total 424 GB RAM.

None


VMware, Inc. 44

https://www.vmware.com/resources/compatibility/pdf/vi_vsan_rn_guide.pdf

Host Boot Device Background Considerations

Minimum boot disk size for ESXi in SCSI-based devices (SAS / SATA / SAN ) is greater than 5 GB. ESXican be deployed using stateful local SAN SCSI boot devices, or by using vSphere Auto Deploy.

What is supported depends on the version of vSAN that your are using:

n vSAN does not support stateless vSphere Auto Deploy

n vSAN 5.5 and greater supports USB/SD embedded devices for ESXi boot device (4 GB or greater).

n Since vSAN 6.0, there is an option to use SATADOM as a supported boot device.

See the VMware vSAN 6.5 Design and Sizing Guide to choose the option that best fits your hardware.

Physical Networking DesignThe physical network uses a leaf-and-spine network architecture.

Figure 2‑3 illustrates a leaf-and-spine network architecture. For additional information, see PhysicalNetwork Architecture.

Figure 2‑3. Leaf-and-Spine Architecture

Leaf Leaf Leaf Leaf Leaf Leaf

Spine Spine Spine

Leaf-and-Spine and Network Virtualization ArchitectureAs virtualization, cloud computing, and distributed cloud become more pervasive in the data center, a shiftin the traditional three-tier networking model is taking place. This shift addresses simplicity and scalability.

Simplicity

The traditional core-aggregate-access model is efficient for north/south traffic that travels in and out of thedata center. This model is usually built for redundancy and resiliency against failure. However, theSpanning Tree Protocol (STP) typically blocks 50 percent of the critical network links to prevent networkloops, which means 50 percent of the maximum bandwidth is wasted until something fails.


VMware, Inc. 45

A core-aggregate-access architecture is still widely used for service-oriented traffic that travelsnorth/south. However, the trends in traffic patterns are changing with the types of workloads. In today’sdata centers east/west or server-to-server traffic is common. If the servers in a cluster are performing aresource-intensive calculation in parallel, unpredictable latency or lack of bandwidth are undesirable.Powerful servers that perform these calculations can attempt to communicate with each other, but if theycannot communicate efficiently because of a bottleneck in the network architecture, wasted capitalexpenditure results.

One way to solve the problem is to create a leaf-and-spine architecture, also known as a distributed core.

A leaf-and-spine architecture has two main components: spine switches and leaf switches.

n Spine switches can be thought of as the core, but instead of being a large, chassis-based switchingplatform, the spine consists of many high-throughput Layer 3 switches with high port density.

n Leaf switches can be treated as the access layer. Leaf switches provide network connection points forservers and uplink to the spine switches.

Every leaf switch connects to every spine switch in the fabric. No matter which leaf switch a server isconnected to, it always has to cross the same number of devices to get to another server (unless theother server is located on the same leaf). This design keeps the latency down to a predictable levelbecause a payload has to hop only to a spine switch and another leaf switch to get to its destination.

Instead of relying on one or two large chassis-based switches at the core, the load is distributed across allspine switches, making each individual spine insignificant as the environment scales out.

Scalability

Several factors, including the following, affect scalability.

n Number of racks that are supported in a fabric.

n Amount of bandwidth between any two racks in a data center.

n Number of paths a leaf switch can select from when communicating with another rack.

The total number of available ports dictates the number of racks supported in a fabric across all spineswitches and the acceptable level of oversubscription.

Different racks might be hosting different types of infrastructure. For example, a rack might host filers orother storage systems, which might attract or source more traffic than other racks in a data center. Inaddition, traffic levels of compute racks (that is, racks that are hosting hypervisors with workloads orvirtual machines) might have different bandwidth requirements than edge racks, which provideconnectivity to the outside world. Link speed as well as the number of links vary to satisfy differentbandwidth demands.

The number of links to the spine switches dictates how many paths are available for traffic from this rackto another rack. Because the number of hops between any two racks is consistent, equal-costmultipathing (ECMP) can be used. Assuming traffic sourced by the servers carry a TCP or UDP header,traffic distribution can occur on a per-flow basis.


VMware, Inc. 46

Figure 2‑4. Leaf-and-Spine and Network VirtualizationSpineSwitch

SpineSwitch

SpineSwitch

ToR Switch

ToR Switch

ToR Switch

ToR Switch




External connection

ToR Switch

ToR Switch

L2

L3

Switch Types and Network ConnectivitySetup of the physical environment requires careful consideration. Follow best practices for physicalswitches, leaf switch connectivity, VLANs and subnets, and access port settings.

Top of Rack Physical Switches

When configuring Top of Rack (ToR) switches, consider the following best practices.

n Configure redundant physical switches to enhance availability.

n Configure switch ports that connect to ESXi hosts manually as trunk ports. Virtual switches arepassive devices and do not send or receive trunking protocols, such as Dynamic Trunking Protocol(DTP).

n Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce thetime it takes to transition ports over to the forwarding state, for example using the Trunk PortFastfeature found in a Cisco physical switch.

n Provide DHCP or DHCP Helper capabilities on all VLANs that are used by Management and VXLANVMkernel ports. This setup simplifies the configuration by using DHCP to assign IP address based onthe IP subnet in use.

n Configure jumbo frames on all switch ports, inter-switch link (ISL) and switched virtual interfaces(SVI's).


VMware, Inc. 47

Leaf Switch Connectivity and Network Settings

Each ESXi host in the compute rack is connected redundantly to the SDDC network fabric ToR switchesby means of two 10 GbE ports, as shown in Figure 2‑5. Configure the ToR switches to provide allnecessary VLANs via an 802.1Q trunk.

Figure 2‑5. Leaf Switch to Server Connection within Compute Racks

Leaf

L3

Server

Spine Spine

Leaf

L210 GigE10 GigE

Each ESXi host in the management/shared edge and compute rack is connected to the SDDC networkfabric and also to the Wide Area Network (WAN) and to the Internet, as shown in Figure 2‑6.


VMware, Inc. 48

Figure 2‑6. Leaf Switch to Server Connection within Management/Shared Compute andEdge Rack

Leaf

L3

Server

Spine SpineWAN/MPLS

InternetWAN/MPLS

Internet

Leaf

L2

10 GigE10 GigE

VLANs and Subnets

Each ESXi host in the compute rack and the management/edge rack uses VLANs and correspondingsubnets for internal-only traffic, as shown in Figure 2‑7.

The leaf switches of each rack act as the Layer 3 interface for the corresponding subnet.

The management/edge rack provides externally accessible VLANs for access to the Internet and/or MPL-based corporate networks.


VMware, Inc. 49

Figure 2‑7. Sample VLANs and Subnets within a PodSp

an o

f VLA

Ns

Layer 3 ToR Switch

Layer 2

Layer 3

VLAN 1612 VLAN 1614 VLAN 1613VLAN 1611

VLAN Trunk (802.1Q)

vMotion VXLAN StorageMgmt172.16.11.0/24

DGW:172.16.11.253

172.16.12.0/24DGW:

172.16.12.253

172.16.13.0/24DGW:

172.16.13.253

172.16.14.0/24DGW:

172.16.14.253

Span

of V

LAN

s

vSphere Host (ESXi)

Routed uplinks (ECMP)

Follow these guidelines.

n Use only /24 subnets to reduce confusion and mistakes when dealing with IPv4 subnetting.

n Use the IP address .1 as the (floating) interface with .2 and .3 for Virtual Router Redundancy Protocol(VRPP) or Hot Standby Routing Protocol (HSRP).

n Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region andanother octet by function. For example, the mapping 172.regionid.function.0/24 results in thefollowing sample subnets.

Note The following VLANs and IP ranges are meant as samples. Your actual implementation dependson your environment.

Table 2‑6. Sample Values for VLANs and IP Ranges

Pod Function Sample VLAN Sample IP range

Management Management 1611 (Native) 172.16.11.0/24

Management vMotion 1612 172.16.12.0/24

Management VXLAN 1614 172.16.14.0/24

Management vSAN 1613 172.16.13.0/24

Shared Edge and Compute Management 1631 (Native) 172.16.31.0/24

Shared Edge and Compute vMotion 1632 172.16.32.0/24

Shared Edge and Compute VXLAN 1634 172.16.34.0/24

Shared Edge and Compute vSAN 1633 172.16.33.0/24


VMware, Inc. 50

Access Port Network Settings

Configure additional network settings on the access ports that connect the leaf switch to thecorresponding servers.

Spanning-Tree Protocol(STP)

Although this design does not use the spanning tree protocol, switchesusually come with STP configured by default. Designate the access portsas trunk PortFast.

Trunking Configure the VLANs as members of a 802.1Q trunk with the managementVLAN acting as the native VLAN.

MTU Set MTU for all VLANS and SVIs (Management, vMotion, VXLAN andStorage) to jumbo frames for consistency purposes.

DHCP helper Configure the VIF of the Management, vMotion and VXLAN subnet as aDHCP proxy.

Multicast Configure IGMP snooping on the ToR switches and include an IGMPquerier on each VLAN.

Region Interconnectivity

The SDDC management networks, VXLAN kernel ports and the edge and compute VXLAN kernelports of the two regions must be connected. These connections can be over a VPN tunnel, Point to Pointcircuits, MPLS, etc. End users must be able to reach the public-facing network segments (publicmanagement and tenant networks) of both regions.

The region interconnectivity design must support jumbo frames, and ensure latency is less then 150 ms.For more details on the requirements for region interconnectivity see the Cross-VC NSX Design Guide.

The design of a region connection solution is out of scope for this VMware Validated Design.

Physical Network Design DecisionsThe physical network design decisions govern the physical layout and use of VLANs. They also includedecisions on jumbo frames and on some other network-related requirements such as DNS and NTP.

Physical Network Design Decisions

The design uses 4 spine switches with 40 GbE ports. As a result, each leaf switch must have 4 uplinkports capable of 40 GbE.

The resulting environment supports fault tolerance and compensates for oversubscription, as follows.

Fault Tolerance In case of a switch failure or scheduled maintenance, switch fabric capacityreduction is 25% with four spine switches.

Oversubscription Oversubscription can occur within a leaf switch. To compute theoversubscription for a leaf switch, use this formula.


VMware, Inc. 51

Total bandwidth available to all connected servers /

aggregate amount of uplink bandwidth

The compute rack and the management/edge rack have 19 ESXi hosts.Each ESXi host has one 10 GbE port connected to each ToR switch,creating up to 190 Gbps of bandwidth. With four 40 GbE uplinks to thespine, you can compute oversubscription as follows (see Figure 2‑8).

190 Gbps (total bandwidth) / 160 Gbps (uplink bandwidth) = 1.2:1

Figure 2‑8. Oversubscription in the Leaf Switches

Leaf Leaf

Spine Spine Spine

19x10 GbE

Leaf

No Oversubscription

4x40 GbE

1.2 : 1

19x10 GbE

4x40 GbE

No Oversubscription

Routing protocols Base the selection of the external routing protocol on your currentimplementation or on available expertise among the IT staff. Takeperformance requirements into consideration. Possible options are OSPF,BGP and IS-IS.

DHCP proxy The DHCP proxy must point to a DHCP server by way of its IPv4 address.See the Planning and Preparation documentation for details on the DHCPserver.


VMware, Inc. 52

Table 2‑7. Physical Network Design Decisions


SDDC-PHY-NET-001

Racks are connected using aleaf-and-spine topology andLayer 3 connectivity.

A Layer 3 leaf-and-spine architecturesupports scale out while maintainingfailure isolation.

Layer 2 traffic is reduced towithin the pod.

SDDC-PHY-NET-002

Only the management andshared edge and compute hostshave physical access to theexternal network by way ofVLANs.

Aggregating physical cabling andnetwork services to the management andshared edge and compute rack reducescosts.

Workloads in compute podslocated in compute racks haveto use network virtualization(NSX for vSphere) for externalnetwork connectivity.

SDDC-PHY-NET-003

Each rack uses two ToRswitches. These switches provideconnectivity across two 10 GbElinks to each server.

This design uses two 10 GbE links toprovide redundancy and reduce overalldesign complexity.

Requires two ToR switches perrack which can increase costs.

SDDC-PHY-NET-004

Use VLANs to segment physicalnetwork functions.

Allow for Physical network connectivitywithout requiring large number of NICs.

Segregation is needed for the differentnetwork functions that are required in theSDDC. This segregation allows fordifferentiated services and prioritizationof traffic as needed.

Uniform configuration andpresentation is required on allthe trunks made available to theESXi hosts.

Additional Design Decisions

Additional design decisions deal with static IP addresses, DNS records, and the required NTP timesource.

Table 2‑8. Additional Network Design Decisions


SDDC-PHY-NET-005

Assign Static IP addresses to allmanagement nodes of the SDDCinfrastructure.

Configuration of static IP addresses avoidconnection outages due to DHCPavailability or misconfiguration.

Accurate IP addressmanagement must bein place.

SDDC-PHY-NET-006

Create DNS records for allmanagement nodes to enableforward, reverse, short and FQDNresolution.

Ensures consistent resolution ofmanagement nodes using both IP address(reverse lookup) and name resolution.

None

SDDC-PHY-NET-007

Use an NTP time source for allmanagement nodes.

Critical to maintain accurate andsynchronized time between managementnodes.

None

Jumbo Frames Design Decisions

IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-framepayload from 1500 bytes to the jumbo frame setting increases the efficiency of data transfer. Jumboframes must be configured end-to-end, which is easily accomplished in a LAN. When you enable jumboframes on an ESXi host, you have to select an MTU that matches the MTU of the physical switch ports.


VMware, Inc. 53

The workload determines whether it makes sense to configure jumbo frames on a virtual machine. If theworkload consistently transfers large amounts of network data, configure jumbo frames if possible. In thatcase, the virtual machine operating systems and the virtual machine NICs must also support jumboframes.

Using jumbo frames also improves performance of vSphere vMotion.

Note VXLANs need an MTU value of at least 1600 bytes on the switches and routers that carry thetransport zone traffic.

Table 2‑9. Jumbo Frames Design Decisions


SDDC-PHY-NET-008

Configure the MTU size to 9000bytes (Jumbo Frames) on theportgroups that support thefollowing traffic types. n NFSn vSANn vMotionn VXLANn vSphere Replication

Setting the MTU to 9000 bytes (JumboFrames) improves traffic throughput.

In order to support VXLAN the MTU settingmust be increased to a minimum of 1600bytes, setting this portgroup also to 9000bytes has no affect on VXLAN but ensuresconsistency across portgroups that areadjusted from the default MTU size.

When adjusting the MTUpacket size, the entire networkpath (VMkernel port,distributed switch, physicalswitches and routers) mustalso be configured to supportthe same MTU packet size.

Physical Storage DesignThis VMware Validated Design relies on both vSAN storage and NFS storage. The "Shared StorageDesign" section explains where the SDDC uses which type of storage and gives background information.The focus of this section is physical storage design.

vSAN Physical DesignSoftware-defined storage is a key technology in the SDDC. This design uses VMware Virtual SAN (vSAN)to implement software-defined storage for the management clusters.

vSAN is a fully integrated hypervisor-converged storage software. vSAN creates a cluster of server harddisk drives and solid state drives, and presents a flash-optimized, highly resilient, shared storagedatastore to hosts and virtual machines. vSAN allows you to control capacity, performance, andavailability on a per virtual machine basis through the use of storage policies.

Requirements and Dependencies

The software-defined storage module has the following requirements and options.

n Minimum of 3 hosts providing storage resources to the vSAN cluster.

n vSAN is configured as hybrid storage or all-flash storage.

n A vSAN hybrid storage configuration requires both magnetic devices and flash caching devices.

n An All-Flash vSAN configuration requires vSphere 6.0 or later.


VMware, Inc. 54

n Each ESXi host that provides storage resources to the cluster must meet the following requirements.

n Minimum of one SSD. The SSD flash cache tier should be at least 10% of the size of the HDDcapacity tier.

n Minimum of two HHDs.

n RAID controller compatible with vSAN.

n 10 Gbps network for vSAN traffic \ with Multicast enabled.

n vSphere High Availability Isolation Response set to power off virtual machines. With this setting,no possibility of split brain conditions in case of isolation or network partition exists. In a split-braincondition, the virtual machine might be powered on by two hosts by mistake. See design decision SDDC-VI-VC-012 for more details.

Table 2‑10. vSAN Physical Storage Design Decision


SDDC-PHY-STO-001

Use one or more 200 GB or greaterSSD and two or more traditional 1 TBor greater HDDs to create at least asingle disk group in the managementcluster.

Using a 200 GB SSD and two 1TB HDD's allows enough capacityfor the management VMs with aminimum of 10% flash-basedcaching.

When using only a single diskgroup you limit the amount ofstriping (performance) capabilityand increase the size of the faultdomain.

Hybrid Mode and All-Flash Mode

vSphere offers two different vSAN modes of operation, all-flash or hybrid.

Hybrid Mode In a hybrid storage architecture, vSAN pools server-attached capacitydevices (in this case magnetic devices) and caching devices, typicallySSDs or PCI-e devices to create a distributed shared datastore.

All-Flash Mode vSAN can be deployed as all-flash storage. All-flash storage uses flash-based devices (SSD or PCI-e) only as a write cache while other flash-based devices provide high endurance for capacity and data persistence.

Table 2‑11. vSAN Mode Design Decision


SDDC-PHY-STO-002

Configure vSAN in hybridmode in the managementcluster.

The VMs in the management cluster,which are hosted within vSAN, do notrequire the performance or expense ofan all-flash vSAN configuration.

vSAN hybrid mode does not providethe potential performance oradditional capabilities such asdeduplication of an all-flashconfiguration.


VMware, Inc. 55

Hardware ConsiderationsYou can build your own VMware vSAN cluster or choose from a list of vSAN Ready Nodes.

Build Your Own Be sure to use hardware from the VMware Compatibly Guide for thefollowing vSAN components:

n Solid state disks (SSDs)

n Magnetic hard drives (HDDs)

n I/O controllers, including vSAN certified driver/firmware combinations

Use VMware vSANReady Nodes

A vSAN Ready Node is a validated server configuration in a tested, certifiedhardware form factor for vSAN deployment, jointly recommended by theserver OEM and VMware. See the VMware Compatibility Guide. The vSANReady Node documentation provides examples of standardizedconfigurations, including the numbers of VMs supported and estimatednumber of 4K IOPS delivered.

As per design decision SDDC-PHY-009, the VMware Validated Design uses vSAN Ready Nodes.

Solid State Disk (SSD) Characteristics

In a VMware vSAN configuration, the SSDs are used for the vSAN caching layer for hybrid deploymentsand for the capacity layer for all flash.

n For a hybrid deployment, the use of the SSD is split between a non-volatile write cache(approximately 30%) and a read buffer (approximately 70%). As a result, the endurance and thenumber of I/O operations per second that the SSD can sustain are important performance factors.

n For an all-flash model, endurance and performance have the same criteria. However many morewrite operations are held by the caching tier, thus elongating or extending the life of the SSDcapacity-tier.

SSD Endurance

This VMware Validated Design uses class D endurance class SSDs for the caching tier.

SDDC Endurance Design Decision Background

For endurance of the SSDs used for vSAN, standard industry write metrics are the primarymeasurements used to gauge the reliability of the drive. No standard metric exists across all vendors,however, Drive Writes per Day (DWPD) or Petabytes Written (PBW) are the measurements normallyused.

For vSphere 5.5, the endurance class was based on Drive Writes Per Day (DWPD). For VMware vSAN6.0 and later, the endurance class has been updated to use Terabytes Written (TBW), based on thevendor’s drive warranty. TBW can be used for VMware vSAN 5.5, VMware vSAN 6.0 and VMware vSAN6.5 and is reflected in the VMware Compatibility Guide.


VMware, Inc. 56

https://www.vmware.com/resources/compatibility/search.php?deviceCategory=Virtual SAN

https://www.vmware.com/resources/compatibility/pdf/vi_vsan_rn_guide.pdf

The reasoning behind using TBW is that VMware provides the flexibility to use larger capacity drives withlower DWPD specifications.

If a SSD vendor uses Drive Writes Per Day as a measurement, you can calculate endurance in TerabytesWritten (TBW) with the following equation.

TBW (over 5 years) = Drive Size x DWPD x 365 x 5

For example, if a vendor specified DWPD = 10 for a 800 GB capacity SSD, you can compute TBW withthe following equation.

TBW = 0.4TB X 10DWPD X 365days X 5yrs

TBW = 7300TBW

That means the SSD supports 7300TB writes over 5 years (The higher the TBW number, the greater theendurance class.).

For SSDs that are designated for caching and all-flash capacity layers, the following table outlines whichendurance class to use for hybrid and for all-flash VMware vSAN.

Endurance Class TBW Hybrid Caching Tier All-Flash Caching Tier All-Flash Capacity Tier

Class A >=365 No No Yes

Class B >=1825 Yes No Yes

Class C >=3650 Yes Yes Yes

Class D >=7300 Yes Yes Yes

Note This VMware Validated Design does not use All-Flash vSAN.

Table 2‑12. SSD Endurance Class Design Decisions


SDDC-PHY-STO-003

Use Class D (>=7300TBW)SSDs for the caching tier of themanagement cluster.

If a SSD designated for the caching tier failsdue to wear-out, the entire VMware vSANdisk group becomes unavailable. The resultis potential data loss or operational impact.

SSDs with higher endurancemay be more expensivethan lower enduranceclasses.

SSD Performance

There is a direct correlation between the SSD performance class and the level of vSAN performance. Thehighest-performing hardware results in the best performance of the solution. Cost is therefore thedetermining factor. A lower class of hardware that is more cost effective might be attractive even if theperformance or size is not ideal.

For optimal performance of vSAN, select class E or greater SSDs. See the VMware Compatibility Guidefor detail on the different classes.


VMware, Inc. 57


SSD Performance Design Decision Background

Select a high class of SSD for optimal performance of VMware vSAN. Before selecting a drive size,consider disk groups and sizing as well as expected future growth. VMware defines classes ofperformance in the VMware Compatibility Guide as follows.

Table 2‑13. SSD Performance Classes

Performance Class Writes Per Second

Class A 2,500 – 5,000

Class B 5,000 – 10,000

Class C 10,000 – 20,000

Class D 20,000 – 30,000

Class E 30,000 – 100,000

Class F 100,000 +

Select an SSD size that is, at a minimum, 10% of the anticipated size of the consumed HDD storagecapacity, before failures to tolerate are considered. For example, select an SSD of at least 100 GB for 1TB of HDD storage consumed in a 2 TB disk group.

Caching Algorithm

Both hybrid clusters and all-flash configurations adhere to the recommendation that 10% of consumedcapacity for the flash cache layer. However, there are differences between the two configurations.

Hybrid vSAN 70% of the available cache is allocated for storing frequently read diskblocks, minimizing accesses to the slower magnetic disks. 30% of availablecache is allocated to writes.

All-Flash vSAN All-flash clusters have two types of flash: very fast and durable write cache,and cost-effective capacity flash. Here cache is 100% allocated for writes,as read performance from capacity flash is more than sufficient.

Use Class E SSDs or greater for the highest possible level of performance from the VMware vSANvolume.

Table 2‑14. SSD Performance Class Selection

Design Quality Option 1 Class E Option 2 Class C Comments

Availability o o Neither design option impacts availability.

Manageability o o Neither design option impacts manageability.

Performance ↑ ↓ The higher the storage class that is used, the better the performance.

Recover-ability o o Neither design option impacts recoverability.

Security o o Neither design option impacts security.

Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.


VMware, Inc. 58


Table 2‑15. SSD Performance Class Design Decisions


SDDC-PHY-STO-004

Use Class E SSDs(30,000-100,000 writes persecond) for the managementcluster.

The storage I/O performancerequirements within the Managementcluster dictate the need for at leastClass E SSDs.

Class E SSDs might be moreexpensive than lower classdrives.

Magnetic Hard Disk Drives (HDD) Characteristics

The HDDs in a VMware vSAN environment have two different purposes, capacity and object stripe width.

Capacity Magnetic disks, or HDDs, unlike caching-tier SSDs, make up the capacityof a vSAN datastore

Stripe Width You can define stripe width at the virtual machine policy layer. vSAN mightuse additional stripes when making capacity and placement decisionsoutside a storage policy.

vSAN supports these disk types:

n Serial Attached SCSI (SAS)

n Near Line Serial Attached SCSI (NL-SCSI). NL-SAS can be thought of as enterprise SATA drives butwith a SAS interface.

n Serial Advanced Technology Attachment (SATA). Use SATA magnetic disks only in capacity-centricenvironments where performance is not prioritized.

SAS and NL-SAS get you the best results. This VMware Validated Design uses 10,000 RPM drives toachieve a balance between cost and availability.

HDD Capacity, Cost, and Availability Background Considerations

You can achieve the best results with SAS and NL-SAS.

The VMware vSAN design must consider the number of magnetic disks required for the capacity layer,and how well the capacity layer will perform.

n SATA disks typically provide more capacity per individual drive, and tend to be less expensive thanSAS drives. However the trade off is performance, because SATA performance is not as good asSAS performance due to lower rotational speeds (typically 7200RPM)

n Choose SAS magnetic disks instead of SATA magnetic disks in environments where performance iscritical.

Consider that failure of a larger capacity drive has operational impact on the availability and recoveryof more components.

Rotational Speed (RPM) Background Considerations

HDDs tend to be more reliable, but that comes at a cost. SAS disks can be available up to 15,000 RPMspeeds.


VMware, Inc. 59

Table 2‑16. vSAN HDD Environmental Characteristics

Characteristic Revolutions per Minute (RPM)

Capacity 7,200

Performance 10,000

Additional Performance 15,000

Cache-friendly workloads are less sensitive to disk performance characteristics; however, workloads canchange over time. HDDs with 10,000 RPM are the accepted norm when selecting a capacity tier.

For the software-defined storage module, VMware recommends that you use an HDD configuration thatis suited to the characteristics of the environment. If there are no specific requirements, selecting 10,000RPM drives achieves a balance between cost and availability.

Table 2‑17. HDD Selection Design Decisions


SDDC-PHY-STO-005

Use 10,000 RPM HDDsfor the managementcluster.

10,000 RPM HDDs achieve a balance betweenperformance and availability for the VMware vSANconfiguration.

The performance of 10,000 RPM HDDs avoids diskdrain issues. In vSAN hybrid mode, the vSANperiodically flushes uncommitted writes to thecapacity tier.

Slower and potentiallycheaper HDDs are notavailable.

I/O ControllersThe I/O controllers are as important to a VMware vSAN configuration as the selection of disk drives.vSAN supports SAS, SATA, and SCSI adapters in either pass-through or RAID 0 mode. vSAN supportsmultiple controllers per host.

n Multiple controllers can improve performance and mitigate a controller or SSD failure to a smallernumber of drives or vSAN disk groups.

n With a single controller, all disks are controlled by one device. A controller failure impacts all storage,including the boot media (if configured).

Controller queue depth is possibly the most important aspect for performance. All I/O controllers inthe VMware vSAN Hardware Compatibility Guide have a minimum queue depth of 256. Consider normalday-to-day operations and increase of I/O due to Virtual Machine deployment operations or re-sync I/Oactivity as a result of automatic or manual fault remediation.


VMware, Inc. 60

About SAS Expanders

SAS expanders are a storage technology that lets you maximize the storage capability of your SAScontroller card. Like switches of an ethernet network, SAS expanders enable you to connect a largernumber of devices, that is, more SAS/SATA devices to a single SAS controller. Many SAS controllerssupport up to 128 or more hard drives.

Caution VMware has not extensively tested SAS expanders, as a result performance and operationalpredictability are relatively unknown at this point. For this reason, you should avoid configurations withSAS expanders.

NFS Physical Storage DesignNetwork File System (NFS) is a distributed file system protocol that allows a user on a client computer toaccess files over a network much like local storage is accessed. In this case the client computer is anESXi host, and the storage is provided by a NFS-capable external storage array.

The management cluster uses VMware vSAN for primary storage and NFS for secondary storage. Thecompute clusters are not restricted to any particular storage technology. For compute clusters, thedecision on which technology to use is based on the performance, capacity, and capabilities (replication,deduplication, compression, etc.) required by the workloads that are running in the clusters.

Table 2‑18. NFS Usage Design Decisions


SDDC-PHY-STO-006

NFS storage is presented to provide thefollowing features.n A datastore for backup datan An export for archive datan A datastore for templates and ISOs

Separate primary virtual machinestorage from backup data in case ofprimary storage failure.

vRealize Log Insight archivingrequires a NFS export.

An NFS capableexternal array isrequired.

Requirements

Your environment must meet the following requirements to use NFS storage in the VMware ValidatedDesign.

n Storage arrays are connected directly to the leaf switches.

n All connections are made using 10 Gb Ethernet.

n Jumbo Frames are enabled.

n 10K SAS (or faster) drives are used in the storage array.

Different disk speeds and disk types can be combined in an array to create different performance andcapacity tiers. The management cluster uses 10K SAS drives in the RAID configuration recommended bythe array vendor to achieve the required capacity and performance.


VMware, Inc. 61

Table 2‑19. NFS Hardware Design Decision


SDDC-PHY-STO-007

Use 10K SASdrives for NFSvolumes.

10K SAS drives achieve a balance between performance andcapacity. Faster drives can be used if desired.

vSphere Data Protection requires high-performance datastoresin order to meet backup SLAs.

vRealize Automation uses NFS datastores for its contentcatalog which requires high-performance datastores.

vRealize Log Insight uses NFS datastores for its archivestorage which, depending on compliance regulations, can use alarge amount of disk space.

10K SAS drives aregenerally moreexpensive than otheralternatives.

Volumes

A volume consists of multiple disks in a storage array that RAID is applied to.

Multiple datastores can be created on a single volume, but for applications that do not have a high I/Ofootprint a single volume with multiple datastores is sufficient.

n For high I/O applications, such as backup applications, use a dedicated volume to avoid performanceissues.

n For other applications, set up Storage I/O Control (SIOC) to impose limits on high I/O applications sothat other applications get the I/O they are requesting.

Table 2‑20. Volume Assignment Design Decisions


SDDC-PHY-STO-008

Use a dedicated NFSvolume to support backuprequirements.

The backup and restore process is I/Ointensive. Using a dedicated NFSvolume ensures that the process doesnot impact the performance of othermanagement components.

Dedicated volumes add managementoverhead to storage administrators.Dedicated volumes might use moredisks, depending on the array andtype of RAID.

SDDC-PHY-STO-009

Use a shared volume forother managementcomponent datastores.

Non-backup related managementapplications can share a commonvolume due to the lower I/O profile ofthese applications.

Enough storage space for sharedvolumes and their associatedapplication data must be available.

Virtual Infrastructure DesignThe virtual infrastructure design includes the software components that make up the virtual infrastructurelayer and that support the business continuity of the SDDC.

These components include the software products that provide the virtualization platform hypervisor,virtualization management, storage virtualization, network virtualization, backup and disaster recovery.VMware products in this layer include VMware vSphere, VMware vSAN, VMware NSX, vSphere DataProtection, and VMware Site Recovery Manager.


VMware, Inc. 62

Figure 2‑9. Virtual Infrastructure Layer in the SDDC

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

Virtual Infrastructure Design OverviewThe SDDC virtual infrastructure consists of two regions. Each region includes a management pod, and ashared edge and compute pod.


VMware, Inc. 63

Figure 2‑10. SDDC Logical Design

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OS

APP

OS

APP

OSAPP

OS

ESXi ESXi

APP

OSAPP

OS

APP

OSAPP

OS


NSXController

(Mgmt)


NSXEdge

(Mgmt)

NSXManager(Mgmt)

NSXManager

(Compute)

NSXEdge

(Compute)



SDDCPayload






Management Cluster






vCenterServer(Mgmt)

vCenterServer

(Compute)

Management PodManagement pods run the virtual machines that manage the SDDC. These virtual machines host vCenterServer, NSX Manager, NSX Controller, vRealize Operations, vRealize Log Insight, vRealize Automation,Site Recovery Manager and other shared management components. All management, monitoring, andinfrastructure services are provisioned to a vSphere cluster which provides high availability for thesecritical services. Permissions on the management cluster limit access to only administrators. This protectsthe virtual machines running the management, monitoring, and infrastructure services.


VMware, Inc. 64

Shared Edge and Compute PodThe virtual infrastructure design uses a shared edge and compute pod. The shared pod combines thecharacteristics of typical edge and compute pods into a single pod. It is possible to separate these in thefuture if required.

This pod provides the following main functions:

n Supports on-ramp and off-ramp connectivity to physical networks

n Connects with VLANs in the physical world

n Hosts the SDDC tenant virtual machines

The shared edge and compute pod connects the virtual networks (overlay networks) provided by NSX forvSphere and the external networks. An SDDC can mix different types of compute-only pods and provideseparate compute pools for different types of SLAs.

ESXi DesignThe ESXi design includes design decisions for boot options, user access, and the virtual machine swapconfiguration.

ESXi Hardware RequirementsYou can find the ESXi hardware requirements in Physical Design Fundamentals. The following designoutlines the design of the ESXi configuration.

ESXi Manual Install and Boot OptionsYou can install or boot ESXi 6.5 from the following storage systems:

SATA disk drives SATA disk drives connected behind supported SAS controllers or supportedon-board SATA controllers.

Serial-attached SCSI(SAS) disk drives

Supported for installing ESXi.

SAN Dedicated SAN disk on Fibre Channel or iSCSI.

USB devices Supported for installing ESXi. 16 GB or larger SD card is recommended.

FCoE (Software Fibre Channel over Ethernet)

ESXi can boot from a disk larger than 2 TB if the system firmware and the firmware on any add-in cardsupport it. See the vendor documentation.

ESXi Boot Disk and Scratch ConfigurationFor new installations of ESXi, the installer creates a 4 GB VFAT scratch partition. ESXi uses this scratchpartition to store log files persistently. By default, vm-support output, which is used by VMware totroubleshoot issues on the ESXi host, is also stored on the scratch partition.


VMware, Inc. 65

An ESXi installation on USB media does not configure a default scratch partition. VMware recommendsthat you specify a scratch partition on a shared datastore and configure remote syslog logging for thehost.

Table 2‑21. ESXi Boot Disk Design Decision


SDDC-VI-ESXi-001

Install and configure all ESXihosts to boot using a SDdevice of 16 GB or greater.

SD cards are an inexpensive and easy toconfigure option for installing ESXi.

Using SD cards allows allocation of all localHDDs to a VMware vSAN storage system.

When you use SD cardsESXi logs are not retainedlocally.

ESXi Host AccessAfter installation, ESXi hosts are added to a VMware vCenter Server system and managed through thatvCenter Server system.

Direct access to the host console is still available and most commonly used for troubleshooting purposes.You can access ESXi hosts directly using one of these three methods:

Direct Console UserInterface (DCUI)

Graphical interface on the console. Allows basic administrative controls andtroubleshooting options.

ESXi Shell A Linux-style bash login on the ESXi console itself.

Secure Shell (SSH)Access

Remote command-line console access.

You can enable or disable each method. By default the ESXi Shell and SSH are disabled to secure theESXi host. The DCUI is disabled only if Strict Lockdown Mode is enabled.

ESXi User AccessBy default, root is the only user who can log in to an ESXi host directly, however, you can add ESXi hoststo an Active Directory domain. After the host has been added to an Active Directory domain, access canbe granted through Active Directory groups. Auditing who has logged into the host also becomes easier.


VMware, Inc. 66

Table 2‑22. ESXi User Access Design Decisions


SDDC-VI-ESXi-002

Add each host to the Active Directorydomain for the region in which it willreside.

Using Active Directory membership allowsgreater flexibility in granting access toESXi hosts.

Ensuring that users log in with a uniqueuser account allows greater visibility forauditing.

Adding hosts to thedomain can add someadministrative overhead.

SDDC-VI-ESXi-003

Change the default ESX Adminsgroup to the SDDC-Admins ActiveDirectory group. Add ESXiadministrators to the SDDC-Adminsgroup following standard accessprocedures.

Having an SDDC-Admins group is moresecure because it removes a knownadministrative access point. In additiondifferent groups allow for separation ofmanagement tasks.

Additional changes to thehost's advanced settingsare required.

Virtual Machine Swap ConfigurationWhen a virtual machine is powered on, the system creates a VMkernel swap file to serve as a backingstore for the virtual machine's RAM contents. The default swap file is stored in the same location as thevirtual machine's configuration file. This simplifies the configuration, however it can cause an excess ofreplication traffic that is not needed.

You can reduce the amount of traffic that is replicated by changing the swap file location to a user-configured location on the host. However, it can take longer to perform VMware vSphere vMotion®

operations when the swap file has to be recreated.

Table 2‑23. Other ESXi Host Design Decisions


SDDC-VI-ESXi-004

Configure all ESXi hosts tosynchronize time with thecentral NTP servers.

Required because deployment ofvCenter Server Appliance on anESXi host might fail if the host is notusing NTP.

All firewalls located between theESXi host and the NTP servers haveto allow NTP traffic on the requirednetwork ports.

vCenter Server DesignThe vCenter Server design includes both the design for the vCenter Server instance and the VMwarePlatform Services Controller instance.

A Platform Services Controller groups a set of infrastructure services including vCenter Single Sign-On,License service, Lookup Service, and VMware Certificate Authority (VMCA). You can deploy the PlatformServices Controller and the associated vCenter Server system on the same virtual machine(embeddedPlatform Services Controller) or on different virtual machines (external Platform Services Controller).

n vCenter Server Deployment

The design decisions for vCenter Server deployment discuss the number of vCenter Server andPlatform Services Controller instances, the type of installation, and the topology.


VMware, Inc. 67

n vCenter Server Networking

As specified in the physical networking design, all vCenter Server systems must use static IPaddresses and host names. The IP addresses must have valid (internal) DNS registration includingreverse name resolution.

n vCenter Server Redundancy

Protecting the vCenter Server system is important because it is the central point of management andmonitoring for the SDDC. How you protect vCenter Server depends on maximum downtimetolerated, and on whether failover automation is required.

n vCenter Server Appliance Sizing

The following tables outline minimum hardware requirements for the management vCenter Serverappliance and the compute vCenter Server appliance.

n vSphere Cluster Design

The cluster design must take into account the workload that the cluster handles. Different clustertypes in this design have different characteristics.

n vCenter Server Customization

vCenter Server supports a rich set of customization options, including monitoring, virtual machinefault tolerance, and so on. For each feature, this VMware Validated Design specifies the designdecisions.

n Use of Transport Layer Security (TLS) Certificates

By default vSphere 6.5 uses TLS/SSL certificates that are signed by VMCA (VMware CertificateAuthority). By default, these certificates are not trusted by end-user devices or browsers. It is asecurity best practice to replace at least user-facing certificates with certificates that are signed by athird-party or enterprise Certificate Authority (CA). Certificates for machine-to-machinecommunication can remain as VMCA-signed certificates.

vCenter Server DeploymentThe design decisions for vCenter Server deployment discuss the number of vCenter Server and PlatformServices Controller instances, the type of installation, and the topology.


VMware, Inc. 68

Table 2‑24. vCenter Server Design Decision

Decision ID Design Decision Design JustificationDesignImplication

SDDC-VI-VC-001

Deploy two vCenter Serversystems in the first availabilityzone of each region.n One vCenter Server

supporting the SDDCmanagementcomponents.

n One vCenter Serversupporting the edgecomponents and computeworkloads.

Isolates vCenter Server failures to management or computeworkloads.

Isolates vCenter Server operations between management andcompute.

Supports a scalable cluster design where the managementcomponents may be re-used as additional compute needs tobe added to the SDDC.

Simplifies capacity planning for compute workloads byeliminating management workloads from consideration in theCompute vCenter Server.

Improves the ability to upgrade the vSphere environment andrelated components by providing for explicit separation ofmaintenance windows:n Management workloads remain available while workloads

in compute are being addressedn Compute workloads remain available while workloads in

management are being addressed

Ability to have clear separation of roles and responsibilitiesto ensure that only those administrators with properauthorization can attend to the management workloads.

Facilitates quicker troubleshooting and problem resolution.

Simplifies Disaster Recovery operations by supporting aclear demarcation between recovery of the managementcomponents and compute workloads.

Enables the use of two NSX managers, one for themanagement pod and the other for the shared edge andcompute pod. Network separation of the pods in the SDDCallows for isolation of potential network issues.

Requireslicenses for eachvCenter Serverinstance.

You can install vCenter Server as a Windows-based system or deploy the Linux-based VMware vCenterServer Appliance. The Linux-based vCenter Server Appliance is preconfigured, enables fast deployment,and potentially results in reduced Microsoft licensing costs.

Table 2‑25. vCenter Server Platform Design Decision


SDDC-VI-VC-002 Deploy all vCenter Serverinstances as Linux-based vCenterServer Appliances.

Allows for rapid deployment,enables scalability, and reducesMicrosoft licensing costs.

Operational staff might need Linuxexperience to troubleshoot theLinux-based appliances.


VMware, Inc. 69

Platform Services Controller Design Decision Background

vCenter Server supports installation with an embedded Platform Services Controller (embeddeddeployment) or with an external Platform Services Controller.

n In an embedded deployment, vCenter Server and the Platform Services Controller run on the samevirtual machine. Embedded deployments are recommended for standalone environments with onlyone vCenter Server system.

n Environments with an external Platform Services Controller can have multiple vCenter Serversystems. The vCenter Server systems can use the same Platform Services Controller services. Forexample, several vCenter Server systems can use the same instance of vCenter Single Sign-On forauthentication.

n If there is a need to replicate with other Platform Services Controller instances, or if the solutionincludes more than one vCenter Single Sign-On instance, you can deploy multiple external PlatformServices Controller instances on separate virtual machines.

Table 2‑26. Platform Service Controller Design Decisions


SDDC-VI-VC-003

Deploy each vCenter Serverwith an external PlatformServices Controller.

External Platform Services Controllers arerequired for replication between PlatformServices Controller instances.

The number of VMs that haveto be managed increases.

SDDC-VI-VC-004

Join all Platform ServicesController instances to asingle vCenter Single Sign-On domain.

When all Platform Services Controllerinstances are joined into a single vCenterSingle Sign-On domain, they can shareauthentication and license data across allcomponents and regions.

Only one Single Sign-Ondomain will exist.

SDDC-VI-VC-005

Create a ring topology for thePlatform Service Controllers.

By default Platform Service Controllers onlyreplicate with one other Platform ServicesController, that creates a single point of failurefor replication. A ring topology ensures eachPlatform Service Controller has tworeplication partners and eliminates any singlepoint of failure.

Command-line interfacecommands must be used toconfigure the ring replicationtopology.

SDDC-VI-VC-006

Use an NSX Edge ServicesGateway as a load balancerfor the Platform ServicesControllers.

Using a load balancer increases theavailability of the PSC’s for all applications.

Configuring the load balancerand repointing vCenter Serverto the load balancers Virtual IP(VIP) creates administrativeoverhead.


VMware, Inc. 70

Figure 2‑11. vCenter Server and Platform Services Controller Deployment Model

Region A: SFO Region B: LAX

Platform Services ControllerAppliance

SFO-1

Management vCenter Server

Appliance

Shared vCenter Single Sign-On Domain

Platform ServicesControllerAppliance

SFO-2

NSX Edge Load Balancer NSX Edge Load Balancer

ComputevCenter Server

Appliance


LAX-2

ComputevCenter Server

Appliance


LAX-1


Appliance LAX LAXSFO SFO

vCenter Server NetworkingAs specified in the physical networking design, all vCenter Server systems must use static IP addressesand host names. The IP addresses must have valid (internal) DNS registration including reverse nameresolution.

The vCenter Server systems must maintain network connections to the following components:

n All VMware vSphere Client and vSphere Web Client user interfaces.

n Systems running vCenter Server add-on modules.

n Each ESXi host.

vCenter Server RedundancyProtecting the vCenter Server system is important because it is the central point of management andmonitoring for the SDDC. How you protect vCenter Server depends on maximum downtime tolerated, andon whether failover automation is required.

The following table lists methods available for protecting the vCenter Server system and the vCenterServer Appliance.


VMware, Inc. 71

Table 2‑27. Methods for Protecting vCenter Server System and the vCenter Server Appliance

Redundancy Method

Protects vCenterServer system(Windows)

Protects PlatformServices Controller(Windows)

Protects vCenterServer (Appliance)

Protects PlatformServices Controller(Appliance)

Automated protection usingvSphere HA.

Yes Yes Yes Yes

Manual configuration andmanual failover. For example,using a cold standby.

Yes Yes Yes Yes

HA Cluster with external loadbalancer

Not Available Yes Not Available Yes

vCenter Server HA Not Available Not Available Yes Not Available

Table 2‑28. vCenter Server Protection Design Decision


SDDC-VI-VC-007 Protect all vCenter Server andPlatform Services Controllerappliances by using vSphere HA.

Supports availability objectives forvCenter Server appliances without arequired manual intervention during afailure event.

vCenter Server will beunavailable during a vSphereHA failover.

vCenter Server Appliance SizingThe following tables outline minimum hardware requirements for the management vCenter Serverappliance and the compute vCenter Server appliance.

Table 2‑29. Logical Specification for Management vCenter Server Appliance

Attribute Specification

vCenter Server version 6.5 (vCenter Server Appliance)

Physical or virtual system Virtual (appliance)

Appliance Size Small (up to 100 hosts / 1,000 VMs)

Platform Services Controller External

Number of CPUs 4

Memory 16 GB

Disk Space 290 GB

Table 2‑30. Logical Specification for Compute vCenter Server Appliance


vCenter Server version 6.5 (vCenter Server Appliance)

Physical or virtual system Virtual (appliance)

Appliance Size Large (up to 1,000 hosts / 10,000 VMs)

Platform Services Controller External

Number of CPUs 16


VMware, Inc. 72

Table 2‑30. Logical Specification for Compute vCenter Server Appliance(Continued)


Memory 32 GB

Disk Space 640 GB

Table 2‑31. vCenter Server Appliance Sizing Design Decisions


SDDC-VI-VC-008

Configure the managementvCenter Server Applianceswith at least the small sizesetting.

Based on the number of managementVMs that are running, a vCenter ServerAppliance installed with the small sizesetting is sufficient.

If the size of the managementenvironment changes, the vCenterServer Appliance size might needto be increased.

SDDC-VI-VC-009

Configure the computevCenter Server Applianceswith at least the large sizesetting.

Based on the number of computeworkloads and NSX edge devicesrunning, a vCenter Server Applianceinstalled with the large size setting isrecommended.

As the compute environmentgrows resizing to X-Large oradding additional vCenter Serverinstances may be required.

vSphere Cluster DesignThe cluster design must take into account the workload that the cluster handles. Different cluster types inthis design have different characteristics.

vSphere Cluster Design Decision Background

The following heuristics help with cluster design decisions.

n Decide to use fewer, larger hosts or more, smaller hosts.

n A scale-up cluster has fewer, larger hosts.

n A scale-out cluster has more, smaller hosts.

n A virtualized server cluster typically has more hosts with fewer virtual machines per host.

n Compare the capital costs of purchasing fewer, larger hosts with the costs of purchasing more,smaller hosts. Costs vary between vendors and models.

n Evaluate the operational costs of managing a few hosts with the costs of managing more hosts.

n Consider the purpose of the cluster.

n Consider the total number of hosts and cluster limits.


VMware, Inc. 73

Figure 2‑12. vSphere Logical Cluster Layout

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

APPOS

MgmtVC

Region AManagement Cluster

ESXi ESXi ESXi ESXi ESXi ESXi ESXi

Region ACompute / Edge Cluster

Region BManagement Cluster

Region BCompute / Edge Cluster

PSC

NSX Edge Load Balancer

NSX Edge Load Balancer

ComputeVC

PSC

MgmtVC

ESXi ESXi ESXi ESXi ESXi ESXi ESXi

PSC

ComputeVC

PSC

vSphere High Availability Design

VMware vSphere High Availability (vSphere HA) protects your virtual machines in case of host failure byrestarting virtual machines on other hosts in the cluster when a host fails.

vSphere HA Design Basics

During configuration of the cluster, the hosts elect a master host. The master host communicates with thevCenter Server system and monitors the virtual machines and secondary hosts in the cluster.

The master hosts detects different types of failure:

n Host failure, for example an unexpected power failure

n Host network isolation or connectivity failure

n Loss of storage connectivity

n Problems with virtual machine OS availability

Table 2‑32. vSphere HA Design Decisions


SDDC-VI-VC-010 Use vSphere HA toprotect all clustersagainst failures.

vSphere HA supports a robust level ofprotection for both host and virtualmachine availability.

Sufficient resources on the remaininghost are required to so that virtualmachines can be migrated to those hostsin the event of a host outage.

SDDC-VI-VC-011 Set vSphere HA HostIsolation Response toPower Off.

vSAN requires that the HA IsolationResponse be set to Power Off and torestart VMs on available hosts.

VMs are powered off in case of a falsepositive and a host is declared isolatedincorrectly.


VMware, Inc. 74

vSphere HA Admission Control Policy Configuration

The vSphere HA Admission Control Policy allows an administrator to configure how the cluster judgesavailable resources. In a smaller vSphere HA cluster, a larger proportion of the cluster resources arereserved to accommodate host failures, based on the selected policy.

The following policies are available:

Host failures the clustertolerates.

vSphere HA ensures that a specified number of hosts can fail and sufficientresources remain in the cluster to fail over all the virtual machines fromthose hosts.

Percentage of clusterresources reserved.

Percentage of cluster resources reserved. vSphere HA ensures that aspecified percentage of aggregate CPU and memory resources arereserved for failover.

Specify Failover Hosts. When a host fails, vSphere HA attempts to restart its virtual machines onany of the specified failover hosts. If restart is not possible, for example thefailover hosts have insufficient resources or have failed as well, thenvSphere HA attempts to restart the virtual machines on other hosts in thecluster.

vSphere Cluster Workload Design

This design defines the following vSphere clusters and the workloads that they handle.

Table 2‑33. vSphere Cluster Workload Design Decisions


SDDC-VI-VC-012

Create a singlemanagement clustercontaining all managementhosts.

Simplifies configuration by isolatingmanagement workloads from computeworkloads.

Ensures that compute workloads haveno impact on the management stack.

You can add ESXi hosts to the clusteras needed.

Management of multiple clusters andvCenter Server instances increasesoperational overhead.

SDDC-VI-VC-013

Create a shared edge andcompute cluster that hostscompute workloads, NSXControllers and associatedNSX Edge gateway devicesused for computeworkloads.

Simplifies configuration and minimizesthe number of hosts required for initialdeployment.

Ensures that the management stackhas no impact on compute workloads.

You can add ESXi hosts to the clusteras needed.

Management of multiple clusters andvCenter Server instances increasesoperational overhead.

Due to the shared nature of the cluster,when compute workloads are added, thecluster must be scaled out to keep highlevel of network performance.

Due to the shared nature of the cluster,resource pools are required to ensureedge components receive all requiredresources.

Management Cluster Design

The management cluster design determines the number of hosts and vSphere HA settings for themanagement cluster.


VMware, Inc. 75

Table 2‑34. Management Cluster Design Decisions


SDDC-VI-VC-014

Create a managementcluster with 4 hosts.

Three hosts are used to provide n+1redundancy for the vSAN cluster. The fourthhost is used to guarantee n+1 for vSANredundancy during maintenance operations.

Additional host resources arerequired for redundancy.

SDDC-VI-VC-015

Configure AdmissionControl for 1 host failureand percentage basedfailover capacity.

Using the percentage-based reservation workswell in situations where virtual machines havevarying and sometime significant CPU ormemory reservations.

vSphere 6.5 automatically calculates thereserved percentage based on host failures totolerate and the number of hosts in the cluster.

In a four host managementcluster only the resources of threehosts are available for use.

SDDC-VI-VC-016

Create a host profile forthe Management Cluster.

Utilizing host profiles simplifies configuration ofhosts and ensures settings are uniform acrossthe cluster.

Anytime an authorized change toa host is made the host profilemust be updated to reflect thechange or the status will shownon-compliant.

The following table summarizes the attributes of the management cluster logical design.

Table 2‑35. Management Cluster Logical Design Background


Number of hosts required to support management hosts with no over commitment . 2

Number of hosts recommended due to operational constraints (Ability to take a host offline withoutsacrificing High Availability capabilities) .

3

Number of hosts recommended due to operational constraints, while using vSAN (Ability to take a hostoffline without sacrificing High Availability capabilities) .

4

Capacity for host failures per cluster . 25% reserved CPU RAM

Shared Edge and Compute Cluster Design

Tenant workloads run on the ESXi hosts in the shared edge and compute cluster. Due to the sharednature of the cluster, NSX Controllers and Edge devices run in this cluster. The design decisionsdetermine the number of hosts and vSphere HA settings and several other characteristics of the sharededge and compute cluster.


VMware, Inc. 76

Table 2‑36. Shared Edge and Compute Cluster Design Decisions


SDDC-VI-VC-017

Create a shared edge andcompute cluster for the NSXControllers and NSX Edgegateway devices.

NSX Manager requires a 1:1 relationshipwith a vCenter Server system.

Each time you provision a ComputevCenter Server system, a new NSXManager is required.

Set anti-affinity rules to keep eachController on a separate host. A 4-node cluster allows maintenancewhile ensuring that the 3 Controllersremain on separate hosts.

SDDC-VI-VC-018

Configure Admission Controlfor 1 host failure andpercentage based failovercapacity.

vSphere HA protects the NSX Controllerinstances and edge services gatewaydevices in the event of a host failure.vSphere HA powers on virtual machinesfrom the failed hosts on any remaininghosts.

Only a single host failure istolerated before potential resourcecontention.

SDDC-VI-VC-019

Create shared edge andcompute cluster with aminimum of 4 hosts.

n 3 NSX Controllers are required forsufficient redundancy and majoritydecisions.

n One host is available for failover and toallow for scheduled maintenance.

4 hosts is the smallest starting pointfor the shared edge and computecluster for redundancy andperformance thus increasing costover a 3 node cluster.

SDDC-VI-VC-020

Set up VLAN-backed portgroups for external accessand management on theshared edge and computecluster hosts.

Edge gateways need access to the externalnetwork in addition to the managementnetwork.

VLAN-backed port groups must beconfigured with the correct numberof ports, or with elastic portallocation.

SDDC-VI-VC-021

Create a resource pool forthe required SDDC NSXControllers and edgeappliances with a CPU sharelevel of High, a memoryshare of normal, and 16 GBmemory reservation.

The NSX components control all networktraffic in and out of the SDDC as well asupdate route information for inter-SDDCcommunication. In a contention situation itis imperative that these virtual machinesreceive all the resources required.

During contention SDDC NSXcomponents receive moreresources then all other workloadsas such monitoring and capacitymanagement must be a proactiveactivity.

SDDC-VI-VC-022

Create a resource pool for alluser NSX Edge devices witha CPU share value of Normaland a memory share value ofNormal.

NSX edges for users, created by vRealizeAutomation, support functions such as loadbalancing for user workloads. These edgedevices do not support the entire SDDC assuch they receive a lower amount ofresources during contention.

During contention these NSX edgeswill receive fewer resources thanthe SDDC edge devices. As aresult, monitoring and capacitymanagement must be a proactiveactivity.

SDDC-VI-VC-023

Create a resource pool for alluser virtual machines with aCPU share value of Normaland a memory share value ofNormal.

Creating virtual machines outside of aresource pool will have a negative impacton all other virtual machines duringcontention. In a shared edge and computecluster the SDDC edge devices must beguaranteed resources above all otherworkloads as to not impact networkconnectivity. Setting the share values tonormal gives the SDDC edges more sharesof resources during contention ensuringnetwork traffic is not impacted.

During contention user workloadvirtual machines could be starvedfor resources and experience poorperformance. It is critical thatmonitoring and capacitymanagement must be a proactiveactivity and that capacity is addedor a dedicated edge cluster iscreated before contention occurs.


VMware, Inc. 77

The following table summarizes the attributes of the shared edge and compute cluster logical design. Thenumber of VMs on the shared edge and compute cluster will start low but will grow quickly as userworkloads are created.

Table 2‑37. Shared Edge and Compute Cluster Logical Design Background


Minimum number of hosts required to support the shared edge and compute cluster 4

Capacity for host failures per cluster 1

Number of usable hosts per cluster 3

Compute Cluster Design

As the SDDC grows, additional compute-only clusters can be configured. Tenant workloads run on theESXi hosts in the compute cluster instances. Multiple compute clusters are managed by the ComputevCenter Server instance. The design determines host-to-rack relationship and vSphere HA settings forthe compute cluster.

Table 2‑38. Compute Cluster Design Decisions


SDDC-VI-VC-024 Configure vSphere HA to usepercentage-based failovercapacity to ensure n+1availability.

Using explicit host failover limitsthe total available resources in acluster.

The resources of one host in thecluster is reserved which can causeprovisioning to fail if resources areexhausted.

vCenter Server CustomizationvCenter Server supports a rich set of customization options, including monitoring, virtual machine faulttolerance, and so on. For each feature, this VMware Validated Design specifies the design decisions.

VM and Application Monitoring Service

When VM and Application Monitoring is enabled, the VM and Application Monitoring service, which usesVMware Tools, evaluates whether each virtual machine in the cluster is running. The service checks forregular heartbeats and I/O activity from the VMware Tools process running on guests. If the servicereceives no heartbeats or I/O activity, it is likely that the guest operating system has failed or that VMwareTools is not being allocated time for heartbeats or I/O activity. In this case, the service determines that thevirtual machine has failed and reboots the virtual machine.

Enable Virtual Machine Monitoring for automatic restart of a failed virtual machine. The application orservice that is running on the virtual machine must be capable of restarting successfully after a reboot orthe VM restart is not sufficient.


VMware, Inc. 78

Table 2‑39. Monitor Virtual Machines Design Decision


SDDC-VI-VC-025

Enable Virtual MachineMonitoring for each cluster.

Virtual Machine Monitoring providesadequate in-guest protection for mostVM workloads.

There is no downside toenabling Virtual MachineMonitoring.

SDDC-VI-VC-026

Create Virtual Machine Groups foruse in startup rules in themanagement and shared edgeand compute clusters.

By creating Virtual Machine groups,rules can be created to configure thestartup order of the SDDC managementcomponents.

Creating the groups is amanual task and addsadministrative overhead.

SDDC-VI-VC-027

Create Virtual Machine rules tospecify the startup order of theSDDC management components.

The rules enforce the startup order ofvirtual machine groups to ensure thecorrect startup order of the SDDCmanagement components.

Creating the rules is a manualtask and adds administrativeoverhead.

VMware vSphere Distributed Resource Scheduling (DRS)

vSphere Distributed Resource Scheduling provides load balancing of a cluster by migrating workloadsfrom heavily loaded hosts to less utilized hosts in the cluster. DRS supports manual and automaticmodes.

Manual Recommendations are made but an administrator needs to confirm thechanges

Automatic Automatic management can be set to five different levels. At the lowestsetting, workloads are placed automatically at power on and only migratedto fulfill certain criteria, such as entering maintenance mode. At the highestlevel, any migration that would provide a slight improvement in balancingwill be executed.

Table 2‑40. vSphere Distributed Resource Scheduling Design Decision


SDDC-VI-VC-028 Enable DRS on all clusters andset it to Fully Automated, withthe default setting (medium).

The default settings provide thebest trade-off between loadbalancing and excessive migrationwith vMotion events.

In the event of a vCenter outage,mapping from virtual machines toESXi hosts might be more difficult todetermine.

Enhanced vMotion Compatibility (EVC)

EVC works by masking certain features of newer CPUs to allow migration between hosts containing olderCPUs. EVC works only with CPUs from the same manufacturer and there are limits to the versiondifference gaps between the CPU families.

If you set EVC during cluster creation, you can add hosts with newer CPUs at a later date withoutdisruption. You can use EVC for a rolling upgrade of all hardware with zero downtime.

Set EVC to the highest level possible with the current CPUs in use.


VMware, Inc. 79

Table 2‑41. VMware Enhanced vMotion Compatibility Design Decision


SDDC-VI-VC-029 Enable Enhanced vMotion Compatibilityon all clusters. Set EVC mode to thelowest available setting supported for thehosts in the cluster.

Allows cluster upgradeswithout virtual machinedowntime.

You can enable EVC only ifclusters contain hosts with CPUsfrom the same vendor.

Use of Transport Layer Security (TLS) CertificatesBy default vSphere 6.5 uses TLS/SSL certificates that are signed by VMCA (VMware CertificateAuthority). By default, these certificates are not trusted by end-user devices or browsers. It is a securitybest practice to replace at least user-facing certificates with certificates that are signed by a third-party orenterprise Certificate Authority (CA). Certificates for machine-to-machine communication can remain asVMCA-signed certificates.

Table 2‑42. vCenter Server TLS Certificate Design Decision


SDDC-VI-VC-030 Replace the vCenter Servermachine certificate and PlatformServices Controller machinecertificate with a certificate signedby a 3rd party Public KeyInfrastructure.

Infrastructure administrators connect toboth vCenter Server and the PlatformServices Controller by way of s Webbrowser to perform configuration,management and troubleshooting activities.Certificate warnings result with the defaultcertificate.

Replacing and managingcertificates is anoperational overhead.

SDDC-VI-VC-031 Use a SHA-2 or higher algorithmwhen signing certificates.

The SHA-1 algorithm is considered lesssecure and has been deprecated.

Not all certificateauthorities supportSHA-2.

Virtualization Network DesignA well-designed network helps the organization meet its business goals. It prevents unauthorized access,and provides timely access to business data.

This network virtualization design uses vSphere and VMware NSX for vSphere to implement virtualnetworking.

n Virtual Network Design Guidelines

This VMware Validated Design follows high-level network design guidelines and networking bestpractices.

n Virtual Switches

Virtual switches simplify the configuration process by providing one single pane of glass view forperforming virtual network management tasks.

n NIC Teaming

You can use NIC teaming to increase the network bandwidth available in a network path, and toprovide the redundancy that supports higher availability.


VMware, Inc. 80

n Network I/O Control

When Network I/O Control is enabled, the distributed switch allocates bandwidth for the followingsystem traffic types.

n VXLAN

VXLAN provides the capability to create isolated, multi-tenant broadcast domains across data centerfabrics, and enables customers to create elastic, logical networks that span physical networkboundaries.

n vMotion TCP/IP Stack

Use the vMotion TCP/IP stack to isolate traffic for vMotion and to assign a dedicated defaultgateway for vMotion traffic.

Virtual Network Design GuidelinesThis VMware Validated Design follows high-level network design guidelines and networking bestpractices.

Design Goals

The high-level design goals apply regardless of your environment.

n Meet diverse needs. The network must meet the diverse needs of many different entities in anorganization. These entities include applications, services, storage, administrators, and users.

n Reduce costs. Reducing costs is one of the simpler goals to achieve in the vSphere infrastructure.Server consolidation alone reduces network costs by reducing the number of required network portsand NICs, but a more efficient network design is desirable. For example, configuring two 10 GbENICs with VLANs might be more cost effective than configuring a dozen 1 GbE NICs on separatephysical networks.

n Boost performance. You can achieve performance improvement and decrease the time that isrequired to perform maintenance by providing sufficient bandwidth, which reduces contention andlatency.

n Improve availability. A well-designed network improves availability, typically by providing networkredundancy.

n Support security. A well-designed network supports an acceptable level of security through controlledaccess (where required) and isolation (where necessary).

n Enhance infrastructure functionality. You can configure the network to support vSphere features suchas vSphere vMotion, vSphere High Availability, and vSphere Fault Tolerance.

Best Practices

Follow networking best practices throughout your environment.

n Separate network services from one another to achieve greater security and better performance.


VMware, Inc. 81

n Use Network I/O Control and traffic shaping to guarantee bandwidth to critical virtual machines.During network contention these critical virtual machines will receive a higher percentage of thebandwidth.

n Separate network services on a single vSphere Distributed Switch by attaching them to port groupswith different VLAN IDs.

n Keep vSphere vMotion traffic on a separate network. When migration with vMotion occurs, thecontents of the guest operating system’s memory is transmitted over the network. You can putvSphere vMotion on a separate network by using a dedicated vSphere vMotion VLAN.

n When using passthrough devices with a Linux kernel version 2.6.20 or earlier guest OS, avoid MSIand MSI-X modes because these modes have significant performance impact.

n For best performance, use VMXNET3 virtual NICs.

n Ensure that physical network adapters that are connected to the same vSphere Standard Switch orvSphere Distributed Switch are also connected to the same physical network.

Network Segmentation and VLANs

Separating different types of traffic is required to reduce contention and latency. Separate networks arealso required for access security.

High latency on any network can negatively affect performance. Some components are more sensitive tohigh latency than others. For example, reducing latency is important on the IP storage and the vSphereFault Tolerance logging network because latency on these networks can negatively affect theperformance of multiple virtual machines.

Depending on the application or service, high latency on specific virtual machine networks can alsonegatively affect performance. Use information gathered from the current state analysis and frominterviews with key stakeholder and SMEs to determine which workloads and networks are especiallysensitive to high latency.

Virtual Networks

Determine the number of networks or VLANs that are required depending on the type of traffic.

n vSphere operational traffic.

n Management

n vMotion

n vSAN

n NFS Storage

n vSphere Replication

n VXLAN

n Traffic that supports the organization’s services and applications.


VMware, Inc. 82

Virtual SwitchesVirtual switches simplify the configuration process by providing one single pane of glass view forperforming virtual network management tasks.

Virtual Switch Design Background

A vSphere Distributed Switch (distributed switch) offers several enhancements over standard virtualswitches.

Centralizedmanagement

Because distributed switches are created and managed centrally on avCenter Server system, they make the switch configuration more consistentacross ESXi hosts. Centralized management saves time, reducesmistakes, and lowers operational costs.

Additional features Distributed switches offer features that are not available on standard virtualswitches. Some of these features can be useful to the applications andservices that are running in the organization’s infrastructure. For example,NetFlow and port mirroring provide monitoring and troubleshootingcapabilities to the virtual infrastructure.

Consider the following caveats for distributed switches.

n Distributed switches are not manageable when vCenter Server is unavailable. vCenter Servertherefore becomes a tier one application.

Health Check

The health check service helps identify and troubleshoot configuration errors in vSphere distributedswitches.

Health check helps identify the following common configuration errors.

n Mismatched VLAN trunks between an ESXi host and the physical switches it's connected to.

n Mismatched MTU settings between physical network adapters, distributed switches, and physicalswitch ports.

n Mismatched virtual switch teaming policies for the physical switch port-channel settings.

Health check monitors VLAN, MTU, and teaming policies.

VLANs Checks whether the VLAN settings on the distributed switch match thetrunk port configuration on the connected physical switch ports.

MTU For each VLAN, health check determines whether the physical accessswitch port's MTU jumbo frame setting matches the distributed switch MTUsetting.

Teaming policies Health check determines whether the connected access ports of thephysical switch that participate in an EtherChannel are paired withdistributed ports whose teaming policy is IP hash.


VMware, Inc. 83

Health check is limited to the access switch port to which the ESXi hosts' NICs connects.

Design ID Design Decision Design Justification Design Implication

SDDC-VI-Net-001

Enable vSphere DistributedSwitch Health Check on allvirtual distributed switches.

vSphere Distributed Switch Health Checkensures all VLANS are trunked to all hostsattached to the vSphere Distributed Switch andensures MTU sizes match the physical network.

You must have a minimum oftwo physical uplinks to use thisfeature.

Note For VLAN and MTU checks, at least two physical NICs for the distributed switch are required. Fora teaming policy check, at least two physical NICs and two hosts are required when applying the policy.

Number of Virtual Switches

Create fewer virtual switches, preferably just one. For each type of network traffic, configure a singleportgroup to simplify configuration and monitoring.

Table 2‑43. Virtual Switch Design Decisions


SDDC-VI-Net-002 Use vSphere DistributedSwitches (VDS).

vSphere Distributed Switchessimplify management.

Migration from a VSS to a VDS requires aminimum of two physical NICs to maintainredundancy.

SDDC-VI-Net-003 Use a single VDS percluster.

Reduces complexity of the networkdesign.

Reduces the size of the faultdomain.

Increases the number of vSphereDistributed Switches that must bemanaged.

Management Cluster Distributed Switches

The management cluster uses a single vSphere Distributed Switch with the following configurationsettings.

Table 2‑44. Virtual Switch for the Management Cluster

vSphere DistributedSwitch Name Function

Network I/OControl

Number of PhysicalNIC Ports MTU

vDS-Mgmt n ESXi Managementn Network IP Storage (NFS)n vSANn vSphere vMotionn VXLAN Tunnel Endpoint (VTEP)n vSphere Replication/vSphere Replication NFCn Uplinks (2) to enable ECMPn External management connectivity

Enabled 2 9000

Table 2‑45. vDS-MgmtPort Group Configuration Settings

Parameter Setting

Failover detection Link status only

Notify switches Enabled


VMware, Inc. 84

Table 2‑45. vDS-MgmtPort Group Configuration Settings(Continued)

Parameter Setting

Failback Yes

Failover order Active uplinks: Uplink1, Uplink2

Figure 2‑13. Network Switch Design for Management Hosts

nic0 nic1

VLAN VSAN

VLAN ESXi Management

VLAN vMotion

VLAN VTEP (VXLAN)

Sample ESXi Management Host

VLAN vSphere Replication/vSphere Replication NFC

vDS-Mgmt

VLAN External Management

VLAN NFS

VLAN Uplink01

VLAN Uplink02

This section expands on the logical network design by providing details on the physical NIC layout andphysical network attributes.


VMware, Inc. 85

Table 2‑46. Management Virtual Switches by Physical/Virtual NIC

vSphere Distributed Switch vmnic Function

vDS-Mgmt 0 Uplink

vDS-Mgmt 1 Uplink

Note The following VLANs are meant as samples. Your actual implementation depends on yourenvironment.

Table 2‑47. Management Virtual Switch Port Groups and VLANs

vSphere Distributed Switch Port Group Name Teaming Policy Active Uplinks VLAN ID

vDS-Mgmt vDS-Mgmt-Management Route based on physical NIC load 0, 1 1611

vDS-Mgmt vDS-Mgmt-vMotion Route based on physical NIC load 0, 1 1612

vDS-Mgmt vDS-Mgmt-VSAN Route based on physical NIC load 0, 1 1613

vDS-Mgmt Auto Generated (NSX VTEP) Route based on SRC-ID 0, 1 1614

vDS-Mgmt vDS-Mgmt-Uplink01 Route based on physical NIC load 0, 1 2711

vDS-Mgmt vDS-Mgmt-Uplink02 Route based on physical NIC load 0, 1 2712

vDS-Mgmt vDS-Mgmt-NFS Route based on physical NIC load 0, 1 1615

vDS-Mgmt vDS-Mgmt-VR Route based on physical NIC load 0, 1 1616

vDS-Mgmt vDS-Mgmt-Ext-Management Route based on physical NIC load 0, 1 130

Table 2‑48. Management VMkernel Adapter

vSphere Distributed Switch Network Label Connected Port Group Enabled Services MTU

vDS-Mgmt Management vDS-Mgmt-Management Management Traffic 1500 (Default)

vDS-Mgmt vMotion vDS-Mgmt-vMotion vMotion Traffic 9000

vDS-Mgmt vSAN vDS-Mgmt-VSAN vSAN 9000

vDS-Mgmt NFS vDS-Mgmt-NFS - 9000

vDS-Mgmt Replication vDS-Mgmt-VR vSphere Replication traffic

vSphere Replication NFC traffic

9000

vDS-Mgmt VTEP Auto Generated (NSX VTEP) - 9000

For more information on the physical network design specifications, see Physical Networking Design.

Shared Edge and Compute Cluster Distributed Switches

The shared edge and compute cluster uses a single vSphere Distributed Switch with the followingconfiguration settings.


VMware, Inc. 86

Table 2‑49. Virtual Switch for the Shared Edge and Compute Cluster

vSphere DistributedSwitch Name Function Network I/O Control

Number of PhysicalNIC Ports MTU

vDS-Comp01 n ESXi Managementn Network IP Storage (NFS)n vSphere vMotionn VXLAN Tunnel Endpoint (VTEP)n Uplinks (2) to enable ECMPn vSANn External customer/tenant connectivity

Enabled 2 9000

Table 2‑50. vDS-Comp01 Port Group Configuration Settings

Parameter Setting

Failoverdetection Link status only


Failback Yes


Network Switch Design for Shared Edge and Compute Hosts



VMware, Inc. 87

Figure 2‑14. Network Switch Design for Shared Edge and Compute Hosts

nic0 nic1

VLAN VSAN


VLAN vMotion

VLAN VTEP (VXLAN)

Sample ESXi Shared Edge and Compute Host

vDS-Comp01

VLAN NFS

VLAN Uplink01

VLAN Uplink02

Table 2‑51. Shared Edge and Compute Cluster Virtual Switches by Physical/Virtual NIC


vDS-Comp01 0 Uplink

vDS-Comp01 1 Uplink


Table 2‑52. Shared Edge and Compute Cluster Virtual Switch Port Groups and VLANs


vDS-Comp01 vDS-Comp01-Management Route based on physical NIC load 0, 1 1631

vDS-Comp01 vDS-Comp01-vMotion Route based on physical NIC load 0, 1 1632

vDS-Comp01 vDS-Comp01-VSAN Route based on physical NIC load 0, 1 1633

vDS-Comp01 vDS-Comp01-NFS Route based on physical NIC load 0, 1 1615

vDS-Comp01 Auto Generated (NSX VTEP) Route based on SRC-ID 0, 1 1634


VMware, Inc. 88

Table 2‑52. Shared Edge and Compute Cluster Virtual Switch Port Groups and VLANs(Continued)


vDS-Comp01 vDS-Comp01-Uplink01 Route based on physical NIC load 0, 1 1635

vDS-Comp01 vDS-Comp01-Uplink02 Route based on physical NIC load 0, 1 2713

Table 2‑53. Shared Edge and Compute Cluster VMkernel Adapter


vDS-Comp01 Management vDS-Comp01-Management Management Traffic 1500 (Default)

vDS-Comp01 vMotion vDS-Comp01-vMotion vMotion Traffic 9000

vDS-Comp01 VSAN vDS-Comp01-VSAN VSAN 9000

vDS-Comp01 NFS vDS-Comp01-NFS - 9000

vDS-Comp01 VTEP Auto Generated (NSX VTEP) - 9000

For more information on the physical network design, see Physical Networking Design.

Compute Cluster Distributed Switches

A compute cluster vSphere Distributed Switch uses the following configuration settings.

Table 2‑54. Virtual Switch for a dedicated Compute Cluster

vSphere Distributed SwitchName Function Network I/O Control

Number of Physical NICPorts MTU

vDS-Comp02 n ESXi Managementn Network IP Storage (NFS)n vSphere vMotionn VXLAN Tunnel Endpoint (VTEP)

Enabled 2 9000

Table 2‑55. vDS-Comp02 Port Group Configuration Settings

Parameter Setting

Failover detection Link status only


Failback Yes



VMware, Inc. 89

Network Switch Design for Compute Hosts

Figure 2‑15. Network Switch Design for Compute Hosts

nic0 nic1

VLAN VSAN


VLAN vMotion

VLAN VTEP (VXLAN)

Sample ESXi Compute Host

VLAN NFS

vDS-Comp02


Table 2‑56. Compute Cluster Virtual Switches by Physical/Virtual NIC


vDS-Comp02 0 Uplink

vDS-Comp02 1 Uplink


Table 2‑57. Compute Cluster Virtual Switch Port Groups and VLANs


vDS-Comp02 vDS-Comp02-Management Route based on physical NIC load 0, 1 1621

vDS-Comp02 vDS-Comp02-vMotion Route based on physical NIC load 0, 1 1622

vDS-Comp02 Auto Generated (NSX VTEP) Route based on SRC-ID 0, 1 1624

vDS-Comp02 vDS-Comp02-NFS Route based on physical NIC load 0, 1 1625


VMware, Inc. 90

Table 2‑58. Compute Cluster VMkernel Adapter


vDS-Comp02 Management vDS-Comp02-Management Management traffic 1500 (Default)

vDS-Comp02 vMotion vDS-Comp02-vMotion vMotion traffic 9000

vDS-Comp02 NFS vDS-Comp02-NFS - 9000

vDS-Comp02 VTEP Auto Generated (NSX VTEP) - 9000

For more information on the physical network design specifications, see Physical Networking Design.

NIC TeamingYou can use NIC teaming to increase the network bandwidth available in a network path, and to providethe redundancy that supports higher availability.

NIC teaming helps avoid a single point of failure and provides options for load balancing of traffic. Tofurther reduce the risk of a single point of failure, build NIC teams by using ports from multiple NIC andmotherboard interfaces.

Create a single virtual switch with teamed NICs across separate physical switches.

This VMware Validated Design uses an active-active configuration using the route that is based onphysical NIC load algorithm for teaming. In this configuration, idle network cards do not wait for a failureto occur, and they aggregate bandwidth.

Benefits and Overview

NIC teaming helps avoid a single point of failure and provides options for load balancing of traffic. Tofurther reduce the risk of a single point of failure, build NIC teams by using ports from multiple NIC andmotherboard interfaces.

Create a single virtual switch with teamed NICs across separate physical switches.

This VMware Validated Design uses an active-active configuration using the route that is based onphysical NIC load algorithm for teaming. In this configuration, idle network cards do not wait for a failureto occur, and they aggregate bandwidth.

NIC Teaming Design Background

For a predictable level of performance, use multiple network adapters in one of the followingconfigurations.

n An active-passive configuration that uses explicit failover when connected to two separate switches.

n An active-active configuration in which two or more physical NICs in the server are assigned theactive role.

This validated design uses an active-active configuration.


VMware, Inc. 91

Table 2‑59. NIC Teaming and Policy

Design Quality Active-Active Active-Passive Comments

Availability ↑ ↑ Using teaming regardless of the option increases the availability of theenvironment.


Performance ↑ o An active-active configuration can send traffic across either NIC, therebyincreasing the available bandwidth. This configuration provides a benefit ifthe NICs are being shared among traffic types and Network I/O Control isused.

Recoverability o o Neither design option impacts recoverability.



Table 2‑60. NIC Teaming Design Decision


SDDC-VI-Net-004 Use the Route based on physical NIC loadteaming algorithm for all port groups exceptfor ones that carry VXLAN traffic. VTEPkernel ports and VXLAN traffic will useRoute based on SRC-ID.

Reduce complexity of thenetwork design and increaseresiliency and performance.

Because NSX does not supportRoute based on physical NICload two different algorithmsare necessary.

Network I/O ControlWhen Network I/O Control is enabled, the distributed switch allocates bandwidth for the following systemtraffic types.

n Fault tolerance traffic

n iSCSI traffic

n vSphere vMotion traffic

n Management traffic

n VMware vSphere Replication traffic

n NFS traffic

n vSAN traffic

n vSphere Data Protection backup traffic

n Virtual machine traffic

How Network I/O Control Works

Network I/O Control enforces the share value specified for the different traffic types only when there isnetwork contention. When contention occurs Network I/O Control applies the share values set to eachtraffic type. As a result, less important traffic, as defined by the share percentage, will be throttled,allowing more important traffic types to gain access to more network resources.


VMware, Inc. 92

Network I/O Control also allows the reservation of bandwidth for system traffic based on the capacity ofthe physical adapters on a host, and enables fine-grained resource control at the virtual machine networkadapter level. Resource control is similar to the model for vCenter CPU and memory reservations.

Network I/O Control Heuristics

The following heuristics can help with design decisions.

Shares vs. Limits When you use bandwidth allocation, consider using shares instead of limits.Limits impose hard limits on the amount of bandwidth used by a traffic floweven when network bandwidth is available.

Limits on CertainResource Pools

Consider imposing limits on a given resource pool. For example, if you puta limit on vSphere vMotion traffic, you can benefit in situations wheremultiple vSphere vMotion data transfers, initiated on different hosts at thesame time, result in oversubscription at the physical network level. Bylimiting the available bandwidth for vSphere vMotion at the ESXi host level,you can prevent performance degradation for other traffic.

Teaming Policy When you use Network I/O Control, use Route based on physical NIC loadteaming as a distributed switch teaming policy to maximize the networkingcapacity utilization. With load-based teaming, traffic might move amonguplinks, and reordering of packets at the receiver can result occasionally.

Traffic Shaping Use distributed port groups to apply configuration policies to different traffictypes. Traffic shaping can help in situations where multiple vSpherevMotion migrations initiated on different hosts converge on the samedestination host. The actual limit and reservation also depend on the trafficshaping policy for the distributed port group where the adapter is connectedto.

Network I/O Control Design Decisions

Based on the heuristics, this design has the following decisions.

Table 2‑61. Network I/O Control Design Decisions


SDDC-VI-NET-005

Enable Network I/O Controlon all distributed switches.

Increase resiliency and performance of thenetwork.

If configured incorrectlyNetwork I/O Control couldimpact networkperformance for criticaltraffic types.

SDDC-VI-NET-006

Set the share value forvMotion traffic to Low.

During times of contention vMotion traffic is not asimportant as virtual machine or storage traffic.

During times of networkcontention vMotion's willtake longer then usual tocomplete.


VMware, Inc. 93

Table 2‑61. Network I/O Control Design Decisions (Continued)


SDDC-VI-NET-007

Set the share value forvSphere Replication traffic toLow.

During times of contention vSphere Replicationtraffic is not as important as virtual machine orstorage traffic.

During times of networkcontention vSphereReplication will takelonger and could violatethe defined SLA.

SDDC-VI-NET-008

Set the share value for vSANto High.

During times of contention vSAN traffic needsguaranteed bandwidth so virtual machineperformance does not suffer.

None.

SDDC-VI-NET-009

Set the share value forManagement to Normal.

By keeping the default setting of Normalmanagement traffic is prioritized higher thenvMotion and vSphere Replication but lower thenvSAN traffic. Management traffic is important as itensures the hosts can still be managed duringtimes of network contention.

None.

SDDC-VI-NET-010

Set the share value for NFSTraffic to Low.

Because NFS is used for secondary storage, suchas VDP backups and vRealize Log Insightarchives it is not as important as vSAN traffic, byprioritizing it lower vSAN is not impacted.

During times of contentionVDP backups will beslower than usual.

SDDC-VI-NET-011

Set the share value forvSphere Data ProtectionBackup traffic to Low.

During times of contention it is more important thatprimary functions of the SDDC continue to haveaccess to network resources over backup traffic.

During times of contentionVDP backups will beslower than usual.

SDDC-VI-NET-012

Set the share value for virtualmachines to High.

Virtual machines are the most important asset inthe SDDC. Leaving the default setting of Highensures that they will always have access to thenetwork resources they need.

None.

SDDC-VI-NET-013

Set the share value for FaultTolerance to Low.

Fault Tolerance is not used in this design thereforeit can be set to the lowest priority.

None.

SDDC-VI-NET-014

Set the share value for iSCSItraffic to Low.

iSCSI is not used in this design therefore it can beset to the lowest priority.

None.

VXLANVXLAN provides the capability to create isolated, multi-tenant broadcast domains across data centerfabrics, and enables customers to create elastic, logical networks that span physical network boundaries.

The first step in creating these logical networks is to abstract and pool the networking resources. Just asvSphere abstracts compute capacity from the server hardware to create virtual pools of resources thatcan be consumed as a service, vSphere Distributed Switch and VXLAN abstract the network into ageneralized pool of network capacity and separate the consumption of these services from the underlyingphysical infrastructure. A network capacity pool can span physical boundaries, optimizing computeresource utilization across clusters, pods, and geographically-separated data centers. The unified pool ofnetwork capacity can then be optimally segmented into logical networks that are directly attached tospecific applications.


VMware, Inc. 94

VXLAN works by creating Layer 2 logical networks that are encapsulated in standard Layer 3 IP packets.A Segment ID in every frame differentiates the VXLAN logical networks from each other without any needfor VLAN tags. As a result, large numbers of isolated Layer 2 VXLAN networks can coexist on a commonLayer 3 infrastructure.

In the vSphere architecture, the encapsulation is performed between the virtual NIC of the guest VM andthe logical port on the virtual switch, making VXLAN transparent to both the guest virtual machines andthe underlying Layer 3 network. Gateway services between VXLAN and non-VXLAN hosts (for example,a physical server or the Internet router) are performed by the NSX Edge Services Gateway appliance.The Edge gateway translates VXLAN segment IDs to VLAN IDs, so that non-VXLAN hosts cancommunicate with virtual machines on a VXLAN network.

The shared edge and compute cluster hosts all NSX Edge instances and all Universal Distributed LogicalRouter instances that are connect to the Internet or to corporate VLANs, so that the network administratorcan manage the environment in a more secure and centralized way.

Table 2‑62. VXLAN Design Decisions


SDDC-VI-Net-015

Use NSX for vSphere to introduceVXLANs for the use of virtual applicationnetworks and tenants networks.

Simplify the networkconfiguration for each tenant viacentralized virtual networkmanagement.

Requires additional compute andstorage resources to deploy NSXcomponents.

Additional training may beneeded on NSX.

SDDC-VI-Net-016

Use VXLAN along with NSX Edgegateways, the Universal DistributedLogical Router (UDLR) and DistributedLogical Router (DLR) to providecustomer/tenant network capabilities.

Create isolated, multi-tenantbroadcast domains across datacenter fabrics to create elastic,logical networks that spanphysical network boundaries.

Transport networks and MTUgreater than 1600 bytes has to beconfigured in the reachabilityradius.

SDDC-VI-Net-017

Use VXLAN along with NSX Edgegateways and the Universal DistributedLogical Router (UDLR) to providemanagement application networkcapabilities.

Leverage benefits of networkvirtualization in themanagement pod.

Requires installation andconfiguration of a NSX forvSphere instance in themanagement pod.

vMotion TCP/IP StackUse the vMotion TCP/IP stack to isolate traffic for vMotion and to assign a dedicated default gateway forvMotion traffic.

By using a separate TCP/IP stack, you can manage vMotion and cold migration traffic according to thetopology of the network, and as required for your organization.

n Route the traffic for the migration of virtual machines that are powered on or powered off by using adefault gateway that is different from the gateway assigned to the default stack on the host.

n Assign a separate set of buffers and sockets.

n Avoid routing table conflicts that might otherwise appear when many features are using a commonTCP/IP stack.


VMware, Inc. 95

n Isolate traffic to improve security.


SDDC-VI-Net-018

Use the vMotionTCP/IP stack forvMotion traffic.

By leveraging the vMotion TCP/IP stack,vMotion traffic can utilize a default gatewayon its own subnet, allowing for vMotiontraffic to go over Layer 3 networks.

The vMotion TCP/IP stack is not availablein the vDS VMkernel creation wizard, andas such the VMkernal adapter must becreated directly on a host.

NSX DesignThis design implements software-defined networking by using VMware NSX™ for vSphere®.With NSX forvSphere, virtualization delivers for networking what it has already delivered for compute and storage.

In much the same way that server virtualization programmatically creates, snapshots, deletes, andrestores software-based virtual machines (VMs), NSX network virtualization programmatically creates,snapshots, deletes, and restores software-based virtual networks. The result is a transformative approachto networking that not only enables data center managers to achieve orders of magnitude better agilityand economics, but also supports a vastly simplified operational model for the underlying physicalnetwork. NSX for vSphere is a nondisruptive solution because it can be deployed on any IP network,including existing traditional networking models and next-generation fabric architectures, from any vendor.

When administrators provision workloads, network management is one of the most time-consuming tasks.Most of the time spent provisioning networks is consumed configuring individual components in thephysical infrastructure and verifying that network changes do not affect other devices that are using thesame networking infrastructure.

The need to pre-provision and configure networks is a major constraint to cloud deployments wherespeed, agility, and flexibility are critical requirements. Pre-provisioned physical networks can allow for therapid creation of virtual networks and faster deployment times of workloads utilizing the virtual network.As long as the physical network that you need is already available on the host where the workload is tobe deployed, this works well. However, if the network is not available on a given host, you must find ahost with the available network and spare capacity to run your workload in your environment.

To get around this bottleneck requires a decoupling of virtual networks from their physical counterparts.This, in turn, requires that you can programmatically recreate all physical networking attributes that arerequired by workloads in the virtualized environment. Because network virtualization supports the creationof virtual networks without modification of the physical network infrastructure, it allows more rapid networkprovisioning.

NSX for vSphere DesignEach NSX instance is tied to a vCenter Server instance. The design decision to deploy two vCenterServer instances per region(SDDC-VI-VC-001) requires deployment of two separate NSX instances perregion.


VMware, Inc. 96

Table 2‑63. NSX for vSphere Design Decisions

Decision ID Design Decision Design Justification Design Implications

SDDC-VI-SDN-001

Use two separate NSX instancesper region. One instance is tied tothe Management vCenter Server,and the other instance is tied tothe Compute vCenter Server.

Software-defined Networking (SDN)capabilities offered by NSX, such as loadbalancing and firewalls, are crucial for thecompute/edge layer to support the cloudmanagement platform operations, and also forthe management applications in themanagement stack that need thesecapabilities.

You must install andperform initialconfiguration of multipleNSX instances separately.

SDDC-VI-SDN-002

Pair NSX Manager instances in aprimary-secondary relationshipacross regions for bothmanagement and computeworkloads.

NSX can extend the logical boundaries of thenetworking and security services acrossregions. As a result, workloads can be live-migrated and failed over between regionswithout reconfiguring the network and securityconstructs.

You must consider thatyou can pair up to eightNSX Manager instances.

Figure 2‑16. Architecture of NSX for vSphere

UniversalController

Cluster

UniversalController

Cluster

Management Cluster - Region A

Shared Edge and Compute Cluster - Region A

Shared Edge and Compute Cluster - Region B

Shared vCenter Single Sign-on Domain

NSX Manager Pairing

NSX Manager Pairing

ManagementvCenter Server

NSX ManagerPrimary

NSX ManagerPrimary

NSX Controller 1

NSX Controller 2

NSX Controller 3

NSX Controller 1

NSX Controller 2

NSX Controller 3

Compute / EdgevCenter Server

Management Cluster - Region B


NSX ManagerSecondary

NSX ManagerSecondary



VMware, Inc. 97

NSX ComponentsThe following sections describe the components in the solution and how they are relevant to the networkvirtualization design.

Consumption Layer

NSX for vSphere can be consumed by the cloud management platform (CMP), represented by vRealizeAutomation, by using the NSX REST API and the vSphere Web Client.

Cloud Management Platform

NSX for vSphere is consumed by vRealize Automation. NSX offers self-service provisioning of virtualnetworks and related features from a service portal. Details of the service requests and their orchestrationare outside the scope of this document and can be referenced in the Cloud Management Platform Designdocument.

API

NSX for vSphere offers a powerful management interface through its REST API.

n A client can read an object by making an HTTP GET request to the object’s resource URL.

n A client can write (create or modify) an object with an HTTP PUT or POST request that includes anew or changed XML document for the object.

n A client can delete an object with an HTTP DELETE request.

vSphere Web Client

The NSX Manager component provides a networking and security plug-in in the vSphere Web Client. Thisplug-in provides an interface to consuming virtualized networking from the NSX Manager for users thathave sufficient privileges.

Table 2‑64. Consumption Method Design Decisions


SDDC-VI-SDN-003

For the shared edge and computecluster NSX instance, end user accessis accomplished by using vRealizeAutomation services. Administratorsuse both the vSphere Web Client andthe NSX REST API.

vRealize Automation services areused for the customer-facing portal.The vSphere Web Client consumesNSX for vSphere resources throughthe Network and Security plug-in. TheNSX REST API offers the potential ofscripting repeating actions andoperations.

Customers typically interactonly indirectly with NSX fromthe vRealize Automation portal.Administrators interact withNSX from the vSphere WebClient and API.

SDDC-VI-SDN-004

For the management cluster NSXinstance, consumption is only byprovider staff via the vSphere WebClient and the API.

Ensures that infrastructurecomponents are not modified bytenants and/or non-provider staff.

Tenants do not have access tothe management stackworkloads.


VMware, Inc. 98

NSX Manager

NSX Manager provides the centralized management plane for NSX for vSphere and has a one-to-onemapping to vCenter Server workloads.

NSX Manager performs the following functions.

n Provides the single point of configuration and the REST API entry-points for NSX in a vSphereenvironment.

n Deploys NSX Controller clusters, Edge distributed routers, and Edge service gateways in the form ofOVF appliances, guest introspection services, and so on.

n Prepares ESXi hosts for NSX by installing VXLAN, distributed routing and firewall kernel modules,and the User World Agent (UWA).

n Communicates with NSX Controller clusters over REST and with hosts over the RabbitMQ messagebus. This internal message bus is specific to NSX for vSphere and does not require setup ofadditional services.

n Generates certificates for the NSX Controller instances and ESXi hosts to secure control planecommunications with mutual authentication.

NSX Controller

An NSX Controller performs the following functions.

n Provides the control plane to distribute VXLAN and logical routing information to ESXi hosts.

n Includes nodes that are clustered for scale-out and high availability.

n Slices network information across cluster nodes for redundancy.

n Removes requirement of VXLAN Layer 3 multicast in the physical network.

n Provides ARP suppression of broadcast traffic in VXLAN networks.

NSX control plane communication occurs over the management network.

Table 2‑65. NSX Controller Design Decision


SDDC-VI-SDN-005

Deploy NSX Controller instances inUniversal Cluster mode with threemembers to provide high availabilityand scale. Provision these threenodes through the primary NSXManager instance.

The high availability of NSXController reduces the downtimeperiod in case of failure of onephysical host.

The secondary NSX Manager willnot deploy controllers.

The controllers from the primaryNSX manager will manage allsecondary resources.

NSX VirtualSwitch

The NSX data plane consists of the NSX virtual switch. This virtual switch is based on the vSphereDistributed Switch (VDS)with additional components to enable rich services. The add-on NSXcomponents include kernel modules (VIBs) which run within the hypervisor kernel and provide servicessuch as distributed logical router (DLR) and distributed firewall (DFW), and VXLAN capabilities.


VMware, Inc. 99

The NSX virtual switch abstracts the physical network and provides access-level switching in thehypervisor. It is central to network virtualization because it enables logical networks that are independentof physical constructs such as VLAN. Using an NSX virtual switch includes several benefits.

n Supports overlay networking and centralized network configuration. Overlay networking enables thefollowing capabilities.

n Facilitates massive scale of hypervisors.

n Because the NSX virtual switch is based on VDS, it provides a comprehensive toolkit for trafficmanagement, monitoring, and troubleshooting within a virtual network through features such as portmirroring, NetFlow/IPFIX, configuration backup and restore, network health check, QoS, and more.

Logical Switching

NSX logical switches create logically abstracted segments to which tenant virtual machines can beconnected. A single logical switch is mapped to a unique VXLAN segment and is distributed across theESXi hypervisors within a transport zone. The logical switch allows line-rate switching in the hypervisorwithout the constraints of VLAN sprawl or spanning tree issues.

Distributed Logical Router

The NSX distributed logical router (DLR) is optimized for forwarding in the virtualized space, that is,forwarding between VMs on VXLAN- or VLAN-backed port groups. DLR has the following characteristics.

n High performance, low overhead first hop routing

n Scales with number of hosts

n Up to 1,000 Logical Interfaces (LIFs) on each DLR

Distributed LogicalRouter Control Virtual Machine

The distributed logical router control virtual machine is the control plane component of the routingprocess, providing communication between NSX Manager and the NSX Controller cluster through theUser World Agent (UWA). NSX Manager sends logical interface information to the control virtual machineand the NSX Controller cluster, and the control virtual machine sends routing updates to the NSXController cluster.

User World Agent

The User World Agent (UWA) is a TCP (SSL) client that facilitates communication between the ESXihosts and the NSX Controller instances as well as the retrieval of information from the NSX Manager viainteraction with the message bus agent.


VMware, Inc. 100

VXLAN Tunnel Endpoint

VXLAN Tunnel Endpoints (VTEPs) are instantiated within the vSphere Distributed Switch to which theESXi hosts that are prepared for NSX for vSphere are connected. VTEPs are responsible forencapsulating VXLAN traffic as frames in UDP packets and for the corresponding decapsulation. VTEPstake the form of one or more VMkernel ports with IP addresses and are used both to exchange packetswith other VTEPs and to join IP multicast groups via Internet Group Membership Protocol (IGMP). If youuse multiple VTEPs, then you must select a teaming method.

Edge Services Gateway

The NSX Edge services gateways (ESGs) primary function is north/south communication, but it alsooffers support for Layer 2, Layer 3, perimeter firewall, load balancing and other services such as SSL-VPN and DHCP-relay.

Distributed Firewall

NSX includes a distributed kernel-level firewall known as the distributed firewall. Security enforcement isdone at the kernel and VM network adapter level. The security enforcement implementation enablesfirewall rule enforcement in a highly scalable manner without creating bottlenecks on physical appliances.The distributed firewall has minimal CPU overhead and can perform at line rate.

The flow monitoring feature of the distributed firewall displays network activity between virtual machinesat the application protocol level. This information can be used to audit network traffic, define and refinefirewall policies, and identify botnets.

Logical Load Balancer

The NSX logical load balancer provides load balancing services up to Layer 7, allowing distribution oftraffic across multiple servers to achieve optimal resource utilization and availability. The logical loadbalancer is a service provided by the NSX Edge service gateway.

NSX for vSphere RequirementsNSX for vSphere requirements impact both physical and virtual networks.

Physical Network Requirements

Physical requirements determine the MTU size for networks that carry VLAN traffic, dynamic routingsupport, type synchronization through an NTP server, and forward and reverse DNS resolution.

Requirement Comments

Any network that carries VXLAN traffic must havean MTU size of 1600 or greater.

VXLAN packets cannot be fragmented. The MTU size must be large enoughto support extra encapsulation overhead.

This design uses jumbo frames, MTU size of 9000, for VXLAN traffic.

For the hybrid replication mode, Internet GroupManagement Protocol (IGMP) snooping must beenabled on the Layer 2 switches to which ESXihosts that participate in VXLAN are attached.IGMP querier must be enabled on the connectedrouter or Layer 3 switch.

IGMP snooping on Layer 2 switches is a requirement of the hybrid replicationmode. Hybrid replication mode is the recommended replication mode forbroadcast, unknown unicast, and multicast (BUM) traffic when deploying intoan environment with large scale-out potential. The traditional requirement forProtocol Independent Multicast (PIM) is removed.


VMware, Inc. 101

Requirement Comments

Dynamic routing support on the upstream Layer 3data center switches must be enabled.

Enable a dynamic routing protocol supported by NSX on the upstream datacenter switches to establish dynamic routing adjacency with the ESGs.

NTP server must be available. The NSX Manager requires NTP settings that synchronize it with the rest ofthe vSphere environment. Drift can cause problems with authentication. TheNSX Manager must be in sync with the vCenter Single Sign-On service on thePlatform Services Controller.

Forward and reverse DNS resolution for allmanagement VMs must be established.

The NSX Controller nodes do not require DNS entries.

NSX Component Specifications

The following table lists the components involved in the NSX for vSphere solution and the requirementsfor installing and running them. The compute and storage requirements have been taken into accountwhen sizing resources to support the NSX for vSphere solution.

Note NSX ESG sizing can vary with tenant requirements, so all options are listed.

VM vCPU Memory Storage Quantity per Stack Instance

NSX Manager 4 16 GB 60 GB 1

NSX Controller 4 4 GB 20 GB 3

NSX ESG 1 (Compact)

2 (Large)

4 (Quad Large)

6 (X-Large)

512 MB (Compact)

1 GB (Large)

1 GB (Quad Large)

8 GB (X-Large)

512 MB

512 MB

512 MB

4.5 GB (X-Large)

(+4 GB with swap)

Optional component. Deployment of the NSXESG varies per use case.

DLR control VM 1 512 MB 512 MB Optional component. Varies with use case.Typically 2 per HA pair.

Guest introspection 2 1 GB 4 GB Optional component. 1 per ESXi host.

NSX data security 1 512 MB 6 GB Optional component.1 per ESXi host.

NSX Edge Service Gateway Sizing

The Quad Large model is suitable for high performance firewall abilities and the X-Large is suitable forboth high performance load balancing and routing.

You can convert between NSX Edge service gateway sizes upon demand using a non-disruptive upgradeprocess, so the recommendation is to begin with the Large model and scale up if necessary. A Large NSXEdge service gateway is suitable for medium firewall performance but as detailed later, the NSX Edgeservice gateway does not perform the majority of firewall functions.

Note Edge service gateway throughput is influenced by the WAN circuit. An adaptable approach, that is,converting as necessary, is recommended.


VMware, Inc. 102

Table 2‑66. NSX Edge Service Gateway Sizing Design Decision


SDDC-VI-SDN-006 Use large size NSX Edgeservice gateways.

The large size provides all the performancecharacteristics needed even in the event of a failure.

A larger size would also provide the performancerequired but at the expense of extra resources thatwouldn't be used.

None.

Network Virtualization Conceptual DesignThis conceptual design provides you with an understanding of the network virtualization design.

The network virtualization conceptual design includes a perimeter firewall, a provider logical router, andthe NSX for vSphere Logical Router. It also includes the external network, internal tenant network, andinternal non-tenant network.

Note In this document, tenant refers to a tenant of the cloud management platform within thecompute/edge stack, or to a management application within the management stack.

Figure 2‑17. Conceptual Tenant Overview

VM VMVM VM VMVM

Internet MPLS

MgmtNetwork

External Network

Perimeter Firewall

Provider Logical Router(PLR) with Firewalling

NSX Logical Distributed Router (LDR)

Internal Tenant Networks (Logical Switches)

vNIC-Level Distributed Firewall


VMware, Inc. 103

The conceptual design has the following key components.

External Networks Connectivity to and from external networks is through the perimeter firewall.The main external network is the Internet.

Perimeter Firewall The physical firewall exists at the perimeter of the data center. Each tenantreceives either a full instance or partition of an instance to filter externaltraffic.

Provider Logical Router(PLR)

The PLR exists behind the perimeter firewall and handles north/south trafficthat is entering and leaving tenant workloads.

NSX forvSphereDistributedLogical Router (DLR)

This logical router is optimized for forwarding in the virtualized space, thatis, between VMs, on VXLAN port groups or VLAN-backed port groups.

Internal Non-TenantNetwork

A single management network, which sits behind the perimeter firewall butnot behind the PLR. Enables customers to manage the tenantenvironments.

Internal TenantNetworks

Connectivity for the main tenant workload. These networks are connectedto a DLR, which sits behind the PLR. These networks take the form ofVXLAN-based NSX for vSphere logical switches. Tenant virtual machineworkloads will be directly attached to these networks.

Cluster Design for NSX for vSphereFollowing the vSphere design, the NSX for vSphere design consists of a management stack and acompute/edge stack in each region.

Management Stack

In the management stack, the underlying hosts are prepared for NSX for vSphere. The managementstack has these components.

n NSX Manager instances for both stacks (management stack and compute/edge stack)

n NSX Controller cluster for the management stack

n NSX ESG and DLR control VMs for the management stack

Compute/Edge Stack

In the compute/edge stack, the underlying hosts are prepared for NSX for vSphere. The compute/edgestack has these components.

n NSX Controller cluster for the compute stack.

n All NSX Edge service gateways and DLR control VMs of the compute stack that are dedicated tohandling the north/south traffic in the data center. A shared edge and compute stack helps preventVLAN sprawl because any external VLANs need only be trunked to the hosts in this cluster.


VMware, Inc. 104

Table 2‑67. vSphere Cluster Design Decisions


SDDC-VI-SDN-007

For the compute stack, donot use a dedicated edgecluster.

Simplifies configuration andminimizes the number of hostsrequired for initial deployment.

The NSX Controller instances, NSX Edgeservices gateways, and DLR control VMs ofthe compute stack are deployed in theshared edge and compute cluster.

The shared nature of the cluster will requirethe cluster to be scaled out as computeworkloads are added so as to not impactnetwork performance.

SDDC-VI-SDN-008

For the management stack,do not use a dedicated edgecluster.

The number of supportedmanagement applications doesnot justify the cost of a dedicatededge cluster in the managementstack.

The NSX Controller instances, NSX Edgeservice gateways, and DLR control VMs ofthe management stack are deployed in themanagement cluster.

SDDC-VI-SDN-009

Apply vSphere DistributedResource Scheduler (DRS)anti-affinity rules to the NSXcomponents in both stacks.

Using DRS prevents controllersfrom running on the same ESXihost and thereby risking theirhigh availability capability.

Additional configuration is required to set upanti-affinity rules.

The logical design of NSX considers the vCenter Server clusters and define the place where each NSXcomponent runs.


VMware, Inc. 105

Figure 2‑18. Cluster Design for NSX for vSphere

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OS

APP

OS

APP

OSAPP

OS

ESXi ESXi

APP

OSAPP

OS

APP

OSAPP

OS


NSXController

(Mgmt)


NSXEdge

(Mgmt)

NSXManager(Mgmt)

NSXManager

(Compute)

NSXEdge

(Compute)



SDDCPayload






Management Cluster






vCenterServer(Mgmt)

vCenterServer

(Compute)

High Availability of NSX for vSphere Components

The NSX Manager instances of both stacks run on the management cluster. vSphere HA protects theNSX Manager instances by ensuring that the NSX Manager VM is restarted on a different host in theevent of primary host failure.

The NSX Controller nodes of the management stack run on the management cluster. The NSX forvSphere Controller nodes of the compute stack run on the shared edge and compute cluster. In bothclusters, vSphere Distributed Resource Scheduler (DRS) rules ensure that NSX for vSphere Controllernodes do not run on the same host.

The data plane remains active during outages in the management and control planes although theprovisioning and modification of virtual networks is impaired until those planes become available again.


VMware, Inc. 106

The NSX Edge service gateways and DLR control VMs of the compute stack are deployed on the sharededge and compute cluster. The NSX Edge service gateways and DLR control VMs of the managementstack run on the management cluster.

NSX Edge components that are deployed for north/south traffic are configured in equal-cost multi-path(ECMP) mode that supports route failover in seconds. NSX Edge components deployed for loadbalancing utilize NSX HA. NSX HA provides faster recovery than vSphere HA alone because NSX HAuses an active/passive pair of NSX Edge devices. By default the passive Edge device becomes activewithin 15 seconds. All NSX Edge devices are also protected by vSphere HA.

Scalability of NSX Components

A one-to-one mapping between NSX Manager instances and vCenter Server instances exists. If theinventory of either the management stack or the compute stack exceeds the limits supported by a singlevCenter Server, then you can deploy a new vCenter Server instance, and must also deploy a new NSXManager instance. You can extend transport zones by adding more shared edge and compute andcompute clusters until you reach the vCenter Server limits. Consider the limit of 100 DLRs per ESXi hostalthough the environment usually would exceed other vCenter Server limits before the DLR limit.

vSphere Distributed Switch Uplink ConfigurationEach ESXi host utilizes two physical 10 Gb Ethernet adapters, associated with the uplinks on the vSphereDistributed Switches to which it is connected. Each uplink is connected to a different top-of-rack switch tomitigate the impact of a single top-of-rack switch failure and to provide two paths in and out of the SDDC.

Table 2‑68. VTEP Teaming and Failover Configuration Design Decision


SDDC-VI-SDN-010

Set up VXLAN TunnelEndpoints (VTEPs) to useRoute based on SRC-ID forteaming and failoverconfiguration.

Allows for the utilization of thetwo uplinks of the vDS resulting inbetter bandwidth utilization andfaster recovery from network pathfailures.

Link aggregation such as LACPbetween the top-of-rack (ToR) switchesand ESXi host must not be configured inorder to allow dynamic routing to peerbetween the ESGs and the upstreamswitches.

Logical Switch Control Plane Mode DesignThe control plane decouples NSX for vSphere from the physical network and handles the broadcast,unknown unicast, and multicast (BUM) traffic within the logical switches. The control plane is on top of thetransport zone and is inherited by all logical switches that are created within it. It is possible to overrideaspects of the control plane.


VMware, Inc. 107

The following options are available.

Multicast Mode The control plane uses multicast IP addresses on the physical network.Use multicast mode only when upgrading from existing VXLANdeployments. In this mode, you must configure PIM/IGMP on the physicalnetwork.

Unicast Mode The control plane is handled by the NSX Controllers and all replicationoccurs locally on the host. This mode does not require multicast IPaddresses or physical network configuration.

Hybrid Mode This mode is an optimized version of the unicast mode where local trafficreplication for the subnet is offloaded to the physical network. Hybrid moderequires IGMP snooping on the first-hop switch and access to an IGMPquerier in each VTEP subnet. Hybrid mode does not require PIM.

Figure 2‑19. Logical Switch Control Plane in Hybrid Mode

VM1 VM2

vSphere Distributed Switch

ESXi Host

VXLAN 5001

VTEP1 10.20.10.10

ESXi HostVTEP2 10.20.10.11

VM3


VM4


MTEP VTEP MTEP VTEP

L2 - IGMP L2 - IGMP

Multicast TrafficUnicast Traffic

ControllerCluster

Legend:

VXLAN Transport Network

This design uses hybrid mode for control plane replication.

Table 2‑69. Logical Switch Control Plane Mode Design Decision


SDDC-VI-SDN-011 Use hybrid mode forcontrol planereplication.

Offloading multicast processing to the physicalnetwork reduces pressure on VTEPs as theenvironment scales out. For largeenvironments, hybrid mode is preferable tounicast mode. Multicast mode is used onlywhen migrating from existing VXLAN solutions.

IGMP snooping must be enabledon the ToR physical switch andan IGMP querier must beavailable.


VMware, Inc. 108

Transport Zone DesignA transport zone is used to define the scope of a VXLAN overlay network and can span one or moreclusters within one vCenter Server domain. One or more transport zones can be configured in an NSX forvSphere solution. A transport zone is not meant to delineate a security boundary.

Table 2‑70. Transport Zones Design Decisions


SDDC-VI-SDN-012

For the compute stack, use auniversal transport zone thatencompasses all shared edgeand compute, and computeclusters from all regions forworkloads that require mobilitybetween regions.

A Universal Transport zone supportsextending networks and securitypolicies across regions. This allowsseamless migration of applicationsacross regions either by cross vCentervMotion or by failover recovery withSite Recovery Manager.

vRealize Automation is not able todeploy on demand network objectsagainst a Secondary NSX Manager.You must consider that you can pairup to eight NSX Manager instances.If the solution grows past eight NSXManager instances, you must deploya new primary manager and newtransport zone.

SDDC-VI-SDN-013

For the compute stack, use aglobal transport zone in eachregion that encompasses allshared edge and compute, andcompute clusters for use withvRealize Automation on demandnetwork provisioning.

NSX Managers with a role ofSecondary can not deploy Universalobjects. To allow all regions to deployon demand network objects a globaltransport zone is required.

Shared Edge and Compute, andCompute Pods have two transportzones.

SDDC-VI-SDN-014

For the management stack, use asingle universal transport zonethat encompasses allmanagement clusters.

A single Universal Transport zonesupports extending networks andsecurity policies across regions. Thisallows seamless migration of themanagement applications acrossregions either by cross-vCentervMotion or by failover recovery withSite Recovery Manager.

You must consider that you can pairup to eight NSX Manager instances.If the solution grows past eight NSXManager instances, you must deploya new primary manager and newtransport zone.

Routing DesignThe routing design considers different levels of routing within the environment from which to define a setof principles for designing a scalable routing solution.

North/south The Provider Logical Router (PLR) handles the north/south traffic to andfrom a tenant and management applications inside of application virtualnetworks.

East/west Internal east/west routing at the layer beneath the PLR deals with theapplication workloads.


VMware, Inc. 109

Table 2‑71. Routing Model Design Decisions


SDDC-VI-SDN-017

Deploy NSX Edge ServicesGateways in an ECMPconfiguration for north/southrouting in both management andshared edge and computeclusters.

The NSX ESG is the recommendeddevice for managing north/south traffic.Using ECMP provides multiple paths inand out of the SDDC. This results infaster failover times than deployingEdge service gateways in HA mode.

ECMP requires 2 VLANS for uplinkswhich adds an additional VLAN overtraditional HA ESG configurations.

SDDC-VI-SDN-018

Deploy a single NSX UDLR forthe management cluster toprovide east/west routing acrossall regions.

Using the UDLR reduces the hop countbetween nodes attached to it to 1. Thisreduces latency and improvesperformance.

UDLRs are limited to 1,000 logicalinterfaces. When that limit isreached, a new UDLR must bedeployed.

SDDC-VI-SDN-019

Deploy a single NSX UDLR forthe shared edge and compute,and compute clusters to provideeast/west routing across allregions for workloads thatrequire mobility across regions.

Using the UDLR reduces the hop countbetween nodes attached to it to 1. Thisreduces latency and improvesperformance.

UDLRs are limited to 1,000 logicalinterfaces. When that limit isreached a new UDLR must bedeployed.

SDDC-VI-SDN-020

Deploy a DLR for the sharededge and compute and computeclusters to provide east/westrouting for workloads that requireon demand network objects fromvRealize Automation.

Using the DLR reduces the hop countbetween nodes attached to it to 1. Thisreduces latency and improvesperformance.

DLRs are limited to 1,000 logicalinterfaces. When that limit isreached a new DLR must bedeployed.

SDDC-VI-SDN-021

Deploy all NSX UDLRs withoutthe local egress option enabled.

When local egress is enabled, control ofingress traffic, is also necessary (forexample using NAT). This becomeshard to manage for little to no benefit.

All north/south traffic is routedthrough Region A until those routesare no longer available. At that time,all traffic dynamically changes toRegion B.

SDDC-VI-SDN-022

Use BGP as the dynamic routingprotocol inside the SDDC.

Using BGP as opposed to OSPF easesthe implementation of dynamic routing.There is no need to plan and designaccess to OSPF area 0 inside theSDDC. OSPF area 0 varies based oncustomer configuration.

BGP requires configuring each ESGand UDLR with the remote routerthat it exchanges routes with.

SDDC-VI-SDN-023

Configure BGP Keep Alive Timerto 1 and Hold Down Timer to 3between the UDLR and all ESGsthat provide north/south routing.

With Keep Alive and Hold Timersbetween the UDLR and ECMP ESGsset low, a failure is detected quicker,and the routing table is updated faster.

If an ESXi host becomes resourceconstrained, the ESG running onthat host might no longer be usedeven though it is still up.


VMware, Inc. 110

Table 2‑71. Routing Model Design Decisions (Continued)


SDDC-VI-SDN-024

Configure BGP Keep Alive Timerto 4 and Hold Down Timer to 12between the ToR switches andall ESGs providing north/southrouting.

This provides a good balance betweenfailure detection between the ToRs andthe ESGs and overburdening the ToRswith keep alive traffic.

By using longer timers to detectwhen a router is dead, a deadrouter stays in the routing tablelonger and continues to send trafficto a dead router.

SDDC-VI-SDN-025

Create one or more static routeson ECMP enabled edges forsubnets behind the UDLR andDLR with a higher admin costthen the dynamically learnedroutes.

When the UDLR or DLR control VM failsover router adjacency is lost and routesfrom upstream devices such as ToR's tosubnets behind the UDLR are lost.

This requires each ECMP edgedevice be configured with staticroutes to the UDLR or DLR. If anynew subnets are added behind theUDLR or DLR the routes must beupdated on the ECMP edges.

Transit Network and Dynamic Routing

Dedicated networks are needed to facilitate traffic between the universal dynamic routers and edgegateways, and to facilitate traffic between edge gateways and the top of rack switches. These networksare used for exchanging routing tables and for carrying transit traffic.

Table 2‑72. Transit Network Design Decisions


SDDC-VI-SDN-026

Create a universal virtual switch for use as thetransit network between the UDLR and ESGs.The UDLR provides east/west routing in bothcompute and management stacks while theESG's provide north/south routing.

The universal virtual switchallows the UDLR and all ESGsacross regions to exchangerouting information.

Only the primary NSXManager can create andmanage universal objectsincluding this UDLR.

SDDC-VI-SDN-027

Create a global virtual switch in each region foruse as the transit network between the DLR andESG's. The DLR provides east/west routing in thecompute stack while the ESG's providenorth/south routing.

The global virtual switch allowsthe DLR and ESGs in eachregion to exchange routinginformation.

A global virtual switch foruse as a transit network isrequired in each region.

SDDC-VI-SDN-028

Create two VLANs in each region. Use thoseVLANs to enable ECMP between the north/southESGs and the ToR switches.

The ToR’s have an SVI on one of the two VLANSand each north/south ESG has an interface oneach VLAN.

This enables the ESGs to havemultiple equal-cost routes andprovides more resiliency andbetter bandwidth utilization inthe network.

Extra VLANs arerequired.

Firewall Logical DesignThe NSX Distributed Firewall is used to protect all management applications attached to applicationvirtual networks. To secure the SDDC, only other solutions in the SDDC and approved administration IPscan directly communicate with individual components. External facing portals are accessible via a loadbalancer virtual IP (VIP). This simplifies the design by having a single point of administration for all firewallrules. The firewall on individual ESGs is set to allow all traffic. An exception are ESGs that provide ECMPservices, which require the firewall to be disabled.


VMware, Inc. 111

Table 2‑73. Firewall Design Decisions


SDDC-VI-SDN-029

For all ESGs deployed as loadbalancers, set the default firewallrule to allow all traffic.

Restricting and granting access ishandled by the distributed firewall. Thedefault firewall rule does not have to doit.

Explicit rules to allow access tomanagement applications mustbe defined in the distributedfirewall.

SDDC-VI-SDN-030

For all ESGs deployed as ECMPnorth/south routers, disable thefirewall.

Use of ECMP on the ESGs is arequirement. Leaving the firewallenabled, even in allow all traffic mode,results in sporadic network connectivity.

Services such as NAT and loadbalancing can not be usedwhen the firewall is disabled.

SDDC-VI-SDN-031

Configure the Distributed Firewallto limit access to administrativeinterfaces in the managementcluster.

To ensure only authorized administratorscan access the administrative interfacesof management applications.

Maintaining firewall rules addsadministrative overhead.

Load Balancer Design

The ESG implements load balancing within NSX for vSphere.

The ESG has both a Layer 4 and a Layer 7 engine that offer different features, which are summarized inthe following table.

Feature Layer 4 Engine Layer 7 Engine

Protocols TCP TCP

HTTP

HTTPS (SSL Pass-through)

HTTPS (SSL Offload)

Load balancing method Round Robin

Source IP Hash

Least Connection

Round Robin

Source IP Hash

Least Connection

URI

Health checks TCP TCP

HTTP (GET, OPTION, POST)

HTTPS (GET, OPTION, POST)

Persistence (keeping clientconnections to the sameback-end server)

TCP: SourceIP TCP: SourceIP, MSRDP

HTTP: SourceIP, Cookie

HTTPS: SourceIP, Cookie, ssl_session_id

Connection throttling No Client Side: Maximum concurrent connections, Maximum newconnections per second

Server Side: Maximum concurrent connections

High availability Yes Yes


VMware, Inc. 112

Feature Layer 4 Engine Layer 7 Engine

Monitoring View VIP (Virtual IP), Pool andServer objects and stats via CLIand API

View global stats for VIP sessionsfrom the vSphere Web Client

View VIP, Pool and Server objects and statistics by using CLIand API

View global statistics about VIP sessions from the vSphereWeb Client

Layer 7 manipulation No URL block, URL rewrite, content rewrite

Table 2‑74. NSX for vSphere Load Balancer Design Decisions


SDDC-VI-SDN-032

Use the NSX load balancer. The NSX load balancer can support theneeds of the management applications.Using another load balancer would increasecost and add another component to bemanaged as part of the SDDC.

None.

SDDC-VI-SDN-033

Use an NSX load balancer inHA mode for all managementapplications.

All management applications that require aload balancer are on a single virtual wire,having a single load balancer keeps thedesign simple.

One management applicationowner could make changes tothe load balancer that impactanother application.

SDDC-VI-SDN-034

Use an NSX load balancer inHA mode for the PlatformServices Controllers.

Using a load balancer increases theavailability of the PSC’s for all applications.

Configuring the PlatformServices Controllers and theNSX load balancer addsadministrative overhead.

Information Security and Access ControlYou use a service account for authentication and authorization of NSX Manager for virtual networkmanagement.

Table 2‑75. Authorization and Authentication Management Design Decisions


SDDC-VI-SDN-035

Configure a service accountsvc-nsxmanager in vCenterServer for application-to-application communicationfrom NSX Manager withvSphere.

Provides the following access control features:n NSX Manager accesses vSphere with the

minimum set of permissions that are requiredto perform lifecycle management of virtualnetworking objects.

n In the event of a compromised account, theaccessibility in the destination applicationremains restricted.

n You can introduce improved accountability intracking request-response interactionsbetween the components of the SDDC.

You must maintain theservice account's lifecycle outside of theSDDC stack to ensure itsavailability

SDDC-VI-SDN-036

Use global permissions whenyou create the svc-nsxmanager service accountin vCenter Server.

n Simplifies and standardizes the deployment ofthe service account across all vCenter Serverinstances in the same vSphere domain.

n Provides a consistent authorization layer.

All vCenter Serverinstances must be in thesame vSphere domain.


VMware, Inc. 113

Bridging Physical WorkloadsNSX for vSphere offers VXLAN to Layer 2 VLAN bridging capabilities with the data path contained entirelyin the ESXi hypervisor. The bridge runs on the ESXi host where the DLR control VM is located. Multiplebridges per DLR are supported.

Table 2‑76. Virtual to Physical Interface Type Design Decision


SDDC-VI-SDN-037

Place all virtual machines, bothmanagement and tenant, on VXLAN-backed networks unless you must satisfyan explicit requirement to use VLAN-backed port groups for these virtualmachines. If VLAN-backed port groupsare required, connect physical workloadsthat need to communicate to virtualizedworkloads to routed VLAN LIFs on a DLR.

Bridging and routing are not possible onthe same logical switch. As a result, itmakes sense to attach a VLAN LIF to adistributed router or ESG and routebetween the physical and virtualmachines. Use bridging only where virtualmachines need access only to thephysical machines on the same Layer 2.

Access to physicalworkloads is routed viathe DLR or ESG.

Region ConnectivityRegions must be connected to each other. Connection types could be point-to-point links, MPLS, VPNTunnels, etc. This connection will vary by customer and is out of scope for this design.

The region interconnectivity design must support jumbo frames, and ensure latency is less then 150 ms.For more details on the requirements for region interconnectivity see the Cross-VC NSX Design Guide.

Table 2‑77. Inter-Site Connectivity Design Decisions

Decision ID Design Decision Design JustificationDesignImplications

SDDC-VI-SDN-038

Provide a connectionbetween regions that iscapable of routing betweeneach pod.

When NSX is configured for cross-vCenter to enableuniversal objects, connectivity between NSX managers,ESXi host VTEPs and NSX controllers to ESXi hostsmanagement interface is required.

To support cross-region authentication, the vCenterServer and Platform Services Controller design requires asingle vCenter Single Sign-On domain.

Portability of management and compute workloadsrequires connectivity between regions.

Jumbo frames arerequired acrossregions.

SDDC-VI-SDN-039

Ensure that the latencybetween regions is lessthan 150 ms.

A latency below 150 ms is required for the followingfeatures.n Cross-vCenter vMotionn The NSX design for the SDDC

None.


VMware, Inc. 114

https://communities.vmware.com/servlet/JiveServlet/downloadBody/32552-102-2-44455/Multi-site%20Options%20and%20Cross-VC%20NSX%20Design%20Guide.pdf

Application Virtual NetworkManagement applications, such as VMware vRealize Automation, VMware vRealize Operations Manager,or VMware vRealize Orchestrator, leverage a traditional 3-tier client/server architecture with apresentation tier (user interface), functional process logic tier, and data tier. This architecture requires aload balancer for presenting end-user facing services.

Table 2‑78. Isolated Management Applications Design Decisions


SDDC-VI-SDN-040

Place the following managementapplications on an application virtualnetwork.n vRealize Automationn vRealize Automation Proxy Agentsn vRealize Businessn vRealize Business collectorsn vRealize Orchestratorn vRealize Operations Managern vRealize Operations Manager remote

collectorsn vRealize Log Insightn Update Manager Download Service

Access to themanagement applicationsis only through publishedaccess points.

The application virtual network isfronted by an NSX Edge device forload balancing and the distributedfirewall to isolate applications fromeach other and external users. Directaccess to application virtual networksis controlled by distributed firewallrules.

SDDC-VI-SDN-041

Create three application virtual networks.n Each region has a dedicated application

virtual network for managementapplications in that region that do notrequire failover.

n One application virtual network isreserved for management applicationfailover between regions.

Using only threeapplication virtualnetworks simplifies thedesign by sharing Layer 2networks with applicationsbased on their needs.

A single /24 subnet is used for eachapplication virtual network. IPmanagement becomes critical toensure no shortage of IP addresseswill appear in the future.

Table 2‑79. Portable Management Applications Design Decision


SDDC-VI-SDN-042 The following management applicationsmust be easily portable between regions.n vRealize Automationn vRealize Orchestratorn vRealize Businessn vRealize Operations Manager

Management applications mustbe easily portable betweenregions without requiringreconfiguration.

Unique addressing isrequired for all managementapplications.

Having software-defined networking based on NSX in the management stack makes all NSX featuresavailable to the management applications.

This approach to network virtualization service design improves security and mobility of the managementapplications, and reduces the integration effort with existing customer networks.


VMware, Inc. 115

Figure 2‑20. Virtual Application Network Components and Design

VC

OSPSC

OSSRM

OSVDP

OS

ECMPESGs

ToRSwitches

Internet/EnterpriseNetwork

Mgmt-Management

Compute-Management

Legend:

Shared Compute and

Edge Pod

192.168.11/24

Transit Networks

Management Application

vRLIvROps CollectorvRA Proxy

vRA/vRO/vRBvROps


ESGLoadBalancer


192.168.31/24


Ext-Management

UMDS

Certain configuration choices might later facilitate the tenant onboarding process.

n Create the primary NSX ESG to act as the tenant PLR and the logical switch that forms the transitnetwork for use in connecting to the UDLR.

n Connect the primary NSX ESG uplinks to the external networks

n Connect the primary NSX ESG internal interface to the transit network.

n Create the NSX UDLR to provide routing capabilities for tenant internal networks and connect theUDLR uplink to the transit network.

n Create any tenant networks that are known up front and connect them to the UDLR.


VMware, Inc. 116

Virtual Network Design ExampleThe virtual network design example illustrates an implementation for a management application virtualnetwork.

Figure 2‑21 shows an example for implementing a management application virtual network. The exampleservice is vRealize Automation, but any other 3-tier application would look similar.

Figure 2‑21. Detailed Example for vRealize Automation Networking

VC

OSPSC

OSSRM

OSVDP

OS

ECMPESGs

ToRSwitches


Mgmt-Management

Compute-Management

Legend:

Shared Compute and

Edge Pod

192.168.11/24

Transit Networks


vRLIvROps CollectorvRA Proxy

vRA/vRO/vRBvROps


ESGLoadBalancer


192.168.31/24


Ext-Management

UMDS


VMware, Inc. 117

The example is set up as follows.

n You deploy vRealize Automation on the application virtual network that is used to fail overapplications between regions. This network is provided by a VXLAN virtual wire (orange network in Figure 2‑21).

n The network that is used by vRealize Automation connects to external networks through NSX forvSphere. NSX ESGs and the UDLR route traffic between the application virtual networks and thepublic network.

n Services such as a Web GUI, which must be available to the end users of vRealize Automation, areaccessible via the NSX Edge load balancer.

The following table shows an example of a mapping from application virtual networks to IPv4 subnets.The actual mapping depends on the customer environment and is based on available IP subnets.

Note The following IP ranges are an example. Your actual implementation depends on yourenvironment.

Application Virtual Network Management Applications Internal IPv4 Subnet

Mgmt-xRegion01-VXLAN vRealize Automation (includes vRealize Orchestrator and vRealizeBusiness)

vRealize Operations Manager

192.168.11.0/24

Mgmt-RegionA01-VXLAN vRealize Log Insight

vRealize Operations Manager Remote Collectors

vRealize Automation Proxy Agents

192.168.31.0/24

Mgmt-RegionB01-VXLAN vRealize Log Insight

vRealize OperationsManager Remote Collectors

vRealize Automation Proxy Agents

192.168.32.0/24

Use of Secure Sockets Layer (SSL) CertificatesBy default NSX Manager uses a self signed SSL certificate. By default, this certificate is not trusted byend-user devices or browsers. It is a security best practice to replace these certificates with certificatesthat are signed by a third-party or enterprise Certificate Authority (CA).


SDDC-VI-SDN-043

Replace the NSX Manager certificatewith a certificate signed by a 3rd partyPublic Key Infrastructure.

Ensures communication between NSXadmins and the NSX Manager areencrypted by a trusted certificate.

Replacing and managingcertificates is anoperational overhead.

Shared Storage DesignThe shared storage design includes design decisions for vSAN storage and NFS storage.

Well-designed shared storage provides the basis for an SDDC and has the following benefits.

n Prevents unauthorized access to business data


VMware, Inc. 118

n Protects data from hardware and software failures

n Protects data from malicious or accidental corruption

Follow these guidelines when designing shared storage for your environment.

n Optimize the storage design to meet the diverse needs of applications, services, administrators, andusers.

n Strategically align business applications and the storage infrastructure to reduce costs, boostperformance, improve availability, provide security, and enhance functionality.

n Provide multiple tiers of storage to match application data access to application requirements.

n Design each tier of storage with different performance, capacity, and availability characteristics.Because not every application requires expensive, high-performance, highly available storage,designing different storage tiers reduces cost.

Shared Storage PlatformYou can choose between traditional storage, VMware vSphere Virtual Volumes, and VMware vSANstorage.

Storage Types

Traditional Storage Fibre Channel, NFS, and iSCSI are mature and viable options to supportvirtual machine needs.

VMware vSAN Storage vSAN is a software-based distributed storage platform that combines thecompute and storage resources of VMware ESXi hosts. When you designand size a vSAN cluster, hardware choices are more limited than fortraditional storage.

VMware vSphere VirtualVolumes

This design does not leverage VMware vSphere Virtual Volumes becauseVirtual Volumes does not support Site Recovery Manager.

Traditional Storage and vSAN Storage

Fibre Channel, NFS, and iSCSI are mature and viable options to support virtual machine needs.

Your decision to implement one technology or another can be based on performance and functionality,and on considerations like the following:

n The organization’s current in-house expertise and installation base

n The cost, including both capital and long-term operational expenses

n The organization’s current relationship with a storage vendor

vSAN is a software-based distributed storage platform that combines the compute and storage resourcesof ESXi hosts. It provides a simple storage management experience for the user. This solution makessoftware-defined storage a reality for VMware customers. However, you must carefully considersupported hardware options when sizing and designing a vSAN cluster.


VMware, Inc. 119

Storage Type Comparison

ESXi hosts support a variety of storage types. Each storage type supports different vSphere features.

Table 2‑80. Network Shared Storage Supported by ESXi Hosts

Technology Protocols Transfers Interface

Fibre Channel FC/SCSI Block access of data/LUN Fibre Channel HBA

Fibre Channel over Ethernet FCoE/SCSI Block access of data/LUN Converged network adapter (hardware FCoE)

NIC with FCoE support (software FCoE)

iSCSI IP/SCSI Block access of data/LUN iSCSI HBA or iSCSI enabled NIC (hardware iSCSI)

Network Adapter (software iSCSI)

NAS IP/NFS File (no direct LUN access) Network adapter

vSAN IP Block access of data Network adapter

Table 2‑81. vSphere Features Supported by Storage Type

TypevSpherevMotion Datastore

Raw DeviceMapping (RDM)

Application orBlock-levelClustering HA/DRS

Storage APIsData Protection

Local Storage Yes VMFS No Yes No Yes

Fibre Channel /Fibre Channel overEthernet

Yes VMFS Yes Yes Yes Yes

iSCSI Yes VMFS Yes Yes Yes Yes

NAS over NFS Yes NFS No No Yes Yes

vSAN Yes vSAN No Yes (via iSCSIInitiator)

Yes Yes

Shared Storage Logical DesignThe shared storage design selects the appropriate storage device for each type of cluster.

The storage devices for use by each type of cluster are as follows.

n Management clusters use vSAN for primary storage and NFS for secondary storage.

n Shared edge and compute clusters can use FC/FCoE, iSCSI, NFS, or vSAN storage. No specificguidance is given as user workloads and other factors determine storage type and SLA for userworkloads.


VMware, Inc. 120

Figure 2‑22. Logical Storage Design

Virtual Appliance

Virtual Appliance

Virtual Appliance

Virtual Appliance

Virtual Appliance

Virtual Appliance APP

OSAPPOS

APPOS

Tenant n

Management Cluster Shared Edge and Compute Cluster

Tenant 1

ESXi Host ESXi Host

Shared Datastores

Mgmt Monitoring Portals

Shared Datastores

PayloadsSLA 1

PayloadsSLA 2

PayloadsSLA N

Software-Defined Storage

Policy-Based Storage ManagementVirtualized Data Services

Hypervisor Storage Abstraction

SAN or NAS or DAS(3rd party or VMware vSAN)

Physical Disks

SSD FC15K FC10K SATA SSD FC15K FC10K SATA

VMDKs

Swap Files + Logs

1500GB

200GB

2048GB

Sample LUN

Table 2‑82. Storage Type Design Decisions


SDDC-VI-Storage-001

In the management cluster, usevSAN and NFS shared storage:n Use vSAN as the primary

shared storage platform.n Use NFS as the secondary

shared storage platform for themanagement cluster.

vSAN as the primary shared storage solutioncan take advantage of more cost-effectivelocal storage.

NFS is used primarily for archival and theneed to maintain historical data. LeveragingNFS provides large, low cost volumes thathave the flexibility to be expanded on aregular basis depending on capacity needs.

The use of two differentstorage technologiesincreases thecomplexity andoperational overhead.

SDDC-VI-Storage-002

In all clusters, ensure that at least20% of free space is alwaysavailable on all non-vSANdatastores.

If the datastore runs out of free space,applications and services within the SDDC,including but not limited to the NSX Edge corenetwork services, the provisioning portal andVDP backups, will fail. To prevent this,maintain adequate free space.

Monitoring and capacitymanagement arecritical, and must beproactively performed.


VMware, Inc. 121

Storage Tiering

Today’s enterprise-class storage arrays contain multiple drive types and protection mechanisms. Thestorage, server, and application administrators face challenges when selecting the correct storageconfiguration for each application being deployed in the environment. Virtualization can make thisproblem more challenging by consolidating many different application workloads onto a small number oflarge devices. Given this challenge, administrators might use single storage type for every type ofworkload without regard to the needs of the particular workload. However, not all application workloadshave the same requirements, and storage tiering allows for these differences by creating multiple levels ofstorage with varying degrees of performance, reliability and cost, depending on the application workloadneeds.

The most mission-critical data typically represents the smallest amount of data and offline data representsthe largest amount. Details differ for different organizations.

To determine the storage tier for application data, determine the storage characteristics of the applicationor service.

n I/O operations per second (IOPS) requirements

n Megabytes per second (MBps) requirements

n Capacity requirements

n Availability requirements

n Latency requirements

After you determine the information for each application, you can move the application to the storage tierwith matching characteristics.

n Consider any existing service-level agreements (SLAs).

n Move data between storage tiers during the application life cycle as needed.

VMware Hardware Acceleration API/CLI for Storage

The VMware Hardware Acceleration API/CLI for storage (previously known as vStorage APIs for ArrayIntegration or VAAI), supports a set of ESXCLI commands for enabling communication between ESXihosts and storage devices. The APIs define a set of storage primitives that enable the ESXi host tooffload certain storage operations to the array. Offloading the operations reduces resource overhead onthe ESXi hosts and can significantly improve performance for storage-intensive operations such asstorage cloning, zeroing, and so on. The goal of hardware acceleration is to help storage vendors providehardware assistance to speed up VMware I/O operations that are more efficiently accomplished in thestorage hardware.

Without the use of VAAI, cloning or migration of virtual machines by the VMkernel data mover involvessoftware data movement. The data mover issues I/O to read and write blocks to and from the source anddestination datastores. With VAAI, the data mover can use the API primitives to offload operations to thearray when possible. For example, when you copy a virtual machine disk file (VMDK file) from one


VMware, Inc. 122

datastore to another inside the same array, the data mover directs the array to make the copy completelyinside the array. If you invoke a data movement operation and the corresponding hardware offloadoperation is enabled, the data mover first attempts to use hardware offload. If the hardware offloadoperation fails, the data mover reverts to the traditional software method of data movement.

In nearly all cases, hardware data movement performs significantly better than software data movement.It consumes fewer CPU cycles and less bandwidth on the storage fabric. Timing operations that use theVAAI primitives and use esxtop to track values such as CMDS/s, READS/s, WRITES/s, MBREAD/s, andMBWRTN/s of storage adapters during the operation show performance improvements.

Table 2‑83. vStorage APIs for Array Integration Design Decision


SDDC-VI-Storage-003

Select an array thatsupports VAAI overNAS (NFS).

VAAI offloads tasks to the array itself, enablingthe ESXi hypervisor to use its resources forapplication workloads and not become abottleneck in the storage subsystem.

VAAI is required to support the desired numberof virtual machine lifecycle operations.

Not all VAAI arrays supportVAAI over NFS. A plugin fromthe array vendor is required toenable this functionality.


VMware, Inc. 123

Virtual Machine Storage Policies

You can create a storage policy for a virtual machine to specify which storage capabilities andcharacteristics are the best match for this virtual machine.

Note vSAN uses storage policies to allow specification of the characteristics of virtual machines, so youcan define the policy on an individual disk level rather than at the volume level for vSAN.

You can identify the storage subsystem capabilities by using the VMware vSphere API for StorageAwareness or by using a user-defined storage policy.

VMware vSphere APIfor Storage Awareness(VASA)

With vSphere API for Storage Awareness, storage vendors can publish thecapabilities of their storage to VMware vCenter Server, which can displaythese capabilities in its user interface.

User-defined storagepolicy

Defined by using the VMware Storage Policy SDK or VMware vSpherePowerCL, or from the vSphere Web Client.

You can assign a storage policy to a virtual machine and periodically check for compliance so that thevirtual machine continues to run on storage with the correct performance and availability characteristics.

You can associate a virtual machine with a virtual machine storage policy when you create, clone, ormigrate that virtual machine. If a virtual machine is associated with a storage policy, the vSphere WebClient shows the datastores that are compatible with the policy. You can select a datastore or datastorecluster. If you select a datastore that does not match the virtual machine storage policy, the vSphere WebClient shows that the virtual machine is using non-compliant storage. See Creating and ManagingvSphere Storage Policies in the vSphere 6.5 documentation.

Table 2‑84. Virtual Machine Storage Policy Design Decision


SDDC-VI-Storage-004 Use the default vSAN storagepolicy for all virtual machines inthe management cluster.

The default vSAN storagepolicy is adequate for themanagement cluster VMs.

If third party or additional VMs havedifferent storage requirements,additional VM storage policies maybe required.

vSphere Storage I/O Control Design

VMware vSphere Storage I/O Control allows cluster-wide storage I/O prioritization, which results in betterworkload consolidation and helps reduce extra costs associated with over provisioning.

vSphere Storage I/O Control extends the constructs of shares and limits to storage I/O resources. Youcan control the amount of storage I/O that is allocated to virtual machines during periods of I/Ocongestion, so that more important virtual machines get preference over less important virtual machinesfor I/O resource allocation.


VMware, Inc. 124

When vSphere Storage I/O Control is enabled on a datastore, the ESXi host monitors the device latencywhen communicating with that datastore. When device latency exceeds a threshold, the datastore isconsidered to be congested and each virtual machine that accesses that datastore is allocated I/Oresources in proportion to their shares. Shares are set on a per-virtual machine basis and can beadjusted.

vSphere Storage I/O Control has several requirements, limitations, and constraints.

n Datastores that are enabled with vSphere Storage I/O Control must be managed by a single vCenterServer system.

n Storage I/O Control is supported on Fibre Channel-connected, iSCSI-connected, and NFS-connectedstorage. RDM is not supported.

n Storage I/O Control does not support datastores with multiple extents.

n Before using vSphere Storage I/O Control on datastores that are backed by arrays with automatedstorage tiering capabilities, check the VMware Compatibility Guide whether the storage array hasbeen certified a compatible with vSphere Storage I/O Control.

Table 2‑85. Storage I/O Control Design Decisions


SDDC-VI-Storage-005

Enable Storage I/OControl with the defaultvalues on all non vSANdatastores.

Storage I/O Control ensures thatall virtual machines on adatastore receive an equalamount of I/O.

Virtual machines that use more I/Oare throttled to allow other virtualmachines access to the datastore onlywhen contention occurs on thedatastore.

Datastore Cluster DesignA datastore cluster is a collection of datastores with shared resources and a shared managementinterface. Datastore clusters are to datastores what clusters are to ESXi hosts. After you create adatastore cluster, you can use vSphere Storage DRS to manage storage resources.

vSphere datastore clusters group similar datastores into a pool of storage resources. When vSphereStorage DRS is enabled on a datastore cluster, vSphere automates the process of initial virtual machinefile placement and balances storage resources across the cluster to avoid bottlenecks. vSphere StorageDRS considers datastore space usage and I/O load when making migration recommendations.

When you add a datastore to a datastore cluster, the datastore's resources become part of the datastorecluster's resources. The following resource management capabilities are also available for each datastorecluster.


VMware, Inc. 125

Capability Description

Space utilization loadbalancing

You can set a threshold for space use. When space use on a datastore exceeds the threshold, vSphereStorage DRS generates recommendations or performs migrations with vSphere Storage vMotion to balancespace use across the datastore cluster.

I/O latency loadbalancing

You can configure the I/O latency threshold to avoid bottlenecks. When I/O latency on a datastore exceedsthe threshold, vSphere Storage DRS generates recommendations or performs vSphere Storage vMotionmigrations to help alleviate high I/O load.

Anti-affinity rules You can configure anti-affinity rules for virtual machine disks to ensure that the virtual disks of a virtualmachine are kept on different datastores. By default, all virtual disks for a virtual machine are placed on thesame datastore.

You can enable vSphere Storage I/O Control or vSphere Storage DRS for a datastore cluster. You canenable the two features separately, even though vSphere Storage I/O control is enabled by default whenyou enable vSphere Storage DRS.

vSphere Storage DRS Background Information

vSphere Storage DRS supports automating the management of datastores based on latency and storageutilization. When configuring vSphere Storage DRS, verify that all datastores use the same version ofVMFS and are on the same storage subsystem. Because vSphere Storage vMotion performs themigration of the virtual machines, confirm that all prerequisites are met.

vSphere Storage DRS provides a way of balancing usage and IOPS among datastores in a storagecluster:

n Initial placement of virtual machines is based on storage capacity.

n vSphere Storage DRS uses vSphere Storage vMotion to migrate virtual machines based on storagecapacity.

n vSphere Storage DRS uses vSphere Storage vMotion to migrate virtual machines based on I/Olatency.

n You can configure vSphere Storage DRS to run in either manual mode or in fully automated mode.

vSphere vStorage I/O Control and vSphere Storage DRS manage latency differently.

n vSphere Storage I/O Control distributes the resources based on virtual disk share value after alatency threshold is reached.

n vSphere Storage DRS measures latency over a period of time. If the latency threshold of vSphereStorage DRS is met in that time frame, vSphere Storage DRS migrates virtual machines to balancelatency across the datastores that are part of the cluster.

When making a vSphere Storage design decision, consider these points:

n Use vSphere Storage DRS where possible.

n vSphere Storage DRS provides a way of balancing usage and IOPS among datastores in a storagecluster:

n Initial placement of virtual machines is based on storage capacity.


VMware, Inc. 126

n vSphere Storage vMotion is used to migrate virtual machines based on storage capacity.

n vSphere Storage vMotion is used to migrate virtual machines based on I/O latency.

n vSphere Storage DRS can be configured in either manual or fully automated modes

vSAN Storage DesignVMware vSAN Storage design in this VMware Validated Design includes conceptual design, logicaldesign, network design, cluster and disk group design, and policy design.

VMware vSAN Conceptual Design and Logical Design

This VMware vSAN design is limited to the management cluster only. The design uses the defaultStorage Policy to achieve redundancy and performance within the cluster.

VMware vSAN Conceptual Design

While vSAN can be used within the shared edge and compute cluster, this design currently gives noguidance for the implementation.


VMware, Inc. 127

Figure 2‑23. Conceptual vSAN Design

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OS

APP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

ESXi ESXi


NSXController

(Mgmt)


NSXEdge

(Mgmt)

NSXManager(Mgmt)

NSXManager

(Compute)

NSXEdge

(Compute)



SDDCPayload


vSAN Datastore (management)


Management Cluster





Management Pod and Shared Edge and Compute Pod

vCenterServer(Mgmt)

vCenterServer

(Compute)

vSAN Logical Design

In a cluster that is managed by vCenter Server, you can manage software-defined storage resources justas you can manage compute resources. Instead of CPU or memory reservations, limits, and shares, youcan define storage policies and assign them to virtual machines. The policies specify the characteristicsof the storage and can be changed as business requirements change.

VMware vSAN Network Design

When performing network configuration, you have to consider the traffic and decide how to isolate vSANtraffic.

n Consider how much replication and communication traffic is running between hosts. With VMwarevSAN, the amount of traffic depends on the number of VMs that are running in the cluster, and onhow write-intensive the I/O is for the applications running in the VMs.


VMware, Inc. 128

n Isolate vSAN traffic on its own Layer 2 network segment. You can do this with dedicated switches orports, or by using a VLAN.

The vSAN VMkernel port group is created as part of cluster creation. Configure this port group on allhosts in a cluster, even for hosts that are not contributing storage resources to the cluster.

The following diagram illustrates the logical design of the network.

Figure 2‑24. VMware vSAN Conceptual Network

VM VM VM

ESXi Host ESXi Host ESXi Host

ESXi Host ESXi Host

vSAN Datastore

vSAN-Enabled Clusters

vSAN Network (VLAN)

Management Network (VLAN)

vMotion Network (VLAN)

Virtual Machine Network(s) (VLAN)

Network Bandwidth Requirements

VMware recommends that solutions use a 10 Gb Ethernet connection for use with vSAN to ensure thebest and most predictable performance (IOPS) for the environment. Without it, a significant decrease inarray performance results.

Note vSAN all-flash configurations are supported only with 10 GbE.

Table 2‑86. Network Speed Selection

Design Quality 1Gb 10Gb Comments



Performance ↓ ↑ Faster network speeds increase vSAN performance (especially in I/O intensive situations).

Recoverability ↓ ↑ Faster network speeds increase the performance of rebuilds and synchronizations in theenvironment. This ensures that VMs are properly protected from failures.




VMware, Inc. 129

Table 2‑87. Network Bandwidth Design Decision


SDDC-VI-Storage-SDS-001

Use only 10 GbE forVMware vSAN traffic.

Performance with 10 GbE is optimal.Without it, a significant decrease inarray performance results.

The physical network must support10 Gb networking between everyhost in the vSAN clusters.

VMware vSAN Virtual Switch Type

vSAN supports the use of vSphere Standard Switch or vSphere Distributed Switch. The benefit of usingvSphere Distributed Switch is that it supports Network I/O Control which allows for prioritization ofbandwidth in case of contention in an environment.

This design uses a vSphere Distributed Switch for the vSAN port group to ensure that priority can beassigned using Network I/O Control to separate and guarantee the bandwidth for vSAN traffic.

Virtual Switch Design Background

Virtual switch type affects performance and security of the environment.

Table 2‑88. Virtual Switch Types

Design QualityvSphereStandard Switch

vSphereDistributed Switch Comments


Manageability ↓ ↑ The vSphere Distributed Switch is centrally managed across allhosts, unlike the standard switch which is managed on each hostindividually.

Performance ↓ ↑ The vSphere Distributed Switch has added controls, such asNetwork I/O Control, which you can use to guarantee performancefor vSAN traffic.

Recoverability ↓ ↑ The vSphere Distributed Switch configuration can be backed upand restored, the standard switch does not have this functionality.

Security ↓ ↑ The vSphere Distributed Switch has added built-in security controlsto help protect traffic.


Table 2‑89. Virtual Switch Design Decision



Use the existing vSphereDistributed Switch instances inthe management clusters.

Provide guaranteed performance forvSAN traffic in case of contention byusing existing networkingcomponents.

All traffic paths are sharedover common uplinks.

Jumbo Frames

VMware vSAN supports jumbo frames for vSAN traffic.


VMware, Inc. 130

A VMware vSAN design should use jumbo frames only if the physical environment is already configuredto support them, they are part of the existing design, or if the underlying configuration does not create asignificant amount of added complexity to the design.

Table 2‑90. Jumbo Frames Design Decision



Configure jumbo frames onthe VLAN dedicated to vSANtraffic.

Jumbo frames are already used toimprove performance of vSpherevMotion and NFS storage traffic.

Every device in the networkmust support jumbo frames.

VLANs

VMware recommends isolating VMware vSAN traffic on its own VLAN. When a design uses multiplevSAN clusters, each cluster should use a dedicated VLAN or segment for its traffic. This approachprevents interference between clusters and helps with troubleshooting cluster configuration.

Table 2‑91. VLAN Design Decision



Use a dedicated VLAN forvSAN traffic for each vSANenabled cluster.

VLANs ensure trafficisolation.

VLANs span only a single pod.

A sufficient number of VLANs are availablewithin each pod and should be used for trafficsegregation.

Multicast Requirements

VMware vSAN requires that IP multicast is enabled on the Layer 2 physical network segment that is usedfor intra-cluster communication. All VMkernel ports on the vSAN network subscribe to a multicast groupusing Internet Group Management Protocol (IGMP).

A default multicast address is assigned to each vSAN cluster at the time of creation. IGMP (v3) snoopingis used to limit Layer 2 multicast traffic to specific port groups. As per the Physical Network Design, IGMPsnooping is configured with an IGMP snooping querier to limit the physical switch ports that participate inthe multicast group to only vSAN VMkernel port uplinks. In some cases, an IGMP snooping querier canbe associated with a specific VLAN. However, vendor implementations might differ.

Cluster and Disk Group Design

When considering the cluster and disk group design, you have to decide on the vSAN datastore size,number of hosts per cluster, number of disk groups per host, and the vSAN policy.

VMware vSAN Datastore Size

The size of the VMware vSAN datastore depends on the requirements for the datastore. Consider costversus availability to provide the appropriate sizing.


VMware, Inc. 131

Table 2‑92. VMware vSAN Datastore Design Decision



Provide the Management clusterwith a minimum of 8TB of rawcapacity for vSAN.

Management cluster virtual machines that usevSAN require at least 8 TB of raw storage.

NFS is used as secondary shared storage ofsome management components such asbackups and log archives.

None


On all VSAN datastores , ensurethat at least 30% of free space isalways available.

When vSAN reaches 80% usage a re-balancetask is started which can be resourceintensive.

Increases the amountof available storageneeded.

Number of Hosts Per Cluster

The number of hosts in the cluster depends on these factors:

n Amount of available space on the vSAN datastore

n Number of failures you can tolerate in the cluster

For example, if the vSAN cluster has only 3 ESXi hosts, only a single failure is supported. If a higher levelof availability is required, additional hosts are required.

Cluster Size Design Background

Table 2‑93. Number of Hosts Per Cluster

Design Quality 3 Hosts 32 Hosts 64 Hosts Comments

Availability ↓ ↑ ↑↑ The more hosts that are available in the cluster, the more failures the clustercan tolerate.

Manageability ↓ ↑ ↑ The more hosts in the cluster, the more virtual machines can be in the vSANenvironment.

Performance ↑ ↓ ↓ Having a larger cluster can impact performance if there is an imbalance ofresources. Consider performance as you make your decision.

Recoverability o o o Neither design option impacts recoverability.

Security o o o Neither design option impacts security.


Table 2‑94. Cluster Size Design Decision



Configure the Managementcluster with a minimum of 4ESXi hosts to support vSAN.

Having 4 hosts addresses the availabilityand sizing requirements, and allows youto take an ESXi host offline formaintenance or upgrades withoutimpacting the overall vSAN cluster health.

The availability requirements forthe management cluster mightcause underutilization of thecluster hosts.


VMware, Inc. 132

Number of Disk Groups Per Host

Disk group sizing is an important factor during volume design.

n If more hosts are available in the cluster, more failures are tolerated in the cluster. This capabilityadds cost because additional hardware for the disk groups is required.

n More available disk groups can increase the recoverability of vSAN during a failure.

Consider these data points when deciding on the number of disk groups per host:

n Amount of available space on the vSAN datastore

n Number of failures you can tolerate in the cluster

The optimal number of disk groups is a balance between hardware and space requirements for the vSANdatastore. More disk groups increase space and provide higher availability. However, adding disk groupscan be cost-prohibitive.

Disk Groups Design Background

The number of disk groups can affect availability and performance.

Table 2‑95. Number of Disk Groups Per Host

Design Quality 1 Disk Group 3 Disk Groups 5 Disk Groups Comments

Availability ↓ ↑ ↑↑ If more hosts are available in the cluster, the clustertolerates more failures. This capability adds cost becauseadditional hardware for the disk groups is required.

Manageability o o o If more hosts are in the cluster, more virtual machines canbe managed in the vSAN environment.

Performance o ↑ ↑↑ If the flash percentage ratio to storage capacity is large, thevSAN can deliver increased performance and speed.

Recoverability o ↑ ↑↑ More available disk groups can increase the recoverabilityof vSAN during a failure.

Rebuilds complete faster because there are more places toplace data and to copy data from.

Security o o o Neither design option impacts security.


Table 2‑96. Disk Groups Per Host Design Decision



Configure vSAN with a singledisk group per ESXi host in themanagement cluster.

Single disk group provides therequired performance andusable space for the datastore.

Losing an SSD in a host takes the diskgroup offline.

Using two or more disk groups canincrease availability and performance.


VMware, Inc. 133

VMware vSAN Policy Design

After you enable and configure VMware vSAN, you can create storage policies that define the virtualmachine storage characteristics. Storage characteristics specify different levels of service for differentvirtual machines. The default storage policy tolerates a single failure and has a single disk stripe. Use thedefault unless your environment requires policies with non-default behavior. If you configure a custompolicy, vSAN will guarantee it. However, if vSAN cannot guarantee a policy, you cannot provision a virtualmachine that uses the policy unless you enable force provisioning.

VMware vSAN Policy Options

A storage policy includes several attributes, which can be used alone or combined to provide differentservice levels. Policies can be configured for availability and performance conservatively to balancespace consumed and recoverability properties. In many cases, the default system policy is adequate andno additional policies are required. Policies allow any configuration to become as customized as neededfor the application’s business requirements.

Policy Design Background

Before making design decisions, understand the policies and the objects to which they can be applied.The policy options are listed in the following table.

Table 2‑97. VMware vSAN Policy Options

Capability Use Case Value Comments

Number of failuresto tolerate

Redundancy Default 1

Max 3

A standard RAID 1 mirrored configuration that provides redundancy for avirtual machine disk. The higher the value, the more failures can be tolerated.For n failures tolerated, n+1 copies of the disk are created, and 2n+1 hostscontributing storage are required.

A higher n value indicates that more replicas of virtual machines are made,which can consume more disk space than expected.

Number of diskstripes per object

Performance Default 1

Max 12

A standard RAID 0 stripe configuration used to increase performance for avirtual machine disk.

This setting defines the number of HDDs on which each replica of a storageobject is striped.

If the value is higher than 1, increased performance can result. However, anincrease in system resource usage might also result.

Flash read cachereservation (%)

Performance Default 0

Max 100%

Flash capacity reserved as read cache for the storage is a percentage of thelogical object size that will be reserved for that object.

Only use this setting for workloads if you must address read performanceissues. The downside of this setting is that other objects cannot use areserved cache.

VMware recommends not using these reservations unless it is absolutelynecessary because unreserved flash is shared fairly among all objects.


VMware, Inc. 134

Table 2‑97. VMware vSAN Policy Options (Continued)

Capability Use Case Value Comments

Object spacereservation (%)

Thickprovisioning

Default 0

Max 100%

The percentage of the storage object that will be thick provisioned upon VMcreation. The remainder of the storage will be thin provisioned.

This setting is useful if a predictable amount of storage will always be filled byan object, cutting back on repeatable disk growth operations for all but new ornon-predictable storage use.

Force provisioning Override policy Default:

No

Force provisioning allows for provisioning to occur even if the currentlyavailable cluster resources cannot satisfy the current policy.

Force provisioning is useful in case of a planned expansion of the vSANcluster, during which provisioning of VMs must continue. VMware vSANautomatically tries to bring the object into compliance as resources becomeavailable.

By default, policies are configured based on application requirements. However, they are applieddifferently depending on the object.

Table 2‑98. Object Policy Defaults

Object Policy Comments

Virtual machine namespace Failures-to-Tolerate: 1 Configurable. Changes are not recommended.

Swap Failures-to-Tolerate: 1 Configurable. Changes are not recommended.

Virtual disk(s) User-Configured Storage Policy Can be any storage policy configured on the system.

Virtual disk snapshot(s) Uses virtual disk policy Same as virtual disk policy by default. Changes are notrecommended.

Note If you do not specify a user-configured policy, the default system policy of 1 failure to tolerate and 1disk stripe is used for virtual disk(s) and virtual disk snapshot(s). Policy defaults for the VM namespaceand swap are set statically and are not configurable to ensure appropriate protection for these criticalvirtual machine components. Policies must be configured based on the application’s businessrequirements. Policies give VMware vSAN its power because it can adjust how a disk performs on the flybased on the policies configured.

Policy Design Recommendations

Policy design starts with assessment of business needs and application requirements. Use cases forVMware vSAN must be assessed to determine the necessary policies. Start by assessing the followingapplication requirements:

n I/O performance and profile of your workloads on a per-virtual-disk basis

n Working sets of your workloads

n Hot-add of additional cache (requires repopulation of cache)

n Specific application best practice (such as block size)


VMware, Inc. 135

After assessment, configure the software-defined storage module policies for availability and performancein a conservative manner so that space consumed and recoverability properties are balanced. In manycases the default system policy is adequate and no additional policies are required unless specificrequirements for performance or availability exist.

Table 2‑99. Policy Design Decision



Use the defaultVMware vSANstorage policy.

The default vSAN storage policy provides the level ofredundancy that is needed within the managementcluster. Additionally, using unique vSAN storagepolicies are not required as the default policy providesthe level of performance that is needed for theindividual management components.

Additional policies might beneeded if 3rd party VMs arehosted in these clusters becausetheir performance or availabilityrequirements might differ fromwhat the default VMware vSANpolicy supports.


Configure the virtualmachine swap fileas a sparse objectson VMware vSAN

Enabling this setting creates virtual swap files as asparse object on the vSAN datastore. Sparse virtualswap files will only consume capacity on vSAN asthey are accessed. The result can be significantly lessspace consumed on the vSAN datastore, providedvirtual machines do not experience memory overcommitment, requiring use of the virtual swap file.

Administrative overhead toenable the advanced setting onall ESXi hosts running VMwarevSAN.

NFS Storage DesignThis NFS design does not give specific vendor or array guidance. Consult your storage vendor for theconfiguration settings appropriate for your storage array.

NFS Storage Concepts

NFS (Network File System) presents file devices to an ESXi host for mounting over a network. The NFSserver or array makes its local file systems available to ESXi hosts. The ESXi hosts access the metadataand files on the NFS array or server using a RPC-based protocol. NFS is implemented using StandardNIC that is accessed using a VMkernel port (vmknic).

NFS Load Balancing

No load balancing is available for NFS/NAS on vSphere because it is based on single sessionconnections. You can configure aggregate bandwidth by creating multiple paths to the NAS array, and byaccessing some datastores via one path, and other datastores via another path. You can configure NICTeaming so that if one interface fails, another can take its place. However these load balancingtechniques work only in case of a network failure and might not be able to handle error conditions on theNFS array or on the NFS server. The storage vendor is often the source for correct configuration andconfiguration maximums.

NFS Versions

vSphere is compatible with both NFS version 3 and version 4.1; however, not all features can be enabledwhen connecting to storage arrays that use NFS v4.1.


VMware, Inc. 136

Table 2‑100. NFS Version Design Decision


SDDC-VI-Storage-NFS-001

Use NFS v3 for all NFSdatastores.

NFS v4.1 datastores are not supportedwith Storage I/O Control and with SiteRecovery Manager.

NFS v3 does not supportKerberos authentication.

Storage Access

NFS v3 traffic is transmitted in an unencrypted format across the LAN. Therefore, best practice is to useNFS storage on trusted networks only and to isolate the traffic on dedicated VLANs.

Many NFS arrays have some built-in security, which enables them to control the IP addresses that canmount NFS exports. Best practice is to use this feature to determine which ESXi hosts can mount thevolumes that are being exported and have read/write access to those volumes. This prevents unapprovedhosts from mounting the NFS datastores.

Exports

All NFS exports are shared directories that sit on top of a storage volume. These exports control theaccess between the endpoints (ESXi hosts) and the underlying storage system. Multiple exports can existon a single volume, with different access controls on each.

Export Size per Region Size

vRealize Log Insight Archive 1 TB

Table 2‑101. NFS Export Design Decisions



Create 1 exports to support thevRealize Log Insight Archivemanagement components.

The storage requirements of thesemanagement components areseparate from the primary storage.

You can add exports if youexpand the design.


Place the vSphere Data Protectionexport on its own separate volume asper SDDC-PHY-STO-008

Backup activities are I/O intensive.vSphere Data Protection or otherapplications suffer if vSphere DataProtection is placed on a sharedvolume.

Dedicated exports can addmanagement overhead tostorage administrators.


For each export, limit access to onlythe application VMs or hosts requiringthe ability to mount the storage.

Limiting access helps ensure thesecurity of the underlying data.

Securing exports individuallycan introduce operationaloverhead.

Cloud Management Platform DesignThe Cloud Management Platform (CMP) layer is the management component of the SDDC. The CMPlayer allows you to deliver tenants with automated workload provisioning by using a self-service portal.


VMware, Inc. 137

The CMP layer includes the Service Catalog, which houses the facilities to be deployed, Orchestrationwhich provides the workflows to get the catalog items deployed, and the Self-Service Portal thatempowers the end users to take full advantage of the Software Defined Data Center. vRealizeAutomation provides the Portal and the Catalog, and vRealize Orchestrator takes care of theOrchestration.

Figure 2‑25. The Cloud Management Platform Layer Within the Software-Defined DataCenter

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

vRealize Automation DesignVMware vRealize Automation provides a service catalog from which tenants can deploy applications, anda portal that lets you deliver a personalized, self-service experience to end users.

vRealize Automation Logical and Physical DesignThe cloud management layer can deliver multi-platform and multi-vendor cloud services.

The cloud management services in the SDDC provide the following advantages.

n Comprehensive and purpose-built capabilities to provide standardized resources to global customersin a short time span.

n Multi-platform and multi-vendor delivery methods that integrate with existing enterprise managementsystems.

n Central user-centric and business-aware governance for all physical, virtual, private, and public cloudservices.

n Design that meets the customer and business needs and is extensible.

Physical Design

The physical design consists of characteristics and decisions that support the logical design.


VMware, Inc. 138

Deployment Considerations

This design uses NSX logical switches to abstract the vRealize Automation application and its supportingservices. This abstraction allows the application to be hosted in any given region regardless of theunderlying physical infrastructure such as network subnets, compute hardware, or storage types. Thisdesign places the vRealize Automation application and its supporting services in Region A. The sameinstance of the application manages workloads in both Region A and Region B.

Table 2‑102. vRealize Automation Region Design Decision


SDDC-CMP-001 Utilize a single vRealizeAutomation installation tomanage both Region A andRegion B deployments from asingle instance.

vRealize Automation can manage one or moreregions. This provides a single consumptionportal regardless of region.

The abstraction of the vRealize Automationapplication over virtual networking allows it tobe independent from any physical sitelocations or hardware.

You must size vRealizeAutomation toaccommodate multi-regiondeployments.

Table 2‑103. vRealize Automation Anti-Affinity Rules


SDDC-CMP-002 Apply vSphere DistributedResource Scheduler (DRS)anti-affinity rules to thevRealize Automationcomponents

Using DRS prevents vRealizeAutomation nodes from residingon the same ESXi host andthereby risking the cluster's highavailability capability

Additional configuration is required to set upanti-affinity rules. Only a single ESXi host inthe management cluster, of the four ESXihosts, will be able to be put intomaintenance mode at a time.

Table 2‑104. vRealize Automation IaaS AD Requirement


SDDC-CMP-003 vRealize Automation IaaSMachines are joined to ActiveDirectory

This is a hard requirement forvRealize Automation

Active Directory access must beprovided using dedicated serviceaccounts


VMware, Inc. 139

Figure 2‑26. vRealize Automation Design Overview for Region A

IWS 1

OS

MSSQL

OS

IMS 1

OS

DEM 1

OS

DEM 2

OS

vRO 1

OS

vRO 2

OS

SVR 2

OS

IWS 2

OS

IAS 1

OS

IAS 2

OS

IWS

IMS

vRO

SVR

BUC 1

OS

BUS 1

OSSVR 1

OS

IMS 2

OS


VIP: 192.168.11.53vra01svr01.rainpole.local

VIP: 192.168.11.56vra01iws01.rainpole.local

VIP: 192.168.11.59vra01ims01.rainpole.local

VIP: 192.168.11.65vra01vro01.rainpole.local

Active Active Standalone Standalone Active Passive




Region Independent Region Specific

NSX Edge Load

Balancer

NSX Edge Load

Balancer

Cluster

Edge Reservation

WAN

Abbreviations

vRA vRealize AutomationvRO vRealize OrchestratorDEM Distributed Execution ManagerSVR vRA ApplianceIWS IaaS Web ServerIMS IaaS Manager ServiceIAS IaaS vSphere Proxy AgentBUS vRealize Business ServerBUC vRealize Business CollectorMSSQL Microsoft SQL

comp01psc01.sfo01.rainpole.local comp01vc01.sfo01.rainpole.local

comp01esx01.sfo01.rainpole.local

comp01esx0n.sfo01.rainpole.local

Shared Compute and Edge ClusterNSX

Manager

Dev Reservation

ProdReservation

192.168.11.56 192.168.11.54192.168.11.55

192.168.11.59 192.168.11.57192.168.11.58

192.168.11.65 192.168.11.63192.168.11.64

192.168.11.53 192.168.11.51192.168.11.52


Standalone


VMware, Inc. 140

Figure 2‑27. vRealize Automation DesignOverview for Region B

IAS 1

OS

IAS 2

OS

BUC 2

OS

Region Specific


Active Active

Edge Reservation

WAN

comp01psc51.lax01.rainpole.local comp01vc51.lax01.rainpole.local

comp01esx51.lax01.rainpole.local

comp01esx5n.lax01.rainpole.local

Shared Compute and Edge ClusterNSX

Manager

Dev Reservation

ProdReservation


vRealize Automation Appliance

The vRealize Automation virtual appliance includes the cloud management Web portal and databaseservices. The vRealize Automation portal allows self-service provisioning and management of cloudservices, as well as authoring blueprints, administration, and governance. The vRealize Automationvirtual appliance uses an embedded PostgreSQL database for catalog persistence and databasereplication. The database is configured between two vRealize Automation appliances for high availability.

Table 2‑105. vRealize Automation Virtual Appliance Design Decisions


SDDC-CMP-004 Deploy two instances of thevRealize Automation virtualappliance to achieve redundancy.

Enable an active/active front-endportal for higher availability.

None

SDDC-CMP-005 Deploy two appliances thatreplicate data using the embeddedPostgreSQL database.

Enable high availability forvRealize Automation.

In this active/passive configuration,manual failover between the twoinstances is required.

SDDC-CMP-006 During deployment configure thevRealize Automation applianceswith 18 GB vRAM.

Supports deployment of vRealizeAutomation in environments withup to 25,000 Active Directoryusers.

For environments with more then25,000 Active Directory users ofvRealize Automation, vRAM mustbe increased to 22 GB.


VMware, Inc. 141

Table 2‑106. vRealize Automation Virtual Appliance Resource Requirements per VirtualMachine


Number of vCPUs 4

Memory 18 GB

vRealize Automation function Portal Web site, Application, service catalog and Identity Manager

vRealize Automation IaaS Web Server

vRealize Automation IaaS Web server provides a user interface within the vRealize Automation portalWeb site for the administration and consumption of IaaS components.

The IaaS Website provides the infrastructure administration and service authoring capabilities to thevRealize Automation console. The Website component communicates with the Model Manager, whichprovides it with updates from the Distributed Execution Manager (DEM), proxy agents and database.

The Model Manager communicates with the database, the DEMs, and the portal website. TheModelManager is divided into two separately installable components — the Model Manager Web serviceand the Model Manager data component.

Note The vRealize Automation IaaS Web server is a separate component from the vRealize Automationappliance.

Table 2‑107. vRealize Automation IaaS Web Server Design Decision


SDDC-CMP-007 Install two vRealizeAutomation IaaS Webservers.

vRealize Automation can support between1,000 and 10,000 virtual machines. TwovRealize Automation IaaS Web Serversprovides redundancy to the IaaS Web Servercomponents.

Operational overheadincreases as more servers aredeployed.

Table 2‑108. vRealize Automation IaaS Web Server Resource Requirements


Number of vCPUs 2

Memory 4 GB

Number of vNIC ports 1

Number of local drives 1

vRealize Automation functions Model Manager (Web)

Operating system Microsoft Windows Server 2012 SP2 R2


VMware, Inc. 142

vRealize Automation IaaS Manager Service and DEM Orchestrator Server

The vRealize Automation IaaS Manager Service and Distributed Execution Management (DEM) serverare at the core of the vRealize Automation IaaS platform. The vRealize Automation IaaS ManagerService and DEM server supports several functions.

n Manages the integration of vRealize Automation IaaS with external systems and databases.

n Provides business logic to the DEMs.

n Manages business logic and execution policies.

n Maintains all workflows and their supporting constructs.

A Distributed Execution Manager (DEM) runs the business logic of custom models by interacting withother vRealize Automation component (repository) as required.

Each DEM instance acts in either an Orchestrator role or a Worker role. The DEM Orchestrator monitorsthe status of the DEM Workers. If a DEM worker stops or loses the connection to the Model Manager orrepository, the DEM Orchestrator puts the workflow back in the queue. It manages the scheduledworkflows by creating new workflow instances at the scheduled time and allows only one instance of aparticular scheduled workflow to run at a given time. It also preprocesses workflows before execution.Preprocessing includes checking preconditions for workflows and creating the workflow's executionhistory.

Note The vRealize Automation IaaS Manager Service and DEM Orchestrator service are separateservices, but are installed on the same virtual machine.

Table 2‑109. vRealize Automation IaaS Model Manager and DEM Orchestrator Server DesignDecision


SDDC-CMP-008 Deploy two virtual machines to runboth the vRealize Automation IaaSModel Manager and the DEMOrchestrator services in a load-balanced pool.

The Automation IaaS ModelManager and DEMOrchestrator share the sameactive/passive applicationmodel.

More resources are required for thesetwo virtual machines to accommodatethe load of the two applications. You canscale up the virtual machines later ifadditional resources are required.

Table 2‑110. vRealize Automation IaaS Model Manager and DEM Orchestrator ServerResource Requirements per Virtual Machine


Number of vCPUs 2

Memory 4 GB



vRealize Automation functions Model Manager , DEM Orchestrator



VMware, Inc. 143

vRealize Automation IaaS DEM Worker Virtual Machine

vRealize Automation IaaS DEM Workers are responsible for the executing provisioning anddeprovisioning tasks initiated by the vRealize Automation portal. DEM Workers are also utilized tocommunicate with specific infrastructure endpoints.

Table 2‑111. vRealize Automation IaaS DEM Worker Design Decision


SDDC-CMP-009 Install three DEMWorker instances perDEM host.

Each DEM Worker can process up to 30concurrent workflows. Beyond this limit, workflowsare queued for execution. If the number ofconcurrent workflows is consistently above 90, youcan add additional DEM Workers on the DEMhost.

If you add more DEM Workers,you must also provideadditional resources to runthem.

Table 2‑112. vRealize Automation DEM Worker Resource Requirements per Virtual Machine


Number of vCPUs 2

Memory 6 GB



vRealize Automation functions DEM Worker


vRealize Automation IaaS Proxy Agent

The vRealize Automation IaaS Proxy Agent is a windows service used to communicate with specificinfrastructure endpoints. In this design, the vSphere Proxy agent is utilized to communicate with vCenter.

The IaaS Proxy Agent server provides the following functions:

n vRealize Automation IaaS Proxy Agent can interact with different types of infrastructure components.For this design, only the vSphere Proxy agent is used.

n vRealize Automation does not itself virtualize resources, but works with vSphere to provision andmanage the virtual machines. It uses vSphere Proxy agents to send commands to and collect datafrom vSphere.


VMware, Inc. 144

Table 2‑113. vRealize Automation IaaS Agent Server Design Decisions


SDDC-CMP-010 Deploy two vRealize AutomationvSphere Proxy Agent virtualmachines.

Using two virtual machinesprovides redundancy for vSphereconnectivity.

More resources are requiredbecause multiple virtual machinesare deployed for this function.

SDDC-CMP-011 Abstract the proxy agent virtualmachines on a separate virtualnetwork for independent failover ofthe main vRealize Automationcomponents across sites.

Allows the failover of the vRealizeAutomation instance from one siteto another independently.

Additional application virtualnetworks and associated edgedevices need to be provisioned forthose proxy agents.

Table 2‑114. vRealize Automation IaaS Proxy Agent Resource Requirements per VirtualMachine


Number of vCPUs 2

Memory 4 GB



vRealize Automation functions vSphere Proxy agent


Load Balancer

Session persistence of a load balancer allows the same server to serve all requests after a session isestablished with that server. The session persistence is enabled on the load balancer to directsubsequent requests from each unique session to the same vRealize Automation server in the loadbalancer pool. The load balancer also handles failover for the vRealize Automation Server (ManagerService) because only one Manager Service is active at any one time. Session persistence is not enabledbecause it is not a required component for the Manager Service.


VMware, Inc. 145

Table 2‑115. Load Balancer Design Decisions


SDDC-CMP-012

Set up a load balancer for allvRealize Automation services thatsupport active/active oractive/passive configurations.

Required to enable vRealize Automation to handle agreater load and obtain a higher level of availabilitythan without load balancers.

Additionalconfiguration isrequired toconfigure the loadbalancers.

SDDC-CMP-013

Configure load balancer forvRealize Automation ServerAppliance, Remote Console Proxy,and IaaS Web to utilize Round-Robin algorithm with Source-IPbased persistence with a 1800second timeout.

Round-robin provides a good balance of clientsbetween both appliances, while the Source-IP ensurethat individual clients remain connected to the sameappliance. 1800 second timeout aligns with thevRealize Automation Appliance Server sessionstimeout value. Sessions that transfer to a differentvRealize Automation Appliance may result in a pooruser experience.

None

SDDC-CMP-014

Configure load balancer forvRealize Iaas Server and vRealizeOrchestrator to utilize Round-Robinalgorithm without persistence.

Round-robin provides a good balance of individualrequests from the vRealize Server to the vRealizeOrchestrator. This will distribute requests equallybetween the configured vRealize Orchestrator serversto allow the performance capacity of both to be bestutilized.

vRealize Automation IaaS Server is Active/stand-byarchitecture, there for all request will go to a singlenode only.

None

Consider the following load balancer characteristics for vRealize Automation.

Table 2‑116. Load Balancer Application Profile Characteristics

Server Role Type Enable SSL Pass-through Persistence Expires in (Seconds)

vRealize Automation - Persistence HTTPS (443) Enabled Source IP 1800

vRealize Automation HTTPS (443) Enabled

Table 2‑117. Load Balancer Service Monitoring Characteristics

Monitor Interval TimeoutMaxRetries Type Expected Method URL Receive

vRealizeAutomationAppliance

3 9 3 HTTPS 204 GET /vcac/services/api/health

vRealizeAutomationIaaS Web

3 9 3 HTTPS GET /wapi/api/status/web REGISTERED

vRealizeAutomationIaaS Manager

3 9 3 HTTPS GET /VMPSProvision ProvisionService

vRealizeOrchestrator

3 9 3 HTTPS GET /vco/api/healthstatus RUNNING


VMware, Inc. 146

Table 2‑118. Load Balancer Pool Characteristics

Server Role Algorithm Monitor Members Port Monitor Port

vRealize AutomationAppliance

Round Robin vRealize AutomationAppliance monitor

vRealize AutomationAppliance nodes

443 443

vRealize AutomationRemote Console Proxy

Round Robin vRealize AutomationAppliance monitor

vRealize AutomationAppliance nodes

8444 443

vRealize Automation IaaSWeb

Round Robin vRealize Automation IaaSWeb monitor

IaaS web nodes 443 443

vRealize Automation IaaSManager

Round Robin vRealize Automation IaaSManager monitor

IaaS Manager nodes 443 443

vRealize Orchestrator Round Robin vRealize AutomationOrchestrator monitor

vRealize Orchestratornodes

8281 8281

Table 2‑119. Virtual Server Characteristics

Protocol Port Default Pool Application Profile

HTTPS 443 vRealize Automation Appliance Pool vRealize Automation - Persistence Profile

HTTPS 443 vRealize Automation IaaS Web Pool vRealize Automation - Persistence Profile

HTTPS 443 vRealize Automation IaaS Manager Pool vRealize Automation Profile

HTTPS 8281 vRealize Orchestrator Pool vRealize Automation Profile

HTTPS 8444 vRealize Automation Remote Console Proxy Pool vRealize Automation - Persistence Profile

Information Security and Access Control in vRealize Automation

You use a service account for authentication and authorization of vRealize Automation to vCenter Serverand vRealize Operations Manager for orchestrating and creating virtual objects in the SDDC.


VMware, Inc. 147



SDDC-CMP-015

Configure a service account svc-vra in vCenter Server forapplication-to-applicationcommunication from vRealizeAutomation with vSphere.

You can introduce improved accountabilityin tracking request-response interactionsbetween the components of the SDDC.

You must maintain the serviceaccount's life cycle outside ofthe SDDC stack to ensure itsavailability

SDDC-CMP-016

Use local permissions when youcreate the svc-vra serviceaccount in vCenter Server.

The use of local permissions ensures thatonly the Compute vCenter Server instancesare valid and accessible endpoints fromvRealize Automation.

If you deploy more ComputevCenter Server instances, youmust ensure that the serviceaccount has been assignedlocal permissions in eachvCenter Server so that thisvCenter Server is a viableendpoint within vRealizeAutomation.

SDDC-CMP-017

Configure a service account svc-vra-vrops on vRealizeOperations Manager forapplication-to-applicationcommunication from vRealizeAutomation for collecting healthand resource metrics for tenantworkload reclamation.

n vRealize Automation accesses vRealizeOperations Manager with the minimumset of permissions that are required forcollecting metrics to determine theworkloads that are potential candidatesfor reclamation.

n In the event of a compromised account,the accessibility in the destinationapplication remains restricted.

n You can introduce improvedaccountability in tracking request-response interactions between thecomponents of the SDDC.

You must maintain the serviceaccount's life cycle outside ofthe SDDC stack to ensure itsavailability.

vRealize Automation Supporting Infrastructure

To satisfy the requirements of this SDDC design, you configure additional components for vRealizeAutomation such as database servers for highly available database service, email server for notification,and vRealize Business for cost management.

Microsoft SQL Server Database

vRealize Automation and vRealize Orchestrator uses a Microsoft SQL Server database to store thevRealize Automation IaaS elements and vRealize Orchestrator data. The database also maintainsinformation about the machines that vRealize Automation manages.


VMware, Inc. 148

Table 2‑121. vRealize Automation SQL Database Design Decisions


SDDC-CMP-018

Set up a Microsoft SQL serverthat supports the availabilityand I/O needs of vRealizeAutomation.

A dedicated or shared SQL server can be usedso long as it meets the requirements ofvRealize Automation.

Requires additionalresources and licenses.

SDDC-CMP-019

Locate the Microsoft SQLserver in the vRealizeAutomation virtual network orset it up to have global failoveravailable.

For simple failover of the entire vRealizeAutomation instance from one region toanother, the Microsoft SQL server must berunning as a VM inside the vRealize Automationapplication virtual network.

If the environment uses a shared SQL server,global failover ensures connectivity from bothprimary and secondary regions.

Adds additional overhead tomanaging Microsoft SQLservices.

SDDC-CMP-020

Set up Microsoft SQL serverwith separate OS volumes forSQL Data, Transaction Logs,TempDB, and Backup.

While each organization might have their ownbest practices in the deployment andconfiguration of Microsoft SQL server, high levelbest practices recommend separation ofdatabase data files and database transactionlogs.

You might need to consultwith the Microsoft SQLdatabase administrators ofyour organization forguidance about productiondeployment in yourenvironment.

Table 2‑122. vRealize Automation SQL Database Server Resource Requirements per VM


Number of vCPUs 8

Memory 16 GB



40 GB (D:) (Application)

40 GB (E:) Database Data

20 GB (F:) Database Log

20 GB (G:) TempDB

80 GB (H:) Backup

vRealize Automation functions Microsoft SQL Server Database

Microsoft SQL Version SQL Server 2012

Microsoft SQL Database Version SQL Server 2012 (110)

Operating system Microsoft Windows Server 2012 R2

PostgreSQL Database Server

The vRealize Automation appliance uses a PostgreSQL database server to maintain the vRealizeAutomation portal elements and services, and the information about the catalog items that the appliancemanages.


VMware, Inc. 149

Table 2‑123. vRealize Automation PostgreSQL Database Design Decision


SDDC-CMP-021 Use the embedded PostgreSQLdatabase within each vRealizeAutomation appliance.

Simplifies the design and enablesreplication of the database across thetwo vRealize Automation appliances.

None.

SDDC-CMP-022 Configure the embeddedPostgreSQL database to utilizeasynchronous replication.

Asynchronous replication offers a goodbalance between availability andperformance.

Asynchronous replicationprovides a good level ofavailability in compliance withthe design objectives.

Notification Email Server

vRealize Automation notification emails are sent using SMTP. These emails include notification ofmachine creation, expiration, and the notification of approvals received by users. vRealize Automationsupports both anonymous connections to the SMTP server and connections using basic authentication.vRealize Automation also supports communication with or without SSL.

You create a global, inbound email server to handle inbound email notifications, such as approvalresponses. Only one, global inbound email server, which appears as the default for all tenants, is needed.The email server provides accounts that you can customize for each user, providing separate emailaccounts, usernames, and passwords. Each tenant can override these settings. If tenant administratorsdo not override these settings before enabling notifications, vRealize Automation uses the globallyconfigured email server. The server supports both the POP and the IMAP protocol, with or without SSLcertificates.

Notifications

System administrators configure default settings for both the outbound and inbound emails servers usedto send system notifications. Systems administrators can create only one of each type of server thatappears as the default for all tenants. If tenant administrators do not override these settings beforeenabling notifications, vRealize Automation uses the globally configured email server.

System administrators create a global outbound email server to process outbound email notifications, anda global inbound email server to process inbound email notifications, such as responses to approvals.

Table 2‑124. vRealize Automation Email Server Configuration


SDDC-CMP-023 Configure vRealize Automation to use aglobal outbound email server to handleoutbound email notifications and a globalinbound email server to handle inboundemail notifications, such as approvalresponses.

Requirement to integratevRealize Automationapprovals and systemnotifications throughemails.

Must prepare the SMTP/IMAP serverand necessary firewall access andcreate a mailbox for inbound emails(IMAP), and anonymous access canbe used with outbound emails.


VMware, Inc. 150

vRealize Business for Cloud

vRealize Business for Cloud provides end-user transparency in the costs that are associated withoperating workloads. A system, such as vRealize Business, to gather and aggregate the financial cost ofworkload operations provides greater visibility both during a workload request and on a periodic basis,regardless of whether the costs are "charged-back" to a specific business unit, or are "showed-back" toillustrate the value that the SDDC provides.

vRealize Business integrates with vRealize Automation to display costing during workload request and onan ongoing basis with cost reporting by user, business group or tenant. Additionally, tenant administratorscan create a wide range of custom reports to meet the requirements of an organization.

Table 2‑125. vRealize Business for Cloud Standard Design Decision


SDDC-CMP-024

Deploy vRealize Business forCloud as part of the cloudmanagement platform andintegrate it with vRealizeAutomation.

Tenant and Workload costing is providedby vRealize Business for Cloud.

Additional appliances need to bedeployed to handle for vRealizeBusiness for Cloud and remotecollectors.

SDDC-CMP-025

Use default vRealize Businessfor Cloud appliance size (8GB).For vRealize Business for Cloudremote collector, utilize areduced memory size of 2GB.

Default vRealize Business for Cloudappliance size supports up to 10,000VMs

Remote Collectors do not run serverservice, and can run on 2GB of RAM.

None.

SDDC-CMP-026

Use default vRealize Businessreference costing database.

Default reference costing is based onindustry information and is periodicallyupdated.

Default reference costing might notaccurately represent actualcustomer costs. vRealize BusinessAppliance requires Internet accessto periodically update the referencedatabase.

SDDC-CMP-027

Deploy vRealize Business as athree-VM architecture withremote data collectors in RegionA and Region B.

For best performance, the vRealizeBusiness collectors should be regionallylocal to the resource which they areconfigured to collect. Because thisdesign supports disaster recovery, theCMP can reside in Region A or RegionB.

In the case where the environmentdoes not implement disasterrecovery support, you must deployan additional appliance, the one forthe remote data collector, althoughthe vRealize Business server canhandle the load on its own.

SDDC-CMP-028

Deploy the vRealize Businessserver VM in the cross-regionlogical network.

The vRealize Business deploymentdepends on vRealize Automation.During a disaster recovery event,vRealize Business will migrate withvRealize Automation.

None.

SDDC-CMP-029

Deploy a vRealize Businessremote data collector VM in eachregion-specific logical network

vRealize Business remote data collectoris a region-specific installation. During adisaster recovery event, the remotecollector does not need to migrate withvRealize Automation.

The communication with vCenterServer involves an additional Layer3 hop through an NSX edgedevice.


VMware, Inc. 151

Table 2‑126. vRealize Business for Cloud Virtual Appliance Resource Requirements perVirtual Machine


Number of vCPUs 4

Memory 8 GB for Server / 2 GB for Remote Collector

vRealize Business function Server or remote collector

vRealize Automation Cloud Tenant DesignA tenant is an organizational unit within a vRealize Automation deployment, and can represent a businessunit within an enterprise, or a company that subscribes to cloud services from a service provider. Eachtenant has its own dedicated configuration, although some system-level configuration is shared acrosstenants.

Comparison of Single-Tenant and Multi-Tenant Deployments

vRealize Automation supports deployments with a single tenant or multiple tenants. System-wideconfiguration is always performed using the default tenant, and can then be applied to one or moretenants. For example, system-wide configuration might specify defaults for branding and notificationproviders.

Infrastructure configuration, including the infrastructure sources that are available for provisioning, can beconfigured in any tenant and is shared among all tenants. The infrastructure resources, such as cloud orvirtual compute resources or physical machines, can be divided into fabric groups managed by fabricadministrators. The resources in each fabric group can be allocated to business groups within eachtenant by using reservations.

Default-TenantDeployment

In a default-tenant deployment, all configuration occurs in the defaulttenant. Tenant administrators can manage users and groups, and configuretenant-specific branding, notifications, business policies, and catalogofferings. All users log in to the vRealize Automation console at the sameURL, but the features available to them are determined by their roles.

Single-TenantDeployment

In a single-tenant deployment, the system administrator creates a singlenew tenant for the organization that use the same vRealize Automationinstance. Tenant users log in to the vRealize Automation console at a URLspecific to their tenant. Tenant-level configuration is segregated from thedefault tenant, although users with system-wide roles can view and


VMware, Inc. 152

manage both configurations. The IaaS administrator for the organizationtenant creates fabric groups and appoints fabric administrators. Fabricadministrators can create reservations for business groups in theorganization tenant.

Multi-TenantDeployment

In a multi-tenant deployment, the system administrator creates new tenantsfor each organization that uses the same vRealize Automation instance.Tenant users log in to the vRealize Automation console at a URL specific totheir tenant. Tenant-level configuration is segregated from other tenantsand from the default tenant, although users with system-wide roles canview and manage configuration across multiple tenants. The IaaSadministrator for each tenant creates fabric groups and appoints fabricadministrators to their respective tenants. Although fabric administratorscan create reservations for business groups in any tenant, in this scenariothey typically create and manage reservations within their own tenants. Ifthe same identity store is configured in multiple tenants, the same userscan be designated as IaaS administrators or fabric administrators for eachtenant.

Tenant Design

This design deploys a single tenant containing two business groups.

n The first business group is designated for production workloads provisioning.

n The second business group is designated for development workloads.

Tenant administrators manage users and groups, configure tenant-specific branding, notifications,business policies, and catalog offerings. All users log in to the vRealize Automation console at the sameURL, but the features available to them are determined by their roles.

Figure 2‑28 illustrates the dual-region tenant design.


VMware, Inc. 153

Figure 2‑28. Rainpole Cloud Automation Tenant Design for Two Regions

Production Business Group

Rainpole Tenanthttps://vra.mycompany.com/vcac/org/rainpole

Business Group Manager

Development Business Group

TenantAdminBusiness Group

Manager

Fabric Admin

IaaSAdmin

ProdReservation

DevReservation

EdgeReservation

Region A Fabric Group

ProdReservation

DevReservation

EdgeReservation

Region B Fabric Group

Region A Data Center Infrastructure Fabric

Region B Data Center Infrastructure Fabric

https://vra.mycompany.com/vcac

• Tenant Creation• System Branding• System Notification Providers• Event LogsSystem Admin

Default Tenant

Fabric Admin

Table 2‑127. Tenant Design Decisions


SDDC-CMP-030

Utilizes vRealize Automation businessgroups for separate business units(instead of separate tenants).

Allows transparency across theenvironments and some level ofsharing of resources and servicessuch as blueprints.

Some elements such as buildprofiles are visible to bothbusiness groups. The designdoes not provide full isolationfor security or auditing.

SDDC-CMP-031

Create separate fabric groups for eachdeployment region. Each fabric grouprepresent region-specific data centerresources. Each of the business groupshave reservations into each of the fabricgroups.

Provides future isolation of fabricresources and potential delegation ofduty to independent fabricadministrators.

Initial deployment will use asingle shared fabric thatconsists of one compute pod.


VMware, Inc. 154

Table 2‑127. Tenant Design Decisions (Continued)


SDDC-CMP-032

Allow access to the default tenant onlyby the system administrator and for thepurposes of managing tenants andmodifying system-wide configurations.

Isolates the default tenant fromindividual tenant configurations.

Each tenant administrator isresponsible for managing theirown tenant configuration.

SDDC-CMP-033

Evaluate your internal organizationstructure and workload needs. ConfigurevRealize Business Groups,Reservations, Service Catalogs, andtemplates based on your specificorganizations needs.

vRealize Automation is designed tointegrate with your individualorganizations needs. Within thisdesign, guidance for Rainpole isprovided as a starting point, but thisguidance may not be appropriate foryour specific business needs.

Partners and Customers willneed to evaluate their specificbusiness needs.

Service Catalog

The service catalog provides a common interface for consumers of IT services to use to request andmanage the services and resources they need.

A tenant administrator or service architect can specify information about the service catalog, such as theservice hours, support team, and change window. While the catalog does not enforce service-levelagreements on services, this service hours, support team, and change window information is available tobusiness users browsing the service catalog.

Catalog Items

Users can browse the service catalog for catalog items they are entitled to request. For some catalogitems, a request results in the provisioning of an item that the user can manage. For example, the usercan request a virtual machine with Windows 2012 preinstalled, and then manage that virtual machineafter it has been provisioned.

Tenant administrators define new catalog items and publish them to the service catalog. The tenantadministrator can then manage the presentation of catalog items to the consumer and entitle new items toconsumers. To make the catalog item available to users, a tenant administrator must entitle the item tothe users and groups who should have access to it. For example, some catalog items may be availableonly to a specific business group, while other catalog items may be shared between business groupsusing the same tenant. The administrator determines what catalog items are available to different usersbased on their job functions, departments, or location.

Typically, a catalog item is defined in a blueprint, which provides a complete specification of the resourceto be provisioned and the process to initiate when the item is requested. It also defines the optionsavailable to a requester of the item, such as virtual machine specifications or lease duration, or anyadditional information that the requester is prompted to provide when submitting the request.

Machine Blueprints

A machine blueprint is the complete specification for a virtual, cloud or physical machine. A machineblueprint determines the machine's attributes, how it is provisioned, and its policy and managementsettings. Machine blueprints are published as catalog items in the service catalog.


VMware, Inc. 155

Machine blueprints can be specific to a business group or shared among groups within a tenant. Tenantadministrators can create shared blueprints that can be entitled to users in any business group within thetenant. Business group managers can create group blueprints that can only be entitled to users within aspecific business group. A business group manager cannot modify or delete shared blueprints. Tenantadministrators cannot view or modify group blueprints unless they also have the business group managerrole for the appropriate group.

If a tenant administrator sets a shared blueprint's properties so that it can be copied, the business groupmanager can also copy the shared blueprint for use as a starting point to create a new group blueprint.

Table 2‑128. Single Machine Blueprints

Name Description

Base Windows Server (Development) Standard Rainpole SOE deployment of Windows 2012 R2 available to the Developmentbusiness group.

Base Windows Server (Production) Standard Rainpole SOE deployment of Windows 2012 R2 available to the Productionbusiness group.

Base Linux (Development) Standard Rainpole SOE deployment of Linux available to the Development businessgroup.

Base Linux (Production) Standard Rainpole SOE deployment of Linux available to the Production business group.

Windows Server + SQL Server(Production)

Base Windows 2012 R2 Server with silent SQL 2012 Server install with customproperties. This is available to the Production business group.

Windows Server + SQL Server(Development)

Base Windows 2012 R2 Server with silent SQL 2012 Server install with customproperties. This is available to the Development business group.

Blueprint Definitions

The following sections provide details of each service definition that has been included as part of thecurrent phase of cloud platform deployment.

Table 2‑129. Base Windows Server Requirements and Standards

Service Name Base Windows Server

Provisioning Method When users select this blueprint, vRealize Automation clones a vSphere virtual machinetemplate with preconfigured vCenter customizations.

Entitlement Both Production and Development business group members.

Approval Process No approval (pre-approval assumed based on approved access to platform).

Operating System and Version Details Windows Server 2012 R2

Configuration Disk: Single disk drive

Network: Standard vSphere Networks

Lease and Archival Details Lease:n Production Blueprints: No expiration daten Development Blueprints: Minimum 30 days – Maximum 270 days

Archive: 15 days

Pre- and Post-DeploymentRequirements

Email sent to manager confirming service request (include description details).


VMware, Inc. 156

Table 2‑130. Base Windows Blueprint Sizing

Sizing vCPU Memory (GB) Storage (GB)

Default 1 4 60

Maximum 4 16 60

Table 2‑131. Base Linux Server Requirements and Standards

Service Name Base Linux Server


Entitlement Both Production and Development business group members.


Operating System and Version Details Red Hat Enterprise Server 6


Network: Standard vSphere networks


Archive: 15 days


Email sent to manager confirming service request (include description details) .

Table 2‑132. Base Linux Blueprint Sizing


Default 1 6 20

Maximum 4 12 20

Table 2‑133. Base Windows Server with SQL Server Install Requirements and Standards



Entitlement Both Production and Development business group members


Operating System and Version Details Windows Server 2012 R2


Network: Standard vSphere Networks

Silent Install: The Blueprint calls a silent script using the vRA Agent to install SQL2012Server with custom properties.


VMware, Inc. 157

Table 2‑133. Base Windows Server with SQL Server Install Requirements and Standards(Continued)



Archive: 15 days


Email sent to manager confirming service request (include description details)

Table 2‑134. Base Windows with SQL Server Blueprint Sizing


Default 1 8 100

Maximum 4 16 400

Branding of the vRealize Automation Console

System administrators can change the appearance of the vRealize Automation console to meet site-specific branding guidelines by changing the logo, the background color, or information in the header andfooter. System administrators control the default branding for tenants. Tenant administrators can use thedefault or reconfigure branding for each tenant.

vRealize Automation Infrastructure as a Service DesignThis topic introduces the integration of vRealize Automation with vSphere resources used to create theInfrastructure as a Service design for use with the SDDC.

Figure 2‑29 illustrates the logical design of the vRealize Automation groups and vSphere resources.


VMware, Inc. 158

Figure 2‑29. vRealize Automation Logical Design

Production Business Group

Rainpole Tenanthttps://vra.mycompany.com/vcac/org/rainpole

Business Group Manager

Development Business Group

TenantAdminBusiness Group

Manager

Fabric Admin

IaaSAdmin

ProdReservation

DevReservation

EdgeReservation

Region A Fabric Group

ProdReservation

DevReservation

EdgeReservation

Region B Fabric Group

Region A Data Center Infrastructure Fabric

Region B Data Center Infrastructure Fabric

https://vra.mycompany.com/vcac

• Tenant Creation• System Branding• System Notification Providers• Event LogsSystem Admin

Default Tenant

Fabric Admin

The following terms apply to vRealize Automation when integrated with vSphere. These terms and theirmeaning may vary from the way they are used when referring only to vSphere.

Term Definition

vSphere (vCenterServer) endpoint

Provides information required by vRealize Automation IaaS to access vSphere compute resources.

Compute resource Virtual object within vRealize Automation that represents a vCenter Server cluster or resource pool, anddatastores or datastore clusters.

Note Compute resources are CPU, memory, storage and networks. Datastores and datastore clusters arepart of the overall storage resources.

Fabric groups vRealize Automation IaaS organizes compute resources into fabric groups.

Fabricadministrators

Fabric administrators manage compute resources, which are organized into fabric groups.

Computereservation

A share of compute resources (vSphere cluster, resource pool, datastores, or datastore clusters), such asCPU and memory reserved for use by a particular business group for provisioning virtual machines.

Note vRealize Automation uses the term reservation to define resources (be they memory, storage ornetworks) in a cluster. This is different than the use of reservation in vCenter Server, where a share is apercentage of total resources, and reservation is a fixed amount.


VMware, Inc. 159

Term Definition

Storage reservation Similar to compute reservation (see above), but pertaining only to a share of the available storageresources. In this context, you specify a storage reservation in terms of gigabytes from an existing LUN orDatastore.

Business groups A collection of virtual machine consumers, usually corresponding to an organization's business units ordepartments. Only users in the business group can request virtual machines.

Reservation policy vRealize Automation IaaS determines its reservation (also called virtual reservation) from which a particularvirtual machine is provisioned. The reservation policy is a logical label or a pointer to the original reservation.Each virtual reservation can be added to one reservation policy.

Build profile A set of user defined properties you apply to a virtual machine when it is provisioned. For example, theoperating system used in a blueprint, or the available networks to use for connectivity at the time ofprovisioning the virtual machine.

Build profile properties determine the specification of the virtual machine, the manner in which it isprovisioned, operations to perform after it is provisioned, or management information maintained withinvRealize Automation.

Blueprint The complete specification for a virtual machine, determining the machine attributes, the manner in which itis provisioned, and its policy and management settings.

Blueprint allows the users of a business group to create virtual machines on a virtual reservation (computeresource) based on the reservation policy, and using platform and cloning types. It also lets you specify oradd machine resources and build profiles.

Figure 2‑30 shows the logical design constructs discussed in the previous section as they apply to adeployment of vRealize Automation integrated with vSphere in a cross data center provisioning.


VMware, Inc. 160

Figure 2‑30. vRealize Automation Integration with vSphere Endpoint

DNS

OSAD

OSDB

OSAPP

OSAPP

OS

Business Group(s)One-click request for VMsprovisioned from the vRA portal.

Console access to cloudVMs using the vRA VMRC.

vSphere SFO EndPoint

vSphere LAX EndPoint

SFO Fabric Group

LAX Fabric Group

SFO Blueprint

VM.Network.Name: SFO-Prod-NetReservation Policy: SFO-RPolicy

Template: Win2012-R2-STD

LAX Blueprint

VM.Network.Name: LAX-Prod-NetReservation Policy: SFO-RPolicy

Template: Win2012-R2-STD

Network Profile: SFO-Prod-Net

Network Profile: LAX-Prod-Net

SFO Reservation Policy

LAX Reservation Policy

vRealize Automation

vSphere ContentLibrary

Templates

Windows 2012 R2 STD 64-bit

Linux 64-bit

Template Synchronization

vSphere ContentLibrary

Templates


Linux 64-bitNetwork Path:SFO-Prod-Net

SFO-Reservation

SFO Shared Edgeand Compute Pod

xx TB Memoryxx TB Storage

Network Path:LAX-Prod-Net

LAX-Reservation

LAX Shared Edgeand Compute Pod

xx TB Memoryxx TB Storage

Layer 3 NetworkRegion A DC Region B DC


VMware, Inc. 161

Infrastructure Source Endpoints

An infrastructure source endpoint is a connection to the infrastructure that provides a set (or multiple sets)of resources, which can then be made available by IaaS administrators for consumption by end users.vRealize Automation IaaS regularly collects information about known endpoint resources and the virtualresources provisioned therein. Endpoint resources are referred to as compute resources(or as computepods, the terms are often used interchangeably).

Infrastructure data is collected through proxy agents that manage and communicate with the endpointresources. This information about the compute resources on each infrastructure endpoint and themachines provisioned on each computer resource is collected at regular intervals.

During installation of the vRealize Automation IaaS components, you can configure the proxy agents anddefine their associated endpoints. Alternatively, you can configure the proxy agents and define theirassociated endpoints separately after the main vRealize Automation installation is complete.

Table 2‑135. Endpoint Design Decisions


SDDC-CMP-034 Create two vSphereendpoints.

One vSphere endpoint is required to connect toeach vCenter Server instance in each region.Two endpoints will be needed for two regions.

As additional regions are broughtonline additional vSphereendpoints need to be deployed.

SDDC-CMP-035 Create one vRealizeOrchestrator endpoint.

vRealize Automation extensibility usesvRealize Orchestrator. One vRealizeOrchestrator cluster exists which requires thecreation of a single endpoint.

Using external vRealizeOrchestrator requires manualconfiguration of a vRealizeOrchestrator endpoint.

Virtualization Compute Resources

A virtualization compute resource is a vRealize Automation object that represents an ESXi host or acluster of ESXi hosts. When a group member requests a virtual machine, the virtual machine isprovisioned on these compute resources. vRealize Automation regularly collects information about knowncompute resources and the virtual machines provisioned on them through the proxy agents.

Table 2‑136. Compute Resource Design Decision


SDDC-CMP-036 Create at least onecompute resources foreach deployed region.

Each region has one computecluster, one compute resource isrequired for each cluster.

As additional compute clusters are created,they need to be added to the existing computeresource in their region or to a new resource,which has to be created.

Note By default, compute resources are provisioned to the root of the compute cluster. In this design,use of vSphere resource pools is mandatory.

Fabric Groups

A fabric group is a logical container of several compute resources, and can be managed by fabricadministrators.


VMware, Inc. 162

Table 2‑137. Fabric Group Design Decisions


SDDC-CMP-037 Create a fabric group for eachregion and include all the computeresources and edge resources inthat region.

To enable region specificprovisioning a fabric group in eachregion must be created.

As additional clusters are added ina region, they must be added to thefabric group.

Business Groups

A business group is a collection of machine consumers, often corresponding to a line of business,department, or other organizational unit. To request machines, a vRealize Automation user must belongto at least one business group. Each group has access to a set of local blueprints used to requestmachines.

Business groups have the following characteristics:

n A group must have at least one business group manager, who maintains blueprints for the group andapproves machine requests.

n Groups can contain support users, who can request and manage machines on behalf of other groupmembers.

n A vRealize Automation user can be a member of more than one Business group, and can havedifferent roles in each group.

Reservations

A reservation is a share of one compute resource's available memory, CPU and storage reserved for useby a particular fabric group. Each reservation is for one fabric group only but the relationship is many-to-many. A fabric group might have multiple reservations on one compute resource, or reservations onmultiple compute resources, or both.

Converged Compute/Edge Clusters and Resource Pools

While reservations provide a method to allocate a portion of the cluster memory or storage withinvRealize Automation, reservations do not control how CPU and memory is allocated during periods ofcontention on the underlying vSphere compute resources. vSphere Resource Pools are utilized to controlthe allocation of CPU and memory during time of resource contention on the underlying host. To fullyutilize this, all VMs must be deployed into one of three resource pools: SDDC-EdgeRP01, User-EdgeRP01, and User-VMRP01. Core-NSX is dedicated for datacenter level NSX Edge components andshould not contain any user workloads. User-EdgeRP01 is dedicated for any statically or dynamicallydeployed NSX components such as NSX Edges or Load Balancers which serve a specific customerworkloads. User-VMRP01 is dedicated for any statically or dynamically deployed virtual machines such asWindows, Linux, databases, etc, which contain specific customer workloads.


VMware, Inc. 163

Table 2‑138. Reservation Design Decisions


SDDC-CMP-038

Create at least one vRealizeAutomation reservation foreach business group at eachregion.

In our example, each resource cluster will havetwo reservations, one for production and onefor development, allowing both production anddevelopment workloads to be provisioned.

Because production anddevelopment share the samecompute resources, thedevelopment business groupmust be limited to a fixed amountof resources.

SDDC-CMP-039

Create at least one vRealizeAutomation reservation foredge resources in eachregion.

An edge reservation in each region allowsNSX to create edge services gateways ondemand and place them on the edge cluster.

The workload reservation mustdefine the edge reservation in thenetwork settings.

SDDC-CMP-040

Configure all vRealizeAutomation workloads toutilize dedicated vCenterResource Pools.

In order to ensure dedicated computeresources of NSX networking components,end-user deployed workloads must beassigned to a dedicated end-user workloadvCenter Resource Pools. Workloadsprovisioned at the root resource pool level willreceive more resources then resource pools,which would starve those virtual machines incontention situations.

Cloud administrators mustensure all workload reservationsare configured with theappropriate resource pool. Thismay be a single resource pool forboth production and developmentworkloads, or two resource pools,one dedicated for theDevelopment Business Groupand one dedicated for theProduction Business Group.

SDDC-CMP-041

Configure vRealizeAutomation reservations fordynamically provisioned NSXEdge components (routedgateway) to utilize dedicatedvCenter Resource Pools.

In order to ensure dedicated computeresources of NSX networking components,end-user deployed NSX edge componentsmust be assigned to a dedicated end-usernetwork component vCenter Resource Pool.Workloads provisioned at the root resourcepool level will receive more resources thenresource pools, which would starve thosevirtual machines in contention situations.

Cloud administrators mustensure all workload reservationsare configured with theappropriate resource pool.

SDDC-CMP-042

All vCenter resource poolsutilized for Edge or Computeworkloads must be created atthe "root" level. Nesting ofresource pools is notrecommended.

Nesting of resource pools can createadministratively complex resource calculationsthat may result in unintended under or overallocation of resources during contentionsituations.

All resource pools must becreated at the root resource poollevel.

Reservation Policies

You can add each virtual reservation to one reservation policy. The reservation from which a particularvirtual machine is provisioned is determined by vRealize Automation based on the reservation policyspecified in the blueprint, if any, the priorities and current usage of the fabric group's reservations, andother custom properties.


VMware, Inc. 164

Table 2‑139. Reservation Policy Design Decisions


SDDC-CMP-043 Create at least one workloadreservation policy for each region.

Reservation policies are used to target adeployment to a specific set of reservations in eachregion. Reservation policies are also used to targetworkloads into their appropriate region, computecluster and/or vSphere resource pool.

None

SDDC-CMP-044 Create at least one reservationpolicy for placement of dynamicallycreated edge services gatewaysinto the edge clusters.

Required to place the edge devices into theirrespective edge clusters and/or vSphere resourcepools.

None

A storage reservation policy is a set of datastores that can be assigned to a machine blueprint to restrictdisk provisioning to only those datastores. Storage reservation policies are created and associated withthe appropriate datastores and assigned to reservations.

Table 2‑140. Storage Reservation Policy Design Decisions


SDDC-CMP-045 Within this design,storage tiers are notused.

The underlying physicalstorage design does not usestorage tiers.

Both business groups will have access to thesame storage.

For customers who utilize multiple datastores withdifferent storage capabilities will need to evaluatethe usage of vRealize Automation StorageReservation Policies.

Template Synchronization

This dual-region design supports provisioning workloads across regions from the same portal using thesame single-machine blueprints. A synchronization mechanism is required to have consistent templatesacross regions. There are multiple ways to achieve synchronization, for example, vSphere ContentLibrary or external services like vCloud Connector or vSphere Replication.

Table 2‑141. Template Synchronization Design Decision


SDDC-CMP-046 This design uses vSphereContent Library tosynchronize templatesacross regions.

The vSphere Content Library is builtinto the version of vSphere being usedand meets all the requirements tosynchronize templates.

Storage space must be provisionedin each region.

vRealize Automation can not directlyconsume templates from vSphereContent Library.

When users select this blueprint, vRealize Automation clones a vSphere virtual machine template withpreconfigured vCenter customizations.


VMware, Inc. 165

Figure 2‑31. Template Synchronization

vSphere Content Library

Source

Templates


Linux 64-bit

DestinationTemplate Synchronization

vSphere Content Library

Templates


Linux 64-bit

VMware Identity Management

The Identity Manager is integrated directly into the vRealize Automation appliance and provides tenantidentity management. The vIDM service synchronizes directly with the Rainpole Active Directory domain.Important users and groups are synced with the Identity Manager. Authentication always takes placeagainst the Active Directory domain, but searches are made against the local Active Directory mirror onthe vRealize Automation appliance.

APP

OS

APP

SAAS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

APP

OS

Proxy

Internal Applications

vRA Services withembedded vIDM and DB

SAP SaaSApps

Horizon View

Load Balancer

Proxy

Shared Infrastructure

vRA Services withembedded vIDM and DB

Active Directory

DNS RSA PKI Radius

DB DB

VMware Identity Manager Proxies Authentication Between Active Directory and vRealize Automation

Table 2‑142. Active Directory Authentication Decision


SDDC-CMP-047 Choose Active Directory withIntegrated WindowsAuthentication as the DirectoryService connection option.

Rainpole uses a single-forest, multiple-domain Active Directory environment.

Integrated Windows Authentication supportsestablishing trust relationships in a multi-domain or multi-forest Active Directoryenvironment.

Requires that the vRealizeAutomation appliances arejoined to the Active Directorydomain.


VMware, Inc. 166

By default the vRealize Automation appliance is initially configured with 18 GB of memory, which isenough to support a small Active Directory environment. An Active Directory environment is consideredsmall if it fewer than 25,000 users in the organizational unit (OU) have to be synced. An Active Directoryenvironment with more than 25,000 users is considered large and needs additional memory and CPU.See the vRealize Automation sizing guidelines for details.

The connector is a component of the vRealize Automation service and performs the synchronization ofusers and groups between Active Directory and the vRealize Automation service. In addition, theconnector is the default identity provider and authenticates users to the service.

Table 2‑143. Connector Configuration Decision


SDDC-CMP-048

To support Directories Servicehigh availability, configure asecond connector thatcorresponds to the secondvRealize Automationappliance.

This design supports high availability byinstalling two vRealize Automation appliancesand using load-balanced NSX Edgeinstances. Adding the second connector tothe second vRealize Automation applianceensures redundancy and improvesperformance by load balancing authenticationrequests.

This design simplifies thedeployment while leveragingrobust built-in HA capabilities.This design uses NSX forvSphere load balancing.

vRealize Orchestrator DesignVMware vRealize Orchestrator is a development and process automation platform that provides a libraryof extensible workflows to allow you to create and run automated, configurable processes to manage theVMware vSphere infrastructure as well as other VMware and third-party technologies.

In this VMware Validated Design, vRealize Administration uses the vRealize Orchestrator Plug-In toconnect to vCenter Server for compute resource allocation.

vRealize Orchestrator Logical DesignThis VMware Validated Design includes this logical design for vRealize Orchestrator.

Table 2‑144. vRealize Orchestrator Hardware Design Decision


SDDC-CMP-VRO-01

Utilize external vRealizeOrchestrator instances withdefault sizing - 2 CPUs, 6 GBmemory, and 17 GB of hard disk.

The vRealize Orchestrator appliancerequires the appropriate resources toenable connectivity to vRealize Automationvia the vRealize Orchestrator Plugin.

External appliances are utilized to ensureisolation between vRealize portalcomponents and customer workflows.

Resources should not bereduced as the vRealizeOrchestrator Appliancerequires this for scalability.

vRealize Orchestrator Authentication

vRealize Orchestrator supports several authentication methods.


VMware, Inc. 167

vRealize Orchestrator supports the following authentication methods:

n vRealize Automation Authentication

n vSphere Authentication

n vCenter Single Sign-On Authentication (SSO Legacy)

vRealize Automation Authentication utilizes the vRealize Automation component registry forauthentication.

vSphere Authentication utilizes the Platform Service Controller in vSphere 6.0 and 6.5 environments.

vCenter Single Sign-On Authentication utilizes the vSphere Legacy SSO in vSphere 5.5 environments.

The only configuration supported for multi-domain Active Directory is domain tree. Forest and externaltrusts are not supported. Multiple domains that have two-way trust, but are not in the same tree, are notsupported and do not work with vRealize Orchestrator.

Table 2‑145. vRealize Orchestrator Directory Service Design Decision


SDDC-CMP-VRO-02

Configure all vRealizeOrchestrator instances withinthe SDDC to use vRealizeAutomation authentication.

LDAP is being depreciated. Supportsexisting design setup utilizing ActiveDirectory services.

This design does not support localauthentication for vRealizeOrchestrator.

SDDC-CMP-VRO-03

Configure vRealizeOrchestrator to utilize thevRealize Automation customertenant for authentication.

The vRealize Automation DefaultTenant users are only administrativeusers. By connecting to the customertenant, workflows executing onvRealize Orchestrator may executewith end-user granted permissions.

End-users who will execute vRealizeOrchestrator workflows will berequired to have permissions on thevRealize Orchestrator server.

Some plug-ins may not functioncorrectly using vRealize AutomationAuthentication.

SDDC-CMP-VRO-04

A vRealize Orchestratorinstallation will be associatedwith only one customer tenant.

To provide best security andsegregation between potentialtenants, vRealize Orchestratorinstallation are associate with a singletenant.

If additional vRealize AutomationTenants are configured, additionalvRealize Orchestrator installationswill be needed.

Network Ports

vRealize Orchestrator uses specific network ports to communicate with other systems. The ports areconfigured with a default value, but you can change the defaults at any time. When you make changes,verify that all ports are available for use by your host. If necessary, open these ports on any firewallsthrough which network traffic for the relevant components flows. Verify that the required network ports areopen before you deploy vRealize Orchestrator.

Default Communication Ports

Set default network ports and configure your firewall to allow incoming TCP connections. Other ports maybe required if you are using custom plug-ins.


VMware, Inc. 168

Table 2‑146. vRealize Orchestrator Default Configuration Ports

Port Number Protocol Source Target Description

HTTP server port 8280 TCP End-user Webbrowser

vRealizeOrchestrator server

The requests sent to Orchestratordefault HTTP Web port 8280 areredirected to the default HTTPS Webport 8281.

HTTPS Server port 8281 TCP End-user Webbrowser

vRealizeOrchestrator server

The SSL secured HTTP protocol usedto connect to the vRealizeOrchestrator REST API.

Web configurationHTTPS access port

8283 TCP End-user Webbrowser

vRealizeOrchestratorconfiguration

The SSL access port for the Web UIfor vRealize Orchestratorconfiguration.

External Communication Ports

Configure your firewall to allow outgoing connections using the external network ports so vRealizeOrchestrator can communicate with external services.

Table 2‑147. vRealize Orchestrator Default External Communication Ports


LDAP 389 TCP vRealizeOrchestrator server

LDAP server Lookup port of your LDAP authenticationserver.

LDAP using SSL 636 TCP vRealizeOrchestrator server

LDAP server Lookup port of your secure LDAPauthentication server.

LDAP using GlobalCatalog

3268 TCP vRealizeOrchestrator server

Global Catalogserver

Port to which Microsoft Global Catalogserver queries are directed.

DNS 53 TCP vRealizeOrchestrator server

DNS server Name resolution

VMware vCenter™Single Sign-Onserver


vCenter SingleSign-On server

Port used to communicate with thevCenter Single Sign-On server.

SQL Server 1433 TCP vRealizeOrchestrator server

Microsoft SQLserver

Port used to communicate with theMicrosoft SQL Server or SQL ServerExpress instances that are configured asthe vRealize Orchestrator database.

PostgreSQL 5432 TCP vRealizeOrchestrator server

PostgreSQL server Port used to communicate with thePostgreSQL Server that is configured asthe vRealize Orchestrator database.

Oracle 1521 TCP vRealizeOrchestrator server

Oracle DB server Port used to communicate with theOracle Database Server that isconfigured as the vRealize Orchestratordatabase.

SMTP Server port 25 TCP vRealizeOrchestrator server

SMTP Server Port used for email notifications.


VMware, Inc. 169

Table 2‑147. vRealize Orchestrator Default External Communication Ports (Continued)


vCenter ServerAPI port


VMware vCenterserver

The vCenter Server API communicationport used by vRealize Orchestrator toobtain virtual infrastructure and virtualmachine information from theorchestrated vCenter Server instances.

vCenter Server 80 TCP vRealizeOrchestrator server

vCenter Server Port used to tunnel HTTPScommunication.

VMware ESXi 443 TCP vRealizeOrchestrator server

ESXi hosts (Optional) Workflows using the vCenterGuest Operations API need directconnection between vRealizeOrchestrator and the ESXi hosts the VMis running on.

vRealize Orchestrator Deployment

This design deploys vRealize Orchestrator as two servers behind a load balancer. A vRealizeOrchestrator appliance is used for each instance required in the SDDC.

Table 2‑148. vRealize Orchestrator Deployment Decision


SDDC-CMP-VRO-05 Deploy a minimum two-nodevRealize Orchestrator externalappliances behind a load balancer.

Supports a highly availablevRealize Orchestratorenvironment.

Cluster setup of vRealizeOrchestrator is required.

An external shared database isrequired.

Table 2‑149. vRealize Orchestrator Anti-Affinity rules


SDDC-CMP-VRO-06

Apply vSphere DistributedResource Scheduler (DRS)anti-affinity rules to thevRealize Orchestratorcomponents

Using DRS prevents vRealizeAutomation nodes from residingon the same ESXi host andthereby risking the cluster's highavailability capability

Additional configuration is required to set upanti-affinity rules. Only a single ESXi host inthe management cluster, of the four ESXihosts, will be able to be put intomaintenance mode at a time.

The vRealize Orchestrator appliance using Linux comes preconfigured, enabling fast deployment.

The vRealize Orchestrator appliance package is distributed with preinstalled software contains thefollowing software components:

n SUSE Linux Enterprise Server 11 SP3 for VMware, 64-bit edition

n PostgreSQL

n Orchestrator

vRealize Orchestrator Server Mode

vRealize Orchestrator supports standalone mode and cluster mode. This design uses cluster mode.


VMware, Inc. 170

vRealize Orchestrator supports the following server modes.

Standalone mode vRealize Orchestrator server runs as a standalone instance. This is thedefault mode of operation.

Cluster mode To increase availability of the vRealize Orchestrator services, and to createa more highly available SDDC, you can configure vRealize Orchestrator towork in cluster mode, and start multiple vRealize Orchestrator instances ina cluster with a shared database. In cluster mode, multiple vRealizeOrchestrator instances with identical server and plug-in configurations worktogether as a cluster, and share a single database.

All vRealize Orchestrator server instances communicate with each other by exchanging heartbeats at acertain time interval. Only active vRealize Orchestrator server instances respond to client requests andrun workflows. If an active vRealize Orchestrator server instance fails to send heartbeats, it is consideredto be non-responsive, and one of the inactive instances takes over to resume all workflows from the pointat which they were interrupted. The heartbeat is implemented through the shared database, so there areno implications in the network design for a vRealize Orchestrator cluster. If you have more than oneactive vRealize Orchestrator node in a cluster, concurrency problems can occur if different users use thedifferent vRealize Orchestrator nodes to modify the same resource.

vRealize Orchestrator SDDC Cluster

The vRealize Orchestrator cluster is deployed to the same cluster as the vRealize Automation instances.

vSphere Cluster Design, defines the following clusters for the two-cluster implementation.

n Management cluster

n Shared Compute and Edge cluster

n Compute payload cluster

The vRealize Orchestrator instance is logically a part of the management cluster.

Table 2‑150. vRealize Orchestrator SDDC Cluster Design Decision


SDDC-CMP-VRO-07

Deploy all vRealize Orchestratorinstances required by the SDDCsolution within the managementcluster.

In this design, only the vRealize Automationcomponent consumes vRealizeOrchestrator.

None

SDDC-CMP-VRO-08

Configure load balancer for vRealizeOrchestrator to utilize round-robinbalancing algorithm withoutpersistence.

Round-robin allows individual workflowrequests to be balanced across vRealizeOrchestrator nodes. Requests do notrequire persistence to a specific vRealizeOrchestrator node.

None

The following tables outline characteristics for this vRealize Orchestrator design.


VMware, Inc. 171

Table 2‑151. Service Monitors Characteristics

Monitor Interval Timeout Retries Type Send String Receive String

vRealize Orchestrator Monitor 3 9 3 HTTPS (443) GET /vco/api/healthstatus RUNNING

Table 2‑152. Pool Characteristics

Pool Name Algorithm Monitors Members Port Monitor Port

vRealize Orchestrator Round Robin withoutPersistence

vRealize OrchestratorMonitor

vRealize Orchestratornodes

8281 8281

Table 2‑153. Virtual Server Characteristics

Name Protocol Service Port Default Pool Name

vRealize Orchestrator HTTPS 8281 vRealize Orchestrator

vRealize Orchestrator Information Security and Access Control

You use a service account for authentication and authorization of vRealize Orchestrator to vCenter Serverfor orchestrating and creating virtual objects in the SDDC.



SDDC-CMP-VRO-09

Configure a service account svc-vro in vCenter Server forapplication-to-applicationcommunication from vRealizeOrchestrator with vSphere.

You can introduce improvedaccountability in tracking request-response interactions betweenthe components of the SDDC.

You must maintain the serviceaccount's life cycle outside of theSDDC stack to ensure its availability

SDDC-CMP-VRO-10

Use local permissions when youcreate the svc-vro serviceaccount in vCenter Server.

The use of local permissionsensures that only the ComputevCenter Server instances arevalid and accessible endpointsfrom vRealize Orchestrator.

If you deploy more Compute vCenterServer instances, you must ensure thatthe service account has been assignedlocal permissions in each vCenterServer so that this vCenter Server is aviable endpoint in vRealizeOrchestrator.

vRealize Orchestrator ConfigurationvRealize Orchestrator configuration includes appliance and client configuration. It also provides guidanceon external database configuration, SSC certificates, and plug-ins.

vRealize Orchestrator Appliance Network Settings and Naming Conventions

Use consistent naming conventions and policies when labeling virtual machines so that their use is clearto any IT staff working with them. It is a best practice to configure common ports, network settings, andappliance names across all virtual appliances to simplify management and maintain consistency. Keep inmind that future extensibility options might affect naming conventions.


VMware, Inc. 172

vRealize Orchestrator Client

The vRealize Orchestrator client is a desktop application that lets you import packages, create, run, andschedule workflows, and manage user permissions.

You can install the vRealize Orchestrator Client standalone on a desktop system. Download the vRealizeOrchestrator Client installation files from the vRealize Orchestrator appliance pageat https://vRO_hostname:8281. Alternatively, you can run the vRealize Orchestrator Client using JavaWebStart directly from the homepage of the vRealize Orchestrator appliance console.

SSL Certificates

The vRealize Orchestrator configuration interface uses a secure connection to communicate with vCenterServer, relational database management systems (RDBMS), LDAP, vCenter Single Sign-On, and otherservers. You can import the required SSL certificate from a URL or file. You can import the vCenter ServerSSL certificate from the SSL Trust Manager tab in the vRealize Orchestrator configuration interface.

Table 2‑155. vRealize Orchestrator SSL Design Decision


SDDC-CMP-VRO-011 Use a CA-signed certificate forcommunication between vCenter Server andvRealize Orchestrator.

Supports requirements for usingCA-signed certificates.

None

vRealize Orchestrator Database

vRealize Orchestrator requires a database. For small-scale deployments, you can use the SQL ServerExpress database that is bundled with vCenter Server, or the preconfigured vRealize Orchestratordatabase. vRealize Orchestrator supports Oracle, Microsoft SQL Server, Microsoft SQL Server Express,and PostgreSQL.

For a complete list of supported databases, see VMware Product InteroperabilityMatrixes at http://www.vmware.com/resources/compatibility/sim/interop_matrix.php.

This design uses an external Microsoft SQL (MSSQL) database.

Table 2‑156. vRealize Orchestrator Database Design Decision


SDDC-CMP-VRO-12

Configure the vRealizeOrchestrator appliance toutilize an external MSSQLdatabase.

The SDDC design is already using an externalMSSQL database for other components. Databasesupport currently includes MSSQL or Oracle. For allsupported versions of databases see the VMwareProduct Interoperability Matrix.

None


VMware, Inc. 173

http://www.vmware.com/resources/compatibility/sim/interop_matrix.php

vRealize Orchestrator Plug-Ins

Plug-ins allow you to use vRealize Orchestrator to access and control external technologies andapplications. Exposing an external technology in a vRealize Orchestrator plug-in allows you to incorporateobjects and functions in workflows that access the objects and functions of the external technology. Theexternal technologies that you can access using plug-ins can include virtualization management tools,email systems, databases, directory services, and remote control interfaces. vRealize Orchestratorprovides a set of standard plug-ins that allow you to incorporate such technologies as the vCenter ServerAPI and email capabilities into workflows.

In addition, the vRealize Orchestrator open plug-in architecture allows you to develop plug-ins to accessother applications. vRealize Orchestrator implements open standards to simplify integration with externalsystems. For information about developing custom content, see Developing with VMware vRealizeOrchestrator.

vRealize Orchestrator and the vCenter Server Plug-In

You can use the vCenter Server plug-in to manage multiple vCenter Server instances. You can createworkflows that use the vCenter Server plug-in API to automate tasks in your vCenter Server environment.The vCenter Server plug-in maps the vCenter Server API to the JavaScript that you can use in workflows.The plug-in also provides actions that perform individual vCenter Server tasks that you can include inworkflows.

The vCenter Server plug-in provides a library of standard workflows that automate vCenter Serveroperations. For example, you can run workflows that create, clone, migrate, or delete virtual machines.Before managing the objects in your VMware vSphere inventory by using vRealize Orchestrator and torun workflows on the objects, you must configure the vCenter Server plug-in and define the connectionparameters between vRealize Orchestrator and the vCenter Server instances you want to orchestrate.You can configure the vCenter Server plug-in by using the vRealize Orchestrator configuration interfaceor by running the vCenter Server configuration workflows from the vRealize Orchestrator client. You canconfigure vRealize Orchestrator to connect to your vCenter Server instances for running workflows overthe objects in your vSphere infrastructure.

To manage the objects in your vSphere inventory using the vSphere Web Client, configure vRealizeOrchestrator to work with the same vCenter Single Sign-On instance to which both vCenter Server andvSphere Web Client are pointing. Also verify that vRealize Orchestrator is registered as a vCenter Serverextension. You register vRealize Orchestrator as a vCenter Server extension when you specify a user(user name and password) who has the privileges to manage vCenter Server extensions.

Table 2‑157. vRealize Orchestrator vCenter Server Plug-In Design Decisions


SDDC-CMP-VRO-13 Configure the vCenter Server plug-in tocontrol communication with the vCenterServers.

Required for communication to vCenterServer instances, and therefore requiredfor workflows.

None

vRealize Orchestrator ScalabilityvRealize Orchestrator supports both scale-up and scale-out scalability.


VMware, Inc. 174

Scale Up

A single vRealize Orchestrator instance allows up to 300 concurrent workflow instances in the runningstate. Workflow instances that are in the waiting or waiting-event states do not count toward that number.You can design long running workflows in a way that preserves resources by using the wait elements ofthe workflow palette. A single vRealize Orchestrator instance supports up to 35,000 managed virtualmachines in its inventory.

Scale Out

You can scale out vRealize Orchestrator using a clustered environment, multiple independent vRealizeOrchestrator instances, or a combination of both.

In a clustered vRealize Orchestrator environment, multiple vRealize Orchestrator instances can beconnected to the same (external) database. Configure all vRealize Orchestrator instances in a clusterusing the same settings. Using a vRealize Orchestrator cluster allows you to increase the number ofconcurrent running workflows, but not the number of managed inventory objects. When clustering avRealize Orchestrator server, choose between these cluster types.

n An active-active cluster with up to five active nodes. VMware recommends a maximum of three activenodes in this configuration.

n An active-passive cluster with only one active node, and up to seven standby nodes.

In a clustered vRealize Orchestrator environment you cannot change workflows while other vRealizeOrchestrator instances are running. Stop all other vRealize Orchestrator instances before you connect thevRealize Orchestrator client and change or develop a new workflow.

You can scale out a vRealize Orchestrator environment by having multiple independent vRealizeOrchestrator instances (each with their own database instance). This option allows you to increase thenumber of managed inventory objects. You can use the vRealize Orchestrator Multinode plug-in toreplicate the vRealize Orchestrator content, and to start and monitor workflow executions.

Table 2‑158. vRealize Orchestrator Active-Passive Design Decision


SDDC-CMP-VRO-14

Configure vRealizeOrchestrator in an active-activecluster configuration.

An active-passive cluster is not currently beingimplemented as a highly available environmentis required.

Active-active clusters allow for both vRealizeOrchestrator servers to equally balanceworkflow execution.

None

SDDC-CMP-VRO-15

vRealize Orchestratorinstallation is sized for use byvRealize Automation for typicalprovisioning operations.

vRealize Orchestrator is designed for customerextensibility. Each customer may utilizevRealize Orchestrator to a different degree asneeded by their specific environment.

Customers must assessthe impact of anycustomer createdworkflows on theenvironment.


VMware, Inc. 175

Operations Infrastructure DesignOperations Management is a required element of a software-defined data center. Monitoring operationssupport in vRealize Operations Manager and vRealize Log Insight provides capabilities for performanceand capacity management of related infrastructure and cloud management components.

Figure 2‑32. Operations Management in the SDDC Layered Architecture

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

n vRealize Operations Manager Design

The foundation of vRealize Operations Manager is a single instance of a 3-node analytics clusterthat is deployed in the protected region of the SDDC, and a 2-node remote collector cluster in eachregion. The clusters run on the management pod in each region.

n vRealize Log Insight Design

vRealize Log Insight design enables real-time logging for all components that build up themanagement capabilities of the SDDC in a dual-region setup.

n vSphere Data Protection Design

Design data protection of the management components in your environment to ensure continuousoperation of the SDDC if the data of a management application is damaged.

n Site Recovery Manager and vSphere Replication Design

To support disaster recovery (DR) in the SDDC, you protect vRealize Operations Manager andvRealize Automation by using vCenter Site Recovery Manager and VMware vSphere Replication.When failing over to a recovery region, these management applications continue the delivery ofoperations management, and cloud platform management functionality.

n vSphere Update Manager Design

vSphere Update Manager pairs with vCenter Server to enable patch and version management ofESXi hosts and virtual machines.


VMware, Inc. 176

vRealize Operations Manager DesignThe foundation of vRealize Operations Manager is a single instance of a 3-node analytics cluster that isdeployed in the protected region of the SDDC, and a 2-node remote collector cluster in each region. Theclusters run on the management pod in each region.

n vRealize Operations Manager Logical and Physical Design

vRealize Operations Manager communicates with all management components in both regions ofthe SDDC to collect metrics about the operations of the objects that these components handle.

n vRealize Operations Manager Nodes

The analytics cluster of the vRealize Operations Manager deployment contains the nodes thatanalyze and store data from the monitored components. You deploy a configuration of the analyticscluster that satisfies the requirements for monitoring the number of virtual machines according to thedesign objectives of this VMware Validated Design.

n Networking Design

You place the vRealize Operations Manager nodes in several network units for isolation and failover.The networking design also supports public access to the analytics cluster nodes.

n Information Security and Access Control

You configure a central source for authentication in vRealize Operations Manager such as an ActiveDirectory service. vRealize Operations Manager also authenticates to vCenter Server and can use alocal user inventory.

n Monitoring and Alerting

vRealize Operations Manager can monitor itself and displays alerts about issues with its operationalstate.

n Management Packs

The SDDC contains several VMware products for network, storage, and cloud management. Youcan monitor and perform diagnostics on all of them in vRealize Operations Manager by usingmanagement packs.

vRealize Operations Manager Logical and Physical DesignvRealize Operations Manager communicates with all management components in both regions of theSDDC to collect metrics about the operations of the objects that these components handle.

Logical Design

In a multi-region Software Defined Data Center (SDDC), you deploy a vRealize Operations Managerconfiguration that consists of the following entities.

n 3-node (medium-size) vRealize Operations Manager analytics cluster that is highly available (HA).This topology provides high availability, scale-out capacity up to sixteen nodes, and failover.


VMware, Inc. 177

n 2 remote collector nodes in each region. The remote collectors communicate directly with the datanodes in the vRealize Operations Manager analytics cluster. For load balancing and fault tolerance,two remote collectors are deployed in each region.

Each region contains its own pair of remote collectors whose role is to ease scalability by performing thedata collection from the applications that are not a subject of failover and periodically sending collecteddata to the analytics cluster. You fail over the analytics cluster only because the analytics cluster is theconstruct that analyzes and stores monitoring data. This configuration supports failover of the analyticscluster by using Site Recovery Manager. In the event of a disaster, Site Recovery Manager migrates theanalytics cluster nodes to the failover region.

Figure 2‑33. Logical Design of vRealize Operations Manager Multi-Region Deployment

RemoteCollector

RemoteCollector

Management/Compute

vCenter Server

vRealizeAutomation

vRealizeLog Insight

vRealizeLog Insight

Remote Collector Cluster

RemoteCollector

NSX

Analytics Cluster

Management/Compute

vCenter Server

NSX

Region A Region B

RemoteCollector

Shared Storage

Remote Collector Cluster

Master MasterReplica Data 1 Data n

Shared Storage

Physical Design

The vRealize Operations Manager nodes run on the management pod in each region of SDDC. Forinformation about the types of pods, see Pod Architecture.

Data Sources

vRealize Operations Manager collects data from the following virtual infrastructure and cloudmanagement components.

n Management pod

n Platform Services Controller

n vCenter Server

n ESXi hosts

n Shared edge and compute pod



VMware, Inc. 178

n vCenter Server

n ESXi hosts

n NSX for vSphere for the management cluster and for the shared edge and compute cluster

n NSX Manager

n NSX Controller Instances

n NSX Edge instances

n vRealize Automation

n vRealize Automation Appliance

n vRealize IaaS Web Server

n vRealize IaaS Management Server

n vRealize IaaS DEM

n vRealize Agent Servers

n vRealize Orchestrator

n Microsoft SQL Server

n vRealize Log Insight

n vRealize Operations Manager

vRealize Operations Manager NodesThe analytics cluster of the vRealize Operations Manager deployment contains the nodes that analyzeand store data from the monitored components. You deploy a configuration of the analytics cluster thatsatisfies the requirements for monitoring the number of virtual machines according to the designobjectives of this VMware Validated Design.

Deploy a 3-node vRealize Operations Manager analytics cluster in the cross-region application virtualnetwork. The analytics cluster consists of one master node, one master replica node, and one data nodeto enable scale out and high availability.

Table 2‑159. Analytics Cluster Node Configuration Design Decision


SDDC-OPS-MON-001

Deploy initially vRealizeOperations Manager as acluster of three nodes: onemaster, one master replicaand one data node.

Provides the initial scale capacityrequired for monitoring up to 1,000 VMsand provides the ability to scale up withadditional data nodes as increasedscale requires.

You must size identically allappliances which increases theresources requirements in theSDDC.

Requires manual installation ofadditional data nodes as per the datanode scale guidelines.

SDDC-OPS-MON-002

Deploy two remote collectornodes per region.

Removes the load from the analyticscluster from collecting metrics fromapplications that do not fail overbetween regions.

When configuring the monitoring of asolution, you must assign a collectorgroup.


VMware, Inc. 179

Sizing Compute Resources

You size compute resources for vRealize Operations Manager to provide enough resources foraccommodating the analytics operations for monitoring the SDDC.

Size the vRealize Operations Manager analytics cluster according to VMware Knowledge Base article 2093783. vRealize Operations Manager is sized so as to accommodate the SDDC design by deployingthe following management packs:

n Management Pack for VMware vCenter Server (installed by default)

n Management Pack for NSX for vSphere

n Management Pack for Storage Devices

n Management Pack for vRealize Log Insight (installed by default)

n Management Pack for vRealize Automation

The sizing of the vRealize Operations Manager instance is calculated with the following two options:

Initial Setup Scaled Setup

4 vCenter Server Appliances 4 vCenter Server Appliances

4 NSX Managers 4 NSX Managers

6 NSX Controllers 6 NSX Controllers

50 ESXi hosts 100 ESXi hosts

4 vSAN datastores 4 vSAN datastores

1,000 virtual machines 10,000 virtual machines

Sizing Compute Resources for the Analytics Cluster Nodes

Deploying 3 medium-size virtual appliances satisfies the initial requirement for retention and formonitoring the expected number of objects and metrics for smaller environments up to 1,000 virtualmachines. As the environment grows, you should deploy more data notes to accommodate the largerexpected number of objects and metrics. Consider deploying additional vRealize Operations Managerdata notes only if more ESXi hosts are added to the management pods to guarantee that the vSpherecluster has enough capacity to host these additional nodes without violating the vSphere DRS anti-affinityrules.

Table 2‑160. Size of a Medium vRealize Operations Manager Virtual Appliance


Appliance size Medium

vCPU 8

Memory 32 GB

Single-Node Maximum Objects 7,000

Single-Node Maximum Collected Metrics (*) 2,000,000

Multi-Node Maximum Objects Per Node (**) 5,000


VMware, Inc. 180

https://kb.vmware.com/kb/2093783

Table 2‑160. Size of a Medium vRealize Operations Manager Virtual Appliance(Continued)


Multi-Node Maximum Collected Metrics Per Node (**) 1,500,000

Maximum number of End Point Operations Management agents per node 1,200

(*) Metric numbers reflect the total number of metrics that are collected from all adapter instances invRealize Operations Manager. To get this number, you can go to the Cluster Management page invRealize Operations Manager, and view the adapter instances of each node at the bottom of the page.You can get the number of metrics collected by each adapter instance. The sum of these metrics is whatis estimated in this sheet.

Note The number shown in the overall metrics on the Cluster Management page reflects the metricsthat are collected from different data sources and the metrics that vRealize Operations Manager creates.

(**) Note the reduction in maximum metrics to permit some head room.


VMware, Inc. 181

Table 2‑161. Analytics Cluster Node Size Design Decision


SDDC-OPS-MON-003

Deploy each node in the analyticscluster as a medium-size appliance.

Provides the scale required tomonitor the SDDC when atfull capacity.

If you use a lower number oflarge-size vRealizeOperations Manager nodes,you must increase theminimum host memory size tohandle the increasedperformance that is the resultfrom stretching NUMA nodeboundaries.

Hypervisor hosts used in themanagement cluster must havephysical CPU processor with aminimum of 8 cores per socket.In total this will utilize 24vCPUs and 96 GB of memoryin the management cluster.

SDDC-OPS-MON-004

Initially deploy 3 medium-size nodes forthe first 1,000 virtual machines in thecompute pod.

Provides enough capacity forthe metrics and objectsgenerated by 100 hosts and1,000 virtual machines whilehaving high availabilityenabled within the analyticscluster.

The first 3 medium-size nodestake more resources per 1,000virtual machines because theyhave to accommodate therequirements for highavailability. Nodes that aredeployed next can spread thisload out more evenly.

SDDC-OPS-MON-005

Add more medium-size nodes to theanalytics cluster if the SDDC expandspast 1,000 virtual machines.

The number of nodes should notexceed number of ESXi hosts inthe management pod - 1.

For example, if the management podcontains 6 ESXi hosts, you deploy amaximum of 5 vRealize OperationsManager nodes in the analytics cluster.

Ensures that the analyticscluster has enough capacityto meet the virtual machineobject and metrics growth upto 10,000 virtual machines.

Ensures that the managementpod always has enoughphysical capacity to take ahost offline for maintenanceor other reasons.

The capacity of the physicalESXi hosts must be largeenough to accommodate virtualmachines that require 32 GBRAM without bridging NUMAnode boundaries.

The management pod musthave enough ESXi hosts sothat vRealize OperationsManager can run withoutviolating vSphere DRS anti-affinity rules.

Sizing Compute Resources for the Remote Collector Nodes

Unlike the analytics cluster nodes, remote collector nodes have only the collector role. Deploying tworemote collector nodes in each region does not increase the capacity for monitored objects.

Table 2‑162. Size of a Standard Remote Collector Virtual Appliance for vRealize OperationsManager


Appliance size Remote Collector - Standard

vCPU 2

Memory 4 GB

Single-node maximum Objects(*) 1,500

Single-Node Maximum Collected Metrics 600,000


VMware, Inc. 182

Table 2‑162. Size of a Standard Remote Collector Virtual Appliance for vRealize OperationsManager (Continued)


Multi-Node Maximum Objects Per Node N/A

Multi-Node Maximum Collected Metrics Per Node N/A

Maximum number of End Point Operations Management Agents per Node 250

Maximum Objects for 16-Node Maximum N/A

Maximum Metrics for 16-Node Configuration N/A

*The object limit for the remote collector is based on the VMware vCenter adapter.

Table 2‑163. Configuration and Resources Requirements of the Remote Collector NodesDesign Decision


SDDC-OPS-MON-006 Deploy the standard-size remotecollector virtual appliances.

Enables metric collection forthe expected number ofobjects in the SDDC when atfull capacity.

You must provide 4 vCPUs and8 GB of memory in themanagement cluster in eachregion.

Sizing Storage

You allocate storage capacity for analytics data collected from the management products and from thenumber of tenant virtual machines that is defined in the objectives of this SDDC design.

This design uses medium-size nodes for the analytics and remote collector clusters. A vRealizeOperations Manager node of a medium size requires 266 GB of free space for data. To collect therequired number of metrics, you must add a 1 TB VMDK to each analytics cluster node.

Sizing Storage for the Analytics Cluster Nodes

The analytics cluster processes a large amount of objects and metrics. As the environment grows, theneed to add more data nodes to the analytics cluster will emerge. Refer to the vRealize OperationsManager sizing guidelines in VMware Knowledge Base article 2093783 to plan the sizing requirements ofyour environment.

Table 2‑164. Analytics Cluster Node Storage Design Decision


SDDC-OPS-MON-007 Provide a 1 TB VMDK foreach analytics cluster node.

Provides enough storage tomeet the SDDC designobjectives.

You must add the 1 TB disk manuallywhile the virtual machine for theanalytics node is powered off.

Sizing Storage for the Remote Collector Nodes

Deploy the remote collector nodes with thin-provisioned disks. Because remote collectors do not performanalytics operations or store data, the default VMDK size is sufficient.


VMware, Inc. 183

https://kb.vmware.com/kb/2093783

Table 2‑165. Remote Collector Node Storage Design Decision


SDDC-OPS-MON-008 Do not provide additional storagefor remote collectors.

Remote collectors do not perform analyticsoperations or store data on disk.

None.

Networking DesignYou place the vRealize Operations Manager nodes in several network units for isolation and failover. Thenetworking design also supports public access to the analytics cluster nodes.

For secure access, load balancing and portability, the vRealize Operations Manager analytics cluster isdeployed in the shared cross-region application isolated network Mgmt-xRegion01-VXLAN, and theremote collector clusters in the shared local application isolated networks Mgmt-RegionA01-VXLAN andMgmt-RegionB01-VXLAN.


VMware, Inc. 184

Figure 2‑34. Networking Design of the vRealize Operations Manager Deployment

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OSAPP

OSAPP

OS


DataCenterUser

ActiveDirectory


Region A(SF001 - San Francisco)


mgmt01vc51.lax01.rainpole.

local

Compute vCenter Server

comp01vc51.lax01.rainpole.

local

rmtcol-01 rmtcol-02


VLAN: vSphere-Mgmt 172.16.11.0/24 VLAN: vSphere-Mgmt172.17.11.0/24


mgmt01vc01.sfo01.rainpole.

local

Compute vCenter Server

comp01vc01.sfo01.rainpole.

local

PhysicalUpstream

Router

mstrn-01 repln-02 datan-03

192.168.11.0/24


vrops-cluster-01.rainpole.local

Analytics Cluster Region A

192.168.31.0/24

Remote Collectors Region A

Region B(LAX01 - Los Angeles)

PhysicalUpstream

Router

rmtcol-51 rmtcol-52


192.168.32.0/24

Remote Collectors

Region B

mstrn-01 repln-02 datan-03

192.168.11.0/24

vrops-cluster-01.rainpole.local

Placeholder Disaster RecoveryAnalytics Cluster

Region B

SFOMGMT-LB01 LAXMGMT-LB01



VMware, Inc. 185

Application Isolated Network Design

The vRealize Operations Manager analytics cluster is installed into the cross-region shared applicationisolated network and the remote collector nodes are installed in their region specific shared applicationisolated networks.

This networking design has the following features:

n Each application component that fails over between regions, such as the analytics cluster forvRealize Operations Manager, is on the same network. vRealize Automation and vRealize Log Insightalso share components on this network.

n All nodes have routed access to the vSphere management network through the NSX UniversalDistributed Logical Router.

n Routing to the vSphere management network and other external networks is dynamic, and is basedon the Border Gateway Protocol (BGP).

For more information about the networking configuration of the application isolated network, see Virtualization Network Design and NSX Design.

Table 2‑166. vRealize Operations Manager Isolated Network Design Decisions


SDDC-OPS-MON-009

Use the existing cross-regionapplication virtual networks forthe vRealize OperationsManager analytics cluster .

Support disaster recovery by isolatingvRealize Operations Manager analytics clusteron the application virtual network Mgmt-xRegion01-VXLAN.

You must implement NSXto support this networkconfiguration.

SDDC-OPS-MON-010

Use the existing region-specific application virtualnetworks for vRealizeOperations Manager remotecollectors.

Ensures collections of metrics locally perregion in the event of a cross-region networkoutage. Additionally, it co-localized metriccollection to the per-region SDDC applicationsusing the virtual networks Mgmt-RegionA01-VXLAN and Mgmt-RegionB01-VXLAN.

You must implement NSXto support this networkconfiguration.

IP Subnets

You can allocate the following example subnets for each cluster in the vRealize Operations Managerdeployment.

Table 2‑167. IP Subnets in the Application Virtual Network of vRealize Operations Manager

vRealize Operations Manager Cluster Type IP Subnet

Analytics cluster in Region A (also valid for Region B for failover) 192.168.11.0/24

Remote collectors in Region A 192.168.31.0/24

Remote collectors in Region B 192.168.32.0/24


VMware, Inc. 186

Table 2‑168. IP Subnets Design Decision


SDDC-OPS-MON-011 Allocate separate subnets foreach application isolatednetwork.

Placing the remote collectors on their own subnetenables them to communicate with the analyticscluster and not be a part of the failover group.

None.

DNS Names

vRealize Operations Manager node name resolution uses a region-specific suffix, such assfo01.rainpole.local or lax01.rainpole.local. The analytics nodes IP addresses and the load balancervirtual IP address (VIP) are mapped to the root domain suffix rainpole.local. Access from the publicnetwork is provided through a VIP, the traffic to which is handled by the NSX Edge service gateway.

Table 2‑169. DNS Names for the Application Virtual Networks

vRealize Operations Manager DNS Name Node Type

vrops-cluster-01.rainpole.local Virtual IP of the analytics cluster

vrops-mstrn-01.rainpole.local Master node in the analytics cluster

vrops-repln-02.rainpole.local Master replica node in the analytics cluster

vrops-datan-03.rainpole.local First data node in the analytics cluster

vrops-datan-0x.rainpole.local Additional data nodes in the analytics cluster

vrops-rmtcol-01.sfo01.rainpole.local First remote collector node in Region A

vrops-rmtcol-02.sfo01.rainpole.local Second remote collector node in Region A

vrops-rmtcol-51.lax01.rainpole.local First remote collector node in Region B

vrops-rmtcol-52.lax01.rainpole.local Second remote collector node in Region B

Networking for Failover and Load Balancing

By default, vRealize Operations Manager does not provide a solution for load-balanced UI user sessionsacross nodes in the cluster. You associate vRealize Operations Manager with the shared load balancer inthe region.

The lack of load balancing for user sessions results in the following limitations:

n Users must know the URL of each node to access the UI. As a result, a single node might beoverloaded if all users access it at the same time.

n Each node supports up to four simultaneous user sessions.

n Taking a node offline for maintenance might cause an outage. Users cannot access the UI of thenode when the node is offline.

To avoid such problems, place the analytics cluster behind an NSX load balancer that is configured toallow up to four connections per node. The load balancer must distribute the load evenly to all clusternodes. In addition, configure the load balancer to redirect service requests from the UI on port 80 to port443.

Load balancing for the remote collector nodes is not required.


VMware, Inc. 187

Table 2‑170. Networking Failover and Load Balancing Design Decisions


SDDC-OPS-MON-012

Use an NSX Edge servicesgateway as a load balancer forthe vRealize OperationManager analytics cluster.

Enables balanced access of tenants andusers to the analytics services with the loadbeing spread evenly across the cluster.

You must manuallyconfigure the NSX Edgedevices to provide loadbalancing services.

SDDC-OPS-MON-013

Do not use a load balancer forthe remote collector nodes.

n Remote collectors must directly accessthe systems that they are monitoring.

n Remote collectors do not require accessto and from the public network.

None.

Information Security and Access ControlYou configure a central source for authentication in vRealize Operations Manager such as an ActiveDirectory service. vRealize Operations Manager also authenticates to vCenter Server and can use a localuser inventory.

Authentication and Authorization

You can allow users to authenticate in vRealize Operations Manager in the following ways:

Import users or usergroups from an LDAPdatabase

Users can use their LDAP credentials to log in to vRealize OperationsManager.

Use vCenter Serveruser accounts

After a vCenter Server instance is registered with vRealize OperationsManager, the following vCenter Server users can log in to vRealizeOperations Manager:

n Users that have administration access in vCenter Server.

n Users that have one of the vRealize Operations Manager privileges,such as PowerUser, assigned to the account which appears at the rootlevel in vCenter Server.

Create local useraccounts in vRealizeOperations Manager

vRealize Operations Manager performs local authentication using theaccount information stored in its global database.


VMware, Inc. 188



SDDC-OPS-MON-014

Use Active Directoryauthentication.

n Provides access to vRealize OperationsManager by using standard Active Directoryaccounts.

n Ensures that authentication is available evenif vCenter Server becomes unavailable.

You must manually configurethe Active Directoryauthentication.

SDDC-OPS-MON-015

Configure a service accountsvc-vrops in vCenter Serverfor application-to-applicationcommunication fromvRealize OperationsManager with vSphere andNSX for vSphere.

Provides the following access control features:n The adapters in vRealize Operations

Manager access vSphere and NSX forvSphere with the minimum set of permissionsthat are required to collect metrics aboutvSphere inventory objects.




SDDC-OPS-MON-016

Configure a service accountsvc-mpsd-vrops in vCenterServer for application-to-application communicationfrom the Storage DevicesAdapters in vRealizeOperations Manager withvSphere.


Manager access vSphere with the minimumset of permissions that are required to collectmetrics about vSphere inventory objects.




SDDC-OPS-MON-017

Use global permissions whenyou create the svc-vrops andsvc-mpsd-vrops serviceaccounts in vCenter Server.

n Simplifies and standardizes the deployment ofthe service accounts across all vCenterServer instances in the same vSpheredomain.


All vCenter Server instancesmust be in the same vSpheredomain.


VMware, Inc. 189

Table 2‑171. Authorization and Authentication Management Design Decisions (Continued)


SDDC-OPS-MON-018

Configure a service accountsvc-vrops-vra in vRealizeAutomation for application-to-application communicationfrom the vRealizeAutomation Adapter invRealize OperationsManager with vRealizeAutomation.

Provides the following access control features:n The adapter in vRealize Operations Manager

accesses vRealize Automation with theminimum set of permissions that are requiredfor collecting metrics about provisioned virtualmachines and capacity management.



n You must maintain theservice account's life cycleoutside of the SDDC stackto ensure its availability.

n If you add more tenants tovRealize Automation, youmust maintain the serviceaccount permissions toguarantee that metricuptake in vRealizeOperations Manager is notcompromised.

SDDC-OPS-MON-019

Configure a local serviceaccount svc-vrops-nsx ineach NSX instance forapplication-to-applicationcommunication from theNSX-vSphere Adapters invRealize OperationsManager with NSX.


Manager access NSX for vSphere with theminimum set of permissions that are requiredfor metrics collection and topology mapping.




Encryption

Access to all vRealize Operations Manager Web interfaces requires an SSL connection. By default,vRealize Operations Manager uses a self-signed certificate. Replace the default self-signed certificateswith a CA-signed certificate to provide secure access to the vRealize Operations Manager user interface.

Table 2‑172. Using CA-Signed Certificates Design Decision


SDDC-OPS-MON-020 Replace the default self-signedcertificates with a CA-signedcertificate.

Ensures that all communication to theexternally facing Web UI is encrypted.

You must contact acertificate authority.

Monitoring and AlertingvRealize Operations Manager can monitor itself and displays alerts about issues with its operationalstate.


VMware, Inc. 190

vRealize Operations Manager display the following administrative alerts:

System alert A component of the vRealize Operations Manager application has failed.

Environment alert vRealize Operations Manager has stopped receiving data from one or moreresources. Such an alert might indicate a problem with system resources ornetwork infrastructure.

Log Insight log event The infrastructure on which vRealize Operations Manager is running haslow-level issues. You can also use the log events for root cause analysis.

Custom dashboard vRealize Operations Manager can show super metrics for data centermonitoring, capacity trends and single pane of glass overview.

Table 2‑173. Monitoring vRealize Operations Manager Design Decisions


SDDC-OPS-MON-021

Configure vRealize OperationsManager for SMTP outboundalerts.

Enables administrators and operators toreceive alerts from vRealize OperationsManager by e-mail.

Requires access to anexternal SMTP server.

SDDC-OPS-MON-022

Configure vRealize OperationsManager custom dashboards.

Provides extended SDDC monitoring,capacity trends and single pane of glassoverview.

Requires manuallyconfiguring the dashboards.

Management PacksThe SDDC contains several VMware products for network, storage, and cloud management. You canmonitor and perform diagnostics on all of them in vRealize Operations Manager by using managementpacks.

Table 2‑174. Management Packs for vRealize Operations Manager Design Decisions


SDDC-OPS-MON-023

Install the following managementpacks:n Management Pack for NSX for

vSpheren Management Pack for Storage

Devicesn Management Pack for vRealize

Automation

Provides additional granular monitoring forall virtual infrastructure and cloudmanagement applications.

You do not have the install the followingmanagement packs because they areinstalled by default in vRealize OperationsManager:n Management Pack for VMware vCenter

Servern Management Pack for vRealize Log

Insight

Requires manuallyinstalling and configuringeach non-defaultmanagement pack.

SDDC-OPS-MON-024

Configure the solutions that fail overbetween regions with the defaultremote collector group.

Provides monitoring for all componentsduring a failover.

Adds minimal additionalload to the analyticscluster


VMware, Inc. 191

vRealize Log Insight DesignvRealize Log Insight design enables real-time logging for all components that build up the managementcapabilities of the SDDC in a dual-region setup.

Logical Design and Data Sources of vRealize Log InsightvRealize Log Insight collects log events from all management components in both regions of the SDDC.

Logical Design

In a multi-region Software Defined Data Center (SDDC) deploy a vRealize Log Insight cluster in eachregion that consists of three nodes. This configuration allows for continued availability and increased logingestion rates.

Figure 2‑35. Logical Design of vRealize Log Insight

Management/Compute

vCenter Server

vRealizeAutomation



NSX

vRealize Operations

Manager

event forwarding

VMDK Storage

Archiving Storage

Management/Compute

vCenter Server



NSX

VMDK Storage

Archiving Storage

Region A Region B

Sources of Log Data

vRealize Log Insight collects logs as to provide monitoring information about the SDDC from a centrallocation.

vRealize Log Insight collects log events from the following virtual infrastructure and cloud managementcomponents.

n Management pod


n vCenter Server

n ESXi Hosts


VMware, Inc. 192

n Shared edge and compute pod


n vCenter Server

n ESXi Hosts

n NSX for vSphere for the management cluster and for the shared compute and edge cluster

n NSX Manager

n NSX Controller instances


n vRealize Automation

n vRealize Automation Appliance



n vRealize IaaS DEM




n vRealize Operations Manager

n Analytics cluster nodes

Cluster Nodes of vRealize Log InsightThe vRealize Log Insight cluster consists of one master node and two worker nodes. You enable theIntegrated Load Balancer (ILB) on the cluster to have vRealize Log Insight to balance incoming trafficfairly among available nodes.

vRealize Log Insight clients, using both the Web user interface, and ingestion through syslog or theIngestion API, connect to vRealize Log Insight that the ILB addresses.

vRealize Log Insight cluster can scale out to 12 nodes, that is, one master and 11 worker nodes.


VMware, Inc. 193

Table 2‑175. Cluster Node Configuration Design Decision


SDDC-OPS-LOG-001

Deploy vRealize Log Insight ina cluster configuration of 3nodes with an integrated loadbalancer: one master and twoworker nodes.

Provides high availability. Usingthe integrated load balancersimplifies the Log Insightdeployment, and prevents from asingle point of failure.

You must size each node identically.

If the capacity requirements for yourvRealize Log Insight cluster grow,identical capacity must be added to eachnode.

SDDC-OPS-LOG-002

Apply vSphere DistributedResource Scheduler (DRS)anti-affinity rules to the vRealizeLog Insight cluster components

Using DRS prevents vRealizeLog Insight nodes from on thesame ESXi host and therebyrisking the cluster's highavailability capability.

Additional configuration is required to setup anti-affinity rules.

Only a single ESXi host in themanagement cluster, of the four ESXihosts, will be able to be put intomaintenance mode at a time.

Sizing Log Insight NodesTo accommodate all log data from the products in the SDDC, you must size the compute resources andstorage for the Log Insight nodes properly.

By default, the vRealize Log Insight virtual appliance uses the predefined values for small configurations,which have 4 vCPUs, 8 GB of virtual memory, and 510 GB of disk space provisioned. vRealize LogInsight uses 100 GB of the disk space to store raw data, index, metadata, and other information.

Sizing Nodes

Select a size for the vRealize Log Insight nodes to collect and store log data from the SDDC managementcomponents and tenant workloads according to the objectives of this design.

Table 2‑176. Compute Resources for a vRealize Log Insight Medium-Size Node


Appliance size Medium

Number of CPUs 8

Memory 16 GB

Disk Capacity 510 GB (490 GB for event storage)

IOPS 1,000 IOPS

Amount of processed log data when using log ingestion 75 GB/day of processing per node

Number of processed log messages 5,000 event/second of processing per node

Environment Up to 250 syslog connections per node


VMware, Inc. 194

Sizing Storage

Sizing is based on IT organization requirements, but this design provides calculations based on a singleregion implementation, and is implemented on a per-region basis. This sizing is calculated according tothe following node configuration per region:

n Management vCenter Servern Platform Services Controller

n vCenter Server

n Compute vCenter Servern Platform Services Controller

n vCenter Server

n Management, shared edge and compute ESXi hosts

n NSX for vSphere for the management cluster and for the shared compute and edge clustern NSX Manager

n NSX Controller instances


n vRealize Automationn vRealize Automation Appliance



n vRealize IaaS DEM




n vRealize Operations Managern Analytics cluster nodes

n Event forwarding configured between vRealize Log Insight clusters

These components aggregate to approximately 210 syslog and vRealize Log Insight Agent sources.Assuming that you want to retain 7 days of data, use the following calculations:

For 210 syslog sources at a basal rate of 150 MB of logs ingested per-day per-source over 7 days, youneed the following storage space:

210 sources * 150 MB of log data ≈ 31.5 GB log data per-day

31.5 GB * 7 days ≈ 220.5 GB log data per vRealize Log Insight node

220.5 GB * 1.7 indexing overhead ≈ 375 GB


VMware, Inc. 195

Based on this example, the storage space that is allocated per medium-size vRealize Log Insight virtualappliance is enough to monitor the SDDC.

Consider the following approaches when you must increase the Log Insight capacity:

n If you must maintain a log data retention for more than 7 days in your SDDC, you might add morestorage per node by adding a new virtual hard disk. vRealize Log Insight supports virtual hard disks ofup to 2 TB. If you must add more than 2 TB to a virtual appliance, add another virtual hard disk.

When you add storage to increase the retention period, extend the storage for all virtual appliances.

Note Do not extend existing retention virtual disks. Once provisioned, do not reduce the size orremove virtual disks to avoid data loss.

n If you must monitor more components by using log ingestion and exceed the number of syslogconnections or ingestion limits defined in this design, you can deploy more vRealize Log Insightvirtual appliances to scale out your environment. vRealize Log Insight can scale up to 12 nodes in anHA cluster.

Table 2‑177. Compute Resources for the vRealize Log Insight Nodes Design Decisions


SDDC-OPS-LOG-003

Deploy vRealizeLog Insight nodesof medium size.

Accommodates the number of expected Syslog and vRealizeLog Insight Agent connections from the following. This isapproximately 210 syslog and vRealize Log Insight Agentsources.n Management & Compute vCenter Server, Platform

Services Controllern Management, and shared edge and compute ESXi hostsn Management and compute components for NSX for

vSpheren vRealize Automation componentsn vRealize Operations Manager componentsn Cross-vRealize Log Insight cluster event forwarding.

Using a medium-size appliances ensures that the storagespace for the vRealize Log Insight cluster is sufficient for 7days of data retention.

You must increase thesize of the nodes if youconfigure Log Insight tomonitor additional syslogsources.

vRealize Log Insight Networking DesignIn both regions, the vRealize Log Insight instances are connected to the region-specific managementVXLANs Mgmt-RegionA01-VXLAN and Mgmt-RegionB01-VXLAN. Each vRealize Log Insight instance isdeployed within the shared management application isolated network.


VMware, Inc. 196

Figure 2‑36. Networking Design for the vRealize Log Insight Deployment

APP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OS

vrli-mstr-51 vrli-wrkr-51

192.168.32.0/24

vRealize Log Insight Region B

vrli-wrkr-52vrli-cluster-51

192.168.31.0/24Mgmt-RegionA01-VXLAN

vRealize Log Insight Region A

vrli-mstr-01 vrli-wrkr-01 vrli-wrkr-02vrli-cluster-01



Application Network Design

This networking design has the following features:

n All nodes have routed access to the vSphere management network through the Management NSXUDLR for the home region.

n Routing to the vSphere management network and the external network is dynamic, and is based onthe Border Gateway Protocol (BGP).

For more information about the networking configuration of the application isolated networks for vRealizeLog Insight, see Application Virtual Network and Virtual Network Design Example.

Table 2‑178. vRealize Log Insight Network Design Decision


SDDC-OPS-LOG-004

Deploy vRealize LogInsight on the region-specific applicationvirtual networks.

n Ensures centralized access to log dataper region if a cross-region networkoutage occurs.

n Co-located log collection to the regionlocal SDDC applications using theregion-specific application virtualnetworks.

n Provides a consistent deployment modelfor management applications.

n Interruption in the cross-regionnetwork can impact eventforwarding between the vRealizeLog Insight clusters and cause gapsin log data.

n You must use NSX to support thisnetwork configuration.

IP Subnets

You can allocate the following example subnets to the vRealize Log Insight deployment.

Table 2‑179. IP Subnets in the Application Isolated Networks

vRealize Log Insight Cluster IP Subnet

Region A 192.168.31.0/24

Region B 192.168.32.0/24


VMware, Inc. 197

vRealize Log Insight DNS Names

vRealize Log Insight node name resolution uses a region-specific suffix, such as sfo01.rainpole.local orlax01.rainpole.local, including the load balancer virtual IP addresses (VIPs). The Log Insight componentsin both regions have the following node names.

Table 2‑180. DNS Names of the vRealize Log Insight Nodes

DNS Name Role Region

vrli-cluster-01.sfo01.rainpole.local Log Insight ILB VIP A

vrli-mstr-01.sfo01.rainpole.local Master node A

vrli-wrkr-01.sfo01.rainpole.local Worker node A

vrli-wrkr-02.sfo01.rainpole.local Worker node A

vrli-cluster-51.lax01.rainpole.local Log Insight ILB VIP B

vrli-mstr-51.lax01.rainpole.local Master node B

vrli-wrkr-51.lax01.rainpole.local Worker node B

vrli-wrkr-52.lax01.rainpole.local Worker node B

Table 2‑181. DNS Names Design Decisions


SDDC-OPS-LOG-005

Configure forward and reverseDNS records for all vRealize LogInsight nodes and VIPs.

All nodes are accessible by usingfully-qualified domain namesinstead of by using IP addressesonly.

You must manually provide a DNSrecord for each node and VIP.

SDDC-OPS-LOG-006

For all applications that fail overbetween regions (such asvRealize Automation and vRealizeOperations Manager), use theFQDN of the vRealize Log InsightRegion A VIP when you configurelogging.

Support logging when not allmanagement applications arefailed over to Region B. Forexample, only one application ismoved to Region B.

If vRealize Automation and vRealizeOperations Manager are failed over toRegion B and the vRealize Log Insightcluster is no longer available in RegionA, update the A record on the childDNS server to point to the vRealize LogInsight cluster in Region B.

vRealize Log Insight Retention and ArchivingConfigure archive and retention parameters of vRealize Log Insight according to the company policy forcompliance and governance.

vRealize Log Insight virtual appliances contain three default virtual disks and can use addition virtualdisks for storage.


VMware, Inc. 198

Table 2‑182. Virtual Disk Configuration in the vRealize Log Insight Virtual Appliance

Hard Disk Size Usage

Hard disk 1 20 GB Root file system

Hard disk 2 510 GB for medium-size deployment Contains two partitions:n /storage/var System logsn /storage/core Storage for Collected logs.

Hard disk 3 512 MB First boot only

Calculate the storage space that is available for log data using the following equation:

/storage/core = hard disk 2 space - system logs space on hard disk 2

Based on the size of the default disk, the storage core is equal to 490 GB.

/storage/core = 510GB - 20 GB = 490 GB

Retention = /storage/core – 3% * /storage/core

If /storage/core is 490 GB, vRealize Log Insight can use 475 GB for retention.

Retention = 490 GB - 3% * 490 ≈ 475 GB

Configure a retention period of 7 days for the medium-size vRealize Log Insight appliance.

Table 2‑183. Retention Period Design Decision


SDDC-OPS-LOG-007 Configure vRealize Log Insight toretain data for 7 days.

Accommodates logs from 750 syslog sources(250 per node) as per the SDDC design.

None

Archiving

You configure vRealize Log Insight to archive log data only if your must retain logs for an extended periodof time, either for compliance, auditability, and so on.

vRealize Log Insight archives log messages as soon as possible. At the same time, the logs are retainedon the virtual appliance until the free local space is almost filled. Data exists on both the vRealize LogInsight appliance and the archive location for most of the retention period. The archiving period must belonger than the retention period.

The archive location must be on an NFS version 3 shared storage. The archive location must be availableand must have enough capacity to accommodate the archives.

Apply an archive policy of 90 days for the medium-size vRealize Log Insight appliance. The vRealize LogInsight appliance will use about 1 TB of shared storage. According to the business complianceregulations of your organization, these sizes might change.


VMware, Inc. 199

Table 2‑184. Log Archive Policy Design Decision


SDDC-OPS-LOG-008

Provide 1 TB of NFSversion 3 sharedstorage to eachvRealize Log Insightcluster.

Archives logs from750 syslog sources.

n You must manually maintain the vRealize Log Insightarchive blobs stored on the NFS store, selectively pruningas more space is required.

n You must enforce the archive policy directly on the sharedstorage.

n If the NFS mount does not have enough free space or isunavailable for a period greater than the retention period ofthe virtual appliance, vRealize Log Insight stops ingestingnew data until the NFS mount has enough free space,becomes available, or archiving is disabled.

vRealize Log Insight AlertingvRealize Log Insight supports alerts that trigger notifications about its health.

Alert Types

The following types of alerts exist in vRealize Log Insight:

System Alerts vRealize Log Insight generates notifications when an important systemevent occurs, for example when the disk space is almost exhausted andvRealize Log Insight must start deleting or archiving old log files.

Content Pack Alerts Content packs contain default alerts that can be configured to sendnotifications, these alerts are specific to the content pack and are disabledby default.

User-Defined Alerts Administrators and users can define their own alerts based on dataingested by vRealize Log Insight.

vRealize Log Insight handles alerts in two ways:

n Send an e-mail over SMTP.

n Send to vRealize Operations Manager.

SMTP Notification

Enable e-mail notification for alerts in vRealize Log Insight.

Table 2‑185. SMTP Alert Notification Design Decision


SDDC-OPS-LOG-009 Enable alerting overSMTP.

Enables administrators and operators toreceive alerts via email from vRealize LogInsight.

Requires access to an externalSMTP server.


VMware, Inc. 200

Integration with vRealize Operations Manager

vRealize Log Insight provide integra with vRealize Operations Manager to provide a central location formonitoring and diagnostics. You can use the following integration points that you can enable separately:

Notification Events Forward notification events from vRealize Log Insight to vRealizeOperations Manager.

Launch in Context Launch vRealize Log Insight from the vRealize Operation Manager userinterface.

Table 2‑186. Integration with vRealize Operations Manager Design Decision


SDDC-OPS-LOG-010

Forward alerts tovRealize OperationsManager.

Provides monitoring and alertinginformation that is pushed fromvRealize Log Insight to vRealizeOperations Manager for centralizedadministration.

You must install the vRealize LogInsight management pack into vRealizeOperation Manager. This managementpack is packaged with vRealizeOperations Manager 6.0 and later.

SDDC-OPS-LOG-011

Allow for Launch InContext with vRealizeOperation Manager

Provides the ability to access vRealizeLog Insight for context-basedmonitoring of an object in vRealizeOperations Manager.

You can register only one vRealize LogInsight cluster with vRealize OperationsManager at a time.

Information Security and Access Control in vRealize Log InsightProtect the vRealize Log Insight deployment by providing centralized role-based authentication andsecure communication with the other components in the Software-Defined Data Center (SDDC).

Authentication

Enable role-based access control in vRealize Log Insight by using the existing rainpole.local ActiveDirectory domain.


VMware, Inc. 201



SDDC-OPS-LOG-012

Use Active Directory forauthentication.

Provides fine-grained role and privilege-basedaccess for administrator and operator roles.

You must provide accessto the Active Directoryfrom all Log Insightnodes.

SDDC-OPS-LOG-013

Configure a service accountsvc-loginsight on vCenterServer for application-to-application communicationfrom vRealize Log Insight withvSphere.

Provides the following access control features:n vRealize Log Insight accesses vSphere with the

minimum set of permissions that are required tocollect vCenter Server events, tasks and alarmsand to configure ESXi hosts for syslogforwarding.


n You can introduce improved accountability intracking request-response interactions betweenthe components of the SDDC.

You must maintain theservice account's lifecycle outside of theSDDC stack to ensure itsavailability.

SDDC-OPS-LOG-014

Use global permissions whenyou create the svc-loginsightservice account in vCenterServer.

n Simplifies and standardizes the deployment ofthe service account across all vCenter Serversin the same vSphere domain.



SDDC-OPS-LOG-015

Configure a service accountsvc-vrli-vrops on vRealizeOperations Manager forapplication-to-applicationcommunication from vRealizeLog Insight for a two-waylaunch in context.

Provides the following access control features:n vRealize Log Insight and vRealize Operations

Manager access each other with the minimumset of required permissions.




Encryption

Replace default self-signed certificates with a CA-signed certificate to provide secure access to thevRealize Log Insight Web user interface.

Table 2‑188. Custom Certificates Design Decision


SDDC-OPS-LOG-016

Replace the default self-signed certificates with a CA-signed certificate.

Configuring a CA-signed certificateensures that all communication to theexternally facing Web UI is encrypted.

The administrator must haveaccess to a Public KeyInfrastructure (PKI) to acquirecertificates.

Configuration for Collecting Logs in vRealize Log InsightAs part of vRealize Log Insight configuration, you configure syslog and vRealize Log Insight agents.


VMware, Inc. 202

Client applications can send logs to vRealize Log Insight in one of the following ways:

n Directly to vRealize Log Insight over the syslog protocol

n By using vRealize Log Insight to directly query the vSphere Web Server APIs

n By using a vRealize Log Insight Agent

Table 2‑189. Direct Log Communication to vRealize Log Insight Design Decisions


SDDC-OPS-LOG-017

Configure syslog sources tosend log data directly tovRealize Log Insight.

Simplifies the designimplementation for log sources thatare syslog capable.

You must configure syslog sources toforward logs to the vRealize Log InsightVIP.

SDDC-OPS-LOG-018

Configure the vRealize LogInsight agent for the vRealizeAutomation Windows serversand Linux appliances.

n Windows does not nativelysupport syslog.

n vRealize Automation requiresthe use of agents to collect allvRealize Automation logs.

You must manually install and configurethe agents on several nodes.

SDDC-OPS-LOG-019

Configure vCenter ServerAppliances and PlatformServices Controller Appliancesas syslog sources to send logdata directly to vRealize LogInsight.

Simplifies the designimplementation for log sources thatare syslog capable.

n You must manually configure syslogsources to forward logs to thevRealize Log Insight VIP.

n Certain dashboards within vRealizeLog Insight require the use of thevRealize Log Insight Agent for properingestion.

n Not all Operating System-level eventsare forwarded to vRealize Log Insight.

SDDC-OPS-LOG-020

Configure vRealize Log Insightto ingest events, tasks, andalarms from the Managementand Compute vCenter Serverinstances .

Ensures that all tasks, events andalarms generated across allvCenter Server instances in aspecific region of the SDDC arecaptured and analyzed for theadministrator.

You must create a service account onvCenter Server to connect vRealize LogInsight for events, tasks, and alarmspulling.

Configuring vSphere Integration withinvRealize Log Insight does not captureevents that occur on the PlatformServices Controller.

SDDC-OPS-LOG-021

Do not configure vRealize LogInsight to automatically updateall deployed agents.

Manually install updated versionsof the Log Insight Agents for eachof the specified components withinthe SDDC for precisemaintenance.

You must maintain manually the vRealizeLog Insight agents on each of the SDDCcomponents.

Time Synchronization and Cluster Connectivity of vRealize Log InsightTime synchronization is critical for the core functionality of vRealize Log Insight. By default, vRealize LogInsight synchronizes time with a predefined list of public NTP servers.


VMware, Inc. 203

NTP Configuration

Configure consistent NTP sources on all systems that send log data (vCenter Server, ESXi, vRealizeOperation Manager). See Time Synchronization in the VMware Validated Design Planning andPreparation documentation.

Table 2‑190. Time Synchronization Design Decision


SDDC-OPS-LOG-022

Configure consistent NTP sources on allvirtual infrastructure and cloudmanagement applications for correct loganalysis in vRealize Log Insight.

Guarantees accurate logtimestamps.

Requires that all applicationssynchronize time to the sameNTP time source.

Cluster Communication

All vRealize Log Insight cluster nodes must be in the same LAN with no firewall or NAT between thenodes.

vRealize Log Insight receives log data over the syslog TCP, syslog TLS/SSL, or syslog UDP protocols.Use the default syslog UDP protocol because security is already designed at the level of the managementnetwork.

Table 2‑191. Syslog Protocol Design Decision


SDDC-OPS-LOG-023

Communicate with the syslog clients,such as ESXi, vCenter Server, NSXfor vSphere, on the default UDPsyslog port.

Using the default syslog portsimplifies configuration for allsyslog sources.

n If the network connection isinterrupted, the syslog traffic islost.

n UDP syslog traffic is not secure.

vRealize Log Insight Event Forwarding Between RegionsvRealize Log Insight supports event forwarding to other clusters and standalone instances. Whileforwarding events, the vRealize Log Insight instance still ingests, stores and archives events locally.

You forward syslog data in vRealize Log Insight by using the Ingestion API or a native syslogimplementation.

The vRealize Log Insight Ingestion API uses TCP communication. In contrast to syslog, the forwardingmodule supports the following features for the Ingestion API.

n Forwarding to other vRealize Log Insight instances.

n Both structured and unstructured data, that is, multi-line messages.

n Metadata in the form of tags.

n Client-side compression.

n Configurable disk-backed queue to save events until the server acknowledges the ingestion.


VMware, Inc. 204

Table 2‑192. Protocol for Event Forwarding Across Regions Design Decision


SDDC-OPS-LOG-024

Forward log eventto the otherregion by usingthe Ingestion API.

Using the forwarding protocolsupports structured andunstructured data provides client-side compression, and eventthrottling to be passed from onevRealize Log Insight cluster to theother. Forwarding ensures thatduring a disaster recovery situationthe administrator has access to alllogs from the two regions althoughone region is offline.

n You must configure each region to forward log datato the other. The configuration requiresadministrative overhead to prevent recursion oflogging between regions via inclusion and exclusiontagging.

n Log forwarding adds more load on each region. Youmust consider log forwarding in the sizingcalculations for the vRealize Log Insight cluster ineach region.

n You must configure identical size on both sourceand destination clusters.

SDDC-OP-LOG-025

Configure logforwarding to useSSL.

Ensures that the log forwardoperations from one region to theother are secure.

n Event forwarding with SSL does not work with theself-signed certificate that is installed on thedestination servers by default. You must set up acustom CA-signed SSL certificate.

n If additional vRealize Log Insight nodes are addedto a region's cluster, the SSL certificate used by theother region's vRealize Log Insight cluster must beinjected into that nodes Java Keystore before SSLcan be used.

SDDC-OP-LOG-026

Configure diskcache for eventforwarding to2,000 MB (2 GB).

Ensures that log forwardingbetween regions has a buffer forapproximately 2 hours if a cross-region connectivity outage occurs.The disk cache size is calculated ata base rate of 150 MB per day persyslog source with 105 syslogsources.

n If the event forwarder of vRealize Log Insight isrestarted during the cross-region communicationoutage, messages that reside in the non-persistentcache will be cleared.

n If a cross-region communication outage exceeds 2hours, the oldest local events are dropped and notforwarded to the remote destination even after thecross-region connection is restored.

vRealize Log Insight Disaster RecoveryEach region is configured to forward log information to the vRealize Log Insight instance in the otherregion.

Because of the forwarding configuration an administrator of the SDDC can use either of the vRealize LogInsight clusters in the SDDC to query the available logs from one of the regions. As a result, you do nothave to configure failover for the vRealize Log Insight clusters, and each cluster can remain associatedwith the region in which they were deployed.

vSphere Data Protection DesignDesign data protection of the management components in your environment to ensure continuousoperation of the SDDC if the data of a management application is damaged.


VMware, Inc. 205

Data backup protects the data of your organization against data loss, hardware failure, accidentaldeletion, or other disaster for each region. For consistent image-level backups, use backup software thatis based on the vSphere APIs for Data Protection (VADP). This design uses vSphere Data Protection asan example. You can use any VADP compatible software. Adapt and apply the design decisions to thebackup software you use.

Table 2‑193. vSphere Data Protection Design Decision


SDDC-OPS-BKP-001

Use VADP compatible backupsoftware to back up allmanagement components suchas vSphere Data Protection.

vSphere Data Protection provides thefunctionality that is required to back upfull image VMs and applications in thoseVMs, for example, Microsoft SQL Server.

vSphere Data Protectionlacks some features that areavailable in other backupsolutions.

Logical DesignvSphere Data Protection protects the virtual infrastructure at the VMware vCenter Server layer. BecausevSphere Data Protection is connected to the Management vCenter Server, it can access all managementESXi hosts, and can detect the virtual machines that require backups.

Figure 2‑37. vSphere Data Protection Logical Design

VM VM

Authentication

Platform Services Controller

vCenter Server

vSphere Data ProtectionRegion A

vSphere Data ProtectionVM Snapshot/Backup Agent

Backup Datastore

Authentication


vCenter Server

vSphere Data ProtectionRegion B

vSphere Data ProtectionVM Snapshot/Backup Agent

Backup Datastore

Backup DatastoreThe backup datastore stores all the data that is required to recover services according to a RecoveryPoint Objective (RPO). Determine the target location and make sure that it meets performancerequirements.


VMware, Inc. 206

vSphere Data Protection uses deduplication technology to back up virtual environments at data blocklevel, which enables efficient disk utilization. To optimize backups and leverage the VMware vSphereStorage APIs, all ESXi hosts must have access to the production storage.

Table 2‑194. VMware Backup Store Target Design Decisions


SDDC-OPS-BKP-002

Allocate a dedicated NFSdatastore for the vSphereData Protection appliance andthe backup data in eachregion according to PhysicalStorage Design.

n vSphere Data Protection emergency restoreoperations are possible even when the primaryVMware vSAN datastore is not availablebecause the vSphere Data Protection storagevolume is separate from the primary vSANdatastore.

n The amount of storage required for backups isgreater than the amount of storage available inthe vSAN datastore.

You must provide anexternal NFS storagearray.

PerformancevSphere Data Protection generates a significant amount of I/O operations, especially when performingmultiple concurrent backups. The storage platform must be able to handle this I/O. If the storage platformdoes not meet the performance requirements, it might miss backup windows. Backup failures and errormessages might occur. Run the vSphere Data Protection performance analysis feature during virtualappliance deployment or after deployment to assess performance.

Table 2‑195. VMware vSphere Data Protection Performance

Total Backup Size Avg Mbps in 4 hours

0.5 TB 306 Mbps

1 TB 611 Mbps

2 TB 1223 Mbps

Volume SizingvSphere Data Protection can dynamically expand the destination backup store from 2 TB to 8 TB. Usingan extended backup storage requires additional memory on the vSphere Data Protection appliance.

Table 2‑196. VMware vSphere Data Protection Sizing Guide

Available Backup Storage Capacity Size On Disk Minimum Appliance Memory

0.5 TB 0.9 TB 4 GB

1 TB 1.6 TB 4 GB

2 TB 3 TB 6 GB

4 TB 6 TB 8 GB

6 TB 9 TB 10 GB

8 TB 12 TB 12 GB


VMware, Inc. 207

Table 2‑197. VMware Backup Store Size Design Decisions


SDDC-OPS-BKP-003

Deploy the vSphere DataProtection virtual applianceinitially for 4 TB of availablebackup storage capacity and (6TB on-disk size. )

Handles the backup of the Managementstack of a single region. Themanagement stack currently consumesapproximately 2 TB of disk space,uncompressed and withoutdeduplication.

You must provide more NFSstorage to accommodateincreased disk requirements.

Other ConsiderationsvSphere Data Protection can protect virtual machines that reside on VMware vSAN datastores from hostfailures. The virtual machine storage policy is not backed up with the virtual machine, but you can restorethe storage policy after restoring the virtual machine.

Note The default vSAN storage policy includes Number Of Failures To Tolerate = 1, whichmeans that virtual machine data is mirrored.

You use vSphere Data Protection to restore virtual machines that fail or whose data must be reverted to aprevious state.

Backup PoliciesUse vSphere Data Protection backup policies to specify virtual machine backup options, the schedulewindow, and retention policies.

Virtual Machine Backup Options

vSphere Data Protection provides the following options for a virtual machine backup:

HotAdd Provides full image backups of virtual machines, regardless of the guestoperating system.

n The virtual machine base disk is attached directly to vSphere DataProtection to back up data. vSphere Data Protection uses ChangedBlock Tracking to detect and back up blocks that are altered.

n The backup and restore performance is faster because the data flow isthrough the VMkernel layer instead of over a network connection.

n A quiesced snapshot can be used to redirect the I/O of a virtualmachine disk .vmdk file.

n HotAdd does not work in multi-writer disk mode.

Network Block Device(NBD)

Transfers virtual machine data across the network to allow vSphere DataProtection to back up the data.

n The performance of the virtual machine network traffic might be lower.


VMware, Inc. 208

n NBD takes a quiesced snapshot. As a result, it might interrupt the I/Ooperations of the virtual machine to swap the .vmdk file or consolidatethe data after the backup is complete.

n The time to complete the virtual machine backup might be longer thanthe backup window.

n NBD does not work in multi-writer disk mode.

vSphere DataProtection Agent InsideGuest OS

Provides backup of certain applications that are running in the guestoperating system through an installed backup agent.

n Enables application-consistent backup and recovery with MicrosoftSQL Server, Microsoft SharePoint, and Microsoft Exchange support.

n Provides more granularity and flexibility to restore on the file level.

Table 2‑198. Virtual Machine Transport Mode Design Decisions


SDDC-OPS-BKP-004

Use HotAdd to back up virtualmachines.

HotAdd optimizes and speeds upvirtual machine backups, and doesnot impact the vSpheremanagement network.

All ESXi hosts need to have thesame visibility of the virtualmachine datastores.

SDDC-OPS-BKP-005

Use the vSphere DataProtection agent for backups ofSQL databases on MicrosoftSQL Server virtual machines.

You can restore application datainstead of entire virtual machines.

You must install the vSphere DataProtection agent and maintain it.

Schedule Window

Even though vSphere Data Protection uses the Changed Block Tracking technology to optimize thebackup data, to avoid any business impact, do not use a backup window when the production storage isin high demand.

Caution Do not perform any backup or other administrative activities during the vSphere Data Protectionmaintenance window. You can only perform restore operations. By default, the vSphere Data Protectionmaintenance window begins at 8 PM local server time and continues uninterrupted until 8 AM or until thebackup jobs are complete. Configure maintenance windows according to IT organizational policyrequirements.


VMware, Inc. 209

Table 2‑199. Backup Schedule Design Decisions


SDDC-OPS-BKP-006

Schedule dailybackups.

Allows for the recovery of virtual machinesdata that is at most a day old

Data that changed since the lastbackup, 24 hours ago, is lost.

SDDC-OPS-BKP-007

Schedule backupsoutside theproduction peaktimes.

Ensures that backups occur when the systemis under the least amount of load. You shouldverify that backups are completed in theshortest time possible with the smallest risk oferrors.

Backups need to be scheduled tostart between 8:00 PM and 8:00 AMor until the backup jobs are complete,whichever comes first.

Retention Policies

Retention policies are properties of a backup job. If you group virtual machines by business priority, youcan set the retention requirements according to the business priority.

Table 2‑200. Retention Policies Design Decision


SDDC-OPS-BKP-008

Retain backups for atleast 3 days.

Keeping 3 days of backups enablesadministrators to restore the managementapplications to a state within the last 72hours.

Depending on the rate of change invirtual machines, backup retentionpolicy can increase the storage targetsize.

SDDC-OPS-BKP-009

Retain backups forcross-region replicatedbackup jobs for atleast 1 day.

Keeping 1 day of a backup for replicated jobsenables administrators, in the event of adisaster recovery situation in which failoverwas unsuccessful, to restore their region-independent applications to a state within thelast 24 hours.

Data that has changed since the lackbackup, 24 hours ago, is lost. This dataloss also increases the storagerequirements for vSphere DataProtection in a multi-regionconfiguration.

Information Security and Access ControlYou use a service account for authentication and authorization of vSphere Data Protection for backup andrestore operations.


VMware, Inc. 210



SDDC-OPS-BKP-010

Configure a service accountsvc-vdp in vCenter Server forapplication-to-applicationcommunication from vSphereData Protection with vSphere.

Provides the following access control features:n vSphere Data Protection accesses vSphere

with the minimum set of permissions that arerequired to perform backup and restoreoperations.



You must maintain theservice account's lifecycle outside of theSDDC stack to ensure itsavailability

SDDC-OPS-BKP-011

Use global permissions whenyou create the svc-vdp serviceaccount in vCenter Server.




Component Backup JobsYou can configure backup for each SDDC management component separately. For this scenario, norequirement to back up the entire SDDC exists, and this design does not imply such an operation. Someproducts can perform internal configuration backups. Use those products in addition to the whole VMcomponent backups as appropriate.

Table 2‑202. Component Backup Jobs Design Decision


SDDC-OPS-BKP-012

Use the internal configurationbackup features withinVMware NSX.

Restoring small configuration files canbe a faster and less destructive methodto achieve a similar restoration offunctionality.

An FTP server is required forthe NSX configuration backup.

Backup Jobs in Region A

Create a single backup job for the components of a management application according to the nodeconfiguration of the application in Region A.

Table 2‑203. VM Backup Jobs in Region A

Product Image VM Backup Jobs in Region AApplication VM Backup Jobs inRegion A

ESXi Backup is not applicable -

Platform Services Controller Part of the vCenter Server backup job -


VMware, Inc. 211

Table 2‑203. VM Backup Jobs in Region A (Continued)

Product Image VM Backup Jobs in Region AApplication VM Backup Jobs inRegion A

vCenter Server Management Jobn Management Job

n mgmt01vc01.sfo01.rainpole.localn mgmt01psc01.sfo01.rainpole.local

n Compute Jobn comp01vc01.sfo01.rainpole.localn comp01psc01.sfo01.rainpole.local

-

NSX for vSphere n Management Jobn mgmt01nsxm01.sfo01.rainpole.local

n Compute Jobn comp01nsxm01.sfo01.rainpole.local

-

vRealize Automation n vra01mssql01.rainpole.localn vra01bus01.rainpole.localn vra01buc01.sfo01.rainpole.localn vra01svr01a.rainpole.localn vra01svr01b.rainpole.localn vra01iws01a.rainpole.localn vra01iws01b.rainpole.localn vra01ims01a.rainpole.localn vra01ims01b.rainpole.localn vra01dem01.rainpole.localn vra01dem02.rainpole.localn vra01vro01a.rainpole.localn vra01vro01b.rainpole.localn vra01ias01.sfo01.rainpole.localn vra01ias02.sfo01.rainpole.local

vra01mssql01.rainpole.local

vRealize Log Insight n vrli-mstr-01.sfo01.rainpole.localn vrli-wrkr-01.sfo01.rainpole.localn vrli-wrkr-02.sfo01.rainpole.local

-

vRealize Operations Manager n vrops-mstrn-01.rainpole.localn vrops-repln-02.rainpole.localn vrops-datan-03.rainpole.localn vrops-rmtcol-01.sfo01.rainpole.localn vrops-rmtcol-02.sfo01.rainpole.local

-

vRealize Orchestrator Part of the vRealize Automation backup job -

vRealize Business Server

vRealize Business DataCollector

Part of the vRealize Automation backup job -


VMware, Inc. 212

Backup Jobs in Region B

Create a single backup job for the components of a management application according tothe node configuration of the application in Region B. The backup jobs in Region B are not applicable to asingle-region SDDC implementation.

Table 2‑204. VM Backup Jobs in Region B

Product Image VM Backups in Region BApplication VM Backup Jobs inRegion B

ESXi Backup is not applicable None

Platform Services Controller Part of the vCenter Server backup job

vCenter Server n Management Jobn mgmt01vc51.lax01.rainpole.localn mgmt01psc51.lax01.rainpole.local

n Compute Jobn comp01vc51.lax01.rainpole.local n comp01psc51.lax01.rainpole.local

NSX for vSphere n Management Jobn mgmt01nsxm51.lax01.rainpole.local

n Compute Jobn comp01nsxm51.lax01.rainpole.local

vRealize Automation n vra01ias51.lax01.rainpole.localn vra01ias52.lax01.rainpole.localn vra01buc51.lax01.rainpole.local

vRealize Log Insight n vrli-mstr-51.lax01.rainpole.localn vrli-wrkr-51.lax01.rainpole.localn vrli-wrkr-52.lax01.rainpole.local

vRealize Operations Manager n vrops-rmtcol-51.lax01.rainpole.localn vrops-rmtcol-52.lax01.rainpole.local

vRealize Business DataCollector

Part of the vRealize Automation backup job

Site Recovery Manager and vSphere Replication DesignTo support disaster recovery (DR) in the SDDC, you protect vRealize Operations Manager and vRealizeAutomation by using vCenter Site Recovery Manager and VMware vSphere Replication. When failingover to a recovery region, these management applications continue the delivery of operationsmanagement, and cloud platform management functionality.


VMware, Inc. 213

The SDDC disaster recovery design includes two locations: Region A and Region B.

Protected Region A inSan Francisco

Region A contains the management stack virtual machine workloads thatare being protected and is referred to as the protected region in thisdocument.

Recovery Region B inLos Angeles

Region B provides an environment to host virtual machines from theprotected region in the case of a disaster and is referred to as the recoveryregion.

Site Recovery Manager can automate the setup and execution of disaster recovery plans between thesetwo regions.

Note A region in the VMware Validated Design is equivalent to the site construct in Site RecoveryManager.

Disaster Recovery Logical DesignCertain SDDC management applications and services must be available in the event of a disaster. Thesemanagement applications are running on vSphere virtual machines, and can have dependencies onapplications and services that run in both regions.

This validated design for disaster recovery defined the following logical configuration of the SDDCmanagement applications:

n Region A has a management cluster of ESXi hosts with management application virtual machinesthat must be protected.

n Region B has a management cluster of ESXi hosts with sufficient free capacity to host the protectedmanagement applications from Region A.

n Each region has a vCenter Server instance for the management ESXi hosts within the region.

n Each region has a Site Recovery Manager server with an embedded Site Recovery Managerdatabase.

n In each region, Site Recovery Manager is integrated with the Management vCenter Server instance.

n vSphere Replication provides hypervisor-based virtual machine replication between Region A andRegion B.

n vSphere Replication replicates data from Region A to Region B by using a dedicated VMkernelTCP/IP stack.

n Users and administrators access management applications from other branch offices and remotelocations over the corporate Local Area Network (LAN), Wide Area Network (WAN), and VirtualPrivate Network (VPN).


VMware, Inc. 214

Figure 2‑38. Disaster Recovery Logical Design

VM VM VM VM VM VM VM VM

Authentication


vCenter Server

vSphere Web Client

Site Recovery Manager

vSphere Replication

Region A Data Center

vSAN

vRealize Automation


vCenter Server

Site Recovery Manager

vSphere Replication

ESXi ESXi

Region B Data Center

vSAN

vRealize Automation

ESXi ESXi

Deployment Design for Site Recovery Manager A separate Site Recovery Manager instance is required for the protection and recovery of managementcomponents in the event of a disaster situation with your SDDC.

Install and configure Site Recovery Manager after you install and configure vCenter Server and thePlatform Services Controller in the region. Site Recovery Manager is a business continuity and disasterrecovery solution that helps you to plan, test, and run the recovery of the management virtual machineswith the VMware Validated Design, providing protection and orchestrated failover between the Region Aand Region B vCenter Server sites.

You have the following options for deployment and pairing of vCenter Server and Site Recovery Manager:

n vCenter Server options

n You can use Site Recovery Manager and vSphere Replication with vCenter Server Appliance orwith vCenter Server for Windows.

n You can deploy a vCenter Server Appliance in one region and a vCenter Server for Windowsinstance in the other region.


VMware, Inc. 215

n Site Recovery Manager options

n You can use either a physical system or a virtual system.

n You can deploy Site Recovery Manager on a shared system, such as the system of vCenterServer for Windows, or on a dedicated system.

Table 2‑205. Design Decisions for Site Recovery Manager and vSphere ReplicationDeployment


SDDC-OPS-DR-001

Deploy Site Recovery Managerin a virtual machine.

All components of the SDDC solution mustsupport the highest levels of availability.When Site Recovery Manager runs as avirtual machine, you can enable theavailability capabilities of vCenter Serverclusters.

None.

SDDC-OPS-DR-002

Deploy each Site RecoveryManager instance in themanagement cluster.

All management components must be in thesame cluster.

None.

SDDC-OPS-DR-003

Deploy each Site RecoveryManager instance with anembedded PostgreSQLdatabase.

n Reduce the dependence on externalcomponents.

n Reduce potential database licensingcosts.

Requires assigning databaseadministrators who have theskills and tools to administerPostgreSQL databases.

SDDC-OPS-DR-004

Deploy each Site RecoveryManager instance with trustedcertificates.

Similarly to vCenter Server, Site RecoveryManager must use trusted CA-signedcertificates.

Replacing the defaultcertificates with trusted CA-signed certificatescomplicates installation andconfiguration.

Sizing Nodes

You must size the host operating system on which the Site Recovery Manager software runs to supportthe orchestrated failover of the SDDC management components according to the objectives of thisdesign.

Table 2‑206. Compute Resources for a Site Recovery Manager Node


Number of vCPUs 2 (running at 2.0 GHz or higher)

Memory 4 GB

Number of virtual machine NIC ports 1

Number of disks 1

Disk size 40 GB

Operating system Windows Server 2012 R2

Sizing is usually done according to IT organization requirements. However, this design uses calculationsthat are based on the management components in a single region. The design then mirrors thecalculations for the other region. Consider the following management node configuration per region:


VMware, Inc. 216

Table 2‑207. SDDC Nodes with Failover Support

Management Component Node Type Number of Nodes

Cloud Management Platform vRealize Automation Appliance 2

vRealize IaaS Web Server 2

vRealize IaaS Management Server 2

vRealize IaaS DEM 2

vRealize Orchestrator Appliance 2

Microsoft SQL Server 1

vRealize Operations Manager vRealize Operations Manager Master 1

vRealize Operations Manager MasterReplica

1

vRealize Operations Manager Data 1

You must protect a total of 14 virtual machines. You use vSphere Replication as the replication solutionbetween the Site Recovery Manager sites, and you distribute the virtual machines in two protectiongroups.

Table 2‑208. Compute Resources for the Site Recovery Manager Nodes Design Decisions


SDDC-OPS-DR-005

Deploy Site Recovery Manager on aMicrosoft Windows Server host OSaccording to the followingspecifications:n 2 vCPUsn 4 GB memoryn 40 GB diskn 1 GbE

Accommodate the protection ofmanagement components to supply thehighest levels of availability. This sizefurther accommodates the following setup:n 14 protected Management virtual

machines as defined in SDDC Nodeswith Failover Support

n Two Protection Groupsn Two Recovery Plans

You must increase thesize of the nodes if youadd more protectiongroups, virtual machinesto protect or recoveryplans.

SDDC-OPS-DR-006

Use vSphere Replication in SiteRecovery Manager as theprotection method for virtualmachine replication.

Enable replication in a vSAN environmentwhere you cannot configure array-basedreplication.

None.

Networking Design for Disaster RecoveryMoving a service physically from one region to another represents a networking challenge, especially ifapplications have hard-coded IP addresses. Network address space and IP address assignmentconsiderations require that you either use the same IP address or a different IP address at the recoveryregion. In many situations, you assign new IP addresses because VLANs do not typically stretch betweenregions.


VMware, Inc. 217

While protecting the management applications, you can simplify the IP address assignment. This designleverages a load balancer to separate a public network segment and a private network segment. Theprivate network can remain unchanged. You only reassign the external load balancer interface.

n On the public network segment, each management application is accessible under one or morevirtual IP (VIP) addresses.

n On the isolated application virtual network segment, the virtual machines of each managementapplication are isolated.

After a failover, the recovered application is available under a different IPv4 address (VIP). The use of thenew IP address requires changes to the DNS records. You can change the DNS records manually or byusing a script in the Site Recovery Manager recovery plan.


VMware, Inc. 218

Figure 2‑39. Logical Network Design for Cross-Region Deployment with ManagementApplication Network Containers

PSC

OSSRM

OSVC

OSVDP

OSPSC

OSSRM

OSVC

OSVDP

OS

Protected Region

ECMPESGs

ToRSwitches


Mgmt-Management

Compute-Management

Legend:

Shared Compute and

Edge Pod

192.168.11/24

Transit Network


vRLIvROps CollectorvRA Proxy / vRB CollectorUMDS

vRA/vRO/vRBvROps

ESGLoadBalancer


192.168.31/24

Mgmt-RegionА01-VXLAN


Recovery Region

ECMPESGs

ToR Switches

vRLIvROps CollectorvRA Proxy / vRB CollectorUMDS

vRA/vRO/vRBvROps

192.168.32/24


192.168.11/24

ESGLoad

Balancer


Failover Components


Shared Compute and

Edge Pod

Ext-Management

The IPv4 subnets (orange networks) are routed within the vSphere management network of each region.Nodes on these network segments are reachable from within the SDDC. IPv4 subnets, such as thesubnet for the vRealize Automation primary components, overlap across a region. Make sure that onlythe active IPv4 subnet is propagated in the region and beyond. The public facing Ext-Mgmt network ofboth regions (grey networks) is reachable by SDDC users and provides connection to external resources,such as Active Directory or DNS. See Virtualization Network Design.


VMware, Inc. 219

NSX Edge devices provide the load balancing functionality, each device fronting a network that containsthe protected components of all management applications. In each region, you use the sameconfiguration for the management applications and their Site Recovery Manager shadow. Active Directoryand DNS services must be running in both the protected and recovery regions.

Information Security and Access ControlYou use a service account for authentication and authorization of Site Recovery Manager to vCenterServer for orchestrated disaster recovery of the SDDC.



SDDC-OPS-DR-007

Configure a service accountsvc-srm in vCenter Server forapplication-to-applicationcommunication from SiteRecovery Manager withvSphere.

Provides the following access control features:n Site Recovery Manager accesses vSphere with

the minimum set of permissions that are requiredto perform disaster recovery failover orchestrationand site pairing.

n In the event of a compromised account, theaccessibility in the destination application remainsrestricted.


You must maintain theservice account's lifecycle outside of theSDDC stack to ensureits availability.

SDDC-OPS-DR-008

Use global permissions whenyou create the svc-srmservice account in vCenterServer.

n Simplifies and standardizes the deployment of theservice account across all vCenter Serverinstances in the same vSphere domain.

n Provides a consistent authorization layer.n If you deploy more Site Recovery Manager

instances, reduces the efforts in connecting themto the vCenter Server instances.


vSphere ReplicationDeploy vSphere Replication for virtual machine replication in Site Recovery Manager considering therequirements for the operation of the management components that are failed over.

Networking Configuration of the vSphere Replication Appliances

vSphere Replication uses a VMkernel management interface on the ESXi host to send replication trafficto the vSphere Replication appliance in the recovery region. To isolate vSphere Replication traffic so thatit does not impact other vSphere management traffic, configure the vSphere Replication network in thefollowing way.

n Place vSphere Replication traffic on a dedicated VMkernel adapter.

n Ensure that the vSphere Replication VMkernel adapter uses a dedicated replication VLAN in theregion.

n Attach the vSphere Replication server network adapter to the dedicated vSphere Replication VLAN inthe region


VMware, Inc. 220

n Enable the service for vSphere Replication and vSphere Replication NFC traffic on the dedicatedvSphere Replication VMkernel adapter.

vSphere Replication appliances and vSphere Replication servers are the target for the replication trafficthat originates from the vSphere Replication VMkernel ports.

For more information about the vSphere Replication traffic on the management ESXi hosts,see Virtualization Network Design.

Table 2‑210. vSphere Replication Design Decisions


SDDC-OPS-DR-009 Set up a dedicated vSphereReplication distributed portgroup.

Ensures that vSphereReplication traffic does notimpact other vSpheremanagement traffic. ThevSphere Replication serverspotentially receive largeamounts of data from theVMkernel adapters on theESXi hosts.

You must allocate a dedicatedVLAN for vSphere Replication.

SDDC-OPS-DR-010 Set up a dedicated VMkerneladapter on the managementESXi hosts

Ensures that the ESXi serverreplication traffic is redirectedto the dedicated vSphereReplication VLAN.

None.

SDDC-OPS-DR-011 Attach a virtual networkadapter for the vSphereReplication VMs to thevSphere Replication portgroup.

Ensures that the vSphereReplication VMs cancommunicate on the correctreplication VLAN.

vSphere Replication VMs mightrequire additional networkadapters for communication onthe management andreplication VLANs.

SDDC-OPS-DR-012 Enable point in time (PIT)instances within vSphereReplication, keeping 3 copiesover a 24-hour period.

Ensures that the managementapplication that is failing overafter a disaster recovery eventoccurs has multiple recoverypoints to ensure applicationintegrity.

Increasing the number ofretained recovery pointinstances increases the diskusage on the vSAN datastore.

Placeholder Virtual Machines

Site Recovery Manager creates a placeholder virtual machine on the recovery region for every machinefrom the Site Recovery Manager protection group. Placeholder virtual machine files are small becausethey contain virtual machine configuration metadata but no virtual machine disks. Site Recovery Manageradds the placeholder virtual machines as recovery region objects to the Management vCenter Server.

Snapshot Space

To perform failover tests, you must provide additional storage for the snapshots of the replicated VMs.This storage is minimal in the beginning, but grows as test VMs write to their disks. Replication from theprotected region to the recovery region continues during this time. The snapshots created during testingare deleted after the failover test is complete.


VMware, Inc. 221

Sizing Nodes

Select a size for the vSphere Replication nodes to facilitate virtual machine replication of the SDDCmanagement components according to the objectives of this design.

Table 2‑211. Compute Resources for a vShere Replication 4 vCPU Node


Number of vCPUs 4

Memory 4 GB

Disk Capacity 18

Environment Up to 2000 replications between nodes

Sizing is done according to IT organization requirements. However, this design uses calculations for asingle region. The design then mirrors the calculations for the other region. You must protect a total of 14virtual machines. For information about the node configuration of the management component per regionthat is used in the calculations, see Table 2‑207.


SDDC-OPS-DR-013

Deploy vSphere Replicationnode of the 4 vCPU size.

Accommodate the replication of the expected 14 virtualmachines of the following components:n vRealize Automation Componentsn vRealize Operations Manager Components

None.

Information Security and Access Control

You use a service account for authentication and authorization of vSphere Replication to vCenter Serverfor managing virtual machine replication and site pairing.



SDDC-OPS-DR-014

Configure a service accountsvc-vr in vCenter Server forapplication-to-applicationcommunication from vSphereReplication with vSphere.

Provides the following access control features:n vSphere Replication accesses vSphere with

the minimum set of permissions that arerequired to virtual machine perform replicationand site pairing.




SDDC-OPS-DR-015

Use global permissions whenyou create the svc-vr serviceaccount in vCenter Server.





VMware, Inc. 222

Messages and Commands for Site Recovery ManagerYou can configure Site Recovery Manager to present messages for notification and acceptacknowledgement to users. Site Recovery Manager also provides a mechanism to run commands andscripts as necessary when running a recovery plan.

You can insert pre-power-on or post-power-on messages and commands in the recovery plans. Thesemessages and commands are not specific to Site Recovery Manager, but support pausing the executionof the recovery plan to complete other procedures, or running customer-specific commands or scripts toenable automation of recovery tasks.

Site Recovery Manager Messages

Some additional steps might be required before, during, and after running a recovery plan. For example,you might set up the environment so that a message appears when a recovery plan is initiated, and thatthe administrator must acknowledge the message before the recovery plan continues. Messages arespecific to each IT organization.

Consider the following example messages and confirmation steps:

n Verify that IP address changes are made on the DNS server and that the changes are propagated.

n Verify that the Active Directory services are available.

n After the management applications are recovered, perform application tests to verify that theapplications are functioning correctly.

Additionally, confirmation steps can be inserted after every group of services that have a dependency onother services. These conformations can be used to pause the recovery plan so that appropriateverification and testing be performed before subsequent steps are taken. These services are defined asfollows:

n Infrastructure services

n Core services

n Database services

n Middleware services

n Application services

n Web services

Details on each message are specified in the workflow definition of the individual recovery plan.

Site Recovery Manager Commands

You can run custom scripts to perform infrastructure configuration updates or configuration changes onthe environment of a virtual machine. The scripts that a recovery plan runs are located on the SiteRecovery Manager server. The scripts can run against the Site Recovery Manager server or can impact avirtual machine.


VMware, Inc. 223

If a script must run on the virtual machine, Site Recovery Manager does not run it directly, but instructsthe virtual machine to do it. The audit trail that Site Recovery Manager provides does not record theexecution of the script because the operation is on the target virtual machine.

Scripts or commands must be available in the path on the virtual machine according to the followingguidelines:

n Use full paths to all executables. For example c:\windows\system32\cmd.exe instead of cmd.exe.

n Call only.exe or .com files from the scripts. Command-line scripts can call only executables.

n To run a batch file, start the shell command with c:\windows\system32\cmd.exe.

The scripts that are run after powering on a virtual machine are executed under the Local SecurityAuthority of the Site Recovery Manager server. Store post-power-on scripts on the Site RecoveryManager virtual machine. Do not store such scripts on a remote network share.

Recovery Plans for Site Recovery ManagerA recovery plan is the automated plan (runbook) for full or partial failover from Region A to Region B.

Startup Order and Response Time

Virtual machine priority determines virtual machine startup order.

n All priority 1 virtual machines are started before priority 2 virtual machines.




n You can additionally set startup order of virtual machines within each priority group.

You can configure the following timeout parameters:

n Response time, which defines the time to wait after the first virtual machine powers on beforeproceeding to the next virtual machine in the plan.

n Maximum time to wait if the virtual machine fails to power on before proceeding to the next virtualmachine.

You can adjust response time values as necessary during execution of the recovery plan test todetermine the appropriate response time values.


VMware, Inc. 224

Recovery Plan Test Network

When you create a recovery plan, you must configure test network options. The following options areavailable.

Isolated Network(Automatically Created)

An isolated private network is created automatically on each ESXi host inthe cluster for a virtual machine that is being recovered. Site RecoveryManager creates a standard switch and a port group on it. A limitation ofthis automatic configuration is that a virtual machine connected to theisolated port group on one ESXi host cannot communicate with a virtualmachine on another ESXi host. This option limits testing scenarios andprovides an isolated test network only for basic virtual machine testing.

Port Group Selecting an existing port group provides a more granular configuration tomeet your testing requirements. If you want virtual machines across ESXihosts to communicate, use either a standard or distributed switch withuplinks to the production network, and create a port group on the switchthat is tagged with a non-routable VLAN. In this way, the network is isolatedand cannot communicate with other production networks.

Because the isolated application networks are fronted by a load balancer, the recovery plan test networkis equal to the recovery plan production network and provides realistic verification of a recoveredmanagement application.

Table 2‑213. Recovery Plan Test Network Design Decision


SDDC-OPS-DR-016

Use the targetrecovery productionnetwork for testing.

The design of the applicationvirtual networks supports theiruse as recovery plan testnetworks.

During recovery testing, a managementapplication will not be reachable using itsproduction FQDN. Either access the applicationusing its VIP address or assign a temporaryFQDN for testing. Note that this approach willresult in certificate warnings due to name andcertificate mismatches.

vSphere Update Manager DesignvSphere Update Manager pairs with vCenter Server to enable patch and version management of ESXihosts and virtual machines.

vSphere Update Manager can remediate the following objects over the network:

n VMware Tools and VMware virtual machine hardware upgrade operations for virtual machines

n ESXi host patching operations

n ESXi host upgrade operations


VMware, Inc. 225

n Physical Design of vSphere Update Manager

You use the vSphere Update Manager service on each vCenter Server Appliance and deploy avSphere Update Manager Download Service (UMDS) in Region A and Region B to download andstage upgrade and patch data.

n Logical Design of vSphere Update Manager

You configure vSphere Update Manager to apply updates on the management components of theSDDC according to the objectives of this design.

Physical Design of vSphere Update ManagerYou use the vSphere Update Manager service on each vCenter Server Appliance and deploy a vSphereUpdate Manager Download Service (UMDS) in Region A and Region B to download and stage upgradeand patch data.

Networking and Application Design

You can use the vSphere Update Manager as a service of the vCenter Server Appliance. The UpdateManager server and client components are a part of the vCenter Server Appliance.

You can connect only one vCenter Server instance to a vSphere Update Manager instance. Because thisdesign uses multiple vCenter Server instances, you must configure a separate vSphere Update Managerfor each vCenter Server. To save the overhead of downloading updates on multiple vSphere UpdateManager instances and to restrict the access to the external network from vSphere Update Manager andvCenter Server, deploy a vSphere Update Manager Download Service (UMDS) in each region. UMDSdownloads upgrades, patch binaries and patch metadata, and stages the downloads on a Web server.The local Update Manager servers download the patches from UMDS.


VMware, Inc. 226

Figure 2‑40. vSphere Update Manager Logical and Networking Design

APPOS

APPOS

UMDSRegion A

Management Cluster

SharedEdge andComputeCluster

ESXi ESXi ESXi ESXiESXi

Management Cluster

SharedEdge andComputeCluster

ESXi ESXi ESXi ESXiESXi

vSphereUpdate

Manager


vSphereUpdate

Manager


192.168.31.0/24


mgmt01umds01.sfo01

UMDSRegion B

192.168.32.0/24


mgmt01umds51.lax01


Region A Region B

vSphereUpdate

Manager


vSphereUpdate

Manager


Deployment Model

vSphere Update Manager is pre-installed in the vCenter Server Appliance. After you deploy or upgradethe vCenter Server Appliance, the VMware vSphere Update Manager service starts automatically.


VMware, Inc. 227

In addition to the vSphere Update Manager deployment, two models for downloading patches fromVMware exist.

Internet-connectedmodel

The vSphere Update Manager server is connected to the VMware patchrepository to download patches for ESXi hosts and virtual appliances. Noadditional configuration is required, other than scan and remediate thehosts as needed.

Proxied access model vSphere Update Manager has no connection to the Internet and cannotdownload patch metadata. You deploy UMDS to download and store patchmetadata and binaries to a shared repository. vSphere Update Manageruses the shared repository as a patch datastore before remediating theESXi hosts.

Table 2‑214. Update Manager Physical Design Decision


SDDC-OPS-VUM-001

Use the vSphere UpdateManager service on each vCenterServer Appliance to provide atotal of four vSphere UpdateManager instances that youconfigure and use for patchmanagement.

A one-to-one mapping of vCenter Server tovSphere Update Manager is required. EachManagement or Compute vCenter Serverinstance in each region needs its own vSphereUpdate Manager.

All physical designdecisions for vCenterServer determine thesetup for vSphere UpdateManager.

SDDC-OPS-VUM-002

Use the embedded PostgresSQLof the vCenter Server Appliancefor vSphere Update Manager .

n Reduces both overhead, and licensing costfor Microsoft or Oracle.

n Avoids problems with upgrades.

The vCenter ServerAppliance has limiteddatabase managementtools for databaseadministrators.

SDDC-OPS-VUM-003

Use the network settings of thevCenter Server Appliance forvSphere Update Manager .

Simplifies network configuration because of theone-to-one mapping between vCenter Serverand vSphere Update Manager. You configurethe network settings once for both vCenterServer and vSphere Update Manager.

None.

SDDC-OPS-VUM-004

Deploy and configure UMDSvirtual machines for each region.

Limits direct access to the Internet from multiplevSphere Update Manager vCenter Serverinstances, and reduces storage requirementson each instance.

None.

SDDC-OPS-VUM-005

Connect the UMDS virtualmachines to the region-specificapplication virtual network.

n Provides local storage and access tovSphere Update Manager repository data

n Avoids cross-region bandwidth usage forrepository access.

n Provides a consistent deployment model formanagement applications.

You must use NSX tosupport this networkconfiguration.

Logical Design of vSphere Update ManagerYou configure vSphere Update Manager to apply updates on the management components of the SDDCaccording to the objectives of this design.


VMware, Inc. 228

UMDS Virtual Machine Specification

You allocate resources to and configure the virtual machines for UMDS according to the followingspecification:

Table 2‑215. vSphere Update Manager Download Service (UMDS) Virtual MachineSpecifications


vSphere Update Manager Download Service vSphere 6.5

Number of CPUs 2

Memory 2 GB

Disk Space 120 GB

Operating System Ubuntu 14.04 LTS

ESXi Host and Cluster Settings

When you perform updates by using the vSphere Update Manager, the update operation affects certaincluster and host base settings. You customize these settings according to your business requirementsand use cases.

Table 2‑216. Host and Cluster Settings That Are Affected by vSphere Update Manager

Settings Description

Maintenance mode During remediation, updates might require the host to enter maintenance mode. Virtual machinescannot run when a host is in maintenance mode. For availability during a host update, virtualmachines are migrated to other ESXi hosts within a cluster before the host enters maintenancemode. However, putting a host in maintenance mode during update might cause issues with theavailability of the cluster.

vSAN When using vSAN, consider the following factors when you update hosts by using vSphere UpdateManager:n Host remediation might take a significant amount of time to complete because, by design, only

one host from a vSAN cluster can be in maintenance mode at any one time.n vSphere Update Manager remediates hosts that are a part of a vSAN cluster sequentially, even if

you set the option to remediate the hosts in parallel.n If the number of failures to tolerate is set to 0 for the vSAN cluster, the host might experience

delays when entering maintenance mode. The delay occurs because vSAN copies data betweenthe storage devices in the cluster.

To avoid delays, set a vSAN policy where the number failures to tolerate is 1, which is the defaultcase.

You can control the update operation by using a set of host and cluster settings in vSphere UpdateManager.


VMware, Inc. 229

Table 2‑217. Host and Cluster Settings for Updates

Level Setting Description

Host settings VM power state when entering maintenancemode

You can configure vSphere Update Manager to power off,suspend or do not control virtual machines during remediation.This option applies only if vSphere vMotion is not available fora host.

Retry maintenance mode in case of failure If a host fails to enter maintenance mode before remediation,vSphere Update Manager waits for a retry delay period andretries putting the host into maintenance mode as many timesas you indicate.

Allow installation of additional software onPXE-booted hosts

You can install solution software on PXE-booted ESXi hosts.This option is limited to software packages that do not require ahost reboot after installation.

Cluster settings Disable vSphere Distributed PowerManagement (DPM), vSphere High Availability(HA) Admission Control, and Fault Tolerance(FT)

vSphere Update Manager does not remediate clusters withactive DPM, HA and FT.

Enable parallel remediation of hosts vSphere Update Manager can remediate multiple hosts.

Note Parallel remediation is not supported if you use vSAN.

Migrate powered-off or suspended virtualmachines

vSphere Update Manager migrates the suspended andpowered-off virtual machines from hosts that must entermaintenance mode to other hosts in the cluster. The migrationis launched on virtual machines that do not prevent the hostfrom entering maintenance mode.

Virtual Machine and Virtual Appliance Update Settings

vSphere Update Manager supports remediation of virtual machines and appliances. You can control thevirtual machine and appliance updates by using the following settings:

Table 2‑218. vSphere Update Manager Settings for Remediation of Virtual Machines andAppliances

Configuration Description

Take snapshots before virtual machine remediation if the remediation fails, you can use the snapshot to return thevirtual machine to the state before the remediation.

Define the window in which a snapshot persists for a remediatedvirtual machine

Automatically clean up virtual machine snapshots that are takenbefore remediation.

Enable smart rebooting for VMware vSphere vApps remediation Start virtual machines post remediation to maintain startupdependencies no matter if some of the virtual machines are notremediated.

ESXi Image Configuration

You can store full images that you can use to upgrade ESXi hosts. You cannot download such imagesfrom the patch repositories and must upload them by using vSphere Update Manager. Import in therepository also the ESXi builds that are available in the environment.


VMware, Inc. 230

By using Image Builder, add the NSX software packages esx-vdpi, esx-vsip and esx-vxlan into the ESXiupgrade image so that you can use the hosts being upgraded in a software-defined networking setup.

Baselines and Groups

vSphere Update Manager baselines and baseline groups are collections of patches that can be assignedto a cluster or host entity in the environment. Depending on the business requirements, the defaultbaselines might not be allowed until patches are tested or verified on development or pre-productionhosts. Baselines can be confirmed so that the tested patches are applied to hosts and only updated whenappropriate.

Two types of baselines exist:

n Dynamic baselines that can change as items are added to the repository.

n Fixed baselines that remain the same.

vSphere Update Manager contains the following default baselines. Each of these baselines is configuredfor dynamic selection of new items.

Critical host patches Upgrades hosts with a collection of critical patches that are high priority asdefined by VMware.

Non-critical hostpatches

Upgrades hosts with patches that are not classified as critical.

VMware Tools Upgradeto Match Host

Upgrades the VMware Tools version to match the host version.

VM Hardware Upgradeto Match Host

Upgrades the virtual machine hardware version to match the host version.

VA Upgrade to Latest Upgrades a virtual appliance to the latest version available.

vSphere Update Manager Logical Design Decisions

This design applies the following decisions on the logical design of vSphere Update Manager and updatepolicy:

Table 2‑219. vSphere Update Manager Logical Design Decisions


SDDC-OPS-VUM-006

Use the default patch repositories byVMware.

Simplifies the configuration becauseyou do not configure additionalsources.

None.

SDDC-OPS-VUM-007

Set the VM power state to Do NotPower Off.

Ensures highest uptime ofmanagement components and computeworkload virtual machines.

You must manuallyintervene if the migrationfails.

SDDC-OPS-VUM-008

Enable parallel remediation of hostsassuming that enough resources areavailable to update multiple hosts atthe same time.

Provides fast remediation of hostpatches.

More resources unavailableat the same time duringremediation.


VMware, Inc. 231

Table 2‑219. vSphere Update Manager Logical Design Decisions (Continued)


SDDC-OPS-VUM-009

Enable migration of powered-offvirtual machines and templates.

Ensures that templates stored on allmanagement hosts are accessible.

Increases the amount oftime to start remediation fortemplates to be migrated.

SDDC-OPS-VUM-010

Use the default critical and non-criticalpatch baselines for the managementcluster and for the shared edge andcompute cluster.

Simplifies the configuration becauseyou can use the default baselineswithout customization.

All patches are added to thebaselines as soon as theyare released.

SDDC-OPS-VUM-011

Use the default schedule of a once-per-day check and patch download.

Simplifies the configuration becauseyou can use the default schedulewithout customization

None.

SDDC-OPS-VUM-012

Remediate hosts, virtual machines,and virtual appliances once a monthor per business guidelines.

Aligns the remediation schedule withthe business policies.

None.

SDDC-OPS-VUM-013

Use Image Builder to add NSX forvSphere software packages to theESXi upgrade image.

n Ensures that the ESXi hosts areready for software-definednetworking immediately after theupgrade.

n Allows for parallel remediation ofESXi hosts.

n Prevents from additional NSXremediation.

n You must enable theImage Builder service.

n NSX for vSphereupdates might requirenew ESXi imagesupdates.

SDDC-OPS-VUM-014

Configure an HTTP Web server oneach UMDS service that theconnected vSphere Update Managerservers must use to download thepatches from.

Enables the automatic download ofpatches on vSphere Update Managerfrom UMDS. The alternative is to copymedia from one place to anothermanually.

You must be familiar with athird-party Web service suchas Nginx or Apache.


VMware, Inc. 232

Documents

Architecture and Design - VMware Validated Design 4 · Architecture and Design ... VMware Validated Design Architecture and Design is intended for ... is tightly coupled with the