Architecture and Design for VMware Enterprise PKS …...Architecture Overview for Enterprise PKS with NSX-T Workload Domains 2 VMware Enterprise PKS is a container services solution

Architecture and Design for VMwareEnterprise PKS with VMware NSX-TWorkload Domains

18 JUL 2019VMware Validated Design 5.1VMware Enterprise PKS 1.4.1

You can find the most up-to-date technical documentation on the VMware website at:

https://docs.vmware.com/

If you have comments about this documentation, submit your feedback to

[email protected]

VMware, Inc.3401 Hillview Ave.Palo Alto, CA 94304www.vmware.com

Copyright © 2019 VMware, Inc. All rights reserved. Copyright and trademark information.

Architecture and Design for VMware Enterprise PKS with VMware NSX-T Workload Domains

VMware, Inc. 2

https://docs.vmware.com/

mailto:[email protected]

http://pubs.vmware.com/copyright-trademark.html

Contents

About Architecture and Design for VMware Enterprise PKS with VMware NSX-TWorkload Domains 4

1 Applying the Guidance for Enterprise PKS with NSX-T Workload Domains 5

2 Architecture Overview for Enterprise PKS with NSX-T Workload Domains 7Physical Network Architecture for Enterprise PKS with NSX-T Workload Domains 7

Network Transport for Enterprise PKS with NSX-T Workload Domains 8

Virtual Infrastructure Architecture for Enterprise PKS with NSX-T Workload Domains 10

Virtual Infrastructure Overview for Enterprise PKS with NSX-T Workload Domains 11

Network Virtualization Components for Enterprise PKS with NSX-T Workload Domains 12

Network Virtualization Services for Enterprise PKS with NSX-T Workload Domains 14

Container Services Architecture for Enterprise PKS with NSX-T Workload Domains 16

3 Detailed Design for Enterprise PKS with NSX-T Workload Domains 18Physical Infrastructure Design for Enterprise PKS with NSX-T Workload Domains 18

Physical Networking Design for Enterprise PKS with NSX-T Workload Domains 19

Virtual Infrastructure Design for Enterprise PKS with NSX-T Workload Domains 24

vSphere Cluster Design for Enterprise PKS with NSX-T Workload Domains 24

Virtualization Network Design for Enterprise PKS with NSX-T Workload Domains 29

NSX Design for Enterprise PKS with NSX-T Workload Domains 38

Container Services Design for Enterprise PKS with NSX-T Workload Domains 57

Sizing Compute and Storage Resources for Enterprise PKS with NSX-T Workload Domains 57

Networking for Enterprise PKS with NSX-T Workload Domains 58

Availability Design for Enterprise PKS with NSX-T Workload Domains 64

Logging Design for Enterprise PKS with NSX-T Workload Domains 69

Monitoring Design for Enterprise PKS with NSX-T Workload Domains 70

Information Security and Access Control for Enterprise PKS with NSX-T Workload Domains 72

Data Protection Design for Enterprise PKS with NSX-T Workload Domains 74

VMware, Inc. 3

About Architecture and Design forVMware Enterprise PKS with VMwareNSX-T Workload Domains

The Architecture and Design for VMware® Enterprise PKS with VMware NSX-T™ Workload Domainsprovides detailed information about the requirements for software and services to implement VMwareEnterprise PKS in a compute cluster in an SDDC that is compliant with VMware Validated Design™ forSoftware-Defined Data Center.

Intended AudienceThis design is intended for architects and administrators who want to deploy VMware Enterprise PKS in avirtual infrastructure workload domain to instantiate Kubernetes cluster workloads.

PrerequisitesDeploy the management workload domain according to VMware Validated Design for Software-DefinedData Center at least in a single region. See the VMware Validated Design documentation page.

Required VMware SoftwareIn addition to the VMware Validated Design for Software-Defined Data Center 5.1 deployment, you mustdownload VMware NSX-T Data Center 2.4.1 and VMware Enterprise PKS 1.4.1. You then deploy andconfigure NSX-T in the shared edge and compute cluster according to this guide. See VMware ValidatedDesign Release Notes for more information about supported product versions

VMware, Inc. 4

https://docs.vmware.com/en/VMware-Validated-Design/index.html

Applying the Guidance forEnterprise PKS with NSX-TWorkload Domains 1The content in Architecture and Design for VMware Enterprise PKS with VMware NSX-T WorkloadDomains replaces certain parts of Architecture and Design in VMware Validated Design for Software-Defined Data Center, also referred to as the Standard SDDC.

Before You Design the Virtual Infrastructure WorkloadDomain for VMware Enterprise PKSBefore you follow this documentation, you must deploy the components for the SDDC managementcluster according to VMware Validated Design for Software-Defined Data Center at least in a singleregion. See Architecture and Design, Planning and Preparation and Deployment of Region A in theVMware Validated Design documentation.

n VMware ESXi™

n VMware Platform Services Controller™ pair and Management vCenter Server®

n VMware NSX® Data Center for vSphere®

n VMware vRealize® Lifecycle Manager™

n vSphere® Update Manager™

n VMware vRealize® Operations Manager™

n VMware vRealize® Log Insight™

n VMware vRealize® Automation™ with embedded vRealize® Orchestrator™

n VMware vRealize® Business™ for Cloud

Designing a Virtual Infrastructure Workload Domain forVMware Enterprise PKSNext, follow the guidance to design a virtual infrastructure (VI) workload domain with VMware EnterprisePKS deployed in this way:

VMware, Inc. 5

https://docs.vmware.com/en/VMware-Validated-Design/5.1/sddc-architecture-and-design/GUID-44471514-D818-424F-AB32-57108003806C.html

https://docs.vmware.com/en/VMware-Validated-Design/5.1/sddc-planning-and-preparation/GUID-75126B85-2186-46F9-9EA0-00FC55451A1F.html

https://docs.vmware.com/en/VMware-Validated-Design/5.1/sddc-deployment-of-region-a/GUID-657DB777-D919-4C23-BA5E-B98D8A91CA8B.html

http://docs.vmware.com/en/VMware-Validated-Design/index.html

In general, use the guidelines about the VI workload domain and shared edge and compute cluster in thefollowing sections of Architecture and Design in VMware Validated Design for Software-Defined DataCenter:

n Architecture Overview > Physical Infrastructure Architecture

n Architecture Overview > Virtual Infrastructure Architecture

n Detailed Design > Physical Infrastructure Design

n Detailed Design > Virtual Infrastructure Design

First-Level Chapter Places to Use the Guidance for Enterprise PKS

Architecture Overview n Physical Infrastructure Architecture

n Workload Domain Architecture

n Cluster Types

n Physical Network Architecture

• Network Transport• Infrastructure Network Architecture

• Physical Network Interfaces

n Virtual Infrastructure Architecture

Detailed Design n Physical Infrastructure Design

n Physical Design Fundamentals

n Physical Networking Designn Physical Storage Design

n Virtual Infrastructure Design

n vCenter Server Design

• vCenter Server Deployment

• vCenter Server Networking

• vCenter Server Redundancy

• vCenter Server Appliance Sizing

• vSphere Cluster Design• vCenter Server Customization

• Use of TLS Certificates in vCenter Server

n Virtualization Network Designn NSX-T Design


VMware, Inc. 6

https://docs.vmware.com/en/VMware-Validated-Design/5.1/sddc-architecture-and-design/GUID-44471514-D818-424F-AB32-57108003806C.html

Architecture Overview forEnterprise PKS with NSX-TWorkload Domains 2VMware Enterprise PKS is a container services solution to put Kubernetes in operation for multi-cloudenterprises and service providers.

VMware Enterprise PKS simplifies the deployment and management of Kubernetes clusters with Day 1and Day 2 operations support. VMware Enterprise PKS manages container deployment from theapplication layer all the way to the infrastructure layer according to the requirements for production-gradesoftware. VMware Enterprise PKS supports high availability, health-checks and self-repairing ofunderlying virtual machines and rolling upgrades for Kubernetes clusters.

Due to its compatibility with Google Container Engine (GKE), VMware Enterprise PKS provides the lateststable Kubernetes release so that developers can have the latest features and tools available.

VMware Enterprise PKS integrates with VMware NSX-T for advanced container networking includingmicro-segmentation, ingress controller, load balancing, and security policy. By using Harbor Registry asits container registry, VMware Enterprise PKS secures container images through vulnerability scanning,image signing, and auditing.

VMware Enterprise PKS exposes Kubernetes in native form without adding any layers of abstraction orproprietary extensions. Developers can use the native and familiar Kubernetes command line interface(CLI). VMware Enterprise PKS can be easily deployed and made operational through Pivotal CloudFoundry (PCF) Ops Manager, which supports a common operating model to deploy VMware EnterprisePKS across multiple IaaS abstractions including VMware vSphere.

This chapter includes the following topics:

n Physical Network Architecture for Enterprise PKS with NSX-T Workload Domains

n Virtual Infrastructure Architecture for Enterprise PKS with NSX-T Workload Domains

n Container Services Architecture for Enterprise PKS with NSX-T Workload Domains

Physical Network Architecture for Enterprise PKS withNSX-T Workload DomainsVMware Validated Designs can use most physical network architectures.

VMware, Inc. 7

Network Transport for Enterprise PKS with NSX-T WorkloadDomainsYou can implement the physical layer switch fabric of an SDDC by offering Layer 2 or Layer 3 transportservices. For a scalable and vendor-neutral data center network, use a Layer 3 transport.

VMware Validated Design supports both Layer 2 and Layer 3 transports. To decide whether to use Layer2 or Layer 3, consider the following factors:

n NSX-T service routers establish Layer 3 routing adjacency with the first upstream Layer 3 device toprovide equal cost routing for workloads.

n The investment you have today in your current physical network infrastructure.

n The benefits and drawbacks for both layer 2 and layer 3 designs.

Benefits and Drawbacks of Layer 2 TransportA design using Layer 2 transport has these considerations:

n In a design that uses Layer 2 transport, top of rack switches and upstream Layer 3 devices, such ascore switches or routers, form a switched fabric.

n The upstream Layer 3 device terminates each VLAN and provides default gateway functionality.

n Uplinks from the top of rack switch to the upstream Layer 3 devices are 802.1Q trunks carrying allrequired VLANs.

Using a Layer 2 transport has the following benefits and drawbacks:

Table 2-1. Benefits and Drawbacks for Layer 2 Transport

Characteristic Description

Benefits n More design freedom.

n You can span VLANs across racks.

Drawbacks n The size of such a deployment is limited because the fabricelements have to share a limited number of VLANs.

n You might have to rely on a specialized data centerswitching fabric product from a single vendor.

n Traffic between VLANs must traverse to upstream Layer 3device to be routed.


VMware, Inc. 8

Figure 2-1. Example Layer 2 Transport

ToR ToR

ESXiHost

Upstream L3 DeviceL3

25 GbE 25 GbE

L2

Upstream L3 Device

Benefits and Drawbacks of Layer 3 TransportA design using Layer 3 transport requires these considerations:

n Layer 2 connectivity is limited within the data center rack up to the top of rack switches.

n The top of rack switch terminates each VLAN and provides default gateway functionality. The top ofrack switch has a switch virtual interface (SVI) for each VLAN.

n Uplinks from the top of rack switch to the upstream layer are routed point-to-point links. You cannotuse VLAN trunking on the uplinks.

n A dynamic routing protocol, such as BGP, connects the top of rack switches and upstream switches.Each top of rack switch in the rack advertises a small set of prefixes, typically one per VLAN orsubnet. In turn, the top of rack switch calculates equal cost paths to the prefixes it receives from othertop of rack switches.

Table 2-2. Benefits and Drawbacks of Layer 3 Transport

Characteristic Description

Benefits n You can select from many Layer 3 capable switch productsfor the physical switching fabric.

n You can mix switches from different vendors because ofgeneral interoperability between their implementation ofBGP.

n This approach is typically more cost effective because ituses only the basic functionality of the physical switches.

Drawbacks n VLANs are restricted to a single rack. The restriction canaffect vSphere Fault Tolerance, and storage networks.


VMware, Inc. 9

Figure 2-2. Example Layer 3 Transport

ToR

L3

ESXiHost

UpstreamSwitch

UpstreamSwitch

WAN/MPLSInternet

WAN/MPLSInternet

ToR

L2

25 GbE25 GbE

Virtual Infrastructure Architecture for Enterprise PKS withNSX-T Workload DomainsThe virtual infrastructure is the foundation of an operational SDDC. It contains the software-definedinfrastructure, software-defined networking and software-defined storage.

In the virtual infrastructure layer, access to the underlying physical infrastructure is controlled andallocated to the management and compute workloads. The virtual infrastructure layer consists of thehypervisors on the physical hosts and the control of these hypervisors. The management components ofthe SDDC consist of elements in the virtual management layer itself.


VMware, Inc. 10

Figure 2-3. Virtual Infrastructure Layer in the SDDC

ServiceManagement

Portfolio Management

OperationsManagement

CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity

Fault Tolerance and Disaster

Recovery

Backup & Restore

Hypervisor

Pools of Resources

Virtualization Control

VirtualInfrastructure

Layer

Compute

Storage

Network

PhysicalLayer

Security

Replication Compliance

Risk

Governance

Virtual Infrastructure Overview for Enterprise PKS with NSX-TWorkload DomainsThe SDDC virtual infrastructure consists of workload domains. The SDDC virtual infrastructure includes amanagement workload domain that contains the management cluster and a virtual infrastructure workloaddomain that contains the shared edge and compute cluster.

Management ClusterThe management cluster runs the virtual machines that manage the SDDC. These virtual machines hostvCenter Server, vSphere Update Manager, NSX Manager, and other management components. Allmanagement, monitoring, and infrastructure services are provisioned to a vSphere cluster which provideshigh availability for these critical services. Permissions on the management cluster limit access only toadministrators. This limitation protects the virtual machines that are running the management, monitoring,and infrastructure services from unauthorized access. The management cluster leverages software-defined networking capabilities in NSX for vSphere.

The management cluster architecture and design is covered in the VMware Validated Design forSoftware-Defined Data Center. The VMware Enterprise PKS validated design does not include the designof the management cluster.

Shared Edge and Compute ClusterThe shared edge and compute cluster runs the NSX-T edge virtual machines and VMware EnterprisePKS management workloads. The edge virtual machines are responsible for North-South routingbetween compute workloads and the external network. This is often referred to as the on-off ramp of theSDDC. The NSX-T edge virtual machines also enable services such as load balancers. The VMwareEnterprise PKS management workloads enable the deployment and management of Kubernetesworkloads in the compute clusters.


VMware, Inc. 11

The hosts in this cluster provide services such as high availability to the NSX-T edge virtual machinesand VMware Enterprise PKS management workloads.

Compute ClustersThe compute clusters host Kubernetes clusters deployed by VMware Enterprise PKS and thecontainerized applications that run on them. On VMware vSphere, Kubernetes clusters consist of a seriesof master and worker nodes that run as virtual machines. As defined within your cluster plans,Kubernetes clusters can either reside within or across physical vSphere Clusters.

Figure 2-4. SDDC Logical Design

APP

OSAPP

OSAPP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OS

NSX-TEdges

PCF OpsManager

Virtual InfrastructureManagement

ESXiTransport

Node

NSX-VController

(Mgmt)

OtherManagementApplications

NSX-VEdge

(Mgmt)

NSX-VManager(Mgmt)

NSX-TManager

(Compute)

ESXi ESXi ESXi

SDDC Kubernetes Workloads Virtual Infrastructure Shared Edge and Compute

NSX-T Transport Zones

N-VDS (Compute) vDS (Mgmt)

NSX-T Transport Zone (Mgmt)

Shared Edge and Compute Cluster3 Compute Clusters Management Cluster

Managed by:Compute vCenter Server

Managed by:Management vCenter Server

Network: External(Internet/Corporate)

Network: Internal SDDC

vCenterServer(Mgmt)

vCenterServer

(Compute)

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node



Kubernetes Clusters

VMwareEnterprice

PKS

BOSHDirector

VMwareHarbor

Registry

ESXi

Network Virtualization Components for Enterprise PKS with NSX-T Workload DomainsThe NSX-T platform consists of several components that are relevant to the network virtualization design.


VMware, Inc. 12

NSX-T PlatformNSX-T creates a network virtualization layer, which is an abstraction between the physical and virtualnetworks. You create all virtual networks on top of this layer.

Several components are required to create this network virtualization layer:

n NSX-T Managers

n NSX-T Edge Nodes

n NSX-T Distributed Routers (DR)

n NSX-T Service Routers (SR)

n NSX-T Segments (Logical Switches)

These components are distributed in different planes to create communication boundaries and provideisolation of workload data from system control messages.

Data plane Performs stateless forwarding or transformation of packets based on tablespopulated by the control plane, reports topology information to the controlplane, and maintains packet level statistics.

The following traffic runs in the data plane:

n Workload data

n N-VDS virtual switch, distributed routing, and the distributed firewall inNSX-T

The data is carried over designated transport networks in the physicalnetwork.

Control plane Contains messages for network virtualization control. You place the controlplane communication on secure physical networks (VLANs) that areisolated from the transport networks for the data plane.

The control plane computes the runtime state based on configuration fromthe management plane. Control plane propagates topology informationreported by the data plane elements, and pushes stateless configuration toforwarding engines.

Control plane in NSX-T has two parts:

n Central Control Plane (CCP). The CCP is implemented as a cluster ofvirtual machines called CCP nodes. The cluster form factor providesboth redundancy and scalability of resources.

The CCP is logically separated from all data plane traffic, that is, afailure in the control plane does not affect existing data planeoperations.


VMware, Inc. 13

n Local Control Plane (LCP). The LCP runs on transport nodes. It is nearto the data plane it controls and is connected to the CCP. The LCP isresponsible for programming the forwarding entries of the data plane.

Management plane Provides a single API entry point to the system, persists user configuration,handles user queries, and performs operational tasks on all management,control, and data plane nodes in the system.

For NSX-T, all querying, modifying, and persisting user configuration is inthe management plane. Propagation of that configuration down to thecorrect subset of data plane elements is in the control plane. As a result,some data belongs to multiple planes. Each plane uses this data accordingto stage of existence. The management plane also queries recent statusand statistics from the control plane, and under certain conditions directlyfrom the data plane.

The management plane is the only source of truth for the logical systembecause it is the only entry point for user configuration. You make changesusing either a RESTful API or the NSX-T user interface.

For example, responding to a vSphere vMotion operation of a virtualmachine is responsibility of the control plane, but connecting the virtualmachine to the logical network is responsibility of the management plane.

Network Virtualization Services for Enterprise PKS with NSX-TWorkload DomainsNetwork virtualization services include segments, gateways, firewalls, and other components of NSX-T.

Segments (LogicalSwitch)

Reproduces switching functionality, broadcast, unknown unicast, andmulticast (BUM) traffic in a virtual environment that is decoupled from theunderlying hardware.

Segments are similar to VLANs because they provide network connectionsto which you can attach virtual machines. The virtual machines can thencommunicate with each other over tunnels between ESXi hosts. EachSegment has a virtual network identifier (VNI), like a VLAN ID. UnlikeVLANs, VNIs scale beyond the limits of VLAN IDs.

Gateway (LogicalRouter)

Provides North-South connectivity so that workloads can access externalnetworks, and East-West connectivity between logical networks.

A Logical Router is a configured partition of a traditional network hardwarerouter. It replicates the functionality of the hardware, creating multiplerouting domains in a single router. Logical Routers perform a subset of thetasks that are handled by the physical router, and each can contain multiple


VMware, Inc. 14

routing instances and routing tables. Using logical routers can be aneffective way to maximize router use, because a set of logical routers withina single physical router can perform the operations previously performed byseveral pieces of equipment.

n Distributed router (DR)

A DR spans ESXi hosts whose virtual machines are connected to thisgateway, and edge nodes the gateway is bound to. Functionally, theDR is responsible for one-hop distributed routing between segmentsand gateways connected to this gateway.

n One or more (optional) service routers (SR).

An SR is responsible for delivering services that are not currentlyimplemented in a distributed fashion, such as stateful NAT.

A gateway always has a DR. A gateway has SRs when it is a Tier-0gateway, or when it is a Tier-1 gateway and has services configured suchas NAT or DHCP.

NSX-T Edge Node Provides routing services and connectivity to networks that are external tothe NSX-T domain through a Tier-0 gateway over BGP or static routing.

You must deploy an NSX-T Edge for stateful services at either the Tier-0 orTier-1 gateways.

NSX-T Edge Cluster Represents a collection of NSX-T Edge nodes that host multiple servicerouters in highly available configurations. At a minimum, deploy a singleTier-0 SR to provide external connectivity.

An NSX-T Edge cluster does not have a one-to-one relationship with avSphere cluster. A vSphere cluster can run multiple NSX-T Edge clusters.

Transport Node Participates in NSX-T overlay or NSX-T VLAN networking. If a nodecontains an NSX-T Virtual Distributed Switch (N-VDS) such as ESXi hostsand NSX-T Edge nodes, it can be a transport node.

If an ESXi host contains at least one N-VDS, it can be a transport node.

Transport Zone A transport zone can span one or more vSphere clusters. Transport zonesdictate which ESXi hosts and which virtual machines can participate in theuse of a particular network.

A transport zone defines a collection of ESXi hosts that can communicatewith each other across a physical network infrastructure. Thiscommunication happens over one or more interfaces defined as TunnelEndpoints (TEPs).


VMware, Inc. 15

When you create an ESXi host transport node and then add the node to atransport zone, NSX-T installs an N-VDS on the host. For each transportzone that the host belongs to, a separate N-VDS is installed. The N-VDS isused for attaching virtual machines to NSX-T Segments and for creatingNSX-T gateway uplinks and downlinks.

NSX-T Controller As a component of the control plane, the controllers control virtual networksand overlay transport tunnels.

For stability and reliability of data transport, NSX-T deploys the NSX-TController as a role in the Manager cluster which consists of three highlyavailable virtual appliances. They are responsible for the programmaticdeployment of virtual networks across the entire NSX-T architecture.

Logical Firewall Responsible for traffic handling in and out the network according to firewallrules.

A logical firewall offers multiple sets of configurable Layer 3 and Layer 2rules. Layer 2 firewall rules are processed before Layer 3 rules. You canconfigure an exclusion list to exclude segments, logical ports, or groupsfrom firewall enforcement.

The default rule, located at the bottom of the rule table, is a catch-all rule.The logical firewall enforces the default rule on packets that do not matchother rules. After the host preparation operation, the default rule is set tothe allow action. Change this default rule to a block action and enforceaccess control through a positive control model, that is, only traffic definedin a firewall rule can flow on the network.

Logical Load Balancer Provides high-availability service for applications and distributes thenetwork traffic load among multiple servers.

The load balancer accepts TCP, UDP, HTTP, or HTTPS requests on thevirtual IP address and determines which pool server to use.

Logical load balancer is supported only in an SR on the Tier-1 gateway.

Container Services Architecture for Enterprise PKS withNSX-T Workload DomainsThe container services layer enables enterprises and service providers to simplify the deployment andoperations of Kubernetes-based container services within the VMware SDDC.


VMware, Inc. 16

In the container services layer, access to production-grade Kubernetes distribution with advancednetworking, built-in private registry, and full lifecycle management support of the clusters is provided ontop of and integrated with the SDDC. The VMware Enterprise PKS platform consists of severalmanagement components.

Pivotal Cloud FoundryOps Manager

Pivotal Cloud Foundry (PCF) Ops Manager is a graphical interface fordeploying and managing BOSH Director, VMware Enterprise PKS, andVMware Harbor Registry application tiles.

PCF Ops Manager also provides a programmatic interface for performinglifecycle management of PCF Ops Manager and application tiles.

BOSH Director BOSH is an open-source tool for release engineering for the deploymentand lifecycle management of large distributed systems. By using BOSH,developers can version, package, and deploy software in a consistent andreproducible manner.

BOSH can support deployments across different IaaS platforms, such as,VMware vSphere, Microsoft Azure, OpenStack, Google Cloud Platform(GCP), and Amazon Web Services EC2 (AWS EC2).

VMware Enterprise PKSControl Plane

The VMware Enterprise PKS control plane is the self-service API for on-demand deployment and lifecycle management of Kubernetes clusters. TheAPI submits requests to BOSH, which automates the creation, update, anddeletion of Kubernetes clusters.

Harbor Registry Harbor is an open-source, enterprise-class container registry service thatstores and distributes Docker images in a private, on-premises registry. Inaddition to providing Role-Based Access Control (RBAC), LightweightDirectory Access Protocol (LDAP), and Active Directory (AD) support,Harbor provides container image vulnerability scanning, policy-basedimage replication, and content-trust and auditing services.

Kubernetes Kubernetes (k8s) is an open-source container orchestration framework.

Containers package applications and their dependencies in containerimages. A container image is a distributable artifact that provides portabilityacross multiple environments, streamlining the development anddeployment of software. Kubernetes orchestrates these containers tomanage and automate resource use, failure handling, availability,configuration, scalability, and desired state of the application.


VMware, Inc. 17

Detailed Design for EnterprisePKS with NSX-T WorkloadDomains 3The detailed design includes a number of design decisions and the justification and implications of eachdecision, primarily affecting the Virtual Infrastructure Design.

This chapter includes the following topics:

n Physical Infrastructure Design for Enterprise PKS with NSX-T Workload Domains

n Virtual Infrastructure Design for Enterprise PKS with NSX-T Workload Domains

n Container Services Design for Enterprise PKS with NSX-T Workload Domains

Physical Infrastructure Design for Enterprise PKS withNSX-T Workload DomainsThe physical infrastructure design includes design decision details for the physical network.

VMware, Inc. 18

Figure 3-1. Physical Infrastructure Design

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

Physical Networking Design for Enterprise PKS with NSX-TWorkload DomainsDesign of the physical SDDC network includes defining the network topology for connecting the physicalswitches and the ESXi hosts, determining switch port settings for VLANs and link aggregation, anddesigning routing. You can use the VMware Validated Design guidance for design and deployment withmost enterprise-grade physical network architectures.

Switch Types and Network Connectivity for Enterprise PKS with NSX-TWorkload DomainsFollow the best practices for physical switches, switch connectivity, VLANs and subnets, and access portsettings.

Top of Rack Physical Switches

When configuring top of rack (ToR) switches, consider the following best practices:

n Configure redundant physical switches to enhance availability.

n Configure switch ports that connect to ESXi hosts manually as trunk ports.

n Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC to reduce thetime to transition ports over to the forwarding state, for example using the Trunk PortFast featurefound in a Cisco physical switch.

n Provide DHCP or DHCP Helper capabilities on all VLANs used by TEP VMkernel ports. This setupsimplifies the configuration by using DHCP to assign IP address based on the IP subnet in use.

n Configure jumbo frames on all switch ports, inter-switch link (ISL), and switched virtual interfaces(SVIs).


VMware, Inc. 19

Top of Rack Connectivity and Network Settings

Each ESXi host is connected redundantly to the ToR switches SDDC network fabric by two 25 GbE ports.Configure the ToR switches to provide all necessary VLANs using an 802.1Q trunk. These redundantconnections use features in vSphere Distributed Switch and NSX-T to guarantee that no physicalinterface is overrun and available redundant paths are used.

Figure 3-2. Host to ToR Connectivity

ToR ToR

ESXiHost

25 GbE 25 GbE

VLANs and Subnets

Each ESXi host uses VLANs and corresponding subnets.

Follow these guidelines:

n Use only /24 subnets to reduce confusion and mistakes when handling IPv4 subnetting.

n Use the IP address .254 as the (floating) interface with .252 and .253 for Virtual Router RedundancyProtocol (VRPP) or Hot Standby Routing Protocol (HSRP).

n Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region andanother octet by function.

Note The following VLANs and IP ranges are samples. Your actual implementation depends on yourenvironment.

Table 3-1. Sample Values for VLANs and IP Ranges

Function Sample VLAN Sample IP Range

Management 1641 172.16.41.0/24

vSphere vMotion 1642 172.16.42.0/24

vSAN 1643 172.16.43.0/24

Host Overlay 1644 172.16.44.0/24

NFS 1645 172.16.45.0/24

Uplink01 1647 172.16.47.0/24

Uplink02 1648 172.16.48.0/24

Edge Overlay 1649 172.16.49.0/24


VMware, Inc. 20

Access Port Network Settings

Configure additional network settings on the access ports that connect the ToR switches to thecorresponding servers.

Spanning Tree Protocol(STP)

Although this design does not use the Spanning Tree Protocol, switchesusually include STP configured by default. Designate the access ports astrunk PortFast.

Trunking Configure the VLANs as members of a 802.1Q trunk with the managementVLAN acting as the native VLAN.

MTU Set MTU for all VLANs and SVIs (Management, vMotion, VXLAN, andStorage) to jumbo frames for consistency purposes.

DHCP Helper Configure a DHCP helper (sometimes called a DHCP relay) on all TEPVLANs.

Physical Network Design Decisions for Enterprise PKS with NSX-T WorkloadDomainsThe physical network design decisions determine the physical layout and use of VLANs. They alsoinclude decisions on jumbo frames and on other network-related requirements such as DNS and NTP.

Physical Network Design Decisions

Routing protocols NSX-T supports only the BGP routing protocol.

DHCP Helper Set the DHCP helper (relay) to point to a DHCP server by IPv4 address.


VMware, Inc. 21

Table 3-2. Physical Network Design Decisions

Decision ID Design Decision Design Justification Design Implication

PKS-PHY-NET-001 Implement the followingphysical network architecture:

n One 25 GbE (10 GbEminimum) port on eachToR switch for ESXi hostuplinks.

n No EtherChannel (LAG/LACP/vPC) configurationfor ESXi host uplinks

n Layer 3 device thatsupports BGP.

n Guarantees availabilityduring a switch failure.

n Uses BGP as the onlydynamic routing protocolthat is supported by NSX-T.

n Might limit the hardwarechoice.

n Requires dynamic routingprotocol configuration inthe physical network.

PKS-PHY-NET-002 Use a physical network that isconfigured for BGP routingadjacency.

n Supports flexibility innetwork design for routingmulti-site and multi-tenancy workloads.

n Uses BGP as the onlydynamic routing protocolthat is supported by NSX-T.

Requires BGP configuration inthe physical network.

PKS-PHY-NET-003 Use two ToR switches foreach rack.

Supports the use of two 10GbE (25 GbE recommended)links to each server andprovides redundancy andreduces the overall designcomplexity.

Requires two ToR switches perrack which can increase costs.

PKS-PHY-NET-004 Use VLANs to segmentphysical network functions.

n Supports physicalnetwork connectivitywithout requiring manyNICs.

n Isolates the differentnetwork functions of theSDDC so that you canhave differentiatedservices and prioritizedtraffic as needed.

Requires uniform configurationand presentation on all thetrunks made available to theESXi hosts.

Additional Design Decisions

Additional design decisions deal with static IP addresses, DNS records, and the required NTP timesource.


VMware, Inc. 22

Table 3-3. IP Assignment, DNS, and NTP Design Decisions


PKS-PHY-NET-005 Assign static IP addresses toall management componentsin the SDDC infrastructureexcept for NSX-T TEPs.

NSX-T TEPs are assigned byusing a DHCP server. Set thelease duration for the TEPDHCP scope to at least 7days.

Ensures that interfaces suchas management and storagealways have the same IPaddress. In this way, youprovide support forcontinuous management ofESXi hosts using vCenterServer and for provisioning IPstorage by storageadministrators.

NSX-T TEPs do not have anadministrative endpoint. As aresult, they can use DHCP forautomatic IP addressassignment. IP pools are anoption but the NSX-Tadministrator must createthem. If you must change orexpand the subnet, changingthe DHCP scope is simplerthan creating an IP pool andassigning it to the ESXi hosts.

Requires accurate IP addressmanagement.

PKS-PHY-NET-006 Create DNS records for allESXi Hosts managementinterfaces to enable forward(A), reverse (PTR), short, andFQDN resolution.

Ensures consistent resolutionof management nodes usingboth IP address (reverselookup) and name resolution.

None.

PKS-PHY-NET-007 Use an NTP time source forall management nodes.

Maintains accurate andsynchronized time betweenmanagement nodes.

None.

Jumbo Frames Design Decisions

IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-framepayload from 1500 bytes to the jumbo frame setting improves the efficiency of data transfer. You mustconfigure jumbo frames end-to-end. Select an MTU that matches the MTU of the physical switch ports.

According to the purpose of the workload, determine whether to configure jumbo frames on a virtualmachine. If the workload consistently transfers large amounts of network data, configure jumbo frames, ifpossible. In that case, confirm that both the virtual machine operating system and the virtual machineNICs support jumbo frames.

Using jumbo frames also improves the performance of vSphere vMotion.

Note The Geneve overlay requires an MTU value of 1600 bytes or greater.


VMware, Inc. 23

Virtual Infrastructure Design for Enterprise PKS with NSX-T Workload DomainsThe virtual infrastructure design includes the NSX-T components that make up the virtual infrastructurelayer.

Figure 3-3. Virtual Infrastructure Layer in the SDDC

ServiceManagement



CloudManagement

Layer

Service Catalog

Self-Service Portal

Orchestration

BusinessContinuity


Recovery

Backup & Restore

Hypervisor

Pools of Resources



Layer

Compute

Storage

Network

PhysicalLayer

Security


Risk

Governance

vSphere Cluster Design for Enterprise PKS with NSX-T WorkloadDomainsThe cluster design must consider the workload that the cluster handles. Different cluster types in thisdesign have different characteristics.

vSphere Cluster Design Decision BackgroundWhen you design the cluster layout in vSphere, consider the following guidelines:

n Use fewer, larger ESXi hosts, or more, smaller ESXi hosts.

n A scale-up cluster has fewer, larger ESXi hosts.

n A scale-out cluster has more, smaller ESXi hosts.

n Compare the capital costs of purchasing fewer, larger ESXi hosts with the costs of purchasing more,smaller ESXi hosts. Costs vary between vendors and models.

n Evaluate the operational costs of managing a few ESXi hosts with the costs of managing more ESXihosts.

n Consider the purpose of the cluster.

n Consider the total number of ESXi hosts and cluster limits.


VMware, Inc. 24

vSphere High Availability Design for Enterprise PKS with NSX-T WorkloadDomainsVMware vSphere High Availability (vSphere HA) protects your virtual machines in case of ESXi hostfailure by restarting virtual machines on other hosts in the cluster when an ESXi host fails.

vSphere HA Design Basics

During configuration of the cluster, the ESXi hosts elect a master ESXi host. The master ESXi hostcommunicates with the vCenter Server system and monitors the virtual machines and secondary ESXihosts in the cluster.

The master ESXi host detects different types of failure:

n ESXi host failure, for example an unexpected power failure

n ESXi host network isolation or connectivity failure

n Loss of storage connectivity

n Problems with virtual machine OS availability

Table 3-5. vSphere HA Design Decisions


PKS-VI-VC-001 Use vSphere HA to protect allclusters against failures.

vSphere HA supports arobust level of protection forboth ESXi host and virtualmachine availability.

vSphere HA is a hard and fastrequirement for VMwareEnterprise PKS.

You must provide sufficientresources on the remaininghosts so that virtual machinescan be migrated to those hostsin the event of a host outage.

PKS-VI-VC-002 Set vSphere HA HostIsolation Response to PowerOff.

Setting the HA IsolationResponse to Power Off willensure VMs are available asquickly as possible in theevent of a network isolationsituation where the NFSdatastores are no longeravailable.

VMs are powered off in case ofa false positive and an ESXihost is declared isolatedincorrectly.

vSphere HA Admission Control Policy Configuration

The vSphere HA Admission Control Policy allows an administrator to configure how the clusterdetermines available resources. In a smaller vSphere HA cluster, a larger proportion of the clusterresources are reserved to accommodate ESXi host failures, based on the selected policy.


VMware, Inc. 25

The following policies are available:

Host failures the clustertolerates

vSphere HA ensures that a specified number of ESXi hosts can fail andsufficient resources remain in the cluster to fail over all the virtual machinesfrom those ESXi hosts.

Percentage of clusterresources reserved

vSphere HA reserves a specified percentage of aggregate CPU andmemory resources for failover.

Specify Failover Hosts When an ESXi host fails, vSphere HA attempts to restart its virtualmachines on any of the specified failover ESXi hosts. If restart is notpossible, for example, the failover ESXi hosts have insufficient resources orhave failed as well, then vSphere HA attempts to restart the virtualmachines on other ESXi hosts in the cluster.

Shared Edge and Compute Cluster Design for Enterprise PKS with NSX-TWorkload DomainsTenant workloads run on the ESXi hosts in the shared edge and compute cluster. Because of the sharednature of the cluster, NSX-T Edge appliances also run in this cluster. To support these workloads, youmust determine the number of ESXi hosts and vSphere HA settings and several other characteristics ofthe shared edge and compute cluster.

Table 3-6. Shared Edge and Compute Cluster Design Decisions


PKS-VI-VC-003 Create a Shared Computeand Edge vSphere Clusterthat contains VMwareEnterprise PKS managementworkloads and NSX-T EdgeTransport Node VMs.

Limits the footprint of thedesign by saving the use of adedicated vSphere edgecluster. Due to inventoryaccess requirements forBOSH Director, VMwareEnterprise PKS managementworkloads should run in thecompute vCenter Server.

In a shared cluster, the VLANsand subnets between theVMkernel ports for ESXi hostoverlay and the overlay ports ofthe edge appliances must beseparate.

PKS-VI-VC-004 Configure Admission Controlfor one ESXi host failure andpercentage-based failovercapacity.

vSphere HA protects theVMware Enterprise PKSmanagement workloads andNSX-T Edge Transport NodeVMs in the event of an ESXihost failure. vSphere HApowers on virtual machinesfrom the failed ESXi hosts onany remaining ESXi hosts.

Only a single ESXi host failureis tolerated before potentialresource contention.


VMware, Inc. 26

Table 3-6. Shared Edge and Compute Cluster Design Decisions (continued)


PKS-VI-VC-005 Deploy NFS-based storagefor shared edge and computecluster use.

Compute clusters have to usea storage platform whereuniform presentation ofstorage resources acrossdisparate vSphere Clusers ispossible, so NFS is used.Leveraging the same storageplatform for the shared edgeand compute cluster providesfor continuity of managementand operations activitiesacross all four vSphereClusters.

You will have to manage andmonitor the external NFSstorage platform and anynetwork connectivity.

PKS-VI-VC-006 Create a shared edge andcompute vSphere Cluster witha minimum of three ESXihosts.

Allocating three ESXi hostsprovides full redundancywithin the cluster.

With the use of NFS forstorage of VMs andKubernetes volumes, you canstart with as few as threenodes.

PKS-VI-VC-007 Create a vSphere ResourcePool for the two large NSX-TEdge virtual machines with aCPU share level of High, amemory share of normal, anda 64-GB memory reservation.

The NSX-T Edge virtualmachines control all networktraffic in and out of the SDDC.In a contention situation,these virtual machines mustreceive all the resourcesrequired.

During contention NSX-Tcomponents receive moreresources than all otherworkloads. As a result,monitoring and capacitymanagement must be aproactive activity.

The resource pool memoryreservation must be expandedas new Edge VMs aredeployed.


VMware, Inc. 27

Table 3-6. Shared Edge and Compute Cluster Design Decisions (continued)


PKS-VI-VC-008 Create a vSphere ResourcePool for VMware EnterprisePKS management workloadswith a CPU share value ofNormal and a memory sharevalue of Normal.

Creating virtual machinesoutside of a resource poolwould have a negative impacton all other virtual machinesduring contention. In a sharededge and compute cluster,the NSX-T Edge virtualmachines must beguaranteed resources aboveall other workloads as to notimpact network connectivity.Setting the share values tonormal gives the NSX-T Edgevirtual machines a highershare of resources duringcontention ensuring networktraffic is not impacted.

During contention, VMwareEnterprise PKS managementworkloads could be starved forresources and experience poorperformance.

Proactively perform monitoringand capacity management, addcapacity before contentionoccurs.

PKS-VI-VC-009 Create a host profile for theshared edge and computecluster.

Using host profiles simplifiesconfiguration of ESXi hostsand ensures settings areuniform across the cluster.

After you add the ESXi hostsas Transport Nodes, the hostprofile are no longer usableand must be removed from thecluster.

The host profile is only usefulfor initial cluster deploymentand configuration.

Compute Cluster Design for Enterprise PKS with NSX-T Workload DomainsAs the SDDC expands, you can add compute clusters.

To support Kubernetes workloads deployed by VMware Enterprise PKS, a series of discrete computeclusters will be deployed. You add compute clusters to increase the available resources for Kubernetesworkloads or to increase application availability across additional VMware Enterprise PKS AvailabilityZones. The design decisions determine the number of vSphere hosts, vSphere HA settings, and othercharacteristics of each compute cluster.


VMware, Inc. 28


PKS-VI-VC-010 Create three computevSphere clusters to whichKubernetes clusters aredeployed.

Segmenting user workloadsfrom management providesfor security, performance, andavailability of the solution.

Multiple discrete computevSphere Clusters fordeploying Kubernetesclusters adds to theperformance and availabilityof the solution. SegmentingVMware Enterprise PKSAvailability Zones based onvSphere Resource Poolsdoes neither of those.

Workload deployment has totake the physically disparateAvailability Zones into account.

PKS-VI-VC-011 Configure Admission Controlfor one ESXi host failure andpercentage-based failovercapacity.

vSphere HA protects tenantworkloads in the event of anESXi host failure. vSphere HApowers on virtual machinesfrom the failed ESXi hosts onany remaining ESXi hosts.

Only a single ESXi host failureis tolerated before potentialresource contention.

PKS-VI-VC-012 Deploy NFS-based storagefor compute cluster use.

NFS storage presented toeach ESXi host in all computeclusters ensures allKubernetes volumes areavailable in any AvailabilityZone.

You will have to manage andmonitor the external NFSstorage platform and anynetwork connectivity.

PKS-VI-VC-013 Create each computevSphere cluster with aminimum of three ESXi hosts.

Allocating three ESXi hostsprovides full redundancywithin the cluster.

With the use of NFS forstorage of VMs andKubernetes volumes, you canstart with as few as threenodes per compute cluster.

PKS-VI-VC-014 Use the shared edge andcompute cluster-based HostProfile.

Using host profiles simplifiesconfiguration of ESXi hostsand ensures settings areuniform across the cluster.

After you add the ESXi hostsas Transport Nodes, the hostprofile are no longer usableand must be removed from thecluster.

Virtualization Network Design for Enterprise PKS with NSX-TWorkload DomainsDesign the virtualization network according to the business goals of your organization. Prevent alsounauthorized access, and provide timely access to business data.

This network virtualization design uses vSphere and NSX-T to implement virtual networking.


VMware, Inc. 29

Virtual Network Design Guidelines for Enterprise PKS with NSX-T WorkloadDomainsThis VMware Validated Design follows high-level network design guidelines and networking bestpractices.

Design Goals

You can apply the following high-level design goals to your environment:

n Meet diverse needs. The network must meet the diverse needs of many different entities in anorganization. These entities include applications, services, storage, administrators, and users.

n Reduce costs. Server consolidation alone reduces network costs by reducing the number of requirednetwork ports and NICs, but you should determine a more efficient network design. For example,configuring two 25 GbE NICs with VLANs might be more cost effective than configuring a dozen 1-GbE NICs on separate physical networks.

n Boost performance. You can achieve performance improvements and decrease the time required toperform maintenance by providing sufficient bandwidth, which reduces contention and latency.

n Improve availability. You usually improve availability by providing network redundancy.

n Support security. You can support an acceptable level of security through controlled access whererequired and isolation where necessary.

n Improve infrastructure functionality. You can configure the network to support vSphere features suchas vSphere vMotion, vSphere High Availability, and vSphere Fault Tolerance.

Best Practices

Follow the networking best practices throughout your environment.

n Separate network services from one another for greater security and better performance.

n Use Network I/O Control and traffic shaping to guarantee bandwidth to critical virtual machines.During network contention, these critical virtual machines receive a higher percentage of thebandwidth.

n Separate network services on an NSX-T Virtual Distributed Switch (N-VDS) by attaching them tosegments with different VLAN IDs.

n Keep vSphere vMotion traffic on a separate network. When migration with vMotion occurs, thecontents of the memory of the guest operating system is transmitted over the network. You can placevSphere vMotion on a separate network by using a dedicated vSphere vMotion VLAN.

n When using pass-through devices with Linux kernel version 2.6.20 or an earlier guest OS, avoid MSIand MSI-X modes. These modes have significant performance impact.

n For best performance, use VMXNET3 virtual machine NICs.

n Ensure that physical network adapters connected to the same virtual switch are also connected to thesame physical network.


VMware, Inc. 30

Network Segmentation and VLANs

You separate different types of traffic for access security and to reduce contention and latency.

High latency on a network can impact performance. Some components are more sensitive to high latencythan others. For example, reducing latency is important on the IP storage and the vSphere FaultTolerance logging network, because latency on these networks can negatively affect the performance ofmultiple virtual machines.

According to the application or service, high latency on specific virtual machine networks can alsonegatively affect performance. Use information gathered from the current state analysis and frominterviews with key stakeholder and SMEs to determine which workloads and networks are especiallysensitive to high latency.

Virtual Networks

Determine the number of networks or VLANs that are required according to the type of traffic.

n vSphere operational traffic.

n Management

n Geneve (overlay)

n vMotion

n vSAN

n NFS Storage

n vSphere Replication

n Traffic that supports the services and applications of the organization.

Virtual Switches for Enterprise PKS with NSX-T Workload DomainsVirtual switches simplify the configuration process by providing single pane of glass view for performingvirtual network management tasks.

Virtual Switch Design Background for Enterprise PKS with NSX-T Workload Domains

vSphere Distributed Switch and NSX-T Virtual Distributed Switch (N-VDS) provide several advantagesover vSphere Standard Switch.

Centralizedmanagement

n A distributed switch is created and centrally managed on a vCenterServer system. The switch configuration is consistent across ESXihosts.

n An N-VDS is created and centrally managed in NSX-T Manager. Theswitch configuration is consistent across ESXi and edge transportnodes.


VMware, Inc. 31

Centralized management saves time and reduces mistakes and operationalcosts.

Additional features Some of the features of distributed switches can be useful to theapplications and services running in the organization’s infrastructure. Forexample, NetFlow and port mirroring provide monitoring andtroubleshooting capabilities to the virtual infrastructure.

Consider the following caveats for distributed switches:

n Distributed switches are manageable only when the vCenter Server instance is available. As a result,vCenter Server becomes a Tier-1 application.

n N-VDS instances are manageable only when the NSX-T Manager cluster is available. As a result, theNSX-T Manager cluster becomes a Tier-1 application.

Virtual Switch Design Decisions for Enterprise PKS with NSX-T Workload Domains

The virtual switch design decisions determine the use and placement of specific switch types.


PKS-VI-NET-001 Use N-VDS for the NSX-Tbased shared edge andcompute cluster, and foradditional NSX-T basedcompute clusters.

The N-VDS is required foroverlay traffic.

Management is shifted fromthe vSphere Client to the NSXManager.

Shared Edge and Compute Cluster Switches for Enterprise PKS with NSX-T WorkloadDomains

The shared edge and compute cluster uses a single N-VDS with a certain configuration for handled traffictypes, NIC teaming, and MTU size.


VMware, Inc. 32

Figure 3-4. Virtual Switch Design for ESXi Hosts in the Shared Edge and Compute Cluster

nic0 nic1

VLAN NFS

VLAN vMotion

VLAN vSAN

Sample ESXi Shared Edge and Compute Host

sfo01-w-nvds01

VLAN ESXi Management

VLAN Trunking Uplink01

VLAN Trunking Uplink02

VLAN ESXi Overlay (Host TEP)

VLAN Trunking Edge Overlay (Edge TEP)

Table 3-7. Virtual Switches for the Shared Edge and Compute Cluster

N-VDS Switch Name FunctionNumber of PhysicalNIC Ports Teaming Policy MTU

sfo01-w-nvds01 n ESXi Management

n vSphere vMotion

n vSAN

n NFS

n Geneve Overlay(TEP)

n Uplink trunking (2)for the NSX-TEdge instances

2 n Load balancesource for the ESXitraffic

n Failover order forthe edge VM traffic

9000

sfo01-w-uplink01 n Uplink for Edge VMnode 1

1 Failover order 9000

sfo01-w-uplink02 n Uplink for Edge VMnode 2



VMware, Inc. 33

Table 3-7. Virtual Switches for the Shared Edge and Compute Cluster (continued)

N-VDS Switch Name FunctionNumber of PhysicalNIC Ports Teaming Policy MTU

sfo01-w-uplink01 n Uplink to enableECMP


sfo01-w-uplink02 n Uplink to enableECMP


Table 3-8. Virtual Switches in the Shared Edge and Compute Cluster by Physical NIC

N-VDS Switch vmnic Function

sfo01-w-nvds01 0 Uplink

sfo01-w-nvds01 1 Uplink

Figure 3-5. Segment Configuration on an ESXi Host That Runs an NSX-T Edge Node

NSX-T Edge Virtual Machine

fp-eth0 (overlay)

sfo01-w-uplink01(VLAN)

fp-eth1 (uplink 1)

fp-eth2 (uplink 2)

eth0 (management)

ESXi Host

sfo01-w-nvds01

vmnic0

vmnic1

sfo01-w-nvds01(Overlay)

sfo01-w-uplink02(VLAN)

sfo01-w-nvds01-uplink01(VLAN Trunking)

sfo01-w-overlay(VLAN Trunking)

sfo01-w-nvds01-uplink02(VLAN Trunking)

sfo01-w-nvds01-management(VLAN 1641)

Table 3-9. Segments in the Shared Edge and Compute Cluster

N-VDS Switch Segment Name

sfo01-w-nvds01 sfo01-w-nvds01-management

sfo01-w-nvds01 sfo01-w-nvds01-vmotion

sfo01-w-nvds01 sfo01-w-nvds01-vsan

sfo01-w-nvds01 sfo01-w-nvds01-overlay

sfo01-w-nvds01 sfo01-w-nvds01-uplink01

sfo01-w-nvds01 sfo01-w-nvds01-nfs

sfo01-w02-uplink01 sfo01-w02-uplink01

sfo01-w02-uplink02 sfo01-w02-uplink02


VMware, Inc. 34

Table 3-10. VMkernel Adapters for the Shared Edge and Compute Cluster

N-VDS Switch Segment Name Enabled Services

sfo01-w-nvds01 sfo01-w-nvds01-management Management Traffic

sfo01-w-nvds01 sfo01-w-nvds01-vmotion vMotion Traffic

sfo01-w-nvds01 sfo01-w-nvds01-vsan vSAN

sfo01-w-nvds01 sfo01-w-nvds01-nfs --

sfo01-w-nvds01 auto created (Host TEP) --

sfo01-w-nvds01 auto created (Host TEP) --

sfo01-w-nvds01 auto created (Hyperbus) --

Note

When the NSX-T Edge appliance is on an N-VDS, it must use a different VLAN ID and subnet from theESXi hosts overlay (TEP) VLAN ID and subnet.

ESXi host TEP VMkernel ports are automatically created when you configure an ESXi host as a transportnode.

NIC Teaming for Enterprise PKS with NSX-T Workload DomainsYou can use NIC teaming to increase the network bandwidth available in a network path, and to providethe redundancy that supports higher availability.

Benefits and Overview

NIC teaming helps avoid a single point of failure and provides options for load balancing of traffic. Toreduce further the risk of a single point of failure, build NIC teams by using ports from multiple NIC andmotherboard interfaces.

Create a single virtual switch with teamed NICs across separate physical switches.

NIC Teaming Design Background

For a predictable level of performance, use multiple network adapters in one of the followingconfigurations.

n An active-passive configuration that uses explicit failover when connected to two separate switches.

n An active-active configuration in which two or more physical NICs in the server are assigned theactive role.

This validated design uses a non-LAG active-active configuration using the route based on physical NICload algorithm for vSphere Distributed Switch and load balance source algorithm for N-VDS. By using thisconfiguration, network cards remain active instead of remaining idle until a failure occurs.


VMware, Inc. 35

Table 3-11. NIC Teaming and Policy

Design Quality Active-Active Active-Passive Comments

Availability ↑ ↑ Using teaming regardless ofthe option increases theavailability of the environment.

Manageability o o Neither design option impactsmanageability.

Performance ↑ o An active-active configurationcan send traffic across eitherNIC, thereby increasing theavailable bandwidth. Thisconfiguration provides a benefitif the NICs are being sharedamong traffic types andNetwork I/O Control is used.

Recoverability o o Neither design option impactsrecoverability.

Security o o Neither design option impactssecurity.

Legend: ↑ = positive impact on quality; ↓ = negative impact on quality; o = no impact on quality.

Table 3-12. NIC Teaming Design Decisions


PKS-VI-NET-002 In the shared edge andcompute cluster, use the Loadbalance source teamingpolicy on N-VDS.

NSX-T Virtual DistributedSwitch(N-VDS) supports Loadbalance source and Failoverteaming policies. When youuse the Load balance sourcepolicy, both physical NICs canbe active and carry traffic.

None.

Geneve Overlay for Enterprise PKS with NSX-T Workload DomainsGeneve provides the overlay capability in NSX-T to create isolated, multi-tenant broadcast domainsacross data center fabrics, and enables customers to create elastic, logical networks that span physicalnetwork boundaries.

The first step in creating these logical networks is to isolate and pool the networking resources. By usingthe Geneve overlay, NSX-T isolates the network into a pool of capacity and separates the consumption ofthese services from the underlying physical infrastructure. This model is similar to the model vSphereuses to abstract compute capacity from the server hardware to create virtual pools of resources that canbe consumed as a service. You can then organize the pool of network capacity in logical networks thatare directly attached to specific applications.

Geneve is a tunneling mechanism which provides extensibility while still using the offload capabilities ofNICs for performance improvement.


VMware, Inc. 36

Geneve works by creating Layer 2 logical networks that are encapsulated in UDP packets. A Segment IDin every frame identifies the Geneve logical networks without the need for VLAN tags. As a result, manyisolated Layer 2 networks can coexist on a common Layer 3 infrastructure using the same VLAN ID.

In the vSphere architecture, the encapsulation is performed between the virtual NIC of the guest VM andthe logical port on the virtual switch, making the Geneve overlay transparent to both the guest virtualmachines and the underlying Layer 3 network. The Tier-0 Gateway performs gateway services betweenoverlay and non-overlay hosts, for example, a physical server or the Internet router. The NSX-T Edgevirtual machine translates overlay segment IDs to VLAN IDs, so that non-overlay hosts can communicatewith virtual machines on an overlay network.

The edge cluster hosts all NSX-T Edge virtual machine instances that connect to the corporate networkfor secure and centralized network administration.

Table 3-13. Geneve Overlay Design Decisions


PKS-VI-NET-003 Use NSX-T to introduceoverlay networks forworkloads.

Simplifies the networkconfiguration by usingcentralized virtual networkmanagement.

n Requires additionalcompute and storageresources to deploy NSX-Tcomponents.

n Might require more trainingin NSX-T.

PKS-VI-NET-004 To provide virtualized networkcapabilities to workloads, useoverlay networks with NSX-TEdge virtual machines anddistributed routing.

Creates isolated, multi-tenantbroadcast domains acrossdata center fabrics to deployelastic, logical networks thatspan physical networkboundaries.

Requires configuring transportnetworks with an MTU size ofat least 1600 bytes.

vMotion TCP/IP Stack for Enterprise PKS with NSX-T Workload DomainsUse the vMotion TCP/IP stack to isolate traffic for vSphere vMotion and to assign a dedicated defaultgateway for vSphere vMotion traffic.

By using a separate TCP/IP stack, you can manage vSphere vMotion and cold migration traffic accordingto the topology of the network, and as required for your organization.

n Route the traffic for the migration of virtual machines by using a default gateway that is different fromthe gateway assigned to the default stack on the ESXi host.

n Assign a separate set of buffers and sockets.

n Avoid routing table conflicts that might otherwise appear when many features are using a commonTCP/IP stack.

n Isolate traffic to improve security.


VMware, Inc. 37

Table 3-14. vMotion TCP/IP Stack Design Decision


PKS-VI-NET-005 Use the vMotion TCP/IP stackfor vSphere vMotion traffic.

By using the vMotion TCP/IPstack, vSphere vMotion trafficcan be assigned a defaultgateway on its own subnetand can go over Layer 3networks.

The vMotion TCP/IP stack isnot available in the VMkerneladapter creation wizard ofvSphere Distributed Switch.You must create the VMkerneladapter directly on the ESXihost.

NSX Design for Enterprise PKS with NSX-T Workload DomainsThis design implements software-defined networking by using VMware NSX-T. By using NSX-T,virtualization delivers for networking what it has already delivered for compute and storage.

In much the same way that server virtualization programmatically creates, takes snapshots of, deletes,and restores software-based virtual machines (VMs), NSX network virtualization programmaticallycreates, takes snapshots of, deletes, and restores software-based virtual networks. As a result, you followa simplified operational model for the underlying physical network.

NSX-T is a nondisruptive solution. You can deploy it on any IP network, including existing traditionalnetworking models and next-generation fabric architectures, regardless of the vendor.

When administrators provision workloads, network management is a time-consuming task. You spendmost time configuring individual components in the physical infrastructure and verifying that networkchanges do not affect other devices that are using the same physical network infrastructure.

The need to pre-provision and configure networks is a constraint to cloud deployments where speed,agility, and flexibility are critical requirements. Pre-provisioned physical networks enable fast creation ofvirtual networks and faster deployment times of workloads using the virtual network. If the physicalnetwork that you need is already available on the ESXi host to run a workload, pre-provisioning physicalnetworks works well. However, if the network is not available on an ESXi host, you must find an ESXi hostwith the available network and allocate capacity to run workloads in your environment.

Decouple virtual networks from their physical counterparts. In the virtualized environment, you mustrecreate all physical networking attributes that are required by the workloads. Because networkvirtualization supports the creation of virtual networks without modification of the physical networkinfrastructure, you can provision the workload networks faster.

NSX-T Design for Enterprise PKS with NSX-T Workload DomainsNSX-T components are not dedicated to a specific vCenter Server or vSphere construct. You can sharethem across different vSphere environments.

NSX-T, while not dedicated to a vCenter Server, supports only single-region deployments in the currentrelease. This design is focused on compute clusters in a single region.


VMware, Inc. 38

Table 3-15. NSX-T Design Decisions

Design ID Design Decision Design Justification Design Implications

PKS-VI-SDN-001 Deploy three NSX-T Managerappliances to configure andmanage all NSX-T basedcompute clusters in a singleregion.

Software-defined networking(SDN) capabilities offered byNSX, such as load balancingand firewalls, are required tosupport the requiredfunctionality in the computeand edge layers.

As of NSX-T 2.4, the theNSX-T Manager also servesthe role of the NSX-TControllers. You deploy threenodes in the cluster foravailability of services.

You must install and configureNSX-T Manager in a highlyavailable management cluster.

PKS-VI-SDN-002 In the management cluster,add the NSX-T Manager tothe NSX for vSphereDistributed Firewall exclusionlist.

Ensures that themanagement plane is stillavailable if a misconfigurationof the NSX for vSphereDistributed Firewall occurs.

None.

Figure 3-6. NSX-T Architecture

Management Workload Domain

Compute Workload Domain Shared Edge and Compute Cluster

NSX-T Edge 1

NSX-T Edge 2

Active/Standby HA Pair

Management vCenter Server

NSX Manager

Compute vCenter Server

NSX-T Manager 1

NSX-T Manager 2

NSX-T Manager 3

NSX-T Components for Enterprise PKS with NSX-T Workload DomainsThe following sections describe the components in the solution and how they are relevant to the networkvirtualization design.


VMware, Inc. 39

NSX-T Manager

NSX-T Manager provides the graphical user interface (GUI) and the RESTful API for creating,configuring, and monitoring NSX-T components, such as segments and gateways.

NSX-T Manager implements the management and control plane for the NSX-T infrastructure. NSX-TManager provides an aggregated system view and is the centralized network management component ofNSX-T. It provides a method for monitoring and troubleshooting workloads attached to virtual networks. Itprovides configuration and orchestration of the following services:

n Logical networking components, such as logical switching and routing

n Networking and edge services

n Security services and distributed firewall

NSX-T Manager also provides a RESTful API endpoint to automate consumption. Because of thisarchitecture, you can automate all configuration and monitoring operations using any cloud managementplatform, security vendor platform, or automation framework.

The NSX-T Management Plane Agent (MPA) is an NSX-T Manager component that is available on eachESXi host. The MPA is in charge of persisting the desired state of the system and for communicating non-flow-controlling (NFC) messages such as configuration, statistics, status, and real-time data betweentransport nodes and the management plane.

NSX-T Manager also contains the NSX-T Controller component. NSX-T Controllers control the virtualnetworks and overlay transport tunnels. The controllers are responsible for the programmatic deploymentof virtual networks across the entire NSX-T architecture.

The Central Control Plane (CCP) is logically separated from all data plane traffic, that is, a failure in thecontrol plane does not affect existing data plane operations. The controller provides configuration to otherNSX-T Controller components such as the segments, gateways, and edge virtual machine configuration.


VMware, Inc. 40

Table 3-16. NSX-T Manager Design Decisions

Decision ID Design Decision Design Justification Design Implications

PKS-VI-SDN-003 Deploy the NSX-T Managerand the Controller nodes aslarge size virtual appliances.

The large-size appliancesupports more than 64 ESXihosts. The small-sizeappliance is for proof ofconcepts and the mediumsize only supports up to 64ESXi hosts.

The large size requires moreresources in the managementcluster.

PKS-VI-SDN-004 n Grant administratorsaccess to both the NSX-TManager UI and its RESTAPI endpoint.

n Restrict end-user accessto the REST API endpointconfigured for end-userprovisioning, such asvRealize Automation orVMware Enterprise PKS.

n Ensures that tenants ornon-provider staff cannotmodify infrastructurecomponents.

n End-users typicallyinteract only indirectlywith NSX-T from theirprovisioning portal.Administrators interactwith NSX-T using its UIand API.

End users have access only toend-point components.

PKS-VI-SDN-005 n Grant administratorsaccess to both the NSX-TManager UI and itsRESTful API endpoint.

n Restrict end-user accessto the RESTful APIendpoint configured forend-user provisioning,such as vRealizeAutomation or VMwareEnterprise PKS.

Ensures that tenants or non-provider staff cannot modifyinfrastructure components.

End-users typically interactonly indirectly with NSX-Tfrom their provisioning portal.Administrators interact withNSX-T using its UI and API.

End users have access only toend-point components.

NSX-T Virtual Distributed Switch

An NSX-T Virtual Distributed Switch (N-VDS) runs on ESXi hosts and provides physical traffic forwarding.It transparently provides the underlying forwarding service that each segment relies on. To implementnetwork virtualization, a network controller must configure the ESXi host virtual switch with network flowtables that form the logical broadcast domains the tenant administrators define when they create andconfigure segments.

NSX-T implements each logical broadcast domain by tunneling VM-to-VM traffic and VM-to-gatewaytraffic using the Geneve tunnel encapsulation mechanism. The network controller has a global view of thedata center and ensures that the ESXi host virtual switch flow tables are updated as VMs are created,moved, or removed.


VMware, Inc. 41

Table 3-17. NSX-T N-VDS Design Decision


PKS-VI-SDN-006 Deploy an N-VDS to eachESXi host in the computecluster.

ESXi hosts in the computecluster create tunnelendpoints for Geneve overlayencapsulation.

None.

Logical Switching

NSX-T Segments create logically abstracted segments to which you can connect tenant workloads. Asingle Segment is mapped to a unique Geneve segment that is distributed across the ESXi hosts in atransport zone. The Segment supports line-rate switching in the ESXi host without the constraints ofVLAN sprawl or spanning tree issues.

Table 3-18. NSX-T Logical Switching Design Decision


PKS-VI-SDN-007 Deploy all workloads on NSX-T Segments (logicalswitches).

To take advantage of featuressuch as distributed routing,tenant workloads must beconnected to NSX-TSegments.

You must perform all networkmonitoring in the NSX-TManager UI, vRealize LogInsight, vRealize OperationsManger, or vRealize NetworkInsight.

Gateways (Logical Routers)

NSX-T Gateways provide North-South connectivity so that workloads can access external networks, andEast-West connectivity between different logical networks.

A Logical Router is a configured partition of a traditional network hardware router. It replicates thefunctionality of the hardware, creating multiple routing domains in a single router. Logical routers performa subset of the tasks that are handled by the physical router, and each can contain multiple routinginstances and routing tables. Using logical routers can be an effective way to maximize router use,because a set of logical routers within a single physical router can perform the operations previouslyperformed by several pieces of equipment.

n Distributed router (DR)

A DR spans ESXi hosts whose virtual machines are connected to this Gateway, and edge nodes theGateway is bound to. Functionally, the DR is responsible for one-hop distributed routing betweensegments and Gateways connected to this Gateway.

n One or more (optional) service routers (SR).

An SR is responsible for delivering services that are not currently implemented in a distributedfashion, such as stateful NAT.

A Gateway always has a DR. A Gateway has SRs when it is a Tier-0 Gateway, or when it is a Tier-1Gateway and has services configured such as NAT or DHCP.


VMware, Inc. 42

Tunnel Endpoint

Tunnel endpoints enable ESXi hosts to participate in an NSX-T overlay. The NSX-T overlay deploys aLayer 2 network on top of an existing Layer 3 network fabric by encapsulating frames inside packets andtransferring the packets over an underlying transport network. The underlying transport network can beanother Layer 2 networks or it can cross Layer 3 boundaries. The Tunnel Endpoint (TEP) is theconnection point at which the encapsulation and decapsulation take place.

NSX-T Edges

NSX-T Edges provide routing services and connectivity to networks that are external to the NSX-Tdeployment. You use an NSX-T Edge for establishing external connectivity from the NSX-T domain byusing a Tier-0 Gateway using BGP or static routing. Additionally, you deploy an NSX-T Edge to supportnetwork address translation (NAT) services at either the Tier-0 or Tier-1 Gateway.

The NSX-T Edge connects isolated, stub networks to shared uplink networks by providing commongateway services such as NAT, and dynamic routing.

Logical Firewall

NSX-T handles traffic in and out the network according to firewall rules.

A logical firewall offers multiple sets of configurable Layer 3 and Layer 2 rules. Layer 2 firewall rules areprocessed before Layer 3 rules. You can configure an exclusion list to exclude segments, logical ports, orgroups from firewall enforcement.

The default rule, that is at the bottom of the rule table, is a catchall rule. The logical firewall enforces thedefault rule on packets that do not match other rules. After the host preparation operation, the default ruleis set to the allow action. Change this default rule to a block action and apply access control through apositive control model, that is, only traffic defined in a firewall rule can flow on the network.

Logical Load Balancer

The NSX-T logical load balancer offers high-availability service for applications and distributes thenetwork traffic load among multiple servers.

The load balancer accepts TCP, UDP, HTTP, or HTTPS requests on the virtual IP address anddetermines which pool server to use.

Logical load balancer is supported only on the Tier-1 Gateway.

NSX-T Network Requirements and Sizing for Enterprise PKS with NSX-TWorkload DomainsNSX-T requirements impact both physical and virtual networks.

Physical Network Requirements

Physical requirements determine the MTU size for networks that carry overlay traffic, dynamic routingsupport, time synchronization through an NTP server, and forward and reverse DNS resolution.


VMware, Inc. 43

Requirement Comments

Provide an MTU size of 1600 or greater on any network thatcarries Geneve overlay traffic.

Geneve packets cannot be fragmented. The MTU size must belarge enough to support extra encapsulation overhead.

This design uses an MTU size of 9000 for Geneve traffic. SeeTable 3-4. Jumbo Frames Design Decisions.

Enable dynamic routing support on the upstream Layer 3devices.

You use BGP on the upstream Layer 3 devices to establishrouting adjacency with the Tier-0 SRs.

Provide an NTP server. The NSX-T Manager requires NTP settings that synchronize itwith the rest of the environment.

Establish forward and reverse DNS resolution for allmanagement VMs.

Enables administrators to access the NSX-T envrionement viaFQDN as opposed to memorizing IP addresses.

NSX-T Component Specifications

When you size the resources for NSX-T components, consider the compute and storage requirements foreach component, and the number of nodes per component type.

Size of NSX Edge services gateways might be different according to tenant requirements. Consider alloptions in such a case.

Table 3-19. Resource Specification of the NSX-T Components

Virtual Machine vCPU Memory (GB) Storage (GB)Quantity per NSX-TDeployment

NSX-T Manager 12 (Large) 48 (Large) 200 (Large) 3

NSX-T Edge virtualmachine

2 (Small, PoC only) 4 (Small, PoC only) 200 (Small, PoC only) Numbers are differentaccording to the usecase. At least two edgedevices are required toenable ECMP routing.

4 (Medium) 8 (Medium) 200 (Medium)

8 (Large) 32 (Large) 200 (Large)

Table 3-20. Design Decisions on Sizing the NSX-T Edge Virtual Machines


PKS-VI-SDN-008 Use large-size NSX-T Edgevirtual machines.

The large-size applianceprovides the requiredperformance characteristics ifa failure occurs.

Virtual Edges providesimplied lifecyclemanagement.

VMware Enterprise PKSrequires large-size NSX-TEdge virtual machines toenable the deployment ofload balancer objects.

Large size Edges consumemore CPU and Memoryresources.


VMware, Inc. 44

NSX-T Network Virtualization Conceptual Design for Enterprise PKS withNSX-T Workload DomainsThis conceptual design for NSX-T provides the network virtualization design of the logical componentsthat handle the data to and from tenant workloads in the environment.

The network virtualization conceptual design includes a perimeter firewall, a provider logical router, andthe NSX-T Gateway. It also considers the external network, internal workload networks, and themanagement network.

Figure 3-7. NSX-T Conceptual Overview

VM VMVM VM VMVM

Internet MPLS

MgmtNetwork

External Network

Perimeter Firewall

Upstream Layer 3Devices

NSX-Т DistributedRouter (DR)

Internal Tenant Networks (Segments)

vNIC-Level Distributed Firewall

NSX-T Service Router(SR)


VMware, Inc. 45

The conceptual design has the following components.

External Networks Connectivity to and from external networks is through the perimeter firewall.

Perimeter Firewall The firewall exists at the perimeter of the data center to filter Internet traffic.

Upstream Layer 3Devices

The upstream Layer 3 devices are behind the perimeter firewall and handleNorth-South traffic that is entering and leaving the NSX-T environment. Inmost cases, this layer consists of a pair of top of rack switches orredundant upstream Layer 3 devices such as core routers.

NSX-T Service Router(SR)

The SR component of the NSX-T Tier-0 Gateway is responsible forestablishing eBGP peering with the Upstream Layer 3 devices and enablingNorth-South routing.

NSX-T DistributedRouter (DR)

The DR component of the NSX-T Gateway is responsible for East-Westrouting.

Management Network The management network is a VLAN-backed network that supports allmanagement components such as NSX-T Manager and NSX-T Controllers.

Internal TenantNetworks

Internal tenant networks are NSX-T Segments and provide connectivity forthe tenant workloads. Workloads are directly connected to these networks.Internal tenant networks are then connected to a DR.

NSX-T Edge Cluster Design for Enterprise PKS with NSX-T Workload DomainsThe NSX-T design uses a management cluster, a shared edge and compute cluster, and computeclusters. You can add additional compute clusters for scale-out or to comply with different workload typesand SLAs.

NSX-T Edge Cluster

The NSX-T Edge cluster is a logical grouping of NSX-T Edge virtual machines. These NSX-T Edge virtualmachines run in the vSphere edge cluster and provide North-South routing for the workloads in thecompute clusters.


VMware, Inc. 46

Table 3-21. NSX-T Edge Cluster Design Decisions


PKS-VI-SDN-009 Deploy at least two large-sizeNSX-T Edge Transport Nodevirtual machines in thevSphere Shared Computeand Edge cluster.

Creates NSX-T Edge Clusterand meets availability andscale requirements.

VMware Enterprise PKSrequires that large-size NSX-T Edge Transport virtualmachines be deployed tosupport deployment of loadbalancer objects.

You must add the edge virtualmachines as transport nodesbefore you add them to theNSX-T Edge Cluster.

PKS-VI-SDN-010 Apply vSphere DistributedResource Scheduler(vSphere DRS) anti-affinityrules to NSX-T EdgeTransport Node VMs in thevSphere edge cluster.

Prevents NSX-T EdgeTransport Nodes from runningon the same ESXi host. Thisimproves the overallavailability of north-southnetwork traffic.

Additional configuration isrequired to set up anti-affinityrules.

High Availability of NSX-T Components

The NSX-T Manager virtual machines run on the management cluster. vSphere HA protects the NSX-TManagers by restarting a NSX-T Manager virtual machine on a different ESXi host if a primary ESXi hostfailure occurs. NSX-T Managers are deployed in a cluster of three nodes to ensure service availability inthe event of a single node failure.

The data plane remains active during outages in the management and control planes although theprovisioning and modification of virtual networks is impaired until those planes become available again.

The NSX Edge virtual machines are deployed on the vSphere Shared Edge and Compute cluster.vSphere DRS anti-affinity rules prevent NSX-T Edge virtual machines that belong to the same NSX-TEdge cluster from running on the same ESXi host.

Replication Mode of Segments for Enterprise PKS with NSX-T WorkloadDomainsThe control plane decouples NSX-T from the physical network, and handles the broadcast, unknownunicast, and multicast (BUM) traffic in the segments (logical switches).

The following options are available for BUM replication on segments.


VMware, Inc. 47

Table 3-22. BUM Replication Mode of NSX-T Segments

BUM Replication Mode Description

Hierarchical Two-Tier In this mode, the ESXi host transport nodes are groupedaccording to their TEP IP subnet. One ESXi host in each subnetis responsible for replication to a ESXi host in another subnet.The receiving ESXi host replicates the traffic to the ESXi hosts inits local subnet.

The source ESXi host transport node knows about the groupsbased on information it has received from the NSX-T controlcluster. The system can select an arbitrary ESXi host transportnode as the mediator for the source subnet if the remotemediator ESXi host node is available.

Head-End In this mode, the ESXi host transport node at the origin of theframe to be flooded on a segment sends a copy to every otherESXi host transport node that is connected to this segment.

Table 3-23. Design Decisions on Segment Replication Mode


PKS-VI-SDN-011 Use hierarchical two-tierreplication on all segments.

Hierarchical two-tierreplication is more efficient byreducing the number of ESXihosts the source ESXi hostmust replicate traffic to.

None.

Transport Zone Design for Enterprise PKS with NSX-T Workload DomainsTransport zones determine which hosts can participate in the use of a particular network. A transportzone identifies the type of traffic, VLAN or overlay, and the N-VDS name. You can configure one or moretransport zones. A transport zone does not represent a security boundary.


VMware, Inc. 48

Table 3-24. Transport Zones Design Decisions


PKS-VI-SDN-012 Create a single transportzone for all overlay trafficacross all workload domains.

Ensures all Segments areavailable to all ESXi hostsand edge virtual machinesconfigured as TransportNodes.

None.

PKS-VI-SDN-013 Create a VLAN transportzone for ESXi host VMkernelports.

Enables the migration of ESXihost VMkernel ports to the N-VDS.

The N-VDS name must matchthe N-VDS name in the overlaytransport zone.

PKS-VI-SDN-014 Create two transport zonesfor edge virtual machineuplinks.

Enables each Edge virtualmachine to peer with a top ofrack switch.

To support VMwareEnterprise PKS, the EdgeVMs cannot leverage ECMPas stateful services arerunning in the T0 Gateway.

You must specify a VLANrange, that is use VLANtrunking, on the segment usedas the uplinks.

You must configure the uplinkfor each Edge VM separately,e.g., Edge-1 would peer toToR-1 and Edge-2 would peerwith ToR-2.

Network I/O Control Design for Enterprise PKS with NSX-T WorkloadDomainsWhen a Network I/O Control profile is attached to an N-VDS, during contention the switch allocatesavailable bandwidth according to the configured shares, limit, and reservation for each vSphere traffictype.

How Network I/O Control Works

Network I/O Control enforces the share value specified for the different traffic types only when there isnetwork contention. When contention occurs, Network I/O Control applies the share values set to eachtraffic type. As a result, less important traffic, as defined by the share percentage, is throttled, grantingaccess to more network resources to more important traffic types.

Network I/O Control also supports the reservation of bandwidth for system traffic according to the overallpercentage of available bandwidth.


VMware, Inc. 49

Network I/O Control Heuristics

The following heuristics can help with design decisions.

Shares vs. Limits When you use bandwidth allocation, consider using shares instead of limits.Limits impose hard limits on the amount of bandwidth used by a traffic floweven when network bandwidth is available.

Limits on NetworkResource Pools

Consider imposing limits on a resource pool. For example, set a limit onvSphere vMotion traffic to avoid oversubscription at the physical networklevel when multiple vSphere vMotion data transfers are initiated on differentESXi hosts at the same time. By limiting the available bandwidth forvSphere vMotion at the ESXi host level, you can prevent performancedegradation for other traffic.

Network I/O Control Design Decisions

Based on the heuristics, this design has the following decisions.

Table 3-25. Network I/O Control Design Decisions


PKS-VI-SDN-015 Create and attach a NetworkI/O Control Policy on all N-DVS switches.

Increases resiliency andperformance of the network.

If configured incorrectly,Network I/O Control mightimpact network performancefor critical traffic types.

PKS-VI-SDN-016 Set the share value forvSphere vMotion traffic toLow (25).

During times of networkcontention, vSphere vMotiontraffic is not as important asvirtual machine or storagetraffic.

During times of networkcontention, vMotion takeslonger than usual to complete.

PKS-VI-SDN-017 Set the share value forvSphere Replication traffic toLow (25).

During times of networkcontention, vSphereReplication traffic is not asimportant as virtual machineor storage traffic.

During times of networkcontention, vSphereReplication takes longer andmight violate the defined SLA.

PKS-VI-SDN-018 Set the share value for vSANtraffic to High (100).

During times of networkcontention, vSAN trafficneeds a guaranteedbandwidth to support virtualmachine performance.

None.


VMware, Inc. 50

Table 3-25. Network I/O Control Design Decisions (continued)


PKS-VI-SDN-019 Set the share value formanagement traffic to Normal(50).

By keeping the default settingof Normal, managementtraffic is prioritized higher thanvSphere vMotion andvSphere Replication but lowerthan vSAN traffic.Management traffic isimportant because it ensuresthat the hosts can still bemanaged during times ofnetwork contention.

None.

PKS-VI-SDN-020 Set the share value for NFStraffic to High (100).

During times of networkcontention, NFS traffic needsa guaranteed bandwidth tosupport virtual machineperformance.

None.

PKS-VI-SDN-021 Set the share value forbackup traffic to Low (25).

During times of networkcontention, the primaryfunctions of the SDDC mustcontinue to have access tonetwork resources withpriority over backup traffic.

During times of networkcontention, backups are slowerthan usual.

PKS-VI-SDN-022 Set the share value for virtualmachines to High (100).

Virtual machines are the mostimportant asset in the SDDC.Leaving the default setting ofHigh ensures that they alwayshave access to the networkresources they need.

None.

PKS-VI-SDN-023 Set the share value forvSphere Fault Tolerance toLow (25).

This design does not usevSphere Fault Tolerance.Fault tolerance traffic can beset the lowest priority.

None.

PKS-VI-SDN-024 Set the share value for iSCSItraffic to Low (25).

This design does not useiSCSI. iSCSI traffic can be setthe lowest priority.

None.

Transport Node and Uplink Policy Design for Enterprise PKS with NSX-TWorkload DomainsA transport node can participate in an NSX-T overlay or NSX-T VLAN network.


VMware, Inc. 51

Several types of transport nodes are available in NSX-T.

ESXi Host TransportNodes

ESXi host transport nodes are ESXi hosts prepared and configured forNSX-T. N-VDS provides network services to the virtual machines runningon these ESXi hosts.

Edge Nodes NSX-T Edge nodes are service appliances that run network services thatcannot be distributed to the hypervisors. They are grouped in one orseveral NSX-T Edge clusters. Each cluster represents a pool of capacity.

Uplink profiles define policies for the links from ESXi hosts to NSX-T Segments or from NSX-T Edgevirtual machines to top of rack switches. By using uplink profiles, you can apply consistent configurationof capabilities for network adapters across multiple ESXi hosts or edge virtual machines. Uplink profilesare containers for the properties or capabilities for the network adapters.

Uplink profiles can use either load balance source or failover order teaming. If using load balance source,multiple uplinks can be active. If using failover order, only a single uplink can be active.

Table 3-26. Design Decisions on Transport Nodes and Uplink Policy


PKS-VI-SDN-025 Create an uplink profile withthe load balance sourceteaming policy with two activeuplinks for ESXi hosts.

For increased resiliency andperformance, supports theconcurrent use of bothphysical NICs on the ESXihosts that are configured astransport nodes.

You can use this policy onlywith ESXi hosts. Edge virtualmachines must use the failoverorder teaming policy.

PKS-VI-SDN-026 Create an uplink profile withthe failover order teamingpolicy with one active uplinkand no standby uplinks foredge virtual machine overlaytraffic.

Provides a profile accordingto the requirements for Edgevirtual machines. Edge virtualmachines support uplinkprofiles only with a failoverorder teaming policy.

VLAN ID is required in theuplink profile. Hence, youmust create an uplink profilefor each VLAN used by theedge virtual machines.

n You create and managemore uplink profiles.

n The VLAN ID used must bedifferent than the VLAN IDfor ESXi host overlaytraffic.

PKS-VI-SDN-027 Create two uplink profiles withthe failover order teamingpolicy with one active uplinkand no standby uplinks foredge virtual machine uplinktraffic.

Provides redundancy at theEdge VM level by ensuringeach Edge VM is peered witha separate ToR switch.

You create and manage moreuplink profiles.


VMware, Inc. 52

Table 3-26. Design Decisions on Transport Nodes and Uplink Policy (continued)


PKS-VI-SDN-028 Create a Transport NodePolicy with the VLAN andOverlay Transport Zones, N-VDS settings, Physical NICsand Network mappings toallow NSX-T to migrate theVMKernel adapter to the N-VDS.

Allows the profile to beassigned directly to thevSphere cluster and ensuresconsistent configurationacross all ESXi HostTransport Nodes.

You must create all requiredTransport Zones andSegments before creating theTransport Node Policy.

PKS-VI-SDN-029 Add as transport nodes allESXi hosts in the SharedEdge and Compute cluster byapplying the Transport NodePolicy to the vSphere clusterobject.

Enables the participation ofESXi hosts and the virtualmachines on them in NSX-Toverlay and VLAN networks.

ESXi hosts VMKernel adaptersare migrated to the N-VDS.Ensure the VLAN ID's for theVMKernel Segments arecorrect to ensure hostcommunication is not lost.

PKS-VI-SDN-030 Add as transport nodes alledge virtual machines.

Enables the participation ofedge virtual machines in theoverlay network and thedelivery of services, such asrouting, by these machines.

None.

PKS-VI-SDN-031 Create an NSX-T Edgecluster with the defaultBidirectional ForwardingDetection (BFD) settingscontaining the edge transportnodes.

Satisfies the availabilityrequirements by default.

Edge clusters are required tocreate services such as NAT,routing to physical networks,and load balancing.

None.

Routing Design by Using NSX-T for Enterprise PKS with NSX-T WorkloadDomainsThe routing design considers different levels of routing in the environment, such as number and type ofNSX-T routers, dynamic routing protocol, and so on. At each level, you apply a set of principles fordesigning a scalable routing solution.

Routing can be defined in the following directions: North-South and East-West.

n North-South traffic is traffic leaving or entering the NSX-T domain, for example, a virtual machine onan overlay network communicating with an end-user device on the corporate network.

n East-West traffic is traffic that remains in the NSX-T domain, for example, two virtual machines on thesame or different segments communicating with each other.


VMware, Inc. 53

Table 3-27. Design Decisions on Routing Using NSX-T


PKS-VI-SDN-032 Create two VLANs to enableBGP peering between theTier-0 Gateway and the Layer3 device (ToR or upstreamdevice).

The ToR switches orupstream Layer 3 deviceshave an SVI on one of thetwo VLANS and each edgevirtual machine has aninterface for its paired ToRswitch.

Enables each Edge virtualmachine to peer with a top ofrack switch.

To support VMwareEnterprise PKS, the EdgeVMs cannot leverage ECMPas stateful services arerunning in the T0 Gateway.

Extra VLANs are required.

Each Edge VM has only asingle uplink. This means that ifthe ToR switch, Edge VM, orphysical network connectivitybetween them was interrupted,Edge VM failover would benecessary.

PKS-VI-SDN-033 Deploy an Active-StandbyTier-0 Gateway.

Supports North-South routingon all Edge VMs in the NSX-TEdge cluster.

You must deploy your Tier-0Gateway in an Active/Standbyconfiguration for use withVMware Enterprise PKS.

Active/Active Tier-0 Gatewayscannot provide services suchas NAT.

PKS-VI-SDN-034 Use BGP as the dynamicrouting protocol.

Enables the dynamic routingby using NSX-T. NSX-Tsupports only BGP .

In environments where BGPcannot be used, you mustconfigure and manage staticroutes.

PKS-VI-SDN-035 Configure BGP Keep AliveTimer to 4 and Hold DownTimer to 12 between the ToRswitches and the Tier-0Gateway.

Provides a balance betweenfailure detection between theToR switches and the Tier-0Gateway and overburdeningthe ToRs with keep alivetraffic.

By using longer timers todetect if a router is notresponding, the data aboutsuch a router remains in therouting table longer. As aresult, the active routercontinues to send traffic to arouter that is down.

PKS-VI-SDN-036 Do not enable GracefulRestart between BGPneighbors.

Avoids loss of traffic. GracefulRestart maintains theforwarding table which in turnwill forward packets to a downneighbor even after the BGPtimers have expired causingloss of traffic.

None.


VMware, Inc. 54

Table 3-27. Design Decisions on Routing Using NSX-T (continued)


PKS-VI-SDN-037 Deploy a Tier-1 Gateway tothe NSX-T Edge cluster andconnect it to the Tier-0Gateway.

Creates a two-tier routingarchitecture that supportsload balancers and NAT.

Because the Tier-1 is alwaysActive/Standby, creation ofservices such as loadbalancers or NAT is possible.

A Tier-1 Gateway can only beconnected to a single Tier-0Gateway.

In scenarios where multipleTier-0 Gateways are required,you must create multiple Tier-1Gateways.

PKS-VI-SDN-038 Deploy Tier-1 Gateways withthe Non-Premptive setting.

Ensures that when the failedEdge Transport Node comesback online it doesn't moveservices back to itselfresulting in a small serviceoutage.

None.

Virtual Network Design Example Using NSX-T for Enterprise PKS with NSX-TWorkload DomainsDesign a setup of virtual networks where you determine the connection of virtual machines to Segmentsand the routing between the Tier-1 Gateway and Tier-0 Gateway, and then between the Tier-0 Gatewayand the physical network.


VMware, Inc. 55

Figure 3-8. Virtual Network Example

Tier-0 Router Active/Standby

Tier-1 to Tier-0 Connection

Legend:VLANs

Logical Switches

Logical Switch 1 - 192.168.100.0/24

Linux Linux

Physical L3 Devices

Edge VM 1 Edge VM 2

Shared Edge and Compute

Logical Switch 2 - 192.168.200.0/24

Tier-1 Router

Windows Linux Linux Windows

Monitoring NSX-T for Enterprise PKS with NSX-T Workload DomainsMonitor the operation of NSX-T for identifying failures in the network setup by using vRealize Log Insightand vRealize Operations Manager.

n vRealize Log Insight saves log queries and alerts, and you can use dashboards for efficientmonitoring.

n vRealize Operations Manger provides alerts, capacity management and custom views anddashboards.


VMware, Inc. 56

Table 3-28. Design Decisions on Monitoring NSX-T


PKS-VI-SDN-039 Install the content pack forNSX-T in vRealize LogInsight.

Add a dashboard and metricsfor granular monitoring of theNSX-T infrastructure.

Requires manually installingthe content pack.

PKS-VI-SDN-040 Configure each NSX-Tcomponent to send loginformation over syslog to thevRealize Log Insight clusterVIP.

Ensures that all NSX-Tcomponents log files areavailable for monitoring andtroubleshooting in vRealizeLog Insight.

Requires manually configuringsyslog on each NSX-Tcomponent.

Use of SSL Certificates in NSX-T for Enterprise PKS with NSX-T WorkloadDomainsBy default, NSX-T Manager uses a self-signed Secure Sockets Layer (SSL) certificate. This certificate isnot trusted by end-user devices or Web browsers.

As a best practice, replace self-signed certificates with certificates that are signed by a third-party orenterprise Certificate Authority (CA).

Table 3-29. Design Decisions on the SSL Certificate of NSX-T Manager

Design ID Design Decision Design Justification Design Implication

PKS-VI-SDN-041 Replace the certificate of theNSX-T Manager instanceswith a certificate that is signedby a third-party Public KeyInfrastructure.

Ensures that thecommunication betweenNSX-T administrators and theNSX-T Manager instance isencrypted by using a trustedcertificate.

Replacing and managingcertificates is an operationaloverhead.

PKS-VI-SDN-042 Replace the NSX-T Managercluster certificate with acertificate that is signed by athird-party Public KeyInfrastructure.

Ensures that thecommunication between thevirtual IP address of the NSX-T Manager cluster andadministrators is encrypted byusing a trusted certificate.

Replacing and managingcertificates is an operationaloverhead.

Container Services Design for Enterprise PKS with NSX-TWorkload DomainsThe design for VMware Enterprise PKS includes design decisions for VMware NSX-T, PCF Ops Manager,BOSH Director, VMware Enterprise PKS control plane, and Harbor Registry. You deploy each componentin standalone mode and vSphere HA provides high-availability for the virtual machines.

Sizing Compute and Storage Resources for Enterprise PKS withNSX-T Workload DomainsYou size the specific compute and storage requirements for the multiple VMware Enterprise PKSmanagement components.


VMware, Inc. 57

Compute and storage requirements for each VMware Enterprise PKS management component are keyconsiderations when you consider how to size for compute clusters.

In addition to these management components, the VMware Enterprise PKS control plane provisionsadditional service virtual machines as-needed for compilation purposes. For example, when you deployan initial Kubernetes cluster, the VMware Enterprise PKS control plane deploys four additional servicevirtual machines for software compilation as a one-time task. When the compilation process finishes, theVMware Enterprise PKS control plane deletes the VMs.

To manage and configure VMware Enterprise PKS after day 0 and day 1 activities, such as deploymentand lifecycle management of Kubernetes clusters, you need VMware Enterprise PKS and Kubernetes CLIcommand-line utilities. These utilities can be installed locally on a workstation which has access to theVMware Enterprise PKS management infrastructure or on a dedicated management virtual machinewhich can be shared with multiple platform operators.

Table 3-30. Resource Requirements for VMware Enterprise PKS Components

Virtual Machine vCPU Memory (GB) Storage (GB)Number perDeployment

PCF Ops Manager 1 8 160 1

BOSH Director 2 8 103 1

VMware Enterprise PKScontrol plane

2 8 29 1

BOSH Compilation VM 4 4 32 4

CLI/API Client 1 2 8 1

VMware Harbor Registry 2 8 167 1

Total 12 38 499 9

Networking for Enterprise PKS with NSX-T Workload DomainsYou place the networks for VMware Enterprise PKS on NSX-T logical switches.

VMware Enterprise PKSManagement Network

This network is used to deploy the VMware Enterprise PKS managementcomponents. Dedicate the network only to the management components.

Kubernetes NodeNetwork

This network is used for the Kubernetes management nodes. A /24 networkis taken from the Node IP Block and is allocated to the master and workernodes. These nodes embed services like the Node Agent to monitor theliveness of the cluster.

Kubernetes PodNetwork

This network is used when an application is requested to deploy onto a newKubernetes namespace. A /24 network is taken from the Pod IP Block andis allocated to a specific Kubernetes namespace allowing for networkisolation and policies to be applied between namespaces. The NSX-TContainer Plug-in automatically creates the NSX-T logical switch and Tier-1router for each namespace.


VMware, Inc. 58

Load Balancer and NATSubnet

This network pool, also known as the Floating IP Pool, provides IPaddresses for load-balancing and NAT services which are required as apart of an application deployment in Kubernetes.

HTTP or HTTPS Proxy Production environments can deny direct access to public Internet servicesand between internal services by placing an HTTP or HTTPS proxy in thenetwork path between Kubernetes nodes and those services.

If your environment includes HTTP or HTTPS proxies, configuring PKS touse these proxies allows the Kubernetes nodes to access public Internetservices and other internal services.

Table 3-31. Supported Network Topologies for VMware Enterprise PKS

NetworkCan be on a vSphereVirtual Switch

Can be on an NSX-TLogical Switch Routable Non-Routable (NAT)

VMware Enterprise PKSManagement Network

Yes Yes Yes Yes

Kubernetes NodeNetwork

Yes Yes Yes Yes

Kubernetes PodNetwork

No Yes Yes Yes

Load Balancer and NATSubnet

N/A N/A Yes No

VMware Enterprise PKSDeployment External toNSX-T

The VMware Enterprise PKS management components are deployed toeither a vSphere Standard Switch or vSphere Distributed Switch.Connectivity between the VMware Enterprise PKS managementcomponents, Kubernetes cluster management, and the NSX-T Tier-0 routeris provided by using a physical or virtual router. Network AddressTranslation (NAT) is automatically configured on the NSX-T Tier-0 router toprovide connectivity to and from the Kubernetes Pod networks.

VMware Enterprise PKSDeployment Internal toNSX-T

The VMware Enterprise PKS management components are deployed to anNSX-T logical switch. Connectivity between the VMware Enterprise PKSmanagement components, Kubernetes cluster management, and NSX-TTier-0 router is provided through the use of a physical or virtual router. Boththe VMware Enterprise PKS management network and the Kubernetescluster management network logical switch can either be routed or can useNAT. NAT is automatically configured on the Tier-0 router to provideconnectivity to and from the Kubernetes Pod networks.


VMware, Inc. 59

Figure 3-9. Logical Architecture of an SDDC with VMware Enterprise PKS

APP

OSAPP

OSAPP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OSAPP

OS

APP

OSAPP

OSAPP

OS

NSX-TEdges

PCF OpsManager

Virtual InfrastructureManagement

ESXiTransport

Node

NSX-VController

(Mgmt)

OtherManagementApplications

NSX-VEdge

(Mgmt)

NSX-VManager(Mgmt)

NSX-TManager

(Compute)

ESXi ESXi ESXi

SDDC Kubernetes Workloads Virtual Infrastructure Shared Edge and Compute

NSX-T Transport Zones

N-VDS (Compute) vDS (Mgmt)

NSX-T Transport Zone (Mgmt)

Shared Edge and Compute Cluster3 Compute Clusters Management Cluster


Managed by:Management vCenter Server

Network: External(Internet/Corporate)


vCenterServer(Mgmt)

vCenterServer

(Compute)

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node

ESXiTransport

Node



Kubernetes Clusters

VMwareEnterprice

PKS

BOSHDirector

VMwareHarbor

Registry

ESXi


VMware, Inc. 60

Figure 3-10. Network Architecture for VMware Enterprise PKS

Shared Edge andCompute Cluster

VMware PKS Management Logical Switch

Kubernetes Cluster Management Logical Switch

Kubernetes Namespace “Prod” - Logical Switch 3

K8s Master K8s Worker K8s Worker K8s Worker

Ops Manager BOSH Director

Tier-0 RouterActive/Standby

EdgeVM 1

Edge VM 2

Tier-1Router

Tier-1Router

Tier-1Router

Tier-1Router

Tier-1Router

Pod 2

Pod 1

Pod 3

Pod 1 Pod 2 Pod 3

Kubernetes Namespace “Dev” - Logical Switch 1

Kubernetes Namespace “Test” - Logical Switch 2

HarborRegistry

PKS ControlPlane


VMware, Inc. 61

Table 3-32. Design Decisions on Networks for VMware Enterprise PKS


PKS-CS-SDN-001 Place the VMware EnterprisePKS managementcomponents on an NSX-Tlogical switch.

The VMware Enterprise PKSmanagement componentsmust be co-located with thevSphere cluster so that youcan provision the BOSHDirector in the same workloaddomain.

You provision the vSphereShared Edge and ComputeCluster with NSX-T and thecluster only contains NSX-Tlogical switches. You alsodeploy the VMwareEnterprise PKS managementworkloads and NSX-T EdgeTransport Node VMs to thatcluster.

The VMware Enterprise PKSmanagement network andKubernetes clustermanagement network must beon separate NSX-T logicalswitches.

PKS-CS-SDN-002 Place the VMware EnterprisePKS management workloadson a dedicated IP subnet.

BOSH Director manages theIP address allocation whendeploying the VMwareEnterprise PKS managementcomponents.

IP addresses in the VMwareEnterprise PKS managementnetwork that are not managedby BOSH Director must beadded to the blacklist in theBOSH Director tile so that its IPaddress assignments do notimpact external workloads.Ensuring that this subnet isdedicated for this purposeremoves any risk of duplicateIP addresses.

PKS-CS-SDN-003 Place the VMware EnterprisePKS managementcomponents on a virtualnetwork that is routable to themanagement network forvSphere and NSX-T.

Provides connectivity to thevSphere and NSX-Tinfrastructure.

Simplifies the network designand reduces the complexity ofnetwork troubleshooting byavoiding the use of NetworkAddress Translation (NAT) forthe VMware Enterprise PKSmanagement workloads.

Increases the network addressmanagement overhead.

PKS-CS-SDN-004 Create a Floating IP Pool forthe NSX-T load balancervirtual IP addresses.

VMware Enterprise PKS usesthe Floating IP Pool toallocate IP addresses to theload balancers created foreach of the Kubernetesclusters. The load balancerroutes the API requests to themaster nodes and the dataplane.

You must manually configurethis IP Pool in NSX-T andVMware Enterprise PKS.


VMware, Inc. 62

Table 3-32. Design Decisions on Networks for VMware Enterprise PKS (continued)


PKS-CS-SDN-005 Specify a /16 network for thePods IP Block ID forKubernetes pods.

The NSX-T Container Plug-inuses this block to assignaddress space to Kubernetespods through the ContainerNetworking Interface. Asingle /24 network segmentfor the Pods IP Block isinstantiated per pod in theKubernetes namespace.

n VMware Harbor Registryuses 172.17.0.0/16 -172.22.0.0/16 andKubernetes uses172.17.0.0/16 for itsinternal networking space.Do not specify these IPsubnets for pod use orKubernetes worker nodeswill be unable to reachHarbor Registry for acontainer image pull.

n If you size the Pods IPBlock incorrectly,Kubernetes workloadprovisioning might fail.

n The subnet size of /24equals a maximum of 256pods per namespace.

n You must manuallyconfigure this IP Block inNSX-T and VMwareEnterprise PKS.


VMware, Inc. 63

Table 3-32. Design Decisions on Networks for VMware Enterprise PKS (continued)


PKS-CS-SDN-006 Specify a /16 network for theNodes IP Block ID forKubernetes nodes.

During a Kubernetes clusterdeployment, VMwareEnterprise PKS instantiates asingle /24 subnet from theNodes IP Block. VMwareEnterprise PKS uses theblock to assign addressspace to the Kubernetesmaster and worker nodeswhen new clusters aredeployed or a clusterincreases scale.

n Harbor Registry uses172.17.0.0/16 -172.22.0.0/16 andKubernetes uses172.17.0.0/16 for itsinternal networking space.Do not specify these IPsubnets for node use orKubernetes worker nodeswill be unable to reachHarbor Registry for acontainer image pull.

n If you size the Nodes IPBlock incorrectly,Kubernetes workloadprovisioning might fail.

n The subnet size of /24equals a maximum of 256pods per namespace.

n You must manuallyconfigure this IP Block inNSX-T and VMwareEnterprise PKS.

PKS-CS-SDN-007 Enable the post-deployscripts when configuring theBOSH Director Tile.

The NSX-T Container Plug-inis automatically deployed aspart of a Kubernetes clusterto ensure that networking viaNSX-T is possible.

None.

Availability Design for Enterprise PKS with NSX-T WorkloadDomainsYou can tolerate failures at the pod, process, virtual machine, and availability zone layers in VMwareEnterprise PKS.

VMware Enterprise PKS provides the following levels of high-availability.

n BOSH Director and Kubernetes

n Pod level (done by Kubernetes)

n Process level (done by BOSH Director)

n Virtual machine level (done by BOSH Director)

n Availability zones

BOSH Director VM and Service RecoveryVMware Enterprise PKS uses BOSH Director to monitor and resurrect Kubernetes processes andunderlying virtual machines.


VMware, Inc. 64

BOSH Director provides a built-in feature VM Resurrector Plugin. You can enable this plug-in toproactively monitor and provide self-repairing capabilities for failed virtual machines.

The VM Resurrector Plugin consists of VM Resurrector and BOSH Agent Process Monitor.

Figure 3-11. Components for VM and Service Recovery in BOSH Director

BOSH Director

VM Resurrector Plugin

Kubernetes Node

BOSH Agent

Kubernetes Node

BOSH Agent

Virtual MachineMonitoring

ProcessMonitoring

ProcessMonitoring

K8sProcesses

K8sProcesses

VM Resurrector The VM Resurrector plugin monitors and heals the following virtualmachines in a VMware Enterprise PKS deployment:

n Kubernetes cluster virtual machines, in the master and worker nodes

n VMware Enterprise PKS control plane virtual machine

BOSH Agent ProcessMonitor

The process monitor restarts a process by interacting with the BOSH agenton each virtual machine that VMware Enterprise PKS is provisions.

The following processes are monitored on the Kubernetes master node:

kube-apiserver n Standard Kubernetes component.

n Component on the master node that exposesthe Kuberenetes API.

n Front end for the Kubernetes control plane.

kube-controller-manager

n Standard Kubernetes component.

n Component on the master node that runscontrollers.

kube-scheduler n Standard Kubernetes component.

n Component on the master that watches newlycreated pods that have no node assigned andselects a node for them to run.


VMware, Inc. 65

etcd n Standard Kubernetes component.

n Consistent and highly available key value storeused as backing Kuberenetes store for allcluster data.

etcd_consistency_checker

n VMware Enterprise PKS add-on component.

n Verifies that the etcd key value store databaseis consistent.

blackbox n Open source add-on component.

n Runs on master and worker nodes.

n Allows probing of endpoints over HTTP, HTTPS,DNS, TCP, and ICMP.

bosh-dns n VMware Enterprise PKS add-on component.


n Keeps fully qualified domain names (FQDNs)for all nodes in a Kubernetes cluster.

bosh-dns-healthcheck



n Maintains local DNS database consistency.

The following processes are monitored on the Kubernetes worker node:

docker n Standard Docker runtime engine.

kubelet n Standard Kubernetes component.

n Represents an agent that runs on each node inthe cluster to ensure that containers are runningin a pod.

kube-proxy n Standard Kubernetes component.

n Enables Kubernetes service abstraction bymaintaining network rules on the host andperforming connection forwarding.

blackbox n Open source add-on component.



VMware, Inc. 66

n Allows probing of endpoints over HTTP, HTTPS,DNS, TCP, and ICMP.

ovsdb-server n VMware Enterprise PKS add-on component.

n A lightweight database server that ovs-vswitchdqueries to obtain its configuration.

ovs-vswitchd n VMware Enterprise PKS add-on component.

n The OVS daemon that implements the virtualswitch on the worker node.

bosh-dns n VMware Enterprise PKS add-on component.


n Keeps fully qualified domain names (FQDNs)for all nodes in a Kubernetes cluster.

bosh-dns-healthcheck



n Maintains local DNS database consistency.

Table 3-33. Design Decisions on Using BOSH VM Resurrector


PKS-CS-AZ-001 Enable the VM RessurectorPlugin feature for BOSHDirector.

Provides automaticmonitoring and self-repairingof failed virtual machines thatVMware Enterprise PKSprovisions.

None.

Availability ZonesAs a fourth level of availability for Kubernetes, VMware Enterprise PKS supports availability zones. If afailure in an availability zone occurs, the Kubernetes cluster remains online.

Availability zones are defined in PCF Ops Manager, where they are associated with either a vSphereCluster or Resource Pool available under a vCenter Server instance. Availability is improved by usingmultiple availability zones and load balancing for Kubernetes cluster deployments and cloud nativeworkloads.

This design calls for a minimum of four availability zones to be deployed. The first availability zonesupports VMware Enterprise PKS management workloads, including PCF Ops Manager, BOSH Director,VMware Enterprise PKS control plane, and VMware Harbor Registry. The last three availability zones willsupport end-user workloads - Kubernetes clusters, including Kubernetes cluster master and worker nodeobjects for cloud native workloads. Kubernetes worker nodes are responsible for handling workloads and


VMware, Inc. 67

can be horizontally scaled to support an increased load. The master node and associated etcd key-valuestore are responsible for the state of the cluster, API endpoints, and cluster management. If more thanone instance of an application is run, the instances are balanced across the worker nodes in allavailability zones that are assigned to the Kubernetes cluster.

Figure 3-12. Availability Design for VMware Enterprise PKS

VMware Enterprise PKS Workload Domain

VMware Enterprise PKS Management

(Resource Pool)

Availability Zone 1 (Compute Cluster 1)

Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster


Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster


Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster

Kubernetes Cluster

NSX-T Edge

NSX-T Edge

NSX-T Edge Cluster

(Resource Pool)

Shared Edge and Compute Cluster

BOSH Director

Ops Manager

VMwareHarbor Registry

VMware Enterprise PKS Control Plane

In a typical Kubernetes cluster, one active master-etcd pair exists. If the master node is down, you cannotdeploy additional containers, scale the environment, or receive telemetry on the Kubernetes cluster.

VMware Enterprise PKS supports the multi-master availability zone or HA cluster feature. You can deploymultiple master-etcd nodes per cluster, across multiple availability zones. Only a single master is active ata time. If a master node or an availability zone fails, to restore the state of the cluster, VMware EnterprisePKS elects one of the master nodes from another availability zone as active.


VMware, Inc. 68

Table 3-34. Design Decisions on Availability of VMware Enterprise PKS

Decision ID Design Decision Decision Justification Decision Implication

PKS-CS-AZ-002 Define the availability zone forVMware Enterprise PKSmanagement workloads as aresource pool in the SharedEdge and Compute vSpherecluster.

Allows for the consolidation ofNSX-T Edge Transport NodeVMs and VMware EnterprisePKS workloads in the samevSphere cluster. If contentionoccurs, it also ensures anadequate resource availabilityfor processing of the networktraffic through NSX-T EdgeTransport Node VMs.

During contention, workloads ina resource pool might lackresources and experience poorperformance. As such,monitoring and capacitymanagement must be aproactive activity.

PKS-CS-AZ-003 Define availability zonesmapped to the root of each ofthe three compute clusters forthe Kubernetes master, theworker, and the poddeployment.

Defining each end-user-facingavailability zone as a vSphereCluster ensures thatavailability zones align withlogical compute failuredomains. In the instancewhere an outage impacts allcompute resources within avSphere cluster, otheravailability zones are notaffected.

Delineation of availability zonesbased on vSphere clusterssignificantly increases thenumber of ESXi hosts neededfor a three availability zonedeployment.

Logging Design for Enterprise PKS with NSX-T Workload DomainsYou enable and configure logging for all VMware Enterprise PKS and NSX-T components to facilitateend-to-end troubleshooting and provide audit capabilities.

You configure the following VMware Enterprise PKS components to forward their logs to the vRealize LogInsight cluster through syslog or native integrations.

n BOSH Director

n VMware Enterprise PKS control plane

n Kubernetes clusters

n Kubernetes pods

n Kubernetes namespaces

n Containers


VMware, Inc. 69

Table 3-35. Design Decisions on Logging for PKS


PKS-CS-LOG-001 Enable the VMwareEnterprise PKS nativeintegration with vRealize LogInsight for log forwarding.

n Simplifies configuration ofVMware Enterprise PKSlogging with nativevRealize Log Insightintegration.

n Aggregates logs basedon cluster, pod,namespace, andcontainer tags.

You must configure VMwareEnterprise PKS to forward logsto the vRealize Log Insightcluster in the tile.

Monitoring Design for Enterprise PKS with NSX-T WorkloadDomainsYou use vRealize Operations Manager to monitor the state of the VMware Enterprise PKS componentsand Kubernetes clusters. You can use the self-monitoring capability of vRealize Operations Manager andreceive alerts about issues that are related to its operational state.

The vRealize Operations Management Pack for Container Monitoring provides visibility and health foreach individual Kubernetes cluster that VMware Enterprise PKS deploys. You can monitor the completeKubernetes topology of nodes, namespaces, pods, containers, and services. In addition, themanagement pack supports highlighting the Key Performance Index and alerts for various objects relatedto Kubernetes clusters that are monitored.


VMware, Inc. 70

Figure 3-13. vRealize Operations Manager Integration with VMware Enterprise PKS andKubernetes Workloads

Metric Adapters

vRealize Operations Manager

Analytics Cluster

Integration

ExternalLoad Balancer

vCenter Server

Access

User Interface

API

vRealizeLog Insight

VMwareEnterprisePKS

Metric Adapters

vCenter Server

NSX

vRealizeLog Insight

vRealizeBusiness

vRealizeAutomation

ManagementPacks

Suite API

Shared Storage

vRealize Operations ManagerRemote Collectors

CollectorGroup

ManagementPacks

Suite API

Remote Collector 2

Remote Collector 1

Shared Storage

StorageDevices

vSAN

Master Replica

Data 1 Data n

KubernetesClusters

SiteRecoveryManager


PKS-CS-MON-001 Install the vRealizeOperations ManagementPack for Container Monitoring

Provides granular monitoringfor the topology of eachKubernetes cluster, includingthe following components:

n Nodes

n Namespaces

n Pods

n Containers

n Services

You must manually install thevRealize OperationsManagement Pack forContainer Monitoring.


VMware, Inc. 71

Information Security and Access Control for Enterprise PKS withNSX-T Workload DomainsTo protect the VMware Enterprise PKS deployment, configure centralized role-based authentication andsecure communication with the components of the SDDC.

AuthenticationThe User Account and Authorization (UAA) module provides the authentication services for VMwareEnterprise PKS. UUA is part of the Ops Manager, BOSH Director, and the VMware Enterprise PKScontrol plane.

UAA supports different authentication methods for each component.

Table 3-36. Supported Authentication Methods for VMware Enterprise PKS Components

PKS ComponentSupported AuthenticationMethods Description

PCF OpsManager n Local UAA

n LDAP

n SAML Identity Provider

The UAA module in Ops Manager and BOSH Directorboth supports an external LDAP authentication source.As a result, you can integrate VMware Enterprise PKSwith external LDAP to enable authentication from anexternally managed enterprise resource.BOSH Director Inherited from PCF Ops Manager

VMware Enterprise PKS controlplane

n Local Account

n LDAP

LDAP can be used to enable authentication from anActive Directory source when accessing the VMwareEnterprise PKS API and command-line interfaces.

Harbor Registry n Local Account

n LDAP

n UAA from VMware EnterprisePKS control plane

n UAA from Pivotal ApplicationService

Configuring VMware Harbor Registry with the UUAinstance in the VMware Enterprise PKS control planeenables authentication from an Active Directory sourcewhen pulling or pushing container images to theregistry.

Figure 3-14. Authentication and Authorization for VMware Enterprise PKS

PCF Ops Manager /

BOSH Director

UAA UAA

VMware Enterprise PKS Control Plane

VMwareHarbor Registry

LDAP

Active Directory


VMware, Inc. 72

Table 3-37. Design Decisions on Authentication for VMware Enterprise PKS


PKS-CS-AUTH-001 Configure PCF Ops Managerto use LDAP authenticationfor UAA.

Provides fine-grained role andprivilege-based access foradministrator and operatorroles.

You must manually configurethe LDAP providers.

PKS-CS-AUTH-002 Use the LDAP integration inthe VMware Enterprise PKScontrol plane.

Provides fine-grained role andprivilege-based access for theadministrator and operatorroles.

You must perform an extraconfiguration step for PCF OpsManager.

PKS-CS-AUTH-003 Use the VMware EnterprisePKS UAA for Harbor Registryauthentication.

Provides fine-grained role andprivilege- based access foradministrator and operatorroles.

None.

EncryptionAccess to all VMware Enterprise PKS management endpoint interfaces requires an SSL connection. Bydefault, the VMware Enterprise PKS components use self-signed certificates. To provide secure access tothe VMware Enterprise PKS components and between SDDC endpoints, replace the default self-signedcertificate with a CA-signed certificate.

You must also provide an NSX Manager Super User Principal Identity Certificate during the VMwareEnterprise PKS installation on NSX-T.

Table 3-38. Design Decisions on Encryption for VMware Enterprise PKS


PKS-CS-SSL-001 Replace the default self-signed certificate on PCF OpsManager with a CA-signedcertificate.

Configuring a CA-signedcertificate ensures that allcommunication to theexternally facing Web UI isencrypted.

Replacing the defaultcertificates with trusted CA-signed certificates from acertificate authority mightincrease the deploymentpreparation time as certificatesrequests are generated anddelivered.

PKS-CS-SSL-002 Add a CA-signed certificate toBOSH Director.

Configuring a CA-signedcertificate ensures that BOSHtraffic is encrypted with atrusted certificate.



VMware, Inc. 73

Table 3-38. Design Decisions on Encryption for VMware Enterprise PKS (continued)


PKS-CS-SSL-003 Replace the default self-signed certificate on theVMware Enterprise PKScontrol plane with a CA-signed certificate.

Configuring a CA-signedcertificate ensures that allcommunication to theexternally facing API isencrypted.


PKS-CS-SSL-004 Replace the default self-signed certificate on VMwareHarbor Registry with a CA-signed certificate.

Configuring a CA-signedcertificate ensures that allcommunication to theexternally facing Web UI isencrypted.


PKS-CS-SSL-005 Configure a certificate torepresent the principal identitywith superuser permissionson NSX-T Manager forapplication-to-applicationcommunication from VMwareEnterprise PKS to NSX-T.

Configuring such a certificateensures that allcommunication betweenVMware Enterprise PKS andNSX-T is encrypted andallows management (create,delete, and modify) of nodenetworking resources.

You must maintain thecertificate lifecycle.

Data Protection Design for Enterprise PKS with NSX-T WorkloadDomainsYou enable and configure data protection for the VMware Enterprise PKS control plane by using BOSHBackup and Recovery. For VMware Harbor Registry, use backup software that is based on the vSphereStorage APIs for Data Protection (VADP) for a consistent image-level backup.

Backups protect your organization against data loss, hardware failure, accidental deletion, or other faults.

Use backup solutions that natively integrate with the virtual infrastructure and container service platformsto ensure a solid data protection strategy.

VMware Enterprise PKS Control PlaneData protection for the VMware Enterprise PKS control plane uses the BOSH Backup and Restore (BBR)framework to back up and restore the management component deployments.

BBR triggers the backup or restore processes on the BOSH deployment, and transfers the backupartifacts to and from the BOSH deployment.

The VMware Enterprise PKS control plane backup includes the following components:

n UAA MySQL database

n VMware Enterprise PKS API MySQL database


VMware, Inc. 74

https://docs.cloudfoundry.org/bbr/index.html

VMware Harbor RegistryFor consistent image-level backups, use backup software that is based on the vSphere Storage APIs forData Protection (VADP) to perform a consistent, image level backup of the Harbor Registry virtualappliance. Adapt and apply the design decisions to the backup software you use.

Table 3-39. Design Decisions on Data Protection for VMware Enterprise PKS


PKS-CS-BKP-001 Use BOSH Backup andRestore to back up theVMware Enterprise PKScontrol plane.

You can back up and restorethe VMware Enterprise PKScontrol plane.

n No native data protectionscheduling for VMwareEnterprise PKS exists. Youmust script it by usingBBR.

n Data protection for VMwareEnterprise PKS does notback up or restoreKubernetes clusters orcloud native applications.

n You must provide a Linuxhost with access to theVMware Enterprise PKSmanagement componentsto install and run BBR.

n BBR stores VMwareEnterprise PKS controlplane backups locally onthe host. You must send ormove them to an externalsecure storage.

PKS-CS-BKP-002 Deploy a Linux jump box forBOSH Backup and Recoveryon the same NSX-T logicalswitch as the VMwareEnterprise PKS managementcomponents.

n BOSH Backup andRecovery connects to themanagement componentson their private IPaddress. The jump boxmust be in the samenetwork because BOSHBackup and Recoverydoes not support SSHgateways.

n The virtual network isroutable to themanagement network forvSphere and NSX-Tenabling the VMwareEnterprise PKS controlplane backup objects tobe sent to an externallocation for long-termsecure storage.

None.


VMware, Inc. 75

Table 3-39. Design Decisions on Data Protection for VMware Enterprise PKS (continued)


PKS-CS-BKP-003 Compress, encrypt, andtransfer backups artifactsfrom the BOSH Backup andRecovery jump box to asecure storage space.

Using compression andencryption minimizes risk byensuring that the backupartifacts and metadata filesfor each backup of theVMware Enterprise PKScontrol plane is securelystored.

n You must perform andmanage the compression,encryption, and transfer ofthe backup artifacts to asecure storage eithermanually or throughcustom scripts.

n You must manage thelifecycle of the backupartifacts.

PKS-CS-BKP-004 Use a backup solution that iscompatible with vSphereStorage APIs - DataProtection (VADP) and canperform image level backupsof the Harbor Registrycomponents.

You can back up and restoreVMware Harbor Registry atthe virtual machine imagelevel.

None.


VMware, Inc. 76

Documents

Architecture and Design for VMware Enterprise PKS …...Architecture Overview for Enterprise PKS with NSX-T Workload Domains 2 VMware Enterprise PKS is a container services solution