Upload
others
View
12
Download
0
Embed Size (px)
Citation preview
Merlin Glynn (VMware)Ramiro Salas (Pivotal)
CNA2006BE
#VMworld #CNA2006BE
Deep Dive: Architecting Container Services with VMware & Pivotal Developer-Ready Infrastructure
VMworld 2017 Content: Not fo
r publication or distri
bution
• This presentation may contain product features that are currently under development.
• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.
• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.
• Technical feasibility and market demand will affect final delivery.
• Pricing and packaging for any new technologies or features discussed or presented have not been determined.
Disclaimer
2
VMworld 2017 Content: Not fo
r publication or distri
bution
Agenda
1 Pivotal Cloud Foundry 101Why do my Developers want it?
2 Kubernetes 101Why do my Developers want it?
3 Ops: Architecture for Containers 101
4 Ops: Network & Security Controls
5 Ops: Monitoring & Logging
6 Ops: Platform as Code{}
7 Ops: PCF+PKS
3
VMworld 2017 Content: Not fo
r publication or distri
bution
Pivotal Cloud Foundry 101Why do my Developers want It?
VMworld 2017 Content: Not fo
r publication or distri
bution
Pivotal Cloud Foundry 101
5
war
Availability Zone 1 Availability Zone 2 Availability Zone 3
Staging
Root
FS
Build
Pack
war
`cf push`
Drop
let
A
I
A
Imyapp.foo.com
*.foo.com = NSX Edge Vip
NSX Edge
PCF Routing PCF Routing PCF Routing
LB Pool Members
“Here is my source code
Run it on the cloud for me
I do not care how”
URL Request:
myapp.foo.com
Developer
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes 101Why do my Developers want It?
VMworld 2017 Content: Not fo
r publication or distri
bution
Kubernetes 101
7
K8s Cluster
Worker
`kubectl apply –f myapp.yml`
Developer
Worker
kube-proxyMaster
etcd
kube-proxy
Service: nodeport | ingress
POD POD
Load Balancer
URL Request:
myapp.foo.com/k8siscool
Docker
Registry
VMworld 2017 Content: Not fo
r publication or distri
bution
Architecting for Containers 101
VMworld 2017 Content: Not fo
r publication or distri
bution
DRI … Architect for Agility
Virtual Data Center
• Architect the right Abstractions
• Automate Everything
• Build for Failure
Control Agility
vSphere NSX vSAN
Pivotal Cloud Foundry
PCF
PKS
BOSH powered Kubernetes
BO
SH
Platform
Operator
Developer
Wavefront
Self Service
• Automation
• Day 2 Operations
• Control
• Application Services or Container Services
• Application Logging & Monitoring
vRLI (Dev)
vRops
vRLI (Ops)
vRNI
VMworld 2017 Content: Not fo
r publication or distri
bution
Ops: Architecting for Availability & Scale
VMworld 2017 Content: Not fo
r publication or distri
bution
vSphere Fundamentals for PCF
11
BOSH
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
Developer
PCF OrgPCF Space
App App
Architecting for Availability & Scale
Virtual Data Center
VMworld 2017 Content: Not fo
r publication or distri
bution
Physical Fault Domains
12
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
Developer
PCF OrgPCF Space
App App
Virtual Data Center
Cluster Design Best Practices
• Enable vSphere HA
• Enable & Tune BOSH HealthMonitor Resurrection
vSphere HA
vSphere HA
BOSH Agent(s)
BOSH
BOSH Hlth
Monitor
VMworld 2017 Content: Not fo
r publication or distri
bution
Physical Fault Domains
13
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
Developer
PCF OrgPCF Space
App App
Virtual Data Center
Cluster Design Best Practices
• Enable vSphere HA
• Enable & Tune BOSH HealthMonitor Resurrection
• Plan For Singletons
– Externalize
– DR (vDP, Image, Snapshot, pgdump)
BOSH Agent(s)
BOSH
webdav
(blob)
BOSH
S3 Compat
Storage
PCF BlobStore
DR
DR DR
VMworld 2017 Content: Not fo
r publication or distri
bution
IaaS Multi Tenancy
14
AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
vSAN/NFS/VMFS
Resource Pool
AZ1 Foundation 1
ESX Cluster
vSAN/NFS/VMFS
ESX Cluster
vSAN/NFS/VMFS
ESX Cluster
Virtual Data Center
Cluster Design Best Practices
• Enable vSphere HA
• Enable & Tune BOSH HealthMonitor Resurrection
• Plan For Singletons
– Externalize
– DR (vDP, Image, Snapshot, pgdump)
• Use Resource Pools & Scale Clusters as needed
BOSH
Resource Pool
AZ2 Foundation 1
Resource Pool
AZ3 Foundation 1
Resource Pool
AZ1 Foundation 2Resource Pool
AZ2 Foundation 2
Resource Pool
AZ3 Foundation 2
Dev|Test|UAT
Foundation
Prod
Foundation
C
P
I
C
P
I
CPI Acct 1 Assigned
vCenter PermsPool Limits & Shares
CPI Acct 2 Assigned
vCenter PermsPool Limits & Shares
AC
L
Quota
VMworld 2017 Content: Not fo
r publication or distri
bution
Recovering the Platform
15
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
PCF OrgPCF Space
App App
BC/DR Best Practices
• Platform as Code{}
BOSH Agent(s)
BOSH
BOSH
S3 Compat
Storage
PCF BlobStore
VMworld 2017 Content: Not fo
r publication or distri
bution
Backup Job
Recovering the Platform
16
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
PCF OrgPCF Space
App App
BC/DR Best Practices
• Platform as Code{}
• Backup Services for Platform Persistent Data
• Backup Services for App Service Persistent Data
– Don’t Forget External App Data not managed by PCF
BOSH Agent(s)
BOSH
BOSH
S3 Compat
Storage
PCF BlobStore
MySql PCF Service
Tile
mysql mysql mysql
VMworld 2017 Content: Not fo
r publication or distri
bution
Backup Job
Recovering the Platform
17
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
PCF OrgPCF Space
App App
BC/DR Best Practices
• Platform as Code{}
• Backup Services for Platform Persistent Data
• Backup Services for App Service Persistent Data
– Don’t Forget External App Data not managed by PCF
• VMotion (Yes)
• SVMotion (NO)
BOSH Agent(s)
BOSH
BOSH
S3 Compat
Storage
PCF BlobStore
*
*
vmdk
VMworld 2017 Content: Not fo
r publication or distri
bution
{}
NSXEdge LTM
Multi-Site Platforms
18
AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
BC/DR Best Practices
• Business Continuity w/ Multi Site
– GSLB
BOSH
GSLB
NSXEdge LTM
Health Checks
Health Checks
{}{}
Common ServiceMeshData
VMworld 2017 Content: Not fo
r publication or distri
bution
19
VMware PKS
Kubernetes on BOSH (Kubo)
BOSH
NSX
Analytics Automation
SecurityOperations
Monitoring
GCP
Service Brokermasteretcd worker
Logging
vSANvSphere
masteretcd workerContainer
Registry
(PKS)
VMworld 2017 Content: Not fo
r publication or distri
bution
What about PKS?
20
BOSH Deploys KUBO
• Same BOSH Availability Zone Constructs are available
• Spread Core K8S Jobs across BOSH Availability Zones
– Master
– ETCD
– Workers
• Multi Site can be GSLB in much the same way as PCF
• BOSH Makes Kubernetes Day 1 & Day 2 easy.
• Does NOT require PCF
VMworld 2017 Content: Not fo
r publication or distri
bution
Architecting the Platform
21
BC/DR Best Practices
• Platform as Code{}
• Backup Services for Platform Persistent Data
• Backup Services for App Service Persistent Data
• Business Continuity w/ Multi Site
Cluster Design Best Practices
• Enable vSphere HA
• Enable & Tune BOSH HealthMonitorResurrection
• Plan For Singletons
• Use Resource Pools & Scale Clusters as needed
• VMotion (Yes)
• SVMotion (NO)
DEVELOPER-READY
INFRASTRUCTURE
Deliver innovation faster
to customers
Architectural Resource(s) Link(s)
VMware VVD (Validated Design) In Progress
Pivotal ’Lite’ Reference Architecture https://docs.pivotal.io/pivotalcf/1-11/refarch/vsphere/vsphere_ref_arch.html
VMworld 2017 Content: Not fo
r publication or distri
bution
Ops: Network & Security Controls
VMworld 2017 Content: Not fo
r publication or distri
bution
http://myapp.default-apps.foo.com
Network Fundamentals for PCF
23
Network Design Best Practices
• Get Wildcard Certs & DNS Approved
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
AP
PS
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub)
Cell
Cell
SS
HTC
P
TC
P
SS
H
AP
PS
A
I
A
I
External Services
Internal Apps
LS: OSPF
app.public-apps.foo.com
CF
ASG
{}
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
AP
PS
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub)
Cell
Cell
SS
HTC
P
TC
P
SS
H
AP
PS
A
I
A
I
External Services
Internal Apps
LS: OSPF
CF
ASG
Network Security & Controls
24
Network Design Best Practices
• Get Wildcard Certs & DNS Approved
• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs
– On Demand: Developer trigger VM provision
– Pre-Provisioned: Ops triggers VM provision
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
AP
PS
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub)
Cell
Cell
SS
HTC
P
TC
P
SS
H
AP
PS
A
I
A
I
External Services
Internal Apps
LS: OSPF
CF
ASG
Network Security & Controls
25
Network Design Best Practices
• Get Wildcard Certs & DNS Approved
• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs
– On Demand: Developer trigger VM provision
– Pre-Provisioned: Ops triggers VM provision
• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
AP
PS
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub)
Cell
Cell
SS
HTC
P
TC
P
SS
H
AP
PS
A
I
A
I
External Services
Internal Apps
LS: OSPF
CF
ASG
Network Security & Controls
26
Network Design Best Practices
• Get Wildcard Certs & DNS Approved
• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs
– On Demand: Developer trigger VM provision
– Pre-Provisioned: Ops triggers VM provision
• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges
• Use NSX Edge for Load Balancing, SSL Termination, & Perimeter FW ACLs
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
LS: Isolation_A
PCF
PCF Isolation Segment
GO
RTRCELL CELL CELL
/22
Public Apps
AP
PS
DNS:
*.public-apps.foo.com
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)
Cell
Cell
SS
HTC
P
ISO
ISO
TC
P
SS
H
AP
PS
A
I
A
I
CF
ASG
External Services
Internal Apps
LS: OSPF
Network Security & Controls
27
Network Design Best Practices
• Get Wildcard Certs & DNS Approved
• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs
– On Demand: Developer trigger VM provision
– Pre-Provisioned: Ops triggers VM provision
• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges
• Use NSX Edge for Load Balancing, SSL Termination, & Perimeter FW ACLs
• Use NSX DLR for PCF Org & Space level segmentation
– Multiple Isolation Segments
– Isolation segments allow Operators to group Diego cells and attach to multiple Logical Swicthes.
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
LS: Isolation_A
PCF
PCF Isolation Segment
GO
RTRCELL CELL CELL
/22
Public Apps
AP
PS
DNS:
*.public-apps.foo.com
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)
Cell
Cell
SS
HTC
P
ISO
ISO
TC
P
SS
H
AP
PS
A
I
A
I
CF
ASG
External Services
Internal Apps
LS: OSPF
Network Security & Controls
28
Network Design Best Practices …
• Use NSX Security Groups for dynamic security principals
– BOSH Integrated NSX (Dynamic Membership)
– Ingress & Egress PCF Org/Space Specific FW
– Dynamic LB Pool Membership
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
BOSH
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
LS: Isolation_A
PCF
PCF Isolation Segment
GO
RTRCELL CELL CELL
/22
Public Apps
AP
PS
DNS:
*.public-apps.foo.com
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)
Cell
Cell
SS
HTC
P
ISO
ISO
TC
P
SS
H
AP
PS
A
I
A
I
CF
ASG
External Services
Internal Apps
LS: OSPF
Network Security & Controls
29
Network Design Best Practices …
• Use NSX Security Groups for dynamic security principals
– BOSH Integrated NSX (Dynamic Membership)
– Ingress & Egress PCF Org/Space Specific FW
– Dynamic LB Pool Membership
• Use Distributed Firewall Policy
– Leverage PCF Integrated Dynamic Security Groups
– Control East+West from single policy engine
– Control App to App at the Org/Space level with Isolation Segments
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
BOSH
{}
VMworld 2017 Content: Not fo
r publication or distri
bution
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
LS: Isolation_A
PCF
PCF Isolation Segment
GO
RTRCELL CELL CELL
/22
Public Apps
AP
PS
DNS:
*.public-apps.foo.com
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub) IaaS: vSphere Security Zone B (Spoke)
Cell
Cell
SS
HTC
P
ISO
ISO
TC
P
SS
H
AP
PS
A
I
A
I
CF
ASG
External Services
Internal Apps
LS: OSPF
Network Security & Controls
30
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
{}
Network Design Best Practices …
• Use NSX Security Groups for dynamic security principals
– BOSH Integrated NSX (Dynamic Membership)
– Ingress & Egress PCF Org/Space Specific FW
– Dynamic LB Pool Membership
• Use Distributed Firewall Policy
– Leverage PCF Integrated Dynamic Security Groups
– Control East+West from single policy engine
– Control App to App at the Org/Space level with Isolation Segments
• Use RFC 1918 for Repeatability
VMworld 2017 Content: Not fo
r publication or distri
bution
Network Security & Controls
31
Platform
Operator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
Network Design Best Practices …
• Platform as Code{} to automate Day 1 & Day 2 ops
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *A
PP
S
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub)
Cell
Cell
SS
HTC
P
TC
P
SS
H
AP
PS
A
I
A
I
External Services
Internal Apps
LS: OSPF
CF
ASG
DNS:
*.sys.pcf.foo.com
*.default-apps.foo.com
Single PCF ‘Foundation’
PCF
Elastic Runtime
** *
LS: Services #
LS: Infra
NSX Edge
LS: ERTLS: Services #
LS: Services #
CF Control
Plane
Ops
Mgr
(PCF)
BOSH GO
RTR
Diego
Brain
TCP
RTR
/26 /22/24(s)
PCF
…
PCF
Rabbit
PCF
Mysql
* * *
AP
PS
Logical Routing (DLR)
IaaS: vSphere Security Zone A (Hub)
Cell
Cell
SS
HTC
P
TC
P
SS
H
AP
PS
A
I
A
I
External Services
Internal Apps
LS: OSPF
CF
ASG
VMworld 2017 Content: Not fo
r publication or distri
bution
Network Security & Controls
PCF OrgPCF Space
AppA AppB AppC
cf create-security-group SECURITY-GROUP PATH-TO-RULES-FILE cf create-security-group dev-mssql mssql.json
PCF Application Security Groups (ASG):
– Uses iptables in the Diego Cell Server
– Controls Egress only at the container source level
– Can control any IP address as the target
• Operator Declares in the Platform
[ {
"protocol": "tcp",
"destination": "10.0.11.0/24",
"ports": "1-65535"
},
{
"protocol": "udp",
"destination": "10.0.11.0/24",
"ports": "1-65535"
} ]
Platform
Operator
Prod Mssql
192.168.11.10
Prod Mssql
10.0.11.10
VMworld 2017 Content: Not fo
r publication or distri
bution
Network Security & Controls
PCF OrgPCF Space
AppA AppB AppC
cf allow-access SOURCE-APP DESTINATION-APP --protocol PROTOCOL --port PORT• cf allow-access “AppA” “Appc” --protocol TCP --port 443
Developer
PCF Container to Container Networking:
– Creates and Overlay (VXLAN)
– Controls ingress & egress between Ais(containers)
– Uses CNI
• Today Flannel
• Tomorrow NSX-T
– Developer can Declare in CI/CD
VMworld 2017 Content: Not fo
r publication or distri
bution
What about PKS?
34
KUBO Networking is less Complex
• Typically multiple smaller K8s Deployments
• The core Kubernetes components need to route to each other
• Services Deployed on an Overlay Network
– NSX-T
• Enterprise Security Policy
• Enterprise Tools & Logging
• Common Ingress Paths:
– kube-proxy running on external gateway
– Load Balance to kube-proxy
Image source: https://github.com/cloudfoundry-incubator/kubo-deployment/blob/master/docs/images/kubo-network.png
Load Balancer
EXTERNAL
SVC
Request
External
Service Gateway
kubeproxy
VMworld 2017 Content: Not fo
r publication or distri
bution
Network Security & Controls
35
DEVELOPER-READY
INFRASTRUCTURE
Deliver innovation faster
to customers
Resource(s) Link(s)
KUBO Git Repo https://github.com/cloudfoundry-incubator/kubo-deployment
VMware PCF & NSX Design Guide Coming Soon
Network Design Best Practices
• Use NSX Security Groups for dynamic security principals
• Use Distributed Firewall Policy
– Control East+West from single policy engine
– Control App to App at the Org/Space level with Isolation Segments
• Use Container to Container Networking to allow developer to define fine grained App level security
• Use RFC 1918 Repeatability
• Platform as Code{} to automate Day 1 & Day 2 ops
Network Design Best Practices …
• Use Multiple NSX Logical Switches & Subnets, 1 per Deployment (PCF Tile) allow Subnet to Service Level ACLs
• Use Application Security Groups (ASGs), App level egress firewall to PCF & external IP ranges
• Use NSX Edge for Load Balancing, SSL Termination, & Perimeter FW ACLs
• Use NSX DLR for PCF Org & Space level segmentation
VMworld 2017 Content: Not fo
r publication or distri
bution
Ops: PCF Monitoring & Logging
VMworld 2017 Content: Not fo
r publication or distri
bution
Monitoring & Logging
37
Developer
Virtual Data Center
– I need to keep my apps healthy
– I need self service to my Apps Log’s
– I need to instrument my Apps (APM)
Platform
Operator
– I need to keep the Platform healthy
– I need to plan capacity
– I need to watch & Alert on KPIs
– I need to audit
VMworld 2017 Content: Not fo
r publication or distri
bution
Monitoring & Logging
38
Developer
Virtual Data Center
– I need to keep my apps healthy
– I need self service to my Apps Log’s
– I need to instrument my Apps (APM)
PCF Metrics
`cf logs appA`
https://metrics.sys.pcf-foundation.io
Nozzle
vRLIhttps://vrli.pcf-foundation.io
Developer Log Access Routes
– `cf logs`: streams single app’s log events for dev to redirect where needed
– PCF Metrics: PCF app correlating App logs, and container Metrics, ~2week retention
– vRLI: Longer term scalable log storage and indexing, dashboards, & alerts
VMworld 2017 Content: Not fo
r publication or distri
bution
Agents Added to Buildpacks
Future !!!
Monitoring & Logging
39
Developer
Virtual Data Center
– I need to keep my apps healthy
– I need self service to my Apps Log’s
– I need to instrument my Apps (APM)
App & App execution specific Metrics
• tc_server: jdbc_query_failed
• custom_app_metric: transaction_response_time
Platform
Operator
Exposed to developers via CF Service Broker
`cf create service my-apm-endpoint`
VMworld 2017 Content: Not fo
r publication or distri
bution
Monitoring & Logging
40
Platform
Operator
– I need to keep the Platform healthy
– I need to plan capacity
– I need to watch & Alert on KPIs
– I need to audit
vRops
vRops Nozzle
Cloud Foundry Metrics (KPIs)
vSphere & NSX Metrics (KPIs)VMworld 2017 Content: Not fo
r publication or distri
bution
Monitoring & Logging
41
Platform
Operator
– I need to keep the Platform healthy
– I need to plan capacity
– I need to watch & Alert on KPIs
– I need to audit
vRLI
vRops
vRops Nozzle
Cloud Foundry Metrics (KPIs)
vSphere & NSX Metrics (KPIs)
Syslog Nozzle
vSphere & NSX Events
CF Platform Events
Thre
shold
s
Ale
rts
Da
sh
bo
ard
s VMworld 2017 Content: Not fo
r publication or distri
bution
Monitoring & Logging
42
Platform
Operator
– I need to keep the Platform healthy
– I need to plan capacity
– I need to watch & Alert on KPIs
– I need to audit
vRLI
vRops
vRops Nozzle
Cloud Foundry Metrics (KPIs)
vSphere & NSX Metrics (KPIs)
Syslog Nozzle
vSphere & NSX Events
CF Platform Events
Thre
shold
s
Ale
rts
Da
sh
bo
ard
s
All App Events
VMworld 2017 Content: Not fo
r publication or distri
bution
Deamon
Set
Deamon
Set
What about PKS?
POD vRLI
POD
vRLI
• App Logging
• System Logging
– OS & Processes not run in Containers
App Logging
• Per App Only
Sidecar
• App Logging @ Pod level
POD
Daemon
Set
(PODs)
vRLI
POD
LOGGER
DOCKERDDOCKERD
vRLI
DaemonSet
• App Logging @ Cluster level
• Cluster Logging
Dockerd
• App Logging @ Cluster level
• Cluster Logging
• Not handled in K8s API
SyslogD
Platform
Operator
Developer
VMworld 2017 Content: Not fo
r publication or distri
bution
What about PKS?
K8s Monitoring Integration w/ Wavefront by VMware
Wavefront Integration can be deployed as containers within the K8s Cluster
– Proxy
– Heapster
• Comprehensive Dashboards
– SaaS
• APM for the Developer
• Cluster KPIs for the Operator
• Integrated with PKS
Image source: https://www.wavefront.com/surf-container-wave-join-wavefront-container-world-santa-clara/
Platform
Operator
Developer
VMworld 2017 Content: Not fo
r publication or distri
bution
45
Platform
Operator
What about PKS?
vRealize Operations & K8s
• Operator KPIs
• Single Pane for SDDC & K8s clusters monitoring
• vRLI Integrated
• Alert on K8s KPIs
• Entity Relationship
• Capacity Planning
• Integrated with PKS
VMworld 2017 Content: Not fo
r publication or distri
bution
Ops: Monitoring & Logging
46
DEVELOPER-READY
INFRASTRUCTURE
Deliver innovation faster
to customers
Resource(s) Link(s)
Wavefront: KUBO Integration https://community.wavefront.com/docs/DOC-1204
Blue Medora : vRops MP https://marketplace.vmware.com/vsx/solutions/blue-medora-mp-for-pivotal-cloud-foundry
Blue Medora : vRLI Pack https://marketplace.vmware.com/vsx/solutions/content-pack-for-pivotal-cloud-foundry
Developer
Virtual Data Center
– I need to keep my apps healthy
– I need self service to my Apps Log’s
– I need to instrument my Apps (APM)
Platform
Operator
– I need to keep the Platform healthy
– I need to plan capacity
– I need to watch & Alert on KPIs
– I need to audit
VMworld 2017 Content: Not fo
r publication or distri
bution
Ops: Platform as Code{}
VMworld 2017 Content: Not fo
r publication or distri
bution
BOSH 101
48
• Built for Platform Operators
• Deploys Complex Distributed Systems
– PCF
– Kubo
• Day 1 & Day 2 Ops
– Initial Deployment
– Updates/Patches
– Maintains Health
Platform
Operator
VMworld 2017 Content: Not fo
r publication or distri
bution
NSX_Config:
edge_vip_1:3
nsxmgr_endpoint: nsxmgr.vmware.io
lswicth_ert_cidr: 192.168.10.0/22
49
AZ1 AZ2 AZ3
Platform
Operator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
Drives NSX-V
NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)
Platform as Code{}
• Declarative
Day 1 & Day 2
YAML
VMworld 2017 Content: Not fo
r publication or distri
bution
Ert_config:
diego_database_instances:3
diego_brain_instances: 3
diego_cell_instances: 9
50
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
PCF OrgPCF Space
App App
BOSH
Drives NSX-V
NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)
Platform as Code{}
• Declarative
Day 1 & Day 2
YAML
VMworld 2017 Content: Not fo
r publication or distri
bution
Ert_config:
diego_database_instances:3
diego_brain_instances: 3
diego_cell_instances: 12
51
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
PCF OrgPCF Space
App App
BOSH
Drives NSX-V
NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)
Platform as Code{}
• Declarative
Day 1 & Day 2
YAML
Cell_3 Cell_3 Cell_3
VMworld 2017 Content: Not fo
r publication or distri
bution
52
AZ1 AZ2 AZ3
Platform
Operator
Ops Manager
(OVA)
cc uaa brain cc uaa brain cc uaa brain
mysql mysql mysql
Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2
go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr
loggre
gator
loggre
gator
loggre
gator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
PCF OrgPCF Space
App App
BOSH
Drives NSX-V
NSX-V (Edge - Load Balancing – Logical Switch – Firewall Services)
Platform as Code{}
• Declarative
• Change Controlled
• Archived
• Audited
Day 1 & Day 2
Cell_3 Cell_3 Cell_3
VMworld 2017 Content: Not fo
r publication or distri
bution
53
Platform
Operator
Ops Manager
(OVA) vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
BOSH
Drives NSX-V
Platform as Code{}
• Declarative
• Change Controlled
• Archived
• Audited
Day 1 & Day 2
NSXEdge LTM
NSXEdge LTM
• Repeat
– Scale
• Repair
– Recovery
• Repave
– Rotate Creds
VMworld 2017 Content: Not fo
r publication or distri
bution
54
Platform
Operator
CVE & Update Patching
The New Stack
• Patch at ANY Layer of the Application Stack
• Address CVE in minutes/hours versus days/weeks
• Simply re-stage all apps when any layer is patched
• Platform as Code{}
Day 1 & Day 2
Developer
PCF ERT Tile
PCF Stemcells
PCF OrgPCF Space
App App
CVE in Root File
System of Container
CVE Exec Layer: TC
Server
CVE on the Container
Host OS
Vulnerability in
Code{}
Restage Applications
PCF BuildPack
VMworld 2017 Content: Not fo
r publication or distri
bution
55
Platform
Operator
CVE & Update Patching
The New Stack
• Stemcells still there …
• Harbor Scans Images for Vulnerability (Clair)
• Address CVE in minutes/hours versus days/weeks
• Platform as Code{}
Developer
Stemcells
CVE in Root File
System of Container
CVE Exec Layer: TC
Server
CVE on the Container
Host OS
Vulnerability in
Code{}
Restage Applications
What about PKS?
Docker
Registry
CVE
FOUND
!!!
BOSH
VMworld 2017 Content: Not fo
r publication or distri
bution
What about PKS?
KUBO Can scale …. A lot
BOSH allows for a repeatablepattern of K8S Clusters as well.
• Many Development teams
• Multiple Security Zones for Applications
• Multi Cluster HA within a DC
• CI/CD Pattern similar to PCF
Platform
Operator
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
vSAN/NFS/VMFS
Resource Pool
ESX Cluster
VCF
BOSH
Developer
Developer
A
BvRA
PKS
VMworld 2017 Content: Not fo
r publication or distri
bution
Ops: Platform As Code{}
57
DEVELOPER-READY
INFRASTRUCTURE
Deliver innovation faster
to customers
Resource(s) Link(s)
Pivotal NSX + PCF Pipeline https://github.com/cf-platform-eng/nsx-ci-pipeline
Pivotal Generic PCF Install & Upgrade pipelines https://github.com/pivotal-cf/pcf-pipelines
Virtual Data Center
CVE & Update Patching
The New Stack
• Patch at ANY Layer of the Application Stack
• Address CVE in minutes/hours versus days/weeks
• Simply re-stage all apps when any layer is patched
• Platform as Code{}
Day 1 & Day 2
• Declarative
• Change Controlled
• Archived
• Audited
• Repeat
– Scale
• Repair
– Recovery
• Repave
– Rotate Creds
VMworld 2017 Content: Not fo
r publication or distri
bution
Wrapping It up …
VMworld 2017 Content: Not fo
r publication or distri
bution
Developer Ready Infrastructure
vSphere NSX vSAN
Pivotal Cloud Foundry
PCF
PKS
BOSH powered KubernetesB
OS
H
Platform
Operator
Developer
Wavefront
Self Service
• Automation
• Day 2 Operations
• Control
• Application Services or Container Services
• Application Logging & Monitoring
Solves for DevOps Reqs …
vRLI (Dev)
vRops
vRLI (Ops)
vRNI
VMworld 2017 Content: Not fo
r publication or distri
bution
60
VMworld US Key Focus Description
CNA1509BU DRI Developer-Ready Infrastructure from VMware & Pivotal
CNA1612BU PCF & KuboUse Cases: Deploying real-world workloads on Kubernetes and Pivotal Cloud
Foundry
CNA2006BU DRIDeep Dive: Architecting Container Services with VMware and Pivotal
Developer Ready Infrastructure
CNA2080BU Kubo Deep Dive: How to Deploy and Operationalize Kubernetes
CNA3429BU KuboBasics of Kubernetes on BOSH: Run Production-grade Kubernetes on the
SDDC
CNA3430BU PCFYour Enterprise Cloud-Native App Platform: An Introduction to Pivotal Cloud
Foundry
MGT2871BUPCF & vRops,
vRLI
Bridging the Operations Gap Between the Software-Defined Data Center
and Pivotal CF for VMware Deployments
NET1523BU PCF & NSX Integrating NSX and Cloud Foundry
PAR4411PU DRIEmerging Technologies with VMware and Pivotal - presented jointly by
VMware, Pivotal and Special Guest Speakers from Cognizant and WWT
Developer Ready Infrastructure @ VMworld
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution
VMworld 2017 Content: Not fo
r publication or distri
bution