Upload
others
View
15
Download
0
Embed Size (px)
Citation preview
1
Run Kubernetes on OpenStack and Bare Metal
Fast
Ramon Acedo RodriguezSenior Principal Product Manager, Red Hat
OPEN
INFRASTRUCTURE SUM
MIT | SHAN
GHAI, NOVEM
BER 4-6 2019
2
OPTIO
NAL SECTION
MARKER O
R TITLE
Open Hybrid CloudVision
3 BARE METAL VIRTUAL PRIVATE CLOUD
The Open Hybrid CloudThe 4 Footprints
PHYSICAL VIRTUALPRIVATECLOUD
PUBLICCLOUD
APP APP APP APP
Goal: Give developers the freedom to innovate faster across on-premises and public clouds
4 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
PHYSICAL
APP APP APP APP
VIRTUALPUBLICCLOUD
VM CC C
On-premises
Kubernetes-powered Open Hybrid Cloud
5 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
VIRTUAL
Containers | VMs | Serverless Apps
BARE METAL PRIVATE CLOUD PUBLIC CLOUDS
Kubernetes on OpenStack is focused on the private cloud
6
Kubernetes on OpenStack
Introduction
7 VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
DATACENTRE
WORKLOADDRIVEN
PROGRAMMATIC API DRIVEN
ACROSS INFRASTRUCTURE
DEEPLYINTEGRATED
Why Kubernetes on OpenStack?Open, scalable, managed, workloads.
SCALE OUT
SOLID FOUNDATION
8
Kubernetes on OpenStack Integrations
Kubernetes and OpenStack Are Complementary
Consumption of resourcesProvides the container platform layer
Exposition of resourcesProvides the infrastructure layer
complementary
Kubernetes on OpenStack Key Integration Points
Kubernetes on OpenStack Integration Example: Cinder
OpenShift on OpenStack Logical Architecture
13
Installation RequirementsOpenShift on OpenStack
Requirements in your OpenStack Platform
Red Hat OpenStack Platform 13 (Queens)
3 Master nodes● At least 16 GB RAM, 4 vCPUs and 25 GB Disk
At least 2 worker nodes● A least 8 GB RAM, 2 vCPUs and 25 GB Disk
Object Storage (Swift)
CoreOS image
OpenStack Resources:● Floating IPs: 2● Security Groups: 3● Security Group Rules: 60● Routers: 1● Subnets: 1● RAM: 112 GB● vCPUs: 28● Volume Storage: 175 GB● Instances: 7● Swift containers: 2● Swift objects: 1● Available space in Swift: at least 10 MB
https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md
15
OpenShift Internal Load Balancing and DNS
DNS
● CoreDNS and mDNS run on all nodes
Internal DNS and Load Balancing
Load Balancing
● HAProxy + keepalived on master nodes
● Use of keepalived VIPs for:
○ Internal API○ Ingress traffic to workloads○ Internal DNS requests
The openshift-installer will configure internal Load Balancing & DNS
https://github.com/openshift/installer/blob/master/docs/design/openstack/networking-infrastructure.md
17
Kuryr-Kubernetes SDNSDN for OpenShift on OpenStack
Kuryr improves the network performance of pods when running on OpenStack.
SDN solution using Kubernetes Container Network Interface (CNI) and OpenStack Neutron.
Provides interconnectivity between Kubernetes pods and OpenStack virtual instances
Kuryr-Kubernetes Logo
Kuryr-Kubernetes CNI
18
Recommended when
● Neutron tenant networks are used for OpenShift Application Nodes
Not recommended when
● Using only OpenStack VLAN-based provider networks (Kuryr not needed)
● Many OpenShift services and few hypervisors:○ For each OpenShift service, Octavia will run a VM with a
load balancer in it
When to use Kuryr
19
Requires
● OpenStack Queens or newer● Octavia Load Balancer● Neutron Trunk Ports
Kuryr Internal Architecture
20
Kuryr integrates with Neutron and Octavia
Pods are connected via Kuryr CNI to Neutron
Kuryr creates Load Balancers with Octavia for the Kubernetes services
21
Bandwidth tests between pods on the same hypervisor show slightly better performance
Kuryr Performance ComparisonPerformance test done with OpenShift 3.11 / Kubernetes 1.11 and OpenStack Queens
22
Bandwidth tests between pods on different hypervisors show up to 9x performance improvement
Kuryr Performance ComparisonPerformance test done with OpenShift 3.11 / Kubernetes 1.11 and OpenStack Queens
23
Kuryr Performance Comparison Blog PostPerformance test done with OpenShift 3.11 and OpenStack Queens
https://blog.openshift.com/accelerate-your-openshift-network-performance-on-openstack-with-kuryr
24
Reference Architecture
OpenShift on OpenStack Reference ArchitectureCurrent: OSP 13 LTS and OCP 3.11 with Kuryr | Next: OSP 13 LTS and OCP 4.3 with Kuryr
25 https://access.redhat.com/documentation/en-us/reference_architectures/2019/html-single/deploying_red_hat_openshift_container_platform_3.11_on_red_hat_openstack_platform_13
26
Ways to Install Kubernetes on
OpenStackIntroduction
27 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
VIRTUALBARE METAL PRIVATE CLOUD
Full Stack AutomationSimplified, opinionated
Installer-provisioned OpenStack resources (IPI)
Pre-existing InfrastructureUser-provisioned
OpenStack resources (UPI)
Kubernetes Installation with OpenShift 4 on OpenStackInstallation Experiences
28 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
VIRTUALBARE METAL PRIVATE CLOUD PUBLIC CLOUDS
Installer Provisions:
Networks
Internal Load Balancers1
Internal DNS1
OpenStack Instances
Red Hat CoreOS
CoreOS Ignition Configs
OpenShift Nodes
OpenShift Cluster Resources
Full Stack Automation InstallationDeploying OpenShift on OpenStack with installer-provisioned infrastructure (IPI)
Red Hat OpenStack Platform
Overcloud
OpenShift Cluster
. . .
openshift-install
1External Load Balancers (routing) and external DNS servers are provided by the user
29 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
VIRTUALBARE METAL
User Provisions:
Networks
Internal Load Balancers1
Internal DNS1
OpenStack Instances
Red Hat CoreOS
CoreOS Ignition Configs
OpenShift Nodes
OpenShift Cluster Resources
Installing on Pre-existing InfrastructureDeploying OpenShift on OpenStack with user-provisioned infrastructure (UPI)
openshift-install
User provisioned resources
Red Hat OpenStack Platform
Overcloud
OpenShift Cluster
. . .
30 VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
Installing on OpenStack DocumentationUser-provisioned Infrastructure | Installer-provisioned Infrastructure
USER PROVISIONED INFRASTRUCTURE
INSTRALLER- PROVISIONED INFRASTRUCTURE
https://github.com/openshift/installer/blob/master/docs/user/openstack/README.mdhttps://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md
OpenShift 4.2 on OpenStack
31 try.openshift.com
32
OpenStack Bare MetalIronic Introduction
Bare Metal On-Trend
OpenStack User Survey 2017
Among users who run Kubernetes on OpenStack, adoption of Ironic is even stronger with 37% relying on it.
OpenStack User Survey 2018
Popular Use Cases
Kubernetes on Bare Metal
High-Performance Computing
Direct Access to Dedicated Hardware Devices
Big Data and Scientific Applications
Bare Metal On-Trend
OpenStack Bare Metal Service - IronicHardware Lifecycle Management
Hardware InspectionServers and Network Switches (via LLDP)
OS Image ProvisioningSupporting qcow2 images
Routed Spine/Leaf NetworkingProvisioning over routed networks
Multi-TenancyWith network isolation between tenants
Node Auto-discovery
Broad Power Management SupportRedfish, iDrac, iRMC, iLo, IPMI, oVirt, vBMC
36
OpenStack Bare MetalFeatures
Tenant-Dedicated NetworksInstead of a shared flat network
Provisioning Over an Isolated Network
Switch Ports Dynamically ConfiguredAt deployment time and on termination
Link Aggregation L2 Switch
BM
NIC NIC
LAG
bond
Configured by ML2 plug-in
Configured by cloud-init using
metadata
L2 Switch
BM
NIC
VLANs set by by ML2 plug-in
BM
NIC
L2 Switch
Multi-Tenant Support with Isolation Between TenantsAvailable from OpenStack Queens
Multi-Tenant Support with Isolation Between TenantsAvailable from OpenStack Queens
Neutron ML2 Networking-Ansible Driver
Multiple Switch Platforms in a Single ML2 DriverLeveraging the Networking Ansible modules
Available in OpenStack Queens | Red Hat OpenStack Platform 13OSP 13 Long Life Support
Provisioning Network is configured in the switch
Boot BM on Tenant
Network
ML2 Plug-in Configures
Switch
BM is Provisioned
ML2 Plug-in Configures
Switch
Tenant Network is configured in the switch
BM is ready
L2 Switch
BMNIC
BMNIC
spine switch
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
spine switch spine switch
L3 routed networks
ToR/leaf switch
Bare Metal
Ironic Node
Ironic Node
Ironic Node
Bare Metal
ToR/leaf switch ToR/leaf switch
DHCP Relay DHCP Relay DHCP Relay
L3 routed networks
L3 Spine and Leaf TopologiesIronic provisioning bare metal nodes over routed networks
DHCP RelayAllowing PXE booting over L3 routed networks
L3 Routed Networks (Spine/Leaf Network Topologies)Available from OpenStack Queens IPv6 Support being added to OpenStack Train and above
BIOS Configurationsdocs.openstack.org/ironic/latest/admin/bios.html
Get and Set BIOS SettingsRetrieve and apply BIOS settings via CLI or REST API. The desired BIOS settings are applied during manual cleaning.
Settings Applied During Node CleaningThe desired BIOS settings are applied during manual cleaning
[{ "name": "hyper_threading_enabled”, "value": "False" }, { "name": "cpu_vt_enabled", "value": "True" }]
Ironic Inspector Nodes Auto-Discovery
Just Power On the NodesNodes PXE boot from the provisioning network used by Ironic
Automatic Node InspectionNodes boot from the network and their hardware is inspected
Nodes Automatically Registered with IronicAfter inspection they are registered with Ironic and ready to be deployed
Use Rules to Set Node Properties E.g. set Ironic driver (iDrac, Redfish…) based on inspection data, set BMC credentials, etc.
cat > rules.json << EOF[ { "description": "Set the vendor driver for Dell hardware", "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true}, {"op": "eq", "field": "data://inventory.system_vendor.manufacturer", "value": "Dell Inc."} ], "actions": [ {"action": "set-attribute", "path": "driver", "value": "idrac"}, {"action": "set-attribute", "path": "driver_info/drac_username", "value": "root"}, {"action": "set-attribute", "path": "driver_info/drac_password", "value": "calvin"}, {"action": "set-attribute", "path": "driver_info/drac_address", "value": "{data[inventory][bmc_address]}"} ] }]EOF
$ openstack baremetal introspection rule import rules.json
Data collected during inspection
E.g: Use the the idrac driver and its credentials if a Dell node is detected
Redfish Support in Ironic
API-driven Remote Management PlatformManage large amounts of physical nodes via API. redfish.dmtf.org
Included in Modern BMCsMost vendors support Redfish in the latest models
Supported in IronicIntroduced in OpenStack Pike, along with the Sushy library
Improvements in OpenStack Train and BeyondOut-of-band inspection of nodes, boot from virtual media (without DHCP) and BIOS configurations
openstack baremetal node create \ --driver redfish \ --driver-info redfish_address=https://example.com \ --driver-info redfish_system_id=/redfish/v1/Systems/CX34R87 \ --driver-info redfish_username=admin \ --driver-info redfish_password=password
43
OpenShift on OpenStack-Managed
Bare MetalDeployment of Kubernetes on the metal
Kubernetes Cluster
Kubernetes on Bare MetalDeploy Kubernetes on OpenStack-managed bare metal nodes
OpenShift Installer
Master Node
Worker Node
Worker Node
Deploy OpenShift
OpenStack with Ironic
https://docs.openshift.com/container-platform/4.2/installing/installing_bare_metal/installing-bare-metal.html
45
Metal3Deployment of Kubernetes on the metal
46 BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
VIRTUAL
Containers | Serverless Apps
BARE METAL PRIVATE CLOUD PUBLIC CLOUDS
Metal3 is focused on Kubernetes on bare metal
Metal3
Metal Kubed, bare metal host provisioning for Kubernetes
Metal3 Kubernetes Operators FrameworkIronic
Metal3 enables bare metal host management with Kubernetes.
48
Metal3 runs on Kubernetes.
And is managed through Kubernetes interfaces.
Metal3
Metal Kubed, bare metal host provisioning for Kubernetes
49
Actuator allows Kubernetes to get Machines the same way it would in a public cloud, using the Kubernetes cluster-api
Operator uses Ironic behind the scenes to manage the physical hardware represented as BareMetalHost objects.
MACHINE CONTROLLER
BARE-METAL ACTUATOR
BARE METAL OPERATOR
Bare Metal Management PodsBare Metal Management PodsBARE-METAL MANAGEMENT PODS
Metal3
Metal Kubed, bare metal host provisioning for Kubernetes
Ironic
50
Try it:
http://metal3.io/try-it.html
Install OpenShift using it:
https://github.com/openshift/installer/blob/master/docs/user/metal/install_ipi.md
Metal3
Metal Kubed, bare metal host provisioning for Kubernetes
51
SummaryKubernetes on OpenStack
52 VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS
Run Kubernetes on OpenStack and Bare Metal Fast
Install Kubernetes on OpenStackhttps://github.com/openshift/installer/blob/master/docs/user/openstack/README.mdhttps://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md
Kuryr performancehttps://blog.openshift.com/accelerate-your-openshift-network-performance-on-openstack-with-kuryr
Reference Architecturehttps://access.redhat.com/documentation/en-us/reference_architectures/2019/html-single/deploying_red_hat_openshift_container_platform_3.11_on_red_hat_openstack_platform_13
Ironic Bare Metalhttps://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/bare_metal_provisioning/index
Metal3 metal3.io
53
Red Hat is the world’s leading provider of enterprise
open source software solutions. Award-winning
support, training, and consulting services make Red
Hat a trusted adviser to the Fortune 500.
Thank you