Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Running Kubernetes on OpenStack and Bare Metal
OpenStack Summit Berlin, November 2018
Ramon Acedo Rodriguez
Product Manager, Red Hat OpenStack Team
@ramonacedo | [email protected]
Bare Metal On-Trend
Bare Metal On-Trend
OpenStack User Survey 2017
Among users who run Kubernetes on OpenStack, adoption of Ironic is even stronger with 37% relying on it.
OpenStack User Survey 2018
Popular Use Cases
Kubernetes on Bare Metal
High-Performance Computing
Direct Access to Dedicated Hardware Devices
Big Data and Scientific Applications
blog.openshift.com/kubernetes-on-metal-with-openshift
Bare Metal On-Trend
Why Kubernetes on OpenStackParticularly, on OpenStack Bare Metal
Why Kubernetes on OpenStack
Datacentre
WORKLOADDRIVEN
PROGRAMMATIC SCALE-OUT
ACROSS INFRASTRUCTURE
DEEPLYINTEGRATED
kubernetes
OpenStack Bare MetalIronic Introduction
OpenStack IronicHardware Lifecycle Management
Hardware InspectionServers and Network Switches (via LLDP)
OS ProvisioningSupporting qcow2 images
Routed Spine/Leaf NetworkingProvision over routed networks
Multi-TenancyML2 Networking Ansible plug-in
Node Auto-discovery
Broad BMC SupportRedfish, iDrac, iRMC, iLo, IPMI, oVirt, vBMC
OpenStack Ironic
Simple Architecture
Highly AvailableRun multiple Ironic instances in HA
Mixed VMs and Bare Metal InstancesSimply add Nova compute nodes
Register Bare Metal
Nodes
OpenStack Admin Workflow
Create Networks
Create Flavors
Upload Images
OpenStack Tenant Workflow
Select Network
Start VM Instances Start BM
Instances
Select OS and Flavor
OpenStack Bare MetalIronic and OpenStack Features
OpenStack Ironic Bare MetalIronic Multi-Tenant with Isolation Between Tenants
Dedicated Provider NetworksInstead of a shared flat network
Provisioning Over an Isolated, Dedicated Network
Physical Switch Ports Dynamically ConfiguredAt deployment time and on termination
Support for Neutron Port Groups and Security GroupsFor Link Aggregation and switch ACLs
L2 Switch
BM
NIC NIC
LAG
bond
Configured by ML2 plug-in
Configured by cloud-init using
metadata
L2 Switch
BM
NIC
VLANs set by by ML2 plug-in
BM
NIC
L2 Switch
Multi-Tenancy
https://docs.openstack.org/ironic/latest/admin/multitenancy.html
https://docs.openstack.org/ironic/latest/install/configure-tenant-networks.html
Port Groups / Bonds
https://docs.openstack.org/ironic/latest/admin/portgroups.html
Multi-tenant Bare Metal as a ServiceUpstream Docs
OpenStack Ironic Bare Metal ML2 Networking Ansible
Neutron ML2 Networking Ansible Driver
Multiple Switch Platforms in a Single ML2 DriverLeveraging the Networking Ansible modules
New in OpenStack Rocky
Provisioning Network is configured in the switch
Boot BM on Tenant
Network
ML2 Plug-in Configures
Switch
BM is Provisioned
ML2 Plug-in Configures
Switch
Tenant Network is configured in the switch
BM is ready
L2 Switch
BMNIC
BMNIC
blogs.rdoproject.org/2018/09/networking-ansible
spine switch
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
spine switch spine switch
L3 routed networks
ToR/leaf switch
Bare Metal
Ironic Node
Ironic Node
Ironic Node
Bare Metal
ToR/leaf switch ToR/leaf switch
DHCP Relay DHCP Relay DHCP Relay
L3 routed networks
OpenStack Ironic Bare Metal L3 Routed Networks (Spine/Leaf Network Topologies)
L3 Spine and Leaf TopologiesIronic provisioning bare metal nodes over routed networks
DHCP RelayAllowing PXE booting over L3 routed networks
OpenStack Bare MetalIronic Inspector Nodes Auto-Discovery
Use Rules to Set Node Properties E.g. set Ironic driver (iDrac, Redfish…) based on inspection data, set BMC credentials, etc.
Just Power On the NodesNodes PXE boot from the provisioning network used by Ironic
Automatic Node InspectionNodes boot from the network and their hardware is inspected
Automatically Registered with IronicAfter inspection they are registered with Ironic and ready to be deployed
cat > rules.json << EOF[ { "description": "Set the vendor driver for Dell hardware", "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true}, {"op": "eq", "field": "data://inventory.system_vendor.manufacturer", "value": "Dell Inc."} ], "actions": [ {"action": "set-attribute", "path": "driver", "value": "idrac"}, {"action": "set-attribute", "path": "driver_info/drac_username", "value": "root"}, {"action": "set-attribute", "path": "driver_info/drac_password", "value": "calvin"}, {"action": "set-attribute", "path": "driver_info/drac_address", "value": "{data[inventory][bmc_address]}"} ] }]EOF
$ openstack baremetal introspection rule import rules.json
Data collected during inspection
E.g: Use the the idrac driver and its credentials if a Dell node is detected
OpenStack Bare MetalRedfish Support in Ironic
API-driven Remote Management PlatformManage large amounts of physical nodes via API. redfish.dmtf.org
Included in Modern BMCsMost vendors support Redfish in the latest models
Supported in IronicIntroduced in Pike along with the Sushy library
OpenStack Stain AdditionOut-of-band inspection of nodes, boot from virtual media (without DHCP) and BIOS configurations
openstack baremetal node create \ --driver redfish \ --driver-info redfish_address=https://example.com \ --driver-info redfish_system_id=/redfish/v1/Systems/CX34R87 \ --driver-info redfish_username=admin \ --driver-info redfish_password=password
Get and Set BIOS SettingsRetrieve and apply BIOS settings via CLI or REST API. The desired BIOS settings are applied during manual cleaning.
Settings Applied During Node CleaningThe desired BIOS settings are applied during manual cleaning
OpenStack Bare MetalIronic BIOS Configuration docs.openstack.org/ironic/latest/admin/bios.html
[{ "name": "hyper_threading_enabled”, "value": "False" }, { "name": "cpu_vt_enabled", "value": "True" }]
Central Site
Ironic Conductor
Bare Metal
Bare Metal
Site B
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
...
Ironic Conductor
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Bare Metal
Site D
Ironic Conductor
Bare Metal
Bare Metal
Bare Metal
Site C
Ironic Controller
Ironic Controller
Ironic Controller
Site A
OpenStack Bare MetalMulti-Site
Ironic Conductor and Node Grouping AffinityUsing the conductor/node grouping affinity spec
Each Ironic Conductor Manages a Group of NodesNo need to expose access to BMC (e.g. IPMI. Redfish, iDrac, iRMC) to the central site
PXE boot or Virtual Media ProvisioningWe will be able to boot nodes without DHCP (see spec Ironic L3 based deployment)
Kubernetes on OpenStack and Bare MetalDeployment of Kubernetes on the metal
Kubernetes Cluster
Kubernetes on Bare MetalDeploy Kubernetes on OpenStack Ironic-managed bare metal nodes
Kubernetes Installer
Master Node
Infra Node
Worker Node
Deploy Kubernetes
OpenStack with IronicOpenStack Installer 1
2
3
Deploy OpenStack with
Ironic
docs.openshift.com/container-platform/3.11/getting_started/install_openshift.htmlWorkflow to Install an OpenShift Cluster on Bare Metal
Kubernetes with OpenShift
Provision Bare Metal NodesIronic provisions the OS image and configures the network
Add DNS EntriesWildcard DNS for container apps and fully-qualified names for the nodes
Distribute SSH keysCluster nodes need to access each other passwordless
Install with the OpenShift Ansible InstallerInstall the openshift-ansibe installer on an admin node and point it to the bare metal nodes
DNS entries with wildcard for apps
Cluster Installation
TripleO-deployed Kubernetes ClusterOpenShift to the Rescue
Kubernetes Cluster
TripleO Node integrates openshift-ansible
Master Nodes
Infra Nodes
Worker Nodes
Deploy an OpenShift/OKD
cluster and a GlusterFS on bare
metal nodes
Kubernetes on Bare MetalProvision nodes and deploy Kubernetes with Ironic in TripleONew in Rocky!
[stack@undercloud-0 ~]$ cat /home/stack/home/stack/openshift_env.yaml[...] OS::TripleO::OpenShiftMaster::Net::SoftwareConfig: /home/stack/master-nic.yaml OS::TripleO::OpenShiftWorker::Net::SoftwareConfig: /home/stack/worker-nic.yaml OS::TripleO::OpenShiftInfra::Net::SoftwareConfig: /home/stack/infra-nic.yaml[...] OpenShiftMasterCount: 3 OpenShiftWorkerCount: 3 OpenShiftInfraCount: 3[...] OpenShiftInfraParameters: OpenShiftGlusterDisks: - /dev/sdb[...]
Kubernetes on Bare MetalProvision nodes and deploy Kubernetes with Ironic in TripleO
Create OpenShift RolesMaster, Workers and Infra nodes in TripleO
Configure the Network Settings in TripleOE.g. Internal, External and Storage networks and the NIC configuration for each node
Set OpenShift and GlusterFS OptionsE.g. Number of nodes, disk for Gluster
Deploy with TripleORun the usual ‘openstack overcloud deploy’ command
[stack@undercloud-0 ~]$ cat overcloud_deploy.sh openstack overcloud deploy \--stack openshift \--templates \-r /home/stack/openshift_roles_data.yaml \-n /home/stack/network_data.yaml \-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \-e /usr/share/openstack-tripleo-heat-templates/environments/openshift.yaml \-e /usr/share/openstack-tripleo-heat-templates/environments/openshift-cns.yaml \-e /home/stack/openshift_env.yaml \-e /home/stack/containers-prepare-parameter.yaml
Kubernetes and TripleO Integrationhttps://github.com/openstack/tripleo-heat-templates
Container Storage Options for Bare MetalGlusterFS, Manila/CephFS, NFS
Container Storage Options for Bare Metal
GlusterFS
NFS/Manila (CephFS)
Storage Should be Highly AvailableGlusterFS and CephFS provide HA
Storage Should Allow RWX ModeAllowing ReadWriteMany is required by some apps. GlusterFS and CephFS are supported backends for RWX access mode
Local
HostPath
Container Storage Options for Bare MetalGlusterFS
Kubernetes Cluster on Bare Metal with Converged GlusterFS Storage
Master Node
Infra Node
Master Node Master Node
Infra Node Infra Node
Worker Node Worker Node Worker Node
InfraGlusterFS
Cluster
AppsGlusterFS
Cluster
OpenStack Storage Not RequiredWe deploy with OpenStack (TripleO) but Kubernetes don’t use OpenStack
TripleO Deploys GlusterFS on Bare MetalOptionally, we can request TripleO to deploy GlusterFS for the OpenShift cluster
GlusterFS Can Be Hosted On the Infra and Worker NodesThe GlusterFS Cluster can be hosted in “converged” mode along with the Infra and Worker nodes
Container Storage Options for Bare MetalManila with CephFS/NFS
Manila Provides RWX AccessPVs can be created with ReadWriteMany (RWX) access mode
Ceph as a Single Storage Backend Manila is backed by CephFS/NFS allowing to use Ceph for OpenStack and OpenShift workloads and infra
Kubernetes Registry on Object Storage from CephCeph RadosGW configured with OpenStack for Object Storage can be used for the registry
Kubernetes Cluster on Bare Metal Consuming Storage from OpenStack Manila Backed by Ceph
Bare Metal Kubernetes
OpenStack IronicManila
Bare Metal Kubernetes
Bare Metal Kubernetes
Ceph Storage Ceph Storage Ceph StorageCeph
Cluster
OpenStack IronicManila
OpenStack IronicManila
Networking on Bare MetalOpenShift Networking Architecture
Kubernetes Cluster on Bare Metal
OpenStack Cluster
Cluster Networking with Bare MetalMore info at docs.openshift.com/container-platform/3.11/architecture/networking/sdn.html
Master Node
Infra Node
Master Node
Master Node
Infra Node Infra Node
Worker Node
Worker Node
Worker Node
Ironic Controller
Ironic Controller
Ironic Controller
Provisioning Network
Data Network
Public Network
Provisioning Network
Data Network
Public Network
Provisioning Network
Data Network
Public Network
Load Balancers
VXLAN (Container to Container)
BMC (IPMI/Redfish/iDrac, etc.)
BMC NetworkIronic manages the servers via their BMC (IPMI, Redfish, iDrac, iLO, iRMC, etc.)
Provisioning NetworkWhen deploying from Ironic, a NIC is used to DHCP/PXE-boot. This is usually a single NIC (or one NIC from a bond with LACP fallback)
Data NetworkPod to pod traffic goes through the data network. A 2-NIC bond is recommended
Open vSwitch and CNIOVS is used for traffic flow within the cluster (pod-to-pod, and node-to-node) and ingress/egress traffic to the cluster. OVS is used as the Container Network Interface (CNI) plug-in for Kubernetes
Thank You
Ramon Acedo Rodriguez
Product Manager, Red Hat OpenStack Team
@ramonacedo | [email protected]