34
Running Kubernetes on OpenStack and Bare Metal OpenStack Summit Berlin, November 2018 Ramon Acedo Rodriguez Product Manager, Red Hat OpenStack Team @ramonacedo | [email protected]

OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

  • Upload
    others

  • View
    9

  • Download
    0

Embed Size (px)

Citation preview

Page 1: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Running Kubernetes on OpenStack and Bare Metal

OpenStack Summit Berlin, November 2018

Ramon Acedo Rodriguez

Product Manager, Red Hat OpenStack Team

@ramonacedo | [email protected]

Page 2: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Bare Metal On-Trend

Page 3: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Bare Metal On-Trend

OpenStack User Survey 2017

Among users who run Kubernetes on OpenStack, adoption of Ironic is even stronger with 37% relying on it.

OpenStack User Survey 2018

Page 4: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Popular Use Cases

Kubernetes on Bare Metal

High-Performance Computing

Direct Access to Dedicated Hardware Devices

Big Data and Scientific Applications

blog.openshift.com/kubernetes-on-metal-with-openshift

Bare Metal On-Trend

Page 5: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Why Kubernetes on OpenStackParticularly, on OpenStack Bare Metal

Page 6: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Why Kubernetes on OpenStack

Datacentre

WORKLOADDRIVEN

PROGRAMMATIC SCALE-OUT

ACROSS INFRASTRUCTURE

DEEPLYINTEGRATED

kubernetes

Page 7: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Bare MetalIronic Introduction

Page 8: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack IronicHardware Lifecycle Management

Hardware InspectionServers and Network Switches (via LLDP)

OS ProvisioningSupporting qcow2 images

Routed Spine/Leaf NetworkingProvision over routed networks

Multi-TenancyML2 Networking Ansible plug-in

Node Auto-discovery

Broad BMC SupportRedfish, iDrac, iRMC, iLo, IPMI, oVirt, vBMC

Page 9: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Ironic

Simple Architecture

Highly AvailableRun multiple Ironic instances in HA

Mixed VMs and Bare Metal InstancesSimply add Nova compute nodes

Page 10: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Register Bare Metal

Nodes

OpenStack Admin Workflow

Create Networks

Create Flavors

Upload Images

Page 11: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Tenant Workflow

Select Network

Start VM Instances Start BM

Instances

Select OS and Flavor

Page 12: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Bare MetalIronic and OpenStack Features

Page 13: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Ironic Bare MetalIronic Multi-Tenant with Isolation Between Tenants

Dedicated Provider NetworksInstead of a shared flat network

Provisioning Over an Isolated, Dedicated Network

Physical Switch Ports Dynamically ConfiguredAt deployment time and on termination

Support for Neutron Port Groups and Security GroupsFor Link Aggregation and switch ACLs

L2 Switch

BM

NIC NIC

LAG

bond

Configured by ML2 plug-in

Configured by cloud-init using

metadata

L2 Switch

BM

NIC

VLANs set by by ML2 plug-in

BM

NIC

L2 Switch

Page 14: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Multi-Tenancy

https://docs.openstack.org/ironic/latest/admin/multitenancy.html

https://docs.openstack.org/ironic/latest/install/configure-tenant-networks.html

Port Groups / Bonds

https://docs.openstack.org/ironic/latest/admin/portgroups.html

Multi-tenant Bare Metal as a ServiceUpstream Docs

Page 15: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Ironic Bare Metal ML2 Networking Ansible

Neutron ML2 Networking Ansible Driver

Multiple Switch Platforms in a Single ML2 DriverLeveraging the Networking Ansible modules

New in OpenStack Rocky

Provisioning Network is configured in the switch

Boot BM on Tenant

Network

ML2 Plug-in Configures

Switch

BM is Provisioned

ML2 Plug-in Configures

Switch

Tenant Network is configured in the switch

BM is ready

L2 Switch

BMNIC

BMNIC

blogs.rdoproject.org/2018/09/networking-ansible

Page 16: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

spine switch

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

spine switch spine switch

L3 routed networks

ToR/leaf switch

Bare Metal

Ironic Node

Ironic Node

Ironic Node

Bare Metal

ToR/leaf switch ToR/leaf switch

DHCP Relay DHCP Relay DHCP Relay

L3 routed networks

OpenStack Ironic Bare Metal L3 Routed Networks (Spine/Leaf Network Topologies)

L3 Spine and Leaf TopologiesIronic provisioning bare metal nodes over routed networks

DHCP RelayAllowing PXE booting over L3 routed networks

Page 17: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Bare MetalIronic Inspector Nodes Auto-Discovery

Use Rules to Set Node Properties E.g. set Ironic driver (iDrac, Redfish…) based on inspection data, set BMC credentials, etc.

Just Power On the NodesNodes PXE boot from the provisioning network used by Ironic

Automatic Node InspectionNodes boot from the network and their hardware is inspected

Automatically Registered with IronicAfter inspection they are registered with Ironic and ready to be deployed

cat > rules.json << EOF[ { "description": "Set the vendor driver for Dell hardware", "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true}, {"op": "eq", "field": "data://inventory.system_vendor.manufacturer", "value": "Dell Inc."} ], "actions": [ {"action": "set-attribute", "path": "driver", "value": "idrac"}, {"action": "set-attribute", "path": "driver_info/drac_username", "value": "root"}, {"action": "set-attribute", "path": "driver_info/drac_password", "value": "calvin"}, {"action": "set-attribute", "path": "driver_info/drac_address", "value": "{data[inventory][bmc_address]}"} ] }]EOF

$ openstack baremetal introspection rule import rules.json

Data collected during inspection

E.g: Use the the idrac driver and its credentials if a Dell node is detected

Page 18: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

OpenStack Bare MetalRedfish Support in Ironic

API-driven Remote Management PlatformManage large amounts of physical nodes via API. redfish.dmtf.org

Included in Modern BMCsMost vendors support Redfish in the latest models

Supported in IronicIntroduced in Pike along with the Sushy library

OpenStack Stain AdditionOut-of-band inspection of nodes, boot from virtual media (without DHCP) and BIOS configurations

openstack baremetal node create \ --driver redfish \ --driver-info redfish_address=https://example.com \ --driver-info redfish_system_id=/redfish/v1/Systems/CX34R87 \ --driver-info redfish_username=admin \ --driver-info redfish_password=password

Page 19: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Get and Set BIOS SettingsRetrieve and apply BIOS settings via CLI or REST API. The desired BIOS settings are applied during manual cleaning.

Settings Applied During Node CleaningThe desired BIOS settings are applied during manual cleaning

OpenStack Bare MetalIronic BIOS Configuration docs.openstack.org/ironic/latest/admin/bios.html

[{ "name": "hyper_threading_enabled”, "value": "False" }, { "name": "cpu_vt_enabled", "value": "True" }]

Page 20: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Central Site

Ironic Conductor

Bare Metal

Bare Metal

Site B

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

...

Ironic Conductor

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Bare Metal

Site D

Ironic Conductor

Bare Metal

Bare Metal

Bare Metal

Site C

Ironic Controller

Ironic Controller

Ironic Controller

Site A

OpenStack Bare MetalMulti-Site

Ironic Conductor and Node Grouping AffinityUsing the conductor/node grouping affinity spec

Each Ironic Conductor Manages a Group of NodesNo need to expose access to BMC (e.g. IPMI. Redfish, iDrac, iRMC) to the central site

PXE boot or Virtual Media ProvisioningWe will be able to boot nodes without DHCP (see spec Ironic L3 based deployment)

Page 21: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Kubernetes on OpenStack and Bare MetalDeployment of Kubernetes on the metal

Page 22: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Kubernetes Cluster

Kubernetes on Bare MetalDeploy Kubernetes on OpenStack Ironic-managed bare metal nodes

Kubernetes Installer

Master Node

Infra Node

Worker Node

Deploy Kubernetes

OpenStack with IronicOpenStack Installer 1

2

3

Deploy OpenStack with

Ironic

Page 23: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

docs.openshift.com/container-platform/3.11/getting_started/install_openshift.htmlWorkflow to Install an OpenShift Cluster on Bare Metal

Kubernetes with OpenShift

Provision Bare Metal NodesIronic provisions the OS image and configures the network

Add DNS EntriesWildcard DNS for container apps and fully-qualified names for the nodes

Distribute SSH keysCluster nodes need to access each other passwordless

Install with the OpenShift Ansible InstallerInstall the openshift-ansibe installer on an admin node and point it to the bare metal nodes

DNS entries with wildcard for apps

Cluster Installation

Page 24: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

TripleO-deployed Kubernetes ClusterOpenShift to the Rescue

Page 25: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Kubernetes Cluster

TripleO Node integrates openshift-ansible

Master Nodes

Infra Nodes

Worker Nodes

Deploy an OpenShift/OKD

cluster and a GlusterFS on bare

metal nodes

Kubernetes on Bare MetalProvision nodes and deploy Kubernetes with Ironic in TripleONew in Rocky!

Page 26: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

[stack@undercloud-0 ~]$ cat /home/stack/home/stack/openshift_env.yaml[...] OS::TripleO::OpenShiftMaster::Net::SoftwareConfig: /home/stack/master-nic.yaml OS::TripleO::OpenShiftWorker::Net::SoftwareConfig: /home/stack/worker-nic.yaml OS::TripleO::OpenShiftInfra::Net::SoftwareConfig: /home/stack/infra-nic.yaml[...] OpenShiftMasterCount: 3 OpenShiftWorkerCount: 3 OpenShiftInfraCount: 3[...] OpenShiftInfraParameters: OpenShiftGlusterDisks: - /dev/sdb[...]

Kubernetes on Bare MetalProvision nodes and deploy Kubernetes with Ironic in TripleO

Create OpenShift RolesMaster, Workers and Infra nodes in TripleO

Configure the Network Settings in TripleOE.g. Internal, External and Storage networks and the NIC configuration for each node

Set OpenShift and GlusterFS OptionsE.g. Number of nodes, disk for Gluster

Deploy with TripleORun the usual ‘openstack overcloud deploy’ command

[stack@undercloud-0 ~]$ cat overcloud_deploy.sh openstack overcloud deploy \--stack openshift \--templates \-r /home/stack/openshift_roles_data.yaml \-n /home/stack/network_data.yaml \-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \-e /usr/share/openstack-tripleo-heat-templates/environments/openshift.yaml \-e /usr/share/openstack-tripleo-heat-templates/environments/openshift-cns.yaml \-e /home/stack/openshift_env.yaml \-e /home/stack/containers-prepare-parameter.yaml

Page 27: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Kubernetes and TripleO Integrationhttps://github.com/openstack/tripleo-heat-templates

Page 28: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Container Storage Options for Bare MetalGlusterFS, Manila/CephFS, NFS

Page 29: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Container Storage Options for Bare Metal

GlusterFS

NFS/Manila (CephFS)

Storage Should be Highly AvailableGlusterFS and CephFS provide HA

Storage Should Allow RWX ModeAllowing ReadWriteMany is required by some apps. GlusterFS and CephFS are supported backends for RWX access mode

Local

HostPath

Page 30: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Container Storage Options for Bare MetalGlusterFS

Kubernetes Cluster on Bare Metal with Converged GlusterFS Storage

Master Node

Infra Node

Master Node Master Node

Infra Node Infra Node

Worker Node Worker Node Worker Node

InfraGlusterFS

Cluster

AppsGlusterFS

Cluster

OpenStack Storage Not RequiredWe deploy with OpenStack (TripleO) but Kubernetes don’t use OpenStack

TripleO Deploys GlusterFS on Bare MetalOptionally, we can request TripleO to deploy GlusterFS for the OpenShift cluster

GlusterFS Can Be Hosted On the Infra and Worker NodesThe GlusterFS Cluster can be hosted in “converged” mode along with the Infra and Worker nodes

Page 31: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Container Storage Options for Bare MetalManila with CephFS/NFS

Manila Provides RWX AccessPVs can be created with ReadWriteMany (RWX) access mode

Ceph as a Single Storage Backend Manila is backed by CephFS/NFS allowing to use Ceph for OpenStack and OpenShift workloads and infra

Kubernetes Registry on Object Storage from CephCeph RadosGW configured with OpenStack for Object Storage can be used for the registry

Kubernetes Cluster on Bare Metal Consuming Storage from OpenStack Manila Backed by Ceph

Bare Metal Kubernetes

OpenStack IronicManila

Bare Metal Kubernetes

Bare Metal Kubernetes

Ceph Storage Ceph Storage Ceph StorageCeph

Cluster

OpenStack IronicManila

OpenStack IronicManila

Page 32: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Networking on Bare MetalOpenShift Networking Architecture

Page 33: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Kubernetes Cluster on Bare Metal

OpenStack Cluster

Cluster Networking with Bare MetalMore info at docs.openshift.com/container-platform/3.11/architecture/networking/sdn.html

Master Node

Infra Node

Master Node

Master Node

Infra Node Infra Node

Worker Node

Worker Node

Worker Node

Ironic Controller

Ironic Controller

Ironic Controller

Provisioning Network

Data Network

Public Network

Provisioning Network

Data Network

Public Network

Provisioning Network

Data Network

Public Network

Load Balancers

VXLAN (Container to Container)

BMC (IPMI/Redfish/iDrac, etc.)

BMC NetworkIronic manages the servers via their BMC (IPMI, Redfish, iDrac, iLO, iRMC, etc.)

Provisioning NetworkWhen deploying from Ironic, a NIC is used to DHCP/PXE-boot. This is usually a single NIC (or one NIC from a bond with LACP fallback)

Data NetworkPod to pod traffic goes through the data network. A 2-NIC bond is recommended

Open vSwitch and CNIOVS is used for traffic flow within the cluster (pod-to-pod, and node-to-node) and ingress/egress traffic to the cluster. OVS is used as the Container Network Interface (CNI) plug-in for Kubernetes

Page 34: OpenStack Summit Berlin, November 2018 Bare Metal Running ...€¦ · GlusterFS Cluster Apps GlusterFS Cluster OpenStack Storage Not Required We deploy with OpenStack (TripleO) but

Thank You

Ramon Acedo Rodriguez

Product Manager, Red Hat OpenStack Team

@ramonacedo | [email protected]