MOSK Reference Architecture Reference... · Introduction Mirantis OpenStack on Kubernetes leverages the container isolation, state enforcement, and declarative definition of resources

MOSK Reference Architectureversion beta

ContentsCopyright notice 1Preface 2

About this documentation set 2Intended audience 2Documentation history 2Conventions 3

Introduction 5Deployment profiles 6

Supported deployment profiles 7Components overview 8Requirements 10

Hardware requirements 11Infrastructure requirements 13

OpenStack 14OpenStack Operator 15

OpenStackDeployment custom resource 16OpenStack Controller 22

OpenStack on Kubernetes architecture 24OpenStack and Ceph controllers integration 27OpenStack and StackLight integration 28OpenStack and Tungsten Fabric controllers integration 29

Networking 30Physical networks layout 31Network types 33Performance optimization 35

Storage 36StackLight 38

Deployment architecture 39Monitored components 42

Tungsten Fabric 43Tungsten Fabric cluster overview 44

MOSK Reference Architecture Beta

©2020, Mirantis Inc. Page i

Tungsten Fabric components 45Tungsten Fabric operator 48Tungsten Fabric traffic flow 51Tungsten Fabric vRouter 53


©2020, Mirantis Inc. Page ii

Copyright notice2020 Mirantis, Inc. All rights reserved.This product is protected by U.S. and international copyright and intellectual property laws. Nopart of this publication may be reproduced in any written, electronic, recording, or photocopyingform without written permission of Mirantis, Inc.Mirantis, Inc. reserves the right to modify the content of this document at any time without priornotice. Functionality described in the document may not be available at the moment. Thedocument contains the latest information at the time of publication.Mirantis, Inc. and the Mirantis Logo are trademarks of Mirantis, Inc. and/or its affiliates in theUnited States an other countries. Third party trademarks, service marks, and names mentionedin this document are the properties of their respective owners.


©2020, Mirantis Inc. Page 1

PrefaceAbout this documentation setThis documentation provides information on how to deploy and operate the Mirantis OpenStackon Kubernetes (MOSK) environment. The documentation is intended to help operators tounderstand the core concepts of the product. The documentation provides sufficient informationto deploy and operate the solution as of its state for the beta release. The beta release can onlybe used in a non-production small environment for demonstration and evaluation purposes.The information provided in this documentation set is being constantly improved and amendedbased on the feedback and kind requests from our beta software consumers.The following table lists the guides included in the documentation set you are reading:

Guides list

Guide PurposeMOSK ReferenceArchitecture

Learn the fundamentals of MOSK reference architecture toappropriately plan your deployment

MOSK Deployment Guide Deploy an MOSK environment of a preferred configuration usingsupported deployment profiles tailored to the demands of specificbusiness cases

MOSK Operations Guide Operate your MOSK environmentMOSK Release notes Learn about new features and bug fixes in the current MOSK

version

The MOSK documentation home page contains references to all guides included in thisdocumentation set. For your convenience, we provide all guides in HTML (default), single-pageHTML, PDF, and ePUB formats. To use the preferred format of a guide, select the required optionfrom the Formats menu next to the guide title.

Intended audienceThis documentation is intended for engineers who have the basic knowledge of Linux,virtualization and containerization technologies, Kubernetes API and CLI, Helm and Helm charts,Docker Enterprise UCP, and OpenStack.

Documentation historyThe following table contains the released revision of the documentation set you are reading:

Release date DescriptionApril 08, 2020 Mirantis OpenStack on Kubernetes (MOSK) beta v1April 30, 2020 Mirantis OpenStack on Kubernetes (MOSK) beta v2May 29, 2020 Mirantis OpenStack on Kubernetes (MOSK) beta v2.1



https://docs.mirantis.com/mosk/beta/

July 22, 2020 Mirantis OpenStack on Kubernetes (MOSK) beta v3

The documentation set refers to MOSK beta as to the latest released beta version of the product.

ConventionsThis documentation set uses the following conventions in the HTML format:

Documentation conventions

Convention Descriptionboldface font Inline CLI tools and commands, titles of the procedures and

system response examples, table titlesmonospaced font Files names and paths, Helm charts parameters and their values,

names of packages, nodes names and labels, and so onitalic font Information that distinguishes some concept or termLinks External links and cross-references, footnotesMain menu > menu item GUI elements that include any part of interactive user interface

and menu navigationSuperscript Some extra, brief information

NoteThe Note block

Messages of a generic meaning that may be useful for the user

Caution!The Caution block

Information that prevents a user from mistakes and undesirableconsequences when following the procedures

WarningThe Warning block

Messages that include details that can be easily missed, butshould not be ignored by the user and are valuable beforeproceeding



https://docs.mirantis.com/mosk

SeealsoThe See also block

List of references that may be helpful for understanding of somerelated tools, concepts, and so on

Learn moreThe Learn moreblock

Used in the Release Notes to wrap a list of internal references tothe reference architecture, deployment and operation proceduresspecific to a newly implemented product feature



IntroductionMirantis OpenStack on Kubernetes leverages the container isolation, state enforcement, anddeclarative definition of resources provided by Kubernetes to deploy and manage OpenStackclusters.The key advantages of the Mirantis OpenStack on Kubernetes product include:

• Containerization of OpenStack servicesAllows for fine-grain and reliable update and upgrade of components since the commonrun-time dependencies between services are minimized.

• Reconciliation of resource stateProvides an extra level of resiliency, as Kubernetes automatically restarts failedprocesses and ensures the number of running processes (pods) for each serviceremains the same. Moreover, the very definition of a failed process can be fine-tuned tothe type of workload running inside the pod through the mechanism of readiness andliveness probes.

• Declarative resource definitionAllows for easier, human- and machine-friendly management of a cluster state.



Deployment profilesA Mirantis OpenStack on Kubernetes (MOSK) deployment profile is a thoroughly tested andofficially supported reference architecture that is guaranteed to work at a specific scale and istailored to the demands of a specific business case, such as generic IaaS cloud, NetworkFunction Virtualisation infrastructure, Edge Computing, and others.A deployment profile is defined as a combination of:

• Services and features the cloud offers to its users.• Non-functional characteristics that users and operators should expect when running the

profile on top of a reference hardware configuration. Including, but not limited to:

• Performance characteristics, such as an average network throughput between VMs inthe same virtual network.

• Reliability characteristics, such as the cloud API error response rate when recovering afailed controller node.

• Scalability characteristics, such as the total amount of virtual routers tenants can runsimultaneously.

• Hardware requirements - the specification of physical servers, and networking equipmentrequired to run the profile in production.

• Deployment parameters that an operator for the cloud can tweak within a certain rangewithout being afraid of breaking the cloud or losing support.

In addition, the following items may be included in a definition:

• Compliance-driven technical requirements, such as TLS encryption of all external APIendpoints.

• Foundation-level software components, such as Tungsten Fabric or Open vSwitch as a backend for the networking service.

NoteMirantis reserves the right to revise the technical implementation of any profile at willwhile preserving its definition - the functional and non-functional characteristics thatoperators and users are known to rely on.



Supported deployment profiles

Supported deployment profiles

Profile Identifier inOsDpl CR Description

Beta PoC compute A small non-production cloud built fromoff-your-shelve hardware and used fordemonstration or evaluation of OpenStack onKubernetes Beta release capabilities with theprimary focus on lifecycle management functions.The profile provides the core set of the services anIaaS vendor would need including some extrafunctionality. The profile is designed to support upto 30 compute nodes and as small number ofstorage nodes.The core set of services provided by the profileincludes:

• Compute (Nova with Ceph as a back end forroot and ephemeral storage)

• Images (Glance)• Networking (Neutron with Open vSwitch as a

back end)• Identity (Keystone)• Block Storage (Cinder with Ceph as a back

end)• Orchestration (Heat)• Load balancing (Octavia)• DNS (Designate)• Secret Management (Barbican with

SimpleCrypto back end)• Web front end (Horizon)

Beta PoC +Tungsten Fabric

compute-tf A variation of the Beta PoC profile with TugstenFabric 5.1 as a back end for networking.Tungsten Fabric is offered as technical preview,meaning that it can be deployed and used as apart of the cloud but no performance or scalabilitytesting was performed as of Beta release.Load balancing capabilities are provided byTungsten Fabric built-in mechanism and aremanaged directly through the Tungsten Fabric API.



Components overviewMirantis OpenStack on Kubernetes (MOSK) includes the following key design elements:

• HelmBundle OperatorThe HelmBundle Operator is the realization of the Kubernetes Operator pattern thatprovides a Kubernetes custom resource of the HelmBundle kind and code running inside apod in Kubernetes. This code handles changes, such as creation, update, and deletion, inthe Kubernetes resources of this kind by deploying, updating, and deleting groups of Helmreleases from specified Helm charts with specified values.

• OpenStackThe OpenStack platform manages virtual infrastructure resources, including virtual servers,storage devices, networks, and networking services, such as load balancers, as well asprovides management functions to the tenant users.Various OpenStack services are running as pods in Kubernetes and are represented asappropriate native Kubernetes resources, such as Deployments, StatefulSets, andDaemonSets.For a simple, resilient, and flexible deployment of OpenStack and related services on top ofa Kubernetes cluster, MOSK uses OpenStack-Helm that provides a required collection of theHelm charts.Also, MOSK uses OpenStack Operator as the realization of the Kubernetes Operator pattern.The OpenStack Operator provides a custom Kubernetes resource of theOpenStackDeployment kind and code running inside a pod in Kubernetes. This code handleschanges such as creation, update, and deletion in the Kubernetes resources of this kind bydeploying, updating, and deleting groups of the HelmBundle resources handled by theHelmBundle Operator to manage OpenStack in Kubernetes through the OpenStack-Helmcharts.

• CephCeph is a distributed storage platform that provides storage resources, such as objects andvirtual block devices, to virtual and physical infrastructure.MOSK uses Rook as the implementation of the Kubernetes Operator pattern that managesresources of the CephCluster kind to deploy and manage Ceph services as pods on top ofKubernetes to provide Ceph-based storage to the consumers, which include OpenStackservices, such as Volume and Image services, and underlying Kubernetes through Ceph CSI(Container Storage Interface).The Ceph controller is the implementation of the Kubernetes Operator pattern, thatmanages resources of the MiraCeph kind to simplify management of the Rook-based Cephclusters.

• StackLight Logging, Monitoring, and AlertingThe StackLight component is responsible for collection, analysis, and visualization of criticalmonitoring data from physical and virtual infrastructure, as well as alerting and errornotifications through a configured communication system, such as email. StackLightincludes the following key sub-components:



• Prometheus• Elasticsearch• Fluentd• Kibana



Requirements



Hardware requirementsThis section provides hardware requirements for the Mirantis OpenStack on Kubernetesreference architecture. More specifically, it defines the basic requirements for server equipment.The hardware nodes present in the Mirantis OpenStack on Kubernetes reference architectureinclude:

• OpenStack control plane nodes and StackLight nodesHost OpenStack control plane services such as database, messaging, API, schedulersconductors, and L3 and L2 agents, as well as the StackLight components.

• Tenant gateway nodesOptional, host OpenStack gateway services including L2, L3, and DHCP agents. Thetenant gateway nodes are combined with OpenStack control plane nodes. The strictrequirement is a dedicated physical network (bond) for tenant network traffic.

• Tungsten Fabric network nodesRequired only if Tungsten Fabric (TF) is enabled as a back end for OpenStacknetworking. These nodes host the TF control plane services such as Cassandradatabase, messaging, API, control, configuration, and analytics services.

• Compute nodesHost OpenStack Compute services such as QEMU, L2 agents, and others.

• Infrastructure nodesRun Kubernetes cluster management and Ceph services. The MCP NG referenceconfiguration requires minimum three infrastructure nodes.

The table below specifies the roles assigned to the hardware nodes in the MCP NG referencearchitecture and required resources per node.

Hardware requirements

Node type # ofservers

CPUcores #

perserver

Memory(GB) perserver

Diskspaceper

server

NICs #per

server

OpenStack control plane,gateway 1, and StackLightnodes

3 32 128 2 TB SSD 5

Tenant gateway (optional) 0-3 32 128 2 TB SSD 5Tungsten Fabric control planenodes

3 32 128 2 TB SSD 1

Compute node 3 (varies) 16 64 500 GBSSD

5

Infrastructure node(Kubernetes clustermanagement)

3 16 64 500 GBSSD

5



Infrastructure node (Ceph) 2 3 16 64 1 SSD 500GB and 2HDDs 2TB each

5

NoteThe exact hardware specifications and number of nodes depend on a cloud configurationand scaling needs.

1 OpenStack gateway services can optionally be moved to separate nodes.2 A Ceph cluster with 3 OSD nodes does not provide hardware fault tolerance

and is not eligible for recovery operations, such as a disk or an entire nodereplacement.

NoteIf you are looking to try MOSK and do not have much hardware at your disposal, you candeploy it in a virtual environment, for example, on top of another OpenStack cloud usingthe sample Heat templates.Please mind, the tooling is provided for reference only and is not a part of the productitself. Mirantis does not guarantee its interoperability with the latest MOSK version.



https://github.com/Mirantis/openstack-k8s-virtual-lab/

Infrastructure requirementsThis section lists the infrastructure requirements for the Mirantis OpenStack on Kubernetesreference architecture.

Infrastructure requirements

Service DescriptionMetalLB MetalLB exposes external IP addresses to access applications in a

Kubernetes cluster.DNS The Kubernetes Ingress NGINX controller is used to expose OpenStack

services outside of a Kubernetes deployment. Access to the Ingressservices is allowed only by its FQDN. Therefore, DNS is a mandatoryinfrastructure service for an OpenStack on Kubernetes deployment.

Seealso

• MOSK Deployment guide: Configure DNS to access OpenStack• MOSK Deployment guide: Deploy MetalLB



https://docs.mirantis.com/mosk/beta/mosk-deployment-guide/access-openstack/dns.html

https://docs.mirantis.com/mosk/beta/mosk-deployment-guide/deploy-prerequisite-components/metallb.html

OpenStack



OpenStack OperatorThe OpenStack Operator component is a combination of the following entities:



OpenStackDeployment custom resourceThe resource of kind OpenStackDeployment (OsDpl) is a custom resource (CR) defined by acorresponding resource of kind CustomResourceDefinition.To obtain the detailed information about schema of an OsDpl CR, run:

kubectl get crd openstackdeployments.lcm.mirantis.com -oyaml

To obtain the definition of a particular OpenStack deployment, run:

kubectl -n openstack get osdpl -oyaml

A minimal example of an OsDpl CR:

apiVersion: lcm.mirantis.com/v1alpha1kind: OpenStackDeploymentmetadata: name: openstack-cluster namespace: openstackspec: openstack_version: train profile: compute size: tiny internal_domain_name: cluster.local public_domain_name: it.just.works features: ssl: public_endpoints: api_cert: |- # Update server certificate content api_key: |- # Update server private key content ca_cert: |- # Update CA certificate content neutron: tunnel_interface: ens3 external_networks: - physnet: physnet1 interface: veth-phy bridge: br-ex network_types: - flat vlan_ranges: null mtu: null floating_network: enabled: False nova: live_migration_interface: ens3



images: backend: local

For the detailed description of the OsDpl main elements, see the tables below:

• The main elements• The Spec elements• The Status elements

The main elements

Element DescriptionapiVersion Specifies the version of the Kubernetes API that is used to create

this object.kind Specifies the kind of the object.metadata:name Specifies the name of metadata. Should be set in compliance with

the Kubernetes resource naming limitations.metadata:namespace Specifies the metadata namespace. While technically it is possible

to deploy OpenStack on top of Kubernetes in other than openstacknamespace, such configuration is not included in the MOSKsystem integration test plans. Therefore, we do not recommendsuch scenario.

WarningBoth OpenStack and Kubernetes platforms provideresources to applications. When OpenStack is running ontop of Kubernetes, Kubernetes is completely unaware ofOpenStack-native workloads, such as virtual machines, forexample.For better results and stability, Mirantis recommends usinga dedicated Kubernetes cluster for OpenStack, so thatOpenStack and auxiliary services, Ceph, and StackLight arethe only Kubernetes applications running in the cluster.



spec Contains the data that defines the OpenStack deployment andconfiguration. It has both high-level and low-level sections.The very basic values that must be provided include:

spec: openstack_version: profile: size: internal_domain_name: public_domain_name:

For the detailed description of the spec subelements, see TheSpec elements

The Spec elements

Element Descriptionopenstack_version Specifies the OpenStack release to deploy.profile Contains an identifier of a curated and supported deployment

profile for OpenStack on Kubernetes.Consult Supported deployment profiles for the list of possiblevalues.

size Contains a separate set of curated values for topology andsettings of OpenStack and auxiliary services. These include anumber of replicas for each service, timeout values for variousoperations, and so on.The list of supported sizes include:

• tiny - for approximately 10 OpenStack compute nodes• small - for approximately 50 OpenStack compute nodes• medium - for approximately 100 OpenStack compute nodes

internal_domain_name Specifies the internal DNS name used inside the Kubernetescluster on top of which the OpenStack cloud is deployed.

public_domain_name Specifies the public DNS name for OpenStack services. This is abase DNS name that must be accessible and resolvable by APIclients of your OpenStack cloud. It will be present in theOpenStack endpoints as presented by the OpenStack Identityservice catalog.The TLS certificates used by the OpenStack services (see below)must also be issued to this DNS name.



features Contains the top-level collections of settings for the OpenStackdeployment that potentially target several OpenStack services.The section where the customizations should take place. Forexample, for a minimal resource with the defined features, see theminimal example of a kind: OpenStackDeployment resourceabove.

features:ssl Contains the content of SSL/TLS certificates (server, key, CAbundle) used to enable a secure communication to publicOpenStack API services.These certificates must be issued to the DNS domain specified inthe public_domain_name field (see above).

features:neutron:tunnel_interface

Defines the name of the NIC device on the actual host that will beused for Neutron.We recommend setting up your Kubernetes hosts in such a waythat networking is configured identically on all of them, and namesof the interfaces serving the same purpose or plugged into thesame network are consistent across all physical nodes.

features:neutron:external_networks

Contains the data structure that defines external (provider)networks on top of which the Neutron networking will be created.

features:neutron:floating_network

If enabled, must contain the data structure defining the floating IPnetwork that will be created for Neutron to provide externalaccess to your Nova instances.

features:nova:live_migration_interface

Specifies the name of the NIC device on the actual host that willbe used by Nova for the live migration of instances.We recommend setting up your Kubernetes hosts in such a waythat networking is configured identically on all of them, and namesof the interfaces serving the same purpose or plugged into thesame network are consistent across all physical nodes.

features:nova:images:backend

Defines the type of storage for Nova to use on the compute hostsfor the images that back up the instances.The list of supported options include:

• local - the local storage is used. The pros include fasteroperation, failure domain independency from the externalstorage. The cons include local space consumption and lessperformant and robust live migration with block migration.

• ceph - instance images are stored in a Ceph pool sharedacross all Nova hypervisors. The pros include faster imagestart, faster and more robust live migration. The cons includeconsiderably slower IO performance, workload operationsdirect dependency on Ceph cluster availability andperformance.

artifacts A low-level section that defines the base URI prefixes for imagesand binary artifacts.



common A low-level section that defines values that will be passed to allOpenStack (spec:common:openstack) or auxiliary(spec:common:infra) services Helm charts.Structure example:

spec: artifacts: common: openstack: values: infra: values:

services Is a section of the lowest level, enables the definition of specificvalues to pass to specific Helm charts on a one-by-one basis:

WarningMirantis does not recommend changing the default settings for spec:artifacts,spec:common, and spec:services elements. Customizations can compromise theOpenStack deployment update and upgrade processes.

The Status elements

Element Descriptionstatus Contains information about the current status of an OpenStack

deployment, which cannot be changed by the user.status:children Specifies the current status of Helm releases that are managed by the

OpenStack Operator. The possible values include:

• True - when the Helm chart is in the deployed state.• False - when an error occurred during the Helm chart deployment.

An example of children output:

children: openstack-block-storage: true openstack-compute: true openstack-coordination: true ...

status:deployed Shows an overall status of all Helm releases. Shows True when allchildren are in the deployed state.



status:fingerprint Is the MD5 hash of the body:spec object. It is passed to all childHelmBundles to thebody:metadata:lcm.mirantis.com/openstack-controller-fingerprintobject. Also, it enables detecting the OsDpl resource version usedwhen applying the child (HelmBundle) resource.

status:version Contains the version of the OpenStack Operator that processes theOsDpl resource. And, similarly to fingerprint, it enables detecting theversion of the OpenStack Operator that processed the child(HelmBundle) resource.

status:health While status:children shows information about any deployed childHelmBundle resources, status:health shows the actual status of thedeployed Kubernetes resources. The resources may be created asdifferent Kubernetes objects, such as Deployments, Statefulsets, orDaemonSets. Possible values include:

• Ready - all pods from the resource are in the Ready state• Unhealthy - not all pods from the resource are in the Ready state• Progressing - Kubernetes resource is updating• Unknown - other, unrecognized states

An example of the health output:

health: barbican: api: generation: 4 status: Ready rabbitmq: generation: 1 status: Ready cinder: api: generation: 4 status: Ready backup: generation: 2 status: Ready rabbitmq: generation: 1 status: Ready ... ...

status:kopf Contains the structure that is used by the Kopf library to store itsinternal data.



OpenStack ControllerThe OpenStack Controller runs in a set of containers in a pod in Kubernetes. The OpenStackController is deployed as a Deployment with 1 replica only. The failover is provided byKubernetes that automatically restarts the failed containers in a pod.However, given the recommendation to use a separate Kubernetes cluster for each OpenStackdeployment, the controller in envisioned mode for operation and deployment will only manage asingle OpenStackDeployment resource, making the proper HA much less of an issue.The OpenStack Controller is written in Python using Kopf, as a Python framework to buildKubernetes operators, and Pykube, as a Kubernetes API client.Using Kubernetes API, the controller subscribes to changes to resources ofkind: OpenStackDeployment, and then reacts to these changes by creating, updating, ordeleting appropriate resources in Kubernetes.The basic child resources managed by the controller are of kind: HelmBundle. They are renderedfrom templates taking into account an appropriate values set from the main and features fieldsin the OpenStackDeployment resource.Then, the common fields are merged to resulting data structures. Lastly, the services fields aremerged providing the final and precise override for any value in any Helm release to bedeployed or upgraded.The constructed HelmBundle resources are then supplied to the Kubernetes API. TheHelmBundle controller picks up the changes in these resources, similarly to the OpenStackOperator, and translates to the Helm releases, deploying, updating, or deleting nativeKubernetes resources.

OpenStack Controller containers

Container Descriptionosdpl The core container that handles changes in the osdpl object.helmbundle The container that watches the helmbundle objects and reports their

statuses to the osdpl object in status:children. See The Status elements fordetails.

health The container that watches all Kubernetes native resources, such asDeployments, Daemonsets, Statefulsets, and reports their statuses to theosdpl object in status:health. See The Status elements for details.

secrets The container that provides data exchange between different componentssuch as Ceph.

node The container that handles the node events.





OpenStack on Kubernetes architectureOpenStack and auxiliary services are running as containers in the kind: Pod Kubernetesresources. All long-running services are governed by one of the ReplicationController-enabledKubernetes resources, which include either kind: Deployment, kind: StatefulSet, orkind: DaemonSet.The placement of the services is mostly governed by the Kubernetes node labels. The labelsaffecting the OpenStack services include:

• openstack-control-plane=enabled - the node hosting most of the OpenStack control planeservices.

• openstack-compute-node=enabled - the node serving as a hypervisor for Nova. The virtualmachines with tenants workloads are created there.

• openvswitch=enabled - the node hosting Neutron L2 agents and OpenvSwitch pods thatmanage L2 connection of the OpenStack networks.

• openstack-gateway=enabled - the node hosting Neutron L3, Metadata and DHCP agents,Octavia Health Manager, Worker and Housekeeping components.

NoteOpenStack is an infrastructure management platform. OpenStack on Kubernetes usesKubernetes mostly for orchestration and dependency isolation. As a result, multipleOpenStack services are running as privileged containers with host PIDs and HostNetworking enabled. You must ensure that at least the user with the credentials used byHelm/Tiller (administrator) is capable of creating such Pods.

Infrastructure services

Service Description



Storage While the underlying Kubernetes cluster is configured to use Ceph CSIfor providing persistent storage for container workloads, for sometypes of workloads such networked storage is suboptimal due tolatency.This is why the separate local-volume-provisioner CSI is deployed andconfigured as an additional storage class. Local Volume Provisioner isdeployed as kind: DaemonSet.

Database A single WSREP (Galera) cluster of MariaDB is deployed as the SQLdatabase to be used by all OpenStack services. It uses the storageclass provided by Local Volume Provisioner to store the actualdatabase files. The service is deployed as kind: StatefulSet of a givensize, which is no less than 3, on any openstack-control-plane node.

Messaging RabbitMQ is used as a messaging bus between the components of theOpenStack services.A separate instance of RabbitMQ is deployed for each OpenStackservice that needs a messaging bus for intercommunication betweenits components.An additional, separate RabbitMQ instance is deployed to serve as anotification messages bus for OpenStack services to post their ownand listen to notifications from other services. StackLight also usesthis message bus to collect notifications for monitoring purposes.Each RabbitMQ instance is a single node and is deployed askind: StatefulSet.

Caching A single multi-instance of the Memcached service is deployed to beused by all OpenStack services that need caching, which are mostlyHTTP API services.

Coordination A separate instance of etcd is deployed to be used by Cinder, whichrequire Distributed Lock Management for coordination between itscomponents.

Ingress Is deployed as kind: DaemonSet.Image pre-caching A special kind: DaemonSet is deployed and updated each time the

kind: OpenStackDeployment resource is created or updated. Itspurpose is to pre-cache container images on Kubernetes nodes, andthus, to minimize possible downtime when updating containerimages.This is especially useful for containers used in kind: DaemonSetresources, as during the image update Kubernetes starts to pull thenew image only after the container with the old image is shut down.

OpenStack services

Service Description



Identity (Keystone) Uses MySQL back end by default.keystoneclient - a separate kind: Deployment with a pod that has theOpenStack CLI client as well as relevant plugins installed, andOpenStack admin credentials mounted. Can be used by administratorto manually interact with OpenStack APIs from within a cluster.

Image (Glance) Supported back end is RBD (Ceph is required).Volume (Cinder) Supported back end is RBD (Ceph is required).Network (Neutron) Supported back end is Open vSwitch. Tungsten Fabric is available as

technical preview.PlacementCompute (Nova) Supported hypervisor is Qemu/KVM through libvirt library.Dashboard (Horizon)DNS (Designate) Supported back end is PowerDNS.Load Balancer(Octavia)Orchestration (Heat)Key Manager(Barbican)

Supported back end is built-in SimpleCrypto.Must be enabled separately through the list of services to bedeployed in the kind: OpenStackDeployment resource:

spec: features: services: - key-manager

Tempest Can be added to the list of services in kind: OpenStackDeployment torun tests against a deployed OpenStack cloud:

spec: features: services: - tempest



OpenStack and Ceph controllers integrationThe integration between Ceph and OpenStack controllers is implemented through the sharedKubernetes openstack-ceph-shared namespace. Both controllers have access to this namespaceto read and write the Kubernetes kind: Secret objects.

As Ceph is required and only supported back end for several OpenStack services, all necessaryCeph pools must be specified in the configuration of the kind: MiraCeph custom resource as partof the deployment. Once the Ceph cluster is deployed, the Ceph controller posts the informationrequired by the OpenStack services to be properly configured as a kind: Secret object into theopenstack-ceph-shared namespace. The OpenStack controller watches this namespace. Oncethe corresponding secret is created, the OpenStack controller transforms this secret to the datastructures expected by the OpenStack-Helm charts. Even if an OpenStack installation istriggered at the same time as a Ceph cluster deployment, the OpenStack Controller halts thedeployment of the OpenStack services that depend on Ceph availability until the secret in theshared namespace is created by the Ceph controller.For the configuration of Ceph RADOS Gateway as an OpenStack Object Storage, the reverseprocess takes place. The OpenStack controller waits for the OpenStack-Helm to create a secretwith OpenStack Identity (Keystone) credentials that RADOS Gateway must use to validate theOpenStack Identity tokens, and posts it back to the same openstack-ceph-shared namespace inthe format suitable for consumption by the Ceph controller. The Ceph controller then reads thissecret and reconfigures RADOS Gateway accordingly.



OpenStack and StackLight integrationThe main option to simultaneously set for StackLight and OpenStackDeployment is theRabbitMQ password for StackLight to connect to the OpenStack Notification messaging bus. Theappropriate stacklight user is created by the OpenStack Operator automatically.Configuration example from the OpenStackDeployment resource:

apiVersion: lcm.mirantis.com/v1alpha1kind: OpenStackDeploymentspec: features: stacklight: password: <stacklight_pwd>

Configuration example from the HelmBundle resouce for StackLight:

apiVersion: lcm.mirantis.com/v1alpha1kind: HelmBundlespec: releases: - chart: stacklight/stacklight values: openstack: enabled: true rabbitmq: credentialsConfig: password: <stacklight_pwd>



OpenStack and Tungsten Fabric controllers integration

NoteThis feature is available as technical preview. Use such configuration for testing andevaluation purposes only.

The integration between the Tungsten Fabric (TF) and OpenStack controllers is implementedthrough the shared Kubernetes openstack-tf-shared namespace. Both controllers have access tothis namespace to read and write the Kubernetes kind: Secret objects.The Keystone administrator credentials and an up-and-running OpenStack Identity service arerequired for the TF controller to initiate the deployment process. The OpenStack controller poststhe data into the openstack-tf-shared namespace required by the TF services. The TF controllerwatches this namespace. Once an appropriate secret is created, the TF controller obtains it intothe internal data structures for further processing.



NetworkingDepending on the size of an OpenStack environment and the components that you use, you maywant to have a single or multiple network interfaces, as well as run different types of traffic on asingle or multiple VLANs.This section provides the recommendations for planning the network configuration andoptimizing the cloud performance.



Physical networks layoutThe image below illustrates the recommended physical networks layout for an MCP NGOpenStack deployment with Ceph.

The image below illustrates the Ceph storage physical network layout.





Network typesWhen planning your OpenStack environment, consider what types of traffic your workloadsgenerate and design your network accordingly. If you anticipate that certain types of traffic,such as storage replication, will likely consume a significant amount of network bandwidth, youmay want to move that traffic to a dedicated network interface to avoid performancedegradation.The MCP NG Kubernetes deployment typically requires the following types of networks:

L3 networks for Kubernetes

Network DescriptionCommon/PXEnetwork

The network used for the provisioning of bare metal servers.

Managementnetwork

The network used for managing of bare metal servers.

Kubernetesworkloadsnetwork

The routable network for communication between containers in Kubernetes.

Storage accessnetwork(Ceph)

The network used for accessing the Ceph storage. We recommended that itis placed on a dedicated hardware interface.

Storagereplicationnetwork(Ceph)

The network used for the storage replication (Ceph). We recommended thatit is placed on a dedicated hardware interface to ensure low latency and fastaccess.

Externalnetworks(MetalLB)

The routable network used for external IP addresses of the KubernetesLoadBalancer services managed by MetalLB.

The MCP NG OpenStack on Kubernetes deployment additionally requires the following types ofnetworks:

L3 networks for OpenStack on Kubernetes

Servicename Network Description

Networking Providernetworks

Typically, a routable network used to provide the externalaccess to OpenStack instances (a floating network). Can be usedby the OpenStack services such as Ironic, Manila, and others, toconnect their management resources.

Networking Overlaynetworks(virtualnetworks)

The network used to provide denied, secure tenant networkswith the help of the tunneling mechanism (VLAN/GRE/VXLAN). Ifthe VXLAN and GRE encapsulation takes place, the IP addressassignment is required on interfaces at the node level.



Compute Livemigrationnetwork

The network used by the OpenStack compute service (Nova) totransfer data during live migration. Depending on the cloudneeds, it can be placed on a dedicated physical network not toaffect other networks during live migration. The IP addressassignment is required on interfaces at the node level.

The way of mapping of the logical networks described above to physical networks and interfaceson nodes depends on the cloud size and configuration. We recommend placing OpenStacknetworks on a dedicated physical interface (bond) that is not shared with storage andKubernetes management network to minimize the influence on each other.



Performance optimizationTo improve the goodput, we recommend that you enable jumbo frames where possible. Thejumbo frames have to be enabled on the whole path of the packets traverse. If one of thenetwork components cannot handle jumbo frames, the network path uses the smallest MTU.To provide fault tolerance of a single NIC, we recommend using the link aggregation, such asbonding. The link aggregation is useful for linear scaling of bandwidth, load balancing, and faultprotection. Depending on the hardware equipment, different types of bonds might be supported.Use the multi-chassis link aggregation as it provides fault tolerance at the device level. Forexample, MLAG on Arista equipment or vPC on Cisco equipment.The Linux kernel supports the following bonding modes:

• active-backup• balance-xor• 802.3ad (LACP)• balance-tlb• balance-alb

Since LACP is the IEEE standard 802.3ad supported by the majority of network platforms, werecommend using this bonding mode.



StorageMirantis OpenStack on Kubernetes uses Ceph as a distributed storage system for block andobject storage. Ceph is deployed using Helm charts with the following components:

• Ceph controller - a Kubernetes controller that obtains the parameters from MirantisOpenStack on Kubernetes through a custom resource (CR), creates CRs for Rook, andupdates its CR status based on the Ceph cluster deployment progress. Ceph controllercreates users, pools, and keys for OpenStack and Kubernetes and provides Cephconfigurations and keys to access them. Also, Ceph controller eventually obtains the datafrom the OpenStack Controller for the Keystone integration and updates the RADOSGateway services configurations to use Kubernetes for user authentication.

• Ceph operator

• Transforms user parameters into Rook credentials and deploys a Ceph cluster usingRook.

• Provides integration of the Ceph cluster with Kubernetes.• Provides data for OpenStack to integrate with the deployed Ceph cluster.

• Custom resource (CR) - represents the customization of a Kubernetes installation andenables you to define the required Ceph configuration before deployment. For example, youcan define the failure domain, pools, node roles, number of Ceph components such as CephOSDs, and so on.

• Rook - a storage orchestrator that deploys Ceph on top of a Kubernetes cluster.A typical Ceph cluster consists of the following components:Ceph Monitors

Three or, in rare cases, five Ceph Monitors.Ceph Managers

Mirantis recommends having three Ceph Managers in every clusterRADOS Gateway services

Mirantis recommends having three or more RADOS Gateway services for HA.Ceph OSDs

The number of OSDs may vary according to the deployment needs.

WarningA Ceph cluster with 3 OSD nodes does not provide hardware fault tolerance and is noteligible for recovery operations, such as a disk or an entire node replacement.

The placement of all Ceph components is defined by specifying proper roles in the MiraCephcustom resource.



Seealso

• OpenStack and Ceph controllers integration• Ceph documentation• Rook documentation



https://docs.ceph.com/docs/master/

https://rook.github.io/docs/rook/master/

StackLightStackLight is the logging, monitoring, and alerting solution that provides a single pane of glassfor cloud maintenance and day-to-day operations as well as offers critical insights into cloudhealth including operational information about the components deployed in Mirantis OpenStackon Kubernetes. StackLight is based on Prometheus, an open-source monitoring solution and atime series database, and Elasticsearch, the logs and notifications storage.



Deployment architectureMirantis OpenStack on Kubernetes deploys the StackLight stack as a release of a Helm chart thatcontains the helm-controller and HelmBundle custom resources. The StackLight HelmBundleconsists of a set of Helm charts describing the StackLight components.

StackLight components overview

StackLight component DescriptionPrometheus server Gathers metrics. Automatically discovers and monitors the

endpoints. By default, the Prometheus database stores metrics forthe past 15 days or up to 15 GB of data depending on the limitthat is reached first.

Prometheus Relay Adds a proxy layer to Prometheus to merge the results fromunderlay Prometheus servers to prevent gaps in case some data ismissing on some servers. Prometheus Relay is available only inthe HA StackLight mode.

Alertmanager Handles the alerts sent by client applications such as Prometheus,deduplicates, groups, and routes alerts to receiver integrations, aswell as performs silencing and inhibition of alerts.

Alerta Receives, consolidates, and deduplicates the alerts sent byAlertmanager and visually represents them through a simple webUI. Using Alerta, you can view the most recent or watched alerts,group, and filter alerts.

Pushgateway Enables ephemeral and batch jobs to expose their metrics toPrometheus. Since these jobs may not exist long enough to bescraped, they can instead push their metrics to Pushgateway,which then exposes these metrics to Prometheus. Pushgateway isnot an aggregator or a distributed counter but rather a metricscache. The pushed metrics are exactly the same as scraped froma permanently running program.

Grafana Builds and visually represents metric graphs based on time seriesdatabases. Grafana supports querying of Prometheus using thePromQL language and includes preconfigured dashboards for mostOpenStack services and infrastructure components.

Logging system Responsible for collecting, processing, and persisting the logs. Thelogging system components include:

• Fluentd, which collects logs, sends them to Elasticsearch,generates metrics based on analysis of incoming log entries,and exposes these metrics to Prometheus.

• Elasticsearch, which stores logs and notifications.• Kibana, which provides real-time visualization and searching

of the data stored in Elasticsearch and enables you to detectissues.



Salesforce integration Salesforce integration includes the following services:

• Salesforce notifier, which enables creating Salesforce casesfrom the Alertmanager notifications and close the cases oncethe alerts are resolved.

• Salesforce reporter, which queries Prometheus for thefollowing metrics data, combines the data, and creates aSalesforce object through API:

• The amount of vCPU, vRAM, and vStorage used andavailable

• The number of VMs running, compute nodes, andtenants/projects

• The availability of Cinder, Nova, Keystone, Glance, andNeutron

By default, Salesforce reporter sends the data to API once perday. Mirantis uses the collected data for further analysis andreports to improve the quality of customer support.

Prometheuselasticsearch exporter

Presents the Elasticsearch data as Prometheus metrics byperiodically sending configured queries to the Elasticsearchcluster and exposing the results to a scrapable HTTP endpoint likeother Prometheus targets.

Metricbeat Collects Kubernetes events and sends them to Elasticsearch forstorage.

Prometheus metricexporters

Export the existing metrics as Prometheus metrics. Prometheusmetric exporters include:

• Prometheus node exporter, which gathers hardware andoperating system metrics exposed by kernel.

• Telegraf agents, which gather basic operating systemmetrics.

• Native exporters and endpoints (libvirt-exporter,nginx-exporter, elasticsearch-exporter, memcached-exporter,rabbitmq-exporter, mysql-exporter).

During the StackLight deployment, you can define the HA or non-HA StackLight architecturetype. The architecture types have the following differences:

StackLight architecture types comparison

Instances # of instances in HA mode # of instances in non-HAmode

Prometheus 2 1Prometheus Relay 1 -Elasticsearch 3 1



The following diagram illustrates the StackLight components layout in the HA and non-HAmodes:



Monitored componentsStackLight measures, analyzes, and reports in a timely manner about failures that may occur inthe following Mirantis OpenStack on Kubernetes components and their sub-components:

• Ceph (OSD, ceph-mon)• Kubernetes services (Kubernetes cluster, Kubernetes containers, Kubernetes deployments,

Kubernetes nodes, Netchecker)• libvirt• Linux operating system• Memcached• MongoDB• MySQL• Node hardware• NTP• OpenStack (Nova, Cinder, Glance, Neutron, Keystone, Horizon, Heat, Octavia)• Open vSwitch• RabbitMQ• SSL certificates• StackLight (Alertmanager, Elasticsearch, Grafana, Prometheus, Prometheus Relay,

Pushgateway, Salesforce integration)

NoteDue to architecture limitations, StackLight in Mirantis OpenStack on Kubernetes does notsupport Calico monitoring and etcd alerting.

SeealsoOpenStack and StackLight integration



Tungsten Fabric

NoteThis feature is available as technical preview. Use such configuration for testing andevaluation purposes only.

Tungsten Fabric provides basic L2/L3 networking to an OpenStack environment running on theUCP cluster and includes the IP address management, security groups, floating IP addresses, androuting policies functionality. Tungsten Fabric is based on overlay networking, where all virtualmachines are connected to a virtual network with encapsulation (MPLSoGRE, MPLSoUDP,VXLAN). This enables you to separate the underlay Kubernetes management network. Aworkload requires an external gateway, such as a hardware EdgeRouter or a simple gateway toroute the outgoing traffic.The Tungsten Fabric vRouter uses different gateways for the control and data planes.



Tungsten Fabric cluster overviewAll services of Tungsten Fabric are delivered as separate containers, which are deployed by theTungsten Fabric Operator (TFO). Each container has an INI-based configuration file that isavailable on the host system. The configuration file is generated automatically upon thecontainer start and is based on environment variables provided by the TFO through KubernetesConfigMaps.The main Tungsten Fabric containers run with the host network as DeploymentSet, without usingthe Kubernetes networking layer. The services listen directly on the host network interface.The following diagram describes the minimum production installation of Tungsten Fabric withMirantis OpenStack on Kubernetes deployment.



Tungsten Fabric componentsThis section describes the Tungsten Fabric services and their distribution across the MirantisOpenStack on Kubernetes deployment.The Tungsten Fabric services run mostly as DaemonSets in a separate container for eachservice. The deployment and update processes are managed by the Tungsten Fabric operator.However, Kubernetes manages the probe checks and restart of broken containers.The following tables describe the Tungsten Fabric services:

• Configuration and control services in Tungsten Fabric controller containers• Analytics services in Tungsten Fabric analytics containers• vRouter services on the OpenStack compute nodes• Third-party services for Tungsten Fabric• Tungsten Fabric plugin services on the OpenStack controller nodes

Configuration and control services in Tungsten Fabric controller containers

Service name Service descriptionconfig-api Exposes a REST-based interface for the Tungsten Fabric API.config-nodemgr Collects data of the Tungsten Fabric configuration processes and

sends it to the Tungsten Fabric collector.control Communicates with the cluster gateways using BGP and with the

vRouter agents using XMPP, as well as redistributes appropriatenetworking information.

control-nodemgr Collects the Tungsten Fabric controller process data and sendsthis information to the Tungsten Fabric collector.

device-manager Manages physical networking devices using netconf or ovsdb. Inmulti-node deployments, it operates in the active-backup mode.

dns Using the named service, provides the DNS service to the VMsspawned on different compute nodes. Each vRouter nodeconnects to two Tungsten Fabric controller containers that run thedns process.

named The customized Berkeley Internet Name Domain (BIND) daemonof Tungsten Fabric that manages DNS zones for the dns service.

schema Listens to configuration changes performed by a user andgenerates corresponding system configuration objects. Inmulti-node deployments, it works in the active-backup mode.



svc-monitor Listens to configuration changes of service-template andservice-instance, as well as spawns and monitors virtual machinesfor the firewall, analyzer services, and so on. In multi-nodedeployments, it works in the active-backup mode.

webui Consists of the webserver and jobserver services. Provides theTungsten Fabric web UI.

Analytics services in Tungsten Fabric analytics containers

Service name Service descriptionalarm-gen Evaluates and manages the alarms rules.analytics-api Provides a REST API to interact with the Cassandra analytics

database.analytics-nodemgr Collects all Tungsten Fabric analytics process data and sends this

information to the Tungsten Fabric collector.analytics-database-nodemgr

Provisions the init model if needed. Collects data of the databaseprocess and sends it to the Tungsten Fabric collector.

collector Collects and analyzes data from all Tungsten Fabric services.query-engine Handles the queries to access data from the Cassandra database.snmp-collector Receives the authorization and configuration of the physical

routers from the config-nodemgr service, polls the physical routersusing the Simple Network Management Protocol (SNMP), anduploads the data to the Tungsten Fabric collector.

topology Reads the SNMP information from the physical router user-visibleentities (UVEs), creates a neighbor list, and writes the neighborinformation to the physical router UVEs. The Tungsten Fabric webUI uses the neighbor list to display the physical topology.

vRouter services on the OpenStack compute nodes

Service name Service descriptionvrouter-agent Connects to the Tungsten Fabric controller container and the

Tungsten Fabric DNS system using the Extensible Messaging andPresence Protocol (XMPP).

vrouter-nodemgr Collects the supervisor vrouter data and sends it to the TungstenFabric collector.



Third-party services for Tungsten Fabric

Service name Service descriptioncassandra

• On the Tungsten Fabric network nodes, maintains theconfiguration data of the Tungsten Fabric cluster.

• On the Tungsten Fabric analytics nodes, stores the collectorservice data.

kafka Handles the messaging bus and generates alarms across theTungsten Fabric analytics containers.

redis Stores the physical router UVE storage and serves as a messagingbus for event notifications.

zookeeper Holds the active-backup status for the device-manager,svc-monitor, and the schema-transformer services. This service isalso used for mapping of the Tungsten Fabric resources names toUUIDs.

rabbitmq Exchanges messages between API servers and original requestsenders.

Tungsten Fabric plugin services on the OpenStack controller nodes

Service name Service descriptionneutron-server The Neutron server that includes the Tungsten Fabric plugin.



Tungsten Fabric operatorThe Tungsten Fabric operator (TFO) is based on the operator SDK project. The operator SDK is aframework that uses the controller-runtime library to make writing operators easier byproviding:

• High-level APIs and abstractions to write the operational logic more intuitively.• Tools for scaffolding and code generation to bootstrap a new project fast.• Extensions to cover common operator use cases.

The TFO deploys the following sub-operators. Each sub-operator handles a separate part of a TFdeployment:

TFO sub-operators

Network DescriptionTFControl Deploys the Tungsten Fabric control services, such as:

• Control• DNS• Control NodeManager

TFConfig Deploys the Tungsten Fabric configuration services, such as:

• API• Service monitor• Schema transformer• Device manager• Configuration NodeManager• Database NodeManager

TFAnalytics Deploys the Tungsten Fabric analytics services, such as:

• API• Collector• Alarm• Alarm-gen• SNMP• Topology• Alarm NodeManager• Database NodeManager• SNMP NodeManager



TFVrouter Deploys a vRouter on each compute node with the following services:

• vRouter agent• NodeManager

TFWebUI Deploys the following web UI services:

• Web server• Job server

TFTool Deploys the following tools to verify the TF deployment status:

• TF-status• TF-status aggregator

TFTest An operator to run Tempest tests.

Besides the sub-operators that deploy TF services, TFO uses operators to deploy and maintainthird-party services, such as different types of storage, cache, message system, and so on. Thefollowing table describes all third-party operators:

TFO third-party sub-operators

Network Descriptioncasandra-operator

An upstream operator that automates the Cassandra HA storage operationsfor the configuration and analytics data.

zookeeper-operator

An upstream operator for deployment and automation of a ZooKeepercluster.

kafka-operator An operator for the Kafka cluster used by analytics services.redis-operator An upstream operator that automates the Redis cluster deployment and

keeps it healthy.rabbitmq-operator

An operator for the messaging system based on RabbitMQ.

The following diagram illustrates a simplified TFO workflow:





Tungsten Fabric traffic flowThis section describes the types of traffic and traffic flow directions in a Mirantis OpenStack onKubernetes cluster.User interface and API traffic

The following diagram illustrates all types of UI and API traffic in a MOSK cluster, includingthe monitoring and OpenStack API traffic. The OpenStack Dashboard pod hosts Horizon andacts as a proxy for all other types of traffic. TLS termination is also performed for this type oftraffic.

SDN traffic

SDN or Tungsten Fabric traffic goes through the overlay Data network and processeseast-west and north-south traffic for applications that run in a MOSK cluster. This networksegment typically contains tenant networks as separate MPLS-over-GRE and MPLS-over-UDPtunnels. The traffic load depends on the workload.The control traffic between the Tungsten Fabric controllers, edge routers, and vRouters usesthe XMPP with TLS and iBGP protocols. Both protocols produce low traffic that does not affectMPLS over GRE and MPLS over UDP traffic. However, this traffic is critical and must bereliably delivered. Mirantis recommends configuring higher QoS for this type of traffic.The following diagram displays both MPLS over GRE/MPLS over UDP and iBGP and XMPPtraffic examples in a MOSK cluster:





Tungsten Fabric vRouterThe Tungsten Fabric vRouter provides data forwarding to an OpenStack tenant instance andreports statistics to the Tungsten Fabric analytics service. The Tungsten Fabric vRouter isinstalled on all OpenStack compute nodes. Mirantis OpenStack on Kubernetes supports thekernel-based deployment of the Tungsten Fabric vRouter.The vRouter agent acts as a local control plane. Each Tungsten Fabric vRouter agent isconnected to at least two Tungsten Fabric controllers in an active-active redundancy mode. TheTungsten Fabric vRouter agent is responsible for all networking-related functions includingrouting instances, routes, and so on.The Tungsten Fabric vRouter uses different gateways for the control and data planes. Forexample, the Linux system gateway is located on the management network, and the TungstenFabric gateway is located on the data plane network.The following diagram illustrates the Tungsten Fabric kernel vRouter setup by the TF operator:

On the diagram above, the following types of networks interfaces are used:

• eth0 - for the management (PXE) network (eth1 and eth2 are the slave interfaces of Bond0)• Bond0.x - for the UCP control plane network• Bond0.y - for the UCP data plane network

SeealsoOpenStack and Tungsten Fabric controllers integration



Documents

MOSK Reference Architecture Reference... · Introduction Mirantis OpenStack on Kubernetes leverages the container isolation, state enforcement, and declarative definition of resources