Datera Kubernetes CSI Driver Deployment Guide

DEPLOYMENT GUIDEKUBERNETES CSI DRIVER

1GET A FREE CONSULTATION. Contact Us | Visit datera.io | Email [email protected]

©2020 Datera, Inc. All Rights Reserved. Datera is a trademark of Datera, Inc. All other trademarks belong to their respective owners. Date: July 2020

Datera Kubernetes CSI Driver Deployment Guide

Contents1. Introduction to Datera and the Kubernetes CSI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2. Datera CSI driver implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3. Kubernetes Volume basics: StorageClasses, PVs, PVCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5. Datera CSI driver - Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6. Datera CSI driver - Troubleshooting and Log collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

7. Datera CSI driver - Support Matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

8. Disaster recovery, Node failures, Node tainting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

9. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

9.1 Creating a volume dynamically and attaching to a App Pod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

9.2 Volume snapshot creation / deletion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

9.3 Datera volume Template override . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

9.4 Volume attachment to Application Pods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

9.5 Volume attachment to Application Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

9.6 Volumes for Statefulset pods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

9.7 Volume resizing or expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

9.8 Volume retention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

9.9 Multi-tenancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

9.10 Driver upgrade and downgrade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

10. References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

JULY 2020

https://datera.io/contact-us/

http://www.datera.io

mailto:sales%40datera.io?subject=My%20Free%20Consultation




1. Introduction to Datera and the Kubernetes CSIDatera is a fully disaggregated scale-out storage platform, that runs over multiple standard protocols (iSCSI, Object/S3), combining both heterogeneous compute platform/framework flexibility (HPE, Dell, Fujitsu, Cisco and others) with rapid deployment velocity and access to data from anywhere.

Datera gives Kubernetes (K8s) enterprise customers the peace of mind of a future-proof data services platform that is ready for diverse and demanding workloads — as K8s continues to dominate the container orchestration arena, it is likely to containerize higher-end workloads, as well.

The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. Using CSI third-party storage providers, such as Datera, can write and deploy plugins exposing new storage systems in Kubernetes without ever having to touch the core Kubernetes code.

Datera’s CSI driver deeply integrates with the K8s runtime. It allows deploying entire stateful multi-site K8s clusters with a single K8s command, and pushing application-specific telemetry to the Datera policy engine, so that it can intelligently adapt the data fabric. Datera’s powerful storage classes, and policy driven workloads are a natural fit with Kubernetes, and our deep CSI integration will be covered in this paper.

K8s Concept Datera Concept

Manifests Templates + CSI driverDeclarative policy (intents) and telemetry (operationalization)Label-based provisioning with seamless integration in K8s manifests

Namespaces TenancyGovernance (operationalization of policy)Single authentication/access/quota mechanism

Quotas Tenancy + QuotasFine-grained controls at tenant and volume level for sandboxing storageContainment for noisy neighbors and rogue resource scalingMakes K8s more safely consumable

Resource Pools“Tainting”

Tenancy + Resource PoolsAbility to restrict media placement to a subset of nodes/resources

Storage Classes Application Classes and Instances + Live Data MobilityJust-in-time non-disruptive resource provisioning, driven by policy:

• No application downtime• No need to respin pods• No need to recreat PVs/PVCs

Live policy (label) changes in AppClasses and/or AppInstances




https://github.com/container-storage-interface/spec/blob/master/spec.md





Datera provides IT a private/hybrid cloud data platform to consolidate both traditional enterprise, bare metal, virtualized and modern cloud-native workloads.

IT operators gain the flexibility to plan, deploy and scale their compute resources independently from their Datera storage resources, while application owners can self-service and consume infrastructure as they go.

K8s Concept Datera Concept

Consistency Groups Application Classes and InstancesSupport for consistency groups at application level (incl. across pods)

Persistent VolumesPersistent Volume Claims

Live Resource Thin ProvisioningNo resource pre-/over-provisioning and caching to placate ops discontinuitiesbetween K8s and storage provider







2. Datera CSI driver implementation The CSI specs (https://github.com/container-storage-interface/spec/blob/master/spec.md) define the boundary between K8S and a CSI Plugin. Datera CSI Plugin is divided into 2 parts.

• Controller plugin• Node plugin

Datera CSI driver implements these plugins along with an Identity service as a single gRPC server. All communication between Kubernetes and the CSI driver happens through well defined Unix Domain Sockets on the nodes. The driver implements all 3 services (Controller, Node and Identity services) in a single binary named ‘dat-csi-plugin’. The corresponding docker image is made available on https://hub.docker.com/repository/docker/dateraiodev/dat-csi-plugin. The Node plugin is deployed as a DaemonSet so that a copy of Node plugin runs on all worker nodes. The Controller plugin is deployed as a StatefulSet with replicas = 1 so that a single copy of provisioner runs on any node in the cluster. The same “dat-csi-plugin” image is used for deploying both DaemonSet and StatefulSet. Check the implementation diagram below.





https://hub.docker.com/repository/docker/dateraiodev/dat-csi-plugin

https://hub.docker.com/repository/docker/dateraiodev/dat-csi-plugin




Here is a more detailed view of how Kubernetes communicates with Datera CSI driver:

The Datera CSI driver code (in Green boxes shown above) is written in Golang and is available under /pkg directory: https://github.com/Datera/datera-csi/tree/master/pkg/driver

The sidecar container images (in Red boxes shown above) are given by Kubernetes CSI community.

The entire set of code necessary for interacting with Datera backend such as login, logout, create volume, delete volume, create snapshot, etc are implemented in Golang SDK which is located here: https://github.com/Datera/go-sdk/tree/master/pkg/dsdk

The driver is installed on a functional Kubernetes cluster by running “kubectl create -f <datera_csi_driver_yaml>” on the Master node. There are certain HW and SW requirements with respect to the Kubernetes master and worker nodes, the iSCSI package availability, etc., to get the driver up and going. This will be detailed in subsequent sections.




https://github.com/Datera/datera-csi/tree/master/pkg/driver

https://github.com/Datera/go-sdk/tree/master/pkg/dsdk




3. Kubernetes Volume basics: StorageClasses, PVs, PVCsPersistent Volumes (PV) and Persistent Volume Claims (PVC) relieves the Application users from knowing anything about the underlying Storage technologies. Note that PVs are cluster-scoped and PVCs are namespace-scoped. The PVs are created dynamically when a PVC claim is submitted to Kubernetes API which in turn calls Datera CSI driver. Datera CSI driver does dynamic provisioning of volumes on Datera cluster nodes. Here is a brief background on manual vs dynamic provisioning.

In manual provisioning, a kubernetes storage administrator would pre-create persistent volumes and make it available for all tenants in the cluster. The persistent volumes could be backed by any of public or private cloud providers. In such cases, an Application user submits a PVC claim referencing a particular PV created by storage administrator and the volume is made available inside the Pods. However, this method binds the Application users to know underlying storage volumes. To solve this problem, Kubernetes provides StorageClasses to dynamically provision persistent volumes.

In dynamic provisioning, storage administrators would create StorageClasses which lets the Application users select the type of persistent storage they want. Every time a PVC claim is submitted, a corresponding PV is dynamically created using a volume provisioner, such as Datera CSI provisioner. With this method, users do not need to know how many and what type of persistent volumes are available in the cluster. Kubernetes will take care of mapping a PVC claim to a certain PV that best matches the storage parameters. Application pods managed by a replication controller ensures that the storage follows the application pods during the pod lifecycle.

1 www.linux-iscsi.org Linux-IO Target (LIO™) is the standard open-source iSCSI target in Linux. It supports all prevalent storage fabrics, including Fibre Channel, FCoE, IEEE 1394, iSCSI, NVMe-OF, iSER, SRP, USB, vHost, etc.







4. Datera CSI driver and K8S RequirementsFrom a network standpoint, the Kubernetes Master node must have IP reachability to Datera Management VIP which was made available to users when the Datera backend system was brought up.

[root@ch3cp ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

0.0.0.0 172.58.4.1 0.0.0.0 UG 102 0 0 eth0

...

[root@ch3cp ~]#

[root@ch3cp ~]# ping 172.58.6.29 -c 3 <-- Datera MGMT VIP

PING 172.58.6.29 (172.58.6.29) 56(84) bytes of data.

64 bytes from 172.58.6.29: icmp _ seq=1 ttl=64 time=0.188 ms



--- 172.58.6.29 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 1999ms

rtt min/avg/max/mdev = 0.165/0.181/0.192/0.019 ms

[root@ch3cp ~]#

Similarly, the kubernetes worker nodes must have IP reachability to Datera Access VIPs.

[root@ch1cp ~]# route -n

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

...

172.129.84.0 172.127.0.0 255.255.255.0 UG 107 0 0 eth2

172.129.85.0 172.127.4.0 255.255.255.0 UG 108 0 0 eth3

...

[root@ch1cp ~]#

[root@ch1cp ~]# ping 172.129.84.4 -c 3 <-- Access VIP1





--- 172.129.84.4 ping statistics ---



[root@ch1cp ~]#







[root@ch1cp ~]# ping 172.129.85.4 -c 3 <-- Access VIP2





--- 172.129.85.4 ping statistics ---



[root@ch1cp ~]#

From a Kubernetes standpoint, following must have been addressed and verified prior to driver installation. Most of them are not a requirement for installing Datera CSI driver itself but will ensure smooth installation and working. These must have been taken care of at the time of Kubernetes installation. Including here for completeness.

• Verify necessary packages exist, such as kubeadm, kubelet, kubelet, iscsi-initiator-utils / open-iscsi, docker, etc.• Ensure a Pod network such as Calico, Flannel, etc, is installed on the K8S cluster.• Verify multipath package (device-mapper-multipath) is installed and enabled on nodes.• Disable the firewall daemon and/or SELinux, and enable IPtables for IPv4 on all nodes.• Ensure the coredns deployment is patched in kube-system namespace with “allowPrivilegeEscalation”:

true and remove the line in coredns configmap which has ‘loop’ in it. This will avoid coredns pods from crashing.

• Ensure kubelet config has enableControllerAttachDetach: true. This is needed for CSI.• Enable the following feature gates on kube-apiserver. • VolumeSnapshotDataSource: true• ExpandCSIVolumes: true• ExpandInUsePersistentVolumes: true• Worker nodes have iscsiadm installed and can perform login to Datera target VIPs.

From the iSCSI standpoint, container-based scsi is no longer supported. The iSCSI daemon must be running on the worker nodes prior to installing the Datera CSI plugin/driver. If it is not running, check whether iscsi-initiator-utils / open-iscsi package is installed based on your distribution.

$ ps -ef | grep iscsid

root 12494 996 0 09:41 pts/2 00:00:00 grep --color=auto iscsid

root 13326 1 0 Dec17 ? 00:00:01 /sbin/iscsid

root 13327 1 0 Dec17 ? 00:00:05 /sbin/iscsid







Datera CSI employs a host-based scsi solution, in which an iscsi-send binary is placed inside the csi-node driver pod and the iscsi-recv binary is run as a service on the host or worker node. The iscsi-recv can be run as a service as shown below.

$ git clone http://github.com/Datera/datera-csi

$ ./assets/setup _ iscsi.sh

[INFO] Dependency checking

[INFO] Downloading iscsi-recv

[INFO] Verifying checksum

[INFO] Changing file permissions

[INFO] Registering iscsi-recv service

Created symlink from /etc/systemd/system/multi-user.target.wants/iscsi-recv.service to /lib/systemd/

system/iscsi-recv.service.

[INFO] Starting iscsi-recv service

[INFO] Verifying service started correctly

root 4879 1 0 19:50 ? 00:00:00 /var/datera/iscsi-recv -addr unix:////var/datera/csi-iscsi.sock

$ systemctl --all | grep iscsi-recv

iscsi-recv.servicevloaded active running iscsi-recv container to host iscsiadm adapter service

The iscsi commands that are executed inside the driver pod are intercepted by iscsi-send program and sent to iscsi-recv program running on host through a UDC socket. The iscsi-recv would further depend on iscsi daemon on host for logins and sessions maintenance.







5. Datera CSI driver - InstallationThe driver installation is controlled by a yaml file that Datera provides. Download the latest yaml file from https://github.com/Datera/datera-csi/tree/master/deploy/kubernetes/release/1.0. At the time of this writing, v1.0.9 is the latest version, hence pick up csi-datera-secrets-1.0.9.yaml file. Check the README available at https://github.com/Datera/datera-csi for supported versions.

There are 2 yaml files for each Datera CSI driver version. One needs datera backend login credentials provided in clear text and the other needs login credentials provided as kubernetes secrets. If you decide to use the yaml which references secrets, then you must create the secrets prior to running the driver installation file.

- name: DAT _ USER

valueFrom:

secretKeyRef:

name: datera-secret

key: username

- name: DAT _ PASS

valueFrom:

secretKeyRef:

name: datera-secret

key: password

5.1. Create Secret

[root@ch3cp ~]# cat /tmp/csi-storage-secret.yaml

apiVersion: v1

kind: Secret

metadata:

name: datera-secret

namespace: kube-system

type: Opaque

data:

# base64 encoded username

# generate this via “$ echo -n ‘your-username’ | base64”

username: YWRtaW4=

# base64 encoded password

# generate this via “$ echo -n ‘your-password’ | base64”

password: cGFzc3dvcmQ=

[root@ch3cp ~]#

[root@ch3cp ~]# kubectl create -f /tmp/csi-storage-secret.yaml

secret/datera-secret created




https://github.com/Datera/datera-csi/tree/master/deploy/kubernetes/release/1.0

https://github.com/Datera/datera-csi




[root@ch3cp ~]#

[root@ch3cp ~]# kubectl get secret datera-secret -n kube-system

NAME TYPE DATA AGE

datera-secret Opaque 2 3h39m

[root@ch3cp ~]#

Now install the Datera CSI driver.

5.2. Install Datera CSI driver

[root@ch3cp ~]# kubectl create -f /tmp/csi-datera-secrets-1.0.9.yaml

storageclass.storage.k8s.io/dat-block-storage created

serviceaccount/csi-datera-controller-sa created

clusterrole.rbac.authorization.k8s.io/csi-datera-provisioner-role created

clusterrolebinding.rbac.authorization.k8s.io/csi-datera-provisioner-binding created

clusterrole.rbac.authorization.k8s.io/csi-datera-attacher-role created

clusterrolebinding.rbac.authorization.k8s.io/csi-datera-attacher-binding created

clusterrole.rbac.authorization.k8s.io/csi-datera-snapshotter-role created

clusterrolebinding.rbac.authorization.k8s.io/csi-datera-snapshotter-binding created

statefulset.apps/csi-provisioner created

serviceaccount/csi-datera-node-sa created

clusterrole.rbac.authorization.k8s.io/csi-datera-node-driver-registrar-role created

clusterrolebinding.rbac.authorization.k8s.io/csi-datera-node-driver-registrar

binding created

daemonset.apps/csi-node created

[root@ch3cp ~]#

[root@ch3cp ~]# kubectl get pod -n kube-system -o wide | grep -E ‘NAME|csi-’

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

csi-node-vzwzz 3/3 Running 1 20m 172.58.4.72 ch13cp <none> <none>

csi-node-wt6k2 3/3 Running 0 20m 172.58.4.16 ch1cp <none> <none>

csi-provisioner-0 6/6 Running 0 20m 172.58.4.72 ch13cp <none> <none>

[root@ch3cp ~]#

Ensure that there is 1 csi-provisioner pod and ‘N’ number of csi-node pods in the kube-system namespace, where N = number of worker nodes. There are multiple sidecar containers in each of the Pod; each responsible for a different function such as volume snapshotting, volume resizing, volume attach/detach, etc.

The csi-provisioner and csi-node pods receives and responds to gRPC calls from Kubernetes. These gRPC calls are interpreted and translated to REST API calls to Datera storage backend by datera go-sdk. It is expected that K8S worker nodes are capable of performing iscsiadm logins to their Datera Targets.







CSI provisioner and node pods are installed in the kube-system namespace and any network policies that restrict traffic in and out of namespaces would also apply to CSI driver pods. Secondly, there is no node affinity specified for CSI provisioner pod (this is per CSI spec), which means that the provisioner pod can land up on either Master or Worker nodes. If colocation of CSI driver pods along with other application pods is undesired, then use Kubernetes scheduling policy features such as Node taints, cordoning, Pod anti-affinities, etc and/or modify the StatefulSet Pod spec in the CSI driver installation file (for ex: csi-datera-1.0.8.yaml).

DO NOT change the number of replicas (set to 1) on the StatefulSet controller. Only one CSI provisioner pod is expected to run per kubernetes cluster. This is again per CSI spec, for the reasons that running multiple CSI provisioner pods behind a load balancing service might end up provisioning or mounting the same volume at the same time and that can cause failures. are interpreted and translated to REST API calls to Datera storage backend by datera go-sdk. It is expected that K8S worker nodes are capable of performing iscsiadm logins to their Datera Targets.

CSI provisioner and node pods are installed in the kube-system namespace and any network policies that restrict traffic in and out of namespaces would also apply to CSI driver pods. Secondly, there is no node affinity specified for CSI provisioner pod (this is per CSI spec), which means that the provisioner pod can land up on either Master or Worker nodes. If colocation of CSI driver pods along with other application pods is undesired, then use Kubernetes scheduling policy features such as Node taints, cordoning, Pod anti-affinities, etc and/or modify the StatefulSet Pod spec in the CSI driver installation file (for ex: csi-datera-1.0.8.yaml).

DO NOT change the number of replicas (set to 1) on the StatefulSet controller. Only one CSI provisioner pod is expected to run per kubernetes cluster. This is again per CSI spec, for the reasons that running multiple CSI provisioner pods behind a load balancing service might end up provisioning or mounting the same volume at the same time and that can cause failures.







6. Datera CSI driver - Troubleshooting and Log collectionWhenever a problem is encountered such as volume provisioning or attach failures, snapshot failures, etc., collect the Datera CSI driver logs as shown below. The csi_log_collect.sh is available at https://github.com/Datera/datera-csi/tree/master/assets. When the script is executed, it will produce a tar.gz file with all the logs from datera CSI driver pods.

[root@ch3cp tmp]# ./csi _ log _ collect.sh

[INFO] Dependency checking

-P, --perl-regexp PATTERN is a Perl regular expression

[INFO] Collecting CSI logs

[INFO] Collecting for pod: csi-node-vzwzz

[INFO] Saving container logfile: node-driver-registrar

[INFO] Saving container logfile: dat-csi-plugin-node

[INFO] Saving container logfile: liveness-probe

[INFO] Collecting for pod: csi-node-wt6k2

[INFO] Saving container logfile: node-driver-registrar

[INFO] Saving container logfile: dat-csi-plugin-node


[INFO] Collecting for pod: csi-provisioner-0

[INFO] Saving container logfile: csi-provisioner

[INFO] Saving container logfile: csi-attacher

[INFO] Saving container logfile: csi-resizer

[INFO] Saving container logfile: csi-snapshotter

[INFO] Saving container logfile: dat-csi-plugin-controller


[INFO] Creating archive: /tmp/csi-logs-ch3cp-gIVcEz.tar.gz

[INFO] Archive size: 301K

[root@ch3cp tmp]#




https://github.com/Datera/datera-csi/tree/master/assets




7. Datera CSI driver - Support Matrix

Datera CSI Plugin Version Supported CSI Version Supported Kubernetes Versions

v1.0.4 v1.0 v1.13.X+

v1.0.5 v1.0 v1.13.X+

v1.0.6 v1.0 v1.13.X+

v1.0.7 v1.0 v1.13.X+

v1.0.8 v1.0 v1.13.X+

v1.0.9 v1.0 v1.13.X+







8. Disaster recovery, Node failures, Node taintingDatera CSI driver pods (csi-provisioner and csi-node) are protected by Kubernetes node failure detection and recovery mechanisms. If the node carrying a driver pod fails, kubernetes will re-spawn the pod on another node. This is because the driver pods are managed by a replication controller. The csi-provisioner pod (although a single pod) is controlled by a StatefulSet and the csi-node pod is controlled by a DaemonSet.

If a csi-provisioner pod is dead, the kubernetes StatefulSet controller would bring up a new pod in a certain time (typically seconds). There is a possibility that a request for volume provisioning (create, delete, etc) would not be serviced by the CSI driver. In such cases, Kubernetes would make retry attempts to establish communication with the csi-provisioner pod.

Similarly, if a worker node dies and stays down, then the Kubernetes DaemonSet controller would kick in and try to establish communication with the csi-node pod. After a timeout, the node would be cordoned off for scheduling purposes and no volumes (old or new) will be available on that node. After the worker node is back online, the DaemonSet will ensure a csi-node pod will run on that node.

Note that there is no affinity set for csi-provisioner pod under the Statefulset and therefore it can be spawned on any of the master or worker nodes. This is per CSI design spec. The csi-node pods are spawned on worker nodes only, using a DaemonSet.

Node affinity, tainting, tolerations and cordoning must be carefully handled on the cluster. They affect the scheduling and placement of driver pods, just like any other application pods.







9. Examples https://github.com/Datera/datera-csi/tree/master/deploy/examples

9.1 Creating a volume dynamically and attaching to a App PodYou would need a StorageClass, a PVC claim and a Pod manifest as shown below.

9.1.1. Create the StorageClass

[root@ch3cp tmp]# cat csi-sc-xfs-no-args.yaml

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

name: csi-sc-xfs-no-args


annotations:

storageclass.kubernetes.io/is-default-class: “true”

provisioner: dsp.csi.daterainc.io

parameters:

replica _ count: “1”

iops _ per _ gb: “5000”

bandwidth _ per _ gb: “5000”

placement _ mode: “hybrid”

round _ robin: “false”

ip _ pool: “default”

template: “”

disable _ template _ override: “true”

read _ iops _ max: “1000”

write _ iops _ max: “2000”

total _ iops _ max: “5000”

read _ bandwidth _ max: “1000”

write _ bandwidth _ max: “2000”

total _ bandwidth _ max: “5000”

csi.storage.k8s.io/fstype: xfs

delete _ on _ unmount: “false”

allowVolumeExpansion: true

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-sc-xfs-no-args.yaml

storageclass.storage.k8s.io/csi-sc-xfs-no-args created

[root@ch3cp tmp]#




https://github.com/Datera/datera-csi/tree/master/deploy/examples




[root@ch3cp tmp]# kubectl get sc -n kube-system

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE

csi-sc-xfs-no-args (default) dsp.csi.daterainc.io Delete Immediate true 27s

[root@ch3cp tmp]#

9.1.2. Create a PVC

[root@ch3cp tmp]# cat csi-pvc-xfs-no-args.yaml

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

name: csi-pvc-xfs-no-args

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 1Gi

storageClassName: csi-sc-xfs-no-args

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-pvc-xfs-no-args.yaml

persistentvolumeclaim/csi-pvc-xfs-no-args created

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

csi-pvc-xfs-no-args Bound pvc-66ff90a3-582a-4e02-a67e-6380e8f81d93 1Gi RWO csi-sc-xfs-no-args 12s

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE

pvc-66ff90a3-582a-4e02-a67e-6380e8f81d93 1Gi RWO Delete Bound default/csi-pvc-xfs-no-args

csi-sc-xfs-no-args 13s

[root@ch3cp tmp]#

9.1.3. Create an App pod which claims the PVC

[root@ch3cp tmp]# cat csi-app-xfs-no-args.yaml

kind: Pod







apiVersion: v1

metadata:

name: csi-app-xfs-no-args

spec:

containers:

- name: my-app-image

image: alpine

volumeMounts:

- mountPath: “/data”

name: my-app-volume

command: [ “sleep”, “1000000” ]

volumes:

- name: my-app-volume

persistentVolumeClaim:

claimName: csi-pvc-xfs-no-args

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-app-xfs-no-args.yaml

pod/csi-app-xfs-no-args created

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl get pod -o wide


csi-app-xfs-no-args 1/1 Running 0 15s 192.168.133.69 ch13cp <none> <none>

[root@ch3cp tmp]#



csi-app-xfs-no-args 1/1 Running 0 15s 192.168.133.69 ch13cp <none> <none>

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl describe pvc csi-pvc-xfs-no-args

Name: csi-pvc-xfs-no-args

Namespace: default

StorageClass: csi-sc-xfs-no-args

Status: Bound

Volume: pvc-66ff90a3-582a-4e02-a67e-6380e8f81d93

Labels: <none>

Annotations: pv.kubernetes.io/bind-completed: yes

pv.kubernetes.io/bound-by-controller: yes

volume.beta.kubernetes.io/storage-provisioner: dsp.csi.daterainc.io

Finalizers: [kubernetes.io/pvc-protection]

Capacity: 1Gi

Access Modes: RWO

VolumeMode: Filesystem







Mounted By: csi-app-xfs-no-args

Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal ExternalProvisioning 115s persistentvolume-controller

waiting for a volume to be created, either by external provisioner “dsp.csi

daterainc.io” or manually created by system administrator

Normal Provisioning 115s dsp.csi.daterainc.io _ ch13cp _ 72f80dc5-

7940-11ea-91a0-003048f5c0bc External provisioner is provisioning volume for claim

“default/csi-pvc-xfs-no-args”

Normal ProvisioningSucceeded 114s dsp.csi.daterainc.io _ ch13cp _ 72f80dc5-7940

11ea-91a0-003048f5c0bc Successfully provisioned volume pvc-66ff90a3-582a-4e02-a67e

6380e8f81d93

[root@ch3cp tmp]#

9.2 Volume snapshot creation / deletion

9.2.1 Create a VolumeSnapshotClass

[root@ch3cp tmp]# cat csi-snapshot-class.yaml

apiVersion: snapshot.storage.k8s.io/v1alpha1

kind: VolumeSnapshotClass

metadata:

name: csi-snap-class

snapshotter: dsp.csi.daterainc.io

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-snapshot-class.yaml

volumesnapshotclass.snapshot.storage.k8s.io/csi-snap-class created

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl get volumesnapshotclass

NAME AGE

csi-snap-class 2m10s

[root@ch3cp tmp]#







9.2.2. Create Volume Snapshot

[root@ch3cp tmp]# cat csi-snapshot.yaml

apiVersion: snapshot.storage.k8s.io/v1alpha1

kind: VolumeSnapshot

metadata:

name: csi-snap

spec:

snapshotClassName: csi-snap-class

source:

name: csi-pvc


[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-snapshot.yaml

volumesnapshot.snapshot.storage.k8s.io/csi-snap created

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl get volumesnapshot

NAME AGE

csi-snap 30s

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl describe volumesnapshot csi-snap

Name: csi-snap

Namespace: default

Labels: <none>

Annotations: <none>

API Version: snapshot.storage.k8s.io/v1alpha1

Kind: VolumeSnapshot

Metadata:

Creation Timestamp: 2020-04-08T02:54:58Z

Finalizers:

snapshot.storage.kubernetes.io/volumesnapshot-protection

Generation: 5

Managed Fields:


Fields Type: FieldsV1

fieldsV1:

f:spec:

.:

f:snapshotClassName:

f:source:

.:







f:kind:

f:name:

Manager: kubectl

Operation: Update

Time: 2020-04-08T02:54:58Z


Fields Type: FieldsV1

fieldsV1:

f:metadata:

f:finalizers:

f:spec:

f:snapshotContentName:

f:source:

f:apiGroup:

f:status:

.:

f:creationTime:

f:readyToUse:

f:restoreSize:

Manager: csi-snapshotter

Operation: Update

Time: 2020-04-08T02:55:00Z

Resource Version: 57899

Self Link:

/apis/snapshot.storage.k8s.io/v1alpha1/namespaces/default

volumesnapshots/csi-snap

UID: 31cc778a-e5f7-4f4b-87c7-72391419e365

Spec:

Snapshot Class Name: csi-snap-class

Snapshot Content Name: snapcontent-31cc778a-e5f7-4f4b-87c7-72391419e365

Source:

API Group: <nil>

Kind: PersistentVolumeClaim

Name: csi-pvc

Status:

Creation Time: 2020-04-08T02:54:58Z

Ready To Use: true

Restore Size: 10Gi

Events: <none>

[root@ch3cp tmp]#







9.2.3 Create a PVC from a Volume Snapshot

[root@ch3cp tmp]# cat csi-pvc-from-snapshot.yaml

apiVersion: v1


metadata:

name: csi-pvc-restore

namespace: default

spec:

storageClassName: dat-block-storage

dataSource:

name: csi-snap

kind: VolumeSnapshot

apiGroup: snapshot.storage.k8s.io

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 1Gi

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-pvc-from-snapshot.yaml

persistentvolumeclaim/csi-pvc-restore created

[root@ch3cp tmp]#



csi-pvc Bound pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0 1Gi RWX dat-block-storage 2m58s

csi-pvc-restore Bound pvc-e6ff26b3-2a3a-4df9-a0bd-e6cb1a6eec36 1Gi RWO dat-block-storage 5s

[root@ch3cp tmp]#

9.2.4 Attach the PVC created from a Volume Snapshot to an App Pod

[root@ch3cp tmp]# cat csi-app-from-snapshot.yaml

kind: Pod

apiVersion: v1

metadata:

name: my-csi-app-restore

spec:

containers:








image: alpine

volumeMounts:


name: my-app-volume

command: [ “sleep”, “1000000” ]

volumes:



claimName: csi-pvc-restore

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-app-from-snapshot.yaml

pod/my-csi-app-restore created

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl get pod

NAME READY STATUS RESTARTS AGE

my-csi-app 1/1 Running 0 5m8s

my-csi-app-restore 1/1 Running 0 15s

[root@ch3cp tmp]#

9.3 Datera volume Template overrideDatera provides storage templates that can be referenced and overridden using Kubernetes Storage class. In this example, the following parameters of the “basic_small” Datera template are overridden.

Template: basic_small

Parameter name Default value Overridden value

replica_count 2 1

placement_mode hybrid default

ip_pool default test

9.3.1. Create a StorageClass and override the template parameters

[root@ch5cp tmp]# cat csi-sc-small-template-override.yaml

kind: StorageClass








metadata:

name: small-template-override


annotations:



parameters:


placement _ mode: “default”

round _ robin: “false”

ip _ pool: “test”

template: “basic _ small”

disable _ template _ override: “false”

fs _ type: “ext4”

fs _ args: “-E lazy _ itable _ init=0,lazy _ journal _ init=0,nodiscard -F”

delete _ on _ unmount: “false”


[root@ch5cp tmp]#

[root@ch5cp tmp]# kubectl create -f csi-sc-small-template-override.yaml

storageclass.storage.k8s.io/small-template-override created

[root@ch5cp tmp]#

[root@ch5cp tmp]# kubectl get sc

NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE

small-template-override (default) dsp.csi.daterainc.io Delete Immediate true 26s

[root@ch5cp tmp]#







9.4 Volume attachment to Application PodsWith Datera CSI driver, the PVs are created dynamically whenever an Application is created with a PVC claim. The volume attachment is automatically handled during the creation of the Application Pod and the volume is available at the mount point specified in the Pod yaml manifest, for example.

Datera CSI driver would automatically perform a filesystem format based on the specification given in the StorageClass right after a Persistent volume is created. Formatting is done by the CSI driver at the time of volume provisioning and not during volume attachment.

In CSI terms, volume provisioning will mount the volume to a Staging path on the worker node (this is called Node staging). And volume attachment will mount the volume from staging path to the given target path inside the App pod (this is called Node publishing). Filesystem formatting happens during the Node staging phase.

9.4.1. Create a PVC

[root@ch3cp tmp]# cat csi-pvc.yaml

apiVersion: v1


metadata:

name: csi-pvc

spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 1Gi


[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-pvc.yaml

persistentvolumeclaim/csi-pvc created

[root@ch3cp tmp]#



csi-pvc Bound pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0 1Gi RWX dat-block-storage 15s



pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0 1Gi RWX Delete Bound default/csi-pvc dat-block-storage 41m

[root@ch3cp tmp]#







9.4.2. Create an Application Pod claiming the PVC

[root@ch3cp tmp]# cat csi-app.yaml

kind: Pod

apiVersion: v1

metadata:

name: my-csi-app

spec:

containers:


image: alpine

volumeMounts:


name: my-app-volume

command: [ “sleep”, “1000000” ]

volumes:



claimName: csi-pvc

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-app.yaml

pod/my-csi-app created

[root@ch3cp tmp]#



my-csi-app 1/1 Running 0 11s

[root@ch3cp tmp]#

9.4.3. Check the PVC is mounted by a Pod

[root@ch3cp tmp]# kubectl describe pvc csi-pvc

Name: csi-pvc

Namespace: default







StorageClass: dat-block-storage

Status: Bound

Volume: pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0

Labels: <none>





Capacity: 1Gi

Access Modes: RWX


Mounted By: my-csi-app

Events:


---- ------ ---- ---- -------

Normal ExternalProvisioning 39m persistentvolume-controller



Normal Provisioning 39m dsp.csi.daterainc.io _ ch12cp _ ab1d1489

79ed-11ea-b72f-003048f5c0b4 External provisioner is provisioning volume for claim

“default/csi-pvc”

Normal ProvisioningSucceeded 39m dsp.csi.daterainc.io _ ch12cp _ ab1d1489

79ed-11ea-b72f-003048f5c0b4 Successfully provisioned volume pvc-e4ce8660-1d85-4407

8bbc-1eda48b162a0

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl describe pv pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0

Name: pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0

Labels: <none>

Annotations: pv.kubernetes.io/provisioned-by: dsp.csi.daterainc.io

Finalizers: [kubernetes.io/pv-protection]


Status: Bound

Claim: default/csi-pvc

Reclaim Policy: Delete

Access Modes: RWX


Capacity: 1Gi

Node Affinity: <none>

Message:







Source:

Type: CSI (a Container Storage Interface (CSI) volume source)

Driver: dsp.csi.daterainc.io

FSType: ext4

VolumeHandle: CSI-pvc-e4ce8660-1d85-4407-8bbc-1eda48b162a0 <-- Datera

Volume

ReadOnly: false

VolumeAttributes: storage.kubernetes.io/csiProvisionerIdent

ty=1586387207322-8081-dsp.csi.daterainc.io

Events: <none>

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl exec -it my-csi-app -- df -Th | egrep ‘Filesystem|data’

Filesystem Type Size Used Available Use% Mounted on

/dev/mapper/36001405186c9c958a344beaaa252e1e3

ext4 975.9M 2.5M 906.2M 0% /data

[root@ch3cp tmp]#

9.5 Volume attachment to Application DeploymentDeployments are used for Stateless applications. Deployments can share an existing PV using ReadOnlyMany and ReadWriteMany access modes. ReadWriteOnly mode is NOT recommended for deployments because the replica pods need to attach and reattach to PV dynamically. If the first pod needs to be detached, the second pod needs to be attached first. However the second pod cannot attach to the PV because the first pod is still attached. This creates a deadlock scenario and neither pod can make progress. StatefulSets can be used to resolve this deadlock.

Note that although Kubernetes allows a single PV in ReadWriteMany mode to be attached to multiple replicas of deployment, the Application owner must take extreme care in allowing this shared Read/Writes to happen to Datera volumes. Some kind of database locking mechanism must be used and its implications must be fully understood.

A quick note on AccessModes in Kubernetes:

• RWO—ReadWriteOnce—Only a single node can mount the volume for reading and writing. • ROX—ReadOnlyMany—Multiple nodes can mount the volume for reading.• RWX—ReadWriteMany—Multiple nodes can mount the volume for both reading and writing.

Note that these access modes RWO, ROX, and RWX pertain to the number of worker nodes that can use the volume at the same time, not to the number of pods!







9.5.1. Create a shared PVC and PV for deployment

[root@ch3cp tmp]# cat csi-pvc-deployment.yaml

apiVersion: v1


metadata:

name: csi-pvc-deployment

spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 50Gi


[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-pvc-deployment.yaml

persistentvolumeclaim/csi-pvc-deployment created

[root@ch3cp tmp]#



csi-pvc-deployment Bound pvc-2972301a-a580-463c-8c53-6906a783dffb 50Gi RWX dat-block-storage 7m55s

[root@ch3cp tmp]#



pvc-2972301a-a580-463c-8c53-6906a783dffb 50Gi RWX Delete Bound default/csi-pvc-deployment

dat-block-storage 8m5s

[root@ch3cp tmp]#

9.5.2. Create the Deployment

[root@ch3cp tmp]# cat csi-app-deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

name: csi-app-deployment

labels:

app: datera

purpose: automation

spec:

replicas: 2

selector:

matchLabels:

app: datera







template:

metadata:

labels:

app: datera

spec:

hostNetwork: true

containers:

- name: csi-app-container

image: alpine

resources:

limits:

cpu: “1”

memory: “1Gi”

requests:

cpu: “1”

memory: “1Gi”

securityContext:

privileged: true

command: [ “/bin/sh”, “-c”, “--” ]

args: [ “while true; do sleep 300; done;” ]

volumeMounts:



volumes:

- name: csi-pvc-deployment


claimName: csi-pvc-deployment

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl create -f csi-app-deployment.yaml

deployment.apps/csi-app-deployment created

[root@ch3cp tmp]#



csi-app-deployment-6cdb499ff7-2scrd 1/1 Running 0 8m54s 172.58.4.72 ch13cp <none> <none>

csi-app-deployment-6cdb499ff7-tshwx 1/1 Running 0 8m54s 172.58.4.71 ch12cp <none> <none>

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl describe pvc csi-pvc-deployment

Name: csi-pvc-deployment

Namespace: default


Status: Bound







Volume: pvc-2972301a-a580-463c-8c53-6906a783dffb

Labels: <none>





Capacity: 50Gi

Access Modes: RWX


Mounted By: csi-app-deployment-6cdb499ff7-2scrd

csi-app-deployment-6cdb499ff7-tshwx

Events:


---- ------ ---- ---- -------

Normal ExternalProvisioning 32m persistentvolume-controller



Normal Provisioning 32m dsp.csi.daterainc.io _ ch12cp _ ab1d1489

79ed-11ea-b72f-003048f5c0b4 External provisioner is provisioning volume for claim

“default/csi-pvc-deployment”

Normal ProvisioningSucceeded 31m dsp.csi.daterainc.io _ ch12cp _ ab1d1489-

8c53-6906a783dffb

[root@ch3cp tmp]#







9.6 Volumes for Statefulset podsWhen your DB application needs to maintain its state in persistent volumes, managing it with a StatefulSet rather than a Deployment is the way to go. Unlike deployments, StatefulSets maintain a persistent identity for each Pod and create a unique PVC for each Pod. StatefulSets will bring up PVCs and Pods in an order. For example: The statefulset controller will create PVC-0 first. Then, Pod-0 is created and PVC-0 is attached to it. Once Pod-0 comes up, PVC-1 is created. Then, Pod-1 is created and PVC-1 is attached to it and so on. Each PVC creates a volume dynamically on the Datera cluster.

This combination of unique podnames and orderliness in Pod and PVC bring up is routinely used for cloud databases such as MongoDB which needs to establish a replication quorum and conduct primary election. The volumes mounted to MongoDB pods are backed by Datera for persistence.

When a StatefulSet is deleted, the order of Pod/PVC deletions are reversed. Your StatefulSet pod should reference the persistent volume claim templates and not a persistent volume claim (PVC). Think of volumeClaimTemplates as one that creates volume claims (PVCs). You would also normally create a “headless” service which frontends the Pods (not shown in the example below).

9.6.1. Create StatefulSet Pods

[root@ch3cp tmp]# cat csi-app-statefulset.yaml

kind: StatefulSet

apiVersion: apps/v1

metadata:

name: csi-app-statefulset

namespace: default

spec:

selector:

matchLabels:

app: csi-app-statefulset

serviceName: “csi-app-ss”

replicas: 2

template:

metadata:

labels:

app: csi-app-statefulset

spec:

hostNetwork: true

containers:

- name: statefulset-pod

imagePullPolicy: Always







image: alpine

volumeMounts:


name: ss

command: [ “sleep”, “1000000” ]

volumeClaimTemplates:

- metadata:

name: ss

spec:

accessModes: [ “ReadWriteOnce” ]

storageClassName: “dat-block-storage”

resources:

requests:

storage: 1Gi

[root@ch3cp tmp]#

[root@ch3cp tmp]# kubectl apply -f csi-app-statefulset.yaml

statefulset.apps/csi-app-statefulset created

[root@ch3cp tmp]#



ss-csi-app-statefulset-0 Bound pvc-4f6e6607-585e-4f2f-b2b7-6f0bef952ec1 1Gi RWO dat-block-storage 24s

ss-csi-app-statefulset-1 Bound pvc-28f91009-ec91-4ec7-ab86-3a064b1d43da 1Gi RWO dat-block-storage 11s

[root@ch3cp tmp]#



csi-app-statefulset-0 1/1 Running 0 35s

csi-app-statefulset-1 1/1 Running 0 22s

[root@ch3cp tmp]#

9.7 Volume resizing or expansionThis is a relatively new feature made available by kubernetes community and support for the feature is available from Datera CSI v1.0.9 onwards. To perform volume resizing, you would need to do the following:

9.7.1. Create cluster role binding

[root@master]# kubectl create clusterrolebinding csi-admin --clusterrole=cluster

admin --serviceaccount=kube-system:csi-datera-controller-sa

clusterrolebinding.rbac.authorization.k8s.io/csi-admin created

[root@master]#







9.7.2. Edit and update feature-gates for kube-apiserver

[root@master]# cat /etc/kubernetes/manifests/kube-apiserver.yaml

- --feature-gates=VolumeSnapshotDataSource=true,ExpandCSIVolumes=true,ExpandInU

ePersistentVolumes=true

[root@master]# !!! Wait until kube-apiserver restarts !!!

[root@master]# ps -ef | grep kube-apiserver

root 26330 26311 5 04:46 ? 00:06:48 kube-apiserver --advertise

address=172.19.1.80

--allow-privileged=true

--authorization-mode=Node,RBAC

--client-ca-file=/etc/kubernetes/pki/ca.crt

--enable-admission-plugins=NodeRestriction

--enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt

--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=

etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --

feature-gates=VolumeSnapshotDataSource=true,ExpandCSIVolumes=true,ExpandInUsePer

istentVolumes=true

--insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver

kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet

client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostn

me --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy

client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed

names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front

proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader

group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User

--secure-port=6443

--service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip

range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private

key-file=/etc/kubernetes/pki/apiserver.key

[root@master]#

Change the accessMode of the PVC to RWX and create the PVC using the StorageClass created above.







9.7.3. Enable allowVolumeExpansion flag

kind: StorageClass


metadata:

name: dat-block-storage


annotations:



parameters:



Change the accessMode of the PVC to RWX and create the PVC using the StorageClass created above.

9.7.4. Change accessMode and create PVC

Save the following as “csi-pvc-deployment.yaml”.

apiVersion: v1


metadata:


spec:

accessModes:

- ReadWriteMany

resources:

requests:

storage: 200Gi


[root@master]# kubectl create -f csi-pvc-deployment.yaml

persistentvolumeclaim/csi-pvc-deployment created

[root@master]#







9.7.5. Create a Pod deployment

Save the following as “csi-app-deployment.yaml”.

apiVersion: apps/v1

kind: Deployment

metadata:

name: csi-app-deployment

labels:

app: datera

purpose: automation

spec:

replicas: 1

selector:

matchLabels:

app: datera

template:

metadata:

labels:

app: datera

spec:

hostNetwork: true

containers:

- name: csi-app-container

image: alpine

resources:

limits:

cpu: “1”

memory: “1Gi”

requests:

cpu: “1”

memory: “1Gi”

securityContext:

privileged: true

command: [ “/bin/sh”, “-c”, “--” ]

args: [ “while true; do sleep 300; done;” ]

volumeMounts:



volumes:

- name: csi-pvc-deployment


claimName: csi-pvc-deployment







9.7.6. Check the status of PVC, PV and volume mount inside Pod

[root@master]# kubectl get pvc


csi-pvc-deployment Bound pvc-5e9e5e98-3692-4958-89f4-b39606193802 200Gi RWX dat-block-storage 5m10s

[root@master]#

[root@master]# kubectl get pv


pvc-5e9e5e98-3692-4958-89f4-b39606193802 200Gi RWX Delete Bound default/csi-pvc-deployment

dat-block-storage 4m41s

[root@master]#

[root@master]# kubectl get pods


csi-app-deployment-9484fb499-7fxx6 1/1 Running 0 36s

[root@master]#

[root@master]# kubectl exec -it csi-app-deployment-9484fb499-7fxx6 -n default --

df -h

Filesystem Size Used Available Use% Mounted on

/dev/mapper/360014053dd95460a6214f2e9b677d4aa 196.6G 59.0M 188.0G 0% /data

[root@master]#

9.7.7. Patch the PVC to increase the claim size

[root@master]# kubectl patch pvc csi-pvc-deployment --patch ‘{“spec”: {“resources”:

{“requests”: {“storage”: “250Gi”}}}}’

persistentvolumeclaim/csi-pvc-deployment patched

[root@master]#

To see the PVC requested size also change, restart the Pod by reducing the replicas to 0 and then back to 1 as shown below. This will also resize the filesystem.

9.7.8. Restart Pod

[root@master]# kubectl scale deployment csi-app-deployment

--replicas=0

deployment.apps/csi-app-deployment scaled

[root@master]#

[root@master]# sleep 30

[root@master]#

[root@master]# kubectl scale deployment csi-app-deployment

--replicas=1

deployment.apps/csi-app-deployment scaled

[root@master]#







After Pod restart, you would see both PVC and PV show 250Gi size, thus concluding Volume expansion. Check the volume size as seen from inside the deployed Pod.

9.7.9. Check the volume is resized



csi-pvc-deployment Bound pvc-5e9e5e98-3692-4958 89f4-b39606193802 250Gi RWX dat-block

storage 46m

[root@master]#

[root@master]# kubectl get pv


pvc-5e9e5e98-3692-4958-89f4-b39606193802 250Gi RWX Delete Bound default/csi-pvc-deployment

dat-block-storage 46m

[root@master]#

[root@master]# kubectl get pod

NAME READY STATUS RESTARTS AGE csi-app-deployment-9484fb499-6dbv8 1/1 Running 0 4m52s

[root@master]#

[root@master]# kubectl exec -it csi-app-deployment

9484fb499-6dbv8 -n default -- df -h

Filesystem Size Used Available Use% Mounted on

/dev/sdc 245.8G 58.6M 235.2G 0% data

[root@master]#







9.8 Volume retentionIn Kubernetes, the Volume lifecycle is independent of Pod lifecycle. The lifespan of the persistent volumes is dictated by the reclamation policy of the Persistent Volume Claim and the default is to bind that lifespan to the lifespan of the Pod that creates the volume. This means that if Pods are deleted or gets deleted, then the volume is deleted as well. If this is not what an Application user needs, then consider changing the reclamation policy to indicate that the persistent volume should be retained.

9.8.1. Create a PVC and set reclamation policy to ‘Retain’



csi-pvc Bound pvc-eb7fad77-a65a-468d-9955-a44ffffeea60 1Gi RWX dat-block-storage 16m

[root@master]#

[root@master]# kubectl patch pv pvc-eb7fad77-a65a-468d-9955-a44ffffeea60 --patch

‘{“spec”:{“persistentVolumeReclaimPolicy”:”Retain”}}’

persistentvolume/pvc-eb7fad77-a65a-468d-9955-a44ffffeea60 patched

[root@master]#

[root@master]# kubectl get pv pvc-eb7fad77-a65a-468d-9955-a44ffffeea60


pvc-eb7fad77-a65a-468d-9955-a44ffffeea60 1Gi RWX Retain Bound default/csi-pvc


[root@master]#

9.8.2. Delete the PVC and check PV is retained

[root@master]# kubectl delete pvc csi-pvc

persistentvolumeclaim “csi-pvc” deleted

[root@master]#

[root@master]# kubectl get pvc csi-pvc

Error from server (NotFound): persistentvolumeclaims “csi-pvc” not found

[root@master]#

[root@master]# kubectl get pv pvc-eb7fad77-a65a-468d-9955-a44ffffeea60


pvc-eb7fad77-a65a-468d-9955-a44ffffeea60 1Gi RWX Retain Released default/csi-pvc


[root@master]#







9.9 Multi-tenancyThe Datera CSI driver is installed at Cluster level, which means tenants in all namespaces would use the same driver for Volume operations. In other words, there is one CSI driver instance per Kubernetes cluster.

9.10 Driver upgrade and downgradeUpgrades and downgrades are very simple. Perform the following 2 steps:

1. $ kubectl delete -f <yaml_file_used_to_install_the_driver>

2. $ kubectl apply -f <new_driver_yaml_file>

10. Referenceshttps://www.linkedin.com/pulse/kubernetes-data-marc-fleischmann/




https://www.linkedin.com/pulse/kubernetes-data-marc-fleischmann/

Documents

Datera Kubernetes CSI Driver Deployment Guide