6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 1/13

Blog

Backup/Restore, HA and DR for IBM CloudAutomation Manager

IBM Cloud Automation Manager (CAM) is designed for production orchestration of cloud workloads andenterprise capabilities of IBM Cloud Private and the Kubernetes environment. In this article we describCAM and strategies around Backup/Restore, High Availability (HA) and Disaster Recovery (DR) for CAM should be taken advantage of when running CAM in a production environment.

These instructions cover IBM Cloud Automation Manager 3.1.0.0 and newer. For previous versions see:and DR for IBM Cloud Automation Manager – Cloud Automation092418

Backup and RestoreLets first start with the basic backup and restore of a CAM environment. CAM deploys and manages womultiple cloud environments. The data that is used to create the deployments and manage the deploymclouds is stored inside CAM’s databases. It is important that a complete and recurring backup strategy CAM data so that in the event of either data loss or data corruption the CAM data can be restored.

Backing up CAM data:CAM data resides in four Persistant Volumes (PVs) that are pre-created prior to the installation of CAM.More on PVs from Kubernetes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

Hugh Hockett, Barry Gower and Tesshu Flower Published on May 8, 2018 / Updated on June 14, 2019

0

Cloud Automation Documentation Tutorials Blogs Videos Why IBM?

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 2/13

The default names of the four CAM PVs and their usages are:

Volume Persistant Volume (PV)name

DefaultSize

Usage

Mongo DB cam-mongo-pv 15 GB MongoDB volume for the ma

Logs cam-logs-pv 10 GB Logs volume for CAM pod lo

Terraform Plugins cam-terraform-pv 15 GB Terraform volume for defaul

plugins

Template Designer

DB

cam-bpd-appdata-pv 15 GB Template Designer MariaDB

For more information on CAM PVs see: https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_create_pv.html

As a best practice these four CAM volumes should reside in a storage location where consistent snapshthat data is consistently backed up from all four volumes simultaneously. If the underlying storage systconsistent snapshots, manual backup actions can still be performed using the procedure below.

Finally, an optional custom encryption password can be configured at CAM chart deploy time to encrypand Terraform container contents with a custom password. It is important that this custom encryption backed up separately (in parallel with the instructions in this document) since it will be needed to decryrestore scenario. Without this password a new install of CAM (for restoring from a backup) will not be aencrypted CAM data.

Manual backup procedureStop CAM First, stop CAM to ensure there are no modifications during the backup. stopCAM.sh

Logs To backup the CAM logs, simply make a copy of the logs volume mount using a shell command like cp o

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 3/13

Terraform Plugins To backup the Terraform Plugins, simply make a copy of the Terraform plugins volume mount using a shor rsync.

Mongo DB If CAM was deployed with an external MongoDB, best practices for MongoDB backup strategies shouldparticularly if the MongoDB instance is replicated or sharded. Find backup and restore best practices indocumentation: https://docs.mongodb.com/manual/core/backups/

Backup bundled CAM MongoDB database

Start just the cam-mongo database. kubectl scale -n services deploy cam-mongo --replicas=1

It is a good idea to separate the system being backed up from the system capturing the backup. Stastandalone MongoDB container in the same ICP cluster where CAM is deployed. This container willthe backup. Note, the port numbers for this command may need to be changed to avoid conflicts. docker run -d --name cam-mongo-backup mongo:3.4

Next, execute the mongodump command, pointing to the cam-mongo service. MONGO_IP=$(kubectl get -n services svc/cam-mongo --no-headers | awk '{print $3}') MONGO_PASSWORD=$(kubectl get -n services secret cam-secure-values-secret -o yaml |mongoDbPassword: | awk '{print $2}' | base64 --decode) Note: Replace “cam-secure-values-secret” in the above command with your cam secret if you use docker exec -it cam-mongo-backup mongodump --uri mongodb://root:$MONGO_PASSWORD@$Marchive=/tmp/mongo_backup.gz --gzip

Once the dump is complete, copy the backup archive off of the standalone MongoDB backup contathis someplace safe. docker cp cam-mongo-backup:/tmp/mongo_backup.gz .

Finally, cleanup the standalone MongoDB backup container: docker stop cam-mongo-backup docker rm cam-mongo-backup

Template Designer There are two different actions to backup the CAM Template Designer data: the MariaDB and some repbpd-ui container.

Backup bundled CAM MariaDB database If CAM was deployed with an external MariaDB, best practices for MariaDB backup strategies shoul

1.

2.

3.

4.

5.

•

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 4/13

particularly if the MariaDB instance is replicated or sharded. Find backup and restore best practicesdocumentation: https://mariadb.com/kb/en/library/backup-and-restore-overview/If CAM was deployed with the bundled MariaDB, follow this procedure:

Start just the cam-bpd-mariabd database. kubectl scale -n services deploy cam-bpd-mariadb --replicas=1

It is a good idea to separate the system being backed up from the system capturing the backupstandalone MariaDB container in the same ICP cluster where CAM is deployed. This container wthe backup. Note, the port numbers for this command may need to be changed to avoid conflictdocker run -d --name cam-maria-backup -e MYSQL_RANDOM_ROOT_PASSWORD=yes -d mari

Next, execute the mysqldump command MARIA_IP=$(kubectl get -n services svc/cam-bpd-mariadb --no-headers | awk '{priMARIA_USERNAME=$(kubectl -n services get secret cam-secure-values-secret -o yammariaDbUsername: | awk '{print $2}' | base64 --decode) MARIA_PASSWORD=$(kubectl -n services get secret cam-secure-values-secret -o yammariaDbPassword: | awk '{print $2}' | base64 --decode) Note: Replace “cam-secure-values-secret” in the above commands with your cam secret if youdocker exec -it cam-maria-backup mysqldump --host=$MARIA_IP --port=3306 --all-duser=$MARIA_USERNAME --password=$MARIA_PASSWORD --result-file=/tmp/mariadb_back

Copy the backup archive off of the standalone MariaDB backup container. Make sure to store thdocker cp cam-maria-backup:/tmp/mariadb_backup.sql .

Finally, cleanup the standalone MariaDB backup container: docker stop cam-maria-backup docker rm cam-maria-backup

Backup CAM BPD UI repositories

Start just the cam-bpd-ui service. kubectl scale -n services deploy cam-bpd-ui --replicas=1

Create an archive of the important files export BPD_UI=$(kubectl get -n services pods | grep cam-bpd-ui | awk '{print $1kubectl exec -it -n services $BPD_UI -- tar -cvzf /tmp/cam-bpd-ui-backup.tgz -Cpatterns/ workspace repositories

Then, copy the archive off the container. Make sure to save this someplace safe. kubectl cp -n services $BPD_UI:/tmp/cam-bpd-ui-backup.tgz .

Finally, cleanup the created archive on the container. kubectl exec -it -n services $BPD_UI -- rm -f /tmp/cam-bpd-ui-backup.tar

1.

2.

3.

4.

5.

•1.

2.

3.

4.

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 5/13

Content Runtime If you have deployed a Content Runtime, follow the steps here to backup the Chef Server, Software RepPattern Manager:https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/content/cam_content_runtime_back

Start CAM Once all manual backup steps are complete, start CAM. startCAM.sh

Manual restore procedurePre-requisite: These manual restore steps assume there is a deployed instance of CAM. This can eithethat the backup was created from which is being repaired, or it can be a new instance of CAM on a sepaCloud Private as part of a larger disaster recovery scenario. If a new instance of CAM is being deployed,encryption password” when deploying the CAM chart if one was specified during the initial deployment

Stop CAM First, stop CAM to ensure there are no modifications during the restore. stopCAM.sh

Logs To restore the CAM logs, simply make a copy of the backed up log files onto the logs volume mount usinlike cp or rsync.

Terraform Plugins To restore the Terraform Plugins, simply make a copy of the backed up Terraform plugins onto the Terramount using a shell command like cp or rsync.

Mongo DB Restoring bundled CAM MongoDB database

Start just the cam-mongo database kubectl scale -n services deploy cam-mongo --replicas=1

It is a good idea to separate the system being restored from the system performing the restore. Stastandalone MongoDB container in the same ICP cluster where CAM is deployed. This container willthe restore. Note, the port numbers for this command may need to be changed to avoid conflicts. docker run -d --name cam-mongo-backup mongo:3.4

1.

2.

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 6/13

Copy the desired backup archive to be restored into the standalone container docker cp mongo_backup.gz cam-mongo-backup:/tmp/mongo_backup.gz

Next, execute the mongorestore command, pointing to the cam-mongo service. MONGO_IP=$(kubectl get -n services svc/cam-mongo --no-headers | awk '{print $3}') MONGO_PASSWORD=$(kubectl get -n services secret cam-secure-values-secret -o yaml |mongoDbPassword: | awk '{print $2}' | base64 --decode) Note: Replace “cam-secure-values-secret” in the above command with your cam secret if you use docker exec -it cam-mongo-backup mongorestore --uri mongodb://root:$MONGO_PASSWORD-archive=/tmp/mongo_backup.gz --gzip --drop

Finally, cleanup the standalone MongoDB container docker stop cam-mongo-backup docker rm cam-mongo-backup

Template Designer

Restoring bundled CAM MariaDB database

Start just the cam-bpd-mariadb database kubectl scale -n services deploy cam-bpd-mariadb --replicas=1

It is a good idea to separate the system being restored from the system performing the restore.standalone MariaDB container in the same ICP cluster where CAM is deployed. This container wthe restore. Note, the port numbers for this command may need to be changed to avoid conflictdocker run -d --name cam-maria-backup -e MYSQL_RANDOM_ROOT_PASSWORD=yes -d mari

Copy the desired backup archive to be restored into the standalone container docker cp mariadb_backup.sql cam-maria-backup:/tmp/mariadb_backup.sql

Next, execute the restore command MARIA_IP=$(kubectl get -n services svc/cam-bpd-mariadb --no-headers | awk '{priMARIA_USERNAME=$(kubectl -n services get secret cam-secure-values-secret -o yammariaDbUsername: | awk '{print $2}' | base64 --decode) MARIA_PASSWORD=$(kubectl -n services get secret cam-secure-values-secret -o yammariaDbPassword: | awk '{print $2}' | base64 --decode) Note: Replace “cam-secure-values-secret” in the above commands with your cam secret if youdocker exec -it cam-maria-backup sh -c "mysql -v --host=$MARIA_IP --port=3306 -user=$MARIA_USERNAME --password=$MARIA_PASSWORD < /tmp/mariadb_backup.sql"

Finally, cleanup the standalone MariaDB container docker stop cam-maria-backup docker rm cam-maria-backup

3.

4.

5.

•1.

2.

3.

4.

5.

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 7/13

Restore CAM BPD UI repositories

Start just the cam-bpd-ui service. kubectl scale -n services deploy cam-bpd-ui --replicas=1

Copy the desired backup archive to be restored into the BPD UI container export BPD_UI=$(kubectl get -n services pods | grep cam-bpd-ui | awk '{print $1kubectl cp -n services cam-bpd-ui-backup.tgz $BPD_UI:/tmp/

Expand the archive on the container kubectl exec -it -n services $BPD_UI -- tar -xvf /tmp/cam-bpd-ui-backup.tgz -C patterns/ workspace repositories

Finally, cleanup the archive on the container. kubectl exec -it -n services $BPD_UI -- rm -f /tmp/cam-bpd-ui-backup.tar

Content Runtime If you have deployed a Content Runtime, follow the steps here to restore the Chef Server, Software RepPattern Manager: https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/content/cam_cont

Start CAM Once all manual restore steps are complete, start CAM. startCAM.sh

High AvailabilityLet's discuss building a CAM environment with high availability to address single points of failure for a bCAM with a single set of pods. First it is important that an ICP be created with multiple worker nodes thmultiple instances of CAM pods with an instances of CAM pods on each node. For more information aboHA environment see the following ICP knowledge center links: https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.2/supported_system_config/hardwarehttps://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0.2/installing/high_availability.html

CAM is composed of a set of pods or microservices. A CAM pod of each type can be run on each workerone of the CAM pods is called cam-iaas which services the core APIs in CAM. If you deploy an ICP with cam-iaas pod can run on each of the three worker nodes. Running multiple cam-iaas pods helps increahandling CAM API calls as well as provide high availability of the CAM APIs themselves. If a worker nodsurviving two worker nodes and their cam-iaas pods could continue to handle the API requests for CAM

One note here: technically a single worker node can be used to provide pod HA by deploying more thanspecific CAM pod on a single worker node. This would allow for surviving the loss of a specific CAM pod

•1.

2.

3.

4.

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 8/13

provide high availability for losing a worker node with all running pods on it (e.g. the VM or physical hardrecommended that CAM pods run on multiple worker nodes to provide high availability that covers the

In CAM there are multiple types pods that handle the front end proxy, API calls, UI calls, service brokerEach type of pod can be configured to run on more than one worker node via a "replica count" configurainstall time. This picture shows when deploying the CAM chart configuring a replica count of 3 for the C

Each type of pod translates to multiple underlying microservices. For example there are multiple UI micmultiple API microservices. Configuring a replica count of 3 for APIs sets the number of pods to 3 for aare used to handle APIs. The types of pods to configure include:

Broker

Proxy

••

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 9/13

APIs

UI

Template Designer CDS

Template Designer MDS

Today when a replica count greater than 1 is configured and there are multiple worker nodes, the pods available worker nodes. For example if a "Replica Count" of 3 is configured and there are 3 worker nodetype will be started on each worker node for high availability. For more information on how Kubernetes http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html

Pods can also be scaled up or down during runtime to address high availability concerns or address percharacteristics of CAM. To scale a specific CAM pod during runtime navigate to the ICP UI under WorkloFind the specific pod (for example cam-iaas in the screenshot below) and under the action menu select

Then enter the number of instances to scale to (3 in the example below) and click "Scale Deployment"

••••

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 10/13

By default CAM ships with a single MongoDB database for storing CAM data, a single Redis for session msingle MariaDB for storing template designer data. To support a highly available CAM environment exteMongoDB, Redis and MariaDB can be setup outside of CAM and then be configured for use by the CAM time.

Cloud Automation Manager can be configured to use an external MongoDB and an external database foManager Template Designer by following the instructions here:https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_using_external_mongodb.html

Similarly an external Redis cluster that can provide high availability can also be configured for use with instructions here: https://www.ibm.com/support/knowledgecenter/SS2L37_2.1.0.2/cam_using_extern

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 11/13

Note: Since these external databases and Redis clusters are created and managed outside of CAM ensuappropriate backup and disaster recovery strategy for these external components in addition to the pra

Disaster RecoveryDisaster recovery (DR) typically covers scenarios where a datacenter is no longer available in either a pevent, and services must be restored at another site that is geographically separated. Implementationstypically vary based on corporate requirements, locations of data centers, available technology and mo

Common technologies used involve site to site continuous replication of databases or underlying data vsoftware or hardware level, and either synchronously (typically shorter distances, no data loss) or asynlonger distances, but with some data loss of transactions in flight that were not fully replicated). A moreinvolve taking regular backups and transferring copies of those backups to a remote site for recovery.

More information on disaster recovery and general practices can be found here:https://en.wikipedia.org/wiki/Disaster_recovery

Replicating CAM data to a backup site is one part of the DR story, but in the event of a disaster (or testinCAM environment must also be installed that uses the replicated data. There are two key considerationbackup CAM environment:

The workloads deployed and managed by CAM in target cloud environments need to be replicated aalong side CAM to the backup site. For example, if you have a target VMware environment, that envworkloads must be replicated or backed up to the backup site outside of CAM. CAM itself does not workloads or data inside deployed workloads, only the data about how to manage the deployed woCAM.

The backup CAM instance must have equivalent network access to target cloud environments. For eVMware vSphere target at the primary datacenter that has deployments, the VMware vSphere envirdeployments must be network accessible in the backup datacenter as they were in the primary datexisting cloud connections in CAM will still be valid and deployments from CAM can be managed byCAM. As another example if a public cloud environment like IBM Cloud is used, equivalent networkmust also be in place for the backup CAM instance to continue to deploy and manage existing IBM

As was discussed in the earlier backup/restore section, CAM stores its data in four persistent volumes. the CAM data and logs that need to be replicated to a backup site. These four volumes should also be pgroup so that data is transferred across all four volumes in the order it is being modified.

1.

2.

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 12/13

Replicated data can then be used with a new install of CAM at the backup site during an unplanned evedisaster), planned event (e.g. planned shutdown of a datacenter) or for a test of disaster recovery pract

In any of these DR scenarios, the restore procedure documented above can be used to restore the replnewly deployed instance of CAM.

For an unplanned event (or test of an unplanned event):

Stop replication and take a consistent snapshot of all four replicated CAM volumes at the backup si

Create four new volumes and populate them with the data from the consistent snapshot

Create four PVs that reference the volumes with restored CAM data

Install CAM and reference these four PVs during the CAM chart install

For a planned event (or test of a planned event)

Stop CAM first to prevent any transactions in flight from being lost.

Take a consistent snapshot of all four replcating CAM volumes at the backup site

Create four new volumes and populate them with the data from the consistent snapshot

Create four PVs that reference the volumes with restored CAM data

Install CAM and reference these four PVs during the CAM chart install

Final ThoughtsThis article provides you with strategies and procedures for implementing Backup/Restore, High AvailaRecovery solutions of CAM instances. Implementations may vary depending on your specific requiremetechnologies but many of the underlying principles around CAM availability and data protection remainUnderstanding these concepts with CAM will allow you to build a robust instance of CAM for deploying production cloud applications.

Special thanks to Jeffrey Luo and Partha Kaushik for their assistance verifying the content of this

1.

2.

3.

4.

1.

2.

3.

4.

5.

6/14/2019 Backup/Restore, HA and DR for IBM Cloud Automation Manager - Cloud Automation

https://developer.ibm.com/cloudautomation/2018/05/08/backup-ha-dr/?preview_id=1937&preview_nonce=1a29837b1d&_thumbnail_id=1231&preview=true 13/13

Contact Privacy Terms of use Accessibility Report Abuse Feedback Cookie preference

TAGS BACKUP, CAM, CLOUD AUTOMATION MANAGER, DISASTER RECOVERY, DR, FEATURED, HA, HIGH AVAILABILITY, RESTORE

Join The Discussion

Enter your comments...