Provision Software-Defined Storage Infrastructure at …...overview of the design and development of a Terraform configuration for provisioning infrastructure and installing the operating

© 2020 Cisco and/or its affiliates. All rights reserved. Page 1 of 20

Provision Software-Defined Storage Infrastructure at

Cloud Scale with Terraform Provider for Cisco Intersight

White paper

Public


Contents

Executive summary 3

Introduction 3

Solution deployment 6

Conclusion 19

For more information 19

About the Author 20


This document describes how to use the Terraform provider for the Cisco Intersight™ to

achieve automated, zero-touch infrastructure deployment for cloud-scale software-defined

storage on the Cisco Unified Computing System™ (Cisco UCS®). This guide provides an

overview of the design and development of a Terraform configuration for provisioning

infrastructure and installing the operating system for a generic software-defined storage

solution using the Terraform provider for the Cisco Intersight platform.

Executive summary

This document describes how to use the Terraform provider for Cisco Intersight™ software with the Cisco

Unified Computing System™ (Cisco UCS®) platform to automate infrastructure provisioning and operating system

deployment for cloud-scale storage. This guide provides the framework for deploying any software-defined

storage solution on three Cisco UCS C240 M5 Rack Server nodes with two Cisco UCS C220 M5 Rack Servers

as controller and load-balancer nodes. Cisco UCS provides computing, network, and storage components as a

unified platform for software-defined storage solutions. Cisco Intersight software provides a systems

management platform that delivers intuitive computing through cloud-powered intelligence. The Cisco

Intersight platform provides infrastructure management for Cisco UCS and Cisco HyperFlex™ systems. It

supports the use of the Terraform provider to develop infrastructure as code, facilitating zero-touch

deployment.

Introduction

This section provides an overview of the Terraform provider, Cisco Intersight and Software-defined storage

solution with Cisco Intersight.

Cisco Intersight platform

The Cisco Intersight platform provides benefits for the entire infrastructure lifecycle. Cisco UCS Manager, Cisco

UCS Director, and Cisco Integrated Management Controller (IMC) focus on day-0 and day-1 activities related to

deployment and configuration. The Cisco Intersight platform augments this focus with benefits for day 2 and

beyond as well. In addition to monitoring and alerting, Cisco Intersight provides integration with the Cisco®

Technical Assistance Center (TAC), predictive analytics, and resource optimization to address ongoing

operations and systems upgrades.

Figure 1 shows a high-level view of multiple management stages with the Cisco Intersight platform.


Figure 1.

Cisco Intersight infrastructure lifecycle

HashiCorp Terraform

Terraform is software for building complex, version-controlled, and collaborative infrastructure with high

productivity. The infrastructure Terraform can manage includes low-level components such as computing

instances, storage, and networking, as well as high-level components such as Domain Name System (DNS)

entries and software-as-a-service (SaaS) features.

The set of files used to describe infrastructure in Terraform is known as a Terraform configuration. The

configuration is written using HashiCorp Configuration Language (HCL), a simple human-readable configuration

language, to define a desired topology of infrastructure resources.

Why Terraform provider for the Cisco Intersight?

The Terraform provider allows organizations to develop Cisco Intersight resources as self-service infrastructure

using code rather than manual provisioning.

This approach provides several benefits:

● You can more quickly and easily scale Cisco Intersight resources. You can provision infrastructure in

minutes, with little effort, using the automated workflows, performing the same tasks that used to

take days.

● The operating model of Terraform is well suited for the Cisco Intersight platform, because it

accommodates the shift from static to dynamic infrastructure provisioning. For example, if a resource

is deleted in the Terraform configuration, it will be reflected in the Cisco Intersight platform when the

new configuration is applied.


● Terraform maintains a state file, which is a record of the currently provisioned resources. State files

provide a version history of Cisco Intersight resources, enabling a detailed audit trail of changes.

● The provider enables idempotency, producing the same result and state with repeated API calls.

Software-defined storage solution with Cisco UCS and Cisco Intersight

The Cisco Intersight platform provides a comprehensive architecture for deploying and managing software-

defined storage. Figure 2 shows the hardware design for a generic software-defined storage solution.

Figure 2.

Solution hardware design

The design includes the following components:

● The Cisco Intersight platform is deployed as a SaaS solution.

● Controller and load-balancer nodes are deployed on Cisco UCS C220 M5 servers.

● Storage nodes are deployed on Cisco UCS C240 M5 servers.

● Cisco UCS C240 and C220 servers are connected to Cisco Nexus® 93240YC-FX2 Switches with 25-

Gbps line speed.


Solution deployment

The deployment of the whole solution consists of several main steps:

● Create an account on the Cisco Intersight platform and claim all storage nodes.

● Create a Terraform configuration environment for the Cisco Intersight platform.

● Apply Terraform configurations for the following:

◦ Update firmware on all storage nodes.

◦ Create server policies and profiles.

◦ Deploy server profiles for all nodes.

◦ Install the operating system on all nodes.

● Deploy software-defined storage software.

Creating an account on the Cisco Intersight platform and claiming nodes

You create an account on the Cisco Intersight platform by claiming a Cisco IMC device. Before you claim the

device, you should perform a preliminary check of the Cisco Intersight device connector in the IMC. The current

device connector requirements are described in

https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Cisco_Intersight_Appliance_Getting_

Started_Guide/b_Cisco_Intersight_Appliance_Install_and_Upgrade_Guide_chapter_010.pdf and summarized in

Table 1.

Table 1. Device connector requirements

Component Minimum software version Supported device connector version

Releases that include supported device connectors

Cisco UCS Manager Release 3.2(I) Release 1.0.9-2290 Release 4.0(2a) or later

Cisco IMC Supervisor For M5 servers: Release 3.1(3a)

For M4 servers: Release 3.0(4)

Release 1.0.9-335 Release 4.0(2c) or later

Cisco HyperFlex Connect and Cisco HyperFlex HX Data Platform

Release 2.6 Release 1.0.9-1335 Release 3.5(2a) or later

To create an account on the Cisco Intersight platform, follow these steps:

1. Log in to https://www.intersight.com/.

2. Sign in with your Cisco ID (Figure 3).

https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Cisco_Intersight_Appliance_Getting_Started_Guide/b_Cisco_Intersight_Appliance_Install_and_Upgrade_Guide_chapter_010.pdf

https://www.cisco.com/c/en/us/td/docs/unified_computing/Intersight/b_Cisco_Intersight_Appliance_Getting_Started_Guide/b_Cisco_Intersight_Appliance_Install_and_Upgrade_Guide_chapter_010.pdf

https://www.intersight.com/


Figure 3.

Cisco Intersight main screen

3. Read the offer description and accept it.

4. Copy the device ID and claim code in the Cisco Intersight Account Creation wizard from the Cisco

IMC.

5. Enter a name for the account and click Create.

The device claim process can take a few minutes. If required, the device connector will automatically be upgraded as part of the process. Account creation and basic configuration are now finished.

6. On the Cisco Intersight dashboard, choose Devices > Claim a New Device (Figure 4).

Figure 4.

Claiming a new device

7. Copy the device ID and claim code for the second Cisco UCS C240 M5L server and click Claim.

8. Repeat the same process for the third Cisco UCS C240 M5L server and for the Cisco UCS C220 M5

servers.

9. After you claim all the devices, you should see the devices listed under the server tab at the left.


Create a Terraform configuration environment for the Cisco Intersight platform

Next you create the Terraform configuration environment.

Install Terraform

On the deployment host, make sure that Terraform is installed. The Terraform binary is available for major

distributions.

1. Download the Terraform zip file from https://www.terraform.io/downloads.html according to the

operating system of your deployment host.

2. Extract the zip file and move it to a directory of your choice.

3. Add the path of the directory to the PATH variable of the system.

For more information about how to install Terraform, see https://learn.hashicorp.com/terraform/getting-

started/install.html.

Clone the repository

Use the following command to clone the repository on your deployment host. This repository contains code to

deploy the whole solution.

# git clone https://github.com/ucs-compute-solutions/terraform-intersight-sds.git

After the repository has been cloned, you should see five directories: firmware_update, create_infra,

provision_infra, os_deployment, and unbind_profiles. Separate directories are created for logical separation

and easy understanding of workflows.

If required, you can merge all the resources into a single file in your deployment with fewer modifications

Copy the Terraform provider binary file

The Terraform binary file for the Cisco Intersight platform is built for Microsoft Windows, Darwin (Mac OS), and

Linux 64-bit architecture systems. Choose the appropriate binary file according to the operating system in your

deployment host.

Follow these steps:

4. Go to https://github.com/cisco-intersight/terraform-provider-intersight.

5. Navigate to the Releases tab. Download the terraform-provider-intersight.zip file for the latest

release.

6. Extract the files and navigate to the terraform-provider-intersight binary file for the operating system

installed on your deployment host.

7. Copy the terraform-provider-intersight binary file to all the directories of the downloaded repository.

Generate API keys

To use the Cisco Intersight provider, you need an API key, a secret key, and the Cisco Intersight endpoint URL.

To generate the keys, follow these steps:


9. On the Settings screen, click the Settings menu.

https://www.terraform.io/downloads.html

https://learn.hashicorp.com/terraform/getting-started/install.html

https://learn.hashicorp.com/terraform/getting-started/install.html

https://github.com/ucs-compute-solutions/terraform-intersight-sds.git

https://github.com/cisco-intersight/terraform-provider-intersight



10. On the General page, choose API > API Keys > Generate API Key.

11. On the Generate New API Key screen, enter the purpose for the API key and click Generate. The API

key ID and RSA private key are displayed.

12. Copy the API key.

13. Save the private key information in a .pem file. Save it in a location in the downloaded repository.

Define the Cisco Intersight provider

Now define the Cisco Intersight provider.

Navigate to the create_infra directory in the cloned repository and open main.tf file. Enter the API key you

copied. Also provide the secret key file. The endpoint changes if you are using Cisco Intersight appliance.

provider "intersight" {

secretkeyfile = "Secret key file generated from previous step "

endpoint = "https://intersight.com"

}

For simplicity in this document, multiple directories are defined for each workflow (to create infrastructure,

apply server profiles, upgrade firmware, etc.). Copy the same main.tf file in all the directories. You can also

combine all the Terraform configurations in one directory.

Configure variables

You need to define some of the basic inputs required for various workflows to provision the infrastructure. You

will define all these as variables in a file named variables.tf.

Each configuration takes a value in the variables.tf file as shown here:

variable "variable_name " {

default = <value of the variable>

}

You need to keep several image files, such as the operating system image file, Cisco Server Configuration

Utility (SCU), and Cisco Host Update Utility (HUU), in a remote server for firmware upgrades and operating

system installation. Download the files from software.cisco.com and keep them in a local Network File System

(NFS), HTTP, or Common Internet File System (CIFS) share that is accessible to the Cisco IMC for the servers.

Then you can edit the variables for the remote server and images.

In variables.tf, define all the VLAN IDs that you need for the management, client, cluster, and replication

networks.

Now you need the managed object ID (MOID) for the organization and claimed storage nodes. To get these

values, follow these steps:


15. From the Help screen, click Get More Help from Cisco Intersight.

16. Navigate to API Documentation.

17. Click API Reference (Figure 5).



Figure 5.

API Reference

18. Search for compute/PhysicalSummary.

19. Click GET: Read a 'compute.PhysicalSummary' resource (Figure 6).

Figure 6.

Supported methods for compute/PhysicalSummary resource


20. In the representational state transfer (REST) client, click Send to get a response from the

compute/PhysicalSummary API (Figure 7).

Figure 7.

REST Client screen

21. The response is an array of physical servers claimed with all the information about the claimed

servers. The first entry is the MOID for the server. Copy the MOIDs for all the claimed servers and

the organization MOID.

22. You have all the basic input required to apply the Terraform configurations. Copy the same

varables.tf file in all the directories.


Understanding the Cisco Intersight provider and Terraform configuration

The resource objects names in the configuration are in this format:

intersight_<model_name_in_snake_case>

For example, intersight_ntp_policy is the resource object for the Network Time Protocol (NTP) policy, and

intersight_storage_disk_group_policy is the resource object for the storage disk group policy. The following is

an example of a complete resource definition for a server profile and the NTP policy attached to the server

profile:

resource "intersight_server_profile" "storage-node1" {

name = "storage-node1"

organization {

object_type = "organization.Organization"

moid = var.org_moid

}

assigned_server {

moid = var.storage-node1

object_type = "compute.RackUnit"

}

}

resource "intersight_ntp_policy" "sds-ntp-policy" {

name = "sds-ntp-policy"

enabled = true

ntp_servers = [

"ntp.esl.cisco.com",

"171.68.38.65",

"173.38.201.115"

]

organization {

object_type = "organization.Organization"

moid = var.org_moid

}

profiles {

moid = intersight_server_profile.storage-node1.moid

object_type = "server.Profile"

}

}


Each resource is assigned a name, which can later be used for tracking and referencing. This name will not be

reflected anywhere in the Cisco Intersight platform. It is only for reference among the .tf files. The NTP policy is

attached to the server profile created earlier. This is accomplished by referencing the storage-node1 profile

in profiles.moid. A resource can point to or reference another resource using the

format <resource>.<resource_name>.<property_name>.

View the logs

TF_LOG is a terraform variable that is used for viewing different categories of logs. By default, this variable is

left empty. To view logs for Terraform operations, this variable must be set to DEBUG.

In Mac OS and Linux, you accomplish this with export TF_LOG=debug. In Windows PowerShell, use

$env:TF_LOG=“DEBUG”.

View documentation

Documentation about provider resources and configuration options can be found at https://github.com/cisco-

intersight/terraform-provider-intersight/tree/master/website/docs.

Implement the Terraform configuration

You need to run terraform init whenever you start with new Terraform code.

The terraform init command will scan the code, identify the provider you are using, and download the

appropriate provider. In the case here, because you copied the provider manually, Terraform use the Cisco

Intersight provider.

You next run the terraform plan command.

The plan command allows you to see what Terraform will do to the resources before any changes are actually

made. This command provides a good way to check the potential results before you make any changes to the

infrastructure.

In the output, the symbols show you the following:

● Resources with a plus sign (+) will be created.

● Resources with a minus sign (-) will be deleted.

● Resources with a tilde (~) will be modified in place.

Finally, run terraform apply to deploy the configuration. Observe that the apply command shows the

same output as the plan command. To actually proceed with the plan, you need to type yes to confirm.

Applying Terraform configurations for workflows

Next apply the Terraform configurations to the various workflows.

Upgrade firmware

You should upgrade the firmware for the Cisco UCS C240 M5 servers to a suggested release. On the Cisco

Intersight platform, you upgrade firmware using a noninteractive Cisco HUU to upgrade the BIOS, Cisco IMC,

PCI adapters, RAID controllers, and other firmware to compatible versions.

You can upgrade firmware either through a network share or using utility storage. In this example, the firmware

is upgraded through a network share using the NFS protocol. The HUU image file is mounted directly in the

Cisco IMC. Therefore, it requires uninterrupted connectivity between the remote file server and Cisco IMC.

https://github.com/cisco-intersight/terraform-provider-intersight/tree/master/website/docs

https://github.com/cisco-intersight/terraform-provider-intersight/tree/master/website/docs


To apply the Terraform configuration for firmware update, follow these steps:

23. Go to the firmware_update directory.

24. Append all the servers that require firmware updates to firmware_update.tf.

25. If running the configuration for the first time, run terraform init.

26. Run terraform apply to initiate the firmware updates.

Create server policies

You need multiple server policies for any generic software-defined storage solution. Table 2 lists the policies

required and the corresponding resource objects in the Terraform provider for the Cisco Intersight platform.

Table 2. Terraform provider policies and resource objects

Policy Terraform rResource oObject Comments

Adapter configuration

intersight_adapter_config_policy Specify the ID of the PCI slot ID in which the Cisco virtual interface card (VIC) adapter is placed.

Ethernet adapter intersight_vnic_eth_adapter_policy

Ethernet network intersight_vnic_eth_network_policy Create these resources based on the number of networks required for the software-defined storage solution. In the example here, four networks are being created.

Ethernet quality of service

intersight_vnic_eth_qos_policy

LAN connectivity intersight_vnic_lan_connectivity_policy

intersight_vnic_eth_if

The first resource creates LAN connectivity policy. The second resource creates multiple virtual interfaces.

NTP intersight_ntp_policy

Disk group intersight_storage_disk_group_policy The purpose of this resource is to create RAID from 2 solid-state disks (SSDs) present in the server. This resource will be used for OS installation.

Storage intersight_storage_storage_policy This resource will include disk group policy and also set policies for virtual drives. This resource will also set storage disks to JBOD mode.

Boot order intersight_boot_precision_policy

To apply the Terraform configuration to create server policies, follow these steps:

27. Go to the create_infra directory.

28. Update the server_profiles.tf file with the number of server profiles required.

29. If you are running the configuration for the first time, run terraform init.



31. After the updates are applied, verify that all the server profiles are created in the Cisco Intersight

portal by navigating to Policies > Server Policies (Figure 8).

Figure 8.

Policies created by Terraform configuration

Associate server profiles

After all the required polices have been created, you need to associate the server profiles with each physical

server. To associate the profiles with the servers, follow these steps:

32. Go to the provision_infra directory.

33. Update the server_profiles.tf file with the resource intersight_server_profile for each server profile.

34. For each server profile, define the server with which you want to associate it in the assigned_server

attribute.

35. If you are running the configuration for the first time, run terraform init.


37. The process for applying the server profiles takes a few minutes. You can see that association is in

progress from the Requests screen (Figure 9).


Figure 9.

Server profile association in progress

38. After the server profiles have been applied, verify server profile association in the Cisco Intersight

portal by navigating to Profiles > Server Profiles (Figure 10).

Figure 10.

Server profile associated with physical servers

Make changes to resources

If any of the resources needs updating, you first need to unbind the server profile, then make the changes, and

then bind the server profiles back again.

Follow these steps to make any changes:

39. Go to the unbind_profiles directory.

40. Update unbind_profiles.tf file with the server profiles that you want to unbind.

41. Run terraform apply.

42. Make any necessary changes in either policies or profiles.

43. Go to provision_infra directory and run terraform apply.


Install the operating system

The operating system used for this document is Red Hat Enterprise Linux (RHEL) 7.6. However, you can choose

a different operating system based on the Cisco UCS Hardware and Software Compatibility Matrix and support

from your storage software vendor.

To install the OS, follow these steps:

44. Go to the os_deployment directory.

45. In repo_setup.tf file, define all the OS images in the resource

intersight_softwarerepository_operating_system_file. Use a separate resource for each OS type.

46. In the same file, configure the resource for the Cisco SCU in the

intersight_firmware_server_configuration_utility_distributable resource.

47. In os_install.tf file, define the resource intersight_os_install for each node.

48. Run terraform apply to install the OS.

Complying with the Cisco Hardware Compatibility List

The Cisco Intersight platform evaluates the compatibility of your Cisco UCS and Cisco HyperFlex systems to

verify that the hardware and software have been tested and validated by Cisco or Cisco partners. The Cisco

Intersight platform reports validation issues after checking the compatibility of the server hardware (server

model, CPU, and server firmware version), server software (current OS vendor and OS version), and adapter

compliance (adapter model, driver protocol, and driver version for the firmware).

You can manually determine the recommended hardware and firmware versions for your server configuration

with the Cisco UCS Hardware and Software Compatibility tool at

https://ucshcltool.cloudapps.cisco.com/public/.

The Cisco Intersight platform provides am open-source tool called the OS Discovery Tool to collect the OS and

driver information need to evaluate Cisco Hardware Compatibility List (HCL) compliance in Linux operating

system versions (Figure 11).

Figure 11.

OS Discovery Tool

https://ucshcltool.cloudapps.cisco.com/public/


Follow these steps to evaluate compliance with the HCL:

49. Set up any Linux virtual or physical machine that has access to the on-premise network. Install

Python 2.7 on this control node.

50. On this control node, clone the Python software development kit (SDK) repository for the Cisco

Intersight platform:

git clone https://github.com/CiscoUcs/intersight-python.git

51. Install the Python SDK:

cd intersight-python

python setup.py install

52. In the intersight-python directory, go to os-discovery-tool.

53. Edit the discovery_config_linux.json file to include intersight_api_key and path

intersight_secret_file.

54. Run the following script:

./get_linux_inv_to_intersight.py --log-inventory --

configfile=discovery_config_linux.json

55. The script collects information about the operating system and device drivers on all the nodes and

routes this information to the Cisco Intersight platform to help evaluate compliance with the HCL.

After this process is complete, you should be able to see the compliance status. In the Servers list,

choose one of the servers and navigate to the HCL tab. You should see the HCL status (Figure 12).

Figure 12.

Hardware and software compatibility status

56. For all the components, you can also see recommended versions and download the drivers by

clicking Get Recommended Drivers (Figure 13).

https://github.com/CiscoUcs/intersight-python.git


Figure 13.

HCL recommendations

Deploy software-defined storage software

After the server profiles are attached and operating system is installed on all the nodes, you can deploy your

software-defined storage solution. Refer to the software-defined storage vendor’s installation guides to deploy

the solution.

Typically, you need to install two components: controllers and load balancer nodes and claiming storage nodes.

You can customize your Terraform resources, such as the number of networks and LAN connectivity policy,

based on the requirements of your storage software vendor.

Conclusion

The Cisco Intersight SaaS platform transforms the way that customers deploy and manage Cisco UCS and

Cisco HyperFlex systems. The Cisco Intersight platform supports the Terraform provider, which offers an

excellent way to easily build, scale, and manage the lifecycle of any scale-out storage software solution with

Cisco UCS servers connected to a switched environment. The Cisco Intersight platform offers resources for

updating firmware, configuring profiles, associating profiles with physical servers, managing profiles, and

installing the operating system.

For more information

For additional information, see the following resources:

● Terraform provider for the Cisco Intersight platform on GitHub:


● GitHub repository for the solution:

https://github.com/ucs-compute-solutions/terraform-intersight-sds


https://github.com/ucs-compute-solutions/terraform-intersight-sds


● Cisco Intersight online help: https://intersight.com/help/home

● Cisco Intersight data sheet: https://www.cisco.com/c/en/us/products/collateral/servers-unified-

computing/intersight/datasheet-c78-739433.html#FlexibleDeploymentOptions

About the Author

Paniraja Koppa, Cisco Systems, Inc.

Paniraja Koppa is a Technical Marketing Engineer for UCS Solutions. He has more than 13 years of experience

with a primary focus on data center technologies such as Cisco UCS, Storage, Operating systems, Automation,

Virtualization and Cloud. In his current role at Cisco Systems, he works on best practices, optimization,

automation and performance tuning of software defined storage on Cisco UCS platforms. Prior to this, he has

led QA efforts for 4 new virtual adapter card’s firmware and software features for Cisco UCS. He also worked

as customer support engineer and advocate in the Data Center Virtualization space.

Acknowledgements

For their support and contribution, the authors would like to thank:

● Vikrant Balyan, Cisco Systems, Inc.

● Aanisha Mishra, Cisco Systems, Inc.

● Chris O'Brien, Cisco Systems, Inc.

● Oliver Walsdorf, Cisco Systems, Inc.

● Jawwad Memon, Cisco Systems, Inc.

● David Soper, Cisco Systems, Inc.

Printed in USA 220104.1 05/20

https://intersight.com/help/home

https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/intersight/datasheet-c78-739433.html#FlexibleDeploymentOptions

https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/intersight/datasheet-c78-739433.html#FlexibleDeploymentOptions

Documents

Provision Software-Defined Storage Infrastructure at …...overview of the design and development of a Terraform configuration for provisioning infrastructure and installing the operating