CycleCloud Quick Start Guideversion 3.2.0

Contents

CycleCloud Quick Start Guide 1Welcome 1

System Prerequisites 1

AWS Prerequisites 1Generating SSH keys 2

Installation 2Download 2

Install 2

Configuration 3Initial Setup Wizard 3

Add Cloud Provider Account 4

Cluster Creation 5Select a Configuration 6

Specify Configuration 6

Cluster Initialization and Data Configuration 7

Starting the Cluster 9

Submitting Jobs to the HTCondor Cluster 10

Terminating the Cluster 11

Conclusion 11

CycleCloud Quick Start Guide

WelcomeThe purpose of this document is to provide a step-by-step guide to setting up an auto-scalingbatch cluster in Amazon AWS using CycleCloud. After completing the steps in this guide, youwill have a fully functional HTCondor cluster that can automatically scale according to thedemands of your application. This Quick Start Guide assumes that you will be runningCycleCloud on a 64-bit Linux machine and that you are familiar with basic systemadministration concepts and tools. The guide assumes that you already have an Amazon AWSaccount, have access to the AWS console, and are not behind a firewall that will block directaccess to AWS services.

System PrerequisitesTo complete the instructions in this guide, you must have root or sudo access. In addition,your system needs to meet the following minimum requirements:

• Linux 64-bit operating system (Centos 6, Ubuntu 12.04 LTS, Ubuntu 14.04 LTS, etc.)• 8GB System Memory• 10GB Free Disk Space

Note

CycleCloud may be installed on physical or virtualized hardware.

AWS PrerequisitesTo complete this Quick Start Guide for CycleCloud, you will need to have the following:

• Amazon EC2 Access Key and Secret Key. If you do not have these, contact your systemadministrator or follow the instructions at:http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html

• Amazon EC2 Key Pair. This file, usually stored with a .pem extension, is used to log in toinstances. It should be available locally on your machine. Please note the location of thisfile for use later in the document. If you do not have a key pair, please contact yoursystem administrator or follow the instructions here:http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

1

Generating SSH keysTo generate your own SSH key, run the following set of commands:

# Ensure your .ssh directory exists$> mkdir -p ~/.ssh# Generate the key pair$> ssh-keygen -f ~/.ssh/cyclecloud -t rsa -b 2048# Do not enter a passphrase to allow CycleCloud's automated processes to work.# Rename the private key to have a .pem extension$> mv ~/.ssh/cyclecloud ~/.ssh/cyclecloud-private.pem

Once you have generated the SSH key pair:

• Log in to AWS console• Go to the EC2 Dashboard• Select Key Pairs• Import the public key “cyclecloud.pub”

Warning

Use phraseless SSH key

If you enter a passphrase when generating your SSH key, the return proxy feature ofCycleCloud will not work, which may cause node reporting and autoscaling to fail.

Installation

DownloadYou should have received a download link from Cycle Computing to access the installer.Click the link in the email and download the installer to your local hard drive. For theremainder of this section, we assume you have saved the installer to $HOME/Downloads.

InstallTo begin the installation, open a terminal session and unpack the CycleCloud installationusing the following command:

$> tar zxf ~/Downloads/cycle_server-cloud-latest-linux64.tar.gz -C /tmp

This will unpack the CycleCloud installation to /tmp/cycle_server. Once the CycleCloudinstallater is unpacked, execute the following commands to install CycleCloud:

2

$> cd /tmp/cycle_server$> sudo ./install.sh --installdir ~/cycle_server

Note

You must have write permission to the directory that you choose to install CycleCloud. Ifthe installation directory is not specified, it will default to /opt/cycle_server. For thepurposes of this document, we will install CycleCloud into your home directory.

Once the installer has finished running, you will be provided a link to complete the installationfrom your browser. Copy the link provided into your web browser.

ConfigurationAfter CycleCloud is installed, it is configured through a web browser.

Initial Setup Wizard

Step 1 - Welcome

Click "Next".

Step 2 - License Agreement

Accept the license agreement and click “Next”.

3

Step 3 - Administrator Account

At this step in the process, you will set up the local administrator account for CycleCloud. Thisaccount is used to administer the CycleCloud application and is not an operating systemaccount. Enter in the desired values and click “Next” to continue. For the remainder of thisguide, we assume the account's user ID is "admin".

Step 4 - CycleCloud Feedback

If you choose to help us improve our produce experience, please select “Yes.” We value yourprivacy and will not collect any data that is not related to our product. Click “Done” tocomplete your installation.

Enter License

You received an email with a license and your download link. Copy the license string fromand paste it into the form. Click “Install License” to continue.

You will see confirmation that the license has been successfully installed.

Add Cloud Provider AccountClick on the “Clusters” menu at the top of the screen to set up your cloud provider account.You will be notified that there are no cloud provider accounts currently set up and provided alink to set up your credentials.

4

Click to add your cloud provider account.

Click the “+” button next to the “Credentials” drop-down to enter your cloud providercredentials.

Enter a name for your cloud provider credentials. This is a descriptive name such as “My AWSAccount.” From the drop-down, select “Amazon” and enter your Access Key, Secret Key, andDefault Region. Click “Test” to validate your account. If you do not know, or have notgenerated your AWS credentials, please visithttp://docs.aws.amazon.com/general/latest/gr/getting-aws-sec-creds.html to learn how tocreate your keys. Once you have successfully validated your credentials, click “Save” toproceed.

Open the Account Bucket drop-down to select an existing S3 bucket to use for storing yourconfiguration and application data for your cluster. If the drop-down is empty, or you wouldlike to create a new S3 bucket, click the “+” button next to the Account Bucket drop-downwhere you will be given the opportunity to create a new bucket. Please remember to follow S3bucket naming conventions and note that bucket names must be unique. For moreinformation, please visithttp://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Once you have completed setting the parameters for your cloud provider account, click“Save” to continue.

CycleCloud is now configured and you are ready to create your first cluster.

Cluster CreationTo add your first cluster to CycleCloud, click the “Clusters” menu in the top level menu. Sincethere are no existing clusters defined, you will be prompted to add a new cluster.

5

Select a ConfigurationClick the “HTCondor” logo from the list of cluster templates. Cluster templates arepre-configured infrastructure definitions that encapsulate the composition and basic attributesof compute environments. These templates can be easily modified by system administratorsand end users to create standard deployments. Cycle Computing supports numerouspre-configured application environments common in industries including life sciences,manufacturing, finance, and more.

Specify ConfigurationThe cluster templates described in the last section include various parameters that control thecompute cluster’s infrastructure, scaling behavior, and provisioning options. The user interfacefor modifying cluster parameters is data-driven and can be easily customized to adapt tospecific customer requirements. The configuration for the HTCondor cluster type is depictedbelow.

Cluster parameters

Parameter Description

Credentials Select the credentials created earlier.

Autoscale Select this checkbox to allow your cluster to dynamically scale-upand scale-down with the workload.

6

Scheduler Type The machine type to use for the scheduler. For this guide, selectm3.large, 2 cores

Scheduler OS The operating system to use for the scheduler. For this giude, selectCentOS 6.

Execute Type The machine type to use for the execute nodes. For this guide,select m3.large.

Execute OS The operating system to use for the execute nodes. For this guide,select CentOS 6.

Initial Cores The number of execute cores to start when the cluster is started.When set to ‘0’, it instructs CycleCloud not to start execute nodeson cluster start.

Max Cores The maximum number of execute cores to autostart.

Use Spot Instances When checked, the execute nodes are spot instances. Whenunchecked, the execute ndoes are on-demand instances.

Bid Price The bid price to use for spot instances. This is not used foron-demand instances.

Cluster-init The folder in the account bucket associated with your clustercredentials. The process of initialization is discussed in the nextsection. For this guide, enter 'condordemo'.

Networking Options for Amazon's Virtual Private Cluster service. For this guide,leave blank.

Keypair The public key name imported into AWS earlier.

Keypair Location The location on the machine running CycleCloud where yourprivate key is stored.

Cluster Name The name of the cluster.

Cluster Initialization and Data ConfigurationYour installation includes a limited version of DataMan, Cycle Computing’s award-winningdata orchestration tool. DataMan provides high-throughput data transfer and orchestrationcapabilities to a variety of user-defined endpoints. For the purposes of this guide, you willconfigure it to set up the cluster’s application executables and data.

Create a Local Endpoint

From the “Data” menu, select the “Transfer Manager”.

7

You will be prompted to set up your initial endpoints. DataMan endpoints are logical storagelocations. For example, AWS S3, AWS Glacier, Azure, local, and remote. For the purposes ofthis guide, you will begin by setting up a local endpoint to upload the demo HTCondorapplication.

Click "Save" to continue. In the left navigation pane of the display, you should now see the"examples" directory that you specified in the "Allowed Paths".

Create an S3 Endpoint

On the right side of the navigation pane, you will configure an S3 endpoint to store yourcluster application and other initialization data. When you created your credentials earlier,you were asked to provide a storage bucket to associate with your CycleCloud account. Youwill now create a DataMan endpoint to this storage location.

Click “Save” to continue.

S3 endpoint parameters

Parameter Description

Name Descriptive name to associate with this endpoint. For this guide, use'MyClusterStorage'.

Type Select 'Amazon S3'

Credential The cloud credentials to use. For this guide, select the credentialsyou set up previously.

Bucket The cloud storage bucket to use. For this guide, select the storagebucket associated with the account you created.

8

Copy HTCondor Cluster Initialization

To copy the Condor cluster initialization example files to the S3 storage account, double-clickon the "examples" folder on the left navigation pane. Select the "cluster-init" folder and clickon the right arrow to transfer the files to S3.

This example creates the initialization scripts and executables required to execute a MonteCarlo approximation of pi. The “cluster-init” folder contains Please refer to the CycleCloudUser Guide for more information on how to create initialization folders.

Starting the ClusterYou have created a cluster definition for and HTCondor cluster and initialized the applicationsand data required to begin working with the cluster. To start the cluster, select “Clusters” fromthe top navigation menu, and select “MyCondorCluster” in the cluster list on the left side.Click "Start" to provision your new cluster.

When you start the cluster, CycleCloud will begin the process of provisioning yourinfrastructure in AWS. You can view the progress of the provisioning process by viewing the"Details" window in the lower right-hand side of the screen.

When the process completes, you will see the "master" node status turn green, indicating"Ready". At any point in the process, you can hover your mouse over the instances to seepop-up text that indicates the current status of the node.

9

Submitting Jobs to the HTCondor ClusterTo connect to your running cluster, select the "master" node in the lower detail pane and clickthe "Connect" button.

This will launch a window with instructions for connecting to the instance.

Copy and paste the SSH command into a terminal to connect to the instance.

To submit the workload, use the condor_submit command:

$> condor_submit /mnt/cluster-init/shared/pi_demo/pi.sub

This will submit 1000 iterations of a simple Newtonian approximation of Pi. To begin with, noexecute nodes have been started to run the workload. In the background, CycleCloud ismonitoring the HTCondor scheduler. When idle jobs are in the queue, CycleCloud will

10

request execute nodes. You can monitor the actions of CycleCloud in the status window in thelower right panel of the Cluster page. For the configuration provided in this guide, you canexpect CycleCloud to provision five execute nodes to run the workload. Near the end of thebilling hour, if the nodes have been idle for at least 20 minutes, CycleCloud will request thatthe execute nodes be terminated.

To view the current status of the HTCondor cluster, you may find the following commandshelpful.

To list cluster nodes:

$> condor_status

To view job status:

$> condor_q

Terminating the ClusterWhen your workfow is completed, you can terminate the cluster. From the Clusters page,select the "MyCondorCluster" cluster in the left-hand navigation pane. Click the "Terminate"button.

ConclusionThrough the course of this Quick Start Guide you have installed CycleCloud, created aworking HTCondor cluster, and submitted and executed a simple workflow. This guide,however, only scratches the surface on what CycleCloud can do. CycleCloud can beconfigured to handle virtually any workflow and can be tailored to integrate with mostenterprise configurations. For advanced configurations, in-depth examples, and customimplementations, please consult the User Guide and Administrator Guide or contactsales@cyclecomputing.com for more information.

11