Bright Computing · Provisioning Monitoring Server Server Server Server Server Server Server Server OS OS OS OS OS OS OS OS Tools Workload Management High Performance Computing (HPC)

Robert Stober

Director of Systems Engineering

February 22, 2016

Advanced IT Management Made Easy

Bright Computing

Agenda

• High performance computing

• Bursting into the public cloud

• Adding big data clusters

• Adding OpenStack

• Adding “bare metal private cloud”

• Putting it all together

Bright Computing makes it easy to deploy

high performance computing (HPC) clusters

and to manage these throughout

their entire lifecycle.

Bright Cluster Manager

Provisioning

Server Server Server Server Server Server Server Server

OS OS OS OS OS OS OS OS

Tools Monitoring Workload

Management

High Performance Computing (HPC) Cluster

With Bright Computing, you can deploy a high performance computing

(HPC) cluster over bare metal quickly and easily.


PBSPro



UGE Slurm


After a quick install, users can immediately log in and begin submitting

jobs. Bright provides this capability out of the box.

OGS Moab Torque Maui LSF Open

Lava


PBSPro



UGE Slurm


After a quick install, users can immediately log in and begin submitting

jobs. Bright provides this capability out of the box.


Lava


PBSPro



UGE Slurm


Meanwhile, Bright is always “on duty.” Extensive monitoring and health

checking features ensure that the cluster is in optimal condition.


Lava


PBSPr

o



UGE Slurm


Meanwhile, Bright is always “on duty.” Extensive monitoring and health

checking features ensure that the cluster is in optimal condition.


Lava


PBSPro



UGE Slurm


Bright’s elegant, “single pane of glass” user interface delivers

unmatched simplicity, visibility, and control to administrators.


Lava

Bright Unified Interface (CMGUI)


PBSPro



UGE Slurm


Bright Cluster Manager is highly scalable and dynamic to address the

intensive and “bursty” resource requirements of HPC workloads.


Lava


PBSPro



UGE Slurm


Bright Cluster Manager is highly scalable and dynamic to address the

intensive and “bursty” resource requirements of HPC workloads.


Lava

node001 node002 node003 node-n-

Bright Computing can extend your high

performance computing (HPC) capacity

into the public cloud.




Workload Management


Bright Cluster Manager integrates with Amazon EC2 to enable dynamic

extension of HPC workloads into the public cloud.

Amazon EC2




Workload Management




Amazon EC2 Server

OS

Server

OS




Workload Management




Amazon EC2 Server

OS

Server

OS

j1 j2




Workload Management


One option is to instantiate and manage an entire HPC cluster inside of

EC2. This is called “Cluster on Demand.”

Amazon EC2 Server

OS

Server

OS

j2 j1




Workload Management


One option is to instantiate and manage an entire HPC cluster inside of

EC2. This is called “Cluster on Demand.”

Amazon EC2 Server

OS

Server

OS




Workload Management


Another option is to dynamically extend your on-premise HPC cluster,

adding and removing nodes as needed. This is “cluster extension.”

Amazon EC2




Workload Management




Amazon EC2

j1




Workload Management




Amazon EC2

j1




Workload Management




Amazon EC2

j1

Server

OS




Workload Management




Amazon EC2 Server

OS j1




Workload Management




Amazon EC2 Server

OS




Workload Management




Amazon EC2




Workload Management




Amazon EC2

Bright Computing can deploy and manage

high performance computing (HPC) and big

data clusters in the same infrastructure.




Hadoop Cluster

Workload Management

With Bright Cluster Manager, you can deploy a Hadoop cluster over bare

metal, leveraging Bright’s unified interface, scalability, and ease.

Spark MapReduce /

Yarn

HDFS /

Lustre

HPC Cluster




Workload Management

HPC Cluster

Bright supports all popular Hadoop distributions and can install and

manage a wide (and steadily growing) range of service components.

Spark

HB

ase

Pig

Hiv

e

Ka

fka

Sqo

op

Accu

mu

lo

Sto

rm

MapReduce /

Yarn

HDFS /

Lustre




Hadoop Cluster Workload Management

HPC Cluster

Administrators can manage HPC and Hadoop clusters together using

Bright’s intuitive, “single pane of glass” interface.

Spark





Hadoop Cluster Workload Management

HPC Cluster

Administrators can manage HPC and Hadoop clusters together using

Bright’s intuitive, “single pane of glass” interface.

Spark


Bright Computing can deploy and manage

OpenStack private clouds, standalone or as

part of a converged IT infrastructure.

Bright OpenStack

Provisioning



Roles Configuration Monitoring

OpenStack Private Cloud

With Bright Computing, you can deploy an OpenStack private cloud over

bare metal quickly and easily.

VM

OS

VM

OS

VM

OS

VM

OS

VM

OS

VM

Windows

VM

Debian

Bright OpenStack

Provisioning





Bright OpenStack includes Bright Managed Instances, which are VMs

provisioned using a standard Bright software image.

VM

OS

VM

OS

VM

OS

VM

OS

VM

OS

VM

Windows

VM

Debian

Bright OpenStack

Provisioning





Bright OpenStack includes Bright Managed Instances, which are VMs

provisioned using a standard Bright software image.

VM

OS

VM

OS

VM

OS

VM

OS

VM

OS

VM

Windows

VM

Debian

Bright OpenStack

Provisioning Roles Configuration Monitoring


Bright’s “cluster-as-a-service” capability makes it easy to spin up a

virtualized HPC cluster, Hadoop cluster, or both inside of OpenStack.

Bright OpenStack

Provisioning Roles Configuration Monitoring


Bright’s “cluster-as-a-service” capability makes it easy to spin up a

virtualized HPC cluster, Hadoop cluster, or both inside of OpenStack.

VM

OS

VM

OS

VM

OS

VM

OS

VM

OS

Bright “Cluster-as-a-Service”

HPC Cluster Hadoop Cluster

With Bright, it’s easy to deploy HPC, Big Data, and OpenStack within a

converged infrastructure and manage them through a unified interface.





HPC

Cluster





HPC

Cluster Hadoop

Cluster




Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster




Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster


Bright Computing enables dynamic

provisioning of physical nodes, i.e. “bare

metal private cloud.”

Bright OpenStack


For sites with highly varying workloads, Bright can switch on, switch off,

and re-provision local nodes based on demand and defined policies.

VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES11sp3

node04

SLES11sp3

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES11sp3

node04

SLES11sp3

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

gA gB gC

LAN

In this scenario, assume three user groups (gA, gB, and gC).

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES11sp3

node04

SLES11sp3

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

LAN

gA gB gC

cA cB cC

Each user group has its own configuration requirements, encapsulated

in the node categories (cA, cB, and cC).

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES11sp3

node04

SLES11sp3

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

LAN

cA cB cC

qA qB qC

gA gB gC

Each user group has its own assigned job queue (qA, qB, and qC).

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES11sp3

node04

SLES11sp3

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

LAN

gA gB gC

qA qB qC

cA cB cC

Initially, nodes 01, 02, and 05 are assigned to the queues as shown;

nodes 03 and 04 are powered off.

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES11sp3

node04

SLES11sp3

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

LAN

gA gB gC

qA qB qC

cA cB cC

User group C (gC) then assigns two new jobs to its queue (qC).

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES12

node04

SLES12

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

LAN

gA gB gC

qA qB qC

cA cB cC cC cC

Rather than keep these jobs pending, Bright changes the node category

of nodes 03 and 04 and powers them on.

Bright OpenStack


VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster

node01

SLES11sp3

node02

SLES11sp3

node03

SLES12

node04

SLES12

node05

SLES12

node06

SLES12

node07

SLES12

node08

SLES12

LAN

gA gB gC

qA qB qC

cA cB cC

After the jobs have finished, Bright switches off nodes 03 and 04 and

returns them to the pool of available resources.

Bright Computing provides “single pane of glass” visibility

and control across your rapidly evolving IT infrastructure.

Leveraging our strong heritage in HPC, we can help you

deploy and manage HPC clusters, Hadoop clusters, and

OpenStack, both on premise and in the cloud.

Bright OpenStack


Let Bright Computing guide you into the “bright future” of the converged,

dynamic datacenter … we can help!

VM

OS

VM

OS

VM

OS

VM

OS

VM

OS




HPC

Cluster Hadoop

Cluster


Questions?

Robert Stober Director of Systems Engineering

Bright Computing

[email protected]

Documents

Bright Computing · Provisioning Monitoring Server Server Server Server Server Server Server Server OS OS OS OS OS OS OS OS Tools Workload Management High Performance Computing (HPC)