Cloud Bursting 101: What to Do When Computing Demand Exceeds Capacity

WEBINAR | February 16, 2017

Housekeeping

2

●Recording

●Today’s Slides

●Attachments

● Questions

●Feedback

Subs

crib

e to

our

Brig

htTa

lk C

hann

el

Note to those new to BrightTalk: Audio access to webinars is only via computer. Phone numbers are not available.

Our Presenters

3

Damian HasakSolutions Architect Lead

Avere Systems

Bernhard BehnPrincipal Technical Marketing Engineer

Avere Systems

4

And When Should You Use It

What is Cloud Bursting?

Cloud Bursting

• Extend existing filesystem(s) into public compute environment

• Retain application compatibility

• Leverage on-demand compute at massive scale

• R/W caching to eliminate WAN latency

Cloud Bursting Benefits

Use as Much (Or As Little) Compute as is Needed• Simple to scale up and down, bursting to the cloud with Avere brings balance and efficiency to

an infrastructure by eliminating the need to house and maintain idle IT resources but still gain the ability to get the job done quickly.

No Need to Rewrite Enterprise Applications• With the Avere vFXT, enterprise apps do not have to be rewritten to run in the compute cloud.

By creating a NAS-based performance storage tier in cloud computing, applications gain access to the data they need, while overcoming inherent NAS verses object API incompatibilities.

Data is Easy to Move, Easy to Manage• Expansion of compute resources no longer involves hardware ordering, shipping, installation,

networking, configuring, or testing.

7

Avere vFXT for Cloud Bursting

AvereOS Overview

• Components• Clustered/Scale-out NAS system• High-performance tier via R+W caching• Global Namespace• Built for parallel access: scale a working set across all nodes

• Scalable system• 3 to 50 nodes within a single cluster• Add nodes for additional performance and/or data locality• High-Availability• Single management interface, regardless of cluster size• Easy expansion

Avere Overview: Public Compute

• Complete port to public compute environments• All features + functionality intact: same look/fit/finish/feel as physical product• Instantiated via dedicated management micro-instance• Or, can be configured via command line interface

• Valuable for orchestration workflows• One command, one cluster (no matter the size)

10

Performance and EfficienciesCloud Bursting for HPC

Cloud Bursting - Networking Challenges

Quick Deployment

SSH Tunneling

Flexible Network

VPN over Internet

Best Performance

Dedicated 10gig Link

-- Performance ++

++ Deployment Ease/Speed --

11

10Gbit

• Compute Cloud Components• Compute Instances• Disks/SSDs• Network Interfaces

• AvereOS Software Components• Install OS onto instance disks• Bootstrap a configuration• Cluster the system together

• vFXT Cluster• IP addresses for clients to mount• Core filer storage configuration• Filesystem namespace access to on-premises data

Cloud Bursting – The Cloud Puzzle Pieces

12

SW/OSSSD

NET

Cloud Bursting – Avere OS Orchestration Options

• Create/Monitor/Destroy vFXT clusters• Interaction with Cloud Provider APIs

• Assemble the cloud components• Interactions with Avere APIs

• Configure the cluster for file access

• Avere vFXT Cluster Manager• Cloud Compute Instance with UI• Avere provides access to disk image• Graphical User Interface• API (XMLRPC based)

• Python Library Orchestration• Provide your own Python environment• Avere provides python package• Command-line orchestration• Programmatic Python calls directly into library

13

AvereOS

SSD

Cloud Bursting – I/O Performance Capabilities

• GCE Instance Types• n1-highmem-8: capable of 8Gbit/sec raw network throughput• n1-highmem-32: capable of 16Gbit/sec raw network throughput

• GCE Disk Types• PersistentSSD: capable of 240MB/sec read, 240MB/sec write• LocalSSD: capable of 1560MB/sec read, 1090MB/sec write

• vFXT Scale-Out Clustering• Adding vFXT nodes to the cluster grows disk cache and disk/network throughput• Balancing client activity across the cluster vFXT nodes yields optimal performance

• vFXT File/Directory Caching• Read or Read+Write caching• Performance vs. Consistency can be dialed-in

14

15

Target Deployment / Use Cases

Use-case: Media and Entertainment

10.1.1.191:/vfx

NFS / SMB VPN

Scenario- Media company requires extra compute to complete a project, but cannot acquire new

hardware within a reasonable timeframe. Beg/borrow/steal isn’t an option.

- Avere vFXT cluster in EC2/GCE/Azure- Render assets lifted from Local NAS to public compute environment- Jobs run/data is processed, output written down to Local NAS- vFXT cluster (and compute) powered down to reduce cost- Data optionally resident in compute environment for next run

Local NAS / Datacenter

Use-case: Financial Services and Technology

10.1.1.192:/marketdata

NFS / SMB VPN

Scenario- Financial Services company would like to process new models at massive scale for 1-2

hours per day, without changing researcher workflow

- Avere vFXT cluster in EC2/GCE/Azure- Researcher generates new model, writing directly to Local NAS as always- vFXT and 10,000-50,000 compute cores spun up to begin work- Output written directly to Local NAS for additional interpretation- Compute and vFXT assets powered down or removed to minimize cost

Local NAS / Datacenter

Use-case: Life Sciences NGS

10.1.1.193:/genomes

NFS / SMB VPN

Scenario- A Drug Discovery company would like to process genetic data from drug trials at

massive scale in the compute cloud, without growing in-house compute infrastructure

- Avere vFXT cluster in EC2/GCE/Azure- Researcher obtains new genomic models, writing directly to Local NAS as always- vFXT and tens of thousands of compute cores spun up to begin work- Output written directly via vFXT funnles back to Local NAS for additional analysis- Compute and vFXT assets powered down or removed to minimize cost

Local NAS / Datacenter

19

Live Demonstration

The Demo Environment

20

Infrastructure• Network Connectivity• Orchestrator• vFXT cluster• Compute Client Instances

Application• Linux NFS grid application• CPU intensive computation• Read intensive I/O workload

On-Prem Datacenter

NAS/on-prem-data/on-prem-archive

Compute Cloud Clients have real-time cached access to on-premises data via GCE-hosted vFXT Cluster

Clients on-premises create the content and consume the results, consisting of desktop users, NAS storage and compute grid

Google Cloud Project

vFXT

/ (root)|--- /on-prem-data|--- /on-prem-archive

compute-node##

GCE vFXT.pyOrchestrator

Pre-Flight Checklist

• Can the orchestrator reach the GCP Project network?• Does the GCP Project network have a network path back to on-premises NAS

core filer?• Can the GCE vfxt.py instance reach the googleapis.com endpoint?• Is the vfxt.py python library installed on the GCE instance?• Does the vfxt.py GCE instance have the proper permissions/credentials to

orchestrate the vFXT cluster creation?• Determine the region/zone to build the cloud infrastructure• Are the project quotas set up for the instances/storage we plan to deploy?

OK! Time for liftoff!!!

21

22

April 24-27, 2017Las Vegas, Nevada

Booth SL9627

May 15-17, 2017Hilton Midtown

New York, New York

May 17-18, 2017Navy Pier

Chicago, Illinois

Questions?

23

Damian HasakSolutions Architect Lead

Avere Systemsdhasak@averesystems.com

Bernhard BehnPrincipal Technical Marketing Engineer

Avere Systemsbbehn@averesystems.com

www.averesystems.com