Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure...

www.nimbusproject.org

Infrastructure Clouds for Science and

Education: Platform Tools

1 11/12/2012

Kate Keahey, John Bresnahan, Patrick Armstrong, Pierre Riteau

Argonne National Laboratory

University of Chicago

The Power of Infrastructure Clouds

Virtualization opens the flood gates

11/12/2012 2

• Outsourcing

• Virtual appliances

– Freeze your stack in time

– Run it anywhere

• Multi-cloud applications

– Run many copies all over the world!

• Elasticity

Harnessing The Power

• Organization tools and techniques

11/12/2012 3

What Needs To Be Harnessed

• VM (appliance) creation and development

– configuration management tools (chef, puppet)

• VM hypervisors

– Infrastructure-as-a-Service (IaaS)

• Cloud applications

– virtual clusters, cloudinit.d, CloudFormation

• Elasticity

– Auto-scaling tools, phantom

11/12/2012 4

What Needs To Be Organized?

• VM hypervisors

– virtual clusters, cloudinit.d, CloudFormation,

• Elasticity

11/12/2012 5

VM Applications

• An entire system frozen in time

– Full software stacks (versions)

– Configuration files

– Important for science!

• A dedicated modular service

– Web service, database, AMQP node, etc

• Demos

• A binary single file (or set of files)

– Easy to freeze

11/12/2012 6

Developing Appliances

• A single binary image?

– Many developers?

– Version control?

– Merging conflicts?

• Base image with a description

– Ex: Ubuntu 11.04 base images plus a set of scripts

• Configuration Management Software

– Chef, Puppet, FG Rain, etc

11/12/2012 7

• Software stack description – ruby and json

• A library of cookbooks

• Cookbooks contain recipes – Ex: apache2 server with php4

• Attributes to customize each recipe – Ex: on what port will apache listen

• Templates for configuration files

• Appliance developers make recipes – Version control can be done with git/svn/cvs…

11/12/2012 8

Example Recipe

11/12/2012 9

app_dir = node[:appdir] ve_dir = node[:virtualenv][:path] git app_dir do repository node[:autoscale][:git_repo] reference node[:autoscale][:git_branch] action :sync user node[:username] group node[:groupname] end execute "run install" do cwd app_dir user node[:username] group node[:groupname] command "python setup.py install" end

Example Template

11/12/2012 10

phantom: system: type: epu rabbit: <%= node[:autoscale][:rabbit_host] %> rabbit_port: <%= node[:autoscale][:rabbit_port] %> rabbit_ssl: False rabbit_user: <%= node[:autoscale][:rabbit_username] %> rabbit_pw: <%= node[:autoscale][:rabbit_password] %> rabbit_exchange: <%= node[:autoscale][:rabbit_exchange] %> authz: type: sqldb dburl: <%= node[:autoscale][:dburl] %>

phantom: system: type: epu rabbit: vm-102.uc.futuregrid.org rabbit_port: 5672 rabbit_ssl: False rabbit_user: XXX rabbit_pw: PPPPPP rabbit_exchange: default_dashi_exchange authz: type: sqldb dburl: mysql://nimbus:XXXX@futuregrid.org/testphantom

• VM hypervisors

• Elasticity

11/12/2012 11

Cloud Applications Virtual Clusters

11/12/2012 12

Linux Clusters

• A cheap answer to Super Computers

– Many “commodity” machines interconnected to operate as single machine

• Load distributed across nodes

• Compute clusters

• A single head node with N worker nodes

11/12/2012 13

Worker

Virtual Clusters

• The same thing

– composed of VMs in a cloud

• Advantages

– A stable, pre-fabricated workspace

– Elastic

– repeatable

• Disadvantages

– Unknown hardware conditions

• Network, noisy neighbors, etc.

11/12/2012 14

Worker

Nimbus One-Click Clusters

• Virtual Clusters In Nimbus Clouds

• Turnkey, tightly-coupled cluster

– Shared trust/security context

– Shared configuration/context information

• Sample images

– NFS file systems, torque queues, GridFTP servers, etc…

• Easily repeatable and distributable

– An xml file and a VM image

11/12/2012 15

Nimbus One-Click Clusters Context

• Personalize a VM instance

– seed them with secrets, access policies, and just-in-time configurations

– Populate /etc/hosts with cluster member addresses

– Set up SSH host-based authentication across all accounts

• Must run a light weight script (context agent) on boot.

– Context broker/agent discussed in detail later

11/12/2012 16

Other Cloud Applications

11/12/2012 17

Cloud Applications

• More than 1 VM needed for the job

• Information exchange is needed

– Manual information exchange

• Multi-cloud

– Cloud independence required

11/12/2012 18

Web Server database

Web Web Web Server

Web Servers

A Simplified Deployment Scenario

11/12/2012 19

cloudinit.d

• Multicloud VM dependency management – Uses the libcloud abstraction library

• Integrated with chef solo

• ini file format descriptions – Coupled with any executable script

• Launch plan end-users/operators – Lightweight

– Copy launch plan and “one click” action

– Easily reconfigured for various clouds

• Launch plan/application developers: – Minimal software assumptions (ssh)

– “Stem cell” deployment approach

– Incremental launch plan development

11/12/2012 20

[svc-alamoHTTP]

iaas_key: XXXXXX

iaas_secret: XXXX

iaas_host: alamo.futuregrid.org

iaas_port: 8443

iaas: Nimbus

image: ubunut10.10

ssh_username: ubuntu

localsshkeypath: ~/.ssh/fg.pem

readypgm: http-test.py

bootpgm: http-boot.sh

cloudinit.d Overview

• Services

• Run Levels

– Collections of

services without

dependencies on

each other

• Launch Plan

– An ordered set of

run levels

11/12/2012 21

cloudinit.d example

11/12/2012 22

Wordpress Web server

MySQL database

• Wordpress example

• 2 virtual machine running in EC2 – MySQL

– Wordpress

• MySQL contact information injected into wordpress

www.nimbusproject.org 11/12/2012 23

• VM hypervisors

• Elasticity

11/12/2012 24

Scaling Considerations

• Reasons to scale – Business vs science

• Cost vs quota

• Lossy environment – VMs fail more often than bare metal

– N preserving

• Spot instances – If the price is right

• Backfill – If resources are idle

11/12/2012 25

Infrastructure Platform Goals • Multi-cloud

– Work across private, community and commercial clouds

• Any Scale – Scale in response to a diverse set of sensors/triggers

– Both system and application sensors

• High Availability – “Any VM can die”: system or user VMs

– Minimizing time to recovery (TTR)

• Your Polices, Our Enactment – User-defined sensors/triggers and policies

• Engineered from the ground up to work with infrastructure clouds

• Easy on the user

11/12/2012 26

Phantom Architecture

11/12/2012 27

REST HTTPS

Web Application HTTPS REST Service

Web Application

FutureGrid Clouds

RabbitMQ EPUM

Provisioner

Zookeeper Cluster

REST Service REST Service IaaS

Clouds

How Can Science Plug Into This Power

Example Embarrassingly Parallel

Scientific Application

Demonstration

11/12/2012 28

M subtask messages

Task Queue

Application Start the workers

Using Nimbus Domains

Preserve N worker VMs

M subtask messages

Cumulus/S3

Message Queue

“N preserving” policy

Infrastructure Compute Cloud

Get task

Results/Checkpoints

Application Start the workers

Using Nimbus Domains

www.nimbusproject.org 11/12/2012 31

One Click Cluster Exercise

• Examine the one click cluster definition

• Launch a virtual cluster

• Inspect the virtual cluster

11/12/2012 32

<?xml version="1.0" encoding="UTF-8"?> <cluster> <workspace> <name>head-node</name> <image>base-cluster-cc14.gz</image> <quantity>1</quantity> <nic wantlogin="true">public</nic> <ctx> <provides> <identity /> <role>nfsserver</role> </provides>

Infrastructure Clouds for Science and Education: Platform Tools · 2012-11-12 · Infrastructure...

Documents

Hybrid Infrastructure Hosting Clouds + Dedicated + Colocated

Loft Your Web Platform into the Clouds with Immutable Infrastructure

SMEs IT Infrastructure Management, Adoption Of The Clouds

First Steps in the Clouds Kate Keahey keahey@mcs.anl.gov University of Chicago Argonne National Laboratory

Mobile Service Clouds: A Self-Managing Infrastructure for ...sadjadi/Publications/... · recongured in response to changing conditions. The Service Clouds infrastructure is designed

Storage Infrastructure Choices for OpenStack Clouds

Virtual Appliances for Scientific Applications Kate Keahey keahey@mcs.anl.gov Argonne National Laboratory University of Chicago

Kate Keahey · CLOUDRESEARCH Kate Keahey keahey@anl.gov FutureCloud Symposium October 20th 2015 ... Cloud research at Scale: Big Data, Big Compute, Big Instrument Big Data Management

Bootstrapping Open Source Clouds - Dell Communityen.community.dell.com/cfs-file/__key/communityserver-blogs...Bootstrapping Open Source Clouds ... Bringing a cloud infrastructure online

HPC Cloud: High Performance Computing in Infrastructure CloudsCloud+ANL+v1.pdf · 2012-06-18 · HPC Cloud: High Performance Computing in Infrastructure Clouds Kate%Keahey% keahey@mcs.anl.gov%

Virtual Workspaces in the Grid Kate Keahey keahey@mcs.anl.gov Argonne National Laboratory Ian Foster, Tim Freeman, Xuehai Zhang, Daniel Galron

Improving Utilization of Infrastructure Clouds · Improving Utilization of Infrastructure Clouds Paul Marshall 1 Department of Computer Science 1 University of Colorado at Boulder

The World of Clouds, Containers, and Complicated Images · The World of Clouds, Containers, and Complicated Images ... Infrastructure Dev(Sec)Ops Physical Infrastructure: Server,

From Sandbox to Playground: Virtual Environments and Quality of Service in the Grids Kate Keahey keahey@mcs.anl.gov Argonne National Laboratory

Private Clouds Will Use Hybrid Infrastructure

OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative Common Execution Infrastructure Kate Keahey, Tim Freeman, Alex Clemesha, David LaBissoniere,

Design Discussion Rain: Dynamically Provisioning Clouds within FutureGrid PI: Geoffrey Fox*, CoPIs: Kate Keahey +, Warren Smith -, Jose Fortes #, Andrew

CMPE 297 Lecture: Building Infrastructure Clouds with OpenStack

Automated Security Analysis of Infrastructure Clouds - DiVA Portal

From Sandbox to Playground: Dynamic Virtual Environments in the Grid Kate Keahey keahey@mcs.anl.gov Argonne National Laboratory Karl Doering University