Introduction to FutureGrid: Towards a Computing Testbed as a Service

https://portal.futuregrid.org

Introduction to FutureGrid:Towards a

Computing Testbed as a Service

Gregor von Laszewski


Use my own cluster?

• Pro:– Full access– Can support the research I am interested in

• Con:– Limited scale– Maintenance cost high– Does often do not integrate with multitenancy

research– Limitation of research topics due to availability of

testbed capabilities (often decommissioned hardware)


Use XSEDE-other than FG?

• Pro:– Full managed environment– Software can be added by staff but must undergo production

testing and application need analysis – Scale

• Con:– Focused on application use not CS research– Software stack is “static”– Research limited to application optimization– Targets production software not testing– Resources are mostly dedicated to production– Experiments could impact production


CS Research Testbeds (A personal view)

• Grids:– Wisconsin Grid Testbed: Compile service, limited testing

• HPC:– XSEDE TIS: Focus on XSEDE resources, production

level, limited innovation ability

• EmuLab:– Focus on network research, bare metal

• FutureGrid:– Integrates with bare metal, IaaS, PaaS, limited network

research


What others want on FG


What others want on FGOpenStack


2009-03-22 - 2009-03-282010-07-11 - 2010-07-172011-10-30 - 2011-11-050

20

40

60

80

100

120

openstackcloudstackubuntu euca-lyptusubuntu openstackredhat openstack

Google Trends


Recent Trends

• FG (Project Trends)– All IaaS same interest

volume

– OpenStack – OpenNebula

– Nimbus – Eucalyptus – Eucalyptus (Class)

• Google (User Trends)– OpenStack – CloudStack

– Eucalyptus – Nimbus


Why don’t we support …?

• IU– Model: provisioning by

users and center– IU supports

• HPC (IU&SDSC)• OpenStack• Eucalyptus (IU&SDSC)• Hadoop• various other activities

• UFL– Model: traditional– Cloudstack

• TACC & UC– Model : traditional – HPC– Nimbus


SW Architecture: Integrative View

https://portal.futuregrid.org http://futuregrid.org

Image Generation

• Users who want to create a new FG image specify the following:o OS typeo OS versiono Architectureo Kernelo Software Packages

• Image is generated, then deployed to specified target.

• Deployed image gets continuously scanned, verified, and updated.

• Images are now available for use on the target deployed system.


Provisioning HPC, Grid, and Cloud Services


Management Services

• Image Management• Dynamic Provisioning• Experiment

Management• Monitoring and

Information Services

Select

Features

• < LAPACK• < MongoDB

Generate Imag

e

• > Meta data• > Raw Image

Store Imag

e

• > Repository• > Local file systemIf

imag

e is

not

ava

ilabl

e


Management Services




Submit

Job

• < Image• < Job description

Provision

Image

• > Access Image• > Place on Resources

Execute

• > Run Job• > Get Results


Experiment ManagementGoals

• Support rigorous experimentation– Define experiments in detail– Record experimental results

• User-specified measurements (placement and granularity)

– Share experiment information• Experiments can be repeated and verified• Variations on experiments can be performed

• Convenient execution of experiments– FutureGrid has distributed resources and services– Supports different user scenarios


Management Services



Information ServicesTaaS

Experiment

Image Generation

Image Repository

Experiment Repository

Dynamic Provisioning

Nimbus

Eucalyptus


FG Hostlist & Grid5000

TakTuk


Using Rain as Experiment Management Tool

• fg-rain –h hostfile –image img• fg-rain –h hostfile –iaas openstack –image img• fg-rain –h hostfile –paas hadoop …

• Users require not a complex experiment environment, but a high-level interface to it

• We need more than a “workflow” enectment engine

• fg-shell > ….. (part of fg-rain)

http://futuregrid.org



Assemble your own Experiment

• Users have control of the entire stack

• Testbed Production systems adapt based on user and service demand

Adaptive

Services

• Move IaaS nodes• SLA’s

Create

Cluster

• PaaS Provisioning

Cloud

Seeding

• IaaS Provisioning

Connectivi

ty

• Network Provisioning


Summary


Selected Testbed Capabilities

Pre FG

Application orientedExperiments in production systemsDIY

FutureGrid I

Dynamic Provisioning of OSPreinstalled IaaS(Preinstalled) PaaSVirtual ClustersVirtual Networks (ViNE)DIWU - Do It with us - RAIN

TestBed aaS

Provision Everything (PEG) - Provision IaaS - Provision Paas - Provision NetworkSynergistic Monitor (SMON) - IaaS, OS, Ganglia, NagiosDevice Level Access (DLA)DIAAS - Do It as a service


Documents

Introduction to FutureGrid: Towards a Computing Testbed as a Service