Using a Widely Distributed Federated Cloud System to Support Multiple Disparate User Communities

www.egi.eu EGI-InSPIRE RI-261323

EGI-InSPIRE


Using a Widely Distributed Federated Cloud System to Support Multiple

Disparate User Communities

David Wallom University of Oxford

Chair, EGI Federated Cloud


Overview

• Why • What • Who


Growth of Providers

Grid Limited academic resource providers

Cloud Diversity and scale of new resource providers


Growth of Userbase

20k researchers 2M researchers!

Few related use cases

Many diverse use cases


Hardware Hardware

Hardware Hardware

Hardware

Cloud Management Cloud Management

Cloud Management Cloud Management

Cloud Management

User Communities

User Communities

User Communities

Federated interfaces

Federated services

• Open Standards

• Technology independant

• Provider agnosticism

• Production integration and reuse

Principles of Federation


EGI Cloud Infrastructure

6

EGI Core Platform

Federated AAI

Service Registry

Monitoring Accounting

EGI Cloud Infrastructure Platform

Instance Mgmt

Information Discovery

Storage Management

Cloud Management Stacks (OpenStack, OpenNebula, Synnefo, …)

Help and Support

Security Co-ordination

Training and Outreach

EGI C

olla

bora

tion

Tool

s

EGI A

pplic

atio

n D

B Im

age

Repo

sito

ry

EGI C

loud

Ser

vice

Mar

ketp

lace

Sustainable Business Models

User Community

GSI GLUE2

OCCI CDMI

SAM UR

OVF

Cloudinit


Compatibility for technology providers

Cloud Mgmt. Fram. Fed. AAI Monitoring Accounting Img. Mgmt. OCCI CDMI

OpenStack Yes Yes* Yes Yes Yes Yes

OpenNebula Yes Yes* Yes Yes Yes Yes

Synnefo Yes Yes Yes - Yes -

Cloudstack - - - - Yes -

Emotive Yes Yes* - N/A N/A Yes

SlipStream Yes N/A N/A Yes Yes Yes*

Perun Yes Yes N/A N/A N/A N/A

7

Example documentation for a CMF:- https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack

https://wiki.egi.eu/wiki/Fedcloud-tf:ResourceProviders:OpenStack�


Geographical dispersion • 12 countries provide 19 certified

resources – Czech Republic, Germany, Greece,

Hungary, Italy, Macedonia, Poland, Slovakia, Spain, Sweden, Turkey, United Kingdom

• 2 countries currently integrating – France, Finland, Portugal, Croatia

• 5 countries interested – Bulgaria, Israel*, The Netherlands,

Switzerland

• Worldwide interest – South Africa* (SAGrid) – South Korea* (KISTI) – United States* (NIST, NSF Centres)

* Not shown on map


Utilisation

9

217352 VMs launched across 25 clouds


Federated Cloud Services

Federated IaaS and STaaS Cloud

EYR TC monthly meeting 10

Tier 1: Reliable

Infrastructure Cloud

Tier 4: Zero ICT

Infrastructures

Tier 3: Platform as a Service

Tier 2: General-purpose platform services

PaaS

PaaS

DB

aaS

Had

oop

aaS

VRE

Secure storage

Key

Mgm

t

Encr

ypti

on

ACL

mgm

t

Virtual eLaboratory

17 Feb 2014


AppDB – Virtual Appliance catalogue

11


EGI FedCloud Communities 9/2014

• Ecology – BioVeL: Biodiversity Virtual e-Laboratory

• Structural biology – WeNMR: a worldwide e-Infrastructure for NMR and structural biology

• Linguistics – CLARIN: ‘British National Corpus’ service (BNCWeb)

• Earth Observation – SSEP: European Space Agency’s Supersites Exploitation Platform for

volcano and earthquakes monitoring (Collaboration with Helix Nebula)

• Software Engineering – SCI-BUS: simulated environments for portal testing

• Software Engineering – DIRAC: deploying ready-to-use distributed computing systems

• Software Engineering – Catania Science Gateway Framework

• Musicology – Peachnote: dynamic analysis of musical scores

• Earth Observation – ENVRI: Common Operations of Environmental Research

infrastructures (collaboration with EISCAT3D)

• Geology – VERCE: Virtual Earthquake and seismology Research

• Ecology – LifeWatch: E-Science European Infrastructure for Biodiversity and Ecosystem

Research

• High Energy Physics – CERN ATLAS: ATLAS processing cluster via HelixNebula

More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users

12


New EGI FedCloud Communities since launch

• Education – Cranfield University distributed systems course

• Cultural Heritage – DCH-RP management of preservation services in the cloud

• Hydrological Modelling – Running Hydrological models to support real time analysis

• Bioinformatics – ELIXIR execution of the Ensamble application in the Federated Cloud

environment

• Systems implementations – deployment of FTK developed tools and services and data

preservation

• Internet of Things – Smart Grid systems investigation

• Software Development – deployment of research PaaS

• RNA Sequencing – deployment of analysis engines in the cloud

• Physiological Modelling – Calibration, scenario mapping and development

More info: https://wiki.egi.eu/wiki/Fedcloud-tf:Users

13


WeNMR e-Infrastructure (VirtualCing)

EGI Federated Cloud Infrastructure

…

NMR Protein DB

ToPoS

Virtual CING Virtual

CING Virtual Cing


CING Virtual Cing


CING Virtual Cing

OCCI client

Virtual Common Interface for NMR structure generation (VirtualCing), developed in WeNMR, has been deployed via the EGI Federated Cloud

14


ESA HN Flagship on the EGI Federated Cloud

The European Space Agency deployed part of the ESA Helix Nebula Flagship into the EGI Federated Cloud:

1. Use SlipStream Infrastructure Broker to deploy a custom computing cluster on multiple Federated Cloud sites (CESNET, GRNET)

2. Connect the computing cluster to the ESA G-POD system and run SAR processing for volcano and earthquake monitoring

15

Automatic Cluster Deployment

EGI Federated Cloud

Computing Cluster

Cluster Head

Working Nodes

OCCI

Globus

Job submission

BLUE BOX

www.egi.eu EGI-InSPIRE RI-261323 16

EISCAT 3D PoC

EISCAT_3D is a three-dimensional imaging radar to be located in the northernmost parts of Europe, to study Earth atmosphere and its coupling to space.

The EISCAT 3D Proof of Concept is a PaaS built on top of the EGI Federated Cloud for Data Access and Dissemination, based on the ENVRI tools, the EGI Federated Cloud Infrastructure and Storage services and the current EISCAT data.


Peachnote

10/9/2014 17

Peachnote is a music score search engine and analysis platform.

Hundreds of thousands of music scores are being digitized by libraries all over the world. In contrast to books, they generally remain inaccessible for content-based retrieval and algorithmic analysis. There is no analogue to Google Books for music scores, and no large corpora exists that can empower advanced analysis on music scores. Peachnote want to help change that providing visitors and researchers access to a massive amount of symbolic music data.

EGI Federated Cloud

OMR Worker

PDF Splitter

OMR Feeder

OMR Worker

…


OpenModeller on the Biovel Portal

10/9/2014 18

The Ecological Niche Modeling (ENM) Workflow takes as input a file containing species occurrence points to create a model with the openModeller Web Service.

• The EUBrazilOpenBio ENM service is exposed through an extended openModeller Web Service interface

• Multi-staging and multi-parametric oM experiments are implemented through COMPSs that dynamically creates the virtual resources to execute the operations.

• An OCCI connector is used for the VMs management while data management supports CDMI endpoints.

ENM Service (OMWS2)

VENUS-C Cloud Middleware

COMPSs Workflow Orchestrator OCCI CDMI

EGI Federated Cloud

Service available at https://portal.biovel.eu/

https://wiki.biovel.eu/display/doc/Ecological+Niche+Modelling+(ENM)+Workflow�

http://www.bsc.es/compss�

https://portal.biovel.eu/�

www.egi.eu EGI-InSPIRE RI-261323 10/9/2014 20

Running GNU Octave and R on the CHAIN-REDS Science Gateway

20

CHAIN-REDS Science Gateway -CHAIN-REDS RI-306819

http://science-gateway.chain-project.eu/�


Strengthening the underpinning platform

Technology Evolution • Broader support for open standards in Cloud

management frameworks – Utilisation of rOCCI for interfaces to commercial cloud

frameworks • New feature additions to foundational tools

depending on requests – Accounting, monitoring, service discovery, Application

DB • Innovative deployments of opensource PaaS • Large improvements on documentation for all 3

stakeholder groups

22

Valu

e ad

ded

serv

ices

for U

ser

Com

mun

ities


Conclusions The EGI Federated Cloud, a federation of institutional private Clouds, offering Cloud

Services to researchers in Europe and worldwide • A single cloud system able to

– Scale to user needs – Integrate multiple different providers to give resilience – Prevent vendor lock-in through open standards – Enable resource provision targeted towards the research community

• Paving the way for a global federated cloud marketplace

– Established best practice – Illustrating leadership – Open standards, open technology – Open membership, open processes – Open competition

Current capability – ~6,000 cores, 300 TB storage Promised through EGI-Inspire – 18,000 cores, 6000 TB storage 2020 Target – 1M cores, 1EB storage

23

Alison Packer, Álvaro López García, Alvaro Simon, Binh Minh Nguyen, Björn Hagemeier, Boris Parak, Boro Jakimovski, Cal

Loomis, Christos Loverdos, Daniele Cesini, Daniele Lezzi, David Blundell, Diego Scardaci, Elisabetta Ronchieri, Emir Imamagic, Enol Fernandez, Feyza Eryol, Florian Feldhaus, Gergely Sipos,

Ivan Diaz, Jan Meizner, John Gordon, Kostas Koumantaros, Malgorzata Krakowian, Marios Chatziangelou, Marco Verlato,

Marica Antonacci , Mattieu Puel, Matteo Turilli, Michel Jouvin, Michel Drescher, Miroslav Ruda, Nuno L. Ferreira, Owen Synge,

Paul Miller, Peter Solagna, Piotr Kasprzak , Roberto Barbera, Ruben Valles, Sándor Ács, Salvatore Pinto, Silvio Spardi,

Soonwook Hwang, Steven Newhouse, Stuart Pullinger, Thijs Metsch, Tomasz Szepieniec, Viet Tran, Zeeshan Ali Shah and all

the other members of the EGI federated Cloud Task Force, activity, workpackage


Thank you!

Questions?

25

Task Force resources

Mailing List: [email protected]

Wiki site: http://go.egi.eu/tf-fedclouds

GitHub: https://github.com/EGI-FCTF

Indico site: https://www.egi.eu/indico/categoryDisplay.py?categId=112