The Glenna Project€¦ · – OpenStack regions concept works in SNIC Cloud • Challenge to find...

The Glenna ProjectNeIC Board Meeting

December 2015

Dan Still (CSC)

Glenna - Project Idea

2

The goal of the project is to share knowledge and setbest practices on managing cloud services and to createa Nordic federated cloud service, driven by the need ofthe Nordic researchers.

Glenna Project Objectives

1. Enable easy access through the KALMAR2 federated trust service2. Sharing technologies to improve quality and security of cloud

services3. Sharing user experiences on a Nordic level to improve quality and

to increase the available set of services4. Sharing cloud administrative work – improving the service

availability to the users5. Enable data sharing (enabling new research) and – increasing

overall availability and security of data (avoiding data loss)6. Create a resource sharing solution to simplifying the usage and

sharing of Nordic Cloud resources7. Enable billing and accounting within the Nordic Cloud – to create

a fair sharing of resources and funding. This will also simplify forexternal funding of research.

3

4

Workpackages

WP0 Project mgmtWP1 Cloud StorageWP2 Federated resources (accounts, billing (technical

aspects), data transfer)WP3 VM mgmt (security, image banks, configuration mgmt, automated

building)WP4 SaaS and PaaS development (initial focus on data analytics

(Hadoop, Spark))WP5 Policy / SLA / Legal (billing and accounting policy)WP6 User training, Documentation and Dissemination (including

scientific portals?)

Glenna Team: November 2015

Gurvinder Singh WP4 UNINETT gurvinder.singh@uninett.noDaniel Huluka WP3 UiB daniel.huluka@uib.noThomas Röblitz WP2 UiO thomas.roblitz@usit.uio.noRoy Dragseth WP5 UiT roy.dragseth@uit.noUwe Grawert WP3 CSC uwe.grawert@csc.fiRisto Laurikainen WP3 CSC Risto.Laurikainen@csc.fiDan Still WP2, WP3, WP5 CSC Dan.Still@csc.fiOlli Tourunen WP4 CSC Olli.Tourunen@csc.fiHjörleifur Sveinbjörnsson WP5 University of Iceland hs@hi.isMagnús Gíslason WP5 University of Iceland magnus@hi.isDiba Markus WP1 DeIC dtma@dtu.dkLars Næsbye Christensen WP1 DeIC lanac@dtu.dkAndreas Hellander WP4 Uppsala Universitet andreas.hellander@it.uu.seSalman Toor WP1 Uppsala Universitet salman.toor@it.uu.seLars Viklund WP3 Umeå Universitet HPC2N lars@hpc2n.umu.seIngemar Fällman WP2 Umeå Universitet HPC2N ingemar.fallman@umu.se

5

Glenna Use Cases: Scientific Application as a Service

A. Hellander, S. Toor, P. Eeerola,B. H. Jonsson, T. Linden

Project Goals: to better understand how scientific applicationsshould be designed and implemented for secure and efficientexecution in hybrid cloud environments. Questions include:• How to design Master/Slave hybrid setups where VMs reside in or

migrate between different clouds?• Where should the multi-cloud logic reside in such hybrid cloud

applications - at the application or IaaS level?• How to build elastic scientific applications in a sensible manner?• Can runtime tools be developed to understand application

performance in (hybrid) cloud environments?

6

Glenna Use Cases: Harmonie MUSC

• HARMONIE is a collaborative NWP development of manyEuropean countries, including all Nordic countries

• HARMONIE MUSC = single-column version of HARMONIE• widely used R&D tool in meteorological institute and

collaborative universitiesBenefits:• much improved overall performance in the installation and

configuration as a cloud image => more time on actual research

• no platform induced differences => inter-comparison betweendifferent user experiments straightforward

• input data sets available into the cloud storage => no tediousdata transfers

• pave the way towards an open science type of use of theHARMONIE system => growing need due to emerging open NWP datain Nordics

=> way to improve the whole HARMONIE system

HARMONIE

Status: reflections on 2H2015

• Promising developments since Aug 2015– DeIC deic.data.dk cloud data service – accessible through Kalmar2– UiO Lifeportal opened to Kalmar users (test-site)– iPython, R notebook service opened in Finland (Beta)– SAML based authentication works for IaaS i.e. OpenStack Kilo

(authorization still a problem)– Mesos Kubernetes cluster in Norway (future federated access)– SeaFile cloud data service in production at UI. (future federated access)– Successful SaaaS use case run over Smog (UPPMAX) and cPouta IaaS

resources (CSC)– Meteo MUSC software cloudified, undergoing testing– OpenStack regions concept works in SNIC Cloud

• Challenge to find and retain personnel– lack of personnel in Sweden and Denmark, challenges in Finland– Norway & Iceland are O.K.

8

The Glenna Environment

9

Glenna image bank:MUSC (use case)SaaaS (use case)Pouta BlueprintsChipsterOpenFoam

DeIC Data Service: data.deic.dk

10

Data.deic.data

• data.deic.dk – an online data storage and synchronization servicefor secure sharing and accessing active research data and storelarge data sets. The service in in pilot-production.

• OwnCloud based service• Command line tools for filetransfer• Opened up through Kalmar2 (works from Finland, Umeå, Bergen)• 2dkr per gbyte

11

UiO Lifeportal – Bioinformatics (Galaxy)

12

UiO Lifeportal

• The Lifeportal at UiO is a (bioinformatics) portal service to easethe access to high performance computing resources forresearchers.

• Galaxy based (https://galaxyproject.org/)• Long history at UiO• Lifeportal testserver currently accessable from Finland,

Sweden(edugain)• Pricing still open

13

CSC cPouta Blueprints: iPython, R (JuPyter) Notebooks

14

Blueprints JuPyter: iPython, R notebooks

• Jupyter Notebook is a web application that allows you to createand share documents that contain live code, equations,visualizations and explanatory text. Uses include: data cleaningand transformation, numerical simulation, statistical modeling,machine learning etc.

• Runs on cPouta – dynamic allocation of computing resources• When the VM dies data is lost• Successful transfer of data to data.deic.dk• Behind the HAKA federation, Kalmar2/EduGain to be added soon

15

University of Iceland: SeaFile Cloud Data Service

16

University of Iceland: Cloud Data Service

• Based on the opensource SeaFile project• DropBox style service (as deic.data.dk)• In use at the University of Iceland• Currently using AD for authentication• Should be possible to add SAML authentication i.e. Kalmar2

17

Google Kubernetes /Apache Mesos at UiT

• Kubernetes is a system for managing containerized applicationsacross multiple hosts, providing basic mechanisms for deployment,maintenance, and scaling of applications.

• Mesos is built using the same principles as the Linux kernel, only ata different level of abstraction. The Mesos kernel runs on everymachine and provides applications (e.g., Hadoop, Spark, Kafka,Elastic Search) with API’s for resource management and schedulingacross cloud environments.

• PI: Gurvinder Singh• Experimental environments particularly geared towards analytics• Plan to get federated access if applicable (may prove too

experimental)

18

Future targets:

Feb 2016 (was Nov 2015): Demonstration of concept with selectedpilots from 2-5 Nordic countries.– Demonstrations run by end-user, showing the functionality of all

needed steps.– Demonstrator of funding scheme process.

June 2016: Fully functional infrastructure– Nordic Cloud infrastructure services operational with Nordic users,

including the AAI, monitoring, billing and balancing of resourcesneeded. Including defined and agreed organization of responsibilitiesbetween the national resource providers and NeIC.

19

Planned for Demo in February 2016

• Enhanced access to the presented services• Federated access to SNIC-Cloud and cPouta• Federated access to Apache Mesos/Kubernetes (may be too

experimental service)• Use case demos (SaaaS case running over multiple OpenStack

clouds)• Preliminary sketch of contractual structure for fair share and

compensation. Discussion on funding schemes.

20

Challenges

• SAML authentication works in OpenStack Kilo but not authorization.Planned for next release of OpenStack but may prove too late forthe Glenna project

• Kubernetes and Mesos can be considered too experimental towarrant general ”Glenna” use

• June 2016: Fully functional infrastructure: contractualcommitments, funding and compensation will be difficult topics toaddress

21