HNSciCloud update @ the World LHC Computing Grid deployment board

HNSciCloud Update

WLCG GDB

12 July 2017Bob Jones

CERNBob.Jones <at> cern.ch

Helix Nebula – The Science Cloud

Helix Nebula – The Science Cloud with Grant Agreement 687614 is a Pre-Commercial Procurement Action funded by H2020 Framework Programme

HNSciCloud Joint Pre-Commercial Procurement

Procurers: CERN, CNRS, DESY, EMBL-EBI, ESRF, IFAE, INFN, KIT, STFC, SURFSaraExperts: Trust-IT & EGI.eu

The group of procurers have committed• Procurement funds• Manpower for testing/evaluation• Use-cases with applications & data• In-house IT resources

Resulting services will be made available to end-users from many research communities

Co-funded via H2020 Grant Agreement 687614

Total procurement budget >5M€

12/07/2017

HNSciCloud project phases

Preparation

•Analysis of requirements, current market offers and relevant standards

•Build stakeholder group

•Develop tender material

Implementation and sharing

Jan’16 Dec’18

Each step is competitive - only contractors that successfully complete the previous step can bid in the next

4/5/2016 4

4 Designs3 Prototypes

2 Pilots

Call-off Feb’17

Call-off Dec’17

Tender Jul’16

Phases of the tender are defined by the Horizon 2020Pre-Commercial Procurement financial instrument

What will be procuredA hybrid cloud platform for the European research community

7/12/2017 5

HNSciCloudPCP

Sou

rce:

Clo

ud

Co

mp

uti

ng

for

Go

vies

, DLT

So

luti

on

s,D

avid

Bla

nke

nh

orn

, V

an R

ista

uan

d C

aro

n B

eesl

ey

Combining services at the IaaS level to support science workflows

The R&D services to be developed are to be integrated withResources in data centres operated by the buyers group

GEANT network

Challenges

Innovative IaaS level services integrated with procurers in-house resources and public e-infrastructure as part of a hybrid cloud to support a range of scientific workloads

Compute and Storagesupport a range of virtual machine and container configurations including HPC working with datasets in the petabyte range

Network Connectivity and Federated Identity Managementprovide high-end network capacity via GEANT for the whole platform with common identity and access management

Service Payment Modelsexplore a range of purchasing options to determine those most appropriate for the scientific application workloads to be deployed

Bob Jones, CERN 6

The Pre-Commercial Procurement process

12/07/2017

12/07/2017

12/07/2017

12/07/2017

12/07/2017

HNSciCloud Prototype Activity Report Unclassified 12

OVERVIEW OF PROTOTYPE CLOUD PLATFORM

Integrated

Realistic

Minimally Intrusive

Scalable

Managed

Sustainable

Functional components to deliver complete data analysis platform…


OVERVIEW OF PROTOTYPE SOLUTION











Martin Gasthuber | Next Steps | 03/04/2017 | Page 17

test phases and resource ramp-up

TP1 TP2 TP3 TP4

2000C

200TB

100C 10TB

3500C

350TB

Activities in Test Phase 1 (TP1): April-June 2017

Establishing network connectivity via GEANT/NRENs,federated identity management via eduGAIN andcreating accounts for tenants

Preliminary testing of the developed services by the Buyers Group has now begun

12/07/2017

Test suiteCERN

• Edugain AAI

• CERN Benchmark Suite

• S3 Endpoint

• PerfSONAR Network

• VM Provisioning and Personalisation

• Grid Jobs in Docker

• Dockerized deployment of EOS+CERNBOX+SWAN

CNRS

• IAAS access via EduGain and local accounts

DESY

• HDF5_IO

EMBL

• CLOUD_BENCHMARK

• ELIXIR_AAI

• data_transparency

ESRF

• FDMNES

IFAE

• monitoring-accounting

• image creation and contextualization

INFN

• BelleII_SIM

• COMPASS/TGEANT_IO

• INFN/GPFS_AFM

• Dynfarm

KIT

• static-batch-extension

• dynamic-batch-extension

• AFM-deployment-test

SURFsara

• BBMRi_data

• dCache_io

• kubertest

12/07/2017

More details: https://cernbox.cern.ch/index.php/s/pjy4lg3cY46vbGt

https://cernbox.cern.ch/index.php/s/pjy4lg3cY46vbGt

Next Steps 20172nd progress review 4-5 September

Scale-up IaaS resourcesMore testing

3rd progress review 9-10 October (DESY)Live webcast demonstrationsFeature freezeScale-up IaaS resources (totalling 10k cores 1PB storage)More testing

End of prototype phase assessment 11-13 December

Selection of 2/3 prototypes for pilot phase

12/07/2017

Pilot Phase 2018

Kick-off event 6-8 Feb 2018 (INFN Bologna)

Progressively scale-up IaaS resourcestotalling 20k cores 2PB storage

Scalability testing

Deployment of applications

Assess procurement/payment models

12/07/2017

Growing the buyers group

Private sector

Gov. agencies

Additional research orgs.

Procuring beneficiaries of

EC project

The initial group of buyers is the set of research orgs. that committed their resources at the start of the project and became beneficiaries of the EC project

Goal is to progressively grow the buyers group by including more publicly funded research organisations: Early adopter group

<see attachment to indico page>

Draft document describesbenefits for the stakeholders,procurement process,steps for implementation

CERN, DESY, EMBL-EBI, ESRF, KIT, CNRS, INFN, PIC-IFAE, SURFsara, STFC

adopter group

Summary

3 prototypes (IBM, RHEA, T-Systems) are being deployed and tested by the Buyers Group

Testing and scaling will continue during 2017

2/3 prototypes will be selected for large-scale pilots in 2018

The Early adopter group will allow more research organisations to procure the resulting services

12/07/2017

Technology

HNSciCloud update @ the World LHC Computing Grid deployment board