EGI technical platforms for advanced · PDF fileEGI technical platforms for advanced computing Tiziana Ferrari Technical Director, EGI Foundation. 2 Outline • Introduction to EGI

www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme

of the European Union under grant number 654142

EGI technical platforms for advanced computing

Tiziana Ferrari Technical Director, EGI Foundation

2

Outline

• Introduction to EGI • Services for distributed computing, data

management and AAI • New requirements, new challenges • Towards the European Open Science Cloud

Astronomical Data Analysis Software and Systems 2016 3

EGIFoundation


EGI Membership

• Major national e-Infrastructures: 22 NGIs • EIROs: CERN and EMBL-EBI • EGI Foundation • (ERICs)

https://www.egi.eu/about/egi-foundation/


International Partnerships

AfricaandArabiaCouncilforScientificandIndustrialResearch,SouthAfrica

IndiaCentreforDevelopmentofAdvancedComp.

ChinaInst.OfHEPChineseAcademyofSciences

LatinAmericaUniversidadeFederaldoRiodeJaneiro

UkraineUkrainianNationalGrid

USA

Canada

AsiaPacificRegionAcademiaSinicaatTaiwan


EGI Federation, 2016 QR3 The largest distributed compute e-Infra worldwide

23Cloudproviders,+300datacentres

+250000instantiatedVMs/year

1.7Millionjobs/day

2.6BillionCPUhours/year+26%

>48000users,+25%


Serving researchers and innovators

ESFRIs,FETflagships

Sizeofindividualgroups

Multinationalcommunities ‘Longtail’

WLCGCTAELIXIREPOSEISCAT_3DBBMRICLARINLOFAREMSOELILifeWatchICOSEMSOCORBELENVRIplus…

VREprojectsWeNMRDRIHMVERCEMuGAgINFRACMMSTLSGCSuperSitesExploitationEnvironmentalsci.neuGRID…

PeachNoteCEBAGalaxyeLabSemiconductordesignMain-beltcometsQuantumpysicsstudiesVirtualimaging(LS)BovinetuberculosisspreadConvergentevol.ingenomesGeographyevolutionSeafloorseismicwaves3DlivermapswithMRIMetabolicratemodellingGenomealignmentTapewormsinfectiononfish…

Industry, SMEs

AgroknowCloudEOCloudSMEEcohydrosgnubilaSinergiseSixSqTEISSTerradueUbercloud…


Supporting international research communities and thematic services

Example:StructuralBiology Distributionofusers(2016,QR3)

➢ 2700users➢ 81countries

(credits:A.Bonvin,WeNMR)

Installedcomputecapacitytrends2011-2016


Astronomy/Astrophysics/Astro-particle physics projects and RIs in EGI

ARGO, AUGER, CTA, KM3NeT, LHCb, LOFAR, Large Synoptic Survey Telescope/LSST, PAMELA, ESA Planck Mission, XENON etc.

2010-2016: > 25 M job


Supporting international research communities and thematic services

StructuralBiology Distributionofusers(2016,QR3)

➢ 2700users➢ 81countries

(credits:A.Bonvin,WeNMR)

Installedcomputecapacitytrends2011-2016


Services Catalogue

http://go.egi.eu/ServiceCatalogue




Runvirtualmachineson-demandwithcompletecontroloverthecomputingresources• On-demandprovisioning• Fullcontrolovercomputingresources

• Standardinterfacetodeployonmultipleserviceproviders

Benefits• Executecompute-anddata-intensiveworkloads,includingGPGPUcomputinginthecloud

• Hostlong-runningservices

Cloud Compute


EGI Federated Cloud• System of cloud

infrastructures • Standard user interfaces

– Clouds and their interconnections are based on open standards, open technologies

– Based on OCCI/OGF and OpenStack

• Harmonised operational behaviour

• Value proposition: distributed cloud computing for analysis of distributed large datasets


Benefits

OpenStack OpenNebula

OpenStackOpenNebula

OpenStack

Synnefo

Harmonisedoperation

CloudregistryInformationsystemVirt.Machinemarketpl.UsageaccountingAccesscontrol

Uniform userinterfaces

OpenStackNova

-Oneverysite

-OnOSsites

CDMI-onanysite• OpenStackSWIFT–onOSsites

VMandblockstoragemanagement: Objectstoragemanagement(optional):


EGI Federated Cloud

EGIFederatedCloudisacollaborationofcommunitiesdeveloping,innovating,operatingandusingcloudfederationsforresearchandeducation.

Today:• 23providersfrom14NGIs

• 15OpenStack• 6OpenNebula• 1Synnefo

• ~7.000coresintotal


Porting of LOFAR calibration pipeline 1/2

Credits: Susana Sánchez Expósito - CSIC


Porting of LOFAR calibration pipeline 2/2


Outcomes

• The computing capabilities fulfil the requirements from the use case

• The memory and cpu needs depends on the specific pipeline step, and the EGI federated cloud allows to configure virtual machines with different capabilities.

• A better storage solution is needed • The user data are too large to be stored in the VM

images. They should be stored in volumes easily mountable from several VMs and synchronized across different cloud providers.

• COMPSs facilitates the porting and deployment of the application


Cloud Container Compute

RunDockercontainerswithinisolateduser-spacewithnooverhead• On-demandprovisioning• Lightweightenvironmentformaximizedperformance

• Standardinterfacetodeployonmultipleserviceproviders

Benefits• Reducetimetoproductionbyremovingfrictionbetweendevelopmentandoperationsenvironments

• Interoperableandtransparent


High-Throughput Compute

Analyzelargedatasetsbyexecutinglargenumbers(thousands)ofcomputationaltasks• Accesstohigh-qualitycomputingresources

• Integratedmonitoringandaccountingtoolstoprovideinformationabouttheavailabilityandresourceconsumption

• Workloadanddatamanagementtoolstomanageallcomputationaltasks

Benefits• Largeamountsofprocessingcapacityoverlongperiodsoftime

• Fasterresultsforyourresearch• Sharedresourcesamongusers,enablingcollaborativeresearch


CTAresourcepoolinEGI:• 20siteswithapprox.8000CPUcores• Disk:1.3PBin6sites(OnlineStorage)• Tape:400TBin3sites(ArchiveStorage)

DIRACforCTA:• CTA-specificextensionofDIRAC• FileCatalogue~21millionfiles• Computingtasks

• 8millionsofar;2.6PBprocessed• Datatransformationtasksvs.Usertasks

CTA:MonteCarlosimulationsandAnalysis

• UseofEGIHighThroughputCompute,OnlineStorage,ArchiveStorage• UseofDIRACforDataCataloguingandWorkloadManagement

Metadataselection

Catalogbrowsing

Queryresult


Online storage

Store,shareandaccessyourlesandtheirmetadataonaglobalscale• Assignglobalidentifierstofiles• Accesshighly-scalablestoragefromanywhere

• Controlthedatayoushare• Organiseyourdatausingaexiblehierarchicalstructure

Benefits• Highlyscalablestoragesystemaccessiblefromanywhere

• Easilysharedata• Accessthroughdifferentinterfaces


Archive storage

Back-upyourdataforthelongtermandfutureuseinasecureenvironment• Storedataforlong-termretention• Storelargeamountofdata• FreeupyouronlinestorageBenefits• Storeslargeamountsofdata• Long-termretention• Reliableandinteroperable


Data Transfer

Transferlargesetsofdatafromoneplacetoanother• Moveresearchdatafast• Specializedanalyticsofon-goingtransfers• Userinterfacetomanagetransferandnetworkresources

Benefits• Idealforverylargefiles• Abletohandlelargeamountsoffiles• Transferprocesswithautomaticretry


EGI AAI New Architecture

EGIServices

EGICheckIn

IdP

AttributeAuthority

EGIUIDFirstname,last

name

email affiliation

Mandatory Attributes


Why a IdP/SP Proxy?

• Service Providers (SPs) can have one statically configured IdP

• No need to run an IdP Discovery Service on each EGI SP

• Connected SPs get harmonised user identifiers and accompanying attribute sets from one or more AAs that can be interpreted in a uniform way for authZ purposes

• External IdPs only deal with a single EGI SP proxy EGI services will not have to deal with the complexity of

multiple IdPs/Federations/Attribute Authorities/technologies.

27

Open Data Platform

• Manage entire data life cycle from raw data to preservation

• Combine efficient computation services with open data managed by federated infrastructures – No local staging of data for processing

• Share public datasets for download or reuse

• Make public datasets discoverable

AstronomicalDataAnalysisSoftwareandSystems2016

28

Open Data Platform interfaces

GUI• Webbased

• Easydatamanagementandsharing,accesscontrol

• • Publicationofdataitems

REST• Advanc

eddataandcollectionmanagementAPIforintegrationwithcommunitytoolsandportals

CDMI• Standar

ddatamanagementoperations

• Advancedmetadataqueries

• Integrationwithfuture

POSIX• Enable

directmountingofspacesinthelocalfilesystemwithoutfulldatatransfer

OAI-PMH• OAI

DataProviderinterface

• DublinCoremetadatabydefault

• Morecomplexmetadatacanbe

HTTP• Direct

downloadofopendatafromURL’s

29

Open Data Platform workflow

Private Resources

Dataset

Share of Dataset

Direct mount Via POSIXCopy of

Data-set-1.1

Private Resources

ViewdatasetpreviewinbrowserusingDOI

Copydataset

Discoverdataset

Mount open data set directlyon desktop or VM (in read only mode)

Create a share from folder or file withMetadata in DC

PublicServicesForDataDiscovery

Register identifier (e.g. DOI)

LazyReplication

Discov

erdat

aset

andm

ountsha

re

Expose share metadata over OAI-PMH


EGI role towards the European Open Science Cloud

• An Open Science Service Exchange as partnership of public/commercial organizations and initiatives responsible for

– Provisioning of wide set of services to researcher and innovators ! consolidation of national e-Infrastructures, open standards, technical and business process integration among the suppliers (e-Infrastructures, Research Infrastructures etc.)

– Platform integration for community-specific capabilities with coordinated outreach

– Aggregation of demand for economies of scale, technical requirements translations, cross-border access via brokering and procurement, end-to-end operations

– Development of human capacity – A “Digital innovation hub” to support innovation with industry/

SMEs

www.egi.eu

Thank you for your attention.

Questions?

This work by Parties of the EGI-Engage Consortium is licensed under a Creative Commons Attribution 4.0 International License.

AcknowledgementsThispresentationusediconsmadebyFreepikfromwww.flaticon.com

http://creativecommons.org/licenses/by/4.0/

Documents

EGI technical platforms for advanced · PDF fileEGI technical platforms for advanced computing Tiziana Ferrari Technical Director, EGI Foundation. 2 Outline • Introduction to EGI