www.egi.eu

European Grid Initiative

www.egi.eu

Federated Cloud update

Peter solagnapeter.solagna@egi.eu

Pre-GDB Workshop

04/19/23 .... 1

www.egi.eu

Principles of federation

.... 2

HardwareHardwareHardwareHardware

HardwareHardwareHardwareHardware

HardwareHardware

Cloud ManagementCloud ManagementCloud ManagementCloud Management

Cloud ManagementCloud ManagementCloud ManagementCloud Management

Cloud ManagementCloud Management

User CommunitiesUser Communities

User CommunitiesUser Communities

User CommunitiesUser Communities

Federated interfaces Federated interfaces

Federated services

Federated services

• Standards and validation: Recommended and common open standards for the interfaces and images – OCCI, CDMI, OVF, GLUE2.

• Resource integration: Cloud Computing to be integrated into the existing production infrastructure.

• Security coordination, operations framework

• Heterogeneous implementation: no mandate on the cloud technology. Operational integration of the interfaces.

www.egi.eu

Services offered

• Service deployment

• HTC over cloud

• Heavy memory applications

• Different access levels: IaaS,PaaS, SaaS• There are already several tools integrated• Continuous activity of integrating more platforms

in the federated cloud

.... 3

www.egi.eu

EGI Cloud infrastructure

.... 4

EGI Core Platform

Federated AAI

ServiceRegistry

Monitoring Accounting

EGI Cloud Infrastructure Platform

Instance Mgmt

Information

Discovery

Cloudinit

Cloudinit

Storage Manageme

ntCDMICDMI

Cloud Management Stack(OpenStack, OpenNebula, Synnefo, …)

Help and Support

Security Co-

ordination

Training and

Outreach

EG

I App

licat

ion

DB

Imag

e R

epos

itory

EG

I Clo

ud S

ervi

ce M

arke

tpla

ce

Sustainable

Business Models

OCCIOCCI

www.egi.eu

VM Management

• Uniform VM Management through OCCI• Plugins/interfaces for Openstack

Opennebula and Synnefo

• Cloud init• Contextualization tool supported Extending

this capability with other configuration tools such as puppet

.... 5

www.egi.eu

Expanding the interfaces supported

• Standard interfaces allow uniform behavior across providers• EGI invested on the common interfaces from the very beginning and is

committed to continue the support to those interfaces also in EGI-Engage

• Top down approach, but this built momentum for the federation

• EGI technological evolution is also user-driven• User communities are already using native interfaces in their

workflows• EGI will support user requirements for the operational integration of

the interfaces• This can be hard work (easy for O.Stack, less easy for O.Nebula), need for

clear use cases from the user communities

• Steps for operational integration• Monitoring of availability• Accounting of usage• Support user authn/authz in the federation

.... 6

www.egi.eu

VMI Management

• A single point for control for the management and control of VO specific application packages across clouds

• Catalogue of available Software Appliances (AppDB)• Support for VO-wide image lists following the HEPiX format • Support requirement for endorsed images only to be distributed• Integration with the EGI Information system• Extended authentication system to support federated AAI through

eduGAIN• Integration with external metadata repositories to provide broader

support to communities and engage with other activities• Automated processes for VMI distribution

• Security policy for VMI endorsement being updated by EGI SPG

.... 7

www.egi.eu

Accounting

• New version of usage record for cloud resources under implementation• Including also VM Images ID

• Publishing scripts have been debugged to report correct usage values

• Accounting probes can report also cloud usage not performed through OCCI interfaces• Depending on the middleware the information can be

more or less complete (user information, VO, …)

• Normalization is still an open issue: collaboration is more than welcome!

.... 8

www.egi.eu

Monitoring

• Currently monitored capabilities:• Standard interfaces: OCCI/CDMI• Accounting• Integration with AppDB: vmcatcher

• Availability calculation engine ARGO allows flexible profiles • Production A/R calculation engine from mid 2015• Will allow easier introduction of custom A/R profiles

• Introduce monitoring for native interfaces• Pros: a lot of probes already available from other

communities. Allow certification for sites exposing only native interfaces

• Cons: manage the credentials used for monitoring

.... 9

www.egi.eu

X509 support

• X509 is still the federated AuthN infrastructure that works™

• X509 support implemented in Keystone, to be used with native OpenStack interfaces and OCCI

• OpenNebula supports X509 only through OCCI

• In EGI-Engage, Cloud services will lead the extension of EGI AAI framework to support new authentication technologies

.... 10

www.egi.eu

Improve user tracking

.... 11

• Most of the science gateways are using robot certificates to generate short-term proxies for the users

• Use of robot certificates proxies can be extended to improve accounting and user tracking

Robot Certificate info VO Information

The same for every user of the gateway

User UIDUser UID

The UID is provided by the science gateway. The user will have the same UID using

different science gateways. It’s a per-user sub-proxy

The UID is provided by the science gateway. The user will have the same UID using

different science gateways. It’s a per-user sub-proxy

www.egi.eu

How big research infrastructures can benefit from the EGI Fedcloud experience?

.... 12

Full integration in the EGI Production infrastructure

Accounting

GOCDB

Monitoring

VMI Management

www.egi.eu

• Thanks for the attention

• Questions?

.... 13