Grid Infrastructures and Standards Example: D-Grid · Grid Infrastructures and Standards Example: D-Grid Wolfgang Gentzsch ... Services Basic Grid Services Distributed Data Archive

ETSI Sophia Antipolis 4th e-Infrastructure Concertation December 2007

4th e-Infrastructure Concertation MeetingETSI, Sophia Antipolis, 5 – 6 December 2007

Grid Infrastructures and StandardsExample: D-Grid

Wolfgang Gentzsch

Water Roads Power Knowledge


Motivation: Towards a Global Society

Old World New World

Static

Silo

Physical

Manual

Application

Dynamic

Shared

Virtual

Automated

Service

From Silo Oriented Architectures to

Service Oriented Architectures

CourtesyMark Linesch, OGF

IT

3

A Grid Computing Timeline

Globus

, Leg

ion, U

nicore

start

’04’03’02 ’06 2007’05‘012000’99’98’971996

CEC FP5 starts

in199

8

GridForu

ms merge,

form

GGF

EGEE, DEISA sta

rt

OGSA v1.0

Open GridForum

EDG, Cross

Grid, e

tcsta

rt

CEC FP6 starts

in200

2

US, Europea

n, AP Grid

Forums

UK e-Science program startsUK e-Science program starts

• DARPA funds Globus Toolkit & Legion• German Govt funds UNICORE project• NSF funds National Technology Grid• NASA starts Information Power Grid

• DARPA funds Globus Toolkit & Legion• German Govt funds UNICORE project• NSF funds National Technology Grid• NASA starts Information Power Grid

Today:• Grid solutions are common for HPC• Grid-based business solutions are

becoming common• Technologies & standards are evolving

Today:• Grid solutions are common for HPC• Grid-based business solutions are

becoming common• Technologies & standards are evolving

Japan Govt funds NAREGIJapan Govt funds NAREGI

Source: OGF, modified

German Govt funds D-GridGerman Govt funds D-GridFrench Grid5000 programFrench Grid5000 program

CEC FP7

4

Three Generations of Grid

Source: Charlie Catlett

• Local “metacomputers“– Distributed file systems– Site-wide single sign-on

• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs-of-concept



5







• Utilize software services and communications protocolsdeveloped by grid projects:

– Condor, Globus, UNICORE, Legion, g-Lite, etc.• Need significant customization to deliver complete solution• Interoperability is still very difficult!



6








– Condor, Globus, UNICORE, Legion, etc.• Need significant customization to deliver complete solution• Interoperability is still very difficult!



• Common interface specifications support interoperabilityof discrete, independently developed services

• Competition and interoperability among applications,toolkits, and implementations of key services



7



Standardization is key for third-generation grids!StandardizationStandardization is key for thirdis key for third--generation grids!generation grids!













8



Standardization is key for third-generation grids!StandardizationStandardization is key for thirdis key for third--generation grids!generation grids!


• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept


• "Metacenters" explore inter-organizational integration• Totally custom-made, top-to-bottom: proofs of concept









We arehere!

We arehere!


Many Grid Projects:

Grid5000

http://www.deisa.org/

http://www.gridpp.ac.uk/

http://grid.infn.it/index.php?infngrid

http://www.westgrid.ca/home.html

http://www.grangenet.net/

http://www.gridlab.org/

http://www.ivdgl.org/grid2003/index.php

http://cagraidsvr06.cs.tcd.ie/index.html

http://www.grid.it/


e-Infrastructures are complex !1. Resources: Networks with computing and data nodes

2. Development/support of standard middleware & grid svcs

3. Internationally agreed AAA infrastructure

4. Discovery services and collaborative tools

5. Data provenance, curation and preservation

6. Open access to data and publications via interoperable repositories7. Remote access to large-scale facilities: Telescopes, LHC, ITER, ..

8. Application- and community-specific portals and workflows

9. Industrial collaboration

10. Service Centers: maintenance, support, training, utility, apps

Courtesy Tony Hey

Standards, Standards, Standards ! ! !


Analysing Basic Grid Services

2. discover resource

3. submit job

4. transfer data

1. secure environment


Analysing Basic Grid Services


3. submit job

4. transfer data



3. submit job

4. transfer data



3. submit job

4. transfer data


discover

submit

Data transfer

UNICOREgLite

Globus

13

Standards Bodies• GGF (Global Grid Forum, 2000) => OGF

− Grid Architecture: OGSA, CDDLM, WS-Agreement, …

• OASIS (Organization for the Advancement ofStructured Information Standards, 1993)− Middleware/Web services focused− WSRF, WS-Notification, WSDM, WS-Security…

• DMTF (Distributed Management Task Force, 1992)− Management and Information models (CIM)− Server management− WS-CIM

• W3C (WWW Consortium, 1994)− WS-Addressing

Not a complete list!Not a complete list!

Logos are trademarks or service marks of their respective owners. Source: OGF, modified

· And IETF, Liberty Alliance, WS-I, EGA

· ETSI (European Telecom. Standards Institute, 1988)− Standardization of ICT in Europe− 2006: ICT GRID Interoperability Testing Framework− ETSI series of GRID Plugtests


Building a National e-Infrastructure for Research and Industry

• 01/2003: Pre-D-Grid Working Groups � Recommendation to Government• 09/2005: D-Grid-1: early adopters, ‘Services for Science’• 07/2007: D-Grid-2: new communities, ‘Service Grids’• …/2008 ? D-Grid-3: Service Grids for research and industry

• D-Grid-1: 25 MEuro > 100 Orgs > 200 researchers• D-Grid-2: 40 MEuro > 100 addl Orgs > 200 addl researchers and industry

� Important:� Sustainable production grid infrastructure after the end of the funding� Integration of new communities� Evaluating business models (operational models) for grid services

*) funded by the German Federal Ministry for Science and Education

Case Study: D-Grid e-Infrastructure *)


D-Grid -1, -2, -32005 - 2011

Generic Grid Middleware and Grid Services

Integration Project DGI-2

Ast

ro-G

rid

C3-

Grid

HEP

-Grid

IN-G

rid

Med

iGrid

ON

TOVE

RSE

WIK

ING

ER

WIS

ENT

Text

grid

. . . . . .

Im W

isse

nsne

tz

Knowledge Management

Business Services, SLAs, SOA Integration, Virtualization

User-friendly Access Layer, Portals


Nutzer

ApplicationDevelopment

andUser Access

GAT API

Data/Software

Resourcesin D-Grid

High-levelGrid

Services

Basic GridServices

DistributedData Archive

User

NetworkInfrastructur

gLite

Globus

AccountingBilling

User/VO-Mngt

SchedulingWorkflow Management

Data management

Security

Plug-In

UNICORE

DistributedCompute

Resources

GridSphere

Monitoring

D-Grid: 3 Grid Middlewares !

gLite

Globus

Unicore

17171717

GlobusGlobus Toolkit ArchitectureToolkit Architecture

Pre-WSAuthenticationAuthorization

GridFTP

GridResource

Allocation Mgmt(Pre-WS GRAM)

Monitoring& Discovery

Services(MDS2)

Web ServicesAuthenticationAuthorization

ReliableFile Transfer(RFT) Service

OGSA DataAccess Interface

(OGSA-DAI)

Web Service GridResource

Allocation Mgmt(WS GRAM)

Web ServiceMonitoring &Discovery

Services (MDS4)

CredentialManagement

MyProxy

ReplicaLocationService

DelegationService

GridWayMetascheduler

C CommonLibraries

Java WS Core

eXtensibleInput-Output

(XIO)

C WS Core

WSWScomponentscomponents

NonNon--WSWScomponentscomponents

CommonRuntime

Security Datamanagement

Executionmanagement

Informationservices

WorkspaceManagement

Web MDS

Courtesy Gabriel Mateescu

1818

Standards inStandards in GlobusGlobus�� Security and clientSecurity and client--service communicationservice communication

�� X.509, XML, WSDL,X.509, XML, WSDL, SOAP,SOAP, SSL/TLS, HTTPSSL/TLS, HTTP�� WSWS--RF,RF, WSWS--Addressing, WSAddressing, WS--Notification, WSNotification, WS--SecuritySecurity�� Under adoption: SAMLUnder adoption: SAML

�� Information System, Monitoring & AccountingInformation System, Monitoring & Accounting�� XML, XMLXML, XML--schema, GLUE Schemaschema, GLUE Schema

�� Job ManagementJob Management�� DRMAA;DRMAA; under adoption: BES, JSDLunder adoption: BES, JSDL

�� Data ManagementData Management�� GridFTPGridFTP, DAIS, DAIS

Courtesy Gabriel Mateescu

http://www.unicore.eu 19http://www.unicore.eu

UNICOREWS-RF

hostingenvironment

XNJS

ArchitectureArchitecture

IDB

UNICOREAtomicServices

OGSA-*

XACMLentity

emergingstandardinterfaces

Grid serviceshosting

job incarnation &authorization

authentication

scientific clientsand applications

parallel scientific jobsof multiple end-users

on target systems

UNICOREWS-RF

hostingenvironment

XNJSIDB

UNICOREAtomicServices

OGSA-*

XACMLentity

Gateway

UNICOREWS-RF

hostingenvironment

ServiceRegistry

Gateway

ByteIO

BES

RUS

ByteIO

BES

RUS

XACML XACML

HPC-P

GPEapplicationclient

command-lineclient

Eclipse-basedclient

Portal client,e.g. GridSphere

WS-ISOAP JSDL

XUUDB

SAML-VOMS

X.509

UR

JSDLHPC-P

UR

JSDL

WS-RF

SAML

X.509

Local RMS (e.g. Torque, LL, LSF, etc.)

Target System Interface

Local RMS (e.g. Torque, LL, LSF, etc.)

Target System InterfaceDRMAA DRMAA

Courtesy Achim Streit

http://www.unicore.eu 20http://www.unicore.eu

Standards inStandards in

� Security�Full X.509 certificates as base line, XACML based access control�Support for SAML-based VOMS & X.509 proxies in development

� Information system, monitoring, accounting�GLUE 2.0 information service in development (strong interaction with

the GLUE WG)�OGSA-RUS for accounting in development (incl. UR for storing)

� Job management�OGSA-BES, HPC-P: creation, monitoring and control of jobs� job definition compliant with JSDL (+ JSDL HPC ext.)�DRMAA communication to local resource manager for job scheduling

� Data management�Fully OGSA-ByteIO compliant for site-to-site transfers

� Web-Services (WS-RF 1.2, SOAP, WS-I) stack !Courtesy Achim Streit

http://www.ogf.org/index.php

21

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688 21

Workload ManagementData Management

Security Information & Monitoring

Access

gLite Grid Middleware Services

API

ComputingElement

WorkloadManagement

MetadataCatalog

StorageElement

DataMovement

File & ReplicaCatalog

Authorization

Authentication

Information &Monitoring

ApplicationMonitoring

Auditing

JobProvenance

PackageManager

CLI

Accounting

Site Proxy

WS-I, X.509+VOMS-ACSAML XACML

RGMA, GLUE v1.3SAGA, GLUE v2.0

WS-I, SRM v2.2, GridFTPDAIR

WS-IBES, JSDL

URRUS

Courtesy Claudio Grandi

22

Enabling Grids for E-sciencE

EGEE-II INFSO-RI-031688 22

Standards in gLite• Security

– Use X.509 certificates and VOMS Attribute Certificates– In future SAML and XACML for attribute and policy management

• Information system, monitoring and accounting– GLUE schema (1.3 now, 2.0 in future) accessed through LDAP

� In future use a SAGA compliant interface for access– UR for description of accounting usage records

� In future will adopt the RUS interface

• Job Management– Adopt BES interface in CREAM (in OMII-Europe)

� but currently not descriptive enough– JSDL (with extensions) used to describe jobs

• Data Management– SRM 2.2 interface for data access and GridFTP for file transfers

• Use a Web Service Interface wherever possible– When performance allows it

Courtesy Claudio Grandi


Standards Summary gLite Globus Unicore----------------------------------------------------------------Security X.509 X.509 X.509

VOMS VOMS VOMSSAML SAML SAMLXACML XACML XACML

----------------------------------------------------------------Information Syst, GLUE GLUE GLUEMonitoring & Acctg XML XML XMLAccounting UR RUS/UR----------------------------------------------------------------Job Management BES BES BES

JSDL JSDL JSDLDRMAA DRMAA DRMAA

----------------------------------------------------------------Data Management GridFTP GridFTP

SRM2.2 DAIS ByteIO----------------------------------------------------------------


Standards Summary gLite Globus Unicore----------------------------------------------------------------Security X.509 X.509 X.509

VOMS VOMS VOMSSAML SAML SAMLXACML XACML XACML

----------------------------------------------------------------Information Syst, GLUE GLUE GLUEMonitoring & Acctg XML XML XMLAccounting UR RUS/UR----------------------------------------------------------------Job Management BES BES BES

JSDL JSDL JSDLDRMAA DRMAA

----------------------------------------------------------------Data Management GridFTP GridFTP

SRM2.2 DAIS ByteIO----------------------------------------------------------------

=> Standards-based Interoperability in OMII-Europe

e-IRG Workshop EML Heidelberg, April 19 – 20, 2007

Germany’s EU Presidency


Our Goal: Sustainability of e-Infrastructurese-IRG Workshop in April ’07:

� Theme 1: Towards a European Grid Infrastructure.Lessons, recommendations: EGI, OMII-UK, HET, CEC, DEISA, Tony Hey

� Theme 2: Sustainability for e-Infrastructures.Sharing policies, resource provisioning, Grid economy, business models,national Grid services, e-social science, large-scale researchinfrastructures

� Theme 3: Bridging the gap between academia and industry.Innovation, industry involvement, EGEE and CoreGRID experience withindustry, strategy for collaboration

e-IRG Workshop EML Heidelberg, April 19 – 20, 2007



Our Goal: Sustainability of e-Infrastructurese-IRG Workshop in April ’07:

� Theme 1: Towards a European Grid Infrastructure.Lessons, recommendations: EGI, OMII-UK, HET, CEC, DEISA, Tony Hey

� Theme 2: Sustainability for e-Infrastructures.Sharing policies, resource provisioning, Grid economy, business models,national Grid services, e-social science, large-scale researchinfrastructures

� Theme 3: Bridging the gap between academia and industry.Innovation, industry involvement, EGEE and CoreGRID experience withindustry, strategy for collaboration

Standards are the key towards achieving these goals !


Courtesy Dieter Kranzlmueller

Last but not least:Standards are a prerequisite for an

International Grid Community

http://www.recherche.gouv.fr/recherche/aci/grid.htm


http://www.ngs.ac.uk/

http://www.grid.it/



http://www.diligentproject.org/


Courtesy Dieter Kranzlmueller

European

GridIn

itiativ

eLast but not least:

Standards enable D-Grid to become part of theInternational Grid Community

http://www.recherche.gouv.fr/recherche/aci/grid.htm


http://www.ngs.ac.uk/

http://www.grid.it/



http://www.diligentproject.org/


Knowledge Engine

[email protected]

Thank You !Combustion Engine

Steam Engine

19th Century

20th Century

21th Century

mailto:[email protected]

Documents

Grid Infrastructures and Standards Example: D-Grid · Grid Infrastructures and Standards Example: D-Grid Wolfgang Gentzsch ... Services Basic Grid Services Distributed Data Archive