EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833

Enabling Grids for E-sciencE

www.eu-egee.org

EGEE is a project co-funded by the European Commission under contract INFSO-RI-508833

Web Services, EGEE and

Richard HopkinsTraining Team, National e-Science Centre, Edinburgh, [email protected]

National Grid Services InductionMarch 11 2005

NGS Induction, NeSc, March 11 2005 – Web Service Grids - 2Richard Hopkins, Training Team, NeSC

Enabling Grids for E-sciencEOutline and Acknowledgements

• Grids and Web Services• EGEE• gLite

Acknowlegements

• This talk has drawn on material produced by many people, including - • Tony Hey• Mike Mineter• EGEE colleagues


Enabling Grids for E-sciencESpecialised DistributedComputing

Grids come within the general field of

• Parallel / Distributed Computing

A computational task involves coordination of components which occur

• Simultaneously

And/or

• At physically separated locations

The Distribution dimension -

Multi-processor

ProcessorPool

Embedded Systems

Flexible Manufacturing . . .

GRIDS

Parallel Distributed


Enabling Grids for E-sciencEGrids & Web Services

A very inclusive formulation of “Grid Computing” is

• Coordination of computational components

• Processing / storage resources

• Of up to international level of geographic separation

• Crossing organisation boundaries

A (Web) service is a

• S/W system designed to support interoperable machine-to-machine interaction over a network.

Similar concepts

• For grids, the focus is on a computational model – Running a job Marshalling the data needed

• For web services – the focus is on service inter-operability A service can be any functionality you want to provide via the web

Need to bring these technologies together

• First we look closer at Web Services


Enabling Grids for E-sciencEWeb Services Approach

Web Services is the next step in the automation of inter-enterprise interaction

Web Browsing

• Human travel agent provides “organise holiday” service by surfing the web to look for and invoking services – book a hotel; book a plane; book a car hire; ….; confirm bookings of best options to meet client needs.

Web Services

• The aspiration of Web services is to provide a framework that allows that same model to be used in writing an application –

• which is itself becomes an “organise a holiday” service, finding and using useful services

Mode human intervention at –

service provider service consumer

E-mail

Web browsing

Web Services

Yes Yes

No Yes

No No


Enabling Grids for E-sciencEService Interaction

I book planes I book car Rentals

I book hotels

I locate services

I organise holidays Get a car rental quote

locate service ask for quote

Is quote good enough?YesReserve it

… get other resources reserved

Confirm booking

Service Identifier

I convert currency

I know the weather

quote


Enabling Grids for E-sciencEEssentials of WS Approach

Need to achieve effective cooperation even though

• the different services are produced by different organisations, without any design collaboration

• the services are autonomously evolving

• Loose coupling –

minimum prior shared information between the designer of the two components of an interaction

This requires

• Self-description – Meta data

• Tolerance of partial understanding

Collaboration is on defining generic standards,

rather than on specific design


Enabling Grids for E-sciencECore WS

Basic Requirements

• Service Description – WSDL • Web services Definition language, defines -

logical service –

Operations. Message structures for each Physical – protocol, location

• Allows service user to bind to the service Statically – use WSDL to generate code Dynamically – using -

• Service Registry and Discovery Services – UDDI• Universal Description, Discovery and integration• Allows one service to register with a registry

known by potential users• User can discover it and obtain its description

• Usability Negotiation – Policies

I book hotels

PolicyQuote- held - 10 minsProv Reserv. Held - 2 hours….

I locate services

register

I organise holidays



Give me a quote for …

Basic Requirements

• Communication Protocol - SOAP

• Request - Response

• Other patterns – e.g notification

Ask car rental service to notify me of any special deals

To notify me if my reservation hold time is about to expire

• Common Language – XML Documents

• Tag-value pair

• Defined by a Schema

• Used for messages, policies, service definition

• Specific extensibility mechanism

• Self Description – tagged values, WSDL

• Necessary for loose coupling

• Autonomous evolution

• Autonomous evolution – discovery service – UDDI

or out of band

Type = QuoteValidity = 1 hourPrice = £200….

Quote


Enabling Grids for E-sciencEEvolving Standards

• Collaboration is on defining generic standards, not specific design

• Two main standards bodies –

• W3C – actually produces “recommendations” – web community

• OASIS – industry – IBM, Microsoft, Sun, ….

• These standards are factored to allow partial adoption and combination

• The core standards

• WS-I – clarifications to aid interoperability

• Higher level standards built on them

XML*

SCHEMAS*WSDL*

DTD

Core WS

SOAP*

*WS-Interoperability

WSRF

WS-addressing

WS-notificationWS-security

WS-TransactionFramework

UDDI*

…

WS-MetaDataExchange



• XML – the standard format for all information

• SCHEMA –the standard language for defining the structure (syntax/type) of a unit of information

DTD is a deprecated predecessor of Schemas

• WSDL – the language for defining a service –

• Operations; Logical Message Structure; Bindings; locations

• SOAP – the standard message format

XML*

SCHEMAS*WSDL*

DTD

Core WS

SOAP*


WSRF

WS-addressing



UDDI*

…

WS-MetaDataExchange


Enabling Grids for E-sciencESome Further Standards

• WS-Security – Framework for authentication and confidentiality

• WS-Transaction Framework – for robustness of correlated interactions, e.g two phase – provisionally book everything, then confirm everything

• UDDI – standard repository interface (included in WS-I)

• WS-MetaDataExchange – how to communicate meta-data

• ….

XML*

SCHEMAS*WSDL*

DTD

Core WS

SOAP*


WSRF

WS-addressing



UDDI*

…

WS-MetaDataExchange


Enabling Grids for E-sciencEWSRF-Related Standards

• WS-Addressing - For communication of identities between services

• WS-Notification - Framework of notification interaction – subscribe, publish

• WSRF – Web Services Resource Framework

– E.g. a quote is a persistent entity which will need to be identified in subsequent interactions to finalise a provisional booking – a resource

– Consistent standard framework for creating, identifying, destroying resources

XML*

SCHEMAS*WSDL*

DTD

Core WS

SOAP*


WSRF

WS-addressing



UDDI*

…

WS-MetaDataExchange


Enabling Grids for E-sciencE Grid Services

Web Service Grids

• Grid middleware can be seen as a special application area of web services dealing mainly with –

VOs data replication job execution

• Many advantages to basing grids on web services Cooperation between autonomously evolving components Mix-and-match of components Combining Aspects of grids with other Web Service application

domains Rapid Development of higher-level services Leverage industrially-produced support packages

• Previously, independent simultaneous developments

• Much of existing middleware components are not based on web services


Enabling Grids for E-sciencEGrid Services

• Now trying to unify these two major developmental thrusts - OGSA – Open Grid Services Architecture

• Quite central to Grids is dynamically created resources, particularly

• Files

• Jobs

• VO membership

• Thus WSRF is important to this unification –

• WSRF originated from the grid world

• A (web) resource is a concept similar to object in O-O programming

• Multiple instances of the same type

• Each with its own (changing) state


Enabling Grids for E-sciencEWeb Service grids : WSRF

A resource can

• Be created

Globally unique id

www/grid:J6

• Change state

• Notify change of state

• Be destroyed

• Self-destruct

(scheduled)

Submit Job

A Grid Access Service

Actual Grid (www/grid) –Services & Resources

Job J6

Notify Started : J6

cancel: J6

Job Id : J6

www/grid



‘WS-I+’ profile minimal for gridsBPEL - workflowsWS-Addressing WS-ReliableMessaging

Web Service Grids: Interoperability

WS-I

Standards that havebroad industry support

and multiple interoperableimplementations

Specifications that are emergingfrom standardisation process

and are recognised as being ‘useful’

Specifications that have/will enter a standardisation processbut are not stable and are still experimental

WS-I – covers Schemas, WSDL, SOAP, UDDI (assumes XML)

‘WS-I+’ next-WSRFNotificationSecurityPortlets


Enabling Grids for E-sciencEOutline

• Grids and Web Services• EGEE• gLite


Enabling Grids for E-sciencEEGEE – towards e-infrastructure

EGEE is building a large-scale production grid service to:

• Underpin research, technology and public service

• Link with and build on national, regional and international initiatives

• Foster international cooperation both in the creation and the use of the e-infrastructure Network

infrastructure& Resource

centres

Op

era

tio

ns

, S

up

po

rt a

nd

tr

ain

ing

Collaboration

Pan-European Grid


Enabling Grids for E-sciencEBackground

• By 2003:• Grid technology shown to be viable

• Large amount of functional middleware

• …thanks to: FP5 : DataGrid, DataTAG, CrossGrid, etc… USA: VDT, Globus, Condor, etc. … and others

• Next step - major production infrastructure• EGEE was proposed to the EU in 2003

• 2 year project began in April 2004, with a 4-year vision.


Enabling Grids for E-sciencEGrids for e-Infrastructure…

• In 2003, what was missing?• Production-quality (stable, mature) Grid middleware• Production-quality operational support

Grid Operation Centres, Helpdesks, etc.• Multi-discipline grid-enabled application environment

Now led by HEP, Bio-info• Administrative and policy decision framework in order to share

resources at pan-European scale (and beyond) Areas such as AAA (Authentication, Authorisation, Accounting) End-to-end issues (Network related) Funding Policies (Grid economics) Resource Sharing Policies Usage Policies

• EGEE project is tackling most of the above issues



In the first 2 years EGEE will

• Establish production quality sustained Grid services • 3000 users from at least 5 disciplines• integrate 50 sites into a common infrastructure• offer 5 Petabytes (1015) storage

• Demonstrate a viable general process to bring other scientific communities on board

• Propose a second phase in mid 2005 to take over EGEE in early 2006

Pilot New


Enabling Grids for E-sciencEEGEE Organisation

• 70 leading institutions in 27 countries, federated in regional Grids

• ~32 M Euros EU funding for first 2 years starting April 2004 (matching funds from partners)

• Leveraging national and regional gridactivities

• Promoting scientific partnershipoutside EU


Enabling Grids for E-sciencEActivities Definition

• Network Activities• NA1: Project Management• NA2: Dissemination and Outreach• NA3: User Training and Induction• NA4: Application Identification and Support• NA5: Policy and International Cooperation

• Service Activities• SA1: Grid Support, Operation and Management• SA2: Network Resource Provision

• Joint Research Activities• JRA1: Middleware Reengineering + Integration• JRA2: Quality Assurance• JRA3: Security• JRA4: Network Services Development

Emphasis in EGEE is on operating a productiongrid and supporting the end-users


Enabling Grids for E-sciencEOperations - Introduction

• Strategy has been to • have a robust certification and testing activity,

• simplify as far as possible what is deployed, and to make that robust and useable.

• In parallel construct the essential infrastructure needed to operate and maintain a grid infrastructure in a sustainable way.

• Current service based on work done in LCG – culminating in the current service (“LCG-2”)• Now at the point where in parallel we need to deploy and understand gLite –

whilst maintaining a reliable production service.


Enabling Grids for E-sciencEEGEE Service Activities

• Create, operate, support and manage a production quality infrastructure

• Offered services:• Middleware deployment and

installation• Software and documentation

repository• Grid monitoring and problem tracking• Bug reporting and knowledge

database• VO services• Grid management services


Enabling Grids for E-sciencESA1 – Operations Structure

• Operations Management Centre (OMC):• At CERN – coordination etc

• Core Infrastructure Centres (CIC)• Manage daily grid operations – oversight,

troubleshooting• Run essential infrastructure services• Provide 2nd level support to ROCs• UK/I, Fr, It, CERN, + Russia (M12)• Taipei also run a CIC

• Regional Operations Centres (ROC)• Act as front-line support for user and operations

issues• Provide local knowledge and adaptations• One in each region – many distributed

• User Support Centre (GGUS)• In FZK – manage PTS – provide single point of

contact (service desk)• Not foreseen as such in TA, but need is clear


Enabling Grids for E-sciencEGrid Operations

• The grid is flat, but

• Hierarchy of responsibility• Essential to scale the operation

• CICs act as a single Operations Centre

• Operational oversight (grid operator) responsibility

• rotates weekly between CICs

• Report problems to ROC/RC

• ROC is responsible for ensuring problem is resolved

• ROC oversees regional RCs

• ROCs responsible for organising the operations in a region

• Coordinate deployment of middleware, etc

• CERN coordinates sites not associated with a ROC

CIC

CICCIC

CICCIC

CICCIC

CICCIC

CICCIC

RCRC

RCRC RCRC

RCRC

RCRC

ROCROC

RCRC

RCRC

RCRCRCRC

RCRCRCRC

ROCROC

RCRC

RCRC RCRC

RCRC

RCRC

ROCROC

RCRC

RCRC

RCRC

RCRC

ROCROC

OMCOMC

RC = Resource Centre


Enabling Grids for E-sciencE(Human) Networking Activities

• Dissemination and Outreach: 5% of EGEE budget• Dissemination – to actively promote and raise awareness of the EGEE project• Outreach – to identify and contact potential new user communities

• Training and Induction: 4% of EGEE budget• Induction – to introduce and orient - users and members• Training – to create, collate, make available and deliver material and courses

• Application Identification and Support• Process for selecting new application areas• Supports selected VO’s in porting applications

• International cooperation


Enabling Grids for E-sciencE Dissemination

• 1st project conference, Cork, April • 2nd conference in The Hague

• 22-26 November• http://public.eu-egee.org/conferences/2nd

• Over 300 delegates

• Websites, Brochures and press releases• For project and general public www.eu-

egee.org• Information packs for the general public,

press and industry


Enabling Grids for E-sciencETraining and Induction

NeSCEdinburgh

UK & Ireland

GUPLinz

Austria

InnsbruckAustria

CESNETPrague

Czech Rep.

BUTEBudapestHungary

ELUBBudapestHungary

MTA SZTAKIBudapestHungary

ICMWarsawPoland

PSNCPoznanPoland

II-SASBratislavaSlovakia

FZKKarlsruheGermany

INFNRomeItaly

KU-NATFAK Copenhagen

Denmark

IHEPProtvinoRussia

IMPB RASMoscowRussia

ITEPMoscowRussia

JINRDubnaRussia

PNPIPetersburgh

Russia

RRCKIMoscowRussia

GRNETAthensGreece

TAUTel Aviv

Isreal

ICIBucharestRomania



Current production m’ware: LCG-2

ReplicaReplicaCatalogueCatalogue

Logging &Logging &Book-keepingBook-keeping

ResourceResourceBrokerBroker

StorageStorageElementElement

ComputingComputingElementElement

Information Information ServiceService

Job Status

DataSets info

Author.&Authen.

Job S

ub

mit

Even

t

Job

Qu

ery

Job

Stat

us

Input “sandbox”

Input “sandbox” + Broker Info

Output “sandbox”

Output “sandbox”

Pu

blis

h

SE & CE info

““User User interface”interface”



• Co-existence with deployed infrastructure

• Co-existence with LCG-2 and OSG (US) are essential for the EGEE Grid services

• Site autonomy• Reduce dependence on ‘global, central’

services

• Open source license

gLite: Guiding Principles

• Service oriented approach• Allow for multiple interoperable

implementations

• Lightweight (existing) services • Easily and quickly deployable• Use existing services where possible

Condor, EDG, Globus, LCG, …

• Portable• Being built on Scientific Linux and Windows

• Security• Sites and Applications

• Performance/Scalability & Resilience/Fault Tolerance

• Comparable to deployed infrastructure

EDGVDT . . .

LCG . . .AliEn


Enabling Grids for E-sciencEgLite Services for Release 1

Grid AccessService

API

Access Services

JobProvenance

Job Management Services

ComputingElement

WorkloadManagement

PackageManager

MetadataCatalog

Data Services

StorageElement

DataManagement

File & ReplicaCatalog

Authorization

Security Services

AuthenticationAuditing

Information &Monitoring

Information & Monitoring Services

Application

Monitoring

Site Proxy

Accounting

JRA3 UK

CERN IT/CZ

Focus on key services


Enabling Grids for E-sciencEApproach

• Role: gLite is intended as the middleware for Pan-European Grid

• Approach:

Combining and re-engineering existing components

(as far as possible)

• Middleware (MW) components now work reasonably well –

problems usually elsewhere

• Substantial body of experience of MW development, packing and use

• Based on LCG2

• What is currently used within the EGEE project

• Can be viewed as an initial version of

Pan-European Grid middleware

• gLIte deals with• LCG2 shortcomings

• Advanced application needs

• State-of-the-art Internet approaches and standards

EDGVDT . . .

LCG . . .AliEn


Enabling Grids for E-sciencEImprovements

• Extended Functionality

• Incorporates features from other products

• Some new functionality

• Increased Modularity

• Previous products tend to be monolithic – assumed to be used in entirety

• Now need to move towards smaller scale of component combinability

• Improved Deployability

• Modularity

• Installation support

• Non-dedicated machines

• Coherent architecture

• extending functionalities and increasing modularity requires –

• a revised picture of the organisation of functions

• rationalisation of interface

• Adopt Web Services Approach and Standards


Enabling Grids for E-sciencEGuiding Principles

• Lightweight - Easily and quickly deployable

• Mix-and-match components

• Multiple services running on the same physical machine (if possible)

• Interoperable

• Client may talk to different independent implementations of the same service

• Resilient and Fault Tolerant

• Co-exist with deployed infrastructure

• Reduce requirements on site components

• Co-existence with LCG-2 essential for the EGEE Grid service

• Platform support- Goal is to have portable middleware

• Building & Integration on RHEL 3 and windows

• Initial testing (at least 3 sites) using different Linux flavours (including free distributions)


Enabling Grids for E-sciencEGuiding Principles

• Service oriented approach

• Based on web services standards

• Service autonomy

• User may talk to services directly or through other services (like access service)

• Open Source software license

• No restriction on usage (academic or commercial) beyond acknowledgement

• That's for MW - for application software, Sites must obtain appropriate licenses before installation

• Main Documentation

Application requirements: http://egee-na4.ct.infn.it/requirements/

Architecture: https://edms.cern.ch/document/476451

Design: https://edms.cern.ch/document/476451

Release plan: https://edms.cern.ch/document/468699


Enabling Grids for E-sciencEApplication communities and EGEE

• LCG and Bio-informatics from day 1

• New application communities are selected by the EGEE Generic Applications Advisory Panel:• For new applications• See: EGEE web site (NA4 activity) and also

http://agenda.cern.ch/age?a042351

• Selected are:• Computational chemistry• Earth sciences• Earth observation• Astrophysics

• Also working with DILIGENT:• Virtual digital data libraries



• Prototyping short development cycles for fast user feedback –

Requirements

Planning & Design

Implementation

Testing

Development Cycle

Globus 2 based Web services based

gLite-2gLite-1LCG-2LCG-1

Deployment : Prototype Infrastructure → Pre-production service → Production Service



Open Source Software License

• The existing EGEE grid middleware (LCG-2) is distributed under an Open Source License developed by EU DataGrid project• Derived from modified BSD - no restriction on usage

(academic or commercial) beyond acknowledgement

• Approved by Open Source Initiative (OSI)

• Same approach for new middleware (gLite)• New license agreed by partners is derived from the

EDG license and takes into account feedback from the World Intellectual Property Office (WIPO)



EGEE www.eu-egee.orgLCG lcg.web.cern.ch/LCG/“Concertation event” and Second EGEE conference:http://public.eu-egee.org/conferences/2nd/programme/outline.html

The Grid Cafe www.gridcafe.org•More EU sites:

•http://www.cordis.lu/ist/grids/fp6_grid_projects.htm•http://www.gridstart.org/concertation_mtg.shtml•“e-Infrastructures Reflection Group http://www.e-irg.org

•NeSC www.nesc.ac.uk

Further Information



• THE END