Grid Standards, Interoperability and Applications UNICORE ...ptvu/gc/UNICOREStandards.pdfCrossCross--Grid Use Case Scenario Grid Use Case Scenario WISDOM (Wide In Silicio Docking on

Grid Standards, Interoperability and Applications Grid Standards, Interoperability and Applications

UNICORE Tutorial UNICORE Tutorial 25.25.--26.07.200726.07.2007

Morris Morris RiedelRiedel

Forschungszentrum Jülichin der Helmholtz-Gesellschaft

Morris Morris [email protected]@fz--juelich.dejuelich.de

OutlineOutline

�Grid Standards

�Motivation

�Grid Islands, Cross-Grid Use Case Scenario, Interoperability Highway

�Overview of UNICORE 6 related and adopted Standards

�OASIS, W3C, OGF, IETF, DMTF

� Interoperability

2 Forschungszentrum Jülich

� Interoperability

�Grid Interoperation Now – Community Group (GIN-CG)

�Open Middleware Infrastructure Institute (OMII) for Europe

�Lessons Learned from Interoperability

�Applications

�Applications of end-users

�Applications for administrators and developers

�Summary & References

Grid StandardsGrid Standards


MotivationMotivation

�Grid Islands: Sending a job from one Grid to another is difficult

�Example: D-Grid Infrastructure, ‘Job Submission Technologies’

�Globus Toolkit 2, (3), 4

– Specification of Jobs via Resource Description Language (RSL)

– Submission to Globus Resource Allocation Manager (GRAM)

�gLite (aka lcg) 3.x

Foster [1]


– Specification of Jobs via Job Definition Language (JDL)

– Submission to Workload Management Service (WMS)

�UNICORE 5

– Specification of Jobs via Abstract Job Objects (AJOs)

– Submission to Network Job Supervisor (NJS)

�Example: DEISA and EGEE European e-Infrastructures

�DEISA, massively parallel jobs: Access via UNICORE

�EGEE, embarrassingly parallel jobs: Access via gLite

Laure et al. [3]

Streit et al. [2]

DEISA [4]

EGEE [5]

CrossCross--Grid Use Case ScenarioGrid Use Case Scenario

�WISDOM (Wide In Silicio Docking on Malaria)

�WISDOM aims at developing new drugs for Malaria

�WISDOM uses EGEE for large scale in silicio docking

�A computational method for prediction of whether one molecule will

bind to another using (using AutoDock and FlexX software)

�AutoDock and FlexX as software provided via gLite in EGEE

WISDOM [6]


�Output is a list of chemical compounds (potential drugs)

�Refine best compound list via molecular dynamics (MD)

�Fast MD computations use highly scalable AMBER in DEISA

�AMBER (Assisted Model Building with Energy Refinement)

�Goal: Accelerate drug discovery using EGEE and DEISA together

�Raises a demand for Interoperability of Grid Middleware Technologies

Vision: Interoperability HighwayVision: Interoperability Highway

EmergingOpen

Standards

End-usersvia clients

& portals

„Interoperability highway“based on open standards

GOAL: Transparencyof Grids for end-users

others


�Open Standards are emerging from various standardization bodies…

GridMiddlewares others

GridResources

Overview of Important StandardsOverview of Important Standards

� Internet Engineering TaskForce (IETF)

�Transport Level Security (TLS)

�X.509 Public Key Infrastructure (PKI)

�Org. for the Advancement of Structured Inf. Standards (OASIS)

�Web Services Resource Framework (WS-RF)

�Used as WS-* infrastructure resource model in UNICORE 6 and GT4


�Used as WS-* infrastructure resource model in UNICORE 6 and GT4

�Security Assertion Markup Language (SAML)

�Extensible Access Control Markup Language (XACML)

�Distributed Management TaskForce (DMTF)

�Common Information Model (CIM)

�Used in conjunction with UNICORE 6 Common Information Service (CIS)

�World Wide Web Consortium (W3C)

�Web Services Addressing (WS-A)

Overview of Important StandardsOverview of Important Standards

�Open Grid Forum (OGF)

�Job Submission and Description Language (JSDL)

�Standardized Job definition for UNICORE 6, gLite (soon) and GT4 (soon)

�OGSA – Basic Execution Services (OGSA-BES)

�Standardized job submission interface to gLite, GT4 & UNICORE 6 (soon)

�High Performance Computing – Profile (HPC-P)


�High Performance Computing – Profile (HPC-P)

�Profile for joint usage of HPC-based JSDL extensions and OGSA-BES

�OGSA – Resource Usage Service (OGSA-RUS)

�Emerging standard exposing OGF usage record formats (URFs)

�OGSA – Byte Input/Output (OGSA-ByteIO)

�Streamable and Random byte transfers for file and data transfers

�Grid Laboratory Uniform Environment (GLUE)

�Used as information model in gLite & GT and UNICORE 6 CIS (soon)

Adoption Adoption ofof JSDL, JSDL, ByteIOByteIO, WS, WS--AA

� JSDL-based

job submission

via UNICORE

Atomic Services


�File Transfer

Service is using

SbyteIO and/or

RByteIO

�Available in

UNICORE 6.0

(August 2007)UNICORE [7]

Adoption Adoption ofof OGSAOGSA--BES & HPCBES & HPC--PP

�HPC-P Compliant UNICORE 6

�JSDL with extensions for HPC

�OGSA-BES interface (v.33, soon 1.0)

�Alternative standardized job submission

interface to UNICORE 6

�OGSA-BES covers functionality from


�OGSA-BES covers functionality from

�Target System Factory (TSF)

�Target System Service (TSS)

�Job Management Service (JMS)

�UNICORE Atomic Services will remain

�GPE plans to adopt OGSA-BES very soon

�OGSA-BES support available in UNICORE 6.1

(end of this year) OMII - Europe [8]

Adoption Adoption ofof SAML & XACMLSAML & XACML

�Next generation VOMS server

releases signed SAML assertions

�Role possession

�Group/Project Membership

�SAML assertions are checked using

Extensible Access Control Markup

Venturi et al. [9]


Extensible Access Control Markup

Language (XACML)-based policies

�SAML is used as delegation mechanism

in UNICORE 6

�Proxies are not needed for this

delegation mechanism

�SAML – VOMS support available in

UNICORE 6.1 (end of this year)

Adoption Adoption ofof CIM & GLUE CIM & GLUE

�WS-based UNICORE Common Information Service

�Based on CIM, plans to adopt upcoming GLUE 2.x specification

�Extensions to CIM:

�UNICORE resource model

�JSDL & OGSA - BES according to OGSA Resource Management design

�XPATH/XQuery as query language, eXist DB (native XML DB)


�XPATH/XQuery as query language, eXist DB (native XML DB)

�Access right control using XACML

�Who is allowed to see

what infos?

�Available at the end

of this year

InteroperabilityInteroperability


Vision: Interoperability HighwayVision: Interoperability Highway

EmergingOpen

Standards

End-usersvia clients

& portals

„Interoperability highway“based on open standards

GOAL: Transparency

of Grids for end-users

others


�Non-trivial : emerging standards are not directly interoperable…

� Different adoption plans and compliance status, different security policies, etc.

GridMiddlewares others

GridResources

Interoperation in Interoperation in ContextContext

� Interoperation

�What needs to be done to get different Grids talking together

�Short-term achievements, using what is available today

�One specific technology works with one other specific technology

�Realized via hacks, adapters, workarounds, etc.

�E.g. work conducted in OGF Grid Interoperation Now (GIN) group


�E.g. work conducted in OGF Grid Interoperation Now (GIN) group

�Example: EGEE-II: gLite and UNICORE 5 interoperation development

�Interoperation scenario for scientific job submissions

�gLite � UNICORE: using CONDOR-U + Trusted Agent

�Sustainability of these interoperations via future interoperability

�Interoperability: native ability to communicate via open standards

Standard Compliance in Standard Compliance in ContextContext

�Standard Compliance and interoperability are not the same…

�Both are different targets, but towards same direction

�Standard Compliance is a prerequisite for interoperability

�Standard Compliance does not implies interoperability

� Interoperability is much more than standard compliance with one specification

�Grid Interoperability: use components together “scenario”


�Grid Interoperability: use components together “scenario”

�e.g. job submission via information service based on information model

�Some Examples for Basic (job) Execution Services (BES)

�In real deployments is not the “vanilla OGSA-BES interface available”

�Same exact “client” works not directly with gLite & UNICORE

�UNICORE Gateway has another WS-Addressing EPR part

�Different security models: X.509 Proxies vs. full certificates

�Different infrastructures (WS-I & WS-RF)… etc.

Driving Grid InteroperabilityDriving Grid Interoperability

OMII-EuropeComponents

Developments:

VOMS,

OMII-Europe Repository

EvaluationInfrastructure

USERS


VOMS,OGSA-BES,OGSA-DAI,OGSA-RUS,GridSphere

Quality Assurance & Compliance Testing of Grid middleware

e.g. Compliance with one specific specification: e.g. OGSA-BES

Interoperability Tests: Integrate and use OMII-Europe components with each other:

e.g. VOMS & OGSA-BES & GridSphere

building the “interoperability highway…”

OMII - Europe [8]

MultiMulti--PlatformPlatform InfrastructureInfrastructure

� JRA3-T2: Integration of all components of OMII – Europe

�OGSA-BES, OGSA-RUS, OGSA-DAI, VOMS, GridSphere

�Middleware: UNICORE, gLite, Globus Toolkits, and CROWN

�Goal: Test and establish interoperability of components


OMII - Europe [8]

LessonsLessons LearnedLearned fromfrom InteropInterop. (1). (1)

�Missing components/functionality in Grid platforms

�UNICORE not relies on Info-Service, but is a requiment for gLite

�Different standard specification versions are implemented in the

Grid middleware platforms

�e.g. OGSA-BES v.26 (Supercomputing 2006) and v.33 public comment

�Different technologies for WS-* infrastructures adopted


�Different technologies for WS-* infrastructures adopted

�WS-RF, WS-I, WS-ResourceTransfer,… � what’s next?

�UNICORE & Globus Toolkit are WS-RF compliant, gLite no plans

�E.g. GetResourceProperty operation of WS-I compliant fails

�Even if both maybe OGSA-BES compliant (aka WS-RF rendering)

�Different versions of the WS-* infrastructures

�UNICORE is WS-RF 1.2 compliant, Globus uses WS-RF 1.0 (1.2 soon)

LessonsLessons LearnedLearned fromfrom InteropInterop. (2). (2)

�Challenges in security are major show stoppers

�Standards specification compliance is only a precondition

�Absence of a widely accepted common security profile

�OMII-Europe (JRA3T1: Common Security Profile) works on that

�Several specifications of OGF (Secure Channel etc.) used

�Only several specifications of OASIS (SAML etc.) are used


�Only several specifications of OASIS (SAML etc.) are used

�Only several specifications of IETF (X.509 etc.) are used

�Different strategies and adoption status in Grid Middleware

�E.g. gLite, Globus: Attribute Certificate VOMS, UNICORE SAML VOMS

�E.g. Delegation mechanisms are extremely different

�Proxies: Globus, gLite – Only optional support by UNICORE

�Explicit Trust Delegation and SAML Assertion Delegation: UNICORE

�Not by gLite & Globus

�One Interoperability scenario within OMII – Europe

�Attribute Authority (AA) VOMS (SAML-based) get central role

�VOMS becomes middleware independent VO service

OMIIOMII--Europe „Success Story“ (1)Europe „Success Story“ (1)


OMII - Europe [8]

ApplicationsApplications


Applications (endApplications (end--users)users)

�Collaborative Online

Visualization and

Steering (COVS)

Service

�One higher-level

service within

UNICORE 6


UNICORE 6

�Goal: User does not

have to deal with

hostnames, usernames

or passwords

(single sign-on)

Riedel et al. [10]


�COVS uses Intel’s Grid Programming Environment (GPE) GridBeans

�COVS GridBean Plugin accessing COVS services in UNICORE 6

�COVS Session Management:

�Create/Join COVS Sessions

�Add/Remove


�Add/Remove Participants

�Works togetherwith scientificvisualizations(e.g. Xnbody)

Link to talk:„Überblick und Programmierung von GridBeans“


�COVS Use Case Scenarios (Grid driving HPC: massively parallel jobs)

�Xnbody: Light-weight 3D visualization application based on VTK

�Used in astro-physics (nbody6++), plasma-physics (PEPC), …Spurzem et al. [16] Gibbon [17]



�European Chemomentum Project

�Workflows and job management based on UNICORE 6

�Application: Quantitative Structure-Activity Relationship (QSAR)

2D-Structures


2D → 3DOptimization

DescriptorsProperties

experimentvalues

y=f(d1, d2, ... , dN )

Computed Values

Chemomentum [12]


�Resource Reservation (Network+CPUs) for UNICORE 5 (move to 6)

UNICORE Client

UNICORE Gateway

UNICORE Gateway

Site A Site B

MetaScheduling Request

(WS-Agreement)

Job submit to UNICORE


Local Scheduler

TSI

Primary NJS

TSI

NJS

TSI

NJS

Local Scheduler

Local Scheduler

Adapter

Job Queue

Adapter Adapter

Job QueueJob Queue

MetaScheduler

Network RMS ARGON

Link Usage

MetaScheduler Response(WS-Agreement)

Adapter

Negotiations and Reservations

Cluster Cluster Cluster

All components will be startedat the same pre-defined time, including

cpu time and network reservations

Eickermann et al. [13]

Applications (administrators)Applications (administrators)

�OGSA-RUS services is onehigher-level service within UNICORE 6

�RUS exposes URFs compliant XML

�URFs are providedvia RMS-specific generators


via RMS-specific generators

�Use case: LLview: User-level Resource Monitoring

Frings et al. [11]

Applications (administrators)Applications (administrators)

29 Forschungszentrum JülichFrings et al. [11]

Applications (developers)Applications (developers)

�A-WARE Service Bus (ASB)

�Goal: Normalised Runtime Environment

�Mediating over the UNICORE 6 Fabric layer (UNICORE Atomic Services)

�Hosting workflow services, bridge between the Client and the Grid

�Client currently based on EnginFrame Portals from NICE

A-WARE [14]


�GridSphere Portals usable for UNICORE 6 (soon)

�GridSphere Application Portlets on top of UNICORE 6

�Last but not least:

�You can easily develop a higher level service for UNICORE!

�Create new application services on top of the core of UNICORE 6

OMII - Europe [8]

GridSphere [15]

Summary and ReferencesSummary and References


SummarySummary

�UNICORE 6 integrates the „most common“ Grid related standards

�WS-RF, JSDL, OGSA-ByteIO, OGSA-BES, OGSA-RUS, SAML, XACML,…

�UNICORE 6 is used by numerous different application domains

�Chemical workflows in Chemomentum, COVS, Meta-Scheduling, …

� Interoperation (work performed within the GIN-CG group)

�What needs to be done to get production Grids talking together


�What needs to be done to get production Grids talking together

�Hacks, workarounds, short-term achievements

�Commonly found in production Grid interoperations

� Interoperability (work performed in OMII – Europe)

�Many components work together to achieve a goal

�Native standard support from middlewares without hacks in future

�Continue work in the open standards working groups

� „Interoperability highway…“ to realize the „true global Grid vision“

References (1)References (1)

� [1] Ian Foster: Globus Toolkit version 4: Software for Service-Oriented Science. In Proceedings of

IFIP International Conference on Network and Parallel Computing, LNCS 3779, pages 213–223.

Springer- Verlag, 2005.

� [2] Streit et al.: UNICORE - From Project Results to Production Grids., L. Grandinetti (Edt.), Grid

Computing: The New Frontiers of High Performance Processing, Advances in Parallel Computing

14, Elsevier, 2005, pages 357-376

� [3] Laure et al.: Programming the Grid with gLite, Computational Methods In Science and

Technology, Volume 12 (1), 2006, pages 33-46

� [4] Distributed European Infrastructure for Supercomputing Applications (DEISA):


� [4] Distributed European Infrastructure for Supercomputing Applications (DEISA):

http://www.deisa.org

� [5] Enabling Grids for EscienE (EGEE): www.eu-egee.org

� [6] WISDOM Project: http://wisdom.eu-egee.fr/

� [7] UNICORE Website: http://www.unicore.eu

� [8] Open Middleware Infrastructure Institute (OMII) for Europe: http://www.omii-europe.org

� [9] Venturi et al.: Using SAML-based VOMS for Authorization within Web Services-based UNICORE

Grids, In Proc. of UNICORE Summit 2007, to appear

� [10] Riedel et al.: Design and Evaluation of a Collaborative Online Visualization and Steering

Framework Implementation for Computational Grids, In Proc. of the 8th IEEE/ACM International

Conference on Grid Computing (Grid 2007), Austin, Texas, to appear

References (2)References (2)

� [11] Frings et al.: LLview: User-Level Monitoring in Computational Grids and e-Science

Infrastructures, In Proc. of German e-Science Conference 2007, Baden-Baden, Germany,

Online-Publication

� [12] Chemomentum project: http://www.chemomentum.org

� [13] Th. Eickermann et al.: Co-allocation of MPI Jobs with the VIOLA Grid MetaScheduling

Framework, In Proc. of German e-Science Conference 2007, Baden-Baden, Germany,

Online-Publication

� [14] A-WARE Project: http://www.a-ware-project.eu/

� [15] GridSphere: http://www.gridsphere.org/


� [15] GridSphere: http://www.gridsphere.org/

� [16] Spurzem et al.: Direct N-body Simulations, The Journal of Computational and Applied

Mathematics Vol. 109, pp. 407-432 (1999)

� [17] Gibbon et al.:. Short Pulse Laser Interactions with Matter: An Introduction, Imperial College

Press/World Scientific, London/Singapore, 2005. ISBN 1-86094-135-4.

IGIIW @ eIGIIW @ e--Science 2007Science 2007

International Grid

Interoperability & Interoperation

Workshopin conjunction with

e-Science 2007, Bangalore, India


e-Science 2007, Bangalore, India

December 10-13, 2007

Call for Paper deadline: 29th of July (next Sunday)

http://omii-europe.org/OMII-Europe/igiiw2007.html

Documents

Grid Standards, Interoperability and Applications UNICORE ...ptvu/gc/UNICOREStandards.pdfCrossCross--Grid Use Case Scenario Grid Use Case Scenario WISDOM (Wide In Silicio Docking on