DEGREE IST 2006- 034619 Dissemination and Exploitation of GRids in Earth sciencE Earth Science Communit y RequirementsRequirements T echnologies FeedbackFeedback

1DEGREE IST 2006- 034619

Dissemination and Exploitation of GRids in Earth sciencE

Earth Earth Science Science

CommunitCommunityy

RequirementsRequirementsRequirementsRequirements

TTechnologiesechnologiesTTechnologiesechnologies

FeedbackFeedbackFeedbackFeedback

ApplicationsApplicationsApplicationsApplications

DisseminationDisseminationDisseminationDissemination

ES ES VOsVOs

EGEE & EGEE & various various GRIDSGRIDS

Dissemination and exploitation of Grids in Earth Science

Ladislav Hluchy,Monique Petitdidier

II SAS, CNRS

DEGREE IST 2005-034619 ─ 3rd EGEE User Forum, Clermont-Ferrand, February 2008 2

Introduction ES community

Atm

osph

ere

Oce

an

Biosphere

Cryosphere

Coupled interdisciplinary processes

Sub-surface complex modelling Complex

web of sensor

Optimal Trajectory

First guess

Complex data analysis

Noisy observations

• Global, regional, local applications– Alternative use of the data at different

time and spatial resolution

• Large historical distributed archives– Long term data archives to be exploited

• Near real-time access to data– For processing, value adding and

dissemination– For now-casting and alert

• Models to provide long term trends and forecast

– Processing-intensive, data-intensive and complex applications

– Data fusion, data assimilation, data mining, modelling …

• Integration of different data sources


Application examples

Production Production (3 algorithms) and (3 algorithms) and validation of 7 years validation of 7 years of of GOMEGOME Ozone Ozone profilesprofiles

Rapid Earthquake Rapid Earthquake analysis (mechanism analysis (mechanism and epicenter) and epicenter) 50- 100CPUs 50- 100CPUs

Modeling seawater Modeling seawater intrusion in costal intrusion in costal aquifer (SWIMED)aquifer (SWIMED)

Geosciences: Geosciences: Geocluster for Geocluster for Academy and industryAcademy and industry

Flood of the Danube Flood of the Danube river-Cascade of modelsriver-Cascade of modelsSpecfem3D: Specfem3D: seismic wave propagation

model Benchmark for MPI (2 to 2000CPUs)Benchmark for MPI (2 to 2000CPUs)


DEGREE project

• DEGREE aims to create a bridge between Earth Science and Grid communities

• In order to achieve this goal, the consortium members will:– Identify key Earth Science

requirements– Disseminate Earth Science

application requirements to Grid projects

– Evaluate Grid middleware, tools and standards regarding Earth Science requirements

– Provide feedbacks to Grid developers


• Report on ES family of applications and their Grid requirements (D1.1)• Document contains:

– Definition of family of applications– Chapters on monitoring requirements – 5 requirements categories:

Security, Data and metadata management, Workflow, Application development, Miscellaneous– Contains both listed and described requirements (providing context)

• D1.1 is a living document: – contains previously stated requirements (DataGrid, EGEE)– contains analysis of 15 application scenarios– After delivery to EU, 5 more scenarios added– (re) analyze requirements– Will be used to monitor progress– Base for deliverable D1.2 Final progress report on

ES families of applications and their Grid requirements– 78 requirements defined

• Document and application descriptions available on DEGREE site

ES requirements


ES key requirements

• Security– Fine grained on group and role level– enforce data policies

• Data and metadata management– Database access, both to databases inside and outside Grid– GIS integration

• Workflow– Complex workflow integration

• Portals– Web service interfaces to Grid middleware functionality

• Miscellaneous– Near Real Time: Not only ‘time to completion/delivery’, but also specific start time

need– Use of licensed software

Requirements might not be ‘unique’ to ES domain, but they are essential for ES

Req. No Need Traceability Families Description

ES-S-1 4.3 [R6], [R9]

1,2,3 Clear and straightforward procedure for obtaining a certificate and joining the Grid

ES-S-2 4.3 [R6], [R9]

1,2,3 Unified, common security protocol, standard for all services and resources

ES-S-3 4.3 [R6] , [R9]

1,2,3 Standard security protocols seamlessly integrated into all components as fundamental design feature

ES-S-4 4.3 [R6] , [R9]

1,2,3 From user perspective, security mechanisms should be non intrusive, transparent and should present similar functionality, independent of the type of Grid resources being accessed

ES-S-5 4 [R6] , [R9]

1,2,3 User may belong to any number and combination of VO(s)/group(s)/role(s)

ES-S-6 3.7 [R6] , [R9]

1,2,3 User may possess a single certificate valid for all associated VO(s)/group(s)/role(s)

ES-S-7 2.3 [R6] , [R9]

1,2,3 Private key stored only on users computer; proxy generation directly from the users computer

ES-S-8 4.3 [R6] , [R9]

1,2,3 User able to specify the required VO/group/role attributes to be associated to the proxy certificate during single sign on

Req. No Need Traceability Families VO management

ES-VO-1 4.3 [R6] , [R9]

1,2,3 Definition of groups within a VO

ES-VO-2 3 [R6] , [R9]

1,2,3 Definition of roles within a VO

ES-VO-3 4.3 [R6] , [R9]

1,2,3 Assign to a user one or more groups and/or one or more roles

ES-VO-4 2.7 [R6] 1,2,3 Web based VO database administrator interface

ES-VO-5 2.7 [R6] 1,2,3 Administrator interface login via the user’s certificate loaded in browser


ES-DM-1 4.7 [R6] 1,2,3 Fast performance of replica catalogue operations, resistance to degradation under heavy loads, whether due to increased data or user numbers

ES-DM-2 3 [R6] , [R9]

1,2,3 QoS response times published by each replica manager service in the Grid information system, retrievable by the end user/application

ES-DM-3 4.3 [R6] 1,2,3 All functions, e.g. creating replicas, file transfers, inserting, updating and retrieving replica catalogue information and metadata, must include proper security checks on the proxy-delegated permissions

ES-DM-4 2.7 [R6] , [R9]

1,2,3 Mechanisms for establishing and operating QoS levels to guarantee validity and integrity of replicated data

ES-DM-5 3.7 [R6] 1,2,3 Data replication (i.e. data transfer and catalogue metadata registration) must be performed as an atomic operation

ES-DM-6 5 [R6] , [R9]

1,2,3 Correct management of disk space quota checking integrated in the data transfer operations

ES-DM-7 4 [R6] 1,2,3 Advance, pre-emptive warnings when available space falls below threshold levels

ES-DM-8 4 [R6] 1,2,3 Automatic data integrity checks embedded in replica manager operations

ES-DM-9 3.3 [R6] , [R9]

2,3 Capability to handle (storing, searching, updating, retrieving) file numbers of the order of millions of files

ES-DM-10 2.3 [R6] , [R9]

3 An application must be able to create, populate, utilize and destroy replica catalogues on the fly, as required by the application, without the need for human administrator intervention, having performed the appropriate security and resource utilization quota checks

ES-DM-11 3.3 [R6] , [R9]

1,2,3 Register both the logical filename and the associated set of metadata keys (list of key=value tuples) in a single operation

ES-DM-12 4.3 [R6] , [R9]

2,3 Retrieve full set of metadata tuples associated to a LFN in a single operation


ES-WL-1 2.7 [R6] , [R9]

1,2,3 Method for jobs running on the worker nodes to retrieve attributes and values that were specified in the JDL

ES-WL-2 4.3 [R6] , [R9]

1,2,3 Improved error messages

ES-WL-3 3.7 [R6] 1,2,3 Provision and enforcement of job submission quotas to prevent overloading the RB

ES-WL-4 3.7 [R6] , [R9]

1,2,3 Monitoring of RB state and job submission timings with results dynamically published in a Grid information service, to enable user/application to choose RB service based on current performance and loading

ES-WL-5 3.3 [R6] 1,2,3 Mechanisms to protect from response time degradation under conditions of heavy loading

ES-WL-6 4 [R9] 2,3 Reliable real-time and instantaneous job submission for high priority jobs for e.g. risk management


ES-SI-1 4.3 [R6] , [R9]

1,2,3 The Grid middleware must provide applications with command-line commands, library embedded APIs and service based access

ES-SI-2 3.7 [R6] 1,2,3 Uniform configuration across the Grid (i.e. available software and installed versions) for each type of resource or service

ES-SI-3 4 [R6] 1,2,3 Standard mechanisms and approach to fault tolerance, error handling and recovery supplied to all middleware components.

ES-SI-4 4.3 [R6] , [R9]

1,2,3 Common standard (guidelines) for command line formats and APIs

ES-SI-5 4.3 [R6] , [R9]

1,2,3 Standardization of error codes, error messages and error handling procedures

ES-SI-6 4 [R9] 1,2,3 Integration on service level definition with GIS-services (OGC) en web services (W3C, OASIS)

Req. No Need Traceability Family Description

ES-AP-1 [R8], [R9]

1,2,3 Grid services shall be in compliance with international standards for interfacing with other (web) services.

ES-AP-2 [R8], [R9]

3 Grid components shall have an (web) service interface, able to handle requests from other (web) services.


ES-O-1 5 [R6, R7, R9]

1,2,3 The Grid M/W services shall provide a predefined Quality of Service, specified by ES users.

ES-O-2 5 [R6, R7, R9]

1,2,3 It shall be possible to monitor this predefined QoS using Grid tools.

ES-O-3 4 [R6, R7, R9]

1,2,3 It shall be possible to establish an SLA between users and providers and use Grid tools to monitor the SLA parameters.


ES-IS-1 4 [R6] 1,2,3 Improved reliability and fault tolerance, eliminate vulnerabilities due to single points of failure

ES-IS-2 3 [R6] , [R9]

3 Schema to include Grid descriptor info to allow applications to determine the type of Grid environment e.g. production Grid, development Grid

ES-IS-3 2.7 [R6] 1,2,3 Mechanisms to enforce security restrictions on published information, e.g. info about VO resources only available to members of the VO

ES-IS-4 2 [R6] , [R9]

2,3 Guaranteed response times within published QoS figures

ES-IS-5 3.7 [R6] , [R9]

1,2,3 Schema to include info on all available services, e.g. resource brokers, replica catalogues, replica optimizers, metadata catalogues, etc.

ES-IS-6 4 [R6] , [R9]

1,2,3 To allow clients to determine whether a service is usable or not, the schema should support inclusion of current serviceability status for all published services


Test suite introduction

• Goals:• Provide Grid developers clear case for our requirements• Monitor on requirements fulfillment of Grid m/w (cover requirements)

Consists of:

• Documentation:– Test suite description– Test procedures– Pass/fail criteria

• ES contact

• Real application*– Application software– Data needed– Data schemas– Workflow schemas

* Application will be provided ‘as it is’ and as policies permit it. No programming done by DEGREE


• Test suite Organization document (high level document)– Test suite organization– Test suite descriptions– Test coverage– Templates:

• Test suite template• Test reporting template

• 8 Test suites, all containing:– Application description– Contact information (for application, data, etc.)– Test environment setup– Test cases– Report template

• All on the DEGREE website:http://www.eu-degree.eu/

Document content


Application Family Special Focus

GOME level 2 product processing (Ozone profile)

Simple Job control, data management

Centroid Seismic Moment Tensor (Seismology)

Simple Job control, data access

GRIMI 2 test suite (Envisat/MIPAS level 2 processor)

Complex MPI need

GeoCluster (GTC) Complex Job control, Portal

GOME validation(Ozone profile validation)

Complex workflow Data policy, data access, Portal

Space Physics Interactive Data resources (SPIDR) (meteorology)

Complex workflow Data access

Flood Forecasting Simulation Cascade (FFSC)

Complex workflow Complex workflow, Portal

PUMA (Atmospheric model) Complex workflow Workflow, data access

Test suites available


ApplicationApplication DB

GIS

Files

GridGrid

FilesFilesFiles

DBDBDB

GISGISGIS

Work doneSurvey of the existing data

technologies in ES

Survey of data usage policies in ES

Grid based data management for ES

application

Test c

ases

Data Management


Data management - Results

• Survey of the existing data technologies and usage policies in ES– more than 20 ES application analyzed – focus on:

• Data provision and data flow• Data access• Data Policies• Authorization, Authentication, Accounting

– requirements gathered• Test suites

– data management test suits developed• Grid based data management for ES applications

– main grid middleware distributions analyzed according to identified requirements: gLite, ARC/NorduGrid, Globus Toolkit, Unicore 5/6, Naregi,GRIA

– gaps identification– analysis of middleware services for specific problem areas (e.g.

metadata services)


Requirements on data management

• Intensive use of file resources – large files / large number of files

• Data access – Databases– GIS systems

• Strong Data Access Policies• Authorization, Authentication, Accounting


Gaps in data management

• metadata management• database access from grid and replication

(mirroring)• toolkits providing integration of existing

functionality as atomic operations• QoS mechanisms for data management• middleware and grid services for specialized

data access protocols and interoperability at data level

AMGA – missing support for spatial metadata attributes

Some functionality required by ES can be achieved by combining functionality of multiple existing grid toolkits:

•Replicated data validity/integrity checks as part of replica management operations.•File registration + metadata registration as an atomic operation

•Support for special protocols (e.g. SEED (seismic data))•Support for data format description•Data format transformation•Interoperability of different ES/environmental risk models

Access protocols and transformation services are often ES specific; however well defined generic framework for specialized data access protocols and data transformation services would be advantageous.

GRelC – filled gap for database acces from grid, important improvement for ES community

Grid middleware for database replication and consistency missing


Job managements

Applications

Workflow managers

Metaschedulers

Tools

Monitoring

gLite ARC

GlobusUnicore

Middleware

LCG

Storage Storage Computing elements

Computing elements


Work done in Job managements

• Survey of existing technologies– Middlewares: LCG, gLite, Unicore– Tools: workflow managers, metaschedulers,

monitoring tools

• Identifying most critical requirements of ES– Near-realtime execution– Reliability– Distributed job management

• Identifying of missing technologies• Creating test suites


Gaps in job management

• Job execution: Near-realtime, reliability• Workflow: automatic workflow composition,

dynamic workflow• Monitoring: expected job start/end time,

notification, progress• MPI support: Unified and transparent way to

submit MPI jobs• Licensing management and scheduling• Co-scheduling of data

gLite supports workflow management through specifications of job dependence in JDL document. Such JDL document must be submitted through a proxy service (WMproxy) which then submits the jobs described in the JDL according to the constraints which are specified for their dependency.

What is missing:• Real-time workflow modifications• User interaction with the workflow management• Fine-grained workflow monitoring• Visualization of the workflow process

• EGEE provide support only for short jobs (Short deadline jobs)

• QoS and fault tolerance are still not guaranteed

Partial support in gLite with job perusal

Difficult to find correct setting for MPI jobs

WMS should have integrated license manager to check license availability


ES portals


ES Portals for Grid, SOA and e-Collaboration

• Objective– Portal is in the critical path between User and Middleware– As such a key element to increase Grid uptake & exploitation– Where do we stand now and what needs to be done to ensure

adequacy for the future• on Middleware (& services) side• on Portals side

• Approach to the task– Establish current state-or-art

• in ES• in other e-Science communities

– Analyze the present solutions• what are ES Portals Requirements• are they met ?• what can be improved and how



• ES Grid Portals Survey– establish current state-of-art baseline– extent of uptake of Grid technologies in ES Portals– methods techniques and middleware used– types of solutions where Grid is used– ES Portals requirements– Deliverable D4.1 (available on the web)

• Generic Grid Portals Survey– looking outside ES Community– other e-science communities approach to Grid Portals– focus on generic Portals & middleware solutions– analysis of ES Grid Portals requirements vs. middleware gaps– recommendations and inputs for ES Grid Roadmap– Deliverable D4.2 (available on the web)



• Results– Wide range of ES Portals scenarios exploiting Grid, SOA,

eCollaboration– ES Portals Classification

• Data Dissemination: "Discover, identify and access ES data"

• Collaborative: "Online collaboration in ES Virtual Communities"

• Grid-based: "ES data intensive processing"

– ES Major Requirements• Integration of heterogeneous distributed services (Grid & Geo-

services)

• Support "Gridifcation" in Geo-services and Spatial Data standards

• Standard "off-the-shelf" tools for integrated Grid Security and User Management

• Big emphasis on Metadata and Data, its Discovery and Access...


Gaps in portal functionality

• Focus must be on ES functionality, grid working as back-end does not have to be visible.

• Higher level components targeted to ES– Computation submission without descending to grid-job level, maybe

even hiding the grid completely

• Big emphasis on Metadata and Data, its Discovery and Access– Browsing and accessing datasets the ES way

• Support for Spatial Data: INSPIRE, SDI, ...• Spatial data searches and OGC services (e.g. WMS)• Tools integrated with the Grid• Interoperability and interchange

– support for standard tools/protocols (ISO19115, OpenDAP, LAS, DODS, NETCDF, integration with OGSA-DAI)

– ontology / semantic web (developing the rudiments)

• Publish, subscribe, notify• Search, locate, access and process ES datasets of interest



• Graphical interfaces for different kinds of ES data, activated by data type– Input specification, e.g. area selection on a map for subset selection

– Output visualization and browsing: components for displaying time series of images, image layering

– Such components exist, but they use different technologies and APIs• Standard "off-the-shelf" tools for integrated Grid Security and User

Management– Interfacing Grid security and ES security Portal login models : User

management integrated with certificate management– Certificates generated on the portal by the portal, transparently– Loging-in into a portal should be enough to authenticate user

• There are existing activities and software to remedy this, e.g. PURSE (EarthScienceGrid), GAMA, ...



• Another approach – integrating grid into existing portals:

• Light-weight grid service interfaces to grid functionality for easy integration and/or mash-ups creation– Like Google Maps but for grids

– Would allow easy integration of grid services into (existing) ES portals


Conclusion

• ES applications have important requirements • The existing technologies have been reviewed

and missing features have been identified:– The features are essential for ES applications

• Expected feedbacks:– Improvement of Grid technologies (both in existing

and new Grid projects) to provide missing features– Better support for porting ES applications to Grid– Wider adoption of Grid technologies in ES community

Documents

DEGREE IST 2006- 034619 Dissemination and Exploitation of GRids in Earth sciencE Earth Science Communit y RequirementsRequirements T echnologies FeedbackFeedback