Die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH) D-Grid Marcel Kunze Steinbuch Centre for Computing (SCC)

Die Kooperation von Forschungszentrum Karlsruhe GmbH

und Universität Karlsruhe (TH)

D-GridMarcel Kunze

Steinbuch Centre for Computing (SCC)

2 | M.Kunze | NORDUnet2008 | 10.04.2008

KIT - Die Kooperation von Forschungszentrum Karlsruhe GmbH


The Three Waves of D-Grid

2003 - 2008 2008 - 2010 2010 - 2015

The Research Wave The Industry Wave The Consumer Wave

Technology, Prototypes Grid-Enabled Products Commodity Virtual Organizations Enterprise Solutions IT Utility, Web2.0 Standards Interoperability Integration OGF, IETF, OASIS OGF, IETF, OASIS Legal, Ethical, Political Orgs

Finally, Grids will “disappear”

3 | M.Kunze | NORDUnet2008 | 10.04.2008



Building a national e-Infrastructure for research and industry

01/2003: Pre-D-Grid Working Groups Recommendation to Government 09/2005: D-Grid-1: early adopters, ‘Research Grids’ 07/2007: D-Grid-2: new communities, ‘Service Grids’ 04/2008: D-Grid-3: Service Grids for research and industry

D-Grid-1: 25 MEuro > 100 Orgs > 200 researchers D-Grid-2: 35 MEuro > 100 addl Orgs > 200 addl researchers and industry D-Grid-3: Call expected in the comíng weeks (~20 MEuro)

D-Grid in a Nutshell

4 | M.Kunze | NORDUnet2008 | 10.04.2008



D-Grid-1

Generic Grid Middleware and Grid Services

Integration Project

As

tro

-Gri

d

C3

-Gri

d

HE

P-G

rid

IN-G

rid

Me

diG

rid

ON

TO

VE

RS

E

WIK

ING

ER

WIS

EN

T

Te

xtg

rid

. . . . . .

Im W

iss

en

sne

tz

User-friendly Access Layer, Portals

5 | M.Kunze | NORDUnet2008 | 10.04.2008



D-Grid-1, -2, -3


Integration Project

As

tro

-Gri

d

C3

-Gri

d

HE

P-G

rid

IN-G

rid

Me

diG

rid

ON

TO

VE

RS

E

WIK

ING

ER

WIS

EN

T

Te

xtg

rid

. . . . . .

Im W

iss

en

sne

tz

Knowledge Management

Business Services, SLAs, SOA Integration, Virtualization


6 | M.Kunze | NORDUnet2008 | 10.04.2008



D-Grid in 2010


As

tro

-Gri

d

C3

-Gri

d

HE

P-G

rid

IN-G

rid

Me

diG

rid

ON

TO

VE

RS

E

WIK

ING

ER

WIS

EN

T

Te

xtg

rid

. . . . . .

Im W

iss

en

sne

tz

Services Level AgreementsVirtual Competence Centers for

Resources, Middleware,Support, Training, Knowledge


Business Services, SLAs, SOA Integration, Virtualization

7 | M.Kunze | NORDUnet2008 | 10.04.2008



Nutzer

ApplicationDevelopment

and User Access

GAT API

Data/Software

Resourcesin D-Grid

High-levelGrid

Services

Basic Grid Services

DistributedData Archive

User

NetworkInfrastructur

LCG/gLite

Globus 4.0.1

AccountingBilling

User/VO-Mngt

SchedulingWorkflow Management

Data management

Security

Plug-In

UNICORE

DistributedCompute Resources

GridSphere

Monitoring

D-Grid Middleware

8 | M.Kunze | NORDUnet2008 | 10.04.2008



Current D-Grid Infrastructure (~20.000 Cores, 5 PB Storage)

Universität Marburg

Universität Wuppertal

Universität Siegen

FZK Karlsruhe

Fraunhofer SCAI

DKRZ Hamburg

FZJ Jülich

Zuse-Institut Berlin

Universität Dortmund

RRZN Hannover

LRZ München

HLRS Stuttgart

Universität Köln

DESY Hamburg

Universität Freiburg

ZIH Dresden

Universität Leipzig

GWDG Göttingen

ARI Heidelberg

9 | M.Kunze | NORDUnet2008 | 10.04.2008



HEP-Grid: p-p collisions at LHC at CERN

Crossing rate 40 MHzEvent Rates: ~109 Hz

Max LV1 Trigger 100 kHzEvent size ~1 MbyteReadout network 1 Terabit/sFilter Farm ~107 Si2KTrigger levels 2Online rejection 99.9997% (100 Hz from 50 MHz)System dead time ~ %Event Selection: ~1/1013

Crossing rate 40 MHzEvent Rates: ~109 Hz

Max LV1 Trigger 100 kHzEvent size ~1 MbyteReadout network 1 Terabit/sFilter Farm ~107 Si2KTrigger levels 2Online rejection 99.9997% (100 Hz from 50 MHz)System dead time ~ %Event Selection: ~1/1013

Event rate

“Discovery” rate

LuminosityLow 2x1033 cm-2 s-1

High 1034 cm-2 s-1

Data analysis: ~1PB/year

Level 1 Trigger

Rate to tape

10 | M.Kunze | NORDUnet2008 | 10.04.2008



AstroGrid

11 | M.Kunze | NORDUnet2008 | 10.04.2008



Climate research moves towards new levels of complexity:

Stepping from Climate (=Atmosphere+Ocean) to Earth System Modelling

Earth system model wishlist:

Higher spatial and temporal resolution

Quality: Improved subsystem models

Atmospheric chemistry (ozone, sulfates,..)

Bio-geochemistry (Carbon cycle, ecosystem dynamics,..)

Increased Computational demand factor: O(1000 -10000)

C3 Grid: Collaborative Climate Community Data and Processing Grid

12 | M.Kunze | NORDUnet2008 | 10.04.2008



GridspezifischeEntwicklungen

Integration project

Cooperation and businessmodels

InGrid: Virtual Prototyping & Modeling in Industry

Molding Metal Forming

Fluid Processes

Groundwater Transportation

Knowledge-based support for engineering-specific

decision support

Support for engineering-specific Workflows

Distributed simulations-based product & process

optimization

Methods and models for solving engineering problems in Grids

Fluid-Structur/ Magneto-Hydro- dynamic Interaction

Security and trust models

Grid-specific developments

AP 2 AP 3 AP 4

13 | M.Kunze | NORDUnet2008 | 10.04.2008



Raw Data

Metadata

Molecule

Homogenization

Target data

Metadata

Population

Metadata

Patient

Metadata

Illness

Metadata

Organ/Tissue

Metadata

Cell

Search, Find, Select

Access Control

Correlate, Process, Analyze

Resulting Data

Presentation

Final Result

MediGrid: Mapping of Characteristics, Features, Raw Data, etc

14 | M.Kunze | NORDUnet2008 | 10.04.2008



TextGrid Modular Platform for distributed cooperative scientific text processing for the humanities

Workbench for Publishing, Text Processing, Text Retrieval, Linking, and Workflow

Focus of TextGrid is on:- Quantity: Full text instead excerpts, text and pictures- Speed of reprography: stability of text, volatility of medium - Precision: Maximum requirements on correctness- Availability: international open standards

Seite 15

WISENT, Knowledge Network for Energy Meteorology

Wettersatellit MSG-1: Die Forscher der Universität Oldenburg und des DLR können damit alle 15 Minuten flächendeckend und präzise den

verfügbaren "Brennstoff" für Solarenergieanlagen bestimmen.Quelle: EUMETSAT

Charakterisation of energy fluctuation from solar and wind energy

Results through interdisciplinary collaboration (Meteorology)

Transformation of wind and solar energy (physics)

Flexible distributed systems (computer science)

16 | M.Kunze | NORDUnet2008 | 10.04.2008



D-Grid-2, Start June 2007

‘Horizontal’ Service Grids: professional Service Providers for heterogeneous user groups in research and industry

‘Vertical’ Community Service Grids using existing D-Grid infrastructure and services, supported by Service Providers

D-Grid extensions, based on a D-Grid 1 gap analysis - Tools for operating a professional grid service - Adding business layer on top of D-Grid infrastructure - Pilot service phase with service providers and ‘customers’

17 | M.Kunze | NORDUnet2008 | 10.04.2008



D-Grid-2 Projects

BauVOGrid: Grid-based platform for VOs in the Construction industry FinGrid: Financial Business Grid: A service grid architecture for the financial service industry AeroGrid: Grid-based collaboration among Aerospace research & industry ProGrid: Automotive: Cooperative product design and development in simulation and production data management BIS-Grid: Grid-based Enterprise Information Systems, integration & orchestration in commercial IT systems Biz2Grid: Enterprise Grids; Acct, Pricing, Billing, Metaschedlg; ERP/SAP PartnerGrid: Cooperation in industrial VOs (SME, supplier, svc providers) GDI-Grid: Geographical Data infrastructure for providing and processing data and simulation for catastrophes, noise, and navigation F&L-Grid: Grid-based IT services for research and education D-MON: Horizontal integration of resource and service Monitoring SUGI: Grid support for small institutions and SMEs

www.BauVOGrid.de

BauVOGridGrid-based platform forvirtual organisations in the construction industry

Seite 19

BIS-Grid OverviewBIS-Grid Overview

Utilisation of Grid technologies to integrate decentralised

business information systems

Seite 20

BIS-Grid OutcomeBIS-Grid Outcome

Horizontal Service Grid

New collaboration and business models for Grid Providing and utilisation

Especially aiming on SMEs

Utilisation of WS-BPEL to orchestrate stateful WSRF-conform Grid Services

Development of a Grid Service-enabled WS-BPEL engine Integrated with UNICORE 6

Evaluation of results in partner-specific application scenarios

Seite 21

BIS-Grid Application ScenarioBIS-Grid Application Scenario

Enterprise Application Integration using Grid technologies

Different sites Extensive amounts of

data and legacy data Interrelations of data

from different sources/databases

2222

Financial Business Grid

FinGrid 23

Research Approaches & Research Objectives

23

Design Science

System Engineeri

ng Approach

Case Studies

Development of 3 Grid-

Prototypes

Field Studies

Development of a

Grid-Economy

FinGrid Standard Project Presentation Release: 2007-08-10

Spatial Data Infrastructure Grid GDI-Grid 24

Geo-Data Infrastructure Grid

• Enhancement of existing GDI applications with Grid technologies:– meet enormous need for processing power– allow access to distributed data sources and huge data storage– combine current base technologies of spatial data infrastructures

(SDI) with Grid middleware (GT4)– enable seamless processing of spatial data in Grid infrastructures

• Development of generic services for the – integration– fusion– management and– processing

of spatial data on the D-Grid infrastructure

• Survey of user requirements, feasibility and marketable potential


Scenarios and Proof of Concept

Three selected scenarios for the evaluation of the implemented GDI-Grid components:– spatial simulation of flood disasters - flood hazard



Three selected scenarios for the evaluation of the implemented GDI-Grid components:– spatial simulation of flood disasters - flood hazard– noise dispersion simulation for noise pollution

predictions



Three selected scenarios for the evaluation of the implemented GDI-Grid components:– spatial simulation of flood disasters - flood hazard– noise dispersion simulation for noise pollution

predictions– route optimization for disaster management

28

PartnerGrid: Project Goal

Development of Grid platform that supports the collaboration within a virtual organisation:

– Definition and execution of complex and distributed process flows

– User interface for access and use of the Grid via portal technology

– Transparent access to distributed data (file systems, data bases)

– Management of distributed, virtual organisations and integrated security concepts

– Interfaces for integration of complex, application specific analysis modules

– Interface to service providers for the execution of compute intensive application

– User-transparent use of D-Grid services

– Secure and liable procurement of resources to the users considering resource provider and resource user

– Resource usage is charged by an accounting and billing function.

PartnerGrid Architecture

Portal

Gridsphere RCE Portlet

RCE-GUI

RCE-Framework

Wra

pper Anwendung

(PAM Crash,Indeed,

Magmasoft)

Services DGI Middleware

Globus UnicoreJob Submission

Datentransport Monitoring

Calana

D-Grid Ressourcen

Firmen-interneRessourcen

ExterneRessourcen

Portal

Gridsphere RCE Portlet

RCE-GUI

RCE-Framework

Wra

pper Anwendung

(PAM Crash,Indeed,

Magmasoft)

Services DGI Middleware

Globus UnicoreJob Submission

Datentransport Monitoring

Calana

D-Grid Ressourcen

Firmen-interneRessourcen

ExterneRessourcen

30

PartnerGrid: Application scenarios

Scenario: Simulation of a founding process– Use case optimisation

• Usage of Grid-resources with parallelism in two dimensions

• Connection of the optimisation-workflows to the grid computing

– Use case enterprise scenario• Deployment of Grid computing in a big enterprise

– Use case outsourcing• MAGMASOFT-Users can use external hardware-, software- and license

resources, that are provided by a independent service provider (MAGMA GmbH)

Scenario: Forming process technique– Use case „project processing between customer, supplier and service provider“

• more efficient project work, reduction of the project management cost

– Use case „provision of compute resources and support“• better utilisation of hard- & software-resources, consulting

service, extension of the service range

31 | M.Kunze | NORDUnet2008 | 10.04.2008



Building a national e-Infrastructure for research and industry

An attractive infrastructure has been set up Increasing number of projects and Grid users Impact of new developments like Cloud Computing?

Important: Operate a sustainable production grid infrastructure after the end of the funding Integration of new communities Evaluating business models (operational models) for grid services

D-Grid Summary

32 | M.Kunze | NORDUnet2008 | 10.04.2008



Acknowledgments

Slides and information have been provided by Wolfgang Gentzsch D-Grid Projects

33 | M.Kunze | NORDUnet2008 | 10.04.2008



Documents

Die Kooperation von Forschungszentrum Karlsruhe GmbH und Universität Karlsruhe (TH) D-Grid Marcel Kunze Steinbuch Centre for Computing (SCC)