Grid ComputingAn Introduction

1

Mathias Dalheimer

2

3

Competence CenterHigh Performance

Computing

4

More material:http://gonium.net/md

5

What is Grid Computing?

6

Virtualization

7

Grid problem: coordinated resource sharing and problem solving in dynamic, multi-institutional

virtual organizations.8

coordinated resource sharing and problem

solving

9 10

11 12

coordinated resource sharing and problem

solving

13

in dynamic, multi-institutional virtual

organizations

14

15 16

in dynamic, multi-institutional virtual

organizations

17

Virtualization?

18

19

vir-tu-al:1: being such in essence or effect though not formally recognized or admitted <a virtual dictator>2: ...

20

21 22

Grid Characteristics

Distributed SystemsSite Autonomy

But also: High securityVirtual resources in a pool

23

Laptop

GatewayEurope

Firewall

Suprenum

Zuse Z1

Eniac

Univac

GatewayUSA

24

Demo

25

2 The Architecture of UNICORE

Figure 1 shows the layered Grid architecture of UNICORE consisting of user,server and target system tier [5]. The implementation of all components shown isrealized in Java. UNICORE meets the Open Grid Services Architecture (OGSA)[6] concept following the paradigm of ’Everything being a Service’. Indeed, ananalysis has shown that the basic ideas behind UNICORE already realizes thisparadigm [7,8].

2.1 User Tier

The UNICORE Client provides a graphical user interface to exploit the entireset of services o!ered by the underlying servers. The client communicates withthe server tier by sending and receiving Abstract Job Objects (AJO) and filedata via the UNICORE Protocol Layer (UPL) which is placed on top of theSSL protocol. The AJO is the realization of UNICORE’s job model and centralto UNICORE’s philosophy of abstraction and seamlessness. It contains plat-form and site independent descriptions of computational and data related tasks,resource information and workflow specifications along with user and securityinformation. AJOs are sent to the UNICORE Gateway in form of serialized andsigned Java objects, followed by an optional stream of bytes if file data is to betransferred.

Fig. 1. The UNICORE architecture.Streit et al.: Unicore - From Project Results to Production Grids

26

from: http://www.globus.org/toolkit/about.html

27

GWD-I (draft-ggf-ogsa-spec-1.5-005) 6 February 2006

ogsa-wg@ggf.org 27

Mathias Dalheimer ! 6.2.06 16:48

Formatted: Danish

Mathias Dalheimer ! 6.2.06 16:50

Mathias Dalheimer ! 6.2.06 16:48

Mathias Dalheimer ! 6.2.06 16:48

Andreas Savva ! 6.2.06 14:23

Formatted: Danish

Mathias Dalheimer ! 6.2.06 16:50

Mathias Dalheimer ! 6.2.06 16:48

Figure 4: Interactions of EMS services to execute a legacy BLAST job

3.5 Data Services

This section describes those OGSA services concerned with the management of, access to and

update of data resources, along with the transfer of data between resources. These are

collectively called “data services”.

3.5.1 Objectives

Data services can be used to move data as required, manage replicated copies, run queries and

updates, and federate data resources. They also provide the capabilities necessary to manage the

metadata that describes this data in particular the provenance of the data itself.

For example, suppose an Execution Management Service (EMS) needs to access data that is

stored elsewhere. Data services allow that EMS to access the data remotely or to stage a copy to a

local machine. If it accesses the data remotely, a cache service may be used to cache some of the

data locally. The data may be available in multiple locations, with the data services ensuring a

consistency policy between the replicas. As with all OGSA services, the data services may

provide specific policy limitations or service guarantees.

Conversely, suppose that a data federation service wishes to define a schema for data stored in

different formats at different locations. It can use OGSA data services to specify how queries

against this schema can be mapped to the underlying resources, where joins or data

transformations should be executed and where the data should be delivered.

The data services do not rely on or specify the semantics of the data they hold. They operate with

generic data.

Provisioning

•Deployment

•Configuration

Information

Services

Service Container

Accounting Services

Execution Planning Services

Candidate Set Generator (Work -Resource mapping)

Job Manager

Reservation

1

0

2 3

6 5

4

Dynamic and static system information

7

Deleted: draft-ggf-ogsa-spec-1.5

Deleted: 005

Deleted: 6 February 2006

Inserted: 6 February 2006

Deleted: 11 January 2006

Andreas Savva ! 6.2.06 18:51

Formatted: English (US)

Andreas Savva ! 6.2.06 19:50

Mathias Dalheimer ! 6.2.06 16:48

Dave Berry ! 26.10.05 22:14

Deleted: <sp>

Deleted: 4

Deleted: data in

GGF OGSA-WG: Open Grid Services Architecture Draft V1.5

28

Key problems

• Security

• Resource Management

• Data Management

• Information Services

29

Grid Security

30

Security = Encryption + Authentication+Authorization

+ etc.

31

Encryption

32

Authentication

33

Passport: X.509

34

35

X.509 is not enough

• Dynamic delegation of rights to a service

• Delegation to dynamically generated services

• Solution:

• Di!erent signatures (Unicore)

• Proxy certi"cates (Globus)

36

Welch et al: X.509 Proxy Certificates for Dynamic Delegation

37

Security = Encryption + Authentication+Authorization

+ etc.

38

Authorization

39

DN -> Privilege mapping

40

Security = Encryption + Authentication+Authorization

+ etc.

41

Grid Resource Management

42

What is a grid resource?

43 44

45 46

47 48

Resource Management

49

!"#$%!&!!'()%!* * +*

*

!"#$

%&!&'()&*+,$)

-.&/"*0'%&!&''''''''''''''''''''''''''1&)&22$2'3.#45!&!".*'''' 1).6"/"*0'%&!&

%&!&'()&*+,$)'7$!8.)9':

3.#45!$)':

1&)&22$2'3.#45!&!".*3.#45!$)';

3.##5*"<&!".*',.)'3.#45!&!".*

7$!8.)9'=

>?@3&6$ >"+5&2"A&!".*

%&!& %&!&'B<<$++ C!.)"*0'%&!&

3.##5*"<&!".*',.)'>"+5&2"A&!".*

7$!8.)9';

C.,!8&)$'D+&0$C.,!8&)$'-"<$*+$

%&!&'C!.)&0$C!.)&0$

)$+.5)<$+

!"#$

%&!&'()&*+,$)

-.&/"*0'%&!&''''''''''''''''''''''''''1&)&22$2'3.#45!&!".*'''' 1).6"/"*0'%&!&

%&!&'()&*+,$)'7$!8.)9':

3.#45!$)':

1&)&22$2'3.#45!&!".*3.#45!$)';

3.##5*"<&!".*',.)'3.#45!&!".*

7$!8.)9'=

>?@3&6$ >"+5&2"A&!".*

%&!& %&!&'B<<$++ C!.)"*0'%&!&

3.##5*"<&!".*',.)'>"+5&2"A&!".*

7$!8.)9';

C.,!8&)$'D+&0$C.,!8&)$'-"<$*+$

%&!&'C!.)&0$C!.)&0$

)$+.5)<$+

*

Figure 2-1: Example schedule

*

*

2.4 Involved resources

,--*./01"*)'*#2#/-#3-4*%4")5%64"*7#8*34*%4954":41*38*:;4*5"4%<*#"*-)0!*#"*:;4*0464""#%8*

74#0"*#%4*/0*=-#64*:)*/0:4!%#:4*:;47*/0:)*:;4*"6;415-/0!*=%)64""(*>/!5%4*?$@*";)A"*:;4*

5"#!4*)'*%4")5%64"*"56;*#"*6)7=5:/0!<*1#:#<*":)%#!4<*04:A)%.*#01*")':A#%4*%4")5%64"<*#"*

A4--*#"*"=46/#-*142/64"(*B5:*/:*6#0*#-")*34*#0:/6/=#:41*:;#:*"4%2/64"<*"40")%"*)%*4240*

;57#0"*7#8*34*:%4#:41*#"*%4")5%64"*/0*#*C%/1*"6;415-/0!*6)0:4D:(**

2.5 Functional requirements

@(' Authentication, authorization, user right delegation & job integrity

verification(*,5:;40:/6#:/)0*#01*#5:;)%/E#:/)0*#%4*4""40:/#-*')%*424%8*C%/1*3#"41*

F)3*"537/""/)0*"640#%/)(*G)*40#3-4*:;4*"6;415-4%*:)*#6:*)0*34;#-'*)'*:;4*5"4%*:;4*

%4"=46:/24*%/!;:"*;#24*:)*34*14-4!#:41*'%)7*:;4*5"4%*:)*:;4*"6;415-4%(*G;/"*5"4*

6#"4*#-")*%495/%4"*:;#:*:;4*/0:4!%/:8*)'*#*F)3*H=#%:"*)'*:;4*F)3I*6#0*34*24%/'/41*

#08:/74*15%/0!*:;4*"6;415-/0!*=%)64""(*

?(' Job parsing & validation. G;4*F)3*14"6%/=:/)0*;#"*:)*34*=#%"41*#01*')%7#--8*

2#-/1#:41*HF)3*=%4$=%)64""/0!I(*

J(' Information retrieval (static & dynamic)(*G)*7#=*:;4*%4")5%64*%4954":"*

6)0:#/041*/0*:;4*F)3*14"6%/=:/)0*)0:)*#2#/-#3-4*%4")5%64"<*/0')%7#:/)0*#3)5:*:;4*

%4")5%64"*#01*:;4/%*":#:5"*;#"*:)*34*%4:%/4241*'%)7*#==%)=%/#:4*40:/:/4"*H#01*)''4%41*

38*:;4"4*40:/:/4"I(*K:*";)5-1*34*=)""/3-4*:)*!#:;4%*":#:/6*HL":#:/6M*A/:;*%4"=46:*:)*:;4*

%50:/74*)'*:;4*F)3I*#01*180#7/6*%4")5%64*/0')%7#:/)0*"4=#%#:4-8*:)*%4":%/6:*:;4*

:/74$6)0"57/0!*180#7/6*/0')%7#:/)0*%4:%/42#-(*

Grid Scheduling Use Cases. Proposed GGF document.

50

51

• Resource Discovery

• Information Gathering

• Job Execution

Schopf: Ten Actions when Grid Scheduling

Grid Resource Management

52

Resource Discovery

• Authorization "ltering

• Application requirement de"nition

• Minimal requirement "ltering

53

Information Gathering

• Dynamic information gathering

• System selection

54

Job Execution

• Advance reservation

• Job submission

• Preparation tasks

• Monitor progress

• Cleanup tasks

55

Di!erent architectures

56

Centralized Scheduling

57

Hierarchical Scheduling

58

Decentralized Scheduler

59

Negotiation

60

Data Management

61

EGEE

62

• 80 GB/sec. continuously

• Data is streamed to computing centers across Europe

• Experiments are run on the stored data

63

• Manage huge amounts of data

• Provide data where it is needed

• Determine where it is needed

• Help to "nd speci"c datasets

Data management challenges

64

GridFTP

• Add-on to FTP:

• Uses GSI to authenticate users

• All data transfers are encrypted

• Allows third-party transfers

• Striped File transfer

• Partial transfers

65

But: Not su#cient

• Doesn’t abstract from the resource

• No search mechanisms for speci"c datasets

• Doesn’t integrate di!erent storage technologies

• No support or replicas and caches

66

67

MCAT server metadatacatalog

SRB server

SRB server

client

68

MCAT Server

UserMCAT Server

User Agent

Metadata Catalog

User Agent

1 2

34 5

67+ 8

9

10

69

Additional concepts

• Fine-grained access control

• Ticket mechanism allows temporary access delegation

• Automatic or manual replication of datasets

• Caching on fast media while archiving on slow media

70

http://gonium.net/md

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 2.0 Germany

License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/2.0/de/ or send a

letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA.

71