Author - Title- Date - n° 1

PartnerLogo

EU DataGrid, Work Package 5http://www.eu-datagrid.org/

The Storage Element

John Gordon - SE Workshop - 24/09/01° 2

An SE in Testbed1

A disk server

running GDMP v2.0

running information providers

running data servers (RFIO, GridFTP)

optionally NFS mount user data on local CE

optionally stage replicated data to MSS

John Gordon - SE Workshop - 24/09/01° 3

Long Term Goals for SE

This is not the long-term vision.

By Testbed2 we should have: replaced current GDMP by WP2 ReplicaManager

a functioning integrated SE Grid Service

This talk is about that Integrated SE Service

John Gordon - SE Workshop - 24/09/01° 4

The SE Provides: Grid interface to permanent storage

Grid security Grid protocols:

Data transfer Control Information

Additional functionality SE provides functionality not supported by underlying storage

system e.g. access control, logging, monitoring

John Gordon - SE Workshop - 24/09/01° 5

StorageElement(WP5)

Resource broker(WP1)

ReplicaCatalog(WP2)

ReplicaManager(WP2)

CE(WP4)

Applications(WP8,9,10)

Network(WP7)

SE details

Data

GridFTP monitoring

Data

Verifi cation

Metadata publication(WP3)

Metadata

StorageElement(WP5)

Resource broker(WP1)

ReplicaCatalog(WP2)

ReplicaManager(WP2)

CE(WP4)

Applications(WP8,9,10)

Network(WP7)

SE details

Data

GridFTP monitoring

Data

Verifi cationSE details

Interaction with other Work Packages

John Gordon - SE Workshop - 24/09/01° 6

Grid Clients Other clients

SE

Mass Storage Systems

John Gordon - SE Workshop - 24/09/01° 7

Interface1

Interface3

Interface2

Message Queue

Session Manager

System Log House Keeping

MetaData

MSSInterface

MSSInterface

MSS1 MSS2

Top layer

Core

Bottom layer

Clients

Storage Element

SE

Architecture

John Gordon - SE Workshop - 24/09/01° 8

Clients

Resource Broker & Job Submission Service (WP1)

Replica Manager (WP2) replicates files between StorageElements

Information services (WP3)

User Applications running on a ComputingElement

command line utilities

John Gordon - SE Workshop - 24/09/01° 9

Control Interface

Control interface allows:

File creation and deletion

Reserving files for reading (“pinning”)

Reserving space for writing

Changing file permissions and lifetime

Delegations

PPDG are developing an SRM API for some of these aspects which we plan to implement.

John Gordon - SE Workshop - 24/09/01° 10

We have written a query program edg_se_query

that performs this query

Information Interface

SE acts as provider for Grid Information Service (MDS)

SE metadata: protocols supported, free space, policies

File metadata: creation date, ownership, access time, pinning status, size, estimated access/transfer time

API provided so clients are not dependent on underlying Information Service

So can support replacement for MDS

John Gordon - SE Workshop - 24/09/01° 11

Data Transfer InterfaceWill support:-

GridFTP Standard FTP protocol extended to provide efficient

transfer

RFIO Used at CERN as interface to CASTOR

NFS Not secure by default, SE must provide secure version

Files are always stored in the storage system, but toclients it appears as if the file is stored in the SE.

John Gordon - SE Workshop - 24/09/01° 12

Client SE

ReplicaManager/Catalog

Storage5’

2

3

4

1

• (1) The Client asks a catalogue to provide the location of a file• (2) The catalogue responds with the name of an SE

• (3) The client asks the SE for the file• (4) The SE asks the storage system to provide the file

•(5,6) The storage system sends the file to the client through the SE or (5’) directly

5

6

John Gordon - SE Workshop - 24/09/01° 13

Supported Storage Systems

UNIX disk systems

HPSS (High Performance Storage System)

CASTOR (through RFIO)

GridFTP servers

DMF

Enstore

SRM?...

????????

John Gordon - SE Workshop - 24/09/01° 14

Security

Will integrate with EDG Security Model authentication

SE access control

File access control

Grid ACL?

John Gordon - SE Workshop - 24/09/01° 15

Client APIs

We will provide

Existing APIs: Compatibility libraries (e.g. RFIO libraries)

Support for existing network protocols (e.g. GridFTP)

SE-specific APIs: Libraries specifically written to access the SE

Command line utilities built using the above

John Gordon - SE Workshop - 24/09/01° 16

Interface Queue

Request Manager

Pipe Manager

Named Pipe

1

2 3

4

5

78Named Pipe

Named Pipe

Named Pipe

Pipe Store

Network

Request processing in the SE

Handler

HandlerHandler

Handler

Handler

Tape

Disk

6

MS

S

Interfaces Core and MSS Layer

John Gordon - SE Workshop - 24/09/01° 17

Summary

In Testbed1 WP5 will continue to enhance capability of RFIO, GridFTP, and Information Providers

By Testbed 2 we plan to have an Integrated SE grid service

which will provide extra functionality to ordinary data servers.

Reference: WP5 Architecture and Design document http://edms.cern.ch/document/336679