IT-SDC : Support for Distributed Computing

Alejandro Alvarez Ayllonon behalf of the DM Clients developers

GFAL/LCGUTILto

GFAL2/GFAL2 migration

2IT-SDC : Support for Distributed Computing

Remember remember the 1st of November

3IT-SDC

The 1st of November

GFAL/LCGUTIL will not be supported They will be removed from Fedora/EPEL

Fear not! We will give support for migration indefinitely

4IT-SDC

Release cycle for gfal2 components

EPEL

• Full production release

Development

• Increase of minor version• New functionalities• Non critical bug fixes• Nightly tests• Deployed in the FTS3

Development Services

RC

• Increase of revision number• Critical bug fixes• Deployed in the FTS3 Pilot

Service

EPEL Testing

• Increase of release number• “Quarantine” stage of a

release• Packaging fixes• Critical patches• Beta testers welcome!

Merges

5IT-SDC : Support for Distributed Computing

Getting started

6IT-SDC

Installation

Enable EPEL (5, 6 or even 7!) Install

gfal2-all Core + Transfer, DCAP, GridFTP, HTTP/DAV,

LFC, RFIO and SRM gfal2-python gfal2-util gfal2-plugin-xrootd for xrootd support

7IT-SDC

Fast check

$ voms-proxy-init --voms dteam$ gfal-ls gsiftp://lxfsra04a04.cern.ch/dpm/drwxr-xr-x 0 0 0 Jul 12 2013 0 cern.ch

8IT-SDC : Support for Distributed Computing

Command Line InterfaceSide by side

9IT-SDC

Description lcg-util gfal2-utilAdd an alias for a given GUID lcg-aa DeprecatedBring SURLS online lcg-bringonline N/A, can be doneCopy files with no catalog lcg-cp gfal-copyCopy file and register lcg-cr gfal-copyDelete a file / directory lcg-del gfal-rmGet checksum lcg-get-checksum gfal-checksumGet TURL/s lcg-getturls gfal-xattrList aliases for a given GUID lcg-la DeprecatedGet the GUID for an LFN lcg-lc gfal-xattrList the replicas for an LFN lcg-lg gfal-xattrDirectory listing lcg-ls gfal-lsRemove an alias lcg-ra Deprecated

10IT-SDC

Description lcg-util gfal2-utilCopy between SEs with registration lcg-rep gfal-copyRegister a file in the catalog lcg-rf gfal-copySet a file status to done lcg-sd DeprecatedGet space tokens lcg-stmd gfal-xattrUnregister a file lcg-uf DeprecatedCreate a directory gfal-mkdirDump stdin into a file gfal-saveDump a file into stdout gfal-cat

11IT-SDC

Two Swiss Army Knifes: gfal-copy

LFC is not a special citizen, but some operations can be mapped easily

Registration = COPY

12IT-SDC

# Copy & Register (lcg-cr)gfal-copy file:///local/file gsiftp://se.cern.ch \ lfc://lfc.cern.ch/lfn

# Copy & Register (lcg-rep)gfal-copy gsiftp://se-a/path gsiftp://se-b/path lfc://lfc.cern.ch/lfn

# Register a file (lcg-rf)gfal-copy gsiftp://se/path lfc://lfc.cern.ch/lfn

13IT-SDC

Two Swiss Army Knifes: gfal-xattr

Get/set extended attributes: key => value

Some “special” mappings user.replicas user.guid user.comment spacetoken

spacetoken.token?<token> spacetoken.description?<description>

14IT-SDC : Support for Distributed Computing

Examples

15IT-SDC : Support for Distributed Computing

GFAL2 Python APISide by side

16IT-SDC

GFAL: Less Pythonicimport gfalgfalDict = {‘surls’: ‘srm://…’}err, obj, msg = gfal.gfal_init(gfalDict)ret, buff, msg = gfal.gfal_ls(obj)gfal.internal_free(obj)

17IT-SDC

GFAL2: More Pythonic

import gfal2ctx = gfal2.creat_context()try:

ctx.listdir(‘srm://…’)except gfal2.Gerror, e:

print “[%d] %s” % (e.code, e.message)

18IT-SDC

Description gfal gfal2

List a directory gfal_ls ctx.listdirctx.opendir + read/readpp

Extend lifetime gfal_pin Not available

Async bring online gfal_prestage ctx.bring_online

TURLS from SURLS gfal_turlsfromsurls ctx.getaxttr

Delete SURLS gfal_deletesurls ctx.unlink

Delete directory gfal_removedir ctx.rmdir

Status of async bring online gfal_prestagestatus ctx.bring_online_poll

Release SURLS gfal_release ctx.release

Copy a file lcg_cp4 ctx.filecopy

Remote random IO supported too, and thread safe!

19IT-SDC

Copy a file

ctx.filecopy(params, source, dest) Params is an instance of ctx.transfer_parameters Members

checksum_check set_user_defined_checksum(algo, value) dst_spacetoken event_callback monitor_callback nbstreams overwrite src_spacetoken timeout

20IT-SDC

Examples

Check our repository

https://svnweb.cern.ch/trac/lcgutil/browser/gfal2-bindings/trunk/example/python

New DM Clients and retirement plans for old ones 21IT-SDC

Do not forget

Supported protocols SRM GridFTP HTTP/DAV LFC RFIO DCAP XROOTD

22IT-SDC

Questions?