LFC tutorial Jean-Philippe Baud, IT-GT, CERN July 2010

LFC tutorial

Jean-Philippe Baud, IT-GT, CERN

July 2010

LFC tutorial

Agenda

• Introduction• DB schema• Authentication and authorization• Installation• Configuration• Log files• Statistics

LFC tutorial

Introduction

• LFC stands for LCG File Catalogue• Development based on lessons learned in DC’s

(2004) • Fixes performance and scalability problems seen in EDG

Catalogs• Cursors for large queries• Timeouts and retries from the client

• Provides more features than the EDG Catalogs• User exposed transaction API• Hierarchical namespace and namespace operations• Integrated GSI Authentication + Authorization• Access Control Lists (Unix Permissions and POSIX ACLs)• Checksums• Bulk methods have been added later

LFC tutorial

Database schema

File Replica

Storage File NameStorage Host

Symlinks

Link Name

File Metadata

Logical File Name (LFN)GUIDSystem Metadata (Ownership,Size, Checksum, ACL)

User Metadata

User Defined Metadata

LFC tutorial

Database tables (1)

• CREATE TABLE Cns_file_metadata (• fileid NUMBER,• parent_fileid NUMBER,• guid CHAR(36),• name VARCHAR2(255),• filemode NUMBER(6),• nlink NUMBER(6),• owner_uid NUMBER(6),• gid NUMBER(6),• filesize NUMBER,• atime NUMBER(10),• mtime NUMBER(10),• ctime NUMBER(10),• fileclass NUMBER(5),• status CHAR(1),• csumtype VARCHAR2(2),• csumvalue VARCHAR2(32),• acl VARCHAR2(3900));

LFC tutorial

Database tables (2)

• CREATE TABLE Cns_file_replica (• fileid NUMBER,• nbaccesses NUMBER,• ctime NUMBER(10),• atime NUMBER(10),• ptime NUMBER(10),• ltime NUMBER(10),• r_type CHAR(1),• status CHAR(1),• f_type CHAR(1),• setname VARCHAR2(36),• poolname VARCHAR2(15),• host VARCHAR2(63),• fs VARCHAR2(79),• sfn VARCHAR2(1103));

LFC tutorial

Database tables (3)

• CREATE TABLE Cns_groupinfo (• gid NUMBER(10),• groupname VARCHAR2(255),• banned NUMBER(10));

• CREATE TABLE Cns_userinfo (• userid NUMBER(10),• username VARCHAR2(255),• banned NUMBER(10));

LFC tutorial

Relationships between tables

GUID

Xxxxxx-xxxx-xxx-xxx-

System Metadata

“size” => 10234“cksum_type” => “MD5”“cksum” => “yy-yy-yy”

Symlink

/grid/dteam/mydir/mylink

Replica

srm://host.example.com/foo/barhost.example.com

Replica


Replica


Replica


Symlink


Symlink


LFN

/grid/dteam/dir1/dir2/file1.root

LFC tutorial

Implementation

• Client-server model• Multi-threaded server

• Server use a pool of threads• Each thread has its own DB connection (max 99 threads)

• Proprietary socket interface (Posix-like)• Server well decoupled from the database backend

• Allow to support easily different backends: Oracle, MySQL, PostgreSQL

• Portable code• Runs on Linux (SL and Debian), OpenSolaris and MacOSx

• Packages exist for SLC4, SL5, Debian 5 and OpenSolaris

LFC tutorial

Client interfaces

• Command Line Interface (Unix-like)• lfc-mkdir, lfc-ls …

• C API (Posix)• lfc_mkdir, lfc_opendir …

• Python and Perl modules

LFC tutorial

Namespace operations

• All names are in a hierarchical namespace• mkdir(), opendir(), etc…• Also chdir()• GUID attached to every directory and file

LFC tutorial

Bulk methods

• Avoid problems with long round-trip times• Examples:

• lfc_getreplicas: get replicas for a list of guids• lfc_getreplicasl: get replicas for a list of lfns• lfc_delfilesbyguid: delete files by guids• lfc_delfilesbyname: delete files by name• lfc_delfilesbypattern: delete files by pattern• lfc_registerfiles: register files with replicas or new replicas

LFC tutorial

Authentication and authorization

• The service has security built-in: GSI or Kerberos 5• The entries in the name space can be protected by

Posix Access Control Lists• All privileged operations can only be done with a

Host Certificate on a trusted host• VOMS integration: groups, sub-groups and roles are

supported

LFC tutorial

VOMS integration (1)

• DNs are mapped to virtual UIDs: the virtual uid is created on the fly the first time the system receives a request for this DN (no pool account)

• VOMS FQANs (groups, sub-groups and roles) are mapped to virtual GIDs, also created on the fly when first received

• A given user may have one DN and several FQANs, so a given user may be mapped to one UID and several GIDs

• Authorization in name space is done using primary and secondary groups

• File group ownership is using the primary group

LFC tutorial

VOMS integration (2)

• Support for normal proxies and VOMS proxies• Integration with CSEC (socket interface) and CGSI

(soap services)• Administrative tools are provided to manually

update the DB mapping table if necessary• To create VO groups in advance• To keep same uid when DN changes• To get same uid for a DN and a Kerberos principal

LFC tutorial

Access Control Lists

• LFC support Posix ACLs based on Virtual Ids• Access Control Lists on files and directories• Default Access Control Lists on directories: they are inherited by

the sub-directories and files under the directory• Example

• lfc-mkdir /grid/dteam/jpb• lfc-setacl -m d:u::7,d:g::7,d:o:5 /grid/dteam/jpb• lfc-getacl /grid/dteam/jpb # file: /grid/dteam/jpb # owner: /C=CH/O=CERN/OU=GRID/CN=Jean-Philippe Baud 7183 # group: dteam user::rwx group::r-x #effective:r-x other::r-x default:user::rwx default:group::rwx default:other::r-x

LFC tutorial

Deployment

• LFC can be deployed as• Central catalogue• Local catalogue• Replica of central catalogue

• Replication uses Oracle STREAMS

LFC tutorial

Installation

• Port to be opened: 5010/tcp• Install host certificate on the server host

• /etc/grid-security/hostcert.pem• /etc/grid-security/hostkey.pem• /etc/grid-security/lfcmgr/lfccert.pem• /etc/grid-security/lfcmgr/lfckey.pem

• Install /opt/lcg/etc/lcgdm-mapfile• Install *.lsc files in /etc/grid-security/vomsdir

LFC tutorial

Configuration

• 3 methods can be used:• Yaim• Quattor• Manual

• /etc/sysconfig/lfcdaemon• RUN_READONLY=“no”• RUN_DISABLEAUTOVIDS="no“• ALLOW_COREDUMP="yes“ (recommended)• NB_THREADS=20 (default, but should be 60 for large VOs)• ORACLE_HOME

• /opt/lcg/etc/NSCONFIG• lcg_lfc_local_test2_w/My_db_pwd@int6r

LFC tutorial

LOGS

02/11 09:16:37 24739 Cns_serv: started (LFC 1.7.0-0)02/11 11:12:09 24739,0 Cns_srv_lstat: NS092 - lstat request

by /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=baud/CN=373165/CN=Jean-Philippe Baud (101,106) from lxb7994.cern.ch

02/11 11:12:09 24739,0 Cns_srv_lstat: NS098 - lstat 0 /02/11 11:12:09 24739,0 Cns_srv_lstat: returns 0

LFC tutorial

Statistics

• LFC installed at 60 sites• LFC used by tens of VOs, including ATLAS and LHCb

LFC tutorial

Monitoring

• Check the maximum number of threads in use• Check that a file entry can be listed• Check that an entry can be created or modified• Look for authentication errors (“Csec” messages)• Look for DB errors (“ORA” errors if the backend is

Oracle). Most of the DB errors are recovered thru internal retries.

• Look for procdirreq, procsessreq and proctransreq errors

LFC tutorial

Documentation and support

• https://svnweb.cern.ch/trac/lcgdm• [email protected]• [email protected]• https://svnweb.cern.ch/trac/lcgdm/roadmap• https://twiki.cern.ch/twiki/bin/view/EGEE/

DMReleaseStatus

Documents

LFC tutorial Jean-Philippe Baud, IT-GT, CERN July 2010