Owen Synge Title of Talk Slide 1

Storage Management

• Owen Synge– Developer, Packager, and first line support to System

Administrators.

• Talks Scope– GridPP for the year ahead.

• Role– RAL employee, RAL data management Team.– Working within GridPP2.– Mass Storage and Local Data Management.

• Project Members– Owen Synge, Jens Jensen, Tara Shah, Glen Johnson

Owen Synge Title of Talk Slide 2

Summary

• Background• History• Current State• Mass Storage

– Mass Storage Future– Mass Storage Future : Upcoming Releases– Mass Storage Future : SRM Release– Mass Storage Future : SRM News

• Worker Node Disks– Local Storage Future : Time Scales– Local Storage Future : Priorities/Features

• Storage Future• Conclusion• Issues

Owen Synge Title of Talk Slide 3

Background

• ADS Service– Peta-byte level Tape Storage Solution

• Grid Summary– Distributed Computing– Commodity Hardware

• European Data Grid Project (EDG)– Europe wide Grid Research Project– Provided much of LCG infrastructure

• SRM Collaboration – Collaboration between Fermilab, Jefferson Lab, Lawrence

Berkeley, RAL, CERN– Contributed to the design of the SRM version 2 protocol

Owen Synge Title of Talk Slide 4

History

• EDG SE History– V1

• HTTP based Storage interface

– V2• Java Web services Storage Interface

– V3 (Not History as Still on V2.2)• SRM Standard Compliant Interface

Owen Synge Title of Talk Slide 5

Current State

• EDG SE Status– Stable Version

• Stable since November.– One Bug Fix (File Truncation)– One Administration Script created.– Atlas Data Transfer

» About 1500 files; 2 TB in total transferred to CERN• Going into LCG Production (Security bug on Client Side).

– ADS Support– Not using Disk, Castor or HPSS interfaces

– Current Development Version • Released this month.• Configuration Upgrade (Layered Template System).• Metadata Upgrade (See Later).

Owen Synge Title of Talk Slide 6

Mass Storage Future

• Addressing issues in EDG-SE– Metadata/Configuration (Due in near future)– Scalability (Number of Files)– Performance (Time taken and resources used)– SRM Compliance

• Addressing issues in GridPP– Disk Resources as MSS system– Checksums (Atlas request)

• Addressing issues for LCG (UK)– Clustering for Bandwidth– Testing frameworks

• Addressing issues (Generic)– Access Control– Job/User Namespaces

Owen Synge Title of Talk Slide 7

Mass Storage Future : Upcoming Release Time Line

• Release 2.2– Renaming of LCG release

• Release 2.3 (May/June 2004)– Layered Configuration– Metadata upgrade

• Release 2.4 (Before August 2004)– LCG Release of 2.3 when Stable and bugs

squashed• Release 3.0 (Before August 2004)

– SRM compliance• Release 3.1

– To be decided

Owen Synge Title of Talk Slide 8

Mass Storage Future : SRM Release

• Extra Features– Multiple Files acted upon with a single operation– Fully Asynchronous (srmStatusOfGetRequest)

• Benefits– Interoperability with major Storage solutions in

Grid community.– GFAL and a large number of other Client tools

available.

Owen Synge Title of Talk Slide 9

Mass Storage Future : SRM News

• SRM2– Finalised now working on 2.1 Revisions

• SRM and the GGF– SRM is going to be a GGF standard (Honolulu)– Specification of Basic/Advanced SRM

• We will provide a service somewhere between these specifications for ADS on the first SRM release

• SRM and Other Storage solutions– SRB

• On going work to support SRM API

– DCache • Currently supports SRM v1 API

Owen Synge Title of Talk Slide 10

Worker Node Disks

• EDG/EGEE Local Data Management– Missing from EDG

• Clean Down Worker Nodes• Reservation

– Remote File Access• RFIO, SlashGRID, Replica Manager/EDG SE Client tools,

GFAL.

• DCache– Aggregate Worker node space into storage– Mature system

Owen Synge Title of Talk Slide 11

Local Storage Future : Time Scales

• PM03 LD1: UK Site Coordinated Deployment and Support Plan for Local Storage Management (LSM) system.

• PM05 LD2: Release of LSM integrating with LCG3 at UK sites. LCG 3 expected in PM05.

• PM07 SD3, LD3: Software prototype release integrating with EGEE DJRA1.3

• PM16 SD4, LD4: software prototype release integrating with EGEE DJRA1.6

• PM26 SD5, LD5: Release of MS and LSM integrating with EGEE follow-on.

Owen Synge Title of Talk Slide 12

Local Storage Future : Priorities/Features

• Requirements– Disk Clean down?– Using Worker Node disk space?– Space management?

• Specification– Need to establish which requirements are

highest priority.

• Implementation– Need to get the plan signed off.– Need to employ a new member of the team.

Owen Synge Title of Talk Slide 13

Storage Future

• SRM (De facto Grid Standard)– SRM1 moving to SRM2– GSM Basic moving to GSM Advanced

• Worker Nodes– Clean down after Jobs– DCache

• Must not forget Tier 2+3 have more storage space than Tier 0+1

Owen Synge Title of Talk Slide 14

Conclusion

• SRM– On going evolution of a standards management

API– Not yet clear where access control is going exist

in this area.– No SRM2/GSM Advanced implementations yet

exist

• LSM– Local Storage management solutions not yet

clear

Owen Synge Title of Talk Slide 15

Issues

• Tracking SRM scope and model • Local Storage Priorities• Recruiting and Training new member of the

team• Testing environments for LCG/EGEE• Representation of files (Metadata Group?)

– Trees/UID– Namespaces

• Job/Service/User/VO/host based