Globus and ESGF

Rachana Ananthakrishnan

University of Chicago

ranantha@uchicago.edu

Reliable, secure, high-performance file transfer and synchronization

• “Fire-and-forget” transfers

• Automatic fault recovery

• Seamless security integration

• Powerful GUIand APIs

DataSource

DataDestination

User initiates transfer request

1

Globus moves and replicates files

2

Globus notifies user

3

3

Recent Transfer Features• Network manager plug-in• Management console

– Task monitoring, cancellation, pause/resume

• Network use controls• Various Storage systems:

– HPSS, Amazon S3, OpenStack Ceph*, SpectraLogic BlackPearl*

Simple, secure sharing off existing storage systems

DataSource

User A selects file(s) to share, selects user or group, and sets permissions

1

Globus tracks shared files; no need to move files to cloud storage!

2

User B logs in to Globus and

accesses shared file

3

• Easily share large data with any user or group

• No cloud storage required

5

Manage shared endpoint access

• Administrators configure:• Sharing as an option• Parts of file system to share from• Users who can create share• Permissions on share (e.g read-only)

• User configure per folder• Share to user, group or all• Invite via email• Level of access: read-only or read/write

6

COG use of Globus Sharing

• NASA JPL has a Globus endpoint on their ESGF data node

• Shared endpoint off folder with public data

• Set permission for public access • COG redirects users to Globus webapp

for destination with shared endpoint as source

Globus Platform-as-a-Service

Identity, Group, Profile Management Services

…

Sharing Service

Transfer Service

Globus Toolkit

Glo

bus

API

s

Glo

bus

Conn

ect

8

NCAR RDA

• Integrate Globus for data distribution

• Shared endpoint with subfolder per request

• Single sign on via streamlined account provisioning

9

Curated publication of data, with relevant metadata for discovery

• Identify• Describe• Curate• Verify• Access• Preserve

Researcher assembles data set; describes it using metadata (Dublin core and domain-specific)

1Peers, public

search and discover data sets; transfer using Globus

3

PublishedDataStore

Curator reviews and approves; data set published on campus or other storage

2Metadata

10

Upcoming Globus Features

• HTTP/S access to files• Globus sharing option for anonymous

access• Streamlined account provisioning

– No need for Globus username/password

• Support for OIDC and OAuth 2.0

11

ESGF use of Globus

• User data download– Integration with COG for restricted data– Requires all data to be published with GridFTP URLs– OAuth 2.0 for credential delegation

• Replication– Globus Transfer as an option in Synda– Performance tuning – Installation of Transfer node for DTNs

• Publication– Ingestion service as part of installer– COG front end to service

12

Opportunities to leverage Globus

• ESGF IdP as alternate IdP– Seamlessly use Globus features

• Facilitate federated IdP use in ESGF– Globus support ~157 federated identities

• Use HPSS endpoints for data access

Thank You

14

Globus Subscriptions• Provider

– Host shared endpoints– Management console– Data publication– Amazon S3 endpoints– Usage reports– Priority support

• Branded Web Site• Alternate Identity Provider (InCommon is standard)• Mass Storage System optimization

globus.org/provider-plans