Scalable Sharing of Network Storage

Micah Beck, Research Assoc. Professor

Director, Logistical Computing & Internetworking (LoCI) Lab

Computer Science DepartmentUniversity of TennesseeCERN May 30, 2002

A Broad View of Networking

• A network is a facility for sharing a pool of resources within a community.

• Sharing is enabled by– Common goals– Standard protocols– Low allocation overhead– Mutual protection

IP Routing as Scalable Sharing of Bandwidth

• The Internet is a collection of links connected by routers

• Internet Protocol (IP) datagrams flow from sender to reciever along some path of links

• The choice of paths is made by routers according to topological considerations (tempered by policy)

Wide Area Network Services Are Like the Network Itself

• Intermittently inaccessible

• Vulnerable to partition

• Prone to corruption in transit

• Unpredictable latencies/jitter

• End-to-End: Never require a network service to be bigger, better or more complex than wide area access allows

Why Is It Called “End-to-End”?

• If the end-points requires a service that is stronger than the network can provide, they must implement it on top of the network.

• The fundamental tools are – Aggregation (retransmission, fragmentation)– Timers– Protocol state maintained at the end-points

How Can Storage Be Shared Scalably?

• Don’t require it to be– Unbounded in size– Unbounded in duration– Perfectly available– Perfectly reliable

• Best effort network storage

• Implement these properties end-to-end

Sharing Storage Enables Advanced Network Apps

• Caching

• Staging (Replication)

• Localization (Rendezvous)

• Distribution (Multicast)

• Spooling (E-Mail)

• Temporary State Management (Checkpointing)

“Logistical” Networking

• Analogy to military or industrial logistics

• Distributing goods requires not only highways but also warehouses (depots)

• Data transmission is the highway

• Storage servers are the depots

• This is not Storage Networking

• Enabling new users and applications

Dimensions in Communication

Routing (spatial) Storage (temporal)

Logistical Networking

Logistical Networking

• Principle Investigators– Micah Beck

– James S. Plank

• Graduate Students– Erica Fuentes

– Xiang Li

• Funding– DoE SciDAC

– NSF Next Gen SW

– NSF Internet Tech.

• Research Staff– Scott Atchley

– Alex Bassi

– Ying Ding

– Hunter Hagewood

– Sharmila Kancherla

– Jeremy Millar

– Terry Moore

– Stephen Soltesz

– Yong Zheng

Internet Backplane Protocol (IBP)

• Servers that make allocation of primitive “byte arrays” available to clients

• Byte arrays are not blocks (more abstract)– Network capabilities

(primitive security)

– Variable extents

• Byte arrays are not files (weaker semantics)– Size & duration are limited

– “Volatile” allocations

– Best effort reliability and availability

– No directory structure, accounting

– No caching, replication

• An end-to-end approach to network storage

IBP Software Structure

• IBP Depots (servers) are daemons that serve local storage to IBP clients.

• IBP clients only link to a protocol library.

• Clients talk to depots using TCP/IP.– Alternatives are being investigated

• Design is for high-performance/scalability.

Allocation Attributes

• Permanent vs. Time-Limited

• Volatile vs. Stable

• Read/Write semantics:– Byte Array– Pipe– Circular Queue– Completely Destructive Write

Impressions of Networking for High Energy Physics (?)

• Reliable throughput is the limiting factor

• Modes of sharing can be “modified”

• The use of specialized data movement middleware is acceptable

• The data transfer network has very few nodes and small user communities

• Little patience for fancy architectural ideas

IBP Enables Heterogeneity

• The tyranny of “TCP-friendly” IP– Datagrams and usage policy must be

homogeneous end-to-end– Non-standard IP policies in the middle can leak

out onto public networks

• Good fences make good neighbors

• IBP can terminate a flow and regenerate it in different network, with localized control

Example:Tsunami

• Uses very aggressive UDP transfer with TCP back-link for control

• Runs using Abilene less-than-best-effort “Scavenger Service”

• Tsunami must not be routed onto a link where Scavenger Service is not supported!

• Solution: do not route the packets, require explicit buffering and retransmission

Scheduling Control Over Sharing Without IP QoS

• Consider a high speed link that operates at 10Gb/s, connected to two types of links: – 100Mb/s and 1Gb/s

• Jobs are 1TB transfers– ~20 hours at 100Mb/s, ~2 hours at 1Gb/s

• Suspending a slow job to allow a fast job through delays it by only 10%

Scheduled Sharing of Network Links Is In Your Future!

• Ignoring buffer control won’t make the need for scheduled sharing go away

• When was more control over time & space ever a bad thing?

• Are we revisiting CERNET/UUCP/BITNET/DECnet?A Short History of Internet Protocols at CERN Ben Segal / CERN PDP-NS April, 1995

• IBP is designed according to end-to-end principles (see SIGCOMM ’02 paper)

The Alternative to Interoperability In Buffering

• High performance data transfer is an isolated service, with “job scheduling” and “data staging” like at computation centers

• As more high performance links are built, concatenation is based on “gateways” rather than a scalable routing infrastructure

• Should networking solutions be built without the benefit of network architecture?

Application Areas

• Source routing• Bandwidth adaption• Reducing (BWdelay)• Reliable multicast• Content Distribution • Remote access to

structured data

• Managing computation state (NetSolve caching)

• Temporary storage• Very large data sets• Collaborative

computing & visualization

Building on IBP

• Many applications assume file semantics– Unbounded size & duration– High reliability & availability– Caching & replication

• In a layered architecture, these are implemented through aggregation and additional intelligence at the next level

The Network Storage Stack

Physical

Access (OS)

IBP (InternetBackplane Protocol)

LogisticalRuntime System

Logistical Tools

Applications

L-Bone exNode

ExNode vs inode

exNode

inode

IBP Allocations

the network

local system

disk blocks

kernel

capabilities

block addresses

user

ExNode Mobility

XML Serialization

The exNode serialization is a portable soft link

Sameple exNodes Services: Fragmentation, Replication

1 2 3

Building a Logistical Operating Environment

• Resource Discovery– Logistical Backbone registry (LBone: Plank)– Can also use DNS, could use MDS

• Programming Tools– exNode library, primitive commands– policy modules (replication, fragmentation)– cool applications (SC demo: IBPster MP3)

Related Work

• Peer-to-Peer Storage– Napster, Gnutella, etc– OceanStore (UC

Berkeley)

• Storage Area Networking– Fiber Channel,

Ultraband, iSCSI– Network Attached

Storage (CMU)

• Overlay Networking– Caching

– Multicast

– Data Grids

• Multimedia Protocols– Content Delivery

– Streaming

– Video on Demand

Logistical Networking: E2E Storage and Bandwidth

• Complex paths and buffers are exposed

• Assumptions are appropriately weak

• Resource management is passive

• Spatial and temporal control

• Applic. autonomy, network transparency

• Resources in the hands of end users

• Futures: more control, greater scope

Logistical Computing and Internetworking Laboratoryhttp://loci.cs.utk.edu

• Documents and software downloads