PanelAbstractions for Large-Scale Distributed

Systems

Henri Balbal@cs.vu.nl

Vrije Universiteit Amsterdam

Background - Projects● Virtual Laboratory for e-Science (VL-e)

● 40 M€ Dutch project

● Generic applicationsupport for e-Science

● Scientific workflow applications

● Ibis: grid programming environment for distributed supercomputing applications

Background – Infrastructure:

DAS-3 Computer Science grid

WAN: 20-40 Gb/s Optical Private Network

● Performance & scalability

● Heterogeneous● Low-level & changing

programming interfaces

Why are distributed applications difficult to

program & deploy?● Connectivity

issues● Fault

tolerance● Malleability

Wide-Area Grid Systems

UseUserr

!

Common (scalable) patterns

• Programming– Master-worker => divide-and-conquer

• Image analysis, N-body, SAT, gene seq, grammar learning, ….

– Asynchronous high-throughput communication• Model checking, search applications

– Mixed task- and data-parallelism• Multimedia content analysis

• Deployment (OS-like functionality)– File I/O, resource allocation, job submission, …

fib(1) fib(0) fib(0)

fib(0)

fib(4)

fib(1)

fib(2)

fib(3)

fib(3)

fib(5)

fib(1) fib(1)

fib(1)

fib(2) fib(2)cpu 2

cpu 1cpu 3

cpu 1

Abstractions we use● Programming

● Divide-and-conquer (Satin)● Fault-tolerant, malleable

● IPL: Java-centric gridcommunication layer

● Run-anywhere, connectivity

● Deployment● JavaGAT, SAGA

(hide low-level APIs)Ibis

Abstractions we need

● Transparent grid libraries● Domain-specific patterns● E.g. imaging operations

● More scheduling functionality● Give me N clusters with

P processors at time T● Map application onto heterogeneous

grid (containing clusters, Cells, GPUs)

● Large-scale e-Science infrastructures● Computing + storage + networking● High-bandwidth, flexible optical WANs

● Supporting large-scale systems● Exploit locality in hierarchical systems● Latency-tolerant algorithms & systems● Bandwidth allocation (light paths)

● Adapt network topology to application: alloc(80-Gb/s, A, B)

● Co-allocate CPU, data, network

Emerging distributed infrastructures (1)

● Very large scale: mobile devices● Smart phones: ubiquitous, location-aware● Distributed smart phone applications

● Social networking, crisis management● Integrated with grid backbone

● Supporting such applications● Location-aware communication (geo-cast)● Mobile ad hoc networks● Dealing with resource constraints

Emerging distributed infrastructures (2)