Douglas Thain, John Bent Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny Computer Sciences...

Douglas Thain, John BentAndrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Miron Livny

Computer Sciences Department, UW-Madison

Gathering at the Well:

Creating Communities for Grid I/O

www.cs.wisc.edu/condor

4 earth-shattering revelations

1) The grid is big.2) Scientific data-sets are large.3) Idle resources are available.4) Locality is good.

www.cs.wisc.edu/condor

How to optimize job placement on the grid?› Move data to the jobs.› Move jobs to the data.› Allow jobs to access data remotely.

› Need framework for evaluation. I/O communities

www.cs.wisc.edu/condor

I/O Communities

UW INFN

www.cs.wisc.edu/condor

I/O communities are an old idea

› File servers and administrative domains

› But, we want more flexible boundaries simple mechanism by which users can

express I/O community relationships

www.cs.wisc.edu/condor

I/O communities› Mechanism which allow either

jobs to move to data, or data to move to jobs, or data to be accessed remotely

› Framework to evaluate these policies

www.cs.wisc.edu/condor

Grocers, butchers, cops› Members of an I/O community

Storage appliances Interposition agents Scheduling systems Discovery systems Match-makers Collection of CPU’s

www.cs.wisc.edu/condor

Storage appliances› Should run without special privilege

Flexible and easily deployable Acceptable to nervous sys admins

› Should allow multiple access modes Low latency local accesses High bandwidth remote puts and gets

www.cs.wisc.edu/condor

Interposition agents› Thin software layer interposed

between application and OS› Allow applications to transparently

interact with storage appliances› Unmodified programs can run in

grid environment

www.cs.wisc.edu/condor

Scheduling systems and discovery

› Top level scheduler needs ability to discover diverse resources

› CPU discovery Where can a job run?

› Device discovery Where is my local storage appliance?

› Replica discovery Where can I find my data?

www.cs.wisc.edu/condor

Match-making› Match-making is the glue which

brings discovery systems together› Allows participants to indirectly

identify each other i.e. can locate resources without

explicitly naming them

www.cs.wisc.edu/condor

Mechanisms not policies

› I/O communities are a mechanism not a policy

› A higher layer is expected to choose application appropriate policies

› We will however demonstrate the strength of the mechanism by defining appropriate policies for one particular application

www.cs.wisc.edu/condor

Experimental results› Implementation› Environment › Application› Measurements› Evaluation

www.cs.wisc.edu/condor

Implementation› NeST

storage appliance› Pluggable File System (PFS)

interposition agent built with Bypass› Condor and ClassAds

scheduling system discovery system match-maker

www.cs.wisc.edu/condor

Two I/O communities› INFN Condor pool

236 machines, about 30 available at any one time

Wide range of machines and networks spread across Italy

Storage appliance in Bologna• 750 MIPS, 378 MB RAM

www.cs.wisc.edu/condor

Two I/O communities› UW Condor pool

911 machines, 100 dedicated for us Each is 600 MIPS, 512 MB RAM Networked on 100 Mb/s switch One was used as a storage appliance

www.cs.wisc.edu/condor

CMS simulator sample run

› Purposefully choose a run with high I/O to CPU ratio

› Accesses about 20 MB of data from a 300 MB database

› Writes about 1 MB of output› ~160 seconds execution time

on a 600 MIPS machine with local disk

www.cs.wisc.edu/condor

Assume the position› We assumed the role of an Italian

scientist› Database stored in Bologna› Need to run 300 instances of simulator

› How to take advantage of UW pool? Three way matching

www.cs.wisc.edu/condor

Three way matching

Machine NeSTJob

JobAd

MachineAd

StorageAdm

atch

Refers toNearestStorage.

Knows whereNearestStorage is.

www.cs.wisc.edu/condor

Two way ClassAdsType = “job”TargetType = “machine”Cmd = “sim.exe”Owner = “thain”Requirements = (OpSys==“linux”)

Job ClassAd

Type = “machine”TargetType = “job”OpSys = “linux”Requirements = (Owner==“thain”)

Machine ClassAd

www.cs.wisc.edu/condor

Three way ClassAdsType = “job”TargetType = “machine”Cmd = “sim.exe”Owner = “thain”Requirements = (OpSys==“linux”) && NearestStorage.HasCMSData

Job ClassAd

Type = “machine”TargetType = “job”OpSys = “linux”Requirements = (Owner==“thain”)NearestStorage = ( Name = “turkey”) && (Type==“Storage”)

Machine ClassAd

Type = “storage”Name = “turkey.cs.wisc.edu”HasCMSData = trueCMSDataPath = /cmsdata”

Storage ClassAd

www.cs.wisc.edu/condor

Policy specification› Run anywhere where data is available

Requirements = (NearestStorage.HasCMSData)

› Run local only Requirements = (NearestStorage.Name == “nestore.bologna”)

› Run local first Requirements = (NearestStorage.HasCMSData) Rank = (NearestStorage.Name == “nestore.bologna” ) ? 10 : 0

› Arbitrarily complex Requirements = ( NearestStorage.Name == “nestore.bologna”)

|| ( ClockHour < 7 ) || ( ClockHour > 18 )

www.cs.wisc.edu/condor

Policies evaluated› INFN local› UW remote› UW stage first› UW local (pre-staged)› INFN local, UW remote› INFN local, UW stage› INFN local, UW local

www.cs.wisc.edu/condor

Completion Time

www.cs.wisc.edu/condor

CPU Efficiency

www.cs.wisc.edu/condor

Conclusions› Locality is good› I/O communities are a natural

structure to expose this locality› Users can use I/O communities to

easily express different job placement policies

www.cs.wisc.edu/condor

Future work› Automation

Configuration of communities Dynamically adjust size as load

dictates› Automation

Selection of movement policy› Automation

www.cs.wisc.edu/condor

For more info› Condor

http://www.cs.wisc.edu/condor› ClassAds

http://www.cs.wisc.edu/condor/classad› PFS

http://www.cs.wisc.edu/condor/pfs› NeST

http://www.nestproject.org

www.cs.wisc.edu/condor

Local only

www.cs.wisc.edu/condor

Remote only

www.cs.wisc.edu/condor

Both local and remote

www.cs.wisc.edu/condor

Grid applications have demanding I/O needs

› Petabytes of data in tape repositories› Scheduling systems have demonstrated

that there are idle CPUs› Some systems

move jobs to data move data to jobs allow job remote access to data

› No one approach is always “best”

www.cs.wisc.edu/condor

Easy come, easy go› In a computation grid, resources

are very dynamic› Programs need rich methods for

finding and claiming resources CPU discovery Device discovery Replica discovery

www.cs.wisc.edu/condor

Bringing it all together

CPU Discovery System

Replica Discovery System

Device Discovery System

Job Agent

Execution siteStorage appliance

Distributed Repository

Short-haul I/O

Long-haul I/O

www.cs.wisc.edu/condor

Conclusions› Locality is good› Balance point between staging data

and accessing it remotely is not static depends on specific attributes of the job

•data size, expected degree of re-reference, etc depends on performance metric

•CPU efficiency or job completion time

www.cs.wisc.edu/condor