34
INFSO-RI-508833 ttp://gridinitiative.ncit.pub.ro Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services Corina Stratan [email protected]

Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

  • Upload
    lethia

  • View
    38

  • Download
    1

Embed Size (px)

DESCRIPTION

Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services. Corina Stratan [email protected]. Outline. Service Oriented Architecture – Introduction Web Services – Standards and protocols Condor Condor BirdBath – Web Service interface for Condor. - PowerPoint PPT Presentation

Citation preview

Page 1: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

INFSO-RI-508833 http://gridinitiative.ncit.pub.ro

Service Oriented Architectures - Introduction.Managing Grid Resources with Condor Web Services

Corina [email protected]

Page 2: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 2

Enabling Grids for E-sciencE

INFSO-RI-508833

Outline

• Service Oriented Architecture – Introduction• Web Services – Standards and protocols• Condor• Condor BirdBath – Web Service interface for Condor

Page 3: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 3

Enabling Grids for E-sciencE

INFSO-RI-508833

SOA – Service Oriented Architectures

• Current requirements in the IT industry: – Collaboration– Sharing of data and resources– Structuring large applications in small blocks that can be re-used

(services)• SOA definition from OASIS:

A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.

• Service Oriented Architecture - components:– Service Providers– Service Requestors– Service Brokers

Page 4: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 4

Enabling Grids for E-sciencE

INFSO-RI-508833

SOA Characteristics

• Interoperability among different platforms and programming languages

• The functional blocks are encapsulated in components that function as services

• Separation of the interface from the implementation• For complex applications, the control of the execution

flow (workflow) is separated from the services• The services can be added/removed dynamically• The correspondence between interfaces and

implementation is done through configuration files and can be adapted

Page 5: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 5

Enabling Grids for E-sciencE

INFSO-RI-508833

Outline

• Service Oriented Architecture – Introduction• Web Services – Standards and protocols• Condor• Condor BirdBath – Web Service interface for Condor

Page 6: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 6

Enabling Grids for E-sciencE

INFSO-RI-508833

Web Services – Brief Overview

• Provide an implementation for the service oriented architecture

• Software components that provide services over the Internet and communicate through standardized XML messages

• A newer form of Remote Procedure Call, with the advantage of standardization

• Designed to be used by applications, not by humans• Self-describing (public interfaces specified with the aid

of WSDL)• Can be published in “Yellow Pages”–like registries

Page 7: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 7

Enabling Grids for E-sciencE

INFSO-RI-508833

Web Services Protocol Stack

Service Transport (HTTP, FTP, …)

XML Messanging (XML-RPC, SOAP)

Service Description (WSDL)

Service Discovery (UDDI)

Transports messages between applications

Encodes messages in the XML formatDescribes the public

interface of the service

Common registry for web services

Page 8: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 8

Enabling Grids for E-sciencE

INFSO-RI-508833

Web Services Scenario

Web Services Registry

ServiceProvider

ServiceRequestor

WSDL

WSDL

SOAP

1

2

3

1 - Service published to the registry

2 – The requestor obtains the public interface of the service

3 – The requestor invokes the service and receives the result

Page 9: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 9

Enabling Grids for E-sciencE

INFSO-RI-508833

Software Platforms for Web Services

• 1998 – 1999: Microsoft starts developing the SOAP standard

• 2000: IBM extends the SOAP specifications (SOAP 1.1)• 2001: the development of the SOAP and WSDL

standards is coordinated by W3C• Software platforms:

– Microsoft: web services support included in the .NET platform– IBM: developed the IBM-SOAP platform, which was

subsequently acquired by Apache and became Apache SOAP– Apache SOAP was followed by Apache Axis– IBM WebSphere contains a web services engine

Page 10: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 10

Enabling Grids for E-sciencE

INFSO-RI-508833

Axis – Apache Extensible Integration System

• Open-source framework for web services programming• Based on the JAX-RPC/JAX-WS APIs• Includes a simple stand-alone web services server• Can be integrated in application servers like Apache Tomcat• Provides tools for generating WSDL descriptions from Java

classes, and for generating Java classes from WSDL descriptions (java2wsdl, wsdl2java)

Page 11: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 11

Enabling Grids for E-sciencE

INFSO-RI-508833

Why Web Services?

• Standardization• Interoperability • Open standards and protocols• Based on HTTP: no problems with the firewalls• Loosely-coupled components – approach specific to

distributed computing

Page 12: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 12

Enabling Grids for E-sciencE

INFSO-RI-508833

Outline

• Service Oriented Architecture – Introduction• Web Services – Standards and protocols• Condor• Condor BirdBath – Web Service interface for Condor

Page 13: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 13

Enabling Grids for E-sciencE

INFSO-RI-508833

Condor - Introduction

• CONDOR: workload management system for computational intensive jobs (High Throughput Computing)

• Availabe since 1984• Functionality is similar with the classical batch

queuing systems (PBS, LSF, ...)• + New concepts: opportunistic computing, resource

classification

Page 14: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 14

Enabling Grids for E-sciencE

INFSO-RI-508833

Resource Management in Condor

• Opportunistic computing: Condor can utilize non-dedicated machines, taking advantage of the time when the owner does not work on them

• The owner of each machine can establish the policy for running jobs on the machine, e.g.:– If the keyboard/mouse have not been used for 15min– During the nights or the week-ends– Only jobs that require less than x MB of RAM

• Condor motto: Leave the owner in control, regardless of the cost.

Page 15: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 15

Enabling Grids for E-sciencE

INFSO-RI-508833

Checkpoints

• The jobs’ memory images can be saved periodically in checkpoint files

• Better fault tolerance• Checkpoints can be used to move a job on another

machine without restarting it from the beginning• The checkpoint files may be stored on the execution

machine or on a dedicated server

Page 16: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 16

Enabling Grids for E-sciencE

INFSO-RI-508833

Resource classification - ClassAds

• Classified Ads = mechanism that allows a machine to publish the available resources and a job submitter to publish the requirements for execution

• Typical attributes for a machine: no. of CPUs, CPU types and benchmark results, the amount of available memory/disk space, the restrictions for running jobs

• Typical attributes for a job: the needed architecture and operating system, the minimum/maximum amount of memory, the estimated execution time

Page 17: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 17

Enabling Grids for E-sciencE

INFSO-RI-508833

Machine ClassAd - ExampleName = "wn1.rogrid.pub.ro"Machine = "wn1.rogrid.pub.ro"CpuBusy = ((LoadAvg - CondorLoadAvg) >= 0.500000)COLLECTOR_HOST_STRING = "lcfg.rogrid.pub.ro"CondorVersion = "$CondorVersion: 6.6.10 Jun 13 2005 $"VirtualMachineID = 1Disk = 6431976LoadAvg = 0.330000Memory = 496Cpus = 1StartdIpAddr = "<141.85.99.135:47385>"Arch = "INTEL"OpSys = "LINUX"KFlops = 280272TotalLoadAvg = 0.330000TotalCondorLoadAvg = 0.000000State = "Unclaimed"Start = TRUERequirements = START…

Page 18: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 18

Enabling Grids for E-sciencE

INFSO-RI-508833

Job ClassAd - ExampleMyType = "Job"ClusterId = 1511Owner = "cmssoft"ExitBySignal = FALSECmd =

"/home/cmssoft/.globus/.gass_cache/local/md5/b7/5f0d6443e430b2e8b56324a66524aa/md5/8d/d25385ef07f36cf8cb05e2290bbcc1/data"

WantCheckpoint = FALSEOut =

"/home/cmssoft/.globus/job/tier2b.cacr.caltech.edu/1027.1121440922/stdout"

Err = "/home/cmssoft/.globus/job/tier2b.cacr.caltech.edu/1027.1121440922/stderr"

ShouldTransferFiles = "NO"DiskUsage = 8Requirements = (OpSys == "LINUX" && Arch == "INTEL") && (Disk >=

DiskUsage) && ((Memory * 1024) >= ImageSize) && (TARGET.FileSystemDomain == MY.FileSystemDomain)

Args = "/raid1/OSG-app/cmssoft /raid1/OSG-app/cmssoft/cms /raid1/OSG-app/cmssoft/cmsi OSCAR_3_9_5 CIT_CMS_PG slc3_ia32_gcc323 install_cms_project_1"

Page 19: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 19

Enabling Grids for E-sciencE

INFSO-RI-508833

Matchmaking

Matchmaker

Agent Resource

1. Jo

b Clas

sAd

1. Machine ClassAd2. Matchmaking algorithm

3. Notification

4. Resource request

Page 20: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 20

Enabling Grids for E-sciencE

INFSO-RI-508833

The Condor Pool

• Condor Pool = cluster of machines + the jobs executing on them

• Machines in a Condor pool:– central manager - unique– execute machine– submit machine

• A machine can have multiple roles

Page 21: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 21

Enabling Grids for E-sciencE

INFSO-RI-508833

Condor Components

• Condor Master: manages all the Condor daemons that are running on a machine

• Condor Startd: runs on the execute machines; starts the jobs and ensures that the owner’s policy is respected

• Condor Schedd: runs on submit machines; keeps a queue with the submitted jobs and publishes their ClassAds

• Condor Collector: collects information about all the resources from the pool

• Condor Negotiator: executes the matchmaking algorithm, taking into account the users’ priorities

Page 22: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 22

Enabling Grids for E-sciencE

INFSO-RI-508833

Condor Universes

• Specify the execution environment for jobs• Possible universes:

– Standard– Vanilla– MPI– PVM– Globus– Java

Page 23: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 23

Enabling Grids for E-sciencE

INFSO-RI-508833

Standard & Vanilla

• Standard – typical universe, Vanilla - “minimal” universe

• Standard: jobs running in Standard universe can be checkpointed and migrated, but the programs must be link-edited with a Condor library

• Vanilla: any executable program can be run in Vanilla universe, but checkpointing and migration are not possible

Page 24: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 24

Enabling Grids for E-sciencE

INFSO-RI-508833

Condor Shadow

• Process that is started on the submit machine, in parallel with the actual execution of the job on the execute machine

• Used to access the environment variables and the local files

• Each system call is redirected to the shadow • Note: because of the shadow processes, even a submit

machine may become heavily loaded

Page 25: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 25

Enabling Grids for E-sciencE

INFSO-RI-508833

Job execution (command line)

• Create a job description file• Use the condor_submit command:

condor@lcfg:~/condor_stuff$ condor_submit calcule.cmdSubmitting job(s).Logging submit event(s).1 job(s) submitted to cluster 14.

Page 26: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 26

Enabling Grids for E-sciencE

INFSO-RI-508833

Job Description File - Exampleexecutable = calcule.condoruniverse = standard

output = calcule.outerror = calcule.err

log = calcule.log

arguments =

should_transfer_files = NO

queue 1

Page 27: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 27

Enabling Grids for E-sciencE

INFSO-RI-508833

The condor_status commandcondor@lcfg:~$ condor_status

Name OpSys Arch State Activity LoadAv Mem ActvtyTime

lcfg.rogrid.p LINUX INTEL Unclaimed Idle 0.780 496 0+00:36:31wn1.rogrid.pu LINUX INTEL Unclaimed Idle 0.120 496 0+00:18:30

Machines Owner Claimed Unclaimed Matched Preempting

INTEL/LINUX 2 0 0 2 0 0

Total 2 0 0 2 0 0

[monalisa@tier2b monalisa]$ condor_status -direct tier2b.cacr.caltech.edu

Name OpSys Arch State Activity LoadAv Mem ActvtyTime

[email protected] LINUX INTEL Owner Idle 0.400 973 4+13:29:[email protected] LINUX INTEL Owner Idle 0.000 973 4+13:29:[email protected] LINUX INTEL Owner Idle 0.000 973 4+13:29:[email protected] LINUX INTEL Owner Idle 0.000 973 4+13:29:00

Machines Owner Claimed Unclaimed Matched Preempting

INTEL/LINUX 4 4 0 0 0 0

Total 4 4 0 0 0

Page 28: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 28

Enabling Grids for E-sciencE

INFSO-RI-508833

The condor_q command[monalisa@tier2b monalisa]$ condor_q

-- Submitter: tier2b.cacr.caltech.edu : <192.168.0.254:46908> : tier2b.cacr.caltech.edu

ID OWNER SUBMITTED RUN_TIME ST PRI SIZE CMD 637.0 uscms01 7/8 16:19 0+00:22:07 I 0 0.0 sleep

57 826.0 uscms01 7/11 14:06 0+00:29:20 I 0 0.3 tcsh -c

id1511.0 cmssoft 7/15 08:22 3+21:02:38 R 0 37.0 data

/raid1/OSG-ap1610.0 sdss 7/18 12:31 0+16:52:48 R 0 9.3

runRemoteJob.sh 031703.0 sdss 7/18 15:04 0+14:18:50 R 0 9.4

runRemoteJob.sh 04

5 jobs; 2 idle, 3 running, 0 held

Page 29: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 29

Enabling Grids for E-sciencE

INFSO-RI-508833

Other Condor Features

• Condor DAGMan: support for executing multiple jobs with dependencies (workflows)

• Condor Quill: builds and supports a mirror database of a Condor queue• Condor-G: task broker that can be used as a front-end to a grid

A

C D

E

B

Page 30: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 30

Enabling Grids for E-sciencE

INFSO-RI-508833

Outline

• Service Oriented Architecture – Introduction• Web Services – Standards and protocols• Condor• Condor BirdBath – Web Service interface for Condor

Page 31: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 31

Enabling Grids for E-sciencE

INFSO-RI-508833

Condor & Web Services

• Condor BirdBath: adds web service interfaces to the Condor daemons

• Web services allow external applications to interact with Condor – job delegation model

• Implementation based on Apache Axis• To use BirdBath for a Condor server:

– Modify the Condor configuration file• To create a BirdBath client:

– Set the environment variables– Use the wsdl2java tool to generate helper classes– Write the client code

Page 32: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 32

Enabling Grids for E-sciencE

INFSO-RI-508833

Services provided by BirdBath

• For the Condor scheduler (schedd daemon):– Job submission– Data transfer and management– Job Monitoring– Queue management operations– Transactions

• For the Condor collector:– Querying the available classAds

Page 33: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 33

Enabling Grids for E-sciencE

INFSO-RI-508833

Resources

• http://cs.wisc.edu/condor - Condor web site• http://cs.wisc.edu/condor/birdbath - Condor BirdBath

page• http://www-128.ibm.com/developerworks/edu/gr-dw-gr-

wscondor.html?S_TACT=105AGX07&S_CMP=HP - Condor Web Services tutorial

• http://www.oreillynet.com/lpt/a//webservices/2002/02/12/webservicefaqs.html - Top Ten FAQs for Web Services

• http://www.xmethods.com – collection of publicly available web services

Page 34: Service Oriented Architectures - Introduction. Managing Grid Resources with Condor Web Services

GridInitiative, University “Politehnica” of Bucharest July 2007 34

Enabling Grids for E-sciencE

INFSO-RI-508833

Proposal for GridInitiative Activities

• Install Condor• Enable the Condor WebServices interface• Create and test a client for Condor WebServices• Model job workflows with Condor DAGMan