Fujitsu Laboratories of Europe © 2003 A Broker/Scheduler Architecture for Grid Services Dr. David Snelling Fujitsu Laboratories of Europe Open Issues in

Fujitsu Laboratories of Europe © 2003Fujitsu Laboratories of Europe © 2003

QuickTime™ and aGIF decompressor

are needed to see this picture.

A Broker/Scheduler Architecture A Broker/Scheduler Architecture for Grid Servicesfor Grid Services

Dr. David SnellingDr. David Snelling

Fujitsu Laboratories of EuropeFujitsu Laboratories of Europe

Open Issues in Grid SchedulingOpen Issues in Grid Scheduling

October 21-22, 2003October 21-22, 2003

How to Build a Schprokerer




What is a Schprokerer?What is a Schprokerer?

The Function of Optimizing Workload on the Grid.The Function of Optimizing Workload on the Grid.

It is not Controlling or Monitoring that Workload.It is not Controlling or Monitoring that Workload.Not the Unicore NJS or GRAM.

Does Include:Does Include:Bids, Optimization, SLAs, etc...

What is Workload?What is Workload?Resource usage over time

“Workload Profile”

Not Necessarily WorkflowNot Necessarily WorkflowSee sidebar.




Workflow SidebarWorkflow Sidebar

What is Workflow?What is Workflow?A description workload consisting of more than one activity.

Activity = task | job | process | functionA “net” to describe the dependencies between activities.

DAGs, DFGs, Pert Charts, BPDs, ...

Categories of WorkflowCategories of WorkflowTask or Job (partial) sequencesData/Task/Message Flow

Workflow Description TypesWorkflow Description TypesStatic vs. Dynamic (changing)Persistent vs. Dynamic (on demand)Explicit (DAGs etc.) vs. Implicit (message sets)Stateful vs. Stateless




QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture.

Workflow Sidebar ContinuedWorkflow Sidebar Continued

Extended FunctionsExtended FunctionsLoops and ConditionalsError management

Workload Exertion TypesWorkload Exertion TypesDeploymentInvocation

EntitiesEntitiesJobsProcesses or Services

Other workflowsPeopleTokens

Implicit or explicit on dependency arcs.

From BPMN © BPMI




Functions - 1Functions - 1

Resource PublicationResource PublicationResource AbstractionResource Abstraction

Compute # of atoms to processor time on machine X.

Resource ReservationResource ReservationIssue of TicketsIssue of TicketsGeneration of Bids for Resource UseGeneration of Bids for Resource Use

Cost + QoS

User IncarnationUser IncarnationAuthorization checking

NegotiationNegotiationSLAs, WS-Agreement, ...





Gather Performance InformationGather Performance Information

Gather Adherence InformationGather Adherence InformationWere SLAs Honored?

Gather Load InformationGather Load InformationLocally and Grid wide.

Make Commitments on behalf of other Make Commitments on behalf of other SchprokerersSchprokerers

Optimize Workload versus Policies Optimize Workload versus Policies A scheduler for example.





Translate Information Models Translate Information Models Ontology for resource description

Convert “Grid Currency” between VOsConvert “Grid Currency” between VOs

Decide to Migrate Workload Decide to Migrate Workload Compute/Data workload and network workload

Not actually migrate the workload

Decide to Preempt Workload Decide to Preempt Workload Not actually preempt the workload

Locate Necessary ResourcesLocate Necessary Resources





Aggregate Workflow Task BidsAggregate Workflow Task BidsE.g. Bidding for complete Workflows

Predicting Actual Workload of a WL Predicting Actual Workload of a WL descriptiondescription

E.g. Workflow loop, conditionals, etc.

Topology checkingTopology checkingData not at the end of a thin pipe.

Prediction of Resource LoadingPrediction of Resource Loading




OGSA Perspective - IOGSA Perspective - I




OGSA Perspective - IIOGSA Perspective - II




OGSA Perspective - IIIOGSA Perspective - III




Architectural AssumptionArchitectural Assumption

Use the WS-Agreement ApproachUse the WS-Agreement Approach

Distributed Scheduling with Local Distributed Scheduling with Local InformationInformation

From the Grid perspective, we can only “plan”

Not Central scheduling based on Not Central scheduling based on distributed informationdistributed information

The Grid cannot control all the resources therefore we can’t “schedule”.




Class RelationshipsClass Relationships

WorkloadSource

“Demand” sideInitiate AgreementsConsign Work

Workload

Describes the work

WorkloadDescription

Describes resource requirementsTerm in agreement

WS-Agreement

Manifest the contract

WorkloadManagement

“Supply” sideProvide load info.

InfoService

Publish information

OntologyService

Translation of resource description

WorkloadOptimizer

Implement workloadmanagement policy

Broker

“Supply” vs “Demand”Policy Optimization

Scheduler

Collective, Temporal Optimization

Dynamic

Runtime workloadoptimization




CRC Analysis - 1CRC Analysis - 1

WorkloadSource: “Demand” resourcesWorkloadSource: “Demand” resourcesInitiate Agreements

Consign workload to ResourceManager

Create workload

Workload: The workWorkload: The workFully describe the work to be done

Provide WorkloadDescriptions

Subtypes:

Workflow, Job, Task, WS-deployment, WS-invocation





WorkloadDescriptions: Resource request WorkloadDescriptions: Resource request Define the (abstract) resource requirements of a workload

Part of an Agreement initiation

WS-Agreement: ContractWS-Agreement: ContractEncapsulate agreed terms of usage contract

Define contract between WorkloadSource and ResourceManager

ResourceManager: “Supplier” of resourcesResourceManager: “Supplier” of resourcesAccess control and management of resources

Supply load information etc. to InformationService

Notify GridMonitor of workload deltas etc.





InformationService: Global Information SourceInformationService: Global Information SourceMaintain and publish information

Gather data from:

ResourceManager

LoadPredictor

GridMonitor

GridSpies

Filter information via ResourceOntologyMapper

ResourceOntologyMapper: TranslationResourceOntologyMapper: TranslationTranslate representations of resources

Provide to InformationService only?





WorkloadOptimizer: Meet policy requirementsWorkloadOptimizer: Meet policy requirementsImplement workload management policy

Create/Negotiate Agreements with WorkloadSources

Check with AuthorizationService

Subtypes:

Broker: Supply vs. Demand policy agreements• Check with the GridBank (exchange rates only)

Scheduler: Collective, temporal optimization

DynamicOptimizer: Runtime workload adaptation





Resource Publication - InformationServiceResource Publication - InformationServiceResource Abstraction - BrokerResource Abstraction - Broker

Compute # of atoms to processor time on machine X.

Resource Reservation - ResourceManagerResource Reservation - ResourceManagerIssue of Tickets - ResourceManagerIssue of Tickets - ResourceManagerGeneration of Bids for Resource Use - ResourceManagerGeneration of Bids for Resource Use - ResourceManager

Cost + QoS

User Incarnation - AuthorizationServiceUser Incarnation - AuthorizationServiceAuthorization checking

Negotiation - WorkLoadOptimizer and WorkloadSourceNegotiation - WorkLoadOptimizer and WorkloadSourceSLAs, WS-Agreement, ...





Gather Performance Information - GridMonitorGather Performance Information - GridMonitor

Gather Adherence Information - GridSpiesGather Adherence Information - GridSpiesWere SLAs Honored?

Gather Load Information - GridMonitorGather Load Information - GridMonitorLocally and Grid wide.

Make Commitments on behalf of other Schprokerers - Make Commitments on behalf of other Schprokerers - BrokerBroker

Optimize Workload versus Policies - WorkloadOptimizerOptimize Workload versus Policies - WorkloadOptimizerA scheduler for example.





Translate Information Models - Translate Information Models - ResourceOntologyMapperResourceOntologyMapper

Ontology for resource description

Convert “Grid Currency” between VOs - GridBankConvert “Grid Currency” between VOs - GridBankDecide to Migrate Workload - Decide to Migrate Workload - DynamicWorkloadOptimizerDynamicWorkloadOptimizer

Compute/Data workload and network workloadNot actually migrate the workload

Decide to Preempt Workload - Decide to Preempt Workload - DynamicWorkloadOptimizerDynamicWorkloadOptimizer

Not actually preempt the workload

Locate Necessary Resources - InformationServiceLocate Necessary Resources - InformationService





Aggregate Workflow Task Bids - SchedulerAggregate Workflow Task Bids - SchedulerE.g. Bidding for complete Workflows

Predicting Actual Workload of a WL Predicting Actual Workload of a WL description - Schedulerdescription - Scheduler

E.g. Workflow loop, conditionals, etc.

Topology checking - SchedulerTopology checking - SchedulerData not at the end of a thin pipe.

Prediction of Resource Loading - Prediction of Resource Loading - LoadPredictorLoadPredictor

Documents

Fujitsu Laboratories of Europe © 2003 A Broker/Scheduler Architecture for Grid Services Dr. David Snelling Fujitsu Laboratories of Europe Open Issues in