Resource Management Resource Management Working GroupWorking Group

SSS Quarterly MeetingSSS Quarterly Meeting

November 28, 2001November 28, 2001

Dallas, TxDallas, Tx

Resource Management and Resource Management and Accounting Working GroupAccounting Working Group

• Working group scope and Working group scope and componentscomponents

• Progress madeProgress made

• Current and future issuesCurrent and future issues

• Next stepsNext steps

Working Group ScopeWorking Group Scope

The Resource Management Working Group The Resource Management Working Group encompasses the areas of resource management, encompasses the areas of resource management, scheduling and accounting.scheduling and accounting.

This working group will focus on the following software This working group will focus on the following software components:components:

• Queue ManagerQueue Manager• SchedulerScheduler• Allocation ManagerAllocation Manager• Meta SchedulerMeta SchedulerOur charter will also encompass the following capabilities:Our charter will also encompass the following capabilities:• AccountingAccounting• Usage ReportsUsage Reports

Phase 1 MilestonesPhase 1 Milestones

• 6 months:6 months: Contribute to checkpoint/restart report with Contribute to checkpoint/restart report with regard to scheduling related aspects regard to scheduling related aspects

• 12 months: Establish and release initial resource 12 months: Establish and release initial resource management interface specificationsmanagement interface specifications

• 12 months: Establishment of the CVS repository and 12 months: Establishment of the CVS repository and module structure, agreement on document module structure, agreement on document conventionsconventions

• 12 months: Finalized API for system initiated 12 months: Finalized API for system initiated checkpoint/restart of parallel MPI jobs on Linux systemscheckpoint/restart of parallel MPI jobs on Linux systems

• 18 months: Release v1.0 of the Center’s resource 18 months: Release v1.0 of the Center’s resource management system based on existing open source management system based on existing open source code and the results of the scalability testing.code and the results of the scalability testing.

High Level ProgressHigh Level Progress

• Establishing high level design covering initial Establishing high level design covering initial component functionality and required component functionality and required interfacesinterfaces

• Determining inter-group requirements (GUI, Determining inter-group requirements (GUI, security, IS, process management, etc)security, IS, process management, etc)

• Preparing existing tools (Maui, Silver, QBank) Preparing existing tools (Maui, Silver, QBank) for use within SSSfor use within SSS

• Creating infrastructure within which to Creating infrastructure within which to develop and test RM deliverablesdevelop and test RM deliverables

• Creating infrastructure within which to Creating infrastructure within which to develop and test intra- and inter-group develop and test intra- and inter-group interfacesinterfaces

Proposed Component Proposed Component ArchitectureArchitecture

QueueManager

AllocationManager

Collector

MetaScheduler

Scheduler

NodeManager

ProcessManager

SecuritySystem

InformationService

DiscoveryService

Color Color KeyKey

Working GroupWorking Group

Resource Resource Management and Management and AccountingAccounting

Execution Execution Management and Management and MonitoringMonitoring

Node Config and Node Config and InfrastructureInfrastructure

Component Interaction Component Interaction DiagramDiagramJob submitted to Queue Job submitted to Queue ManagerManager

UserInterface

CollectorMetaScheduler

Queue Manager

Allocation Manager

Scheduler ProcessManager

21

34

65

7

98

1011

Component Interaction Component Interaction TraceTraceJob submitted to Queue Job submitted to Queue ManagerManager

1. A user submits a job to the Queue Manager2. The Queue Manager does a sanity balance check with the Bank3. The Queue Manager notifies the Scheduler that a new job has arrived4. The Scheduler queries node and job status until job can run5. A bank reservation is made with the Allocation Manager6. The Scheduler requests the Queue Manager to run the job7. The Queue Manager passes job control to the Process Manager8. The Process Manager notifies Queue Manager of job completion9. The Queue Manager notifies Scheduler of job completion10. A bank withdrawal is made with the Allocation Manager11. The user is notified of job completion

Component Interaction Component Interaction Diagram Diagram Job submitted to Meta Job submitted to Meta SchedulerScheduler

UserInterface

CollectorMetaScheduler

Queue Manager

Allocation Manager

Scheduler ProcessManager

21

34

65

87

109

11

1312

1415

Component Interaction Component Interaction TraceTraceJob submitted to Meta Job submitted to Meta SchedulerScheduler

1. A user submits a job to the Meta Scheduler2. The Meta Scheduler contacts Schedulers to determine which systems

could run the job the soonest3. The Schedulers request quotes from Allocation Banks to determine

which systems would run the job for the lowest cost4. A Scheduler reservation is created for the job on the resource

providing the best service -- this reservation can be moved or improved upon until the job is staged

5. The job is staged and queued at the system where it is to run6. The Queue Manager notifies the Scheduler that a new job has arrived7. The Scheduler queries node and job status until job can run8. A bank reservation is made with the Allocation Manager9. The Scheduler requests the Queue Manager to run the job10. The Queue Manager passes job control to the Process Manager11. The Process Manager notifies Queue Manager of job completion12. The Queue Manager notifies Scheduler of job completion13. A bank withdrawal is made with the Allocation Manager14. The Scheduler notifies the Meta Scheduler of job completion15. The user is notified of job completion

Design/Interface ProgressDesign/Interface Progress

• Initial high level RMS architecture definedInitial high level RMS architecture defined• Resource management dictionary created Resource management dictionary created

defining objects within resource management defining objects within resource management ‘world’‘world’

• Object ‘tokens’ declared for major objectsObject ‘tokens’ declared for major objects• Component functional interfaces identifiedComponent functional interfaces identified• Initial XML request/response syntax proposedInitial XML request/response syntax proposed• Prototypes being constructed to test Prototypes being constructed to test

communication protocolscommunication protocols• Initial detailed extra-group component Initial detailed extra-group component

requirements document createdrequirements document created

Local Scheduler RationaleLocal Scheduler Rationale

Local interfaces with majority of inter and intra Local interfaces with majority of inter and intra RM componentsRM components

Establish test platform from which interfaces can Establish test platform from which interfaces can be testedbe tested

Leverage existing capabilities to accelerate SSS Leverage existing capabilities to accelerate SSS development development

Establish infrastructure within which scheduling Establish infrastructure within which scheduling and metascheduling services and capabilities and metascheduling services and capabilities can be developedcan be developed

Establish ‘driver’ to evaluate other resource Establish ‘driver’ to evaluate other resource management componentsmanagement components

Local Scheduler ProgressLocal Scheduler Progress

• Baseline scheduler established (Maui 3.2) for SSS Baseline scheduler established (Maui 3.2) for SSS scheduling services integrating production and scheduling services integrating production and development capabilitiesdevelopment capabilities

• Prototype interface enabling XML communication Prototype interface enabling XML communication with queue manager, metascheduler, and node with queue manager, metascheduler, and node managermanager

• Extended QoS infrastructure integratedExtended QoS infrastructure integrated• Extended Job prioritization infrastructure Extended Job prioritization infrastructure

integratedintegrated• Prototype created for object-oriented data accessPrototype created for object-oriented data access• Advanced metascheduling interface integratedAdvanced metascheduling interface integrated

Meta Scheduler ProgressMeta Scheduler Progress

• Initial distribution packaging created to Initial distribution packaging created to allow collaborative developmentallow collaborative development

• Documentation enhanced and extendedDocumentation enhanced and extended

• Prototype XML scheduler to Prototype XML scheduler to metascheduler query interface metascheduler query interface developeddeveloped

• Initial fault tolerance framework Initial fault tolerance framework designeddesigned

Queue Manager DesignQueue Manager Design

• Established need for unified queue manager Established need for unified queue manager design common to Scheduler and design common to Scheduler and MetaschedulerMetascheduler

• Queue manager will interface directly with Queue manager will interface directly with Process managerProcess manager

• In process of refining the queue manager In process of refining the queue manager taskstasks

• Queue manager will provide an interface to Queue manager will provide an interface to obtain information about any job regardless of obtain information about any job regardless of job state including completed jobs (i.e. it will job state including completed jobs (i.e. it will maintain a job information archive)maintain a job information archive)

Allocation Manager ProgressAllocation Manager Progress

• QBank placed under revision controlQBank placed under revision control

• Java prototype created which sends Java prototype created which sends requests in XMLrequests in XML

• Experimenting with protocol Experimenting with protocol frameworks (simple octet-counting, frameworks (simple octet-counting, octet-stuffing, SOAP, BEEP)octet-stuffing, SOAP, BEEP)

Next Steps (In Progress)Next Steps (In Progress)

• Software Lifecycle InfrastructureSoftware Lifecycle Infrastructure– Online intra-RM schedule and dependencies documentOnline intra-RM schedule and dependencies document– Detailed extra-RM working group requirements Detailed extra-RM working group requirements – Coordinate creation of component level regression test Coordinate creation of component level regression test

suite suite – Bug tracking systems activated (used to track internal Bug tracking systems activated (used to track internal

defects and development plans)defects and development plans)• InterfaceInterface

– Produce validating intra-RM XML schema Produce validating intra-RM XML schema – Produce prototype RM components communicating in Produce prototype RM components communicating in

initial protocolinitial protocol• Feature EnhancementsFeature Enhancements

– Contribution to checkpoint/restart reportContribution to checkpoint/restart report– Creation of queue manager prototypeCreation of queue manager prototype

Next Steps (6 Months)Next Steps (6 Months)

• UsabilityUsability– GUI-server interface, GUI format, security determined and GUI-server interface, GUI format, security determined and

prototypes createdprototypes created– Documentation of initial meta job constraints/features and Documentation of initial meta job constraints/features and

specification languagespecification language• Inter-group CollaborationInter-group Collaboration

– Creation of early scheduler XML implementation for use as RM Creation of early scheduler XML implementation for use as RM driverdriver

– Development of initial dynamic job scheduler-queue manager Development of initial dynamic job scheduler-queue manager interfaceinterface

– Extension of RM specifications/requirement documentExtension of RM specifications/requirement document– Extension of internal component test infrastructureExtension of internal component test infrastructure– Determination of ‘best practices’ in documentation maintenanceDetermination of ‘best practices’ in documentation maintenance– Evaluation and adoption of web project management and Evaluation and adoption of web project management and

collaboration toolscollaboration tools– Creation of prototype queue manager with scheduler/task Creation of prototype queue manager with scheduler/task

manager interfaces manager interfaces

Next Steps (6 Months)Next Steps (6 Months)

• Fault ToleranceFault Tolerance– Enhance metascheduler to ‘survive’ local daemon failureEnhance metascheduler to ‘survive’ local daemon failure– Enhancement of threaded scheduling interface. Enhancement of threaded scheduling interface. – Development of threaded metascheduling interface.Development of threaded metascheduling interface.

• Resource OptimizationResource Optimization– Development of local optimization features of meta workloadDevelopment of local optimization features of meta workload

• Feature EnhancementsFeature Enhancements– Creation of resource manager extension features.Creation of resource manager extension features.– Development of direct metascheduler to queue manager Development of direct metascheduler to queue manager

staging roadmap.staging roadmap.• InterfacesInterfaces

– Specification of ‘best guess’ security infrastructure and Specification of ‘best guess’ security infrastructure and evaluation of impact on system internals and communication evaluation of impact on system internals and communication protocolsprotocols

Next Steps (1 year)Next Steps (1 year)

• Software Lifecycle InfrastructureSoftware Lifecycle Infrastructure– Create multi-component regression testsCreate multi-component regression tests– Generate ‘alpha’ package of scheduling, metascheduling, and Generate ‘alpha’ package of scheduling, metascheduling, and

allocation management packages.allocation management packages.• InterfacesInterfaces

– Development of functional XML interfaces for all componentsDevelopment of functional XML interfaces for all components– Early adoption of security infrastructureEarly adoption of security infrastructure– Creation of optional information service interfacesCreation of optional information service interfaces– Admin and end-user GUI’s proposed to enable use of new Admin and end-user GUI’s proposed to enable use of new

functionalityfunctionality• Inter-group CollaborationInter-group Collaboration

– Enhanced suspend/resume and checkpoint/restart features Enhanced suspend/resume and checkpoint/restart features with detailed roadmap specified for all remaining with detailed roadmap specified for all remaining suspend/resume and checkpoint restart deliverablessuspend/resume and checkpoint restart deliverables

Current IssuesCurrent Issues

• Should there be an enveloping protocol Should there be an enveloping protocol framework which handles framing (where the framework which handles framing (where the XML document begins and ends), authentication, XML document begins and ends), authentication, multiplexing, streaming data, etc? (should we multiplexing, streaming data, etc? (should we look at something like BEEP, or start from scratch look at something like BEEP, or start from scratch and invent something of our own?)and invent something of our own?)

• The queue manager/collector to node/process The queue manager/collector to node/process manager functionality and data interface requires manager functionality and data interface requires further refinement.further refinement.

• Queue manager/collector and node/process Queue manager/collector and node/process manager development schedules must be manager development schedules must be determined and coordinated.determined and coordinated.

IssuesIssues

• Continued effort is required to complete Continued effort is required to complete an ‘intra-RM’ XML schema to handle initial an ‘intra-RM’ XML schema to handle initial RMS interaction needs. Boundaries RMS interaction needs. Boundaries between internal ‘intra-RM’ and global between internal ‘intra-RM’ and global XML schema is needed.XML schema is needed.

• Understanding of open source Understanding of open source requirements (I.e. can software be requirements (I.e. can software be included in SSS distribution that requires included in SSS distribution that requires registration and usage agreements) registration and usage agreements)

Inter-Group IssuesInter-Group Issues

• Need for coordination of resource Need for coordination of resource management system across working groups management system across working groups – so that the pieces all function together – so that the pieces all function together properly and no part is overlooked. Need to properly and no part is overlooked. Need to coordinate schedules for delivery of RMWG-coordinate schedules for delivery of RMWG-dependent non-RMWG components.dependent non-RMWG components.

• Early vendor/industry collaborations (We’d Early vendor/industry collaborations (We’d better do this while it can still influence our better do this while it can still influence our design. Need to talk to decision makers and design. Need to talk to decision makers and develop business plans)develop business plans)

Inter-group IssuesInter-group Issues

•Information service – should we rather Information service – should we rather be looking for something existing? (i.e. be looking for something existing? (i.e. MDS2)MDS2)

•Need to solidify SSS-wide standards for Need to solidify SSS-wide standards for packaging, revision control, packaging, revision control, documentation content, format, and documentation content, format, and packaging, problem tracking, … and packaging, problem tracking, … and establish mechanisms and places to establish mechanisms and places to home them.home them.

•Creation of regression and integration Creation of regression and integration test suite (w/ Validation and Testing WG – test suite (w/ Validation and Testing WG – we need this from an early stage)we need this from an early stage)

ConclusionsConclusions

• Questions…Questions…