EGEE is a project funded by the European Union under contract INFSO-RI-508833
Practical approaches to Grid workload management in the
EGEE project
Massimo SgaravattoINFN Padova
On behalf of the EGEE JRA1 IT-CZ cluster
CHEP 2004
www.eu-egee.org
Chep 2004 - 2
EGEE project
• EGEE project Aim: build a consistent, robust and secure Grid infrastructure Focus first on two pilot applications areas (HENP, Biomedical
applications)• But the goal is to take other researchers in academia and industry
• Middleware activity (JRA1) Re-engineer Grid software to provide production quality
middleware Evolution towards emerging standards, based on Service
Oriented Architectures Taking into account application requirements and production/
deployment/ management needs
• See talk #247 (E. Laure)
Chep 2004 - 3
Workload management
• Grid workload and resource management is one of the key Grid middleware functionality How to efficiently schedule a big number of different data-intensive jobs,
submitted by a distributed community of users, to a Grid encompassing many and heterogeneous resources
• Progress was made in various projects with different integrated software solutions: DataGrid Workload Management System Condor EuroGrid-Unicore resource broker …
• Still a lot to do Scalability, reliability Identification and handling of failures originating from different software
layers, and possibly from 'foreign' Grid system and resources Distributed (hierarchical ?) super-scheduling Proper semantics of resource information collection and distribution (push,
pull, index, cache, refresh) …
Chep 2004 - 4
Workload Management System
• Provision of Grid Workload Management System services assigned to the “EGEE JRA1 Italian Czech cluster” CESNET Datamat S.p.A. INFN
• Architecture of the EGEE WMS designed and being implemented Taking into account feedback and requirements from reference applications
and deployment/production/management activities Taking into account previous experiences from other Grid projects (in
particular the DataGrid WMS) Set of Grid services
• Workload Manager (WM)• Computing Element (CE): Resource access• Logging & Bookkeeping (L&B)• Job Provevance (JP)• Grid Accounting service
Interoperating among them and with other EGEE Grid Services
Chep 2004 - 6
Workload Manager
Job managementrequests (submission, cancellation) expressed
via a Job DescriptionLanguage (JDL)
Chep 2004 - 7
Workload Manager
Keeps submission requests
Requests are kept for a while
if no matchingresources available
Chep 2004 - 8
Workload Manager
Repository of resource information
available to matchmaker
Updated via notifications and/or active
polling on sources
Chep 2004 - 9
Workload Manager
Finds an appropriateCE for each submission
request, taking into account job requests and preferences, Grid status, utilization policies
on resources
Chep 2004 - 10
Scheduling policies
• Different possible policies Eager scheduling: a job is bound to a resource as soon as possible
• Job is then forwarded to that CE, where very likely it will end up in a queue
Lazy scheduling: job held by the WM until a resource becomes available• Job then forwarded to that CE for immediate execution
• WM architecture able to accommodate both models (and the intermediate solutions) Eager scheduling: matching a job against multiple resources Lazy scheduling: matching a resource against multiple jobs
• Needed to better investigate strengths and weaknesses of different policies in different scenarios Evaluation of relevant metrics, covering both resource utilization and user
satisfaction
Chep 2004 - 11
Computing Element
• Service representing a computing resource• Main functionality: job management
Run jobs Cancel jobs Suspend and resume jobs Provide info on “quality of service”
• How many resources match the job requirements ?• What is the estimated time to have the job starting its execution ?• …
…• Used by the WM or by any other client (e.g. end-user)• CE architecture accommodated to support both push and pull model
Push model: the job is pushed to the CE by the WM Pull model: the CE asks the WM for jobs
• These two models are somewhat mirrored in the resource information flow In order to 'pull' a job a resource must choose where to 'push' information
about itself
Chep 2004 - 12
CECECECE
LSFLSFLSFLSF
Worker NodesWorker NodesWorker NodesWorker Nodes
PBSPBSPBSPBS ????
MonMonMonMon
ClientClient
WEB WEB
CE Architecture
JobSubmitJobAssess
JobKillJobSuspendJobResume
JobGetStatus
Web serviceaccepting jobmanagement
requests
Chep 2004 - 13
CECECECE
LSFLSFLSFLSF
Worker NodesWorker NodesWorker NodesWorker Nodes
PBSPBSPBSPBS ????
MonMonMonMon
ClientClient
WEB WEB
CE Architecture
Notifications
Job requests
Async. notificationsabout job/CE events
Job requests (forCE working in pullmode)
Chep 2004 - 14
Logging & Bookkeeping
• Collects and manages job-related events (e.g. submission, suitable CE found, start of execution, …) from the WMS components
• Processes these events to give a higher level view on job states
• Both job states and raw data available to users Also via Web Service interface
• Possible to subscribe to receive notifications on particular job state changes
• LB event trail can be analyzed to identify problems with resources ("black holes", unusual failure rates, etc).
• See poster #419 for more details
Chep 2004 - 15
Job Provenance
• Keeps track of definition of submitted jobs, execution conditions and job life cycle for a long time Job life logs (JDL, timestamps, jobids, …) Executable and input/output files Execution environment (OS, installed software version, …) Custom data provided by user
• Used for Debugging Post-portem analysis Comparison of job executions in an evolving environment
• Service components Primary Storage Server
• Keeps data in the most compact and economic form Index Servers
• Configured to support a set of queryable attributes
• See poster # 419 for more details
Chep 2004 - 16
Grid Accounting
• Accumulates information about the usage of Grid resources by users / groups (e.g. VOs)
• To be used To track resource usage To discover abuses (and help avoiding them)
• Also possible to charge users for the resources they have used
• Allows implementation of submission policies based on resource usage Exchange market among Grid users and Grid resource owners,
which should result in market equilibrium Load balancing on the Grid
Chep 2004 - 17
Accounting architecture
Accounting
ComputingElement
StorageElement
Resource metering:
getting infoabout resource
usage
Resource metering:
getting infoabout resource
usage
Chep 2004 - 18
Accounting architecture
Accounting
ComputingElement
StorageElement
Reports aboutresource usage per user / VO/
resource
Chep 2004 - 19
Accounting architecture
Accounting
ComputingElement
StorageElement
Resourcepricing
Resource owner
Chep 2004 - 20
Accounting architecture
Accounting
ComputingElement
StorageElement
Resourcepricing
Resource owner
Costcomputation
Chep 2004 - 21
Status
• Workload Manager, Logging & Bookkeeping, Grid Accounting software inherited by DataGrid WMS software Being revised and complemented according to the new architecture
• E.g. Information Supermarket, TaskQueue new developments• Web services interfaces
First implementation already deployed in the EGEE GLITE prototype testbed
• Computing Element New fresh developments CEMon prototype already implemented
• Job Provenance New component being implemented