Upload
jagsekaran
View
234
Download
0
Embed Size (px)
Citation preview
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
1/24
Job Scheduling: History and Evolution
November 17, 2010Welkin Associates
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
2/24
Job Scheduler Terminology and Components
The Scheduler Study Overview
Brief History of Job Scheduling
Job Scheduler Functions
Job Scheduler Architectures
Evaluation of Job Schedulers and Vendors
Q/A
Agenda
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
3/24
TerminologyJob: An executable unit. Can be a systems command, a script, anapplication program, or a collection of executable units. Jobs can have oneor more steps, each containing an executable unit.Job-Dependencies: Logical events or conditions that must be satisfied inorder for jobs to be scheduled for execution. Predecessor and Successor.Job-Flow: A collection of two or more jobs related by job-dependencies.Schedule: The current active set of scheduled jobs and job-flows ready tobe released for execution and monitored by the Master Scheduler.
ComponentsMaster Scheduler: Central software component that defines, modifies,stores job and job-flow definitions, dependencies, security information,events, and schedules. Provides automatic submission of executables,monitors status of all jobs and job-flows, provides alerts, controls starting,stopping, restarting of jobs and job-flows. Creates Audit trail of all events.
Agent:A program that resides on each server that is controlled by theMaster. The agent monitors jobs on its server, communicates with theMaster about the status of jobs, and executes commands relayed by theMaster.GUI Console: Console display for Administrators and Operators.Relational Database: The central repository used by the Master to store all
job, job-flow, dependencies, security information, schedules and audit trails.
Job Scheduler Terminology and Components
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
4/24
Job Scheduler Terminology and Components (Cont.)
Agent
Master Scheduler
RDMS
Operator GUI
console
Administrator
Agent
Agent
Logs
Administrator
GUI console
Operator
Job-Flows
Job-Flows
Job-Flows
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
5/24
OPUS: Legacy Job Scheduler used for Satellite dataprocessing
Developed by Space Telescope Science Institute (STScI ) in 1994 for HSTVery fast and efficient pipeline processor. Powerful job-flow features andGUI console
Adopted and customized for NOAA as a Job Scheduler on AIX p-series formigration from z/OS Mainframe (2005)OPUS requires replacement by a COTS Scheduler:
STScI no longer supports OPUSOPUS GUI written in Java has compatibility issues with latest version ofJava
The Scheduler Study
Evaluated using two in-house Schedulers (NDE and STARS)Evaluated COTS Schedulers and vendors.
Scheduler Study Overview
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
6/24
In the beginning there were 80 column Hollerith cards
The term batch was derived from stacks of punched cards
The first true Job Schedulers were people and card readers
Therefore, originally Job Scheduling known as batch processing
1964 IBM OS/360 JCL presented first job dependency functions
Brief History of Job Scheduling
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
7/24
Early Mainframe era: Batch processing using card decks
Later Mainframe era: JCL on IBM Mainframes.
Jobs: Dependencies/Priorities/Queues/TOD-Calendar/Alerts/Reruns
Evolution of 3rdparty vendor Mainframe Job SchedulersIBM Tivoli Workload Scheduler (TWS)
CA 7 Mainframe Scheduler
BMC Control-M
ASG Zeke Scheduler
Unix systems: crons
Today: Job Scheduling has greatly matured, with a number of
software vendors that have developed and marketed their Job
Scheduler products for over 25 years
Brief History of Job Scheduling (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
8/24
Workload Automation Maturity Pyramid
Brief History of Job Scheduling (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
9/24
Gartner: The future automation model will be IT Workload Automation Broker (ITWAB)
Brief History of Job Scheduling (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
10/24
Scheduler Functions:
Create and edit jobs, job-flows, users, groups, schedules
Calendar scheduling
File-Watchers (Trigger file events)
Job parameters within job-flows available
Job Queues, Classes, Priority
Virtual resources to control jobs and files
High Availability / Failover capabilityReliable and fast Scheduler database with open schema
Scheduler commands to start, stop, suspend, resume, and restart jobs
Alerts generated for job errors and sent to Operators (GUI console, email, cell)
Reporting tools
GUI console:
Windows, Web-based, Unix/Linux
Flexible and configurable for Administrators and Operators, easy to use
Graphical display of jobs and job-flows. Click and drag highly preferable
Detailed display of job-flow and job status, dependencies, errors/alerts
Job Scheduler Functions
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
11/24
Architecture: Master-Agent vs. Peer-to-Peer
Job Scheduler Architectures
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
12/24
Architecture: MasterAgent
Job Scheduler Architectures (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
13/24
EMA Criteria and Methodology for Evaluating Job Schedulers
Research/Advisory Firms Job Scheduler Evaluations
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
14/24
EMA Radar Report for Job Schedulers 01/2010
Research/Advisory Firms Job Scheduler Evaluations (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
15/24
Forrester List of Evaluated Job Schedulers
November 2009 Market Overview: Workload Automation, Q3 2009
Workload Automation Vendors
Research/Advisory Firms Job Scheduler Evaluations (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
16/24
Gartner Job Scheduler Magic Quadrant Report 04/2009
The Gartner Magic Quadrant is copyrighted
(04/28/2009) by Gartner, Inc., and is reused with
permission. The Magic Quadrant is a graphicalrepresentation of a marketplace at and for a specific
time period. It depicts Gartnersanalysis of how certain
vendors measure against criteria for that marketplace,
as defined by Gartner. Gartner does not endorse any
vendor, product or service depicted in the Magic
Quadrant, and does not advise technology users to
select only those vendors placed in the Leaders
quadrant. The Magic Quadrant is intended solely as a
research tool, and is not meant to be a specific guide
to action. Gartner disclaims all warranties, express or
implied, with respect to this research, including any
warranties of merchantability or fitness for a particular
purpose.
Source: Gartner, Inc.
Magic Quadrant for Job Scheduling
28 April 2009, Report Id Number: G00166953
Milind Govekar, Gartner Analyst
Research/Advisory Firms Job Scheduler Evaluations (Cont.)
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
17/24
Q/A
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
18/24
Additional Slides for reference
18
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
19/24
Workload Automation Example
19
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
20/24
Future Growth of WLA and Job Scheduling
Forrester: Forecast for Growth of WLA and Job Scheduling
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
21/24
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
22/24
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product NameOpen Source,
Freeware,COTS
Corporate Practical Solutions Bqplus COTS
Cronless.com cronless Open Source
David G. Walker php JobScheduler Free Ware
Exact JobBOSS COTS
Exspans Automan COTS
Flux Corp. Flux COTS
Grand Logic JobServer COTS
Help/Systems Robot/SCHEDULE Enterprise COTS
Honico Batchman COTS
Hux Software HQ Scheduler COTSIBM
Tivoli Workload Automation SuiteCOTS
IndependIT BICsuite Scheduler Open Source
ISE EnterpriseSCHEDULE COTS
JOX IT JOX COTS
LXI LXI Scheduler COTS
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
23/24
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product NameOpen Source,
Freeware,COTS
MJTNet Macro Scheduler COTS
MVP Systems JAMS COTS
Neteject.com VisualCron COTS
Network Automation Automate7 COTS
Nguyen Anthuan SetCronJob Free Ware
OAR Team OAR Open Source
Open Systems Mgmt Cosbatch COTS
OpsWise Automation Center COTS
ORSYP Dollar Universe COTS
OW2 Consortium ProActive Parallel Suite Open Source
Platform Computing Load Sharing Facility (LSF) COTS
PTC Software PTC Scheduler COTS
Redwood Software Cronacle COTS
ROC Software Maestro Scheduler COTS
Runtime Design
AutomationNetwork Computer (NC) COTS
SMA OpCon/xps COTS
SoftTree Tech 24x7 Scheduler COTS
8/10/2019 NESDIS Scheduler Presentation Welkin LBL V0.4
24/24
Evaluation of Job Schedulers and Vendors
Vendor Scheduler Product NameOpen Source,
Freeware,COTS
Sourceforge Jcrontab Open Source
Stonebranch Indesca COTS
Sun Microsystems N1 Grid Engine COTS
Sypsoft Job2Do COTSTenth Generation Job Scheduler for Windows COTS
Terracotta Quartz Enterprise Job Scheduler Open Source
The Coens Group Jobstream Open Source
UC4 Workload Automation Suite COTS
Univa UD Grid MP COTS
University of Warwick TITAN Grid Scheduler Open Source
University of Wisconsin Condor Open Source
Vinzant Global ECS (GECS) COTS
Xi Software GNUbatch Open Source
XuiS Software EnterpriseSchedule COTS