Upload
buinguyet
View
235
Download
6
Embed Size (px)
Citation preview
© 2013 IBM Corporation
Nobody Does Java Batch Like
WebSphere Application Server! (And It is All Included.)
Chris Vignola, WebSphere Architecture, IBM
TAW-1443
22 © 2013 IBM Corporation
Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.
44 © 2013 IBM Corporation
Agenda
• Extra value in WAS v8.5
• All about WebSphere Batch
• Introducing JSR 352 (Batch Applications for Java)
55 © 2013 IBM Corporation5
Health Management
Self-ProtectingSelf-Healing
Application Edition
ManagementSelf-Managing
A.1A.2A.3
B.7B.8
Dynamic Clustering
Self-Optimizing
Batch processing
Before
After
Install WebSphere
Virtual Enterprise
Install WebSphere
Compute Grid
Install WAS V8 or earlier
Install WAS V8.5
WAS V8.5: Virtual Enterprise & Compute Grid (Batch)
66 © 2013 IBM Corporation
IBM Compute Grid V8 and IBM WAS V8.5
The IBM WebSphere Java Batch function is provided in two ways today:
IBM WebSphere
Compute Grid
Version 8
IBM WebSphere
Application Server
Version 7 or 8
Operating Systems Supported:AIX, IBM i, Linux, Windows, HP-UX,
Solaris, Linux for System z, z/OS
Add the function ("Augment")
IBM WebSphere
Application Server
Version 8.5
Operating Systems Supported:AIX, IBM i, Linux, Windows, HP-UX,
Solaris, Linux for System z, z/OS
Compute Grid V8 function
incorporated into WAS V8.5Java Batch
Function
Java
Execution
Runtime
Function is identical between the two environments
Compute Grid V8 available for those who have not yet migrated their execution runtimes to WAS V8.5
77 © 2013 IBM Corporation
WebSphere Java Batch
Key Features:
Java Batch programming model
Java Batch container built on WAS QoS
Development and deployment tooling
Batch execution environment
Concurrent OLTP and batch workloads
Enterprise scheduler integration
Parallel processing of batch jobs
Container based checkpoint and restart
Skip record processing
COBOL support on z/OS
WebSphere Application Server v8.5 integrates capabilities from WebSphere
Compute Grid and delivers a complete enterprise level Java batch
processing solution
Compute Grid
capabilities
integrated
into
WAS 8.5
88 © 2013 IBM Corporation
WebSphere Batch – Key Use Cases
Batch Modernization – Migrate from a native batch runtime, typically
developed in programming languages like C, C++, PL/I, and COBOL, to Java.
Highly Parallel Batch Jobs – Execute a single large batch job that is broken
into chunks and executed concurrently across a grid of resources.
Dynamic OLTP & Batch Runtime – Dynamically provision resources for
execution to meet operational goals.
Batch as a Service – Expose business capabilities as a service and leverage
usage accounting features for tracking and chargeback.
Replace Homegrown Batch Frameworks – Eliminate costly proprietary
batch infrastructures and focus development resources on business logic.
Shared business logic across OLTP and Batch – Leverage the proven
WebSphere platform to share logic across both batch and OLTP.
Evolve to a single infrastructure for both OLTP and Batch that enables you to
leverage existing applications and focus resources on business logic
99 © 2013 IBM Corporation
Batch Essentials - Quick Review
1010 © 2013 IBM Corporation
Batch Container Added to the WAS Runtime
At a very high-level, you may think the IBM WebSphere Java Batch function as a
"batch container" operating alongside the other containers of WAS itself:
Container-managed Services
Web Container
Application
Web Modules
Container-managed Services
EJB Container
Application
EJB Modules
Container-managed Services
Batch Container
Batch Applications
WebSphere Application Server Runtime Environment
Batch job dispatching and
management system
Job resiliency services
(skip record, step retry)
Data record read and
write support services
Parallel job management
and execution services
Checkpoint and job
restart services
COBOL module call
services
1111 © 2013 IBM Corporation
Overview of the Management and Execution ModelThis picture illustrates some of the key components of the WebSphere Java
Batch model as provided in Compute Grid V8 and WAS V8.5:
Job
Dispatching
FunctionJob Properties
Declaration File
Job Execution
Endpoint
Batch
Applications
Job Execution
Endpoint
Batch
Applications
Development
Libraries and
Tooling Support
Job
Management
Console1
2
3
4
5
1. Job Management Console (JMC) provides a view into the batch
environment and allows you to submit and manage jobs
2. Job declaration file (xJCL) provides information about the job to be
run, such as the steps, the data input and output streams and the
batch class files to invoke
3. The Job Dispatching function interprets the xJCL, dispatches the
job to the endpoint where the batch application resides, and
provides ability to stop and restart jobs
4. The Execution Endpoint is a WAS server in which the deployed
batch applications run
5. The development libraries and tooling assist in the creation of the
batch applications
A comprehensive Java batch execution platform
Built on the proven Java runtime environment
of WebSphere Application Server
1212 © 2013 IBM Corporation
Batch Job and Batch Job Steps
A batch job consists of one or more steps executed in order specified in xJCL:
xJCL
Properties of the overall job
Job Step 1• Java class
• Input and output declarations
• Other properties of the step
Job Step 2• Java class
• Input and output declarations
• Other properties of the step
Job Step n• Java class
• Input and output declarations
• Other properties of the step
Job The xJCL is submitted through the Job
Management ConsoleInterfaces provided: HTTP browser, command Line, Web Services, RMI
The Job Dispatching function interprets
xJCL and determines which endpoint has
batch application class files deployed
Dispatching Function invokes job and
passes to the endpoint an object
containing all the properties in xJCL
Steps are executed in order, with
conditional step processing if declared
Dispatching Function maintains
awareness of job state
When job ends, job output file accessible
through Job Management Console
1313 © 2013 IBM Corporation
Transactional Checkpoint ProcessingThe batch container provides the ability to checkpoint at intervals based on
either record count or time. The container keeps track of last checkpoint.
Batch Container
Java Batch
Application
xJCL says:
Checkpoint = 5
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Commit
Processing
Last good
checkpoint
persisted
Checkpoint interval (record or
time) specified in the xJCL
This is a function of the batch
container, not your application
code
As checkpoint intervals are
reached, container commits and
records the checkpoint attained
In the event of a failure, job may
be restarted at the last good
checkpoint
Set the checkpoint interval based
on your knowledge of balance
between recoverability and
efficiency
1414 © 2013 IBM Corporation
New Project Type for WebSphere Batch
application
Project and Job Creation Wizards help guide
user through batch programming model
semantics
Editors for xJCL Job Control
Source code wizards for Java code skeleton
Menu choices for
– generating required additional files
– deploying to unit test or
remote Compute Grid runtime environment
– submitting the batch job Job output log
viewer
Samples / Help
Rational Tool Support
Available since RAD v8.0.3 Wizard-driven
xJCL Editor
Unit Test Support
1515 © 2013 IBM Corporation
WAS 8.5 Feature Focus1
1 Also available in WebSphere Compute Grid Version 8.0
1616 © 2013 IBM Corporation
Integration with Enterprise Scheduler FunctionsThe Job Dispatching Function has a Message Driven Bean (MDB) interface. IBM
supplies a program that integrates schedulers with WebSphere Java Batch:
Enterprise SchedulerExample: IBM Tivoli
Workload Scheduler, CA
Workload Automation CA 7,
or BMC Control-M
WSGRID ProgramShell script, BAT file or JCL job
Input Queue
Output Queue
Message Driven
Bean Interface
WSGRID is seen by Scheduler as any other batch job it starts and monitors
WSGRID interacts with Job Dispatching, submitting the job and processing
Java batch job output back to STDOUT or JES Spool if z/OS
WSGRID program stays up for life of job in WebSphere Java Batch
To the Scheduler, WGRID is the Java Batch job ... but behind WSGRID is all
the WebSphere Java Batch function we'll discuss
WebSphere MQ or the integrated
Default Messaging of WAS
1717 © 2013 IBM Corporation
Skip-Record Processing
Provides a container-managed way of tolerating data read or write errors so the
job itself may continue on. Information about data errors may be logged.
Batch Data Stream
Framework
Java Batch
Application
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
Data Record
xJCL tells BDSF:
• How many data read
or write exceptions to
consume
• What exceptions to
consider for skip-
record processing
• Alternatively, what
exceptions to exclude
from skip-record
processing
Objective: allow job to continue if a data read or
write exception occurs in BDSFWhy fail a million-record job just because of one or two read or write
exceptions? Better to complete the job and allow auditors to go back
and investigate the few exceptions.
Skip-Record processing allows BDSF to keep
exception and not surface it to your applicationThis takes burden off your application code to explicitly handle data
read or write exceptions that may occur
A "skip-record listener" may be called so your
code may log information about skipped recordsMore on "batch listeners" coming up
xJCL properties allow you to specify how many
records may be skipped and what exceptions to
include or exclude from consideration
When skip limit is reached, further exceptions are
surfaced to application. That may result in job
failing and going into a restartable stateNormal restart-at-checkpoint would occur
1818 © 2013 IBM Corporation
Retry-Step ProcessingProvides a means of retrying a job step in the event of an exception thrown. If
successful on retry then the job continues and your processing completes.
xJCL tells Container:
• How many step retries may be
attempted
• What exceptions to consider for
retry-step processing
• Alternatively, what exceptions to
exclude from retry-step processing
• Whether to process a delay before
attempting a retry of the step
Objective: retry step in attempt to allow overall job
to continue and complete when an unanticipated
exception is thrown
This is at level higher than skip-record ... this is if
an unhandled exception is thrown when the job
step function is called
Batch container falls back to last good checkpoint
and restarts from there
A "retry-step listener" may be called so you can
perform custom action upon retry-step processingMore on "batch listeners" coming up
xJCL properties allow you to specify how many
retry attempts will be performed and what
exceptions to include or exclude from
consideration
When retry limit is reached, job will go into
restartable stateNormal restart-at-checkpoint would occur
On exception, retry
up to n times
1919 © 2013 IBM Corporation
Batch "Listeners"These are callout points where your customer "listener" code will be called
when key events occur. The callouts are managed by the batch container:
Job Listener• Callouts occur:
Start of the job; Start of each step; End of each step; End of job
• Register your code to container with property in xJCL
• Use this to perform any special setup or cleanup actions at those
points in the lifecycle of a batch job
Retry-Step Listener• Callouts occur:
When the exception is thrown; When the retry is attempted
• Register your listener with code in application createJobStep()
method
• Use this to take action at these points, such as logging
information about the exception and the point in the processing
where it occurred
Skip-Record Listener• Callouts occur:
On skipped read or skipped write operation
• Register your listener with code in application createJobStep()
method
• Use this to take action at these points, such as logging
information about the exception and the record skipped
Job Listener
Retry-Step Listener
Skip-Record Listener
Listeners provide ability
to have your code called
at key points during batch
job execution
2020 © 2013 IBM Corporation
Parallel Job Manager
The Parallel Job Manager (PJM) provides a way to "parameterize" logic so
parallel sub-jobs may act on a slice of the overall batch job data:
One job processing 1M customer records
1 - 100K
100K - 200K
900K - 1M
Ten sub-jobs
acting on a
1/10th slice of
data each
Sub-job
Sub-job
Sub-job
or
Time = 0 Time = 1 Time = 10
Objective is reduction in
overall job completion timeWhich shortens overall batch window if other
jobs are dependent on this job for completion
xJCL specifies whether job is to be
run in parallel, and if so how:• One JVM, multiple threads
• Multiple JVMs
Your "parameterizer" code is called
at start so data range may be
segmented into sub-job slices
Job is submitted, then PJM
dispatches "sub-jobs" to act on
each data range"Parameterizer" code constructs data range query
strings to be used by each sub-job
PJM manages "top-job" and all
subordinate "sub-jobs" to
completion
2121 © 2013 IBM Corporation
COBOL ContainerThe COBOL Container provides a way to call and execute COBOL modules in
the WAS z/OS server address space ... a very efficient way to call COBOL
1. Batch application runs in the WAS
z/OS servant region address space
2. The COBOL container is created as a
separate LE enclave in the address
space
3. COBOL DLLs are accessed using
STEPLIB or LIBPATH
4. COBOL Container code provides the
"glue" between the Java environment
and the native COBOL
5. Java batch code uses supplied class
methods to create the container and
use it
6. Call stubs provide an easy way to call
the COBOL DLL and marshal data
back and forth
7. The call stubs are generated by a
supplied utility that uses COBOL
source to understand data bindings
8. JDBC Type 2 connections created in
the Java batch program may be
shared into the COBOL module in the
COBOL Container
COBOL
Container
Call Code
WebSphere Java
Batch Container
COBOL Container LE Enclave
COBOL Module
Call
Stubs
WAS z/OS Servant Region Address SpaceSeparate LE Enclave from COBOL Container
Call Stub
Generator
IBM Rational
Application Developer
Compiled COBOL LibraryPDSE or USS Directory
1
2
3
4
5
6
7
Lines of code needed to invoke COBOL many times less than other means of calling COBOL from Java
8
2222 © 2013 IBM Corporation
OSGi Batch Applications
• Enables use of OSGi for batch
applications development
• Full batch programming model
available to OSGi framework
• Supports standard and
blueprint bundles
• Enterprise Bundle Archive
deployment
bundle bundle…
.eba
WCG/WAS
WCGJob
2323 © 2013 IBM Corporation
Record Metrics
• Key processing metrics
collected by Batch Container:
‒ Skipped record count
‒ Retry count
‒ Records/second
‒ Total processing time
• Values written to job log at
end of job step.
• Available to batch application
through JobStepContext
object.
Job step
JobStepContext
Job log
…Step ended.N records skipped.M retries.R records/sec.T seconds.
Programmaticaccess to metrics
Batch Containercollects metrics
Metrics reportedin job log
2424 © 2013 IBM Corporation
Group Security for Jobs
• Group level security for jobs
• Users have access to jobs
based on group membership
• Configurable to work both with
and without security roles
• Leverages WAS group security
features
• Job Console (JMC) views
based on group settings
• JMC group filtering to simplify
administration
• Multiple System Programming
Interfaces for customization
job
JGroupA
job
JGroupB
user
UGroupX
user
UGroupY
UGroupX has access to both groups JGroupA and JGroupB
UGroupY has access to JGroupB only
Group membership in console
2525 © 2013 IBM Corporation
Memory Overload Protection for Jobs
• Protection against
over-scheduling jobs to an
application server
• Batch Container monitors job
memory demand against
available JVM heap space
• Prevents Java
OutOfMemoryError
• Automatic real time job memory
estimation with declarative xJCL
override
JVM Heap
free space
job
job job job job
job
Room for next job?
job…
incoming jobs
running jobs
<job name=… [memory=N] … >xJCL:
2626 © 2013 IBM Corporation
SMF Type 120 Subtype 9
• Optional Subtype 9 usage
record1 for each WCG job
• Subtype 9 is rich, including:
‒ zIIP, zAAP, and general purpose
processor time
‒ Times given in CPU seconds and
service units
• Subtype 9 augmented with
additional data for
jobs:
‒ Accounting string
‒ Jobid
‒ Submitter1 z/OS only
job job job…
SMFRepository
SMF 120/9
SMF 120/9
SMF 120/9
…
running jobs
2727 © 2013 IBM Corporation
JSR 352 Batch Applications for the Java Platform
2828 © 2013 IBM Corporation
• Adds new application type (batch) to Java
• Expert Group (operation April 2013-present) : ‒ IBM (Lead) - Credit Suisse‒ Oracle - Clarkson University‒ Redhat - Industry Consultants (2)‒ VMWare
• Specification approved on 8 April 2013.
• Part of Java EE 7 (Reference Implementation is GlassFish)
• Separate Reference Implementation available for Java SE
JSR 352 Summary
2929 © 2013 IBM Corporation
Batch for Java Landscape
• Jobs (JSL) are started through the JobOperator interface.
• A typical job is a series of one or more steps, where each step is comprised of a
reader, processor, and writer.
• Typical job steps read and process a series of input records and produce
output records in a long running loop with periodic checkpoints to support restart.
• Job history is stored in a repository and is available through the JobOperator
interface.
3030 © 2013 IBM Corporation
JSR 352 Comparison to WebSphere Batch
Feature JSR 352 WAS Batch
Transactional Batch Model Yes Yes
Compute Intensive Model (Batchlet) Yes Yes
Partitioned Step Execution Yes Yes
Job and Step Contexts Yes Yes
Job and Step Listeners Yes Yes
Job Specification Language (JSL) Yes Yes
JSL Substitution Yes Yes
Operations Interface Yes Yes
Concurrent Steps Yes NO
Restart Substitution Yes NO
CDI Integration Yes NO
Job Dispatcher NO Yes
Job Logs NO Yes
Distributed Parallelization NO Yes
3131 © 2013 IBM Corporation
Theoretical JSR 352/WebSphere Batch Integration
wBatch
Job
Scheduler
JSR 352
JSL
wBatch
xJCLor
job dispatch
submit
wBatch Endpoint
Common
Job
loggingCommon
Job
Repository
Common Batch Runtime
JSR 352
Job
wBatch
Job
3232 © 2013 IBM Corporation
• JSR 352 Home Page
• JSR 352 Expert Group Proceedings
• JSR 352 Public Wiki
• JSR 352 Public Mailing List
• JSR 352 Issues Mailing List
• JSR 352 Issue Tracker (Bugzilla)
• JSR 352 Source Repository (Git)
JSR 352 Links
3333 © 2013 IBM Corporation
Summary
• WAS v8.5 delivers enhanced value for batch workloads.
• Full WebSphere Compute Grid feature-set now part of WAS
‒ Increases WAS value
‒ Simplifies ownership
• Proven function encourages customer adoption where warranted by
business need.
• JSR 352 delivers industry standardization for Java batch applications.
3434 © 2013 IBM Corporation
We love your Feedback!
Don’t forget to submit your Impact session and speaker feedback!
• Your feedback is very important to us – we use it to improve next year’s
conference
• Go to the Impact 2013 SmartSite (http://impactsmartsite/com):
‒ Use the session ID number to locate the session
‒ Click the “Take Survey” link
‒ Submit your feedback
3535 © 2013 IBM Corporation
3636 © 2013 IBM Corporation
Legal Disclaimer
• © IBM Corporation 2013. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by
you will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs
and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM
Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).
Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your
presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in
your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International
Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and
other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of
others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta
Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration
purposes only.