Upload
raffael-schmid
View
1.852
Download
0
Embed Size (px)
DESCRIPTION
Talk at Java User Group Switzerland in Zürich
Citation preview
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
2013 © Trivadis
Spring Batch Lessons Learned out of a real life banking system.
Michael Beer
Senior Consultant
Raffael Schmid
Consultant
December 3rd, 2013
Lessons Learned out of a real life banking system
1
2013 © Trivadis
About us
Michael Beer
Senior Consultant
December 3rd, 2013
Lessons Learned out of a real life banking system
2
Trivadis since 2001
design and development of
web based applications
part of the Trivadis APM
team
Trivadis since 2010
design and development of
web based applications
interested in performance
related things on the JVM
Raffael Schmid
Consultant
2013 © Trivadis
AGENDA
December 3rd, 2013
Lessons Learned out of a real life banking system
3
Initial
position
Requirements
and Topics
Lessons
learned Conclusion
2013 © Trivadis
Initial position
December 3rd, 2013
Lessons Learned out of a real life banking system
4
Initial
position
Requirements
and Topics
Lessons
learned Conclusion
2013 © Trivadis
common usage
Initial and target system context
Automatic load out of thirdparty
systems
multiple times per day
export / import as CSV file
Manual load by user
at irregular time intervals
Migration load due to thirdparty
system decommissioning
run once
different volumes
December 3rd, 2013
Lessons Learned out of a real life banking system
5
Web Interface
Rule Engine
File Interface
Thirdparty system
Thirdparty system
2013 © Trivadis
Why we used Spring Batch?
It was already part of the technology stack within the customers
environment.
There were no real (free) alternatives at that time (or at least we didn‘t
know any).
Exceptional permit was needed for Spring Integration because it was
shipped with Spring Batch Admin.
December 3rd, 2013
Lessons Learned out of a real life banking system
6
2013 © Trivadis
Terminology of Spring Batch jobs
December 3rd, 2013
Lessons Learned out of a real life banking system
7
Job
Job Instance
Job Execution
Step
Step Execution
*
*
*
*
*
Job Execution
Context
Step Execution
Context
* Job Parameter
Tasklet *
2013 © Trivadis
Requirements and Topics
December 3rd, 2013
Lessons Learned out of a real life banking system
8
Initial
position
Requirements
and Topics
Lessons
learned Conclusion
2013 © Trivadis
Requirement: Performance
parallelized on multiple threads
order of execution matters
number of threads can be changed before or after job execution
December 3rd, 2013
Lessons Learned out of a real life banking system
9
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Reprocess failed items
rerunning a job processes failed items (only)
process state of item therefore needs to be maintained
December 3rd, 2013
Lessons Learned out of a real life banking system
10
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Summary mail
write a detailed execution log that contains
Exceptions
messages out of the Rule Engine
aggregate to summary
Number of errors, warnings, etc.
worst status level
send summary to submitter and operator
December 3rd, 2013
Lessons Learned out of a real life banking system
11
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Gather detailed job information
rerun failed single item in trace mode
collect diagnostic information
e.g. out of the Rule Engine
December 3rd, 2013
Lessons Learned out of a real life banking system
12
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Trigger jobs
periodically
either fixed delay or fixed rate
on event
e.g. new data arrived in database
December 3rd, 2013
Lessons Learned out of a real life banking system
13
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Control load
prevent too many jobs in parallel
conditions might prevent job execution
requests should prevent system shutdown
December 3rd, 2013
Lessons Learned out of a real life banking system
14
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Deactivate jobs
deactivate job execution
set job execution on hold
December 3rd, 2013
Lessons Learned out of a real life banking system
15
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirement: Inter-job dependencies
finished jobs might trigger dependencies
December 3rd, 2013
Lessons Learned out of a real life banking system
16
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
2013 © Trivadis
Requirements grouped into five different topics
December 3rd, 2013
Lessons Learned out of a real life banking system
17
Control load Deactivate jobs Inter-job
dependencies Trigger jobs
Reprocess
failed items Gather detailed
job information Summary mail Performance
Sourcing
Job Control
Error
Handling Monitoring & Tracing Partitioning
2013 © Trivadis
Lessons learned
December 3rd, 2013
Lessons Learned out of a real life banking system
18
Initial
position
Requirements
and Topics
Lessons
learned Conclusion
2013 © Trivadis
Topics we will cover
December 3rd, 2013
Lessons Learned out of a real life banking system
19
Error
Handling
Monitoring
& Tracing
Job Control Sourcing Partitioning
depends on
depends on
depends on
firing trigger
supervising
2013 © Trivadis
Sourcing
December 3rd, 2013
Lessons Learned out of a real life banking system
20
Error
Handling
Monitoring
& Tracing
Job Control Sourcing Partitioning
depends on
depends on
depends on
firing trigger
supervising
2013 © Trivadis
Load data into staging table
December 3rd, 2013
Lessons Learned out of a real life banking system
21
CAT_ID | PERSON_ID | AMOUNT | CURRENCY | VALID_FORM | LAST_CHANGE|
310 | 2908 | 200000 | USD | 03.12.2013 | 02.02.2013 |
310 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
311 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
PARTITION LINE TYPE STATE FIELD01 FIELD02 FIELD03 FIELD04
PT_000122 171 D 02 441 1804 400000 EUR
PT_000122 172 D 02 441 1002 221000 EUR
PT_000123 1 H 01 CAT_ID PERSON_ID AMOUNT CURRENCY
PT_000123 2 D 01 310 2908 200000 USD
PT_000123 3 D 01 310 1608 100000 CHF
PT_000123 4 D 01 311 1608 100000 CHF
Sourcing
2013 © Trivadis
Allows partitioning, single record execution
Partitioning out of the box
December 3rd, 2013
Lessons Learned out of a real life banking system
22
PARTITION LINE TYPE STATE FIELD01 FIELD02 FIELD03 FIELD04
PT_000123 1 H 01 CAT_ID PERSON_ID AMOUNT CURRENCY
PT_000123 2 D 01 310 2908 200000 USD
PT_000123 3 D 01 310 1410 100000 CHF
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND LINE BETWEEN 0 AND 99;
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND STATE = ‘01’
Reexcution of failed records
Execution of single record
SELECT * FROM LOAD
WHERE PARTITION =‘PT_000123’
AND LINE = 8
Sourcing
2013 © Trivadis
Job Control
December 3rd, 2013
Lessons Learned out of a real life banking system
23
Error
Handling
Monitoring
& Tracing
Job Control Sourcing Partitioning
depends on
depends on
depends on
firing trigger
supervising
2013 © Trivadis
Trigger
Simplest way of triggering a job
December 3rd, 2013
Lessons Learned out of a real life banking system
24
0 0/5 * * * ? Spring Batch
(SimpleJobLauncher) job-requests
Job Control
<int:inbound-channel-adapter method="create" channel="job-requests"
ref="requestFactory">
<int:poller cron="0 0/5 * * * ?" />
</int:inbound-channel-adapter>
Request Factory
<int:inbound-channel-adapter method="create" channel="job-requests"
ref="requestFactory">
<int:poller fixed-rate="300000"/>
</int:inbound-channel-adapter>
Or even simpler (PeriodicTrigger)
Define polling channel adapter (CronTrigger)
2013 © Trivadis
Persist job launch requests into database
December 3rd, 2013
Lessons Learned out of a real life banking system
25
job launch
request job-
requests
Data import job
(database)
Periodic jobs 0 0/5 * * * ?
*JobQueueMessageHandler job queue
*JobLaunchingMessageHandler
update status
save job dependencies
Guard
verification *JobLauncher
Job Control
persist
2013 © Trivadis
Partitioning
December 3rd, 2013
Lessons Learned out of a real life banking system
26
Error
Handling
Monitoring
& Tracing
Job Control Sourcing Partitioning
depends on
depends on
depends on
firing trigger
supervising
2013 © Trivadis
Performance
December 3rd, 2013
Lessons Learned out of a real life banking system
27
Type Local/Remote Description
Multi-threaded Step Local A step is multithreaded (TaskExecutor)
Parallel Steps Local Executes steps in parallel using multithreading
Partitioning Step Local
Remote Partitions data and splits up processing
Remote Chunking Remote Distributed chunk processing to remote nodes
Partitioning
2013 © Trivadis
Partitioning overview
December 3rd, 2013
Lessons Learned out of a real life banking system
28
Job
Master
Step
Step Slave
Slave
Slave
Slave
Slave
Partitioning
2013 © Trivadis
Partitioning detail
December 3rd, 2013
Lessons Learned out of a real life banking system
29
StepScopes
Master
Slave [1]
Slave [2]
Slave [3]
Slave [4]
Slave [0]
Execution
Context [1]
Execution
Context [2]
Execution
Context [3]
Execution
Context [4]
Execution
Context [0]
rowId % 5
Partitioning
2013 © Trivadis
Characteristics of data
There might be dependencies between records. The order of execution
matters at some point.
December 3rd, 2013
Lessons Learned out of a real life banking system
30
CAT_ID | PERSON_ID | AMOUNT | CURRENCY | VALID_FORM | LAST_CHANGE|
310 | 2908 | 200000 | USD | 03.12.2013 | 02.02.2013 |
310 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
311 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
312 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
313 | 1608 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
310 | 1410 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
390 | 1108 | 100000 | CHF | 03.12.2013 | 02.01.2013 |
310 | 2908 | 500000 | USD | 04.12.2013 | 03.02.2013 |
Partitioning
Insert
Update
Read Write
Process / Write
2013 © Trivadis
Partitioning detail – Spring Batch Admin
December 3rd, 2013
Lessons Learned out of a real life banking system
31
Partitioning
2013 © Trivadis
Performance – Read / Write / Process
December 3rd, 2013
Lessons Learned out of a real life banking system
32
Open
Transaction
Begin Step
Commit
Transaction ItemReader
Item read()
ItemProcessor
Process(Item)
ItemWriter
Write(List<Item>)
Finish Step
Rollback
Transaction
RuntimeException RuntimeException RuntimeException
Partitioning
For each item For each item
2013 © Trivadis
Performance
December 3rd, 2013
Lessons Learned out of a real life banking system
33
No error Error on each item
• ~ 22 sec
• 0 Rollbacks
• ~ 50 sec
• 1‘100 Rollbacks
• 5‘500 Filter
• 1‘000 Write Skips
Partitioning
2013 © Trivadis
Error Handling
December 3rd, 2013
Lessons Learned out of a real life banking system
34
Error
Handling
Monitoring
& Tracing
Job Control Sourcing Partitioning
depends on
depends on
depends on
firing trigger
supervising
2013 © Trivadis
Restartability
December 3rd, 2013
Lessons Learned out of a real life banking system
35
Job 1
Instance 1
Execution 1
Job 1
Instance 1
Execution 2
read
new/error read
new/error
Error
Handling
regular start restart
2013 © Trivadis
Restartability configuration
December 3rd, 2013
Lessons Learned out of a real life banking system
36
Error
Handling
Job
Step
initialization
Step
process
Step
finalize
initial (failed) execution
Job
Step
initialization
Step
process
Step
finalize
restarted execution
Execution
Context
cleanup
serialized
context
2013 © Trivadis
Item based error handling
December 3rd, 2013
Lessons Learned out of a real life banking system
37
Feature When? What? Where?
Skip For nonfatal
exceptions
Keeps processing for an
incorrect item Chunk-oriented step
Retry For transient
exceptions
Makes new attemps on an
operation
Chunk-oriented step,
application code
Error
Handling
2013 © Trivadis
Monitoring
December 3rd, 2013
Lessons Learned out of a real life banking system
38
Error
Handling
Monitoring
& Tracing
Job Control Sourcing Partitioning
depends on
depends on
depends on
firing trigger
supervising
2013 © Trivadis
Monitoring – Spring Batch Admin
December 3rd, 2013
Lessons Learned out of a real life banking system
39
Monitoring &
Tracing
2013 © Trivadis
Monitoring – logging to a database
December 3rd, 2013
Lessons Learned out of a real life banking system
40
Step
Read
Process
Write
LoggerService @Transactional(propagation =
Propagation.REQUIRES_NEW)
2 calls on error
Listener
Before
OnSkip
[WARN]
OnError
[ERROR]
After
[INFO]
Monitoring &
Tracing
2013 © Trivadis
Monitoring - Mail
December 3rd, 2013
Lessons Learned out of a real life banking system
41
Step
ReadProcessWrite
Job
FAILED
Error Mail OK Mail
Monitoring &
Tracing
2013 © Trivadis
Tracing – Explain plan
December 3rd, 2013
Lessons Learned out of a real life banking system
42
Step
ReadProcessWrite
Partitioner
1 Record
Read
1 Record
Process
Debug mode
Step
…
Writer
does nothing
Operator
explicit job start
Monitoring &
Tracing
2013 © Trivadis
Monitoring / Tracing summary
December 3rd, 2013
Lessons Learned out of a real life banking system
43
OK/ERROR Mail
ERROR
UI
Log
Spring Batch
Admin
UI
Explain plan User Operator
Monitoring &
Tracing
2013 © Trivadis
Conclusion
December 3rd, 2013
Lessons Learned out of a real life banking system
44
Initial
position
Requirements
and Topics
Lessons
learned Conclusion
2013 © Trivadis
Challenges – Error Handling
December 3rd, 2013
Lessons Learned out of a real life banking system
45
Fire and forget
Commit until first error
- Skip
- Retry
Depending on the characteristic of data
Rollback
- Whole job
- Partition
- Chunk
Job flow
Deciders
Operator takes care about the errors
2013 © Trivadis
Why we still use Spring Batch
Out of 155 SE tests in the JSR-352 TCK, Spring Batch 3.0 (milestone 1)
passes 70.
The terminology stays more or less the same: Job, Step, Chunk, Item,
ItemProcessor, JobInstance, JobExecution.
December 3rd, 2013
Lessons Learned out of a real life banking system
46
Spring Batch JSR 352 Comments
Tasklet Batchlet
ItemReader / ItemStream ItemReader JSR-352‘s ItemReader includes Spring Batchs
ItemStream capabilities
ItemWriter / ItemStream ItemWriter JSR-352‘s ItemReader includes Spring Batchs
ItemStream capabilities
JobExecutionListener JobListener
StepExecutionListener StepListener
2013 © Trivadis
Suitable for different applications
It’s very easy to customize existing parts at your needs. Different level of
customization is possible:
December 3rd, 2013
Lessons Learned out of a real life banking system
47
Spring Batch supports implementing reliable batch jobs with small effort.
In few cases even no Java code is necessary.
configuration partial
implementation job control
2013 © Trivadis
Questions and answers ...
2013 © Trivadis
BASEL BERN BRUGG LAUSANNE ZUERICH DUESSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
Michael Beer
+41 58 459 51 90
Raffael Schmid
+41 58 459 52 34
December 3rd, 2013
Lessons Learned out of a real life banking system
48