Placement of BPM runtime components in an SOA environment

© 2012 IBM Corporation

IBM Software Group Placement of BPM runtime components in an SOA environment

Presenter: Kim Clark Authors: Kim Clark ([email protected]) Brian Petrini ([email protected]) Version 1.6

IBM Software Group


Related article

Please note, the following article has been published that covers the concepts in the first half of this presentation: “The placement of BPM runtime components in an SOA” http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html

25/01/14

IBM Software Group


How complicated can a simple sequence of boxes be?

Process

IBM Software Group


Common questions arising when considering BPM and SOA

https://collaboration.opengroup.org/projects/soa-ref-arch

•  Should all requests go through business process management (BPM)? •  Is BPM above or below the enterprise service bus (ESB)? •  Are processes also services? •  Should BPM handle page navigation? •  Should BPM perform integration?

IBM Software Group


Consumers

Are business processes just another type of consumer?

Services (Exposure)

Operational Systems (Applications & Data)

Process Runtime (BPM as a consumer)

Other consumer

Service Providers Service C

onsumers

Other consumer

Could/should we remove the business process layer altogether? Removes temptation to assert that all requests should go through business process layer.

IBM Software Group


Sequences of steps Where could they occur / Where should they occur?

Service Exposure


Integration

Consumers

Business Process

IBM Software Group


People based processes

Service Exposure


Integration

Consumers

Business Process

IBM Software Group


What makes a business process suitable for BPM?

Performing the process provides value to the business The process contains individually business relevant steps Business relevant data flows through the process The process follows a relatively structured path The steps within the process are performed by multiple roles/teams. The process changes over time as a result of changes in the business

Create Account

Capture Details

Send Documents

Sign Documents

Activate Account

Customer

Call centre

Back office

Notify Customer

New revenue!

Existing customer

New customer

-- - --- -- --- -- -- --- -- ------ -- --- -- -

-- - --- -- --- -- -- --- -- ------ -- --- -- -

-- - --- -- --- -- -- --- -- ------ -- --- -- -

-- - --- -- --- -- -- --- -- ------ -- --- -- - -- - --- -- --- -- -- --- -- ------ -- --- -- -

-- - --- -- --- -- -- --- -- ------ -- --- -- -

IBM Software Group


Consumers

BPM as a consumer of services

Services (Exposure)


Business Process (Orchestration)

Process Runtime (as a consumer)


onsumers

A business process orchestrates (consumes) services in order to automate steps in the process

Human task

System task

Exposed service

Back end system

Misleading representation

of an interaction

Interaction

Key

IBM Software Group


Consumers

Positioning of the integrated BPM UI

Services (Exposure)



Process Runtime (consumer)


onsumers

Typically an internal BPM UI is provided to enable users to view and work their tasks.

Integrated BPM UI

IBM Software Group


Consumers

Introducing an external BPM user interface

Services (Exposure)




Integrated BPM UI


onsumers

Often some, or all of the human tasks need to be served up by an external custom portal due to the specific requirements of the graphical user interface.

retrieveTask() updateTask() completeTask()

Independent BPM UI

IBM Software Group


Systems acting as both providers and consumers simultaneously

§  The layered architecture is a logical model, not a physical deployment

§  Components may appear more than once if they perform different roles

§  Many operational systems are both providers and consumers

Services (Exposure)


System A (as a provider)


onsumers

Consumers System B (as a consumer)

System A (as a consumer)

System B (as a provider)

IBM Software Group


Consumers

The process runtime acting as a provider of services

Services (Exposure)



Process Runtime (as a consumer)

Integrated BPM UI

Process Runtime (as a provider)

Independent BPM UI


onsumers

In reality, the business process runtime also becomes a service provider in this situation providing services for external systems to read and manipulate tasks. So the BPM component now lives in two places on the diagram, each with a specific role. BPM is both above and below the service layer

retrieveTask() updateTask() completeTask()

Inte

rnal

inte

ract

ion

with

in

busi

ness

pro

cess

runt

ime

API

IBM Software Group


Pros and Cons of fully decoupling the process API Pros: •  Enables hiding of routing, versioning, security models etc. •  Becomes a standardize process capability, independent of technology and

implementation •  Standardized monitoring and controlling service level agreements, throttling

traffic •  Standardized governance: Enforcing policies for things such as encryption,

deprecation. •  Optionally enables standardization of data model, but see Cons below. Cons: •  Significant up front work before the first project can use the API •  Results in another layer to be maintained, more skillsets involved operationally. •  May render the standard API product documentation less effective, and new

documentation may need to be created and maintained. Particularly true if a new data model is introduced.

Recommendations •  Only consider decoupling where multiple highly independent consumers are

present, each with strong reliance on well defined service level agreements. •  Focus on requirements around traffic management, and security. •  Avoid replacing/translating the APIs data model. This is a lot of work both up

front and ongoing for very little gain.

IBM Software Group


Page navigation

Service Exposure


Integration

Consumers

Business Process

IBM Software Group


Separating Process, Page Navigation and Page Design

Loan Requester

Loan Approver

System

Approve Loan

Process Loan

Capture Loan Details

Capture Personal Details

Capture Financial Details

First name

Second name

Date of birth

Personal details

etc.

What’s wrong with this process model?

Multiple steps in the same human swimlane of a BPMN diagram suggest there is an issue with the granularity of the process modeling. It implies they could have been performed by the same person in one go, so they are not really separate steps. We should not model this low level page navigation in the BPMN diagram.

IBM Software Group


Separating Process, Page Navigation and Page Design

Loan Requester

Loan Approver

Capture Loan Application

System

Approve Loan

Process Loan

Process


Loan Details

Personal Details

Financial Details

First name

Second name

Date of birth

Personal details

etc.

Page navigation

Page

Page navigation should be modeled separately from the BPMN swimlane diagram.

IBM Software Group


Separation of concerns when creating a custom UI for BPM

Independent user interfaces for BPM should also take on the page navigation. Example issues •  Page navigation depends on the

page data •  There are often data dependencies

across pages •  Pages/data may need to be pre-

loaded based on navigation •  The ideal UI data model will be

different to the process model

Process Layer

Presentation Layer

Page navigation

Page

Process

IBM Software Group


Consumers

Is a process also a service?

Services (Exposure)




Integrated BPM UI


onsumers

Process Runtime (provider)

submitLoanApplication()

Service Components

startProcess()

User Interface

Inte

rnal

inte

ract

ion

with

in

busi

ness

pro

cess

eng

ine

API

“Is a process a service?” – No! But a service can result in the initiation of a process.

IBM Software Group


Processes with integration

Service Exposure


Integration

Consumers

Business Process

IBM Software Group


Single Transaction for all invocations

Multiple Transactions across invocations

Human Tasks

Terminology: Orchestration vs. Process vs. Composition? Orchestration is a sequence of “invocations” of systems, or humans

Appear to the business as a single step.

Do not include human interaction from the business.

Take long enough that the business need to be aware of their intermediate in-progress states.

Include human interaction, or interaction with slow responding systems.

Orchestration

Composition Process

Note: These are not industry standard terms, but derive from common usage observed in the field

IBM Software Group


Core types of Orchestration: Process vs. Composition

Process Makes calls to mature high level services Often triggered (i.e. one way call) rather than invoked as a two way call Where it is invoked as a two way interaction, the caller is typically asynchronous (i.e. not a user) and therefore the service level agreement is throughput based rather than response time based Stateful persistence of the steps in the process Events can correlate with the running process Often involves human interaction to perform some tasks within the process

Composition Grouping of relatively fine grained interactions Typically called in real-time in a request/response (two way) interaction style. Response time is the primary driver for the service level agreement Common for aggregation functions Some or all the granular interactions may not themselves be exposed as re-usable services Generally state free Never involves human interaction during the composition

IBM Software Group


Separating the business process from low level composition

Loan Requester

Loan Approver

System

Approve Loan

Save Loan Details


What’s wrong with this process model?

Multiple steps in the same system swimlane of a BPMN diagram suggest there is an issue with the granularity of the process modeling. It is likely we are modeling the separate systems users have to update today. In the future process this multiple update will be automated, so it is no longer be relevant to model the individual steps on the business process diagram. We should not model this low level integration logic in BPMN diagram.

Save Personal Details

Save Financial Details

IBM Software Group


Pros/cons of moving composition out of the process layer

Orchestration modelled and implemented within the process layer. Good for visibility, but process much more complex. Granular services exposed for re-use. Interactions traverse many layers (poor performance)

Orchestration modelled and implemented in the integration layer. Hides integration complexity from process. Course grained services exposed for re-use Interactions are close to systems (good performance)

Service Exposure


Integration Layer

Business Process Layer

Composition

Business Process

a b c

Service Exposure


Integration Layer

Business Process Layer

Business Process

b a c

IBM Software Group


Just a selection of the different patterns of orchestration

Integrated workflow

Stateful integration

Aggregation Isolated

transaction

Composition

Exceptions only

Process

Stateless* engine Stateful* engine

* This refers to persistence of orchestration state

Real time retrieval of data from

multiple systems

Real time updates to multiple systems

that can be combined into a single update

Straight through processing across

systems that can’t be combined

transactionally

People based exception handling

Processes integrating people

and systems

IBM Software Group


“Process” patterns

Integrated workflow


Exceptions only

Process

Stateful engine

IBM Software Group


Pure workflow process People based process with no system integration

All data is stored in the process. State is stored as each user works a task Model changes can be in more in the hands of the users. Examples •  Paper based processes •  Procedures involving multiple teams of people Primary benefits: •  Reduction/removal of paper •  Improved work distribution/prioritisation •  Management visibility on process status •  Common understanding of process model Implementation: Stateful orchestration engine with good task user interface capabilities.

System based activity

People-based activity

Queued events

State persistence

IBM Software Group


Integrated workflow process People and systems are involved in the process

Enables progressive automated of tasks Users directly involved in modelling of process. IT required for integration. Examples •  Processes where elements of data are duplicated across multiple systems •  Processes where system updates are unnecessarily complex for users Benefits: •  Benefits of Pure Workflow •  Reduce/re-assign workforce •  Improve end-to-end time •  Improve data consistency Implementation: Stateful orchestration engine with good integration capabilities. Significantly simplified if implemented in conjunction with a service oriented architecture.



Queued events

State persistence

IBM Software Group


Process with people for exceptions only Main path is straight through, people handle known exception paths

Users can progressively refine exception path rules to increase automation. A step in the direction of Straight Through Processing (STP) Examples • Approval processes rules enable some auto-approval Benefits: •  Benefits of Integrated Workflow Process •  Manage proportionally higher volumes Implementation: Orchestration engine with integration and rules capabilities.



Queued events

80%

20%

State persistence

IBM Software Group


“Composition” patterns


Aggregation Isolated

transaction

Composition

Stateless engine

IBM Software Group


Aggregation No transactions required, just gathering data

Data gathered from multiple sources and merged into a single result. By far the most common requirement. Most solutions need convenient access to rich data. Examples •  Gathering quotations from multiple insurance brokers (parallel) •  Combining customer data from multiple systems (sequenced) Benefits •  Relatively simple implementation since no data integrity/transactionality requirements. •  Failures part way through can be simply re-tried •  Hides complexities of data translation/merge and connectivity to disparate systems Implementation: Ideally performed as close to the back end systems as possible. Integration engine preferable due to strong data merging capabilities, and more direct connectivity to back end systems. Challenges •  Should you wait for all responses •  How do you merge parallel requests •  Which layer in the architecture should this be performed.


IBM Software Group


Isolated transaction Multiple transactional calls to the same system

Provides a real-time response regarding the status of the update to the caller. Commonly used to commit changes requested by user interfaces providing immediate feedback of success. Examples •  Updates to multiple tables in the same database. e.g to create an order and its order items. Benefits •  Ensures data integrity across multiple calls to the same system •  Failures will be fully rolled back, so requests can be simply re-tried. Implementation: Ideally done as close to the data as possible for performance, and simplicity of the transactional protocols. Highest performance will be within the back end system (e.g. a stored procedure). Next best is a integration engine that can handle the transactional protocol (such as JDBC). Challenges •  Transactions typically hold resources (e.g. threads/memory) whilst in progress •  The aggregate of all the steps may take longer than the transaction timeout. •  Transactions containing multiple steps can cause deadlocks if coding guidelines are not adhered to.

Transaction Boundary


Single phase commit

IBM Software Group


Distributed transaction Data integrity critical across multiple systems

Requires a independent transaction controller to perform a global transaction across more than one system. This requires that the systems are enabled for two phase commit transactions. Examples •  Committing changes to multiple databases •  Committing changes to a queue and a database. In reality two phase commit is more commonly used “within” a system, rather than across multiple systems. For the reasons noted in “Challenges” Benefits •  Ensures data integrity across calls multiple systems •  Failures will be fully rolled back, so requests can be simply re-tried.

Challenges •  Few system owners will allow a two phase commit to be performed against their systems by external parties due concerns over resource locking. •  Two phase commit require the maintaining of a transaction log so systems can re-align should a failure happen mid transaction. This adds interdependencies that complicate design of high availability and disaster recovery. •  Deadlocks are much more difficult to diagnose than with single phase commit. •  Two phase commit is much more heavyweight in CPU both for the end systems, and for the caller than single phase commit.



Two phase commit

IBM Software Group


Enriched update Only the final update matters, everything else is preparation



read non-critical update

critical update

*Definition: “Critical Update” Any request that performs a change to data (create, update or delete) that MUST be resolved in the case of a failure, either with a transactional rollback, or with compensatory actions if the update has already been committed.

A complex scenario is simplified to focus on the one key update, thereby removing the need for intermediate state. Examples •  Checking for existence of a parent objects before creating a child object – e.g. checking customer exists before creating order. •  Validating data before performing update – e.g. checking for stock inventory before submitting an order. Benefits •  Enables a complex composition yet still removes the need for stateful orchestration. Challenges •  Re-tries result in repeating all of the pre-critical steps •  Invocations deemed to be “non-critcial” may result in confusion during diagnosis of issues.

IBM Software Group


Event distribution/broadcast We only need to know it will be done, not that it has been done

Messages are placed in a number of queues within a single transaction. Examples •  Useful for updates to systems that are not expected to be instantaneously up to date. E.g. data warehouses, audit logs •  Useful for notification type requests (SMS, email etc.) Benefits •  Provides a swift answer to the caller •  With persistent queues, the messages should never be lost, and will be applied to their destinations eventually. Challenges •  Destination systems have not received the data when the response is given •  The time taken for destinations to receive the update is not in the control of the composition •  Should problems occur processing the messages, the composition, and therefore also the caller are unaware. Therefore reliant on external exception handling.



Queued events

IBM Software Group


Basic Interaction Types

Provider Requester getCustomer(CustomerID)

returns Customer

Provider Requester submitOrder(Order)

a)  Fire and forget

b) Request-response

A messaging transport is used. The request is “one-way” sending data to the provider, but with no response. The requester need only wait for the message to be received by the transport – i.e. it is non-blocking. This is the style typically used to initiate a process. i.e. we do not wait for the process to complete.

A synchronous transport is used. The requester requires response data, and must wait for the provider to respond. The provider must be available at the time of the request and must process the request immediately. This is the style typically used with compositions. i.e. the caller requires a response from the composition.

IBM Software Group


Stateful Integration – The pattern with two homes

Integrated workflow Aggregation Isolated

transaction

Composition

Exceptions only

Process

Stateless engine Stateful engine


IBM Software Group


Stateful Integration Process Zero or near zero human interaction

All steps are system invocations. Multiple separate transactional interactions are required. State must be stored in between each step, ideally as part of the same transaction. Examples •  Data synchronisation. e.g. “Change of Address” •  “Straight Through Processing” (STP). e.g. payments processing Benefits •  Enables orders of magnitude higher volumes. e.g. moving sales to the internet •  Enables precise data integrity across disparate systems Implementation: Depends significantly on non-functional requirements – throughput/volumes. See next slide for options.


Queued events

State persistence


IBM Software Group


Implementation options for Stateful Integration Process

Orchestrated interactions Pros Mature products available (e.g. BPM) First class notion of “process” Modelled process visible in implementation Monitoring/reporting at process level Process version migration support Clearer modelling of exception paths Easier to introduce human interaction Cons Higher CPU/network due to persistence Solution will be split between business process integration layers

Chained interactions Pros Higher performance due to minimal persistence Can be implemented entirely in the integration layer Cons Largely a custom solution No notion of “process”, only the individual steps Process “model” not visible in implementation Custom work for effective monitoring/reporting Exception paths harder to understand Custom work to migrate to new processes Custom work to introduce human interaction

Queued events State persistence

IBM Software Group


What orchestration patterns are required?

Service Exposure


Integration

Consumers

Business Process Integrated workflow

Page Navigation

Enriched Update

Aggregation Stateful integration

IBM Software Group


Core characteristics for assessing orchestration requirements

Time-span Granularity

Human interaction

Principal objects

Application integration

Complexity

Monitoring

Flow ownership

Dynamicity Transactionality

Performance

Volumes

Business value

State management

Security

Infrastructure

IBM Software Group


Many patterns, many possible implementation options

Integration flow pub/

sub

Business process

short running

Business process briefly

persisted

Integration flow 1:n

Integration flow 1:1

with translation

Business Process

long running system centric

Business Process

long running human centric

Integration flow 1:1

routing only

Many attempts at decision trees for implementation choices fail because they try to cover the whole many-to-many mapping from all possible requirements to all possible implementations. This is typically an impossibly hard decision tree to draw, and inconceivably complex to navigate.

Implementation Options

Requirements

more… more…

IBM Software Group


Narrowing the complexity using a two stage decision tree

Patterns

Implementation Types

Use architectural principles to determine the relevant implementation choices for that pattern

Use requirements to choose the pattern

IBM Software Group


Types of orchestration in relation to transactionality Do any of the requests perform critical updates*?

No

Aggregation Enriched update

Yes

Can all critical updates* be performed in one transaction?

Distributed Transaction

Stateful Integration

Yes No

How many critical updates*?

=1 >1

*Definition: “Critical Update” Any request that performs a change to data (create, update or delete) that MUST be resolved in the case of a failure, either with a transactional rollback, or with compensatory actions if the update has already been committed.

Non-transactional Isolated transaction Global transaction Multi-transaction

IBM Software Group


No single answer Process solutions are always a combination of orchestration patterns

Service Exposure


Integration

Consumers

Business Process

Data synchronization

Microflow

In-app. workflow

Screenflow/ page navigation

Stored procedure

Straight-through processing

Business process

IBM Software Group


References

The placement of BPM runtime components in an SOA http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html

Process implementation types: Patterns based design for process-based solutions http://www.ibm.com/developerworks/websphere/library/techarticles/1004_clark/1004_clark.html Process-oriented modeling for SOA http://www.ibm.com/developerworks/library/ar-procmod1