BizTalk ESB Toolkit 2.1 and Real World Best Practices Prepared by: Joe Simon, Lead Architect, Stott Creations August 18, 2012 New projects always start out with the best intentions. Teams are energized by the prospect of

delivering a clean low-maintenance solution to the business. Often, new projects are driven by new

technology platforms that boast the best set of features for the job. However, what is also all too

common is the low ramp up time team members are given to fully understand the new technology. This

leads to a final product that is hard to understand and a backlog full of “would of”, “could of”, and

“should of”s.

The ESB Toolkit is not excluded from this scenario. If an ESB implementation moves too far down the

wrong track, it can negate the primary benefits of using an ESB. Fortunately, some best practices have

precipitated from leveraging the ESB toolkit in real world scenarios. The following whitepaper discusses

these best practices and the advantages of the ESB model in further detail.

Top 5 List of BizTalk Real-World Best Practices Best Practice Benefits and Explanation

In design, use the ESB Toolkit as the starting point – but not the endpoint.

The ESB portal as delivered is to expose the capabilities of the ESB toolkit data and their relationships but is not a production grade site as-is.

Just Enough Logging (JEL) - scale back logging post-go live to sane levels. Do not enable tracking for ports and orchestrations, and minimize the tracking of MsgBody. Do enable tracking on Biztalk server hosts.

Logging every message is demanding on database and storage resources, and over time becomes less valuable. After deployment, administrators and users only care if BizTalk is running and if they have the ability to drill in on errors. As a general rule, the greater the amount of logging, the less said logging is used. Minimize and target the tracking and logging you actually need.

Avoid complex deployment scenarios using shared custom core libraries

As a rule of thumb, always exhaust all possible scenarios using out-of-the-

box (OOTB) components to solve an issue before moving to custom code.

Most scenarios can be accomplished by leveraging the samples provided, at

the very least as a starting point. Shy away from starting from scratch. This

also eases deployment and maintenance (no custom core components

means nothing in the config files. Everything can be deployed in a

comprehensive package)

Write tightly focused, small components

This promotes solid designs and reusability. Break up development in small units of work and string these solutions together in the itinerary.

Your endpoints should always be resolved/configured via BRE over UDDI

Power users often want to change endpoints and map types; the BRE as a

resolver engine is the most powerful and flexible engine available.

Comparatively UDDI both lacks intelligence and is has a complex interface

configuration.

What can the ESB Toolkit offer my organization? The essence of the ESB model in BizTalk is a standardized way to build declarative services that can be

easily modified due to mediated layers. This is NOT P2P – new receive locations / endpoints can be

added without affecting older customers. If we’re using Business Rules Editor, deployments are easy and

done with a new BRE policy version – without changing the itinerary or the actual BizTalk server

configuration. Itinerary changes can be made without changing the BizTalk config; and new

transformations and orchestrations can be rolled out without disrupting existing services.

The ESB toolkit offers the following benefits:

Increased flexibility

Easier enterprise-wide deployment versus point-to-point solutions

More configuration versus integration coding

No central rules-engine or broker

Refactorable, incremental patching across the enterprise with zero-downtime.

Designed to facilitate workflow routing

Designed to complement, not replace, existing Biztalk artifacts (pipelines, orchestrations, etc).

Since these artifacts remain unchanged from the BizTalk stack, this shortens the learning curve

across development teams.

A much improved notification process for configured alerts that negate the use of sql

notification services or custom notifications processes (some clients have been forced to use

SQL Server 2005 notification services for equivalent functionality)

What are the shortcomings and limitations of the ESB Toolkit? It should be noted that in our experience the ESB model in itself is not the be-all, end-all for every

integration solution. For example, it has the following shortcomings:

- Not designed out of the box to aggregate data

- Not everything will be message based. For example, some operations still will work

faster and will offer better exception handling in batch mode

- The framework for exception handling is not complete OOTB

The stated purpose of the BizTalk ESB Toolkit itself makes it very plain that the BizTalk toolkit is just a

starting point:

“The Microsoft BizTalk ESB Toolkit 2.1 provides architectural guidance, patterns, and a collection of

BizTalk Server and .NET Framework components to simplify the development of an Enterprise Service Bus

(ESB) on the Microsoft platform and to allow Microsoft customers to extend their own messaging

and integration solutions.”

That sentence states it all –it is intended as a framework and set of patterns that can and should be

extended based on the customer’s needs – not as a design endpoint. This collection of architectural

patterns is not new but is based on traditional enterprise application integration (EAI), message-oriented

middleware, web services, and host system integration.

At a high level, BizTalk ESB Best Practices and BizTalk Best Practices are the same, since the ESB toolkit is

built on core BizTalk components. Receive ports still receive messages, pass them into orchestrations for

handling business logic and processing, or directly onto send ports. These are just done using BizTalk ESB

pipeline components – all extensions of standard BizTalk components, as can be seen below.

Development Approach Most organizations have, at the minimum, a few development standards in place. Although team

methodology is too far of a scope for this document, there are a few points to keep in mind when

working with the ESB Toolkit. One may even consider adopting these points as part of their

development practices.

First, use what BizTalk provides. As discussed above, the ESB toolkit is just that, a toolkit. The toolkit

completely relies on standard BizTalk artifacts such as pipelines, orchestrations, maps, etc… So when

trying to solve a problem with the ESB, it helps to think of your solution using BizTalk artifacts such as

pipelines, orchestrations, maps etc… The ESB Toolkit truly boils down to the itinerary which facilitates

where your message goes and who it talks to. That’s it. Once a team goes down the path of constructing

custom components to fit their view of what an ESB is, the solution instantly becomes more difficult to

understand, monitor, and maintain. As a rule of thumb, always exhaust all possible scenarios using out-

of-the-box (OOTB) components to solve an issue before moving to custom code.

Using what’s provided out of the box with any technology platform over custom code is not

revolutionary. However, it is critical enough to call out. For example, one development shop using the

ESB Toolkit never took the proper time to understand the technology and therefore, in the interest of

project deadlines, constructed custom artifacts to “get the job done”. At the end of their project that

particular shop was left with an environment that required every BizTalk hosted application to be

completely uninstalled before their custom core components could be updated. In addition, this shop

constructed a single all-encompassing orchestration to perform the majority of their ESB activities. This

one orchestration to rule them all required that each application had its own set of bindings that were

re-configured once any new release installs were complete. The binding files alone were a maintenance

nightmare vulnerable to human error and lacked the flexibility and loose coupling of a more proper ESB

implementation. The design went so far as to implement a custom XML disassembler – and a non-

threadsafe one - over the one provided natively in BizTalk! This code eventually had to be scrapped

completely, costing the company nearly a million dollars in unnecessary development costs.

Leveraging OOTB BizTalk artifacts ensures that your team members’ knowledge of the ESB stays

consistent with what an ESB actually is or does. Again, using the monolithic orchestration mentioned

earlier as an example, custom core components can cloud a team member’s understanding of how the

ESB works, and, ultimately, this leads to design decisions that are clunky and less efficient. A different

company tried a much less creative but ultimately far more successful approach, sticking closely to the

book and avoiding custom extension points and configurations.

Continuing with the OOTB theme, it is always a good idea to start any complex solution with the ESB

Toolkit samples. Most scenarios can be accomplished by leveraging the samples provided. Shy away

from starting from scratch and use the samples to gain knowledge on how to solve a particular scenario.

Constructing an ESB solution in small units of work is also beneficial. Small units of work help by allowing

developers to work in parallel if the solution requires multiple artifacts such as schemas, maps, or

orchestrations. The itinerary can then be pieced together as items are completed.

Soft-Code and Soft-Code Extreme If one were to ask a developer which is better “soft-coded” or “hard-coded”? Majority of the time, the

developer would answer the former. In BizTalk it is easy to say “my endpoints are soft-coded” because a

developer has configured them in the BizTalk administration console. Even though the endpoints

specified via the admin console are soft-coded, this approach does not allow for the same level of

flexibility built into the ESB Toolkit.

The real power of the ESB Toolkit comes into play when discussing the routing and resolving capabilities

that comes with the toolkit. For each step in a given itinerary, the ESB Toolkit provides the ability to

resolve what map or address should be executed at that point in time. Leveraging resolvers, such as the

BRE resolver, the Toolkit adds a level of flexibility to endpoint management and, when compared to the

administration console, takes some maintenance points away. For example, addresses and maps can

change at any time without the need to reset any host instances. In addition, the same workflow

specified in an itinerary can be utilized to call different versions of services or maps allowing an

enterprise to truly leverage the benefits of an ESB and Service oriented architecture. All items of your

ESB solution should be constructed to take advantage of Toolkit’s resolver components. One should

truly determine if a static endpoint is needed before rolling one into a production environment. Even if

your requirement states that the endpoint will, “never change”, you should still resolve the address for

the moment that it does.

Resolving addresses and maps at runtime also further extends the loose coupling story of the ESB.

Declaring that all your ESB solutions require non-static locations ensures that your enterprise truly

leverages loose coupling and will allow your applications to mold and change as your enterprise

landscape changes with time.

The Business Rules Editor (BRE) versus UDDI The ESB Toolkit’s resolver components take advantage of a few different implementations. The most

interesting of the two are the BRE and UDDI implementations. Both provide an external and flexible way

to configure your endpoints and maps. However, there are a few distinctions between the two that

separate them when making the critical decision of which resolver implementation to use.

The first distinction is intelligence. The BRE is packed with intelligence. The BRE is a complex rules engine

that demands a chapter or two in every BizTalk technology book. The capabilities to smartly route

messages to the proper address or map are almost endless with the BRE. On the other hand, UDDI lacks

the same level of flexibility and intelligence that the BRE provides. Lastly, it is good to point out that the

idea behind UDDI’s administration portal is a good one, its implementation is lacking given its

complexity. In a real world scenario, it is not common to see power users changing endpoints and map

types. If developers or administration users will be in charge of these endpoints, you might as well

provide as much flexibility as possible by using the BRE.

Logging The ESB Toolkit provides OOTB logging capabilities. These capabilities are a great way to track the health

of your ESB by monitoring the Business Activity Monitoring (BAM) and ESB Exception data. The

Exception Management Framework provided with ESB receives messages from the Exception

Publication Service and Failed Message Routing mechanisms and publishes them in an easy to access

and – more importantly – encapsulate and aggregate them.

It’s hard to overstate the benefits message-based fault handling offers to the enterprise. The robustness

of a system is determined, not by how it handles normal operating conditions, but when messages or

entire systems are malformed or unavailable. By this standard, BizTalk ESB is very robust indeed. A

developer could log failures to the event log for SCOM or custom logging service to handle. Quickly users

and administrators would find themselves drowned in a flurry of alert messages, unable to separate out

the truly important system errors. In contrast, the Exception Management Framework delivers the

following benefits:

can version/deploy new exception types without disrupting production BizTalk processes

call multiple handlers, for example to group failed orders together or to separate out critical

errors

allow repair and resubmit of failed messages

Often an enterprise will want to log every message as it passes through the BizTalk application. In

reality, these messages are usually quite interesting for the first few days of a production go-live. After

those days are past, people realize a couple of things. One, logging every message is very demanding on

database resources. And, two, most power users and system admins only care whether BizTalk is

running correctly and what errors have occurred. So before allowing anyone to determine that all

messages must be logged, consider that the ESB Toolkit comes out of the box with just enough logging

to show that BizTalk is running and any faults that have happened along the way.

Just Enough Logging (JEL) is a beautiful thing. For example, a JEL approach in a logging/management

dashboard will provide the following:

Auditing trails for repair and resubmit

Historical views of exception data, including all BizTalk context properties

Filtering of exceptions based on application

Remote web-based access

Dashboarding and graphical metrics

Repair/resubmittal of failed threads

Alerting – especially summation alerting, so if an endpoint goes offline, thousands of messages

are not sent

The built-in management console for ESB, as can be seen from the screenshots below, does meet these

criteria – on a basic level. However, as can be seen in the following section, there is significant room for

improvement.

SharePoint Management Portal As stated above, the out-of-the-box portal provided as part of the ESB toolkit is a good introductory tool

that offers some visibility and monitoring options to the end user. However, there are features that

would make for a more complete portal application. For example:

1) Track all activity regardless of failures

2) Deployable to an Enterprise’s existing infrastructure

3) Easy install and configuration

4) Customizable tracking to allow the portal to mold to the customer’s data schema

The goal of the SharePoint Management Portal is to provide the features mentioned above and give the

end user the ultimate visibility into their ESB. Leveraging SharePoint also allows the portal to work with

the latest versions of Internet Explorer, Fire Fox, and Chrome. This section describes, in further detail,

the capabilities of the SharePoint Management Portal.

Comprehensive Environment View One issue facing every organization’s ESB/BizTalk implementation is how to gain visibility over the

numerous environments constructed to develop an ESB/BizTalk application. Environments can be as

simple as a two server development to production scenario or as complex as a 20 machine

development, quality assurance, staging, and production environment. How does one monitor all ESB

activity for every environment in an enterprise’s landscape?

The SharePoint Management Portal solves this question. By configuring a webpart and associated

SharePoint list, the user is able to add as many environments to monitor as they’d like.

It is common to see unique tracking requirements per ESB solution and/or per environment. To solve

this issue the Management Portal has the ability to track a custom list of “views” per environment.

Display All Activity

The SharePoint Management Portal shows all activity running in a given ESB environment. This gives ultimate visibility to the end user and allows them to gain a confident sense in how their ESB solutions are performing.

The grid gives the user a clear and comprehensive view of how their ESB is performing. The grid supports typical functionality such as sorting, date controls, paging, and keyboard navigation. In addition, the grid automatically updates itself giving the user a more desktop like experience. One can literally open a view and watch their ESB environment in real-time. Date Control

Use mouse or keyboard arrows to navigate paging

Individual Itinerary The SharePoint Management portal provides and individual itinerary view that displays detailed information regarding the lifespan of the itinerary. This view completes the relationship of itineraries by showing the entire itinerary flow, its associated stages, faults, and resubmission history.

Fault Viewing The SharePoint Management Portal provides the ability to view all faults that occurred during the processing of an itinerary. The SharePoint management portal expands on this concept by delivering the faults on the web page in the context of the parent entity, the itinerary. This gives the user an immediate view of every fault and their resubmission attempts.

The actual fault message lives in a pop up window that presents itself when the user hovers over the fault item. This delivers the full fault message to the user in a non-obtrusive way while adhering to the SharePoint Management Portal’s theme of all information in a single location. Resubmission Capabilities and the Friendly XML Editor The SharePoint Management Portal provides users the ability to edit and resubmit messages to the ESB. This provides an enterprise the ability to manage their ESB solutions with a recoverable process in the

event the itinerary experiences issues during its flight. The SharePoint Management portal takes this concept further with a few features such as friendly XML editing and resubmission history. In the friendly XML editor, every XML message, as long as it is valid XML, is displayed in a tree like structure with color coded values. This provides an intuitive navigation experience for the end user. Virtually anybody with little to no XML experience can start correcting data issues and resubmitting messages. The editor also shows what has changed and what is currently being edited. The Reset button allows the user to roll back to the last saved message and, ultimately, Resubmit sends the message back to the ESB.

If more extensive editing of XML is needed, the Raw button allows adding/removing nodes.

In the event the message is not valid XML, the editor will default to the raw message so the itinerary can still be worked and recovered to completion. Bulk Resubmission The SharePoint Management Portal also has the ability to resubmit failed itineraries in bulk. For example, bulk resubmission can be especially helpful when a large amount of itineraries fail for a third party being down or any other similar situation.

REST URL Approach The SharePoint Management Portal takes advantage of REST URLs to keep an intuitive history as a user queries the ESB portal. This can be especially helpful when the power user needs to provide an executive with status of their ESB operations. Instead of installing a thick client on the executives machine or taking a screen shot, the user can simply cut and past the URL and an instant message to the executive. This also allows users to bookmark views without having to remember the steps taken to arrive at that particular view. Address bar that is an exact snapshot of current page, so sending exact address bar information shows the same thing to recipient.

Custom Data Points

Custom Search Capabilities

Deployment and Configuration