How everyone should test their BizTalk based solutions

Darren JeffordSolution ArchitectMicrosoft UKdarrenj@microsoft.comhttp://blogs.msdn.com/darrenj

Objectives And Takeaways

Objective(s): Some complex BizTalk solutions run into problemsNONE should!

Raise awareness of key tools and techniques to dramatically improve customers experience

Based on real world experienceDrive the quality of enterprise-class BizTalk solutions up

Administration is also key..

BizTalk, Testing, No Time!

We all know testing is important for every solutionCustomers rarely test well enough... but get away with itSolutions using BizTalk tend to be extremely business critical and tolerate no downtimePoor testing runs the risk of huge technical and political problemsExtensive testing must always be performed, no excuses!

Testing approach

“Test early, often and comprehensively”Ensure that a demonstrator is created early on to prove concepts, patterns and indicative performanceToo many solutions are found to perform badly in the last few weeks for any number of reasons

Highlights sub-optimal design up-frontIdentifies where the bottlenecks are likely to beIncreases confidence

We sometimes hear...

“it doesn’t scale”“it doesn’t meet our performance requirements”“we were told it would do XX msgs/sec”BizTalk is not

A universal panaceaOptimised for every customer scenario out-of-the-boxAble to make bad code, good code

BizTalk is part of the overall end-to-end solution

A great “case study”

“ [BizUnit,LoadGen,MSTest] provides the ability to reliably and automatically execute hundreds of test cases that otherwise would have to be performed manually”

“improved turn around times by a factor of upwards of 5. For a test pack containing 100 test cases which took up to 3 days to complete manually we can now execute in approx 3 hrs”

Laying the foundations

Critical to achieving representative performance within your environment

Benchmark your environmentFile Adapter -> File Adapter (3K) message should get ~500msgs/sec

Don’t run perf tests on workstation hardwareBeware of virtual machinesInvest in SQL Server Hardware and DisksBeware of UNC FILE Adapter locationsMessage TrackingRemember Virus Scanners

Foundation Checklist Databases and Logs separated onto separate drive letters and

therefore separate physical hard drives? Are the drives used for Databases and Logs configured on a

fast disk-array and with RAID10/0+1? Next generation SAN technology tends to use huge

caches so RAID choice is becoming unimportant Default Database Autogrowth settings changes or AutoGrowth

disabled? SQL Server Agent Running? Have you created separate Receive, Orchestration and Send

Hosts and moved the appropriate Adapters and Orchestrations to the appropriate host?

Concurrent Web Service Calls configuration file setting changed?

<system.net> <connectionManagement> <add address="*” maxconnection="80"/> </connectionManagement></system.net>

PerfMon Counters..... ??

A huge selection of performance counters are made available by the BizTalk engineIt can be time consuming to identify the counters you need especially when you have multiple machinesCollection of counters is key

Perfmon Countersdemo

Excel spreadsheet to define counter collection for multiple machines

Visual Studio Profiler

New with Visual Studio Team DeveloperIdeal for profiling custom code: custom pipeline components, .NET class librariesProfiler can be installed standalone for servers..

vs_profiler.exe located in the \vs\wcu\profiler directory of the VSTS media

Must be driven via the command line to attach to a service

Configuring the VS ProfilerVSPerfClrEnv /globalsampleon

VSPerfCmd /start:sample /output:“c:\dir\profileroutput"

VSPerfCmd /attach:<PID>VSPerfCmd /detach:<PID>VSPerfCmd /shutdown

VSPerfClrEnv /globaloff

Code Coverage

Enables you to understand what code has been executed during your test run

Identifies holes in your testingIdeal for custom pipeline components, helper assemblies, web services, etc.vsinstr –coverage “YOURASSEMBLY”

vsperfmon –coverage –output “filename.coverage”

vsperfcmd /shutdown

How do I test this?

BAMPrimaryImport

BizTalk: DarrenJ Travel Services

Hotel Orchestratio

n

Flight Orchestratio

n

Hire Car Orchestratio

nTravel

Insurance Orchestratio

n

OrchestrationEventStream

Wiley Flight Services Booking

Web Service

DirectEventStream

Itinerary Message[ASMX]

Itinerary Orchestratio

n MSMQ

BizUnitTest Cases are constructed from generic and reusable test stepsTest cases described using an XML syntax

Driven from NUNIT or MSTestModel every “use case” within your solution

Run a testIf you get a “green light” your solution functions at a requirements levelIncredibly powerful, not just for the development lifecycle but for changes, new releases.....

Not just about BizTalk

Test Steps available “in box”BAMDeploymentStepCheckPop3MailStepContextManipulatorStepCrossReferenceData-

VerificationStepDatabaseDeleteStepDatabaseRowCountStepDBQueryStepDelayStepDotNetObjectInvokerStepEventLogCheckStepEventLogClearStepExecuteCommandStepFileCreateStepFileMultiValidateStepFilesExistStepFilesMoveStepFileDeleteMultipleStep

FileDeleteStepFileMoveStepFileValidateStepHttpPostStepHttpRequestResponseStepMQSeriesClearQueueStepMQSeriesGetStepMQSeriesPutStepMSMQCreateQueueStepMSMQDeleteQueueStepMSMQQueuePurgeStepMSMQReadStepMSMQWriteStepOrchestrationConductorStepReceiveLocationEnabledSte

pReceivePortConductorStep

RenameDirectoryStepRuleEngineStepSMTPReadStepSOAPHTTPRequest-

ResponseStepWaitOnFileStep OutlookReadStep

Your own.....

BizUnit Test Case Format

SetupInitialise the platform ready to perform the test

ExecutionThe execution of the test

CleanupRestore the platform to its state prior to test execution

TestSetup Stage

TestCleanup Stage

TestExecution Stage

Test Step: MSMQWriteStep

Test Step: MSMQReadStep

Test Step: DBQueryStep

Validation Step: XmlValidationStep

Test Step: ExecuteCommandStep

Test Step: MSMQQueuePurgeStep

Invoke a web service

<TestStep assemblyPath="" typeName="SOAPHTTPRequestResponseStep">

<WebServiceWSDLURL>http://..</WebServiceWSDLURL> <ServiceName>MyWebServiceName</ServiceName> <WebMethod>SubmitItinerary</WebMethod> <InputMessageTypeName>Itinerary</InputMessageTypeName> <MessagePayload>ItineraryMessage.xml</MessagePayload>

<ContextLoaderStep assemblyPath="" typeName="XmlContextLoader">

<XPath contextKey="itineraryReference"> //ItineraryReceived/attribute::ItineraryReference </XPath> </ContextLoaderStep></TestStep>

Validate a MSMQ message

<TestStep assemblyPath="" typeName=“MSMQReadStep">

<QueuePath>.\Private$\HotelQ</QueuePath> <Timeout>10000</Timeout>

<ValidationStep assemblyPath="“ typeName=“BinaryValidationStep"> <ComparisonDataPath>

HotelFragment.xml </ComparisonDataPath> <CompareAsUTF8>true</CompareAsUTF8> </ValidationStep>

</TestStep>

Validate database rows<TestStep assemblyPath="" typeName=“DBQueryStep"> <DelayBeforeCheck>25</DelayBeforeCheck> <ConnectionString>...</ConnectionString> <SQLQuery> <RawSQLQuery> SELECT * FROM dbo.bam_Itinerary_Completed WHERE ActivityID='{0}‘ </RawSQLQuery> <SQLQueryParams> <SQLQueryParam takeFromCtx="itineraryReference“/> </SQLQueryParams> </SQLQuery> <Rows><Columns>

<CustomerName>Darren Jefford</CustomerName> </Columns></Rows></TestStep>

BizUnitdemo

Writing a test for a end-to-end scenario

Orchestration Profiler

Code Coverage tools don’t really help with OrchestrationsHow do you understand where time is being spend inside Orchestrations?

Enabling tracking and using HAT is the only real way right now

No way to understand which Orchestrations are not being fully tested

Edge ScenariosError paths

Orchestration Profilerdemo

“Code Coverage” your orchestrations

Identify slow performing shapes and orchestrations

LoadGenBizTalk Load Generation Tool

New 2007 release provides WCF supportGenerate messages to test how servers handle normal and extreme workload conditionsData GenerationLoad Throttling

Find the maximum sustainable rate of your rig, automatically.

Multithreaded“Throw away all those custom test harnesses”, please...

“in-box” Load Generators

FILE Outputs files in a destination folderHTTP Supports both asynchronous and request-response messaging.SOAP Supports both asynchronous and request-response messagingMSMQ Supports both asynchronous and request-response messagingMQSeries Message can be local or remote, in-order or unordered, persistent or

non-persistent.WSE Supports both asynchronous and request-response messaging.WSS Windows® SharePoint® Services.Custom Base your own on the FileTransport sample

In-box Message Creators

CustomMCCreate new messages based on a templateReplace Items within the message with differing data

SizeMCExplodes a sample message to a certain size (big batch files)

In-box Throttle Monitors

SQLSpoolEnsure the Spool depth remains within a certain range

Rate100 msgs/sec..

CPUKeep the BizTalk servers within 80-90% CPU utilisation

PerfMonMonitor any PerfMon counter

LoadGendemo

Generating load into your BizTalk solutions

LoadGen and BizUnit

LoadGenFantastic load generation tool

Generate increasing loadFind the maximum sustainable point of your solution

BizUnitExtensive library of Test Steps

Collecting PerfMon and EventViewer LogsStoring Test InformationValidating Test Results

BizUnit can drive LoadGenAutomate all of your performance testing end-to-end!!

LoadGen and BizUnitdemo

Driving test cases using LoadGen and BizUnit

•Shameless Plug Further ReadingIn depth chapters covering Testing,

Performance & Scalability..

Summary

Get the environment right before any testingStart testing early, and often. A “demonstrator” is key for complex scenarios Use BizUnit for modelling “use cases”Use LoadGen for generating loadUse Orchestration Profiler to ensure your “touching” all parts of your solution during testingUse BizUnit and LoadGen together to automate your tests

Tools

PerfMon Counter spreadsheetOrchestration ProfilerBizTalk DocumenterLoadGen 2007BizUnit

© 2007 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after

the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.