Connected Systems Consulting Ltd

Sponsored & Brought to you by

The Analysis Part of Integration ProjectsMichael Stephenson

https://twitter.com/michael_stephen

https://www.linkedin.com/in/michaelstephensonuk1

The Analysis Part of Integration Projects

Connected Systems Consulting Ltd

Connected Systems Consulting Ltd

Who am I?Michael Stephenson

– UK-Based Freelance Consultant specializing in:• BizTalk• Windows Azure• Integration

– Microsoft Integration MVP for 6+ years– Pluralsight Author– Azure Insider/Advisor– One of organizers of UK Connected Systems User Group– Founder of BizTalk Maturity Assessment – www.biztalkmaturity.com– Worked on approx. 30 projects that have leveraged Azure– First project went live about 4 years ago

– Contact Info:• Blog: http://microsoftintegration.guru• Twitter: @michael_Stephen• Linked In: http://www.linkedin.com/in/michaelstephensonuk1• Email: michael_stephensonuk@yahoo.co.uk

Thanks to….

Question

“How often do you feel thrown in at the deep end when starting an integrationdevelopment project because no one seems to have a clear understanding ofwhat is required”?

+1 in the chat window if you think this is common

The Usual Story

Common Scenario 1

“I need a feed of customer data from system A to system B”

Err…. Ok so what does this mean?

Common Scenario 2

Contoso

Fabrikam

Acme

CRM System

ERPSystem

FinanceSystem

MarketingSystem

ProductsSystem

ProductsSystem

OrdersSystem

OrdersSystem

OrdersSystem

CustomersSystem

Its this interface here, can we just do this?

Those arrows are pretty but we don’t know anything about what it means

Common Scenario 3

“I need an additional data attribute added to an Existing integration process”

This may be straightforward and we could just get on

with it… but lets get a little info to be sure

Common Scenario 4

“We need to change the existing new customer processSo it also updates the new CRM system”

I hope we have all of the data we

might need

The Problem

What systems are involved?

What does the data look like?

How much data do we

need to process?

What do things happen?

What is the process flow?

Are there any alternative or

exception scenarios?

How do I talk to each

application?

Are there any SLA’s?

Is there any business logic?

What are the data

transformation rules?

Will we need any new

infrastructure?

Are there any performance

requirements?

Are there any security

requirements?

Is there any documentation

?

When do things

happen?

“There’s just so much unknown that affects how we design and build the system”

How Important is Analysis?

Integration is easy when…• The solution is very like something you have done before

• You are reusing stuff you have build in the past

• The applications involved are well understood

• Only a small number of people are involved

• The data is not complicated

• The process logic is simple

• You have a good test environment setup

• Everyone understands the process to be built

• You have good testing

• Your dependencies are well understood

Integration is hard when…• You need new infrastructure

• You have lots of dependencies or they are poorly understood

• Lots of external vendors or people are involved

• You need to use patterns you have not implemented before

• The applications involved are not well understood

• The applications involved are unreliable

• You have a poor test environment setup

• You follow good ALM practices

• You don’t have enough information

• Your requirements are not well understood or clear

• You have a poor or incorrect solution design

• Your making it up as you go along

Getting it wrong

Not enough analysis• Developer gets things wrong

• Architect makes bad assumptions

• Poor documentation

• Testing problems

• Delays to redo and fix things

• Developer ends up chasing around to do analysis that wasn’t done

• Takes longer to deliver value

• Unhappy team

• Unhappy customer

Too much analysis• Wasted time

• Too much documentation

• Lots of the information doesn’t add value

• Takes longer to deliver

• Costs too much

• Over complicates things

• Difficult to deal with change

• Time to delivering value is high

• Unhappy customer

Amount of Analysis

Amount of Analysis

High

HighLowSweet spot = Just enough

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 300

20

40

60

80

100

120

Chart Title

Cost of Project Risk

We are usually in one of these places

MethodologiesI have a methodology doesn’t this solve the problem?

Methodologies – In Theory

Waterfall Agile

Analysis

DesignDevelop

Analysis Design Develop

Methodologies – In Practice

Waterfall AgileAnalysis & Developme

nt

Re-doing Analysis & Develop that

we missed or got wrong

Analysis Design Develop

Analysis Design Develop

Time Spent

Value Delivered

What happened to architecture?

Agile v’s Waterfall

Agile• Just give it to the scrum team

• We are doing agile so things are assumed to be simpler

• The team will figure it out

• Often teams don’t do analysis sprints

• Often This should be a sprint 0 type activity

• You usually have more stakeholders than just the product owner

• Encourages POC of risk areas which is good

Waterfall• Lots of analysis and design before any real world

validation

• Analyst/Architect not close enough to delivery to handle change

• Analysis sometimes too detailed and overkill in terms of presentation format

• Analysis that doesn’t add value

“In the real world a methodology often doesn’t solve the problemas they aren’t getting the right amount of the right information”

Doing it effectively

Lets take the best things and make

them work?

“Gathering the information we need to allow us to design, build and test the solution the customer needs, and being able to communicate that information to the entire team”

Analysis is….

“Just enough analysis to do an effective job”

“Analyst and Architect working together”

The Project

Developer

Legacy ApplicationTeam

ERP System Team

Infrastructure Team Support Team Security Team

Finance Team

Project ManagerSenior Management

Change Team

Definition of Ready

Do we feel ready?• Do we feel confident about what

we need to do• Is the project broken down into

stories or features?• Do we need or have the right

analysis artefacts

Measuring Readiness• INVEST• I – Independent (of others)• N – Negotiable• V – Valuable• E – Estimatable• S – Small• T - Testable

Analysis Iterations

• Iteration 1 – High Level• Iteration 2 - More Detail• Iteration 3 – Reduce Risk

Iteration 1

Iteration 2

Iteration n

Iteration 1 - Analysis

Step 1 – The User Story

StructureIn order to (why is it important)As a ………. (who wants it)I want …… (what do you need)

ExampleIn order to collect money from customers

As a sales director

I want the CRM system to process credit card payments on a monthly basis

“If you cant explain the value statement then why do we want to do it”?

Step 2 – The Context Diagram

Old CRM New CRM

Orders Website

Lookup Customer

Customer Transfer

Orders System

Process Orders

Step 3 – Integration Use Cases

Step 4 – High Level Process Logic

• Could you easily explain the process logic to someone

Step 5 – High Level Data Items

• What entities?

• Any relationships?

• Any key data attributes?

• Where does data come from?

Step 6 – Applications Involved

• What are the supported interfaces?

• What protocols are supported?

• Where are the applications?

• Who manages them?

• Who are the key contacts?

Step 7 – Non-Functional Requirements

• Is there a general list of NFR’s

• Are there any specific NFR’s raised by the business

• Are their any IT NFR’s

• How should the system perform?

Step 8 – Stakeholder Map

Keep Satisfied

• CRM Development Team

• Infrastructure Team

Actively Engage

• Product Owner• HR Department• Support Team

Monitor

• Warehouse Dept

Keep Informed

• ERP Development Team

• Finance Dept

• Make sure we identify stakeholders and where they fit into the map so we can make sure we talk to the right people at the right time

• Make sure owners of error conditions are identified not just owners of processes

Step 9 – Which type of Integration?

• API & Services

• ESB / Messaging

• Batch Based

• Simple Orchestration

• Complex Orchestration

• Mixed

Step 10 – Candidate Architecture

• Is it something we already do?

• What patterns do we need?

• How might we build it?

• How confident are we about the design?

Step 11 – Create Integration Catalogue• Agree a friendly Interface Name

• Identify Source and Destination Systems

• Identify Candidate Integration Type• Batch• API• ESB/Messaging• Complex Orchestration

• Identify Types of Data involved

• Identify grouping of interfaces

• Identify related Use Cases

• Relate non functional requirements

• Relate candidate architecture

Checklist

What have we done• Created an Integration Catalogue for project which

pulls together:• User Stories• High Level Context• Identified Use Cases• Draw Process Logic• High Level Data Model• Applications Involved• Stakeholders• Non-Functional Req’s• Candidate architecture

How do we feel• Are we confident about process

• Are we confident about data

• Are we confident about candidate architecture

• Does it feel simple or complex

• Do we feel we have a good understanding

Iteration Summary

Iteration 2 – Analysis & Design

“We have decided we need more information & detail before proceeding, what do we get next?”

What next?

Where do we look – Key Areas

1 Processing Logic • What processing logic is required

• What data transformation is required

• Are there any patterns

2 Application Connectivity

• How might we connect to the application

• Is the application interface well defined and easy to work with

3 The Data • Does the data make sense• Does the data map between

systems well

1

2

2

2

2

3

3

3

Step 1 – Flush out features

• Find alternative scenarios

• Specification by example

• Find exceptions

• Confirm owner of exception scenarios

Step 2 – Define Data

• Get sample data

• Data Model

• Get data specifications if existing• XSD/XML• WSDL/Swagger• JSON• Flat File Structure• HL7• EDI

Step 3 – Define Data Mappings

• Encoding formats

• Reference data mappings (Look up values)• Mr AA• Mrs AB• Miss AC

• Number formatting• Leading zeros• Decimal places

• Text• Length• Alignment• Padding

• Date formatting

Important this can often be an areaWhich can throw up more integration use cases or maintenance challenges

Step 4 – Application Integration Capabilities

• Do we have connectors?

• Are the connectors well understood?

• Are there challenges doing the connection?

• How will security work?

• Are there any throttling requirements?

Step 5 – Non Functional’s

• Revisit the NFR’s and validate with stakeholders

Step 6 – Patterns

• Are there any obvious integration patterns involved

• Patterns can help create a common understanding

Step 7 – Design Decisions

• Are there any key design decisions to make• Who will be involved• How will the decision be made• How do we communicate to the decision to everyone• How do we ensure the decision is followed?

Step 8 – Update Candidate Architecture

• Is our candidate still good?

• Are there any new things

Checklist

What have we done• More detailed process definition using

Specification by Example

• Defined data in more detail

• Define mappings

• Reviewed NFR’s

• Identify Patterns

• Update Architecture

How do we feel• Are we confident about process

• Are we confident about data

• Are we confident about candidate architecture

• Do we understand the complexity

Iteration Summary

Iteration 3 – Reduce Risk

“How do we reduce the risk around the areas we have concerns with”

What next?

Step 1 – Logical Process POC

• What to do• Implement light weight stubs to prove the applications will

work

• What does it achieve• Allows focus on “will the applications work”• Mitigates risk that the process logic will not work

Step 2 – Application Integration POC

• What to do• Test we can connect to the application• Create BDD style Specflow tests of the Application interface

• What does this achieve• Test the application behaves the way we expect it to• Tests the data looks like what we expect it to• Tests assumptions about error conditions

Step 3 – Update Candidate Architecture

• Is our candidate still good?

• Are there any new things

Checklist

What have we done• POC the key risk areas

• Updated architecture candidate

How do we feel• Do we have any concerns?

• Have we mitigated risks?

Iteration Summary

Analysis Iterations

• Just enough analysis to set us up to be successful

• Use the iterations that are needed

• Make sure information is clear and can be understood by anyone in the team

Iteration 1

Iteration 2

Iteration n

Takeaway

• Define your organisations definition of ready?• What analysis artefacts do you usually need?• Share it?• Who is up for some community guidance around this?• Technet wiki or something?