View
9.939
Download
10
Category
Tags:
Preview:
DESCRIPTION
A historic view on software integration on how we've failed with our current approach to SOA (also know as SOA 1.0). We look at the problem, what to do about them using Events and the changes that we need to apply to our design philosophy and architecture to really obtain loose coupling, reuse (or replaceability) and a reactive system.
Citation preview
Jeppe Cramon – Partner in TigerTeam ApS
AANUG ConferenceSeptember 2013
SOA & EDA
There are problems with SOA
• Systems are more fragile• Development and Maintenance costs are
higher• Your services are not being reused• You thought SOA would solve your integration
problems, but they’ve gotten worse• No one wants to build or maintain services• System performance is worse
If you have arrived at the realization that SOA is a pain and costs too much without enabling business agility
or reducing your costs, you are not the only one.
You will have to change your approach to achieve the benefits of
SOA
Let’s take a trip back in time
And look at how we could do business
Sales Department
Old style SOA
Ingredients of Blue Satin Dress
Design Department Customer Department Finance Department Fabric Department
The Customer
Create Sales Order
Get Fabric Status and Size
Get Fabric PriceGet Cus
tomer
Details
?
This is slow
We are not selling enough!!
Sales Department
Old style SOA II
Ingredien
ts of B
lue Sati
n Dress
Design Department Customer Department Finance Department Fabric Department
Create Sales Order
Get Fabric Status and Size
Get Fabric PriceGe
t Cus
tom
er D
etai
ls
Reply
We need to speed things up!
Hackel walks between desks? Way too slow.
We need to speed things up!
He needs to call So we make sure thathe can
Actually, call a lot!
We need to speed things up!
But Hackel still have to wait Even if we get creative
We need to speed things up!
Something needs to be
done!!
Speed up even more!
Lets get an automatic switchboard!
Speed up even more!
New technology, same problem
Even if we get creative again
Or do things Async
So Hackel can send request to Ingeborg and Kresten
which can respond to Hackel when they’ve got an
answer
But even that doesn’t scale well, there’s still only one Ingeborg and Kresten
Speed up even more!
So Kresten and Ingeborg adds more staff
Scalability is expensiveSales Department
Design Department Customer Department Finance Department Fabric Department
Coordinator Hackel
Same problem different technology
Data Storage
Data Storage
Data Storage
FinanceService
DataService
DesignService
Hackel CoordinatorService
Hackel CoordinatorService
Sales Order Service
Sales Order Service
Customer Customer
FabricService
CustomerService
Data Storage
Client
Process Service layer
Activity Service layer
Data Service layer
Aka. Layered SOACaching to Solve
latency problems?
Resemble much?
Now back to the story…
• Scaling up Sales requires scaling up other departments relatively much
• People cost money and why have 9 Finance people for every 10 sales people just to answer questions about pricing
• There’s got to be a better way to handle this• And there is
Sales Department
Event Driven SOA and local caching
Ingredients of Blue Satin Dress
Design Department Customer Department Finance Department Fabric Department
The Customer
Create Sales OrderCusto
mer Deta
ils
Change
dFabric details and stock
Status for Satin Updated
Price for Satin
Changed
Scalability is cheaperSales Department
Design Department Customer Department Finance Department Fabric Department
This is just the beginning
Have our mindset changed over the last 40 years?
Paper work, file cabinets and manual workflows
Next step was automating paper forms(aka. The birth of CRUD)
Database integration
Enterprise Application Integration (EAI)
WebServices to the Rescue?
ESB’s the the Rescue?
ESB Reality
I’ve got a cheaper solution
To be clear – none of the examples represent in my opinion SOA
The 4 tenets of SOA1. Services are autonomous
– Encapsulation & Cohesion at a bigger scale. – A service is autonomous if it doesn’t rely on other services to complete its job
2. Services have explicit boundaries– Services should not be too familiar with each other. There’s a clear boundary
to where the responsibilities of each services starts and ends– Boundaries are made unclear when a service rely on other services to
complete its job3. Services share contract and schema, not class or type
– Encapsulation improved. We need not worry about how things are implemented (languages or platform dependent information)
4. Service interaction is controlled by a policy– Controls under which rules/form of technical communication between
services take place (e.g. using encryption incl. algorithm and keys)
What we’ve seen is just old EAI wine on new SOA bottles
S O A
WebServices and in general synchronous integration has nothing to do with real SOA
Some call this pattern for SOA 1.0 to distinguish them selves from the old
approach and the mess it causes
Layered Architectures typically leaves all orchestration to a central manager (ESB)
where business processes are coordinated through spaghetti code (BPEL)
These BPEL processes typically break all Service encapsulation as they are data and feature
envious
This hurts our coupling an autonomy even further
What we have with classic layered SOA is a HARD COUPLED
architecture
But what about reuse?• Layered SOA has typically been recommended because it increases reuse (at the
expense of a hard coupled architecture)• The thesis is that the more we reuse, the faster we will be done• But the thesis rest on the false assumption that writing code is the most expensive
part of a project– Not all code takes the same amount of time to write. Some code is very trivial while other
code is very hard to write• The real money and time consumers on any project are:
– Figuring out what the customer needs– The time it takes for the customer to figure what they really needed and the resulting rework– Meetings– Integration work, databases, webservices – Fix bugs– Debugging– Deployment– Test– Ship
Failed?
Yes
No
The more dependencies we have, the worse and more expensive this becomes
Service reuse multiplies our direct and especially indirect dependencies which creates high coupling
My new Service that
wants to reuse other
services
Service that is going to be reused
Reusable Service
Reusable Service
Reusable Service
Reusable Service
Reusable Service
DB servic
e
Another Service that is going to be reused
Can’t we just avoid Services and make one big application?
Retail Domain
InvoicingSubdomain
Product Catalog
Subdomain
Shipping Subdomain
Sales (Order) Subdomain
Inventory Subdomain
Pricing
One Domain Model to Rule them All?
Combining subdomains into one big domain model increases complexity as everything tends to be connected to and depend on
everything else
One big model to capture it ALL?
….…..…..
….…..…..
….…..…..
….…..…..
Result: Our queries get messy
The Query of Despair
ONE MODEL TO RULE THEM ALL
ONE MODEL TO FIND THEM
ONE MODEL TO BRING THEM ALL AND IN THE DARKNESS BIND THEM
The database determines our scalability
And databases often scale poorly
To solve the performance problems for a few, you have to upgrade it all
Do not design for locality and try to distribute
Design for distribution, take advantage of locality
Alternatively we can use Read replicas
Master Slave SlaveSlaveSlave
Read
Update
You’re now eventually consistent
Or introduce caching…
Read
Update
What’s your synchronization mechanism?
?
Many perspectives on data
Customer
Retail System
Pricing
Inventory
Sales
Product
Unit PricePromotional PricePromotion End Date
Stock Keeping Unit (SKU)Quantity On Hand (QOH)Location Code
PriceQuantity OrderedName
The lifecycle for Data is VERY important!
Forecasting?
Smaller models & clear data ownership
SalesProduct
ProductIDNameDescriptionQuantity Ordered…
Sales
InventoryProduct
ProductIDSKUQOHLocation Code… Inventory
PricingProduct
ProductIDUnit PricePromotional Price…
Pricing
DDD:Bounded Context
SOA:Service
Retail System
Shared Entity identity
A Service is
• A technical authority for a specific business capability – very similar to Bounded Contexts
• The owner of all the data and business rules that support this business capability – like Bounded Contexts
A Service is equivalent to a Bounded Context
What a Service is NOT
• A service with only functionality (and no data) is a FUNCTION– Like: check if order is valid
• A service that only has data is a DATABASE– Like: CRUD of an entity
This is typically seen in a lot of layered SOA usages where a function calls a function that calls a function that calls a database
Autonomy becomes the next (and biggest) problem
Can you rely on others?
Data Layer
Service Orchestration issue
Data Access Layer
Service Layer
Interface/Display Layer
System CSystem BSystem A
UI
Service Service
DAL DAL DAL
Schema Schema Schema
Local transactionspanning all
modules
Service
Service autonomy
Service B Service C
Service A
System XService A
System Y
Service B Service C
System X
Slow/unreliable networkDifferent SLASlow system
Orchestration using synchronouscommunication and local transactions
System CSystem BSystem A
UI
Service Service
DAL DAL DAL
Schema Schema Schema
B:Service() call C:Service() call B:DAL() call A:Service() commit()
Service
Solution for when going distributed?
Can’t we just distributed transactions?(XA / 2 phase commit)
Distributed Transactions…. Oh my
Sales system
Sale
Update customer status (e.g. Gold customer)
Bookkeeping
Deliver goods
Deliverysystem Deliveries
Customer/CRM system Customer
SAPBookkeeping
CompletePurchase
TransactionCoordinator
TransactionalResource
Request-to-Prepare
Commit
Prepared
Done
PreparePhase
CommitPhase
2 Phase Commit
What’s wrong with distributed transactions?
• Transactions lock resources while active• Services are autonomous– Can’t be expected to finish within a certain time
interval• Locking keeps other transactions from
completing their job• Locking doesn’t scale• X Phase Commit is fragile by design
Orchestration using synchronouscommunication and without distributed transactions
System CSystem BSystem A
UI
Service Service
DAL DAL DAL
Schema Schema Schema
ServiceB:Service() call C:Service() call B:DAL() call A:Service() if (A:Call-Failed?) call A:Service() if (A:Call-Failed?) Wait-A-While() call A:Service() if (A:Call-Failed-Due-To-IO?) Save-We-Need-Check-If-Call-A-Succeded-After-All AND We-Need-To-Retry call C:Service and call B:DAL AND Tell-Customer-That-This-Operation-Perhaps-Went-Well if (A:Call-Went-Well) commit()
More of the same clearly can’t be
The answer…
The more autonomous services are, the more loosely coupled they are.
Loose coupling is what drives business agility – which is why we wanted SOA in the
first place
Loose coupling
Requires good encapsulation and high cohesion
Return of the SILOS!!!!
We need SMALL SILOS that SHARE very LITTLE with each other
Sales Pricing InventoryShipping
✕ ✕ ✕
A silo/bounded-context contains all parts related to it
DB Schema
Domain Services
Application Services
DomainModel
Aggregates, Entities,
Value Objects, Events
Integration Endpoints
(REST, SOAP,
Pub/Sub)
User Interface
Silo
A silos parts are not limited to deployment at a single location
Subdomain
User Interface
Integration Endpoints
(REST, SOAP,
Pub/Sub)
Domain Services
Application Services
Aggregates, Entities,
Value Objects, Events
DB Schema
Units of individual deployment
What should our silos share?
Bounded Context
Degrees of coupling
UI
UI
Service
ServiceData
Data Events
Events
Composite UI’sRetail System
Sales Pricing Inventory
Web/Applicationtier
Background servertier
Storage tier
Composite UI
UI LayeredApplicationData Access Layer
Read
mod
el
Writ
e m
odel
✕ ✕
Remember
Layers != Tiers
Composite UI - exampleContext:
Book ISBN-10 0-321-12521-5
Composite UI - HTML
#DIV – BookReviews
#DIV – BookImage
#DIV – BookTitleAndAuthor
#DIV – BookPricing
#DIV – BookAvailability
#DIV - Header
#DIV - Body
Composite UI
The Service owns it UI in all Contexts and for all Composite UI’s
Not just for HTML clients
Invoice Composite UI example
InvoiceHeader
Order:ShippingInfo
Invoice:NextInvoiceNu
mberInvoice:Data and Due
date
Order:RelationInformation
Ord
er:It
em-Q
ty
Prod
uct:
Item Product:
DescriptionOrder:
Item-Unit-Price
Order:Item-Total-Price
Order:Total
Billing:Balance
All Services participate at the UI level for
each individual Item in the
Order
When won’t a Composite UI make sense?
For certain types of UI’s that built outside of the organization, e.g. customer specific netbank apps and mobile banking application we need to provide a synchronous integration API that will allow them to integrate with the business domains without having to coordinate with the business domains UI team.
The integration API could be built using both a REST* and/or a SOAP approach.
* It’s not such a distant thought to see a Composite UI part as a REST resource with a rendering matching the requested MimeType encoding.
That covers data presentation
What about transactions?
Next Step – Event Driven Architecture
Why Events?
Synchronous calls lowers our Fault tolerance
• When servers crashes• When databases are down• When deadlocks occurs in the database– Do you retry?
With synchronous RPC style Services interaction we loose business data. There’s no automatic retry and idempotence is often an after though.
Cure: Messaging
• Message based communication is asynchronous and thereby breaks temporal coupling
• Messages can be exchanged between services over a message channel
• The Message channel is responsible for delivering the message(s) to the relevant parties (consumers). If something goes wrong, the message will be put back on the Message channel (rollback) and will be resent later
Let’s make the implicit explicit!Old wisdom seems to have been forgotten. Let’s introduce:
Business EventsWhich:• Signal that something has happened• Closely aligned to the Domain Model• Are handled by a messaging system• They are in the past tense:
– CustomerBilled– ParcelShipped– CustomerCreated– ReviewCreated– CommentAdded– CommentDeleted
Business Events as Messages
A Business Event can be transferred over a message channel, such as a Queue or a Topic, in
many different formats
Business Event as XML Message<OrderWasAccepted> <CustomerId>50D1F244-ABBC-4EC7-BDCA-E4934C124A89</CustomerId> <OrderId>C199322A-01F1-4E56-918E-7A63529F8FA3</OrderId> <ShippingAddress> ... </ShippingAddress> <BillingAddress> ... </BillingAddress> <Items> <Item ProductId="4CD22C4B-600C-4477-B5BF-48ABDEE4DA61" Amount="100" AmountUnit="Pieces" UnitPrice="100,10" UnitCurrency="EUR"/> <Item ProductId="56E6BD19-660C-464A-9120-100DAF579855" Amount="10" AmountUnit="Litres" UnitPrice="56,95" UnitCurrency="CHF"/> </Items></OrderWasAccepted>
Business Event as JSON Message{
EventType: "OrderWasAccepted",CustomerId: "50D1F244-ABBC-4EC7-BDCA-E4934C124A89",OrderId: "C199322A-01F1-4E56-918E-7A63529F8FA3",ShippingAddress: { ... }BillingAddress: { ... }Items: [
{ ProductId: "4CD22C4B-600C-4477-B5BF-
48ABDEE4DA61",Amount: "100", AmountUnit: "Pieces", UnitPrice: "100,10",UnitCurrency: "EUR"
},{
ProductId: "56E6BD19-660C-464A-9120-100DAF579855",
Amount: "10", AmountUnit: "Litres", UnitPrice: "56,95", UnitCurrency: "CHF"
}]
}
Business Events help us achieve autonomy, loose coupling and encapsulation
• Encapsulation - because we don’t need to supply our services internal data (because no on else needs to know them) – internal data will only be accessible through the Services UI (which takes part in a Composite UI)
• Events only need to explain WHAT happened and what it relates to and very few business data (which are typically only needed for replication scenarios)
Business Events Messagesand
Business Processes
By publishing Events messages from our Services we can communicate with each other and also drive Business Processes
Using Business Events to drive Business Processes
Sales Service
Shipping
Invoicing
Sales
CustomersM
essa
ge
Chan
nel
The sales fulfillment processing can now begin…
Online Ordering System
Web Shop(Composite
UI)
Invoicing Service
Shipping ServiceOrder
Accepted
Event
Business Events example
Sales Service
Order Accepte
d
Invoicing Service
Retail System
Order Accepte
d
Customer Billed
Mes
sage
Cha
nnel
We use the Order Accepted event message published from the Sales Serviceto drive the Invoicing/Billing of the customer. The billing part of the process also use Business Events, in this case Customer BilledEvent, to indicate that its part of the process is completed.
Because we use asynchronous messaging we can still accept orders in the sales service even though the invoicing services is down. The Order Accepted event message will remain in the Message Channel until the Invoicing Service is ready to process it.
Introducing Choreographed Processes
Sales ServiceOrder
Accepted
Invoicing Service
Shipping Process Manager
(Saga)
Shipping Service
Retail System
Mes
sage
Ch
anne
l
Order Accepte
d
Order Accepte
d
Customer Billed
Customer Billed
Order Ready
for Shipmen
t
Order Ready
for Shipmen
t
Works as a Finite State Machine
(WorkFlow)handling the life cycle of Shipping
and thereby forms a very central new
Aggregate in the System
Process Managers
• Process Managers are essential to the coordination and monitoring of long running processes.
• They work as a Finite State Machines (WorkFlow) which handling the life cycle of Process (e.g. Shipping an Order) and thereby forms a very central new Aggregate in the System
• They can include manual steps/person intervention• Sometimes these Process Managers belong naturally
within a specific Subdomain and other times they are truly and thing by themselves and often form new Subdomains that way
Companies often derive their competitive advantages from their Processes.
A Process Manager allows you coordinate Business Processes on the basis of Events
Domain Events can also be used for data replication
This is typically used for read-mostly master data that is needed in many business-domains. This way we avoid having to call a synchronous
services to query the latest value(s)
Event Based ReplicationSubdomain Z
Repository
Repository
Z DataSubdomain Y
Service Repository
Message channel
Event ReceiverY Event Receiver and Local Cache Builder
Y DataCache
Y Events
Y Events
Questions?@jeppec on Twitterjeppe@tigerteam.dk
@TigerTeamDK on Twitterhttp://tigerteam.dk
Recommended