Upload
guido-schmutz
View
218
Download
0
Embed Size (px)
Citation preview
BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
Microservices with KafkaEcosystemGuido SchmutzFrankfurt, 12.12.2017
@gschmutz guidoschmutz.wordpress.com
Guido Schmutz
Working at Trivadis for more than 21 yearsOracle ACE Director for Fusion Middleware and SOAConsultant, Trainer Software Architect for Java, Oracle, SOA andBig Data / Fast DataHead of Trivadis Architecture BoardTechnology Manager @ Trivadis
More than 30 years of software development experience
Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz
Microservices with Kafka Ecosystem2
COPENHAGEN
MUNICH
LAUSANNEBERN
ZURICHBRUGG
GENEVA
HAMBURG
DÜSSELDORF
FRANKFURT
STUTTGART
FREIBURG
BASEL
VIENNA
With over 600 specialists and IT experts in your region.
14 Trivadis branches and more than600 employees
200 Service Level Agreements
Over 4,000 training participants
Research and development budget:CHF 5.0 million
Financially self-supporting and sustainably profitable
Experience from more than 1,900 projects per year at over 800customers
Microservices with Kafka Ecosystem4
Agenda
1. Where do we come from?2. What are Microservices?3. Why not Event Driven?4. Why Kafka for Event-Driven Microservices?5. Asynchronous Microservices in Enterprise Architecture6. Summary7. References
Microservices with Kafka Ecosystem5
Microservices with Kafka Ecosystem6
Where do we come from?
Shop Rich UI
Shop Backend Application
Traditional Approach
Search Facade
Customer DAO
Order DAO
Order Facade
Shop UI
Product DAO
UI Logic
DataBusiness
GUI
Customer Fat Client App
Customer BOCustomer UIDataGUI
Data Storage
Shared Database
Microservices with Kafka Ecosystem7
sync request/response
Shop UI App
Business Activity Service
SOA Approach
Search BAS
Customer DAO
Order DAO
Order BAS
Shop UI
Product DAO
UI Logic
Data
GUI
Service
Business Entity ServiceShop Web App
Shop UI UI LogicGUI
Data Storage
CustomerDatabase
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order andProduct DBSOAP
SOAP
SOAP SOAP
SOAP
SOAP
SOAP
Microservices with Kafka Ecosystem8
Shop UI App
Business Activity Service
Virtualized SOA Approach
Search BASCustomer DAO
Order DAO
Order BAS
Shop UI UI LogicGUI
Business Entity Service
Shop Web App
Shop UI UI LogicGUI
Data Storage
CustomerDatabase
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order andProduct DB
Service Virtualization Layer
Service Bus
SOAP SOAP
SOAP
SOAP
SOAP
SOAP
SOAP
9
Microservices with Kafka Ecosystem10
What are Microservices?
Key Principles of Microservices
Tightly Scoped
Tightly Encapsulated behind concrete interfaces
Responsible for managing their own data
Highly cohesive
Highly decoupled
Independently deployable, self-contained and autonomous
Microservices with Kafka Ecosystem11
Microservice Approach Customer Microservice
{}
Customer APICustomer
Customer Logic
Order Microservice
{}
Order API OrderOrder Logic
Product Microservice
{}
Product API ProductProduct Logic
Stock Microservice
{}
Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
Microservices with Kafka Ecosystem12
Microservice Approachwith API Gateway
Customer Microservice
{}
Customer APICustomer
Customer Logic
Order Microservice
{}
Order API OrderOrder Logic
Product Microservice
{}
Product API ProductProduct Logic
Stock Microservice
{}
Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
API Gateway
Microservices with Kafka Ecosystem13
Synchronous World of Request-Response leads to tight, point-to-point couplings
Microservices with Kafka Ecosystem14
problem in lower end of chain have a ripple effect on the other service• crash of service• overloaded service / slow response time• change of interface
Service 2Service 1
{}
APILogic
{}
API Logic
StateState
Service 3
{}
API Logic
State
Service 4
{}
API Logic
State
Service 5
{}
API Logic
State
Service 7
{}
API Logic
State
Service 6
{}
API Logic
State
Microservices with Kafka Ecosystem15
Why not Event-Driven?
3 mechanisms through which services interact
Request-Driven Event Driven
Service
Logic
State
Event Consume“IPad OrderedEvent”Command
”Order IPad”order(iPad) : boolean
EventPublish“OrderValidatedEvent”
Query”Retrieve my Orders)getAllOrders(myself)
Event Broker
Microservices with Kafka Ecosystem16
Request-Driven Communication – Highest Coupling
Shop UI Application
UI Logic
Order Microservice
Logic State
{}
API
Stock Microservice
State
{}
API Logic
Call PlaceOrderAPI
Call UpdateStockAPI
Maybe Call Reorder API
Microservices with Kafka Ecosystem18
Decoupling through Events – Lowest Coupling
Event Broker
Microservices with Kafka Ecosystem19
Order Microservice
Logic State
{}
API
Stock Microservice
State
{}
API Logic3) Raise OrderConfirmedEvent
4) Listen toOrderConfirmedEvent
5) Maybe Raise Reorder Event
2) Listen toOrderRequested
Event
1) RaiseOrderRequestedEvent
sync request/response
async request/response
async, event pub/sub
Event-Driven (Async)Microservice Approach
Customer Microservice
{}
Customer APICustomer
Customer Logic
Order Microservice
{}
Order API OrderOrder Logic
Product Microservice
{}
Product API ProductProduct Logic
Stock Microservice
{}
Stock API StockStock Logic
Shop Web App
Shop UI UI LogicGUI
REST
REST
REST
REST
API Gateway
Microservices with Kafka Ecosystem20
Event Broker
sync request/response
async request/response
async, event pub/sub
Shop UI App
Business Activity Service
Process & Virtualized SOA Approach – Sync & Async
Search BASCustomer DAO
Order DAO
Order BAS
Shop UI UI LogicGUI
Business Entity Service
Shop Web App
Shop UI UI LogicGUI
Data Storage
CustomerDatabase
Customer BES
Payment BES
Product BES
Order BES
Custer BAS
Order andProduct DB
Service Virtualization Layer
Service Bus
Orchestration
Microservices with Kafka Ecosystem21
Command Query Responsibility Segregation (CQRS)
Optimize different nonfunctional requirements for read and write behavior
interface to the service is split between • commands that trigger changes in state• queries that provide read access to the state
of resources
support services with higher performance and capacity requirements for readingdata than for writing data
scare reading capacity independently of writing
Data Storage
Write Data Store
Read Data Store(Materialized Views)
Service
Command Service
Query Service
App
UI
ProjectionService
UI Logic
Microservices with Kafka Ecosystem22
Event Sourcing
persists the state of a business entity as a sequence of state-changing events
Whenever state of business entity changes, a new event is appended to the list of events
Saving an event is a single operation and is inherently atomic
The application reconstructs an entity’s current state by replaying the events
Data Storage
Event Store
Service
Event Service
Publisher
App
UI
UI Logic Replayer
Other App
Microservices with Kafka Ecosystem23
Event Sourcing & CQRS
Event sourcing is commonly combined with the CQRS pattern
materializing views from the stored events
Optionally Commands can be stored in event store and transformed into events by the command handler
Data Storage
Event Store
Service
Command Service
App
UI
UI Logic
Query Service Read Data Store(Materialized Views)
ProjectionService
Command Handler
Microservices with Kafka Ecosystem24
Using Event Sourcing with Microservices
Microservices with Kafka Ecosystem25
“Event sourcing enables building a forward-compatible application architecture — the ability to add more applications in the future that need to process the same event but create a different materialized view.”
Neha Narkhede, Confluent Blog
Microservice
State
Command Handler
{}
API
REST
Event Store
Projection Handler(s)
Query Logic
Event Handler(s)
Event Subscribe
How many Event Stores do we need ?
Microservice
{}
API StateLogic
REST
Event Store
Event
Microservice
{}
API StateLogic
REST
Event Store
Event
Microservice
{}
API StateLogic
REST
Event Store
Event
Microservice
{}
API StateLogic
REST
Microservice
{}
API StateLogic
REST
Event Store
Event
Microservice
{}
API StateLogic
REST
OR
Microservices with Kafka Ecosystem26
Have only one „source of truth“
Avoid double write!• Would need distributed transactions
Write Event first then consume it from same micro service • “eat your own dog food”
Microservice
{}
API StateLogic
REST
Event Store
Event
Microservice
{}
API StateConsumer
REST
Event Store
Event
Publisher
Microservices with Kafka Ecosystem27
Data Store
Publish Events through Database
Customer
Event Store
Event
Integration
Microservice
StateLogic
CDCCDC Connector
Customer Fat Client App
Customer BOCustomer UI
Microservices with Kafka Ecosystem28
Why Kafka for Event-Driven Microservices?
Microservices with Kafka Ecosystem29
What is apache Apache Kafka?
The who is who• Producers write data to brokers.• Consumers read data from
brokers.• All this is distributed.
The data• Data is stored in topics.• Topics are split into partitions,
which are replicated.
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer
Broker 1 Broker 2 Broker 3
ZookeeperEnsemble
Microservices with Kafka Ecosystem31
Apache Kafka – scalable message processing and more!
Microservices with Kafka Ecosystem32
Source Connector
trucking_driver
Kafka BrokerSink
Connector
Stream Processing
Leverage the Log
at the heart of Apache Kafka sits a distributed log
collection of messages, appended sequentially to a file
service ‘seeks’ to the position of the last message it read, then scans sequentially, reading messages in order
log-structured character makes Kafka well suited to performing the role of an Event Store in Event Sourcing
Event Hub
01020304050607080910111213141516171819202122
Reads are a singleseek & scan
Writes are append only
Microservices with Kafka Ecosystem33
Scale-Out Architecture
Microservices with Kafka Ecosystem34
topic consists of many partitions
producer load load-balanced over all partitions
consumer can consume with as many threads as there are partitions
Producer 1
Consumer 1Broker 1
Producer 2
Producer 3
Broker 2
Broker 3
Consumer 2
Consumer 3
Consumer 4
ConsumerGroup1
ConsumerGroup2
KafkaCluster
Strong Ordering Guarantees
most business systems need strong ordering guarantees
messages that require relative ordering need to be sent to the same partition
supply same key for all messages that require a relative order
To maintain global ordering use a single partition topic
Producer 1
Consumer 1
Broker 1
Broker 2
Broker 3
Consumer 2
Consumer 3
Key-1
Key-2
Key-3Key-4
Key-5
Key-6
Key-3
Key-1
Microservices with Kafka Ecosystem35
Replay-ability – Logs never forget
by keeping events in a log, we have a version control system for our data
if you were to deploy a faulty program, the system might become corrupted, but it would always be recoverable
sequence of events provides an audit point, so that you can examine exactly what happened
rewind and reply events, once service is back and bug is fixed
Event Hub
01020304050607080910111213141516171819202122
Replay
RewindService
Logic State
Microservices with Kafka Ecosystem38
Hold Data for Long-Term – Data Retention
Producer 1
Broker 1
Broker 2
Broker 3
1. Never
2. Time based (TTL) log.retention.{ms | minutes | hours}
3. Size based log.retention.bytes
4. Log compaction based (entries with same key are removed):
kafka-topics.sh --zookeeper zk:2181 \--create --topic customers \--replication-factor 1 \--partitions 1 \--config cleanup.policy=compact
Microservices with Kafka Ecosystem39
Keep Topics in Compacted Form
0 1 2 3 4 5 6 7 8 9 10 11
K1 K2 K1 K1 K3 K2 K4 K5 K5 K2 K6 K2
V1 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11
OffsetKey
Value
3 4 6 8 9 10
K1 K3 K4 K5 K2 K6
V4 V5 V7 V9 V10 V11
OffsetKey
Value
Compaction
Microservices with Kafka Ecosystem40
V1 V2
V3
V4 V5
V6
V7
V8
V9V10 V11
K1 K3 K4 K5K2 K6
Topic Viewed as Event Stream or State Stream (Change Log)
Event Stream State Stream (Change Log Stream)
2017-10-02T20:18:46 11,Normal,41.87,-87.67
2017-10-02T20:18:55 11,Normal,40.38,-89.17
2017-10-02T20:18:59 21,Normal,42.23,-91.78
2017-10-02T20:19:01 21,Normal,41.71,-91.32
2017-10-02T20:19:02 11,Normal,38.65,-90.2
2017-10-02T20:19:23 21,Normal41.71,-91.32
11 2017-10-02T20:18:46,11,Normal,41.87,-87.67
11 2017-10-02T20:18:55,11,Normal,40.38,-89.17
21 2017-10-02T20:18:59,21,Normal,42.23,-91.78
21 2017-10-02T20:19:01,21,Normal,41.71,-91.32
11 2017-10-02T20:19:02,11,Normal,38.65,-90.2
21 2017-10-02T20:19:23,21,Normal41.71,-91.32
Microservices with Kafka Ecosystem41
Keep Topics both in Original and Compacted Form
Microservices with Kafka Ecosystem42
Kafka BrokerABCBAEKAEACK
BEACK
Producer
Kafka BrokerABCBAEKAEACK
BEACK
Producer
ConsumeandProduce
OR
TX
Legacy Microservice
Change Data Capture (CDC)
Microservices with Kafka Ecosystem43
RDBMS cdc-source trucking_driver
Driver Topicelasticsearch
-sink NoSQL
Enrich Stream with Static Data with Kafka Streams
Microservices with Kafka Ecosystem44
Movement Topic ConsumeandProduce
DriverTable
Driver Topic DriverHandler
Join
Driver Local State
Movement & Driver Topic
Building Embedded, Materialized View with Kafka Streams
Microservices with Kafka Ecosystem45
Movement & Driver TopicConsumeandProduce
JoinStore
Engine Metric Topic
Join
Join State
WindowAggregation
ResultStore
Result State
Building Embedded, Queryable Materialized View with Kafka Streams
Microservices with Kafka Ecosystem46
Movement & Driver Topic ConsumeandProduce
JoinStore
Engine Metric Topic
Join
Join State
WindowAggregation
ResultStore
Result State
API Application
Asynchronous Microservices in Enterprise Architecture
Microservices with Kafka Ecosystem47
Hadoop ClusterdHadoop ClusterBig Data Cluster
Asynchronous Microservices in Enterprise Architecture
Location
Social
Clickstream
Sensor Data
Billing &Ordering
CRM / Profile
MarketingCampaigns
CallCenter
MobileApps
SQL
Search
BITools
Enterprise Data Warehouse
Search/Explore
Online&MobileApps
File Import / SQL Import
WeatherData
Event Hub
Parallel Processing
Storage
Storage
Raw
Ref
ined
Results
Microservice Cluster
Microservice State
{}
API
Stream Analytics Cluster
StreamProcessor
State
{}
API
EventStream
EventStream
Service
48
Summary
Microservices with Kafka Ecosystem49
Summary
Microservices with Kafka Ecosystem50
• service autonomy is key in a Microservices Architecture!
• not all communication need to be synchronous => separate into• commands
• events
• queries
• Kafka is well suited as an event broker / event store• brings many more interesting features beyond just “message passing”
References
Microservices with Kafka Ecosystem51
References
Microservices with Kafka Ecosystem52
Microservices Blog Series, Ben Stopford, Confluent:• https://www.confluent.io/blog/tag/microservicesApache Kafka for Microservices: A Confluent Online Talk Series:• https://www.confluent.io/landing-page/microservices-online-talk-series/Turning the database inside-out with Apache Samza, Martin Kleppmann, Con• https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/Event sourcing, CQRS, stream processing and Apache Kafka: What’s the connection?, Neha Narkhede, Confluent:• https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/Immutability Changes Everything, Pat Helland, Salesforce:• http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdfCommander: Better Distributed Applications through CQRS and Event Sourcing, Bobby Calderwood:• https://www.youtube.com/watch?v=B1-gS0oEtYc
Technology on its own won't help you.You need to know how to use it properly.
Microservices with Kafka Ecosystem53