Upload
confluent
View
1.900
Download
4
Embed Size (px)
Citation preview
The Enterprise Kafka Summit 2016
Rimas Silkaitis
how Heroku leveraged Kafka to realize the original vision of the enterprise service bus (esb)
Rimas Silkaitis
Product
Heroku Postgres Heroku Redis
What is an ESB?
Analyst Definitions
ESBs receive service consumption requests, route the requests to the correct service provider, transform the requests to a format compatible with the service provider, wait for the results, and deliver them back to the service consumer.
IDChttp://www.idc.com/2010st/esbcm.html
Uses a bus architecture for scalability and reliability, establishes a unified communications channel, provides integration, mediation and communication services and is standards based.
http://redmonk.com/public/OpenSourceESBs.pdf
Redmonk
Vendor Definitions
Oracle Service Bus transforms complex and brittle architectures into agile integration networks by connecting, virtualizing, and managing interactions between services and applications.
http://www.oracle.com/technetwork/middleware/service-bus/overview/index.html
Oracle
Mule ESB is a lightweight Java-based enterprise service bus (ESB) and integration platform that allows developers to connect applications together quickly and easily, enabling them to exchange data.
https://www.mulesoft.com/resources/esb/what-mule-esb
Mulesoft
How should we define it?
¯\_(ツ)_/¯
App
App
App
App
Finance
HR
BizOps
Inventory
ESBRouting
Mediation
Messaging
EventProcessing
ServiceOrchestration
ESB
Push
Centralized
Architecture
API
CLI GIT Dashboard
Data Runtime BizOps
$ heroku addons:create heroku-postgresql:standard-0 -a sushiCreating postgresql-reticulated-74861... done, ($50.00/month)Adding postgresql-reticulated-74861 to neovintage... doneSetting HEROKU_POSTGRESQL_GRAY_URL and restarting sushi... done, v249The database should be available in 3-5 minutes. ! The database will be empty. If upgrading, you can transfer ! data from another database with pg:copy.Use `heroku pg:wait` to track statusUse `heroku addons:docs heroku-postgresql` to view documentation.
API
CLI GIT Dashboard
Data Runtime BizOps
API
CLI GIT Web
Data Runtime BizOps
THE API IS OUR ESB!
Problems Emerge
• Bottleneck in getting features released that touch API
• Difficult to get Analytics across the entire Org
• Hiring has become difficult
• API is down. So is the entire platform.
PUSH vs PULLhttp://kafka.apache.org/documentation.html#design_pull
PUSH
PULLProducers Consumers
ESB Redefinition
• Pull over Push
• Consumers dictate their own consumption rate
• Independent development, scaling and scheduling
• Stop the world processing doesn’t exist
CLI GIT Dashboard
API Data Runtime BizOps
Other Use Cases Around Heroku
Operational Experience
aggregation
router
aggregation
router
metrics-ingest
router
metrics-ingest
router
ZK postgres
metrics-ingest
router
aggregation
router
aggregation
router
aggregation
router
sink
router
aggregation
router
aggregation
router
metrics-api
router
$ heroku kafka:list -a metaas-addons=== Kafka Topics on KAFKA_URL
Name Messages Traffic─────────────────────────────────── ──────────────
─────────────ROUTER_REQUEST.v1 132,823.12/sec 14.6 mb/sec
DYNO_MEMORY.v1 1,478.97/sec 187 kb/secDYNO_LOAD.v1 1,468.01/sec 153 kb/sec
ROUTER_STATUS_SUMMARY.v1 987.4/sec 28.3 kb/secSYSLOG.v1 691.9/sec 76.8 kb/sec
ROUTER_LATENCY_SUMMARY.v1 678.84/sec 37.5 kb/secROUTER_ERROR.v1 476.12/sec 93.9 kb/sec
DYNO_ERROR.v1 354.06/sec 38.5 kb/secDYNO_LOAD_SUMMARY.v1 241.33/sec 9.73 kb/sec
DYNO_MEMORY_SUMMARY.v1 219.26/sec 25.1 kb/secDYNO_ERROR_SUMMARY.v1 73.49/sec 1.83 kb/sec
ROUTER_ERROR_SUMMARY.v1 12.35/sec 276 bytes/sec__consumer_offsets 4.1/sec 405 bytes/sec
ROUTER_STATUS_PARTIAL_SUMMARY.v1 0.0/sec 0 bytes/secROUTER_LATENCY_PARTIAL_HISTOGRAM.v1 0.0/sec 0 bytes/sec
ROUTER_REQUESTS.v1 0.0/sec 0 bytes/sec
What do we need to do next?
Schema Registry
Security
as a Service
Creating kafka-lively-27896... done, (free)Adding kafka-lively-27896 to sushi... doneKafka cluster has been created and will be available shortlyUse `heroku addons:docs heroku-kafka` to view documentation.
=== KAFKA_URLName: kafka-silhouetted-79119Created: 2016-03-24 21:21 UTCPlan: Beta 3Status: availableVersion: 0.9.0.0Topics: 0 topics (see heroku kafka:list)Connections: 0 consumers (0 applications)Messages: 0.0 messages/secTraffic: 0 Bytes/sec in / 0 Bytes/s out
$ heroku kafka:info -a sushi
$ heroku addons:create heroku-kafka:beta-3 -a sushi
Public Betahttps://heroku.com/kafka
Come See Us At Our BoothWe’re Hiring!