Towards Improved Data Dissemination
of Publish-Subscribe Systems
Srinath Perera, Ramith JayasingheDinesh Gamage
Lanka Software Foundation
Outline
● Outline of Pub/sub Paradigm● 3 challenges ● Avoiding Blocking IO● Avoiding Message accumulation through parallel
message delivery● Working around Slow and unreliable consumers/
publishers● OGCE workflow suite● Conclusions
Publish/ Subscribe Paradigm and Message Broker
● Many Event sources that generate events● Subscribers notify their interest through
subscriptions ● Broker Matches and deliver events to
Subscribers
Motivating Usecase
Goals of Message Broker
● High throughput ● Preserving Message Order for messages
generated from same event source● Reduce publish to delivery time
Basic Message Broker Architecture
Goals of this Paper
● Three architectural challenges● Blocking IO● Parallel Message Delivery ● Unreliable Consumers
● We explore architectural options in addressing these challenges
● We have improved OGCE Messenger based on our observations and used that the test bed for this study.
Challenge 1: Blocking IO
● Blocking IO assigns each request to a thread. Then the number of parallel clients are limited by number of threads.
● Potential alternative is non-blocking IO, which uses an event based model and minimize the thread blocking due to IO.
● Message broker has an IO dominated workload, and therefore we believed an non-blocking approach can provide major improvements.
● We took advantage of Axis2 supports a pluggable transport architecture and setup the broker with NIO – transport from Apache
Experimental setup● Loaded system with XML messages, over a
constant set of subscriptions● Loaded the system with out loading the
network.● Statistics were calculated periodically (e.g. 2
seconds)● 10 topics, 1000 messages per topic, 200
consumers
NIO vs. Blocking IO
● NIO transport increases the throughput of the system.
● NIO is able to handle more concurrent connections (publishers) with less resources.
Challenge 2: Message Accumulation
● If message reception rate is lower than message dissemination rate,
● => messages accumulate => system slows down and crash
● Often one incoming message need to be delivered to multiple consumers. With high number of Consumers, very high chance of above problem.
● Single delivery thread could pause major limitations.
● But Naive Parallel solution will break the order of message delivery
Parallel Message Delivery
● Two parallelization strategies considered● Topic based
– Each thread is assigned a set of Topic or Xpath Expressions
– Thread will deliver a message to Consumers if it matches to a topic/Xpath handled by it.
● Consumer (EPR) based – Each active consumer is assigned a standing job and
a message queue– Job delivers messages accumulated in message
queues
Topic Based Message Delivery
● Queue for each topic, and matching messages for the topic are place in the queue.
● A thread assigned to each queue pick up messages and delivers.
● Concerns ● If a message matches multiple subscriptions submitted
by the same consumer, it will be delivered multiple times. ● If two subscriptions for same consumer is handled by
different threads, how can system preserve order of messages?
Consumer (EPR) based parallelization
● There is a queue for each consumer.
● Messages for that consumer are placed on that queue.
● A thread assigned to the queue delivers messages in the queue to the consumer.
● Facts● Since only thread delivers messages to a consumer, order is
preserved. ● Jobs and queue are created only when messages are
available for a consumer.● Queue eventually expires when messages are not available
Scheduling Standing jobs
● We use read-write locks to maximize concurrency
● We can not assign a static thread to each consumer as that will not scale with large number of consumers => we use a thread pool.
● Dynamic Thread pool● Standing Job will drain the queue entirely and try deliver
● Will not release the control as long as messages are in queue.
● Potential starvation
● Static Thread pool (size is configurable)● Each thread will iterate over standing jobs assigned to it.
● Each thread will drain the queue partially (configurable)
● Standing job will release control after delivering drained messages
● Optimized system by allowing greater concurrency for message filtering
Performance: Throughput
Performance: Round Trip Time
Performance: Summary● Both parallel implementation performance better
than serial ( throughput & round trip time)● Static thread pool reports better round trip times.● Dynamic thread pool increases throughput
Slow and Unreliable Consumers/Publishers
● Distributed in a heterogeneous environment and unpredictable (beyond the control of the system)
● Effects the performance of middleware● Round trip time● Throughput
● Delay incurred by delivering to slow consumers will be propagated to fast consumers as well.● E.g. connection timeouts will block the thread until
time out period
Solution 1: Soft State Subscriptions
● Forcing consumers to renew ● But the problem persists until the timeout
happens.
Solution 2: Blacklisting Schema
● Consumers are uniquely identified by their EPR● If message delivery fails (e.g. times-out)
repeatedly for a consumer – it will be blacklisted● System doesn’t try to send subsequent messages
to black-listed consumers (for a configurable time period)
● Facts ● Minimize the overhead incurred by message delivery
failures
WS-Messenger
● Part of NSF funded “Open Grid Computing Environments – OGCE” project and Full Opensource.
● Implements WS-Eventing and WS-Notifications.
● Supports Topic based and Xpath Based Subscriptions
● New version works on Axis2 http://ws.apache.org/axis2/
● Multiple deployment options● Standalone distribution● Embedded in Servlet Container ( e.g. Tomcat)
Future Directions
● Improved static thread pool based parallelization
● Ensure equal thread utilization ● Implement a work stealing mechanism or hand
over jobs to idle threads.
● Analyze performance impact on parallelization strategies when number consumers are increased.● Memory requirements ● Throughput, round trip time
Important Info
● Open Grid Computing Environments (www.collab-ogce.org) provides a SOA based workflow suite for scientific use cases.
● WS-Messenger ● http://www.collab-
ogce.org/ogce/index.php/Messaging● http://www.collab-
ogce.org/ogce/index.php/Messaging_User_Guide
Questions?