Fault-Tolerant Reliable Delivery of Messages in Distributed Publish/Subscribe Systems

Shrideep Pallickara, Hasan Bulut & Geoffrey FoxCommunity Grids Lab

Indiana University

Reliable Delivery: Desiderata• Cope with node/link failures & unpredictable

links Links may duplicate, garble and lose messages

• Many-to-Many reliable delivery across sessions Support recovery from failures or disconnects Support replays

• Transport-independent• Support exactly-once delivery

Order, duplicate detection• Authorized reliable delivery

Reliable Delivery

RA

RDEFRB

RLM

Publishers or Subscribers

RX Repository for a reliable-topic X

· Persist Events· Maintain Registered Entities· Track Delivery Sequences

• One Repository associated with a reliable topic

• A Repository can manage multiple reliable topics

Reliable DeliveryP

UB

LIS

HE

R

RE

PO

SIT

OR

YP2R-Retransmit

R2P-NAK

R2P-ACK

P2R-Order

SU

BS

CR

IBE

R

R2S*-Persistent

S2R-ACK

R2S-Sync

R2S-RectifyS2R-NAK

R2S-Retransmit

Regular ExchangeError DetectionError Correction

Reliable delivery: Advantages• Reliable delivery ONLY for authorized entities

Coexists with entities not interested in reliable delivery• Storage, is not communal, and should be

provisioned by the topic owner. • Control Messages are issued over different topics.

Discovery constraints can be imposed e.g. Restrict replays• Different QoS can be associated with control topics.

Require signed acknowledgements (Non-repudiation) Buffering & Jitter reduction services for replayed messages.

• Streams replayed at say 24 fps instead of 500 fps• Easy to maintain audit trails

Track client loss rates, NAKs, disconnects & recoveries• Lends itself naturally for greater redundancy

{Z}

Repository Redundancy

WX

Y

Z

RA2

RA1RA3

RA# Repository for a reliable-topic A

RA3

RA4

{W,X}

{Y}

{RA1 , RA2 , RA3 , RA4} – Repository bundle for reliable topic A

Repository Redundancy• Multiple repositories constitute a repository bundle

A given repository can be part of multiple bundles• Associate a repository-bundle for a given topic for

greater redundancy Sustain repository failures and downtimes

• Fine tune redundancy associated with a bundle Graceful addition & removal of constituent repositories

• Clients leverage network proximity through bundle By choosing a closer repository, communication latencies

are reduced. Retransmissions and recoveries are faster.• Repository with which a client actively interacts with

is its steering repository.• Set aside repositories for recovery and replays

Repository Bundle: Publisher

Repo

sito

ry

Publ

ishe

r

Repo

sito

ryRe

posi

tory

P2R-Retransmit

R2P-NAKR2P-ACK

P2R-Order

P2R-Order

P2R-Order

P2R-Order

R2P-ACK

R2P-NAK

P2R-Retransmit

P2R-Retransmit

P2R-Retransmit

Repository Bundle: Subscriber

Repo

sito

ryRe

posi

tory

Subs

crib

er

R2S*-PersistentS2R-ACK

R2S-Sync

R2S-Rectify

R2S-Retransmit

R2S*-Persistent

S2R-ACK

R2S-Sync

R2S-RectifyS2R-NAK

R2S-Retransmit

S2R-ACK

S2R-ACKRe

posi

tory

S2R-NAK

0 2 4 6 8

10 12 14 16

100 1000 10000

Tim

e (M

illise

cond

s)

Payload Size (Bytes)

Delivery overheads in different Topologiesfor different message payload sizes

3 Brokers, Best effort 3 Brokers, 1 Repository

3 Brokers, 3 Repositories

Ongoing Activities• Repository placement schemes

Reduction of publisher & subscriber overheads Facilitate faster recoveries and error-corrections Dedicated repositories for use in replays

• Current Deployment eSports System – To facilitate recording,

annotation and replays of multimedia stream• Release Schedule

Will be released as part of NaradaBrokering 3.0 in June 2007