A Gentle Introduction to OpenSplice DDS

Angelo Corsaro, Ph.D.Chief Technology Officer

OMG DDS SIG [email protected]

Delivering Performance, Openness, and Freedom

OpenSplice DDS

A Gentle Introduction

mailto:[email protected]


D e l i v e r i n g P e r f o r m a n c e , O p e n n e s s , a n d F r e e d o m

OpenSplice DDS

“Historical” Perspective

© 2009, PrismTech. All Rights Reserved

Addressing Data Distribution Challenges

The OMG DDS Standard‣ Introduced in 2004 to address the Data

Distribution challenges faced by a wide class of Defense and Aerospace Applications

‣ Key requirement for the standard were its ability to deliver very high performance while seamlessly scaling from embedded to ultra-large-scale deployments

‣ Today recommended by key administration worldwide and widely adopted across several different application domains, such as, Automated Trading, Simulations, SCADA, Telemetry, etc.

DDS is standard designed to address the data-distribution challenges across a wide class of Defense and Aerospace Applications


The OMG Data Distribution Service (DDS)

DDS v1.2 API Standard‣ Language Independent, OS and HW architecture

independent

‣ DCPS. Standard API for Data-Centric, Topic-Based, Real-Time Publish/Subscribe

‣ DLRL. Standard API for creating Object Views out of collection of Topics

DDSI/RTPS v2.1 Wire Protocol Standard‣ Standard wire protocol allowing interoperability

between different implementations of the DDS standard

‣ Interoperability demonstrated among key DDS vendors in March 2009

Object/Relational Mapping

Ownership DurabilityContent

Subscription

Minimum Profile

Data Centric Publish/Subscribe (DCPS)

Data Local Reconstruction Layer (DLRL)

DDS Interoperability Wire Protocol

Application

UDP/IP

Real-Time Publish/Subscribe Protocol

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


DDS RecommendationsIncreasingly Mandated/Recommended by Administrations

‣US Navy: Open Architecture

‣DISR/DISA: Net-centric Systems

‣EuroControl: Air Traffic Control Center Operational Interoperability

‣QinetiQ: Recommending DDS for VSI

A Great Time for DDS!

! The infrastructure evolution cycle is churning…– New -> Emerging -> Standard -> Commodity

– Middleware is emerging as OS declines

! …DDS is maturing…– OMG focus

– Wire spec

– Tools

– Enterprise integration

– Multiple products fielded

– Deployed applications!

! …and adoption is on the rise– Navy

– DISR

– FCS/SoSCOE

– Many significant applications


OpenSplice DDS

As Simple as it Gets


As Simple as it Gets‣ DDS is based around the concept

of a fully distributed Global Data Space (GDS)

Global Data Space

DDS




‣ Publishers and Subscribers can join and leave the GDS at any time

Global Data Space

DDS





Global Data Space

DDS

Subscriber

Subscriber

Subscriber

Publisher

Publisher

Publisher





‣ Publishers and Subscribers express their intent to produce/consume specific type of data, e.g., Topics

Global Data Space

DDS

Subscriber

Subscriber

Subscriber

Publisher

Publisher

Publisher






Global Data Space

DDS

Subscriber

Subscriber

Subscriber

Publisher

Publisher

Publisher






‣ Data flows from Publisher to Subscribers

Global Data Space

DDS

Subscriber

Subscriber

Subscriber

Publisher

Publisher

Publisher






‣ Data flows from Publisher to Subscribers

Global Data Space

DDS

Subscriber

Subscriber

Subscriber

Publisher

Publisher

Publisher


OpenSplice DDS

Defining Data


DDS Topics

Topic

‣ Unit of information exchanged between Publisher and Subscribers.

‣ An association between a unique name, a type and a QoS setting

struct ShapeType { long x; long y; long shapesize; string color;};#pragma keylist ShapeType color

{ShapeType}

{Circle, Square, Triangle}

{...}

Topic Type.

‣ Type describing the data associated with one or more Topics

‣ A Topic type can have a key represented by an arbitrary number of attributes

‣ Expressed in IDL


DDS Topics

Circle


Topic QoS


DDS Topics

Triangle


Topic QoS


DDS Topic Instances and Samples

Topic Instances

‣ Each key value identifies a unique Topic Instance,

‣ Topic’s instance lifetime can be explicitly managed in DDS

Topic Samples

‣ The values assumed by a Topic Instance over time are referred as Instance Sample



Topic/Instances/Samples Recap.Topics Instances

Samples

timeti tj tnow


Content Filtering

‣DDS allows to specify content-filtered Topics for which a subset of SQL92 is used to express the filter condition

‣Content filters can be applied on the entire content of the Topic Type

‣Content filters are applied by DDS each time a new sample is produced/delivered

X0 X1

Y0

Y1

X0 <= X <= X1

Y0 <= Y <= Y1


Filtering Market Data

‣ DDS continuous queries can be used for filtering the stream of data produced for a specific stock symbol.

0

10

20

30

40

Microsoft

MSFT

154.0

156.5

159.0

161.5

164.0

Apple

AAPL

MSFT > $35AAPL < $157 | | AAPL > $162


Local Queries‣ Subscribed Topics can be seen locally as “Tables”

‣ A subset of SQL92 can be used for performing queries on multiple topics as well as natural joins

Circle Topic

SELECT * FROM ShapeType s WHERE s.x > 25 AND s.y < 55

color x y shapesizeyellow 30 25 50

color x y shapesizeredblue

yellow

57 62 5090 85 5030 25 50

‣ Queries are performed under user control and provide a result that depends on the current snapshot of the system, e.g., samples currently available

X0

Y0Y0

Y1

Y0 <= Y <= Y1


OpenSplice DDS

Organizing Data


DDS Partitions

‣ All DDS communication is happens within a Domain

‣ Domain can divided into Partitions

‣ Topics are published and subscribed across on or more Partitions

A

B

CD

E

F

m

J

K

Publisher

SubscriberPublisher

Publisher

Subscriber

Subscriber

DomainPartition


OpenSplice Network Partitions‣ OpenSplice DDS allows to

define network partitions along with DDS partitions

‣ Network partitions are bound to a list of unicast/multicast network addresses

‣ Partition.Topic combination can be mapped into OpenSplice DDS Network Partitions

‣ Wildcards can be used when defining the mapping, and in case of multiple matches OpenSplice DDS will always consider the best match

A

B

CD

E

F

m

J

K

Publisher

SubscriberPublisher

Publisher

Subscriber

Subscriber

"Red"

"Green"

"Yellow"

Yellow.* => 224.1.1.1 Green.* => 224.1.1.2Green.E => 224.1.1.3


OpenSplice DDS

Quality of Service


Anatomy of a DDS Application

1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

Instances1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

Instances1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

InstancesDataReaderDataReaderDataReaderDataReaderDataReaderDataReaderDataWriterDataWriterDataWriterDataWriter

struct TempSensor { int tID; float temp; float humidity; };#pragma keylist TempSensor tID



1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

Instances1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

Instances1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic


Partition


PublisherSubscriber

Arrows show structural relationships, not data-flows



1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

Instances1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic

Instances1 21 62

2 20 61

3 25 70 25

1 22 62 1 23 63

2 19 60

3 25 71 3 25 74 3 26 77

Samples

Topic


PublisherSubscriberPartition

Domain

Domain Participant


Arrows show structural relationships, not data-flows


QoS Model

‣ QoS-Policies are used to control relevant properties of OpenSplice DDS entities, such as:‣ Temporal Properties

‣ Priority

‣ Durability

‣ Availability

‣ ...

‣ Some QoS-Policies are matched based on a Request vs. Offered Model thus QoS-enforcement

‣ Publications and Subscriptions match only if the declared vs. requested QoS are compatible ‣ e.g., it is not possible to match a publisher which delivers data unreliably with a subscriber which requires reliability

Publisher

DataWriter

Topic

Type

QoS

Name

writes

QoS

DataWriter

Topic

Typewrites

Subscriber

DataReaderreads

DataReaderreads

...

QoS

Name

QoS

QoS QoS

QoS matching

......

QoS QoS

Type Matching

DomainParticipant DomainParticipant

QoS QoS


QoS Policies

‣ Rich set of QoS allow to configure several different aspects of data availability, delivery and timeliness

‣ QoS can be used to control and optimize network as well as computing resource

QoS Policy Applicability RxO Modifiable

DURABILITY

DURABILITY SERVICE

LIFESPAN

HISTORY

PRESENTATION

RELIABILITY

PARTITION

DESTINATION ORDER

OWNERSHIP

OWNERSHIP STRENGTH

DEADLINE

LATENCY BUDGET

TRANSPORT PRIORITY

TIME BASED FILTER

RESOURCE LIMITS

USER_DATA

TOPIC_DATA

GROUP_DATA

T, DR, DW Y N Data AvailabilityT, DW N N

Data Availability

T, DW - Y

Data Availability

T, DR, DW N N

Data Availability

P, S Y N Data DeliveryT, DR, DW Y N

Data Delivery

P, S N Y

Data Delivery

T, DR, DW Y N

Data Delivery

T, DR, DW Y N

Data Delivery

DW - Y

Data Delivery

T, DR, DW Y Y Data TimelinessT, DR, DW Y Y

Data Timeliness

T, DW - Y

Data Timeliness

DR - Y Resources

T, DR, DW N N

Resources

DP, DR, DW N Y ConfigurationT N Y

Configuration

P, S N Y

Configuration

Publisher

DataWriter

Topic

Type

QoS

Name

writes

QoS

DataWriter

Topic

Typewrites

Subscriber

DataReaderreads

DataReaderreads

...

QoS

Name

QoS

QoS QoS

QoS matching

......

QoS QoS

Type Matching

DomainParticipant DomainParticipant

QoS QoS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Mapping QoS

Data Latency

LatencyBudget TransportPriority

Throughput

DeadlineTimeBasedFilter

Control over Latency/Throughput tradeoff Control over data latency Control over data priority

Which properties does QoS controls?

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Mapping QoS

Data Latency


Throughput



Data Availability

OwnershipOwnershipStrength

Durability

History

Lifespan

Control over data queueing Control over data persistency Control over data sources hot-swap


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Mapping QoS

Data Latency


Throughput



Data Availability

OwnershipOwnershipStrength

Durability

History

Lifespan

Control over data queueing Control over data persistency Control over data sources hot-swap

Data Delivery

Reliability

DestinationOrder

Presentation

Control over data distribution reliability Control over data ordering Control over presentation

OpenSplice DDS provides programmatic QoS-driven support for configuring the most important properties of data distribution!



ReliabilityThe reliability with which data is delivered to applications is impacted in DDS by the following qualities of service

‣ RELIABILITY‣ BEST_EFORT

‣RELIABLE

‣ HISTORY‣ KEEP_LAST (K)

‣ KEEP_ALL

‣ Theoretically, the only way to assure that an application will see all the samples produced by a writer is to use RELIABLE+KEEP_ALL. Any other combination could induce to samples being discarded on the receiving side because of the HISTORY depth


Real-TimeThe real-time properties with which data is delivered to applications is impacted in DDS by the following qualities of service:

‣ TRANSPORT_PRIORITY

‣ LATENCY_BUDGET

‣ In addition, DDS provides means for detecting performance failure, e.g., Deadline miss, by means of the DEADLINE QoS

‣ Given a periodic task-set {T} with periods Di (with Di < Di+1) and deadline equal to the period, than QoS should be set as follows:

‣ Assign to each task Ti a TRANSPORT_PRIORITY Pi such that Pi > Pi+1

‣ Set for each task Ti a DEADLINE QoS of Di

‣ For maximizing throughput and minimizing resource usage set for each Ti a LATENCY_BUDGET QoS between Di /2 and Di/3 (this is a rule of thumb, the upper bound is Di-(RTT/2))

Publisher Subscriber

DeadlineDeadlineDeadlineDeadlineDeadline

Deadline Violation

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


High Availability‣ A Topic can have Shared or Exclusive Ownership

‣ Exclusively owned Topics can be modified by a single writer

‣ Writer strength is used to coordinate replicated writers

R1

R2

R3

symbol: "MSFT"name: "Microsoft Corp."exchange: "NASD"quote: 33.73

StockQuote

symbol: "GOOG"name: "Google Inc."exchange: "NASD"quote: 663.97

StockQuote

W1

W2’’

STRENGTH=3

STRENGTH=1


StockQuote


StockQuotesymbol: "AAPL"name: "Apple Inc."exchange: "NASD"quote: 165.37

StockQuote

W1’

STRENGTH=2


StockQuote



StockQuote


StockQuote


StockQuote

symbol: "AAPL"name: "Apple Inc."exchange: "NASD"quote: 165.37


StockQuote


StockQuote

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.





R1

R2

R3


StockQuote


StockQuote

W1

W2’’

STRENGTH=3

STRENGTH=1


StockQuote


StockQuote

W1’

STRENGTH=2


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.





R1

R2

R3


StockQuote


StockQuote

W1

W2’’

STRENGTH=3

STRENGTH=1


StockQuote

W1’

STRENGTH=2


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


High Availability ‣ A Topic can have Shared or Exclusive Ownership




StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote

R1

R2

R3W2’’

STRENGTH=1

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


High Availability ‣ A Topic can have Shared or Exclusive Ownership




StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuote


StockQuoteR1

R2

R3W2’’

STRENGTH=1

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Eventual Consistency & R/W Caches

Under an Eventual Consistency Model, DDS guarantees that all matched Reader Caches will eventually be identical of the respective Writer Cache

Topic

DataReader

1 1

2 1

3 1

DataReader Cache Topic

1 2

2 3

3 1

DataWriter Cache

DataWriter

1 2

2 2 2 3

Topic

DataReader

1 1

2 1

3 1

DataReader Cache

Topic

DataReader

1 1

2 1

3 1

DataReader Cache

DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


QoS Impacting the Consistency Model

The DDS Consistency Model is a property that can be associated to Topics or further refined by Reader/Writers. The property is controlled by the following QoS Policies:

‣ DURABILITY ‣ VOLATILE | TRANSIENT_LOCAL | TRANSIENT | PERSISTENT

‣ LIFESPAN

‣ RELIABILITY‣ RELIABLE | BEST_EFFORT

‣ DESTINATION ORDER‣ SOURCE_TIMESTAMP | DESTINATION_TIMESTAMP

QoS Policy Applicability RxO Modifiable

DURABILITY

LIFESPAN

RELIABILITY

DESTINATION ORDER

T, DR, DW Y N

T, DW - Y

T, DR, DW Y N

T, DR, DW Y N

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


QoS Impacting the Consistency Model

DURABILITY RELIABILITY DESTINATION_ORDER LIFESPAN

Eventual Consistency (No Crash / Recovery)

Eventual Consistency (Reader Crash / Recovery)

Eventual Consistency(Crash/Recovery)

Eventual Consistency(Crash/Recovery)

Weak Consistency

Weak ConsistencyWeak Consistency

VOLATILE RELIABLE SOURCE_TIMESTAMP INF.

TRANSIENT_LOCAL RELIABLE SOURCE_TIMESTAMP INF.

TRANSIENT RELIABLE SOURCE_TIMESTAMP INF.

PERSISTENT RELIABLE SOURCE_TIMESTAMP INF.

ANY ANY DESTINATION_TIMESTAMP ANY

ANY BEST_EFFORT ANY ANY

ANY ANY ANY N

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Eventual Consistency @ Work

A

B

CD

E

F

m

J

K

P1

P2

S1P = {A, B}

S = {A, D}

P = {D, C, J}

S4

S = {A}

Eventual Consistency (Reader Crash / Recovery)Eventual Consistency(Crash/Recovery)




{A}

{B}

Weak Consistency ANY ANY ANY N {J}

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K

A

P1

P2

S1P = {A, B}

S = {A, D}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K

A

P1

P2

S1

A

P = {A, B}

S = {A, D}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



B A

B

CD

E

F

m

J

K

A

P1

P2

S1

A

P = {A, B}

S = {A, D}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K

A

P1

P2

S1B

A

P = {A, B}

S = {A, D}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K

A

P1

P2

S1

S2

B

A

B

A

P = {A, B}

S = {A, D}

S= {A, B, J}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K

A

P1

P2

S1

S2

B

A

B

A

P = {A, B}

S = {A, D}

S= {A, B, J}

P = {D, C, J}B

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

KP2

S1

S2

A

BA

BB

S = {A, D}

S= {A, B, J}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

KP2

S1

S2

S3

A

BA

B

S= {A, B, D, J}

B

S = {A, D}

S= {A, B, J}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

KP2

S1

S2

S3

A

BA

B

S= {A, B, D, J}

J

B

D

J

S = {A, D}

S= {A, B, J}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

KP2

S1

S2

S3

A

BA

B

S= {A, B, D, J}

B

D

J

S = {A, D}

S= {A, B, J}

P = {D, C, J}

S4

S = {A}

A





{A}

{B}


Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Design Guidelines

‣ For all (non-periodic) Topics for which an eventually consistent model is required use the following QoS settings:

‣ For information produced periodically, with a period P, where P is small enough to be acceptable as a consistency convergence delay, the following QoS settings will provide an approximation of the eventual consistency:

Eventual Consistency (Crash / Recovery)



Eventual Consistency (Crash / Recovery)

VOLATILE BEST_EFFORT SOURCE_TIMESTAMP INF.



OpenSplice DDS

OpenSplice DDS Architecture


OpenSplice DDS Architectural Outlook

Architectural Highlights

‣ Shared-Memory based architecture for minimizing intra-nodal latency, as well as maximizing nodal scalability

‣ Pluggable Service Architecture

‣ Full control over network scheduling


Real-Time Networking Technology

Scalability and Efficiency

‣ Single shared library for applications & services

‣ Ring-fenced shared memory segment

‣ Data urgency driven network-packing

Architecture

‣ Network-channels

‣ Priority bands

‣ Network-partitions

‣ Multicast Groups

‣ Traffic-shaping‣ Burst/Throughput

Determinism & Safety

‣ Preemptive network-scheduler

‣ Data importance based network-channel selection

‣ Partition based multicast-group selection

‣ Managed critical network-resource

Traffic Shaping

OpenSplice DDSBinding

Networking

Shared

Memory


Networking

Shared

Memory

Network Channels

Priority Bands

Pre-emptive Network Scheduler

Priority Scheduler

Data Urgency Traffic Pacing

Single Copy per Node

Pack Across Topics/Applications

Optimal Unmarshaling

Fault-Tolerance

‣ Active Channels

‣ Fall back on next highest priority active channel


Durable Data Technology

Goal

‣ Transient QoS. Keep state-data outside the scope/lifecycle of its publishers

‣ Persistence QoS. Keep persistent settings to outlive the system downtime

Architecture

‣ Fault-Tolerant Data Availability‣ Transient -- on memory

‣ Persistent -- on disk

‣ Partitioning‣ DDS Partitions

‣ Alignment‣ Dedicated Channels

Features

‣ Fault-tolerant availability of non-volatile data

‣ Efficient delivery of initial data to late-joining applications

‣ Pluggable Durability Service

‣ Automatic alignment of replicated durability-services


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Dedicated Persistence Service Alignment Channel

Persist Partitions

Persistent Data on Local Disk

Transient Data in Memory

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Durability Configuration

A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.



A

B

CD

E

F

m

J

K


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


Networking

Shared Memory


Durability

Disk


OpenSplice DDS

Product Organization

Enterprise Ed.

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


OpenSplice DDS

‣ Product reorganized into Editions‣ Community Edition

‣ Compact Edition

‣ Professional Edition

‣ Enterprise Edition

‣ Product Editions provide a a growing set of functionality to address the needs of increasingly more sophisticated users

‣ The Community Edition is Open Source

‣ Compact, Enterprise and Professional Edition are available only through Commercial Subscriptions

Enterprise Ed.

Community Ed.

Compact Ed.

Professional Ed.

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Community EditionFeatures

‣ OMG DDS v1.2 DCPS‣Minimum Profile

‣ Content Subscription Profile

‣ Durability Profile

‣ Ownership Profile

‣ Networking‣ DDSI v2.1 Implementation

‣ Real-Time Networking Implementation


Subscription

Minimum Profile

Real-Time Pub/Sub (DCPS)

Networking Technology

Application

UDP/IP

Interoperable Wire Protocol (DDSI)

DD

SI v

2.1

DD

S v

1.2

Real-Time Networking

Open Source LGPL Licensed Edition -- The Best Way to Get Started with DDS

Licensing

‣ LGPLv3

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Compact EditionFeatures

‣ OMG DDS v1.2 DCPS‣Minimum Profile






‣ PowerTools‣MDE PowerTools

‣Tuner


Subscription

Minimum Profile



Application

UDP/IP


DD

SI v

2.1

DD

S v

1.2


PowerTools

MDE

Tuner

Commercially Supported, Boosting Your Development with PowerTools

Licensing

‣ Commercial + PrismTech Source Code License

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Professional EditionFeatures

‣ OMG DDS v1.2 DCPS+DLRL‣Minimum Profile




‣ Data Local Reconstruction Layer



‣ PowerTools‣MDE Power Tools

‣ Tuner



Subscription

Minimum Profile


Object-Oriented Pub/Sub (DLRL)

Application

UDP/IP

DD

SI v

2.1

DD

S v

1.2

SOAP

Connectors

PowerTools

MDE

Tuner




Unleash the Full Power of the DDS Technology!

Licensing


‣ Connectors‣ SOAP Connector

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Enterprise EditionFeatures

‣ OMG DDS v1.2 (DCPS+DLRL)‣ Minimum Profile




‣ Data Local Reconstruction Layer



‣ Spiked Absorber

‣ Secure Networking

‣ PowerTools‣ MDE Power Tools

‣ Tuner


Ownership PersistenceContent

Subscription

Minimum Profile

Real-Time Pub/Sub(DCPS)

Object-Oriented Pub/Sub (DLRL)


Application

UDP/IP


DD

SI v

2.1

DD

S v

1.2


SOAPSecure Networking

DBMS

Spike Absorber

Connectors

PowerTools

MDE

Tuner

Connectors

Licensing


Connect Your Enterprise Seamlessly and in Real-Time!

‣ Connectors‣ SOAP Connector

‣ DBMS Connector


OpenSplice DDS

Performance

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Performance on Commodity HW

!"#$%&#'

()(!%(*'

&")&%+)'

&!(,%!)'

),,%$#'

!,*%))'(("%!"'

&&)%*$' $"%"$' ,+%($',($%!+'

,*+%&)'

$&"%,,'

*!#%,&'

)$,%$*'

)"#%*)'

"!+%++'

"*(%)+' ")(%&!' ""&%,&'

,++%++'

!++%++'

$++%++'

#++%++'

*++%++'

)++%++'

"++%++'

&+++%++'

&&++%++'

(+%++'

&+(+%++'

(+(+%++'

,+(+%++'

!+(+%++'

$+(+%++'

#+(+%++'

)' &#' ,(' #!' &()' ($#' $&(' &+(!' (+!)' !+"#'

!"#$%

&'$($)$*+%

!*$$,(*%-./*%

-./0/1/23' 456/'

Test Scenario

‣ Single Threaded Application (multi-threaded networking service)

‣ 8192 bit message batches

Throughput Latency

Inter-Node Latency

‣ 60 usec

Inter-Core Read-Latency

‣ 2 usec

Inter-Core Latency

‣ <10 usec

HW:

‣ Dell blade-server

‣ Dual-core, Dual-CPU, AMD Opteron 2.4 GhzOS

‣ Linux 2.6.21-1.3194.fc7Network

‣ Gigabit Ethernet cards

‣ Dell PowerConnect 5324 switch


OpenSplice DDS

Use Cases

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


European Flight Data Processor

‣ Large program to replace existing Flight Data Processors (FDPs)‣ 5 Centers in France‣ 4 Centers in Italy‣ 2 Centers in Switzerland

ATCCBrest

ATCCReimsATCC

Paris

ATCCMilano

ATCCBrindisi

ATCCAix-en-

Provence

ATCCBordeaux

ATCCRoma

ATCCPadova

ATCCGeneve

ATCCZurich

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


DDS in CoFlight -- FDP Core

‣OpenSplice DDS glues together the most critical components of the CoFlight FDP running at a SWAL-2 (same as DO-178B Level B) assurance level

‣ In this context OpenSplice DDS distributes flights data plans of redundant LANs

DDS

Flight Data Processing Servers

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


DDS in CoFlight -- CWP

‣ OpenSplice DDS is used within CoFlight to distribute the “external” Flight Data Plan to Controller Working Positions

DDS


DDS

Controllers

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


DDS in CoFlight -- IOP‣ OpenSplice DDS is used

to integrate CoFlight-based Centers

‣ OpenSplice DDS is used to provide interoperability with other Interoperable Centers (as per ICOG-2)

DDS

Controllers


Air Traffic Control Center

DDS

Controllers



DDS

Controllers



A

B

CD

E

F

m

J

K

DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Some Other Use CasesDefense & Aerospace

‣ Combat Management Systems

‣e.g. THALES, Northrop Grumman

‣ Vetronics

‣e.g. European SI

‣ Tactical Links

‣e.g. ULTRA

‣ Simulation ‣ e.g. MIT Lincoln Labs

SCADA/Utilities

‣ Industrial Automation ‣ e.g. Chemtech

‣ Rocket Launch Systems (Telemetry)‣ e.g. NASA

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Some Use Cases

Transportation

‣ Drones‣e.g. Navcom

Financial Services

‣ Automated Trading Firms‣e.g. ThinkTrade


OpenSplice DDS

Concluding Remarks


OpenSplice DDS

AMQP vs DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Scope of Standardization



Subscription

Minimum Profile




Application

UDP/IP


DD

SI

v2

.1D

DS

v1

.2

Interoperability Wire Protocol

(TCP, SCTP, UDP)/IP

Advanced Message Queuing Protocol

AM

QP

v0

.10

API (?)

Application

AMQP OMG DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.





Subscription

Minimum Profile




Application

UDP/IP


DD

SI

v2

.1D

DS

v1

.2


(TCP, SCTP, UDP)/IP


AM

QP

v0

.10

API (?)

Application

Wire Protocol Standard

AMQP OMG DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.





Subscription

Minimum Profile




Application

UDP/IP


DD

SI

v2

.1D

DS

v1

.2


(TCP, SCTP, UDP)/IP


AM

QP

v0

.10

API (?)

Application


Pub/Sub Standard API

AMQP OMG DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.





Subscription

Minimum Profile




Application

UDP/IP


DD

SI

v2

.1D

DS

v1

.2


(TCP, SCTP, UDP)/IP


AM

QP

v0

.10

API (?)

Application


Pub/Sub Standard API

Object Relational Mapping Standard API

AMQP OMG DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Architectural Style

Publisher

SubscriberPublisher

Publisher

Subscriber

Subscriber

Broker

OMG DDS

Publisher

SubscriberPublisher

Publisher

Subscriber

Subscriber

Broker

Broker

Broker

Broker

Broker

AMQP

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Key Differences in Summary

‣ Brokered Communication Architecture

‣ Peer-to-Peer Communication Architecture

AMQP

‣ Wire Protocol Standard‣ Interoperability across AMQP

implementations

‣ Wire Protocol + API Standard‣ Interoperability across DDS implementations

‣ Application Portability across DDS Implementation

‣ Header Based Routing ‣ Content-Based Filtering/Quering

‣ Not geared for High-Performance ‣ Designed bottom-up for maximum performance and determinism

DDS


OpenSplice DDS

29West vs DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


29West vs DDS



29West

‣ Proprietary technology ‣ Wire Protocol + API Standard‣ Interoperability across DDS implementations

‣ Application Portability across DDS Implementation

‣ Topic are blobs of data

‣ No content-based routing‣ Topics have Types and Qos

‣ Content-Based Filtering/Quering

‣ Designed for High-Performance ‣ Designed bottom-up for High-Performance, Determinism, Reliability

DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


29West vs DDS#include <lbm.h>

int main(int, char**) {

lbm_context_t *ctx; lbm_topic_t *topic; lbm_src_t *src; int lbm_failed;lbm_failed = lbm_context_create(&ctx, NULL, NULL, NULL);/* Create a Topic*/lbm_failed = lbm_src_topic_alloc(&topic, ctx, "Greeting", NULL);! /* Create a source*/lbm_failed = lbm_src_create(&src, ctx, topic, NULL, NULL, NULL);!SLEEP(3);! /* Send data*/lbm_failed = lbm_src_send(src, "Hello!", 6, LBM_MSG_FLUSH | LBM_SRC_BLOCK);/* Finished all sending to this topic, delete the source object. */ lbm_src_delete(src);/* Do not need to delete the topic object - LBM keeps track of topic * objects and deletes them as-needed. *//* Finished with all LBM functions, delete the context object. */ lbm_context_delete(ctx);return 0;

}

#include <dds/dds.hpp>

int main(int, char**) {

dds::Runtime::start();/* Create a Topic with default QoS. */ dds::Topic<dds::StringTopicType> strTopic(“Greeting”);/* Create a Writer with default QoS. */dds::Writer<dds::StringTopicType> writer(strTopic);/* Create a sample and write data. */dds::StringTopicType msg(“Hello”);writer.write(msg);dds::Runtime::stop();

}

29West DDS

Pro

prie

tary

Info

rmat

ion

- D

istri

butio

n w

ithou

t Exp

ress

ed W

ritte

n P

erm

issi

on is

Pro

hibi

ted.


Concluding Remarks‣ OpenSplice DDS is the ideal fit for addressing

several of the data distribution and management challenges faced by financial firms, such as:‣ Market Data Distribution

‣ High Performance Caching

‣ Real-Time Business Intelligence

‣ OpenSplice DDS has great performance on standard COTS HW and its architecture is designed for exploiting at best high-end multi-cores such as the Intel XEON 5500

‣ OpenSplice DDS is Commercially Supported Open Source Software

Delivering Performance, Openness, and Freedom

OpenSplice DDS

Enterprise Ed.

Community Ed.

Compact Ed.

Professional Ed.

Embrace Performance, Openness and Freedom. Join the OpenSplice DDS Community at www.opensplice.org

http://www.opensplice.org

http://www.opensplice.org

Online Resources

http://www.opensplice.com/

emailto:[email protected]

http://www.youtube.com/OpenSpliceTube http://opensplice.blogspot.com

http://bit.ly/1Sreg


http://www.slideshare.net/angelo.corsaro

http://twitter.com/acorsaro/

http://www.prismtech.com/opensplice-dds








http://bit.ly/1Sreg

http://bit.ly/1Sreg



