Upload
gerardo-pardo-castellote
View
2.365
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Citation preview
Your systems. Working as one.
Introduc)on to OMG DDS
OMG Technical Mee9ng, Washington D.C., March 2013 Gerardo Pardo-‐Castellote, Ph.D. [[email protected]] CTO, Real-‐Time Innova9ons, Inc. [www.r9.com] Co-‐chair of the OMG Data-‐Distribu9on SIG
Systems that interact with the Real World
• Must adapt to changing environment • Cannot stop processing the informa9on • Live within world-‐imposed 9ming
Beyond tradi9onal interpreta9on of real-‐9me
2 © 2012 RTI • ALL RIGHTS RESERVED
3
Challenge: More Data, More Speed, More Sources
TRENDS: • Growing Informa9on Volume • Lowering Decision Latency • Increasing System Availability • Accelera9ng technology inser9on and deployment
Next-‐genera)on systems needs: • Scalability • Integra9on & Evolu9on • Robustness & Availability • Performance • Security
3 © 2012 RTI • ALL RIGHTS RESERVED
“Real World” Systems are integrated using a Data Model
• Grounded on the “physics” of the problem domain – Tied to the nature of the sensors and real objects in the system (vehicles,
device types, …)
• Provides governance across disparate teams & organiza9ons – The “N^2” integra9on problem is reduced to a “N” problem
• Increased decoupling from use-‐cases and components – Avoids over constraining applica9ons
• Open, Evolvable, Plaborm-‐Independent – The use-‐cases, algorithms might change between missions or versions of
the system
Realizing this data-‐model requires a middleware infrastructure
App App App
4 © 2012 RTI • ALL RIGHTS RESERVED
DDS: Standards-‐based Integra9on Infrastructure for Cri9cal Applica9ons
Streaming Data
Sensors Events
Real-‐Time Applica)ons
Enterprise Applica)ons
Actuators
5 © 2012 RTI • ALL RIGHTS RESERVED
RPC over DDS
2013
Family of Specifica9ons
6
DDS Implementa9on
Network / TCP / UDP / IP
App
DDS Implementa9on
App
DDS Implementa9on DDS
Interoperablity
2006
UML Profile for DDS
2008
DDS for Lw CCM
2009
DDS X-‐Types
2010
DDS Security
2013 2010
DDS-‐STD-‐C++ DDS-‐JAVA5
App
6 © 2012 RTI • ALL RIGHTS RESERVED
DDS Spec
2004
7
Broad Adop9on
• Vendor independent – API for portability – Wire protocol for interoperability
• Mul9ple implementa9ons – 10 of API – 8 support RTPS
• Heterogeneous – C, C++, Java, .NET (C#, C++/CLI), ADA, Python, Scala, …
– Linux, Windows, VxWorks, Integrity, other embedded & real-‐9me
• Loosely coupled
DDS Real-‐Time Publish-‐Subscribe Wire Protocol (RTPS)
Middleware
DDS API
Cross-‐vendor portability!
Cross-‐vendor interoperability!
© 2012 RTI • ALL RIGHTS RESERVED
8
Interoperability between the applica)ons implemented by six different vendors (March 2012)
OCI ETRI PrismTech IBM RTI TwinOaks
DDS mandated by key DoD Programs • UK Generic Vehicle Architecture
– Mandates DDS for vehicle comm. – Mandates DDS-‐RTPS for interop.
• DISR – Mandates DDS for Pub-‐Sub API – Mandates DDS-‐RTPS for Interop
• Army, OSD – UCS, Unmanned Vehicle Control – FACE (Future Airborne Capability
Environment) • US Navy Open Architecture
– Mandates DDS for Pub-‐Sub • SPAWAR NESI
– Mandates DDS for Pub-‐Sub SOA
9 © 2012 RTI • ALL RIGHTS RESERVED
DDS Deployed Applica9on Examples
© 2012 RTI • ALL RIGHTS RESERVED 10
Everyday Example: Calendaring
Alterna)ve Process #1 (message-‐centric): 1. Email: “Mee9ng Monday at 10:00.”
2. Email: “Here’s dial-‐in info for mee9ng…”
3. Email: “Mee9ng moved to Tuesday”
4. You: “Where do I have to be? When?”
5. You: (siFing through email messages…)
11 © 2012 RTI • ALL RIGHTS RESERVED
Example: Calendaring
Alterna)ve Process #2: 1. Calendar: (add meeIng Monday at 10:00) 2. Calendar: (add dial-‐in info) 3. Calendar: (move meeIng to Tuesday)
4. You: “Where do I have to be? When?” 5. You: (check calendar. Contains
consolidated-‐state)
The difference is state! The infrastructure consolidates changes and maintains it
12 © 2012 RTI • ALL RIGHTS RESERVED
Copyright © 2010 RTI -‐ All rights Reserved. . 13
Data-‐Centric Middleware allows applica9ons to be integrated to the Informa9on Model
APP APP APP APP
DDS Global Data Space
Data Model
Standard Mapping(*)
Standard API
No custom mappings / code necessary Direct support for data-‐centric ac9ons: create, dispose, read/take
Copyright © 2010 RTI -‐ All rights Reserved. . 14
Integra9ng components to generic middleware technology
Comp Comp Comp Comp
Middleware Ar)facts
Data Model
Custom Mapping
Custom Integra9on
Akin to implemen9ng an OO design on a Procedural Language: Requires mapping inheritance, encapsula9on, excep9ons, …
Tradi9onal data-‐centric technologies not suited to scalable near real-‐9me systems
Other data-‐centric technologies: – Databases: SQL – Web: HTTP (mostly)
• …assume the world changes slowly • …use network resources inefficiently • …are highly centralized
State
Server App
App
App App
App
App
Slow A few updates/sec
Not scalable 100 apps => 100x load
Unreliable Failure here kills many apps 15 © 2012 RTI • ALL RIGHTS RESERVED
DDS is decentralized. Can be deployed without servers/brokers
DDS: • …allows you to observe frequent changes • …uses network resources efficiently • …is peer-‐to-‐peer and decentralized
App App App App App App
Fast 100,000’s updates/sec
Scalable Load indep. # apps
Reliable No single pt. failure
Managed with QoS
DDS Data-‐Centric Bus
16 © 2012 RTI • ALL RIGHTS RESERVED
The data-‐centric communicaIons model
© 2012 RTI • ALL RIGHTS RESERVED 17
DDS Adressing: Data-‐Objects in the Global Data Space
• Domain: world you’re talking about • Topic: group of similar objects
– Similar structure (“type”) what – Similar way they change when
over 9me (“QoS”) how • Instance: individual object
• DataWriter: source of observa9ons about a set of data-‐objects (Topic)
• DataReader: observer of a set of data-‐objects (Topic)
Domain (e.g. Yellowstone Park)
Topic (e.g. bears in the park)
Topic Snow Depth Sensors
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
ID GPS value
Instance (e.g. Yogi the bear)
Booboo Instance
Data-‐Centric Qos-‐Aware Pub-‐Sub Model
Persistence Service
Recording Service
Virtual, decentralized global data space
CRUD opera9ons
Source (Key) Speed Power Phase
WPT1 37.4 122.0 -12.20
WPT2 10.7 74.0 -12.23
WPTN 50.2 150.07 -11.98
19 © 2012 RTI • ALL RIGHTS RESERVED
Demo: Publish-‐Subscribe ShapesDemo
20 © 2012 RTI • ALL RIGHTS RESERVED
Data Reader “Alarm”
Domain Par9cipant
Data Writer “Alarm”
Domain Par9cipant
Data-‐Centric Communica9ons Model
• Par)cipants scope the global data space (domain) • Topics define the data-‐objects (collec9ons of subjects) • DataWriters publish data on Topics • DataReaders subscribe to data on Topics • QoS Policies are used configure the system • Listeners are used to no9fy the applica9on of events
Listener Offered QoS Listener
Got new data
Requested QoS
New subscriber!
example
21 © 2012 RTI • ALL RIGHTS RESERVED
Quality of Service (QoS)
• Aside from the actual data (WHAT) to be delivered, users ozen need to specify HOW to send it … … reliably (or “send and forget”) … how much data (all data , last 5 samples, every 2 secs) … how long before data is regarded as ‘stale’ and is discarded … how many publishers of the same data is allowed … how to ‘failover’ if an exis9ng publisher stops sending data … how to detect “dead” applica9ons … …
• These op9ons are controlled by formally-‐defined Quality of Service (QoS) policies
Real-‐Time Quality of Service Control
• Quality of Service (QoS) determined per-‐en6ty
• QoS Contract: Request -‐ Offered
• Publishing and subscribing applica9ons can be no9fied when QoS contract is violated
– e.g. Messages lost or deadlines missed
• High availability via automa9c failover
Subscriber
Subscriber
Subscriber
Collision avoidance application
Radar Track Publisher
Reliable; 1 Hz;
get history
Best effort; 0.2 HZ; get history
Airline operations center
Best effort; 0.01 Hz; no history
Database, real-time log
• Publish reliably • 10 Hz • Keep 10 minute history (6000 samples)
Backup Publisher
Real-‐Time Quality of Service (QoS)
QoS Policy DURABILITY
HISTORY
LIFESPAN
WRITER DATA LIFECYCLE
READER DATA LIFECYCLE
ENTITY FACTORY
RESOURCE LIMITS
RELIABILITY
TIME BASED FILTER
DEADLINE
CONTENT FILTERS
Cache
User Q
oS
Delivery
Presenta)on
Availability
Resources
Transport
QoS Policy USER DATA
TOPIC DATA
GROUP DATA
PARTITION
PRESENTATION
DESTINATION ORDER
OWNERSHIP
OWNERSHIP STRENGTH
LIVELINESS
LATENCY BUDGET
TRANSPORT PRIORITY
Qos in Ac9on
• Detec9ng presence
• History cache • Dele9ng objects
• Ownership • Liveliness • Filtering • Durability
© 2009 Real-‐Time Innova9ons, Inc. 25
ShapesDemo
OperaIonal robustness and performance
© 2012 RTI • ALL RIGHTS RESERVED 26
Architecture for the next-‐genera9on systems
• Exis9ng technologies are reaching robustness/performance/scalability limits
• DDS provides a fundamentally new DataBus architecture and approach – Powerful data-‐centric model
– Ultra-‐scalable and robust – Fully decentralized, peer-‐to-‐peer, “no bo}lenecks” architecture – Superior Wire Protocol – Standards-‐based, mul9-‐plaborm
Single-‐lane traffic No priori9za9on
Brokers as choke-‐points Connext DDS Approach
27 © 2012 RTI • ALL RIGHTS RESERVED
0
50
100
150
200
250
300
350
400
Average Laten
cy (M
icrosecond
s)
Throughput (Messages per Seconds)
1 (1 per CPU and NIC)
20 (1 per CPU and NIC)
40 (1 per CPU, 2 per NIC)
Performance
• Reliable mul9cast • Fully meshed, reliable
Number of Subscribers
Orders of magnitude faster than IT solu9ons
Fastest DDS solu)on
28 © 2012 RTI • ALL RIGHTS RESERVED
Performance results are vendor-‐specific These results are for RTI Connext DDS.
Scalability
1 ~1000 subscribers, < 15% throughput decrease
600,000
500,000
400,000
300,000
200,000
100,000
0
Messages pe
r Second
Pe
r Subscriber (2
00 Bytes)
0 200 400 600 800 1,000
Number of Subscribers
• Scalable Performance!!• Millions of data
elements!• .5m updates/sec
(batched)!• 10s µs latency!• 1000s consumers
per update!• Orders of magnitude more scalable than IT solu9ons
29 © 2012 RTI • ALL RIGHTS RESERVED
Scalability results are vendor-‐specific These results are for RTI Connext DDS.
Comparison with other technologies
Message Length (samples)
Adapted from Vanderbilt presenta9on at July 2006 OMG Workshop on RT Systems 30 © 2012 RTI • ALL RIGHTS RESERVED
Comparison results are vendor-‐specific These results compare with RTI Connext DDS.
Common use-‐cases / paRerns
© 2012 RTI • ALL RIGHTS RESERVED 31
Common Use Cases
• Isola9ng subsystems • Detec9ng presence of applica9ons • Discovering publishers and subscribers • Keeping a “last-‐value” cache • Monitoring the health of applica9ons • Monitoring the health of data • Reliable data delivery • Building a highly-‐available system • Managing network load and behavior
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 32
Isola9ng Subsystems
• Concept of Domains and Domain Par9cipants
N1 App 1 Pub/Sub (A,B/C,D)
N2 App 2 Subscribe (C)
N4 App 4 Pub/Sub (D/C,E,F)
N4 App 5 Publish (C)
N3 App 3 Pub/Sub (E,F/A,C)
N5 App 6 Subscribe (B,C)
Domain
Single ‘Domain’ System
• Container for applica9ons that want to communicate
• Applica9ons can join or leave a domain in any order
• New Applica9ons are “Auto-‐Discovered”
• An applica9on that has joined a domain is also called a “Domain Par9cipant”
Isola9ng Subsystems
• Use mul9ple domains tor scalability, modularity and isola9on
Node 1 -‐ App 1 Pub/Sub
Node 2 -‐ App 1 Subscribe
Node 4 -‐ App 1 Pub/Sub
Node 4 -‐ App 2 Publish
Node 3 -‐ App 1 Pub/Sub
Node 5 -‐ App 1 Subscribe
Domain A
Node 5 -‐ App 2 Pub/Sub
Node 6 -‐ App 1 Pub/Sub
Domain B
Domain C Added Func.
Mul)ple Domain System
Detec9ng presence of applica9on
• DDS has a buil9n discovery service • It provides the means for an applica9on to discover the presence of other par9cipants on the Domain – The Topic “DCPSPar9cipants” can be read as a regular Topic to see when DomainPar9cipants join and leave the network
• Applica9ons can also include meta-‐data that is sent along by DDS discovery
Discover Publishers and Subscribers
• DDS provides a mechanism to detect en99es, their capabili9es and requirements
• An applica9on can discover all the other en99es in the system that match the requirements – The Topics “DCPSPublica9ons”, “DCPSSubscrip9ons”, “DCPSTopics”, and “DCPSPar9cipants” can be read to observe the other en99es in the domain
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 36
Publishing data that outlives it source
• The concept of durability in the global dataspace – Vola9le
• No durability – Transient local
• Durability maintained by the publishing applica9on – Transient
• Durability maintained by a 3rd party applica9on (persistence service)
– Persistent • Durability of data is maintained by a 3rd party and saved to disk
• Durability maintained azer restart
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 37
Keeping a “last value” cache
• A last-‐value cache is already built-‐in into every Writer in the system
• A late joiner will automa9cally ini9alize to the last value
• Last value cache can be configure with history depth greater than 1
• The Persistence Service can be used to provide a last value cache for durable data
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 38
Monitoring the health of applica9ons
• DDS has mechanisms to monitor presence, health and ac9vity of all en99es
• Supports a concept of liveliness – Automa9c
• Managed by the infrastructure – Manually asserted
• Managed by the applica9on
• What does it mean if the applica9on no longer publishes data? – No news, good news? – Applica9on has crashed? – Applica9on is disconnected?
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 39
Monitor the health of data-‐objects
• Concept of deadline – DDS can monitor the ac9vity of each individual data-‐instance in the system
– If an instance is not updated according to the requirements of the receiving applica9on, the applica9on is no9fied
– Trigger for built-‐in failover mechanism
• Concept of lifespan – DDS understands if a data-‐object has outlived its purpose and is considered ‘stale’ data
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 40
Ensuring a reliable data delivery
• DDS supports a transport independent efficient reliability protocol – In-‐order delivery – Reliable delivery – Detec9on and no9fica9on of data loss – Very configurable
• Warning thresholds
• Heartbeats, gaps, acks, nacks, blocking or non-‐blocking behaviour
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 41
Building a high-‐available system
• High Availability has mul9ple facets, many of which can be handled by DDS – Detec9on of presence
• DISCOVERY – Detec9on of health and ac9vity
• LIVELINESS, DEADLINE, LIFESPAN – Survive applica9on and system failures
• DURABILITY – Ability to handle redundant data source and failover
• OWNERSHIP – Reliable data transfer
• RELIABILITY
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 42
Manage network load & behaviour
• DDS provides mechanism to limit the data-‐rates – Filter by 9me – Filter by content – Par99ons – Use of mul9cast – Configured reliability level – TransportPriority QoS policy – LatencyBudget QoS policy
3/24/13 © 2012 RTI • COMPANY CONFIDENTIAL 43
Conclusions • DDS is a mature interna9onal Standard from OMG
– Plaborm Neutral: Opera9ng systems and Programming Languages
– Deployed worldwide in Military systems and other Demanding real-‐9me applica9ons
• DDS Is mandated by DoD for Publish-‐Subscribe and data-‐distribu9on applica9ons
• DDS is an ideal integra9on plaborm for Intelligent Systems
– Highly Tunable via Quality of Service (QoS) – Rich services (persistence, filtering, high-‐availability)
44 © 2012 RTI • ALL RIGHTS RESERVED
Find out more… www.r9.com
community.r9.com
demo.r9.com
www.youtube.com/real9meinnova9ons
blogs.r9.com
www.twi}er.com/RealTimeInnov
www.facebook.com/RTIsozware
www.slideshare.net/GerardoPardo
dds.omg.org
www.omg.org
© 2012 RTI • ALL RIGHTS RESERVED 45
Thank You!
46 © 2012 RTI • ALL RIGHTS RESERVED
dds.omg.org
www.omg.org