Extensible and Dynamic Topic Types For DDS (out of date)

Extensible and Dynamic Topic Types for DDS

MARS – San Antonio, TX – Sept., 2009

Rick Warren, RTI

[email protected]

document number: 2009-09-08

IntroductionDDS Types: Assessment and Challenges

Good

Unmatched type safety => fewer defects

Unique type awareness => faster integration

Unique type awareness => higher performance

Bad

No standard API to define / access types at runtime

Hard to change types without rebuilding, redeploying

Hard to upgrade systems one subsystem at a time

© 2009 RTI - All rights Reserved 2

Vendor-Specific Solutions

OMG-Standard Solutions

IntroductionScope

Improve Portability Clarify applicability of existing IDL type system

Codify existing DDS type system features, especially keys

Extend IDL to support new concepts

Improve Type System Expressiveness Support maps (i.e. associative collections)

Omit irrelevant fields, if desired

Make RTPS Discovery encapsulation first class, usable by any type/topic


IntroductionScope

Enable Agility Support type evolution, subsystem by subsystem

Send and receive data using types unknown at compile time

Define and modify types programmatically Discover remote type definitions Get and set data member values by name

Simplify Integration XML type definitions connect DDS to the XML-

speaking world

Built-in types help users start using DDS quickly


IntroductionOverview

1. Type System: abstract definition of what types can exist Expressed as UML meta-model Mostly familiar from IDL

IDL-compatible primitives Strings and aliases (typedefs) Arrays, sequences, and maps Structures (incl. aspects of valuetypes) and unions Enumerations and bit sets Annotations

2. Type Representations: languages for describing types IDL (with extensions) XML and XSD TypeCode



3. Data Representations: languages for describing samples CDR (with extensions)

XML (TBD)

4. Language Binding: programming APIs “Plain language”: extension of existing IDL-to-

language bindings

Dynamic: reflective API for types and samples, defined in UML (PIM) and IDL (PSM)

5. Use by DDS: defines how DDS uses all of the above Everything else is DDS-independent!

(so far, maybe only 90% independent)




pkg SubmissionOv erv iew

DataRepresentation LanguageBindingTypeRepresentation

Defines the type system for DDS Topic Types.The set of types that can be sent/received by DDS

Defines the way objects of a type are accessed from a programming language

Defines the ways objects can be serialized for networktransmission of storage on a stream (e.g. fi le)

Defines the way a type can be represented in a externalized form for networktransmission or storage on a stream (e.g. fi le)

TypeSystem


IntroductionOverview: Example

TypeRepresentation

LanguageBinding

DataRepresentation

IDL:Foo.idl

struct Foo { string name; long ssn;};

IDL to Language Mapping:Foo.hFoo.cFooTypeSupport.c

struct Foo { char *name; int ssn;};

Foo f = {"hello", 2};

IDL to CDR:

0000000668656C6C6F00000000000002

TypeSystem implicitly definedby the Type Representation

IntroductionTechnology Highlights

Dynamic Language Binding: dynamic types, dynamic data

XML Type Representations: XML and XSD

IDL Type Representation: …with extensions

CDR Data Representation: extensible binary data


Dynamic Types, Dynamic DataDynamic Types: Overview

Summary: Define and/or modify type definitions at runtime

Informed by CORBA TypeCode

Interoperable with statically defined types

Small number of fundamental classes: DynamicType: A programmatic type description

DynamicTypeFactory: Creates new types

DynamicTypeSupport: Registers types with DDS


Dynamic Types, Dynamic DataDynamic Types: Example #1

Create the equivalent:

struct MyType {long my_integer;

};

C++ code:DynamicTypeFactory* factory =DynamicTypeFactory::get_instance();

DynamicType* type = factory->create_type(TypeDescriptor("MyType", STRUCTURE_KIND));

type->add_member(MemberDescriptor(

"my_integer",factory->get_primitive_type(

INTEGER_KIND)));


Dynamic Types, Dynamic DataDynamic Types: Example #2

Subscribe to a type you’ve never seen before

Java code:DynamicType type =myPubBuiltinTopicData.type;

DynamicTypeSupport support =DynamicDataTypeFactory.get_instance().

create_type_support(type);support.register_type(myParticipant,myPubBuiltinTopicData.type_name);

// … init QoS based on built-in topic dataTopic topic = myParticipant.create_topic(myPubBuiltinTopicData.name,myPubBuiltinTopicData.type_name,myTopicQos, myTopicListener);

…© 2009 RTI - All rights Reserved 12

Dynamic Types, Dynamic DataDynamic Data: Overview

Summary: Introspect and modify samplesat runtime

Informed by CORBA DynAny, JMS MapMessage,TIBCO Rendezvous self-describing messages

Small number of fundamental classes: DynamicData: A sample of any (single) type

DynamicDataFactory: “Instantiates” a dynamic type

DynamicDataWriter: Writes samples of any (single) type, each represented by a DynamicData object

DynamicDataReader: Reads samples of any (single) type, each represented by a DynamicData object


Dynamic Types, Dynamic DataDynamic Data: Names and IDs

DynamicData gets / sets member valuesin 2 ways:

By name: Obvious, human-readable approach

By ID number: Potentially faster, more efficient search

Example in C++:DynamicData* sample = DynamicDataFactory::get_instance()->

create_data(my_type);

long my_value = 5;

sample->set_int("my_integer", // Specify name or ID;MEMBER_ID_ANY, // no need for both.my_value); // <-- Value to

set.© 2009 RTI - All rights Reserved 14

XML Type RepresentationsXML Type Representations

#1: XSD

Based on IDL-to-WSDL mapping

Allows type sharing between DDS and web services

But…

Very verbose

Hard to read, hard to write, hard to parse

#2: XML

Straightforward mapping of IDL into XML

Suitable for embedding into other XML documents (e.g. QoS profile files)

Easy to read, easy to write, easy to parse


XML Type RepresentationsXML Type Representations: Examples

#1: XSD<xsd:complexType name="T">

<xsd:sequence>

<xsd:element name="value" minOccurs="1" maxOccurs="1" type="char"/>

</xsd:sequence>

</xsd:complexType>

#2: XML

<struct name="T">

<member name="value" type="char"/>

</struct>


IDL Type RepresentationIDL Extension: Annotations

Summary: Bringing Java-style annotations to IDL Don’t just solve today’s problem;

…also provide an extensible tool we can reuse tomorrow

Example:

@AnAnnotationAppliedToTheType

struct TheType

@AnAnnotationAppliedToTheMember

long the_member;

};© 2009 RTI - All rights Reserved 17

IDL Type RepresentationIDL Extension: Annotations

Define annotations:

@Annotation local interface MyAnnotation {

long value() default 32;

};

* Why “@Annotation” instead of “@interface”? Coming up…

Apply annotations:

struct MyStruct {

@MyAnnotation long my_field;

};

Code gen translates to Java annotations one-to-one


IDL Type RepresentationIDL Extension: Compatible Annotation Syntax

Assumption: Some vendors will implement this spec sooner than others

Assumption: Users share IDL files among products from multiple vendors

Problem: How to take advantage of new features w/o breaking old IDL compilers with new syntax?

Solution: Put annotations in single-line “comment,” e.g.:

You can do this: @Key long my_key_field;

…or equivalently: long my_key_field; //@Key


IDL Type RepresentationIDL Extension: Built-in Annotations

@Key A member is part of its enclosing type’s key

@ID Specify the ID of a particular type member

If omitted, ID is assigned automatically

@Optional A type member may be omitted from data samples

@Extensible A type might change in the future,

…so DDS will use a more-verbose, more-flexible encapsulation


CDR Data RepresentationExtensible Binary Data: Challenge

Problem: CDR encoded data looks like this:

10110010001011100001110101000111

…So how do I know if I’ve added / removed a field?

Possible Solution #1: Include type definition with every sample

Uses too much bandwidth

Parsing is too slow

Possible Solution #2: Have DataWriter publish its type definition; DataReader uses that to parse samples

Parsing is still too slow


CDR Data RepresentationExtensible Binary Data: Solution

Solution: Use member IDs w/ RTPS param CDR

+----+----+----+----+----+----+----+----+

| ID (16) | size (16) |

+----+----+----+----+----+----+----+----+

| data (size) ... |

+----+----+----+----+----+----+----+----+

Benefits: Skip fields you don’t understand

Easily determine which fields are (not) present

Reorder fields

Brings RTPS simple discovery into type system mainstream

Cost: 4 bytes (+ alignment) per member© 2009 RTI - All rights Reserved 22

CDR Data RepresentationExtensible Binary Data: Details

Types marked as “extensible” encapsulated w/ parameterized CDR, all others w/ “plain” CDR

Extensible types can contain objects of non-extensible types and vice versa

100% compatible with existing types and serialization(…since built-in topic data are the only “extensible” types that exist today )

Lots of members in your type? Data sizes > 64 KB?

+----+----+----+----+----+----+----+----+

| ID sentinel (16) | size = 8 (16) |

+----+----+----+----+----+----+----+----+

| extended ID (32) |

+----+----+----+----+----+----+----+----+

| extended size (32) |

+----+----+----+----+----+----+----+----+

| data (extended size) ... |

+----+----+----+----+----+----+----+----+


CDR Data RepresentationExtensible Binary Data: Type Versioning

Problem: How can types change over time – and at different times from subsystem to subsystem?

Possible Solution: Give types version number, keep track of past versions and who is using which version.

Too slow: must consult per-remote-writer definition to deserialize

Requires tracking lots of state – including every version of every type that’s ever existed (e.g. for late joiners)

Requires network propagation of all type information: increased bandwidth use, possible security objections

Solution: Do nothing. Member IDs already provide compact per-member “version.”

Adding / changing a member? Define a new ID. Old applications can ignore it.

“Removing” a member? Just stop sending it. Doesn’t invalidate existing persisted data.


Summary: Specification Overview


pkg SubmissionOv erv iew

DataRepresentation LanguageBindingTypeRepresentation

Defines the type system for DDS Topic Types.The set of types that can be sent/received by DDS

Defines the way objects of a type are accessed from a programming language

Defines the ways objects can be serialized for networktransmission of storage on a stream (e.g. fi le)

Defines the way a type can be represented in a externalized form for networktransmission or storage on a stream (e.g. fi le)

TypeSystem

Summary: Goals and Benefits

Portable, consistent type system Standardized syntax for keys and other DDS concepts General-purpose extensibility (annotations) reusable by future

specifications, vendors, and users Same features usable by user-defined types as built-in types

Dynamic access to type definitions Introspect types for remote publications, subscriptions Define and modify types at runtime Publish and subscribe to dynamically defined types Interoperate with statically defined types and across vendors

Simplified integration of DDS into other systems XML and XSD type definitions to integrate w/ XML-centric systems Built-in types for simplified programming, especially for new users

Fast: little or no performance overhead in steady state


Summary: Progress

Many more details than previous submission Full specification of IDL extensions

Unification of CDR and parameterized CDR data representations

Concrete dynamic type, dynamic data APIs

Refinements to automatic type conversions

Some work remains XML data representation

Refinements to XML type representation grammars

Refinements to endpoint matching rules

Targeted for completion by Dec. 2009 meeting


Q & A
